A fast Ruby link checker with support for multiple HTTP libraries. Does not parse documents, just checks links. Fast. Anecdotal benchmarking on a M1 mac and T1 Internet yields ~50 URLs per second with LinkChecker::Typhoeus::Hydra.
The LinkChecker::Typhoeus::Hydra link checker is recommended.
Add typhoeus and ruby-link-checker to your Gemfile and run bundle install.
gem 'typhoeus'
gem 'ruby-link-checker'require 'typhoeus'
require 'ruby-link-checker'
# create a new checker instance
checker = LinkChecker::Typhoeus::Hydra::Checker.new
# queue URLs to check
links = [...]
links.each do |url|
checker.check url
end
# run the checks
checker.run
# display buckets of results
checker.results.each_pair do |bucket, results|
puts "#{bucket}: #{results.size}"
endYou can pipe custom options through check and retrieve them in events as follows.
checker.check 'https://www.example.org', { location: 'page.html' }
checker.on :success do |result|
result.options # contains { location: 'page.html' }
endFast link checker that uses Typhoeus.
require 'typhoeus'
require 'ruby-link-checker'
# create a new instance of a checker
checker = LinkChecker::Typhoeus::Hydra::Checker.new(
hydra: {
# lower than the Typhoeus default of 200, seems to start breaking around 50+
max_concurrency: 25
}
)
# log requests and response codes
checker.logger.level = Logger::INFO
links = [...] # array of URLs
links.each do |url|
checker.check url
end
# examine failures and errors as they come
checker.on :error, :failure do |result|
puts "FAIL: #{result}"
end
# execute Hydra#run, will block until all requests have completed
checker.run
# examine results
checker.results.each_pair do |bucket, results|
puts "#{bucket}: #{results.size}"
endYou can pass Typhoeus timeout options into a new instance of a checker, or configure timeouts globally.
LinkChecker::Typhoeus::Hydra.configure do |config|
config.timeout = 5
config.connecttimeout = 10
endFast, concurrent link checker that uses async-http.
require 'async'
require 'async/http'
require 'ruby-link-checker'
# create a new instance of a checker
checker = LinkChecker::Async::HTTP::Checker.new
# log requests and response codes
checker.logger.level = Logger::INFO
links = [...] # array of URLs
links.each do |url|
checker.check url
end
# examine failures and errors as they come
checker.on :error, :failure do |result|
puts "FAIL: #{result}"
end
# runs all queued checks concurrently, will block until all requests have completed
checker.run
# examine results
checker.results.each_pair do |bucket, results|
puts "#{bucket}: #{results.size}"
endYou can pass Async::HTTP timeout options into a new instance of a checker, or configure timeouts globally.
LinkChecker::Async::HTTP.configure do |config|
config.read_timeout = 5
config.open_timeout = 10
endYou can also force a specific HTTP protocol version, :http1 or :http2, per instance or globally. By default, async-http negotiates the protocol version with the server.
LinkChecker::Async::HTTP.configure do |config|
config.protocol = :http2
endAsync::HTTP reuses connection-pooling clients per scheme/host/port. Its default concurrency is unlimited; set max_concurrency to bound active requests without counting delayed retries against that limit.
checker = LinkChecker::Async::HTTP::Checker.new(max_concurrency: 25)Slow, sequential checker.
require 'net/http'
require 'ruby-link-checker'
# create a new instance of a checker
checker = LinkChecker::Net::HTTP::Checker.new
# log requests and response codes
checker.logger.level = Logger::INFO
links = [...] # array of URLs
links.each do |url|
checker.check url
end
# examine results
checker.results.each_pair do |bucket, results|
puts "#{bucket}: #{results.size}"
endYou can pass Net::HTTP timeout options into a new instance of a checker, or configure timeouts globally.
LinkChecker::Net::HTTP.configure do |config|
config.read_timeout = 5
config.open_timeout = 10
endBy default link checkers do not retry. Set retries to enable retries for transient connection errors/timeouts and HTTP statuses 408, 425, 429, 500, 502, 503, and 504. Permanent HTTP failures and invalid URLs are not retried. HEAD-to-GET fallback still applies independently of the retry policy. The retry budget is shared across methods: with retries: 2, repeated transient failures produce HEAD, HEAD, HEAD, then GET.
checker = LinkChecker::Net::HTTP::Checker.new(retries: 1)Checkers follow up to 20 redirects per attempt and detect redirect loops. Relative redirects are resolved against the current response URL. Set max_redirects to change the limit, or use 0 to disable redirects.
checker = LinkChecker::Typhoeus::Hydra::Checker.new(max_redirects: 10)Set retry_after to a minimum delay in seconds (default 0). Optional retry_backoff multiplies that delay on successive retries (default 1), and retry_jitter adds a random fraction between 0 and the configured value (default 0, maximum 1). All delays are capped by max_retry_delay (default 60 seconds).
checker = LinkChecker::Typhoeus::Hydra::Checker.new(
retries: 2,
retry_after: 1,
retry_backoff: 2,
retry_jitter: 0.1,
respect_retry_after: true,
max_retry_delay: 60
)With respect_retry_after: true (default false), HTTP Retry-After delta-seconds and dates provide a minimum delay, subject to the same cap. Invalid or past dates are ignored. Async delays yield without blocking other requests. Hydra queues delayed retries after the active batch drains, so actual delays can exceed the configured minimum without blocking completion callbacks. Delays apply to retries, not ordinary HEAD-to-GET fallback.
Customize retry_status_codes to choose retryable HTTP statuses. Set it to nil to restore the previous behavior of retrying every failure and error, including malformed URLs and programming errors.
checker = LinkChecker::Net::HTTP::Checker.new(retries: 1, retry_status_codes: [429, 503])By default checkers collect results.
checker = LinkChecker::Net::HTTP::Checker.new(results: false)
...
checker.run
checker.results # => { error: [...], failure: [...], success: [...] }You can disable this with results: false.
checker = LinkChecker::Net::HTTP::Checker.new(results: false)
...
checker.run
checker.results # => nilBy default checkers try a HEAD request, followed by a GET if HEAD fails. You can change this behavior by specifying other methods.
The following examples disables GET and only makes HEAD requests.
checker = LinkChecker::Net::HTTP::Checker.new(methods: %w[HEAD])Set deduplicate: true to coalesce identical in-flight requests and reuse successful results for the lifetime of the checker. Failures and errors are rechecked after completion. Each submission still receives its own result events and metadata, including events emitted before it joined an in-flight check. Deduplication is disabled by default.
checker = LinkChecker::Typhoeus::Hydra::Checker.new(deduplicate: true)
checker.check('https://www.example.org', location: 'one.html')
checker.check('https://www.example.org', location: 'two.html')
checker.run # one check, two sets of result eventsIdentity includes the complete URL (including query values and fragment), transport, methods, headers, user-agent, and request-related settings. Changing those settings prevents reuse. URLs containing user-info credentials bypass deduplication and caching.
Persistent success caching is disabled by default. Enable it with an explicit file path and positive TTL in seconds. Recent successful checks can then be reused across checker instances; failures and errors are never stored. Cache writes are atomic and locked, and unreadable or malformed files fall back to normal checking.
checker = LinkChecker::Typhoeus::Hydra::Checker.new(
deduplicate: true,
cache: { file: 'tmp/link-checker/cache.json', ttl: 3600 }
)Each cache hit emits :result and :success with the new submission's metadata and cached? == true. Persistent hits return LinkChecker::CachedResult with the HTTP method, status, original/final URIs, and metadata; native request/response objects and headers are not persisted. Redirect targets containing user-info credentials are not written. Add the cache directory to your ignore rules.
Caching alone does not coalesce in-flight checks. With deduplication enabled, successful results remain reusable within that checker even after the persistent TTL expires. Cached successes do not replay historical redirect/retry events.
Pass your own logger.
checker = LinkChecker::Net::HTTP::Checker.new(logger: Logger.new(STDOUT))Pass your own user-agent. Default is Ruby Link Checker/x.y.z.
checker = LinkChecker::Net::HTTP::Checker.new(user_agent: 'Custom Agent/1.0')Some sites reject or return errors (403, 404) to requests that look like bots. Pass additional headers to mimic a browser.
checker = LinkChecker::Net::HTTP::Checker.new(
headers: {
'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language' => 'en-US,en;q=0.5',
'DNT' => '1'
}
)All options can also be configured globally.
LinkChecker.configure do |config|
config.user_agent = 'Custom Agent/1.0'
config.headers = { 'DNT' => '1' }
config.methods = ['HEAD', 'GET']
config.logger = ::Logger.new(STDOUT)
endEvents enable processing of results as they become available.
checker.on :result do |result|
puts result # any result
end
checker.on :error, :failure do |result|
puts result # error or failure
endCheckers support the following events.
| Event | Description |
|---|---|
| :retry | A request is being retried on failure or error. |
| :result | A new result, any of success, failure, or error. |
| :success | A valid URL, usually a 2xx response from the server. |
| :failure | A failed URL, usually a 4xx or a 5xx response from the server. |
| :error | An error, such as an invalid URL or a network timeout. |
Events are called with results, which contain the following properties.
| Property | Description |
|---|---|
| :url | The original URL before redirects. |
| :result_url | The last URL, different from url in case of redirects. |
| :method | The result HTTP method. |
| :code | HTTP error code. |
| :request_headers | Request headers. |
| :redirect_to | A redirect URL in case of redirects. |
| :error | A raised error in case of errors. |
| :error_type | :timeout, :connection, or :other for errors; otherwise nil. |
| :response_headers | Response headers, accessible using []. |
| :cached? | Whether this result was reused from a successful check. |
See result.rb for more details.
See benchmarks/check.rb for a script comparing the performance of the three checkers (LinkChecker::Net::HTTP, LinkChecker::Typhoeus::Hydra, and LinkChecker::Async::HTTP) against a fixed set of URLs.
bundle exec ruby -Ilib benchmarks/check.rb
Anecdotal results show LinkChecker::Async::HTTP using significantly less CPU time than LinkChecker::Typhoeus::Hydra (both concurrent checkers), while wall-clock time is comparable between the two and depends on timeouts and the responsiveness of the URLs being checked. LinkChecker::Net::HTTP checks links sequentially and is significantly slower in wall-clock time for large batches of URLs; it's commented out by default in the script.
You're encouraged to contribute to ruby-link-checker. See CONTRIBUTING for details.
Copyright (c) Daniel Doubrovkine and Contributors.
This project is licensed under the MIT License.