diff --git a/CHANGELOG.md b/CHANGELOG.md index f787a20..2f8101d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ * [#23](https://github.com/dblock/ruby-link-checker/pull/23): Fixed chained relative redirects, bounded redirect following with `max_redirects`, enabled plain HTTP in the Net::HTTP checker, and corrected retry documentation - [@dblock](https://github.com/dblock), [@Copilot](https://github.com/Copilot). * [#24](https://github.com/dblock/ruby-link-checker/pull/24): Added transient-only retries, opt-in retry delays/backoff/jitter and `Retry-After` handling, transport error categories, and an Async concurrency limit - [@dblock](https://github.com/dblock), [@Copilot](https://github.com/Copilot). +* [#25](https://github.com/dblock/ruby-link-checker/pull/25): Added opt-in URL deduplication and atomic TTL-based success caching, preserving per-submission metadata and query values - [@dblock](https://github.com/dblock), [@Copilot](https://github.com/Copilot). * [#8](https://github.com/dblock/ruby-link-checker/issues/8): Added `protocol` configuration option to `LinkChecker::Async::HTTP` to force HTTP/1 or HTTP/2 - [@dblock](https://github.com/dblock), [@Copilot](https://github.com/Copilot). * Your contribution here. diff --git a/README.md b/README.md index d0df735..74785e1 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,8 @@ A fast Ruby link checker with support for multiple HTTP libraries. Does not pars - [Retry Policy](#retry-policy) - [Results](#results) - [Methods](#methods) + - [Deduplication](#deduplication) + - [Caching](#caching) - [Logger](#logger) - [User-Agent](#user-agent) - [Headers](#headers) @@ -297,6 +299,34 @@ The following examples disables `GET` and only makes `HEAD` requests. checker = LinkChecker::Net::HTTP::Checker.new(methods: %w[HEAD]) ``` +#### Deduplication + +Set `deduplicate: true` to coalesce identical in-flight requests and reuse successful results for the lifetime of the checker. Failures and errors are rechecked after completion. Each submission still receives its own result events and metadata, including events emitted before it joined an in-flight check. Deduplication is disabled by default. + +```ruby +checker = LinkChecker::Typhoeus::Hydra::Checker.new(deduplicate: true) +checker.check('https://www.example.org', location: 'one.html') +checker.check('https://www.example.org', location: 'two.html') +checker.run # one check, two sets of result events +``` + +Identity includes the complete URL (including query values and fragment), transport, methods, headers, user-agent, and request-related settings. Changing those settings prevents reuse. URLs containing user-info credentials bypass deduplication and caching. + +#### Caching + +Persistent success caching is disabled by default. Enable it with an explicit file path and positive TTL in seconds. Recent successful checks can then be reused across checker instances; failures and errors are never stored. Cache writes are atomic and locked, and unreadable or malformed files fall back to normal checking. + +```ruby +checker = LinkChecker::Typhoeus::Hydra::Checker.new( + deduplicate: true, + cache: { file: 'tmp/link-checker/cache.json', ttl: 3600 } +) +``` + +Each cache hit emits `:result` and `:success` with the new submission's metadata and `cached? == true`. Persistent hits return `LinkChecker::CachedResult` with the HTTP method, status, original/final URIs, and metadata; native request/response objects and headers are not persisted. Redirect targets containing user-info credentials are not written. Add the cache directory to your ignore rules. + +Caching alone does not coalesce in-flight checks. With deduplication enabled, successful results remain reusable within that checker even after the persistent TTL expires. Cached successes do not replay historical redirect/retry events. + #### Logger Pass your own logger. @@ -377,6 +407,7 @@ Events are called with results, which contain the following properties. | :error | A raised error in case of errors. | | :error_type | `:timeout`, `:connection`, or `:other` for errors; otherwise `nil`. | | :response_headers | Response headers, accessible using `[]`. | +| :cached? | Whether this result was reused from a successful check. | See [result.rb](lib/ruby-link-checker/result.rb) for more details. diff --git a/lib/ruby-link-checker.rb b/lib/ruby-link-checker.rb index e6cf1ca..ce08bc2 100644 --- a/lib/ruby-link-checker.rb +++ b/lib/ruby-link-checker.rb @@ -9,6 +9,7 @@ require_relative 'ruby-link-checker/tasks' require_relative 'ruby-link-checker/checker' require_relative 'ruby-link-checker/result' +require_relative 'ruby-link-checker/cache' require_relative 'ruby-link-checker/net/http' require_relative 'ruby-link-checker/typhoeus/hydra' require_relative 'ruby-link-checker/async/http' diff --git a/lib/ruby-link-checker/cache.rb b/lib/ruby-link-checker/cache.rb new file mode 100644 index 0000000..2ec983d --- /dev/null +++ b/lib/ruby-link-checker/cache.rb @@ -0,0 +1,101 @@ +# frozen_string_literal: true + +require 'digest' +require 'fileutils' +require 'json' +require 'tempfile' +require 'uri' + +module LinkChecker + class CachedResult < Result + attr_reader :code + + def initialize(data, uri) + @code = data.fetch('code') + super(URI(data.fetch('result_uri')), data.fetch('method'), URI(uri.to_s)) + @cached = true + end + + def success? + true + end + end + + class Cache + VERSION = 1 + + def initialize(options, logger:) + unless options.is_a?(Hash) && options[:file].is_a?(String) && !options[:file].strip.empty? + raise ArgumentError, 'Cache requires a file path' + end + + @ttl = options[:ttl] + unless @ttl.is_a?(Numeric) && @ttl.real? && @ttl.finite? && @ttl.positive? + raise ArgumentError, 'Cache requires a positive, finite TTL in seconds' + end + + @file = File.expand_path(options[:file]) + @logger = logger + end + + def fetch(key, uri) + data = read_entries[key] + return unless fresh?(data) && data['code'].is_a?(Integer) && data['code'].between?(200, 299) + return unless data['method'].is_a?(String) && data['result_uri'].is_a?(String) + + parsed = URI(data['result_uri']) + return unless %w[http https].include?(parsed.scheme) && parsed.host && !parsed.userinfo + + CachedResult.new(data, uri) + rescue URI::InvalidURIError + nil + end + + def store(key, result) + return unless result.success? + return if URI(result.result_uri.to_s).userinfo + + directory = File.dirname(@file) + FileUtils.mkdir_p(directory) + File.open("#{@file}.lock", File::RDWR | File::CREAT, 0o600) do |lock| + lock.flock(File::LOCK_EX) + entries = read_entries.select { |_entry_key, data| fresh?(data) } + entries[key] = { + 'checked_at' => Time.now.to_f, + 'method' => result.method, + 'code' => result.code, + 'result_uri' => result.result_uri.to_s + } + Tempfile.create(['link-checker-', '.json'], directory) do |file| + file.write(JSON.generate('version' => VERSION, 'entries' => entries)) + file.flush + File.rename(file.path, @file) + end + end + rescue IOError, SystemCallError, JSON::GeneratorError + @logger.warn("Unable to write link cache #{@file}; result was not cached") + end + + private + + def fresh?(data) + return false unless data.is_a?(Hash) + + timestamp = data['checked_at'] + now = Time.now.to_f + timestamp.is_a?(Numeric) && timestamp.finite? && timestamp <= now && now - timestamp < @ttl + end + + def read_entries + data = JSON.parse(File.read(@file)) + return {} unless data.is_a?(Hash) && data['version'] == VERSION && data['entries'].is_a?(Hash) + + data['entries'] + rescue Errno::ENOENT + {} + rescue IOError, SystemCallError, JSON::ParserError + @logger.warn("Unable to read link cache #{@file}; checking links normally") + {} + end + end +end diff --git a/lib/ruby-link-checker/checker.rb b/lib/ruby-link-checker/checker.rb index 129e0bd..7cfa19a 100644 --- a/lib/ruby-link-checker/checker.rb +++ b/lib/ruby-link-checker/checker.rb @@ -14,6 +14,9 @@ def initialize(options = {}) raise ArgumentError, "Missing methods." if methods&.none? @logger ||= options[:logger] || LinkChecker::Config.logger || LinkChecker::Logger.default @results = { error: [], failure: [], success: [] } unless options.key?(:results) && !options[:results] + @pending = {} + @successful = {} + @cache_store = Cache.new(cache, logger: @logger) if cache end def task_klass @@ -24,6 +27,37 @@ def task_klass end def check(uri, options = {}) + return execute_check(uri, options) unless deduplicate || @cache_store + + key = request_key(uri) + return execute_check(uri, options) unless key + + subscriber = [uri.to_s.dup, options] + if deduplicate && (entry = @pending[key]) + entry[:subscribers] << subscriber + entry[:events].each { |event, result| publish(event, copy_result(result, subscriber)) } + return + end + + cached_result = @successful[key] if deduplicate + cached_result ||= @cache_store&.fetch(key, uri) + if cached_result + @successful[key] ||= cached_result if deduplicate + result = copy_result(cached_result, subscriber) + result.cached = true + publish(:result, result) + publish(:success, result) + return + end + + entry = { subscribers: [subscriber], events: [] } + @pending[key] = entry if deduplicate + execute_check(uri, options) { |event, result| handle_shared_result(key, entry, uri, event, result) } + end + + private + + def execute_check(uri, options, &handler) tasks = Tasks.new( self, task_klass, @@ -32,10 +66,49 @@ def check(uri, options = {}) options ) tasks.on do |event, *args| - results[event] << args.first if @results && %i[error failure success].include?(event) - callback event, *args + handler ? handler.call(event, *args) : publish(event, args.first) end tasks.execute! end + + def publish(event, result) + result.checker = self + results[event] << result if @results && %i[error failure success].include?(event) + callback(event, result) + end + + def handle_shared_result(key, entry, uri, event, result) + entry[:events] << [event, result] + subscribers = entry[:subscribers].dup + if %i[error failure success].include?(event) + @pending.delete(key) if @pending[key].equal?(entry) + if event == :success && request_key(uri) == key + @successful[key] = result if deduplicate + @cache_store&.store(key, result) + end + end + subscribers.each { |subscriber| publish(event, copy_result(result, subscriber)) } + end + + def copy_result(result, subscriber) + result.dup.tap do |copy| + copy.uri = URI(subscriber.first) + copy.result_uri = result.result_uri.dup + copy.options = subscriber.last + end + end + + def request_key(uri) + parsed = URI(uri.to_s) + return unless %w[http https].include?(parsed.scheme) && parsed.host && !parsed.userinfo + + attributes = Config::ATTRIBUTES - %i[logger cache deduplicate] + attributes += %i[timeout connecttimeout read_timeout open_timeout protocol].select { |key| respond_to?(key) } + settings = attributes.to_h { |key| [key, send(key)] } + settings[:headers] = headers.sort_by { |key, _value| key.to_s } + Digest::SHA256.hexdigest(JSON.generate([task_klass.name, parsed.to_s, settings])) + rescue URI::InvalidURIError, JSON::GeneratorError, TypeError + nil + end end end diff --git a/lib/ruby-link-checker/config.rb b/lib/ruby-link-checker/config.rb index 20eba0c..73ed31d 100644 --- a/lib/ruby-link-checker/config.rb +++ b/lib/ruby-link-checker/config.rb @@ -17,6 +17,8 @@ module Config retry_jitter max_retry_delay respect_retry_after + deduplicate + cache ].freeze attr_accessor(*Config::ATTRIBUTES) @@ -34,6 +36,8 @@ def reset self.retry_jitter = 0 self.max_retry_delay = 60 self.respect_retry_after = false + self.deduplicate = false + self.cache = nil end def retries=(value) diff --git a/lib/ruby-link-checker/result.rb b/lib/ruby-link-checker/result.rb index 8bfa082..ce1541b 100644 --- a/lib/ruby-link-checker/result.rb +++ b/lib/ruby-link-checker/result.rb @@ -4,6 +4,7 @@ module LinkChecker class Result attr_accessor :uri, :result_uri, :method, :options, :checker + attr_writer :cached def initialize(current_uri, method, original_uri, options = {}) @uri = original_uri @@ -16,6 +17,10 @@ def success? false end + def cached? + !!@cached + end + def failure? false end diff --git a/spec/ruby-link-checker/checker_spec.rb b/spec/ruby-link-checker/checker_spec.rb index c759cc3..5eb8e67 100644 --- a/spec/ruby-link-checker/checker_spec.rb +++ b/spec/ruby-link-checker/checker_spec.rb @@ -2,9 +2,76 @@ require 'spec_helper' describe LinkChecker::Checker do - context 'config' do + describe LinkChecker::Cache do + let(:directory) { Dir.mktmpdir('link-checker-spec-') } + let(:file) { File.join(directory, 'cache.json') } + let(:cache) { described_class.new({ file: file, ttl: 60 }, logger: LinkChecker::Logger.default) } + let(:result) do + LinkChecker::Net::HTTP::Result.new( + URI('https://www.example.org/final'), 'GET', URI('https://www.example.org/'), + {}, Net::HTTPOK.new('1.1', '200', 'OK'), { location: 'private.html' } + ) + end + + after do + FileUtils.remove_entry(directory) + end + + it 'expires successful entries using the configured TTL' do + cache.store('key', result) + allow(Time).to receive(:now).and_return(Time.now + 61) + expect(cache.fetch('key', 'https://www.example.org/')).to be_nil + end + + it 'retains the final URL and status without persisting metadata or headers' do + cache.store('key', result) + restored = cache.fetch('key', 'https://www.example.org/') + expect(restored.result_uri.to_s).to eq('https://www.example.org/final') + expect(restored.code).to eq(200) + data = JSON.parse(File.read(file)).fetch('entries').fetch('key') + expect(data.keys).to contain_exactly('checked_at', 'method', 'code', 'result_uri') + end + + it 'merges writes from separate cache instances' do + other_cache = described_class.new({ file: file, ttl: 60 }, logger: LinkChecker::Logger.default) + cache.store('one', result) + other_cache.store('two', result) + expect(cache.fetch('one', 'https://www.example.org/')).not_to be_nil + expect(cache.fetch('two', 'https://www.example.org/')).not_to be_nil + end + + it 'recovers from corrupted cache files' do + File.write(file, 'invalid JSON') + expect(cache.fetch('key', 'https://www.example.org/')).to be_nil + cache.store('key', result) + expect(cache.fetch('key', 'https://www.example.org/').code).to eq(200) + end + + it 'ignores malformed cache entries' do + File.write(file, JSON.generate('version' => 1, 'entries' => { 'key' => true })) + expect(cache.fetch('key', 'https://www.example.org/')).to be_nil + end + + it 'does not store credentials in redirect URLs' do + result.result_uri = URI('https://user:password@www.example.org/') + cache.store('key', result) + expect(File).not_to exist(file) + end + + it 'requires an explicit file and TTL' do + expect { described_class.new({}, logger: LinkChecker::Logger.default) }.to raise_error ArgumentError + expect { described_class.new({ file: file }, logger: LinkChecker::Logger.default) }.to raise_error ArgumentError + end + + it 'does not turn a cache write failure into a failed link' do + allow(FileUtils).to receive(:mkdir_p).and_raise(Errno::EACCES) + expect { cache.store('key', result) }.not_to raise_error + end + end + + context 'config' do it 'requires at least one method' do - expect { LinkChecker::Checker.new(methods: []) }.to raise_error ArgumentError, 'Missing methods.' + expect { described_class.new(methods: []) }.to raise_error ArgumentError, 'Missing methods.' end it 'rejects an invalid redirect limit' do diff --git a/spec/ruby-link-checker/typhoeus/hydra/checker_spec.rb b/spec/ruby-link-checker/typhoeus/hydra/checker_spec.rb index 3f33233..37da24a 100644 --- a/spec/ruby-link-checker/typhoeus/hydra/checker_spec.rb +++ b/spec/ruby-link-checker/typhoeus/hydra/checker_spec.rb @@ -137,4 +137,44 @@ def check(url, options = {}) expect(checker.results[:error].first.error_type).to eq(:connection) end end + + describe 'in-flight deduplication' do + let(:checker) { described_class.new(methods: ['GET'], deduplicate: true) } + let(:url) { 'https://www.example.org/' } + + around do |example| + VCR.turned_off { example.run } + end + + it 'fans out events to duplicates queued before completion' do + metadata = [] + stub_request(:any, url).to_return(status: 200) + checker.on(:result) { |result| metadata << result.options } + checker.check(url, location: 'one.html') + checker.check(url, location: 'two.html') + checker.run + expect(a_request(:any, url)).to have_been_made.once + expect(metadata).to eq([{ location: 'one.html' }, { location: 'two.html' }]) + end + + it 'replays events when a subscriber joins during a result callback' do + stub_request(:any, url).to_return(status: 200) + checker.on(:result) { |result| checker.check(url, location: 'two.html') if result.options[:location] == 'one.html' } + checker.check(url, location: 'one.html') + checker.run + expect(checker.results[:success].map(&:options)).to eq([{ location: 'one.html' }, { location: 'two.html' }]) + expect(a_request(:any, url)).to have_been_made.once + end + + it 'does not memoize a response if settings changed while queued' do + stub_request(:any, url).to_return(status: 200) + checker.check(url) + checker.headers = { 'Accept' => 'text/html' } + checker.run + checker.headers = {} + checker.check(url) + checker.run + expect(a_request(:any, url)).to have_been_made.twice + end + end end diff --git a/spec/support/link_checker.rb b/spec/support/link_checker.rb index 6cded56..d99b705 100644 --- a/spec/support/link_checker.rb +++ b/spec/support/link_checker.rb @@ -137,6 +137,108 @@ end end + context 'with deduplication' do + let(:checker) { described_class.new(methods: ['GET'], deduplicate: true) } + let(:url) { 'https://www.example.org/' } + + around do |example| + VCR.turned_off { example.run } + end + + it 'checks once and preserves metadata for every submission' do + stub_request(:any, url).to_return(status: 200) + checker.check(url, location: 'one.html') + checker.check(url, location: 'two.html') + checker.run if checker.respond_to?(:run) + expect(a_request(:any, url)).to have_been_made.once + expect(checker.results[:success].map(&:options)).to eq([{ location: 'one.html' }, { location: 'two.html' }]) + expect(checker.results[:success].map(&:uri)).to eq([URI(url), URI(url)]) + end + + it 'rechecks failures instead of retaining them' do + stub_request(:any, url).to_return({ status: 404 }, { status: 200 }) + checker.check(url) + checker.run if checker.respond_to?(:run) + checker.check(url) + checker.run if checker.respond_to?(:run) + expect(a_request(:any, url)).to have_been_made.twice + expect(checker.results[:failure].size).to eq(1) + expect(checker.results[:success].size).to eq(1) + end + + it 'keeps query values distinct' do + stub_request(:any, "#{url}?id=1").to_return(status: 200) + stub_request(:any, "#{url}?id=2").to_return(status: 404) + checker.check("#{url}?id=1") + checker.check("#{url}?id=2") + checker.run if checker.respond_to?(:run) + expect(checker.results[:success].first.uri.query).to eq('id=1') + expect(checker.results[:failure].first.uri.query).to eq('id=2') + end + + it 'includes headers in request identity' do + stub_request(:any, url).to_return(status: 200) + checker.check(url) + checker.run if checker.respond_to?(:run) + checker.headers = { 'Accept' => 'text/html' } + checker.check(url) + checker.run if checker.respond_to?(:run) + expect(a_request(:any, url)).to have_been_made.twice + end + + context 'without result collection' do + let(:checker) { described_class.new(methods: ['GET'], deduplicate: true, results: false) } + + it 'still delivers results to every subscriber' do + completed = [] + stub_request(:any, url).to_return(status: 200) + checker.on(:success) { |result| completed << result.options } + checker.check(url, location: 'one.html') + checker.check(url, location: 'two.html') + checker.run if checker.respond_to?(:run) + expect(completed).to eq([{ location: 'one.html' }, { location: 'two.html' }]) + expect(checker.results).to be_nil + end + end + end + + context 'with persistent caching' do + let(:directory) { Dir.mktmpdir('link-checker-spec-') } + let(:cache_options) { { file: File.join(directory, 'cache.json'), ttl: 60 } } + let(:checker) { described_class.new(methods: ['GET'], cache: cache_options) } + let(:next_checker) { described_class.new(methods: ['GET'], cache: cache_options) } + let(:url) { 'https://www.example.org/' } + + around do |example| + VCR.turned_off { example.run } + end + + after do + FileUtils.remove_entry(directory) + end + + it 'reuses a success across checker instances with fresh metadata' do + stub_request(:any, url).to_return(status: 200) + checker.check(url, location: 'one.html') + checker.run if checker.respond_to?(:run) + next_checker.check(url, location: 'two.html') + next_checker.run if next_checker.respond_to?(:run) + expect(next_checker.results[:success].first.cached?).to be true + expect(next_checker.results[:success].first.options).to eq(location: 'two.html') + expect(a_request(:any, url)).to have_been_made.once + end + + it 'does not persist failed checks' do + stub_request(:any, url).to_return({ status: 404 }, { status: 200 }) + checker.check(url) + checker.run if checker.respond_to?(:run) + next_checker.check(url) + next_checker.run if next_checker.respond_to?(:run) + expect(next_checker.results[:success].first.cached?).to be false + expect(a_request(:any, url)).to have_been_made.twice + end + end + context 'default' do subject do described_class.new