Add opt-in URL deduplication and persistent success caching

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 * [#23](https://github.com/dblock/ruby-link-checker/pull/23): Fixed chained relative redirects, bounded redirect following with `max_redirects`, enabled plain HTTP in the Net::HTTP checker, and corrected retry documentation - [@dblock](https://github.com/dblock), [@Copilot](https://github.com/Copilot).
 * [#24](https://github.com/dblock/ruby-link-checker/pull/24): Added transient-only retries, opt-in retry delays/backoff/jitter and `Retry-After` handling, transport error categories, and an Async concurrency limit - [@dblock](https://github.com/dblock), [@Copilot](https://github.com/Copilot).
+* [#25](https://github.com/dblock/ruby-link-checker/pull/25): Added opt-in URL deduplication and atomic TTL-based success caching, preserving per-submission metadata and query values - [@dblock](https://github.com/dblock), [@Copilot](https://github.com/Copilot).
 * [#8](https://github.com/dblock/ruby-link-checker/issues/8): Added `protocol` configuration option to `LinkChecker::Async::HTTP` to force HTTP/1 or HTTP/2 - [@dblock](https://github.com/dblock), [@Copilot](https://github.com/Copilot).
 * Your contribution here.
 

README.md

@@ -24,6 +24,8 @@ A fast Ruby link checker with support for multiple HTTP libraries. Does not pars
     - [Retry Policy](#retry-policy)
     - [Results](#results)
     - [Methods](#methods)
+    - [Deduplication](#deduplication)
+    - [Caching](#caching)
     - [Logger](#logger)
     - [User-Agent](#user-agent)
     - [Headers](#headers)
@@ -297,6 +299,34 @@ The following examples disables `GET` and only makes `HEAD` requests.
 checker = LinkChecker::Net::HTTP::Checker.new(methods: %w[HEAD])
 ```
 
+#### Deduplication
+
+Set `deduplicate: true` to coalesce identical in-flight requests and reuse successful results for the lifetime of the checker. Failures and errors are rechecked after completion. Each submission still receives its own result events and metadata, including events emitted before it joined an in-flight check. Deduplication is disabled by default.
+
+```ruby
+checker = LinkChecker::Typhoeus::Hydra::Checker.new(deduplicate: true)
+checker.check('https://www.example.org', location: 'one.html')
+checker.check('https://www.example.org', location: 'two.html')
+checker.run # one check, two sets of result events
+```
+
+Identity includes the complete URL (including query values and fragment), transport, methods, headers, user-agent, and request-related settings. Changing those settings prevents reuse. URLs containing user-info credentials bypass deduplication and caching.
+
+#### Caching
+
+Persistent success caching is disabled by default. Enable it with an explicit file path and positive TTL in seconds. Recent successful checks can then be reused across checker instances; failures and errors are never stored. Cache writes are atomic and locked, and unreadable or malformed files fall back to normal checking.
+
+```ruby
+checker = LinkChecker::Typhoeus::Hydra::Checker.new(
+  deduplicate: true,
+  cache: { file: 'tmp/link-checker/cache.json', ttl: 3600 }
+)
+```
+
+Each cache hit emits `:result` and `:success` with the new submission's metadata and `cached? == true`. Persistent hits return `LinkChecker::CachedResult` with the HTTP method, status, original/final URIs, and metadata; native request/response objects and headers are not persisted. Redirect targets containing user-info credentials are not written. Add the cache directory to your ignore rules.
+
+Caching alone does not coalesce in-flight checks. With deduplication enabled, successful results remain reusable within that checker even after the persistent TTL expires. Cached successes do not replay historical redirect/retry events.
+
 #### Logger
 
 Pass your own logger.
@@ -377,6 +407,7 @@ Events are called with results, which contain the following properties.
 | :error            | A raised error in case of errors.                               |
 | :error_type       | `:timeout`, `:connection`, or `:other` for errors; otherwise `nil`. |
 | :response_headers | Response headers, accessible using `[]`.                        |
+| :cached?          | Whether this result was reused from a successful check.         |
 
 See [result.rb](lib/ruby-link-checker/result.rb) for more details.
 

lib/ruby-link-checker.rb

@@ -9,6 +9,7 @@
 require_relative 'ruby-link-checker/tasks'
 require_relative 'ruby-link-checker/checker'
 require_relative 'ruby-link-checker/result'
+require_relative 'ruby-link-checker/cache'
 require_relative 'ruby-link-checker/net/http'
 require_relative 'ruby-link-checker/typhoeus/hydra'
 require_relative 'ruby-link-checker/async/http'

lib/ruby-link-checker/cache.rb

@@ -0,0 +1,101 @@
+# frozen_string_literal: true
+
+require 'digest'
+require 'fileutils'
+require 'json'
+require 'tempfile'
+require 'uri'
+
+module LinkChecker
+  class CachedResult < Result
+    attr_reader :code
+
+    def initialize(data, uri)
+      @code = data.fetch('code')
+      super(URI(data.fetch('result_uri')), data.fetch('method'), URI(uri.to_s))
+      @cached = true
+    end
+
+    def success?
+      true
+    end
+  end
+
+  class Cache
+    VERSION = 1
+
+    def initialize(options, logger:)
+      unless options.is_a?(Hash) && options[:file].is_a?(String) && !options[:file].strip.empty?
+        raise ArgumentError, 'Cache requires a file path'
+      end
+
+      @ttl = options[:ttl]
+      unless @ttl.is_a?(Numeric) && @ttl.real? && @ttl.finite? && @ttl.positive?
+        raise ArgumentError, 'Cache requires a positive, finite TTL in seconds'
+      end
+
+      @file = File.expand_path(options[:file])
+      @logger = logger
+    end
+
+    def fetch(key, uri)
+      data = read_entries[key]
+      return unless fresh?(data) && data['code'].is_a?(Integer) && data['code'].between?(200, 299)
+      return unless data['method'].is_a?(String) && data['result_uri'].is_a?(String)
+
+      parsed = URI(data['result_uri'])
+      return unless %w[http https].include?(parsed.scheme) && parsed.host && !parsed.userinfo
+
+      CachedResult.new(data, uri)
+    rescue URI::InvalidURIError
+      nil
+    end
+
+    def store(key, result)
+      return unless result.success?
+      return if URI(result.result_uri.to_s).userinfo
+
+      directory = File.dirname(@file)
+      FileUtils.mkdir_p(directory)
+      File.open("#{@file}.lock", File::RDWR | File::CREAT, 0o600) do |lock|
+        lock.flock(File::LOCK_EX)
+        entries = read_entries.select { |_entry_key, data| fresh?(data) }
+        entries[key] = {
+          'checked_at' => Time.now.to_f,
+          'method' => result.method,
+          'code' => result.code,
+          'result_uri' => result.result_uri.to_s
+        }
+        Tempfile.create(['link-checker-', '.json'], directory) do |file|
+          file.write(JSON.generate('version' => VERSION, 'entries' => entries))
+          file.flush
+          File.rename(file.path, @file)
+        end
+      end
+    rescue IOError, SystemCallError, JSON::GeneratorError
+      @logger.warn("Unable to write link cache #{@file}; result was not cached")
+    end
+
+    private
+
+    def fresh?(data)
+      return false unless data.is_a?(Hash)
+
+      timestamp = data['checked_at']
+      now = Time.now.to_f
+      timestamp.is_a?(Numeric) && timestamp.finite? && timestamp <= now && now - timestamp < @ttl
+    end
+
+    def read_entries
+      data = JSON.parse(File.read(@file))
+      return {} unless data.is_a?(Hash) && data['version'] == VERSION && data['entries'].is_a?(Hash)
+
+      data['entries']
+    rescue Errno::ENOENT
+      {}
+    rescue IOError, SystemCallError, JSON::ParserError
+      @logger.warn("Unable to read link cache #{@file}; checking links normally")
+      {}
+    end
+  end
+end

lib/ruby-link-checker/checker.rb

@@ -14,6 +14,9 @@ def initialize(options = {})
       raise ArgumentError, "Missing methods." if methods&.none?
       @logger ||= options[:logger] || LinkChecker::Config.logger || LinkChecker::Logger.default
       @results = { error: [], failure: [], success: [] } unless options.key?(:results) && !options[:results]
+      @pending = {}
+      @successful = {}
+      @cache_store = Cache.new(cache, logger: @logger) if cache
     end
 
     def task_klass
@@ -24,6 +27,37 @@ def task_klass
     end
 
     def check(uri, options = {})
+      return execute_check(uri, options) unless deduplicate || @cache_store
+
+      key = request_key(uri)
+      return execute_check(uri, options) unless key
+
+      subscriber = [uri.to_s.dup, options]
+      if deduplicate && (entry = @pending[key])
+        entry[:subscribers] << subscriber
+        entry[:events].each { |event, result| publish(event, copy_result(result, subscriber)) }
+        return
+      end
+
+      cached_result = @successful[key] if deduplicate
+      cached_result ||= @cache_store&.fetch(key, uri)
+      if cached_result
+        @successful[key] ||= cached_result if deduplicate
+        result = copy_result(cached_result, subscriber)
+        result.cached = true
+        publish(:result, result)
+        publish(:success, result)
+        return
+      end
+
+      entry = { subscribers: [subscriber], events: [] }
+      @pending[key] = entry if deduplicate
+      execute_check(uri, options) { |event, result| handle_shared_result(key, entry, uri, event, result) }
+    end
+
+    private
+
+    def execute_check(uri, options, &handler)
       tasks = Tasks.new(
         self,
         task_klass,
@@ -32,10 +66,49 @@ def check(uri, options = {})
         options
       )
       tasks.on do |event, *args|
-        results[event] << args.first if @results && %i[error failure success].include?(event)
-        callback event, *args
+        handler ? handler.call(event, *args) : publish(event, args.first)
       end
       tasks.execute!
     end
+
+    def publish(event, result)
+      result.checker = self
+      results[event] << result if @results && %i[error failure success].include?(event)
+      callback(event, result)
+    end
+
+    def handle_shared_result(key, entry, uri, event, result)
+      entry[:events] << [event, result]
+      subscribers = entry[:subscribers].dup
+      if %i[error failure success].include?(event)
+        @pending.delete(key) if @pending[key].equal?(entry)
+        if event == :success && request_key(uri) == key
+          @successful[key] = result if deduplicate
+          @cache_store&.store(key, result)
+        end
+      end
+      subscribers.each { |subscriber| publish(event, copy_result(result, subscriber)) }
+    end
+
+    def copy_result(result, subscriber)
+      result.dup.tap do |copy|
+        copy.uri = URI(subscriber.first)
+        copy.result_uri = result.result_uri.dup
+        copy.options = subscriber.last
+      end
+    end
+
+    def request_key(uri)
+      parsed = URI(uri.to_s)
+      return unless %w[http https].include?(parsed.scheme) && parsed.host && !parsed.userinfo
+
+      attributes = Config::ATTRIBUTES - %i[logger cache deduplicate]
+      attributes += %i[timeout connecttimeout read_timeout open_timeout protocol].select { |key| respond_to?(key) }
+      settings = attributes.to_h { |key| [key, send(key)] }
+      settings[:headers] = headers.sort_by { |key, _value| key.to_s }
+      Digest::SHA256.hexdigest(JSON.generate([task_klass.name, parsed.to_s, settings]))
+    rescue URI::InvalidURIError, JSON::GeneratorError, TypeError
+      nil
+    end
   end
 end

lib/ruby-link-checker/config.rb

@@ -17,6 +17,8 @@ module Config
       retry_jitter
       max_retry_delay
       respect_retry_after
+      deduplicate
+      cache
     ].freeze
 
     attr_accessor(*Config::ATTRIBUTES)
@@ -34,6 +36,8 @@ def reset
       self.retry_jitter = 0
       self.max_retry_delay = 60
       self.respect_retry_after = false
+      self.deduplicate = false
+      self.cache = nil
     end
 
     def retries=(value)

lib/ruby-link-checker/result.rb

@@ -4,6 +4,7 @@
 module LinkChecker
   class Result
     attr_accessor :uri, :result_uri, :method, :options, :checker
+    attr_writer :cached
 
     def initialize(current_uri, method, original_uri, options = {})
       @uri = original_uri
@@ -16,6 +17,10 @@ def success?
       false
     end
 
+    def cached?
+      !!@cached
+    end
+
     def failure?
       false
     end

spec/ruby-link-checker/checker_spec.rb

@@ -2,9 +2,76 @@
 require 'spec_helper'
 
 describe LinkChecker::Checker do
-    context 'config' do
+  describe LinkChecker::Cache do
+    let(:directory) { Dir.mktmpdir('link-checker-spec-') }
+    let(:file) { File.join(directory, 'cache.json') }
+    let(:cache) { described_class.new({ file: file, ttl: 60 }, logger: LinkChecker::Logger.default) }
+    let(:result) do
+      LinkChecker::Net::HTTP::Result.new(
+        URI('https://www.example.org/final'), 'GET', URI('https://www.example.org/'),
+        {}, Net::HTTPOK.new('1.1', '200', 'OK'), { location: 'private.html' }
+      )
+    end
+
+    after do
+      FileUtils.remove_entry(directory)
+    end
+
+    it 'expires successful entries using the configured TTL' do
+      cache.store('key', result)
+      allow(Time).to receive(:now).and_return(Time.now + 61)
+      expect(cache.fetch('key', 'https://www.example.org/')).to be_nil
+    end
+
+    it 'retains the final URL and status without persisting metadata or headers' do
+      cache.store('key', result)
+      restored = cache.fetch('key', 'https://www.example.org/')
+      expect(restored.result_uri.to_s).to eq('https://www.example.org/final')
+      expect(restored.code).to eq(200)
+      data = JSON.parse(File.read(file)).fetch('entries').fetch('key')
+      expect(data.keys).to contain_exactly('checked_at', 'method', 'code', 'result_uri')
+    end
+
+    it 'merges writes from separate cache instances' do
+      other_cache = described_class.new({ file: file, ttl: 60 }, logger: LinkChecker::Logger.default)
+      cache.store('one', result)
+      other_cache.store('two', result)
+      expect(cache.fetch('one', 'https://www.example.org/')).not_to be_nil
+      expect(cache.fetch('two', 'https://www.example.org/')).not_to be_nil
+    end
+
+    it 'recovers from corrupted cache files' do
+      File.write(file, 'invalid JSON')
+      expect(cache.fetch('key', 'https://www.example.org/')).to be_nil
+      cache.store('key', result)
+      expect(cache.fetch('key', 'https://www.example.org/').code).to eq(200)
+    end
+
+    it 'ignores malformed cache entries' do
+      File.write(file, JSON.generate('version' => 1, 'entries' => { 'key' => true }))
+      expect(cache.fetch('key', 'https://www.example.org/')).to be_nil
+    end
+
+    it 'does not store credentials in redirect URLs' do
+      result.result_uri = URI('https://user:password@www.example.org/')
+      cache.store('key', result)
+      expect(File).not_to exist(file)
+    end
+
+    it 'requires an explicit file and TTL' do
+      expect { described_class.new({}, logger: LinkChecker::Logger.default) }.to raise_error ArgumentError
+      expect { described_class.new({ file: file }, logger: LinkChecker::Logger.default) }.to raise_error ArgumentError
+    end
+
+    it 'does not turn a cache write failure into a failed link' do
+      allow(FileUtils).to receive(:mkdir_p).and_raise(Errno::EACCES)
+      expect { cache.store('key', result) }.not_to raise_error
+    end
+  end
+
+  context 'config' do
     it 'requires at least one method' do
-      expect { LinkChecker::Checker.new(methods: []) }.to raise_error ArgumentError, 'Missing methods.'
+      expect { described_class.new(methods: []) }.to raise_error ArgumentError, 'Missing methods.'
     end
 
     it 'rejects an invalid redirect limit' do