Add transient retries, backoff, and Async concurrency limits

AGENTS.md

@@ -0,0 +1,23 @@
+# Contributor instructions
+
+Keep changes focused and preserve existing behavior unless a change is explicitly requested. Follow CONTRIBUTING.md and document public behavior in README.md.
+
+## Changelog and pull requests
+
+Add entries under the current `(Next)` heading in CHANGELOG.md and retain `Your contribution here`. Include the implementation PR number, its URL, and contributor attribution using this format:
+
+```markdown
+* [#123](https://github.com/dblock/ruby-link-checker/pull/123): Describe the change - [@contributor](https://github.com/contributor).
+```
+
+Predict the PR number before opening a new PR, then verify the assigned number and correct the entry if necessary. Issue references may supplement but do not replace the implementation PR link.
+
+Keep Markdown paragraphs on one line. Keep each PR branch to one commit, squash follow-up changes before pushing, and use `--force-with-lease` when updating rewritten PR history. Never force-push the default branch.
+
+## Validation
+
+Run focused RSpec tests for behavior changes and `bundle exec rubocop` before committing. `bundle exec rake` runs both the full specs and RuboCop. Avoid repeated intermediate test runs; rerun when failures or subsequent changes require it.
+
+For changelog or README changes, run `bundle exec danger dry_run --base=origin/main`; Danger checks changelog entries and README table-of-contents consistency. Keep the temporary `git < 6` dependency pin until Danger supports git 6; the Gemfile comment links the upstream fix.
+
+Do not commit Gemfile.lock; this repository ignores it. Include `Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>` in AI-assisted commits and disclose AI involvement when posting GitHub comments on behalf of the user.

CHANGELOG.md

@@ -1,6 +1,7 @@
 ### 0.5.0 (Next)
 
-* Fixed chained relative redirects, bounded redirect following with `max_redirects`, enabled plain HTTP in the Net::HTTP checker, and corrected retry documentation - [@dblock](https://github.com/dblock), [@Copilot](https://github.com/Copilot).
+* [#23](https://github.com/dblock/ruby-link-checker/pull/23): Fixed chained relative redirects, bounded redirect following with `max_redirects`, enabled plain HTTP in the Net::HTTP checker, and corrected retry documentation - [@dblock](https://github.com/dblock), [@Copilot](https://github.com/Copilot).
+* [#24](https://github.com/dblock/ruby-link-checker/pull/24): Added transient-only retries, opt-in retry delays/backoff/jitter and `Retry-After` handling, transport error categories, and an Async concurrency limit - [@dblock](https://github.com/dblock), [@Copilot](https://github.com/Copilot).
 * [#8](https://github.com/dblock/ruby-link-checker/issues/8): Added `protocol` configuration option to `LinkChecker::Async::HTTP` to force HTTP/1 or HTTP/2 - [@dblock](https://github.com/dblock), [@Copilot](https://github.com/Copilot).
 * Your contribution here.
 

README.md

@@ -20,6 +20,8 @@ A fast Ruby link checker with support for multiple HTTP libraries. Does not pars
   - [Options](#options)
     - [Retries](#retries)
     - [Redirects](#redirects)
+    - [Retry After](#retry-after)
+    - [Retry Policy](#retry-policy)
     - [Results](#results)
     - [Methods](#methods)
     - [Logger](#logger)
@@ -180,6 +182,12 @@ LinkChecker::Async::HTTP.configure do |config|
 end
 ```
 
+Async::HTTP reuses connection-pooling clients per scheme/host/port. Its default concurrency is unlimited; set `max_concurrency` to bound active requests without counting delayed retries against that limit.
+
+```ruby
+checker = LinkChecker::Async::HTTP::Checker.new(max_concurrency: 25)
+```
+
 #### [LinkChecker::Net::HTTP](lib/ruby-link-checker/net/http/checker.rb)
 
 Slow, sequential checker.
@@ -218,7 +226,7 @@ end
 
 #### Retries
 
-By default link checkers do not retry. Set `retries` to retry errors and failures. The retry budget is shared across methods: with `retries: 2` and the default methods, repeated failures produce `HEAD`, `HEAD`, `HEAD`, then `GET`.
+By default link checkers do not retry. Set `retries` to enable retries for transient connection errors/timeouts and HTTP statuses 408, 425, 429, 500, 502, 503, and 504. Permanent HTTP failures and invalid URLs are not retried. HEAD-to-GET fallback still applies independently of the retry policy. The retry budget is shared across methods: with `retries: 2`, repeated transient failures produce `HEAD`, `HEAD`, `HEAD`, then `GET`.
 
 ```ruby
 checker = LinkChecker::Net::HTTP::Checker.new(retries: 1)
@@ -232,6 +240,31 @@ Checkers follow up to 20 redirects per attempt and detect redirect loops. Relati
 checker = LinkChecker::Typhoeus::Hydra::Checker.new(max_redirects: 10)
 ```
 
+#### Retry After
+
+Set `retry_after` to a minimum delay in seconds (default `0`). Optional `retry_backoff` multiplies that delay on successive retries (default `1`), and `retry_jitter` adds a random fraction between `0` and the configured value (default `0`, maximum `1`). All delays are capped by `max_retry_delay` (default `60` seconds).
+
+```ruby
+checker = LinkChecker::Typhoeus::Hydra::Checker.new(
+  retries: 2,
+  retry_after: 1,
+  retry_backoff: 2,
+  retry_jitter: 0.1,
+  respect_retry_after: true,
+  max_retry_delay: 60
+)
+```
+
+With `respect_retry_after: true` (default `false`), HTTP `Retry-After` delta-seconds and dates provide a minimum delay, subject to the same cap. Invalid or past dates are ignored. Async delays yield without blocking other requests. Hydra queues delayed retries after the active batch drains, so actual delays can exceed the configured minimum without blocking completion callbacks. Delays apply to retries, not ordinary HEAD-to-GET fallback.
+
+#### Retry Policy
+
+Customize `retry_status_codes` to choose retryable HTTP statuses. Set it to `nil` to restore the previous behavior of retrying every failure and error, including malformed URLs and programming errors.
+
+```ruby
+checker = LinkChecker::Net::HTTP::Checker.new(retries: 1, retry_status_codes: [429, 503])
+```
+
 #### Results
 
 By default checkers collect results. 
@@ -342,6 +375,8 @@ Events are called with results, which contain the following properties.
 | :request_headers  | Request headers.                                                |
 | :redirect_to      | A redirect URL in case of redirects.                            |
 | :error            | A raised error in case of errors.                               |
+| :error_type       | `:timeout`, `:connection`, or `:other` for errors; otherwise `nil`. |
+| :response_headers | Response headers, accessible using `[]`.                        |
 
 See [result.rb](lib/ruby-link-checker/result.rb) for more details.
 

lib/ruby-link-checker/async/http/checker.rb

@@ -3,7 +3,7 @@ module Async
     module HTTP
       class Task < ::LinkChecker::Task
         def run!
-          checker._queue { _run! }
+          checker._queue(delay: delay) { _run! }
         end
 
         private
@@ -42,25 +42,36 @@ def initialize(options = {})
           end
           @queue = []
           @clients = {}
+          if max_concurrency && (!max_concurrency.is_a?(Integer) || max_concurrency < 1)
+            raise ArgumentError, "Invalid concurrency limit: #{max_concurrency}"
+          end
+
           super
         end
 
         # Runs all queued checks concurrently and blocks until they complete.
         def run
+          require 'async/semaphore'
+
           Barrier do |barrier|
             @barrier = barrier
+            @semaphore = ::Async::Semaphore.new(max_concurrency) if max_concurrency
             _flush!
           end
         ensure
           @barrier = nil
+          @semaphore = nil
           _close_clients!
         end
 
-        def _queue(&block)
+        def _queue(delay: 0, &block)
           if @barrier
-            @barrier.async(&block)
+            @barrier.async do
+              sleep(delay) if delay.positive?
+              @semaphore ? @semaphore.acquire(&block) : block.call
+            end
           else
-            @queue << block
+            @queue << [delay, block]
           end
         end
 
@@ -94,7 +105,7 @@ def _endpoint_options
         def _flush!
           queue = @queue
           @queue = []
-          queue.each { |block| @barrier.async(&block) }
+          queue.each { |delay, block| _queue(delay: delay, &block) }
         end
 
         def _close_clients!

lib/ruby-link-checker/async/http/config.rb

@@ -10,6 +10,7 @@ module Config
           read_timeout
           open_timeout
           protocol
+          max_concurrency
         ].freeze
 
         attr_accessor(*Config::ATTRIBUTES)
@@ -18,6 +19,7 @@ def reset
           self.read_timeout = nil
           self.open_timeout = nil
           self.protocol = nil
+          self.max_concurrency = nil
         end
       end
 

lib/ruby-link-checker/async/http/result.rb

@@ -58,6 +58,10 @@ def request_headers
           Headers.new(request.headers)
         end
 
+        def response_headers
+          response&.headers || {}
+        end
+
         def redirect_to
           return nil unless response
 

lib/ruby-link-checker/checker.rb

@@ -6,11 +6,10 @@ class Checker
     include LinkChecker::Callbacks
 
     attr_reader :results
-    attr_accessor(*(Config::ATTRIBUTES - %i[max_redirects]))
 
     def initialize(options = {})
       LinkChecker::Config::ATTRIBUTES.each do |key|
-        send("#{key}=", options[key] || LinkChecker.config.send(key))
+        send("#{key}=", options.fetch(key) { LinkChecker.config.send(key) })
       end
       raise ArgumentError, "Missing methods." if methods&.none?
       @logger ||= options[:logger] || LinkChecker::Config.logger || LinkChecker::Logger.default

lib/ruby-link-checker/config.rb

@@ -11,6 +11,12 @@ module Config
       logger
       retries
       max_redirects
+      retry_status_codes
+      retry_after
+      retry_backoff
+      retry_jitter
+      max_retry_delay
+      respect_retry_after
     ].freeze
 
     attr_accessor(*Config::ATTRIBUTES)
@@ -22,6 +28,12 @@ def reset
       self.logger = nil
       self.retries = 0
       self.max_redirects = 20
+      self.retry_status_codes = [408, 425, 429, 500, 502, 503, 504]
+      self.retry_after = 0
+      self.retry_backoff = 1
+      self.retry_jitter = 0
+      self.max_retry_delay = 60
+      self.respect_retry_after = false
     end
 
     def retries=(value)
@@ -35,6 +47,44 @@ def max_redirects=(value)
 
       @max_redirects = value
     end
+
+    def retry_status_codes=(value)
+      unless value.nil? || (value.is_a?(Array) && value.all? { |code| code.is_a?(Integer) && code.between?(100, 599) })
+        raise ArgumentError, "Invalid retry status codes: #{value}"
+      end
+
+      @retry_status_codes = value&.dup
+    end
+
+    def retry_after=(value)
+      validate_number!(:retry_after, value, 0)
+      @retry_after = value
+    end
+
+    def retry_backoff=(value)
+      validate_number!(:retry_backoff, value, 1)
+      @retry_backoff = value
+    end
+
+    def retry_jitter=(value)
+      validate_number!(:retry_jitter, value, 0)
+      raise ArgumentError, "Invalid retry_jitter: #{value}" if value > 1
+
+      @retry_jitter = value
+    end
+
+    def max_retry_delay=(value)
+      validate_number!(:max_retry_delay, value, 0)
+      @max_retry_delay = value
+    end
+
+    private
+
+    def validate_number!(name, value, minimum)
+      return if value.is_a?(Numeric) && value.real? && value.finite? && value >= minimum
+
+      raise ArgumentError, "Invalid #{name}: #{value}"
+    end
   end
 
   class << self

lib/ruby-link-checker/errors.rb

@@ -1,3 +1,4 @@
 require_relative 'errors/base_error'
 require_relative 'errors/redirect_loop_error'
 require_relative 'errors/redirect_limit_error'
+require_relative 'errors/connection_error'

lib/ruby-link-checker/errors/connection_error.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+module LinkChecker
+  module Errors
+    class ConnectionError < BaseError
+      TRANSIENT_CODES = %i[
+        couldnt_resolve_proxy couldnt_resolve_host couldnt_connect
+        partial_file got_nothing send_error recv_error http2 http2_stream
+      ].freeze
+
+      attr_reader :return_code
+
+      def initialize(return_code, message)
+        @return_code = return_code
+        super(message)
+      end
+
+      def error_type
+        TRANSIENT_CODES.include?(return_code) ? :connection : :other
+      end
+    end
+  end
+end

lib/ruby-link-checker/net/http/checker.rb

@@ -3,6 +3,7 @@ module Net
     module HTTP
       class Task < ::LinkChecker::Task
         def run!
+          sleep(delay) if delay.positive?
           ::Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == 'https') do |http|
             http.read_timeout = checker.read_timeout if checker.read_timeout
             http.open_timeout = checker.open_timeout if checker.open_timeout

lib/ruby-link-checker/net/http/result.rb

@@ -30,6 +30,10 @@ def request_headers
           request
         end
 
+        def response_headers
+          response || {}
+        end
+
         def redirect_to
           return nil unless response
 

lib/ruby-link-checker/result.rb

@@ -1,3 +1,6 @@
+require 'timeout'
+require 'socket'
+
 module LinkChecker
   class Result
     attr_accessor :uri, :result_uri, :method, :options, :checker
@@ -33,6 +36,14 @@ def request_headers
       {}
     end
 
+    def response_headers
+      {}
+    end
+
+    def error_type
+      nil
+    end
+
     def code
       nil
     end
@@ -70,5 +81,14 @@ def error?
     def code
       error.class.name
     end
+
+    def error_type
+      return :timeout if error.is_a?(Timeout::Error)
+      return :timeout if defined?(::Async::TimeoutError) && error.is_a?(::Async::TimeoutError)
+      return error.error_type if error.is_a?(LinkChecker::Errors::ConnectionError)
+      return :connection if [IOError, SystemCallError, SocketError].any? { |klass| error.is_a?(klass) }
+
+      :other
+    end
   end
 end

lib/ruby-link-checker/task.rb

@@ -3,6 +3,7 @@ class Task
     include LinkChecker::Callbacks
 
     attr_reader :uri, :original_uri, :method, :logger, :options, :checker
+    attr_accessor :delay
 
     def initialize(checker, uri, method, original_uri, options = {})
       @checker = checker
@@ -11,6 +12,7 @@ def initialize(checker, uri, method, original_uri, options = {})
       @original_uri = original_uri || @uri
       @method = method
       @options = options
+      @delay = 0
     end
 
     def run!