Propose honoring stated retry times (add-retry-after)

openspec/changes/add-retry-after/adr.md

@@ -0,0 +1,16 @@
+# ADR Review
+
+## Reviewed
+
+ADR 0018 (health-driven eligibility) defines the breaker, its cooldown and that a cooling target is left out of
+a plan while another remains. This change only changes how long an endpoint cools after a failure that states a
+retry time; eligibility, ordering and fail-open behavior are untouched. ADR 0013 (decisions are metadata only)
+is unaffected: only a duration is kept, never body text.
+
+## Outcome
+
+Amends ADR 0018's cooldown rule, with the amendment recorded in the ADR rather than a new one.
+
+## New ADRs
+
+None; ADR 0018 gains an "Amended" note describing stated retry times.

openspec/changes/add-retry-after/design.md

@@ -0,0 +1,41 @@
+# Design
+
+## Context
+
+`Breaker::failure` counts consecutive failures and, at `failure_threshold`, opens for
+`cooldown_secs` doubled per earlier trip up to `max_cooldown_secs`. `fails_over` in `pool.rs` decides which
+errors count (429, 408, 5xx, transport and timeout). A 429 from a free tier usually comes with how long to wait.
+
+## Decisions
+
+- **One seam: `retry_after(error) -> Option<Duration>`.** Today it parses the body; when Switchyard's error
+  carries headers it reads `Retry-After` first (delta-seconds or an HTTP date) and falls back to the body. The
+  breaker and everything downstream see only a duration, so adopting #904 is a change to one function.
+- **Only a few well-known body shapes.** Case-insensitive phrases `try again in <d>`, `retry after <d>`,
+  `retry in <d>` and `please retry in <d>`, and the JSON fields `retryDelay` (Google, `"28s"`),
+  `retry_after` and `retry-after` (seconds). A duration is `<n>ms`, `<n>s`, `<n>m`, or a combination such as
+  `2m3.5s`, with a bare number meaning seconds only for the JSON fields. Only the first 4 KiB of the body is
+  read. Guessing wider would turn arbitrary error prose into cooldowns.
+- **Open immediately on a hint.** The provider has said it will not serve before then, so counting to
+  `failure_threshold` first wastes requests and, on free tiers, can burn the quota that is about to reset.
+  Without a hint, behavior is exactly as today.
+- **Exactly the hinted time, not doubled, not capped by `max_cooldown_secs`.** The existing cap exists because
+  a guessed cooldown can be wrong; a stated one is not a guess. It is bounded by its own limit,
+  `max_retry_after_secs` (default 300), so an absurd value (or a hostile body) cannot park an endpoint for
+  hours. At least 1 second, so a `0` cannot make a busy loop.
+- **A late failure does not shorten or extend it.** The breaker already ignores failures that arrive while it is
+  open; the hinted cooldown follows the same rule. A failed probe after the cooldown is a new failure and, if it
+  carries a hint, a new cooldown.
+- **429 and 503 only.** Those are the statuses that mean "come back later". Other failures keep the guessed
+  cooldown.
+- **Untrusted text.** The body is provider-controlled. The parser reads bounded input, returns a number, and
+  nothing from the body is logged or stored beyond that number.
+
+## Risks / trade-offs
+
+- A provider whose phrasing is not recognized falls back to today's behavior, so the interim helps the common
+  providers and loses nothing on the others.
+- A hint can be shorter than the real recovery time (the next request is rate-limited again): the new cooldown
+  then follows from that response, so it converges.
+- Two requests in flight to one endpoint can both fail with hints; the second finds the breaker already open
+  and changes nothing.

openspec/changes/add-retry-after/proposal.md

@@ -0,0 +1,41 @@
+# Proposal
+
+## Why
+
+A rate-limited free endpoint tells the caller when it will serve again. humpyard cools a failing endpoint for a
+guessed time (10 seconds, doubling to a cap of 35, after `failure_threshold` consecutive failures), so it either
+keeps sending requests into a 429 or comes back too early. ModelRelay honors the provider's retry time; this is
+step 1 of "Replacing ModelRelay".
+
+Switchyard's HTTP client reads `Retry-After` but, in 0.3.0, does not put it on the error it returns
+(`UpstreamHttp { status, body }`), so the header cannot be read by humpyard yet. Upstream PR
+NVIDIA-NeMo/Switchyard#904 (closing #903) adds the headers to that error. Until a release has it, many
+providers repeat the same information in the error body ("Please try again in 7.66s", or a `retryDelay` field),
+which humpyard can read now.
+
+## What Changes
+
+- **A retry time is read from a failed attempt**: from the upstream error body now (an interim), and from the
+  `Retry-After` header once Switchyard exposes it (a follow-up that changes one function). The body is read for
+  a small set of well-known phrasings and fields only.
+- **A rate-limited endpoint (429, or 503) with a retry time is cooled for that long**, immediately, without
+  waiting for `failure_threshold` failures, and without the doubling. When the cooldown ends, one request probes
+  it as before.
+- **Bounded**: the retry time is clamped to at least 1 second and at most `[health] max_retry_after_secs`
+  (default 300); anything unreadable or out of range is ignored, and the endpoint behaves exactly as today.
+- **Visible**: endpoint health shows the cooldown's source (`retry_after` or `failures`), decisions and
+  `route/explain` already show a cooling target as skipped, and a counter records how often a retry time was
+  honored and where it came from.
+- **Out of scope:** reading `x-ratelimit-reset-*` or per-minute quota headers, predicting limits before a 429,
+  changing Switchyard's own internal retries (`max_retries`), and any change to routes or fallback order.
+
+## Capabilities
+
+### Modified Capabilities
+- `routing-policy`: endpoint health and cooldown.
+
+## Impact
+
+`src/health.rs` (a cooldown from a given duration), `src/pool.rs` (read the hint on a failed attempt), a small
+parser module (`src/retry_after.rs`), `src/config` (`max_retry_after_secs`), `src/metrics.rs`, `src/server.rs`
+(health snapshot). Docs: `routing.md` (Endpoint health), `configuration.md`, `metrics.md`.

openspec/changes/add-retry-after/specs/routing-policy/spec.md

@@ -0,0 +1,33 @@
+# Spec Delta
+
+## ADDED Requirements
+
+### Requirement: Stated retry times
+When an attempt fails with status 429 or 503 and the failure states a retry time, the gateway SHALL cool that endpoint for exactly that time (at least 1 second and at most `[health] max_retry_after_secs`, default 300), opening it immediately instead of after `failure_threshold` failures. The retry time SHALL be read from the upstream response when it is available as a header, and otherwise from a small set of well-known phrasings and fields in the error body, reading at most its first 4 KiB. A failure with no readable retry time SHALL behave as before.
+
+#### Scenario: A 429 with a retry time
+- **WHEN** an endpoint returns 429 with a body that says to try again in 20 seconds
+- **THEN** the endpoint is skipped for 20 seconds even on its first failure, and one request probes it afterwards
+
+#### Scenario: The cap
+- **WHEN** the stated retry time is larger than `max_retry_after_secs`
+- **THEN** the endpoint cools for `max_retry_after_secs`
+
+#### Scenario: Unreadable or out of range
+- **WHEN** the body has no recognized retry time, or one that is zero or negative or not a number
+- **THEN** the failure is handled as before (counted toward `failure_threshold`, guessed cooldown)
+
+#### Scenario: Other statuses
+- **WHEN** a 500 response body says to try again in 20 seconds
+- **THEN** the stated time is ignored
+
+#### Scenario: Late failures
+- **WHEN** a request sent before the endpoint cooled fails while it is cooling
+- **THEN** the cooldown is neither shortened nor extended
+
+### Requirement: Cooldown source is visible
+Endpoint health SHALL report whether the current cooldown came from a stated retry time or from consecutive failures, and the gateway SHALL count honored retry times by provider and source.
+
+#### Scenario: Health shows the source
+- **WHEN** an endpoint is cooling because of a stated retry time
+- **THEN** its health entry shows the source `retry_after` and the time remaining

openspec/changes/add-retry-after/tasks.md

@@ -0,0 +1,14 @@
+# Tasks
+
+## 1. Parser and breaker
+- [ ] 1.1 `src/retry_after.rs`: the body parser (phrases, `retryDelay`, `retry_after`, durations in ms/s/m/combinations), bounded input; unit tests for each shape, units, case, junk, zero, negative, huge values and a body past 4 KiB
+- [ ] 1.2 `[health] max_retry_after_secs` (default 300): config, validation and a docs row
+- [ ] 1.3 `Breaker::failure_for(duration)`: open immediately for exactly that long, no doubling, late failures ignored; unit tests for each scenario and that the guessed path is unchanged
+
+## 2. Wiring and visibility
+- [ ] 2.1 `src/pool.rs`: on a 429 or 503 failure read the retry time and call the breaker; a counter `humpyard_retry_after_honored_total{provider,source}` and docs in `metrics.md`
+- [ ] 2.2 Endpoint health snapshot shows the cooldown source; a test through `/v1/health` or the explain output
+- [ ] 2.3 End to end: a mock upstream answering 429 with "try again in N seconds" is skipped for that time while another target serves, then probed; a hint-less 429 behaves as before
+
+## 3. Docs and follow-up
+- [ ] 3.1 `routing.md` (Endpoint health), `configuration.md`, the ADR 0018 amendment, the roadmap marks step 1 shipped with the header read still to follow Switchyard#904