fix(backtest): a failed engine call is one lost cell, and a spent quota stops that engine — never the campaign

[modelmirror]

Closes #1957.

What changed

A cert back-test campaign no longer ends when one engine call fails; it lands a report that names what it could not score.

• A failed engine call is one lost cell. _replay_item_cells in cert_backtest.py catches EngineFailed beside EngineUnavailable, records the cell under provenance.lost_cells as engine-failed, and continues. The scorer treats it as a loss exactly as a missing or invalid cell; short entries still sort below full ones.
• A terminal quota is not transient. pipeline/runner.py gains EngineQuotaExhausted(EngineFailed), raised after one attempt and no backoff when the captured stderr carries a terminal-quota marker (TerminalQuotaError, exhausted your daily quota, both gemini's, named to that engine in the docstring). Ordinary throttles (HTTP 429, Quota exceeded for quota metric, retry-after) stay on the transient retry path; tests pin both sides. The replay distinguishes the two by exception type, not message text.
• A spent quota stops the engine's remaining cells. Keyed on the resolved backend, kept as its own quota_exhausted set (never merged into unavailable: a missing binary and a spent allowance are different facts and the CLI prints a distinct drop line for each). Every further cell of that engine is recorded as quota-exhausted without being attempted, so the report's counts stay honest.
• Schema: CertBacktestCellLoss.reason gains engine-failed and quota-exhausted (extracted to a named CertBacktestLossReason alias); two stale field descriptions on the provenance block corrected; schemas/ regenerated. The metrics-refresh PR body's per-loss gloss now reads (no score — unreadable, failed, or not attempted after a spent quota), pinned by test.
• Docs: metrics/README.md's cert-backtest reading rules now state that a quota-exhausted truncation keeps the prefix of the dispatched replay order, so the surviving slice is differently composed (floor, band mix, base rate and provisioning mix all follow the prefix) and such an entry is not comparable with a full one; and that a quota drop carries a loss per petition while a missing-binary drop carries none. docs/pipeline.md and docs/cli.md updated; config/tracking.yaml's runner comment and RunnerConfig's docstring now state the terminal carve-out from max_attempts, since the same runner serves predict and evaluate.

Executed checks

• Offline stub campaign over the fixture corpus (--engine stub): a report lands with all five entries and empty lost_cells, before and after the review fixes.
• New tests: the campaign survives EngineFailed and lands a report with the cell under lost_cells while the other predictors score; a spent quota means exactly one gemini attempt across two petitions with both cells recorded as losses and both other engines scoring both petitions; terminal-quota stderr → one attempt, no sleep, EngineQuotaExhausted; ordinary throttles still transient.
• scripts/gate.sh lock, lint, types, test (4984 passed, 2 pre-existing skips), data, schemas — all pass. uv run fedcourts process-digest --all byte-identical to origin/staging, re-verified after the tracking.yaml comment edit.

What is not yet verified on a runner: the terminal-quota classification reads the subprocess's captured stderr; the marker text is quoted from run 35643195366's log, where the classifier demonstrably read that same stream, but the tests prove the classifier given the string, not the stream. The first real gemini quota event settles it. The campaign-survival half (item 1) is stream-independent and fully covered.

Review

code-reviewer, docs-reviewer, stats-reviewer: six blockers, all fixed (the "three backoffs apiece" wording that this change itself made false; the tracking.yaml comment; the metrics-refresh gloss; two stale schema descriptions; the order-determined truncation reading rule; the asymmetry between the two whole-predictor drops). Three recommendations declined, stated here: a rejected codex login now shows as one engine-failed loss per petition rather than a campaign crash (rerouting it into the missing-binary path is its own change); the metrics-refresh headline can still name an offline baseline "top predictor" when a concrete --engine dispatch truncates every engine (pre-existing, orthogonal); which engine ran out is not carried on ReplayOutcome (recoverable from lost_cells and the CLI drop line).

Merge: left for the maintainer — it changes the shared runner's retry classification. Effect check for the promotion body: the next run-backtest on a day gemini's allowance is spent lands a report whose provenance.lost_cells carries quota-exhausted for gemini's cells and the job log prints the quota drop line; on a clean day the report is unchanged in shape and lost_cells is empty.

🤖 Generated with Claude Code