Repository navigation
[Bug]: Reset-derived cooldowns can miss early recovery while another pool account remains eligible #915
Description
Client or integration
Codex App
Area
Authentication and account pool
Summary
A reset-derived Codex cooldown can remain after the account recovers early when another pool account remains eligible.
An actual upstream 429 with reset headers creates a non-persistent cooldown. The cooled account is excluded, and the current recovery probe is acquired only after routing selects that account. If another account can always serve traffic, the cooled account is never selected, so it never receives the bounded probe. A forced fresh WHAM read may show recovery, but updating quota cache does not clear the matching routing cooldown.
Persistently pausing an account at displayed 100% is not a valid workaround: OpenAI can reset earlier than the predicted timestamp, 100% can remain advisory while an in-progress task still succeeds, and pause can outlive the restriction.
Expected behavior:
- Create cooldown only after an actual upstream rejection; a WHAM
100%snapshot alone must not create cooldown, pause, or move an existing thread. - Keep reset-derived exclusion non-persistent.
- At a bounded interval, permit one single-flight fresh quota refresh for a due reset-derived candidate even while another account serves requests.
- Fence by account identity, credential generation, quota scope, and cooldown generation.
- Clear only the matching reset-derived scope/generation when a complete fresh authoritative snapshot proves recovery.
- Retain cooldown on still-
100%, incomplete, stale, unavailable, unrelated-scope, or replaced-credential results. - Do not clear explicit
Retry-Afteror headerless/default cooldowns through this path. - Fail closed without delaying ordinary traffic on the healthy alternate.
Relevant source:
src/codex/routing.ts: reset-derived health, generations, scope, selection, and probe leases.src/codex/auth-context.ts: probe admission currently occurs only after selection.src/codex/auth-api.ts: main/pool forced refresh and existing single-flight.src/codex/quota.ts: freshness and authoritative exhaustion.
Required tests: early recovery with another eligible account; still-exhausted/unknown/fetch-failure cases; concurrent single-flight; stale-generation rejection; main and pool accounts; shared vs Spark scope separation; explicit Retry-After untouched; all pool strategies re-include the recovered account; paused unchanged; and no cooldown/rebinding from a standalone 100% snapshot.
Related: #433 (bounded probe/cap), #590/#599 (scoped cooldowns), #584/#585 (same-request failover after real rejection), and #657/#866 (separate reset-credit semantics).
Reproduction
- Configure accounts A and B.
- Let A receive an actual upstream
429with reset headers. - Keep B eligible so traffic continues through B.
- Let OpenAI reset A earlier than the recorded deadline.
- Force a fresh authoritative quota read while continuing traffic through B.
Current: A's quota cache updates, but its routing cooldown remains because A is never selected for the existing probe.
Expected: one bounded refresh clears only A's matching reset-derived generation, A becomes eligible immediately, and no persistent pause or percentage-only thread move occurs.
Version
2.10.0; the relevant routing and quota-refresh behavior remained on dev at 075c2f34c060d068ed56985e76924a9f23aca39d when checked on 2026-08-03.
Operating system
Windows 11 Pro 10.0.26200 (platform-independent routing gap).
Provider and model
OpenAI (Codex login / forward account pool); native Codex models and confirmed quota scopes.
Logs or error output
Selected Codex account is cooling down
Screenshots and supporting files
None. Exact quota values, reset times, and account identifiers are omitted.
Redacted configuration
{
"providers": {
"openai": {
"adapter": "openai-responses",
"authMode": "forward",
"codexAccountMode": "pool"
}
},
"accountPoolStrategy": "fill-first",
"autoSwitchThreshold": 100,
"activeCodexAccountId": "__main__"
}Checks
- I searched existing issues and documentation.
- I removed secrets, tokens, account details, request credentials, and personal data.
Activity
github-actions commentedon Aug 3, 2026
Automated translation bookkeeping — detected language: unknown.
lidge-jun commentedon Aug 3, 2026
Triaged against current
dev. Real, and it needs more than a guard — recording the analysis so the fix does not start from scratch.The mechanism you describe is exact. Cooled accounts are filtered out before all strategy selection (
src/codex/routing.ts:734-760), and the alternate selectors only ever see the eligible list (:927-965). Probe eligibility does exist and is correctly limited to non-Retry-Aftercooldowns with one lease per interval (:369-423) — butresolveCodexAuthContext()selects the account first (src/codex/auth-context.ts:214-226) and only then checks that selected account's lease (:237-253). So with B eligible, A is never selected and never reaches the lease code. Selection-before-probe, exactly as you put it.Your point about WHAM is right too: pool results call
setAccountQuotaFromParsed()(src/codex/auth-api.ts:590-626), which writes the quota cache (src/codex/quota.ts:134-179) and has no authority over routing cooldown generations.Why this is scheduled rather than patched now: the fix is a background recovery worker with a claim/settle contract fenced on cooldown generation, quota scope, and credential generation — not a change to account selection. Two constraints already established, so nobody rediscovers them:
clearCodexAccountCooldown()must not be used; it clears every scope and carries no credential fence.src/codex/auth-api.ts:646-688is already generation-aware and single-flight, so it is the thing to join rather than duplicate.One qualification on the report: "a 100% WHAM snapshot must never move an existing thread under any strategy" is a separate policy change. The quota strategy deliberately rebinds an over-threshold thread (
src/codex/routing.ts:1179-1200) and has tests for it. Fill-first and round-robin already preserve affinity. Worth deciding on its own rather than bundling into the recovery fix.Written up in
devlog/_plan/260803_cooldown_recovery_probe/000_plan.mdondev.docs(triage): record the stack outcome and what the audit gates caught
Merge pull request #955 from lidge-jun/codex/915-cooldown-recovery-probe
lidge-jun commentedon Aug 4, 2026
Fixed by #955, merged into
devas48338a1a4.A cooled account is now probed for early recovery instead of waiting to be selected, so it rejoins the pool as soon as its reset actually lands rather than at the next request that happens to pick it. Layer 5 of the bug stack; plan and evidence in
devlog/_plan/260803_bug_backlog_stack/030_cooldown_recovery.md.Two defects in the original implementation were found in review by @Yuxin-Qiao (#967) and fixed before merge: monthly-classified snapshots were being rejected outright, and the probe's own token refresh looked like a credential replacement. Both are covered by regression tests.
Closing. Reopen if you see an account stay cooled past its reset.
fix(codex): probe reset-derived cooldowns without waiting to be selec…
fix(codex): classify prolite as a weekly plan (lidge-jun#915)
fix(codex): share one window rule instead of maintaining a plan allow…
fix(codex): probe reset-derived cooldowns without waiting to be selec…
fix(codex): classify prolite as a weekly plan (lidge-jun#915)
fix(codex): share one window rule instead of maintaining a plan allow…
docs(triage): record the stack outcome and what the audit gates caught
Merge pull request lidge-jun#955 from lidge-jun/codex/915-cooldown-re…