Skip to content

ops(llm): OpenRouter has been quota-exhausted on every call since 2026-09-12 #523

Description

@FlyM1ss

Every platform-credits LLM call on prod attempts openrouter first, gets provider_quota_exhausted, and fails over to commonstack. From credit_llm_reservations, attempt 0 only:

date openrouter attempt 0 outcome
09-22 47 all quota-exhausted
09-20 191 all quota-exhausted
09-19 42 all quota-exhausted
09-18 243 all quota-exhausted
09-17 94 all quota-exhausted
09-16 69 all quota-exhausted
09-15 196 all quota-exhausted
09-14 43 all quota-exhausted
09-13 273 270 quota-exhausted, 2 settled, 1 invalid
09-12 142 118 quota-exhausted, 24 unavailable

1,313 quota-exhausted first attempts in 11 days, against 2 OpenRouter successes total. It has been failing on essentially every call since at least 2026-09-12.

Measured cost: a median 12.0s per call (attempt-0 → attempt-1 gap on run agent_20260922_202119_fc7bd724; n=47, min 11.0, max 14.0) — 16% of every decision bar, on every LLM backtest the platform runs.

It is invisible because PROVIDER_QUOTA_EXHAUSTED is in _PLATFORM_FAILOVER_CATEGORIES (infrastructure/llm/execution/service.py:61), so runs still succeed. This is the fail-closed-is-not-fail-visible shape from CLAUDE.md: a dead primary credential and a healthy one produce identical output.

  • Top up / rotate the OpenRouter credential — operational, no code change.
  • Warn when a provider's failover rate stays near 100% over a window. A per-call category cannot report a wholesale credential outage; that is what let this run for 11 days.

Adjacent, same data: commonstack answers response_invalid on 9/47 calls of that run (19%) and 37/191 on 09-20. Those bars fall back to rule-based and do not count toward H6 LLM coverage. Worth a look if it persists.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions