diff --git a/CLAUDE.md b/CLAUDE.md index 6c332417..712da9a5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -71,10 +71,17 @@ python dashboard/scripts/backtest_hourly_agent.py # main hourly agent backte - `ADMIN_BOOTSTRAP_SECRET` (optional, **minimum 32 characters**): shared secret for `POST /api/admin/bootstrap`, which promotes the **signed-in** caller to `role=admin`. One-shot: it refuses once any admin account exists (break-glass after that is SQL). A value shorter than `_BOOTSTRAP_MIN_LENGTH` (32) is refused as if unset — this secret grants admin with no account behind it and no lockout to hide behind, so its entropy is the only thing bounding a guesser, and that cannot be left to whoever filled in the Render field. **Unset, too weak, and wrong all answer identically (403)**, so a caller cannot learn whether a deployment is bootstrappable; the operator's signal is a server-log line (same call the repo makes for `LEADERBOARD_DAILY_REFRESH_SECRET`, which 401s either way). Wrong guesses are rate-limited three ways — per user, per client IP (5 / 15 min each) and a server-wide 20 / 15 min ceiling — but the global budget is consulted **after** the compare, so wrong guesses can never refuse a *correct* secret. Do not "tidy" that ordering back: checked first, 20 guesses a window (re-spent every window, from any account, and signup is open) locked the real operator out indefinitely, and the window it blocked is exactly the fresh-deploy window this route exists for. The secret is compared via SHA-256 + `secrets.compare_digest` — the hash is load-bearing because `compare_digest` raises `TypeError` on a non-ASCII `str` (a JSON body can send one) and runs in time proportional to the shorter operand, leaking the expected length. It does *not* raise on a length mismatch; unequal buffers just compare false. Set it in the **Render dashboard** for a fresh deploy (already set there), then leave it (it is inert after the first admin) or unset it. `tests/conftest.py` strips it so the suite sees a known-unset baseline. - `DEFAULT_MAX_CONCURRENT_BACKTESTS` (optional, default **5**, range 0–20): concurrent protocol runs an account gets before an admin edits its row. Nothing seeds a `user_entitlements` row at signup and nothing backfills, so this constant is the **live limit for every account that already exists** — it silently became everyone's quota on the deploy that shipped the entitlement plane. It matches `MAX_ACTIVE_RUNS_PER_AGENT` (5) for that reason: before the plane, an account's concurrency was bounded only by the per-agent cap times however many agents it owned, so a lower default demotes every multi-agent user at once with no remedy short of an admin. Lower it deliberately or not at all. A junk or out-of-range value falls back to 5 with a log line rather than raising at import. - `CREDITS_METERING_ENABLED` (optional) and `DEFAULT_CREDITS` (optional, default **100**, range 0–1,000,000): the **superseded** one-credit-per-run metering policy in `domain/entitlements/credits.py`. It is on no request path. `3eebb7da` *feat: add secure llm worker handoff* (2026-08-24) replaced it with the unified LLM execution layer in the next bullet; what survives is the module, its unit tests, and `GET /api/admin/stats` reporting `credits_metering_enabled` through `admin_users.py:179` — the only non-test import of `domain/entitlements` in the backend. **Arming the flag today moves one admin boolean and nothing else**: `authorize_llm_run` and `refund_llm_run` are called only by `tests/test_credit_metering.py`. `DEFAULT_CREDITS` still seeds `user_entitlements.credits`, an admin-granted allowance no live path spends (the LLM lane bills `credit_llm_usage_entries` instead). ⚠ **Do not read "only tests call it" as "spend is unmetered" and re-wire the debit.** That inference has already been drawn once, from exactly this evidence, and it is wrong — a control that was deleted is not a control that is absent until you have looked for its replacement. The retraction is recorded in `docs/superpowers/plans/2026-09-11-backtest-p1-p2-plan.md`. `tests/conftest.py` strips both. -- **LLM backtest billing — the live path, and it has no env var of its own.** `POST /backtest/run` with `decision_source='llm'` on the pipeline runtime is **sign-in-only** (401) and **requires `billing_mode`** (422), one of `byok` or `platform_credits` (`api/routers/backtests.py:2755-2841`). The route preflights the credential and the provider/model route, then hands the worker a signed, secret-free envelope on **stdin** (`create_execution_handoff` → `consume_execution_handoff`, `dashboard/scripts/backtest_hourly_agent.py:170`) — a raw key never reaches subprocess arguments or the environment. `infrastructure/llm/execution/service.py` then bills **per call, by usage, not per run**: `_execute_platform` reserves a ceiling as a `credit_llm_reservations` row, `_settle` debits the provider-reported tokens into `credit_llm_usage_entries` at a pricing snapshot, and `_release_after_failure` hands the hold back. Not `credit_ledger_entries` — that table's `entry_type` CHECK admits only `purchase` / `refund` / `admin_grant_assign` / `admin_grant_reclaim`, so it can hold neither row. (This bullet named the wrong table until 2026-09-14.) Settlement **fails closed** — a status other than `settled`, or an amount disagreeing with the computed one, raises `BILLING_FAILED` rather than letting the run pass as billed. BYOK never touches the ATL ledger: `finalize_run` is an explicit no-op for that lane, so a BYOK run cannot quietly spend Credits. The **parent** repeats `finalize_run` in its `finally` (`backtests.py:1627-1634`) because the child's own cleanup does not run when the subprocess is killed — by the timeout, or by the cancel route added in #460 — and a killed child would otherwise strand a reservation. Design: `docs/superpowers/plans/2026-08-24-unified-llm-execution-layer.md`. A run killed by that timeout now finalizes as its own terminal outcome rather than a generic exception: `/backtest/status` answers `timed_out` with a `timeout` block naming the budget, and the analytics event stays `backtest_failed` with `error_category="run_timeout"`. ⚠ **The lane and the settled spend (`credits_service.sum_run_llm_spend`) in that block are owner-only** — `_status_snapshot_for` strips `spent_micro`/`model_calls`/`billing_mode` for any caller who does not satisfy `_slot_owned_by`. The read rule the route authorizes with (`_slot_visible_to`) is deliberately weaker than ownership: for a built-in agent it admits anyone holding the slot's `session_id`, which the *running* branch of this same route publishes to every visitor, so without the strip an anonymous caller could read a signed-in stranger's Credits charge out of `_recent_slots`. Do not widen it back by reusing `_slot_visible_to` for the whole payload — the outcome is public on purpose (every poller branches on `timed_out`) and only the account-level facts are withheld. The spend lookup also runs on a daemon thread joined with `TIMEOUT_SPEND_LOOKUP_SECONDS`: `except Exception` answers a query that raises, not one that never returns, and inline it blocked finalize and stranded a concurrency slot for a dead child. Design: `docs/superpowers/specs/2026-09-14-backtest-timeout-outcome-design.md`. +- **LLM backtest billing — the live path, and it has no env var of its own.** `POST /backtest/run` with `decision_source='llm'` on the pipeline runtime is **sign-in-only** (401) and **requires `billing_mode`** (422), one of `byok` or `platform_credits` (`api/routers/backtests.py:2755-2841`). The route preflights the credential and the provider/model route, then hands the worker a signed, secret-free envelope on **stdin** (`create_execution_handoff` → `consume_execution_handoff`, `dashboard/scripts/backtest_hourly_agent.py:170`) — a raw key never reaches subprocess arguments or the environment. `infrastructure/llm/execution/service.py` then bills **per call, by usage, not per run**: `_execute_platform` reserves a ceiling as a `credit_llm_reservations` row, `_settle` debits the provider-reported tokens into `credit_llm_usage_entries` at a pricing snapshot, and `_release_after_failure` hands the hold back. Not `credit_ledger_entries` — that table's `entry_type` CHECK admits only `purchase` / `refund` / `admin_grant_assign` / `admin_grant_reclaim`, so it can hold neither row. (This bullet named the wrong table until 2026-09-14.) Settlement **fails closed** — a status other than `settled`, or an amount disagreeing with the computed one, raises `BILLING_FAILED` rather than letting the run pass as billed. BYOK never touches the ATL ledger: `finalize_run` is an explicit no-op for that lane, so a BYOK run cannot quietly spend Credits. The **parent** repeats `finalize_run` in its `finally` (`backtests.py:1627-1634`) because the child's own cleanup does not run when the subprocess is killed — by the timeout, or by the cancel route added in #460 — and a killed child would otherwise strand a reservation. Design: `docs/superpowers/plans/2026-08-24-unified-llm-execution-layer.md`. ATL Credits try platform providers in `ATL_PLATFORM_PROVIDER_ORDER` order (below; CommonStack first by default) and fail over on `_PLATFORM_FAILOVER_CATEGORIES`. The first `PROVIDER_QUOTA_EXHAUSTED` a platform provider raises in a process prints `ERROR: llm.platform_quota_exhausted provider= fallback=`, once and flushed, and the parent's `_drain_stream` echoes every child line starting `ERROR: llm.` to its own stdout as it arrives, through `_redact_credentials` like the dump — the timeout path never dumps the child's capture and a long run's middle is elided from it, so without the relay the line would not reach the service log. The end-of-run dump drops those lines (`_without_relayed_lines`) so each reaches the log exactly once; the capture itself keeps them, because it also feeds the failure summary. It exists because CommonStack has no balance endpoint, so a failed call is the only way to see it drain. BYOK never prints it. A run killed by that timeout now finalizes as its own terminal outcome rather than a generic exception: `/backtest/status` answers `timed_out` with a `timeout` block naming the budget, and the analytics event stays `backtest_failed` with `error_category="run_timeout"`. ⚠ **The lane and the settled spend (`credits_service.sum_run_llm_spend`) in that block are owner-only** — `_status_snapshot_for` strips `spent_micro`/`model_calls`/`billing_mode` for any caller who does not satisfy `_slot_owned_by`. The read rule the route authorizes with (`_slot_visible_to`) is deliberately weaker than ownership: for a built-in agent it admits anyone holding the slot's `session_id`, which the *running* branch of this same route publishes to every visitor, so without the strip an anonymous caller could read a signed-in stranger's Credits charge out of `_recent_slots`. Do not widen it back by reusing `_slot_visible_to` for the whole payload — the outcome is public on purpose (every poller branches on `timed_out`) and only the account-level facts are withheld. The spend lookup also runs on a daemon thread joined with `TIMEOUT_SPEND_LOOKUP_SECONDS`: `except Exception` answers a query that raises, not one that never returns, and inline it blocked finalize and stranded a concurrency slot for a dead child. Design: `docs/superpowers/specs/2026-09-14-backtest-timeout-outcome-design.md`. +- `ATL_PLATFORM_PROVIDER_ORDER` (optional, default **`commonstack,openrouter`**): the order in which ATL Credits (`billing_mode=platform_credits`) try platform providers. `platform_provider_order()` in `domain/model_providers/service.py` reads it per call, and both `resolve_platform_execution_candidates` and `list_execution_options` use it. + - **Why CommonStack is first:** its prepaid balance should be spent before OpenRouter's (#522). The order was hard-coded OpenRouter-first until 2026-09, and CommonStack carried all the traffic only because OpenRouter's key was quota-dead (#523). Topping that key up would have silently stranded CommonStack's balance. + - **Leaving a provider out** removes it from routing entirely. That is the no-deploy way to pull a drained lane. An explicit `provider_id` in the request body is honoured first **only when it is in the order** — otherwise an API caller naming the pulled lane would reopen it (the dashboard never sends one for ATL Credits). An admin-added platform lane is therefore reachable only once it is listed here. If nothing routable is left, the route answers 422 rather than guessing. + - **An invalid token** rejects the whole value, prints one `WARNING`, and falls back to the default. A typo'd order must not half-apply. That includes a well-formed id the provider registry does not know (`commonstak`): both call sites (`resolve_platform_execution_candidates`, `list_execution_options`) pass the registry's ids in, because a one-letter typo passes the syntax check and would otherwise silently drop the lane it misnames. Do not add a third caller without them. + - **Options order:** in `list_execution_options`, only **platform-only** lanes (not `byok_enabled`, i.e. CommonStack) move — to the end, in this order. Every BYOK-capable provider keeps repository order, OpenRouter included even though it is also a platform lane: `app.js` takes `providers[0]` of the BYOK list as the default, and ranking OpenRouter moved it behind any BYOK provider named after it. The list order is cosmetic for ATL Credits (model dropdown order); routing order is decided by the route. + - **Scope:** BYOK ignores it. It is read **only in the route process**: the handoff carries the ordered tuple whole and `_execute_with_platform_failover` treats it as authoritative. The worker used to widen a lone `("openrouter",)` to OpenRouter → CommonStack — hard-coded OpenRouter-first, and reading the env without the registry, so it re-added a lane the route had pulled or rejected. A lone candidate is what the route decided; do not reintroduce worker-side routing. + - `tests/conftest.py` strips it. Design: `docs/superpowers/specs/2026-09-23-llm-backtest-step-latency-design.md` §5. - `ATL_STRIPE_TEST_BILLING_ENABLED` (optional, **strict opt-in**, default off) plus `STRIPE_SECRET_KEY` and `STRIPE_WEBHOOK_SECRET` (both required once enabled; `PUBLIC_APP_URL` supplies the Checkout return URLs): the Stripe **Test Mode** Credits purchase flow (`domain/credits/*`, `api/routers/credits.py`, `/api/webhooks/stripe`). Enabled with any value missing, `BillingConfig.ready` is false and every billing route answers **503** rather than half-working. A `sk_live_` key is refused outright (`BillingConfigurationError`) — this release has no Live Mode. That error subclasses `ValueError`, so it must stay mapped ahead of the generic `ValueError` arm in `_raise_billing_http_error` or an *operator* misconfiguration gets reported as a **422 blaming the caller**. **Credits purchased here are spendable now, and this bullet used to say the opposite**: they land in `credit_ledger_entries`, and `get_balance_projection` sums that table with `credit_llm_usage_entries`, which is where the LLM execution layer settles (see the LLM backtest billing bullet above) — so a purchase raises the very balance an LLM run spends down, through a different table rather than the same one, and `grant_default_signup_credits` seeds new accounts from `api/auth.py:205`. The claim that they bought nothing was true of the Stripe release and was pinned by `test_balance_does_not_claim_credits_are_spendable`; that test no longer exists, which is why nothing failed when the statement stopped being true. Self-service refunds remain out of scope. ⚠ **The purchase ledger rides `USERS_DATABASE_URL`, not its own var** (`_build_credits_store()`): a deployment that set only `CONTENT_DATABASE_URL`/`AGENT_RUNS_DATABASE_URL` silently stores **real-money purchase records on ephemeral SQLite** and destroys them on the next redeploy. The one warning is a boot line — `credits_store backend: postgres (/)` or `credits_store backend: sqlite (ephemeral on Render)` — so check it after any billing deploy. The webhook is the only unauthenticated route in the feature: its body is capped (`_MAX_WEBHOOK_BODY_BYTES`, 256 KiB) because signature verification needs the raw bytes and therefore *must* read before it can authenticate, and it carries its own flood limiter. Every non-settling webhook outcome prints a `[credits] ERROR|WARN` line — deliberately unconditional, because the failure it exists to catch (an upstream field rename rejecting every payment while customers are charged) is wholesale, and the route answers 200 either way so Stripe's dashboard shows "delivered". - `MAX_ACTIVE_DASHBOARD_BACKTESTS` (optional, default **5**, 0 disables all dashboard backtests): server-wide ceiling on concurrent `POST /backtest/run` runs, the outer bound on the per-owner slot ledger in `api/routers/backtests.py`. Deliberately equal to `DEFAULT_MAX_CONCURRENT_BACKTESTS` so one default-entitlement account can actually reach its own quota, and no higher: a dashboard backtest is a **subprocess** — unlike the protocol surfaces' in-process step sessions, whose global caps are 50/100 — and each pins a loaded bar window. That was sized against a 512MB free instance; since the 2026-09-11 move to Standard (2GB) RAM is no longer what holds it at 5. It is also — and now principally — the LLM **spend** bound, because before this module grew slots the runner was single-flight and the ceiling was exactly 1; a large value here multiplies operator API cost by the same factor. The per-owner cap above it is keyed on the **caller's** browser session (never on the built-in agent's session the results file under, which would put every anonymous visitor in one bucket), so like every session-keyed cap in this repo it is an incentive fix rather than a bound against someone rotating the header — this global number is what actually holds. A junk or negative value falls back to 5 with a log line rather than raising at import; **an unparseable value used to kill app boot**, since it was read with a bare `int()` at module scope. -- `ATL_BACKTEST_WORKER` (**never set by an operator**): `run_backtest_background` sets it to `1` in a dashboard backtest child's environment and nowhere else. Every Postgres store twin that owns a schema — **eleven of the twelve pairs in `tests/test_store_twin_parity.py::_TWINS`**; the twelfth, `PostgresValueAnalyticsStore`, composes the other stores and has no `_init_schema`, which that file's `_NO_OWN_DDL_TWINS` already records — routes its constructor's DDL through `db_url.init_schema_unless_worker`, which skips it on the literal `1` and prints ` backend: schema init skipped (backtest worker)`, and otherwise runs it, times it into `db_url.schema_init_seconds()` and prints ` backend: schema init s`. The parent ran that DDL at import, and the child was paying a pooled Neon checkout plus a batch of round trips per store before reading a bar: on today's import graph six twins construct in a child — `PostgresBacktestDatabase` (55 statements), `PostgresModelProviderStore` (18), `PostgresAnalyticsStore` (16), `PostgresAgentStore` (14), `PostgresCreditsStore` (3), `BrokerConnectionStorePostgres` (1), 107 statements over six checkouts. A warm schema does not shrink that: 72 of the 107 are `ADD COLUMN IF NOT EXISTS` and the other five `ALTER`s are constraint drops and re-adds, so they round-trip whether or not the column is there. ⚠ **`_init_schema` is not purely DDL, so the guard skips more than its name says.** Three of the eleven twins run data statements inside it — `PostgresCreditsStore` seeds the `default` grant pool and backfills settled reservation amounts plus ledger bucket/operation columns, `PostgresModelProviderStore` scrubs `api_key_enc` on revoked credentials and seeds `SEEDED_PROVIDERS`, `PostgresUserStore` repairs `users.user_group` values outside the allowed set. A child skips those too, and they are safe to skip **only because the parent applies them at boot before it spawns anything** — a property of how dashboard backtests are launched rather than of this code, so a worker that ever ran without a parent ahead of it (a bare CLI run with the flag exported, a future worker-only service) would silently not have them. `test_backtest_worker_schema_skip.py` holds that set as a declared registry and fails in both directions, so a fourth cannot arrive unnoticed and a stale entry cannot linger; read it before adding a migration to any guarded `_init_schema`. **Both log directions are deliberate:** the timed line is the parent's own boot cost, which is the only pre-change baseline the deploy that removes the child's copy can still produce, and a silent skip would look exactly like a build with no guard. ⚠ **All eleven are guarded, not just those six, on purpose.** Which stores a child imports is an accident of the import graph, and the accident arrives by **two independent routes**. The script imports `db` deliberately (`backtest_hourly_agent.py:51`) and then reaches `credits_store` and `model_provider_store` through the two service imports at `:188-189` (`credits/service.py:27`, `model_providers/service.py:38`), with `model_providers/repository.py:13-14` pulling in `agents/repository.py` and `brokers/repository.py` — five of the six twins, with `domain/analytics/service.py` never imported. Only the sixth, `analytics_store`, needs the analytics chain those same service modules also pull in (`domain/analytics/instrumentation.py:15` → `analytics/service.py:17`, whose module-level `analytics_service = _build_analytics_service()` at `:253` then constructs a value store that resolves `credits_store`, `model_provider_store`, `agent_store` and `run_store` as constructor-time defaults — the first three already built by route one, `run_store` (like `analytics_store`) built here for the first time). `run_store` does not move the count: it is a seventh module-level singleton (`domain/runs/repository.py:469`), not a seventh twin — `domain/runs/` has no `repository_postgres.py`, so protocol runs are SQLite either way and nothing here guards them. Neither route is legible from the child's own import block, which names `db` and two service modules and nothing else, so one new module-level import in the engine's reach would otherwise re-add an unguarded payer with nothing going red. `test_backtest_worker_schema_skip.py` pins the invariant that survives that, in the AST rather than as a substring: no non-test `*_postgres.py` calls **any** `self._…schema…()` method straight from `__init__`, any twin defining a schema method names `init_schema_unless_worker`, and guarded ∪ exempt equals `_TWINS`. It matched the literal `"self._init_schema()"` until 2026-09-21, which is a check on one spelling — a twin calling `self._ensure_schema()` passed it while doing the forbidden thing. One shared helper rather than eleven copies, for the reason `db_url.py`'s header gives about its scrubber — an inverted copy fails in the *parent*, as an `UndefinedTable` on prod from a store nobody edited, and a dropped log line makes "skipped" and "never constructed" the same empty stdout. Named for the process role, not the DDL, so nobody exports it globally and later child-only behaviour has a home. The child also publishes pre-loop **phases** to its progress file (`engine.py:PROGRESS_PHASES`: `starting`, `loading_bars`, `indicators`, `first_decision`, `running`, `saving`), which `/backtest/status` turns into the card's sentence and which double as the start-up measurement — read `phases[]` off a run's progress file for the per-phase timings, or the child's `⏱ phase …` stdout lines out of the service log, which is the copy that survives (the parent unlinks the progress file when the run ends). Every finished entry carries `duration_seconds`, measured on `time.monotonic()` — do **not** re-derive a duration by differencing `started_at`/`ended_at`, which are deliberately wall-clock stamps kept for correlating against a log line: `time.time()` is not monotonic, so an NTP step between two reads distorts the interval and a backward step yields a negative one. ⚠ `starting` is the one phase whose duration *is* that wall-clock difference, and it cannot be otherwise — it spans the parent→child boundary (it opens at the parent's `--launched-at` and closes in the child) and `monotonic()` has a **per-process epoch**, so the parent's reading and the child's are not comparable at all. `_init_progress_phases` leaves the steady mark `None` to say exactly that, and `_set_progress_phase` reads `None` as "fall back to the wall clock"; every later phase is entirely in-child and uses the steady clock. The `starting` entry additionally carries `child_entered_at`, `imports_done_at` and `schema_init_seconds`, splitting that one interval into spawn, imports-including-stores, of-which-DDL and preflight: one phase name, four numbers, because a single figure covering all four cannot say which of them an optimisation moved. ⚠ **That exemption covers the phase's total and exactly one of those four splits — do not read it as covering the record.** Only `spawn+interpreter` (`child_entered_at - started_at`) reaches back into the parent; `imports+stores` and `preflight` both begin and end inside the child, so both are measured on the steady clock and published as `imports_seconds` and `preflight_seconds` beside the stamps. The launch script takes a second, steady pair for this (`CHILD_ENTERED_STEADY`/`IMPORTS_DONE_STEADY`, handed over in `startup_clock`) and the engine closes `preflight` against its own `steady_clock()` read, which is the same clock in the same process. Those two marks are deliberately **not** published — a raw `monotonic()` reading has a per-process epoch and is meaningless in a file — so what `phases[]` gains is the two durations, not the marks. The wall-clock fallback for a caller that hands over only the three stamps is kept and is *silent*, so it is pinned in the source rather than at runtime: `test_backtest_launch_phases.py` AST-checks that the script still hands over the steady pair, that both are `time.monotonic()` reads, and that they bracket the imports. Shipping the #509 fix with these two differences still in the breakdown line is the mistake that guard exists to prevent repeating. `loading_bars` carries `fetch_seconds` for the same reason and prints `fetch s | aggregate+verify s` on the transition out: the phase is one name, but with the bar cache warm the residual **is** the aggregation cost, measured rather than inferred — which is what decides whether caching the aggregated output is worth a second change. `record_phase_metric` attaches a number to whichever phase is open — and drops it when none is — and extras are cleared on every transition, so a figure can never be reported against the wrong phase. The startup clock is seeded only when `--launched-at` actually opens `starting`; a bare CLI run passes the clock without a launch time, and those keys must not surface on `loading_bars`. `tests/conftest.py` strips the flag. +- `ATL_BACKTEST_WORKER` (**never set by an operator**): `run_backtest_background` sets it to `1` in a dashboard backtest child's environment and nowhere else. Every Postgres store twin that owns a schema — **eleven of the twelve pairs in `tests/test_store_twin_parity.py::_TWINS`**; the twelfth, `PostgresValueAnalyticsStore`, composes the other stores and has no `_init_schema`, which that file's `_NO_OWN_DDL_TWINS` already records — routes its constructor's DDL through `db_url.init_schema_unless_worker`, which skips it on the literal `1` and prints ` backend: schema init skipped (backtest worker)`, and otherwise runs it, times it into `db_url.schema_init_seconds()` and prints ` backend: schema init s`. The parent ran that DDL at import, and the child was paying a pooled Neon checkout plus a batch of round trips per store before reading a bar: on today's import graph six twins construct in a child — `PostgresBacktestDatabase` (55 statements), `PostgresModelProviderStore` (18), `PostgresAnalyticsStore` (16), `PostgresAgentStore` (14), `PostgresCreditsStore` (3), `BrokerConnectionStorePostgres` (1), 107 statements over six checkouts. A warm schema does not shrink that: 72 of the 107 are `ADD COLUMN IF NOT EXISTS` and the other five `ALTER`s are constraint drops and re-adds, so they round-trip whether or not the column is there. ⚠ **`_init_schema` is not purely DDL, so the guard skips more than its name says.** Three of the eleven twins run data statements inside it — `PostgresCreditsStore` seeds the `default` grant pool and backfills settled reservation amounts plus ledger bucket/operation columns, `PostgresModelProviderStore` scrubs `api_key_enc` on revoked credentials, seeds `SEEDED_PROVIDERS` and backfills the CommonStack model allowlist into an already-seeded row, `PostgresUserStore` repairs `users.user_group` values outside the allowed set. A child skips those too, and they are safe to skip **only because the parent applies them at boot before it spawns anything** — a property of how dashboard backtests are launched rather than of this code, so a worker that ever ran without a parent ahead of it (a bare CLI run with the flag exported, a future worker-only service) would silently not have them. `test_backtest_worker_schema_skip.py` holds that set as a declared registry and fails in both directions, so a fourth cannot arrive unnoticed and a stale entry cannot linger; read it before adding a migration to any guarded `_init_schema`. **Both log directions are deliberate:** the timed line is the parent's own boot cost, which is the only pre-change baseline the deploy that removes the child's copy can still produce, and a silent skip would look exactly like a build with no guard. ⚠ **All eleven are guarded, not just those six, on purpose.** Which stores a child imports is an accident of the import graph, and the accident arrives by **two independent routes**. The script imports `db` deliberately (`backtest_hourly_agent.py:51`) and then reaches `credits_store` and `model_provider_store` through the two service imports at `:188-189` (`credits/service.py:27`, `model_providers/service.py:38`), with `model_providers/repository.py:13-14` pulling in `agents/repository.py` and `brokers/repository.py` — five of the six twins, with `domain/analytics/service.py` never imported. Only the sixth, `analytics_store`, needs the analytics chain those same service modules also pull in (`domain/analytics/instrumentation.py:15` → `analytics/service.py:17`, whose module-level `analytics_service = _build_analytics_service()` at `:253` then constructs a value store that resolves `credits_store`, `model_provider_store`, `agent_store` and `run_store` as constructor-time defaults — the first three already built by route one, `run_store` (like `analytics_store`) built here for the first time). `run_store` does not move the count: it is a seventh module-level singleton (`domain/runs/repository.py:469`), not a seventh twin — `domain/runs/` has no `repository_postgres.py`, so protocol runs are SQLite either way and nothing here guards them. Neither route is legible from the child's own import block, which names `db` and two service modules and nothing else, so one new module-level import in the engine's reach would otherwise re-add an unguarded payer with nothing going red. `test_backtest_worker_schema_skip.py` pins the invariant that survives that, in the AST rather than as a substring: no non-test `*_postgres.py` calls **any** `self._…schema…()` method straight from `__init__`, any twin defining a schema method names `init_schema_unless_worker`, and guarded ∪ exempt equals `_TWINS`. It matched the literal `"self._init_schema()"` until 2026-09-21, which is a check on one spelling — a twin calling `self._ensure_schema()` passed it while doing the forbidden thing. One shared helper rather than eleven copies, for the reason `db_url.py`'s header gives about its scrubber — an inverted copy fails in the *parent*, as an `UndefinedTable` on prod from a store nobody edited, and a dropped log line makes "skipped" and "never constructed" the same empty stdout. Named for the process role, not the DDL, so nobody exports it globally and later child-only behaviour has a home. The child also publishes pre-loop **phases** to its progress file (`engine.py:PROGRESS_PHASES`: `starting`, `loading_bars`, `indicators`, `first_decision`, `running`, `saving`), which `/backtest/status` turns into the card's sentence and which double as the start-up measurement — read `phases[]` off a run's progress file for the per-phase timings, or the child's `⏱ phase …` stdout lines out of the service log, which is the copy that survives (the parent unlinks the progress file when the run ends). Every finished entry carries `duration_seconds`, measured on `time.monotonic()` — do **not** re-derive a duration by differencing `started_at`/`ended_at`, which are deliberately wall-clock stamps kept for correlating against a log line: `time.time()` is not monotonic, so an NTP step between two reads distorts the interval and a backward step yields a negative one. ⚠ `starting` is the one phase whose duration *is* that wall-clock difference, and it cannot be otherwise — it spans the parent→child boundary (it opens at the parent's `--launched-at` and closes in the child) and `monotonic()` has a **per-process epoch**, so the parent's reading and the child's are not comparable at all. `_init_progress_phases` leaves the steady mark `None` to say exactly that, and `_set_progress_phase` reads `None` as "fall back to the wall clock"; every later phase is entirely in-child and uses the steady clock. The `starting` entry additionally carries `child_entered_at`, `imports_done_at` and `schema_init_seconds`, splitting that one interval into spawn, imports-including-stores, of-which-DDL and preflight: one phase name, four numbers, because a single figure covering all four cannot say which of them an optimisation moved. ⚠ **That exemption covers the phase's total and exactly one of those four splits — do not read it as covering the record.** Only `spawn+interpreter` (`child_entered_at - started_at`) reaches back into the parent; `imports+stores` and `preflight` both begin and end inside the child, so both are measured on the steady clock and published as `imports_seconds` and `preflight_seconds` beside the stamps. The launch script takes a second, steady pair for this (`CHILD_ENTERED_STEADY`/`IMPORTS_DONE_STEADY`, handed over in `startup_clock`) and the engine closes `preflight` against its own `steady_clock()` read, which is the same clock in the same process. Those two marks are deliberately **not** published — a raw `monotonic()` reading has a per-process epoch and is meaningless in a file — so what `phases[]` gains is the two durations, not the marks. The wall-clock fallback for a caller that hands over only the three stamps is kept and is *silent*, so it is pinned in the source rather than at runtime: `test_backtest_launch_phases.py` AST-checks that the script still hands over the steady pair, that both are `time.monotonic()` reads, and that they bracket the imports. Shipping the #509 fix with these two differences still in the breakdown line is the mistake that guard exists to prevent repeating. `loading_bars` carries `fetch_seconds` for the same reason and prints `fetch s | aggregate+verify s` on the transition out: the phase is one name, but with the bar cache warm the residual **is** the aggregation cost, measured rather than inferred — which is what decides whether caching the aggregated output is worth a second change. `record_phase_metric` attaches a number to whichever phase is open — and drops it when none is — and extras are cleared on every transition, so a figure can never be reported against the wrong phase. The startup clock is seeded only when `--launched-at` actually opens `starting`; a bare CLI run passes the clock without a launch time, and those keys must not surface on `loading_bars`. `tests/conftest.py` strips the flag. - `MAX_AI_HEDGE_FUND_TRADING_DAYS` (optional, default **10**, range 0–60, **0 disables the hosted runtime**): the pre-flight bound `POST /backtest/run` applies to an AI Hedge Fund window (`_enforce_ai_hedge_fund_window` in `api/routers/backtests.py`). The hosted runtime spends one upstream **subprocess per trading day**, each loading its own lookback window beside a parent already holding uvicorn, FastAPI and the Postgres pools, and the kernel's victim is the whole web process — one backtest denies service to everyone (issue #308). An over-long window answers **422** naming the bound and the requested size (the caller can shorten it); `0` answers **503** (the deployment's own configuration, nothing the caller can change). Measured in trading days via `_estimated_decision_days`, the same upper bound `_backtest_subprocess_timeout` sizes the parent budget from, so the two cannot disagree about how big a window is. Junk/negative/over-ceiling values log and fall back rather than raising at import — a bare `int()` at module scope in this very module once killed app boot. **The default stays 10 even though prod is now 2GB** (2026-09-11): nothing has ever measured one child's resident set, so 10 was a guess against the old ceiling and a larger guess against a larger ceiling is the same mistake with more RAM behind it. Profile a run, then raise it from the Render dashboard — no deploy needed. - `MAX_LEGACY_ACTIVE_PER_SESSION` / `MAX_LEGACY_ACTIVE_GLOBAL` (optional, defaults **5** / **50**, 0 disables): concurrency budgets for the legacy `/api/v1/backtest/*` surface. That surface authenticates nothing — `_require_session` accepts any non-empty `X-Session-Id` — and writes **no `protocol_runs` row**, so the per-agent, per-account and global protocol caps are all blind to its runs; before these it was the one unbounded path into the same engine. The per-session budget bounds a looping client (its key is caller-chosen, so it is not a bound against someone who rotates it); the global one is the memory bound that holds regardless, since every live session pins a loaded bar window. Enforced via `start_backtest(enforce_session_cap=True)`, which **only** the legacy route passes — the protocol surfaces reach the same function through `run_service.create_run`, which has already applied its three caps. - `ANALYTICS_DAILY_JOB_INTERVAL_SECONDS` (optional, default **300**, range 5–3600): how often the analytics daily-facts worker (`domain/analytics/daily_job.py`, started from `app.py` beside the run reaper) wakes to ask whether yesterday's `user_daily_facts` row set is due. It runs on **its own thread**, not the 60-second reaper tick — the reaper exists to keep run heartbeats fresh against `RUN_HEARTBEAT_STALE_SECONDS`, and a whole-population batch across three databases on that thread would let a slow analytics night mark live runs as orphaned (design D23). An idle wake costs exactly one store call (the refused day claim on `analytics_projection_jobs`), so the interval buys freshness after midnight, not cost. Junk or out-of-range values log and fall back to 300 rather than raising at import — a bare `int()` at module scope has killed app boot in this repo before. `tests/conftest.py` strips it. The worker also runs the idempotent `user_activity` seed and the eight-week history copy (`domain/analytics/facts_migration.py`) once before its first tick, and owns `rollup_day` and the analytics retention coordinator; the reaper keeps only the two throttled snapshot repairs in `maintenance.py` until PR B deletes them. Design: `docs/superpowers/specs/2026-09-15-admin-layer-redesign-design.md` §6.9, §6.12. diff --git a/dashboard/backend/api/routers/backtests.py b/dashboard/backend/api/routers/backtests.py index 32d32bd6..16b9f964 100644 --- a/dashboard/backend/api/routers/backtests.py +++ b/dashboard/backend/api/routers/backtests.py @@ -1765,6 +1765,7 @@ def run_backtest_background( stdin_payload=execution_handoff_payload or "", timeout=subprocess_timeout, live_run_id=resolved_live_run_id, + redact_secret=financial_datasets_api_key, ) # Print script output for debugging @@ -1775,14 +1776,18 @@ def run_backtest_background( # byte the child ever wrote held in parent RAM for the whole run; the # head this comment used to promise (universe, decision source, FX # bootstrap) is exactly what the head half of that buffer keeps. - if result.stdout: + # Relayed ERROR: llm. lines were already printed live by _drain_stream; + # dumping them again would count every quota event twice in the log. + dumped_stdout = _without_relayed_lines(result.stdout or "") + dumped_stderr = _without_relayed_lines(result.stderr or "") + if dumped_stdout: print( - f"STDOUT:\n{_redact_credentials(result.stdout, financial_datasets_api_key)}", + f"STDOUT:\n{_redact_credentials(dumped_stdout, financial_datasets_api_key)}", flush=True, ) - if result.stderr: + if dumped_stderr: print( - f"STDERR:\n{_redact_credentials(result.stderr, financial_datasets_api_key)}", + f"STDERR:\n{_redact_credentials(dumped_stderr, financial_datasets_api_key)}", flush=True, ) print(f"Return code: {result.returncode}", flush=True) @@ -2544,7 +2549,27 @@ def text(self) -> str: ) -def _drain_stream(stream: Any, capture: _BoundedStreamCapture) -> None: +_RELAYED_CHILD_LINE_PREFIX = "ERROR: llm." + + +def _without_relayed_lines(text: str) -> str: + """Drop the lines ``_drain_stream`` already echoed to the service log. + + Applied to the end-of-run dump only, so a relayed line reaches the log + once. The capture itself keeps them: it also feeds the failure summary. + """ + return "".join( + line + for line in text.splitlines(keepends=True) + if not line.startswith(_RELAYED_CHILD_LINE_PREFIX) + ) + + +def _drain_stream( + stream: Any, + capture: _BoundedStreamCapture, + redact_secret: Optional[str] = None, +) -> None: """Copy one child stream into a bounded capture until EOF. This is what makes ``Popen`` + ``wait`` safe: without a reader the child @@ -2555,6 +2580,16 @@ def _drain_stream(stream: Any, capture: _BoundedStreamCapture) -> None: try: for line in iter(stream.readline, ""): capture.feed(line) + if line.startswith(_RELAYED_CHILD_LINE_PREFIX): + # Echoed live, not left to the capture: the timeout path never + # dumps it, a normal exit dumps it only when the run ends, and + # a long run's middle is elided. Redacted like the dump, since + # the prefix is all that selects a line for this path. + print( + _redact_credentials(line, redact_secret), + end="", + flush=True, + ) except (OSError, ValueError): # The pipe was closed under us, which is the kill path doing its job. # Whatever was read before that still stands and is still worth logging. @@ -2586,6 +2621,7 @@ def _run_backtest_subprocess( stdin_payload: str, timeout: int, live_run_id: Optional[str], + redact_secret: Optional[str] = None, ) -> _BacktestSubprocessOutcome: """Run the backtest child, draining its output into bounded buffers. @@ -2632,10 +2668,14 @@ def _run_backtest_subprocess( stderr_capture = _BoundedStreamCapture() readers = [ _StreamReaderThread( - target=_drain_stream, args=(process.stdout, stdout_capture), daemon=True + target=_drain_stream, + args=(process.stdout, stdout_capture, redact_secret), + daemon=True, ), _StreamReaderThread( - target=_drain_stream, args=(process.stderr, stderr_capture), daemon=True + target=_drain_stream, + args=(process.stderr, stderr_capture, redact_secret), + daemon=True, ), ] for reader in readers: diff --git a/dashboard/backend/domain/model_providers/repository.py b/dashboard/backend/domain/model_providers/repository.py index 58a19cb2..64b6e39b 100644 --- a/dashboard/backend/domain/model_providers/repository.py +++ b/dashboard/backend/domain/model_providers/repository.py @@ -19,7 +19,9 @@ CredentialNotFoundError, CredentialOwnershipError, ProviderNotFoundError, + COMMONSTACK_ALLOWLIST_BACKFILLS, SEEDED_PROVIDERS, + commonstack_allowlist_backfill, deserialize_capabilities, serialize_capabilities, validate_adapter_type, @@ -156,6 +158,7 @@ def _init_schema(self) -> None: ), ) self._migrate_legacy_openrouter_platform_flag(conn) + self._migrate_commonstack_allowlist(conn) conn.commit() conn.close() @@ -205,6 +208,40 @@ def _migrate_legacy_openrouter_platform_flag(conn: sqlite3.Connection) -> None: (migration_id, _utcnow_iso()), ) + @staticmethod + def _migrate_commonstack_allowlist(conn: sqlite3.Connection) -> None: + """Backfill newly verified CommonStack models into a seeded row, once. + + Each backfill appends only the ids it introduced, and is recorded even + when nothing changed, so an admin who removes one of those models -- + before or after it runs -- is not overridden on a later boot. + """ + + for migration_id, model_ids in COMMONSTACK_ALLOWLIST_BACKFILLS: + if conn.execute( + "SELECT 1 FROM model_provider_migrations WHERE migration_id = ?", + (migration_id,), + ).fetchone(): + continue + now = _utcnow_iso() + provider = conn.execute( + "SELECT capabilities_json FROM provider_registry WHERE provider_id = 'commonstack'" + ).fetchone() + updated = ( + commonstack_allowlist_backfill(provider["capabilities_json"], model_ids) + if provider + else None + ) + if updated is not None: + conn.execute( + "UPDATE provider_registry SET capabilities_json = ?, updated_at = ? WHERE provider_id = 'commonstack'", + (updated, now), + ) + conn.execute( + "INSERT INTO model_provider_migrations (migration_id, applied_at) VALUES (?, ?)", + (migration_id, now), + ) + @staticmethod def _ensure_user_credential_columns(conn: sqlite3.Connection) -> None: columns = { diff --git a/dashboard/backend/domain/model_providers/repository_common.py b/dashboard/backend/domain/model_providers/repository_common.py index 44e693e0..dc7f463d 100644 --- a/dashboard/backend/domain/model_providers/repository_common.py +++ b/dashboard/backend/domain/model_providers/repository_common.py @@ -133,6 +133,17 @@ def secret_fingerprint(secret: str) -> str: "anthropic/claude-sonnet-4-6", "deepseek/deepseek-v4-pro", "qwen/qwen3.7-plus", + "anthropic/claude-haiku-4-5", +) + +# The seed below is ``ON CONFLICT DO NOTHING``, so an id appended to the +# allowlist above never reaches a deployment whose row already exists. Each +# addition therefore ships a one-shot backfill naming ONLY the ids it +# introduced -- never "whatever the allowlist now holds", which would re-add +# every model an admin had removed from the live row. Append a new entry +# (``-v2``, ...) with the next addition; never edit a shipped one. +COMMONSTACK_ALLOWLIST_BACKFILLS: tuple[tuple[str, tuple[str, ...]], ...] = ( + ("commonstack-allowlist-v1", ("anthropic/claude-haiku-4-5",)), ) @@ -253,6 +264,36 @@ def serialize_capabilities(value: ProviderCapabilities | dict) -> str: return json.dumps(capabilities.model_dump(), sort_keys=True, separators=(",", ":")) +def commonstack_allowlist_backfill( + capabilities_json: str | None, + model_ids: tuple[str, ...], +) -> str | None: + """Return ``capabilities_json`` with ``model_ids`` appended where missing. + + Only appends: an id an admin added stays and nothing is reordered. Returns + None when there is nothing to add, when the stored allowlist is empty (an + empty allowlist routes nothing, so it is how an admin turns the lane off, + and adding one model would turn it back on), or when the stored row cannot + be read -- ``deserialize_capabilities`` turns an unreadable row into + default capabilities, and writing that back would erase whatever it held. + """ + try: + capabilities = ProviderCapabilities.model_validate( + json.loads(capabilities_json or "") + ) + except (TypeError, ValueError): + return None + current = capabilities.model_allowlist + if not current: + return None + missing = tuple(model_id for model_id in model_ids if model_id not in current) + if not missing: + return None + return serialize_capabilities( + capabilities.model_copy(update={"model_allowlist": current + missing}) + ) + + def deserialize_capabilities(value: str | None) -> ProviderCapabilities: try: return ProviderCapabilities.model_validate(json.loads(value or "{}")) diff --git a/dashboard/backend/domain/model_providers/repository_postgres.py b/dashboard/backend/domain/model_providers/repository_postgres.py index 31960a46..188350c0 100644 --- a/dashboard/backend/domain/model_providers/repository_postgres.py +++ b/dashboard/backend/domain/model_providers/repository_postgres.py @@ -20,7 +20,9 @@ CredentialNotFoundError, CredentialOwnershipError, ProviderNotFoundError, + COMMONSTACK_ALLOWLIST_BACKFILLS, SEEDED_PROVIDERS, + commonstack_allowlist_backfill, deserialize_capabilities, serialize_capabilities, validate_adapter_type, @@ -230,6 +232,7 @@ def _init_schema(self) -> None: ), ) self._migrate_legacy_openrouter_platform_flag(cur) + self._migrate_commonstack_allowlist(cur) @staticmethod def _migrate_legacy_openrouter_platform_flag(cur) -> None: @@ -282,6 +285,42 @@ def _migrate_legacy_openrouter_platform_flag(cur) -> None: (migration_id, now), ) + @staticmethod + def _migrate_commonstack_allowlist(cur) -> None: + """Backfill newly verified CommonStack models into a seeded row, once. + + Each backfill appends only the ids it introduced, and is recorded even + when nothing changed, so an admin who removes one of those models -- + before or after it runs -- is not overridden on a later boot. + """ + + for migration_id, model_ids in COMMONSTACK_ALLOWLIST_BACKFILLS: + cur.execute( + "SELECT 1 FROM model_provider_migrations WHERE migration_id = %s", + (migration_id,), + ) + if cur.fetchone(): + continue + now = _utcnow_iso() + cur.execute( + "SELECT capabilities_json FROM provider_registry WHERE provider_id = 'commonstack'" + ) + provider = cur.fetchone() + updated = ( + commonstack_allowlist_backfill(provider["capabilities_json"], model_ids) + if provider + else None + ) + if updated is not None: + cur.execute( + "UPDATE provider_registry SET capabilities_json = %s, updated_at = %s WHERE provider_id = 'commonstack'", + (updated, now), + ) + cur.execute( + "INSERT INTO model_provider_migrations (migration_id, applied_at) VALUES (%s, %s)", + (migration_id, now), + ) + def list_enabled_providers(self, *, mode: str = "byok") -> list[dict[str, Any]]: if mode not in {"byok", "platform"}: raise ValueError("unsupported provider mode") diff --git a/dashboard/backend/domain/model_providers/service.py b/dashboard/backend/domain/model_providers/service.py index a11e523d..30e6924a 100644 --- a/dashboard/backend/domain/model_providers/service.py +++ b/dashboard/backend/domain/model_providers/service.py @@ -2,7 +2,7 @@ from __future__ import annotations -from collections.abc import Callable +from collections.abc import Callable, Collection from datetime import datetime, timezone from dataclasses import dataclass import json @@ -42,6 +42,7 @@ ) from .repository_common import ( CredentialConflictError, + ModelProviderStoreError, ProviderNotFoundError, canonical_request_digest, secret_fingerprint, @@ -98,6 +99,55 @@ def _environment_platform_secret(provider_id: str) -> str | None: return secret or None +DEFAULT_PLATFORM_PROVIDER_ORDER: tuple[str, ...] = ("commonstack", "openrouter") +_PLATFORM_PROVIDER_ORDER_ENV = "ATL_PLATFORM_PROVIDER_ORDER" +_warned_platform_provider_orders: set[str] = set() + + +def platform_provider_order( + known_provider_ids: Collection[str] | None = None, +) -> tuple[str, ...]: + """Return the ATL Credits provider preference, read per call. + + CommonStack comes first by default so its prepaid balance is spent + before OpenRouter's (#522). The order used to be hard-coded + OpenRouter-first, and CommonStack carried the traffic only because + OpenRouter's key was quota-dead (#523). + + The value is read per call, never at import, so a bad value cannot kill + boot. Any invalid token rejects the whole value, because a typo'd order + must not half-apply. A provider left out is never an automatic + candidate, which is the operator's no-deploy way to pull a drained lane. + + Pass ``known_provider_ids`` wherever the registry is at hand: a one-letter + typo (``commonstak``) is a well-formed id, and without the registry it + would silently drop the lane it misnames. + """ + + raw = os.getenv(_PLATFORM_PROVIDER_ORDER_ENV, "") + tokens = [token.strip().lower() for token in raw.split(",") if token.strip()] + if not tokens: + return DEFAULT_PLATFORM_PROVIDER_ORDER + ordered: list[str] = [] + try: + for token in tokens: + provider_id = validate_provider_id(token) + if known_provider_ids is not None and provider_id not in known_provider_ids: + raise ModelProviderStoreError("unknown provider id") + if provider_id not in ordered: + ordered.append(provider_id) + except ModelProviderStoreError: + if raw not in _warned_platform_provider_orders: + _warned_platform_provider_orders.add(raw) + print( + f"WARNING: {_PLATFORM_PROVIDER_ORDER_ENV} is not a comma-separated list " + "of known provider ids; using " + f"{','.join(DEFAULT_PLATFORM_PROVIDER_ORDER)}" + ) + return DEFAULT_PLATFORM_PROVIDER_ORDER + return tuple(ordered) + + def _utcnow_iso() -> str: return datetime.now(timezone.utc).replace(microsecond=0).isoformat() @@ -171,10 +221,14 @@ def list_execution_options( default_counts[provider_id] = default_counts.get(provider_id, 0) + 1 options: list[ExecutionProviderOption] = [] - for raw_provider in self.store.list_all_providers(): + byok_enabled_ids: set[str] = set() + raw_providers = self.store.list_all_providers() + for raw_provider in raw_providers: provider = ProviderRecord.model_validate(raw_provider) if provider.status != "enabled": continue + if provider.byok_enabled: + byok_enabled_ids.add(provider.provider_id) platform = self.store.get_platform_credential_public( provider.provider_id ) @@ -210,9 +264,26 @@ def list_execution_options( ), ) ) - # Preserve the repository's existing order while keeping OpenRouter as - # the preferred platform lane before the CommonStack fallback. - options.sort(key=lambda option: option.provider_id == "commonstack") + # Only platform-only lanes move: they go last, in + # ATL_PLATFORM_PROVIDER_ORDER order. Every BYOK-capable provider -- + # OpenRouter included, though it is also a platform lane -- keeps + # repository order, so the BYOK list app.js takes providers[0] from + # cannot be reordered by this env var or by a newly added provider. + # Platform routing order is decided by the route, not by this list. + platform_ids = platform_provider_order( + {str(raw.get("provider_id")) for raw in raw_providers} + ) + rank = { + provider_id: index + for index, provider_id in enumerate(platform_ids) + if provider_id not in byok_enabled_ids + } + options.sort( + key=lambda option: ( + option.provider_id in rank, + rank.get(option.provider_id, 0), + ) + ) return options def resolve_platform_execution_candidates( @@ -232,9 +303,13 @@ def resolve_platform_execution_candidates( if preferred_provider_id and preferred_provider_id.strip() else None ) + configured = platform_provider_order(providers) + # An explicit provider_id is honoured first only when it is in the + # configured order: leaving a lane out is the operator's no-deploy kill + # switch, and an API caller naming the pulled lane must not reopen it. ordered_ids: list[str] = [] - for provider_id in (preferred, "openrouter", "commonstack"): - if provider_id and provider_id not in ordered_ids: + for provider_id in (preferred, *configured): + if provider_id and provider_id in configured and provider_id not in ordered_ids: ordered_ids.append(provider_id) candidates: list[str] = [] for provider_id in ordered_ids: diff --git a/dashboard/backend/infrastructure/llm/execution/service.py b/dashboard/backend/infrastructure/llm/execution/service.py index a41bea42..73e27ae9 100644 --- a/dashboard/backend/infrastructure/llm/execution/service.py +++ b/dashboard/backend/infrastructure/llm/execution/service.py @@ -3,6 +3,7 @@ from __future__ import annotations import json +import threading from collections.abc import Callable from dashboard.backend.domain.credits.models import LLMSettlementResult @@ -12,16 +13,9 @@ ) from dashboard.backend.domain.analytics import instrumentation as analytics_instrumentation from dashboard.backend.domain.model_providers.models import ProviderRecord -from dashboard.backend.domain.model_providers.execution_catalog import ( - UnsupportedExecutionModel, -) -from dashboard.backend.domain.model_providers.repository_common import ( - ProviderNotFoundError, -) from dashboard.backend.domain.model_providers.service import ( ModelProviderService, ResolvedCredential, - CredentialResolutionError, ) from dashboard.backend.infrastructure.llm.execution.adapters.base import ( AdapterResponse, @@ -62,6 +56,25 @@ } ) +# CommonStack has no balance endpoint (every candidate path 404s as of +# 2026-09-23), so a drained platform lane is only observable as a failed +# call. Once per provider per process: a backtest child is one run, so a +# drained lane costs one line per run, not one per call. +_quota_exhausted_reported: set[str] = set() +_quota_exhausted_lock = threading.Lock() + + +def _report_platform_quota_exhausted(provider_id: str, fallback: str | None) -> None: + with _quota_exhausted_lock: + if provider_id in _quota_exhausted_reported: + return + _quota_exhausted_reported.add(provider_id) + print( + "ERROR: llm.platform_quota_exhausted " + f"provider={provider_id} fallback={fallback or 'none'}", + flush=True, + ) + # The provider receives the serialized messages, but its tokenizer is not # necessarily the same as ATL's estimator. Reserving the UTF-8 byte count is a @@ -362,25 +375,13 @@ def _execute_with_platform_failover( ) -> LLMExecutionResult: """Try ordered platform candidates, retaining one requested identity.""" + # The route's ordered tuple is authoritative: it already applied + # ATL_PLATFORM_PROVIDER_ORDER against the registry, and the handoff + # carries it whole. The worker used to re-derive routing here, turning + # a lone ("openrouter",) into OpenRouter -> CommonStack -- hard-coded + # OpenRouter-first, and blind to an order the route had rejected. A + # lone candidate is what the route decided, so it is not widened. candidates = tuple(request.provider_ids or (request.provider_id,)) - # Direct service callers predating the candidate-list handoff still get - # the established OpenRouter -> CommonStack fallback when the route is - # available. New handoffs always carry the complete ordered tuple. - if candidates == ("openrouter",): - try: - self.providers.preflight_execution_model( - "commonstack", request.model_id - ) - self.providers.preflight_platform_credential("commonstack") - except ( - ProviderNotFoundError, - CredentialResolutionError, - UnsupportedExecutionModel, - ): - pass - else: - candidates = ("openrouter", "commonstack") - requested_provider_id = candidates[0] last_error: LLMExecutionError | None = None for attempt_index, provider_id in enumerate(candidates): @@ -398,6 +399,13 @@ def _execute_with_platform_failover( ) except LLMExecutionError as exc: last_error = exc + if exc.category is ExecutionErrorCategory.PROVIDER_QUOTA_EXHAUSTED: + _report_platform_quota_exhausted( + provider_id, + candidates[attempt_index + 1] + if attempt_index + 1 < len(candidates) + else None, + ) if exc.category not in _PLATFORM_FAILOVER_CATEGORIES: raise assert last_error is not None diff --git a/dashboard/backend/tests/conftest.py b/dashboard/backend/tests/conftest.py index 2b9cd7de..342fda34 100644 --- a/dashboard/backend/tests/conftest.py +++ b/dashboard/backend/tests/conftest.py @@ -195,6 +195,11 @@ # unrelated failures. os.environ.pop("PIPELINE_SECONDS_PER_LLM_CALL", None) +# Decides which platform provider every ATL Credits call tries first. A +# developer configured like prod would otherwise see the routing and +# failover tests' order assertions fail as unexplained mismatches. +os.environ.pop("ATL_PLATFORM_PROVIDER_ORDER", None) + @atexit.register def _cleanup_test_db_dir() -> None: diff --git a/dashboard/backend/tests/domain/model_providers/test_execution_catalog.py b/dashboard/backend/tests/domain/model_providers/test_execution_catalog.py index 56b6b57d..bce9a8f7 100644 --- a/dashboard/backend/tests/domain/model_providers/test_execution_catalog.py +++ b/dashboard/backend/tests/domain/model_providers/test_execution_catalog.py @@ -104,9 +104,10 @@ def test_custom_provider_allowlist_rejects_invalid_model_ids(): _provider("openai_compatible", allowlist=("not allowed?",)) -def test_platform_candidates_prefer_openrouter_and_support_commonstack_only( +def test_platform_candidates_prefer_commonstack_and_follow_the_env_order( tmp_path, monkeypatch ): + monkeypatch.delenv("ATL_PLATFORM_PROVIDER_ORDER", raising=False) store = ModelProviderStore(tmp_path / "providers.db") service = ModelProviderService(store=store) monkeypatch.setenv("OPENROUTER_API_KEY", "sk-openrouter-test-abcd") @@ -114,9 +115,99 @@ def test_platform_candidates_prefer_openrouter_and_support_commonstack_only( assert service.resolve_platform_execution_candidates( "qwen/qwen3.7-plus" + ) == ("commonstack", "openrouter") + # Haiku reaches CommonStack only through #535's allowlist backfill. + assert service.resolve_platform_execution_candidates( + "anthropic/claude-haiku-4-5" + ) == ("commonstack", "openrouter") + + # An explicit provider_id that IS in the order is still honoured first. + assert service.resolve_platform_execution_candidates( + "qwen/qwen3.7-plus", preferred_provider_id="openrouter" ) == ("openrouter", "commonstack") + monkeypatch.setenv("ATL_PLATFORM_PROVIDER_ORDER", "openrouter,commonstack") + assert service.resolve_platform_execution_candidates( + "qwen/qwen3.7-plus" + ) == ("openrouter", "commonstack") + + # Leaving a lane out takes it out of automatic routing... + monkeypatch.setenv("ATL_PLATFORM_PROVIDER_ORDER", "openrouter") + assert service.resolve_platform_execution_candidates( + "qwen/qwen3.7-plus" + ) == ("openrouter",) + # ...and an explicit provider_id naming the pulled lane cannot reopen it. + assert service.resolve_platform_execution_candidates( + "qwen/qwen3.7-plus", preferred_provider_id="commonstack" + ) == ("openrouter",) + + # An order naming nothing routable yields no candidates; the route 422s. + monkeypatch.setenv("ATL_PLATFORM_PROVIDER_ORDER", "anthropic") + assert service.resolve_platform_execution_candidates("qwen/qwen3.7-plus") == () + + monkeypatch.delenv("ATL_PLATFORM_PROVIDER_ORDER") monkeypatch.delenv("OPENROUTER_API_KEY") assert service.resolve_platform_execution_candidates( "qwen/qwen3.7-plus" ) == ("commonstack",) + + monkeypatch.setenv("OPENROUTER_API_KEY", "sk-openrouter-test-abcd") + monkeypatch.delenv("COMMONSTACK_API_KEY") + assert service.resolve_platform_execution_candidates( + "qwen/qwen3.7-plus" + ) == ("openrouter",) + + +def test_platform_provider_order_parses_and_rejects_junk_whole(monkeypatch, capsys): + from dashboard.backend.domain.model_providers import service as service_module + + monkeypatch.setattr(service_module, "_warned_platform_provider_orders", set()) + + monkeypatch.delenv("ATL_PLATFORM_PROVIDER_ORDER", raising=False) + assert service_module.platform_provider_order() == ("commonstack", "openrouter") + monkeypatch.setenv("ATL_PLATFORM_PROVIDER_ORDER", " ") + assert service_module.platform_provider_order() == ("commonstack", "openrouter") + + monkeypatch.setenv( + "ATL_PLATFORM_PROVIDER_ORDER", " OpenRouter , commonstack,,openrouter " + ) + assert service_module.platform_provider_order() == ("openrouter", "commonstack") + assert capsys.readouterr().out == "" + + for junk in ("commonstack;openrouter", "Common Stack", "commonstack,open-router"): + monkeypatch.setenv("ATL_PLATFORM_PROVIDER_ORDER", junk) + assert service_module.platform_provider_order() == ( + "commonstack", + "openrouter", + ) + assert service_module.platform_provider_order() == ( + "commonstack", + "openrouter", + ) + out = capsys.readouterr().out + assert out.count("WARNING: ATL_PLATFORM_PROVIDER_ORDER") == 1 + assert junk not in out + + +def test_misspelled_provider_in_order_rejects_the_whole_value( + tmp_path, monkeypatch, capsys +): + """A one-letter typo passes the id syntax check; it must still not + half-apply by silently dropping the lane it misnames.""" + from dashboard.backend.domain.model_providers import service as service_module + + monkeypatch.setattr(service_module, "_warned_platform_provider_orders", set()) + store = ModelProviderStore(tmp_path / "providers.db") + service = ModelProviderService(store=store) + monkeypatch.setenv("OPENROUTER_API_KEY", "sk-openrouter-test-abcd") + monkeypatch.setenv("COMMONSTACK_API_KEY", "cs-commonstack-test-abcd") + monkeypatch.setenv("ATL_PLATFORM_PROVIDER_ORDER", "commonstak,openrouter") + + for _ in range(2): + assert service.resolve_platform_execution_candidates( + "qwen/qwen3.7-plus" + ) == ("commonstack", "openrouter") + + out = capsys.readouterr().out + assert out.count("WARNING: ATL_PLATFORM_PROVIDER_ORDER") == 1 + assert "commonstak" not in out diff --git a/dashboard/backend/tests/domain/model_providers/test_repository_contract.py b/dashboard/backend/tests/domain/model_providers/test_repository_contract.py index 8d40716a..37fa4863 100644 --- a/dashboard/backend/tests/domain/model_providers/test_repository_contract.py +++ b/dashboard/backend/tests/domain/model_providers/test_repository_contract.py @@ -2,6 +2,7 @@ from __future__ import annotations +import json import sqlite3 import pytest @@ -16,7 +17,12 @@ resolve_execution_model_route, ) from dashboard.backend.domain.model_providers.models import ProviderRecord -from dashboard.backend.domain.model_providers.repository_common import CredentialConflictError +from dashboard.backend.domain.model_providers.repository_common import ( + COMMONSTACK_ALLOWLIST_BACKFILLS, + COMMONSTACK_MODEL_ALLOWLIST, + CredentialConflictError, + commonstack_allowlist_backfill, +) @pytest.fixture(autouse=True) @@ -101,6 +107,7 @@ def test_seeded_commonstack_is_platform_only_with_verified_model_allowlist(store "anthropic/claude-sonnet-4-6", "deepseek/deepseek-v4-pro", "qwen/qwen3.7-plus", + "anthropic/claude-haiku-4-5", ) @@ -108,6 +115,7 @@ def test_commonstack_routes_only_the_verified_catalog_models(store): provider = ProviderRecord.model_validate(store.get_provider("commonstack")) assert [route.catalog_id for route in list_execution_model_routes(provider)] == [ + "anthropic/claude-haiku-4-5", "anthropic/claude-sonnet-4-6", "openai/gpt-5.5", "google/gemini-3.1-pro-preview", @@ -115,7 +123,7 @@ def test_commonstack_routes_only_the_verified_catalog_models(store): "qwen/qwen3.7-plus", ] with pytest.raises(UnsupportedExecutionModel): - resolve_execution_model_route(provider, "anthropic/claude-haiku-4-5") + resolve_execution_model_route(provider, "nvidia/nemotron-3-nano-30b-a3b") def test_legacy_openrouter_platform_flag_migrates_when_environment_key_exists( @@ -136,6 +144,7 @@ def test_legacy_openrouter_platform_flag_migrates_when_environment_key_exists( assert reopened.get_provider("openrouter")["platform_enabled"] is True marker = reopened._get_connection().execute( "SELECT migration_id FROM model_provider_migrations" + " WHERE migration_id = 'openrouter-platform-key-v1'" ).fetchone() assert marker[0] == "openrouter-platform-key-v1" @@ -171,7 +180,10 @@ def test_legacy_migration_respects_admin_platform_disable( assert reopened.get_provider("openrouter")["platform_enabled"] is False assert ( reopened._get_connection() - .execute("SELECT COUNT(*) FROM model_provider_migrations") + .execute( + "SELECT COUNT(*) FROM model_provider_migrations" + " WHERE migration_id = 'openrouter-platform-key-v1'" + ) .fetchone()[0] == 0 ) @@ -325,3 +337,92 @@ def test_legacy_sqlite_schema_is_migrated_without_losing_active_rows(tmp_path): label="Old", secret="sk-reused-label-1234", )["label"] == "Old" + + +def _set_commonstack_allowlist(database_path, allowlist, *, forget_migration): + conn = sqlite3.connect(database_path) + row = conn.execute( + "SELECT capabilities_json FROM provider_registry WHERE provider_id = 'commonstack'" + ).fetchone() + capabilities = json.loads(row[0]) + capabilities["model_allowlist"] = list(allowlist) + conn.execute( + "UPDATE provider_registry SET capabilities_json = ? WHERE provider_id = 'commonstack'", + (json.dumps(capabilities),), + ) + if forget_migration: + conn.executemany( + "DELETE FROM model_provider_migrations WHERE migration_id = ?", + [(migration_id,) for migration_id, _ in COMMONSTACK_ALLOWLIST_BACKFILLS], + ) + conn.commit() + conn.close() + + +def test_commonstack_allowlist_backfills_a_row_seeded_before_haiku(tmp_path): + # Prod's row was seeded before Haiku joined the allowlist, and the seed is + # ON CONFLICT DO NOTHING -- only the backfill can reach it. It adds Haiku + # and nothing else: gemini/sonnet/deepseek were removed from this row (by + # an admin, as far as the store can tell) and must stay removed. An id an + # admin added survives too. + database_path = tmp_path / "provider-backfill.db" + ModelProviderStore(database_path) + _set_commonstack_allowlist( + database_path, + ("openai/gpt-5.5", "qwen/qwen3.7-plus", "admin/extra-model"), + forget_migration=True, + ) + + store = ModelProviderStore(database_path) + + assert store.get_provider("commonstack")["capabilities"].model_allowlist == ( + "openai/gpt-5.5", + "qwen/qwen3.7-plus", + "admin/extra-model", + "anthropic/claude-haiku-4-5", + ) + + +def test_commonstack_allowlist_backfill_keeps_an_emptied_lane_off(tmp_path): + # An empty allowlist routes nothing, so it is how an admin pulls the lane; + # a backfill adding one model would silently reopen it. + database_path = tmp_path / "provider-backfill-empty.db" + ModelProviderStore(database_path) + _set_commonstack_allowlist(database_path, (), forget_migration=True) + + store = ModelProviderStore(database_path) + + assert store.get_provider("commonstack")["capabilities"].model_allowlist == () + + +def test_every_commonstack_backfill_id_is_in_the_seeded_allowlist(): + # A backfill may only add what a fresh deployment would seed; and ids are + # one-shot per migration, so two entries must not share an id. + migration_ids = [migration_id for migration_id, _ in COMMONSTACK_ALLOWLIST_BACKFILLS] + assert len(migration_ids) == len(set(migration_ids)) + for _migration_id, model_ids in COMMONSTACK_ALLOWLIST_BACKFILLS: + assert model_ids + assert set(model_ids) <= set(COMMONSTACK_MODEL_ALLOWLIST) + + +def test_commonstack_allowlist_backfill_runs_once_so_admin_removals_stick(tmp_path): + database_path = tmp_path / "provider-backfill-once.db" + ModelProviderStore(database_path) + _set_commonstack_allowlist( + database_path, ("openai/gpt-5.5",), forget_migration=False + ) + + store = ModelProviderStore(database_path) + + assert store.get_provider("commonstack")["capabilities"].model_allowlist == ( + "openai/gpt-5.5", + ) + + +def test_commonstack_allowlist_backfill_leaves_an_unreadable_row_alone(): + haiku = ("anthropic/claude-haiku-4-5",) + assert commonstack_allowlist_backfill("{not json", haiku) is None + assert commonstack_allowlist_backfill(None, haiku) is None + assert commonstack_allowlist_backfill( + json.dumps({"model_allowlist": list(COMMONSTACK_MODEL_ALLOWLIST)}), haiku + ) is None diff --git a/dashboard/backend/tests/domain/model_providers/test_service.py b/dashboard/backend/tests/domain/model_providers/test_service.py index cb0adebf..770f651c 100644 --- a/dashboard/backend/tests/domain/model_providers/test_service.py +++ b/dashboard/backend/tests/domain/model_providers/test_service.py @@ -472,20 +472,38 @@ def test_verified_stored_commonstack_credential_precedes_environment_key( assert resolved.secret == "cs-fake-stored-test-wxyz" -def test_execution_options_keep_openrouter_ahead_of_commonstack( +def test_execution_options_follow_platform_order_and_keep_byok_order( tmp_path, monkeypatch ): - service, _store = _service(tmp_path, FakeAdapter()) + monkeypatch.delenv("ATL_PLATFORM_PROVIDER_ORDER", raising=False) + service, store = _service(tmp_path, FakeAdapter()) monkeypatch.setenv("OPENROUTER_API_KEY", "or-fake-options-abcd") monkeypatch.setenv("COMMONSTACK_API_KEY", "cs-fake-options-wxyz") + # A BYOK provider whose display name sorts after "OpenRouter": moving + # OpenRouter (a BYOK *and* platform lane) to the end would make this the + # BYOK default that app.js takes from providers[0]. + store.upsert_provider( + provider_id="xai", + display_name="xAI", + adapter_type="openai_compatible", + approved_base_url="https://api.x.ai/v1", + capabilities=ProviderCapabilities(model_allowlist=()), + byok_enabled=True, + platform_enabled=False, + status="enabled", + ) - provider_ids = [ - option.provider_id - for option in service.list_execution_options(7) - if option.platform_credits_available - ] - - assert provider_ids.index("openrouter") < provider_ids.index("commonstack") + # Every BYOK-capable provider keeps repository (display-name) order; + # only the platform-only CommonStack lane moves, to the end. + expected = ["anthropic", "gemini", "openai", "openrouter", "xai", "commonstack"] + for order in (None, "openrouter,commonstack", "commonstack,openrouter,openai"): + if order is None: + monkeypatch.delenv("ATL_PLATFORM_PROVIDER_ORDER", raising=False) + else: + monkeypatch.setenv("ATL_PLATFORM_PROVIDER_ORDER", order) + assert [ + option.provider_id for option in service.list_execution_options(7) + ] == expected, order assert "cs-fake-options-wxyz" not in repr(service.list_execution_options(7)) diff --git a/dashboard/backend/tests/infrastructure/llm/test_platform_credits_env_fallback.py b/dashboard/backend/tests/infrastructure/llm/test_platform_credits_env_fallback.py index be878793..224d65ef 100644 --- a/dashboard/backend/tests/infrastructure/llm/test_platform_credits_env_fallback.py +++ b/dashboard/backend/tests/infrastructure/llm/test_platform_credits_env_fallback.py @@ -167,6 +167,17 @@ def _request(run_id: str) -> LLMExecutionRequest: ) +def _failover_request(run_id: str) -> LLMExecutionRequest: + """A platform request carrying the route's OpenRouter -> CommonStack tuple. + + The worker never widens a lone candidate, so failover tests hand over the + ordered tuple exactly as the route would. + """ + return _request(run_id).model_copy( + update={"provider_ids": ("openrouter", "commonstack")} + ) + + def _execution_service(tmp_path, monkeypatch, adapter, *, adapters=None): monkeypatch.setenv("OPENROUTER_API_KEY", "sk-or-execution-test-abcd") provider_store = ModelProviderStore(tmp_path / "providers.db") @@ -298,7 +309,7 @@ def test_platform_quota_error_retries_once_through_commonstack( }, ) assert service.providers.store.get_provider("commonstack")["platform_enabled"] is True - request = _request("failover-run").model_copy( + request = _failover_request("failover-run").model_copy( update={ "model_id": "qwen/qwen3.7-plus", "reasoning_effort": "high", @@ -356,7 +367,7 @@ def test_non_quota_platform_failures_do_not_fail_over( ) with pytest.raises(LLMExecutionError) as exc_info: - service.execute(_request(f"no-failover-{category.value}")) + service.execute(_failover_request(f"no-failover-{category.value}")) assert exc_info.value.category is category assert fallback.calls == [] @@ -391,7 +402,7 @@ def test_provider_selection_failures_fail_over_to_commonstack( adapters={"openrouter": primary, "commonstack": fallback}, ) - result = service.execute(_request(f"failover-{category.value}")) + result = service.execute(_failover_request(f"failover-{category.value}")) assert result.provider_id == "commonstack" assert result.requested_provider_id == "openrouter" @@ -451,7 +462,7 @@ def test_fallback_failure_returns_commonstack_safe_category(tmp_path, monkeypatc adapters={"openrouter": primary, "commonstack": fallback}, ) with pytest.raises(LLMExecutionError) as exc_info: - service.execute(_request("dual-failure")) + service.execute(_failover_request("dual-failure")) assert exc_info.value.category is ExecutionErrorCategory.PROVIDER_TIMEOUT assert len(primary.calls) == 1 assert len(fallback.calls) == 1 @@ -503,7 +514,7 @@ def fail_release(*_args, **_kwargs): monkeypatch.setattr(service.credits, "release_llm_credits", fail_release) with pytest.raises(LLMExecutionError) as exc_info: - service.execute(_request("release-failure")) + service.execute(_failover_request("release-failure")) assert exc_info.value.category is ExecutionErrorCategory.BILLING_FAILED assert fallback.calls == [] @@ -531,7 +542,7 @@ def test_two_quota_failures_stop_after_commonstack(tmp_path, monkeypatch): adapters={"openrouter": primary, "commonstack": fallback}, ) with pytest.raises(LLMExecutionError) as exc_info: - service.execute(_request("double-quota")) + service.execute(_failover_request("double-quota")) assert exc_info.value.category is ExecutionErrorCategory.PROVIDER_QUOTA_EXHAUSTED assert len(primary.calls) == 1 assert len(fallback.calls) == 1 @@ -653,3 +664,164 @@ def test_restricted_account_execution_error_is_actionable( assert exc_info.value.category is ExecutionErrorCategory.ACCOUNT_RESTRICTED assert expected in exc_info.value.safe_message assert "CreditAccountRestrictedStoreError" not in exc_info.value.safe_message + + +def _platform_request(run_id: str, provider_ids: tuple[str, ...]) -> LLMExecutionRequest: + return LLMExecutionRequest( + user_id=USER_ID, + run_id=run_id, + call_index=0, + billing_mode=BillingMode.PLATFORM_CREDITS, + provider_id=provider_ids[0], + provider_ids=provider_ids, + model_id=MODEL_ID, + system_message="Return one trading decision.", + messages=(LLMMessage(role="user", content="Analyze the market."),), + usage_policy=UsagePolicy(max_output_tokens=100), + ) + + +def _quota_error() -> ProviderExecutionError: + return ProviderExecutionError(ExecutionErrorCategory.PROVIDER_QUOTA_EXHAUSTED) + + +def _ok_response() -> AdapterResponse: + return AdapterResponse( + text="BUY", + model_id=MODEL_ID, + usage=LLMUsage(input_tokens=40, output_tokens=20), + finish_reason="stop", + ) + + +@pytest.fixture +def fresh_quota_reports(monkeypatch): + monkeypatch.setattr(execution_service_module, "_quota_exhausted_reported", set()) + + +def test_drained_commonstack_prints_one_operator_error_per_process( + tmp_path, monkeypatch, capsys, fresh_quota_reports +): + monkeypatch.setenv("COMMONSTACK_API_KEY", "cs-fake-drained-abcd") + commonstack = ScriptedExecutionAdapter([_quota_error(), _quota_error()]) + openrouter = ScriptedExecutionAdapter([_ok_response(), _ok_response()]) + service, _store = _execution_service( + tmp_path, + monkeypatch, + openrouter, + adapters={"commonstack": commonstack, "openrouter": openrouter}, + ) + + for run_id in ("drained-1", "drained-2"): + result = service.execute( + _platform_request(run_id, ("commonstack", "openrouter")) + ) + assert result.provider_id == "openrouter" + + lines = [ + line + for line in capsys.readouterr().out.splitlines() + if "llm.platform_quota_exhausted" in line + ] + assert lines == [ + "ERROR: llm.platform_quota_exhausted provider=commonstack fallback=openrouter" + ] + + +def test_quota_exhaustion_with_no_next_candidate_names_none( + tmp_path, monkeypatch, capsys, fresh_quota_reports +): + monkeypatch.setenv("COMMONSTACK_API_KEY", "cs-fake-drained-only-abcd") + commonstack = ScriptedExecutionAdapter([_quota_error()]) + service, _store = _execution_service( + tmp_path, + monkeypatch, + commonstack, + adapters={"commonstack": commonstack}, + ) + + with pytest.raises(LLMExecutionError) as exc_info: + service.execute(_platform_request("drained-only", ("commonstack",))) + + assert exc_info.value.category is ExecutionErrorCategory.PROVIDER_QUOTA_EXHAUSTED + assert ( + "ERROR: llm.platform_quota_exhausted provider=commonstack fallback=none" + in capsys.readouterr().out + ) + + +def test_byok_quota_exhaustion_prints_no_operator_error( + tmp_path, monkeypatch, capsys, fresh_quota_reports +): + service, _store = _execution_service( + tmp_path, + monkeypatch, + ScriptedExecutionAdapter([]), + ) + + def fail_byok_once(*_args, **_kwargs): + raise LLMExecutionError(ExecutionErrorCategory.PROVIDER_QUOTA_EXHAUSTED) + + monkeypatch.setattr(service, "_execute_once", fail_byok_once) + request = _request("byok-drained").model_copy( + update={"billing_mode": BillingMode.BYOK} + ) + + with pytest.raises(LLMExecutionError): + service.execute(request) + + assert "llm.platform_quota_exhausted" not in capsys.readouterr().out + + +def test_quota_report_is_once_under_concurrency_and_flushed( + capsys, fresh_quota_reports +): + import inspect + import threading + + barrier = threading.Barrier(8) + + def report(): + barrier.wait() + execution_service_module._report_platform_quota_exhausted( + "commonstack", "openrouter" + ) + + threads = [threading.Thread(target=report) for _ in range(8)] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + + assert capsys.readouterr().out.count("llm.platform_quota_exhausted") == 1 + # A killed child's block-buffered stdout dies with it, and the 3600s + # timeout kill is exactly when this line matters. + assert "flush=True" in inspect.getsource( + execution_service_module._report_platform_quota_exhausted + ) + + +def test_worker_never_widens_the_route_s_lone_candidate( + tmp_path, monkeypatch, capsys, fresh_quota_reports +): + """A lone ("openrouter",) is what the route decided -- CommonStack pulled + from ATL_PLATFORM_PROVIDER_ORDER (including by a typo the route rejected), + or ineligible for the model. The worker used to re-derive routing and add + CommonStack back; with the default order and CommonStack fully eligible + it must still bill only the lane it was handed.""" + monkeypatch.delenv("ATL_PLATFORM_PROVIDER_ORDER", raising=False) + monkeypatch.setenv("COMMONSTACK_API_KEY", "cs-fake-pulled-abcd") + openrouter = ScriptedExecutionAdapter([_quota_error()]) + commonstack = ScriptedExecutionAdapter([_ok_response()]) + service, _store = _execution_service( + tmp_path, + monkeypatch, + openrouter, + adapters={"openrouter": openrouter, "commonstack": commonstack}, + ) + + with pytest.raises(LLMExecutionError) as exc_info: + service.execute(_platform_request("pulled-lane", ("openrouter",))) + + assert exc_info.value.category is ExecutionErrorCategory.PROVIDER_QUOTA_EXHAUSTED + assert commonstack.calls == [] diff --git a/dashboard/backend/tests/infrastructure/llm/test_pricing.py b/dashboard/backend/tests/infrastructure/llm/test_pricing.py index 93bb5391..a2c7ead2 100644 --- a/dashboard/backend/tests/infrastructure/llm/test_pricing.py +++ b/dashboard/backend/tests/infrastructure/llm/test_pricing.py @@ -56,3 +56,10 @@ def test_snapshot_default_source_version_follows_the_table(): snapshot.input_usd_per_million_tokens, snapshot.output_usd_per_million_tokens, ) == (2.50, 10.0) + + +def test_haiku_on_commonstack_prices_at_the_listed_rate(): + # CommonStack lists anthropic/claude-haiku-4-5 at $1 / $5 per million + # (checked 2026-09-23). The claude-haiku-4 entry already matches, so the + # #535 allowlist addition needed no pricing-table change. + assert pricing.price_for_model("anthropic/claude-haiku-4-5") == (1.0, 5.0) diff --git a/dashboard/backend/tests/test_admin_analytics_frontend.py b/dashboard/backend/tests/test_admin_analytics_frontend.py index 9caaa666..8e89742b 100644 --- a/dashboard/backend/tests/test_admin_analytics_frontend.py +++ b/dashboard/backend/tests/test_admin_analytics_frontend.py @@ -162,7 +162,7 @@ def test_app_lifecycle_and_cache_versions_are_wired(): # Lockstep owner for the console's bumped tags and the /admin page's pins: # every bump edits this test in the same change (Global Constraints). assert 'styles.css?v=150' in APP_HTML - assert 'app.js?v=144' in APP_HTML + assert 'app.js?v=145' in APP_HTML assert 'js/admin-tabs.js?v=12' in APP_HTML for tag in ( 'href="admin.css?v=4"', diff --git a/dashboard/backend/tests/test_analytics_frontend.py b/dashboard/backend/tests/test_analytics_frontend.py index da47d6a5..67f70549 100644 --- a/dashboard/backend/tests/test_analytics_frontend.py +++ b/dashboard/backend/tests/test_analytics_frontend.py @@ -41,7 +41,7 @@ def _function_body(source: str, signature: str) -> str: def test_analytics_script_loads_between_app_and_page_scripts(): - app_at = APP_HTML.index('') + app_at = APP_HTML.index('') analytics_at = APP_HTML.index( '' ) diff --git a/dashboard/backend/tests/test_backtest_cancel.py b/dashboard/backend/tests/test_backtest_cancel.py index 5f6bbdc8..42e39515 100644 --- a/dashboard/backend/tests/test_backtest_cancel.py +++ b/dashboard/backend/tests/test_backtest_cancel.py @@ -1038,3 +1038,69 @@ def explode(): bt._max_ai_hedge_fund_trading_days() == bt._DEFAULT_MAX_AI_HEDGE_FUND_TRADING_DAYS ) + + +def test_drain_relays_operator_llm_errors_to_the_parent_log_live(capsys): + """The quota ERROR line must reach the service log even if the child is + killed. The timeout path never dumps the child's capture, and a long run's + middle is elided from it, so the parent echoes the line as it arrives.""" + import io + + capture = bt._BoundedStreamCapture(head_chars=10, tail_chars=10) + stream = io.StringIO( + "filler line that fills the head\n" + "ERROR: llm.platform_quota_exhausted provider=commonstack fallback=openrouter\n" + "ordinary progress line\n" + "more filler to push the error out of the retained tail\n" + ) + + bt._drain_stream(stream, capture) + + out = capsys.readouterr().out + assert out == ( + "ERROR: llm.platform_quota_exhausted provider=commonstack fallback=openrouter\n" + ) + assert "provider=commonstack" not in capture.text() + + +def test_drain_redacts_relayed_lines_like_the_dump(capsys, monkeypatch): + """The prefix is all that selects a line for the live relay, so the relay + must scrub it exactly as the end-of-run dump would.""" + import io + + monkeypatch.setenv("IFIND_REFRESH_TOKEN", "ifind-refresh-secret-123") + capture = bt._BoundedStreamCapture() + stream = io.StringIO( + "ERROR: llm.example key=fd-secret-abc token=ifind-refresh-secret-123 " + "access_token=bearer-xyz\n" + ) + + bt._drain_stream(stream, capture, "fd-secret-abc") + + out = capsys.readouterr().out + assert out.startswith("ERROR: llm.example") + for secret in ("fd-secret-abc", "ifind-refresh-secret-123", "bearer-xyz"): + assert secret not in out + assert out.count("[REDACTED]") == 3 + + +def test_relayed_lines_are_not_dumped_a_second_time(): + """_drain_stream already printed them; the end-of-run dump repeating them + made every quota event count twice in the service log.""" + import inspect + + text = ( + "starting\n" + "ERROR: llm.platform_quota_exhausted provider=commonstack fallback=openrouter\n" + "ERROR: something else stays\n" + "done" + ) + assert bt._without_relayed_lines(text) == ( + "starting\nERROR: something else stays\ndone" + ) + assert bt._without_relayed_lines("") == "" + + source = inspect.getsource(bt.run_backtest_background) + assert "_without_relayed_lines(result.stdout" in source + assert "_without_relayed_lines(result.stderr" in source + assert "redact_secret=financial_datasets_api_key" in source diff --git a/dashboard/backend/tests/test_backtest_comparison_frontend.py b/dashboard/backend/tests/test_backtest_comparison_frontend.py index 5b6a2db8..a2726fa9 100644 --- a/dashboard/backend/tests/test_backtest_comparison_frontend.py +++ b/dashboard/backend/tests/test_backtest_comparison_frontend.py @@ -191,7 +191,7 @@ def test_exact_raw_ties_mark_every_tied_series_best(): def test_comparison_script_and_semantic_table_ship_before_app(): helper = '' - app = '' + app = '' assert 'href="styles.css?v=150"' in APP_HTML assert APP_HTML.index(helper) < APP_HTML.index(app) for element_id in ( diff --git a/dashboard/backend/tests/test_backtest_worker_schema_skip.py b/dashboard/backend/tests/test_backtest_worker_schema_skip.py index 76cc3e86..d9d70660 100644 --- a/dashboard/backend/tests/test_backtest_worker_schema_skip.py +++ b/dashboard/backend/tests/test_backtest_worker_schema_skip.py @@ -240,7 +240,8 @@ def test_the_guarded_list_accounts_for_every_twin(): "amounts plus ledger bucket/operation columns" ), "domain/model_providers/repository_postgres.py": ( - "scrubs api_key_enc on revoked credentials and seeds SEEDED_PROVIDERS" + "scrubs api_key_enc on revoked credentials, seeds SEEDED_PROVIDERS " + "and backfills the CommonStack model allowlist" ), "users_postgres.py": ( "repairs users.user_group values outside the allowed set" diff --git a/dashboard/backend/tests/test_byok_backtest_frontend.py b/dashboard/backend/tests/test_byok_backtest_frontend.py index d682931a..affb1f57 100644 --- a/dashboard/backend/tests/test_byok_backtest_frontend.py +++ b/dashboard/backend/tests/test_byok_backtest_frontend.py @@ -57,10 +57,13 @@ def test_atl_credits_hides_provider_and_omits_provider_payload(): assert body.index("payload.billing_mode = selectedBillingMode") < body.index( "payload.provider_id = selectedProviderId" ) - _assert_contains( - APP_JS, - "ATL Credits automatically use OpenRouter first, then CommonStack if needed.", - ) + # Names no provider and no order: ATL_PLATFORM_PROVIDER_ORDER can reorder + # or remove either lane with no deploy, so copy naming them (or promising + # a fallback between them) would go stale silently. + _assert_contains(APP_JS, "ATL Credits cover the model calls. ATL picks an available provider automatically.") + hint = _function_body("setRunBacktestBillingMode") + assert "OpenRouter" not in hint + assert "CommonStack" not in hint def test_atl_model_options_are_merged_without_duplicate_ids(): diff --git a/dashboard/backend/tests/test_frontend_fast_boot.py b/dashboard/backend/tests/test_frontend_fast_boot.py index 61948d81..7b961a99 100644 --- a/dashboard/backend/tests/test_frontend_fast_boot.py +++ b/dashboard/backend/tests/test_frontend_fast_boot.py @@ -191,7 +191,7 @@ def test_cache_busters_bumped(): # the next bump, so the exact one looks like the broken guard and gets # "fixed" by loosening it. That collision has already cost this repo one # round of follow-ups (#347/#348). - assert "app.js?v=144" in APP_HTML + assert "app.js?v=145" in APP_HTML assert "js/agent-editor.js?v=32" in APP_HTML assert "styles.css?v=150" in APP_HTML assert "js/leaderboard.js?v=33" in APP_HTML diff --git a/dashboard/backend/tests/test_model_provider_store_postgres.py b/dashboard/backend/tests/test_model_provider_store_postgres.py index feec57db..a86d371b 100644 --- a/dashboard/backend/tests/test_model_provider_store_postgres.py +++ b/dashboard/backend/tests/test_model_provider_store_postgres.py @@ -2,6 +2,7 @@ from __future__ import annotations +import json import os import psycopg @@ -19,6 +20,7 @@ resolve_execution_model_route, ) from dashboard.backend.domain.model_providers.repository_common import ( + COMMONSTACK_ALLOWLIST_BACKFILLS, CredentialConflictError, CredentialOwnershipError, ) @@ -65,6 +67,38 @@ def test_postgres_store_rejects_non_postgres_url_before_connecting(): PostgresModelProviderStore("sqlite:///tmp/not-postgres.db") +@pg_only +def test_postgres_commonstack_allowlist_backfills_a_row_seeded_before_haiku( + postgres_store, +): + # The seed is ON CONFLICT DO NOTHING, so prod's pre-Haiku row is reachable + # only through the one-shot backfill. It adds Haiku alone: the seeded ids + # this row lacks were removed and stay removed; an admin-added id survives. + with psycopg.connect(TEST_POSTGRES_URL, autocommit=True) as conn: + with conn.cursor() as cur: + cur.execute( + "SELECT capabilities_json FROM provider_registry WHERE provider_id = 'commonstack'" + ) + capabilities = json.loads(cur.fetchone()[0]) + capabilities["model_allowlist"] = ["openai/gpt-5.5", "admin/extra-model"] + cur.execute( + "UPDATE provider_registry SET capabilities_json = %s WHERE provider_id = 'commonstack'", + (json.dumps(capabilities),), + ) + cur.executemany( + "DELETE FROM model_provider_migrations WHERE migration_id = %s", + [(migration_id,) for migration_id, _ in COMMONSTACK_ALLOWLIST_BACKFILLS], + ) + + reopened = PostgresModelProviderStore(TEST_POSTGRES_URL) + + assert reopened.get_provider("commonstack")["capabilities"].model_allowlist == ( + "openai/gpt-5.5", + "admin/extra-model", + "anthropic/claude-haiku-4-5", + ) + + @pg_only def test_postgres_seeded_commonstack_is_platform_only_with_allowlisted_models( postgres_store, @@ -81,12 +115,14 @@ def test_postgres_seeded_commonstack_is_platform_only_with_allowlisted_models( "anthropic/claude-sonnet-4-6", "deepseek/deepseek-v4-pro", "qwen/qwen3.7-plus", + "anthropic/claude-haiku-4-5", ) provider_record = ProviderRecord.model_validate(provider) assert [ route.catalog_id for route in list_execution_model_routes(provider_record) ] == [ + "anthropic/claude-haiku-4-5", "anthropic/claude-sonnet-4-6", "openai/gpt-5.5", "google/gemini-3.1-pro-preview", @@ -94,7 +130,7 @@ def test_postgres_seeded_commonstack_is_platform_only_with_allowlisted_models( "qwen/qwen3.7-plus", ] with pytest.raises(UnsupportedExecutionModel): - resolve_execution_model_route(provider_record, "anthropic/claude-haiku-4-5") + resolve_execution_model_route(provider_record, "nvidia/nemotron-3-nano-30b-a3b") @pg_only diff --git a/dashboard/frontend/app.html b/dashboard/frontend/app.html index e39ec79a..84cb41eb 100644 --- a/dashboard/frontend/app.html +++ b/dashboard/frontend/app.html @@ -2547,7 +2547,7 @@

Refund Credits purchase

- + diff --git a/dashboard/frontend/app.js b/dashboard/frontend/app.js index c529a0ab..5b392ea9 100644 --- a/dashboard/frontend/app.js +++ b/dashboard/frontend/app.js @@ -9882,7 +9882,7 @@ function setRunBacktestBillingMode( hint.textContent = runBacktestBillingMode === 'byok' ? 'Provider charges go directly to your API key. ATL Credits are not deducted.' : (runBacktestBillingMode === 'platform_credits' - ? 'ATL Credits automatically use OpenRouter first, then CommonStack if needed.' + ? 'ATL Credits cover the model calls. ATL picks an available provider automatically.' : 'Choose an available AI billing method.'); } } diff --git a/docs/superpowers/specs/2026-09-01-platform-provider-auto-routing-design.md b/docs/superpowers/specs/2026-09-01-platform-provider-auto-routing-design.md index 5ce08057..0626f992 100644 --- a/docs/superpowers/specs/2026-09-01-platform-provider-auto-routing-design.md +++ b/docs/superpowers/specs/2026-09-01-platform-provider-auto-routing-design.md @@ -1,5 +1,7 @@ # Platform Provider Auto-Routing Design +> **Amended 2026-09-26:** the OpenRouter-first order this document specifies is superseded. ATL Credits now follow `ATL_PLATFORM_PROVIDER_ORDER`, CommonStack first by default. See `2026-09-23-llm-backtest-step-latency-design.md` §5. + ## Goal When a user runs an LLM backtest with `Use ATL Credits`, the UI exposes only the approved model list. The backend automatically tries OpenRouter first and falls back to CommonStack when the first provider cannot serve the request because of quota, balance, credential availability, timeout, or temporary provider unavailability. BYOK keeps its explicit provider selection.