ops(backtests): read runtime_type/decision_source in prod to settle the LLM-vs-rule-based mix

[FlyM1ss]

From §12 of docs/superpowers/specs/2026-09-21-backtest-bar-cache-design.md (#506).
Companion to #502 — same trip to prod, different data source (the run history DB, not the
Render deploy log).

Both #501 and #506 shorten the dark window before the first bar: schema-init skipping, and
now an on-disk bar cache that removes loading_bars (~86% of that window, measured locally)
on a warm key. Neither touches the decision loop.

So the ceiling on every further change of this kind is the mix of what prod actually runs.
Read it, once, rather than guessing:

SELECT runtime_type, decision_source, COUNT(*), AVG(...)
FROM agent_runs
WHERE created_at > now() - interval '30 days'
GROUP BY 1, 2;

against AGENT_RUNS_DATABASE_URL (the ATL-runs-main Neon project — run history has its own
database, see CLAUDE.md).

Why it decides something: if most runs are decision_source='llm', wall-clock is
dominated by model latency the cache cannot touch, and further start-up work has a small
ceiling — spend the effort on the LLM lane instead. If most are rule-based, the start-up
window is a real share of total runtime and the next cache (aggregated output, the
loading_bars residual that #506 split out as fetch_seconds vs aggregate+verify) is worth
building.

Entry point: §1-C of the speed-and-trust design. Record the numbers on this issue; that
is the deliverable.

[FlyM1ss]

Read against ATL-runs-main on 2026-09-22. 625 rows total, first 2026-05-25, last 2026-09-22 20:15.

The answer

The dashboard lane is majority rule-based, so start-up latency is a real share of total runtime and further work here has a genuine ceiling.

window	runs	llm-driven	rule-based
mode='backtest', last 30d — the lane the bar cache serves	126	54 (42.9%)	72 (57.1%)
all modes, last 30d	226	54 (23.9%)	172
all modes, last 7d	41	6 (14.6%)	35
all time	625	180 (28.8%)	445

leaderboard rows (100 of the last 30d) are cached reruns of the curated roster, not user traffic — excluded from the headline figure.

Three caveats that matter more than the numbers

1. runtime_type is never recorded. Zero occurrences in agent_runs.metadata across all 625 rows. That half of this issue is not answerable from this table — it would need the field to start being written first.
2. decision_source is recorded on only 45 of the 400 most recent rows. The rest is inferred from llm_calls > 0. The inference is sound (an LLM run bills calls) but it is an inference.
3. Run duration is not derivable here. updated_at - created_at is a median of 0.0s for rule-based runs, i.e. the row is inserted at finalize rather than at start. So this table cannot give the wall-clock split the decision would ideally rest on. Worth fixing separately if that split is ever wanted.

Also: created_at is a TEXT column in the Postgres twin (SQLite heritage), so any date predicate needs created_at::timestamp — a bare > now() - interval '30 days' raises UndefinedFunction.

Unasked-for bonus, and it settles the other open question

While I had the table open I measured window reuse, which is what the bar cache's "altitude limit" note in bar_cache.py:42-55 is about:

126 dashboard runs over 13 distinct (start, end) windows
runs landing on an ALREADY-SEEN exact window:  113/126 = 89.7%

top windows:  2026-05-04 -> 2026-05-12   x63
              2026-04-01 -> 2026-04-30   x13
              2026-04-15 -> 2026-04-16    x6

Simulating a day-partitioned key over the same traffic:

fully covered by earlier runs:  115 (91.3%)
partially covered:                4  (3.2%)
no overlap at all:                7  (5.6%)

89.7% → 91.3% is +1.6 points. The design's own tail note warns that the verbatim-window key "does NOT speed up an arbitrary window a user types, which is most of the problem this track exists for." On real traffic that is not the case — users overwhelmingly re-run the same handful of windows, so the narrow key already captures nearly all the reuse a general one would. Filing that separately as measured-and-not-worth-building rather than leaving it as an open design debt.