Say plainly whether 2s/10s is reachable: one yes, one no

[adamjohnwright]

T022. Measured on astream_answer — the path the endpoint uses — over five questions × three runs per arm, with wall clock to the first token recorded independently of the phase timings so the breakdown has to account for the total rather than being assumed to.

	default	expansion off
preprocess	1.77s	1.65s
retrieval	2.84s	0.26s
answer model (residual)	0.66s	0.90s
first token, p50	5.27s	2.81s
first token, p90	10.57s	3.81s
complete, p50	9.64s	7.04s

The plain answer

10s complete: met. 9.64s today, 7.04s with expansion off.

2s to first token: not met, either way, and not close on the default path. Disabling query expansion reaches 2.81s — short of the target but the same order as it rather than double. So FR-005a stays a target with a named blocker.

A number I cannot reconcile

The spec records the answer model's time to first token at 6.1s, "the largest block left". This measurement puts it at 0.66s, with retrieval the largest at 2.84s.

I can offer the mechanism but not the proof. My figure is a residual — first token minus preprocess minus retrieval — so it absorbs anything unattributed, which biases it upward, not down. The two disagree by a factor of nine and only one can describe today's code.

Recorded as unreconciled rather than quietly replacing the old number, because "the answer model is the problem" has been steering this spec's priorities and now looks wrong. Someone should settle it before acting on either.

The variance matters more than the median

p90 first token is 10.57s against a 5.27s median — one request in ten waits twice the median, on a panel that renders progressively, so the wait is visible. Expansion off collapses p90 to 3.81s. Any future latency requirement should be stated at p90, because a p50 target hides exactly the requests that feel broken.

What would have to change, in measured order

1. Retrieval, 2.84s — largest block, and 2.58s of it is query expansion. QUERY_EXPANSION_ALTERNATES=0 removes it today; recall cost unmeasured beyond thirteen questions (T020).
2. Preprocessing, 1.77s — whether a search-page question needs all four calls is still open (T020c).
3. The answer model, 0.66s — no longer worth attention on these numbers, which is why the discrepancy above matters.

Documentation only; no code changes.

🤖 Generated with Claude Code