Streaming Anthropic responses undercount uncached input by the cache-read amount #435

What happens

For an Anthropic upstream, usage.input_tokens already excludes cache reads — the cached part is reported separately as cache_read_input_tokens. The non-streaming path honours that: parse_usage (src/gateway.rs:71-89) returns (input, cached, output, cache_write) with no subtraction, and the module states the rule at src/gateway.rs:57 — for anthropic, read usage.cache_read_input_tokens, and do not subtract it, because input_tokens already excludes cache hits (the source comment is in Chinese; rendered here in English).

The streaming path does not honour it. UsageCapture::finish ends with one statement, applied to every protocol alike:

((input - cached).max(0.0), cached, output, cache_write)

src/gateway.rs:878

For the OpenAI-shaped protocols that subtraction is correct, because prompt_tokens includes the cached part. For Anthropic it subtracts tokens that were never inside input_tokens: uncached input is undercounted by cache_read_input_tokens, and clamped to zero when cached >= input.

So the same upstream response bills differently depending on whether the client asked for a stream.

Worked example

Catalogue price for claude-sonnet-5.5 (config/config.example.toml): input $2.00/M, cache hit $0.10/M, output $10.00/M.

A streaming call where Anthropic reports input_tokens = 2,000, cache_read_input_tokens = 60,000, output_tokens = 1,000:

A fifth of the call goes unbilled, and the entire uncached segment drops to zero. The identical response fetched non-streaming bills $0.020000. In points, $0.004000 is about 0.0288.

The repository's own streaming test reproduces the same arithmetic at a smaller scale: src/gateway.rs:3028-3039 feeds input_tokens = 80 with cache_read_input_tokens = 70 and asserts input == 10.0. The correct value is 80 — the assertion currently pins the defect.

Who is affected

• Streaming requests (stream: true) to an Anthropic-protocol upstream, and only when that upstream reports cache_read_input_tokens > 0.
• Non-streaming Anthropic requests are already correct and bill the full input_tokens.
• Each affected call settles low by cache_read_input_tokens × input_per_m. Long prompts with a large cached prefix — the case caching exists for — are exactly where the loss is largest.
• Cross-protocol streaming (Anthropic-shaped usage converted for an OpenAI-shaped client) reads a separate slot in sse.rs and is a different path.

What fixing it does to existing bills

It raises them, for calls settled after it lands. Recorded transactions and usage_records rows are written at settlement time and are not rewritten, so history stays exactly as recorded — but a reconciliation window that straddles the change will show a step, and a user who watches their spend will see the same traffic cost more afterwards. That is the honest summary: this fix recovers revenue the platform is currently giving away, and it is a visible change for anyone reading the ledger closely.

Why it was not folded into #433 or #434

#433 introduced the cache-write price and #434 exposed it in the admin model editor. This arithmetic predates both, and correcting it moves real balances, so it was pinned as-is rather than silently changed inside a pricing PR. It belongs in its own change, with the bill impact stated up front — which is this issue.

Proposed fix

In UsageCapture::finish, subtract the cached part only for the protocols whose input token count includes it, and leave anthropic at input unchanged — the same rule parse_usage already applies and src/gateway.rs:57 already documents. The test at src/gateway.rs:3039 changes its expectation from 10.0 to 80.0, and a test should pin that the non-streaming path keeps billing the full input_tokens, so a later refactor cannot "unify" the two paths back into the bug.

Notes

• The default branch already carries feat(billing): bill cache-write (cache-creation) tokens #433 and feat(admin): accept the cache-write price in the model editor #434; nothing here is a regression from either.
• Please comment before starting if you would rather split the correction from the test expectation change.