[elyasmnvidian]

When Switchyard translates an upstream response into Chat Completions, Pi and Oh My Pi see zero prompt-cache writes and show a lower cost than their configured rates give. Switchyard reports cache writes as prompt_tokens_details.cache_creation_tokens, but both clients read cache writes only from prompt_tokens_details.cache_write_tokens. They count the cache-write tokens as ordinary input instead.

For example, an Anthropic upstream reports 50 input tokens, 20 cache-read tokens, 30 cache-write tokens, and 10 output tokens. Switchyard returns prompt_tokens: 100 with cache_creation_tokens: 30. Pi records 80 input tokens and 0 cache writes, so it never applies the model's cache-write rate.

This PR renames the field to cache_write_tokens in the Chat Completions usage encoder. Buffered and streamed responses use the same encoder, so both change. The Responses encoder already uses cache_write_tokens. When Switchyard reads Chat Completions usage from an upstream, it still accepts both names.

A client that reads cache_creation_tokens from a translated Chat Completions response must read cache_write_tokens instead. I kept one name instead of sending both because Pi, Oh My Pi, and Switchyard's own Responses encoder use cache_write_tokens, and no code in this repository reads cache_creation_tokens from Switchyard's Chat Completions output.

The translation test for Responses to Chat Completions now checks cache_write_tokens. It fails with the old field name.

Earlier versions of this PR also added session cost reporting. #1045 has since added that to main, so this PR now contains only the field rename.

Local run

A local HTTP stub served Anthropic Messages responses with the usage above for both buffered and streamed requests. Switchyard used this config:

schema_version = 1

[llm_clients.upstream]
format = "anthropic_messages"
base_url = "http://127.0.0.1:55098"

[targets.claude]
id = "claude-stub"
llm_client = "upstream"

[routes.claude]
id = "route-claude"
type = "passthrough"
target = "claude"

The server ran with switchyard-server --config routes.toml --host 127.0.0.1 --port 55099. Pi 0.84.3 used a custom openai-completions provider that points to http://127.0.0.1:55099/v1, with rates of $1 input, $5 output, $0.10 cache read, and $2 cache write per million tokens. Each run sent one buffered request and one streamed request with curl and one pi -p "hi" --mode json turn, then printed the usage.

Before, on main at fd37aa762:

buffered: {"prompt_tokens":100,"completion_tokens":10,"total_tokens":110,"prompt_tokens_details":{"cached_tokens":20,"cache_creation_tokens":30}}
streamed: {"prompt_tokens":100,"completion_tokens":10,"total_tokens":110,"prompt_tokens_details":{"cached_tokens":20,"cache_creation_tokens":30}}
pi:       {"input":80,"output":10,"cacheRead":20,"cacheWrite":0,...,"cost":{...,"cacheWrite":0,"total":0.000132}}

After:

buffered: {"prompt_tokens":100,"completion_tokens":10,"total_tokens":110,"prompt_tokens_details":{"cached_tokens":20,"cache_write_tokens":30}}
streamed: {"prompt_tokens":100,"completion_tokens":10,"total_tokens":110,"prompt_tokens_details":{"cached_tokens":20,"cache_write_tokens":30}}
pi:       {"input":50,"output":10,"cacheRead":20,"cacheWrite":30,...,"cost":{...,"cacheWrite":0.00006,"total":0.000162}}

These rates are only for the example and do not reflect a real provider bill. I did not run Oh My Pi. I read its installed openai-completions provider instead, and it reads cache writes only from prompt_tokens_details.cache_write_tokens.

[github-actions]

PR Preview Action v1.8.2
🚀 View preview at https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-947/
Built to branch gh-pages at 2026-10-27 22:25 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[coderabbitai]

No actionable comments were generated in the recent review.

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA-NeMo/Switchyard/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: caae6ca2-d51c-4bb2-b879-616d7e2e00c5

📥 Commits

Reviewing files that changed from the base of the PR and between 3a918cb and f58951a.

📒 Files selected for processing (26)

• crates/libsy-llm-client/tests/observability.rs
• crates/protocol/src/llm.rs
• crates/switchyard-nemo-relay-plugin/src/runtime.rs
• crates/switchyard-server/src/cli.rs
• crates/switchyard-server/src/lib.rs
• crates/switchyard-server/src/pricing.rs
• crates/switchyard-server/src/routing_log.rs
• crates/switchyard-server/src/usage_metrics.rs
• crates/switchyard-server/tests/cli.rs
• crates/switchyard-server/tests/server.rs
• crates/switchyard-translation/src/codecs/anthropic/buffered.rs
• crates/switchyard-translation/src/codecs/anthropic/stream.rs
• crates/switchyard-translation/src/codecs/common.rs
• crates/switchyard-translation/src/codecs/openai_chat/buffered.rs
• crates/switchyard-translation/src/codecs/openai_chat/stream.rs
• crates/switchyard-translation/src/codecs/responses/buffered.rs
• crates/switchyard-translation/src/codecs/responses/stream.rs
• crates/switchyard-translation/tests/response_translation.rs
• crates/switchyard-translation/tests/stream_translation.rs
• docs/integrations/costs.md
• docs/integrations/oh_my_pi.md
• docs/integrations/pi.md
• examples/costs/claude_statusline.py
• examples/costs/display.ts
• examples/costs/omp.ts
• examples/costs/pi.ts

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The change adds provider-reported and estimated cost tracking, including per-model pricing, cache-duration usage metadata, routing-log and session cost totals, and CLI configuration. It also records additional failed and routing calls, preserves cost metadata across supported usage translations, and adds client status displays and setup documentation.

Changes

Session Cost Tracking

Layer / File(s)	Summary
Usage metadata and translation crates/protocol/src/llm.rs, crates/switchyard-translation/src/codecs/*, crates/switchyard-translation/tests/*, crates/libsy-llm-client/tests/observability.rs, crates/switchyard-nemo-relay-plugin/src/runtime.rs	Usage types and codecs capture and encode provider costs and one-hour cache-write counts. Translation tests cover buffered and streamed usage.
Pricing rules and configuration crates/switchyard-server/src/pricing.rs, crates/switchyard-server/src/cli.rs, crates/switchyard-server/src/lib.rs, crates/switchyard-server/tests/cli.rs	Pricing files are parsed and validated, and cost calculation prefers valid provider charges before estimating from model rates and usage. The CLI loads pricing into the routing log and requires a routing-log file.
Routing records and session totals crates/switchyard-server/src/routing_log.rs, crates/switchyard-server/src/lib.rs, crates/switchyard-server/src/usage_metrics.rs, crates/switchyard-server/tests/server.rs	Routing records store calculated costs, and session snapshots aggregate cost totals. Observers record failed answers and classifier or judge calls; usage metrics record stream failures. Server tests check cost accounting across requests and models.
Client displays and guidance examples/costs/*, docs/integrations/costs.md, docs/integrations/pi.md, docs/integrations/oh_my_pi.md	Examples fetch session statistics and display cost totals. Integration documentation describes pricing setup, report fields, and client-specific display options.

Priority: Normal

Estimated code review effort: 4 (Complex) | ~60 minutes

Merge Risk: Minimal · up to f5895

No merge-blocking issue was identified in the reviewed session-cost changes; normal checks remain appropriate before merging.

Pre-merge checks: 4

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title accurately describes the Chat Completions cache-write field change, which is a real part of the pull request. It does not describe the broader session-cost reporting work, but the title chec…

Full details: Title check

Explanation

The title accurately describes the Chat Completions cache-write field change, which is a real part of the pull request. It does not describe the broader session-cost reporting work, but the title check allows a title that covers a valid aspect of the changeset.

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{A rabbit counts the tokens bright,
And checks the rates by moonlit light.
“Reported cost,” it nibbles, “wins;
Estimated sums come next in bins.”
Session totals hop into view,
With cache-write counts included too.}

_{Comment @coderabbitai help to get the list of available commands.}