Skip to content

feat(server): report per-request prefill and decode timings in api responses - #814

Open
void-mckenzie wants to merge 1 commit into
FlashML-org:mainfrom
void-mckenzie:feat/server-timings-block
Open

void-mckenzie wants to merge 1 commit into
FlashML-org:mainfrom
void-mckenzie:feat/server-timings-block

Conversation

@void-mckenzie

Copy link
Copy Markdown
Contributor

Closes #813. Production deployments front FreeToken with proxies and dashboards that track per-request stats, and those read prefill/decode rates and cache hits from the response itself; without this block the numbers silently go missing. Every chat completion, completion and Responses response carries a llama.cpp-shaped timings object next to usage (fields and arithmetic from server_slot_stats::to_json), so frontends like llama-swap and Open Web UI show them. On streams it rides the final finish chunk as well as the include_usage chunk, like llama.cpp.

  • prompt_n counts only the tokens evaluated after the cached prefix, and decode rates divide by n_gen - 1 (the first token rides the last prompt batch), matching llama.cpp exactly, so cache_n + prompt_n reconstructs the prompt total.
  • Spans match feat(server): publish cumulative prefill and decode timing on /v1/stats #553's /v1/stats cumulative timing: prefill = request start to the first output reply, decode = between output replies. The two features report identical spans for the same request.
  • cache_n is unconditional (llama.cpp and stats semantics); --enable-cache-report still gates only usage.prompt_tokens_details.cached_tokens.
  • Engine and scheduler untouched; capture lives in the ack-consume loops, emission in the adapters.
  • Anthropic /v1/messages is left out on purpose: llama.cpp emits no timings there either (to_json_anthropic keeps the strict Anthropic schema, with the cache hit in usage.cache_read_input_tokens), and FreeToken already reports that field under --enable-cache-report. Adding an off-spec key would be looser than llama.cpp itself.

Verified

  • pytest tests/server/: 627 passed, 3 new tests (llama.cpp arithmetic, edge shapes, span capture) in the existing accounting file.
  • Live serving, unsloth GLM-5.3-Flash UD-IQ4_XS (RTX 5090, ft serve ... --moe-cache-size 800 --max-prefill-length 12288): a 19,008-token cold prefill reports 1727 t/s over 11.0 s (2 chunks); a 1,800-token coding decode reports 16.31 t/s / 61.29 ms per token; a repeated 1,300-token prompt reports cache_n: 1280 and prompt_n: 20, summing to the prompt total. All four surfaces (chat stream + full, completions, Responses) emit the block.

@void-mckenzie
void-mckenzie marked this pull request as ready for review October 23, 2026 19:36
@void-mckenzie

Copy link
Copy Markdown
Contributor Author
Image

Works with llama-swap now. When speculative decoding is eventually added, drafted/accepted fields can also be filled in to maintain parity with llama-server stats reporting.

KarrAcaRn pushed a commit to KarrAcaRn/FreeToken-ByAI that referenced this pull request Oct 24, 2026
…refill and decode timings in api responses

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Report per-request prefill/decode timings in API responses (llama.cpp-style timings block)

1 participant