Repository navigation
feat(server): report per-request prefill and decode timings in api responses - #814
Open
void-mckenzie wants to merge 1 commit into
Open
void-mckenzie wants to merge 1 commit into
void-mckenzie wants to merge 1 commit into
Conversation
void-mckenzie
marked this pull request as ready for review
October 23, 2026 19:36
Contributor
Author
KarrAcaRn
pushed a commit
to KarrAcaRn/FreeToken-ByAI
that referenced
this pull request
Oct 24, 2026
…refill and decode timings in api responses
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Closes #813. Production deployments front FreeToken with proxies and dashboards that track per-request stats, and those read prefill/decode rates and cache hits from the response itself; without this block the numbers silently go missing. Every chat completion, completion and Responses response carries a llama.cpp-shaped
timingsobject next tousage(fields and arithmetic fromserver_slot_stats::to_json), so frontends like llama-swap and Open Web UI show them. On streams it rides the final finish chunk as well as theinclude_usagechunk, like llama.cpp.prompt_ncounts only the tokens evaluated after the cached prefix, and decode rates divide byn_gen - 1(the first token rides the last prompt batch), matching llama.cpp exactly, socache_n + prompt_nreconstructs the prompt total.cache_nis unconditional (llama.cpp and stats semantics);--enable-cache-reportstill gates onlyusage.prompt_tokens_details.cached_tokens./v1/messagesis left out on purpose: llama.cpp emits notimingsthere either (to_json_anthropickeeps the strict Anthropic schema, with the cache hit inusage.cache_read_input_tokens), and FreeToken already reports that field under--enable-cache-report. Adding an off-spec key would be looser than llama.cpp itself.Verified
pytest tests/server/: 627 passed, 3 new tests (llama.cpp arithmetic, edge shapes, span capture) in the existing accounting file.ft serve ... --moe-cache-size 800 --max-prefill-length 12288): a 19,008-token cold prefill reports 1727 t/s over 11.0 s (2 chunks); a 1,800-token coding decode reports 16.31 t/s / 61.29 ms per token; a repeated 1,300-token prompt reportscache_n: 1280andprompt_n: 20, summing to the prompt total. All four surfaces (chat stream + full, completions, Responses) emit the block.