5080deploy: OpenAI compat: stream/non-stream parser parity, chat logprobs under the parsers, logprobs/usage fixes, cache report + decode interleave by gdevenyi · Pull Request #92 · gdevenyi/FreeToken

gdevenyi · 2026-10-15T15:48:40Z

Stacked on #91 (the rebuild). This is the new 5080deploy tip c23d2d5d, deployed 2026-10-15. It makes the OpenAI-compatible endpoints correct and complete for this deployment's qwen3 reasoning parser and qwen3_coder tool parser.

1. Streaming and non-streaming now agree (5080/r6-oai-P, fixes #87, #88, #89, #90). For the same tokens, both modes return identical content, reasoning_content and tool calls:

qwen3 reasoning ends at the first </think> and never re-enters; a later <think>/</think> (in code or a tool argument) stays literal text. One whitespace rule around </think> in both modes. The one-shot parse replays the streaming parser.
qwen3_coder non-streaming keeps the text between tool calls; the whitespace rule after a call moved from per-chunk suppress_ws into the detector, so it no longer depends on chunking.
A bare <function=name> (no <tool_call>) is a call in both modes, but only for a declared tool name followed by <parameter= or </function>, so <function= in prose or code stays text. A <tool_call> named in prose stays content; a repeated opener counts as one block; a stray </tool_call> after a bare call is dropped; a function name stops at the next tag.
Parity tests feed every case token by token and character by character.

Visible change: with thinking on, production streamed every answer with a leading "\n\n" content delta that non-streaming did not have; that is gone. Non-streaming content now keeps whitespace before a tool call ("Let me check.\n\n"), as streaming always did.

2. Chat logprobs with the parsers on (5080/r6-oai-L; was a 400):

choices[].logprobs = {"content": [...], "refusal": null} covers visible content tokens only: reasoning, tool-call markup and hidden/special tokens are excluded. A segmenter keyed on token ids (<think>/</think> are single ids here) plus a shadow qwen3_coder detector decides per token, so stream and non-stream entry lists are identical for any chunking; leftovers ride the finish chunk.
Stop strings: entries are held until their text is released, and entries a stop string trims are marked and dropped (chat and completions).
Built only when logprobs is requested. Other parser families keep the 400 unless their markers are single ids.

3. Fixes in the existing logprobs/usage code (5080/r6-oai-F):

bytes is the token's real bytes (ByteLevel inverse, added tokens, padded ids), so a client can rebuild a split character: 🦀 = 3 entries whose bytes join to f0 9f a6 80.
/v1/completions: text_offset stamped by the detokenizer, correct across multi-byte text and held stop prefixes; streaming no longer drops entries of acks with no text; a top_logprobs key collision keeps the most likely candidate.
compute_logprobs builds its row index on the host, removing a host-device sync on every step where any row asks for logprobs.
/v1/responses usage.output_tokens_details.reasoning_tokens was hardcoded 0; now real (Codex reads it). reasoning_tokens is also right when generation stops inside the think block.
Non-stream chat choices carry "logprobs": null when not requested.
Hot path: the detokenizer's hold bookkeeping is skipped when nothing is held (+0.1-0.24 us/msg left over the base, measured).

Each branch was adversarially reviewed and its confirmed findings fixed with tests; the integration merge was reviewed the same way.

Launcher (run-freetoken.sh, backup run-freetoken.sh.bak-2026-10-15):

--enable-cache-report: usage.prompt_tokens_details.cached_tokens (and Responses input_tokens_details.cached_tokens) are now reported; on /v1/messages, input_tokens excludes the cached prefix (Anthropic semantics).
--decode-interleave-every 4 (carried feat(scheduler): interleave decode steps into long prefill runs FlashML-org/FreeToken#484, upstream issue DSV4: prefill priority starves in-flight decodes for the whole chunked-prefill burst FlashML-org/FreeToken#483). A/B with one stream decoding while a cold prompt prefills beside it (2 rounds):

	off	4	2
decoding stream's worst gap, 30K / 120K prefill	9.2 / 38.1 s	6.2 / 6.5 s	3.3 / 3.4 s
the long prompt's TTFT, 30K / 120K	10.6 / 39.6 s	10.9-11.0 / 41.2-41.7 s	11.3-11.6 / 43.0-43.7 s

The cost only applies while another request is decoding.

Live checks on the deployed server (oai_check.py, greedy, prefix pre-warmed so both modes see the same cache state):

check	previous deploy	this PR
tool-call turns, stream == non-stream (5 prompts)	3 of 3 thinking-on turns differ	all equal
chat logprobs with tools / without, thinking on and off	400	200; stream == non-stream; joined bytes == content
🦀 rebuilt from logprob bytes	400 (chat)	OK
responses reasoning_tokens	0	53
cached_tokens on a repeated 2.6K prompt	absent	2560

Cold greedy fingerprints are unchanged (4ac964c4… / 08996814… / 8c72d19d…), quality 14/15, 255K + side calls peak 15606 MiB, side calls 0.85-0.91 s. 15-minute agentic soak: pass (67 main turns, 43 side calls, 419 tool calls), no tracebacks since the deploy.

Tests: full not-slow GPU suite 3761 passed, 0 failed.

Not done: FlashML-org#324's defensive <tool_call>-inside-reasoning rule (no occurrence seen here); tool_choice: "required", seed and response_format are still not enforced (FlashML-org#692, FlashML-org#693). Note: asked to write a literal <think> into a file, Qwen3.8 itself emits <|im_start|>/<|im_end|> (raw tokens checked), so that case cannot be fixed server-side.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

compute_logprobs ran torch.nonzero over a CUDA bool mask, a host-device sync on every step where any row asked for logprobs, which stalls the overlap scheduler for every request in the batch. Sampler.prepare already has the per-row flags as a host list, so upload the requested-row index with make_device_tensor (pinned, non-blocking) and skip the gather and scatter entirely when every row asks. Values are unchanged. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

usage.output_tokens_details.reasoning_tokens was hardcoded to 0 on /v1/responses although GenResult and GenDone carry the detokenizer's count, so Codex always showed no reasoning output. Pass it through on the non-stream response and every streamed terminal event. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

A byte-level token that holds part of a multi-byte character decodes to U+FFFD on its own, so every such entry reported bytes [239, 191, 189] and a client could not rebuild the character (the 3 tokens of a crab emoji). Read the vocab piece instead: map it back through the GPT-2 byte table when the decoder is ByteLevel, take <0xNN> pieces as their byte under ByteFallback, and use the content of added tokens, which the byte table cannot invert. Ids past the real vocab (the padded logits tail that topk can return) report an empty token rather than risking the worker. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

Non-stream /v1/completions added up len(token) per entry, and a byte-level piece of a multi-byte character decodes to one U+FFFD each, so offsets drifted past the real text (the crab emoji's three pieces gave 0, 1, 2). Use the length of the text already sent when each entry's ack arrived, as streaming does, so a split character's pieces all point at its start. The legacy top_logprobs dict is keyed by token text, so partial-UTF-8 candidates collide on U+FFFD and the last (least likely) one won. Keep the most likely candidate per key. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

Streaming /v1/completions attached an entry only to its own ack's chunk and sent no chunk for an ack with empty incremental_output, so the entries of held text (the leading pieces of a multi-byte character, a partial stop-string prefix) and of EOS were dropped. Carry them to the next chunk with text, and put any left at the end on the finish chunk, so the joined stream equals the non-stream list. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

…he block reasoning_tokens counted generated tokens up to the </think> tag and reported 0 when that tag never came, so a reply cut by max_tokens (or an EOS) inside the qwen3 think block showed no reasoning at all. The detokenizer cannot see the prompt, so the tokenizer worker marks a prompt whose last reasoning tag is an opening one (the template pre-opened <think>), the flag rides SamplingParams and DetokenizeMsg like skip_special_tokens, and an unclosed block then counts every completion token. A <think> the model emits itself opens the block too. Thinking off renders a pre-closed block, so it still reports 0. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

OpenAI's chat.completion always carries choices[].logprobs, null when the request did not ask for logprobs; the key was left out. Streaming chunks are left as they are for now: chat logprobs under the parsers rework that path. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

…d one-shot The qwen3 parser disagreed with itself. Streaming re-entered reasoning on the first <think> seen after </think> (anywhere, including inside tool-call arguments), deleting the markers and moving the text between them into reasoning_content; one-shot deleted every <think> and split at the first </think>. Editing code that quotes the markers corrupted the tool call in both modes, differently. Whitespace also differed: one-shot stripped content after </think> and reasoning at both ends, streaming emitted the "\n\n" after </think> and the newline before it. qwen3 now gets its own parser: reasoning opens from the start (template pre-open) or on an explicit <think> at the very start of the generation, ends at the first </think> and is never re-entered; later markers are literal text. Reasoning, and content after a real </think>, are trimmed at both ends in both modes (streaming holds whitespace-only tails until text follows). Start-in-content output stays verbatim. The one-shot parse replays the streaming machine so the two paths cannot drift. glm and minimax keep the shared ThinkReasoningParser unchanged. The semantic-token test put a hidden <|im_start|> before <think>; Qwen never emits that (the prompt ends in "<|im_start|>assistant\n"), so the marker moves after the think block to keep testing the filter. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

…g responses One-shot qwen3_coder parsing kept only the text before the first <tool_call> (stripped) plus whatever followed the last </tool_call>, so prose between two calls vanished from non-streaming content while streaming showed it. Streaming had its own wobble: suppress_ws dropped a whitespace-only gap after a call only when it arrived as its own chunk. The detector now settles the gap itself, the same for any chunking: whitespace after a closed block is held, dropped when another block or the end follows (markup separation, what suppress_ws meant), and kept verbatim when text follows. detect_and_parse replays the streaming state machine on a fresh detector, so the one-shot content is every text segment the stream emits, in order, and the calls are the stream's calls (a call cut off mid-arguments falls back to the parse ledger, as the stream close does). parse_non_stream returns that result as is, and the serving layer accepts a parse that dropped markup without producing a call (an empty block), as streaming already did. Other tool formats keep their one-shot parse. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

…g too When the model skipped the <tool_call> wrapper, the old one-shot parse returned a tool call and also left the raw <function=...> markup in content, while streaming never looked for the bare form and emitted it all as content. (After the previous commit both modes showed it as text.) Both modes now parse it as a call and keep the markup out of content: an agent client needs the call executed, and the model only writes this shape when it means one. Streaming detection is made robust instead of greedy. A bare opener counts only when it names a DECLARED tool (not the unknown-tool forwarding a wrapped block gets) and is followed, after optional whitespace, by <parameter= or </function>. The name is bounded to 128 non-space chars, so a literal "<function=" in prose or code is held for at most that long and then released. The only new hold is a trailing partial of "<function="; '<' was already held as a possible "<tool_call>". The bare call ends at </function>, and the whitespace that follows it is treated like the whitespace after a closed block. The one-shot parse replays the same machine, so the two modes cannot disagree. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

text_offset was the length of the text already sent when a token's ack arrived. While a stop-string prefix is held back that lags the token's real start, so under the common stop=["\n\n"] the token after a single "\n" pointed at the "\n" (non-stream regressed in f1cede1; streaming had the same lag before it). The detokenizer is the only place that knows where each token's text lands, so it now reports the decoded length before the token (prior to holdback) on every reply. Both completions paths use it and clip to the returned text, which a matched stop string trims. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

…_in_reasoning A raw /v1/completions or /generate prompt that merely mentions <think> ends on an opening tag, so the flag is set for it. Nothing reads the count on those paths (only chat and responses report reasoning_tokens, and their prompts are always templated), so record the caveat rather than gate the scan on templating, which would also drop a raw prompt that deliberately ends inside <think>. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

A stop string whose prefix repeats ("aab", "\n\n\n") holds back more than one whole token, so mid-stream a pending entry's offset can run past the text sent so far. That is where its text will land; clipping it to the sent length gave "aaax" under stop=["aab"] offsets [0,1,1,3] instead of [0,1,2,3] and broke parity with non-stream. Clip only once the text is final: the finished ack's chunk, the finish chunk and non-stream. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

…reams The qwen3_coder detector now decides itself whether whitespace after a call is a separator (dropped) or part of the text that follows (kept), and emits "\n\n<div>x" as one piece. The special-token filter then splits that piece at the "<" and the serving layer's suppress_ws dropped the "\n\n" it had kept, so streaming lost whitespace the one-shot response returned. Leave that decision to detectors that replay their stream for the one-shot parse. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

The name pattern "<function=([^>]*)>" ran across newlines into "</tool_call>" when the opener had no ">" of its own: a well-formed stream with such a malformed opener produced a call named "x\n\n</tool_call>", and end-of-stream recovery, which appends "</tool_call>" to the leftover buffer, invented one from a max_tokens cut inside the tag that the one-shot response did not have. Exclude "<" from the name so the opener stays incomplete and yields no call. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

Since the one-shot parse replays the stream, a <tool_call> that the model only mentions ("the parser looks for <tool_call> tags") opened a block in both modes and dropped everything after it, where the old one-shot parse kept the text. Like a bare <function= lookalike, an opener whose next non-whitespace text cannot start an invoke or </tool_call> is now text. That also surfaces a non-qwen3_coder body (e.g. hermes JSON) as content instead of dropping it. Only qwen3_coder sets the flag; minimax keeps its behaviour. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

When the model skips the <tool_call> opener it can still emit the closer after </function>. The bare call was parsed, but the orphan "</tool_call>" leaked into content in both modes. Swallow it, whitespace allowed in between, right after a bare call closes. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

The qwen3 reasoning parser now opens on <think> only as the generation's first non-whitespace text, so a marker leaked before it leaves "<think>" in the reasoning. The test input was moved for that reason; record it at the input. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

…lock The literal-opener check made the first of "<tool_call>\n<tool_call>\n<function=..." text, leaking "<tool_call>\n" into content where the block used to ignore the stutter. An opener followed by another opener now opens the block again. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

The detokenizer sent one logprob entry per sampled token on that token's own reply, while it held the token's text back as a possible stop-string prefix and cut it at the match. The API layer only sees characters, so entries of trimmed tokens reached clients: non-streaming chat and /v1/completions listed them, and when a held prefix was released by the next token (stop "bc" over "b", "b", "c") a streaming chunk carried the entry of a token whose text never appeared. With stop strings, an entry now waits in the detokenizer for the reply that releases its token's text, and the finished reply carries the entries of trimmed tokens marked stop_trimmed, which chat and completions drop. UserReply.logprobs becomes a list for this. Requests without stop strings release each entry on its own reply, as before. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

…sers Chat logprobs returned 400 whenever a reasoning parser or tool parsing was on, which is always the case for Qwen3.8 in production. A LogprobSegmenter now picks the message-content entries out of the sampled ones, so reasoning and tool-call tokens never appear in choices[].logprobs.content. The segmenter reads only the entry sequence (token ids and token strings), so streaming and non-streaming select the same entries for any chunking. - The reasoning split is keyed on the <think>/</think> ids and follows the qwen3 parser: a leading <think> opens reasoning once, the first </think> ends it for good, and content after it is trimmed at both ends. - Tool markup is decided by a shadow qwen3_coder parser fed one token string at a time, so the literal-opener, bare <function=, stray-closer and value-containing-</tool_call> rules are the parser's own. A token is content when any of its characters is emitted as text. - Special and hidden ids (EOS included) are never content; non-streaming passthrough chat used to list the EOS entry. Streamed content entries ride content deltas; entries left at the end go on the finish chunk through GenDone.logprobs. The response shape is {"content": [...], "refusal": null}. The 400 stays for reasoning parsers other than qwen3, for think markers that are not single tokens, and for tool parsers whose one-shot parse does not replay their stream. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

…s and stream entries, host-built logprob row index, responses and truncated reasoning_tokens, logprobs null)

The Qwen3.8 tokenizer merges a bare call's closing ">" with the next character (">n", ">The"), so the text the shadow tool parser emits for that token sits at the end of "</function>n". The segmenter's leftmost-character fallback then marked the first matching character, which for "n", "t" or "c" is inside "function": logprobs.content listed the "function" markup token and dropped the real ">n" content token. Anchor emitted text at the region's end when it is a suffix, before falling back. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

…ow parse After </think> the qwen3 reasoning parser strips content's leading whitespace, so the tool parser sees "<function=" when the model writes " <function=". The segmenter fed the shadow tool parser the whole first token " <", which emitted the space as text and listed the token in logprobs.content although the reply had no content, only the call. Feed the first content token's text without its leading whitespace, so it counts only when a non-space character of it is emitted. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

…text When a stop string ended inside a token, the detokenizer released that token's entry unmarked (often before the cut, as soon as any of its text was out), and the chat segmenter judged it on the full token string. With stop "!" over " !" the entry was listed although its only visible text, a space, is trimmed from content; with stop "_c" over <tool_call> the segmenter took the id as a call opener while content showed "<tool", which then had no entry. With stop strings an entry now waits until all its token's text is out, and the finished reply marks the token a stop cuts inside with stop_visible, the length of its released text. The segmenter blank-checks and shadow-parses only that prefix and never treats such a token as a think marker. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

…d stream/non-stream parser consistency With stop strings the detokenizer now holds a logprob entry until its text is released, so one reply can carry several entries or none, and a per-reply text_offset can no longer describe them. The detokenizer stamps each entry with its own token's start instead, and UserReply.text_offset is gone; completions reads the offset from the entry, keeps the pending carry-forward for text-less replies, and drops stop_trimmed entries on both paths. The non-stream chat choice keeps the always-present logprobs key, now with the segmenter's {"content", "refusal"} object. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

The stop-string hold block ran for every message, building and replacing empty lists even for requests that never asked for logprobs, about 0.25 us per message (6-14% of the detokenizer's per-message cost on the Qwen3.8 tokenizer). Move it into DecodeStatus.release_logprobs and call it only when the message carries an entry or entries are still held. The new test pins that held entries are still released by a message without its own entry. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

_generate_full_impl ran untrimmed() on every ack even for requests without logprobs, and its result then kept whatever raw, unsegmented entries the acks carried. Read the flag once and skip both, so the protocol-neutral result carries entries only for a request that asked for them, as the streaming path already does. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

gdevenyi · 2026-10-15T16:00:54Z

Post-deploy checks for the interleave flag and the two agent-client paths:

--decode-interleave-every 4 under the slot checks. The deployed config with FREETOKEN_MOE_PREFETCH_VERIFY=meta and FREETOKEN_MOE_SLOT_AUDIT=64 (1550 slots), while cold 30K and 120K prompts prefilled beside a decoding stream (5 runs):
- verify: 58 windows, about 0.7 M decode layer calls checked, 0 bad;
- slot audit: 228 audits of all 1550 held slots, byte-checked, 0 bad;
- no warnings or tracebacks.
/v1/messages (Claude Code shape), streamed, thinking + tool: a thinking block, then a tool_use block with valid input, stop reason tool_use. On the second turn over the same system prompt, usage is input_tokens 49 and cache_read_input_tokens 1472 (--enable-cache-report).
/v1/responses (Codex shape), streamed, tool: output reasoning, then function_call with read({"path":"src/app.py"}), and reasoning_tokens 14.

Scripts are ~/models/bench/integ6/interleave/verify.sh and ~/models/bench/integ6/client_check.py.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

gdevenyi and others added 28 commits October 15, 2026 09:11

merge fork: logprobs and usage fixes (token bytes, completions offset…

33ee00e

…s and stream entries, host-built logprob row index, responses and truncated reasoning_tokens, logprobs null)

Navigation Menu

Repository navigation

5080deploy: OpenAI compat: stream/non-stream parser parity, chat logprobs under the parsers, logprobs/usage fixes, cache report + decode interleave - #92

5080deploy: OpenAI compat: stream/non-stream parser parity, chat logprobs under the parsers, logprobs/usage fixes, cache report + decode interleave#92
gdevenyi wants to merge 28 commits into
5080/42-rebuild-0bca88f7from
5080/43-openai-compat

gdevenyi commented Oct 15, 2026 •

edited

Loading

Uh oh!

gdevenyi commented Oct 15, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

gdevenyi commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gdevenyi commented Oct 15, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

gdevenyi commented Oct 15, 2026 •

edited

Loading