Design: how chat logprobs should behave when a reasoning or tool parser is active #719

What happened

On main, /v1/chat/completions ignores logprobs: true (#693). #224 adds logprobs and returns 400 whenever the server runs a reasoning parser or the request enables tool parsing, and leaves the combined case for later. So any deployment of a thinking model (for example Qwen3.8 with --reasoning-parser qwen3) has no chat logprobs, even with enable_thinking: false. Nothing defines yet what the entries should contain when the parsers hide or reshape tokens. This issue proposes rules for that.

Reference behaviour, from the current sources:

• vLLM (entrypoints/openai/chat_completion/serving.py) builds chat logprobs from all output token_ids, reasoning and tool-call tokens included. When streaming, it attaches each step's entries to whatever chunk that step produces. It drops them only when include_reasoning=false.
• SGLang (entrypoints/openai/serving_chat.py) attaches a step's entries to the first chunk emitted that step (reasoning, tool call or content). When a parser is on, it flushes leftovers in a separate empty-delta chunk.
• OpenAI documents choices[].logprobs.content as the message content tokens, with a separate refusal list. Its reasoning models do not return logprobs.

Neither vLLM nor SGLang gives the meaning OpenAI documents. Clients that use logprobs for confidence or scoring of the answer then get reasoning and tool-markup tokens mixed into content.

Minimal reproduction (CPU only). It shows how the parser reshapes text compared with token boundaries, and that the markers are single token ids:

# PYTHONPATH=python MODEL=nvidia/Qwen3.8-Flash-Next-NVFP4 python repro.py
import os
from transformers import AutoTokenizer
from freetoken.server.reasoning_parser import ReasoningParser
tok = AutoTokenizer.from_pretrained(os.environ["MODEL"])
for m in ["<think>", "</think>", "<tool_call>", "</tool_call>", "<|im_end|>"]:
    print(f"{m!r:16} -> {tok.encode(m, add_special_tokens=False)}")
text = "Let me think.</think>\n\nUse a <b> tag."
p = ReasoningParser("qwen3", force_reasoning=True)
for i in tok.encode(text, add_special_tokens=False):
    piece = tok.decode([i]); r, c = p.parse_stream_chunk(piece)
    print(f"{i:>7} {piece!r:12} reasoning={r!r:10} content={c!r}")
print("non-stream", ReasoningParser("qwen3", force_reasoning=True).parse_non_stream(text))

'<think>'        -> [248068]
'</think>'       -> [248069]
'<tool_call>'    -> [248058]
'</tool_call>'   -> [248059]
'<|im_end|>'     -> [248046]
...
     13 '.'          reasoning='.'        content=''
 248069 '</think>'   reasoning=''         content=''
    271 '\n\n'       reasoning=''         content='\n\n'
    361 ' <'         reasoning=''         content=' '
     65 'b'          reasoning=''         content='<b'
...
non-stream ('Let me think.', 'Use a <b> tag.')

Cause (current main, 9b68e00)

• The parsers work on text, not tokens. server/reasoning_parser.py holds back a partial marker (_split_trailing_partial, line 217). In the output above, the < of token ' <' is sent with the next token. detect_and_parse strips the text around </think> (lines 134, 153), but the streaming path does not, so \n\n streams as content.
• server/generation.py adds more reshaping on top: the stream loop (lines 705-722) and the tool-text router, which drops whitespace between tool calls (suppress_ws, lines 633-670).
• To line entries up with the emitted text, both parsers would have to report character offsets. feat(server): OpenAI-compatible logprobs for chat and legacy completions #224 notes the same: "needs the reasoning-parser interface to expose per-token classification".
• For the qwen3 and qwen3_coder formats, the markers are each exactly one token id. Each sampled id can be classified from the ids alone, without changing the parsers.

Proposed fix

Proposed rules for choices[].logprobs with a parser active:

1. logprobs.content has an entry for each sampled token that becomes visible message content. Reasoning tokens, tool-call tokens, marker tokens (<think>, </think>, <tool_call>, </tool_call>) and special/EOS ids get no entry. A turn that only calls tools has content: []. refusal is null. When a stop string cuts a token, its entry is kept only if part of its text is still visible.
2. top_logprobs stays the model's raw top-k at that position. An alternative such as </think> is fine there.
3. Streaming and non-streaming give the same list: the content entries of all chunks, joined in order, equal the non-streaming list. Whitespace that one path strips (after </think>, between tool calls) is left out in both paths.
4. Each content entry goes with the first content delta sent at or after that token's step. A token whose text the parser holds back therefore goes with the later delta. A hidden token's entry is never attached to a visible delta.
5. Entries still held at the end of the stream go on the finish_reason chunk, not dropped.
6. With enable_thinking: false and tool_choice: "none", the result is the same as with no parser.

Mechanism: classify each sampled token by its id, with a small state machine that runs alongside the existing text parsers. The states are reasoning, content and tool call, and the marker ids switch between them. The starting state is the same force_reasoning that _make_reasoning_parser uses (generation.py:389-422). Resolve the marker ids once from the tokenizer. Use a marker only if encode(marker) gives exactly one id that decodes back to it. Parser families whose state cannot be read from single marker ids keep the 400 (for example Harmony, where the channel name follows <|channel|> as plain text). One exception needs a signal from the text side: qwen3_coder also accepts a bare <function= with no <tool_call> (function_call_parser.py:1953). When the tool parser starts a call while the classifier is in content, the held entries should be relabelled as tool-call tokens.

Optional: a vLLM-style "all tokens" scope, or a separate reasoning_content logprob list behind a request flag, for clients that want reasoning confidence as well.

Environment

Seen serving nvidia/Qwen3.8-Flash-Next-NVFP4 on an RTX 5080 16 GB; the reproduction above is CPU-only on main 9b68e00.