Intermittent '</think>' in message.content on a tool call (qwen3 reasoning parser, 1 of 13) #6

Activity

1. gdevenyi commented on Sep 15, 2026

   

   gdevenyi

   on Sep 15, 2026

   OwnerAuthor

   More actions

   Not reproduced on the current stack; parser mechanism confirmed offline

   Setup: 5080deploy @ 2d56708 (adds FlashML-org#475, FlashML-org#339, FlashML-org#338, FlashML-org#169, FlashML-org#231, FlashML-org#389 over the 45373a6 where this was seen), production config from #17: FP8 dense + FP8 lm_head, nvfp4 KV, default sampling (temperature 1.0, top_k 20, top_p 0.95).

   Chat completions: the exact request above, sent 60 times. All 60 returned content == "", non-empty reasoning_content and the list_files tool call.

   Raw completions: to see what the model emits before any parser, the same prompt was rendered with the checkpoint's chat template (tools, reasoning_effort="low"). It is 373 tokens, the same count the server reports for the chat request. It was sent 100 times to /v1/completions. </think> is a non-special added token (id 248069), so it survives detokenization. All 100 completions contain exactly one </think>, e.g.:

   The user wants me to list the 7 most recent files using the list_files tool. This is straightforward - I just need to call the tool with a limit of 7.\n</think>\n\n<tool_call>\n<function=list_files>\n<parameter=limit>\n7\n</parameter>\n</function>\n</tool_call>
   

   How the reported output can arise (offline, through the real generate_full / generate_events path with reasoning_parser="qwen3" and tool_call_parser="qwen3_coder", using the tests/server/test_streaming_model_matrix.py fakes):

   raw completion	non-stream content	stream content	tool call
   reasoning</think>\n\n<tool_call>...	''	'\n\n'	ok
   reasoning</think>\n\n</think>\n\n<tool_call>...	'</think>'	'\n\n</think>\n\n'	ok
   reasoning</think>\n\n<tool_call>...</tool_call>\n</think>	'\n</think>'	'\n\n</think>'	ok

   • Cause: a doubled </think> from the model gives exactly the reported content == "</think>" next to a correct tool call. ThinkReasoningParser.detect_and_parse splits on the first closer only, and for qwen3 no special-token strip runs on content (_leaked_special_tokens strips think tags only for deepseekv32). So the most likely explanation is a rare sampled duplicate closer. The run where it was seen used bf16 dense and lm_head; the logits differ slightly under feat(qwen4_exp): load-time per-tensor FP8 dense projections (W8A8 via _scaled_mm) FlashML-org/FreeToken#389's FP8, which may be why it's rarer now.
   • Why no PR: it isn't reproduced, so a fix can't be shown to fail before and pass after on real output.
   • If it recurs: a narrow hardening would treat any </think> directly following the first one, with only whitespace between, as part of the closer, in both detect_and_parse and the streaming path. A closer with no opener can't be meaningful content.