[zaneni6]

What

continue_final_message (vLLM's field, Hugging Face's apply_chat_template flag) on POST /v1/chat/completions. When the last message is a partial assistant message, the model continues it instead of starting a new reply:

{"messages": [
   {"role": "user", "content": "Is Paris the capital of France? Answer yes or no."},
   {"role": "assistant", "content": "The answer is:"}],
 "continue_final_message": true, "max_tokens": 1, "logprobs": true, "top_logprobs": 5}

On llama3.2:1b the one logprobs entry is the next-token distribution after the prefix: ' Yes' −0.098, ' No' −2.97. This scores a classification or multiple-choice answer through the model's own chat template. /v1/completions and the Ollama endpoints are unchanged.

How

• Prompt. The template is rendered with add_generation_prompt=false and cut right after the final message's text, using transformers' algorithm (src/common/chat/chat_continue.cpp). The same template gives the same prompt here as in vLLM/HF. Every model renders through one helper, AutoModel::_render_chat_template.
• Models. Models opt in through supports_continue_final_message().
  • In continue mode a model keeps every prompt token, so no generation-prompt trim, and feeds no think preamble.
  • Supported: every multi-turn model except gpt-oss (Harmony).
  • Single-turn (flash) models and gpt-oss return 400.
• Validation. Returns 400 invalid_request_error when:
  • the last message is not from the assistant;
  • the final message has tool_calls, images or audio, or no text;
  • the request asks for response_format JSON or a tool_choice that forces a call;
  • add_generation_prompt: true is sent with the flag, which vLLM also refuses;
  • the template rewrites the message (for example a closed <think> block). HF raises in this case too.
• Output.
  • content is only the continuation, with its whitespace as written (' Yes'). logprobs cover exactly the continued tokens.
  • The reasoning state is read from the continued turn only.
  • A model that reasons first (DeepSeek-R1, qwen3-tk, nanbeige) may read the message as its reasoning and close it with </think>. The text before that tag goes to reasoning_content, as vLLM's reasoning parsers do. A stream from such a model holds its output until the tag or the end of the reply.

Fixes found on the way (all models, not only continuation)

• max_tokens was off by one. The token sampled after prefill was emitted but not counted, so max_tokens: N produced N+1 tokens while usage.completion_tokens reported N. This is fixed in _shared_generate and in every model's own decode loop, including the MTP speculative loop, which now caps each draft at the tokens left.
• token_history could drift from the KV cache.
  • A token cut off by max_tokens, a cancel or the context cap was recorded as run.
  • Several custom loops forwarded their first sample without recording it.
  • Either way, the next request's prefix match could skip or misplace a token.
• SentencePiece byte tokens (<0x0D>) were printed as their name. gemma3 wrote a literal <0x0D> before a JSON answer.
• Single-turn models refused conversation history with HTTP 200 and an error body. They now return 400.
• The --always-thinks test runs gave reasoning models no room to reason before answering, so content, JSON and tool calls never arrived. The harness now adds a reasoning budget (--reasoning-budget, default 1024) to every answer budget. On main, qwen3-tk failed 33 tests, nanbeige 17 and lfm2.5-tk 10 for this reason.

Tests

• src/test/chat_continue: CPU-only checks of the cut on the template shapes FLM ships (Llama 3, ChatML, Qwen3 think, Gemma, DeepSeek, VL typed content) and of the output parser for continued text, streaming included. It is added to the Ubuntu CI as the chat-continue job.
• openai_api_test/test_continue_final_message.py covers:
  • continuation instead of a new reply, and the next-token score;
  • logprobs coverage and stream/non-stream parity;
  • typed content and a cache-resumed continuation;
  • a follow-up turn, tools under auto, and the 400 cases.
• test_logprobs.max_tokens_is_exact.
• On NPU hardware, before the last round of fixes, llama3.2:1b, qwen3:0.6b, qwen3-it, qwen2.5, phi4-mini, gemma3:4b, lfm2, minicpm-v, qwen2.5vl, qwen3vl, gemma4 12B, qwen3.6-35B and qwen3.8-27B passed the whole suite. The full matrix on the final build is still running; I'll post it here when it finishes.

Docs

New sections in request_parameters.md (Continuing an Assistant Message) and openapi.md (Score the Next Token After a Partial Answer), both labelled as available from v1.0.11. Merging this publishes them on fastflowlm.com straight away.

🤖 Generated with Claude Code

[zaneni6]

NPU hardware results (final build)

Run on a Strix Point laptop against 139252ff (code is unchanged since; e62c97e1 only changes a test budget).

Full openai_api_test suite, one server session per model:

Model	Passed	Failed	Skipped
qwen3.5:0.8b	69	0	12
hy-mt2-flash:1.8b	52	0	29
llama3.2:1b	65	0	16
qwen3:0.6b	68	0	13
gemma3:1b	65	0	16
gemma4-it:e2b	81	0	0
gemma4e-flash:e2b	12	0	12 (continuation is refused on single-turn models)

The modules this PR changes (continue_final_message / logprobs / response_shape), with a fresh server per module:

Model	continue_final_message	logprobs	response_shape
qwen3-tk:4b (--always-thinks)	12 / 0 / 0	11 / 1 / 0 *	6 / 0 / 0
lfm2.5-tk:1.2b (--always-thinks)	10 / 0 / 2	12 / 0 / 0	6 / 0 / 0
nanbeige4.1:3b (--always-thinks)	11 / 0 / 1	12 / 0 / 0	6 / 0 / 0
qwen3-it:4b	12 / 0 / 0	11 / 0 / 1	6 / 0 / 0
qwen2.5-it:3b	12 / 0 / 0	11 / 0 / 1	6 / 0 / 0
phi4-mini-it:4b	12 / 0 / 0	10 / 0 / 2	5 / 0 / 1
gemma3:4b	12 / 0 / 0	10 / 0 / 2	5 / 0 / 1
lfm2:1.2b	12 / 0 / 0	11 / 0 / 1	6 / 0 / 0
minicpm-v-4.7:1b	12 / 0 / 0	12 / 0 / 0	6 / 0 / 0
qwen2.5vl-it:3b	12 / 0 / 0	10 / 0 / 2	5 / 0 / 1
qwen3vl-it:4b	12 / 0 / 0	11 / 0 / 1	6 / 0 / 0
gemma4-it:12b	12 / 0 / 0	12 / 0 / 0	6 / 0 / 0
qwen3.6-moe:35b-a3b	12 / 0 / 0	12 / 0 / 0	6 / 0 / 0
qwen3.8-mtp:27b	11 / 0 / 1	12 / 0 / 0	running

Each cell is passed / failed / skipped. Skips are capability skips: no tool syntax, no reasoning, or no cache resume to compare against.

* top_logprobs_default asked for 16 tokens, which a model that always reasons spends entirely on reasoning. e62c97e1 gives it an answer budget. Every module that ran after that change passed.

Notes:

• max_tokens_is_exact passes on every model above, including qwen3.8-mtp's speculative path. That covers the off-by-one fix in every model's decode loop.
• On main, the always-thinking models failed 10–33 tests each because reasoning used up the answer budget. Here they pass with --always-thinks.
• gpt-oss was not run. deepseek-r1 and deepseek-r1-0528 are still running; I'll add them here.
• The two failed sweep-* CI jobs ran no steps. Both sat on the xsjnuckrck01-flm-sweep runner for 10 minutes, which is a runner problem, not this change.

[zaneni6]

NPU hardware results: the rest

These finish the table above. Each cell is passed / failed / skipped, with one server session per module.

Model	continue_final_message	logprobs	response_shape
qwen3.8-mtp:27b	11 / 0 / 1	12 / 0 / 0	6 / 0 / 0
deepseek-r1-0528:8b (--always-thinks)	11 / 0 / 1	11 / 0 / 1	5 / 0 / 1
deepseek-r1:8b (--always-thinks)	11 / 0 / 1	11 / 0 / 1	5 / 0 / 1

• I re-ran top_logprobs_default on qwen3-tk:4b after e62c97e1, and it passes.
• The continuation skips on these three models are cached_continuation_matches_cold. None of them resumes a two-message continuation from cache, so there is nothing to compare, and the test skips instead of passing without checking anything.
• Both DeepSeek models are where the reasoning-first parsing matters most. Before it, they returned a stray </think> in content; every continuation case now passes.

Every model except gpt-oss now has results on the final build, with no failures left.