feat(server): continue_final_message on /v1/chat/completions (+ exact max_tokens)

.github/workflows/ubuntu-build.yml

@@ -25,6 +25,19 @@ jobs:
           cmake --build build-d3-contract -j"$(nproc)"
           ctest --test-dir build-d3-contract --output-on-failure
 
+  # continue_final_message (src/common/chat): where the prompt is cut on the template shapes the
+  # models ship, and how the output parser reads the continued text. CPU only, no NPU.
+  chat-continue:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+      - name: Build and run the chat continue test
+        run: |
+          cmake -S src/test/chat_continue -B build-chat-continue -DCMAKE_BUILD_TYPE=Release
+          cmake --build build-chat-continue -j"$(nproc)"
+          ctest --test-dir build-chat-continue --output-on-failure
+
   package-deb:
     runs-on: ubuntu-latest
     strategy:

docs/docs/instructions/server/openapi.md

@@ -17,6 +17,8 @@ parent: Local Server (Server Mode)
   - [Example: Use a File as the Prompt](#-example-use-a-file-as-the-prompt)
   - [Example: Batch Requests (Multiple Prompts)](#-example-batch-requests-multiple-prompts)
   - [Example: Use Temperature, Top-p, and Presence Penalty](#-example-use-temperature-top-p-and-presence-penalty)
+  - [Example: Token Log Probabilities](#-example-token-log-probabilities)
+  - [Example: Score the Next Token After a Partial Answer](#-example-score-the-next-token-after-a-partial-answer)
   - [Example: Multi-Image Input](#-example-multi-image-input)   
   - [Example: Multi-Modal Input](#️-example-multi-modal-input)   
   - [Example: Flash Models (Single-Turn, Pinned System Prompt)](#-example-flash-models-single-turn-pinned-system-prompt)   
@@ -358,6 +360,41 @@ Streaming works the same way: each content chunk carries `choices[0].logprobs.co
 
 ---
 
+## 🎯 Example: Score the Next Token After a Partial Answer
+
+> Available from FLM v1.0.11.
+
+To see how likely each answer is, end `messages` with the start of the assistant's reply and set `continue_final_message` (an `extra_body` field). The model continues that text instead of starting a new reply, and the first log probability entry is its distribution for the very next token:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://127.0.0.1:52625/v1", api_key="flm")
+
+response = client.chat.completions.create(
+    model="llama3.2:1b",
+    messages=[
+        {"role": "user", "content": "Is Paris the capital of France? Answer yes or no."},
+        {"role": "assistant", "content": "The answer is:"},
+    ],
+    max_tokens=1,
+    temperature=0,
+    logprobs=True,
+    top_logprobs=5,
+    extra_body={"continue_final_message": True},
+)
+
+for alt in response.choices[0].logprobs.content[0].top_logprobs:
+    print(f"{alt.token!r:10} {alt.logprob:8.4f}")
+```
+
+- The prompt goes through the model's own chat template, so the score is for the context the model is used to.
+- `content` is only the continuation, without your text, and keeps its whitespace as the model wrote it (`' Yes'`, not `'Yes'`).
+- Use a model with thinking off (`think: false`, or a non-reasoning model). A model that always reasons first reads your text as its reasoning, so its first token is not the answer.
+- Continuing works on every chat model except single-turn (flash) models and `gpt-oss`. What else is refused, and how the prompt is cut: [Continuing an Assistant Message](https://fastflowlm.com/docs/instructions/server/request_parameters/#-continuing-an-assistant-message-continue_final_message).
+
+---
+
 ## 📷 Example: Multi-Image Input
 
 Send multiple images together with a text prompt.  

docs/docs/instructions/server/request_parameters.md

@@ -20,6 +20,7 @@ This page says what FLM does with the fields of a `/v1/chat/completions` or `/v1
 - [The Penalty Window](#-the-penalty-window)
 - [`seed` and Repeatable Output](#-seed-and-repeatable-output)
 - [Reasoning in `reasoning_content`](#-reasoning-in-reasoning_content)
+- [Continuing an Assistant Message (`continue_final_message`)](#-continuing-an-assistant-message-continue_final_message)
 - [Fields FLM Doesn't Know](#-fields-flm-doesnt-know)
 - [Long Prompts: Prefill Chunking](#-long-prompts-prefill-chunking)
 
@@ -179,6 +180,54 @@ Turning thinking on and off, streaming examples and the full message format: [Re
 
 ---
 
+## ⏩ Continuing an Assistant Message (`continue_final_message`)
+
+> Available from FLM v1.0.11, on `/v1/chat/completions`.
+
+By default, a request whose last message is from the assistant gets a **new** reply: the chat template closes that message and opens a fresh assistant turn. Set `continue_final_message: true` and the model **continues** the last message instead, from exactly where its text ends. The field is vLLM's, and Hugging Face's `apply_chat_template` has the same flag.
+
+```python
+response = client.chat.completions.create(
+    model="llama3.2:1b",
+    messages=[
+        {"role": "user", "content": "Is Paris the capital of France? Answer yes or no."},
+        {"role": "assistant", "content": "The answer is:"},
+    ],
+    max_tokens=1,
+    logprobs=True,
+    top_logprobs=5,
+    extra_body={"continue_final_message": True},
+)
+first = response.choices[0].logprobs.content[0]
+print(repr(first.token), first.logprob)   # ' Yes' -0.098 on llama3.2:1b
+```
+
+Use it to score what the model would say at an exact point, through the model's own chat template: classification, multiple-choice evaluation, or checking how confident the model is in an answer. With `logprobs`, the first entry is the model's distribution for the token right after your text (see [the example](https://fastflowlm.com/docs/instructions/server/openapi/#-example-score-the-next-token-after-a-partial-answer)).
+
+What you get back:
+
+- **`content` is only the continuation.** Your text isn't repeated, so `prefix + content` is the whole message. Its whitespace is kept as the model wrote it (`' Yes'`, not `'Yes'`), even when the continuation is only whitespace. In a stream, trailing whitespace arrives once the text after it, or the end of the reply, does.
+- **`logprobs` cover only the continued tokens.** Your text is part of the prompt, so it has no entries.
+- **`usage.prompt_tokens` includes your text**, and `completion_tokens` counts the continuation.
+- **The prompt is cut the way Hugging Face cuts it.** FLM renders the chat with no generation prompt, finds the last occurrence of the final message's text, and stops right after it. One exception: a few templates write a finished message differently from how they open a reply (Qwen3.5 leaves out the empty `<think></think>` block, Gemma4 12B with thinking off its empty thought channel, LFM2.5-Thinking its opening `<think>`). Cut that way, the prompt looks like a finished turn and the model ends it, so for those templates FLM puts your text after the reply's opening instead. Templates that trim message content also drop trailing whitespace from your text, so `"The answer is: "` is read as `"The answer is:"`. A template that keeps the space keeps it, unless your text also starts with whitespace (Hugging Face drops the trailing space then too), and the model then sees a separate space token, which changes what it predicts next.
+- **Models that reason first may read your text as reasoning.** `deepseek-r1`, `deepseek-r1-0528`, `qwen3-tk`, `lfm2.5-tk`, `nanbeige4.1`, and any model with `think` on, start every reply by reasoning. They can treat the message you continue as the start of that reasoning: they go on, close it with `</think>`, then answer. FLM then returns the continuation in `reasoning_content` and the answer after it in `content`, as vLLM's reasoning parsers do. A stream from such a model sends nothing until it either writes that tag or ends. To score the next token, use a model with `think` off: a reasoning model's first token is reasoning, not the answer.
+
+Rejected with HTTP 400 (`invalid_request_error`), for streaming requests too:
+
+| Request | Why |
+|---|---|
+| `continue_final_message` that isn't a boolean | |
+| The last message isn't from the assistant | There is nothing to continue. |
+| `add_generation_prompt: true` together with it | They ask for opposite things; vLLM refuses this too. Sending `add_generation_prompt: false` is fine. |
+| The last message has `tool_calls`, images or audio, or no text | Only text can be continued. |
+| `response_format` (JSON), or `tool_choice` `"required"` / a named function | The grammar for these starts at the beginning of a reply. `tool_choice` `"auto"` and `"none"` work. |
+| The chat template rewrites the last message, for example by moving a closed `<think>…</think>` block out of it | Its text no longer appears as you sent it, so there is no place to continue from. An unclosed `<think>` is kept, and the model goes on reasoning. |
+| The model can't continue a message: single-turn (flash) models and `gpt-oss` | The message is `continue_final_message is not supported by <model>`. |
+
+`continue_final_message: false` or `null` keeps the default behaviour. FLM does not read `add_generation_prompt` otherwise.
+
+---
+
 ## 📨 Fields FLM Doesn't Know
 
 **FLM ignores fields it doesn't read. It doesn't reject them.** OpenAI SDKs, LangChain and agent frameworks send fields that FLM has no use for, and they still work against FLM. The downside is that a misspelt field (`temprature`) is dropped without any error or log line, so check your spelling if a setting seems to have no effect.
@@ -191,7 +240,7 @@ Turning thinking on and off, streaming examples and the full message format: [Re
 | An `options` object (Ollama style) on a `/v1` endpoint | Ignored. Put sampling fields at the top level of the request. |
 | A sampling field outside its range | Ignored with a warning in the server log. The previous value stays in effect ([see above](#️-sampling-fields)). |
 | A sampling field of the wrong JSON type, such as `"temperature": "0.7"` | The request fails with an error. Send numbers as JSON numbers. |
-| A malformed `messages`, `tool_choice`, `parallel_tool_calls`, `response_format`, `logprobs`, `top_logprobs`, `seed` or `repeat_last_n` | Rejected with HTTP 400 and `"type": "invalid_request_error"` before anything is generated, for streaming requests too. |
+| A malformed `messages`, `tool_choice`, `parallel_tool_calls`, `response_format`, `logprobs`, `top_logprobs`, `seed`, `repeat_last_n` or `continue_final_message` | Rejected with HTTP 400 and `"type": "invalid_request_error"` before anything is generated, for streaming requests too. |
 
 FLM's own extensions to the OpenAI request, all top-level fields (`extra_body` in the OpenAI SDK):
 
@@ -201,6 +250,7 @@ FLM's own extensions to the OpenAI request, all top-level fields (`extra_body` i
 | `top_k`, `repetition_penalty` | [Sampling Fields](#️-sampling-fields) |
 | `repeat_last_n` | [The Penalty Window](#-the-penalty-window) |
 | `backend` | [Choose an Execution Backend](https://fastflowlm.com/docs/instructions/cli/#-choose-an-execution-backend) |
+| `continue_final_message` (vLLM's name) | [Continuing an Assistant Message](#-continuing-an-assistant-message-continue_final_message) |
 
 ---
 

src/common/AutoModel/automodel.cpp

@@ -6,6 +6,7 @@
 /// \note This is a source file for the auto_model class
 
 #include "AutoModel/automodel.hpp"
+#include "chat/chat_continue.hpp"
 #include "utils/npu_platform.hpp"
 
 #include <filesystem>
@@ -40,12 +41,13 @@ int resolve_context_length(const nlohmann::ordered_json& model_info,
 
 
 ModelRequestError::ModelRequestError(
-    int http_code, bool session_cleared, std::string message)
+    int http_code, bool session_cleared, std::string message, std::string type)
     : std::runtime_error(std::move(message)),
-      http_code_(http_code), session_cleared_(session_cleared) {}
+      http_code_(http_code), session_cleared_(session_cleared), type_(std::move(type)) {}
 
 int ModelRequestError::http_code() const noexcept { return http_code_; }
 bool ModelRequestError::session_cleared() const noexcept { return session_cleared_; }
+const std::string& ModelRequestError::type() const noexcept { return type_; }
 
 AutoModel::AutoModel(flm_rt::device* npu_device_inst, std::string current_model) {
     this->npu_device_inst = npu_device_inst;
@@ -537,6 +539,13 @@ bool AutoModel::_shared_insert(chat_meta_info_t& meta_info, std::vector<int>& to
         clear_context();
         skip_count = 0;
     }
+    else if (skip_count >= tokens.size() && !tokens.empty()) {
+        // The engine holds the whole prompt already (the same request again), but
+        // sampling needs the logits of its last token, and the cache cannot step
+        // back one: read it in again from the start.
+        clear_context();
+        skip_count = 0;
+    }
     tokens.erase(tokens.begin(), tokens.begin() + skip_count);
 
     if (this->total_tokens + tokens.size() >= this->MAX_L){
@@ -630,17 +639,51 @@ buffer<bf16> AutoModel::_chunked_insert(chat_meta_info_t& meta_info, std::vector
     return y;
 }
 
+std::string AutoModel::_render_chat_template(minja::chat_template_inputs& inputs, bool continue_final_message,
+                                             const minja::chat_template_options& options) {
+    if (!continue_final_message) {
+        inputs.add_generation_prompt = true;
+        return this->chat_tmpl->apply(inputs, options);
+    }
+    if (inputs.messages.empty() || inputs.messages.back().value("role", "") != "assistant") {
+        throw ModelRequestError(400, false, "continue_final_message requires the last message to have role \"assistant\"",
+                                "invalid_request_error");
+    }
+    const auto render = [&](const nlohmann::ordered_json& messages, bool add_generation_prompt) {
+        minja::chat_template_inputs chat = inputs;
+        chat.messages = messages;
+        chat.add_generation_prompt = add_generation_prompt;
+        return this->chat_tmpl->apply(chat, options);
+    };
+    try {
+        chat_continued continued = chat_continue_prompt(render, inputs.messages);
+        // the output parser looks for an open reasoning block in this turn only, not in what the user wrote earlier
+        this->continued_turn_ = std::move(continued.turn);
+        return continued.prompt;
+    } catch (const std::invalid_argument& e) {
+        throw ModelRequestError(400, false, e.what(), "invalid_request_error");
+    }
+}
+
 void AutoModel::_begin_turn(chat_meta_info_t& meta_info, const std::vector<int>& prompt_tokens) {
     // keep the end of the prompt as the model sees it: generation continues from
-    // there, so it tells the output parser whether a reasoning block is already open
+    // there, so it tells the output parser whether a reasoning block is already open.
+    // A continued message may have opened one anywhere in its text, so then it is
+    // the whole continued turn.
     constexpr size_t PROMPT_TAIL_TOKENS = 32;
     meta_info.prompt_tail.clear();
-    for (size_t i = prompt_tokens.size() > PROMPT_TAIL_TOKENS ? prompt_tokens.size() - PROMPT_TAIL_TOKENS : 0;
-         i < prompt_tokens.size(); i++) {
-        meta_info.prompt_tail += this->token_text(prompt_tokens[i]);
+    if (meta_info.continue_final_message) {
+        meta_info.prompt_tail = this->continued_turn_;
+    }
+    else {
+        for (size_t i = prompt_tokens.size() > PROMPT_TAIL_TOKENS ? prompt_tokens.size() - PROMPT_TAIL_TOKENS : 0;
+             i < prompt_tokens.size(); i++) {
+            meta_info.prompt_tail += this->token_text(prompt_tokens[i]);
+        }
     }
     if (meta_info.constraint) {
-        meta_info.constraint->start(chat_format_prefill(this->chat_format, meta_info.prompt_tail));
+        meta_info.constraint->start(chat_format_prefill(this->chat_format, meta_info.prompt_tail,
+                                                        meta_info.continue_final_message));
     }
 }
 
@@ -788,6 +831,8 @@ std::string AutoModel::_shared_generate(chat_meta_info_t& meta_info, int length_
     stop_reason_t reason = EOT_DETECTED;
     int last_sampled_token = this->last_token;
     this->token_history.push_back(this->last_token);
+    // the sample insert() took is the reply's first token, and counts toward the limit
+    meta_info.generated_tokens++;
     if (this->is_normal_token(last_sampled_token) && last_sampled_token != -1){
         std::string token_str = this->tokenizer->run_time_decoder(last_sampled_token);
         result += token_str;
@@ -807,10 +852,11 @@ std::string AutoModel::_shared_generate(chat_meta_info_t& meta_info, int length_
     if (this->total_tokens >= decode_cap){
         header_print("WARNING", "Max length reached, stopping generation...");
         reason = MAX_LENGTH_REACHED;
-        meta_info.stop_reason = reason;
-        return result;
     }
-    while (true){
+    else if ((length_limit > 0) && (meta_info.generated_tokens >= length_limit)){
+        reason = MAX_LENGTH_REACHED;
+    }
+    else while (true){
         // Running out of window is a truncation, not the model choosing to
         // stop, and the caller has to be able to tell those apart: this is
         // what becomes `done_reason` over the API. Testing the cap here rather
@@ -864,6 +910,11 @@ std::string AutoModel::_shared_generate(chat_meta_info_t& meta_info, int length_
     }
     meta_info.decoding_duration = (uint64_t)(time_utils::cast_to_us(this->profiler_list[DECODING_TIME].get_total_time()).first) * 1e3;
     meta_info.stop_reason = reason;
+    if (reason != EOT_DETECTED) {
+        // Cut short after sampling its last token: no engine has run that one,
+        // so the next prompt has to carry it.
+        this->token_history.pop_back();
+    }
     if (this->total_tokens >= decode_cap){
         header_print("WARNING", "Max length reached, stopping generation...");
     }

src/common/AutoModel/modeling_gemma3.cpp

@@ -34,12 +34,11 @@ void Gemma3::setup_tokenizer(std::string model_path) {
     auto tokenizer_config = this->_shared_setup_tokenizer(model_path);
 }
 
-std::string Gemma3::apply_chat_template(nlohmann::ordered_json& messages, nlohmann::ordered_json tools) {
+std::string Gemma3::apply_chat_template(nlohmann::ordered_json& messages, nlohmann::ordered_json tools, bool continue_final_message) {
     minja::chat_template_inputs inputs;
-    inputs.add_generation_prompt = true;
     inputs.messages = messages;
     inputs.extra_context = this->extra_context;
-    return this->chat_tmpl->apply(inputs);
+    return this->_render_chat_template(inputs, continue_final_message);
 }
 
 bool Gemma3::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function<bool()> is_cancelled) {
@@ -110,7 +109,7 @@ bool Gemma3::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std:
             filtered_messages.push_back(filtered_message);
         }
         header_print("FLM", "Total images: " << total_images);
-        templated_text = this->apply_chat_template(filtered_messages);
+        templated_text = this->apply_chat_template(filtered_messages, nlohmann::ordered_json::object(), meta_info.continue_final_message);
 
         if (total_images > 0) {
             pixel_values.resize(3 * 896 * 896 * sizeof(bf16) * total_images);

src/common/AutoModel/modeling_gemma3_text.cpp

@@ -36,12 +36,11 @@ void Gemma3_Text_Only::setup_tokenizer(std::string model_path) {
     auto tokenizer_config = this->_shared_setup_tokenizer(model_path);
 }
 
-std::string Gemma3_Text_Only::apply_chat_template(nlohmann::ordered_json& messages, nlohmann::ordered_json tools) {
+std::string Gemma3_Text_Only::apply_chat_template(nlohmann::ordered_json& messages, nlohmann::ordered_json tools, bool continue_final_message) {
     minja::chat_template_inputs inputs;
-    inputs.add_generation_prompt = true;
     inputs.messages = messages;
     inputs.extra_context = this->extra_context;
-    return this->chat_tmpl->apply(inputs);
+    return this->_render_chat_template(inputs, continue_final_message);
 }
 
 bool Gemma3_Text_Only::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function<bool()> is_cancelled) {
@@ -53,7 +52,7 @@ bool Gemma3_Text_Only::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& i
         return false;
     }
     if (!input.messages.empty()) { // already a formated messages, usually from REST API
-        templated_text = this->apply_chat_template(input.messages);
+        templated_text = this->apply_chat_template(input.messages, nlohmann::ordered_json::object(), meta_info.continue_final_message);
     }
     else if (!input.prompt.empty()) { // a pure text, usually from the cli
         nlohmann::ordered_json messages;