fix(server): each turn reuses the previous one, long prompts share time, late system messages work

[davide221]

Fixes for problems found by serving the Manage models the way Manage does (Qwen3.8-Flash-Next split over an R9700 + Strix Halo, Qwen 3.8 27B on the R9700, DeepSeek V4.1 Flash on an R9700 + 192 GB Strix Halo) to scripted chats and agents and to real Claude Code and Codex sessions.

What was wrong

What a person saw	Models	Cause
A follow-up question on a long document re-read the whole document	all three: 103 s (Flash-Next, 130K tokens), 532 s (Qwen 27B, 118K), 72 s (V4.1, 33K)	The prefix cache saved a turn before its last message, so the message the follow-up extends was never saved.
Every reply in a chat re-read the whole conversation; an agent re-read up to 4K tokens per turn	Flash-Next	Restore points 4096 tokens apart (#828) moved every save down to the last of them.
Agents running at once rarely reused their own turns	Flash-Next, Qwen 27B	The prefix cache held one capture at a time, so requests prefilling beside it saved nothing.
A person chatting beside a long prompt got 1.6 tok/s (Flash-Next) or 0.6 tok/s (Qwen 27B)	Flash-Next, Qwen 27B	Each step paired a decode token with a prompt chunk; on Qwen 27B such a step also runs without the drafter.
A long document sent to Qwen 27B on its own crashed the server 79 s in	Qwen 27B	Paged attention asked for more partials than the R9700 had left (ROCm error: out of memory).
An app that sends a system message after the first user message failed on every request (HTTP 500); Claude Code 2.1 does this and retried 10 times	Flash-Next	The model's template rejects a late system message.
Anthropic API clients asking for adaptive thinking (Claude Code does) got none; replies in every app started with blank lines; OpenAI API clients (Codex, for one) showed 0 cached tokens	all	adaptive thinking turned thinking off; the newlines after </think> were sent as text; usage had no cached-token field.

Changes

1. fix(server): the prefix cache keeps each turn's end, for every request at once. Each prefilling request holds its own capture reservation, and concurrent serving prunes the snapshots a new capture supersedes, as single-sequence serving does.
2. feat(server): requests decoding beside a long prompt keep half the time (StepPlanLimits::prefill_time_share), and a prompt that fits in one step runs first.
3. fix(server): a late system message no longer fails a request, replies start without blank lines. For any app: a template that rejects a late system or developer message gets it as a user turn, in place; adaptive thinking means the model's default; a template error answers 400.
4. feat(server): usage reports the prompt tokens the prefix cache served (prompt_tokens_details.cached_tokens, input_tokens_details.cached_tokens).
5. fix(server): a late system message stays one where the template takes it.
6. fix(server): every model keeps each turn's end, and the turn before it for edits.
   • Every backend saves the prompt up to the end of its last message, the point the next turn extends.
   • A conversation's first turn still saves its system head, so new conversations on the same system prompt restore it, unless the head is short beside a long message.
   • A new snapshot keeps the previous turn's while the resident budget holds it, so a person who edits their last message resumes there. Eviction drops these kept turns first.
7. fix(ggml-cuda): paged attention keeps its partials within 256 MiB. A chunk with many rows covers the context with fewer, longer partitions.
8. feat(qwen35): requests decoding beside a long prompt keep half the time, for prompts of more than 8192 new tokens (StepPlanLimits::time_share_prompt_tokens), so an agent's next step keeps every step.
9. perf(qwen4exp): an agent's next step no longer gives up time to decoding requests: the same 8192-token size on Flash-Next.

Measured

Each model served with its Manage command on one box (R9700 + Strix Halo, 192 GB), greedy, from a cold start. Scenarios: an 8-turn chat; one agent with tools over this repository (13.7K-token system prompt); four agents at once; a long document and a follow-up while a person chats; a chat on a 4K-token system prompt whose last message is edited and sent again, then a new conversation on the same prompt.

Qwen3.8-Flash-Next (split, 262K context, four slots), main (b8b3cf5) against this PR:

Scenario	main	This PR
Chat: prompt tokens prefilled, 8 turns	8,465	1,951
Chat: time to first token, mean	1.42 s	1.26 s
One agent: time to first token after step 0, mean / max	2.92 / 5.07 s	2.50 / 3.52 s
Four agents: wall time	180.9 s	159.4 s
Four agents: prompt tokens prefilled	87,082	54,918
Four agents: time to first token, mean / max	3.33 / 7.52 s	3.29 / 5.95 s
Follow-up on a 130K-token document	102.9 s	2.7 s
Person chatting beside the document: decode	1.6 tok/s	19 tok/s
Document's first answer, with the person beside it	111.7 s	150.9 s

Qwen 3.8 27B (R9700, 131K context, eight slots, built with Manage's flags for it: HIP graphs off), Manage's pin (e6f643d) against this PR:

Scenario	Manage pin	This PR
118K-token document sent on its own	server aborts 79 s in (out of memory)	answers in 524 s
Follow-up on that document	532 s	3.8 s
Person chatting beside the document: decode	0.6 tok/s	47 to 60 tok/s
Document's first answer, with the person beside it	530 s	540 s
Four agents: wall time	169.1 s	121.9 s
Four agents: prompt tokens prefilled	58,792	36,457
Four agents: time to first token, mean / max	16.4 / 43.5 s	10.8 / 24.0 s
Chat, one agent, edit: time to first token, mean	0.66 / 5.53 / 1.22 s	0.60 / 4.07 / 1.22 s
Published harness, HumanEval prompts (10 x 2, 512 tokens)	175.3 tok/s	174.9 tok/s, same text 10/10

DeepSeek V4.1 Flash (R9700 + Strix Halo, ds41-gorgon, one request at a time, HIP graphs off as Manage builds it): the follow-up on a 33K-token document restores it and prefills 79 tokens in 1.5 s, against 33,233 tokens in 63 s; its time to first token drops from 72 s to 11 s, the rest being the wait behind a person's request. The agent, edit and chat scenarios take the same time as at Manage's pin (55.6 / 25.3 / 80.4 s against 56.0 / 26.6 / 79.2 s), and the R9700's memory stays at 27.3 to 28.0 GiB as it does there.

Real apps. None of the changes is specific to one app; they apply to any client of the OpenAI or Anthropic API. We checked them with two coding agents against this PR on Flash-Next, Claude Code 2.1.296 on a clean config (22 tools) and Codex 0.155.1: an 11-turn Claude Code task in 73 s (every request failed on main) and a 10-turn Codex task in 84 s. From the second turn on each turn restores the previous turn's end. On Qwen 3.8 27B, the same Claude Code task finishes in 97 s, and Codex in 52 s with 66,895 of its 82,926 input tokens served from the prefix cache.

Costs:

• Snapshot saves. Saving at each turn's end takes about 10 µs per token on Flash-Next (0.29 s at 25K tokens), in that turn's time to first token.
• Memory. Keeping the previous turn holds one more snapshot per conversation, within the resident budget; it is the first to go when the budget is short.
• An edit re-reads the previous reply. The previous turn's snapshot ends before the reply, so an edited last message prefills the reply and the message (223 tokens, 0.78 s on Qwen 27B, against 0.69 s before).
• Output can differ from main. A hit restores at the previous turn's end, so its prefill forwards are cut at different places than a cold run of the same prompt; greedy outputs can differ at near ties.

Tests

Test	Result
test_server_unit	659 passed
test_parallel_prefix_txn	passes

New and updated tests: the capture is the end of the last message on every backend; a first turn saves its head; captures in flight hold their slots and bytes; the previous turn survives pruning while the budget holds it and an edit restores it; kept turns are evicted first; the scheduler captures for two requests admitted together, keeps the previous turn and frees the one before; decode-only steps run between the chunks of a long prompt and not beside a prompt under the engine's long-prompt size; a short prompt runs alone before a long one; a strict template gets a late system message as a user turn and a permissive one keeps it; adaptive thinking is the default; the cached-token usage fields.

Built for gfx1151 + gfx1201.

🤖 Generated with Claude Code

[davide221]

Code review

Found 1 issue:

1. Every non-first system/developer message is silently demoted to a user turn for all Jinja templates · likely (server/src/server/chat_template.cpp:1084-1086)

_{Reviewed 980e3f43 · 1 candidate issue checked, kept at ≥70% confidence · add the code-review label to review again}

[davide221]

Every non-first system/developer message is silently demoted to a user turn for all Jinja templates · likely

The change rewrites m.role to "user" whenever a system or developer message appears after the first message, for every Jinja template, not only for templates that reject late system messages. Most ChatML-style templates (including Qwen's) accept a system message at any position and render it as a system block. A normal multi-system conversation — e.g. OpenWebUI's per-chat notes appended after the main system prompt, or an app injecting a mid-conversation system instruction — now has those instructions rendered as user text, changing model behaviour, and it also changes the rendered prompt for conversations whose prefix is already cached, silently invalidating existing prefix-cache entries for that turn. A fix that only rewrites the role when the template actually cannot take a late system message (or only when the first render raises) would keep Claude Code working without altering prompts for templates that support it.

Checked (80%): chat_template.cpp:1084-1086 unconditionally rewrites any non-first system/developer message to role "user" for every Jinja template, with no check of whether the template actually rejects late system messages. Mid-conversation system messages are a normal input (the PR itself cites Claude Code doing this), so templates that previously rendered them as system blocks now render them as user text, a real behavior change introduced by these changed lines; the only mitigating uncertainty is how harmful the demotion is in practice.