fix(SERVE-STREAM-USAGE): the chat role frame waits for the first engine result, so TTFT stops measuring an empty frame (#1982) by localai-bot · Pull Request #1994 · mudler/vllm.cpp

localai-bot · 2026-08-26T15:47:59Z

ChatSseStream::next wrote the /v1/chat/completions role frame without ever
calling WaitOutput. The buffering loop lived inside the
if (usage_.include_continuous_usage) arm only, so on the default path a
request that had produced nothing had already been answered 200 with a frame.
Remove that guard, so the buffering runs in every usage mode.

Upstream builds the role chunk under if first_iteration: inside
async for res in result_generator:
(vllm/entrypoints/openai/chat_completion/serving.py:477,487 at the parity pin
555967922). Its comment at :484-486 gives the reason, and the reason has
nothing to do with usage: "if there are exceptions in the result_generator, it
needs to be sent as the FIRST response (by the try...catch)".

Two things followed from writing the frame early, and the second is why this is
a correctness fix rather than a tidy-up.

A request that died before its first token had already had a 200 and a role
frame written to it, so the failure could not be the first response.

vllm/benchmarks/lib/endpoint_request_func.py:404-408 stamps TTFT on the first
chunk carrying a choices key, whatever delta.content holds. Our role frame
carries delta.content = "" and no usage, so it satisfied that guard. Any
TTFT taken against this endpoint with vllm bench serve --backend openai-chat
was the HTTP round trip to an empty frame: near zero, and independent of load.
vLLM and SGLang order their role frame after the first result, so their rows on
the same harness measured the real quantity and only ours did not. The artifact
flattered this engine and only this engine, and it blocked the #1574
three-engine TTFT row.

.agents/specs/stream-options.md:110-113 and :197-199 scoped the buffering to
continuous usage deliberately. Both passages are true about continuous usage and
both stop there: the recorded reasoning treated the buffering as a means to a
native prompt count and weighed neither the error ordering nor the TTFT stamping
rule. Both are corrected here rather than left contradicting the code, and
.agents/specs/chat-role-frame-ordering.md carries the reversal.

What it costs a real client

Stated plainly, because this is a latency change on a hot path. TTFT as measured
by that harness gets WORSE, and should, because it was previously measuring
nothing. A client that renders a typing indicator on the role frame loses that
early signal by the true prefill time.

The first TOKEN is not delayed. The token that used to ride in frame two now
rides in frame three, at the same instant, out of the one RequestOutput the
stream already buffered.

Worker-thread occupancy does not change. create_chat_completion still returns
without waiting, and the wait moves from the second next() call to the first
on the same cpp-httplib worker that was going to block either way. AsyncLLM
keeps batching every other request.

Tests

tests/vllm/entrypoints/openai/test_chat_stream_first_frame.cpp holds both
halves through the production ApiServer::handle_chat_completions dispatch,
over a real AsyncLLM whose model runner the test clocks.

A role-frame SHAPE assertion passes on both sides of this change and proves
nothing. The discriminating assertion is a negative one taken while a gated
runner is held inside sample_tokens: the case asserts its own precondition
(sampled_steps() == 0, so no token exists anywhere in the engine) and then
asserts that no frame has arrived. The 300 ms is a grace for the defective path
to show itself, never a deadline the correct path must beat, so a slower box
makes the case more reliable rather than less.

The error case runs a runner that throws inside sample_tokens, and asserts
that the FIRST next() call surfaces the failure with a sentinel chunk still
unwritten.

The drain that follows guards the one regression this change could introduce.
The buffered first result must be delivered rather than swallowed, so the
concatenated content across every frame has to carry the fixture token twice.
Counting the text rather than the frames keeps that independent of collector
merging.

Red before, on the parent tree: 2 cases failed, 4 assertions, the offending
frame logged in the failure text. Green after: 2 passed, 21 of 21.

Two mutations on the final head, tree restored and sha256sum -c verified after
each, then green re-confirmed. Reinstating the include_continuous_usage guard
returns the same 4 failures. Deleting the production call site
out.sse_stream = std::move(result.sse_stream) in handle_chat_completions
takes the file to 2 failed cases at 6 assertions, which is what proves the gate
enters through the route handler rather than around it.

Interaction with #1999

2a42cb369 landed ClampMaxNumSeqsToStateBudget, which bounds max_num_seqs
by the seats the KV budget affords. This row's fixture sets max_num_seqs = 8
against 1024 blocks, close enough to the clamped geometry to be worth computing
rather than assuming. It cannot fire, for three independent reasons, and the
fixture now records all three so that an edit breaking one still leaves two:

ComputeHybridKvBudget returns early when no MambaSpec is present, and this
config carries a single FullAttentionSpec, so the budget stays
kStateSeqsUnbounded (-1) and ClampMaxNumSeqsToStateBudget passes the
configured value straight through.
The only production caller is src/vllm/entrypoints/model_loader.cpp, and the
fixture builds the Scheduler and AsyncLLM directly without the loader.
Each case issues exactly one streaming request, so 8 is headroom rather than a
requirement; even a clamp to a single seat would leave both cases passing.

Not fixed here

Neither stream converts an engine exception into a data: {"error": ...} frame
the way chat_completion/serving.py:827-833 does. The exception reaches the
cpp-httplib content provider, which logs to stderr and truncates the body.
After this change no role frame precedes that truncation, which is an
improvement and not a fix. Filed as #1992, owned by this row's spec under
## Owed.

Closes #1982

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5-1m [Claude Code]

…ngine is read, so our TTFT measures an empty frame (#1982) `ChatSseStream::next` emits the `/v1/chat/completions` role frame without ever calling `WaitOutput` on the default path. The `WaitOutput` loop lives inside the `if (usage_.include_continuous_usage)` arm only, so a request that has produced nothing has already had a 200 and a role frame written to it. Upstream builds the role chunk under `if first_iteration:` inside `async for res in result_generator:` (`vllm/entrypoints/openai/chat_completion/serving.py:477,487` at the parity pin `555967922`) and states the reason at `:484-486`: an exception in the generator "needs to be sent as the FIRST response". There is a second consequence that the record never weighed. `vllm/benchmarks/lib/endpoint_request_func.py:404-408` stamps TTFT on the first chunk carrying a `choices` key, whatever `delta.content` holds. Our role frame carries `delta.content = ""` and no `usage`, so it satisfies that guard. Any TTFT measured against our chat endpoint with `vllm bench serve --backend openai-chat` is therefore the HTTP round trip to an empty frame, near zero and independent of load. vLLM and SGLang order their role frame after the first result, so their rows on the same harness are honest and only ours is not. That blocks the #1574 three-engine TTFT row. This commit lands the spec and the records. `specs/stream-options.md:110-113` and `:197-199` scoped the buffering to continuous usage deliberately, so both passages are corrected here rather than left contradicting the code that follows. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5-1m [Claude Code]

…ne result, so TTFT stops measuring an empty frame (#1982) `ChatSseStream::next` buffered the first `RequestOutput` before the role frame only when continuous usage asked for its native prompt count. Remove that guard so the buffering runs in every usage mode, which is what upstream does. Upstream builds the role chunk under `if first_iteration:` inside `async for res in result_generator:` (`vllm/entrypoints/openai/chat_completion/serving.py:477,487` at the parity pin `555967922`). Its comment at `:484-486` gives the reason and it has nothing to do with usage: "if there are exceptions in the result_generator, it needs to be sent as the FIRST response (by the try...catch)". Two things follow from writing the frame early, and the second one is why this is a correctness fix rather than a tidy-up. A request that died before its first token had already been answered 200 with a role frame, so the failure could not be the first response. `vllm/benchmarks/lib/endpoint_request_func.py:404-408` stamps TTFT on the first chunk carrying a `choices` key, whatever `delta.content` holds. Our role frame carries `delta.content = ""` and no `usage`, so it satisfied that guard, and any TTFT taken against this endpoint with `vllm bench serve --backend openai-chat` was the HTTP round trip to an empty frame: near zero, and independent of load. vLLM and SGLang order their role frame after the first result, so their rows on the same harness measured the real quantity and only ours did not. That blocked the #1574 three-engine TTFT row. The change is a latency change on a hot path and it is worth stating plainly. TTFT as measured by that harness gets WORSE, because it was previously measuring nothing. A client that renders a typing indicator on the role frame loses that early signal by the true prefill time. The first TOKEN is not delayed: the token that used to ride in frame two now rides in frame three, at the same instant, out of the one `RequestOutput` the stream already buffered. Worker-thread occupancy is unchanged, because the wait moves from the second `next()` call to the first on the same cpp-httplib worker, and `AsyncLLM` keeps batching every other request. `tests/vllm/entrypoints/openai/test_chat_stream_first_frame.cpp` holds both halves through the production `ApiServer::handle_chat_completions` dispatch. A role-frame SHAPE assertion passes on both sides of this change and proves nothing, so the discriminating assertion is a negative one taken while a gated model runner is held inside `sample_tokens`: the case asserts its own precondition (0 sampled steps, so no token exists) and then asserts that no frame has arrived. The error case runs a runner that throws, and asserts that the FIRST `next()` call surfaces the failure with a sentinel `chunk` still unwritten. `/v1/completions` is untouched. It already withholds the empty chunked-prefill delta, mirroring `vllm/entrypoints/openai/completion/serving.py:368-374`. Neither stream converts an engine exception into a `data: {"error": ...}` frame the way `chat_completion/serving.py:827-833` does; the exception reaches the cpp-httplib content provider, which logs and truncates the body. After this change no role frame precedes that truncation, which is an improvement and not a fix. Recorded under `## Owed` in the spec. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5-1m [Claude Code]

localai-bot · 2026-08-26T20:01:15Z

CI lane triage: both reds are pre-existing and named

Recording this here rather than in the commit message, because it is lane triage rather than a description of the change.

`sanitize-cpu (thread)` — #1862, filed 2026-08-24, two days before this branch

The failure is test_cpu_kernel_bench_cli, at tests/scripts/test_cpu_kernel_bench.py:42:

AssertionError: 1 not greater than 1

which is self.assertGreater(got["timing"]["calls_per_sample"], 1).

It is not a ThreadSanitizer finding. grep -c "WARNING: ThreadSanitizer: data race" over the job log returns 0.

Mechanism, at examples/cpu_kernel_bench/main.cpp::Benchmark (the calibration loop): calls_per_sample starts at 1 and doubles until one sample takes >= max(2.0e6 ns, 1000 * overhead). So the loop breaks on its first iteration whenever a single call already fills the 2 ms budget, and line 42 is really asserting "one q4_k GEMM at m=1,n=8,k=256 finishes in under 2 ms on this machine". That is a property of the runner, not of the tree, and it is a threshold, so it is bimodal — which is exactly why the same lane is green on main and red here without either tree differing in any relevant way.

Reproduced on this branch's own TSan build, by moving per-call cost across the threshold and changing nothing else. One binary, one commit:

shape	ns per call	`calls_per_sample`	line 42
n=8 k=256	67,792	8	PASS
n=8 k=1024	1,149,534	2	PASS
n=16 k=1024	1,170,215	1	FAIL: `1 not greater than 1`
n=32 k=1024	5,969,152	1	FAIL
n=64 k=2048	9,525,998	1	FAIL

At the test's own shape this box reports calls_per_sample=8 at ~23.5 us per call under TSan, roughly 85x below the 2 ms floor, so the suite passes here (Ran 3 tests ... OK). A hosted runner is slower and lands on the other side of the same threshold.

Attribution. #1862 records the identical failure on PR #1856, whose diff contains zero C++ files — a stronger control than anything this branch could construct. I did check whether the bench even links this PR's translation unit, and it does (nm -C finds ChatSseStream, OpenAIServingChat), so I am not resting on causal independence by absence; the evidence above is behavioural.

The assertEqual(first.returncode, 0) variant that appears when running this locally is a separate local-only artifact: TSan aborts with unexpected memory mapping unless ASLR is disabled. setarch -R fixes it; CI runners already have it off.

A rerun of the job on the identical head is in flight as run_attempt=2.

`windows-msvc-cpu` / `windows-msvc-vulkan` — #1979

test_openai_api_server.exe fail-fasts with 0xC0000409. Same binary, same status on PRs #1986 and #1971, neither of which touches the OpenAI serving path. These jobs are PR-only, so they have no main baseline to bisect against; #1979 records exactly that.

Not owed by this row

Both are open issues with their own owners. Neither is fixed here, and this PR adds no new red.

`2a42cb369` (#1999) landed the recurrent-state KV budget, which made this branch read CONFLICTING. Both conflicts are in files this row appends to. `.agents/issue-index.md` is append-only, so the union is the resolution. Verified by row-ID set difference rather than by eye: 729 rows at the merge base, plus #1982 and #1992 from this branch and #1983 from `main`, giving 732 with zero lost, zero invented and zero duplicated ids. `tests/CMakeLists.txt` is not append-only and several rows added targets in the same region today. This branch's block stays inside `if(VLLM_CPP_SERVER)` next to `test_openai_api_server`, and #1999's `test_hybrid_kv_budget` and `test_loaded_engine_dense` registrations are intact. No `vllm_cpp_add_test` target name is duplicated across the file. #1999's `ClampMaxNumSeqsToStateBudget` cannot reach this row's fixture. The following commit records why in the fixture itself. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5-1m [Claude Code]

…not reach this fixture The merge with `2a42cb369` brought in `ClampMaxNumSeqsToStateBudget`, which bounds `max_num_seqs` by the seats the KV budget affords. This fixture sets `max_num_seqs = 8` against 1024 blocks, which is close enough to the clamped geometry to be worth checking rather than assuming. It cannot fire, and the note records all three reasons so that an edit which breaks one still has two. `ComputeHybridKvBudget` returns early when no MambaSpec is present and this config carries a single FullAttentionSpec, so the budget stays `kStateSeqsUnbounded` and the clamp is a pass-through. The only production caller is the model loader, which this fixture does not use. And each case issues exactly one streaming request, so the value is headroom rather than a requirement. Comment only; no behaviour changes. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5-1m [Claude Code]

…ults, so landing this re-baselines our arm (#1985, #1984) The premise was verified against the public `Qwen/Qwen3.8-27B` file; the operator confirmed it on the checkpoint the project actually benchmarks. `qwen3.8-27b-nvfp4-mtp-sm121-r0b0tlab` ships `temperature: 1.0`, `top_k: 20`, `top_p: 0.95`, so every benchmark run on it has sampled the full 248,320-token vocabulary while vLLM sampled from twenty candidates. That is a per-token cost we inflicted on ourselves and an asymmetry inside the comparison itself. The consequence is recorded because it is easy to misread later: a number taken before this merges is not comparable with one taken after, and not because anything regressed -- the two are different sampling configurations. The arm re-baselines at the merge commit, and `--generation-config vllm` reproduces the old resolution exactly if an old figure has to be re-derived. Also recorded: #1994 voids TTFT and TPOT measured on older trees, and nothing in this row quotes either. The only measured figures it carries are `cuda_sample.cu`'s own ~7.5 ms/token greedy-scan anchor, which is a per-kernel attribution rather than a harness TPOT, and #1929's 16.9 tok/s, which is throughput. #1994's own spec lists `/v1/completions` among the paths it does not touch, so the acceptance arms are unaffected; on the chat backend the residual bias would overstate the greedy-versus-default gap by about 1.6%, which moves the acceptance bar by under a millisecond. And the CUDA half now has a compile verdict instead of a hope: `cuda-fat-build` returned success at `e5b9b0045` across all ten shipped architectures, with both sanitizers green. It is a compile verdict only -- CI has no device, so no CUDA test ran -- and it belongs to that tree, so the merged tree owes its own. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: claude-code:claude-opus-5-1m [Claude Code]

main advanced 45 files past this row's last merge base (2a42cb3) and GitHub reported the PR `CONFLICTING`. Four of this row's twelve files overlap that delta, and every one was resolved by the AGENTS.md `## Records` rule rather than by accepting the automatic merge. `.agents/issue-index.md` was union-appended and then checked by row-ID set difference rather than by reading the diff: 730 base + 2 ours (#1963, #1966) + 15 theirs = 747 expected, 747 actual, 0 lost, 0 invented, 0 duplicated. Every merged row is byte-identical to a row in one of the three sources, and the preamble is byte-identical to base on all three sides. GitHub does not honour the `merge=union` driver, which is why this file is the whole reason the PR read dirty. `tests/CMakeLists.txt` was rebuilt rather than trusted: origin/main's complete file (blob 6a616d7) with this row's five lines re-applied after `vllm_cpp_add_test(test_loaded_engine_dense ...)`, an anchor asserted to occur exactly once. The rebuild is byte-identical to the three-way result, and `git diff origin/main -- tests/CMakeLists.txt` carries those five lines and nothing else. `docs/FEATURES.md` and `src/vllm/entrypoints/model_loader.cpp` take edits in disjoint regions. Both were checked in both directions: `git diff origin/main` carries only this row's hunks, byte-identical to `git diff 2a42cb3 4f8638a`, and `git diff 4f8638a` reproduces origin/main's entire 45-file, 6785-insertion, 333-deletion delta with nothing dropped. No conflict region was reconstructed, so the brace-hoist failure this branch already recorded cannot recur here; the resolved tree was built and run anyway. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code]

main advanced 45 files past this row's last merge base (2a42cb3) and GitHub reported the PR `CONFLICTING`. Two of this row's six files overlap that delta, and both were resolved by the AGENTS.md `## Records` rule rather than by accepting the automatic merge. `.agents/issue-index.md` was union-appended and then checked by row-ID set difference rather than by reading the diff: 730 base + 2 ours (#2008, #2009) + 15 theirs = 747 expected, 747 actual, 0 lost, 0 invented, 0 duplicated. Every merged row is byte-identical to a row in one of the three sources, and the preamble is byte-identical to base on all three sides. GitHub does not honour the `merge=union` driver, which is why this file is the whole reason the PR read dirty. `tests/CMakeLists.txt` was rebuilt rather than trusted: origin/main's complete file (blob 6a616d7) with this row's seven lines re-applied after the `test_dflash2_ctx_capacity` source line, an anchor asserted to occur exactly once. The rebuild is byte-identical to the three-way result. Nothing in the delta touches `runner.h` or `runner.cpp`, so this row's request-keyed draft context arrives unchanged: `git diff origin/main` is this row's original six files, 786 insertions and 68 deletions, and `git diff 3d895a2` reproduces origin/main's entire 45-file, 6785- insertion, 333-deletion delta with nothing dropped. No conflict region was reconstructed, so no take-both brace hoist is possible here; the resolved tree was built and run anyway. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code]

…, so the budget bounds the allocation again (#1963, #1966) (#2000) `KVCacheGroupSpec::layer_names` is upstream's per-layer name list, and upstream bounds its KV allocation with `max(len(group.layer_names) ...)` over that same list (`vllm/v1/core/kv_cache_utils.py:1399`), dividing the budget by the count it multiplies the allocation by (`:1005-1008`, `:1409-1416`). The count is one expression over one list, so the product is bounded by construction. Thirty-three of our thirty-four registries publish a single PLACEHOLDER string per group instead — `{"fa"}`, `{"gdn"}`, `{"mla"}`, `{"kda"}`, `{"fa_draft"}`, `{"encoder"}` — and two consumers read `layer_names.size()` as if it were a layer count. `KVBytesPerBlock` divides an absolute `--kv-cache-memory` budget by ONE layer's page while `GPUModelRunner::initialize_kv_cache` allocates one buffer per layer. Measured on `dgx:gpu0` before this change: `--kv-cache-memory 1073741824` logged `page_size_bytes=131072 num_blocks=4096`, and 16 full-attention layers plus the draft layer make that **8.5 GiB of buffers for a 1 GiB budget** (#1963). `recurrent_state_bytes` reports 0.90 GiB against a 43.40 GiB allocation at `--max-num-seqs 32`, `k=8`, so the #371 guard — which exists to turn an OOM reboot into a refusal — does not fire on the only family it can fire for (#1966). Together they account for the 98.2 GiB the watchdog saw at `--kv-cache-memory 6GiB`: 51.00 GiB of paged pool plus 43.40 GiB of recurrent state, 94.40 GiB of the 100.6 GiB the watchdog measured. `ResolveKVCacheGroupLayerNames` rewrites the placeholders into real per-layer names, and `LoadedEngine::MakeKVCacheMaybeSpec` calls it. That is the single funnel every architecture returns through — the speculative branch and `ModelRegistry::MakeKVCache` both — and `MakeKVCacheResolved` passes the probe and the resized config through it, so one call site reaches all thirty-four registries. The classification is the runner's own predicate, not a second derivation of the model's shape: a layer is recurrent iff the config has a Mamba group and `layer_types[l] == "linear_attention"`; the target attention group is the first non-eagle attention group and covers every other layer; a second attention group is the speculative draft head, one layer at index `num_hidden_layers`; a third gets an empty list, because the runner allocates no buffer for it. Reproducing the allocator's predicate is what makes the accounting unable to disagree with the allocation. `NemotronHForCausalLM` already publishes real names and is left alone: one resolvable name anywhere and the resolver returns untouched. Its `layer_types` is empty and its MoE blocks cache nothing, so the fallback would re-introduce the 52-against-6 mis-classification #810 removed. `LayerIndexOfName` moves out of the runner's anonymous namespace to `vllm::v1::KVCacheLayerIndexOfName` so both sides parse a name with one function. `GPUModelRunner::kv_cache_allocated_bytes()` and its paged half sum the size every `CacheBuffer` was constructed with, so the gate compares the sizing arithmetic against what the allocator DID rather than against a second copy of the same formula. Six cases, driven from `MakeQwen3_5KVCacheSpec` through the `LoadedEngine` constructor rather than from a hand-built `KVCacheGroupSpec`, on a 6-layer hybrid with TWO full-attention and FOUR GDN layers. The 4-layer config that file already had has exactly one full-attention layer, which is the config in which this bug is invisible — and `tests/vllm/v1/test_kv_cache_interface.cpp` hands the function `KVCacheGroupSpec{{"layer1", "layer2"}, ref}`, a shape no registry emits, which is why the pre-existing unit coverage could not see it. With the resolver call site deleted in a scratch copy: ``` CHECK( allocated <= kv_cache_memory_bytes ) 2097152 <= 1048576 CHECK( KVBytesPerBlock * num_blocks == ... ) 1048576 == 2097152 CHECK( groups[0].layer_names.size() == 2 ) 1 == 2 CHECK( groups[1].layer_names.size() == 4 ) 1 == 4 CHECK( recurrent_state_bytes(cfg,4) == ... ) 4992 == 19968 ``` Exactly 2x on the paged half and 4x on the recurrent half, which is the defect in bytes. Restored: 21 of 21 assertions pass and 627 of 627 ctest cases. `--kv-cache-memory` now buys as many times fewer blocks as the model has attention layers, because those blocks were never inside the budget. `docs/USAGE.md` says so beside the flag, and its `8589934592` example becomes honest rather than a footgun; `docs/FEATURES.md` corrects the "group-aware divisor" claim, which named the wrong instrument. ### Merged with `KV-GDN-STATE-BUDGET` (#1999), which landed first Five files overlap; four three-way-merged and one conflicted. Every one was resolved by the AGENTS.md `## Records` rule rather than by accepting the automatic merge: take `origin/main`'s complete file, prove it byte-identical, re-apply this row's scoped edit at an anchor asserted unique, then confirm `git diff origin/main -- <file>` carries only this row's lines. The index was union-appended and checked by row-ID set difference (729 base + 2 ours + 1 theirs = 732; 0 lost, 0 invented, 0 duplicated). `ResolveMaxNumSeqs` and every other line #1999 added is byte-identical to `origin/main` in this branch. Case 4 changed, and it is a correctness change rather than a textual one. It asserted `recurrent_state_bytes(cfg, params.max_num_seqs)` while #1999 makes the constructor hand the runner the RESOLVED concurrency. The two agree only while `ResolveMaxNumSeqs` does not clamp — true here by a 64x margin — so it now reads `eng.max_num_seqs()`, which is right by construction. **The first attempt at the conflicted file passed every static check and did not compile.** Taking the two conflict sides verbatim looked right: git had hoisted the closing `}` both blocks end with out of the conflict region as shared trailing context, so each side arrived one brace short. Marker count zero, `TEST_CASE` names unique, identifiers unique, includes unique, `git diff` clean and purely additive — six agreeing instruments, none measuring whether the file parsed. The compiler was, in ten lines. The check added for it is a brace balance against the pre-merge file, and the real remedy is the `## Records` procedure above, which never reconstructs a block from a conflict region. ### The two fixes compose; they do not fight `ComputeHybridKvBudget` never reads `layer_names`. The only input of its arithmetic this row moves is `kv_cfg.num_blocks` — and upstream's `num_blocks` is a PER-LAYER count (`kv_cache_utils.py:1008` divides by `num_layers`), which is the meaning its unification against one attention page assumes. Before this row the byte-budget path handed it a count inflated by the layer count, so its clamp was too permissive. Feeding it a truthful pool is what makes its seat count correct. Worked through for the device run below: `unified_block_tokens` = 32 x ceil(3371008 / 131072) = **832**, `unified_num_blocks` = 481 x 32 / 832 = **18**, `slots_per_seq` = 9, so **2 seats** and `max_num_seqs` clamps 32 -> 2; the recurrent allocation is 3371008 x 48 x 18 = **2.71 GiB**. Whole-engine KV **3.71 GiB against the base tree's 51.90 GiB** at the same flag. That arithmetic is checkable against #1999's own output rather than against itself: its engine prints `The KV pool (3072 blocks) holds 118 unified pages of 832 tokens`, and 3072 x 32 / 832 = 118. The formula reproduces both numbers. Consequence worth stating because it looks like a regression and is not: at a fixed `--kv-cache-memory` the seat count now falls by the same factor the pool does, 8.5x on this spec-on launch. To seat 32 sequences at k=8 the budget must be >= 16684941312 (15.54 GiB). `--num-blocks` is unaffected and always was — `ResolveNumBlocks` arm 1 returns it verbatim, and only the byte-budget path converts differently. ### Device confirmation, predicted before the run At `--kv-cache-memory 1073741824` on the 27B the `[kv-alloc]` line should read `num_blocks=481` rather than `4096`, for 1071775744 B (0.998 GiB) paged against the 1 GiB asked, plus 2.71 GiB recurrent on the integrated tree. ### Re-merge onto `main` (`b758127ec`) `origin/main` moved to `b758127ec` (#1994) and GitHub reported this PR `CONFLICTING`. Re-merged; head is `1548bc620`. The sole conflict under GitHub's merge is `.agents/issue-index.md` — the `merge=union` driver resolves it locally and GitHub does not honour that driver. `tests/CMakeLists.txt` merges clean either way, and no source file conflicted, so no take-both resolution exists on this branch. `.agents/issue-index.md` was union-appended and then checked by row-ID set difference rather than by reading the diff: base 730 + ours 2 (#1963, #1966) + main 15 = 747 expected, 747 actual, **0 lost / 0 invented / 0 duplicated**, with every merged row byte-identical to a row in one of the three sources. `tests/CMakeLists.txt` was rebuilt from `origin/main` (blob `6a616d781`) with this row's edit re-applied after the `test_loaded_engine_dense` registration line, an anchor asserted to occur exactly once. The rebuild is byte-identical to the three-way result; `git diff origin/main` is **5 added, 0 deleted**. Nothing this row owns moved. Comparing the reviewed pre-merge delta against the post-merge delta, normalised for hunk offsets, the added and removed line sets are **identical (906 added, 53 removed)** — the only difference is two context lines, where main's #1982 and #1992 index rows now sit adjacent to this row's. Every file main did not touch (`kv_cache_interface.{h,cpp}`, `runner.{h,cpp}`, `test_loaded_engine_dense.cpp`, `test_nemotron_h_scaffold.cpp`, `docs/USAGE.md`) is blob-identical to the reviewed head `4f8638a7c`, so the divisor arithmetic is untouched. **The full `ctest` suite has NO local verdict on this head, and CI carries it.** The box was at 98% disk; a full build drove the root filesystem to 100% with 4.1 GiB free, which is the ENOSPC state that makes unrelated checkers emit false policy refusals, so it was aborted and the space reclaimed. The focused suites were re-run after the abort from a clean rebuild — `test_hybrid_kv_budget` 8/8, `test_kv_state_budget` 5/5, `test_kv_cache_interface` 43/43, `test_runner` 20/20, `test_nemotron_h_scaffold` 14/14, `test_kv_cache_fp8_wiring` 31/31, `test_loaded_engine_dense` 30/30 (128 assertions), every one rc=0 — and `scripts/agent-preflight.sh --staged` is green. Closes #1963 Closes #1966 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

… draw that scanned 248,320 tokens on one thread (#1984, #1985) (#1997) On `Qwen/Qwen3.8-27B` vLLM draws its next token from 20 candidates and we draw ours from 248,320, because we read the checkpoint's `generation_config.json` for `eos_token_id` and discard every sampling key in it. `vllm bench serve` no longer sends `--temperature`, so both engines resolve 1.0 and both take the random-sampling path: that is a correctness divergence on the workload the parity gate runs, not a cosmetic default. The same fact explains the second defect. `RandomSampleKernel` was launched `<<<n, 1>>>` and walked the whole vocabulary on one lane, computing two SplitMix64 rounds and an f64 `log` per element. Eleven lines above it, `cuda_sample.cu` already records that a single-block single-thread scan of a ~151k vocab cost ~7.5 ms/token, and that is why greedy argmax was rewritten into the two-pass grid-strided reduction sitting there now. The Gumbel draw never got the same treatment, and its answer is the same lowest-index argmax that reduction already computes. ## What changed **The defaults (#1985).** The sibling read that already produced `generation_config_eos_ids` now also yields the six keys `ModelConfig.get_diff_sampling_param` narrows. `GetDiffSamplingParam` mirrors that narrowing and vLLM's `--generation-config auto|vllm|<dir>` selector. Both OpenAI handlers resolve a request against it with upstream's exact precedence: an explicitly sent field wins, an omitted field takes the checkpoint's value, and only a knob neither declares reaches the neutral OpenAI default. `nullptr` and an empty `DefaultSamplingParams` both reproduce the old resolution, which is what `--generation-config vllm` gives a user who wants it back. **The kernel (#1984).** `ArgmaxPartialKernel` is parameterised on the score, and `random_sample` instantiates it rather than growing a second hand-written reduction. The output is **bit-identical**, not merely equivalent: every element's score is the same `GumbelScore(prob, seed, row, j)` on the same device libm, and only the order in which those identical floats are combined changes, while `ArgReduce` compares the true global index rather than thread or block order. The serial kernel is retained behind `VT_FAST_RANDOM_SAMPLE=0`, mirroring `VT_FAST_ARGMAX`, so the equality gate is a same-binary A/B rather than a claim. `SplitMix64`, `ExpNoise`, `ArgReduce` and the launch partition moved to `include/vt/sample_common.h`, which `cpu_sample.cpp` and `cuda_sample.cu` both include. They had been written out three times, and the CPU copy is the reference the device copies are gated against, so a drift between any two of them was a token difference nothing looked for. ## Red before green `test_openai_protocol` failed **3 cases / 9 assertions** for the intended reason against a stubbed resolution, then passed 37/37 over 269 assertions. One expectation had to be corrected rather than the code: temperature 0 is greedy and `__post_init__` clears `top_p`, `top_k` and `min_p` upstream and here, so a checkpoint's `top_k: 20` cannot survive a `--temperature 0` request. That interaction now has its own case, because it is what keeps the SACRED greedy path unchanged. Six mutations, each applied, measured, and restored byte for byte: | mutation | gate | result | |---|---|---| | drop `bi < ai` from `ArgReduce` | `test_ops_sample` | RED, 35 assertions | | `prob * q` instead of `prob / q` in `GumbelScore` | `test_ops_sample` | RED, 2 assertions | | delete the sibling sampling read in `hf_config.cpp` | `test_generation_config` / `test_hf_config` / `test_openai_serving` | RED 3 / RED 1 / RED 3 | | revert `to_sampling_params(..., &default_sampling_params_)` in the completion handler | `test_openai_serving` | RED (reachability) | | the same in the chat handler | `test_openai_serving` | RED (reachability) | | delete the `--generation-config` argument branch | `test_serve_recipe_args` | RED, 12 assertions | The second row is the honest one, and it is why the table names where each mutation landed. `GumbelScore` is shared by the CPU reference and by the decomposition the equivalence cases drive, so a change to the score moves both sides together and those cases cannot see it. What caught it was the pre-existing `large-N empirical frequency approximates softmax probs` case. That is the right division of labour rather than a gap: the equivalence cases own the ORDER, the distribution case owns the FORMULA. ## What is not gated here **There is no CUDA toolkit on the implementing host, so `cuda_sample.cu` was not compiled locally.** Its first compile is this pull request's `cuda-fat-build`, and its first execution is the operator's leased box. The property the kernel rests on is gated on CPU against the production inlines — `ArgReduce`, `GumbelScore` and `ArgBlocksPerRow` under the kernel's exact two-pass partition, compared with `vt::RandomSample` over ten vocabulary widths including 248,320, six row shapes chosen for ties and masked zeros, and four seeds. The kernel's launch geometry, shared-memory sizing and scratch lifetime are not, and `compute-sanitizer` is requested for exactly that reason: this change adds a second persistent device scratch and a new grid geometry, and #1958 is an illegal memory access on this same sampler surface. **Reachability is proven at three hops of four.** The CLI hop re-execs the real `VllmServerMain`. The handler hop enters through `create_completion` / `create_chat_completion` with the defaults produced by the real `LoadHfConfig` and `GetDiffSamplingParam` off a real `generation_config.json`. The two-line join in `server_main.cpp` sits after model load and no committed generative checkpoint fixture can reach it, so no CPU test turns red when it is deleted. Stated as a gap, not waived: the smallest thing that would close it is a tiny on-disk causal-LM fixture of the `tests/vllm/models/fixtures/llama_embed_e2e` kind, which this row did not add. ## The measurement this does not merge without `AGENTS.md` and #1975 settle it between them: a performance change here does not merge on a green compile, and a per-kernel figure is not evidence. #1929 landed a 708 us to 40 us top-k kernel with fresh review, mutation proofs and a green CUDA build, and cost 16.9 tok/s end to end. The request is therefore stated before it is run, and it is in `## Now` of the row's spec. `vllm bench serve` against our server and the pinned vLLM on the identical `Qwen3.8-27B` artifact, vLLM's graphed production configuration as the denominator, **no `--temperature` on either side**, in four same-binary arms so each half of the row is attributable on its own: | arm | `--generation-config` | `VT_FAST_RANDOM_SAMPLE` | isolates | |---|---|---|---| | A | `vllm` | `0` | the pre-change baseline | | B | `auto` | `0` | the config read alone | | C | `vllm` | `1` | the kernel alone | | D | `auto` | `1` | the shipping default | **`--speculative-config` would measure nothing either**, and the standing baseline recipe carries it ([#2002](#2002), filed from this row). `GPUModelRunner::sample_tokens` branches on `exec_state_.step.num_draft_tokens > 0` alone and returns the greedy-only `RejectionSampler`'s output, so with drafts present `Sampler::forward` — and with it `vt::RandomSample` — is never called, whatever the temperature. Every arm above therefore runs without it. The same routing is a correctness divergence in its own right: at temperature 1.0 under spec decode we emit the target argmax where vLLM draws from the residual distribution, and `include/vllm/v1/spec_decode/rejection_sampler.h` already states the contract nothing enforces. **`--temperature 0` would measure nothing**, and it is what every harness in `tools/bench/` passes today (`run_qwen35_4b_ab.sh` sends `--temperature 0`, `profile_vllm_online_gate.py` sets `temperature=0.0`). At temperature 0 `PostInit` clears `top_p`, `top_k` and `min_p`, the batch is all-greedy, and `RandomSample` is never called, so all four arms are the same run and four identical numbers read exactly like "no effect". Predicted, and marked as prediction: **D >= C > A**, with C - A the large term, of order 10 ms/step recovered at B=32. That figure is INFERRED from this file's own recorded ~7.5 ms/151k anchor scaled by vocabulary and by the extra per-element transcendental, not measured. B - A is predicted small and possibly negative on throughput; the reason to want it is that it makes the two engines sample the same distribution. Two outcomes would falsify the design rather than the tuning, and both are named in advance: D materially slower than C means the top-k path costs more than the distribution it saves; D no faster than A means the sampler was never the bottleneck and the per-step term lives in the `cudaMalloc`/`cudaFree` and the blocking download recorded under `## Owed`. ## Landing this re-baselines every benchmark number we have **Do not compare a number taken before this merges with one taken after.** The benchmark checkpoint `qwen3.8-27b-nvfp4-mtp-sm121-r0b0tlab` ships a `generation_config.json` containing `{"do_sample": true, "temperature": 1.0, "top_k": 20, "top_p": 0.95}`. Today we ignore it and sample over the full 248,320-token vocabulary while vLLM samples from top-20 — so our arm has been paying a per-token cost we inflict on ourselves, on a live benchmark asymmetry, and the tokens we emit are drawn from a different distribution than the oracle's. After this lands, an unparameterised request on that checkpoint samples top-20 / top-p-0.95 like vLLM's. That is the correct behaviour and it is the point of #1985, but it moves the numbers. Our arm must be re-baselined at the merge commit, and any figure quoted across the boundary is comparing two different sampling configurations rather than two engines. `--generation-config vllm` restores the old resolution exactly if an old number ever needs reproducing. ## What #1994 does and does not invalidate here Nothing in this pull request or in the row's spec quotes a TTFT or a TPOT figure, so there is nothing to void. Checked rather than assumed: the only measured numbers either document carries are `~7.5 ms/token` — `cuda_sample.cu`'s own long-standing anchor for the single-thread greedy scan, a per-kernel decode-path attribution rather than a harness TPOT — and `16.9 tok/s`, an end-to-end throughput figure from #1929. Throughput figures are unaffected by #1994. Everything else here is explicitly marked as prediction. The acceptance measurement in `## Now` is likewise safe, and the reason is in #1994's own spec rather than in its summary: it lists `/v1/completions` under what it does **not** touch, and scopes the defect to `vllm bench serve --backend openai-chat`. Should any arm be run on the chat backend instead, the residual bias is bounded and small — TTFT was understated by roughly one decode step, and `TPOT = (E2E - TTFT)/(tokens - 1)` spreads that over 63 tokens, so the greedy arm inflates by about 1.7 ms and the default arm by about 5.4 ms, overstating the greedy-versus-default gap by about 1.6%. That shifts the acceptance bar by well under a millisecond and changes no verdict. ## Owed, with anchors - `src/vllm/v1/sample/sampler.cpp:344` allocates the `[n, vocab]` probs buffer through `Backend::Alloc`/`Free` (`sampler.cpp:41,47` to `src/vt/cuda/cuda_backend.cu:80-85`) — a 30.3 MiB raw `cudaMalloc` plus a device-synchronising `cudaFree` every decode step at B=32. - `src/vllm/v1/sample/sampler.cpp:358` `rs.download(...)` is a blocking `Synchronize` every step, because the zero-sync device-resident path at `sampler.cpp:456-461` requires `sm.all_greedy` and is structurally unreachable at temperature 1.0. - `src/vt/rocm/rocm_sample.hip::RandomSampleK` carries the identical `<<<n, 1>>>` defect and its own copy of `ExpNoise` (`rocm_sample.hip:38`). Left untouched rather than changed on hardware this row cannot gate. - `ExpNoise` computes `-log(u)` in **double**, where upstream's default is f32 (`use_fp64_gumbel: bool = False`). That is an unannotated widening of the kind `.agents/porting.md` says a token gate cannot see, and on a part with 1:64 f64 throughput it is the dominant remaining cost of the parallel kernel. Narrowing it changes which token is drawn, so it is a separate row with its own gate. - `--override-generation-config`, the `max_new_tokens` server-wide output cap, and an offline `LLM.get_default_sampling_params` equivalent for the C ABI. Closes #1984 Closes #1985 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: claude-code:claude-opus-5-1m [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

origin/main gained #1994 (b758127), #2000 (61ba99f, the KV group layer-count fix) and #1997 (2c27f0d, the sampling defaults plus the Gumbel kernel) while this row was in flight. Merged rather than rebased. The .agents/issue-index.md union is verified by row-ID SET DIFFERENCE rather than by a clean automerge, because GitHub ignores this repository`s union driver and two relocations can automerge into a duplicate. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]

…d batch row cannot take it from its owner (#2008) (#2010) DFlash2 served exactly one sequence. At concurrency 2 the engine refused on its own position invariant one request in, `EngineCore` stopped, and every later request on that server came back `[request submitted to a stopped AsyncLLM]`. This is the blocking defect for the #1574 three-engine comparison: vLLM and SGLang were both measured at c=1..32 with the same DFlash2 draft at k=8, and we could produce c=1 and nothing above it. Closes #2008. ## Why the accumulation desynchronises The runner held the draft context in four arrays indexed by **batch row**. A row index is not stable for a request's lifetime here: `InputBatch::condense` slides a live request down into the hole a finished neighbour left (`input_batch.cpp:686-706`), and `swap_states` exchanges two live rows (`:762-847`). Both permute every per-slot array they own — `req_ids`, `num_computed_tokens_cpu`, `num_accepted_tokens`, `last_sampled_tokens`, `prefill_len`, the block-table rows, the index-keyed sampling maps — and neither knows these four exist, because they live on `GPUModelRunner` rather than in `InputBatch`. So when the first of two concurrent requests finished, the survivor was moved onto the departed request's bookkeeping. The reused-slot test at `runner.cpp:2895-2906` read a changed occupant, allocated a fresh empty store and set `ctx_len = 0` — under a request still using it — and the invariant at `:2939-2945` then refused rather than drafting from a foreign context. `ok=1` at c=2 is that mechanism's signature, not an incidental count. At c=1 the only row is row 0: a request finishes, the next is admitted into the same row, and the reset is *correct*, because the new occupant is a fresh prefill whose first position is 0. The condense move that breaks the state exists only when a second live request has to be slid over a departed one. The operator's isolation had already settled the layer: with `--speculative-config` omitted and everything else identical, the same two concurrent requests both complete. This tree already names the bug class, in a comment above the log written to solve it for a different array (`input_batch.h:240-252`): *"Upstream needs no equivalent because it never condenses ... the slot index is stable for the request's lifetime. This log is the price of our condensed dense batch."* The DFlash2 arrays never got that treatment. ## Upstream Read beyond our parity pin at `b389ac2946` (the clone is shallow and checked out at the pin, so anchors were read with `git show`). **Upstream has no analogue of the host-side counter at all.** DFlash and DFlash2 address the draft KV by absolute position (`dflash/speculator.py:562-590`), re-read their one anchor from the target's own positions every step (`:553`), and index every cross-step tensor by the persistent request slot through `req_state_idx = idx_mapping[req_idx]` (`:536`, `dflash2/speculator.py:95`) — in a V2 runner that has no `condense` at all (`gpu/states.py:29,100,132`). Its legacy V1 runner does condense, and there the draft's block-table row moves with the request (`gpu_input_batch.py:786` → `block_table.py:367-373`). Both upstream shapes key the draft context to the **request**. Ours keyed it to neither. ## The change One `unordered_map<string, DflashReqCtx>` holding the store, the context length and the disabled flag. Every row permutation the batch can perform — today's two and any future one — is now a no-op for the draft context. The reused-slot test goes away with it. "Has this row's occupant changed" was only ever a proxy for "is this state this request's", and the proxy is precisely what broke: after a condense move the answer was yes for a request whose context was perfectly valid. Keyed by id the question cannot be asked wrongly; a first sight of an id constructs the entry, and that construction *is* the reset. `dflash_ctx_disabled_`'s own comment — "the flag is a property of the REQUEST, not of the row" — becomes literally true. The **decode-first reorder** is fixed for free, and it is a second live trigger rather than a hypothetical one. `reorder_batch_to_split_decodes_and_prefills` runs unconditionally every step (`runner.cpp:1324`) and swaps live rows through `swap_states`. It emits no swap only while arrival order already puts decodes before prefills — the common case, since condense keeps older decoding requests at low rows and a new arrival appends at the end as a prefill — which is why the #2008 measurement met the condense move first. A batch whose regions are out of order does swap, and on the pre-change code that exchanged two live requests' draft contexts. (The commit body and an earlier draft of the spec called this path "inert for a Qwen3 target today"; that was wrong and the spec now says so — the reorder has no model-family gate at its call site.) Entries are pruned each propose against `InputBatch`'s membership — the authority on residency, unlike `exec_state_.req_ids`, which lists only the rows scheduled this step. Each entry owns a device allocation, so pruning is part of the change rather than a follow-up. One detail a reviewer will want checked: the prune is guarded by `size() > num_reqs()`, and a departure paired with an arrival in the same step can leave the sizes equal with a stale entry present. That entry survives **exactly one further step** and cannot be read in the meantime, because lookup is by current request id. It is then reclaimed: the resolve loop inserts the newcomer, so the next prune sees `size() > num_reqs()` and fires. The lag is bounded at one step and is self-correcting; it does not accumulate. **Adopting upstream's paged shape** — the draft context as a real KV cache group carried by `MultiGroupBlockTable::move_row` — is the right end state and is recorded as owed, not attempted. It replaces `DflashDeviceKVStore`, changes how #1919 sizes the draft budget and how #2007 splits the pools, and would land the concurrency repair behind a rewrite. ## The gate `tests/vllm/v1/spec_decode/test_dflash2_concurrency.cpp` drives two concurrent requests through the synchronous production front and **reproduces the discontinuity on CPU**, which is what converts this from something only a GPU run can see. Its load-bearing leg is not "it did not throw". Marking the moved row disabled — the #1919 fallback, which `continue`s *before* the invariant — also makes the throw stop and emits **identical tokens**, because the verify is lossless and a request that stops speculating just runs slower. So the gate asserts that the survivor keeps proposing, and at every step it is alive for. ## Mutations Each rival repair was **built on the pre-change code and measured**, not argued about. | Mutation | Result | |---|---| | **A′ — the fallback repair.** On a position mismatch, disable the row instead of asserting. | **CAUGHT.** Case 1 goes green and the propose count reads `none_lines == 9` — the survivor stopped proposing for all nine remaining steps. | | **B′ — "silence the check."** Delete the position invariant, let the reset context stand. | **NOT CAUGHT.** Filed as #2009. | | **Reachability.** Delete the `propose_drafts_block` call site in `propose_drafts_dflash`. | Red, by name. | **One leg was written, run, and removed rather than shipped.** Comparing the drafted blocks against a solo control is a **tautology on this fixture**: with the invariant deleted and the context reset at every row move, the synthetic draft still emits `12 12 12` at every step of both runs — seeded-noise weights over a 24-token vocabulary, and the selector walk collapses to one id. Nine passing string comparisons measured nothing. The test file and the spec both record it, because the next person to reach for that leg will find the same fixture. That B′ is consequently ungated is #2009, not a thing this row quietly left out. ## What this does not do, and what must be measured before it is trusted **A concurrency change does not merge here on a green compile.** The end-to-end ladder is the operator's to run — rungs c = 1, 2, 4, 8, 16 at the #2008 flags. Predictions and falsifiers are in the spec under `## Gates`. In short: c=1 must be **inert** (24.70 out tok/s, TPOT 37.90 ms), no rung may VOID or report `failed > 0`, and c=2 must not come in below c=1. A rung that lands low but completes is #2007 and the `P == 1` capture gate being measured for the first time — this change is what lets a batch reach `P > 1` at all — and those are named, owned and out of scope. ## Gates run | Gate | Result | |---|---| | `cmake --build build -j 4` (CI's CPU configuration) | **rc 0**, no `error:` lines | | `ctest --test-dir build --output-on-failure` | **rc 0 — 628 tests, 0 failed**, 5 pre-existing skips | | `test_dflash2_concurrency` in that suite | Passed, 37.73 s | | `test_dflash2_runner_reach` (sibling DFlash2 gate) | Passed, 45.78 s | | `scripts/agent-preflight.sh --staged` | **rc 0**, 0 gates failed | | `check-agent-record.py` | rc 0 | | `check-commit-trailers.py` / `check-commit-style.py` | rc 0 | Exit codes were captured explicitly rather than read off the tail, which is how the first preflight run's `role-undeclared` failure was caught behind a clean-looking output. The staged preflight's trailer check initially **SKIPPED** because `origin/main` had moved and was no longer an ancestor of HEAD — a diff-scoped gate that skips still exits zero. `origin/main` is merged (bringing #1999) and both checks were then rerun against the new base. ## Found and not fixed - **#2009** — the position invariant is ungated; deleting it leaves this row's own gate green. Closing it needs a fixture whose draft is sensitive to its context, which this one is not. - **UNVERIFIED hypothesis, labelled rather than asserted: a prefix-cache hit or a resumed request may trip the same invariant at c=1.** The reasoning is that such a request is admitted with `num_computed_tokens > 0` and has no draft context for the cache-supplied tokens, so its first propose would read `positions[rows[0]] > 0` against `L == 0`. **It was not reproduced.** A throwaway probe forced `enable_prefix_caching = true` and issued the same 20-token prompt twice; both completed, nothing threw, and the engine's own `prefix_cache_metrics()` reported `queries=40 hits=0` — the cache never engaged, so the probe measured nothing about the hypothesis. The fixture's target is a GDN hybrid, the family upstream defaults prefix caching OFF for. Reported as a hypothesis, not a defect; confirming it needs a decoder-only DFlash2-capable target. - **#2007** — the two-pool allocation. Noted, not widened into. ## Re-merge onto `main` (`b758127ec`) `origin/main` moved to `b758127ec` (#1994) and GitHub reported this PR `CONFLICTING`. Re-merged; head is `2ea92d2fc`. The sole conflict under GitHub's merge is `.agents/issue-index.md` — the `merge=union` driver resolves it locally and GitHub does not honour that driver. `tests/CMakeLists.txt` merges clean either way, and no source file conflicted, so no take-both resolution exists on this branch. `.agents/issue-index.md` was union-appended and then checked by row-ID set difference rather than by reading the diff: base 730 + ours 2 (#2008, #2009) + main 15 = 747 expected, 747 actual, **0 lost / 0 invented / 0 duplicated**, with every merged row byte-identical to a row in one of the three sources. `tests/CMakeLists.txt` was rebuilt from `origin/main` (blob `6a616d781`) with this row's edit re-applied after the `test_dflash2_ctx_capacity` source line, an anchor asserted to occur exactly once. The rebuild is byte-identical to the three-way result; `git diff origin/main` is **7 added, 0 deleted**. Nothing this row owns moved. Comparing the reviewed pre-merge delta against the post-merge delta, normalised for hunk offsets, the added and removed line sets are **identical (792 added, 74 removed)** — the only difference is two context lines, where main's #1982 and #1992 index rows now sit adjacent to this row's. Main's delta touches neither `runner.h` nor `runner.cpp`, and both, along with `test_dflash2_concurrency.cpp`, are blob-identical to the reviewed head `3d895a202`, so the request-id keying of the draft context is untouched. Re-merged again onto `2c27f0d57`, which carries #2000 and #1997. Head is `53e144f37`. #2000 is the first commit on `main` to touch `runner.h` and `runner.cpp`; #1997 touches neither. Both runner files three-way merged with no conflict because the edits are disjoint against the common base `b758127ec` — #2000 at `runner.h` 357-474 and `runner.cpp` 334-823, this row at `runner.h` 836-870 and `runner.cpp` 2752-3148 — and the merge is verified in both directions, each side's per-file delta byte-identical to its own source diff at shifted offsets. Brace balance checked arithmetically anyway: `runner.h` expected 55, merged 55; `runner.cpp` expected 337, merged 337; both for `{` and `}`. Index re-checked against the current base: 745 + 2 ours (#2008, #2009) + 5 theirs (#1963, #1966, #1984, #1985, #2002) = 752 expected, 752 actual, **0 lost / 0 invented / 0 duplicated**. `tests/CMakeLists.txt` rebuilt from blob `d4d6cff71` at the same uniqueness-asserted anchor, 7 added / 0 deleted, with #2000's and #1997's own lines intact. The reviewed change is unchanged across all three re-merges: added/removed line sets identical to `git diff 2a42cb3 3d895a2` (786/68 excluding file headers, 792/74 including them). Re-merged onto `c714b0234`. Head is `822b90b6e`. **This merge is CODE-TOUCHING and takes a fresh CI cycle:** #1977 adds 88 lines to `src/vllm/v1/worker/gpu/runner.cpp`, the file this row rewrites, so the merged translation unit is not the one CI built (green blob `59265ee22`, merged blob `f7ed973af`), and `test_runner` moves 20/544 to 22/567 with #1977's own cases. The merge took no conflict because the edits are disjoint against base `2c27f0d57` (#1977 at `runner.cpp` 353-607, this row at 2770-3166) and is verified in both directions, each side's per-file delta byte-identical to its own source diff. Brace balance expected 342, merged 342, for `{` and `}`. `runner.h`, `test_dflash2_concurrency.cpp` and the spec stay blob-identical to `53e144f37`. Index re-checked against the current base: 750 + 2 ours (#2008, #2009) + 14 theirs = 766 expected, 766 actual, **0 lost / 0 invented / 0 duplicated**. `tests/CMakeLists.txt` rebuilt from blob `21d878869` at the same uniqueness-asserted anchor, 7 added / 0 deleted. The reviewed change is unchanged across all four re-merges: added/removed line sets identical to `git diff 2a42cb3 3d895a2` (786/68 excluding file headers). Local gates on this head: `BUILD_EXIT=0`, `ctest -R dflash` 27/27, `test_dflash2_concurrency` 2/2 (21 assertions), `test_runner` 22/22 (567), `agent-preflight.sh --staged` green over 109 gates. **The full `ctest` suite has NO local verdict on this head, and CI carries it.** The box was at 98% disk; a full build drove the root filesystem to 100% with 4.1 GiB free, which is the ENOSPC state that makes unrelated checkers emit false policy refusals, so it was aborted and the space reclaimed. The focused suites were re-run after the abort from a clean rebuild — `ctest -R dflash` 27/27, `test_dflash2_concurrency` 2/2 (21 assertions), `test_dflash2_runner_reach` 8/8, `test_dflash2_ctx_capacity` 6/6, `test_runner` 20/20, every one rc=0 — and `scripts/agent-preflight.sh --staged` is green. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: CLAUDE-CODE:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

… default server never sets, so a non-speculative capture cudaMallocs (#2029) (#2047) With DFlash2 speculation off the engine could not warm up at concurrency 8. It died inside CUDA graph capture, and every request after that came back 500 `[request submitted to a stopped AsyncLLM]`. ``` engine-fatal: EngineCore busy loop threw: vt cuda: cudaMalloc: operation not permitted when stream is capturing ``` That is the configuration a user gets by omitting `--speculative-config`, so this is the default server and not a benchmark artefact. Closes #2029. ## The asymmetry is not in the speculative code #2029 records that the same binary WITH `--speculative-config` warms and serves c=1/2/4 cleanly, and concludes that "the failing allocation is on a path taken only when speculation is off". That is exactly true, and the path is the **absence** of the #1380 capture pre-grow. Both Qwen3.5 decode-graph drivers compute ```cpp const bool dbuf = impl_->dbuf || spec_step; // qwen3_5.cpp:10764, :11315 ``` with `impl_->dbuf = enabled && DecodeGraphDoubleBufferEnabled()` (`:10529`, `:11082`), `DecodeGraphDoubleBufferEnabled()` false unless `VT_ASYNC_EXECUTOR=1` (`:10302-10308`), and `spec_step = gdn_meta.num_spec_decodes > 0` (`:10715`, `:11267`). And `Pool(b).PreGrowForCapture(b, s.demand)` sat **inside** `if (dbuf)` — `:10885`/`:10907` in the 35B MoE driver, `:11439`/`:11461` in the 27B dense one, which is the driver the #1574 subject runs on. So on the default server neither disjunct holds, the pre-grow never executed, and the driver opened `vt::GraphCaptureScope` over a `DevicePool` nobody had prepared. A `Get` miss inside that region is `DevicePool::Get`'s `b.Alloc(key)` (`device_pool.h:212`), which on CUDA is the `cudaMalloc` in the message. #1380 named the same message from the same driver. `s.demand` and `Pool(b).MarkStepBoundary()` were already recorded **unconditionally** (`:11607`/`:11251`, `:11046`/`:10697`). Only the consumer was gated, and the guard is an accident of history rather than a decision: that block exists for the `VT_ASYNC_EXECUTOR` parity ring's drain and its persistent step inputs, and #1393 replaced the single-block pre-grow in place without revisiting what it sat under. **The `!dbuf` arm needs it at least as much as the `dbuf` arm.** It passes `persistent_sdi == nullptr` (`:11531`, `:10974`), so `BuildStepDevInputs` and `MaybeBuildAttnCosSin` run from the main pool INSIDE the captured region. That is also what makes `s.demand` exact for it: the cold step takes the identical `nullptr` branch, so its per-class peak IS the capture's demand. On the `dbuf` arm the captured region's main-pool demand is a strict subset of the cold step's, which is the containment #1393 recorded. ## The change One statement moves in each driver, out of `if (dbuf)` and in front of it. Everything else stays: `b.Synchronize` drains an in-flight replay only the parity ring can leave behind, and the `StepDevInputs` / `s.pin` construction is the persistent-input path the ring and the spec capture need. The pre-grow now precedes that `b.Synchronize` rather than following it. A pre-grow is `Backend::Alloc` and nothing else — it enqueues no work and reads no in-flight buffer — and the drain still happens before `BeginCapture`, which is the property it was added for. ## Prefix caching drops the same fault to c=1, and this finding predicts it Measured by the operator on `dgx:gpu0` while this was in flight: same `3d895a202` binary, speculation off, plus `--enable-prefix-caching --scheduling-policy lpm`, **concurrency 1** — the identical message, `illegal memory access` 0, `position discontinuity` 0, `ok=0 failed=8`. Nothing in this change was ever scoped to batch 8. The pre-grow is skipped on EVERY non-speculative step at EVERY padded size, and **this pull request's own gate is a single-request `S = 1` case**. What the two configurations differ in is only whether the free list happens to be short when the capture opens. The rival reading — that some buffer is sized from a per-step quantity — was checked against the code and does not hold on this driver: | Candidate | Why not | |---|---| | `block_table_num_cols` / `max_blocks` | fixed once by `BlockTable`'s constructor from `max_model_len` (`block_table.cpp:49`), read unchanged by `gather_block_table` (`runner.cpp:1185-1187`). It cannot move between two steps, and prefix caching does not touch it | | FA-2 decode scratch | `DecodeShapeKey` is `{batch, hq, heads, groups, head_dim, max_blocks, page_size, num_splits}` (`cuda_flash_attn_fa2.cu:1013`), all key-determined or constant — and it carries its OWN capture refusal at `:1021-1029` with a different message, so it cannot produce this one | | the forward's knowledge of prefix caching | it has none; `enable_prefix_caching` appears nowhere in the runner's forward path | **Which size class is short in either run is NOT determined here, and cannot be from a CPU box.** The fix does not depend on the answer — an unprepared pool is unprepared whatever empties it — but the issue stays open until a device says so. See *What is not verified* below. Not the same fault as #2042 (prefix caching + DFlash2 at c=1, the draft's position invariant). With speculation ON, prefix caching fails through #2042; with speculation OFF, through this. ## The gate Two cases in `tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp`, one per driver, on the existing CPU seam in `decode_graph_seam_harness.h`. The assertion is the guarantee, not the call: **zero `Backend::Alloc` between `BeginCapture` and `EndCaptureGraph`**, on a step where `num_spec_decodes == 0` and `VT_ASYNC_EXECUTOR` is unset. `CHECK(the pre-grow was called)` would be a transcription — it stays green when the demand profile is wrong, which is the half #1393's own body recorded as ungated: *"the fix rests on the captured forward demanding no more blocks of any size class than the eager forward at that shape did ... no test asserts it."* `CaptureCapableCpuBackend` gains the capture-window split for it: one counter, incremented under a flag the existing `BeginCapture` / `EndCaptureGraph` overrides set. **The pool is drained between the cold step and the capture step**, and that is the load-bearing part of the construction. In production the free list is SHORT rather than empty — every captured `SizeSlot` retains its `[S, vocab]` logits and `[S, H]` hidden forever, `DevicePool` is keyed by SIZE CLASS, and a ramping server captures more shapes. Reproducing that by arithmetic would make the case depend on whether two of the tiny model's tensors happen to collide in a class, i.e. on a coincidence rather than on the rule. `Drain` is the same condition taken to its limit, it is a production API called at phase changes, and it is what also settles the premise above: an empty free list can only be served by the pre-grow, so green says the cold step's profile COVERS the capture. `CHECK(allocs() > 0)` is the non-vacuity guard. The pre-grow is itself a driver allocation made outside the region, so a fixed driver must allocate in this step and must allocate none of it under capture. ## The `sanitize-cpu (thread)` red, and a correction That job refused the two new cases on the PRECONDITION, not on the guarantee: ``` tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp:948: FATAL ERROR: REQUIRE( freed > 0 ) is NOT correct! values: REQUIRE( 0 > 0 ) ``` **It is not ThreadSanitizer and not its allocator.** `.github/workflows/ci.yml:1598` sets `VT_POOL_BYPASS: "1"` for that job, on **both** the `address,undefined` and the `thread` lane. The identical `REQUIRE( 0 > 0 )` reproduces at the same two lines on an ordinary **non-sanitized** Release build with that one variable set and nothing else changed. (The `address,undefined` lane read `pending`, not green — it had not finished, so it never contradicted this.) Under bypass every `Get` is a raw `Backend::Alloc` and every `Put` a real `Free` (`device_pool.h:113-127`, `:246-252`), so there is no free list, `Drain` reports 0, and `PreGrowForCapture` returns before it grows anything (`:429`). The guarantee these cases assert is **false by design** there, and false identically for the fixed and the unfixed driver. So both cases now carry `doctest::skip(PoolBypassLane())`, with the reason in the case NAME so a reader who sees the skip count can recover it, and the predicate mirroring `DevicePool::Bypass()` byte for byte rather than approximating it. `REQUIRE(freed > 0)` is **unchanged** — M3 measured that it is what stops the pooled lane from asserting nothing. **A correction, because the first reading of this failure was backwards.** The guard did **not** save a vacuous pass. Removed, the cases reach the capture assertion and **FAIL** it, at **107** and **195** driver allocations inside the capture, because under bypass every `Get` is a driver call. What the guard buys is an inevitable failure that names the PRECONDITION instead of the SYMPTOM. Worth having, and a smaller claim than the one first made for it. **The guarantee is therefore not exercised by `sanitize-cpu` — and that is the job's configuration, not a limit of the instrument.** Measured on a real TSan build, `-DVLLM_CPP_SANITIZE=thread`, `setarch -R`: | Environment | `test_qwen3_5_decode_graph_seam` | |---|---| | `VT_POOL_BYPASS=1`, as CI sets it | exit **0**, 8 passed / **2 skipped**, 138 assertions, 0 TSan warnings | | bypass unset, the pool ENABLED | exit **0**, **10 passed / 0 skipped**, 156 assertions, `0` driver allocations inside both captures, **0 TSan warnings** | ThreadSanitizer runs this guarantee green when the pool is on. Nothing in the test file can opt one case back in — `Bypass()` is read once into a process-wide function-local static (`device_pool.h:480-486`), so no scope, no locally constructed `DevicePool` and no `ActivePoolScope` reaches it — and unsetting the variable for these cases would be worse than the gap, because the pool would then retain blocks that the job's own `ASAN_OPTIONS=detect_leaks=1` reports as leaks. Filed as **#2059**. ## Evidence `mudler-ubuntu-box`, x86-64, CPU-only. `Release`, `-DVLLM_CPP_CUDA=OFF -DVLLM_CPP_TRITON=OFF -DVLLM_CPP_SERVER=OFF`, Ninja, `-j 4`. | Reading | Result | |---|---| | RED, both new cases, before the fix | exit **1**, 2 cases / 0 passed / **2 failed**. `45` driver allocations inside the dense capture, `42` inside the MoE one | | GREEN, both new cases | exit **0**, `0` and `0` inside the capture, `49` in the step — all of them the pre-grow, outside the region | | `test_qwen3_5_decode_graph_seam`, whole file, three runs | exit **0** each, 10 cases / 156 assertions | | `test_qwen3_decode_graph_seam` | exit 0, 4 cases / 231 assertions | | `test_qwen3_moe_decode_graph_seam` | exit 0, 3 / 228 | | `test_deepseek_v2_decode_graph_seam` | exit 0, 3 / 230 | | `test_voxtral_decode_graph_seam` | exit 0, 3 / 230 | | `test_qwen3_dflash_decode_graph_seam` | exit 0, 4 / 23 | | `test_qwen3_dflash2_draft` | exit 0, 43 / 449 | | `test_moe_async_device_ids` | exit 0, 6 / 191 | | `test_device_pool` | exit 0, 11 / 59 | | `test_breakable_graph` | exit 0, 30 / 265 | | `test_persistent_step_input` | exit 0, 10 / 66 | | `test_qwen35_paged_forward` | exit 0, 7 / 63 | | `test_qwen3_5_gdn_spec_routing` | exit 0, 6 / 52 | | `test_qwen3_5_fa2_class` | exit 0, 6 / 15 | | `scripts/agent-preflight.sh` | exit **0**, every gate green, none skipped | | bypass lane, `VT_POOL_BYPASS=1`, non-sanitized | exit **0**, 8 passed / **2 skipped**, 138 assertions | | TSan build, CI environment | exit **0**, 8 passed / 2 skipped, 0 TSan warnings | | TSan build, pool ENABLED | exit **0**, 10 passed / 0 skipped, 156 assertions, 0 TSan warnings | Every row above was re-measured on the merged tree after `origin/main` moved to `2c27f0d57` (#1994, #2000, #1997), because merge-tree clean is not merge-tree builds and that merge touched `qwen3_5.cpp` from both sides. The four `test_qwen3_5_decode_graph_seam` runs also read on #1390, which reports that file exiting 139 with a non-reproducible assertion line: four runs here, exit 0 every time, 10 cases and 156 assertions every time. That is one build on one box and it does not close #1390; it is recorded because a flaky neighbour is otherwise something a reviewer has to re-derive. ## Mutations Each compiled clean, each restored by `sha256sum -c` against the baseline before the next. | Mutation | Result | |---|---| | **M1**, the production call site: put `PreGrowForCapture` back inside `if (dbuf)` | **RED** — 10 cases / 8 passed / **2 failed**, and the two failures are the new cases and nothing else | | **M2**, the pool half: `PreGrowForCapture` returns 0 before it grows anything | **RED** — `test_qwen3_5_decode_graph_seam` 8/10 and `test_device_pool` 10/11 | | **M3**, the case's own construction: remove the `Drain`, on the UNFIXED driver | the capture-window assertion goes **vacuously green** — `0` allocations inside the capture and `0` in the whole step — and only `allocs() > 0` fires | | **M4**, the skip predicate: `PoolBypassLane()` returns false, run under `VT_POOL_BYPASS=1` | **RED** — 10 cases / 8 passed / **2 failed**, **146 assertions**: a byte-for-byte reproduction of the `sanitize-cpu (thread)` reading, on a non-sanitized build. The skip is load-bearing, not a decorative no-op | **All four were re-run after the fixture changed**, because a fixture change can disarm a mutation proof. M1, M2 and M3 read identically before and after. **M3 is the one worth reading.** It says the drain is load-bearing rather than decorative: without it this case cannot detect the defect at all, and without the non-vacuity guard beside it the case would have reported a pass while measuring nothing. M1 is also the reachability mutation "Nothing lands dead" asks for: deleting the production call site reds the focused gate. ## What is not verified **No GPU was available to this implementer**, and nothing here is reported as device-verified. The change edits no `.cu`, so the CUDA arm is a `cuda-fat-build` compile verdict and never a run verdict either way. #2029 therefore stays **open**. This removes a proven, non-speculative-only gap that produces exactly its message, at every padded size including the c=1 one prefix caching exposes. It does not prove that gap was the only one. Three experiments settle it, in this order, each on the operator's own invocation with `--speculative-config` removed and `--enable-prefix-caching --scheduling-policy lpm` added, at c=1: 1. **`VT_ASYNC_EXECUTOR=1` on the UNCHANGED `3d895a202` binary.** That flips `impl_->dbuf` true and turns the pre-grow on with no rebuild and no patch. A run that SERVES proves the missing pre-grow is the cause, on the device, against the exact binary #2029 was measured on. A run that still dies is **inconclusive rather than a refutation**, and the asymmetry has to be stated when the result is read: the same flag opens the parity ring, which retains a second `[S, vocab]` logits and `[S, H]` hidden per size and can empty the free list by a route of its own. 2. **The same configuration on a binary built from this branch**, environment untouched. Serving says this closes that entry point; dying says a second site exists. 3. **`VLLM_CPP_QWEN3_DENSE_DECODE_GRAPH=0`, as the control.** Serving confirms the fault is inside this driver's capture; dying says the capture that allocates is elsewhere and 1 and 2 answered the wrong question. If 2 still dies, the next instrument is a backtrace at `CudaBackend::Alloc` — how #1380 was located, and exactly what #2037 exists to make unnecessary. `VT_POOL_STATS` cannot substitute: it prints per-pool totals at destruction, after the engine is already dead, and names no class. ## Found and NOT fixed, each with an issue - **#2035** — seven decode-graph drivers (`qwen3.cpp`, `qwen3_moe.cpp`, `deepseek_v2.cpp`, `deepseek_v4.cpp`, `voxtral.cpp`, `laguna.cpp`, `qwen3_dflash.cpp`) open a capture with **no pre-grow and no demand profile at all**. A grep for the #1380 machinery outside `device_pool.h` returns `qwen3_5.cpp` and nothing else. Three of them argue in comments that their cold step makes the capture allocation-free, which is the reasoning #1380 measured and refuted. Each needs its own `MarkStepBoundary` / `StepDemandProfile` wiring and its own gate, so it is a different change with a different blast radius. - **#2036** — `DenseAlignFor` (`qwen3_5.cpp:2825-2849`) makes five raw `d.b.Alloc` calls **and calls `d.b.Synchronize(d.q)` at `:2846`** on an `M` miss, and `EnsureCtmp` (`cuda_marlin_dense.cu:74-89`) grows with `cudaMallocAsync` at `:85` — both with no `cudaStreamIsCapturing` refusal, unlike the six sibling shape-keyed caches that have one. Neither is #2029's cause: both are keyed by a quantity the cold step already visits at the same shape. - **#2037** — the fatal handler prints `e.what()` and no backtrace while `core_client.h:63` promises "See stack trace (above)". #1380 closed only because somebody instrumented `CudaBackend::Alloc` by hand; #2028 and #2029 both record the gap; this change was located by reading the tree rather than by reading the failure. - **#2059** — `sanitize-cpu` sets `VT_POOL_BYPASS=1` on **both** lanes, so the `DevicePool` free list, size-class ladder, best-fit borrow (#1922) and capture pre-grow (#1380) are unexecuted under ASan *and* TSan. The stated justification is ASan's `detect_leaks`; ThreadSanitizer has no leak detector and gains nothing, while losing the one allocator whose `std::mutex` and shared maps that lane exists to inspect. Measured above: with the pool enabled the `thread` lane is 10/10 and silent. Out of scope and deliberately untouched: #2028 (the illegal memory access with speculation ON), #2007 (two pools), and the `P == 1` capture gate at `qwen3_dflash.cpp:1577`. ## Records `.agents/specs/cudagraph-pregrow-nonspec.md` is committed **before** the implementation, and `.agents/issue-index.md` gains four appended rows (#2029, #2035, #2036, #2037). `origin/main` moved three times while this was in flight; the last merge brought #1994 (`b758127ec`), #2000 (`61ba99ffd`) and #1997 (`2c27f0d57`). **The `.agents/issue-index.md` union is verified by row-ID SET DIFFERENCE rather than by a clean automerge**, because GitHub ignores this repository's union driver and two relocations can automerge into a duplicate: 750 rows before, **755** after, **+5 added and 0 removed**, every key `origin/main` holds still present, the only keys not on `origin/main` being this branch's own five, and **755 unique numeric keys for 755 rows** — no duplicate. One of those rows is then EDITED, which the file's own rule forbids, and the commit that does it argues for the exception rather than hiding it: `check-agent-record.py` refused a hyphen where the schema wants an em dash, the row had never left this branch so no other branch can hold a copy to merge against, and the net diff against `origin/main` is still a pure append. An edit to a row that exists on `main` is the case the rule is about, and this is not it. ## Re-merge onto `main` (`dde045419`) Re-merged after #2010 landed. Head is `8defcf5c5`. **CODE-TOUCHING, so this head takes a fresh CI cycle:** 108 files differ from the CI-green head `625b11b83`, 58 of them compiled sources, and one is this row's own `src/vllm/model_executor/models/qwen3_5.cpp` — #2019 (QWEN4-EXP W6a) edits it, so the merged translation unit is not the one CI built (green blob `b8b99b7ef`, merged blob `d86206193`). **The #2010 interaction is now real and is gated rather than assumed.** #2010 landed as `dde045419`, so its `runner.h`, `runner.cpp` and `test_dflash2_concurrency.cpp` are in this tree for the first time. No file is touched by both rows, so there is no textual interaction; the semantic one is covered by running both surfaces on the merged tree — `test_runner` 22/22 (567 assertions), `test_dflash2_concurrency` 2/2 (21), `test_dflash2_runner_reach` 8/8 (144), all rc=0. The merge took no conflict because the edits are disjoint against base `2c27f0d57`: #2019 at `qwen3_5.cpp` 1194-9202, this row at 10848-11409. Verified in both directions, each side's per-file delta byte-identical to its own source diff at shifted offsets. The spec, the seam harness and the seam test are blob-identical to `625b11b83`. Brace balance 1543 base / 1543 ours / 1542 theirs, expected 1542, merged 1542 for `{` and `}` — though the authority there is the compiler, and `cmake --build` returns 0 with no warnings. Index re-checked against the current base: 750 + 5 ours (#2029, #2035, #2036, #2037, #2059) + 16 theirs = 771 expected, 771 actual, **0 lost / 0 invented / 0 duplicated**. `tests/CMakeLists.txt` is not in this row's change, so no rebuild of it is owed. The `VT_POOL_BYPASS=1` lane this row already repaired still behaves as designed on the merged tree: the seam test is 10 cases / 156 assertions pooled, and 8 cases with 2 skipped / 138 assertions under the flag. The skip is scoped and the remaining assertions still run, so the green is not bought by weakening a check. Local gates: `test_qwen3_5_decode_graph_seam` 10/10, `test_device_pool` 11/11, `test_breakable_graph` 30/30, `agent-preflight.sh --staged` green over 109 gates. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

localai-bot force-pushed the row/SERVE-CHAT-ROLE-FRAME-ORDER branch 4 times, most recently from c21c968 to 801825b Compare August 26, 2026 16:38

localai-bot force-pushed the row/SERVE-CHAT-ROLE-FRAME-ORDER branch from 801825b to 5b5523b Compare August 26, 2026 16:40

mudler added 2 commits August 26, 2026 20:09

localai-bot force-pushed the row/SERVE-CHAT-ROLE-FRAME-ORDER branch from facd01f to 0ee83eb Compare August 26, 2026 20:09

localai-bot merged commit b758127 into main Aug 27, 2026
25 of 27 checks passed

localai-bot mentioned this pull request Aug 27, 2026

feat(SAMPLE-CORE): the checkpoint's sampling defaults, and the Gumbel draw that scanned 248,320 tokens on one thread (#1984, #1985) #1997

Merged

localai-bot mentioned this pull request Aug 27, 2026

fix(SPEC-DFLASH2): key the draft context by request id, so a condensed batch row cannot take it from its owner (#2008) #2010

Merged

localai-bot mentioned this pull request Aug 27, 2026

DFlash2 leaves the paged graph fast path at every c>1 and re-attends the whole batch context every decode step, which is the c=4 -> c=8 scaling stall #2087

Closed

Navigation Menu

fix(SERVE-STREAM-USAGE): the chat role frame waits for the first engine result, so TTFT stops measuring an empty frame (#1982) - #1994

fix(SERVE-STREAM-USAGE): the chat role frame waits for the first engine result, so TTFT stops measuring an empty frame (#1982)#1994
localai-bot merged 4 commits into
mainfrom
row/SERVE-CHAT-ROLE-FRAME-ORDER

localai-bot commented Aug 26, 2026 •

edited

Loading

Uh oh!

localai-bot commented Aug 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

localai-bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What it costs a real client

Tests

Interaction with #1999

Not fixed here

Uh oh!

localai-bot commented Aug 26, 2026

CI lane triage: both reds are pre-existing and named

sanitize-cpu (thread) — #1862, filed 2026-08-24, two days before this branch

windows-msvc-cpu / windows-msvc-vulkan — #1979

Not owed by this row

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

localai-bot commented Aug 26, 2026 •

edited

Loading

`sanitize-cpu (thread)` — #1862, filed 2026-08-24, two days before this branch

`windows-msvc-cpu` / `windows-msvc-vulkan` — #1979