fix(ENG-CUDAGRAPH-BREAK): the capture pre-grow is gated on a flag the default server never sets, so a non-speculative capture cudaMallocs (#2029)

[localai-bot]

With DFlash2 speculation off the engine could not warm up at concurrency 8. It died inside CUDA graph capture, and every request after that came back 500 [request submitted to a stopped AsyncLLM].

engine-fatal: EngineCore busy loop threw: vt cuda: cudaMalloc: operation not permitted when stream is capturing

That is the configuration a user gets by omitting --speculative-config, so this is the default server and not a benchmark artefact.

Closes #2029.

The asymmetry is not in the speculative code

#2029 records that the same binary WITH --speculative-config warms and serves c=1/2/4 cleanly, and concludes that "the failing allocation is on a path taken only when speculation is off". That is exactly true, and the path is the absence of the #1380 capture pre-grow.

Both Qwen3.5 decode-graph drivers compute

const bool dbuf = impl_->dbuf || spec_step;   // qwen3_5.cpp:10764, :11315

with impl_->dbuf = enabled && DecodeGraphDoubleBufferEnabled() (:10529, :11082), DecodeGraphDoubleBufferEnabled() false unless VT_ASYNC_EXECUTOR=1 (:10302-10308), and spec_step = gdn_meta.num_spec_decodes > 0 (:10715, :11267). And Pool(b).PreGrowForCapture(b, s.demand) sat inside if (dbuf) — :10885/:10907 in the 35B MoE driver, :11439/:11461 in the 27B dense one, which is the driver the #1574 subject runs on.

So on the default server neither disjunct holds, the pre-grow never executed, and the driver opened vt::GraphCaptureScope over a DevicePool nobody had prepared. A Get miss inside that region is DevicePool::Get's b.Alloc(key) (device_pool.h:212), which on CUDA is the cudaMalloc in the message. #1380 named the same message from the same driver.

s.demand and Pool(b).MarkStepBoundary() were already recorded unconditionally (:11607/:11251, :11046/:10697). Only the consumer was gated, and the guard is an accident of history rather than a decision: that block exists for the VT_ASYNC_EXECUTOR parity ring's drain and its persistent step inputs, and #1393 replaced the single-block pre-grow in place without revisiting what it sat under.

The !dbuf arm needs it at least as much as the dbuf arm. It passes persistent_sdi == nullptr (:11531, :10974), so BuildStepDevInputs and MaybeBuildAttnCosSin run from the main pool INSIDE the captured region. That is also what makes s.demand exact for it: the cold step takes the identical nullptr branch, so its per-class peak IS the capture's demand. On the dbuf arm the captured region's main-pool demand is a strict subset of the cold step's, which is the containment #1393 recorded.

The change

One statement moves in each driver, out of if (dbuf) and in front of it. Everything else stays: b.Synchronize drains an in-flight replay only the parity ring can leave behind, and the StepDevInputs / s.pin construction is the persistent-input path the ring and the spec capture need.

The pre-grow now precedes that b.Synchronize rather than following it. A pre-grow is Backend::Alloc and nothing else — it enqueues no work and reads no in-flight buffer — and the drain still happens before BeginCapture, which is the property it was added for.

Prefix caching drops the same fault to c=1, and this finding predicts it

Measured by the operator on dgx:gpu0 while this was in flight: same 3d895a202 binary, speculation off, plus --enable-prefix-caching --scheduling-policy lpm, concurrency 1 — the identical message, illegal memory access 0, position discontinuity 0, ok=0 failed=8.

Nothing in this change was ever scoped to batch 8. The pre-grow is skipped on EVERY non-speculative step at EVERY padded size, and this pull request's own gate is a single-request S = 1 case. What the two configurations differ in is only whether the free list happens to be short when the capture opens.

The rival reading — that some buffer is sized from a per-step quantity — was checked against the code and does not hold on this driver:

Candidate	Why not
block_table_num_cols / max_blocks	fixed once by BlockTable's constructor from max_model_len (block_table.cpp:49), read unchanged by gather_block_table (runner.cpp:1185-1187). It cannot move between two steps, and prefix caching does not touch it
FA-2 decode scratch	DecodeShapeKey is {batch, hq, heads, groups, head_dim, max_blocks, page_size, num_splits} (cuda_flash_attn_fa2.cu:1013), all key-determined or constant — and it carries its OWN capture refusal at :1021-1029 with a different message, so it cannot produce this one
the forward's knowledge of prefix caching	it has none; enable_prefix_caching appears nowhere in the runner's forward path

Which size class is short in either run is NOT determined here, and cannot be from a CPU box. The fix does not depend on the answer — an unprepared pool is unprepared whatever empties it — but the issue stays open until a device says so. See What is not verified below.

Not the same fault as #2042 (prefix caching + DFlash2 at c=1, the draft's position invariant). With speculation ON, prefix caching fails through #2042; with speculation OFF, through this.

The gate

Two cases in tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp, one per driver, on the existing CPU seam in decode_graph_seam_harness.h.

The assertion is the guarantee, not the call: zero Backend::Alloc between BeginCapture and EndCaptureGraph, on a step where num_spec_decodes == 0 and VT_ASYNC_EXECUTOR is unset. CHECK(the pre-grow was called) would be a transcription — it stays green when the demand profile is wrong, which is the half #1393's own body recorded as ungated: "the fix rests on the captured forward demanding no more blocks of any size class than the eager forward at that shape did ... no test asserts it." CaptureCapableCpuBackend gains the capture-window split for it: one counter, incremented under a flag the existing BeginCapture / EndCaptureGraph overrides set.

The pool is drained between the cold step and the capture step, and that is the load-bearing part of the construction. In production the free list is SHORT rather than empty — every captured SizeSlot retains its [S, vocab] logits and [S, H] hidden forever, DevicePool is keyed by SIZE CLASS, and a ramping server captures more shapes. Reproducing that by arithmetic would make the case depend on whether two of the tiny model's tensors happen to collide in a class, i.e. on a coincidence rather than on the rule. Drain is the same condition taken to its limit, it is a production API called at phase changes, and it is what also settles the premise above: an empty free list can only be served by the pre-grow, so green says the cold step's profile COVERS the capture.

CHECK(allocs() > 0) is the non-vacuity guard. The pre-grow is itself a driver allocation made outside the region, so a fixed driver must allocate in this step and must allocate none of it under capture.

The sanitize-cpu (thread) red, and a correction

That job refused the two new cases on the PRECONDITION, not on the guarantee:

tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp:948:
  FATAL ERROR: REQUIRE( freed > 0 ) is NOT correct!  values: REQUIRE( 0 > 0 )

It is not ThreadSanitizer and not its allocator. .github/workflows/ci.yml:1598 sets VT_POOL_BYPASS: "1" for that job, on both the address,undefined and the thread lane. The identical REQUIRE( 0 > 0 ) reproduces at the same two lines on an ordinary non-sanitized Release build with that one variable set and nothing else changed. (The address,undefined lane read pending, not green — it had not finished, so it never contradicted this.)

Under bypass every Get is a raw Backend::Alloc and every Put a real Free (device_pool.h:113-127, :246-252), so there is no free list, Drain reports 0, and PreGrowForCapture returns before it grows anything (:429). The guarantee these cases assert is false by design there, and false identically for the fixed and the unfixed driver.

So both cases now carry doctest::skip(PoolBypassLane()), with the reason in the case NAME so a reader who sees the skip count can recover it, and the predicate mirroring DevicePool::Bypass() byte for byte rather than approximating it. REQUIRE(freed > 0) is unchanged — M3 measured that it is what stops the pooled lane from asserting nothing.

A correction, because the first reading of this failure was backwards. The guard did not save a vacuous pass. Removed, the cases reach the capture assertion and FAIL it, at 107 and 195 driver allocations inside the capture, because under bypass every Get is a driver call. What the guard buys is an inevitable failure that names the PRECONDITION instead of the SYMPTOM. Worth having, and a smaller claim than the one first made for it.

The guarantee is therefore not exercised by sanitize-cpu — and that is the job's configuration, not a limit of the instrument. Measured on a real TSan build, -DVLLM_CPP_SANITIZE=thread, setarch -R:

Environment	test_qwen3_5_decode_graph_seam
VT_POOL_BYPASS=1, as CI sets it	exit 0, 8 passed / 2 skipped, 138 assertions, 0 TSan warnings
bypass unset, the pool ENABLED	exit 0, 10 passed / 0 skipped, 156 assertions, 0 driver allocations inside both captures, 0 TSan warnings

ThreadSanitizer runs this guarantee green when the pool is on. Nothing in the test file can opt one case back in — Bypass() is read once into a process-wide function-local static (device_pool.h:480-486), so no scope, no locally constructed DevicePool and no ActivePoolScope reaches it — and unsetting the variable for these cases would be worse than the gap, because the pool would then retain blocks that the job's own ASAN_OPTIONS=detect_leaks=1 reports as leaks. Filed as #2059.

Evidence

mudler-ubuntu-box, x86-64, CPU-only. Release, -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_TRITON=OFF -DVLLM_CPP_SERVER=OFF, Ninja, -j 4.

Reading	Result
RED, both new cases, before the fix	exit 1, 2 cases / 0 passed / 2 failed. 45 driver allocations inside the dense capture, 42 inside the MoE one
GREEN, both new cases	exit 0, 0 and 0 inside the capture, 49 in the step — all of them the pre-grow, outside the region
test_qwen3_5_decode_graph_seam, whole file, three runs	exit 0 each, 10 cases / 156 assertions
test_qwen3_decode_graph_seam	exit 0, 4 cases / 231 assertions
test_qwen3_moe_decode_graph_seam	exit 0, 3 / 228
test_deepseek_v2_decode_graph_seam	exit 0, 3 / 230
test_voxtral_decode_graph_seam	exit 0, 3 / 230
test_qwen3_dflash_decode_graph_seam	exit 0, 4 / 23
test_qwen3_dflash2_draft	exit 0, 43 / 449
test_moe_async_device_ids	exit 0, 6 / 191
test_device_pool	exit 0, 11 / 59
test_breakable_graph	exit 0, 30 / 265
test_persistent_step_input	exit 0, 10 / 66
test_qwen35_paged_forward	exit 0, 7 / 63
test_qwen3_5_gdn_spec_routing	exit 0, 6 / 52
test_qwen3_5_fa2_class	exit 0, 6 / 15
scripts/agent-preflight.sh	exit 0, every gate green, none skipped
bypass lane, VT_POOL_BYPASS=1, non-sanitized	exit 0, 8 passed / 2 skipped, 138 assertions
TSan build, CI environment	exit 0, 8 passed / 2 skipped, 0 TSan warnings
TSan build, pool ENABLED	exit 0, 10 passed / 0 skipped, 156 assertions, 0 TSan warnings

Every row above was re-measured on the merged tree after origin/main moved to 2c27f0d57 (#1994, #2000, #1997), because merge-tree clean is not merge-tree builds and that merge touched qwen3_5.cpp from both sides.

The four test_qwen3_5_decode_graph_seam runs also read on #1390, which reports that file exiting 139 with a non-reproducible assertion line: four runs here, exit 0 every time, 10 cases and 156 assertions every time. That is one build on one box and it does not close #1390; it is recorded because a flaky neighbour is otherwise something a reviewer has to re-derive.

Mutations

Each compiled clean, each restored by sha256sum -c against the baseline before the next.

Mutation	Result
M1, the production call site: put PreGrowForCapture back inside if (dbuf)	RED — 10 cases / 8 passed / 2 failed, and the two failures are the new cases and nothing else
M2, the pool half: PreGrowForCapture returns 0 before it grows anything	RED — test_qwen3_5_decode_graph_seam 8/10 and test_device_pool 10/11
M3, the case's own construction: remove the Drain, on the UNFIXED driver	the capture-window assertion goes vacuously green — 0 allocations inside the capture and 0 in the whole step — and only allocs() > 0 fires

| M4, the skip predicate: PoolBypassLane() returns false, run under VT_POOL_BYPASS=1 | RED — 10 cases / 8 passed / 2 failed, 146 assertions: a byte-for-byte reproduction of the sanitize-cpu (thread) reading, on a non-sanitized build. The skip is load-bearing, not a decorative no-op |

All four were re-run after the fixture changed, because a fixture change can disarm a mutation proof. M1, M2 and M3 read identically before and after.

M3 is the one worth reading. It says the drain is load-bearing rather than decorative: without it this case cannot detect the defect at all, and without the non-vacuity guard beside it the case would have reported a pass while measuring nothing. M1 is also the reachability mutation "Nothing lands dead" asks for: deleting the production call site reds the focused gate.

What is not verified

No GPU was available to this implementer, and nothing here is reported as device-verified. The change edits no .cu, so the CUDA arm is a cuda-fat-build compile verdict and never a run verdict either way.

#2029 therefore stays open. This removes a proven, non-speculative-only gap that produces exactly its message, at every padded size including the c=1 one prefix caching exposes. It does not prove that gap was the only one.

Three experiments settle it, in this order, each on the operator's own invocation with --speculative-config removed and --enable-prefix-caching --scheduling-policy lpm added, at c=1:

1. VT_ASYNC_EXECUTOR=1 on the UNCHANGED 3d895a202 binary. That flips impl_->dbuf true and turns the pre-grow on with no rebuild and no patch. A run that SERVES proves the missing pre-grow is the cause, on the device, against the exact binary c=8 with speculation OFF dies in CUDA graph capture: cudaMalloc while the stream is capturing #2029 was measured on. A run that still dies is inconclusive rather than a refutation, and the asymmetry has to be stated when the result is read: the same flag opens the parity ring, which retains a second [S, vocab] logits and [S, H] hidden per size and can empty the free list by a route of its own.
2. The same configuration on a binary built from this branch, environment untouched. Serving says this closes that entry point; dying says a second site exists.
3. VLLM_CPP_QWEN3_DENSE_DECODE_GRAPH=0, as the control. Serving confirms the fault is inside this driver's capture; dying says the capture that allocates is elsewhere and 1 and 2 answered the wrong question.

If 2 still dies, the next instrument is a backtrace at CudaBackend::Alloc — how #1380 was located, and exactly what #2037 exists to make unnecessary. VT_POOL_STATS cannot substitute: it prints per-pool totals at destruction, after the engine is already dead, and names no class.

Found and NOT fixed, each with an issue

• Seven decode-graph drivers open a capture with no pool pre-grow and no demand profile at all #2035 — seven decode-graph drivers (qwen3.cpp, qwen3_moe.cpp, deepseek_v2.cpp, deepseek_v4.cpp, voxtral.cpp, laguna.cpp, qwen3_dflash.cpp) open a capture with no pre-grow and no demand profile at all. A grep for the A speculative decode-graph capture does a cudaMalloc inside the captured region and throws on sm_110 #1380 machinery outside device_pool.h returns qwen3_5.cpp and nothing else. Three of them argue in comments that their cold step makes the capture allocation-free, which is the reasoning A speculative decode-graph capture does a cudaMalloc inside the captured region and throws on sm_110 #1380 measured and refuted. Each needs its own MarkStepBoundary / StepDemandProfile wiring and its own gate, so it is a different change with a different blast radius.

• Two Marlin-path caches allocate — and one SYNCHRONIZES — with no CUDA-graph capture guard #2036 — DenseAlignFor (qwen3_5.cpp:2825-2849) makes five raw d.b.Alloc calls and calls d.b.Synchronize(d.q) at :2846 on an M miss, and EnsureCtmp (cuda_marlin_dense.cu:74-89) grows with cudaMallocAsync at :85 — both with no cudaStreamIsCapturing refusal, unlike the six sibling shape-keyed caches that have one. Neither is c=8 with speculation OFF dies in CUDA graph capture: cudaMalloc while the stream is capturing #2029's cause: both are keyed by a quantity the cold step already visits at the same shape.

• engine-fatal prints only what(), so the promised "See stack trace (above)" never exists #2037 — the fatal handler prints e.what() and no backtrace while core_client.h:63 promises "See stack trace (above)". A speculative decode-graph capture does a cudaMalloc inside the captured region and throws on sm_110 #1380 closed only because somebody instrumented CudaBackend::Alloc by hand; CUDA illegal memory access under sustained c=8 load at seqs=16/ctx=8192, after c=1/2/4 serve cleanly (not #1983) #2028 and c=8 with speculation OFF dies in CUDA graph capture: cudaMalloc while the stream is capturing #2029 both record the gap; this change was located by reading the tree rather than by reading the failure.

• sanitize-cpu sets VT_POOL_BYPASS=1 on BOTH lanes, so neither sanitizer ever exercises the DevicePool — and the thread lane gains nothing from it #2059 — sanitize-cpu sets VT_POOL_BYPASS=1 on both lanes, so the DevicePool free list, size-class ladder, best-fit borrow (Memory grows monotonically per request at 12288 context: three requests exhaust the box #1922) and capture pre-grow (A speculative decode-graph capture does a cudaMalloc inside the captured region and throws on sm_110 #1380) are unexecuted under ASan and TSan. The stated justification is ASan's detect_leaks; ThreadSanitizer has no leak detector and gains nothing, while losing the one allocator whose std::mutex and shared maps that lane exists to inspect. Measured above: with the pool enabled the thread lane is 10/10 and silent.

Out of scope and deliberately untouched: #2028 (the illegal memory access with speculation ON), #2007 (two pools), and the P == 1 capture gate at qwen3_dflash.cpp:1577.

Records

.agents/specs/cudagraph-pregrow-nonspec.md is committed before the implementation, and .agents/issue-index.md gains four appended rows (#2029, #2035, #2036, #2037).

origin/main moved three times while this was in flight; the last merge brought #1994 (b758127ec), #2000 (61ba99ffd) and #1997 (2c27f0d57). The .agents/issue-index.md union is verified by row-ID SET DIFFERENCE rather than by a clean automerge, because GitHub ignores this repository's union driver and two relocations can automerge into a duplicate: 750 rows before, 755 after, +5 added and 0 removed, every key origin/main holds still present, the only keys not on origin/main being this branch's own five, and 755 unique numeric keys for 755 rows — no duplicate.

One of those rows is then EDITED, which the file's own rule forbids, and the commit that does it argues for the exception rather than hiding it: check-agent-record.py refused a hyphen where the schema wants an em dash, the row had never left this branch so no other branch can hold a copy to merge against, and the net diff against origin/main is still a pure append. An edit to a row that exists on main is the case the rule is about, and this is not it.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

[localai-bot]

Device proof: your root cause is CONFIRMED on hardware, no rebuild required

Ran experiment 1 from your report on dgx:gpu0 (GB10) under rc lease 6e1589ea, on the unchanged 3d895a202 binary — artifact-gated vllm-server 101,151,128 B, flash_fwd=1792, SpecDecodeFA2Bf16=1.

Configuration, identical in both arms except one environment variable:
--num-blocks 3744 --max-num-seqs 16 --max-model-len 8192 --enable-prefix-caching --scheduling-policy lpm, no --speculative-config, c=1, vllm bench serve --dataset-name random --backend openai-chat, 1024 in / 512 out.

arm	result
unchanged binary	VOID ok=0 failed=8 — cudaMalloc: operation not permitted when stream is capturing x4
+ VT_ASYNC_EXECUTOR=1	SERVES ok=8/8, 10.82 out tok/s, TTFT mean 162.69 ms, TPOT 92.27 ms

Fault counters on the passing run: stream is capturing 0, illegal memory access 0, position discontinuity 0, engine-fatal 0. Both server: prefix caching enabled and vllm.cpp: Asynchronous scheduling is enabled (max_concurrent_batches=2) present, so the flag took effect and APC was genuinely on.

VT_ASYNC_EXECUTOR=1 flips impl_->dbuf true, which is precisely what makes the gated Pool(b).PreGrowForCapture(b, s.demand) execute. Turning that call on — with no code change at all — turns the fault off.

By your own stated asymmetry, this is the informative direction: serving proves the missing pre-grow is the device cause, whereas dying would have been inconclusive because the same flag also opens the parity ring. It served.

The throughput figure is not a performance datapoint — speculation is off and the async executor is on — it is only evidence that requests completed.

Experiments 2 (a binary built from row/ENG-CUDAGRAPH-BREAK-2029) and 3 (VLLM_CPP_QWEN3_DENSE_DECODE_GRAPH=0) were not run: my lease expires and a queued job takes the box. Neither is needed to establish the cause, though 2 remains worth running to confirm the fix closes this entry point on the device rather than only in the seam test.