fix(ENG-CUDAGRAPH-BREAK): the capture pre-grow is gated on a flag the default server never sets, so a non-speculative capture cudaMallocs (#2029)

.agents/issue-index.md

@@ -766,6 +766,10 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#2015](https://github.com/mudler/vllm.cpp/issues/2015) | `ENG-HYBRID-PLACEMENT` | Map llama.cpp's four placement surfaces (`-ot` general, `-cmoe` and `-ncmoe N` as sugar over it, `--fit` auto) onto the `vllm_cpp` extension key of `--offload-config` instead of a bespoke flag, per developer direction 2026-08-26, and re-anchor the spec at the recorded pin `b10451` — `237ad9b96` was superseded on 2026-08-22 and none of its cited line numbers resolve | record |
 | [#2005](https://github.com/mudler/vllm.cpp/issues/2005) | `BACKEND-TENSTORRENT` | **No Tenstorrent sibling of `gpu_clock_state.py` existed, so every TT speed figure on record was clock-unattributed — including #2003's inversion, which could have been a clock excursion.** Landed `tools/bench/tt_clock_state.py` (+`tt_refold_busy.py`): AICLK via 430 ms `tt-smi -s` snapshots at 1 Hz, NVIDIA thresholds copied constant-for-constant (spread 5%, cross-arm median AND mean 1%, 30 busy samples, majority-busy), stop-only summaries (#1657 shape), NOT-APPLICABLE fields stated (no persistence/applications knobs; no live throttle bitmap → carried caveat), claimed-max as provenance-carrying argument, busy proxied by the leg pid holding `/dev/tenstorrent` fds checked live per interval, machine identity compared unconditionally even under a boot waiver. First wired use re-measured #2003 same-day: the P150 AICLK governor proved TWO-STATE (800 idle / pegged cap under load); raw windows refuse within-run spread 40.74% by construction of that platform, and the live-recorded busy flags enable a criterion-independent busy-slice refold in which **all six windows carry exactly one distinct value {1350}, spread 0.00%, judge PASS** — the inverted ratio is clock-attributed parity. Owed: verified claimed-max pin (1350 is class folklore today), in-process pyluwen sampler for sub-second cadence, and a policy decision on whether two-state governors deserve a spread rule scoped to busy slices inside the tool itself rather than the offline refold script | perf |
 | [#2032](https://github.com/mudler/vllm.cpp/issues/2032) | `ENG-HYBRID-PLACEMENT` | W3a: `MoePlacementPlan`, resolving a name-keyed `DevicePlacement` to the per-layer decision the forward can actually read, against llama.cpp's own GGUF tensor spelling — and REFUSING a partial placement, where a layer's gate, up and down do not share a device, because the MoE block runs one grouped GEMM over the three | feature |
+| [#2029](https://github.com/mudler/vllm.cpp/issues/2029) | `ENG-CUDAGRAPH-BREAK` | With DFlash2 speculation OFF the engine dies at c=8 in CUDA graph capture: `cudaMalloc: operation not permitted when stream is capturing`. Located statically: `Pool(b).PreGrowForCapture(b, s.demand)` — the #1380 capture pre-grow — sits INSIDE `if (dbuf)` in both Qwen3.5 decode-graph drivers (`qwen3_5.cpp:10885/10907`, `:11439/11461`), and `dbuf = impl_->dbuf \|\| spec_step` is false on the DEFAULT server, where `VT_ASYNC_EXECUTOR` is unset and no step is speculative. So the pre-grow is exactly the "path taken only when speculation is off" the issue names, by its absence. Spec [cudagraph-pregrow-nonspec.md](specs/cudagraph-pregrow-nonspec.md) | bug |
+| [#2035](https://github.com/mudler/vllm.cpp/issues/2035) | `ENG-CUDAGRAPH-BREAK` | Seven decode-graph drivers (`qwen3.cpp`, `qwen3_moe.cpp`, `deepseek_v2.cpp`, `deepseek_v4.cpp`, `voxtral.cpp`, `laguna.cpp`, `qwen3_dflash.cpp`) open a `vt::GraphCaptureScope` with no `DevicePool::PreGrowForCapture` and no demand profile at all — only `qwen3_5.cpp` uses any of the #1380 machinery. Found while fixing #2029; owed under `## Owed` in [cudagraph-pregrow-nonspec.md](specs/cudagraph-pregrow-nonspec.md) | bug |
+| [#2036](https://github.com/mudler/vllm.cpp/issues/2036) | `ENG-CUDAGRAPH-BREAK` | `DenseAlignFor` (`qwen3_5.cpp:2825-2849`) allocates five blocks and calls `d.b.Synchronize(d.q)` at `:2846` on an `M` miss, and `EnsureCtmp` (`cuda_marlin_dense.cu:74-89`) grows with `cudaMallocAsync` at `:85`, both with no `cudaStreamIsCapturing` refusal — unlike the six sibling shape-keyed caches that have one. Latent today (the cold step visits the same key), found while fixing #2029; owed under `## Owed` in [cudagraph-pregrow-nonspec.md](specs/cudagraph-pregrow-nonspec.md) | bug |
+| [#2037](https://github.com/mudler/vllm.cpp/issues/2037) | — | `EngineDeadError` promises "See stack trace (above)" (`include/vllm/v1/engine/core_client.h:63`) and the fatal handler prints only `e.what()` (`src/vllm/v1/engine/core_client.cpp:36-38`), so no trace is ever emitted. #1380 closed only because somebody instrumented `CudaBackend::Alloc` by hand; #2028 and #2029 both record the gap. Owed under `## Owed` in [cudagraph-pregrow-nonspec.md](specs/cudagraph-pregrow-nonspec.md) | bug |
 | [#2026](https://github.com/mudler/vllm.cpp/issues/2026) | `ENG-HYBRID-PLACEMENT` | W3b: route a placed layer's routed-expert compute to the placement device, with the activation round trip at the boundary — and refuse the fp4-resident arm, whose device residents are built eagerly at load so placing it would upload every expert and then compute across the bus, a defect a token gate cannot see | feature |
 | [#2040](https://github.com/mudler/vllm.cpp/issues/2040) | `ENG-HYBRID-PLACEMENT` | W3c: replace W3b's hand-written per-model round trip with ONE shared seam every architecture routes through, exploiting that every MoE block already has the same `(Dev, weights, params, [T,H] dh, T)` shape — and correct `docs/ENVIRONMENT.md` and `docs/FEATURES.md`, which W3b made false by leaving them saying the knobs place nothing | bug |
 | [#2049](https://github.com/mudler/vllm.cpp/issues/2049) | `MODEL-MM-QWEN4-EXP` | **The row spec's settled-decisions list stated the REFUTED QSA mapping**, telling a fresh W4/W5 implementer to build QSA on MiniMax-M3 and calling the DeepSeek-V4 lane "the wrong port" — the exact reverse of the correction recorded in the same file's Port map and Design section, in the matrix row, and in [#1978](https://github.com/mudler/vllm.cpp/issues/1978). Pre-existing on `main`; found while reviewing the W6a merge ([#2019](https://github.com/mudler/vllm.cpp/pull/2019)) and fixed in that same flow per AGENTS.md "Every change starts from an issue". Load-bearing rather than cosmetic: item 2 sits in the section written so an implementer does NOT re-derive it, and a top-down reader hits the stale instruction before the corrected Design section. | doc |
@@ -774,6 +778,7 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1984](https://github.com/mudler/vllm.cpp/issues/1984) | `SAMPLE-CORE` | `RandomSampleKernel` is launched `<<<n, 1>>>` and scans a 248,320-wide vocab on one thread per row, computing two `SplitMix64` rounds and an f64 `log` per element. Eleven lines above it the same file records that a single-block single-thread scan of a ~151k vocab cost ~7.5 ms/token, which is why greedy argmax was rewritten into `ArgmaxPartialKernel`/`ArgmaxFinalKernel`; the Gumbel draw never got that treatment. Upstream is whole-tensor (`vllm/v1/sample/ops/topk_topp_sampler.py::sample_with_exponential_noise`), so this is a mirror obligation. Reached by every non-greedy row through `ModelRunner::execute_model` -> `Sampler::forward` -> `vt::RandomSample`. Spec: [sample-gen-config-and-parallel-gumbel.md](specs/sample-gen-config-and-parallel-gumbel.md) | perf |
 | [#1985](https://github.com/mudler/vllm.cpp/issues/1985) | `SAMPLE-CORE` | `generation_config.json` is read for `eos_token_id` only (`hf_config.cpp::ReadGenerationConfigEosIds`), so `Qwen/Qwen3.8-27B`'s shipped `top_k: 20` / `top_p: 0.95` never reach `SamplingParams` and `to_sampling_params` resolves omitted knobs straight to the neutral OpenAI defaults, which disable both filters. vLLM applies them through `ModelConfig.get_diff_sampling_param` -> `OpenAIServing*.default_sampling_params` -> `to_sampling_params`. Since `vllm bench serve` stopped sending `--temperature`, both engines sample at temperature 1.0 and vLLM draws from 20 candidates while we draw from 248,320: different sampling on two sides of a parity benchmark. Spec: [sample-gen-config-and-parallel-gumbel.md](specs/sample-gen-config-and-parallel-gumbel.md) | bug |
 | [#2002](https://github.com/mudler/vllm.cpp/issues/2002) | `SAMPLE-CORE` | With `--speculative-config` set, `GPUModelRunner::sample_tokens` branches on `num_draft_tokens > 0` alone and returns the greedy-only `RejectionSampler`'s output, so `Sampler::forward` and `vt::RandomSample` are never called and a `temperature: 1.0` request decodes GREEDILY. `include/vllm/v1/spec_decode/rejection_sampler.h` states the contract it violates in its own deferral list ("a temperature > 0 request must NOT be routed here yet"); neither the runner nor `RejectionSampler::forward` enforces it. Found while writing #1984's acceptance measurement against a baseline recipe carrying `--speculative-config`, where the sampler under test would never have been launched and the null result would have read as "the change did nothing" | bug |
+| [#2059](https://github.com/mudler/vllm.cpp/issues/2059) | — | `.github/workflows/ci.yml:1598` sets `VT_POOL_BYPASS: "1"` for BOTH `sanitize-cpu` lanes, so the `DevicePool` free list, size-class ladder, best-fit borrow (#1922) and capture pre-grow (#1380) are unexecuted under ASan AND TSan. The stated justification is ASan's `detect_leaks`; ThreadSanitizer has no leak detector and gains nothing. MEASURED: under `-DVLLM_CPP_SANITIZE=thread` with the pool ENABLED, `test_qwen3_5_decode_graph_seam` is 10/10, 156 assertions, exit 0, zero TSan warnings. Found while repairing the #2047 red; owed under `## Owed` in [cudagraph-pregrow-nonspec.md](specs/cudagraph-pregrow-nonspec.md) | bug |
 | [#2055](https://github.com/mudler/vllm.cpp/issues/2055) | `ENG-UPSTREAM-LTX2-PIN` | `tools/oracle/ltx2_oracle.py` asserts the LTX-2 oracle's revision and its resolved `ltx_core` / `ltx_pipelines` origins in the PARENT, then renders in a child started with `python -m`, which puts the CURRENT WORKING DIRECTORY on that child's `sys.path[0]`. The parent's `importlib.util.find_spec` never consults it, so a directory holding a decoy `ltx_pipelines`, made the CWD, is imported by the process that loads the weights while the process that checked identity sees nothing. MEASURED by a fresh reviewer of [#2053](https://github.com/mudler/vllm.cpp/pull/2053) on a scratch copy: the parent printed `IDENTITY_OK` naming the pinned tree and the child then printed `DECOY ti2vid_one_stage RUNNING -- the parent's assert never saw me`. This is the `.agents/specs/ltx-2-5.md` §7.0(b) decoy failure surviving in the one process that touches weights. The 2026-08-27 reference render is NOT affected and its `gateable = yes` stands: `render.sh` issues no `cd`, `/workspace/ltx2-oracle/` holds no `ltx_*` package, and the committed manifest records module origins inside the pinned clone — so the defect is an OVER-CLAIMED guarantee, not a wrong measurement. Fix: `-P` (or `PYTHONSAFEPATH=1`) on the child plus an explicit `cwd=`, with the reviewer's decoy as the red-first test. NOT fixed in flow and deliberately: the script's sha256 equals the one the worker executed and printed, and that equality is the provenance chain `.agents/oracles/ltx-2.md`'s `gateable = yes` rests on, so editing the file for a hardening that changed no result would trade a verifiable fact for a better comment. Owned by `ENG-UPSTREAM-LTX2-PIN` and listed under `## Owed` in [oracle-ltx-2-pin.md](specs/oracle-ltx-2-pin.md) | bug |
 | [#2056](https://github.com/mudler/vllm.cpp/issues/2056) | `MODEL-MM-QWEN4-EXP` | **`check-agent-record.py` accepts TWO claim files owning the same matrix row**, so a claim collision merges clean and silent. Measured on this branch: copying W6a's `CLAIM-MODEL-MM-QWEN4-EXP.md` beside W1's `CLAIM-MODEL-MM-QWEN4-EXP-W1.md` gives `agent record OK`, rc=0, with both files asserting ownership of `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` and both marked `ACTIVE`. Git cannot conflict on it because the two sides touch different PATHS. The matrix owner cell holds exactly ONE value, so the record goes silently ambiguous. Resolved here by merge ORDER, which is an operator remembering rather than a gate. NOT fixed in flow: it changes checker semantics and owes its own row, spec and red-before test per AGENTS.md. Listed under `## Owed` in [qwen4-exp-flash-next.md](specs/qwen4-exp-flash-next.md). | bug |
 | [#1973](https://github.com/mudler/vllm.cpp/issues/1973) | `KV-DSV4-MULTICACHE` | **`MakeDeepseekV4KVCache` publishes ONE placeholder group for a model that needs 167 caches, and the runner would drop the real topology in silence.** W2 of [#1925](https://github.com/mudler/vllm.cpp/issues/1925), after W1 ([#1960](https://github.com/mudler/vllm.cpp/issues/1960)) landed the allocation metadata and published none of it. `deepseek_v4_registry.cpp:126-148` emits one `"mla"` group at `head_size = head_dim + qk_rope_head_dim = 576` and calls itself a STUB; upstream at the pin `5559679229bc961848b121ccdeaa8fa5d79bec98` publishes **167 entries in seven groups** across DeepSeek-V4-Flash's 43 layers -- 21 C4A + 20 C128A compressed latents and 21 indexer key caches as `MLAAttentionSpec` (`vllm/models/deepseek_v4/attention.py:631-645`, `:669-684`), 43 SWA caches (`vllm/v1/attention/backends/mla/sparse_swa.py:86-101`) and 21 + 21 + 20 compressor states (`vllm/models/deepseek_v4/compressor.py:188-200`) as `SlidingWindowMLASpec`, at block sizes 256, 64, 4 and 8. **The factory is REACHED from production** -- it is `ModelFactory::make_kv_cache` (`:120`), called from `LoadedEngine`'s constructor through `MakeKVCacheResolved`/`MakeKVCacheMaybeSpec` (`src/vllm/entrypoints/model_loader.cpp:1394-1404`, `:1681`) -- so what it publishes reaches `GPUModelRunner::initialize_kv_cache` unfiltered, and that loop (`src/vllm/v1/worker/gpu/runner.cpp:577-597`) has two arms and no `else`: `kSlidingWindowMla` matches nothing and a SECOND `kMlaAttention` group is passed over by the `full_attn_group_id_ < 0` guard, with no diagnostic either way. **Worse than one dropped group, and this is what W1 could not see from the selection loop alone:** `membership_by_name` is set only inside `if (has_mamba_group)` (`runner.cpp:820-845`), DeepSeek-V4 has no Mamba group, so the allocation loop falls into `is_full_attn = !is_gdn` and allocates ONE buffer per HIDDEN LAYER sized from the target group's page -- 43 buffers of one page for a model needing 167 of seven. A silently short KV allocation is a wrong-tokens failure, not a crash. FIXED by publishing the seven groups WITH a `VT_CHECK` refusal naming any published group the runner does not allocate, which is `AGENTS.md`'s "refuse an unimplemented arm with a message that names the missing part" rather than a new policy. **Consequence stated rather than implied: DeepSeek-V4 on the server path now refuses at engine construction instead of running.** What it loses is a full-recompute path with no decode step -- `Forward`/`ForwardDevice` discard `attn_kv` (`src/vllm/model_executor/models/deepseek_v4.cpp:2886-2887`, `:2959-2960`) -- and `examples/deepseek_v4_gen` does not go through the runner and is unaffected. Two further refusals are NOT in scope and are named so nobody takes them for this: `HybridKVCacheCoordinator`'s uniform-block-size `assert`, inert under `NDEBUG` (`src/vllm/v1/core/kv_cache_coordinator.cpp:340-346`, W4), and the missing third `ModelForwardInput` channel (`include/vllm/model_executor/models/model_registry.h:303-315`, W3). Nothing published is consumed; reachability is owed to W3 and W5 under `## Owed` in [kv-dsv4-multicache.md](specs/kv-dsv4-multicache.md) | bug |