From fc7be7c101337bb1bd7bc6d2d743c9641acc6e00 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 11:49:05 +0000 Subject: [PATCH 1/5] spec(KV-DSV4-MULTICACHE): W2 -- publish the seven-group topology, and refuse the group the runner drops in silence (#1973, #1974) W1 declined to publish DeepSeek-V4's cache topology because the runner drops an unrecognised group with no diagnostic. That reading is correct, and tracing it end to end makes it worse rather than better: MakeDeepseekV4KVCache is ModelFactory::make_kv_cache and is reached from LoadedEngine's constructor, and because DeepSeek-V4 has no Mamba group the allocation loop never reaches the by-name path at all -- it allocates one buffer per hidden layer sized from the first group's page. Publishing seven groups without a refusal would allocate 43 buffers of one page for a model that needs 167 of seven, and say nothing. So W2 is resequenced in content rather than in order: it publishes the topology AND makes the drop impossible to take silently, which is the polarity AGENTS.md already requires of an unimplemented arm. This section records the seven groups with the upstream site each value came from, the four details a careless port gets wrong -- the prefix is `attn` and not `self_attn`, kv_quant_mode is passed on two of the four construction sites, the indexer and compressor specs carry no model_version, and the indexer key width defaults to 132 and not 68 -- the page sizes as literals, and the byte-neutrality pins. It also states the cost: a DeepSeek-V4 engine now refuses instead of running a full recompute that has no decode step in it. #1974 is the spec_equal observation W1 named and correctly left alone unverified. It is verified here, filed, and fixed in flow with W2. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-fable-5 [Claude Code] --- .agents/issue-index.md | 2 + .agents/specs/kv-dsv4-multicache.md | 257 +++++++++++++++++++++++++++- 2 files changed, 251 insertions(+), 8 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 2902da28d..e78ad2600 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -741,3 +741,5 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1870](https://github.com/mudler/vllm.cpp/issues/1870) | `BACKEND-ROCM` | **`VT_GGUF_KEEP_QUANT=0` is documented as a same-binary opt-out and is unreachable on a 16 GiB discrete ROCm card, failing with a raw allocator throw rather than a refusal that names the cause.** `engine-fatal: EngineCore busy loop threw: vt rocm: hipMalloc: out of memory` on **both** `Qwen3.6-14B-A3B-VibeForged-v2-Q4_K_M` (7.87 GiB) and `Ornith-1.5-9B-Q4_K_M` (5.23 GiB) on `4b1154bc5`, RX 9060 XT (gfx1200, 15.92 GiB), ROCm 7.2.3, `--device auto`, with free VRAM asserted above 13 GiB and no resident model process; both run normally on the keep-quant default at 13.0-13.1 and 18.4-18.6 tok/s. The OOM itself is arithmetic and expected — a Q4_K_M expands roughly 4x to bf16, so 5.23 GiB becomes about 20 GiB and does not fit — and three things around it are the defect. (1) `docs/ENVIRONMENT.md:94` reads "`0` disables it and expands to BF16" and states **no memory precondition**, so the documented behavior is unreachable on this class of board and the document is wrong by omission. (2) `AGENTS.md` requires an unreachable arm to refuse with a message naming the missing part, and `hipMalloc: out of memory` names neither the knob, nor the expansion, nor the budget required. (3) It **removes the same-binary A/B lever** that `AGENTS.md` requires before a performance result is accepted, which is not hypothetical: it blocked the keep-quant attribution [#1863](https://github.com/mudler/vllm.cpp/issues/1863) wanted, and it is why [#1876](https://github.com/mudler/vllm.cpp/issues/1876) had to carry its own `VT_ROCM_Q8K_BLOCK` lever instead. Related gap in the same area: `kMoeGroupedGemmBf16` is unregistered on ROCm (`rocm_ops.hip` has zero occurrences, CUDA has it), so even where memory allowed the expansion the bf16 MoE arm has no provider. Split out of [#1506](https://github.com/mudler/vllm.cpp/issues/1506), whose title claim stopped being true when [#523](https://github.com/mudler/vllm.cpp/pull/523) registered `kMatmulBTQuant` on ROCm on 2026-08-21; its surviving `1.73x peak RSS` finding is this, re-measured, and on this card the penalty is no longer a ratio but a refusal to run. Filed separately rather than by re-scoping that issue, because the index is append-only and an edited row is duplicated rather than merged. A fix shape is bounded but NOT designed here: resolve the expanded residency requirement at load, compare against the device budget, and refuse by name before allocating; whether the knob should instead be ignored with a warning is a product decision this row does not settle | bug | | [#1914](https://github.com/mudler/vllm.cpp/issues/1914) | `ENG-WEIGHT-OFFLOAD` | **Four measured ROCm device facts for the weight-offload row, from a throwaway gfx1200 spike that was never merged.** The row mirrors vLLM's `cpu_offload_gb` ([#797](https://github.com/mudler/vllm.cpp/issues/797), the dense half of [#149](https://github.com/mudler/vllm.cpp/issues/149)) and its config surface has landed, but **none of it has been measured on AMD** and `specs/weight-offload-uva.md`'s scope table names no ROCm arm. Spike `5056bbf90` on `spike/rocm-523`, 2026-08-19, base `7b9e207b1`, RX 9060 XT (gfx1200, 15.92 GiB), ROCm 7.2.3, +86 lines across three files, inert with no environment variable set so the OFF arm is the unmodified upload path in the same binary. (1) **The premise works:** `Qwen3.6-35B-A3B-UD-Q4_K_S` at 19.45 GiB dies on `hipMalloc: out of memory` and, with large weights kept host-resident and handed to the kernel as a device-readable pointer, loads and generates; on the 14B, offloading 2.00 of 6.39 GiB of experts (31%) gave **byte-identical tokens** for **10.6%**. No new backend virtual was needed for the pinned arm, because on ROCm `hipHostMalloc` returns a pointer the device reads directly and `hipHostGetDevicePointer` returns the SAME value. (2) **A slab-read microbenchmark overpredicts by about 3x:** 23 GB/s idealised streaming against roughly 8 for the real GEMM, and at 47% offloaded the same 60 GEMM dispatches went 8.03 -> 31.20 ms/token, so an offload budget sized from a streaming-bandwidth number will be optimistic. (3) **The budget is NOT monotonic:** 6 GiB gives 3.21 tok/s and 7 GiB gives 7.67, because 6 leaves almost nothing for KV and allocator slack — a SMALLER budget is 2.4x slower, the cliff is reproducible, and its mechanism is **unexplained**, so a naive "offload as little as possible" policy walks into it. (4) **`hipMallocManaged` does NOT migrate on this part:** it allocates past VRAM and the device can write to it, so it looks like it works, but a paired A/B against pinned was identical (7.64/7.77 vs 7.63/7.77 tok/s) and `mem_info_gtt_used` stayed flat at 0.43 GiB while `vram_used` filled to 15.76 — consistent with the managed-memory note in `docs/ROCM.md` (`:56` on current main; the spike cited `:148` before that file was rewritten), so on discrete AMD the pinned-host path is what works and managed memory is not a shortcut to a UVA tier. `Backend::AllocManaged`/`FreeManaged` default to `nullptr` meaning "this backend has no managed allocator", so a caller falls back rather than assuming. **What the spike is NOT, and its code must not be lifted:** no `WeightOffloader`, no canonical-name targeting, no `cpu_offload_gb`, no `supports_weight_offload`; selection is raw byte size against a counter. Caveats stated rather than implied: one board, one ROCm version, one model family; the measurements sit on a base now **259 commits stale**, and both [#1402](https://github.com/mudler/vllm.cpp/pull/1402) and [#523](https://github.com/mudler/vllm.cpp/pull/523) landed afterwards and change decode cost, so the RATIOS are the durable part and the absolute tok/s figures are not; host contention was not controlled to benchmark standard; finding 3's mechanism is unexplained and findings 2 and 4 are single-board observations. Adjacent: llama.cpp's Vulkan backend loads the same 19.45 GiB file by spilling into the 31.35 GiB GTT the amdgpu driver exposes while its HIP backend refuses as we do ([#1400](https://github.com/mudler/vllm.cpp/issues/1400)), and [#1870](https://github.com/mudler/vllm.cpp/issues/1870) makes keep-quant residency load-bearing on a 16 GiB card, which changes what an offload budget competes for | record | | [#1960](https://github.com/mudler/vllm.cpp/issues/1960) | `KV-DSV4-MULTICACHE` | **`SlidingWindowMLASpec` is a declared enumerator with no struct behind it, and `MLAAttentionSpec` carries none of the four DeepSeek-V4 fields, so 105 of V4's 167 cache entries cannot be sized at all.** W1 of [#1925](https://github.com/mudler/vllm.cpp/issues/1925). `KVCacheSpecKind::kSlidingWindowMla` is declared at `include/vllm/v1/kv_cache_interface.h:89` and the port's deferral list names the class as omitted (`:46-52`); it is the spec class of the SWA cache (43 entries, `vllm/v1/attention/backends/mla/sparse_swa.py:86-101`) and of both compressor-state populations (41 + 21, `vllm/models/deepseek_v4/compressor.py:188-200`). `MLAAttentionSpec` (`kv_cache_interface.h:242-261`) adds no fields over `FullAttentionSpec` where upstream carries `cache_dtype_str`, `alignment`, `compress_ratio` and `model_version` (`vllm/v1/kv_cache_interface.py:381-388`), so the compressed latent is sized `block_size` rows per page where upstream stores `block_size // compress_ratio`, and the 584-byte `fp8_ds_mla` token (`:396-405`) throws by name instead (`src/vllm/v1/kv_cache_interface.cpp:64-71`). `_apply_alignment_padding` (`:345-351`) has no twin, so no V4 page reaches its 576B/512B alignment. Pure allocation metadata: nothing constructs either spec outside tests, because publishing before W3 would allocate a silent subset (`src/vllm/v1/worker/gpu/runner.cpp:577-597` drops an unmatched group kind with no diagnostic). | bug | +| [#1973](https://github.com/mudler/vllm.cpp/issues/1973) | `KV-DSV4-MULTICACHE` | **`MakeDeepseekV4KVCache` publishes ONE placeholder group for a model that needs 167 caches, and the runner would drop the real topology in silence.** W2 of [#1925](https://github.com/mudler/vllm.cpp/issues/1925), after W1 ([#1960](https://github.com/mudler/vllm.cpp/issues/1960)) landed the allocation metadata and published none of it. `deepseek_v4_registry.cpp:126-148` emits one `"mla"` group at `head_size = head_dim + qk_rope_head_dim = 576` and calls itself a STUB; upstream at the pin `5559679229bc961848b121ccdeaa8fa5d79bec98` publishes **167 entries in seven groups** across DeepSeek-V4-Flash's 43 layers -- 21 C4A + 20 C128A compressed latents and 21 indexer key caches as `MLAAttentionSpec` (`vllm/models/deepseek_v4/attention.py:631-645`, `:669-684`), 43 SWA caches (`vllm/v1/attention/backends/mla/sparse_swa.py:86-101`) and 21 + 21 + 20 compressor states (`vllm/models/deepseek_v4/compressor.py:188-200`) as `SlidingWindowMLASpec`, at block sizes 256, 64, 4 and 8. **The factory is REACHED from production** -- it is `ModelFactory::make_kv_cache` (`:120`), called from `LoadedEngine`'s constructor through `MakeKVCacheResolved`/`MakeKVCacheMaybeSpec` (`src/vllm/entrypoints/model_loader.cpp:1394-1404`, `:1681`) -- so what it publishes reaches `GPUModelRunner::initialize_kv_cache` unfiltered, and that loop (`src/vllm/v1/worker/gpu/runner.cpp:577-597`) has two arms and no `else`: `kSlidingWindowMla` matches nothing and a SECOND `kMlaAttention` group is passed over by the `full_attn_group_id_ < 0` guard, with no diagnostic either way. **Worse than one dropped group, and this is what W1 could not see from the selection loop alone:** `membership_by_name` is set only inside `if (has_mamba_group)` (`runner.cpp:820-845`), DeepSeek-V4 has no Mamba group, so the allocation loop falls into `is_full_attn = !is_gdn` and allocates ONE buffer per HIDDEN LAYER sized from the target group's page -- 43 buffers of one page for a model needing 167 of seven. A silently short KV allocation is a wrong-tokens failure, not a crash. FIXED by publishing the seven groups WITH a `VT_CHECK` refusal naming any published group the runner does not allocate, which is `AGENTS.md`'s "refuse an unimplemented arm with a message that names the missing part" rather than a new policy. **Consequence stated rather than implied: DeepSeek-V4 on the server path now refuses at engine construction instead of running.** What it loses is a full-recompute path with no decode step -- `Forward`/`ForwardDevice` discard `attn_kv` (`src/vllm/model_executor/models/deepseek_v4.cpp:2886-2887`, `:2959-2960`) -- and `examples/deepseek_v4_gen` does not go through the runner and is unaffected. Two further refusals are NOT in scope and are named so nobody takes them for this: `HybridKVCacheCoordinator`'s uniform-block-size `assert`, inert under `NDEBUG` (`src/vllm/v1/core/kv_cache_coordinator.cpp:340-346`, W4), and the missing third `ModelForwardInput` channel (`include/vllm/model_executor/models/model_registry.h:303-315`, W3). Nothing published is consumed; reachability is owed to W3 and W5 under `## Owed` in [kv-dsv4-multicache.md](specs/kv-dsv4-multicache.md) | bug | +| [#1974](https://github.com/mudler/vllm.cpp/issues/1974) | `KV-DSV4-MULTICACHE` | **`spec_equal`'s `default:` arm returns false for `kMlaAttention` and `kSlidingWindowMla`, so two identical MLA specs never merge into one `SpecGroup`.** Observed by W1 ([#1960](https://github.com/mudler/vllm.cpp/issues/1960)) from a read of the switch and correctly left alone unverified; verified and fixed in flow with W2 ([#1973](https://github.com/mudler/vllm.cpp/issues/1973)). `spec_equal` (`src/vllm/v1/core/kv_cache_coordinator.cpp:17-67`) is our port of the frozen-dataclass `__eq__` and carries arms for `kFullAttention`, `kSlidingWindow`, `kChunkedLocalAttention` and `kMamba` only; `HybridKVCacheCoordinator::verify_and_split_kv_cache_groups` (`:353-376`) uses it to batch groups sharing a spec. Upstream cannot answer `false` there: every spec class is `@dataclass(frozen=True, kw_only=True)` (`vllm/v1/kv_cache_interface.py:380-381`, `:610-611`), so `__eq__` is generated over all fields and two identical `MLAAttentionSpec`s are equal. **Latent rather than observed at runtime, and that is stated rather than implied:** the coordinator needs two groups, and every MLA model in the tree publishes exactly one MLA group (`deepseek_v2_registry.cpp`, `deepseek_v4_registry.cpp`, `glm4_moe_lite_registry.cpp`, `kimi_k3_registry.cpp`, `kimi_linear_registry.cpp`, `minicpm3_registry.cpp`, `dots3_note.cpp`), so the helper is never called on two of them today. It stops being latent with #1973, which publishes three `kMlaAttention` and four `kSlidingWindowMla` groups for DeepSeek-V4. The cost of a wrong `false` is a second `SpecGroup`, so `find_longest_cache_hit` runs per group instead of per distinct spec and the eagle-bit propagation (`:395-402`) is computed over a different partition than upstream's; a wrong `true` would be the dangerous direction, and this is the safe one, which is why it is a mirror divergence rather than a live miscompute. Fixed by adding both arms, comparing `FullAttentionSpec`'s / `SlidingWindowSpec`'s fields plus the four DeepSeek-V4 fields `cache_dtype_str`, `alignment`, `compress_ratio` and `model_version`; `default:` keeps returning `false` for the kinds with no ported struct | bug | diff --git a/.agents/specs/kv-dsv4-multicache.md b/.agents/specs/kv-dsv4-multicache.md index 1d5ecb324..c4e1f6b97 100644 --- a/.agents/specs/kv-dsv4-multicache.md +++ b/.agents/specs/kv-dsv4-multicache.md @@ -13,14 +13,22 @@ recorded at `include/vllm/model_executor/models/deepseek_v4.h:13`. ## Now -`ACTIVE` — W1 ([#1960](https://github.com/mudler/vllm.cpp/issues/1960)) is -claimed and its design is `### W1 design — allocation metadata`. W1 changes the -KV-cache spec hierarchy only: it adds `SlidingWindowMLASpec`, the four -DeepSeek-V4 fields on `MLAAttentionSpec`, both `storage_block_size()` overrides, -both `real_page_size_bytes` special cases and the alignment-padding helper. -**Nothing constructs either spec outside tests**, so no topology is published -and nothing is reachable from a production entry point yet; W2 owns publication -and W3 owns consumption. W2 through W7 remain proposals with no owner. +`ACTIVE` — W1 ([#1960](https://github.com/mudler/vllm.cpp/issues/1960)) landed +as `c1e6f3fb9`: the KV-cache spec hierarchy gained `SlidingWindowMLASpec`, the +four DeepSeek-V4 fields on `MLAAttentionSpec`, both `storage_block_size()` +overrides, both `real_page_size_bytes` special cases and the alignment-padding +helper, and published none of it. + +W2 ([#1973](https://github.com/mudler/vllm.cpp/issues/1973)) is claimed and its +design is `### W2 design — publishing the topology, and the refusal that makes it +safe`. It publishes DeepSeek-V4's seven groups / 167 cache entries from +`MakeDeepseekV4KVCache`, teaches `GPUModelRunner::initialize_kv_cache` to REFUSE +a published group it does not allocate rather than drop it in silence, and fixes +`spec_equal`'s missing MLA arms +([#1974](https://github.com/mudler/vllm.cpp/issues/1974)) in flow. **Nothing +consumes the published topology**, so the refusal is what a DeepSeek-V4 engine +now hits; W3 owns carrying the groups and W5 owns reading them. W3 through W7 +remain proposals with no owner. ## Scope @@ -648,6 +656,208 @@ the page formula; it does not supply the store. The refusal therefore stays and its message is sharpened to say which half now exists, and the obligation moves to **W5**, the wave that lands the read and write side. +### W2 design — publishing the topology, and the refusal that makes it safe ([#1973](https://github.com/mudler/vllm.cpp/issues/1973)) + +Committed before the implementation. Every value below was read from +`/home/mudler/_git/vllm` at the pin `5559679229bc961848b121ccdeaa8fa5d79bec98`, +and where this section and `## The geometry, derived from source` disagree, +upstream wins and the disagreement is named. + +**The blocking question this wave had to answer first.** W1 declined to publish +because `src/vllm/v1/worker/gpu/runner.cpp:577-597` drops an unrecognised group +with no diagnostic. That reading is CORRECT and the situation is worse than it +records, which is why the answer is not "publish anyway" and not "defer W2 behind +W3" but "publish, and make the drop impossible to take silently". + +Traced end to end rather than read off one loop: + +1. `MakeDeepseekV4KVCache` is `ModelFactory::make_kv_cache` + (`deepseek_v4_registry.cpp:120`), reached from a production entry point — + `LoadedEngine`'s constructor through `MakeKVCacheResolved` / + `MakeKVCacheMaybeSpec` (`src/vllm/entrypoints/model_loader.cpp:1394-1404`, + `:1681`) and `ModelRegistry::MakeKVCache` + (`src/vllm/model_executor/models/model_registry.cpp:380-386`). What it + publishes reaches `GPUModelRunner::initialize_kv_cache` unfiltered. +2. The selection loop has two arms and no `else`: the FIRST non-eagle + `kFullAttention`/`kMlaAttention` group becomes `full_attn_group_id_`, a + `kMamba` group becomes `gdn_group_id_`. A `kSlidingWindowMla` group matches + nothing. A SECOND `kMlaAttention` group is passed over by the + `full_attn_group_id_ < 0` guard. Both ids are plain `int`s + (`runner.h:571-572`). +3. **The allocation loop then does not even see the groups.** + `membership_by_name` is set only when the model has a recurrent group + (`runner.cpp:820-845` — `gdn_layer_mask` is computed inside + `if (has_mamba_group)` and `membership_by_name = gdn_layer_mask.has_value()`). + DeepSeek-V4 has no Mamba group, so the loop falls into the historical + predicate `is_full_attn = !is_gdn` and allocates ONE `CacheBuffer` per HIDDEN + LAYER, every one sized from the target group's `page_size_bytes()` + (`runner.cpp:895-995`). Publishing the seven groups below without the refusal + allocates 43 buffers of ONE page for a model that needs 167 buffers of seven + different pages — and reports nothing. + +That is a silently short KV allocation, which is a wrong-tokens failure and not +a crash. `AGENTS.md` requires an unimplemented arm to refuse with a message that +names the missing part, so the refusal is policy, not taste. + +**What the refusal costs, stated rather than implied.** DeepSeek-V4 on the +server path stops constructing an engine. Today it loads and produces correct +tokens by full recompute, because `Forward`/`ForwardDevice` discard `attn_kv` +(`deepseek_v4.cpp:2886-2887`, `:2959-2960`) — so what is lost is a path that has +no decode step and whose tok/s `### The speed question, answered` already says is +not a decode rate. `examples/deepseek_v4_gen` does not go through the runner +(`DeepseekV4ForwardGgufCached`, `main.cpp:198`) and is unaffected. Recorded here +because it is a user-visible behaviour change that a reader must be able to find +without reading the diff. + +**What lands.** + +1. **The runner refuses a published group it does not allocate.** After the + selection loop, every group is one of exactly four things: the target + attention group, the recurrent group, the single `fa_draft` draft-KV slot + (the second `kFullAttention` group, allocated at `runner.cpp:1115-1150`), or + a defect. The fourth case becomes a `VT_CHECK` naming the group index, its + spec kind and its first layer name. Byte-neutral for every model shipping + today: each publishes exactly the groups the runner consumes, which the + byte-neutrality pins below assert directly rather than infer. + + The draft slot is tolerated on its KIND rather than on `spec_on()`, mirroring + the allocation loop's own predicate (`runner.cpp:1128-1130` skips any group + whose kind is not `kFullAttention`). A `kFullAttention` group published with + speculation OFF is therefore still tolerated and still unallocated; tightening + that is W3's, and it is listed under `## Owed` rather than left to be found. + +2. **`MakeDeepseekV4KVCache` publishes the real topology.** Seven groups, 167 + entries, one group per distinct published spec — which is upstream's own + grouping rule and, for DeepSeek-V4-Flash, resolves to seven. + + **Which dtype arm, and why it is not the one our cache-dtype resolution can + reach.** `DeepseekV4Attention.use_fp8_ds_mla_layout` is + `ClassVar[bool] = True` on the base class (`attention.py:140`) and only the + FlashInfer-sparse SM120 subclass sets it `False` + (`nvidia/flashinfer_sparse.py:163`); `_resolve_dsv4_kv_cache_dtype` then + writes `cache_config.cache_dtype = "fp8_ds_mla"` back onto the cache config + and returns `torch.uint8` (`attention.py:89-119`). The default DeepSeek-V4 + cache format upstream IS `fp8_ds_mla`, so that is the arm this wave mirrors: + `cache_dtype_str = "fp8_ds_mla"`, `alignment = 576`, 1-byte storage. Our + `ParseCacheDType` refuses the string `fp8_ds_mla` by name + (`include/vllm/v1/kv_cache_dtype.h:87-90`) and our factory signature carries + no cache dtype at all, so the arm is published unconditionally rather than + selected — the `alignment = 512` non-`fp8_ds_mla` arm is NOT published, and + that omission is named under `## Owed` against W5, which owns lifting the + refusal together with the store path. + +3. **`spec_equal` gains the two MLA arms** + ([#1974](https://github.com/mudler/vllm.cpp/issues/1974)) — the observation + W1 recorded and correctly declined to act on. Verified: every upstream spec + class is `@dataclass(frozen=True, kw_only=True)` + (`kv_cache_interface.py:380-381`, `:610-611`), so `__eq__` is generated over + all fields and two identical `MLAAttentionSpec`s are equal, while + `src/vllm/v1/core/kv_cache_coordinator.cpp:17-67` returns `false` for them + from its `default:` arm. Fixed in flow. + +**The seven groups, with the upstream site each value came from.** + +| # | group | spec class | upstream site | `block_size` | `head_size` | dtype | window | layers | +|---:|---|---|---|---:|---:|---|---:|---:| +| 1 | C4A compressed latent | `MLAAttentionSpec` | `attention.py:631-645` | `cache_config.block_size` | 512 | u8 | — | 21 | +| 2 | C128A compressed latent | `MLAAttentionSpec` | same site, ratio 128 | `cache_config.block_size` | 512 | u8 | — | 20 | +| 3 | indexer key cache | `MLAAttentionSpec` | `attention.py:669-684` | `cache_config.block_size` | 132 | u8 | — | 21 | +| 4 | SWA cache | `SlidingWindowMLASpec` | `sparse_swa.py:86-101` | 64 | 512 | u8 | 128 | 43 | +| 5 | C4 attention-compressor state | `SlidingWindowMLASpec` | `compressor.py:188-200` | 4 | 2048 | f32 | 8 | 21 | +| 6 | C4 indexer-compressor state | `SlidingWindowMLASpec` | same site at `head_dim=128` (`attention.py:768-777`) | 4 | 512 | f32 | 8 | 21 | +| 7 | C128 compressor state | `SlidingWindowMLASpec` | same site at ratio 128 | 8 | 1024 | f32 | 128 | 20 | + +21 + 20 + 21 + 43 + 21 + 21 + 20 = **167**. + +**Four upstream details a careless port gets wrong, each read rather than +assumed.** + +- **The prefix is `attn`, not `self_attn`.** `DeepseekV4DecoderLayer` builds its + attention as `prefix=f"{prefix}.attn"` + (`vllm/models/deepseek_v4/nvidia/model.py:808-813`) under + `prefix=f"{prefix}.layers"` (`:1015`) and `maybe_prefix(prefix, "model")` + (`:1409`). Every published name is therefore `model.layers..attn...`, and + `LayerIndexOfName` (`runner.cpp:343-359`) resolves each to ``. This is the + same class of trap as W1's branch order: the `## The geometry, derived from + source` section of this document says `{layer}.swa_cache` without fixing what + `{layer}` is, and the answer is not the `self_attn` every other architecture + in this tree uses. +- **`kv_quant_mode` is passed on two of the four construction sites and not the + other two.** The SWA cache (`sparse_swa.py:100`) and the compressed latent + (`attention.py:644`) pass `get_kv_quant_mode(...)`, which returns + `FP8_PER_TENSOR` for any string starting with `fp8` + (`kv_cache_interface.py:70-71`). The indexer key cache (`attention.py:669-684`) + and the compressor state (`compressor.py:188-200`) pass nothing and default to + `NONE`. Mirrored exactly. This is what makes the published topology exercise + W1's branch-order guarantee on the real geometry: those two specs carry a + non-NONE quant mode AND `cache_dtype_str == "fp8_ds_mla"`, and our + `real_page_size_bytes` must reach the 584-byte branch before the quant-mode + guard throws. +- **The indexer key cache and the compressor state carry NO `model_version` and + NO `cache_dtype_str`.** Both take the element formula, not the 584-byte + branch: `64 * 1 * 132 * 1 = 8448` and `4 * 1 * 2048 * 4 = 32768`. Only + `alignment` is passed on those two sites. +- **The indexer key width is 132, not 68.** `use_fp4_kv` reads + `attention_config.use_fp4_indexer_cache`, whose default is `False` + (`vllm/config/attention.py:64`), so the FP8 branch + `head_dim + head_dim // quant_block_size * 4 = 128 + 128 // 128 * 4 = 132` + is the default (`attention.py:738`, `:751-760`). The MXFP4 68-byte arm is not + published and is named under `## Owed`. + +**The published page sizes, as literals, at `block_size = 256`.** The +configured block size is 256 on this geometry for the reason +`sparse_swa.py:76-83` and `compressor.py:174-178` both state: the C4A KV block +shape `[256//4, head_dim] = [64, head_dim]` is what fixes the SWA block size at +64 and the compressor block sizes at 4 and 8. Every number below is the value +W1's formulas produce from the constructor arguments in the table above, and the +test states them as literals so a change to any of them is a red test rather +than a silently different pool. + +| # | `storage_block_size` | `real_page_size_bytes` | `page_size_bytes` (padded) | +|---:|---:|---:|---:| +| 1 | 64 | `64*584 = 37376` | 37440 | +| 2 | 2 | `2*584 = 1168` | 1728 | +| 3 | 64 | `64*1*132*1 = 8448` | 8640 | +| 4 | 64 | `64*584 = 37376` | 37440 | +| 5 | 4 | `4*1*2048*4 = 32768` | 32832 | +| 6 | 4 | `4*1*512*4 = 8192` | 8640 | +| 7 | 8 | `8*1*1024*4 = 32768` | 32832 | + +Groups 1 and 4 reach 37440 by different routes — an `MLAAttentionSpec` at +`block_size=256, compress_ratio=4` and a `SlidingWindowMLASpec` at +`block_size=64, compress_ratio=1` — and upstream fixes the SWA block size at 64 +precisely so that they agree (`sparse_swa.py:76-83`, the shared physical +tensor). The test asserts the equality directly, not only the two values. + +**`block_size` is refused rather than mis-sized.** `storage_block_size` is +`block_size / compress_ratio`, so a `block_size` that is not a multiple of 128 +gives the C128A group a `storage_block_size` of 0 and a zero-byte page. +Upstream has no such check because its own comments derive the whole geometry at +256; ours refuses by name in `MakeDeepseekV4KVCache`, naming 128 and the two +upstream comments. That refusal is the reason a wrong engine block size is a +loud failure here rather than a zero-sized pool. + +**Byte-neutrality, proved rather than asserted.** This wave changes no spec +class and no page formula, so no existing page size can move; what could move is +the runner's behaviour for every model and `spec_equal`'s partition. Both are +pinned directly: + +- The tolerated group shapes are enumerated in a test: `{target attention}`, + `{target attention, recurrent}`, `{target attention, recurrent, fa_draft}` + all construct, and each of `kSlidingWindowMla`, `kSlidingWindow`, + `kChunkedLocalAttention` and a second `kMlaAttention` group refuses. +- Every KV-cache factory shipping today publishes exactly the groups the runner + consumes — asserted as literals over the seven `MLAAttentionSpec` call sites' + registries plus the hybrid registries, so the refusal cannot fire for them. +- `spec_equal`'s existing arms keep their answers; the two new arms are red-first + from a pair of identical specs that compares unequal today. + +**Nothing published is consumed, and the commit says so.** No `attn_kv` reaches +`DeepseekV4Model::Forward`, no runner allocates any of the seven groups, and the +refusal is what makes that visible rather than silent. Reachability is owed to +W3 and W5 under [#1925](https://github.com/mudler/vllm.cpp/issues/1925), and +listed under `## Owed`. + ### The speed question, answered **No. A server-side tok/s measurement of DeepSeek-V4 is not meaningful before @@ -790,6 +1000,37 @@ config parse and upstream's disagree about the layer partition (that would be a exist. W1 landed the page formula the refusal names as missing. - [#1960](https://github.com/mudler/vllm.cpp/issues/1960) — W1. Owned by this row, closed by W1. +- [#1973](https://github.com/mudler/vllm.cpp/issues/1973) — W2. Owned by this + row, closed by W2. +- [#1974](https://github.com/mudler/vllm.cpp/issues/1974) — `spec_equal`'s + missing `kMlaAttention` / `kSlidingWindowMla` arms. Owned by this row, FIXED + IN FLOW with W2 and closed by it. Listed rather than omitted because the index + row has to name an owner. +- **The published topology is UNREACHED.** `MakeDeepseekV4KVCache` publishes + seven groups that nothing allocates and nothing reads: the runner refuses them + by name (`### W2 design — publishing the topology, and the refusal that makes it + safe`), `DeepseekV4Model::Forward` still discards `attn_kv`, and no + `PagedKvCache` is built from any of them. Owned by this row. The wiring falls + due at **W3** (the runner carrying more than one attention group and more than + one cache per layer) and the consumption at **W5**. Tracked under + [#1925](https://github.com/mudler/vllm.cpp/issues/1925) and + [#1973](https://github.com/mudler/vllm.cpp/issues/1973). +- **The runner tolerates a second `kFullAttention` group on its KIND, not on + `spec_on()`.** W2's refusal mirrors the draft-KV allocation loop's own + predicate (`src/vllm/v1/worker/gpu/runner.cpp:1128-1130`), so a `fa_draft` + group published with speculation OFF is still tolerated and still unallocated. + Owned by this row, falls due at **W3**, which generalizes + `full_attn_group_id_` and can then account for the draft slot positively + instead of by kind. +- **Only the `fp8_ds_mla` arm of the topology is published.** Upstream selects + between a 576B-aligned `fp8_ds_mla` geometry and a 512B-aligned plain + bf16/fp8 geometry on `use_fp8_ds_mla_layout` (`attention.py:140`, + `nvidia/flashinfer_sparse.py:163`); W2 publishes the default `fp8_ds_mla` arm + unconditionally because our factory signature carries no cache dtype and + `ParseCacheDType` refuses the string by name + (`include/vllm/v1/kv_cache_dtype.h:87-90`). The second arm, and the MXFP4 + 68-byte indexer width (`attention.py:751-755`), are owed to **W5** together + with the store path. ## Evidence From 0b01c8bab5539eeedbd632e0e3a0158a84328d1a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 12:14:51 +0000 Subject: [PATCH 2/5] feat(KV-DSV4-MULTICACHE): W2 -- DeepSeek-V4's real seven-group cache topology, and a runner that names the group it cannot carry (#1973, #1974) MakeDeepseekV4KVCache published ONE placeholder group called "mla", 576 bytes wide, for a model that upstream gives 167 cache entries across 43 layers. It now publishes all of them: 21 C4A and 20 C128A compressed latents plus 21 indexer key caches as MLAAttentionSpec, and 43 SWA caches plus 21 + 21 + 20 compressor states as SlidingWindowMLASpec, at block sizes 256, 64, 4 and 8, with the real module paths as layer names. That could not land alone. GPUModelRunner::initialize_kv_cache selects the first full-attention or MLA group and one Mamba group and has no else, so a kSlidingWindowMla group and a second kMlaAttention group were both dropped with no buffer and no message. Measured on the pre-fix binary: three published groups, constructs without throwing, full_attn_group_id 0, attn_kv().size() 4 -- one buffer per hidden layer off group 0's 1024-byte page -- and zero bytes for the group whose own page is 37440. Publishing seven groups into that would have allocated a subset of the topology in silence, which is a wrong-tokens failure rather than a crash. So the runner now refuses, naming every group it does not allocate, its kind, its first layer and what its page would have cost. The consequence is that a DeepSeek-V4 engine refuses at KV-cache construction instead of running. What it loses is a path with no decode step in it: Forward and ForwardDevice discard attn_kv and recompute the whole prefix per token. examples/deepseek_v4_gen does not go through the runner and is unaffected. docs/FEATURES.md is updated to say so. Four upstream details this port had to get right and a careless one would not: the published prefix is `attn`, not the `self_attn` every other architecture in this tree uses; kv_quant_mode is passed on two of the four construction sites and not the other two, so the SWA and latent specs carry FP8_PER_TENSOR and must still reach the 584-byte branch before W1's quant guard throws; the indexer and compressor specs carry no model_version and take the element formula; and the indexer key width defaults to 132 rather than the MXFP4 68. #1974 rides along: spec_equal's default arm answered false for kMlaAttention and kSlidingWindowMla, so two identical MLA specs never merged into one SpecGroup. Upstream's frozen dataclasses cannot answer that. Latent until this commit published three MLA groups; fixed in flow with both arms comparing the four DeepSeek-V4 fields. NOTHING CONSUMES THE PUBLISHED TOPOLOGY. No runner allocates any of the seven groups and DeepseekV4Model::Forward still discards attn_kv. The wiring is owed to W3 and the consumption to W5, both tracked under #1925 and both listed under `## Owed` in .agents/specs/kv-dsv4-multicache.md, together with the fp8_ds_mla arm being the only one published and the draft slot being tolerated on its kind. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-fable-5 [Claude Code] --- docs/FEATURES.md | 2 +- .../models/deepseek_v4_registry.cpp | 228 +++++++++++++++-- src/vllm/v1/core/kv_cache_coordinator.cpp | 42 +++ src/vllm/v1/worker/gpu/runner.cpp | 88 +++++++ .../vllm/models/test_deepseek_v4_scaffold.cpp | 239 ++++++++++++++++++ tests/vllm/v1/test_kv_cache_coordinator.cpp | 85 +++++++ tests/vllm/v1/worker/test_runner.cpp | 160 ++++++++++++ 7 files changed, 828 insertions(+), 16 deletions(-) diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 7ed0ba57a..0ad745d39 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -144,7 +144,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `MiniCPM3ForCausalLM` | openbmb/MiniCPM3-4B (MLA) | near-tie 16/16 vs vLLM 0.25.0 | pending | | `Olmo2ForCausalLM`, `Olmo3ForCausalLM` | allenai/OLMo-2-0425-1B; OLMo-3 (Olmo2 factory alias) | OLMo-2 strict 16/16; OLMo-3 oracle-blocked (vLLM 0.25.0 cannot build it) | pending | | `DeepseekV2ForCausalLM` | DeepSeek-V2-Lite (MLA) | strict 8/8 vs vLLM 0.25.0 | speed short, attributed | -| `DeepseekV4ForCausalLM` | DeepSeek-V4-Flash GGUF (ds4 q2-imatrix, UD-IQ2); the SAFETENSORS arms now get past the tokenizer ([#1924](https://github.com/mudler/vllm.cpp/issues/1924)) | coherent near-tie vs ds4 oracle (vLLM cannot fit one GB10). Tokenizer ids are exact vs HF `tokenizers` on the checkpoint's own 6.4 MB `tokenizer.json`, and the GGUF arm's `joyai-llm` pre no longer resolves to an APPROXIMATION | decode beats ds4 1.144x, default on, via the `deepseek-v4-gen` CLI; the registered engine forward is a W3 stub (`ARCH-ONE-SURFACE` fold) | +| `DeepseekV4ForCausalLM` | DeepSeek-V4-Flash GGUF (ds4 q2-imatrix, UD-IQ2); the SAFETENSORS arms now get past the tokenizer ([#1924](https://github.com/mudler/vllm.cpp/issues/1924)) | coherent near-tie vs ds4 oracle (vLLM cannot fit one GB10). Tokenizer ids are exact vs HF `tokenizers` on the checkpoint's own 6.4 MB `tokenizer.json`, and the GGUF arm's `joyai-llm` pre no longer resolves to an APPROXIMATION | decode beats ds4 1.144x, default on, via the `deepseek-v4-gen` CLI; the registered engine now REFUSES at KV-cache construction rather than running: it publishes DeepSeek-V4's real seven-group / 167-entry cache topology ([#1973](https://github.com/mudler/vllm.cpp/issues/1973)) and the runner carries at most one attention group, so it names every group it cannot allocate instead of allocating a subset in silence; the forward is still a W3 stub (`ARCH-ONE-SURFACE` fold) | | `Glm4ForCausalLM` | GLM-4-9B-0414 | near-tie 16/16 vs vLLM 0.25.0 | pending | | `Glm4MoeLiteForCausalLM` | zai-org/GLM-4.7-Flash (31.2B, MLA MoE) | near-tie 8/8 vs vLLM 0.25.0 | pending | | `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on, via the `laguna-gen` CLI; the registered engine forward VT_CHECKs non-bf16 (`ARCH-ONE-SURFACE` fold) | diff --git a/src/vllm/model_executor/models/deepseek_v4_registry.cpp b/src/vllm/model_executor/models/deepseek_v4_registry.cpp index bf528f825..99bbc7732 100644 --- a/src/vllm/model_executor/models/deepseek_v4_registry.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_registry.cpp @@ -27,8 +27,11 @@ #include "vllm/model_executor/models/model_registry.h" #include +#include #include +#include #include +#include #include "vllm/model_executor/models/deepseek_v4.h" #include "vllm/model_executor/models/qwen3_5.h" // ForwardLogits carrier @@ -125,25 +128,220 @@ const ModelFactory kDeepseekV4Factory{ v1::KVCacheConfig MakeDeepseekV4KVCache(const HfConfig& config, int block_size, int num_blocks) { - // STUB (W3): V4's TRUE KV topology is the fp8_ds_mla UE8M0 576B-paged latent - // (attention.py:89) PLUS the DSA indexer/compressor caches — a multi-cache - // geometry not yet representable. We emit ONE placeholder MLA group sized to - // the compressed latent + rope so the arch RESOLVES and the spec builder is - // wired; the real topology (and the indexer/compressor caches) is a named W3 - // residual. Never exercised this pass — the forward VT_CHECKs pending. + // KV-DSV4-MULTICACHE W2 (#1973): the REAL topology, replacing the one + // placeholder `"mla"` group this function used to emit. + // + // DeepSeek-V4 publishes one cache per (layer x cache role), not one per + // layer. Four upstream construction sites produce them, and for + // DeepSeek-V4-Flash's 43 layers they resolve to 167 entries in SEVEN groups + // (one group per distinct published spec, which is upstream's own grouping + // rule): + // + // (a) `DeepseekV4Attention.get_kv_cache_spec` — the compressed MLA latent, + // `MLAAttentionSpec`, on the 41 layers with compress_ratio > 1 and + // `None` on the other two (`vllm/models/deepseek_v4/attention.py:626-645`). + // Splits by ratio: 21 at C4A, 20 at C128A. + // (b) `DeepseekV4IndexerCache.get_kv_cache_spec` — the indexer key cache, + // `MLAAttentionSpec`, on the 21 layers with ratio == 4 (`:669-684`), + // built at `k_cache_head_dim` bytes rather than a semantic width + // (`:751-760`). + // (c) `DeepseekV4SWACache.get_kv_cache_spec` — the sliding-window cache, + // `SlidingWindowMLASpec`, on ALL 43 attention layers including the two + // that have no MLA cache at all + // (`vllm/v1/attention/backends/mla/sparse_swa.py:86-101`, constructed + // unconditionally at `attention.py:315-321`). + // (d) `CompressorStateCache.get_kv_cache_spec` — the compressor state, + // `SlidingWindowMLASpec`, f32, once per compressor + // (`vllm/models/deepseek_v4/compressor.py:188-200`). TWO populations: + // the attention layer's own compressor (41, `attention.py:333-343`, + // head_dim 512) and the indexer's (21, `attention.py:768-777`, + // head_dim 128). The attention population splits by ratio into 21 + 20. + // + // 21 + 20 + 21 + 43 + 21 + 21 + 20 = 167. + // + // NOTHING CONSUMES THIS. No runner allocates any of these groups and + // `DeepseekV4Model::Forward` still discards `attn_kv`; the runner REFUSES a + // group it cannot allocate (`src/vllm/v1/worker/gpu/runner.cpp`, #1973) + // rather than dropping it in silence, so a DeepSeek-V4 engine now refuses to + // construct instead of allocating a subset of this topology and saying + // nothing. Carrying the groups is row KV-DSV4-MULTICACHE W3; reading them is + // W5. Both are tracked under #1925 and listed under `## Owed` in + // `.agents/specs/kv-dsv4-multicache.md`. const DeepseekV4Params p = ParseDeepseekV4Params(config); - // Placeholder latent width: the wkv-compressed KV (num_key_value_heads * 512 - // rows in the checkpoint) is head-independent for MLA; use head_dim + rope as - // an honest per-page upper bound until the fp8_ds_mla geometry lands. - const int head_size = - static_cast(p.head_dim + p.qk_rope_head_dim); // 512 + 64 = 576 (W3 TODO) + + // THE fp8_ds_mla ARM, and why it is published unconditionally. + // `DeepseekV4Attention.use_fp8_ds_mla_layout` is `ClassVar[bool] = True` on + // the base (`attention.py:140`); only the FlashInfer-sparse SM120 subclass + // sets it False (`vllm/models/deepseek_v4/nvidia/flashinfer_sparse.py:163`). + // `_resolve_dsv4_kv_cache_dtype` then writes `cache_config.cache_dtype = + // "fp8_ds_mla"` back onto the cache config and returns `torch.uint8` + // (`attention.py:89-119`), so the DEFAULT DeepSeek-V4 cache format upstream + // is fp8_ds_mla and that is the arm mirrored here. Our factory signature + // carries no cache dtype and `ParseCacheDType` refuses the string by name + // (`include/vllm/v1/kv_cache_dtype.h:87-90`), so the 512B-aligned plain + // bf16/fp8 arm is NOT published — owed to W5 with the store path. + const std::string kCacheDtypeStr = "fp8_ds_mla"; + const std::string kModelVersion = "deepseek_v4"; + constexpr int kAlignment = 576; // `sparse_swa.py:99`, `attention.py:642` + constexpr int kSwaBlockSize = 64; // `sparse_swa.py:80`, fixed by tensor sharing + // 1-byte storage, mirroring upstream's `torch.uint8`. `SizeOf(kI8) == 1` is + // what the indexer's element formula needs; the 584-byte branch does not read + // the dtype at all. + const vt::DType kByteDType = vt::DType::kI8; + + // `get_kv_quant_mode(cache_dtype)` returns FP8_PER_TENSOR for any string + // starting with "fp8" (`vllm/v1/kv_cache_interface.py:70-71`), and upstream + // passes it on EXACTLY TWO of the four sites: the SWA cache + // (`sparse_swa.py:100`) and the compressed latent (`attention.py:644`). The + // indexer key cache and the compressor state pass nothing and default to + // NONE. Mirrored exactly, including the asymmetry — which is also what makes + // this topology exercise W1's branch order, since those two specs carry a + // non-NONE quant mode AND `cache_dtype_str == "fp8_ds_mla"` and must reach + // the 584-byte branch before the quant-mode guard throws. + const v1::KVQuantMode kLatentQuantMode = v1::KVQuantMode::kFp8PerTensor; + + // The published module path, which is what `LayerIndexOfName` + // (`src/vllm/v1/worker/gpu/runner.cpp`) resolves back to a layer index once + // W3 reads these names. THE SEGMENT IS `attn`, NOT `self_attn`: + // `DeepseekV4DecoderLayer` builds its attention as `prefix=f"{prefix}.attn"` + // (`vllm/models/deepseek_v4/nvidia/model.py:808-813`) under + // `prefix=f"{prefix}.layers"` (`:1015`) and `maybe_prefix(prefix, "model")` + // (`:1409`). Every other architecture in this tree spells it `self_attn`, so + // this is worth stating rather than pattern-matching. + const auto attn_prefix = [](int64_t l) { + return "model.layers." + std::to_string(l) + ".attn"; + }; + + // Per-group layer-name lists, filled by ONE walk over the layers. + std::vector c4a_latent, c128a_latent, indexer_key, swa; + std::vector c4_attn_state, c4_indexer_state, c128_attn_state; + bool has_c4 = false, has_c128 = false; + + for (int64_t l = 0; l < p.num_hidden_layers; ++l) { + // `max(1, config.compress_ratios[layer_id])` — upstream's own guard, and + // the value every downstream branch tests against (`attention.py:205-212`). + // Our parser keeps the raw 0 for layers 0 and 1. + const int64_t raw = p.compress_ratio(l); + const int64_t ratio = raw < 1 ? 1 : raw; + VT_CHECK(ratio == 1 || ratio == 4 || ratio == 128, + std::string("deepseek-v4 kv-cache: unsupported compress_ratio ") + + std::to_string(ratio) + " on layer " + std::to_string(l) + + "; upstream accepts 1, 4 or 128 only " + "(vllm/v1/attention/backends/mla/sparse_swa.py:44-55)"); + + // (c) Every attention layer has a SWA cache, unconditionally. + swa.push_back(attn_prefix(l) + ".swa_cache"); + + if (ratio == 1) continue; // `attention.py:626-630` returns None: no MLA cache. + + // (a) The compressed latent. `head_size = self.head_dim` (512) — NOT + // head_dim + rope, which is what the placeholder this replaces used. + (ratio == 4 ? c4a_latent : c128a_latent).push_back(attn_prefix(l)); + // (d) The attention layer's own compressor state. + (ratio == 4 ? c4_attn_state : c128_attn_state) + .push_back(attn_prefix(l) + ".compressor.state_cache"); + if (ratio == 4) { + has_c4 = true; + // (b) + (d) The indexer's key cache and its own compressor state. + indexer_key.push_back(attn_prefix(l) + ".indexer.k_cache"); + c4_indexer_state.push_back(attn_prefix(l) + + ".indexer.compressor.state_cache"); + } else { + has_c128 = true; + } + } + + // `storage_block_size` is `block_size / compress_ratio`, so a block size that + // is not a multiple of the ratio gives a truncated page and one that is + // smaller than the ratio gives a ZERO-byte page. Upstream has no such check + // because its own comments derive the whole geometry at 256 + // (`sparse_swa.py:76-83` and `compressor.py:174-178` both spell + // `[256//4, head_dim] = [64, head_dim]`). Refuse by name rather than publish + // a pool that cannot hold a token. + const auto check_ratio_fits = [&](int ratio) { + VT_CHECK(block_size % ratio == 0 && block_size / ratio >= 1, + std::string("deepseek-v4 kv-cache: block_size ") + + std::to_string(block_size) + + " cannot express a compress_ratio-" + std::to_string(ratio) + + " page (storage_block_size = block_size / compress_ratio " + "would be " + std::to_string(block_size / ratio) + + "). Upstream derives this geometry at block_size 256 " + "(vllm/v1/attention/backends/mla/sparse_swa.py:76-83, " + "vllm/models/deepseek_v4/compressor.py:174-178)"); + }; + if (has_c4) check_ratio_fits(4); + if (has_c128) check_ratio_fits(128); + + // `head_dim bytes = 128 fp8 + 4 fp32 scale = 132` (`attention.py:756-759`), + // with `quant_block_size = 128` (`:738`). The MXFP4 68-byte arm is selected by + // `attention_config.use_fp4_indexer_cache`, whose default is False + // (`vllm/config/attention.py:64`), so the FP8 width is the default and the + // other arm is owed to W5. + constexpr int64_t kIndexerQuantBlock = 128; + const int indexer_head_size = static_cast( + p.index_head_dim + p.index_head_dim / kIndexerQuantBlock * 4); + + const int head_size = static_cast(p.head_dim); // 512 + const int swa_window = static_cast(p.sliding_window); // 128 + + // `state_dim = 2 * coff * head_dim` with `coff = 1 + (compress_ratio == 4)`, + // `sliding_window = coff * compress_ratio`, and block_size 4 for ratio 4 / 8 + // for ratio 128 (`compressor.py:168-200`). dtype is f32, which upstream + // asserts (`:170`). + const int c4_attn_state_dim = static_cast(2 * 2 * p.head_dim); // 2048 + const int c4_indexer_state_dim = static_cast(2 * 2 * p.index_head_dim); // 512 + const int c128_state_dim = static_cast(2 * 1 * p.head_dim); // 1024 v1::KVCacheConfig kv; kv.num_blocks = num_blocks; - kv.kv_cache_groups.emplace_back( - std::vector{"mla"}, - std::make_shared(block_size, head_size, - v1::ResolveKvCacheDType())); + + const auto add_mla = [&](std::vector names, int hs, int ratio, + bool ds_mla_layout) { + if (names.empty()) return; + kv.kv_cache_groups.emplace_back( + std::move(names), + std::make_shared( + block_size, hs, kByteDType, /*num_kv_heads=*/1, + ds_mla_layout ? kLatentQuantMode : v1::KVQuantMode::kNone, + /*page_size_padded=*/std::nullopt, + /*indexes_kv_by_block_stride=*/false, + ds_mla_layout ? std::optional(kCacheDtypeStr) + : std::nullopt, + kAlignment, ratio, + ds_mla_layout ? std::optional(kModelVersion) + : std::nullopt)); + }; + const auto add_swa_mla = [&](std::vector names, int bs, int hs, + vt::DType dt, int window, bool ds_mla_layout) { + if (names.empty()) return; + kv.kv_cache_groups.emplace_back( + std::move(names), + std::make_shared( + bs, /*num_kv_heads=*/1, hs, dt, window, + ds_mla_layout ? std::optional(kCacheDtypeStr) + : std::nullopt, + kAlignment, /*compress_ratio=*/1, + ds_mla_layout ? std::optional(kModelVersion) + : std::nullopt, + ds_mla_layout ? kLatentQuantMode : v1::KVQuantMode::kNone)); + }; + + // Group order is ours and is documented rather than incidental: the + // `MLAAttentionSpec` groups first (latent by ratio, then the indexer key), + // then the `SlidingWindowMLASpec` groups (SWA, then the three compressor + // state populations). + add_mla(std::move(c4a_latent), head_size, /*ratio=*/4, /*ds_mla_layout=*/true); + add_mla(std::move(c128a_latent), head_size, /*ratio=*/128, true); + add_mla(std::move(indexer_key), indexer_head_size, /*ratio=*/4, + /*ds_mla_layout=*/false); + add_swa_mla(std::move(swa), kSwaBlockSize, head_size, kByteDType, swa_window, + /*ds_mla_layout=*/true); + add_swa_mla(std::move(c4_attn_state), /*bs=*/4, c4_attn_state_dim, + vt::DType::kF32, /*window=*/8, /*ds_mla_layout=*/false); + add_swa_mla(std::move(c4_indexer_state), /*bs=*/4, c4_indexer_state_dim, + vt::DType::kF32, /*window=*/8, /*ds_mla_layout=*/false); + add_swa_mla(std::move(c128_attn_state), /*bs=*/8, c128_state_dim, + vt::DType::kF32, /*window=*/128, /*ds_mla_layout=*/false); return kv; } diff --git a/src/vllm/v1/core/kv_cache_coordinator.cpp b/src/vllm/v1/core/kv_cache_coordinator.cpp index ec4481a32..846a2b8c7 100644 --- a/src/vllm/v1/core/kv_cache_coordinator.cpp +++ b/src/vllm/v1/core/kv_cache_coordinator.cpp @@ -32,6 +32,48 @@ bool spec_equal(const KVCacheSpec& a, const KVCacheSpec& b) { fa.attention_chunk_size == fb.attention_chunk_size && fa.non_causal == fb.non_causal; } + // #1974: `kMlaAttention` and `kSlidingWindowMla` used to fall into + // `default:` and compare UNEQUAL to a structurally identical twin, so two + // identical MLA groups never merged into one SpecGroup. Upstream cannot + // answer that: every spec class is `@dataclass(frozen=True, kw_only=True)` + // (`vllm/v1/kv_cache_interface.py:380-381` MLAAttentionSpec, `:610-611` + // SlidingWindowMLASpec), so `__eq__` is generated over ALL fields — the + // parent's plus the four DeepSeek-V4 fields — and two identical specs are + // equal. Latent until KV-DSV4-MULTICACHE W2 (#1973) published three + // `kMlaAttention` and four `kSlidingWindowMla` groups for DeepSeek-V4, + // because every MLA model in the tree publishes exactly one MLA group. + case KVCacheSpecKind::kMlaAttention: { + const auto& ma = static_cast(a); + const auto& mb = static_cast(b); + return ma.num_kv_heads == mb.num_kv_heads && + ma.head_size == mb.head_size && ma.head_size_v == mb.head_size_v && + ma.dtype == mb.dtype && ma.kv_quant_mode == mb.kv_quant_mode && + ma.page_size_padded == mb.page_size_padded && + ma.indexes_kv_by_block_stride == + mb.indexes_kv_by_block_stride && + ma.sliding_window == mb.sliding_window && + ma.attention_chunk_size == mb.attention_chunk_size && + ma.non_causal == mb.non_causal && + ma.cache_dtype_str == mb.cache_dtype_str && + ma.alignment == mb.alignment && + ma.compress_ratio == mb.compress_ratio && + ma.model_version == mb.model_version; + } + case KVCacheSpecKind::kSlidingWindowMla: { + const auto& sa = static_cast(a); + const auto& sb = static_cast(b); + return sa.num_kv_heads == sb.num_kv_heads && + sa.head_size == sb.head_size && sa.head_size_v == sb.head_size_v && + sa.dtype == sb.dtype && sa.kv_quant_mode == sb.kv_quant_mode && + sa.page_size_padded == sb.page_size_padded && + sa.indexes_kv_by_block_stride == + sb.indexes_kv_by_block_stride && + sa.sliding_window == sb.sliding_window && + sa.cache_dtype_str == sb.cache_dtype_str && + sa.alignment == sb.alignment && + sa.compress_ratio == sb.compress_ratio && + sa.model_version == sb.model_version; + } case KVCacheSpecKind::kSlidingWindow: { const auto& sa = static_cast(a); const auto& sb = static_cast(b); diff --git a/src/vllm/v1/worker/gpu/runner.cpp b/src/vllm/v1/worker/gpu/runner.cpp index cbca63f13..360c8294d 100644 --- a/src/vllm/v1/worker/gpu/runner.cpp +++ b/src/vllm/v1/worker/gpu/runner.cpp @@ -357,6 +357,26 @@ std::optional LayerIndexOfName(std::string_view name) { return value; } +// The enumerator's own name, for the refusal below. Nothing else in the tree +// spells a KVCacheSpecKind, and a refusal that prints an integer names nothing. +const char* KVCacheSpecKindName(KVCacheSpecKind kind) { + switch (kind) { + case KVCacheSpecKind::kFullAttention: return "kFullAttention"; + case KVCacheSpecKind::kMlaAttention: return "kMlaAttention"; + case KVCacheSpecKind::kSlidingWindow: return "kSlidingWindow"; + case KVCacheSpecKind::kSlidingWindowMla: return "kSlidingWindowMla"; + case KVCacheSpecKind::kMamba: return "kMamba"; + case KVCacheSpecKind::kChunkedLocalAttention: + return "kChunkedLocalAttention"; + case KVCacheSpecKind::kSinkFullAttention: return "kSinkFullAttention"; + case KVCacheSpecKind::kEncoderOnlyAttention: + return "kEncoderOnlyAttention"; + case KVCacheSpecKind::kCrossAttention: return "kCrossAttention"; + case KVCacheSpecKind::kUnknown: return "kUnknown"; + } + return "kUnknown"; +} + // The per-layer membership mask of one KV cache group, or nullopt when the // group does not publish per-layer names. // @@ -595,6 +615,74 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { } } + // KV-DSV4-MULTICACHE W2 (#1973): REFUSE a published group this runner does + // not allocate, instead of dropping it in silence. + // + // The loop above has exactly two arms and NO `else`. A `kSlidingWindowMla` + // group matched nothing, and a SECOND `kMlaAttention` group is passed over by + // the `full_attn_group_id_ < 0` guard — and neither produced a buffer or a + // message. That is not "one group is missing": `membership_by_name` below is + // reached only when the model has a recurrent group, so a model without one + // (DeepSeek-V4 has none) falls into `is_full_attn = !is_gdn` and allocates + // ONE buffer per HIDDEN LAYER, every one of them sized from the TARGET + // group's page. Publishing DeepSeek-V4's seven groups without this check + // allocates 43 buffers of one page for a model that needs 167 of seven, and + // reports nothing. A silently short KV allocation is a wrong-tokens failure, + // not a crash, which is why this refuses rather than warns + // (AGENTS.md: "Refuse an unimplemented arm with a message that names the + // missing part"). + // + // A published group is exactly one of four things: the TARGET attention + // group, the RECURRENT group, the single `fa_draft` draft-KV slot allocated + // at `draft_attn_buf_` below, or a defect. The draft slot is recognised on + // its KIND, mirroring that block's own predicate (`kind() != kFullAttention` + // => continue), so a `fa_draft` group published with speculation OFF is + // tolerated here and still unallocated — tightening that belongs with the + // generalization of `full_attn_group_id_` (row KV-DSV4-MULTICACHE W3, which + // is where a positive per-group accounting becomes possible). + // + // BYTE-NEUTRAL for every model shipping today: each publishes exactly the + // groups this runner consumes, which `test_runner.cpp` asserts directly + // rather than leaves to be inferred. + { + std::string unallocated; + int unallocated_count = 0; + bool draft_slot_taken = false; + for (int g = 0; + g < static_cast(kv_cache_config.kv_cache_groups.size()); ++g) { + if (g == full_attn_group_id_ || g == gdn_group_id_) continue; + const auto& group = + kv_cache_config.kv_cache_groups[static_cast(g)]; + const KVCacheSpecKind kind = group.kv_cache_spec->kind(); + if (kind == KVCacheSpecKind::kFullAttention && !draft_slot_taken) { + draft_slot_taken = true; // the `fa_draft` draft-KV slot + continue; + } + ++unallocated_count; + unallocated += "\n group "; + unallocated += std::to_string(g); + unallocated += " kind="; + unallocated += KVCacheSpecKindName(kind); + unallocated += " layers="; + unallocated += std::to_string(group.layer_names.size()); + unallocated += " first='"; + unallocated += group.layer_names.empty() ? std::string("") + : group.layer_names.front(); + unallocated += "' page_size_bytes="; + unallocated += + std::to_string(group.kv_cache_spec->page_size_bytes()); + } + VT_CHECK(unallocated_count == 0, + std::string("runner: ") + std::to_string(unallocated_count) + + " published KV cache group(s) get NO cache from this runner, " + "which carries at most ONE attention group, ONE recurrent " + "group and ONE fa_draft slot. Refusing rather than allocating " + "a SUBSET of the published topology in silence. Unallocated:" + + unallocated + + "\n(row KV-DSV4-MULTICACHE W3 owns carrying more than one " + "attention group and more than one cache per layer; #1973)"); + } + // Allocate one PagedKvCache per full-attn layer and one GdnStateCache per GDN // layer, in LAYER ORDER (matches Qwen3_5Model::Forward's per-layer fa_idx / // gdn_idx indexing). As in upstream, MambaSpec is the source of truth for the diff --git a/tests/vllm/models/test_deepseek_v4_scaffold.cpp b/tests/vllm/models/test_deepseek_v4_scaffold.cpp index e5db6f10d..21aa43842 100644 --- a/tests/vllm/models/test_deepseek_v4_scaffold.cpp +++ b/tests/vllm/models/test_deepseek_v4_scaffold.cpp @@ -8,13 +8,17 @@ #include "vllm/model_executor/models/deepseek_v4.h" #include "vllm/model_executor/models/deepseek_v4_probe.h" #include "vllm/model_executor/models/model_registry.h" +#include "vllm/v1/kv_cache_interface.h" #include #include +#include #include #include +#include +#include #include #include #include @@ -170,3 +174,238 @@ TEST_CASE("deepseek-v4 scaffold: parse REJECTS an unrepresentable config") { bad2.raw["scoring_func"] = "sigmoid"; // V4 is sqrtsoftplus-only CHECK_THROWS_AS(ParseDeepseekV4Params(bad2), std::runtime_error); } + +// ─── KV-DSV4-MULTICACHE W2 (#1973) — the published cache topology ──────────── +// +// G1, the topology gate of `.agents/specs/kv-dsv4-multicache.md`, on the half +// that needs no checkpoint, no GPU and no forward: upstream's cache topology is +// a pure function of the config, so the whole spec set can be compared exactly. +// Every expectation below was derived from an upstream CONSTRUCTION SITE at the +// pin 5559679229bc961848b121ccdeaa8fa5d79bec98, not from this test's own +// arithmetic, and the page sizes are stated as LITERALS so a change to any of +// them is a red test rather than a silently different pool. +// +// NOTHING CONSUMES THIS. The runner refuses these groups by name (#1973); the +// wiring is owed to W3 and the consumption to W5, both under #1925. +TEST_CASE("deepseek-v4 kv-cache: the published topology is upstream's 167 entries") { + // 256 is not a guess. `sparse_swa.py:76-83` and `compressor.py:174-178` both + // derive the geometry from `[256//4, head_dim] = [64, head_dim]`, and the SWA + // block size of 64 and the compressor block sizes of 4 and 8 only hold there. + const vllm::v1::KVCacheConfig kv = + vllm::MakeDeepseekV4KVCache(RealConfig(), /*block_size=*/256, + /*num_blocks=*/8); + REQUIRE(kv.kv_cache_groups.size() == 7); + CHECK(kv.num_blocks == 8); + + size_t entries = 0; + for (const auto& g : kv.kv_cache_groups) entries += g.layer_names.size(); + CHECK(entries == 167); // 21 + 20 + 21 + 43 + 21 + 21 + 20 + + using vllm::v1::KVCacheSpecKind; + using vllm::v1::KVQuantMode; + using vllm::v1::MLAAttentionSpec; + using vllm::v1::SlidingWindowMLASpec; + + // (a) The compressed MLA latent, C4A. `attention.py:631-645`: block_size from + // the cache config, num_kv_heads 1, head_size = self.head_dim (512 — NOT + // head_dim + rope), uint8, compress_ratio 4, cache_dtype_str "fp8_ds_mla", + // alignment 576, model_version "deepseek_v4", kv_quant_mode FP8_PER_TENSOR. + { + const auto& g = kv.kv_cache_groups[0]; + const auto* sp = dynamic_cast(g.kv_cache_spec.get()); + REQUIRE(sp != nullptr); + CHECK(sp->kind() == KVCacheSpecKind::kMlaAttention); + CHECK(sp->block_size == 256); + CHECK(sp->storage_block_size() == 64); // 256 / 4 + CHECK(sp->num_kv_heads == 1); + CHECK(sp->head_size == 512); + CHECK(sp->dtype == vt::DType::kI8); + CHECK(sp->compress_ratio == 4); + CHECK(sp->alignment == std::optional(576)); + CHECK(sp->cache_dtype_str == std::optional("fp8_ds_mla")); + CHECK(sp->model_version == std::optional("deepseek_v4")); + // The branch order W1 landed, on the real geometry: this spec carries a + // NON-NONE quant mode AND cache_dtype_str "fp8_ds_mla", so the 584-byte + // branch must be reached before the quant-mode guard throws. + CHECK(sp->kv_quant_mode == KVQuantMode::kFp8PerTensor); + CHECK(sp->real_page_size_bytes() == 37376); // 64 * 584 + CHECK(sp->page_size_bytes() == 37440); // round_up(37376, 576) + CHECK(g.layer_names.size() == 21); + CHECK(g.layer_names.front() == "model.layers.2.attn"); + CHECK(g.layer_names.back() == "model.layers.42.attn"); + } + + // (a) The compressed MLA latent, C128A. Same site at ratio 128. + { + const auto& g = kv.kv_cache_groups[1]; + const auto* sp = dynamic_cast(g.kv_cache_spec.get()); + REQUIRE(sp != nullptr); + CHECK(sp->block_size == 256); + CHECK(sp->storage_block_size() == 2); // 256 / 128 + CHECK(sp->head_size == 512); + CHECK(sp->compress_ratio == 128); + CHECK(sp->real_page_size_bytes() == 1168); // 2 * 584 + CHECK(sp->page_size_bytes() == 1728); // round_up(1168, 576) + CHECK(g.layer_names.size() == 20); + CHECK(g.layer_names.front() == "model.layers.3.attn"); + CHECK(g.layer_names.back() == "model.layers.41.attn"); + } + + // (b) The indexer key cache. `attention.py:669-684` passes NO cache_dtype_str + // and NO model_version, so it takes the ELEMENT formula and not the 584-byte + // branch, and it defaults kv_quant_mode to NONE. Its width is byte-derived: + // head_dim + head_dim // quant_block_size * 4 = 128 + 128 // 128 * 4 = 132 + // (`:738`, `:756-759`); the MXFP4 68-byte arm needs + // `use_fp4_indexer_cache`, whose default is False (`config/attention.py:64`). + { + const auto& g = kv.kv_cache_groups[2]; + const auto* sp = dynamic_cast(g.kv_cache_spec.get()); + REQUIRE(sp != nullptr); + CHECK(sp->block_size == 256); + CHECK(sp->storage_block_size() == 64); // 256 / 4 + CHECK(sp->head_size == 132); + CHECK(sp->dtype == vt::DType::kI8); + CHECK(sp->compress_ratio == 4); + CHECK(sp->alignment == std::optional(576)); + CHECK_FALSE(sp->cache_dtype_str.has_value()); + CHECK_FALSE(sp->model_version.has_value()); + CHECK(sp->kv_quant_mode == KVQuantMode::kNone); + CHECK(sp->real_page_size_bytes() == 8448); // 64 * 1 * 132 * 1 + CHECK(sp->page_size_bytes() == 8640); // round_up(8448, 576) + CHECK(g.layer_names.size() == 21); + CHECK(g.layer_names.front() == "model.layers.2.attn.indexer.k_cache"); + } + + // (c) The sliding-window cache, on ALL 43 attention layers — including + // layers 0 and 1, which have NO MLA cache at all. `sparse_swa.py:86-101`. + { + const auto& g = kv.kv_cache_groups[3]; + const auto* sp = + dynamic_cast(g.kv_cache_spec.get()); + REQUIRE(sp != nullptr); + CHECK(sp->kind() == KVCacheSpecKind::kSlidingWindowMla); + CHECK(sp->block_size == 64); + CHECK(sp->storage_block_size() == 64); // compress_ratio 1 + CHECK(sp->num_kv_heads == 1); + CHECK(sp->head_size == 512); + CHECK(sp->dtype == vt::DType::kI8); + CHECK(sp->sliding_window == std::optional(128)); + CHECK(sp->alignment == std::optional(576)); + CHECK(sp->cache_dtype_str == std::optional("fp8_ds_mla")); + CHECK(sp->model_version == std::optional("deepseek_v4")); + CHECK(sp->kv_quant_mode == KVQuantMode::kFp8PerTensor); + CHECK(sp->real_page_size_bytes() == 37376); // 64 * 584 + CHECK(sp->page_size_bytes() == 37440); + CHECK(g.layer_names.size() == 43); + CHECK(g.layer_names.front() == "model.layers.0.attn.swa_cache"); + CHECK(g.layer_names.back() == "model.layers.42.attn.swa_cache"); + } + + // (d) The compressor states. `compressor.py:168-200`: f32, state_dim = + // 2 * coff * head_dim with coff = 1 + (ratio == 4), sliding_window = + // coff * ratio, block_size 4 at ratio 4 and 8 at ratio 128. NO + // cache_dtype_str and NO model_version, so the element formula again. + { + const auto& g = kv.kv_cache_groups[4]; // attention compressor, ratio 4 + const auto* sp = + dynamic_cast(g.kv_cache_spec.get()); + REQUIRE(sp != nullptr); + CHECK(sp->block_size == 4); + CHECK(sp->storage_block_size() == 4); + CHECK(sp->head_size == 2048); // 2 * 2 * 512 + CHECK(sp->dtype == vt::DType::kF32); + CHECK(sp->sliding_window == std::optional(8)); // coff * ratio + CHECK_FALSE(sp->cache_dtype_str.has_value()); + CHECK_FALSE(sp->model_version.has_value()); + CHECK(sp->real_page_size_bytes() == 32768); // 4 * 1 * 2048 * 4 + CHECK(sp->page_size_bytes() == 32832); // round_up(32768, 576) + CHECK(g.layer_names.size() == 21); + CHECK(g.layer_names.front() == + "model.layers.2.attn.compressor.state_cache"); + } + { + const auto& g = kv.kv_cache_groups[5]; // the INDEXER's own compressor + const auto* sp = + dynamic_cast(g.kv_cache_spec.get()); + REQUIRE(sp != nullptr); + CHECK(sp->block_size == 4); + CHECK(sp->head_size == 512); // 2 * 2 * 128, built at head_dim=128 + CHECK(sp->dtype == vt::DType::kF32); + CHECK(sp->sliding_window == std::optional(8)); + CHECK(sp->real_page_size_bytes() == 8192); // 4 * 1 * 512 * 4 + CHECK(sp->page_size_bytes() == 8640); // round_up(8192, 576) + CHECK(g.layer_names.size() == 21); + CHECK(g.layer_names.front() == + "model.layers.2.attn.indexer.compressor.state_cache"); + } + { + const auto& g = kv.kv_cache_groups[6]; // attention compressor, ratio 128 + const auto* sp = + dynamic_cast(g.kv_cache_spec.get()); + REQUIRE(sp != nullptr); + CHECK(sp->block_size == 8); + CHECK(sp->head_size == 1024); // 2 * 1 * 512 + CHECK(sp->dtype == vt::DType::kF32); + CHECK(sp->sliding_window == std::optional(128)); // coff * ratio + CHECK(sp->real_page_size_bytes() == 32768); // 8 * 1 * 1024 * 4 + CHECK(sp->page_size_bytes() == 32832); + CHECK(g.layer_names.size() == 20); + CHECK(g.layer_names.front() == + "model.layers.3.attn.compressor.state_cache"); + } + + // THE PORT'S OWN SELF-CHECK. Upstream fixes the SWA block size at 64 + // *because* the SWA and C4A blocks share one physical tensor and must + // therefore have one page size (`sparse_swa.py:76-83`). Our two classes reach + // that page by different routes — an MLAAttentionSpec at block_size 256 / + // compress_ratio 4 and a SlidingWindowMLASpec at block_size 64 / + // compress_ratio 1 — so the equality holds only if both storage_block_size + // overrides, both 584-byte branches and the shared 576-byte padding are all + // right at once. Asserted directly, not just as two literals. + CHECK(kv.kv_cache_groups[0].kv_cache_spec->page_size_bytes() == + kv.kv_cache_groups[3].kv_cache_spec->page_size_bytes()); + CHECK(kv.kv_cache_groups[0].kv_cache_spec->storage_block_size() == + kv.kv_cache_groups[3].kv_cache_spec->storage_block_size()); +} + +// The published names carry `.attn.`, NOT the `.self_attn.` every other +// architecture in this tree uses: `DeepseekV4DecoderLayer` builds its attention +// as `prefix=f"{prefix}.attn"` (`vllm/models/deepseek_v4/nvidia/model.py:808-813`) +// under `prefix=f"{prefix}.layers"` (`:1015`) and `maybe_prefix(prefix, "model")` +// (`:1409`). W3 resolves these back to layer indices, and a name that does not +// parse makes GroupLayerMask fall back wholesale rather than fail, so the +// spelling is gated here where it is still cheap. +TEST_CASE("deepseek-v4 kv-cache: names are the upstream module paths") { + const vllm::v1::KVCacheConfig kv = + vllm::MakeDeepseekV4KVCache(RealConfig(), 256, 8); + for (const auto& g : kv.kv_cache_groups) { + for (const std::string& n : g.layer_names) { + CHECK(n.rfind("model.layers.", 0) == 0); + CHECK(n.find(".attn") != std::string::npos); + CHECK(n.find(".self_attn") == std::string::npos); + } + // Within one group every name resolves to a DISTINCT layer index, which is + // GroupLayerMask's all-or-nothing precondition (`runner.cpp`). + std::vector idx; + for (const std::string& n : g.layer_names) { + const size_t at = n.find(".layers.") + 8; + idx.push_back(n.substr(at, n.find('.', at) - at)); + } + std::vector uniq = idx; + std::sort(uniq.begin(), uniq.end()); + uniq.erase(std::unique(uniq.begin(), uniq.end()), uniq.end()); + CHECK(uniq.size() == idx.size()); + } +} + +// `storage_block_size` is `block_size / compress_ratio`, so a block size below +// the ratio produces a ZERO-byte page rather than a refusal. Upstream has no +// such check because its comments derive the geometry at 256; ours refuses. +TEST_CASE("deepseek-v4 kv-cache: a block_size that cannot hold a C128A row is refused") { + CHECK_THROWS_AS(vllm::MakeDeepseekV4KVCache(RealConfig(), 16, 8), + std::runtime_error); + CHECK_THROWS_AS(vllm::MakeDeepseekV4KVCache(RealConfig(), 192, 8), + std::runtime_error); + // 128 divides 128 exactly and gives storage_block_size 1: representable. + CHECK_NOTHROW(vllm::MakeDeepseekV4KVCache(RealConfig(), 128, 8)); +} diff --git a/tests/vllm/v1/test_kv_cache_coordinator.cpp b/tests/vllm/v1/test_kv_cache_coordinator.cpp index 498cb2ac3..6762fa81c 100644 --- a/tests/vllm/v1/test_kv_cache_coordinator.cpp +++ b/tests/vllm/v1/test_kv_cache_coordinator.cpp @@ -20,6 +20,7 @@ #include #include +#include #include #include @@ -63,6 +64,22 @@ std::shared_ptr MakeFullSpec() { /*head_size=*/1, DType::kF32); } +std::shared_ptr MakeMlaSpec(int compress_ratio = 1) { + return std::make_shared( + kBlockSize, /*head_size=*/1, DType::kF32, /*num_kv_heads=*/1, + vllm::v1::KVQuantMode::kNone, /*page_size_padded=*/std::nullopt, + /*indexes_kv_by_block_stride=*/false, /*cache_dtype_str=*/std::nullopt, + /*alignment=*/std::nullopt, compress_ratio, + /*model_version=*/std::nullopt); +} + +std::shared_ptr MakeSlidingMlaSpec( + int sliding_window = 4) { + return std::make_shared( + kBlockSize, /*num_kv_heads=*/1, /*head_size=*/1, DType::kF32, + sliding_window); +} + std::shared_ptr MakeMambaSpec() { return std::make_shared( kBlockSize, std::vector>{{1, 1}}, @@ -275,6 +292,74 @@ TEST_CASE("HybridKVCacheCoordinator: equal sliding specs share one SpecGroup") { CHECK(hyb->attention_groups[1].group_ids == std::vector{0, 2}); } +// #1974 (KV-DSV4-MULTICACHE W2). `spec_equal`'s `default:` arm returned false +// for `kMlaAttention` and `kSlidingWindowMla`, so two structurally identical MLA +// specs compared UNEQUAL and each became its own SpecGroup. Upstream cannot +// answer that: every spec class is `@dataclass(frozen=True, kw_only=True)` +// (`vllm/v1/kv_cache_interface.py:380-381`, `:610-611`), so `__eq__` is +// generated over all fields — including the four DeepSeek-V4 fields — and two +// identical specs are equal. RED-first: before the fix these reported 3 groups. +// +// The third (full-attention) group is not decoration. `HybridKVCacheCoordinator` +// asserts `attention_groups.size() > 1`, so a config of two identical specs +// alone would trip that assert once they correctly merge. +TEST_CASE("HybridKVCacheCoordinator: equal MLA specs share one SpecGroup") { + KVCacheConfig cfg; + cfg.num_blocks = 100; + cfg.kv_cache_groups.emplace_back(std::vector{"mla0"}, + MakeMlaSpec()); + cfg.kv_cache_groups.emplace_back(std::vector{"full0"}, + MakeFullSpec()); + cfg.kv_cache_groups.emplace_back(std::vector{"mla1"}, + MakeMlaSpec()); + auto coord = MakeCoordinator(std::move(cfg)); + auto* hyb = dynamic_cast(coord.get()); + REQUIRE(hyb != nullptr); + REQUIRE(hyb->attention_groups.size() == 2); + CHECK(hyb->attention_groups[0].spec->kind() == + vllm::v1::KVCacheSpecKind::kFullAttention); + CHECK(hyb->attention_groups[1].spec->kind() == + vllm::v1::KVCacheSpecKind::kMlaAttention); + CHECK(hyb->attention_groups[1].group_ids == std::vector{0, 2}); +} + +// The four DeepSeek-V4 fields PARTICIPATE. Two MLA specs identical in every +// inherited field but differing in `compress_ratio` are two SpecGroups, which +// is what upstream's generated `__eq__` answers and what DeepSeek-V4's C4A and +// C128A latent groups depend on. +TEST_CASE("HybridKVCacheCoordinator: MLA specs differing only in compress_ratio do NOT merge") { + KVCacheConfig cfg; + cfg.num_blocks = 100; + cfg.kv_cache_groups.emplace_back(std::vector{"mla_c1"}, + MakeMlaSpec(/*compress_ratio=*/1)); + cfg.kv_cache_groups.emplace_back(std::vector{"full0"}, + MakeFullSpec()); + cfg.kv_cache_groups.emplace_back(std::vector{"mla_c2"}, + MakeMlaSpec(/*compress_ratio=*/2)); + auto coord = MakeCoordinator(std::move(cfg)); + auto* hyb = dynamic_cast(coord.get()); + REQUIRE(hyb != nullptr); + CHECK(hyb->attention_groups.size() == 3); +} + +TEST_CASE("HybridKVCacheCoordinator: equal sliding-window MLA specs share one SpecGroup") { + KVCacheConfig cfg; + cfg.num_blocks = 100; + cfg.kv_cache_groups.emplace_back(std::vector{"swamla0"}, + MakeSlidingMlaSpec()); + cfg.kv_cache_groups.emplace_back(std::vector{"full0"}, + MakeFullSpec()); + cfg.kv_cache_groups.emplace_back(std::vector{"swamla1"}, + MakeSlidingMlaSpec()); + auto coord = MakeCoordinator(std::move(cfg)); + auto* hyb = dynamic_cast(coord.get()); + REQUIRE(hyb != nullptr); + REQUIRE(hyb->attention_groups.size() == 2); + CHECK(hyb->attention_groups[1].spec->kind() == + vllm::v1::KVCacheSpecKind::kSlidingWindowMla); + CHECK(hyb->attention_groups[1].group_ids == std::vector{0, 2}); +} + TEST_CASE("HybridKVCacheCoordinator: equal chunked-local specs share one SpecGroup") { KVCacheConfig cfg; cfg.num_blocks = 100; diff --git a/tests/vllm/v1/worker/test_runner.cpp b/tests/vllm/v1/worker/test_runner.cpp index f33dfda5b..b210e0dec 100644 --- a/tests/vllm/v1/worker/test_runner.cpp +++ b/tests/vllm/v1/worker/test_runner.cpp @@ -1581,3 +1581,163 @@ TEST_CASE("runner: initialize_kv_cache refuses a non-multiple-of-16 block size") doctest::Contains("block_size not supported"), std::runtime_error); } + +// ─── KV-DSV4-MULTICACHE W2 (#1973) — the runner refuses what it cannot carry ── +// +// The selection loop above this comment's subject (`runner.cpp`, the +// full_attn/gdn resolution) has exactly two arms and no `else`, and the +// allocation loop keys on `!is_gdn` when the model has no recurrent group. So +// before this row a published group of any other kind produced NO buffer and NO +// message. MEASURED on the pre-fix binary with a throwaway probe over +// `MakeFaOnlyKvConfig` plus a `kSlidingWindowMla` group and a second +// `kMlaAttention` group: the runner CONSTRUCTED, reported +// `full_attn_group_id = 0`, `gdn_group_id = -1`, `attn_kv().size() = 4` (one +// buffer per HIDDEN LAYER, all sized from group 0's `fa_page_size_bytes = 1024`) +// and allocated 0 bytes for the group whose own `page_size_bytes()` is 37440. +// A silently short KV allocation is a wrong-tokens failure, not a crash. +// +// These cases are the gate for the refusal that replaces that silence, and the +// tolerated-shape case beside them is its byte-neutrality contract: every model +// shipping today publishes exactly the groups this runner consumes. +TEST_CASE("runner: a published KV group it cannot allocate is REFUSED by name") { + const HfConfig c = MakeConfig(); + const Qwen3_5MoeWeights w = MakeWeights(c); + // One argument, so the doctest macros below do not split on the ctor commas. + const auto construct = [&](const KVCacheConfig& kvc) { + GPUModelRunner runner(c, w, kvc, Q(), /*max_num_reqs=*/8, kMaxModelLen, + /*max_num_batched_tokens=*/64); + (void)runner; + }; + const auto refusal_message = [&](const KVCacheConfig& kvc) { + try { + construct(kvc); + } catch (const std::runtime_error& e) { + return std::string(e.what()); + } + return std::string(""); + }; + + // The exact spec class DeepSeek-V4 publishes for 105 of its 167 entries. + SUBCASE("a kSlidingWindowMla group is named and refused") { + KVCacheConfig kv = MakeFaOnlyKvConfig(c); + kv.kv_cache_groups.emplace_back( + std::vector{"model.layers.0.attn.swa_cache"}, + std::make_shared( + /*block_size=*/64, /*num_kv_heads=*/1, /*head_size=*/512, + DType::kI8, /*sliding_window=*/128, + /*cache_dtype_str=*/std::string("fp8_ds_mla"), /*alignment=*/576, + /*compress_ratio=*/1, + /*model_version=*/std::string("deepseek_v4"))); + CHECK_THROWS_AS(construct(kv), std::runtime_error); + const std::string msg = refusal_message(kv); + // It names HOW MANY, WHICH KIND, WHICH LAYER and WHAT IT WOULD HAVE COST. + CHECK(msg.find("1 published KV cache group(s)") != std::string::npos); + CHECK(msg.find("group 1") != std::string::npos); + CHECK(msg.find("kSlidingWindowMla") != std::string::npos); + CHECK(msg.find("model.layers.0.attn.swa_cache") != std::string::npos); + CHECK(msg.find("page_size_bytes=37440") != std::string::npos); + } + + // The other silently-passed-over shape: a SECOND kMlaAttention group, which + // the `full_attn_group_id_ < 0` guard skips. DeepSeek-V4 publishes three. + SUBCASE("a second kMlaAttention group is named and refused") { + KVCacheConfig kv = MakeFaOnlyKvConfig(c); + kv.kv_cache_groups.emplace_back( + std::vector{"model.layers.2.attn"}, + std::make_shared(kBlockSize, /*head_size=*/512, + DType::kI8)); + CHECK_THROWS_AS(construct(kv), std::runtime_error); + const std::string msg = refusal_message(kv); + CHECK(msg.find("kMlaAttention") != std::string::npos); + CHECK(msg.find("model.layers.2.attn") != std::string::npos); + } + + // A plain SlidingWindowSpec and a ChunkedLocalAttentionSpec matched no arm + // either. No registry builds one today, which is exactly why nobody noticed. + SUBCASE("a kSlidingWindow group is named and refused") { + KVCacheConfig kv = MakeFaOnlyKvConfig(c); + kv.kv_cache_groups.emplace_back( + std::vector{"swa0"}, + std::make_shared( + kBlockSize, /*num_kv_heads=*/2, /*head_size=*/8, DType::kBF16, + /*sliding_window=*/4)); + CHECK_THROWS_AS(construct(kv), std::runtime_error); + CHECK(refusal_message(kv).find("kSlidingWindow ") != std::string::npos); + } + SUBCASE("a kChunkedLocalAttention group is named and refused") { + KVCacheConfig kv = MakeFaOnlyKvConfig(c); + kv.kv_cache_groups.emplace_back( + std::vector{"chunk0"}, + std::make_shared( + kBlockSize, /*num_kv_heads=*/2, /*head_size=*/8, DType::kBF16, + /*attention_chunk_size=*/4)); + CHECK_THROWS_AS(construct(kv), std::runtime_error); + CHECK(refusal_message(kv).find("kChunkedLocalAttention") != + std::string::npos); + } + + // EVERY unallocated group is named, not just the first — a refusal that + // stopped at the first one would understate a seven-group topology as one. + SUBCASE("all unallocated groups are named together") { + KVCacheConfig kv = MakeFaOnlyKvConfig(c); + kv.kv_cache_groups.emplace_back( + std::vector{"model.layers.0.attn.swa_cache"}, + std::make_shared( + /*block_size=*/64, /*num_kv_heads=*/1, /*head_size=*/512, + DType::kI8, /*sliding_window=*/128, + /*cache_dtype_str=*/std::string("fp8_ds_mla"), /*alignment=*/576, + /*compress_ratio=*/1, + /*model_version=*/std::string("deepseek_v4"))); + kv.kv_cache_groups.emplace_back( + std::vector{"model.layers.2.attn"}, + std::make_shared(kBlockSize, /*head_size=*/512, + DType::kI8)); + const std::string msg = refusal_message(kv); + CHECK(msg.find("2 published KV cache group(s)") != std::string::npos); + CHECK(msg.find("kSlidingWindowMla") != std::string::npos); + CHECK(msg.find("kMlaAttention") != std::string::npos); + } +} + +// BYTE-NEUTRALITY. The four group shapes every model in the tree publishes +// today still construct, so the refusal above cannot fire for any of them. The +// draft slot is tolerated on its KIND rather than on `spec_on()`, mirroring the +// draft-KV allocation loop's own predicate; that is deliberate and is listed +// under `## Owed` against W3 in `.agents/specs/kv-dsv4-multicache.md`. +TEST_CASE("runner: the group shapes shipped today still construct") { + const HfConfig c = MakeConfig(); + const Qwen3_5MoeWeights w = MakeWeights(c); + const int Hkv = static_cast(c.num_key_value_heads); + const int Dh = static_cast(c.head_dim); + + SUBCASE("one full-attention group (dense Qwen3)") { + GPUModelRunner runner(c, w, MakeFaOnlyKvConfig(c), Q(), 8, kMaxModelLen, 64); + CHECK(runner.full_attn_group_id() == 0); + CHECK(runner.gdn_group_id() == -1); + } + SUBCASE("one MLA group (every MLA model in the tree)") { + KVCacheConfig kv; + kv.num_blocks = kNumBlocks; + kv.kv_cache_groups.emplace_back( + std::vector{"mla"}, + std::make_shared( + kBlockSize, /*head_size=*/576, vllm::v1::ResolveKvCacheDType())); + GPUModelRunner runner(c, w, kv, Q(), 8, kMaxModelLen, 64); + CHECK(runner.full_attn_group_id() == 0); + } + SUBCASE("full-attention + recurrent (the hybrid gate models)") { + GPUModelRunner runner(c, w, MakeKvConfig(c), Q(), 8, kMaxModelLen, 64); + CHECK(runner.full_attn_group_id() == 0); + CHECK(runner.gdn_group_id() == 1); + } + SUBCASE("full-attention + recurrent + fa_draft (num_spec>0)") { + KVCacheConfig kv = MakeKvConfig(c); + kv.kv_cache_groups.emplace_back( + std::vector{"fa_draft"}, + std::make_shared(kBlockSize, Hkv, Dh, + vllm::v1::ResolveKvCacheDType())); + GPUModelRunner runner(c, w, kv, Q(), 8, kMaxModelLen, 64); + CHECK(runner.full_attn_group_id() == 0); + CHECK(runner.gdn_group_id() == 1); + } +} From fcd4e171ee57e57f20acb912ffb6773f8382357c Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 12:55:27 +0000 Subject: [PATCH 3/5] record(ENG-RELEASE-WINDOWS): file the repository-wide windows-msvc-vulkan red found while gating W2 (#1979) Not this branch's defect and not this branch's fix. `windows-msvc-vulkan` stops on `test_openai_api_server.exe exited with status -1073740791` on this pull request and on #1975, #1969 and #1967, whose diffs share no file under src/vllm/entrypoints/. The job is PR-only and never runs on main, so there is no green reference commit to bisect against, which is why four authors could each meet this red without any of them being able to attribute it. It rides here rather than in its own record-only pull request because the alternative to filing it now is not filing it: AGENTS.md requires a bug found during other work to get an issue that names an owner, and this one is owned by ENG-RELEASE-WINDOWS rather than by the row that found it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-fable-5 [Claude Code] --- .agents/issue-index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index e78ad2600..5ea88c645 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -743,3 +743,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1960](https://github.com/mudler/vllm.cpp/issues/1960) | `KV-DSV4-MULTICACHE` | **`SlidingWindowMLASpec` is a declared enumerator with no struct behind it, and `MLAAttentionSpec` carries none of the four DeepSeek-V4 fields, so 105 of V4's 167 cache entries cannot be sized at all.** W1 of [#1925](https://github.com/mudler/vllm.cpp/issues/1925). `KVCacheSpecKind::kSlidingWindowMla` is declared at `include/vllm/v1/kv_cache_interface.h:89` and the port's deferral list names the class as omitted (`:46-52`); it is the spec class of the SWA cache (43 entries, `vllm/v1/attention/backends/mla/sparse_swa.py:86-101`) and of both compressor-state populations (41 + 21, `vllm/models/deepseek_v4/compressor.py:188-200`). `MLAAttentionSpec` (`kv_cache_interface.h:242-261`) adds no fields over `FullAttentionSpec` where upstream carries `cache_dtype_str`, `alignment`, `compress_ratio` and `model_version` (`vllm/v1/kv_cache_interface.py:381-388`), so the compressed latent is sized `block_size` rows per page where upstream stores `block_size // compress_ratio`, and the 584-byte `fp8_ds_mla` token (`:396-405`) throws by name instead (`src/vllm/v1/kv_cache_interface.cpp:64-71`). `_apply_alignment_padding` (`:345-351`) has no twin, so no V4 page reaches its 576B/512B alignment. Pure allocation metadata: nothing constructs either spec outside tests, because publishing before W3 would allocate a silent subset (`src/vllm/v1/worker/gpu/runner.cpp:577-597` drops an unmatched group kind with no diagnostic). | bug | | [#1973](https://github.com/mudler/vllm.cpp/issues/1973) | `KV-DSV4-MULTICACHE` | **`MakeDeepseekV4KVCache` publishes ONE placeholder group for a model that needs 167 caches, and the runner would drop the real topology in silence.** W2 of [#1925](https://github.com/mudler/vllm.cpp/issues/1925), after W1 ([#1960](https://github.com/mudler/vllm.cpp/issues/1960)) landed the allocation metadata and published none of it. `deepseek_v4_registry.cpp:126-148` emits one `"mla"` group at `head_size = head_dim + qk_rope_head_dim = 576` and calls itself a STUB; upstream at the pin `5559679229bc961848b121ccdeaa8fa5d79bec98` publishes **167 entries in seven groups** across DeepSeek-V4-Flash's 43 layers -- 21 C4A + 20 C128A compressed latents and 21 indexer key caches as `MLAAttentionSpec` (`vllm/models/deepseek_v4/attention.py:631-645`, `:669-684`), 43 SWA caches (`vllm/v1/attention/backends/mla/sparse_swa.py:86-101`) and 21 + 21 + 20 compressor states (`vllm/models/deepseek_v4/compressor.py:188-200`) as `SlidingWindowMLASpec`, at block sizes 256, 64, 4 and 8. **The factory is REACHED from production** -- it is `ModelFactory::make_kv_cache` (`:120`), called from `LoadedEngine`'s constructor through `MakeKVCacheResolved`/`MakeKVCacheMaybeSpec` (`src/vllm/entrypoints/model_loader.cpp:1394-1404`, `:1681`) -- so what it publishes reaches `GPUModelRunner::initialize_kv_cache` unfiltered, and that loop (`src/vllm/v1/worker/gpu/runner.cpp:577-597`) has two arms and no `else`: `kSlidingWindowMla` matches nothing and a SECOND `kMlaAttention` group is passed over by the `full_attn_group_id_ < 0` guard, with no diagnostic either way. **Worse than one dropped group, and this is what W1 could not see from the selection loop alone:** `membership_by_name` is set only inside `if (has_mamba_group)` (`runner.cpp:820-845`), DeepSeek-V4 has no Mamba group, so the allocation loop falls into `is_full_attn = !is_gdn` and allocates ONE buffer per HIDDEN LAYER sized from the target group's page -- 43 buffers of one page for a model needing 167 of seven. A silently short KV allocation is a wrong-tokens failure, not a crash. FIXED by publishing the seven groups WITH a `VT_CHECK` refusal naming any published group the runner does not allocate, which is `AGENTS.md`'s "refuse an unimplemented arm with a message that names the missing part" rather than a new policy. **Consequence stated rather than implied: DeepSeek-V4 on the server path now refuses at engine construction instead of running.** What it loses is a full-recompute path with no decode step -- `Forward`/`ForwardDevice` discard `attn_kv` (`src/vllm/model_executor/models/deepseek_v4.cpp:2886-2887`, `:2959-2960`) -- and `examples/deepseek_v4_gen` does not go through the runner and is unaffected. Two further refusals are NOT in scope and are named so nobody takes them for this: `HybridKVCacheCoordinator`'s uniform-block-size `assert`, inert under `NDEBUG` (`src/vllm/v1/core/kv_cache_coordinator.cpp:340-346`, W4), and the missing third `ModelForwardInput` channel (`include/vllm/model_executor/models/model_registry.h:303-315`, W3). Nothing published is consumed; reachability is owed to W3 and W5 under `## Owed` in [kv-dsv4-multicache.md](specs/kv-dsv4-multicache.md) | bug | | [#1974](https://github.com/mudler/vllm.cpp/issues/1974) | `KV-DSV4-MULTICACHE` | **`spec_equal`'s `default:` arm returns false for `kMlaAttention` and `kSlidingWindowMla`, so two identical MLA specs never merge into one `SpecGroup`.** Observed by W1 ([#1960](https://github.com/mudler/vllm.cpp/issues/1960)) from a read of the switch and correctly left alone unverified; verified and fixed in flow with W2 ([#1973](https://github.com/mudler/vllm.cpp/issues/1973)). `spec_equal` (`src/vllm/v1/core/kv_cache_coordinator.cpp:17-67`) is our port of the frozen-dataclass `__eq__` and carries arms for `kFullAttention`, `kSlidingWindow`, `kChunkedLocalAttention` and `kMamba` only; `HybridKVCacheCoordinator::verify_and_split_kv_cache_groups` (`:353-376`) uses it to batch groups sharing a spec. Upstream cannot answer `false` there: every spec class is `@dataclass(frozen=True, kw_only=True)` (`vllm/v1/kv_cache_interface.py:380-381`, `:610-611`), so `__eq__` is generated over all fields and two identical `MLAAttentionSpec`s are equal. **Latent rather than observed at runtime, and that is stated rather than implied:** the coordinator needs two groups, and every MLA model in the tree publishes exactly one MLA group (`deepseek_v2_registry.cpp`, `deepseek_v4_registry.cpp`, `glm4_moe_lite_registry.cpp`, `kimi_k3_registry.cpp`, `kimi_linear_registry.cpp`, `minicpm3_registry.cpp`, `dots3_note.cpp`), so the helper is never called on two of them today. It stops being latent with #1973, which publishes three `kMlaAttention` and four `kSlidingWindowMla` groups for DeepSeek-V4. The cost of a wrong `false` is a second `SpecGroup`, so `find_longest_cache_hit` runs per group instead of per distinct spec and the eagle-bit propagation (`:395-402`) is computed over a different partition than upstream's; a wrong `true` would be the dangerous direction, and this is the safe one, which is why it is a mirror divergence rather than a live miscompute. Fixed by adding both arms, comparing `FullAttentionSpec`'s / `SlidingWindowSpec`'s fields plus the four DeepSeek-V4 fields `cache_dtype_str`, `alignment`, `compress_ratio` and `model_version`; `default:` keeps returning `false` for the kinds with no ported struct | bug | +| [#1979](https://github.com/mudler/vllm.cpp/issues/1979) | `ENG-RELEASE-WINDOWS` | **`windows-msvc-vulkan` is red on every open PR: `test_openai_api_server.exe` fail-fasts with `0xC0000409`, and the job has no `main` baseline to bisect against.** Found while gating [#1973](https://github.com/mudler/vllm.cpp/issues/1973) (PR [#1977](https://github.com/mudler/vllm.cpp/pull/1977)); PRE-EXISTING and not that row's defect. `scripts/build-windows-release.ps1:31` throws on `build-pr-windows-vulkan\tests\Release\test_openai_api_server.exe exited with status -1073740791`, which is `0xC0000409` `STATUS_STACK_BUFFER_OVERRUN`, the MSVC fail-fast code. It is a CRASH and not a doctest failure: the log carries no `[doctest]` summary and no assertion line for that binary, so nothing reports which case was running; the last lines before it are ordinary chat-completions request logs. MEASURED 2026-08-26 by reading four job logs directly -- [#1977](https://github.com/mudler/vllm.cpp/pull/1977) job 98175568298, [#1975](https://github.com/mudler/vllm.cpp/pull/1975) job 98169254199, [#1969](https://github.com/mudler/vllm.cpp/pull/1969) job 98152969359 and [#1967](https://github.com/mudler/vllm.cpp/pull/1967) job 98144800167 -- all four stopping on the SAME binary. Those four pull requests touch the KV-cache interface, the runner, the LTX-2.5 lane and the record surfaces, and the intersection of their diffs contains no file under `src/vllm/entrypoints/`, so the common factor is the job and not the change. **The second half of the defect is that `windows-msvc-cpu` and `windows-msvc-vulkan` are PR-ONLY jobs that never run on `main`**, so there is no green reference commit to bisect against and no signal for when this started -- which makes a repository-wide red something every author must independently re-attribute, and this issue exists so that stops happening a fifth time. NOT ESTABLISHED and stated rather than implied: no Windows host was available, so nothing was reproduced locally, the crashing case is not identified, the first bad commit is not identified and no cause is proposed; `httplib`'s `stop()` being a no-op on a never-listened server so `SO_REUSEPORT` splits the port (`.agents/environment.md`) is named as an ADJACENT shape to check first, NOT as an attribution. NOT FIXED IN FLOW: it needs a Windows host to reproduce, it sits in a server path unrelated to the KV-cache row that found it, and identifying the crashing case is its own investigation | bug | From 97ceb5674f959fa9539e32cbf92578d57079dc7e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 18:54:53 +0000 Subject: [PATCH 4/5] record(KV-DSV4-MULTICACHE): W2 -- say that G1's oracle side never ran, instead of leaving the row's own gate silent (#1973) `## Gates` declares G1 as the gate available at W2 and calls it "the gate that would have caught the stub". W2 landed without mentioning it. The topology is gated instead by doctest cases that pin every published entry as a literal, and the scaffold case comment calls itself G1 "on the half that needs no checkpoint, no GPU and no forward" -- but the expected values are READ from upstream construction sites and `RealConfig()` is a transcription of the artifact's `config.json`, so BOTH sides of that comparison are source inspection. That is real evidence against a stub and against drift. It is not the oracle execution AGENTS.md asks for, and the difference was nowhere in the record. AGENTS.md allows a gate to be pending on a named resource. It does not allow silence, and silence is what a reader of the spec would have taken for a run G1. So this states the result and names what it is pending on, measured on the host this wave was gated on rather than assumed: `vllm.models.deepseek_v4.attention` does not import here (`cbor2`, `pyzmq`, `msgspec` and `cloudpickle` absent, then `ImportError: cannot import name 'ALLOWED_LAYER_TYPES' from 'transformers.configuration_utils'` because the installed `transformers` predates the API the pin uses), and no DeepSeek-V4-Flash `config.json` is on the host cache or the NAS mount. No product code changes and no gate moves. The debt is owned by this row, falls due at the next wave with that environment, and is tracked under #1925. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-fable-5 [Claude Code] --- .agents/specs/kv-dsv4-multicache.md | 49 +++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/.agents/specs/kv-dsv4-multicache.md b/.agents/specs/kv-dsv4-multicache.md index c4e1f6b97..ecd923fdf 100644 --- a/.agents/specs/kv-dsv4-multicache.md +++ b/.agents/specs/kv-dsv4-multicache.md @@ -903,6 +903,45 @@ Deliberately not stated: how much faster a cached path would be. Nothing here measured anything, no GPU was leased for this spike, and an estimate would be the "a number quoted often becomes treated as measured" failure. +### W2 gate result — G1's oracle side was NOT run, and is PENDING on a named resource ([#1973](https://github.com/mudler/vllm.cpp/issues/1973)) + +`## Gates` declares **G1** as the gate available at W2 and calls it "the gate +that would have caught the stub". G1 has two sides: dump upstream's 167-entry +spec set by instantiating the four `get_kv_cache_spec` sites, and dump ours from +`MakeDeepseekV4KVCache`, then compare entry for entry. + +**Only our side was executed.** `tests/vllm/models/test_deepseek_v4_scaffold.cpp` +runs `MakeDeepseekV4KVCache` and pins every published entry as a LITERAL — group +count, layer names, `block_size`, `storage_block_size`, `head_size`, dtype, +`sliding_window`, `compress_ratio`, `alignment`, `cache_dtype_str`, +`model_version`, `kv_quant_mode`, `real_page_size_bytes`, `page_size_bytes` — +and its case comment calls itself G1 "on the half that needs no checkpoint, no +GPU and no forward". That framing is worth tightening rather than repeating: the +expected values are READ from upstream construction sites at the pin, and +`RealConfig()` is itself a transcription of the artifact's `config.json`. Both +sides of that comparison are therefore source inspection. It is real evidence +against a stub and against drift, and it is not the oracle execution `AGENTS.md` +asks for when it says to check every change in two ways. + +MEASURED on 2026-08-26, on the host this wave was gated on, rather than assumed: + +- **`vllm.models.deepseek_v4.attention` does not import here.** `cbor2`, + `pyzmq`, `msgspec` and `cloudpickle` are absent from the ambient environment + and were installed into a scratch virtual environment; the import then stops + at `ImportError: cannot import name 'ALLOWED_LAYER_TYPES' from + 'transformers.configuration_utils'`, because the installed `transformers` + predates the API the pin uses. Repairing that is a dependency resolution + against the pin, not a step this wave can take incidentally. +- **No DeepSeek-V4-Flash `config.json` is present.** Neither the host's + HuggingFace cache nor the NAS mount carries one, so the config G1 is a pure + function of is not on this box either. `docs/USAGE.md` names the artifact repo + `0xSero/deepseek-v4-flash-0731-spark` @ + `22f28d32b9b29b4352eaa380ff8c2c170b2847ab`. + +So G1's result at W2 is **PENDING on a named resource** — a vLLM environment at +the pin that imports `vllm.models.deepseek_v4`, plus that repo's `config.json`. +It is not satisfied, and it is not waived. Listed under `## Owed`. + ## Gates vLLM implements `DeepseekV4ForCausalLM` at the pin, so vLLM is the primary @@ -1031,6 +1070,16 @@ config parse and upstream's disagree about the layer partition (that would be a (`include/vllm/v1/kv_cache_dtype.h:87-90`). The second arm, and the MXFP4 68-byte indexer width (`attention.py:751-755`), are owed to **W5** together with the store path. +- **G1, this row's own topology gate, has NOT been run.** `## Gates` calls it + "the gate that would have caught the stub", and W2 published the topology + gated by literals read from upstream source rather than by a dump from the + running oracle. PENDING on a named resource — a vLLM environment at the pin + that imports `vllm.models.deepseek_v4`, plus the artifact repo's + `config.json`, neither of which is on the host this wave was gated on. The + blockers are measured in `### W2 gate result — G1's oracle side was NOT run, + and is PENDING on a named resource`. Owned by this row; falls due at the next wave that has + that environment, and W7 needs it anyway. Tracked under + [#1925](https://github.com/mudler/vllm.cpp/issues/1925). ## Evidence From ef548dfff4f21161e52728704fa0f49e8c8064d2 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 20:47:29 +0000 Subject: [PATCH 5/5] fix(KV-DSV4-MULTICACHE): W2 review -- enter the topology gate through the factory pointer, and arm the seven dead spec_equal comparisons (#1973, #1974) The W2 review found the gate did not measure the production seam. Three scaffold cases called `vllm::MakeDeepseekV4KVCache` as a free function, so the only thing holding `kDeepseekV4Factory.make_kv_cache` to the seven-group factory was `test_model_registry.cpp:68`'s `!= nullptr`. Non-null is not "points at the seven-group factory": the reviewer repointed the initializer at the pre-W2 placeholder -- one group named `"mla"`, head_size 576, `ResolveKvCacheDType()` -- and the scaffold stayed at 8/8 and 666 assertions with the whole CPU gate green. A merge resolution or a W3 refactor could put the placeholder back and nothing would say so. The three topology cases now go through `reg.factory->make_kv_cache`, the same pointer `LoadedEngine` dereferences via `MakeKVCacheResolved` -> `MakeKVCacheMaybeSpec` -> `ModelRegistry::MakeKVCache`. Re-running that mutation here: `ninja rc=0` at 4 steps, and the scaffold goes to **3 cases red, 5 assertions failed**. Read the CASE count -- a `REQUIRE` aborts its case, so the assertion line falls to 71 and only the case count tells the story. Seven of the eight `spec_equal` DeepSeek-V4 field comparisons were unreachable from any test, because `MakeMlaSpec` varied only `compress_ratio` and `MakeSlidingMlaSpec` varied nothing. Both helpers now take all four fields, and two subcased test cases separate a pair on ONE field at a time. `alignment` needs care to isolate: `ApplyAlignmentPadding` writes `page_size_padded` whenever the rounding moves the number, and `page_size_padded` is compared by an inherited line older than these four fields, so aligning to the real page size -- which rounds to itself -- leaves `alignment` the only difference, and the subcase asserts padding stayed unset on both sides. Re-running the reviewer's mutations: deleting `cache_dtype_str`, `alignment` and `model_version` from the `kMlaAttention` arm is `ninja rc=0` at 3 steps and **1 case red, 3 assertions failed**; answering `true` for all four in the `kSlidingWindowMla` arm is `ninja rc=0` at 3 steps and **1 case red, 4 assertions failed**. Both were green before this change. The block size a default engine actually hits is now executable rather than derived. `EngineParams::block_size` is 32, `has_c128` is true for Flash, so `check_ratio_fits(128)` evaluates `32 % 128 != 0` and the factory's own `VT_CHECK` throws inside the `kv_cfg_` member initializer -- which precedes `runner_` in the `LoadedEngine` initializer list -- before any group is built and before `GPUModelRunner::initialize_kv_cache` is entered. So the runner's by-name group refusal is reachable for V4 only at `--block-size` 128 or 256, and a default run reads a message about the block size, not about the topology. The refusal case gates 32 alongside 16 and 192, and the pull request body says which message a user sees. Two upstream anchors were off. At the pin 5559679229bc961848b121ccdeaa8fa5d79bec98 `self.block_size = 64` is `sparse_swa.py:82`, not `:80` -- `:80` is the comment line above it -- and the string is unique in that file. `get_kv_cache_spec` runs `:87-102`, not `:86-101`; the second correction was not in the review's list and is called out here rather than folded in silently. Baselines, with the delta accounted: `test_deepseek_v4_scaffold` 8 cases and 666 -> **669** assertions (+2 for the factory non-null REQUIREs, +1 for the block-size-32 refusal); `test_kv_cache_coordinator` 19 -> **21** cases and 117 -> **142** assertions (+11 for the MLA case's three subcases plus two padding checks, +14 for the sliding case's four subcases plus two). `test_runner` 567 and `test_model_registry` 941 are unchanged, as a test-only repair plus two comment corrections should leave them. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-fable-5 [Claude Code] --- .../models/deepseek_v4_registry.cpp | 4 +- .../vllm/models/test_deepseek_v4_scaffold.cpp | 45 +++++-- tests/vllm/v1/test_kv_cache_coordinator.cpp | 118 +++++++++++++++++- 3 files changed, 150 insertions(+), 17 deletions(-) diff --git a/src/vllm/model_executor/models/deepseek_v4_registry.cpp b/src/vllm/model_executor/models/deepseek_v4_registry.cpp index 99bbc7732..c12c2b56e 100644 --- a/src/vllm/model_executor/models/deepseek_v4_registry.cpp +++ b/src/vllm/model_executor/models/deepseek_v4_registry.cpp @@ -148,7 +148,7 @@ v1::KVCacheConfig MakeDeepseekV4KVCache(const HfConfig& config, int block_size, // (c) `DeepseekV4SWACache.get_kv_cache_spec` — the sliding-window cache, // `SlidingWindowMLASpec`, on ALL 43 attention layers including the two // that have no MLA cache at all - // (`vllm/v1/attention/backends/mla/sparse_swa.py:86-101`, constructed + // (`vllm/v1/attention/backends/mla/sparse_swa.py:87-102`, constructed // unconditionally at `attention.py:315-321`). // (d) `CompressorStateCache.get_kv_cache_spec` — the compressor state, // `SlidingWindowMLASpec`, f32, once per compressor @@ -183,7 +183,7 @@ v1::KVCacheConfig MakeDeepseekV4KVCache(const HfConfig& config, int block_size, const std::string kCacheDtypeStr = "fp8_ds_mla"; const std::string kModelVersion = "deepseek_v4"; constexpr int kAlignment = 576; // `sparse_swa.py:99`, `attention.py:642` - constexpr int kSwaBlockSize = 64; // `sparse_swa.py:80`, fixed by tensor sharing + constexpr int kSwaBlockSize = 64; // `sparse_swa.py:82`, fixed by tensor sharing // 1-byte storage, mirroring upstream's `torch.uint8`. `SizeOf(kI8) == 1` is // what the indexer's element formula needs; the 584-byte branch does not read // the dtype at all. diff --git a/tests/vllm/models/test_deepseek_v4_scaffold.cpp b/tests/vllm/models/test_deepseek_v4_scaffold.cpp index 21aa43842..8669b5f71 100644 --- a/tests/vllm/models/test_deepseek_v4_scaffold.cpp +++ b/tests/vllm/models/test_deepseek_v4_scaffold.cpp @@ -75,6 +75,22 @@ HfConfig RealConfig() { }; return c; } + +// THE PRODUCTION SEAM. An engine never calls `vllm::MakeDeepseekV4KVCache`. It +// reaches this topology as `LoadedEngine` -> `MakeKVCacheResolved` -> +// `MakeKVCacheMaybeSpec` (`entrypoints/model_loader.cpp:1394-1404`) -> +// `ModelRegistry::MakeKVCache` (`model_executor/models/model_registry.cpp:381-386`), +// which dereferences `registration().factory->make_kv_cache` — the pointer +// `kDeepseekV4Factory` sets at `deepseek_v4_registry.cpp:123`. Calling the free +// function proves the function builds seven groups; it does NOT prove that +// pointer still names it, and a merge resolution or a W3 refactor that repoints +// it would put the one 576-byte `"mla"` placeholder back with every gate green. +// So the topology gates below enter through the pointer, not around it. +vllm::v1::KVCacheConfig RegistryKVCache(int block_size, int num_blocks) { + const HfConfig cfg = RealConfig(); + const vllm::ModelRegistration& reg = ModelRegistry::Resolve(cfg); + return reg.factory->make_kv_cache(cfg, block_size, num_blocks); +} } // namespace TEST_CASE("deepseek-v4 scaffold: DeepseekV4ForCausalLM RESOLVES through the registry") { @@ -191,9 +207,14 @@ TEST_CASE("deepseek-v4 kv-cache: the published topology is upstream's 167 entrie // 256 is not a guess. `sparse_swa.py:76-83` and `compressor.py:174-178` both // derive the geometry from `[256//4, head_dim] = [64, head_dim]`, and the SWA // block size of 64 and the compressor block sizes of 4 and 8 only hold there. + const HfConfig cfg = RealConfig(); + const vllm::ModelRegistration& reg = ModelRegistry::Resolve(cfg); + REQUIRE(reg.factory != nullptr); + REQUIRE(reg.factory->make_kv_cache != nullptr); + // Not `&MakeDeepseekV4KVCache` by name: the point is the value the loader + // will dereference, so the gate dereferences the same one. const vllm::v1::KVCacheConfig kv = - vllm::MakeDeepseekV4KVCache(RealConfig(), /*block_size=*/256, - /*num_blocks=*/8); + RegistryKVCache(/*block_size=*/256, /*num_blocks=*/8); REQUIRE(kv.kv_cache_groups.size() == 7); CHECK(kv.num_blocks == 8); @@ -376,8 +397,7 @@ TEST_CASE("deepseek-v4 kv-cache: the published topology is upstream's 167 entrie // parse makes GroupLayerMask fall back wholesale rather than fail, so the // spelling is gated here where it is still cheap. TEST_CASE("deepseek-v4 kv-cache: names are the upstream module paths") { - const vllm::v1::KVCacheConfig kv = - vllm::MakeDeepseekV4KVCache(RealConfig(), 256, 8); + const vllm::v1::KVCacheConfig kv = RegistryKVCache(256, 8); for (const auto& g : kv.kv_cache_groups) { for (const std::string& n : g.layer_names) { CHECK(n.rfind("model.layers.", 0) == 0); @@ -402,10 +422,17 @@ TEST_CASE("deepseek-v4 kv-cache: names are the upstream module paths") { // the ratio produces a ZERO-byte page rather than a refusal. Upstream has no // such check because its comments derive the geometry at 256; ours refuses. TEST_CASE("deepseek-v4 kv-cache: a block_size that cannot hold a C128A row is refused") { - CHECK_THROWS_AS(vllm::MakeDeepseekV4KVCache(RealConfig(), 16, 8), - std::runtime_error); - CHECK_THROWS_AS(vllm::MakeDeepseekV4KVCache(RealConfig(), 192, 8), - std::runtime_error); + CHECK_THROWS_AS(RegistryKVCache(16, 8), std::runtime_error); + CHECK_THROWS_AS(RegistryKVCache(192, 8), std::runtime_error); + // 32 is `EngineParams::block_size`'s default (`entrypoints/model_loader.h:120`, + // and the `params.block_size > 0 ? params.block_size : 32` fallback in the + // `LoadedEngine` ctor). `has_c128` is true for Flash, so `32 % 128 != 0` and + // THIS refusal is what a default-configured engine hits — inside the + // `kv_cfg_` member initializer, which precedes `runner_` in the ctor's + // initializer list. The runner's own by-name group refusal (#1973) is + // therefore reachable for V4 only at `--block-size` 128 or 256, and the + // message a default run reads names the block size, not the topology. + CHECK_THROWS_AS(RegistryKVCache(32, 8), std::runtime_error); // 128 divides 128 exactly and gives storage_block_size 1: representable. - CHECK_NOTHROW(vllm::MakeDeepseekV4KVCache(RealConfig(), 128, 8)); + CHECK_NOTHROW(RegistryKVCache(128, 8)); } diff --git a/tests/vllm/v1/test_kv_cache_coordinator.cpp b/tests/vllm/v1/test_kv_cache_coordinator.cpp index 6762fa81c..52bafe8b9 100644 --- a/tests/vllm/v1/test_kv_cache_coordinator.cpp +++ b/tests/vllm/v1/test_kv_cache_coordinator.cpp @@ -22,6 +22,7 @@ #include #include #include +#include #include #include "vllm/sampling_params.h" @@ -64,20 +65,30 @@ std::shared_ptr MakeFullSpec() { /*head_size=*/1, DType::kF32); } -std::shared_ptr MakeMlaSpec(int compress_ratio = 1) { +// Every one of the four DeepSeek-V4 fields is settable here, because +// `spec_equal` compares all four and a helper that can only vary one leaves the +// other three comparisons unreachable from any test (#1974 review finding 2). +std::shared_ptr MakeMlaSpec( + int compress_ratio = 1, + std::optional cache_dtype_str = std::nullopt, + std::optional alignment = std::nullopt, + std::optional model_version = std::nullopt) { return std::make_shared( kBlockSize, /*head_size=*/1, DType::kF32, /*num_kv_heads=*/1, vllm::v1::KVQuantMode::kNone, /*page_size_padded=*/std::nullopt, - /*indexes_kv_by_block_stride=*/false, /*cache_dtype_str=*/std::nullopt, - /*alignment=*/std::nullopt, compress_ratio, - /*model_version=*/std::nullopt); + /*indexes_kv_by_block_stride=*/false, std::move(cache_dtype_str), + alignment, compress_ratio, std::move(model_version)); } std::shared_ptr MakeSlidingMlaSpec( - int sliding_window = 4) { + int sliding_window = 4, int compress_ratio = 1, + std::optional cache_dtype_str = std::nullopt, + std::optional alignment = std::nullopt, + std::optional model_version = std::nullopt) { return std::make_shared( kBlockSize, /*num_kv_heads=*/1, /*head_size=*/1, DType::kF32, - sliding_window); + sliding_window, std::move(cache_dtype_str), alignment, compress_ratio, + std::move(model_version)); } std::shared_ptr MakeMambaSpec() { @@ -342,6 +353,101 @@ TEST_CASE("HybridKVCacheCoordinator: MLA specs differing only in compress_ratio CHECK(hyb->attention_groups.size() == 3); } +// `compress_ratio` above is ONE of four. The reviewer of #1973 proved the other +// seven DeepSeek-V4 comparisons in `spec_equal` were unreachable: deleting +// `cache_dtype_str`, `alignment` and `model_version` from the `kMlaAttention` +// arm, and replacing all four in the `kSlidingWindowMla` arm with `true`, both +// left this file green. Upstream generates `__eq__` over every field of a +// frozen dataclass, so each of the four splits a group on its own, and +// upstream's own `alignment = 576 if uses_fp8_ds_mla_layout else 512` split +// (`sparse_swa.py:99` against `compressor.py:199`) is exactly this shape. +// +// `alignment` needs care to isolate. `ApplyAlignmentPadding` writes +// `page_size_padded` whenever the rounding moves the number, and +// `page_size_padded` is compared by an inherited line that predates these four +// fields — so a pair that differs in `alignment` AND in `page_size_padded` +// would go red with the `alignment` comparison deleted. Aligning to the real +// page size rounds to itself, so padding stays unset on both sides and +// `alignment` is the ONLY field that differs. The subcase asserts that. +TEST_CASE( + "HybridKVCacheCoordinator: MLA specs differing in ONE DeepSeek-V4 field do " + "NOT merge") { + const auto base = MakeMlaSpec(); + const auto page = static_cast(base->real_page_size_bytes()); + std::shared_ptr other; + SUBCASE("cache_dtype_str") { + other = MakeMlaSpec(/*compress_ratio=*/1, /*cache_dtype_str=*/"fp8_ds_mla"); + } + SUBCASE("alignment") { + other = MakeMlaSpec(/*compress_ratio=*/1, /*cache_dtype_str=*/std::nullopt, + /*alignment=*/page); + // The isolation itself: padding did not fire, so the inherited + // `page_size_padded` comparison cannot be what separates these two. + CHECK_FALSE(base->page_size_padded.has_value()); + CHECK_FALSE(other->page_size_padded.has_value()); + } + SUBCASE("model_version") { + other = MakeMlaSpec(/*compress_ratio=*/1, /*cache_dtype_str=*/std::nullopt, + /*alignment=*/std::nullopt, + /*model_version=*/"deepseek_v4"); + } + REQUIRE(other != nullptr); + + KVCacheConfig cfg; + cfg.num_blocks = 100; + cfg.kv_cache_groups.emplace_back(std::vector{"mla_a"}, base); + cfg.kv_cache_groups.emplace_back(std::vector{"full0"}, + MakeFullSpec()); + cfg.kv_cache_groups.emplace_back(std::vector{"mla_b"}, other); + auto coord = MakeCoordinator(std::move(cfg)); + auto* hyb = dynamic_cast(coord.get()); + REQUIRE(hyb != nullptr); + CHECK(hyb->attention_groups.size() == 3); +} + +// The same four fields on the sliding-window class. `MakeSlidingMlaSpec` never +// varied ANY of them before this case, so all four comparisons in the +// `kSlidingWindowMla` arm were dead. +TEST_CASE( + "HybridKVCacheCoordinator: sliding-window MLA specs differing in ONE " + "DeepSeek-V4 field do NOT merge") { + const auto base = MakeSlidingMlaSpec(); + const auto page = static_cast(base->real_page_size_bytes()); + std::shared_ptr other; + SUBCASE("cache_dtype_str") { + other = MakeSlidingMlaSpec(/*sliding_window=*/4, /*compress_ratio=*/1, + /*cache_dtype_str=*/"fp8_ds_mla"); + } + SUBCASE("alignment") { + other = MakeSlidingMlaSpec(/*sliding_window=*/4, /*compress_ratio=*/1, + /*cache_dtype_str=*/std::nullopt, + /*alignment=*/page); + CHECK_FALSE(base->page_size_padded.has_value()); + CHECK_FALSE(other->page_size_padded.has_value()); + } + SUBCASE("compress_ratio") { + other = MakeSlidingMlaSpec(/*sliding_window=*/4, /*compress_ratio=*/2); + } + SUBCASE("model_version") { + other = MakeSlidingMlaSpec(/*sliding_window=*/4, /*compress_ratio=*/1, + /*cache_dtype_str=*/std::nullopt, + /*alignment=*/std::nullopt, + /*model_version=*/"deepseek_v4"); + } + REQUIRE(other != nullptr); + + KVCacheConfig cfg; + cfg.num_blocks = 100; + cfg.kv_cache_groups.emplace_back(std::vector{"swamla_a"}, base); + cfg.kv_cache_groups.emplace_back(std::vector{"full0"}, + MakeFullSpec()); + cfg.kv_cache_groups.emplace_back(std::vector{"swamla_b"}, other); + auto coord = MakeCoordinator(std::move(cfg)); + auto* hyb = dynamic_cast(coord.get()); + REQUIRE(hyb != nullptr); + CHECK(hyb->attention_groups.size() == 3); +} + TEST_CASE("HybridKVCacheCoordinator: equal sliding-window MLA specs share one SpecGroup") { KVCacheConfig cfg; cfg.num_blocks = 100;