feat(KV-DSV4-MULTICACHE): W2 -- DeepSeek-V4's real seven-group cache topology, and a runner that names the group it cannot carry (#1973, #1974)

[localai-bot]

MakeDeepseekV4KVCache published ONE placeholder group called "mla", 576 bytes
wide, for a model that upstream gives 167 cache entries across 43 layers. It now
publishes all of them: 21 C4A and 20 C128A compressed latents plus 21 indexer key
caches as MLAAttentionSpec, and 43 SWA caches plus 21 + 21 + 20 compressor
states as SlidingWindowMLASpec, at block sizes 256, 64, 4 and 8, with the real
module paths as layer names.

That could not land alone. GPUModelRunner::initialize_kv_cache selects the
first full-attention or MLA group and one Mamba group and has no else, so a
kSlidingWindowMla group and a second kMlaAttention group were both dropped
with no buffer and no message. Measured on the pre-fix binary with a throwaway
probe: three published groups, constructs without throwing,
full_attn_group_id = 0, attn_kv().size() = 4 — one buffer per HIDDEN LAYER
off group 0's 1024-byte page — and zero bytes for the group whose own page is
37440. Publishing seven groups into that would have allocated a subset of the
topology in silence, which is a wrong-tokens failure rather than a crash. So the
runner now refuses, naming every group it does not allocate, its kind, its first
layer and what its page would have cost.

The consequence is that a DeepSeek-V4 engine refuses at KV-cache construction
instead of running. What it loses is a path with no decode step in it: Forward
and ForwardDevice discard attn_kv and recompute the whole prefix per token.
examples/deepseek_v4_gen does not go through the runner and is unaffected.
docs/FEATURES.md is updated to say so.

Four upstream details this port had to get right and a careless one would not:
the published prefix is attn, not the self_attn every other architecture in
this tree uses; kv_quant_mode is passed on two of the four construction sites
and not the other two, so the SWA and latent specs carry FP8_PER_TENSOR and
must still reach the 584-byte branch before W1's quant guard throws; the indexer
and compressor specs carry no model_version and take the element formula; and
the indexer key width defaults to 132 rather than the MXFP4 68.

#1974 rides along: spec_equal's default: arm answered false for
kMlaAttention and kSlidingWindowMla, so two identical MLA specs never merged
into one SpecGroup. Upstream's frozen dataclasses cannot answer that. Latent
until this change published three MLA groups; fixed in flow with both arms
comparing the four DeepSeek-V4 fields.

Red before

test_deepseek_v4_scaffold.cpp:197: FATAL ERROR: REQUIRE( kv.kv_cache_groups.size() == 7 ) is NOT correct! values: REQUIRE( 1 == 7 )

test_runner.cpp:1631: ERROR: CHECK_THROWS_AS( construct(kv), std::runtime_error ) did NOT throw at all!

test_kv_cache_coordinator.cpp:318: FATAL ERROR: REQUIRE( hyb->attention_groups.size() == 2 ) is NOT correct! values: REQUIRE( 3 == 2 )

Build for the red run: ninja rc=0 at 6/6, 0 FAILED.

Green after

ninja rc=0 at 7/7, 0 FAILED. test_runner 22 cases / 567 assertions,
test_deepseek_v4_scaffold 8 / 666, test_kv_cache_coordinator 19 / 117 — all
0 failed.

Affected suites: 89 built (ninja rc=0 at 162/162, 0 FAILED) and run,
ctest rc=0, 100% passed. 25 of the 89 report ZERO doctest assertions —
they are checkpoint-gated skips wearing a pass, and test_qwen35_paged_engine
did not run at all (Skipped, no checkpoint). Stated rather than implied: the
64 suites that carry assertions are the evidence, led by
test_dots3_note_scaffold 110818, test_nemotron_h_scaffold 38308,
test_parser_engine_assembly 5038, test_nemotron_h_paged_forward 3269,
test_model_registry 941, test_llm_engine 493, test_runner 567,
test_kimi_linear_paged 206, test_dflash2_runner_reach 144,
test_mtp_depth 123. Full ctest was NOT run: the disk stood at 95-96% used
and this box hit ENOSPC once during this work.

Mutations

Ten, each rebuilt from source, each verified to have LANDED before running, each
restored byte-for-byte with sha256 verified after.

mutation	ninja	run	verdict
refusal deleted (VT_CHECK always true)	rc=0, 3 steps	1 case red, 16 assertions failed	RED
kSlidingWindowMla added to the tolerated kinds	rc=0, 3 steps	1 case red, 8 assertions failed	RED
refusal stops at the FIRST unallocated group	rc=0, 3 steps	1 case red, 2 assertions failed	RED
published prefix is .self_attn instead of .attn	rc=0, 4 steps	2 cases red, 344 assertions failed	RED
latent head_size is head_dim + rope (the old placeholder 576)	rc=0, 3 steps	1 case red, 3 assertions failed	RED
SWA cache only on the compressed layers	rc=0, 3 steps	1 case red, 3 assertions failed	RED
alignment 512 instead of 576	rc=0, 3 steps	1 case red, 10 assertions failed	RED
block_size refusal disabled	rc=0, 4 steps	1 case red, 2 assertions failed	RED
spec_equal MLA arm ignores compress_ratio	rc=0, 4 steps	1 case red, 1 assertion failed	RED
spec_equal kSlidingWindowMla arm answers false for identical specs	rc=0, 4 steps	1 case red, 1 assertion failed	RED

Two of these were first written in forms that FAILED TO BUILD (rc=1) — a build
failure re-runs the previous binary and reads as a pass. They were caught by
recording ninja's exit code, rewritten and rerun.

Byte-neutrality

No spec class and no page formula changes, so no existing page size can move.
What could move is the runner's behaviour for every model and spec_equal's
partition, and both are pinned directly: test_runner asserts that all four
group shapes shipped today still construct (one full-attention; one MLA; full-
attention + recurrent; full-attention + recurrent + fa_draft), and every KV
factory in the tree publishes one group or two (kimi_linear: MLA + KDA), each
of them a tolerated shape. The fa+gdn shape is executed by
test_nemotron_h_paged_forward (3269) and test_kimi_linear_paged (206); the
fa_draft slot by test_dflash2_runner_reach (144) and test_mtp_depth (123).

Nothing lands dead

The published topology is UNREACHED and this PR does not claim otherwise. No
runner allocates any of the seven groups and DeepseekV4Model::Forward still
discards attn_kv. What IS reached from a production entry point is the refusal:
MakeDeepseekV4KVCache is ModelFactory::make_kv_cache, reached from
LoadedEngine's constructor through MakeKVCacheResolved and
ModelRegistry::MakeKVCache, and the runner check sits on
GPUModelRunner::initialize_kv_cache, which every engine crosses — the first
mutation deletes it and the gate goes red.

The wiring is owed to W3 (the runner carrying more than one attention group
and more than one cache per layer) and the consumption to W5, both tracked
under #1925 and both listed under ## Owed in
.agents/specs/kv-dsv4-multicache.md, together with two more items this wave
adds there: only the fp8_ds_mla arm of the topology is published (our
ParseCacheDType refuses the string by name and the factory signature carries no
cache dtype), and the draft slot is tolerated on its KIND rather than on
spec_on().

CI

gh pr checks reports nothing on this branch and that is not a pass. The
pull_request run on 0b01c8bab was CANCELLED by the push of the third commit,
and GitHub then created no run at all for fcd4e171e — not after the push, and
not after a close/reopen twenty minutes later, while other pull requests' runs
started normally in the same window. A cancelled run reads as failed and a run
that never starts is not a slow green, so the head was verified another way:

workflow_dispatch run 32975253972 on fcd4e171e, the exact tree of this
pull request. 14 success, 2 failure, 3 skipped:

• success: build-test-cpu, build-test-cpu-arm64, build-test-cpu-arm64-full,
  build-newest-gcc, build-test-vulkan, cuda-fat-build, cuda-arch-features,
  sanitize-cpu (address,undefined), sanitize-cpu (thread), macos-metal-mlx,
  device-leakage, vulkan-spirv-freshness, agent-record, last-gated-commit.
• skipped: pr-size, documentation-checkpoint, commit-protocol-tag — they
  carry pull_request-only conditions and cannot run on a dispatch. All three
  passed on the pull_request run at 0b01c8bab, and
  scripts/agent-pr-body.py --pr 1977 passes on the body as it stands.
• failure: windows-msvc-cpu and windows-msvc-vulkan, below.

An operator merging this needs the PR's own checks populated; any fresh push or
a re-run from the forge will do it.

Both windows-msvc-* jobs are RED and it is not this branch's defect.
test_openai_api_server.exe fail-fasts with -1073740791 (0xC0000409,
STATUS_STACK_BUFFER_OVERRUN) on both jobs here and on #1975, #1969 and #1967, whose diffs
share no file under src/vllm/entrypoints/; the job is PR-only and never runs
on main, so nobody has a green reference commit to bisect against. Filed as
#1979 against ENG-RELEASE-WINDOWS with the four job logs, and the index row
rides in the third commit here rather than in its own record-only pull request,
because the alternative to filing it now was not filing it. Every other check
that has settled is green.

Closes #1973
Closes #1974

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-fable-5 [Claude Code]