fix(FIX-KV-GROUP-LAYER-COUNT): resolve the placeholder KV group names, so the budget bounds the allocation again (#1963, #1966)

[localai-bot]

KVCacheGroupSpec::layer_names is upstream's per-layer name list, and upstream
bounds its KV allocation with max(len(group.layer_names) ...) over that same
list (vllm/v1/core/kv_cache_utils.py:1399), dividing the budget by the count
it multiplies the allocation by (:1005-1008, :1409-1416). The count is one
expression over one list, so the product is bounded by construction.

Thirty-three of our thirty-four registries publish a single PLACEHOLDER string
per group instead — {"fa"}, {"gdn"}, {"mla"}, {"kda"}, {"fa_draft"},
{"encoder"} — and two consumers read layer_names.size() as if it were a
layer count.

KVBytesPerBlock divides an absolute --kv-cache-memory budget by ONE layer's
page while GPUModelRunner::initialize_kv_cache allocates one buffer per
layer. Measured on dgx:gpu0 before this change: --kv-cache-memory 1073741824 logged page_size_bytes=131072 num_blocks=4096, and 16
full-attention layers plus the draft layer make that 8.5 GiB of buffers for a
1 GiB budget (#1963). recurrent_state_bytes reports 0.90 GiB against a
43.40 GiB allocation at --max-num-seqs 32, k=8, so the #371 guard — which
exists to turn an OOM reboot into a refusal — does not fire on the only family
it can fire for (#1966). Together they account for the 98.2 GiB the watchdog
saw at --kv-cache-memory 6GiB: 51.00 GiB of paged pool plus 43.40 GiB of
recurrent state, 94.40 GiB of the 100.6 GiB the watchdog measured.

ResolveKVCacheGroupLayerNames rewrites the placeholders into real per-layer
names, and LoadedEngine::MakeKVCacheMaybeSpec calls it. That is the single
funnel every architecture returns through — the speculative branch and
ModelRegistry::MakeKVCache both — and MakeKVCacheResolved passes the probe
and the resized config through it, so one call site reaches all thirty-four
registries.

The classification is the runner's own predicate, not a second derivation of
the model's shape: a layer is recurrent iff the config has a Mamba group and
layer_types[l] == "linear_attention"; the target attention group is the first
non-eagle attention group and covers every other layer; a second attention
group is the speculative draft head, one layer at index num_hidden_layers; a
third gets an empty list, because the runner allocates no buffer for it.
Reproducing the allocator's predicate is what makes the accounting unable to
disagree with the allocation.

NemotronHForCausalLM already publishes real names and is left alone: one
resolvable name anywhere and the resolver returns untouched. Its layer_types
is empty and its MoE blocks cache nothing, so the fallback would re-introduce
the 52-against-6 mis-classification #810 removed. LayerIndexOfName moves out
of the runner's anonymous namespace to vllm::v1::KVCacheLayerIndexOfName so
both sides parse a name with one function.

GPUModelRunner::kv_cache_allocated_bytes() and its paged half sum the size
every CacheBuffer was constructed with, so the gate compares the sizing
arithmetic against what the allocator DID rather than against a second copy of
the same formula.

Six cases, driven from MakeQwen3_5KVCacheSpec through the LoadedEngine
constructor rather than from a hand-built KVCacheGroupSpec, on a 6-layer
hybrid with TWO full-attention and FOUR GDN layers. The 4-layer config that
file already had has exactly one full-attention layer, which is the config in
which this bug is invisible — and tests/vllm/v1/test_kv_cache_interface.cpp
hands the function KVCacheGroupSpec{{"layer1", "layer2"}, ref}, a shape no
registry emits, which is why the pre-existing unit coverage could not see it.

With the resolver call site deleted in a scratch copy:

CHECK( allocated <= kv_cache_memory_bytes )   2097152 <= 1048576
CHECK( KVBytesPerBlock * num_blocks == ... )  1048576 == 2097152
CHECK( groups[0].layer_names.size() == 2 )    1 == 2
CHECK( groups[1].layer_names.size() == 4 )    1 == 4
CHECK( recurrent_state_bytes(cfg,4) == ... )  4992 == 19968

Exactly 2x on the paged half and 4x on the recurrent half, which is the defect
in bytes. Restored: 21 of 21 assertions pass and 627 of 627 ctest cases.

--kv-cache-memory now buys as many times fewer blocks as the model has
attention layers, because those blocks were never inside the budget.
docs/USAGE.md says so beside the flag, and its 8589934592 example becomes
honest rather than a footgun; docs/FEATURES.md corrects the "group-aware
divisor" claim, which named the wrong instrument.

Merged with KV-GDN-STATE-BUDGET (#1999), which landed first

Five files overlap; four three-way-merged and one conflicted. Every one was
resolved by the AGENTS.md ## Records rule rather than by accepting the
automatic merge: take origin/main's complete file, prove it byte-identical,
re-apply this row's scoped edit at an anchor asserted unique, then confirm
git diff origin/main -- <file> carries only this row's lines. The index was
union-appended and checked by row-ID set difference (729 base + 2 ours + 1
theirs = 732; 0 lost, 0 invented, 0 duplicated). ResolveMaxNumSeqs and every
other line #1999 added is byte-identical to origin/main in this branch.

Case 4 changed, and it is a correctness change rather than a textual one. It
asserted recurrent_state_bytes(cfg, params.max_num_seqs) while #1999 makes the
constructor hand the runner the RESOLVED concurrency. The two agree only while
ResolveMaxNumSeqs does not clamp — true here by a 64x margin — so it now reads
eng.max_num_seqs(), which is right by construction.

The first attempt at the conflicted file passed every static check and did not
compile. Taking the two conflict sides verbatim looked right: git had hoisted
the closing } both blocks end with out of the conflict region as shared
trailing context, so each side arrived one brace short. Marker count zero,
TEST_CASE names unique, identifiers unique, includes unique, git diff clean
and purely additive — six agreeing instruments, none measuring whether the file
parsed. The compiler was, in ten lines. The check added for it is a brace
balance against the pre-merge file, and the real remedy is the ## Records
procedure above, which never reconstructs a block from a conflict region.

The two fixes compose; they do not fight

ComputeHybridKvBudget never reads layer_names. The only input of its
arithmetic this row moves is kv_cfg.num_blocks — and upstream's num_blocks
is a PER-LAYER count (kv_cache_utils.py:1008 divides by num_layers), which
is the meaning its unification against one attention page assumes. Before this
row the byte-budget path handed it a count inflated by the layer count, so its
clamp was too permissive. Feeding it a truthful pool is what makes its seat
count correct.

Worked through for the device run below: unified_block_tokens = 32 x
ceil(3371008 / 131072) = 832, unified_num_blocks = 481 x 32 / 832 = 18,
slots_per_seq = 9, so 2 seats and max_num_seqs clamps 32 -> 2; the
recurrent allocation is 3371008 x 48 x 18 = 2.71 GiB. Whole-engine KV
3.71 GiB against the base tree's 51.90 GiB at the same flag. That arithmetic
is checkable against #1999's own output rather than against itself: its engine
prints The KV pool (3072 blocks) holds 118 unified pages of 832 tokens, and
3072 x 32 / 832 = 118. The formula reproduces both numbers.

Consequence worth stating because it looks like a regression and is not: at a
fixed --kv-cache-memory the seat count now falls by the same factor the pool
does, 8.5x on this spec-on launch. To seat 32 sequences at k=8 the budget must
be >= 16684941312 (15.54 GiB). --num-blocks is unaffected and always was —
ResolveNumBlocks arm 1 returns it verbatim, and only the byte-budget path
converts differently.

Device confirmation, predicted before the run

At --kv-cache-memory 1073741824 on the 27B the [kv-alloc] line should read
num_blocks=481 rather than 4096, for 1071775744 B (0.998 GiB) paged against
the 1 GiB asked, plus 2.71 GiB recurrent on the integrated tree.

Closes #1963
Closes #1966

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [Claude Code]

[localai-bot]

DEVICE EVIDENCE — the prediction was stated first, and it is exact

Operator run on dgx:gpu0 (GB10), idle under an rc lease. Binary from integ4/1574 @ 3d137890f (= main + #1994 + #1997 + #1999 + #2000), artifact-gated before use: flash_fwd=1792, SpecDecodeFA2Bf16=1, CUDA FA2 compiled-arch manifest: [121a].

Launch: --kv-cache-memory 1073741824 --max-num-seqs 1 --max-model-len 8192 --no-enable-prefix-caching --speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}', VT_KV_ALLOC_LOG=1.

[kv-alloc] source=spec kind=0 block_size=32 num_kv_heads=4 head_size=256 dtype=2 page_size_bytes=131072 num_blocks=481

	base (measured earlier)	with #2000	predicted
num_blocks for a 1 GiB request	4096	481	481
paged bytes allocated	17 x 4096 x 131072 = 8.50 GiB	17 x 481 x 131072 = 0.998 GiB	0.998 GiB
overshoot against the flag	8.5x	1.00x	—

page_size_bytes=131072 is read off the log rather than assumed, which was the caveat this PR attached to its own prediction: it confirms the 17-page divisor (16 target full-attention layers + 1 draft) and therefore that the 8.5x multiple quoted in #1963 was correct. Had the value differed, that multiple would have been wrong too — the falsifier was stated in advance and did not fire.

We now allocate what the operator asks for.

Alongside this, the #371 guard repair is also confirmed on device

Separately measured on the same integrated binary at --num-blocks 7488 --max-num-seqs 32:

server: fatal: To serve 32 concurrent sequences, 43.40 GiB of recurrent (Mamba/GDN)
state is needed, which is larger than the available 38.70 GiB. Speculative decoding
widens that state to 9 snapshot slots per sequence (num_speculative_tokens=8 + 1),
so it costs 9x the non-speculative state. Reduce --max-num-seqs, lower
num_speculative_tokens, or run without speculative decoding.

The guard that computed 0.90 GiB before this row now computes 43.40 GiB — the 48x repair — and refuses by name rather than letting the box march to an OOM reboot, which is what it did earlier the same day. That is #1966 closed on hardware.

Note on merge order

main is now 2a42cb369 (#1999 merged), and this PR conflicts with it in five places including src/vllm/entrypoints/model_loader.cpp, where #1999 added ResolveMaxNumSeqs. That conflict was anticipated in this PR's own analysis, along with the fix for its case 4 — asserting against eng.max_num_seqs() rather than params.max_num_seqs, which is correct by construction rather than by margin now that a clamp exists.

[localai-bot]

Confirmed independently: the take-both resolution really does lose a brace

This PR warned that the rehearsed "take both blocks" resolution of tests/vllm/entrypoints/test_loaded_engine_dense.cpp passes every static check and does not compile, because git hoists the closing } that both blocks end with out of the conflict region as shared trailing context.

I made the same resolution on my measurement branch and pushed it. Measured on origin/integ4/1574:

164 open braces, 163 close, delta = 1, balanced = False

Exactly the described defect, in a tree that passed the six checks I ran at the time — 0 markers, 30 TEST_CASEs, 0 duplicate names, 0 repeated identifiers, 0 duplicate includes, purely additive diff. Six instruments agreed and none of them measured whether the file parsed.

It escaped notice only because my device build targets vllm-server, which does not compile that translation unit. So the #1574 ladder numbers and the kv-alloc verification taken from /usr/local/vcpp/build14 are unaffected — by luck, not by care. integ4/1574 should not be used as a starting point by anyone; the four PRs themselves are the source of truth.

The remedy stands as this PR states it: the §Records procedure — take the target-branch file whole, re-apply scoped edits at uniquely-asserted anchors — never reconstructs a block from a conflict region and so cannot lose a brace. That it produced a -0 deletions diff here is the evidence that it did not.

On the corrected recurrent figure

Accepted, and the falsifiability is what makes it credible: the formula reproduces the engine's own printed line (3072 blocks -> 118 unified pages of 832 tokens; 3072 x 32 / 832 = 118.15 -> 118) as well as predicting 2.71 GiB at --kv-cache-memory 1 GiB. That is a check against the engine rather than against itself.

So on the integrated tree the whole-engine KV at that flag is 3.71 GiB against the base tree's 51.90 GiB, and the guard's 43.40 GiB figure I measured belongs to the --num-blocks 7488 --max-num-seqs 32 launch, not to this one. To seat 32 sequences at k=8 via the byte flag now requires >= 15.54 GiB, which is the number an operator needs and which no previous version of this engine could have told them.