Attention and recurrent state are allocated from two separate pools, so c=32 at k=8 is unservable where vLLM and SGLang serve it

[localai-bot]

#1999 mirrors upstream's page-unification arithmetic but the runner still allocates attention and recurrent state as separate pools. The consequence is measurable: on a GB10 we cannot serve 32 concurrent sequences at num_speculative_tokens=8, where vLLM and SGLang both can on the same model and draft.

Measured, on the integrated tree

integ4/1574 @ 3d137890f = main + #1994 + #1997 + #1999 + #2000, artifact-gated (flash_fwd=1792, FA2 manifest [121a]). --num-blocks 7488 --max-num-seqs 32 --max-model-len 8192 --speculative-config '{"method":"dflash","num_speculative_tokens":8}':

server: fatal: To serve 32 concurrent sequences, 43.40 GiB of recurrent (Mamba/GDN)
state is needed, which is larger than the available 38.70 GiB. Speculative decoding
widens that state to 9 snapshot slots per sequence (num_speculative_tokens=8 + 1),
so it costs 9x the non-speculative state. Reduce --max-num-seqs, lower
num_speculative_tokens, or run without speculative decoding.

That refusal is correct and is #1966 working — the same guard computed 0.90 GiB before that row repaired its layer count. This issue is not about the guard. It is about why the budget it checks against is too small.

The structural difference

We allocate four independent buffer sets, src/vllm/v1/worker/gpu/runner.cpp:

buffer	line	sized by
ssm_buf_	:942	gdn_state_slots_ = seats x (k+1), per GDN layer
conv_buf_	:946	same
full_attn_buf_	:992	num_blocks_, per full-attention layer
draft_attn_buf_	:1149	num_blocks_

--num-blocks / --kv-cache-memory bounds only the last two. The first two come out of whatever is left.

Upstream allocates one pool whose tensors are shared by layers from every group. vllm/v1/core/kv_cache_utils.py:1410-1413 builds shared_by across groups and :1409-1416 allocates exactly group_size tensors of page_size * num_blocks; _unify_hybrid_kv_cache_specs (:1089 and around) pads the mamba page so every group's page is the same size. Attention and recurrent state are therefore drawn from the same budgeted allocation, and the total is bounded by construction.

#1999 reproduces upstream's unification arithmetic to derive a seat count — correctly, and it fixed a real collapse (verified: residency slope 1.22-1.50 GiB/seq to 0.0000, --max-num-seqs 28 from 0.23 to 17.37 tok/s). But the allocation underneath it is still two pools, so the seat count it computes from a unified page and the memory the runner actually consumes are two different quantities. At --num-blocks 7488 the arithmetic says 32 seats while the physical requirement is 14.6 GiB of attention plus 41.3 GiB of state, and the guard correctly refuses.

Why this blocks the #1574 comparison

vLLM and SGLang were both measured at --max-num-seqs 32 on this box with the same 27B NVFP4 target and DFlash2 draft at k=8, and both served. Upstream's per-sequence recurrent cost is not cheaper than ours — same mamba_ssm_dtype from the checkpoint, same (1 + num_speculative_blocks) slots (vllm/v1/kv_cache_interface.py:718). Their advantage is entirely that the state comes out of one profiled pool rather than beside it.

So the three-engine ladder currently has vLLM and SGLang rows at c=32 and no ours row, for a reason that is architectural rather than a tuning choice.

Scope

Allocate attention and recurrent state from one budgeted pool, mirroring _unify_hybrid_kv_cache_specs and the shared_by allocation, so that a single flag bounds the total and the seat count derives from the same bytes the runner consumes.

Explicitly not in scope: the #371 guard (#1966, already repaired), the block divisor (#1963, repaired in #2000), and the seat clamp (#1999). Those three are correct and this row builds on them.

Owed evidence

A device run at --max-num-seqs 32, k=8, showing the engine serves where it now refuses, with the total bounded by the flag. Reproduction of the refusal above is one command; both binaries are on the box at /usr/local/vcpp/build14 (integrated) and /usr/local/vcpp/build13 (#1999 alone).