At ctx=32768 --max-num-seqs 32 our engine consumes ~108 GB during load and never serves; vLLM and SGLang both serve there

[localai-bot]

At the context ceiling both vLLM and SGLang reach on this box, our engine cannot load. It consumes ~108 GB of unified memory during load and is killed before serving a single request.

Measured

Box: dgx:gpu0 (GB10, 119 GB unified), idle, under an rc lease, 114 GB available at launch.
Build: integ2/1574 @ ba3dd7cb4 = main + #1929 + #1930 + #1932 + #1952, clean CUDA build for sm_121a with -DVLLM_CPP_CUTLASS_FETCH=ON. Artifact-gated before timing: flash_fwd=1792, SpecDecodeFA2Bf16=1, CUDA FA2 compiled-arch manifest: [121a].
Launch: --max-model-len 32768 --max-num-seqs 32 --kv-cache-memory 6GiB --no-enable-prefix-caching --speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'
Checkpoints: r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121 target, qwen3.8-27b-dflash2 draft.

Memory, sampled once per second by the watchdog:

07:38:10 avail 36 GiB
07:38:11 avail 21 GiB
07:38:12 avail 13 GiB
07:38:13 avail 10 GiB
07:38:14 avail  7 GiB   -> BREACH, container killed

From 114 GB to 7 GiB. The last 29 GB went in three seconds. Had the watchdog not fired, this is an OOM reboot — the same failure that took this box down for 5h45m earlier the same night.

Where it dies

The captured server log ends immediately after the draft loads and shares its embedding table:

vllm.cpp: DFlash draft loaded from safetensors /draft (k=8, taps=5, mask=248070, vocab=248320, shared head from the target safetensors shards)
vllm.cpp: DFlash draft embed SHARED with the target (one device copy, 2542796800 B saved)

So the collapse is after the embed dedup (#1946/#1952) has already saved its 2.54 GB, and before the server reports healthy. It is not the embedding table.

Why this matters

vLLM and SGLang both loaded and served at exactly these settings, and both were probed at 32768 surviving three sequential ~15,858-token prompts at 512 output tokens. Their ceilings are recorded as 32768. Ours does not reach the starting line, so the #1574 three-engine comparison currently has no ours row at the context both competitors reach.

Not yet attributed — candidates, in order

1. The DFlash2 draft context store, which has never run on a device. fix(SPEC-DFLASH2): a >4K-token prompt killed EngineCore, because the draft context store was 4096 slots no matter what max_model_len said (#1919) #1932 replaced a 4096-slot constant with a resolution from max_model_len capped by an 8 GiB aggregate budget, divided by max_num_reqs. Its own spec records that budget as gated on CPU arithmetic only. This run is the first device exercise of that path. At the published draft geometry (5 layers, Hkv 4, head_dim 256 ≈ 20 KiB per context token) an uncapped store would want 32768 x 32 x 20 KiB ≈ 21.5 GB — large, but not 108 GB on its own.
2. Something scaling with max_model_len x max_num_seqs outside the store and outside --kv-cache-memory, which was set explicitly to 6 GiB and therefore is not the KV cache.
3. DevicePool retention: the pool never returns a block to the driver and device_pool_cap_bytes is 0 (uncapped) on every platform, so any transient during load stays resident. That converts a spike into a floor but does not by itself create one.

Owed

• A per-allocation census during load (ReportLoadBytes / device_upload= deltas) to attribute the 108 GB, rather than the guesswork above.
• The ceiling probe stepping 32768 -> 16384 -> 8192 -> 4096 at --max-num-seqs 1, to separate the context term from the concurrency term. Running now; result to be attached.
• Whether fix(SPEC-DFLASH2): a >4K-token prompt killed EngineCore, because the draft context store was 4096 slots no matter what max_model_len said (#1919) #1932's 8 GiB aggregate is actually enforced on the device path.

Filed from an operator GPU session; the watchdog killed the container rather than letting the box reboot, so the box is healthy and the evidence above is complete.

[localai-bot]

It is the CONTEXT, not the concurrency

The ceiling probe was run to separate the two terms. It launches the same binary at the same --max-model-len 32768 but with --max-num-seqs 1 instead of 32.

It drove the box to the same place. Within ~4 minutes of === probing ctx=32768 === the host stopped answering ssh — ICMP still replied, but the connection timed out during banner exchange, which is what sshd does when it cannot allocate — and the rc controller lost contact with the device.

So the ~108 GB is not driven by max_num_seqs. One sequence at a 32768 context is enough. That rules out, or at least heavily de-weights, every candidate that scales with concurrency:

• the DFlash2 draft context store's aggregate term (budget / max_num_reqs) — at max_num_reqs = 1 the per-request slice is the whole 8 GiB budget, but the budget is the cap, so this cannot exceed 8 GiB unless the cap is not enforced on the device path;
• anything sized max_num_seqs x max_model_len;
• DevicePool retention across concurrent requests.

What survives as a candidate is something scaling with max_model_len alone, allocated per engine at load, outside --kv-cache-memory (which was pinned at 6 GiB in the c=32 run and is therefore excluded).

Suggested next probe

Step the context down with --max-num-seqs 1 fixed and find the largest that loads: 16384, 8192, 4096. If the memory scales linearly in max_model_len, the slope gives the per-context-token cost directly, and that number should identify the allocation. For reference, the published draft geometry (5 layers, Hkv 4, head_dim 256) is ~20 KiB per context token, so a single-sequence draft store at 32768 should be ~0.67 GB — three orders of magnitude below what was consumed.

Both engines we compare against load at this context: vLLM and SGLang each survived three sequential ~15,858-token prompts at 512 output tokens with --max-num-seqs 32.

[localai-bot]

Root cause: the KV block count is computed with a divisor that counts ONE layer per group, while the runner allocates one buffer PER layer

This is not a max_model_len term at all — which is why it reproduced at --max-num-seqs 1. Everything that genuinely scales with max_model_len on the load path sums to about 155 KB of host memory.

Three anchors, each verified by reading the tree at integ2/1574 @ ba3dd7cb4:

1. The registry publishes one PLACEHOLDER name per group. src/vllm/model_executor/models/qwen3_5_common.cpp:68 emits std::vector<std::string>{"fa"}, :75 emits {"gdn"}, :101 emits {"fa_draft"}. The runner states this itself at src/vllm/v1/worker/gpu/runner.cpp:834-841: "Every registry shipping today publishes a single PLACEHOLDER name per group."

2. The divisor weights by that name count. src/vllm/v1/kv_cache_interface.cpp:156-157:

bytes += attn->page_size_bytes() *
         static_cast<int64_t>(group.layer_names.size());

With one name per group that is page x 1 + page x 1, not page x 16 + page x 1.

3. The allocator allocates per LAYER. src/vllm/v1/worker/gpu/runner.cpp:974-977 pushes a CacheBuffer of num_blocks_ * l_page inside the per-layer loop, once for each of the 16 full-attention layers.

The arithmetic

block_size          = 32
page_size_bytes     = 32 * 4 * (256 + 256) * 2 = 131,072 B
KVBytesPerBlock     = 131,072 (fa) + 131,072 (fa_draft) = 262,144 B   <- 1 layer each
num_blocks          = 6,442,450,944 / 262,144 = 24,576

full-attention pool = 16 * 24,576 * 131,072 = 51,539,607,552 B = 48.00 GiB
draft pool          =  1 * 24,576 * 131,072 =  3,221,225,472 B =  3.00 GiB

We were asked for 6 GiB and allocated 51 GiB — 8.5x the flag.

The recurrent half is allocated from a separate axis that --kv-cache-memory never sees: gdn_state_slots_ = max_num_reqs * (num_spec + 1) (runner.cpp:555), one conv + one SSM buffer per GDN layer (:924-931). At k=8:

per slot per layer = 48*128*128*4 (f32 ssm) + (2*16*128 + 48*128)*(4-1+8)*2 (conv) = 3,371,008 B
--max-num-seqs 32 : 3,371,008 * 288 * 48 = 46,600,814,592 B = 43.40 GiB
--max-num-seqs 1  : 3,371,008 *   9 * 48 =  1,456,275,456 B =  1.36 GiB

--max-num-seqs	full-attn	draft	GDN state	total from a 6 GiB request
32	48.00 GiB	3.00 GiB	43.40 GiB	94.40 GiB
1	48.00 GiB	3.00 GiB	1.36 GiB	52.36 GiB

Plus ~21 GB of NVFP4 shards, that is ~115 GB on a 119 GB box — exactly the observed collapse. The --max-num-seqs 1 arm still asks 9x what was budgeted, which is why the ceiling probe wedged the host too.

What upstream does

vLLM divides by the layer count of the largest group and then allocates exactly that many tensors, so the total is bounded by construction:

• vllm/v1/core/kv_cache_utils.py:1399 — group_size = max(len(group.layer_names) for group in kv_cache_groups), and upstream's groups carry REAL per-layer names, not a placeholder
• :1405-1407 -> :1008 — num_blocks = int(available_memory // page_size // num_layers)
• :1409-1416 — allocates group_size tensors of page_size * num_blocks

For the same 6 GiB on this model upstream lands on 6442450944 // 131072 // 16 = 3072 blocks and allocates 6.00 GiB. It also pads the Mamba page up to the common page size (:1091-1131, _unify_hybrid_kv_cache_specs) so recurrent state is INSIDE the budget rather than beside it.

#1932's aggregate budget is NOT implicated

ResolveCtxStoreSizing (qwen3_dflash.cpp:1135) is one pure host function with one call site (runner.cpp:2772), so there is no separate device path. At max_num_reqs = 1 the cap does not even bind: round_down16(8GiB / 20480) = 419,424 slots against a want of 32,784, giving 0.63 GiB. At 32 it binds at 256 MiB/request, 8 GiB aggregate. Neither is material, and the store allocates lazily on first draft, after the point where the process died.

Confirming probe (owed, box is recovering)

VT_KV_ALLOC_LOG=1 ... 2>&1 | grep kv-alloc

prints from runner.cpp:768-780 BEFORE the allocation loop. Expect num_blocks=24576 for --kv-cache-memory 6442450944. If it reads 3072, this analysis is wrong.

Cheapest confirmation that also yields a serving engine with no rebuild: --num-blocks 3072 --max-num-seqs 1, which is arm 1 of ResolveNumBlocks (model_loader.cpp:1410-1412) and bypasses the divisor entirely.

[localai-bot]

CONFIRMED ON THE DEVICE — and the engine serves once the pool is sized correctly

The prediction was stated before the run, so it could not be fitted afterwards. With --kv-cache-memory 1073741824 (1 GiB) and --max-num-seqs 1:

• if the divisor counts 1 layer per group: 1 GiB / 262144 = 4096 blocks -> 16 * 4096 * 131072 = 8 GiB allocated
• if the divisor were correct (16 + 1 layers): 1 GiB / 2228224 = 481 blocks -> 0.9 GiB allocated

Measured, from VT_KV_ALLOC_LOG=1 (printed at src/vllm/v1/worker/gpu/runner.cpp:768-780, before the allocation loop):

[kv-alloc] source=spec kind=0 block_size=32 num_kv_heads=4 head_size=256 dtype=2 page_size_bytes=131072 num_blocks=4096

4096. We were asked for 1 GiB and sized a pool of 8 GiB — an 8x overshoot, exactly as the source analysis predicted. The root cause stands: KVBytesPerBlock (src/vllm/v1/kv_cache_interface.cpp:156-157) weights by group.layer_names.size(), which is 1 because MakeQwen3_5KVCacheSpec publishes one placeholder name per group (qwen3_5_common.cpp:68,75,101), while initialize_kv_cache allocates one CacheBuffer per real layer (runner.cpp:974-977).

The engine is fine; only the sizing was wrong

Same binary, same --max-model-len 32768, with the smaller request so the 8x overshoot stays survivable:

container srvb Up 2 minutes
avail 89 GB
curl /health -> HEALTHY
server: listening on http://0.0.0.0:8100 (model 'qwen38-27b', HTTP worker pool 5 fixed)

This is the first time this engine has served at ctx=32768 — the context both vLLM and SGLang were already measured at. Nothing about the model, the draft, or the speculative path was the obstacle.

Workaround in use for the #1574 ladder

--num-blocks 3072 (arm 1 of ResolveNumBlocks, model_loader.cpp:1410-1412) bypasses the divisor entirely and yields 16 * 3072 * 131072 = 6.00 GiB, the same KV budget vLLM and SGLang were given. The ours ladder is running on that now, so the three-engine comparison is like-for-like.

Note this does not bound the GDN recurrent state, which is allocated from a separate axis (gdn_state_slots_ = max_num_reqs * (num_spec + 1), runner.cpp:555) that no flag reaches — 43.40 GiB at --max-num-seqs 32, per #1966.

Fix shape

Mirror upstream and make it bounded by construction: divide by the real per-layer count of the largest group (vllm/v1/core/kv_cache_utils.py:1399 -> :1008 -> :1409-1416), and either publish real per-layer names from the registries or have every consumer take the layer count from the config rather than from layer_names.size(). Both KVBytesPerBlock and the #1966 guard read that same field, so a fix has to cover both.

A gate has to assert, on a real registry config, that the sum of what initialize_kv_cache allocates is bounded by what was budgeted. The existing unit test cannot catch this: tests/vllm/v1/test_kv_cache_interface.cpp:425-433 hands the function {"layer1","layer2"}, a shape no shipping registry emits.