DFlash2 speculative decoding serves only ONE sequence: at c=2 the draft context accumulation desynchronises and EngineCore stops

[localai-bot]

DFlash2 speculative decoding serves one sequence. At concurrency 2 the engine refuses on its own invariant and EngineCore stops. This is the blocking defect for the #1574 three-engine comparison: vLLM and SGLang were both measured at c=1..32 with the same DFlash2 draft at k=8, and we cannot produce a rung above c=1.

Measured

integ4/1574 @ 3d137890f (main + #1994 + #1997 + #1999 + #2000), artifact-gated (flash_fwd=1792, SpecDecodeFA2Bf16=1, FA2 manifest [121a]), on an idle leased GB10. --num-blocks 3744 --max-num-seqs 16 --max-model-len 8192 --no-enable-prefix-caching --speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}', via vllm bench serve --backend openai-chat, input 1024 / output 512:

rung	result
c=1	24.70 out tok/s, 75.40 total, TTFT mean 1363.78 ms / p99 1388.08, TPOT 37.90 ms, 8/8 ok
c=2	VOID — ok=1, failed=7

The c=2 failure:

engine-fatal: EngineCore busy loop threw: vt: propose_drafts_block: context position
discontinuity (accumulation out of sync with the target's committed positions)
at src/vllm/v1/worker/gpu/runner.cpp:2961

Every later request returns 500 [request submitted to a stopped AsyncLLM]. The ladder's failed==0 assertion recorded the rung as VOID and wrote no row, which is the harness working.

The invariant, and why it is right to assert

src/vllm/v1/worker/gpu/runner.cpp:2955-2963:

// Invariant: this step's first committed token sits at absolute position L
// (== current context length). A violation means the accumulation lost sync
// (the I5e async-input-combine bug class) — assert rather than corrupt.
const int64_t L = dflash_ctx_len_[static_cast<size_t>(i)];
VT_CHECK(step.positions[static_cast<size_t>(rows[0])] == L,
         "propose_drafts_block: context position discontinuity (accumulation "
         "out of sync with the target's committed positions)");

This is a guarded refusal, not a crash — the code deliberately stops rather than drafting from a desynchronised context, and the comment names the failure class it is defending against. The defect is that the accumulation desynchronises at all once more than one request is in flight.

The matching half is the capture gate at src/vllm/model_executor/models/qwen3_dflash.cpp:1577, which admits the capture-safe paged path only when P == 1. Above that the fallback re-materialises each request's context from the paged store every propose step, and dflash_ctx_len_[i] no longer tracks what the target committed for request i.

Relationship to the other rows

Distinct from all four fixes landed today, and it is what remains after them:

• At ctx=32768 --max-num-seqs 32 our engine consumes ~108 GB during load and never serves; vLLM and SGLang both serve there #1963 / The #371 recurrent-state OOM guard is 48x under and passes a config that allocates 43.4 GiB, because it counts placeholder layer names #1966 (divisor, guard) — repaired; the guard now refuses 32 seats by name where it once silently allowed a 94.40 GiB allocation.
• fix(KV-GDN-STATE-BUDGET): bound the recurrent-state pool by the KV budget, not by --max-num-seqs (#1983) #1999 (recurrent state in the KV budget) — verified on device: residency slope 1.22-1.50 GiB/seq to 0.0000, --max-num-seqs 28 from 0.23 to 17.37 tok/s.
• Attention and recurrent state are allocated from two separate pools, so c=32 at k=8 is unservable where vLLM and SGLang serve it #2007 (attention and state still allocated from two pools) — why c=32 at k=8 is unservable on this box.
• This issue — why any concurrency above 1 is unservable with speculation on, regardless of memory. At c=2 the box had ample headroom; the refusal is arithmetic, not capacity.

It is also the surviving candidate for the crash half of #1983: the same P > 1 region, reached under a configuration where the position accumulation desynchronises rather than the assertion catching it first.

What c=1 says about the rest

Worth recording, because it changes the priority. At c=1 on this build we are competitive: 24.70 out tok/s against vLLM's 24.36 and SGLang's 25.20, with TPOT 37.90 ms against SGLang's 37.93 — per-token decode at parity with the engine we are chasing. Caveats: ours ran at --max-model-len 8192 where the recorded competitor rows used 32768, and at --max-num-seqs 16 with an explicit 3744-block pool against their defaults, so this is a same-band result rather than a like-for-like one.

The gap to close is therefore concurrency, not per-token cost.

Owed

• A red-before test that drives two concurrent requests through propose_drafts_block and reproduces the discontinuity on CPU, so the fix is gated rather than measured.
• Whether the same configuration serves at c>=2 with --speculative-config omitted. If it does, that isolates the defect to the draft path and gives an interim comparison point.
• Reproduction is one command; the binary is at /usr/local/vcpp/build14 and the ladder at /usr/local/vcpp/ladder.sh.

[localai-bot]

Isolated: the engine handles concurrency correctly — the defect is the draft path alone

The obvious question this issue owed was whether concurrency works at all with the draft removed. It does.

Same binary (integ4/1574 @ 3d137890f, artifact-gated), same box, same pool (--num-blocks 3744 --max-num-seqs 16 --max-model-len 8192 --no-enable-prefix-caching). The only difference is --speculative-config omitted. Two concurrent requests, 852-token prompt, 32 output each:

health=UP (spec OFF)
  req0: ok completion=32 elapsed=127.6s
  req1: ok completion=32 elapsed=127.5s
  fatal_lines=0

configuration	c=2
--speculative-config dflash k=8	VOID — propose_drafts_block: context position discontinuity, EngineCore stops, later requests 500
no --speculative-config	both requests complete, zero discontinuity or engine-fatal lines

(The 127 s is cold-start CUDA graph capture on a fresh server — both requests were the first through it. It is not a throughput number and should not be read as one.)

What this narrows

Batching, scheduling, the paged KV cache, the block tables and the sampler all handle two concurrent sequences correctly. The failure is confined to the DFlash2 draft's per-request context accumulation — dflash_ctx_len_[i] against step.positions[rows[0]] at src/vllm/v1/worker/gpu/runner.cpp:2955-2963, and the P == 1 capture gate at src/vllm/model_executor/models/qwen3_dflash.cpp:1577.

So the fix does not need to touch the engine's concurrency machinery, and a change that did would be aimed at the wrong layer.

Consequence for the #1574 comparison

An ours row above c=1 is currently reachable only with speculation off, which is not the configuration vLLM and SGLang were measured in (both ran the same DFlash2 draft at k=8). A no-spec ladder would measure a different engine configuration and must not be published in the same table as their spec-on rows without saying so.

The c=1 row stands and is spec-on: 24.70 out tok/s, TPOT 37.90 ms, 8/8 ok, against vLLM 24.36 / 38.53 and SGLang 25.20 / 37.93.