Prefix caching + DFlash2 kills the engine on the first request at c=1, which makes the SGLang-compat lpm scheduler unreachable

[localai-bot]

Enabling prefix caching kills the engine on the first request when DFlash2
speculative decode is on — at concurrency 1, where the identical binary serves 8/8
with prefix caching off.

engine-fatal: EngineCore busy loop threw: vt: propose_drafts_block: context position
  discontinuity (accumulation out of sync with the target's committed positions)
  at /src/src/vllm/v1/worker/gpu/runner.cpp:2959

Every subsequent request returns 500 [request submitted to a stopped AsyncLLM]. The
rung reads ok=0 failed=8.

Measured

3d895a202 (PR #2010), sm_121a, artifact-gated: vllm-server 101,151,128 B,
flash_fwd=1792, SpecDecodeFA2Bf16=1. dgx:gpu0 (GB10), idle under an rc lease.
--num-blocks 3744 --max-num-seqs 16 --max-model-len 8192, DFlash2 k=8,
vllm bench serve --dataset-name random --backend openai-chat, 1024 in / 512 out.

prefix caching	c=1	c=2	c=4
--no-enable-prefix-caching	OK 24.84 out tok/s, 8/8	OK 40.11, 8/8	OK 59.13, 16/16
--enable-prefix-caching --scheduling-policy lpm	VOID, ok=0 failed=8	not reached	not reached

illegal memory access 0, stream is capturing 0 — this is the position invariant
refusing, not #2028 and not #2029.

Mechanism

With APC on, a request whose prefix is served from cache has its committed positions
begin at the cached prefix length, not at 0. The DFlash2 draft context starts its
accounting at 0 for a new request, so the invariant comparing
step.positions[rows[0]] against the accumulated ctx_len fails on the first
propose. The draft path assumes positions start at zero; prefix caching breaks that
assumption.

Consequence: a shipped SGLang-compat capability is unreachable

ENG-SGLANG-BEHAVIOR-FLAG SW1 exposes --scheduling-policy lpm (SGLang alias
--schedule-policy), cache-aware longest-prefix-match admission ordering. Per
src/vllm/entrypoints/openai/server_main.cpp:1266-1275, lpm needs prefix caching to
have any effect and degrades to fcfs without it. Since prefix caching cannot be
enabled alongside DFlash2, lpm is unreachable in any speculative configuration — the
flag parses, the server boots, and the feature can never do anything. This was its
first-ever load test.

It also means our benchmark's --no-enable-prefix-caching is forced, not a choice,
while the SGLang arm it is compared against runs its radix cache enabled (their flag is
--disable-radix-cache; --disable-radix-attention does not exist upstream and is our
alias only). That asymmetry cannot currently be closed from our side.

Not diagnosed

Whether prefix caching works with speculation OFF is being measured now and will be
posted here; that control separates "APC is broken" from "the APC x DFlash2 interaction
is broken".

Whether the invariant is the DEFECT or the DETECTOR is also open: it may be correctly
refusing a genuinely inconsistent state, in which case the repair is in the draft
context's position accounting, not in the check.

Found while running the first load test of the SW1 lpm path during #2028 ladder work.
Not fixed in flow: it needs its own spec and touches the draft accounting that #2010
just changed.

[localai-bot]

Control result: the two failures are DISTINCT, and prefix caching is broken on both paths

The control this issue promised — prefix caching with speculation off — has run. It also fails at c=1, but with a different fault, so "APC is broken" and "the APC x DFlash2 interaction is broken" are both true, independently.

Same binary (3d895a202), same box and settings, --enable-prefix-caching --scheduling-policy lpm, --speculative-config removed:

engine-fatal: EngineCore busy loop threw: vt cuda: cudaMalloc: operation not permitted when stream is capturing

Counts: stream is capturing x4, position discontinuity 0, illegal memory access 0. That is #2029, not this issue.

speculation	prefix caching	c=1 result
on (DFlash2 k=8)	on	this issue — propose_drafts_block: context position discontinuity
off	on	#2029 — cudaMalloc while the stream is capturing
on	off	OK — 24.84 out tok/s, 8/8
off	off	OK at c=2 (measured during #2008 isolation)

What survives, and what is corrected

Survives: the mechanism described above — with APC on, committed positions begin at the cached prefix length while the draft context starts accounting at 0, so the invariant refuses on the first propose. Nothing in the control contradicts it, because the control never reaches the draft path at all.

Corrected: this issue can no longer claim that disabling speculation is a workaround for prefix caching. It is not. Prefix caching is currently unusable in either configuration, for two unrelated reasons.

Unchanged consequence: SW1 lpm remains unreachable. It requires prefix caching, and prefix caching does not work with speculation on (this issue) or off (#2029). The flag parses and the server boots, so the feature looks live and does nothing.

Still not diagnosed: whether the invariant here is the defect or the detector. It may be correctly refusing a genuinely inconsistent state, in which case the repair belongs in the draft context's position accounting rather than in the check.

[localai-bot]

Context: the SGLang TTFT comparison this issue's fix bears on

Re-baselined on main @ 583b9a9e4 (post-#1994, so TTFT measures a real first token rather than an empty SSE frame), c=1, ctx 32768, 1024 in / 512 out, matched workload, dgx:gpu0 GB10 under an rc lease:

	ours	vLLM	SGLang
TTFT mean	~1346 ms	1331 ms	940 ms
out tok/s	24.76–25.15	24.36	25.20
TPOT	37.2–37.8 ms	38.53	37.93

We match vLLM on TTFT and trail SGLang by 43%. SGLang's arm runs with its radix cache ON; ours is forced to --no-enable-prefix-caching by THIS issue. So part of that 43% may be prefix caching we structurally cannot use.

The control I tried to run, and why it is unresolved

I attempted SGLang at c=1 with --disable-radix-cache (their real flag — --disable-radix-attention does not exist upstream; it is our alias only) to separate cache from engine. It refuses to start, twice, with 114 GB free on a 119 GB box:

ValueError: Loaded weights leave no GPU memory for the KV cache under --mem-fraction-static=0.7.
  Raise --mem-fraction-static above 0.241 (minimum viable = 1 - available/pre = 0.2404).
  If using speculative decoding, draft weights are now counted.

The identical configuration WITH radix enabled loads and serves normally — that is where the 940.30 ms figure comes from. So disabling radix changes SGLang's memory accounting enough that our matched configuration becomes unservable.

I could start it by raising --mem-fraction-static, but a different memory configuration is a different measurement and would not be comparable to the number I am comparing against. So the control is PARKED AS UNRESOLVED rather than run at a config that would not answer the question.

Consequence for this issue: SGLang's TTFT advantage is currently unattributed — neither confirmed as cache nor as engine. Fixing this issue is worth doing regardless, because prefix caching is the standard TTFT optimisation and we cannot use it at all, and because it is what makes the shipped SW1 lpm scheduler reachable. But nobody should claim yet that closing it will close the 43%.