CUDA illegal memory access under sustained c=8 load at seqs=16/ctx=8192, after c=1/2/4 serve cleanly (not #1983)

[localai-bot]

At --max-num-seqs 16 --max-model-len 8192 with DFlash2 speculation on, the engine
serves c=1, c=2 and c=4 cleanly and then dies under sustained c=8 load with a CUDA
illegal memory access, roughly 30 requests in.

engine-fatal: EngineCore busy loop threw: vt cuda: cudaMemcpyAsync: an illegal memory access was encountered
async-llm: output handler saw engine death: EngineCore encountered an issue.
sse: stream aborted mid-flight: ...

This is NOT #1983. That issue's own
isolation matrix records --max-model-len 8192 as not crashing (it reports a 360 s
timeout at seqs=32, "no crash"), and its crash fires on the first real request at
seqs=32 / ctx=32768. This one fires at seqs=16 / ctx=8192, after ~30 requests
have completed successfully. #1983 was also measured on main @ 993b62e4e, before
#1999 and #2000 changed KV sizing.

Measured

row/FIX-DFLASH2-CONCURRENCY @ 3d895a202 (= PR #2010), sm_121a, artifact-gated:
vllm-server 101,151,128 B, flash_fwd=1792, SpecDecodeFA2Bf16=1. dgx:gpu0 (GB10),
idle under rc lease 6e1589ea. --num-blocks 3744 (6.00 GiB pool),
--no-enable-prefix-caching, --speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}',
workload 1024 in / 512 out.

c	out tok/s	TPOT ms	n	ok
1	24.84	39.97	8	8
2	40.11	49.05	8	8
4	59.13	62.97	16	16
8	VOID	—	32	31, failed=1

The c=16 rung never ran; the ladder terminates on a VOID.

Not introduced by #2010. Before that fix c>=2 aborted at the
propose_drafts_block invariant, so c=8 was unreachable and this defect was
unobservable. #2010 exposes it.

The isolation attempt is inconclusive, and found a second defect

Same binary, same pool, --speculative-config removed, c=8: the engine dies during
warmup instead, with a different fault and illegal memory access count 0:

engine-fatal: EngineCore busy loop threw: vt cuda: cudaMalloc: operation not permitted when stream is capturing

filed separately. So "is the IMA in the draft path?" is unanswered — the control
cannot reach the load phase. A valid control needs CUDA graphs disabled, or
compute-sanitizer.

Not diagnosed

No stack trace is emitted despite the message saying "See stack trace (above)"; that
diagnostic gap is part of what makes this expensive to chase.

Found while running the #1574 concurrency ladder for PR #2010. Not fixed in flow: the
fault is in device memory management under concurrency and needs its own spec and a
compute-sanitizer run under a lease.

[localai-bot]

Correction and a much sharper characterisation

Two corrections to the issue as filed.

1. The reported frame is uninformative. CUDA illegal accesses are asynchronous and
sticky: the error surfaces at the next call that checks, and cudaMemcpyAsync is simply
the commonest such call. cudaMemcpyAsync: an illegal memory access therefore does
not localise the fault to a copy, and the title's emphasis on it should not be read
as a pointer to the faulting site.

2. It is NOT concurrency alone. The trigger is the PRODUCT of concurrency and
--max-model-len. The original filing said the c=8 crash occurred where #1983's matrix
records ctx=8192 as safe, and inferred a context-independent trigger. A second ladder at
--max-model-len 32768 falsified that: c=2 now VOIDs (ok=1 failed=7, illegal memory access x4, stream is capturing x0), where the same binary served c=2 at 8/8 with
ctx=8192.

Every rung measured on this binary (3d895a202, artifact-gated flash_fwd=1792,
dgx:gpu0 GB10, --num-blocks 3744, --max-num-seqs 16, --no-enable-prefix-caching,
DFlash2 k=8, 1024 in / 512 out):

--max-model-len	c	c x ctx	result
8192	1	8,192	OK (24.84 out tok/s)
8192	2	16,384	OK (40.11)
8192	4	32,768	OK (59.13)
8192	8	65,536	IMA, ok=31 failed=1
32768	1	32,768	OK (25.13)
32768	2	65,536	IMA, ok=1 failed=7

A monotone split: everything at or below 32,768 total tokens serves, both points at
65,536 die. Memory is not the cause — 114 GiB available on a 119 GiB box during the
failure.

This also refines #1983, which concluded "the trigger is concurrency configuration,
not context length". On this evidence it is neither alone; #1983's own two crashing
cells (seqs=32/ctx=32768) and its safe cells are consistent with a product bound.

Leading hypothesis, from the #2010 author's static analysis

The P > 1 fallback at src/vllm/model_executor/models/qwen3_dflash.cpp:1888-1927 is a
per-propose-step device-allocation burst that does not exist at P == 1, and its own
comment says it is "not capture-targeted". Every decode step it allocates 2 x L
buffers of [C, kdim] plus, per request per layer, tmpk/tmpv [cr, kdim] and two
index buffers, then frees them. C ~ sum(num_ctx) scales with concurrency x context —
i.e. with exactly the product that predicts the failures above.

This may share a root with #2029 (cudaMalloc: operation not permitted when stream is capturing): both are consistent with device allocation occurring where a capture scope
is open.

Ruled out: PR #2010 is not the cause. Its request-id keying contributes no device
index — every slot, bound and offset is re-derived from store.num_ctx/store.max_pages
and cross-checked by VT_CHECK on all backends; unordered_map::operator[] has no miss
path; the sole cudaMemcpyAsync in the touched path is a fixed 4-byte H2D with no index;
and no store pointer can dangle or outlive its step. #2010 made c>=2 reachable and thereby
exposed this; it did not introduce it.

Falsifiable prediction, registered before the run

If the product bound is real, --max-model-len 16384 must serve c=2 (32,768) and
fail c=4 (65,536). That test is running now; the result will be posted here whichever
way it goes.

[localai-bot]

The product bound is CONFIRMED — prediction registered before the run, and it held

The previous comment predicted: at --max-model-len 16384, c=2 (32,768) must serve and
c=4 (65,536) must fail. Both happened.

• c=2 @ ctx 16384: 42.61 out tok/s, 130.09 total, TTFT mean 184.31 ms, TPOT 45.77 ms, ok=8/8
• c=4 @ ctx 16384: VOID, ok=15 failed=1, same signature — illegal memory access x3, stream is capturing x0

Six points across three context lengths, perfect separation, no exceptions:

--max-model-len	c	c x ctx	result
8192	1	8,192	OK — 24.84 out tok/s
8192	2	16,384	OK — 40.11
8192	4	32,768	OK — 59.13
16384	2	32,768	OK — 42.61
32768	1	32,768	OK — 25.13
8192	8	65,536	IMA — ok=31 failed=1
16384	4	65,536	IMA — ok=15 failed=1
32768	2	65,536	IMA — ok=1 failed=7

The engine dies exactly when concurrency x max-model-len reaches 65,536 tokens.
Everything at or below 32,768 serves. Memory is not the cause: 114 GiB available on a
119 GiB box during the failures.

This is a much sharper handle than "crashes at high concurrency", and it should let the
fixer find the bound by arithmetic rather than by bisection. 65,536 = 2^16, which is
where I would look first: a 16-bit index, an int16_t/uint16_t count, or a buffer
sized in units that overflow or wrap at 65,536 total tokens across the batch. Note the
draft advertises "speculative context limited to 8208 tokens (513 pages x 16)" at
max_num_seqs 16 — 8208 x 8 = 65,664, close to but not equal to the bound, so the page
accounting is worth checking against the exact 65,536 figure rather than assumed to be it.

All rungs measured on 3d895a202 (PR #2010), artifact-gated flash_fwd=1792
SpecDecodeFA2Bf16=1, vllm-server 101,151,128 B, dgx:gpu0 GB10 idle under rc lease,
--num-blocks 3744 --max-num-seqs 16 --no-enable-prefix-caching, DFlash2 k=8,
vllm bench serve --dataset-name random, 1024 in / 512 out.

[localai-bot]

RETRACTION: the product bound is FALSIFIED

My previous comment claimed the engine dies exactly when concurrency x max-model-len reaches 65,536, on six points with perfect separation, and treated the confirmed c=4 @ ctx16384 prediction as establishing a law. A seventh point kills it.

c=8 @ ctx 4096 = 32,768 total tokens — the SAFE side of the claimed bound — VOIDs anyway. ok=31 failed=1, illegal memory access x3, discontinuity 0, stream is capturing 0. Only 1 request finished before the engine died, against ~30 at ctx 8192.

The earlier confirmation was a coincidence of which points I happened to test, not a law. I should not have written it up as one on six co-linear points.

Full table, all eight points

ctx	c	c x ctx	result
8192	1	8,192	OK — 24.84
8192	2	16,384	OK — 40.11
16384	2	32,768	OK — 42.61
8192	4	32,768	OK — 59.13
32768	1	32,768	OK — 25.13
4096	8	32,768	IMA — ok=31 failed=1
8192	8	65,536	IMA — ok=31 failed=1
16384	4	65,536	IMA — ok=15 failed=1
32768	2	65,536	IMA — ok=1 failed=7

What the data now support — stated as a hypothesis, not a law

A KV-capacity reading fits seven of eight points better than the product bound did. --num-blocks 3744 at 16 tokens/block is ~59,904 paged KV tokens. Every failure has c x ctx above that, and every success below it — except c=8 @ 4096.

So the evidence is consistent with two mechanisms rather than one:

• (a) capacity: when c x max-model-len exceeds the paged KV pool, the engine takes an IMA instead of refusing gracefully. Whatever the root cause, overcommitting the pool should be a clean refusal naming the shortfall, never an illegal access.
• (b) something specific to c=8, independent of capacity — c=8 has now failed at ctx 4096 and 8192, and has never once succeeded.

The discriminating test for (b) — c=8 at ctx 2048 (16,384 tokens, a quarter of the pool) — is running now and will be posted here whichever way it goes. If it fails, capacity is not the explanation for c=8 and (b) is a distinct defect.

Unchanged: the reported cudaMemcpyAsync frame is still not the faulting site (async, sticky). Memory is still not the cause (114 GiB free). #2010 is still excluded as the cause.

Also note --num-blocks 3744 is itself a workaround for the #1963 divisor bug, so the pool is far smaller than the box could support — vLLM on the same box reports GPU KV cache size: 647,682 tokens, roughly 10x ours.

[localai-bot]

c=8 SERVES at ctx 2048 — so "c>=8 is broken" is dead too, and the boundary is not a clean product

c=8 @ ctx 2048: 70.08 out tok/s, 210.19 total, TTFT mean 175.62 ms / p99 224.08 ms, TPOT 108.40 ms, n=32 ok=32. Zero failures. This is the project's first successful c=8 rung.

So hypothesis (b) from my last comment — something capacity-independent specific to c=8 — is falsified. c=8 works fine when the context is small enough.

Every point measured, by concurrency

c	ctx values tested	largest working	first failing	first failing c x ctx
1	8192, 32768	32768	none	—
2	8192, 16384, 32768	16384	32768	65,536
4	8192, 16384	8192	16384	65,536
8	2048, 4096, 8192	2048	4096	32,768

The boundary depends on both factors but is NOT their product. c=4 @ 8192 (32,768) serves at 59.13 tok/s; c=8 @ 4096 (32,768) dies. Same total, opposite outcome. c=8's ceiling is half the product c=2 and c=4 tolerate.

I am recording this as a measured boundary and NOT proposing a third law. My first two — the 2^16 product bound, and then a c=8-specific defect — were each consistent with everything I had measured at the time and each died to the next point. The pattern in my own errors is generalising from points chosen where rival explanations agree.

What a fixer can use

• A cheap reproducer exists: c=8 @ ctx 4096 dies after 1 completed request (against ~30 at ctx 8192), so the failure is fast to reach.
• A working control sits right next to it: c=8 @ ctx 2048, same binary, same pool, 32/32 clean. The pair differs only in --max-model-len.
• Whatever is sized from max_model_len per sequence is the place to look, since the survivable context halves as concurrency doubles from 4 to 8 — but that reasoning is exactly the kind that has misled me twice here, so treat it as a starting point, not a conclusion.

All runs: 3d895a202 (PR #2010), artifact-gated flash_fwd=1792 SpecDecodeFA2Bf16=1, vllm-server 101,151,128 B, dgx:gpu0 GB10 idle under an rc lease, --num-blocks 3744 --max-num-seqs 16 --no-enable-prefix-caching, DFlash2 k=8, vllm bench serve --dataset-name random --backend openai-chat, 1024 in / 512 out.