[diwakergupta]

What this changes

exl3.Dense zeroed its split-K counters before every 128-row block of every call: one cuMemsetD8Async per projection per decode step, and per 128 prompt rows. Each block's last arriving program in exl3_linear.cu already resets its own counter once the partials are summed (a.counters[pass * NB + nb] = 0), so the counters only need zeroing when the buffer grows.

• Dense.grownCounters zeroes the counters on the calls' stream when they grow, and returns the same buffer otherwise. blocks() no longer memsets.
• reserve() takes that stream, so the zeroing happens at load and a graph capture allocates and memsets nothing.
• This reaches every EXL3 checkpoint on the native Dense path, MoE included. The arithmetic is unchanged.

This is what's left of the prompt-tiles PR. The 64-row tensor-core tiles are dropped: on the GB10 they changed tokens against main (7 of 12 prompt rows), because the tiles sum in a different order from the linear. #1467's mid-M kernels keep the linear's bits.

Receipt

• Environment. RTX 3090 Ti (24 GB, sm_86). turboderp/Qwen3.8-27B-exl3 branch 3.50bpw (rev 8351c54e), z-lab/Qwen3.8-27B-DFlash2 (rev 50307d4c) at 4 bits, bf16 KV cache, --context 32768. Both binaries came from zig build native -Dsm=86, one from main (a0d24896) and one from this branch. Each ran twice, alternating (main, this, main, this), on a restarted server.

• Exactness. tools/bench_concurrent.py --tokens 384 --reps 3 --alone --serial, the default code and chat cells (greedy and sampled) at --levels 1,4, and --levels 1 with --prompts of two long prompts (5,080 and 2,278 tokens):

  • Concurrent == alone and drafted == "draft": false in every cell on both binaries: 0 unequal, 0 failed.
  • token_sha256 of this branch equals main's in 528/528 replies.

• Decode speed. Per-stream tok/s, runs 1 / 2 (each a median of 3). Tokens per round are equal except in the sampled 4-stream cells, where they vary from run to run on both binaries:

  Cell	main	this PR	change
  code, sampled, 1 stream	190.1 / 190.1	192.1 / 192.2	+1.1%
  code, sampled, 4 streams	132.6 / 128.8	132.8 / 135.2	+2.5%
  chat, sampled, 1 stream	116.0 / 116.0	117.4 / 117.4	+1.2%
  chat, sampled, 4 streams	88.8 / 89.1	90.2 / 91.8	+2.3%
  code, greedy, 1 stream	183.7 / 183.6	186.0 / 185.9	+1.3%
  code, greedy, 4 streams	128.9 / 129.0	130.4 / 130.4	+1.1%
  chat, greedy, 1 stream	107.9 / 107.9	109.2 / 109.2	+1.2%
  chat, greedy, 4 streams	88.2 / 88.3	89.2 / 89.2	+1.1%
  long prompts, sampled	112.7 / 113.2	113.9 / 113.9	+0.8%
  long prompts, greedy	115.8 / 117.5	118.8 / 118.7	+1.8%

• Prompt speed. Three distinct cold ~8,145-token prompts per run (cached_tokens 0, max_tokens 1), prefill_seconds from the reply: main 14.61 s mean, this PR 14.57 s (+0.3%).

• Tests. The full checklist suite (zig build test test-kernel-contract test-golden test-server-security test-pull-security fuzz-cortheon test-cuda-host) plus test-exl3-cuda-host and test-exl3-prompt-host: 485/488 steps succeeded. The one failure is keepalive_test "a silent engine's stream gets keepalives before its first token", which crashes the same way on main (a0d24896) on this machine. lean_new.py against main: 0 new problems. New host test: the counters are zeroed on the calls' stream only as they grow (exl3.zig, with exl3_mock's memset counts). It pins one memset per growth on the named stream; on main there is no such function, and blocks() memsets every call.

• Not run. GB10 / sm_121. Metal (not affected). MoE EXL3 checkpoints, which take the same blocks().

Checklist

• Every token still goes through the lane rounds.
• Drafted output equals "draft": false, concurrent equals alone, and this branch's tokens equal main's.
• No precision traded for speed: no arithmetic changed.
• No GPU kernel changed (one comment in exl3_linear.cu).
• Prompt processing unchanged in bits.
• Platforms: CUDA sm_86 only.
• The new test pins the new behaviour (no equivalent function exists before the change).
• Comments and docstrings are one line.
• No personal data, local paths or AI attribution lines.

[ashhart]

Thanks @diwakergupta. This covers everything we asked on #1407: the tiles are opt-in, the choice depends only on the layer and the dtypes, the counters are zeroed on the calls' stream with a test that fails on the legacy stream, and the 16/17-row receipts on the bf16 cache are the ones we wanted. Merged onto current main, fmt, lean, test-exl3-cuda-host, test-exl3-prompt-host, test-cuda-host and the CUDA cross-compile all pass.

1. Please rebase onto qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts #1465 instead of qwen3_5 cuda: serve dense EXL3 checkpoints with the DFlash2 drafter #1474, which we're closing as a duplicate of qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts #1465. Only cuda_exl3_weights.zig:30 and :148 touch the loader, so it should move across cleanly.
2. exl3 cuda: the 27B's EXL3 speed series on the native Dense module (#260), stacked on #1465 #1467 (also on qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts #1465) speeds up the same prompt rows without a second arithmetic. Its mid-M kernels give 17-128 rows their decode bits, and its receipt shows about 2.7x cold prefill on a GB10. With both landed, the tiles would take dense prompts away from that path. A separate prompt arithmetic is only worth it if it's clearly faster, so we'll time both on a GB10 with the same prompts. The same comparison on your 3090 Ti would help too.
3. The counters change reaches the MoE EXL3 checkpoints too, because blocks no longer zeroes per call (exl3.zig:300). The bits stay the same, and this is the version we want; exl3 cuda: the 27B's EXL3 speed series on the native Dense module (#260), stacked on #1465 #1467 does it on the legacy stream.

Once it's rebased, we'll run the dense pack on a GB10 with tails of 16 and 17 rows past a kept state, concurrent == solo, drafted == plain, and cold prefill against #1465 and #1467, plus a MoE EXL3 checkpoint against main.

[ashhart]

Thanks @diwakergupta for the quick rebase. The last commit now sits on #1465 and is line for line the commit we reviewed, and the description now says the counters change reaches the MoE EXL3 checkpoints too. Merged onto current main, fmt, lean (no new findings), test-exl3-cuda-host, test-exl3-prompt-host, test-qwen35-formats-host, the EXL3 admission and owner steps, test-cuda-host and the CUDA cross-compile all pass.

Two small things:

1. The title still says "stacked on qwen3_5 cuda: serve dense EXL3 checkpoints with the DFlash2 drafter #1474"; please make it qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts #1465.
2. The Dense doc comment at zig/src/cuda/exl3.zig:184 still says prompt rows run in 128-row blocks. With prompt_tiles that's only the default; one line saying an engine can opt in to the tiles would keep it true.

If you get the chance, the same three cold ~8,145-token prompts on #1467 on your 3090 Ti would still help us choose between the two prompt paths (its kernels build for sm_86). It isn't a blocker; our GB10 timing will decide.

A heads-up: this PR and #1507 both add tests at the same place in exl3.zig, so whichever lands second needs a trivial rebase that keeps both.

Next we'll run the dense pack on a GB10: resumed against fresh with tails of 16 and 17 rows past a kept state (greedy and seeded), concurrent == solo, drafted == plain, cold prefill against #1465 and #1467 on the same prompts, and a MoE EXL3 checkpoint against main by token hash.

[diwakergupta]

Thanks @ashhart. Pushed ee02a88e:

1. The title now says qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts #1465.
2. The Dense doc comment at exl3.zig:184 now reads "prompt rows take 128-row blocks unless an engine opts in to tiles", on one line.

I timed the three cold ~8,145-token prompts on the 3090 Ti, alternating #1465, #1467 (c9588c91, -Dsm=86) and this branch twice, each on a restarted server:

	#1465	#1467	this PR
mean cold prefill	14.61 s	8.82 s	8.37 s

So on sm_86 the tiles are about 5% faster than #1467's mid-M path, a small margin. The per-prompt numbers are in the description. I'd defer to your GB10 timing. If the margin is similar there, keeping one prompt arithmetic seems the better trade, and I'm happy to close this, or to cut it down to the counters change if you want that on its own. Noted on the shared test spot in exl3.zig with #1507; I'll rebase whichever lands second.

[ashhart]

Thanks @diwakergupta, both are done, and thank you for timing #1467 on the 3090 Ti. With the tiles about 5% ahead
there, our GB10 timing will decide between the two prompt paths. If the margin is similar on a GB10, cutting this down
to the counters change is a good outcome.

Next, after #1465, we'll run the dense pack with DFlash2 on a GB10:

• resumed against fresh, with tails of 16 and 17 rows past a kept state, greedy and seeded;
• concurrent against alone, and drafted against plain;
• cold prefill on the same three ~8,145-token prompts, against qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts #1465 and exl3 cuda: the 27B's EXL3 speed series on the native Dense module (#260), stacked on #1465 #1467;
• a MoE EXL3 checkpoint against main by token hash.

We've started the CI runs that were waiting on our approval.

[diwakergupta]

Hi @ashhart, checking in on the GB10 timing of the tiles against #1467. Once it's in, I can cut this down to the counters change if the margin is small. zig-windows on ee02a88e was cancelled, not failed, so it needs a rerun.

[ashhart]

Thanks @diwakergupta. We ran this on a GB10 (sm_121) against main, with Qwen3.8-27B EXL3 3.50bpw, fresh servers per arm and alternating base/new rounds.

The prefill gain is real and large: 2,048-token prompts go from 507 to 820 tok/s (+62%), 8,192 from 492 to 800 (+63%), and 32,768 from 473 to 754 (+59%).

Two things stop us landing it as it stands:

• The output tokens change. 7 of the 12 prompt-test rows produce different tokens from main. TensorFold's rule is exact output, so the 64-row tensor-core tile path has to give the same tokens as the path it replaces. If the reduction order changes inside the tile, that is the likely cause.
• Some decode cells are slower: decode after a 32,768-token prompt is 37.3 -> 35.8 tok/s (-4.0%), and 8 streams are 167.4 -> 160.4 (-4.2%). One to four streams are within about 1.5%.

We're parking this for now, not closing it. Once the tokens match main, we'll rerun the same cells and land it if they're clean. If you'd like, we can share the exact prompts behind the differing rows.

[diwakergupta]

Thanks @ashhart, and for the GB10 numbers. The tiles sum in a different order from the linear, so their tokens can't match main without giving up most of the gain, and #1467's mid-M kernels already keep the linear's bits. So I've cut this down to the counters change, as you suggested. Pushed 17d96c40, on current main (#1465 has landed):

• Dense zeroed its split-K counters before every 128-row block of every call, which is one memset per projection per decode step. The kernel's last program in each block already resets its counter, so they're now zeroed only when they grow, on the calls' stream.
• On the 3090 Ti against main: 528/528 token hashes equal, with 0 unequal in concurrent vs alone and drafted vs plain. Decode is +0.8% to +2.5% in every cell at 1 and 4 streams, and cold prefill is unchanged (14.61 s vs 14.57 s).

The description has the full table.