[BHCC2025]

Stacked on #1465 (its commit comes first here; this PR's own commits are the last six). It carries #260's EXL3 speed series, from the Python engine, onto the native Dense module, as the closing note on #260 suggested once the native engine had an EXL3 reader.

What this changes

• exl3 cuda, kernels.
  • Weight.sk/wk let a family fix a shape's plan (K splits, warps). No family here fixes one: every launch is plan()'s. A fixed plan keeps the linear.
  • The mid-M kernels and linear_wc cover 17-128 rows (prompt rows and wide verify windows) with linear_body's bits. exl3.pick is matmul's choice among them and the linear.
  • matmulGroup runs up to three layers of one K over the same rows in one rotation and one launch.
  • Programmatic dependent launch on the rotations and decode-row linears, on sm_90 and newer (TF_EXL3_PDL=0: plain). Below sm_90 the waits compile to nothing and the launches are plain.
  • tf-cuda-test exl3-midm, exl3-plans and exl3-shapes check and time them; exl3-contract runs their kernel-contract registrations alone.
• qwen3_5 cuda, family.
  • projMany puts an EXL3 engine's decode-row members in one group launch (q/k/v, qkv/z, gate/up).
  • Dense.reserveGroup sizes a group's scratch at load (after the layer's MLP is set), before graph capture.
  • Dense.reserve sizes split-K scratch by planOf. A grown counter buffer is zeroed on the call's stream.

The arithmetic

• Same bits throughout. The mid-M kernels, linear_wc, the group launch and rot_group return the bytes of the one-weight linear_body (and rot_in) launch at plan()'s plan. Every projection keeps plan()'s launch, so no summation order changes.

Receipt

• Environment. main at b30d5b9d, plus qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts #1465, plus this. One DGX Spark (GB10, sm_121, 128 GB unified memory), Ubuntu 24.04.5, driver 580.178.04, nvcc 13.0.88, Zig 0.17.0, zig build install native -Dsm=121.

  • Checkpoint turboderp/Qwen3.8-27B-exl3@113cf7ab (4.00bpw), drafter z-lab/Qwen3.8-27B-DFlash2@50307d4c.
  • --context 131072; greedy unless stated. Same machine, prompts and settings as qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts #1465's receipt.

• Same bits (tf-cuda-test, synthetic weights at every 27B shape, mul1 4-bit; the head 6-bit).

  • exl3-midm: the mid-M kernels and linear_wc equal the linear byte for byte at 17, 24, 32, 33, 48, 49, 64, 100 and 128 rows, and the mid-M kernels alone at 24, 48, 64 and 128. The group launches (q k v, gate up, qkv z) equal each member's own launch at 1, 7 and 16 rows. PASS exl3.midm, PASS exl3.group.
  • exl3: 20 pass, 0 fail. The 9 skips are the Python engine's fixture checks (no fixture directory given).
  • exl3-tiles: 5 pass, 0 fail.

• Exactness, served.

  • tf-qwen35-cuda ... step|window|tree over 48 ids: window logits equal step logits byte for byte, and so do tree logits.
  • bench_concurrent.py --alone --serial --levels 1,4 --temperatures 0,1.0 (drafts on): alone 20/20 equal, serial 20/20 equal, 0 failed.
  • Drafted equals "draft": false: an 8,020-token chat, 256 reply tokens. Fresh, the cached repeat and "draft": false give one reply hash.
  • Resumed equals fresh: turn 2 of a 30.4k-token two-turn chat resumed at 28,672 cached tokens (time to first token 2.26 s, against 36.62 s cold).
  • Cancellation: two streamed 32k-token completions closed mid-prefill. The chat reply before them and after each one has the same sha256, and /health stays ok.

• Speed, against qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts #1465 alone on the same machine:

  	qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts #1465	with this
  bench_openai drafts on, fibonacci-raw / gpu-chat-no-think	71.9 / 45.8 tok/s	74.8 / 46.1
  bench_openai, no drafter	15.5 / 15.3	15.7 / 15.6
  bench_concurrent 4 streams, code / chat, greedy	137.4 / 101.2	209.7 / 133.2
  bench_concurrent 4 streams, code / chat, temperature 1.0	140.9 / 101.4	202.8 / 123.0

  Cold prefill: bench_cache.py --require-cold --prompt-file, --prompt-cache-gib 0, a fresh server per length. These are qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts #1465's exact token arrays (this repository's zig/src/**/*.zig under the checkpoint's tokenizer). All four qualified cold.

  prompt	qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts #1465 ttft_s	with this	tok/s
  2,048	5.71	2.20	359 → 933
  8,192	23.07	8.88	355 → 922
  32,768	94.11	38.22	348 → 857
  65,536	193.98	82.59	338 → 794

• Kernel contract.

  • The 73 new launches are in registry.zig (exl3_dense_records.zig), and 60 existing exl3_linear records are refreshed. All come from kernel-contract-limits over fatbins for sm_80/86/89/120/121 with nvcc 13.0.88; the audit has no stale or failing EXL3 record.
  • zig build kernel-contract on the GB10 (sm_121), this branch: SUMMARY 426 kernels: 426 pass gates 1-4, 0 fail, 5 known findings; 318 controls: 318 red as planted, 0 missed; 195 limit problems.
  • None of the 195 limit problems is an EXL3 launch. They are in qwen35/qwen35_kv8 attention, qwen4_hc, qwen4_ple, qwen4_projection, qwen4_gdn and nvfp4_experts_tile records, which this PR does not touch. On qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts #1465's branch the run lists the same gate and control findings up to the limits section (322 identical FAIL/KNOWN lines).
  • Gates 1 to 4 (zig/tests/cuda/contract_exl3.zig): eight replacements of linear_body, each with a planted control that goes red at gate 3, plus rot_group against rot_in.
    • The mid-M kernels at 3, 4, 5 and 6 bits, and linear_wc at 4 and 6 bits, at 1 to 128 rows. Their served shapes reach every form plan() gives those widths: both row passes, 4 and 8 warps, split and unsplit K, and every linear_wc tile.
    • The groups at 1 to 16 rows.
    • Gate 4's stress weights are all-ones states, one weight at every width.
    • exl3-contract: SUMMARY exl3 8 kernels: 8 pass gates 1-4, 0 fail; 6 controls: 6 red as planted, 0 missed.
  • A host test sweeps K and N to 16384 through exl3.pick and groupable. It fails on any form the dispatch reaches that no registration's gate-3 shapes run.

• Tests run.

  • zig build test on Linux x86_64: 0 failed.
  • zig fmt --check is clean.
  • python3 tools/zig/lean_check.py: 428 problems, the same as main. Zero new.

• Not run, and why.

  • Only GB10 (sm_121) ran on a GPU. sm_80/86/89/120 are compiled and their limits checked, but not run. The RTX 3090 cross-checks offered on Port Qwen3.8-27B (EXL3 3.00bpw + DFlash2) to the native CUDA engine on Ampere sm_86 (RTX 3090) — proposal and gate #603 would cover sm_86 and the plain-launch path.
  • No teacher-forced comparison against ExLlamaV3 itself.
  • No macOS build: the change is in CUDA-only files and their host tests.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
• No precision traded for speed: the new kernels keep the linear's bits, and every launch keeps plan()'s summation order.
• New and changed kernels are registered with the kernel contract (gate 5), with the SUMMARY line above.
• Prompt processing is no slower than before: 2.3-2.6× faster than qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts #1465.
• The description says which platforms I ran (CUDA GB10 only) and which I could not.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[BHCC2025]

Pushed 969c4128, which fixes the CI failure and two things the full SM build turned up:

• CI (zig, zig-windows). kernel_symbols failed because tf_exl3_group_8_2_4 and _8 were generated by a macro, so their names never appeared in the source. They are now written out.
• sm_80-89 builds. The programmatic-dependent-launch helpers used griddepcontrol, which exists only on sm_90 and newer. They now compile to nothing below sm_90, as in qmm_frag.cuh, and Linear.load enables dependent launches only on compute capability 9.0 and newer. Before this, exl3_linear did not build for sm_80, 86 or 89.
• Kernel contract, gate 5. The 73 new launches (2-warp linears, mid-M, linear_wc, groups, rot_group) are registered in exl3_dense_records.zig. Their usage comes from nvcc 13.0.88 fatbins for sm_80/86/89/120/121, and the existing exl3_linear records are refreshed. tf-cuda-test kernel-contract-limits now reports nothing stale or failing for exl3_linear. Its only remaining FAILs are the existing qwen35 tf_q35_attn_partial and qwen35_kv8 tf_q35_attn_partial8 records, which this PR does not touch.
• Host checks. zig build test passes, zig fmt --check is clean, and lean_check.py reports 428 problems, matching main.

Still to do before review: gates 1-4 for the new kernels against linear_body (bit-equal at 1-128 rows), and the device receipt from this branch.

[ashhart]

Thanks @BHCC2025. The same-bits parts read as same-bits to us. The mid-M kernels and linear_wc keep linear_body's K ranges, warp-order sums and split-order partials, and a group member runs linear_body on its own blocks. Every PDL launch also waits before it touches global memory. Merged onto main with #1465, fmt, lean, test-exl3-cuda-host, test-kernel-contract, test-cuda-host and the CUDA cross-compile all pass.

1. Fixed plans. CONTRIBUTING requires a changed reduction order to keep native error no worse than the current reference. Your table shows higher RMS for six of the seven changed shapes, and a worse worst column for down, k_v, qkv and out_o. Please keep plan() (drop the table at cuda_exl3_weights.zig:130), or keep only plans that are no worse at both 16 and 128 rows. Then re-measure with the same-bits parts alone.
2. Load order. reserveGroups runs at cuda_exl3_weights.zig:302, before l.mlp is set, so it reads a freshly allocated layer's uninitialised mlp, and the real gate/up group is never reserved. It's harmless on the 27B only because other reserves already cover it. Please move the call below the MLP. A host test that decodeGroup allocates nothing after a dense load would catch this.
3. Zeroing. scratch() zeroes grown buffers on the legacy stream (exl3.zig:376), and the per-call zeroing is gone, but the engine's stream is non-blocking. Please zero the counters on the call's stream, as exl3: zero the split-K counters once as they grow, not before every call #1475's grownCounters does.
4. Contract. The new launches have limit records only. Please register the mid-M kernels, linear_wc, the groups and rot_group in gates 1-4 as byte-equal replacements of linear_body at 1-128 rows, each with a planted control. Please also add host tests for groupable, wcPick and planOf.

Then we'll run exl3-midm, kernel-contract and its limits (nvcc 13.0 and 13.3) on a GB10, the dense pack against #1465, and a MoE EXL3 checkpoint against main, since PDL and the groups reach it too.

[BHCC2025]

Thanks for the careful read. All four are addressed in be7c2a7:

1. Fixed plans. Dropped: the table in cuda_exl3_weights.zig is gone and every projection takes plan()'s launch. None of the eight plans met "no worse at both 16 and 128 rows" (only q's, which was already plan()'s). Weight.sk/wk and planOf stay, so a family can still fix a plan that does qualify. exl3-plan-f64 went with the table. With plan(), q/k/v and qkv/z no longer share a warp count, so only gate/up groups on the 27B. The re-measure below is the same-bits parts alone.

2. Load order. reserveGroups now runs after l.mlp is set, so the gate/up group is reserved. New host test (exl3_dense_test.zig, mock driver): after a dense layer's reserves, groupScratch at 1 to 16 rows allocates nothing and zeroes nothing. Without the group reserve the same call allocates, so the test catches the missing reservation. decodeGroup takes its scratch through groupScratch, the function the test exercises.

3. Zeroing. scratch() no longer zeroes. grownCounters(s, bytes) zeroes a grown counter buffer on the call's stream, as exl3: zero the split-K counters once as they grow, not before every call #1475 does, and reserve/reserveGroup take the stream (the loader passes the engine's). A host test checks that the fill goes to the given stream once, and not again while the buffer is big enough.

4. Contract. zig/tests/cuda/contract_exl3.zig registers six replacements of linear_body in gates 1 to 4:

   • the mid-M kernels and linear_wc, each at 4 and 6 bits, 1 to 128 rows on the 27B's shapes. A new Linear.min_rows (17 by default) is set to 1 there, so they run below 17 rows too and every row count's bytes are compared;
   • the projection groups at 1 to 16 rows (gate up; q, gate, qkv at 8 warps; z, k_v at 4 warps with split K), against each member's own launch;
   • tf_exl3_rot_group against each member's tf_exl3_rot_in.

   Gate 1 is linear_body's bytes with planted NaN and both infinities reaching the outputs. Gate 3 is byte-equal at every served shape and row edge. Gate 4 cannot make EXL3's K sums exact (decoded fp16 weights), so its stress input is a constant weight (a 2-bit period at every code's offset) times rotated inputs of 4096. fp16 sums overflow in every order while fp32 stays in its bound. Four planted controls each go red at gate 3: the mid-M kernels and linear_wc at another K split, a group with one member's scales swapped, and rot_group with one member's suh swapped. tf-cuda-test exl3-contract runs just these. Host tests for groupable, wcPick and planOf are in exl3_dense_test.zig.

Receipt for be7c2a7, on one GB10 (sm_121, nvcc 13.0.88, driver 580.178.04):

• exl3-contract: SUMMARY exl3 6 kernels: 6 pass gates 1-4, 0 fail; 4 controls: 4 red as planted, 0 missed
• kernel-contract: SUMMARY 424 kernels: 424 pass gates 1-4, 0 fail, 5 known findings; 316 controls: 316 red as planted, 0 missed; 195 limit problems (none of the 195 limit problems is an EXL3 launch: they are qwen4 hc/ple/projection/gdn, qwen35 and qwen35_kv8, nvfp4_experts_tile and the dsv41_x3gm optional shared-memory notes; every EXL3 launch passes the live limits)
• exl3-midm, exl3, exl3-tiles: 2, 20 and 5 pass, 0 fail
• zig build test (Linux x86_64) 0 failed, zig fmt --check clean, lean_check.py 428 problems (main's baseline, 0 added).

Same-bits parts alone against #1465, same checkpoint, prompts and settings as before:

	#1465	earlier draft (fixed plans)	now (plan())
Drafts on, code / chat (tok/s)	71.9 / 45.8	76.4 / 50.8	74.8 / 46.1
No drafter, code / chat	15.5 / 15.3	16.2 / 16.1	15.7 / 15.6
4 streams greedy, code / chat	137.4 / 101.2	209.6 / 142.7	209.7 / 133.2
4 streams temperature 1, code / chat	140.9 / 101.4	207.5 / 124.9	202.8 / 123.0
Cold prefill 2k / 8k / 32k / 64k (tok/s)	359 / 355 / 348 / 338	989 / 975 / 904 / 832	933 / 922 / 857 / 794

Exactness on this build: window == step and tree == step (48 steps, 64.0 ms a step); alone and serial 20/20 equal, 0 failed; drafted = "draft": false on the 8k chat; resume a 30,445-token second turn resumed from 28,672 cached tokens (ttft 2.26 s against 36.62 s cold); cancellation the reply after two cancellations equal to the one before.

[ashhart]

Thanks @BHCC2025. On be7c2a7f, the plans are back to plan(), reserveGroups runs after the MLP is set, and the counters are zeroed on the call's stream. The host and compile checks pass here.

One contract gap is left. mpg_variants includes halves 6 and 10, and the default prompt dispatch picks them from 17 rows on. But contract_exl3.Set.add registers mid-M only at halves 8 and 12 (zig/src/cuda/exl3.zig:59, :102, :283; zig/tests/cuda/contract_exl3.zig:409). The loader already admits 3-bit and 5-bit dense projections. Please register those widths through gates 1-4 too, with both row-pass forms and a split-K shape, or keep them on the old linear until they're qualified. Their random float64 checks and launch records don't cover the whole replacement contract.

[BHCC2025]

Thanks. The gap was wider than 3 and 5 bits, so I checked the whole dispatch rather than only those two widths. Fixed in 59123e5.

What was missing. I listed every kernel form matmul's default dispatch can reach for a dense projection the loader admits. A form is a width, a row pass (P1 up to 48 rows, P2 above), 4 or 8 warps, split or unsplit K, and for linear_wc its tile. There are 45 reachable forms across the mid-M kernels and linear_wc. The old registrations ran 14 of them. Besides 3 and 5 bits, they missed:

• 6-bit at 8 warps;
• 4-bit at 4 warps with unsplit K;
• 16 linear_wc tile and split forms;
• the projection group at 4 warps with unsplit K.

The fix.

• Registrations. contract_exl3 registers the mid-M kernels at 3, 4, 5 and 6 bits, with new 3- and 5-bit controls, and linear_wc at 4 and 6 bits. Their served shapes cover every form plan() gives at each width: both row passes, both warp counts, split and unsplit K, every linear_wc tile. Small-K shapes (K 512 to 1408) reach the unsplit and 8-warp forms the 27B's K never produces. The groups gain a K-512 member set (4 warps, unsplit).
• A dispatch test. exl3.pick is now matmul's choice of kernel alone, and matmul launches whatever it returns. A host test in contract_exl3.zig sweeps K and N from 128 to 16384 at 17-128 rows through pick, and the groups through groupable. It fails on any form that no registration's gate-3 shapes run. With the 3-bit registration removed it fails with unregistered: mpg 1 split false (3-bit, 17 rows, n 128, k 128). A new mid-M width or tile can't reach the default path without a registration.
• Fixed plans. A family that fixes a plan (Weight.sk set) now keeps the old linear, so a form outside plan()'s set (for example 2 warps) is never dispatched unqualified. No family sets one.
• Gate 4. The stress weights are now all-ones states with scales of 1, which decode to one weight at every width. My earlier note was wrong at odd widths: the old 0xaaaa fill decodes to two alternating weights at 3 and 5 bits, because their code windows start at odd offsets.

Receipt for 59123e5, on one GB10 (sm_121, nvcc 13.0.88, driver 580.178.04):

• exl3-contract: SUMMARY exl3 8 kernels: 8 pass gates 1-4, 0 fail; 6 controls: 6 red as planted, 0 missed
• kernel-contract: SUMMARY 426 kernels: 426 pass gates 1-4, 0 fail, 5 known findings; 318 controls: 318 red as planted, 0 missed; 195 limit problems. 195, as on be7c2a7, all in qwen4, qwen35 and nvfp4 records. None is an EXL3 launch.
• exl3-midm and exl3: 2 and 20 pass, 0 fail
• zig build test (Linux x86_64) 0 failed, zig fmt --check clean, lean_check.py 428 problems (main's baseline, 0 added).

The served path is unchanged: no family fixes a plan, so every 27B projection takes the same launch as on be7c2a7, and that receipt still applies.

[ashhart]

@BHCC2025, good news and one ask. Our last run of this on a GB10 (the cut-down head, with the dense EXL3 27B at 4.00 bpw) was GREEN: replies equal to main and no cell slower. It was ready to land.

Since then #1475 landed. It zeroes the split-K counters once as they grow, on the calls' stream (Dense.grownCounters, with a test), and reserve now takes the stream. Your review-fix commit does the same thing its own way, so the two now conflict in zig/src/cuda/exl3.zig (the counters field, reserve, grownCounters, reserveGroup/groupScratch) and in qwen3_5/cuda_exl3_weights.zig (the reserve call), across several of your commits.

Could you rebase onto current main and keep main's grownCounters as the one place counters are zeroed? Your groups then just call it with their summed bytes. You know these commits best, and a hand-merge from us across six of them is where mistakes would creep in. Once you push, we'll gate it and re-run the same GB10 cells straight away. It's the only thing between this and main.