[diwakergupta]

What and why

Enable dense qwen3_5 ExLlamaV3 (EXL3) checkpoints with the 4-bit DFlash2 drafter on the native CUDA engine, and close the prompt-prefill, VRAM-headroom, and speculative-decode gaps on 24 GB GPUs:

1. Dense qwen3_5 EXL3 admission and DFlash2 attachment (cuda_geometry.zig, cuda_exl3_weights.zig, cuda_native.zig):
   • Admit non-MoE qwen3_5 EXL3 checkpoints (g.pocket == 0), guard MoE grouped-expert workspace allocation on g.moe(), load dense SwiGLU mlp.{gate,up,down}_proj via dense(), and allow openDrafter on EXL3 targets.
2. 64-row tensor-core EXL3 prompt tiles, one-time counter zero-init, and .x3 lm_head span slicing (exl3.zig, exl3_candidate.zig, cuda_draft_frame.zig):
   • Preload exl3_candidate during Dense.reserve and route Dense.prompt (rows > 16, BF16 activations) to tf_exl3_prompt_2 (64-row tensor-core tiles).
   • Zero-fill Dense.counters once on allocation/growth since exl3_matmul_kernel resets counters before exiting.
   • Support column-sliced Weight views in Dense.candidateWeight / candidate.launch (n_blocks = stride_k / (bits * 8)) and slice .x3 lm_head in cuda_draft_frame.zig across the two active vocabulary spans (0..98304 and 248064..248320, 39.7% of the 248,320 vocabulary).
3. Host-mapped BF16 embed_tokens and disjoint mixer/MLP scratch union (cuda_exl3_weights.zig, cuda_engine.zig):
   • Pin BF16 embed_tokens.weight in mapped host memory (HostBuffer.allocMapped) so embedRows gathers token rows over PCIe without occupying 2.37 GiB of device arena memory.
   • Overlap the sequentially executed DeltaNet mixer, full-attention mixer, and dense MLP scratch regions in Engine.scratch, reclaiming 0.46 GiB of device arena memory (2.83 GiB total).
4. Gate 27B root_copies on long_match (20 tokens) (cuda_native.zig):
   • Set facts.copy_match = lanes.windows.long_match (20) alongside facts.root_copies = true on 27B dense (hidden == 5120, layers == 64, experts == 0) so short 5-token suffix matches do not preempt DFlash2 drafting.

How it was tested

• Host and contract suites:
  • zig build
  • zig build test test-kernel-contract test-golden test-server-security test-pull-security fuzz-cortheon test-cuda-host
  • zig build test-kernel-contract -Dsm=80,86,89,120,121
  • python3 tools/zig/lean_new.py (0 new problems)
• End-to-end GPU evaluation on NVIDIA GeForce RTX 3090 Ti (24 GB VRAM, sm_86, CUDA 13.0 driver) with turboderp/Qwen3.8-27B-exl3 (3.50bpw) + z-lab/Qwen3.5-27B-DFlash (4-bit) + --kv-cache fp8:
  • Model + scratch VRAM: 17.21 GiB -> 14.38 GiB (-2.83 GiB), expanding --context 32768 capacity from 2 to 4 concurrent streams and single-stream capacity from 147,456 to 237,568 tokens (192k 3-needle NIAH passes 3/3 at 50.7 tok/s).
  • Prompt prefill TTFT: retrieval-8192 15.08s -> 8.79s (-42%), cache_seed (3.6k tokens) 6.90s -> 3.93s (-43%), vision 0.462s -> 0.320s (-31%), retrieval-65536 122.49s -> 89.46s (-27%).
  • Speculative decode throughput: tool 154.9 -> 362.6 tok/s (2.34x), retrieval-8192 122.7 -> 223.5 tok/s (1.82x), cache_followup 30.8 -> 60.9 tok/s (1.98x), 6-task Quality suite 6/6 at 170.8 tok/s average (198.2 tok/s on systemd, 187.5 tok/s on debug-db_leak), Multi-Turn Agentic Stress 4/4 (100%, CAI 92.4, 0 violations), and tool-eval-bench --short 93/100 (15/15 pass).

Checklist

• One logical change in this pull request
• zig build test test-kernel-contract test-golden test-server-security test-pull-security fuzz-cortheon test-cuda-host passes
• python3 tools/zig/lean_check.py has no new problems (tools/zig/lean_new.py against origin/main)
• Every changed behavior has a test that fails without the change
• If .cu or .metal files changed, the kernel contract was updated

[ashhart]

Thanks @diwakergupta, this is a substantial piece of work and the speed and memory numbers on a 24 GB card are
impressive. We rebased it onto current main (clean) and fmt, zig build test, check-native-cuda-host and the
qwen3_5 host steps all pass. Before we can land it, a few things need to change, mostly because some of it reaches
paths other models already use.

1. Prompt arithmetic. Dense.prompt now sends prompts above 16 rows to the candidate tile kernel for every
   EXL3 checkpoint. That includes the Qwen3.5-MoE EXL3 checkpoints main already serves, so their prefill bits
   change, and it contradicts exl3.md ("a prompt row's bits equal its decode-row bits"). A separate prompt
   arithmetic can be exact when the engine keeps prompt ends, but here the kernel depends on whether a chunk has
   more than 16 rows. Could you keep the default as it is and opt the dense path in explicitly, with receipts on
   the bf16 cache? We need drafted == "draft": false, a resumed prompt == a fresh one (with tails of 16 and 17
   rows past a kept state), and concurrent == solo. Please also update exl3.md.
2. Counters. Leaning on the kernel's own reset is right. But zeroing a grown buffer with fill8(0, null) runs
   on the legacy stream, and the engine's stream is non-blocking. Please zero on the call's stream, and refresh the
   "zeroed before every call" comment.
3. Pinned embedding. If open fails after allocMapped, host_embed is never freed, about 2.5 GB of pinned
   memory per failed open. An errdefer in Engine.init would cover it.
4. Default paths. The scratch union applies to every qwen3_5 CUDA checkpoint, and copy_match = 20 to the 27B
   in every format. We'll A/B both on the default MLX 27B on a GB10. Splitting them into their own PR would make
   that review quick.
5. Housekeeping. After rebasing, qwen3_5/cuda_engine.zig is 603 lines (the limit is 600). Please state the
   licences of turboderp/Qwen3.8-27B-exl3 and the DFlash drafter, and update docs/recipes/qwen3.8-27b.md,
   which still says CUDA reads no EXL3 checkpoint.

A split would help: (a) dense EXL3 admission, DFlash2 on EXL3 targets and the head slicing; (b) the prompt tiles;
(c) the memory changes; (d) the copy policy. Then (a) could land first once it shows the exactness receipts. Thanks
again. The dense EXL3 path is something we want.

[diwakergupta]

@ashhart thanks for the feedback! I've split this up into 4 new PRs and addressed all the feedback. We can safely close this PR.

[ashhart]

Thanks @diwakergupta, agreed: we'll close this one now that it's split up. Each of the asks from our earlier comment has landed in one of the new PRs, which made this easy to follow. For the record, here is where each part goes. The dense EXL3 path continues in #1465, which makes the same change as #1474 and is already queued for our exactness receipts on a GB10. The prompt tiles are #1475, the memory changes #1476 and the copy policy #1477. We've left a note on each. The drafter's EXL3 head column views from #1474 would be welcome as a small PR on top of #1465.