Repository navigation
Proposal: DFlash2 on HIP for Qwen3.8-27B (gfx1151: 2.0-2.5x one stream, same tokens) #1563
Description
Summary
The HIP engine refuses --drafter today ("the HIP engine runs no DFlash2 drafter yet"). I have a working port of the CUDA DFlash2 design to HIP for the dense qwen3_5 family: the HIP target taps the drafter's layers into a per-stream ring, a HIP drafter implements core/lanes Drafter (chains first), and the native HIP server wraps the target with Drafted. Drafted output equals plain in every check I ran. On a Ryzen AI Max+ 395 (gfx1151), Qwen3.8-27B decodes 2.0-2.5x faster on one stream and 10-17% faster at four.
Before opening anything I'd like to agree the split, order and a few open points with you, since it is about 5k lines.
Results
Environment: GMKtec EVO-X2, Ryzen AI Max+ 395 (gfx1151), 128 GB unified memory (Linux sees 107 GiB), Ubuntu 26.04.1, kernel 7.0, Ubuntu's HIP runtime 7.1.0 (libamdhip64 7.1.0) and hipcc (clang 21.1.8), Zig 0.17.0. Target: TensorFold/Qwen3.8-27B-MLX-4bit at revision 22d8d538154e0e5b6f8dcb1fcc74a60a0798104c (all three shards' sha256 match the Hub's). Drafter: z-lab/Qwen3.8-27B-DFlash2 (model.safetensors sha256 67fc76d6...4bab65c), quantized on load to affine-4 g64 with the CUDA drafter's arithmetic. Branch base: b30d5b9d.
Served decode, tensorfold-native serve --context 4096 --parallel 4 --prompt-cache-gib 0, tools/bench_concurrent.py --levels 1,4 --tokens 256 --reps 3 --temperatures 0 --alone --serial, plain and drafted alternated A1 B1 A2 B2 (spread within 0.3 tok/s per arm):
| plain tok/s | DFlash2 tok/s | ||
|---|---|---|---|
| code, 1 stream | 13.65 | 33.8 | 2.48x |
| chat, 1 stream | 14.05 | 28.25 | 2.01x |
| code, 4 streams (aggregate) | 45.75 | 53.6 | +17.2% |
| chat, 4 streams (aggregate) | 45.45 | 50.05 | +10.1% |
- Every cell:
--aloneand--serialfound no unequal reply, and the token hashes are the same in all four runs. - First-token latency moves by 0.02-0.06 s.
- Prompt speed and cold prefill: not measured yet. The drafter only changes decode, but I can add the 2k-64k receipts if you want them.
Exactness:
tf-qwen35-test check <27B> --drafter <DFlash2>gives 49 passed / 0 failed / 1 skipped. The fourlanes {short,long} {greedy,seeded t=1}: drafted == serialchecks pass, as do solo == together, graph == eager, resumed and radix under the drafter.- Without
--drafter, the check gives 45 / 0 / 5, as before. - A planted control that keeps one rejected draft fails all four drafted == serial checks (36 / 10 / 1).
Drafter quality against the plain target's greedy continuation (3 chat prompts x 160 steps): the first draft is right 85% of the time, with a mean acceptance length of 4.86 tokens a round. That is in the README's 4.1-5.5 range at block 8.
What it changes
| # | Change | Lines (approx.) |
|---|---|---|
| 1 | Target taps: tf_plan_tap / tf_tap_rows. The ring address is a slot-descriptor word before the page table, and prepare_features / features go in the HIP lanes backend. Contract cases and records are included, plus a taps invariant (ring == traced residuals). |
~630 |
| 2 | Drafter config and weights for HIP: the window checked against the tap ring, and CUDA's host affine-4 quantize, which is already the layout HIP's MLX projections read. | ~770 |
| 3 | Drafter kernels, as a new draft code-object group: gathers from the tap ring, KV ring writes, dynamic conv, mask rows, windowed block attention with split-K partials and merge. Float64-referenced kernel tests, contract cases and records are included. |
~1400 |
| 4 | The HIP Draft engine: absorb and hold, chains, frames batched over up to 16 streams. | ~1100 |
| 5 | Server wiring: taps set before streams are sized, the drafter counted in the memory plan, and Drafted. It adds refusals for --tp > 1, other --drafter-bits values and other HIP families, benchmark --backend rocm --drafter, and tf-qwen35-test check --drafter. |
~450 |
| 6 | Depth control for shared rounds: a hold runs only the rows its drafts need, so the lane core's step_ms x drafts price is true and its existing rule drafts shallower at four streams. Before this, 4-stream chat was 13.5% slower than plain. |
~60 |
Nothing changes in CUDA, Metal or core/lanes. With no --drafter, the HIP plan and startup line are unchanged.
Questions
- Split and order: would you take it as the six chained PRs above, or another split?
- Contract records: which hipcc should the new records come from? My toolchain (Ubuntu clang 21) reports about 513 existing records as stale against yours, and I left those untouched.
- Coverage: I only have gfx1151. The new kernels have records for gfx1030, gfx1100, gfx1151 and gfx1201, but I have only run them on gfx1151. Is that acceptable for an opt-in path, with the other chips listed as unrun?
- Earlier work: the branch also sits on two earlier HIP stream-tile changes that are not upstream either (row-block interleave for wide rounds, and code pairs made once with 4-bit BF16 pairs from bytes). Together they make the 27B verify for 9-16 rows about 12% faster. They are independent of DFlash2. Would you like them proposed first, separately?
Known limits: chains only, with trees left for later; one rank, so --tp > 1 is refused; prompts reach the drafter whole, because HIP's prefill step doesn't report fill yet.
Activity
AbrahamCete commentedon Oct 28, 2026
/atc claim
AbrahamCete commentedon Oct 29, 2026
Two receipts that were missing from the proposal, measured on the same machine (GMKtec EVO-X2, gfx1151) with the same checkpoint and drafter.
Cold prefill. I used
tensorfold-native serve --context 66560 --parallel 1 --prompt-cache-gib 0, with a fresh server for each run, and measured withtools/bench_cache.py --require-cold. Every request reportedcached_tokens = 0. The runs alternated U P D U P D, giving 6 cold samples per cell. U ismainat 7c06889, P is the branch without--drafter, and D is the branch with--drafter.mainat every size.Against llama.cpp on the same machine. I ran llama.cpp b11739 (Vulkan) with
ggml-org/Qwen3.8-27B-GGUFat revisiona316ea52: the Q4_K_M model, plusdflash-Qwen3.8-27B-Q4_0.gguffor--spec-type draft-dflashandmtp-Qwen3.8-27B-Q4_0.gguffordraft-mtp, with-ngl 99 -fa on -c 16384 --parallel 4. Both engines were measured with the samebench_concurrent.py --levels 1,4 --tokens 256 --reps 3 --temperatures 0, with the arms alternated twice. Results in aggregate decode tok/s:Caveats:
n_max = 3drafts. Its acceptance was about 0.60, and I did not tune it.llama-benchgives pp512 at 394.6 tok/s, against about 238 tok/s for TensorFold HIP at 2k cold. I'm looking at that separately, and it is not part of this proposal.Branch status. The branch is rebased on
mainat 7c06889 with no conflicts and revalidated. I'll rebase it again over #1542; one line inzig/build/hip.zigwill need a trivial merge. It also includes follow-ups from review: a pinned acceptance bound for truncated frames, a floor on the measured draft step, and a test that--no-draftsloads no drafter. It still waits on your answer about the split and the records toolchain before I open anything.ashhart commentedon Oct 30, 2026
Thanks @AbrahamCete. A planted control that fails all four drafted == serial checks, the cold-prefill table and the
llama.cpp comparison answer most of what we'd have asked.
--drafter.(hipcc and clang versions) in the PR. We'll re-derive every HIP record with one pinned hipcc before landing, and
from then on the contract doc will name it, as it does nvcc for CUDA.
from the compile. If anyone with one of those chips can run the drafter checks, that would be welcome.
change the 27B's verify path for everyone on HIP, so they need before/after receipts and tokens equal to main.
#1542 is on main now, so please rebase over it before opening. Chains only and
--tp > 1refused are fine as statedlimits.
AbrahamCete commentedon Oct 30, 2026
Thanks @ashhart, that's clear.
main. One thing from those receipts: on Qwen3.6-35B-A3B MTP at four streams, the cheaper wide rounds tip the lane core's draft depth into a slower mode more often. That is -4.5% on average in code, with the same bytes, andmainhas that mode too. The details are in perf(hip): wide stream rounds reuse cached weights, and a chunk's code pairs are made once #1694 and lanes: on HIP, Qwen3.6-35B-A3B MTP at four streams falls into a slower, deeper-drafting mode that holds per server process #1695.1 and 3. The DFlash2 branch sits on perf(hip): wide stream rounds reuse cached weights, and a chunk's code pairs are made once #1694's two commits. Once perf(hip): wide stream rounds reuse cached weights, and a chunk's code pairs are made once #1694 is reviewed, I'll open it as six chained PRs in the proposal's order, rebased over
main(it already includes build(hip): opt-in -Dhip-relax-branches for gfx1100 under clang 21 #1542), with gfx1030, gfx1100 and gfx1201 listed as unrun. If you'd rather see the first one sooner, I can open it now with perf(hip): wide stream rounds reuse cached weights, and a chunk's code pairs are made once #1694's commits underneath.ashhart commentedon Oct 31, 2026
Thanks @AbrahamCete. Please wait for #1694's review before opening the series, so the six PRs sit on a settled base
and nobody rebases twice. The Qwen3.6-35B-A3B MTP note (-4.5% in code at four streams, the same bytes) belongs with
#1694 and #1695, and we'll look at it there.