HIP Flash Next on gfx1151: results and approach of a complete branch, as a reference for the open series #1661

We have a gfx1151 HIP path for Qwen3.8-Flash-Next that we measured and tested on an AMD Strix Halo (gfx1151) machine with 192 GiB of unified memory and ROCm 7.15. This issue records the results and the approach. A draft pull request will refer to it as a reference. It does not compete with the HIP Flash Next series that is open now.

Relation to the open HIP Flash Next work

These issues and pull requests are in the same area, and our branch touches many of the same files:

• issues: HIP Flash Next: WMMA sparse attention for prompt chunks on gfx11 #1597, HIP Flash Next: the loader requires the MTP head under language_model.mtp, so oMLX checkpoints fail #1598, HIP Flash Next: a shared expert wider than the routed experts cannot load #1599, HIP: the prompt GEMM pays a group-32 scale and bias fold every K stage #1600, HIP: a 16/32-row routed-expert prompt tile for many-expert MoE (Flash Next) #1601, HIP Flash Next: the first request after load decodes at half speed #1602, HIP Flash Next: about 850 small kernels per verify round cost a fifth of decode #1603, HIP Flash Next: the prompt's n-gram gather is a single-threaded host copy #1606, Flash Next on HIP: which tensors an oQ checkpoint can narrow for bandwidth-bound decode #1617, HIP Flash Next on gfx1151: defaults for prompt step and MTP drafts #1649;
• pull requests: hip flashnext: parallel n-gram gather, release weight pages after upload #1611, hip: load a shared expert wider than the routed experts beside the stack #1612, hip flashnext: take the MTP head's root from the index #1613, hip: decode router over the bf16 rows the checkpoint holds #1614, hip flashnext: random-access advice on the n-gram table mappings #1615, hip flashnext: WMMA sparse attention for prompt chunks (sparse=wmma) #1622, hip: an opt-in prompt GEMM tile that stages dequantized bf16 weights (dq=on) #1650, hip: 4096-row prompt chunks by default on the RDNA 3.5 iGPUs #1651, hip: a 64-row routed prompt tile for many-expert plans on gfx1151 #1658.

Areas that overlap, by topic:

• prompt GEMM tiles (dense and routed expert): HIP: the prompt GEMM pays a group-32 scale and bias fold every K stage #1600, HIP: a 16/32-row routed-expert prompt tile for many-expert MoE (Flash Next) #1601, hip: an opt-in prompt GEMM tile that stages dequantized bf16 weights (dq=on) #1650, hip: a 64-row routed prompt tile for many-expert plans on gfx1151 #1658;
• WMMA sparse attention for prompt chunks: HIP Flash Next: WMMA sparse attention for prompt chunks on gfx11 #1597, hip flashnext: WMMA sparse attention for prompt chunks (sparse=wmma) #1622;
• kernel count and host work in a verify round: HIP Flash Next: about 850 small kernels per verify round cost a fifth of decode #1603, hip: decode router over the bf16 rows the checkpoint holds #1614;
• prompt chunk and MTP draft defaults: HIP Flash Next on gfx1151: defaults for prompt step and MTP drafts #1649, hip: 4096-row prompt chunks by default on the RDNA 3.5 iGPUs #1651;
• loader and table handling: HIP Flash Next: the loader requires the MTP head under language_model.mtp, so oMLX checkpoints fail #1598, HIP Flash Next: a shared expert wider than the routed experts cannot load #1599, HIP Flash Next: the first request after load decodes at half speed #1602, HIP Flash Next: the prompt's n-gram gather is a single-threaded host copy #1606, hip flashnext: parallel n-gram gather, release weight pages after upload #1611, hip: load a shared expert wider than the routed experts beside the stack #1612, hip flashnext: take the MTP head's root from the index #1613, hip flashnext: random-access advice on the n-gram table mappings #1615.

atc overlap finds textual conflicts with #1612, #1614, #1622, #1650, #1651 and #1658. #1613 and #1615 merge without conflicts.

We can split our work onto those pull requests, pair with their author, or review them, as the maintainers and the author prefer. We do not ask to replace that series.

Results of the fork's numerics=inexact path (not in #1667)

These numbers and charts were measured with the rounding-class kernels described below, under --policy numerics=inexact. #1667 no longer contains those kernels; its exact-path receipts will be posted there.

Served, greedy, thinking off, one stream, prompt cache off.

Checkpoint TensorFold/Qwen3.8-Flash-Next-MLX-4bit-MTP. --policy numerics=inexact. A 17,715-token coding prompt and 1,024 output tokens. Medians of 4 timed runs, with a fresh server for each arm.

One more change (the next layer's expert down codes read on a side stream during one-row decode) adds +2.6 % to serial decode (38.24 / 38.32 to 39.24 / 39.30 tok/s on two seeds) with the same text. Default MTP and prefill do not change.

A prompt-length sweep (1,024 output tokens each) gives:

The first 64K and 128K runs were contended by other host activity; both points were re-measured on a quiet machine, and the table and charts show the re-runs. Default MTP decode follows the text's acceptance more than the length. Peak memory is the Default MTP arm (serial is within 0.6 GiB); host memory is the n-gram tables.

Findings on the rounding-class kernels (not proposed; to fix before any later proposal)

The fork's rounding-class kernels ran only under --policy numerics=inexact. They are not part of #1667, because they do
not meet CONTRIBUTING's rules:

1. FP64 references. On identical input rows, every rounding-class kernel's error against an independent float64
   reference is larger than the exact path's, in aggregate and in the worst column. Per change and projection class
   (rms_rel inexact / exact; worst-column error inexact / exact):

The one-launch HC up+mix wrote the chain's bytes on these rows, but its reordered sum does not guarantee that.
On decode rows the prompt-only changes run the exact kernels. All lines: fp64-references.txt.

2. Teacher-forced fidelity (numerics=exact against numerics=inexact, 65,520 positions each):

Every difference, with both runs' top-1 and top-2 logits and margins:
wikitext-2, CPython.
No tie policy is applied.

3. A lead: one large deviation. At wikitext position 4031 the exact path predicts the target with a margin of 10.1,
   and the inexact path predicts another token (15.38 against 15.25). That is far beyond rounding noise and points to an
   edge case in one of the rounding-class kernels; it is not bisected yet.

4. The result depends on how the rows are grouped. Under numerics=inexact, a turn resumed from 10,240 cached
   tokens differed from a fresh run of the full prompt (6 rewind replies), and a sampled decoder changed while a long
   prompt ran beside it. Both contracts hold under numerics=exact. Two causes:

   • some rounding-class kernels are chosen by how many rows a launch holds, so the same row takes another arithmetic
     path in a different batch;
   • the chunked (WY) DeltaNet scan computes each row relative to its 32-token chunk's start, so a row's bytes depend on
     the prompt's chunk grid, and the row-by-row decode scan sums it another way.

5. Kernel contract gate 4 (sums). On the fork's inexact tip, gfx1151: 104 kernels: 97 pass gates 1-4, 7 fail.
   Every failure is in the sums gate:

   • affine_wmma_dense_inexact<2560> and <6144>, affine_wmma_moe_down_inexact<4> and affine_wmma_router<2560>:
     142x to 219x (router 11.4x) over the exact-sum bound, "not fp32 sums". The matrix cores' accumulation on gfx1151
     is not an exact fp32 sum.
   • tf_gdn_wy_prep and tf_gdn_wy_scan (y): 12,416x over the bound at k 128.
   • tf_fn_idx_scores_mma and the WY state: the stress inputs do not separate fp32 from fp16 sums (2.6x and 0.0x,
     the gate needs 100x), so the gate cannot pass these as written.

Each of these must be fixed, and each change given its own gate here, before any of these kernels is proposed again.

Approach

• Prompt (in hip: Qwen3.8-Flash-Next on gfx1151, a reference branch for review (to be split) #1667, same bytes as the base path): WMMA tiles for the dense and routed-expert GEMMs, the DeltaNet prompt recurrence and its prework in fewer launches, the indexer at 4 lanes a score, a prompt's picks and route plan, the router tile.
• Prompt (fork only, rounding class, see Findings): the chunked (WY) DeltaNet scan, the matrix-core indexer scores and sparse attention tiles, bf16-dequantized prompt tiles.
• Decode and verify: a router with a fused top-k pick, fused small kernels in the verify round, and a side stream that reads the next experts' down codes.
• hip: Qwen3.8-Flash-Next on gfx1151, a reference branch for review (to be split) #1667 now holds only changes whose output bytes equal the base path's. The rounding-class kernels above stay on a fork branch.

The pull request gives the list of its changes, their contract registration and the test output.

🤖 Generated with Claude Code