Pre-sm70: remaining launch-config fixes on the Qwen3.8-Flash-Next path #628

Before you start

• I have read the Roadmap and this is not already planned there.
• I have read the FAQ.
• I have searched existing issues and found no duplicate.

Applies to

Engine

What problem does this solve

Pre-sm70 GPUs are outside the supported range in the FAQ, but #605, #612, #617 and #618 are bringing the Qwen3.8 path up on them. These are the remaining launch-config and capability fixes on that path that have no issue or PR yet. All were measured on a GTX 1080 Ti (sm_61) with Qwen3.8-Flash-Next NVFP4. They are gated below sm_70, and their outputs are identical or within the existing test tolerances.

Proposed solution

1. QSA indexer score (qsa_mqa_paged): use 32-key tiles. That means 1 tile per program on 4 warps for ≤ 32 rows, and 32 tiles per program on 1 warp otherwise. The default 64-wide 2-warp tile pads 4 index heads to a 16-wide FMA dot. Results:

   • a 2048-row chunk at 64K: 1401 -> 151 ms per layer
   • the same at 262K: 5694 -> 666 ms
   • a decode row at 64K: 5.9 -> 1.55 ms

   gdevenyi@3ff2d7e73904

2. QSA sparse attend: 16-wide tiles on 4 warps, plus a retry ladder that halves the tile on OutOfResources, because the byte estimate undercounts Triton's operand staging at the 48 KiB limit. A 2048-row chunk drops from 599 to 145 ms per layer. gdevenyi@1308d030a874, gdevenyi@0f767f2a31b4

3. 32-bit stream memops: pre-Volta reports CAN_USE_64_BIT_STREAM_MEM_OPS = 0 even with NVreg_EnableStreamMemOPs=1, so the CPU MoE flag handshake and the PLE row-store wait fell back to host funcs. Probing 64-bit first and then 32-bit works: the flags only hold small values, so the 32-bit ops act on the low word of each little-endian int64 flag. Decode goes from 18.15 to 19.08 tok/s. gdevenyi@45adf4d11cd3

4. NVFP4 experts:

   • Decode e2m1 codes arithmetically (place exp/mantissa in an fp16 field) instead of through the LUT gather, which costs an L2 round trip per element on Pascal.
   • Use sm_61-swept tiles for the decode GEMV and the prefill GEMM.

   With these, the expert GEMVs run about 12× faster. gdevenyi@e71faea3afc5, gdevenyi@fe1ac1059258. Related: perf(moe): dequant nvfp4 prefill experts by bit placement and load activations contiguously #578 dequantizes the prefill weights by bit placement too.

5. MXFP8 GEMV: the dot GEMV's M_TILE 256 needs 128 KB of shared memory and runs on FMAs, so take dequant + cuBLAS for M > 1. Self-lower the cap on OutOfResources, and use 512-wide K tiles for M = 1 (~330 GB/s of fp8). gdevenyi@84e642dd3d7d

6. Staged moe_align (the atomic-free path of fix(moe): route moe_align to the atomic-free staged path below sm_70 #612): it took token counts as constexprs, so almost every request's tail chunk recompiled three kernels inside the request. They are now runtime arguments. gdevenyi@5ded569cc15b

Alternatives considered

Leaving pre-sm70 to fail or to run at ~1% of FMA peak.