Skip to content

cuda: add Blackwell (SM120) MMVQ parameter table and tune FATTN config for Blackwell - #3

Open
sunagent wants to merge 1 commit into
z-lab:masterfrom
sunagent:pr-blackwell-mmvq-fattn
Open

cuda: add Blackwell (SM120) MMVQ parameter table and tune FATTN config for Blackwell#3
sunagent wants to merge 1 commit into
z-lab:masterfrom
sunagent:pr-blackwell-mmvq-fattn

Conversation

@sunagent

Copy link
Copy Markdown

Summary

Add a dedicated MMVQ parameter table for NVIDIA Blackwell (sm_120) and tune one flash-attention MMA config case used on Blackwell.

mmvq.cu

SM120 previously fell back to the GENERIC parameter table. This adds MMVQ_PARAMETERS_BLACKWELL with values verified by A/B recompile-and-measure on 2x RTX 5060 Ti (16 GB, sm_120) with a Qwen3.8-27B NVFP4 model (tensor split + MTP speculative decoding):

  • nwarps=1 for NVFP4 at ncols_dst=1 (decode): VDR=8 fits a single warp and skips the shared-memory reduction entirely (measured +5% tg).
  • nwarps=4 for other types at ncols_dst=1 (e.g. Q8_0 output layer; measured neutral vs 2).
  • nwarps=8 for ncols_dst 2-4 (batch path).
  • nwarps=2 for ncols_dst 5-8: this is the main speculative-decode path (MTP verify batches of 5 tokens). Measured +4-5% tg vs the previous nwarps=4 (A/B: 55.9 -> 58.1-58.7 t/s, 500-token runs, server-side tg).
  • rows_per_block=2 at decode (improves SM occupancy; rows_per_block=4 measured neutral).

Device detection: __CUDA_ARCH__ >= GGML_CUDA_CC_BLACKWELL in the device branch, and ggml_cuda_highest_compiled_arch >= GGML_CUDA_CC_BLACKWELL on the host side.

All table entries were chosen from measured A/B comparisons on sm_120; the alternatives (nwarps 1/4/8, rows_per_block 1/2/4) either regressed or were neutral.

fattn-mma-f16.cuh

  • In the Ampere config set (used by sm_120): (256, 256, 8, 64, 4, ..., true) -> (256, 256, 8, 128, 2, ..., false). Q_in_reg=false measured +4-7% at 20K ctx (63.85 vs 59.69 t/s). Note: measured on a single config / short context only.

Test environment

  • 2x RTX 5060 Ti 16 GB (PCIe x16, no NVLink), Ryzen 5 5600, 32 GB RAM
  • Qwen3.8-27B-Cold-Fusion-GAIN-V1.1, NVFP4 weights, q4_0 KV, CUDA graphs, MTP spec decode
  • Long-context (100-200K) sustained 40-50 t/s in production after these changes.

…g for Blackwell

- mmvq: new MMVQ_PARAMETERS_BLACKWELL table (SM120). nwarps: NVFP4 decode=1,
  other types decode=4, batch 2-4=8, batch 5-8=2 (+4-5% tg measured, MTP
  verify path); rows_per_block=2 at decode.
- fattn: (256,256,8,128,2,...,false) Q_in_reg=false, +4-7% measured at 20K ctx.

Measured on 2x RTX 5060 Ti (sm_120) with Qwen3.8-27B NVFP4 + MTP spec decode.
Long-context (100-200K) sustained 40-50 t/s.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant