cuda: add Blackwell (SM120) MMVQ parameter table and tune FATTN config for Blackwell - #3
Open
sunagent wants to merge 1 commit into
Open
cuda: add Blackwell (SM120) MMVQ parameter table and tune FATTN config for Blackwell#3sunagent wants to merge 1 commit into
sunagent wants to merge 1 commit into
Conversation
…g for Blackwell - mmvq: new MMVQ_PARAMETERS_BLACKWELL table (SM120). nwarps: NVFP4 decode=1, other types decode=4, batch 2-4=8, batch 5-8=2 (+4-5% tg measured, MTP verify path); rows_per_block=2 at decode. - fattn: (256,256,8,128,2,...,false) Q_in_reg=false, +4-7% measured at 20K ctx. Measured on 2x RTX 5060 Ti (sm_120) with Qwen3.8-27B NVFP4 + MTP spec decode. Long-context (100-200K) sustained 40-50 t/s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a dedicated MMVQ parameter table for NVIDIA Blackwell (sm_120) and tune one flash-attention MMA config case used on Blackwell.
mmvq.cu
SM120 previously fell back to the GENERIC parameter table. This adds
MMVQ_PARAMETERS_BLACKWELLwith values verified by A/B recompile-and-measure on 2x RTX 5060 Ti (16 GB, sm_120) with a Qwen3.8-27B NVFP4 model (tensor split + MTP speculative decoding):nwarps=1for NVFP4 atncols_dst=1(decode): VDR=8 fits a single warp and skips the shared-memory reduction entirely (measured +5% tg).nwarps=4for other types atncols_dst=1(e.g. Q8_0 output layer; measured neutral vs 2).nwarps=8forncols_dst2-4 (batch path).nwarps=2forncols_dst5-8: this is the main speculative-decode path (MTP verify batches of 5 tokens). Measured +4-5% tg vs the previous nwarps=4 (A/B: 55.9 -> 58.1-58.7 t/s, 500-token runs, server-side tg).rows_per_block=2at decode (improves SM occupancy; rows_per_block=4 measured neutral).Device detection:
__CUDA_ARCH__ >= GGML_CUDA_CC_BLACKWELLin the device branch, andggml_cuda_highest_compiled_arch >= GGML_CUDA_CC_BLACKWELLon the host side.All table entries were chosen from measured A/B comparisons on sm_120; the alternatives (nwarps 1/4/8, rows_per_block 1/2/4) either regressed or were neutral.
fattn-mma-f16.cuh
(256, 256, 8, 64, 4, ..., true)->(256, 256, 8, 128, 2, ..., false). Q_in_reg=false measured +4-7% at 20K ctx (63.85 vs 59.69 t/s). Note: measured on a single config / short context only.Test environment