Repository navigation
CUDA: add IQ1_M to MMQ - #304
danielhanchen wants to merge 4 commits into
Conversation
IQ1_M weights are w = d*(2*s+1)*(g + delta) with g in {-1,0,1}, a 3 bit
sub-scale s per 16 values and delta = +-1/8 per 8 values. The delta is folded
into the int8 tile values (8*g +- 1 in {-9,-7,-1,1,7,9}) and the per-16 scale
becomes d*(2*s+1)/8, which is exact. The tile then uses the same Q8_0_16 data
layout and vec_dot (DP4A and MMA) as IQ2_XS, with D4 q8_1 activations.
Adds test-backend-ops coverage for IQ1_M/IQ1_S MUL_MAT and MUL_MAT_ID with
many experts, and perf cases for 1-bit MoE up projections.
IQ1_M uses the IQ2_XS/IQ2_S tile layout, so its switch cases, loader, template declaration and generator entry now sit beside IQ2_S. No functional change.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 64b2a7e7bd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // IQ1_M: w = d * (2*s + 1) * (g + delta), g in {-1, 0, 1}, delta = +-1/8 per 8 values, s 3 bit per 16 values. | ||
| // The per-8 delta is folded into the int8 values: 8*(g + delta) = 8*g +- 1 in {-9, -7, -1, 1, 7, 9}, | ||
| // and the per-16 scale becomes d * (2*s + 1) / 8. This is exact, the dot product then uses the | ||
| // same 16-value-scale data layout and vec_dot as IQ2_XS (Q8_0_16 / SRAM layout Q3_K). |
There was a problem hiding this comment.
This new four-line comment exceeds the repository's usual one-to-two-line limit and splits a sentence across lines 1317-1318, making the kernel rationale harder to scan; condense it to one or two comment lines without wrapping mid-sentence.
AGENTS.md reference: AGENTS.md:L77-L84
Useful? React with 👍 / 👎.
| // row counts with and without the out-of-bounds fallback, and MoE shapes with many experts. | ||
| for (ggml_type type_a : {GGML_TYPE_IQ1_M, GGML_TYPE_IQ1_S}) { | ||
| for (int64_t k : {256, 1024, 4096, 5376}) { | ||
| for (int64_t n : {1, 2, 8, 9, 16, 64, 128, 512, 2048}) { |
There was a problem hiding this comment.
Despite the preceding claim that every MMQ J tile is covered, on the Ampere non-fallback path these batch sizes only select J=16, 64, or 128 once n<=8 stays on MMVQ. The newly enabled J=24, 32, 40, 48, 80, 96, and 112 configurations therefore receive no correctness coverage, so a tile-specific indexing or launch regression can pass this suite; include sizes such as 17, 25, 33, 41, 65, 81, and 97 that actually select those specializations.
Useful? React with 👍 / 👎.
On Turing and newer, MMQ for dense IQ1_M is slower than dequantization + cuBLAS once the batch reaches 128 tokens (B200, k = 4096: 2.0x-2.3x slower at n = 512). Keep MMQ for MoE (MUL_MAT_ID), where it is ~10x faster, and for dense batches below 128.
|
Pushed 33fdcde: dense IQ1_M at batch sizes of 128 and up goes back to dequantize + cuBLAS on Turing and newer; MoE ( Routing every batched dense IQ1_M matmul through MMQ was slower than base at larger batches on a B200 (m=4096, n=512: 56 us on base, 128 us with MMQ). Medians of 3 interleaved rounds on an exclusive B200, k=4096, relative to base (A/A spread in brackets):
MMQ still wins at 96 tokens and loses at 128, so the cut is Not measured: other NVIDIA architectures (the crossover was taken on B200), AMD (unchanged by this commit), and end to end llama-bench. The pin in |
Summary
IQ1_M has no MMQ kernel in the CUDA backend. MMVQ covers it, so decode is fine, but every batched IQ1_M matmul falls back to dequantize + cuBLAS. For MoE models (MUL_MAT_ID) that fallback is a per-expert loop and dominates prompt processing. This adds IQ1_M to
mul_mat_qfor both MUL_MAT and MUL_MAT_ID.On
unsloth/Qwen3-30B-A3B-Instruct-2507-GGUFUD-IQ1_M, prompt processing on a B200 goes from about 2500 to 7700 t/s (3.1x at pp512). Decode is unchanged: n <= 8 still runs MMVQ.Pin for the prebuilt: one change on top of upstream
d80619397(b11756), the base the nightly picks now.Design
An IQ1_M weight is
w = d * (2*s + 1) * (g + delta), withgin {-1, 0, 1} from the 11-bit grid index,delta = +-1/8(one sign per 8 values),sa 3-bit sub-scale per 16 values anddthe fp16 super-block scale. The per-8 delta is folded into the integer and the per-16 scale is divided by 8:After the load the tile is "int8 values with one float scale per 16 values", the same format IQ2_XS, IQ2_S and Q3_K already use, so no new dot product is needed:
MMQ_Q8_1_DS_LAYOUT_D4(the delta is inside the int8, so no activation-sum term).MMQ_DP4A_TXS_Q8_0_16(DP4A) and the Q3_K SRAM layout (MMA).ggml_cuda_mmq_vec_dot_q8_0_16_q8_1_dp4a/_mma, unchanged. Each 16-value integer dot is bounded by 169127, so int32 cannot overflow.ggml_cuda_mmq_load_tiles_iq1_m, 8 threads per 256-value row.mmq-config-*.cuh; on RDNA3 IQ1_M gets the same batch-size cap as IQ2_XS/IQ2_S.generate_cu_files.pyentry. The IQ1_M entries sit next to IQ2_S, so the change does not touch the lines around IQ1_S.Correctness
test-backend-ops, CUDA vs CPU, B200 (sm_100):
1242be66fEnd to end,
llama-perplexity --kl-divergenceon wiki.test (c = 1024, 48 chunks),unsloth/Qwen3-30B-A3B-Instruct-2507-GGUFUD-IQ1_M. The reference is the same file run on the CPU backend (CUDA_VISIBLE_DEVICES=,-ngl 0), so the KLD measures only how far each CUDA path is from the CPU result for the same weights:1242be66f)CPU PPL on the same chunks: 7.7312 +- 0.1270. The MMQ path is as close to the CPU result as the old path (the KLD difference is inside the error bars), so output quality is unchanged.
Speed
llama-bench -p 512,2048 -n 128 -ub 512,2048 -b 2048 -r 3 -ngl 99,unsloth/Qwen3-30B-A3B-Instruct-2507-GGUFUD-IQ1_M, one exclusive B200, interleaved base / head / base / head / base / head (3 rounds of 3 reps each). Base = upstream master1242be66f, head = the same commit + this change (the MMQ files are identical between b11756 and that master). Values are the per-round means; the A/A spread is the base range over its 3 rounds.tg128 is unchanged: decode stays on MMVQ (n <= 8), which this change does not touch. The one low head round (165 / 166 t/s) coincided with a CPU-heavy job on the same host; the other two head rounds match base.
A separate decode-only rerun on an idle exclusive B200 (
-p 0 -n 128 -r 5, base / head interleaved, 4 rounds each): base 253.3, 252.7, 252.7, 253.5 t/s (median 253.0, A/A spread 0.3%), head 253.3, 251.8, 252.8, 252.4 t/s (median 252.6). The -0.2% difference is inside the A/A spread.Pin set
llama_pinset.py simwith this pin appended to the currentpr-set.json: merges cleanly and every pin stays intact (pin_contract) on b11756, b11757 and b11759.verify --cudaon b11756 and b11759: CPU gate, full CUDA build, test-chat, test-backend-ops MUL_MAT 2582/2582, MUL_MAT_ID 1027/1027, PAD 37/37, GET_ROWS 349/349, and all 13 feature checks on GPU.Not tested