Skip to content

cuda: tall one-column mat-vec for Q8_0/Q4_0/Q2_0/Q4_K/Q6_K on Hopper, and allow folded Gemma 4 weights - #413

Merged
bri-prism merged 2 commits into
prismfrom
gemma/tall-ext
Oct 29, 2026
Merged

bri-prism merged 2 commits into
prismfrom
gemma/tall-ext

Conversation

@bri-prism

Copy link
Copy Markdown
Collaborator

What

  • Extends the 8-rows-per-block one-column mat-vec geometry, which only PQ2_0 used, to Q8_0, Q4_0, Q2_0, Q4_K and Q6_K on Hopper when the matrix has at least 4096 rows (a multiple of 8). Default on; GGML_CUDA_MMVQ_TALL_EXT=0 restores the old PQ2_0-only behavior.
  • Adds Gemma 4 to the prism.hadamard architecture whitelist.

Why

  • Output heads and other very tall matrices in these types ran as one 128-thread block per row. A 262144 x 2816 Q8_0 head reached only about 1.6 TB/s on H200.
  • Gemma 4 routes every folded weight (attention q/k/v/o, shared FFN, routed experts) through build_lora_mm / build_lora_mm_id, so the activation transform is applied correctly. Until now the loader refused folded Gemma 4 GGUFs.

How

  • Same kernel, same K split and per-row reduction order, so every output row is bit-identical to the default geometry. Only the rows-per-block launch parameter changes.

Testing (H200)

  • test-backend-ops MUL_MAT 1669/1669 and MUL_MAT_ID 1035/1035 pass on this branch (CUDA vs CPU).
  • Greedy output is byte-identical with the switch on and off on several models.
  • No change on PQ2_0 models, which already used the tall path (27B PQ2_0 decode 172.0 vs 172.1 tok/s).
  • Decode: +3.8% on a 2B Q8_0 model and about +7% on a Gemma 4 MoE model with a Q8_0 head.
  • Hopper only; other architectures are unchanged.

…=4096 on Hopper (GGML_CUDA_MMVQ_TALL_EXT)

Extend the 8-rows-per-CTA one-column mat-vec geometry, previously used for PQ2_0 only,
to Q8_0, Q4_0, Q2_0, Q4_K and Q6_K when the matrix has at least 4096 rows (a multiple of 8)
on Hopper. The K partition and reduction order per row are unchanged, so every output row
is bit-identical to the default geometry.

GGML_CUDA_MMVQ_TALL_EXT=0 restores the PQ2_0-only behavior (default: enabled).
Every folded Gemma 4 weight (attention q/k/v/o, shared FFN, routed experts) goes through build_lora_mm / build_lora_mm_id, which apply the activation transform.
@bri-prism
bri-prism merged commit ddfa3dc into prism Oct 29, 2026
8 checks passed
bri-prism added a commit that referenced this pull request Oct 29, 2026
Resolve the prism.hadamard loader conflict by keeping load_hparams_hadamard()
and adding Gemma 4 to its arch whitelist (#413). Add the SWIGLU_CLAMP case to
the PTQ1_0 warp-per-row epilogue from #415, since this branch has that op.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant