Skip to content

cuda: bit-serial Q1_0 dot and four rows per CTA for one-column mat-vec - #420

Merged
bri-prism merged 2 commits into
prismfrom
cuda/q1_0-bit-serial-mmvq
Oct 30, 2026
Merged

bri-prism merged 2 commits into
prismfrom
cuda/q1_0-bit-serial-mmvq

Conversation

@bri-prism

Copy link
Copy Markdown
Collaborator

What

  • One-column Q1_0 mat-vec uses a bit-serial dot. The activation is quantized into a bit-plane layout (GGML_CUDA_Q8_1_Q1B: word j of a 32-block holds elements j, 8+j, 16+j, 24+j, plus the exact integer block sum), and each 32-block becomes 8 dp4a over masked weight bits: sumi = 2*s - isum.
  • One-column Q1_0 launches use four rows per CTA (second commit, one line).

Why

  • The existing Q1_0 mat-vec is limited by instruction issue, not memory bandwidth: unpacking bits into ±1 bytes took about 2.5 byte-permutes per dp4a. The bit-serial form needs no unpacking.

How

  • The integer dot is exact and the float epilogue is unchanged, so outputs are identical.
  • Enabled on NVIDIA from Turing on; HIP, MUSA and the batch-invariant mode keep the existing layout.

Testing

  • test-backend-ops MUL_MAT, MUL_MAT_ID and MUL_MAT_VEC_FUSION pass (CUDA vs CPU).
  • Greedy output is byte-identical before and after on a binary (Q1_0) model.
  • H200 tg128 (alternating runs) on that model: 99.0 -> 103.3 tok/s (+4.3%), 104.4 tok/s (+5.5%) with four rows per CTA. pp512 unchanged.
  • Tested on H200 only.

Quantize the activation for one-column Q1_0 mat-vecs into a bit-plane layout
(word j of a 32-block holds elements j, 8+j, 16+j, 24+j) with the exact integer
block sum, and compute each 32-block as 8 dp4a over masked weight bits:
sumi = 2*s - isum. The integer dot is exact and the float epilogue is
unchanged, so outputs are identical. HIP and MUSA keep the existing path.
@bri-prism
bri-prism merged commit 01ea7fc into prism Oct 30, 2026
8 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant