[sb32445]

Overview

Rebased on current prism (c96baf4ab) and squashed to one commit; this fixes the build error from the review (ptq1_0_pt_smem_bytes has a new parameter) and the test hunk that no longer applied. The environment switches are gone, the test cases are now part of the PR.

The dedicated PTQ1_0 kernel (mul_mat_vec_ptq1_0_pt) is templated for gate + up + SwiGLU fusion with any column count, but the graph check, an assert in ggml_cuda_mul_mat_vec_q and the launcher only allowed ncols_dst == 1. So gate + up + SwiGLU ran as three kernels in every FFN block of a 2-4 token verify step (speculative decoding, small batches).

• The fused path is allowed for the dedicated kernel (plain 2D, K a multiple of 128, shared memory within the limit; the graph check calls the same helper, ggml_cuda_mmvq_ptq1_0_can_fuse_mc, so guard and launch agree). With more than one column the bias tensors have to be contiguous. Other types keep the one-column rule.
• In 24 of 64 FFN blocks of the Bonsai 2 verify graph the fusion was refused because ggml_cuda_check_fusion_memory_ranges saw the GLU output overlap src1 (ggml-alloc hands it the block src1 released). The overlap is real when the q8_1 rows written by ggml_cuda_try_fwht_q8 live in that buffer, so the check was right. Now ggml_cuda_try_fwht_q8 puts the rows into a pool block when a following GLU of a consumer pair overlaps the buffer (the mechanism it already has for out_aliases_in), and the check ignores an overlap with an input whose q8 rows are registered in a pool block. All 64 FFN blocks fuse.

RTX 4070, Bonsai 2 27B PTQ1_0 + MTP head (n-max 2), q4_0 K/V: decode 115.98 -> 117.16 tok/s (+1.02 %, 95 % CI [+0.96, +1.08] %, 8 interleaved pairs), outputs identical in all runs, acceptance unchanged.

Additional information

• One commit, 5 files (+83/-11), no environment switches. g_fwht_q8_ctx is reset by a small scope guard after the graph evaluation (it is read only while a graph is evaluated).
• Tests: new MUL_MAT_VEC_FUSION cases for gate fused PTQ1_0 mat-vec with 2 to 4 columns (538 -> 546 cases).
• What the tests cover: test-backend-ops exercises the fused kernel (2 to 4 columns) but not the buffer aliasing handled by the pool block, because that depends on how ggml-alloc places the tensors of a real graph. For the aliasing I rely on identical outputs of the full model. The original comment in ggml_cuda_try_fwht_q8 describes the failure mode (rows stored early over input that other blocks still read: sporadic garbage in the recurrent state), so please review that part with it in mind.
• I did not prove by inspection that the fused SwiGLU epilogue is bitwise equal to the separate GLU kernel; I only observed identical outputs.
• Measured on one Ada GPU and one model family. HIP: the helper returns false there.
• Earlier numbers of this PR (environment-switch A/B on an older base): the fusion alone +1.01 %, keeping it over the overlap another +0.33 %. These are not repeated here because the switches no longer exist; the number above is the whole change against plain prism.

Test results

• Hardware / software: RTX 4070 12 GB (AD104, cc 8.9), Linux 6.18, NVIDIA driver 615.71, CUDA 13.4, GCC 16.2; Release build, -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89.
• Base: prism at c96baf4ab (builds without errors, test-backend-ops run on it).
• Model: Ternary-Bonsai-2-27B (PTQ1_0) with a community MTP draft head (Q8_0), --spec-type draft-mtp --spec-draft-n-max 2 (the verify step runs 3 columns), q4_0 K/V cache, one slot, context 114688.
• Method: separate builds of prism and the branch, 4 greedy prompts x 256 tokens, alternating runs, 8 pairs, paired differences with a 95 % bootstrap interval (minimum detectable effect 0.10 %).
• Decode: 115.98 -> 117.16 tok/s, +1.02 % (CI [+0.96, +1.08] %), outputs identical in all runs, acceptance 74.0 % in both.
• Cost of removing the change from the full patch stack (separate measurement, my stack on prism): -0.65 % at depth 0 and -0.37 % at 114688 tokens.
• test-backend-ops test -b CUDA0: MUL_MAT_VEC_FUSION 546/546 (538 on prism), MUL_MAT 1669/1669, SWIGLU 24/24, RMS_NORM 51/51.
• Not tested: other GPUs, more than 4 columns (they keep the old path), HIP, models other than PTQ1_0, depths other than the short-prompt benchmark in this run.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: The patches were developed with Claude Code (Anthropic's coding agent): it wrote the code, the measurement scripts and the first drafts of the commit messages and PR texts. I decided what to work on (which kernels and host paths to optimise, based on profiles of my own decode setup). The measurements and checks listed in the PR texts were run in the Claude Code sessions; I did not re-run them independently. I will maintain the changes. Commits where Claude Code was used carry a Co-Authored-By trailer.

🤖 Generated with Claude Code

[bri-prism]

Agent review: posted by the maintainer's coding agent at their request.

This needs a rebase. Against current prism (73bfe6c), the CUDA build fails:

ggml/src/ggml-cuda/mmvq.cu(1602): error: too few arguments in function call
      const size_t smem = ptq1_0_pt_smem_bytes((int) (src0->ne[0] / 128), ncols_dst, (int) src0->ne[1], true);

ptq1_0_pt_smem_bytes has gained a parameter on prism since this branch was cut, and the test-backend-ops hunk no longer applies either. I'll re-run the H200 A/B (test-backend-ops MUL_MAT_VEC_FUSION, greedy identity, 2-4-column decode) once it's rebased.

[sb32445]

Rebased on current prism (c96baf4ab) and squashed to one commit (869bc93a0). The ptq1_0_pt_smem_bytes build error is fixed, the test cases are now part of the PR, the environment switches are gone. MUL_MAT_VEC_FUSION 546/546 on CUDA0 (RTX 4070), decode +1.02 % (CI [+0.96, +1.08] %), outputs identical. The PR description is updated. Ready for the H200 re-run whenever convenient.