vulkan : add GGML_VK_MMV_MAX_COLS override for mul_mat_vec dispatch (#25356)

[djcdevelopment]

Overview

Adds the ability to set batch size at runtime with a variable vs the hardcode 8 currently set.

static constexpr uint32_t mul_mat_vec_max_cols = 8;

Increase the number and increase the thruput. Only works is model fits completely in VRAM., Influences dense models more. I got a 6.9x gain from Gemma-3-27B.

Additional information

Benchmarking ladders, no change comparison at default setting, methodology.
github.com/djcdevelopment/vulkancliff

Requirements

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES (Claude). I designed the experiments, monitored the results, and approved at each stage. Claude built the test harnesses, executed the tests and correlated/summarized all the results. Technically Claude found the line of code that was causing me pain as well.

[ggml-gh-bot]

Hi @djcdevelopment, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[0cc4m]

Please do not use AI like that, it is unreadable and blocks actual conversation. If you have something to say, write the text yourself as per contribution guidelines and also keep it as short as possible.

[djcdevelopment]

Thanks for the cross check and RADV data. docs updated; you're credited.
https://github.com/djcdevelopment/vulkancliff/blob/main/data/data-correctness.md#3-perplexity-ab-end-to-end-numerical-comparison--corrected-2026-08-24

...cross-vendor confirmation on Xe2/Windows is exactly what #25356 was missing, and I'm glad the env-var name from the test builds turned out to be a usable interface.
[...]
On this GPU matmul beats MMV at B=16 by ~42 %, so the crossover sits between 12 and 16, and the local optimum is 12, not 16.

Practical consequence for the "per-vendor defaults can come later" line: please don't carry 16 over to RDNA/RADV without measuring. Default 8 plus the env knob, as this PR has it, is the right shape; a per-vendor default of 16 would be a regression here at exactly the batch sizes speculative decoding lands on.

Agreed on all of it. Your B=16 inversion is exactly why this PR keeps the default at 8 and proposes no per-vendor raise. Nothing should get carried to any vendor without that vendor's measurements. For what it's worth, Xe2's crossover sits past 16 (MMV 158.5 vs matmul 45.4 at B=16 on Mistral-24B), which confirms your point from the other side: the crossover is vendor physics, not a constant.

[...]
But I don't think "PPL identical to every digit" should be offered as evidence of equivalence unless the ubatch actually put dispatches inside the window — otherwise the check passes without touching the code under test.

Fair catch. I ran A/B at the default ubatch, so it never touched the path it claimed to test. Re-ran it your way on Xe2: -ub 12 -b 12 -c 512 --chunks 12, Mistral-Small-24B Q4_K_M, one Arc Pro B70:

GGML_VK_MMV_MAX_COLS	path at ub=12	PPL	s/pass
8	matmul	6.3148 ± 0.28436	40.7
12	MMV	6.3130 ± 0.28423	5.8
16	MMV	6.3130 ± 0.28423	3.1