Vulkan: batched decode throughput cliff at n_tokens=9 on many-expert MoE (fixed 8-token thresholds in MMV dispatch)

[ftoleedo]

Summary

Batched decoding of a many-expert MoE model (Qwen3-Coder-Next 30B-A3B, 512 experts, top-10,
Q4_K_XL) on an AMD Strix Halo iGPU (gfx1151, RADV, Vulkan backend) shows a large throughput
cliff at exactly 9 concurrent sequences: aggregate TG drops from 122.5 t/s (B=8) to
82.9 t/s (B=9) and only recovers above B≥24. I traced it to two fixed batch-size thresholds
in the Vulkan backend that switch from the mul_mat_vec kernels to the tiled matmul kernels
above 8 tokens. Raising them removes the cliff on this hardware/model: +56% at B=9, +34% at
B=16, +20% at B=32, with monotonic scaling through B=32 and no regression at B≤8.

Environment

• HW: AMD Ryzen AI MAX+ 395 (Strix Halo), Radeon 8060S iGPU (gfx1151), 128 GB unified LPDDR5X
• Driver: RADV (Mesa), Vulkan backend, coopmat enabled
• llama.cpp: reproduced on b9293 and on current master (20a04b2)
• Model: Qwen3-Coder-Next-UD-Q4_K_XL (30B-A3B MoE, 512 experts, top-10)

Reproduction

llama-batched-bench -m Qwen3-Coder-Next-UD-Q4_K_XL.gguf -c 32768 -ngl 999 -fa 1 \
  -ctk q4_0 -ctv q4_0 --no-mmap -npp 512 -ntg 128 -npl 1,2,4,8,9,12,16,24,32

Aggregate TG t/s, master (20a04b2), stock:

B	1	2	4	8	9	12	16	24	32
stock	45.4	69.0	96.2	122.5	82.9	93.1	103.8	118.9	130.9

The cliff is at the B=8→9 boundary and recovers slowly — a dispatch-boundary signature, not
bandwidth saturation. GGML_VK_DISABLE_COOPMAT=1 makes B=16 worse (86.4), so it is not a
coopmat tile issue: any mul_mm path at small n loses to the MMV path on this bandwidth-bound
device.

Root cause

Two fixed thresholds switch away from the mul_mat_vec kernels above 8 tokens:

1. mul_mat_vec_max_cols = 8 (ggml-vulkan.cpp) — dense projections;

2. ggml_vk_use_mul_mat_vec_id(): src2->ne[1] <= 8 — the MUL_MAT_ID expert path, which
   dominates. GGML_VK_PERF_LOGGER=1 shows the expert matmuls growing super-linearly across
   the boundary (per-call, decode phase):

   • MUL_MAT_ID q4_K m=512 n=10 k=2048 n_expert=512: 166 µs at batch=8 → 440 µs at batch=12 (2.65×)
   • MUL_MAT_ID q5_K m=2048 n=10 k=512 n_expert=512: 213 µs → 488 µs (2.29×)

   Dense MUL_MAT_VEC entries scale sub-linearly per token across the same boundary (healthy).

The <= 8 heuristic makes sense for few-expert MoEs (8-expert Mixtral-style), where tokens
share experts and the gather-based mm-id amortizes weight reads. For a 512-expert/top-10
model, token–expert overlap at these batch sizes is near zero (B=12 → 120 draws over 512
experts), so switching to mm-id amortizes nothing and pays the tile-inefficiency cost.
vec-id dispatches per token (descriptor sets = nei1) and has no structural limit.

Experiment: raising the thresholds

Local change: mul_mat_vec_max_cols 8→16, and the vec-id gate ne[1] <= 8 → <= 32.

B	8	9	12	16	24	32
stock	122.5	82.9	93.1	103.8	118.9	130.9
patched	126.5	129.4	135.7	139.0	150.9	157.2

• Monotonic through B=32; B=1–4 unchanged (49.1/71.5/99.5 vs 45.4/69.0/96.2).
• test-backend-ops test -o MUL_MAT and -o MUL_MAT_ID pass vs CPU.
• temp=0 output is byte-identical to stock (same model/prompt).
• End-to-end (llama-server, real prompts, ~1600-tok streams): +41% aggregate at 16 concurrent
  slots, J/token floor drops 0.61 → 0.506 on this device.

Suggested direction

A flat constant is probably wrong in both directions — for few-expert MoEs the mm-id switch
is right where it is; for many-expert MoEs vec-id keeps winning far beyond 8. A density-based
gate for MUL_MAT_ID would capture both regimes, e.g. stay on vec-id while expected rows per
activated expert is low:

use_vec_id = n_tokens * n_experts_used <= C * n_expert   // C ≈ 1–2, to be tuned

(Qwen 512/top-10 → vec-id up to ~50–100 tokens; Mixtral 8/top-2 → switches at ~8, like today.)
For the dense mul_mat_vec_max_cols, per-arch tuning similar to what PR #20905 did for the
CUDA backend may apply; on gfx1151/RADV, 16 cols showed no downside in my runs.

I'm happy to run benchmarks/validation on this hardware (Strix Halo / gfx1151 / RADV) if
useful — full benchmark logs and the perf-logger dumps are available.

[FJR-Data]

Independent cross-hardware validation of the density-based gate you suggested above — implemented and benchmarked on a different GPU family (AMD RX 7900 XTX, RDNA3, vs. your gfx1151/RDNA3.5 Strix Halo), for whoever picks this up.

Implementation: exactly your suggested formula, use_vec_id = n_tokens * n_experts_used <= C * n_expert, with C = 2 and a cap at n_tokens <= 64 (empirical, not guessed — beyond that the tiled path wins regardless of density). Left the separate dense mul_mat_vec_max_cols = 8 threshold untouched, per your note that it likely needs its own per-arch tuning (à la #20905 for CUDA) rather than being folded into this gate.

Correctness: test-backend-ops -o MUL_MAT_ID: 850/850 baseline → 870/870 with the gate (added coverage for 128-expert/top-8 and 512-expert/top-10 shapes at n_tokens = 9/16/32/64, matching Qwen3-class configs — there was no existing coverage in that many-expert × token-count range).

Kernel-level A/B (q4_K, production-shaped): 128e/top-8, m=768 k=2048, n=9: 263.4µs → 47.5µs (5.5x); n=32: 2.9x. 512e/top-10: up to 6.7x. n≤8, n≥64, and low-expert-count (32e) shapes: 0.95–1.03x (no regression).

End-to-end (llama-batched-bench, OLMoE-1B-7B Q4_K_M, fully offloaded): stock has the same cliff signature you describe, −41% at B=9 (1214→714 t/s); gated version removes it (1120 t/s, +57% at B=9), +47% at B=12, +31% at B=16, B=24/32 within ±1% of stock. A residual (smaller) dip at B=9 remains, consistent with it coming from the untouched dense threshold rather than this gate.

Caveat: both this run and yours are single-datapoint on C and the 64 cap — RDNA3 + RDNA3.5 agreeing is encouraging but it's still only 2 GPU families.

Disclosure in the interest of your project's stance on AI-authored contributions (AGENTS.md): this investigation, implementation, and benchmarking was run autonomously by an AI agent (Claude) at my direction, not hand-written by me. I'm not opening a PR for that reason — posting here as a validated reference implementation + cross-hardware data point in case a maintainer wants to take it from here. Happy to share the branch/diff directly if useful.

[ftoleedo]

Cross-validation of the density gate on gfx1151 (RDNA3.5) — closing the loop from the other direction: I ran your exact implementation (use_vec_id = n_tokens * n_experts_used <= 2 * n_expert, cap n_tokens <= 64) on the hardware from the original report. With this we now have 2 GPU families × 2 independent implementations × 3 models, all agreeing.

Setup: Strix Halo / Radeon 8060S iGPU (gfx1151, RDNA3.5), RADV, 128 GB unified. Base = master 20a04b2. Single binary with the gate behind an env var (GGML_VK_DENSITY_GATE=1) and the dense threshold runtime-selectable (GGML_VK_MMV_MAX_COLS, default 8 = stock), so every A/B below is the same build — no rebuild noise. Same bench protocol as the report: llama-batched-bench -c 65536 -ngl 999 -fa 1 -npp 512 -ntg 128.

Correctness: full test-backend-ops test -o MUL_MAT_ID passes with the gate enabled.

Qwen3-Coder-Next 30B-A3B (512 experts / top-10, Q4_K_XL) — aggregate TG t/s

B	1	2	4	8	9	12	16	24	32	48	64
stock	46.5	68.4	95.9	122.3	80.8	92.3	102.4	118.7	129.5	102.3	139.9
gate	46.8	68.4	96.6	122.9	112.0	123.5	133.6	145.9	151.8	159.8	164.7
gate + dense 16	47.7	69.8	98.8	125.3	128.8	134.4	138.2	148.0	153.6	161.4	156.5

• Gate alone: +39% at B=9, gains through B=64, no regression at B≤8 — so the cap of 64 holds on RDNA3.5 too (for 512e/top-10 the density formula alone would allow ~100 tokens; the cap is doing its job as the tiled-path crossover).
• New datapoint: stock has a second dip at B=48 (102.3, −21% vs B=32) that wasn't in my original report; the gate removes it (+56%).
• Your residual-dip observation replicates exactly: gate-only still dips at B=9 (112.0 vs 122.9 at B=8), and it's the dense threshold — sweeping mul_mat_vec_max_cols at B=9: 8 → 112.0, 10 → 127.0, 12 → 126.6, 16 → 128.8 (monotonic curve through B=48 at 16). This matches the flat-threshold experiment from the issue body (129.4) while keeping the correct density form for the expert path, and supports handling the dense side as per-arch tuning à la Optimize MOE GEMV kernel for BS > 1. #20905, separate from this gate.

OLMoE-1B-7B-0125-Instruct (64 experts / top-8, Q4_K_M) — your protocol

B	1	2	4	8	9	12	16	24	32
stock	205.7	289.9	369.7	446.4	274.1	322.8	382.9	481.4	569.9
gate	212.8	290.0	372.6	447.9	419.8	456.9	494.3	482.5	570.1

Δ: +53% (B=9), +42% (B=12), +29% (B=16), ±0% (B=24/32) — against your RDNA3 end-to-end run (+57/+47/+31/±1): agreement within a few points on a different GPU family. Few-expert regimes (B×top-k > 2×64 from B≥17) hand off to the tiled path exactly as intended.

Summary for whoever picks this up

• Density gate n_tokens * n_used <= 2 * n_expert, cap 64: validated on RDNA3 (RX 7900 XTX) and RDNA3.5 (gfx1151/RADV), two independent implementations, three model shapes; removes the B=9 cliff (+39–57%), fixes a previously unreported B=48 dip, no regressions observed outside the target regime.
• The remaining B=9 dip is the dense mul_mat_vec_max_cols = 8; on gfx1151/RADV the optimum is ~16 (dip gone from 10 upward). Per-arch treatment like the CUDA side (Optimize MOE GEMV kernel for BS > 1. #20905) seems right.
• C = 2 and the 64 cap now have two GPU-family datapoints each; other vendors (NVIDIA/Intel Vulkan) still untested.

Disclosure, same as above: this run was executed by an AI agent (Claude) at my direction.

[ftoleedo]

Same env-gated build, same OLMoE-1B-7B llama-batched-bench sweep, on two GPUs. Aggregate
decode S_TG (t/s):

B	AMD gfx1151 stock	AMD gate		NVIDIA 5070 Ti stock	NVIDIA gate
8	444	375		555	555
9	216	356 (+65%)		596	560 (−6%)
12	235	363 (+54%)		715	565 (−21%)
16	279	355 (+27%)		876	570 (−35%)
24	355	346		1172	1172

(AMD: 1 sweep, consistent with the rigorous multi-rep result on Qwen3.6-35B-A3B, gate +62% at B=9.
NVIDIA: 6 clean reps, B=1-ratio sanity filter, std ≤0.01.)

On AMD the n_tokens<=8 heuristic drops off the fast path one step too early → a −51% cliff at
B=9 (444→216); the density gate recovers it. On NVIDIA there is no cliff (decode rises
monotonically) and forcing the same path regresses 6–35% at B=9–16. So the change must be
conditioned on the AMD/RADV driver (or a runtime/vendor-detected parameter), like the per-arch
tuning in the CUDA backend (#20905). A blanket change regresses NVIDIA-Vulkan users.

[ftoleedo]

Why does AMD cliff but NVIDIA not? I profiled the same MUL_MAT_ID (q4_K, m=1024, k=2048,
n_expert=64) with GGML_VK_PERF_LOGGER=1, at decode B=16 (~2 tokens/expert) and at large batch
(64 tokens/expert, tiles full):

	small-M (B=16)	full tiles	efficiency
AMD gfx1151 (KHR coopmat)	810 GFLOP/s	5,900	13.7 %
NVIDIA 5070 Ti (coopmat2)	2,350 GFLOP/s	22,700	10.4 %

The important correction to my earlier guess: both vendors pad the coopmat tiles at small M
(2 real rows in a 16-row tile ⇒ ~87 % waste; efficiency is ~10–14 % of peak on both). NVIDIA
does not avoid the padding — its matrix cores are simply ~3.8× more powerful (peak 22.7 vs
5.9 TFLOP/s), so even the padded path (2,350) still beats the MMV alternative. On AMD the padded
path (810) falls below MMV (≈1,360 GFLOP/s measured), so switching to MMV wins — that is the
cliff, and why the gate helps AMD but hurts NVIDIA. The crossover is set by raw matrix-core
throughput, not by any NVIDIA-only extension. (gfx1151 exposes only VK_KHR_cooperative_matrix;
the 5070 Ti also has VK_NV_cooperative_matrix2 — but the profiling shows that alone doesn't
rescue small-M, so it's not the deciding factor here.)

[ftoleedo]

The MMV gate is a band-aid — even MMV only reaches ~23 % of AMD's peak. The cure is to fill the
coopmat tiles at small tokens-per-expert:

1. Segmented / grouped mul_mat_id — pack the tiny (1–2 row) per-expert GEMMs of several
   experts into full 16-row coopmat tiles. Eliminates the padding, uses the matrix cores properly.
   Helps both vendors (lifts AMD from 810 and NVIDIA from 2,350 toward their peaks); on AMD it
   would rise above MMV and make the gate unnecessary. Highest-value, structural.
2. Wave/subgroup + tile tuning per arch (e.g., wave32 vs wave64 on RDNA3.5) — smaller gain,
   arch-specific.

Where: ggml/src/ggml-vulkan/vulkan-shaders/mul_mm_id*.comp + the coopmat pipeline selection
and mul_mat_id dispatch in ggml-vulkan.cpp.

The hardware can already do it — ROCm proves the ceiling. On the same gfx1151, the ROCm/HIP
backend (rocBLAS/hipBLASLt) has no cliff and is 2–2.6× faster than Vulkan in this batch
range (OLMoE decode S_TG t/s):

B	Vulkan stock	Vulkan+gate	ROCm/HIP
8	444	375	264
9	216	356	294
12	235	363	565
16	279	355	724
32	429	425	880

Vulkan only wins at B≤8 (single/low stream); from B≥12 ROCm is ~2× even the gated Vulkan, and
ROCm B=16 (724) approaches the NVIDIA-Vulkan number (876). So the small-M MoE matmul is a
solvable Vulkan-backend problem — rocBLAS already packs these tiles well; the Vulkan
mul_mat_id should aim for the same.

I can share the full profiling traces (both vendors), the batched-bench data, and re-run anything.
Data provider here, not proposing to author the shader — flagging the root cause + headroom so the
fix can be scoped. (AI-assisted analysis; measurements on my hardware, reproducible.)

[hogeheer499-commits]

I repeated this on gfx1151/RADV from official llama.cpp b9979 (4114ba18b208c2e9c1689a8316d410e649309dbc) with controlled cooldowns and two local MoE shapes.

The test build keeps upstream behavior by default and exposes two independent opt-in switches:

• GGML_VK_DENSITY_GATE=1: n_tokens <= 64 && n_tokens * n_experts_used <= 2 * n_expert
• GGML_VK_MMV_MAX_COLS=16: separately selects the ordinary dense MMV threshold

This is deliberately not a blanket default because of the NVIDIA-Vulkan regression already documented above.

Protocol: llama-batched-bench, pp512/tg128 per sequence, Q4_0 KV, 65,536-token context. Each controlled run started below 50 C. Values below are aggregate decode t/s, mean (standard deviation).

Qwen3-Coder 30B-A3B, 128 experts / top-8

Route	np8	np9	np12	np16
b9979 stock	228.18 (0.04)	147.19 (0.09)	177.02 (0.08)	212.66 (0.06)
density	228.36 (0.21)	210.07 (0.08)	236.70 (0.05)	266.07 (0.17)
density + dense16	228.11 (0.08)	234.12 (0.58)	246.13 (0.76)	227.20 (0.11)
Lemonade ROCm b1259	191.78 (0.38)	200.89 (0.49)	240.79 (1.10)	287.64 (0.84)

Five Vulkan repeats and three ROCm repeats. Stock loses 35.5% from np8 to np9. Density recovers 42.7% versus stock at np9 without materially changing np8; dense16 reaches +59.1% at np9, but regresses versus density alone at np16.

Qwen3-Next 80B-A3B, 512 experts / top-10

Route	np8	np9	np12	np16
b9979 stock	144.88 (0.24)	100.15 (0.13)	113.26 (0.06)	126.61 (0.12)
density	145.02 (0.57)	125.48 (0.11)	138.46 (0.24)	150.82 (0.21)
density + dense16	144.76 (0.35)	142.72 (0.30)	144.97 (0.38)	131.08 (0.10)
Lemonade ROCm b1259	112.19 (0.37)	116.90 (0.36)	130.50 (0.36)	143.32 (0.39)

Three repeats per route. Stock loses 30.9% from np8 to np9. Density recovers 25.3% versus stock at np9; dense16 reaches +42.5%, but again falls behind density alone at np16. This artifact matches the 512-expert/top-10 topology, although it is not the exact model artifact from the issue body.

Correctness checks:

• stock, density, and density+dense16 each passed 790/790 Vulkan MUL_MAT_ID backend tests
• stock and density+dense16 each passed 956/956 ordinary Vulkan MUL_MAT tests
• fixed-prompt, temperature-zero generated text was identical across the three Vulkan modes

Controlled repeats stayed at or below 87 C for 30B and 85 C for 80B. The Lemonade result is a useful existing ROCm comparator, not a same-commit backend A/B.

The practical conclusion matches the profiling above: the density gate is a useful AMD/RADV recovery policy, but the dense threshold is workload/concurrency-sensitive and should not simply become 16 globally. Backend crossover is also model-dependent: ROCm leads this 30B matrix at np16, while tuned Vulkan density leads this 80B matrix at np16.

Evidence and patch:

• repeat tables, method, correctness, and interpretation
• repeat-aware summary CSV
• raw logs, telemetry, wrappers, host snapshot, and opt-in patch

Execution and write-up were AI-assisted at my direction; measurements were run locally and the raw artifacts are public.

[djcdevelopment]

I upgraded my whole system: new CPU, 4x the RAM, new OS, a year-newer driver,
same two Arc Pro B70s. And hit the exact same 8-stream bottleneck, at the exact
same count?! Turns out it's a hard-coded limit in the Vulkan dispatch that acts
as a concurrency ceiling, and it hits dense models hard on the non-CUDA vendors
(the CUDA backend already merged its own override in #26079).

So, adding the data point this thread is missing, from a different vendor, OS,
and code path: the same cliff exists on Intel Arc (Xe2) under Windows, and on
the dense mul_mat_vec side rather than the MUL_MAT_ID side discussed
above. Dense models fall off it much harder than the MoE numbers in this thread,
and on this hardware the fix is not a band-aid: the raised MMV path beats the
general matmul path at every batch size I can reach.

Setup: Intel Arc Pro B70 (32 GB, Xe2, matrix cores: KHR_coopmat), Windows 11,
driver 32.0.101.8974, master e85caa8 built with MSVC + Vulkan SDK 1.4.357.
Protocol matches this thread: llama-batched-bench -c 65536 -ngl 999 -fa 1 -npp 512 -ntg 128, B swept 1..32, warm reps reported (first use of each NUM_COLS
pipeline variant pays a compile tax), cool-start gated (<50 C), and VRAM
spill-guarded (a run that silently demotes to shared memory is discarded, not
reported).

Dose-response. I patched mul_mat_vec_max_cols into a runtime knob (same
GGML_VK_MMV_MAX_COLS name the test builds above used; default 8 = stock
behavior, clamped 1..16, one file) and ran the same binary at limits 4 / 8 / 16.
Mistral-Small-24B Q4_K_M, single card, aggregate decode S_TG (t/s):

B	limit 4	limit 8 (stock)	limit 16
4	78.3	78.2	78.4
6	23.3	106.4	107.0
8	29.9	125.0	125.2
9	19.7	19.7	131.5
12	29.4	29.4	147.4
16	45.5	45.4	158.5
24	92.4	92.1	92.3
32	140.1	139.5	139.5

The cliff lands at exactly the first batch past whatever the limit says, and past
the cliff all three configs converge on the same matmul floor. On this GPU the
MMV path at B=16 (158.5) outruns the matmul path at B=32 (139.5).

It is not one model. Seven models, stock vs GGML_VK_MMV_MAX_COLS=16, same
binary, aggregate S_TG:

model	arch	B=9 stock	B=9 @16	B=16 stock	B=16 @16
Phi-4 14B	dense	28.7	195.1 (6.8x)	65.8	240.3 (3.7x)
Gemma-3-27B	dense	16.1	110.3 (6.8x)	35.6	131.4 (3.7x)
Qwen2.5-32B	dense	12.8	88.1 (6.9x)	29.1	108.0 (3.7x)
Mistral-Small-24B	dense	19.7	130.6 (6.6x)	45.4	158.4 (3.5x)
Llama-3.3-70B (2 GPU, -ts 1,1)	dense	7.3	25.4 (3.5x)	16.1	29.9 (1.9x)
gpt-oss-20b	MoE	34.8	63.2 (1.8x)	63.4	106.7 (1.7x)
Qwen3-30B-A3B	MoE	65.8	74.3 (1.1x)	111.4	134.1 (1.2x)

Two details that speak to the discussion above:

• The dense/MoE split is clean and supports the two-gate picture: the MoE models
  only gain the share of their compute that goes through dense mul_mat_vec
  (attention projections); their expert FFNs sit behind the MUL_MAT_ID gate
  this thread already covers. On the dense side there is no coopmat-tile-filling
  alternative to wait for; the dispatch gate itself is the binding constraint.
• On Gemma-3-27B stock, the matmul path never catches the B=8 MMV number at any
  batch size I tested (B=32 reaches 96.3 vs 106.8 at B=8). With the stock
  constant, that model's total throughput can never exceed its 8-batch ceiling on
  this GPU.

Everything is public — the one-file patch, every raw batched-bench output, the
harnesses (cool-start gated and VidMm-spill guarded), and per-dataset writeups:
https://github.com/djcdevelopment/vulkancliff

Correctness. test-backend-ops -o MUL_MAT -b Vulkan0 with
GGML_VK_MMV_MAX_COLS=16: all pass (cols 9..16 exercised against the CPU
reference). At the default the patched build is stock within noise (+-3 % on a
pp/tg sanity bench vs the b10549 prebuilt). Perplexity cross-check env 8 vs 16
(same text, 12 chunks, Mistral-24B): Final estimate: PPL = 6.3139 +/- 0.28430
under both settings, identical to every printed digit.

I've opened #27652 with the env knob (default 8, so stock behavior is untouched;
GGML_CUDA_MMVQ_MAX from #26079 is the CUDA precedent). Per-vendor defaults can
come later if other Xe2/RDNA data confirms; the NVIDIA numbers earlier in this
thread suggest it should stay at 8 there.

Disclosure: this investigation and the patch were done with an AI assistant
(Claude); I directed the work and reviewed the code and every number by hand.