vulkan: use small Lightning Indexer CM for batches 4-15

[pepuscz]

Overview

DSpark speculative verification can send 2–15 target tokens to the Lightning Indexer in one step. In measurements using Strix Halo llama.cpp v0.6.6, the cooperative-matrix shader lightning_indexer_cm_small_f16 was faster than the scalar lightning_indexer_f16 indexer for batches 4–15. The patch therefore keeps batches 2–3 on the scalar path and routes only batches 4–15 through small-CM.

Additional information

lightning_indexer_cm_small_f16 is selected for DSpark verifier batches 4–15
using:

GGML_VK_LIGHTNING_INDEXER_SMALL_CM=1

v0.6.6 versus 4–15 small-CM patch

Prompt tokens	Allocation	v0.6.6 prefill	v0.6.6 + patch prefill	Change	v0.6.6 decode	v0.6.6 + patch decode	Change
2,040	128K	32.36 tok/s	32.07 tok/s	-0.91%	40.96 tok/s	40.79 tok/s	-0.41%
59,933	128K	235.49 tok/s	235.00 tok/s	-0.21%	35.02 tok/s	36.48 tok/s	+4.16%
122,879	128K	218.08 tok/s	215.96 tok/s	-0.97%	30.55 tok/s	32.60 tok/s	+6.71%
163,840	256K	207.28 tok/s	206.92 tok/s	-0.17%	28.51 tok/s	30.47 tok/s	+6.87%
212,992	256K	193.79 tok/s	193.56 tok/s	-0.12%	25.98 tok/s	28.59 tok/s	+10.04%
245,760	256K	186.25 tok/s	185.88 tok/s	-0.20%	24.50 tok/s	27.21 tok/s	+11.09%
491,520	512K	146.28 tok/s	145.81 tok/s	-0.32%	17.19 tok/s	20.01 tok/s	+16.37%

Benchmarked on the same dataset as Strix Halo DeepSeek V4 Flash repository.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES for benchmarking

[Nathanw1014]

Hey @pepuscz very much appreciated for your work on this, the correctness numbers came our great, It did lead me down the next path of optimisations, ive got a new set of commits carried in the below branch :

https://github.com/Nathanw1014/llama.cpp/tree/strix-halo-vulkan-indexer-gen2

https://github.com/Nathanw1014/strix-halo-llamacpp/releases/tag/dev-20260826-7c025e5

Your work was what let do these changes, id be keen for you to test it and see what further performance gains can be had.

If you woul dlike to test, can you please test with and without the spec decoding:
llama-server -m DeepSeek-V4-Flash-UD-IQ3_XXS-00001-of-00004.gguf -ngl 999 -c 524288 -fa on -ctk q8_0 -ctv q8_0

llama-server -m DeepSeek-V4-Flash-UD-IQ3_XXS-00001-of-00004.gguf -ngl 999 -c 524288 -fa on -ctk q8_0 -ctv q8_0 -md DeepseekV4-Flash-20260731-DSpark.gguf -ngld 999 --spec-type draft-dspark --spec-draft-n-max 4

[aic0d3r]

Independent confirmation on a second rig — thanks @pepuscz, this patch (via the v0.7.0 release) checks out end to end on my machine.

Rig: ASUS Flow Z13 (Ryzen AI MAX+ 395 / 8060S, 128 GiB), CachyOS, official v0.7.0 portable, UD-IQ3_XXS + DSpark drafter (Q2_K_S), --spec-draft-n-max 4, f16 KV. Note: 70 W power cap and stock CPU policy, so my absolute numbers sit below the 120 W tables above — the deltas are what carry over.

One-variable A/B on the same v0.7.0 binary using the GGML_VK_LIGHTNING_INDEXER_DECODE_CM_BATCH=0 kill switch, 41,134-token context, 3 trials per arm:

dispatch	trials (t/s)	mean
4-15 decode-CM (default)	25.92 / 27.40 / 24.31	25.88
kill switch (=0)	25.54 / 24.93 / 24.01	24.83

+4.2% at 41k — right on your curve at an earlier depth than your first table point (+4.16% at 60k), with run-to-run variance of roughly ±1.5 t/s.

End-to-end vs the v0.6.3-beta3-class build (same session, same config): prefill +8.4% at pp2048 (304.5 vs 281.0 t/s) and +8.1% at pp4096; decode flat from 9k to 41k context (25.2 → 25.3 t/s) where the old build slides 26.0 → 24.7. Everything I have is consistent with the depth-scaling model in the release notes.

Your retrieval protocol replicated at 59k tokens (synthetic filler, five keys at 2/20/50/80/98% depth, temp 0 / top-k 1 / top-p 1, 256-token cap, cold KV, thinking off): 29.3 t/s on f16 KV (84.4% draft acceptance, all keys recovered byte-exact); 26.9 t/s on q8_0 KV. The absolute gap to your 36.5 t/s @ 120 W is fully accounted for by the power cap (+~1 t/s per 10 W on this SoC), your CPU 2.0 GHz/boost-off policy freeing package power for the GPU, and q8_0 KV — which your stack needs for the 491k allocation but measures ~9% slower than f16 at mid depths on my box. Full write-up of that replication and the config sweeps: our r/LocalLLaMA guide — DeepSeek V4 Flash 0731 at 27+ t/s decode on Strix Halo (v0.7.0 A/B and the retrieval replication are in the Aug 27 edit note).

One data point that may be useful for the docs: on this fork, --spec-draft-n-max 4 beats both 2 and 64 at ≥8k context on a code-gen workload (n2 24.6 / n3 25.7 / n4 26.0 / n64 22.2 t/s at 9k ctx) — the "2 wins at long context" guidance from the ROCm-side repos doesn't transfer to the Vulkan stack.

[pepuscz]

Independent v0.7.0 qualification result

We qualified the unmodified official strix-halo-llamacpp v0.7.0 portable release, source 95c828e. No local source patch or Lightning Indexer override was applied.

The matched profile used UD-IQ3_XXS, the Q2_K/Q8_0 DSpark drafter, full target and draft GPU offload (-ngl 999, -ngld 999), flash attention on (-fa on), n_max=4, q8_0 K/V, one slot, batch 2048, and microbatch 1024. The comparison baseline was v0.6.6 source 7b6c613 plus the 4-15 small-CM dispatch from this PR, with otherwise identical model and runtime parameters. Each row reused the same frozen prompt hash and allocation with cold KV state, thinking off, temperature 0, top-k 1, top-p 1, a 256-token output cap, and five byte-exact retrieval keys.

Prompt	Allocation	v0.6.6 prefill	v0.7.0 prefill	Change	v0.6.6 decode	v0.7.0 decode	Change
2,040	128K	246.07	245.23	-0.34%	39.55	40.04	+1.24%
3,840	128K	258.87	260.12	+0.48%	38.75	38.98	+0.59%
7,680	128K	252.71	254.07	+0.54%	37.29	37.70	+1.09%
15,359	128K	250.50	254.94	+1.77%	39.89	40.09	+0.52%
30,720	128K	244.95	250.59	+2.30%	38.41	39.48	+2.79%
59,933	128K	235.00	244.19	+3.91%	36.48	37.90	+3.89%
122,879	128K	215.96	226.82	+5.03%	32.60	35.73	+9.60%
163,840	256K	206.92	218.56	+5.62%	30.47	34.12	+11.99%
212,992	256K	193.56	206.65	+6.76%	28.59	32.77	+14.61%
245,760	256K	185.88	200.21	+7.71%	27.21	31.77	+16.74%
491,520	512K	145.81	163.23	+11.95%	20.01	25.02	+25.05%

Units are tokens/s. All eleven v0.7.0 rows recovered 5/5 keys. Geometric-mean improvements were 4.10% for prefill and 7.74% for decode.

For the requested -c 524288 speculative A/B at 491,520 actual prompt tokens, v0.7.0 with DSpark reached 163.226 prefill / 25.021 decode tokens/s; the same binary and configuration without the drafter reached 162.214 / 11.882. DSpark therefore delivered 2.1058x decode throughput, with 51 of 52 drafted tokens accepted.

The reduced public dataset, production manifest, and chart are published here: https://github.com/pepuscz/strix-halo-deepseek-v4-flash/releases/tag/v1.4.0