Skip to content

Sync prism-v7 into prism: PQ2_0 legacy ftype name, shared Hadamard transform, PTQ1_0 - #150

Merged
khosravipasha merged 3 commits into
prismfrom
prism-v7-rebased
Sep 4, 2026
Merged

khosravipasha merged 3 commits into
prismfrom
prism-v7-rebased

Conversation

@khosravipasha

Copy link
Copy Markdown
Collaborator

prism-v7 received three PRs that were meant for prism as well (#134, #137, #148). The branches had diverged (prism carries #133 and doc edits prism-v7 lacks), so this is prism-v7 rebased onto prism: the three commits cherry-picked in order with -x, zero conflicts, history linear.

commit origin
llama: name the pre-rename PQ2_0 ftype instead of reporting unknown #134
hadamard: share the activation transform between folded weights that share an input #137
ggml: add PTQ1_0, ternary at group 128 (1.75 bpw, lossless vs PQ2_0) #148

Verification: git diff prism-v7 prism-v7-rebased touches only .github/workflows/release-prism.yml, AGENTS.md, CONTRIBUTING.md, i.e. exactly the four commits that were only on prism.

Merge with Rebase and merge (it is a fast-forward of prism). Afterwards prism-v7 can be reset to prism so the two lines are identical again.

…134)

ggufs packed before the Q2_0_G128 -> PQ2_0 rename carry the old ftype value 142.
The format did not change, only the enum did, so these files load and compute
correctly but print

  print_info: file type   = unknown, may not work

which reads as a broken file. With this they print

  print_info: file type   = PQ2_0 - 2.13 bpw (group 128, legacy ftype)

Naming only. No load path, no quantizer path, and no behaviour changes; the new
constant exists so the switch has a case for the value.

Worth noting the collision that makes this confusing: 142 is also
GGML_TYPE_PQ2_0, the tensor type id, which is current and correct. Only the
ftype was renumbered, to 141.

Verified on a published gguf that carries ftype 142: reports unknown on the
current branch, reports PQ2_0 with this change, same output either way.
…share an input (#137)

* hadamard: share the activation transform between folded weights that share an input

Folded weights that consume the same activation were each building their own
identical transform. ggml does no common-subexpression elimination, so the
sign-multiply and the FWHT ran once per weight rather than once per activation.

Three groups share an input on this architecture:

  ffn_gate + ffn_up      build_ffn LLM_FFN_PAR, both take cur     64 redundant
  attn_qkv + attn_gate   qwen35.cpp:237,241, both take input      48 redundant
  attn_q + attn_k + attn_v  qwen35.cpp:270,282,285, all take cur  32 redundant

That is 144 of 401 folded weights doing a transform that already exists, so 36%
of the FWHT calls per token were duplicates.

build_lora_mm and build_lora_mm_id now memoize on (input, rotation). Both must
match for the transform to be shared: same input tensor and same rotation, which
also implies the same width and therefore the same sign vector. The map lives for
one graph build.

Measured on M5, Metal, a folded low-bit checkpoint, three interleaved pairs
on a warm machine:

  pp512   358.29 -> 370.74   +3.5%
  tg128    23.20 ->  23.91   +3.1%

Run-to-run spread was at most 1.1 on pp512 and 0.11 on tg128, so the gap is well
outside the noise. Greedy output is unchanged, byte for byte, on the same prompt.

The size of the win matches the prediction: the transform stack was about 10% of
traced GPU time, and removing 36% of it should be worth about 3.5%.

* hadamard: trim the memo comment to two lines per AGENTS.md
…148)

* ggml: add PTQ1_0, Prism ternary at group 128 (1.75 bpw)

Upstream TQ1_0 packs ternary weights base-3 at 5 trits per byte, but carries a
single fp16 scale per 256 weights. The ternary checkpoints this targets carry a
scale per 128, so a 256-wide scale has to discard one of the two group scales it
straddles. Measured on a ternary checkpoint of this family: every trit survives and the zero pattern
is preserved exactly, but the smaller-alpha group of each block is reconstructed
at the larger alpha, giving 6.43% weight error that passes into activations at
full strength because it is multiplicative on the weight rather than noise.

PTQ1_0 keeps TQ1_0's trit packing and our group of 128:

  qs[24] + qh[2] + fp16 d = 28 bytes / 128 weights = 1.7500 bpw

against PQ2_0's 34 bytes = 2.1250 bpw, so 17.6% smaller with no loss of
information: PQ2_0 spends two bits on a value carrying log2(3) = 1.585 bits.

test-quantize-fns reports ptq1_0 identical to pq2_0 on both metrics, absolute
quantization error 0.008678 and dot product error 0.141111, versus tq1_0's
0.008681 and 0.141345. A round trip of real folded ternary weights is
bit-identical to the PQ2_0 reconstruction, so the only error either format
carries is the shared fp16 scale rounding.

A distinct type id (143) rather than changing TQ1_0 in place, for the same reason
PQ2_0 is distinct from upstream's group-64 Q2_0: TQ1_0 is a wire format, and
reinterpreting its block width would silently misread every existing TQ1_0 gguf.

TQ1_0's qs staging is fixed at 32-then-16 bytes, which cannot cover 24, so the
stages are generalised to 32/16/8. At TQ1_0's 48-byte qs that reduces to exactly
its original behaviour.

Scope: codec, ggml traits, CPU vec_dot against Q8_0, and the ftype plumbing so
llama-quantize can target it. The vec_dot decodes a block to element order before
the dot, sharing the traversal with dequantize so the two cannot drift; it is
generic C on every arch, with no SIMD path yet. CUDA, Metal and Vulkan kernels,
and the repack gemv/gemm paths, are not implemented.

* ggml: wire PTQ1_0 into quantize dispatch, ftype map and row validation

Three plumbing gaps the traits entry alone did not cover, each of which failed
end-to-end quantization in a different way:

  ggml_quantize_chunk   - without a dispatch case the returned byte count did not
                          match the row size and GGML_ASSERT(result == nrows *
                          row_size) aborted a worker thread mid-tensor, leaving a
                          truncated output file behind.
  ggml_ftype -> wtype   - the ftype could not resolve to a ggml type.
  validate_row_data     - "invalid type 143", so quantization refused the result
                          it had just produced.

Verified end to end on a 1.7B ternary model with the embedding and output tensors
pinned to f16 in both runs, so only the quantized tensors differ:

  quantized tensors  PQ2_0 357.47 MiB -> PTQ1_0 294.47 MiB, 17.62% smaller
                     (1.75/2.125 bpw predicts 17.65%)
  perplexity, 4 chunks at ctx 512, identical to every printed digit:
                     per chunk 11.3796 / 14.7182 / 14.5014 / 12.6944 for both,
                     final 12.6944 +/- 1.12596 for both

The perplexity run goes through the new CPU vec_dot, so that path is exercised
rather than only the codec.

* metal: PTQ1_0 kernels — correct, but 4.2x slower decode than PQ2_0

Adds the Metal path for PTQ1_0: an element accessor for the base-3 layout, both
dequantize entry points, a mat-vec kernel, mul_mm and mul_mm_id instantiations
for f32 and f16, get_rows, the simdgroup constants and both pipeline-selection
switches. Also enumerates PTQ1_0 in test-backend-ops, which otherwise silently
skipped the new type and reported a clean pass.

Correct: 1251/1251 MUL_MAT tests pass on Apple M5 Pro. End to end on
a 1.7B ternary model the Metal perplexity is identical to PQ2_0 to every printed
digit, 12.6935 +/- 1.12567 for both.

Slow, and the reason is structural. Measured on M5 Pro:

                    PQ2_0            PTQ1_0
  pp128        4505 tok/s       2590 tok/s   0.57x
  tg32          182.5 tok/s      43.6 tok/s   0.24x

Base-3 packing makes element order non-positional, so a thread owning 16
contiguous weights cannot index bytes directly, and threads owning different trit
indices reload the same bytes. Hoisting the trit factor out of the inner loop for
spans below element 80 took decode from 29.8 to 43.6 tok/s, but the remaining gap
needs the thread mapping changed so one thread consumes all five trits of a byte
it loads once, instead of five threads each loading it for one trit.

This reproduces what the earlier TQ1_0 work already found on other backends:
base-3 is ALU-bound rather than bandwidth-bound, and the packing win does not pay
for itself at decode without that restructure. The 17.6% file-size reduction is
real and lossless; the speed is not there yet.

* metal: PTQ1_0 decode 29.8 -> 165.3 tok/s, now 0.89x of PQ2_0

Four changes, each measured on Apple M5 Pro against a 1.7B ternary model, with
MUL_MAT staying 1251/1251 and Metal perplexity identical to PQ2_0 throughout:

  1. byte-owning threads         29.8 -> 136.4   a thread now consumes all five
     trits of a byte it loads once, instead of eight threads each reloading the
     same qs bytes for one trit index; the block's 26 bytes are read once total
  2. y staged in registers      136.4 -> 143.5   y was being re-read from device
     memory once per row; it is now gathered once per block and reused, with the
     ternary offset folded into a single sumy subtraction
  3. N_R0 8 -> 4                143.5 -> 163.8   rows-per-thread was the real
     limiter: 16 rows gives 94 tok/s, 8 gives 144, 4 gives 164, 2 gives 163
  4. balanced qh                163.8 -> 165.3   qh's 8 elements went to threads
     0 and 1 only, so those did 19 elements while the rest did 15 and the whole
     simdgroup waited; one element per thread makes it 16 each and drops yl to 16

Trit extraction is the base-3 remainder recurrence, t = (v*3)>>8 with
v = (v*3)&0xFF, two integer ops per trit with no table. A 256-entry lookup was
tried and is not better for decode: it adds a dependent load per byte, 26 extra
per block against PQ2_0's 32 total loads. It does help the mul_mm path, which
must produce 16 contiguous elements and cannot use the byte-owning mapping, so
dequantize keeps the table.

Final, PQ2_0 vs PTQ1_0 on M5 Pro:

  pp128   4541 -> 3998 tok/s   0.88x
  tg32     186 ->  165 tok/s   0.89x
  size    357 ->  294 MiB      17.6% smaller, and lossless

Prefill's remaining gap is structural: mul_mm's dequantize contract is 16
contiguous elements, so five threads still touch the same 16 bytes for different
trit indices. Closing it needs mul_mm itself to stage a decoded block in
threadgroup memory, which is a larger change than this.

* cuda: PTQ1_0 dequantize and MMVQ paths — NOT COMPILED, no toolchain here

Adds the CUDA surface for PTQ1_0, mirroring PQ2_0: type traits, a device trit
accessor, the float2 dequantize entry point, the six dequant dispatch sites in
convert.cu, get_rows, the MMVQ dot with its VDR constant and three dispatch
sites, and both supports_op type lists.

**This code has never been compiled or executed.** The machine it was written on
is an Apple M5 Pro with no CUDA toolchain, so nvcc was unavailable. It must be
built and run on a CUDA host before anyone relies on it.

What *is* verified is the part where a bug would actually hide. The device
accessor's element mapping is pure integer logic, so it was transcribed verbatim
to the host and checked against the CPU codec's own traversal over 2,560,000
element positions across 20,000 random blocks: zero mismatches. That check is
kept as tests/test-ptq1_0-element-map.cpp so it can be rerun.

Trit extraction uses the same base-3 remainder recurrence as the Metal path,
t = (v*3)>>8 with v = (v*3)&0xFF. The MMVQ dot gathers four trits at a time and
packs them into an int as signed int8 lanes so accumulation still goes through
dp4a, since base-3 packing leaves no byte-aligned run inside a 32-element chunk.

MMQ is deliberately not implemented. Types without it fall back to dequantize
plus cuBLAS for prefill, which is correct if not optimal, and writing an MMQ
tile loader blind against tile layouts I cannot test would be worse than leaving
the fallback in place.

* metal: tune PTQ1_0 to N_SG=1, and record what optimising it ruled out

N_R0=4, N_SG=1. Decode lands at 162-166 tok/s across runs against PQ2_0's 186,
so 0.87-0.89x; the spread between N_SG 1 and 2 is inside run-to-run noise
(+/- 3.4 on a single measurement).

Five further optimisations were tried and MEASURED, and four of them lost. Kept
here so nobody spends the same day twice:

  256-entry trit lookup table       142.1  vs 143.5 recurrence   no decode gain
                                    (helps mul_mm, which keeps it)
  independent pow3 extraction       159.8  vs 166.0 recurrence   WORSE
  contiguous byte ownership         162.0  vs 165.3 scattered    WORSE
  N_R0 = 3 / 5 / 6                  129.8 / 156.7 / 149.6        all worse than 4
  N_SG = 8                          161.1                        worse than 1-2

What that rules out. The lookup table shows decode is not bound by trit
arithmetic. The pow3 variant shows it is not bound by the recurrence's five-deep
serial dependency either -- four rows per thread already supply enough
independent work, so fewer ops beat more ILP. Contiguous byte runs made it worse
because thread 5 straddles the qs 16/8 chunk boundary, and the resulting per-byte
branch cost more than consolidating the loads saved.

Where the remaining 11% actually lives, per 128 weights:

  PQ2_0   128 x (shift, mask, cvt, fma)                = 512 ops, 34 B read
  PTQ1_0  128 x (mul, shift, cvt, fma, mask-for-next)  = 640 ops, 28 B read
          ALU 1.25x, bytes 0.82x

A purely ALU-bound kernel would sit at 0.80x and a purely bandwidth-bound one at
1.21x. Measured 0.89x sits between them, which is about where this instruction
and byte mix should land. Extracting a base-3 digit is inherently a multiply
where a 2-bit field is a shift, so this gap is the price of packing at the
1.585-bit information limit rather than on a 2-bit boundary, not an
implementation defect.

The one lever left with a real mechanism behind it is prefill, not decode. In
decode each row has its own weights so there is nothing to share, but mul_mm
reuses a weight tile across many output columns, and its dequantize contract
forces 16 contiguous elements, so five threads still touch the same 16 bytes.
Staging a cooperatively decoded block in threadgroup memory would remove that
redundancy outright. Prefill is at 0.87x (3946 vs 4492).

Perplexity remains identical to PQ2_0 at 12.6935 +/- 1.12567, MUL_MAT 1251/1251.

* metal, cpu: close PTQ1_0 type-list gaps, add the missing mul_mv_ext r1_5

Self-review compared every switch that names PQ2_0 against those naming PTQ1_0
and found the new type absent from ten support enumerations: seven in
ggml-cpu/ops.cpp, including ggml_compute_forward_get_rows_f32, one in
ggml-metal-ops.cpp and two in ggml-metal-device.m. PTQ1_0 is traits-compatible
with PQ2_0 there -- same block size, same to_float and from_float shape -- so it
belongs in each.

Closing those gaps then exposed a real defect. Adding the type to the Metal
support lists enabled the mul_mv_ext path for wider n, which immediately failed
with "kernel not found in any metal library" for
kernel_mul_mv_ext_ptq1_0_f32_r1_5: PQ2_0 instantiates r1_2 through r1_5 and only
r1_2 through r1_4 had been written, because the grep that enumerated them was
truncated by head. Instantiating r1_5 fixes it, and the instantiation set is now
derived from PQ2_0's rather than transcribed.

Worth noting the earlier 1251/1251 passes were not wrong, they were narrow: with
the type missing from the support lists the wider-n path was never selected, so
the absent kernel could not be reached. The gap and the defect concealed each
other.

After: MUL_MAT 1251/1251 on M5 Pro, Backend MTL0 OK, Metal perplexity identical
to PQ2_0 at 12.6935 +/- 1.12567, decode 165 tok/s and prefill 3947.

* tests: verify the CUDA MMVQ dot numerically on the host

The CUDA path still cannot be compiled here, but its arithmetic can be checked.
This stubs the three intrinsics the kernel uses -- __dp4a, __low2float and the
4-byte int load -- transcribes vec_dot_ptq1_0_q8_1 and the device trit accessor
verbatim, and compares them against a float reference built by dequantising the
block and the q8_1 activations independently.

Result over 5000 random weight blocks with random q8_1 activations:

  exact integer-accumulator checks   20000, mismatches 0
  error / sum of term magnitudes     2.399e-08

The integer test is the one that matters: the kernel accumulates in int via dp4a
before scaling, so if the trit extraction, the int8 lane packing or the chunk-to-
q8_1 pairing were wrong, that accumulator would differ. It does not, on any of
the 20000 chunks.

A first pass reported 7.062e-05 "relative error" and looked like a failure. That
was cancellation: the reference total is a signed sum that can land near zero, so
dividing by it inflates the ratio. Measured against the sum of term magnitudes
instead, which cancellation cannot distort, the error is 2.4e-08, i.e. float32
rounding of the four per-chunk products.

Together with tests/test-ptq1_0-element-map.cpp this leaves only compilation
unproven for the CUDA path, not correctness of its math.

* cuda: move ptq1_0_trit to common.cuh so vecdotq.cuh can see it — BUILDS NOW

First compile of the CUDA path, on an RTX 5090 with CUDA 12.8 and
CMAKE_CUDA_ARCHITECTURES=120, failed with

  vecdotq.cuh(822): error: identifier "ptq1_0_trit" is undefined

The accessor was defined in dequantize.cuh, which vecdotq.cuh does not include.
PQ2_0's dot never needed a shared helper -- it decodes 2-bit fields inline with
byte_perm -- so nothing in the file it was mirrored from exposed the dependency.
Both headers include common.cuh, so the helper belongs there.

Verified on the 5090 after the fix:

  test-backend-ops -o MUL_MAT   1283/1283 passed, Backend CUDA0: OK
                                45 ptq1_0 cases OK, 0 failures
  host element-map test         2,560,000 positions, 0 mismatches
  host MMVQ dot test            20,000 integer checks, 0 mismatches

The two host tests were already green before this, which is the point worth
recording: they verified the arithmetic and could not have caught a missing
declaration. Compilation was the only gate that would find this, and the earlier
commit saying the CUDA path had never been built was accurate about the risk.

* vulkan: PTQ1_0 support, mirrored from Q1_0 not Q2_0 — UNTESTED as committed

Adds the Vulkan path for PTQ1_0: the block struct and QUANT_K defines, the trit
accessor plus dequantize/dequantize4/get_dm, the mul_mm load branch, a standalone
dequant shader, the shader-generator registration, and 16 pipeline sites in
ggml-vulkan.cpp mirrored from Q1_0's.

Mirrored from Q1_0 deliberately. Q1_0 is type 41 at group 128 with correct Vulkan
kernels; Q2_0 in this tree is group 128 in ggml-common.h but its Vulkan side still
declares QUANT_K_Q2_0 64, because upstream's group-64 kernels were merged over a
group-128 type and only some of the GLSL was re-derived. Pattern-matching off Q2_0
would have inherited that. Q1_0 also has no MMVQ entry, so this follows the same
dequantize-based path rather than inventing one.

One divergence from Q1_0 worth naming: PTQ1_0 puts the scale LAST in the block
(qs[24], qh[2], then d) where Q1_0 puts d first. The GLSL struct mirrors the C
field order exactly. Getting that wrong misindexes every block silently instead of
failing, which is the same class of error as the Q2_0 breakage above.

Not wired, and it is a real gap rather than an oversight: on-device quantization
(cpy_f32_quant, set_rows, cpy_quant_f32). Those need a base-3 ENCODER in GLSL,
which is a separate piece of work and is not needed to run a model -- weights are
quantized offline by llama-quantize.

**This has not been compiled or run.** There was no Vulkan toolchain on the
machine it was written on. It is pushed so it can be built and tested on a GPU
box, and it should be treated as unverified until test-backend-ops says otherwise.

* cuda: PTQ1_0 MMVQ takes the whole block — measured, and still 6-9x slower

Benchmarked the CUDA path for the first time. It is correct and slow, and the
numbers matter more than the change: us/run on a 4090 at m=4096, k=14336, against
pq2_0 on the same run.

        n=1     n=2     n=3     n=8    n=512
  q1_0   8.88    9.82   13.05   30.14   225.4
  pq2_0  9.75   11.05   14.92   31.27   232.4
  ptq1_0 46.8    64.6    99.1   287.3   525.6

The per-chunk version asked "which byte holds element e" per element, a branch plus
a variable-length recurrence, and iqs differs per lane so the branch diverged. Taking
the whole 128-weight block per call (VDR 1 -> 4) makes every lane walk the same 26
bytes in the same order. That bought about 9% at n=2, 70.6 -> 64.6, and no more.

So divergence was not the dominant cost, and the comment now says so. What remains is
load shape: this issues 128 scalar int8 reads of the q8_1 activations, one per element,
where pq2_0 issues eight dp4a calls over 4-byte words.

The fix is identified but deliberately not guessed at. Below element 80 the element
stride is 16 and 4 divides 16, so four consecutive elements are four consecutive BYTES
at the same trit index. Four recurrences packed into one int would feed dp4a against a
4-byte q8_1 load, matching pq2_0's shape. That needs a build cycle to verify, and this
session has already shown twice -- on the CPU vec_dot and on the Metal mapping -- that
predictions about where GPU time goes are wrong about as often as they are right.

Correctness held throughout: test-backend-ops -o MUL_MAT 1283/1283 on both an RTX 5090
and an RTX 4090, Backend CUDA0 OK, 45 ptq1_0 cases OK and 0 bad, before and after.

Note n=512 is the cuBLAS dequantize path rather than MMVQ, and is 2.3x rather than 6-9x,
which is consistent with the problem being in the MMVQ load pattern specifically.

* vulkan: fix three build breaks — shared ptq1_0.glsl, skip coopmat2, drop cm2 pipelines

The Vulkan commit did not build. Three errors, one root cause each, all found by a
teammate on an Intel Arc B390 and independently by windows CI:

1. mul_mm_cm2.comp: 'dequantFuncA' undeclared. Adding the type to type_names in
   vulkan-shaders-gen.cpp auto-enrols it in coopmat2 generation, but
   dequant_funcs_cm2.glsl has no PTQ1_0 decoder (Q1_0 has six entries). That failed
   the whole Vulkan shader build, not just this type. Now skipped for ptq1_0, which
   falls back to the scalar and coopmat1 paths that are actually implemented.

2. mul_mm_funcs.glsl: 'ptq1_0_trit' no matching function. The helper lived in
   dequant_funcs.glsl; mul_mm.comp includes types/dot_product/mul_mm_id/mul_mm_funcs
   and never dequant_funcs.glsl. **This is the same bug as the CUDA one fixed an hour
   earlier, same helper, different backend**: defined in the dequantize chain, called
   from the matmul chain. Fixed the same way, with a shared header
   vulkan-shaders/ptq1_0.glsl behind an include guard, included from both. Not
   duplicated: two copies would have to stay in step with the CPU codec in
   ggml-quants.c, and a divergence there appears as wrong matmul results, never as a
   build error.

3. ggml-vulkan.cpp: 'matmul_ptq1_0_f16_cm2_len' undeclared. Registrations for cm2
   pipelines whose shaders no longer exist once (1) is fixed. Both removed, with a
   comment recording that the absence is deliberate. (1) and (3) had to be fixed
   together or one error just trades for the other.

Verified locally this time, which is the other half of the lesson: glslc, molten-vk,
vulkan-headers and vulkan-loader were already installed on the machine that wrote the
original commit. It was pushed unverified on the false assumption that no toolchain
was present, and a local build finds all three in minutes.

  -DGGML_VULKAN=ON            builds clean
  SPIR-V generated            1991 shaders, 54 of them ptq1_0, incl. coopmat1, no cm2
  test-backend-ops MUL_MAT    1009/1009, Backend Vulkan0: OK, no regression

That local run does NOT verify the kernel. MoltenVK reports every ptq1_0 case "not
supported", and it reports pq2_0 the same way (q1_0 gets 28 of 156), so this is an
Apple/MoltenVK coverage limit rather than a defect in the new type. Kernel correctness
comes from the Arc B390 run on the patched tree: 1009/1009 with 78 ptq1_0 cases
actually executed and 0 failures, on KHR_coopmat.

Still absent by design: MMVQ (no q8_1 pipelines, absent from should_use_mmvq),
coopmat2, and on-device quantization, which needs a base-3 encoder in GLSL.

* vulkan: make PTQ1_0 reachable, and guard the coopmat2 path

The PTQ1_0 pipelines were created but the type was absent from every
selection and support enumeration, so supports_op returned false, the
scheduler never offered MUL_MAT/MUL_MAT_ID/GET_ROWS to Vulkan, and the
generated shaders were unreachable. The build looked green while nothing
executed on the backend. Add the type alongside GGML_TYPE_Q1_0 in the
five pipeline getters and the two supports_op gates.

Adding it there alone is not safe, because PTQ1_0 is the only type in
those lists with no coopmat2 pipelines: dequant_funcs_cm2.glsl carries no
decoder for it, so vulkan-shaders-gen skips cm2 generation. The cm2 slots
are still allocated by vk_matmul_pipeline2's constructor, so the mm getter
would return a non-null pipeline whose l/m/s are all null, and the
caller's "mmp == nullptr" fallback would not fire; the mul_mat_id getter
would instead trip GGML_ASSERT(support_fp32acc). Both getters now return
nullptr when the cm2 slots are empty, which routes the caller to the
dequant + f16 matmul fallback it already implements. That path needs
ggml_vk_get_to_fp16, which is one of the gates opened above.

The cpy, SET_ROWS and DUP gates are deliberately left alone:
copy_to_quant.comp has no PTQ1_0 block, so quantizing to this type on
device is unimplemented and a refusal there is correct.

Verified on Intel Arc B390 (KHR_coopmat): ptq1_0 goes from 0 OK / 78
declined to 28 OK / 50 declined / 0 failures, matching q1_0 and q2_0 case
for case, byte-identical across two runs. The coopmat2 guard is reasoned
from the pipeline-creation structure and is not covered by that run,
which is a coopmat1 device; it has no coopmat2 hardware behind it yet.

* cuda: accelerate PTQ1_0 MMVQ with packed DP4A

Assisted-by: OpenAI Codex
@khosravipasha
khosravipasha merged commit 01fd952 into prism Sep 4, 2026
8 of 11 checks passed
@khosravipasha
khosravipasha deleted the prism-v7-rebased branch September 23, 2026 23:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants