Repository navigation
Conversation
bri-prism
left a comment
bri-prism
left a comment
There was a problem hiding this comment.
Agent review: posted by the maintainer's coding agent at their request. Measurements are from a second agent session on an Arc B390 (Xe3, Windows, oneAPI 2025.3.2).
Tested as authored (b61a07146, which is #327 plus f0a22003d).
-
test-backend-ops: passes.
-
Greedy output: byte-identical to #327 on every prompt, so the fusions add no drift.
-
vs #327:
model pp512 tg128 2B +5% +11% 9B +6% +2% 27B +14% +3%
As with #327, this needs a rebase onto #356, and it has to go together with or after #327. Most of the fusion set already exists on tip. This comes from reading tip's source and env knobs, not from a per-op trace:
| fusion in this PR | already on tip |
|---|---|
| Hadamard sign + FWHT + int8 quantization in one kernel | GGML_SYCL_PTQ1_T2_FWHTQ |
| SWIGLU folded into the activation quantization | ptq1-t2.cpp, GGML_SYCL_PTQ1_GLU1 |
| shared quantized activations | GGML_SYCL_PTQ1_T2_QCACHE |
| shared-src1 multi-matmul | GGML_SYCL_T2_MULTI |
| silu·gate / +residual / +bias / sigmoid epilogues on the DPAS accumulator | ternsycl/epilogue_dev.hpp |
With T2=all, tip already reaches 27B pp 490 vs 480 here. The parts that look new are:
- the deferred gate;
- the PQ2_0 (XMX) side of these fusions: 9B pp 1392 here vs tip's best of 1145.
Could you rebase onto #356, drop what T2 already covers, and say which fusions remain? Please also keep Xe3 decode on the reorder path (27B tg 16.4 on tip's default vs 15.2 here).
Replace the ESIMD kernels of the PQ2_0/PTQ1_0 XMX path with SIMT SYCL kernels ported from Intel's TernSYCL (int2_via_int2_x_int8_dpas, BSD-3-Clause): inline-vISA s2 x s8 DPAS and 2D block I/O, a mat-vec kernel with a K split reduced in SLM for up to 8 tokens, and a GEMM with 256 GRF whose tile depends on the batch size and which splits K over work-groups when a small batch would leave most of the GPU idle. - PQ2_0 layout: uint32 [K/16][N] s2 codes (one 2D block load gives a sub-group 16 columns x 128 weights in the DPAS operand layout), then the fp16 scales [K/128][N]. - PTQ1_0 stays at 1.75 bits: [K/128][7][N] dwords (qs, then qh and the scale). The kernels decode the trits through a 256-entry table in SLM, in byte-major K order so the 10-bit table entries concatenate into the s2 dwords; the activation is quantized in the same permuted order. PTQ1_0 weights no longer expand to 34 bytes a block. For large batches the GEMM decodes each weight block once per work-group and shares it through SLM. - Rows are padded to 16 in the allocation, so any row count works (e.g. 151669-token vocabs). The gate, the layout flag, the capability check and the fallbacks of ggml-org#294 are unchanged.
…othing is shared PTQ1_0 decodes its trits in the kernels, so with few tokens the decode, not the weight reads, set the speed, and the 27B decoded 8 and 16 parallel sequences 3-5% slower than with the expanded weights of ggml-org#294. - The GEMM decodes in registers when a work-group has one sub-group row (WG_M == 1), without the SLM round trip and barrier that only pay off when sub-groups share the decoded block. - PTQ1_0 picks its own tiles: 16x16 and 32x16 per sub-group for 5-32 tokens (each decoded block feeds 2-4 row blocks), 32x32 with one sub-group row for 33-64 tokens with K >= 6144, and the 4-row mat-vec for 2 tokens (the 2-row variant is slower). Arc Pro B50, Bonsai 2 27B PTQ1_0, llama-batched-bench decode t/s at 8 / 16 / 32 sequences: 77.4 / 93.1 / 110.7 -> 82.2 / 105.7 / 119.7 (prism: 80.1 / 98.1 / 110.8).
On top of the XMX kernels, the graph loop now fuses around PQ2_0/PTQ1_0 mat-muls on the XMX layout, beside what the TernSYCL T2 path fuses for its own weights: - Epilogues (TernSYCL postop 1 and 2) in the mat-vec and GEMM stores: gate + up + SWIGLU runs as two mat-muls from one quantized activation, the gate writing silu(gate) * up; mat-mul + residual ADD (also through reshapes) writes the sum. - Shared activations: a quantized activation is kept for the length of a graph compute and reused by every XMX mat-mul that reads it (q/k/v, qkv and z, gate and up). Entries live in a separate pool, as the VMM pool only frees in reverse order. - Hadamard folding (Bonsai 2): when every user of a rotated input is an XMX mat-mul, the sign flip, the 1024-wide FWHT and the int8 quantization run in one kernel and the FWHT output is never written. Other rotations are left to ggml_sycl_op_fwht_signed. - A mat-mul whose only use is the gate of a later SWIGLU (z in gated delta net layers) runs at the GLU, with the SWIGLU in its store. GGML_SYCL_ENABLE_FUSION=0 turns this off.
b61a071 to
3fbc0e6
Compare
|
Rebased onto the current tip ( What T2 already covers is left to T2. These fusions only run on weights in the XMX layout. A weight T2 takes keeps T2's own fusions (
What remains is all on the XMX layout, which means PQ2_0 and any PTQ1_0 that T2 isn't asked for:
Xe3 decode stays on tip's reorder path. Arc Pro B50 (Xe2, Linux), llama-bench pp512 / tg128, 3 repetitions. test-backend-ops: MUL_MAT 1484/1484, plus ARCLAB_T2_MULTI 24/24, GLU_ROT 15/15 and ADD_NORM_ROT 15/15, with and without
The 27B PTQ1_0 T2=all figure in this table is from before the look-ahead fix; after it, the result is the 782.6 above. On PQ2_0, #328 is ahead of tip T2=all on decode for every size; on prefill it is 3% behind on 1.7B, even on 8B and 1.5% ahead on 27B. |
Overview
Depends on #327 (its two commits are in this branch: the first and the third, a cherry-pick of #327's follow-up; review #327 first). This PR adds the second commit: the graph loop now fuses the PQ2_0/PTQ1_0 XMX mat-muls with the ops around them, so far fewer small kernels run per token.
{mul_mat(gate), mul_mat(up), SWIGLU}runs as two mat-muls from one quantized activation, the gate writingsilu(gate) * up;mul_mat -> (reshape) -> ADD(the residual adds afterffn_down,ssm_outand the attention output) writes the sum.xmx_act_pool()), because the VMM pool only frees in reverse order.x * signs -> FWHT_1024 (hinted mul_mat) -> mul_mat. The sign flip, the FWHT and the int8 quantization now run in one kernel (the butterflies followggml_sycl_op_fwht). When every user of the FWHT output is an XMX mat-mul (checked against the graph's use counts), that output is never written; otherwise one kernel does sign flip + FWHT for the other users.zof the gated delta net layers, three nodes before its GLU) keeps its quantized activation and runs at the GLU, with the SWIGLU in its store.GGML_SYCL_ENABLE_FUSION=0turns all of this off. Other models and types take exactly the paths of #327.The fusion rules are independent of the weight type; only
ggml_sycl_pq2_xmx_eligible()(the side-effect-free half of #294's gate, split out here) decides which mat-muls take part.Results
Same setup as #327: Intel Arc Pro B50 (16 GB), oneAPI 2026.1, Level Zero 1.14.37020, Linux 7.0
xedriver,-ngl 99. "#327" is the first commit alone; all columns measured in the same session.llama-bench, t/s:-fa 1)-fa 1)Parallel decode (
llama-batched-bench -npp 128 -ntg 64), aggregate decode t/s, #327 -> this PR:Speculative decoding, Bonsai 2 27B PTQ1_0 with the grafted MTP head (sudoingx/Ternary-Bonsai-2-27B-PTQ1_0-MTP-GGUF),
llama-server, 3 chat prompts (prose, code, bash) x 256 tokens, greedy, thinking off, decode t/s:--spec-type draft-mtp --spec-draft-n-max 1--spec-type draft-mtp --spec-draft-n-max 2With this stack the B50 decodes Bonsai 2 27B at ~39 t/s (code and bash ~42, prose ~33) on one 16 GB, 70 W card, short context.
Correctness
test-backend-ops -b SYCL0, run per op over all 130 ops: MUL_MAT 1401/1401, MUL_MAT_ID 1035/1035, MUL_MAT_VEC_FUSION 506/506 (gate/up + GLU and mat-mul + bias ADD on PQ2_0 and PTQ1_0, 1-8 tokens), ADD 99/99, GET_ROWS 119/119. The ops that do not pass fully (CONV_2D, FLASH_ATTN_EXT, LIGHTNING_INDEXER, ROLL; CPY and SET_ROWS abort) do the same on prism.GGML_SYCL_DEBUG=1, that every Hadamard site of the 27B decode graph takes the fused path (257 sites per decode graph).-c 512, 5 chunks of the repository'sdocs/*.md): identical to sycl: PQ2_0/PTQ1_0 XMX kernels from TernSYCL, PTQ1_0 kept at 1.75 bpw #327 for 1.7B b1/b512, 4B b1/b512, 1.7B-ngl 0, 4B-ngl 20; 27B PQ2_0/PTQ1_0 b512 6.2356 (sycl: PQ2_0/PTQ1_0 XMX kernels from TernSYCL, PTQ1_0 kept at 1.75 bpw #327) -> 6.2367 (this PR), prism 6.2370; 27B PTQ1_0 with-ub 166.2389.Additional information
Requirements