Merge upstream master (d854d98ba) into prism by bri-prism · Pull Request #352 · PrismML-Eng/llama.cpp

bri-prism · 2026-10-11T23:51:15Z

What

Brings prism (through cuda: run head size 72 flash attention on the MMA kernel by zero-padding to 80 (9B vision encoder 2.2x at 2048²) #390 and cuda: fuse the o_proj/ffn_down Hadamard with q8_1 quantization for PQ2_0 by default #402) up to upstream master at d854d98. This replaces the 2026-10-11 merge that this
PR carried before; its commits are still in the history.
Three follow-up fixes the merge needs:
- graph: keep the direct recurrent-state gather when there is one state, so the CUDA gather fusions still match.
- cuda: handle the new SWIGLU_CLAMP GLU op in the PQ2_0 multi-column fused mat-vec.
- cuda: handle SWIGLU_CLAMP in the PTQ1_0 one-column warp-per-row epilogue. Without it that path computed gate*up,
  so decode disagreed with prefill for models that use the clamped SwiGLU.

Why

The fork was about 1,300 upstream commits behind. Upstream CUDA work since then (fused MoE weighted reduction,
float4 RMS norm, faster low-bit MMVQ, top-k MoE fusions) is the base for the next fork PRs.

How

Prism quantization type IDs and kernels are kept, and the model, batch and graph-cache paths are adapted to the
upstream APIs.
Existing packed models decode exactly as before by default: the scalar CUDA RMS reduction stays the default
(GGML_CUDA_RMS_NORM_VECTOR=1 opts into the float4 path), and the Qwen3.5 GDN normalization and the short-context
GQA decode dispatch keep their current arithmetic. Switching the RMS default is a separate PR.

Testing

test-backend-ops (CUDA vs CPU) all pass: MUL_MAT 2925, MUL_MAT_ID 1097, MUL_MAT_VEC_FUSION 1618, RMS_NORM, NORM, ADD,
MUL, GET_ROWS, SET_ROWS, SOFT_MAX, ROPE, CPY, SSM_CONV, GATED_DELTA_NET.
Greedy output is byte-identical to current prism on the packed models (27B and 9B PQ2_0, 4B PTQ1_0, 2B Q2_0).
Architectures whose upstream graph changed produce different output, as expected.
H200, current prism vs this branch (alternating runs):

model pp512 tg128

27B PQ2_0 3260 -> 3411 172.1 -> 175.0

9B PQ2_0 9233 -> 9630 414.2 -> 411.1

4B PTQ1_0 11279 -> 11925 507.2 -> 513.8

2B Q2_0 19128 -> 20171 740.2 -> 734.5

The 9B decode drop (-0.7%) is recovered by two PQ2_0 decode tuning commits that follow in a separate PR (9B reaches
about 430 tok/s and 27B about 181 tok/s with them). The 2B difference is within run-to-run noise on current prism
(its two runs were 743.8 and 736.7).

+						fix(fixer) {
+							// Add a second newline so the two siblings are separated by a
+							// blank line, keeping the trailing indentation.
+							return fixer.replaceText(textNode, textNode.value.replace(/\n/, '\n\n'));


+            const int leaf_backend_id = tensor_backend_id(leaf);
+            GGML_ASSERT(leaf_backend_id != -1); // all leafs should be assigned by now
+
+            if (tensor_id_copy(leaf_id, leaf_backend_id, 0) == NULL) {


+                    }
+                    ggml_set_input(tensor_copy);
+                    ggml_set_output(tensor_copy); // prevent ggml-alloc from overwriting the tensor
+                    tensor_id_copy(leaf_id, leaf_backend_id, c) = tensor_copy;


+    const int split_backend_id = split->backend_id;
+    ggml_backend_t split_backend = sched->backends[split_backend_id];
+    ggml_backend_t input_backend = ggml_backend_sched_get_tensor_backend(sched, input);
+    struct ggml_tensor * input_cpy = tensor_copy(input, split_backend_id, sched->cur_copy);


+                                        wdata_src_b + i1n, 0,
+                                        wdata_kernel + i01*ne00*ne03 + i00*ne03, 0, 1);
+                            }
+                            dst_data[(i11*stride + i01)*ne0 + i10*stride + i00] += v;


khosravipasha

LGTM. This is not really possible to review so need to see what subset of functionatlies/hardware we want to test before mering.

…rg#30421) Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* llama : respect no_alloc in memory caches Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB) * llama : TODO: move no_alloc out of llama_hparams Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB) * llama : TODO: move no_alloc out of llama_hparams Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

) (ggml-org#29755)

…based expert detection (ggml-org#30820) * llama-quant : fix quantizing 4D tensors nrows_total was ne[1]*ne[2], so for a 4D tensor only the first ne[2] slices were quantized and the rest of the rows were dropped. Use ggml_nrows(tensor) to count all rows, including ne[3]. Fixes ggml-org#29916 * llama-quant : detect expert tensors by name and fix imatrix size for 4D tensors The MXFP4_MOE path used ne[2] > 1 to detect MoE expert tensors. This is wrong for non-expert tensors with 3 or more dims, e.g. MLA projections (BailingMoE3) and 4D conv kernels. Use the tensor name instead: all tensors used by GGML_OP_MUL_MAT_ID are named *_exps or *_chexps. This also removes the BailingMoE3 special case. The imatrix holds one vector of ne[0] values per 2D slice, so the expected size is ne[0]*ne[2]*ne[3], not ne[0]*ne[2]. This matches how the imatrix is indexed per row for 4D tensors in llama_tensor_quantize_impl. Assisted-by: Claude Opus 5.5

…l-org#31035)

…0977)

* qwen: merge tensors on convert * convert: use generator functions * qwen4exp: remove conts and pre-scale

…ml-org#31153) * common, grammar: add recursive depth limit, make it a requirement * catch more recursive functions * common : tidy up peg parser changes * grammar : reject candidates in a loop, without a depth limit The depth limit in llama_grammar_reject_candidates_for_stack counted the code points of a vocab token, so any token of 256 chars or more was always rejected under a grammar, such as the 512-space token of GLM-4.7-Flash or the long space tokens of Laguna-S-2.1. The rejection now walks the code points with its own stack of levels instead of recursing: there is no limit and no stack risk, whatever the length of a token from the GGUF, and the masks match master. * grammar : track rejected candidates as indices into the input At a given code point a candidate only differs from its input by the position of its code point pointer, so the levels now carry indices into the input candidates instead of copies. The first level needs no special case and the rejects map back to the input directly. Grammar sampling gets faster than master on GLM-4.7, Laguna, Qwen3.6 and gemma-4 with the json, list and spaces grammars, same masks. * nit : shorten the comment of llama_grammar_reject_step * common : add depth_guard for schema walks * grammar : move GRAMMAR_RECURSIVE_LIMIT into the .cpp * common : reduce recursion limit for peg parser * improve --------- Co-authored-by: Alde Rojas <hello@alde.dev> Co-authored-by: Pascal <admin@serveurperso.com>

* cuda : use std::string instead of VLA for archName Modernize the code (old C style) and fix the following warnings: ``` ggml/src/ggml-cuda/ggml-cuda.cu:180:19: warning: variable length arrays in C++ are a Clang extension [-Wvla-cxx-extension] 180 | char archName[archLen + 1]; | ^~~~~~~~~~~ ggml/src/ggml-cuda/ggml-cuda.cu:180:19: note: read of non-const variable 'archLen' is not allowed in a constant expression ggml/src/ggml-cuda/ggml-cuda.cu:179:9: note: declared here 179 | int archLen = strlen(devName); | ^ ``` Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Add a comment about archName Signed-off-by: Adrien Gallouët <adrien@gallouet.fr> --------- Signed-off-by: Adrien Gallouët <angt@huggingface.co> Signed-off-by: Adrien Gallouët <adrien@gallouet.fr>

Brings prism (through #390, #402) and the 2026-10-11 upstream merge candidate together.

Keep the Prism quantization type IDs and kernels, and adapt the model, batch and graph-cache paths to the upstream APIs. Keep the scalar CUDA RMS reduction by default (GGML_CUDA_RMS_NORM_VECTOR=1 opts into the float4 path), the baseline Qwen3.5 GDN normalization and the short-context GQA decode dispatch, so existing packed models decode exactly as before. Mark observed Hadamard fixture intermediates as outputs.

With one sequence and one state there is no relocation tail, so return the gathered rows directly. The CUDA gather fusions match on that node.

The upstream merge added the clamped SwiGLU GLU op. Pass glu_limit through and apply it in the epilogue instead of falling back to gate*up.

The one-column PTQ1_0 path had no case for the clamped SwiGLU and computed gate*up, so decode disagreed with prefill for models that use it.

+        for (int input_id = 0; input_id < split->n_inputs; input_id++) {
+            if (!ggml_backend_sched_is_host_weight(split->inputs[input_id])) {
+                ggml_tensor * input = split->inputs[input_id];
+                ggml_tensor * input_cpy = tensor_copy(input, split_backend_id, sched->cur_copy);


The merge switched the condition to crop_before_nextn() and left the local unused, which fails builds with LLAMA_FATAL_WARNINGS.

Fixes -Werror=reorder in the llm_graph_context constructor after the merge.

Fixes -Werror=unused-variable in the MTP draft paths.

…rk drafter in test-llama-archs common_chat_msg gained extra_json in the merge, so the aggregate initializer hit -Wmissing-field-initializers. DSpark is a drafter that needs a target, like EAGLE3 and DFLASH, which the generated-model test already skips.

The fork drops empty type/oneOf/anyOf unions instead of rejecting them (#333), and the merge kept that behavior. Replace upstream's two rejection asserts with a test of the fork behavior.

Resolve the prism.hadamard loader conflict by keeping load_hparams_hadamard() and adding Gemma 4 to its arch whitelist (#413). Add the SWIGLU_CLAMP case to the PTQ1_0 warp-per-row epilogue from #415, since this branch has that op.

…d reduction After the merge, upstream's moe_weighted_reduction matched first and took every pattern, so the fork's exact kernel (#107) never ran and MOE_WEIGHTED_REDUCE, which requires a bit-exact result, failed. Try the exact kernel first; shapes it does not take still use upstream's reduction.

bri-prism requested a review from khosravipasha October 11, 2026 23:51

github-actions Bot added documentation Improvements or additions to documentation ggml Vulkan testing Apple Metal SYCL IBM zDNN build devops android server model nix jinja parser Ascend NPU OpenCL Hexagon WebGPU OpenVINO conversion CUDA server/ui mtmd vendor labels Oct 11, 2026

github-advanced-security Bot found potential problems Oct 11, 2026

View reviewed changes

Comment thread tools/ui/eslint.config.js

fix(fixer) {

// Add a second newline so the two siblings are separated by a

// blank line, keeping the trailing indentation.

return fixer.replaceText(textNode, textNode.value.replace(/\n/, '\n\n'));

github-advanced-security Bot found potential problems Oct 12, 2026

View reviewed changes

bri-prism mentioned this pull request Oct 12, 2026

cuda: keep the quantized K/V type in the short-query MMA FA fallback #353

Closed

khosravipasha approved these changes Oct 12, 2026

View reviewed changes

opencl: fix OOB tail access for adreno noshuffle gemv kernels (ggml-o…

9166cea

…rg#30421) Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

ggerganov and others added 15 commits October 25, 2026 11:56

grammar: limit recursion depth to prevent stack overflow (ggml-org#29495

f1168ac

) (ggml-org#29755)

llama: fix exhausting CPU and MEM caused by dry sequence breaker (ggm…

63412d8

…l-org#31035)

ci : disable test-llama-context with fa off in the webgpu (ggml-org#3…

1a0b56c

…0977)

qwen: merge tensors on convert (ggml-org#31028)

2280f0f

* qwen: merge tensors on convert * convert: use generator functions * qwen4exp: remove conts and pre-scale

tests : adjust default threads for test-llama-context (ggml-org#31157)

6050b46

Merge the upstream-merge candidate into current prism

e880914

Brings prism (through #390, #402) and the 2026-10-11 upstream merge candidate together.

graph: keep the direct recurrent-state gather when there is one state

d0f258c

With one sequence and one state there is no relocation tail, so return the gathered rows directly. The CUDA gather fusions match on that node.

cuda: handle SWIGLU_CLAMP in the PQ2_0 multi-column fused mat-vec

0085e51

The upstream merge added the clamped SwiGLU GLU op. Pass glu_limit through and apply it in the epilogue instead of falling back to gate*up.

cuda: handle SWIGLU_CLAMP in the PTQ1_0 warp-per-row mat-vec epilogue

c2734e3

The one-column PTQ1_0 path had no case for the clamped SwiGLU and computed gate*up, so decode disagreed with prefill for models that use it.

Merge the DSpark script fixes from the previous upstream-merge branch

0c7907e

bri-prism changed the title ~~Merge upstream master into prism (2026-10-11)~~ Merge upstream master (d854d98ba) into prism Oct 29, 2026

github-actions Bot added the AMD ZenDNN label Oct 29, 2026

This was referenced Oct 29, 2026

cuda: 5120-wide fused norm + Hadamard with PDL, and two-row PQ2_0 decode tiles on Hopper #417

Merged

cuda: use the float4 RMS norm path by default, with a pinned rounding order #418

Merged

github-advanced-security Bot found potential problems Oct 29, 2026

View reviewed changes

bri-prism added 7 commits October 29, 2026 00:13

qwen35moe: use narrow_last for the last-layer row crop

61cb746

The merge switched the condition to crop_before_nextn() and left the local unused, which fails builds with LLAMA_FATAL_WARNINGS.

graph: initialize mtp_chain_embd in declaration order

c897791

Fixes -Werror=reorder in the llm_graph_context constructor after the merge.

speculative: drop two row_bytes locals left unused by the merge

b4c2b0b

Fixes -Werror=unused-variable in the MTP draft paths.

tests: expect empty unions to parse as any schema

c4595c5

The fork drops empty type/oneOf/anyOf unions instead of rejecting them (#333), and the merge kept that behavior. Replace upstream's two rejection asserts with a test of the fork behavior.

bri-prism merged commit f802296 into prism Oct 30, 2026
19 of 43 checks passed

bri-prism mentioned this pull request Oct 30, 2026

build: fix RPC, Vulkan and SYCL builds after the upstream merge #421

Merged

noonghunna mentioned this pull request Oct 31, 2026

Prism b12128: pin it noonghunna/club-3090#1873

Merged

Navigation Menu

Repository navigation

Merge upstream master (d854d98ba) into prism - #352

Merge upstream master (d854d98ba) into prism#352
bri-prism merged 1290 commits into
prismfrom
merge/upstream-20261011

bri-prism commented Oct 11, 2026 •

edited

Loading

Uh oh!

khosravipasha left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

model	pp512	tg128
27B PQ2_0	3260 -> 3411	172.1 -> 175.0
9B PQ2_0	9233 -> 9630	414.2 -> 411.1
4B PTQ1_0	11279 -> 11925	507.2 -> 513.8
2B Q2_0	19128 -> 20171	740.2 -> 734.5

Conversation

bri-prism commented Oct 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

khosravipasha left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

bri-prism commented Oct 11, 2026 •

edited

Loading