Repository navigation
Sync upstream b11887 - #327
Conversation
The clamp kernel indexed source and destination as one flat contiguous array, so a strided view as input (the new CLAMP v=1 case) read the wrong elements. Add a strided kernel_clamp_nc, as the other unary ops have, and take it whenever the source is not contiguous.
* spec : add lilicorr support * convert : use convention for hparam assignment * add todo for hparam use in llama-batch.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * typo fix --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…g#30498) * server: save only restorable slot states to the prompt cache The idle-slot save loop stores every idle slot whenever a task starts, while only completion tasks restore from the prompt cache. Rerank, embedding, decision and infill states fill the cache up to its limit without ever being read back, and slots already cached are offered again at every task start. Each slot now tracks with cache_pending whether its state is restorable and missing from the cache: a completion release or an explicit slot restore sets it, prompt_save consumes it, prompt_clear resets it. Both save paths share this rule, and the LRU path restores only for completion tasks. * server: test that decision slots skip the prompt cache Decision tasks never mark their slot as cacheable, so starting new tasks must not save their idle slots to the RAM prompt cache, with or without a unified KV cache. * server: restore any reusing task from the prompt cache Embedding and rerank tasks never reuse their state, so their slots are cleared on release like child slots and the cache only holds states that a later task can restore. Infill and decision tasks now restore from the cache too, which replaces the cache_pending flag. Decision models disable the prompt cache unless --cache-ram is given, since their states seldom repeat.
An embedding-only graph has no logits, and build_pooling() returns early when embeddings are off, so since the models no longer expand t_embd the graph ends up empty and its inputs are never allocated. Expand t_embd in that case.
* llama: honor PARTIAL_ONLY in the plain KV cache A non-SWA llama_kv_cache rolls back with seq_rm, so it holds no partial state, yet state_write ignored the flag and serialized every cell. A draft context made of a plain KV cache (MTP head, dense -md draft) therefore copied its whole KV into each server checkpoint, growing it by the draft KV size per token. The cache now writes and reads nothing for PARTIAL_ONLY unless it is SWA, and the iSWA wrapper no longer gates its base cache since it filters itself. * Update src/llama-kv-cache.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * llama: delegate PARTIAL_ONLY to the caches of every memory wrapper The hybrid, hybrid-idx and dsa-iswa wrappers skipped their attention caches on PARTIAL_ONLY, which also dropped an SWA window that is partial state. Each cache now applies the flag itself, so the wrappers only delegate. The indexer is marked stale only when a restore rewrites cells. test-llama-context gains a case checking that the partial state is not larger than the full state and, without SWA, does not grow with the tokens. --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…ml-org#30491) * metal : add BF16 variant of the tensor API flash attention kernel * cont : move tensor FA kernels to kernels/fa_tensor.metal
…-org#29200) * cpu: add F16 SWIGLU_OAI reference implementation and F16 activation op test coverage Adds ggml_compute_forward_swiglu_oai_f16() to the CPU backend, mirroring the existing F32 path, and extends test-backend-ops with F16 test cases for SWIGLU_OAI so downstream backends(e.g. Hexagon) can be CPU-diffed for correctness. * ggml-openvino: fix f16 SWIGLU_OAI type mismatch in alpha/one constants translate_glu_swiglu_oai creates alpha_const and one as f32 constants unconditionally, so f16 inputs hit a type mismatch in Swish/Add on GPU ("Swish inputs must have the same type but they are: f16 and f32"). translate_glu_swiglu_clamp already upcasts to f32 and converts back at the end; apply the same pattern here. --------- Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
…y channels (ggml-org#27703) * vulkan : copy strided f16 KV once so reads spread across memory channels Assisted-by: Claude (Opus 4.8 / Opus 5) * tests : cover the f16 KV copy to scratch Assisted-by: Claude (Opus 4.8 / Opus 5) * vulkan : gate the f16 KV scratch copy to AMD Assisted-by: Claude (Opus 4.8 / Opus 5) * vulkan : reuse the generic copy shader for the f16 KV scratch fill The f16 clause needs only a strided copy, not a dequant, so drop the dedicated shader and dispatch the existing copy pipeline instead. The copy is described in source memory order so the strided access lands on the writes, which stream, instead of the reads, which would concentrate on a few memory channels. Assisted-by: Claude (Opus 4.8 / Opus 5 / Fable 5) * vulkan : extend the strided KV scratch copy to bf16 Assisted-by: Claude Opus 5 * vulkan : clean up comments Assisted-by: Claude Opus 5 * consolidate copy shaders * disable on RDNA4 --------- Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
…g#30532) * llama : split llama_prefetch() into merge and prefetch steps Merge the ranges first, then call the system functions. This fixes -Wunreachable-code-return on Windows. Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Use const_cast Signed-off-by: Adrien Gallouët <angt@huggingface.co> --------- Signed-off-by: Adrien Gallouët <angt@huggingface.co>
…#30485) * unicode : reject overlong UTF-8 and out-of-range codepoints * Update tests/test-unicode.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * tests: document byte-wise UTF-8 error recovery * break out of first test on failure --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Assisted-by: Codex
…rg#30421) Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* replace oneAPI package by DLE+oneDNN package * support to install oneDNN in windows, in release and docker * for internal test * remove test code * Update intel.Dockerfile
* vulkan : fix concurrent first device init * tests : load models in parallel in test-thread-safety Loading them serially never exercised concurrent first backend init, so the crash reported in ggml-org#29764 was invisible to this test. * vulkan : create fence before retaining device in buffer context
tojson wrote nan, inf and -inf for these floats, as values and as dict keys; json.dumps writes NaN, Infinity and -Infinity. The string and safe filters of a float keep nan and inf, as Python's str() does, so they no longer use the tojson function. float_to_json is renamed float_repr, as it now gives str(); float_json gives the json.dumps text. Also say in a comment why a last digit 9 is not bumped at a power of 2.
Assisted-by: Codex
* Add Support to maion-coder arch * iAdd Support to support Draft models * Address comments * Remove maion context shift function --------- Co-authored-by: Karthik Vijayan <kavijay@microsoft.com>
…gml-org#30582) The registry hard-coded 10 for AGENTIC_MAX_TURNS while DEFAULT_AGENTIC_CONFIG said 100. The registry seeds SETTING_CONFIG_DEFAULT, so the loop's `|| DEFAULT_AGENTIC_CONFIG.maxTurns` fallback never fires and users got 10 turns. Reference the shared constant, as the adjacent MCP timeout does. Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
* bump ty to 0.0.88 * arange -> linspace.tolist() to pass expected type
…ggml-org#30567) Slot selection takes free slots by id or by last use only, so child tasks often land on a slot holding a parent state while empty slots are idle, and overwrite it without saving it to the prompt cache. Free slots are now ordered empty first, then least recently used, for both the LRU fallback and the child slots, and a child task saves the state it overwrites. With 4 grouped decision requests in flight and 12 states in rotation, prompt cache reads go from 5 to 36. [no release]
The slot file held the target context and the context checkpoints, but never the draft context, so a restored slot drafted with an empty or stale draft KV until the cache refilled. The file now ends with the live draft state, the full draft context and the speculative state, in the same appendix style as the checkpoints. A draft state that does not fit the draft context is dropped and the draft starts empty, while older files and older servers keep working. [no release]
…rg#30578) * model : mellum - read MoE gating params from GGUF, unify RoPE - read/write expert_weights_norm and expert_gating_func instead of hardcoded values (defaults unchanged) - use per-layer rope params via get_rope_freq_base/scale - check n_expert once, outside the layer loop * Update conversion/mellum.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> [no release]
The first evaluated token attends to the positions above pos - n_swa, so a SWA state starting at pos - n_swa + 1 holds the whole window, but the reuse threshold asked for two more cells. A saved SWA state keeps only the cells inside its window, so a state read back from a slot file or from the RAM prompt cache was always rejected and the slot rolled back to a checkpoint. The SWA threshold now matches the window, while a state without SWA keeps the exact bound. The test harness gains override_kv, used to give the SWA test model a window smaller than the test context. [no release]
Small helper around the GitHub Actions API: lists the workflow runs that are not finished yet (in_progress, queued, pending, waiting, requested), grouped by status. Supports selecting runs by status, by branch (server-side, to reduce API traffic) or by a regex on the branch name, and can cancel the selected runs via --kill. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
…gml-org#30520) * add support * remove unneeded bos fix * add video flag * fix flake error * remove video-specific budgeting * convert : fix lilicorr-dflash bid clash * spec : enforce lilicorr n_tokens condition * spec : fix lilicorr walk [no release]
…gml-org#31100) The two T4 runners in the ggml-ci group (ggml-6-x86-vulkan-t4, ggml-8-x86-nvidia-a10) share the same labels and can both pick up the same jobs, so extend the per-runner results dir from ggml-org#31046 to them. Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
ggml-org#30978) Use the dropdown menu surface (border, radius, background, shadow, padding) for the hover card and the context gauge popup, and fix the hover card open/close animation to use bits-ui data-state attributes. Assisted-by: Claude (Opus 5.5 Medium)
* ui : move models manager to its own route, keep Discover as a dialog Models now lives at #/models instead of a dialog. Discover opens as a standalone dialog over it, and leaving the page returns to the chat it was opened from. Assisted by: Claude (Opus 5.5 Medium) * ui : polish the models page layout and filters Give the models page a page header with the Discover action, calmer column labels, tinted section and family headers, and an empty state. Load the model list on a direct visit, slide the detail pane over the table below xl, and draw the filter toggles as one box matching the context select, with room for their shadow. Assisted by: Claude (Opus 5.5 Medium) * ui : round the models table family headers Assisted by: Claude (Opus 5.5 Medium) * ui : fix models page navigation and Discover gating Record the models page's way back after every navigation, so Back, a typed URL or a reload no longer return to a stale chat. Escape closes the detail pane, then leaves the page. Gate every Discover entry point on one shared check, recheck the API key on /models, skip the duplicate router fetch on a direct load, and share the route load between pages. Assisted by: Claude (Opus 5.5 Medium) * ui : show model details in a drawer Open the models manager's detail pane in a drawer, from the right on desktop and from the bottom on a phone, instead of an inline side panel and a full-screen cover. Pad the pane on both sides, drop its close button on a phone, and assert that filtered rows are gone from the page rather than merely hidden. Assisted by: Claude (Opus 5.5 Medium) * ui : make whole models table rows clickable A click anywhere on a model or quant row opens it, unless it lands on one of the row's own controls (load toggle, actions menu). The name button stays the keyboard path to the same action. Assisted by: Claude (Opus 5.5 Medium)
The src1 power-of-two rescale in kernel_mul_mm_id is superseded by the per expert-token amax scaling done in the compute graph (ref: ggml-org#30861), which keeps the MoE down-projection activations within f16 range before the op runs. Drop the two-stage reduction kernels, their scratch buffer and the GGML_PREC_F32 dispatch added in ggml-org#26223, as well as the test cases that drive MUL_MAT_ID src1 out of f16 range. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* server_role * cont
…org#31055) * hexagon: KV-head team split for HMX flash attention in multicore * hexagon: merge FA_HEAD_SPLIT and FA_SPLIT into single GGML_HEXAGON_FA_SPLIT flag 0 = off (token-split only); 1 = head-split, plus team-split when n_kv_heads < N_cores so stranded cores still participate (e.g. gemma-4 with 2 KV heads on 4 cores). Where heads divide evenly across cores, plain head-split is used.
* jinja : fix select and selectattr argument handling * jinja : address review feedback * jinja : Simplified test attr
* model : add DeepSeek-V4.1-Flash (deepseek41) * deepseek41 : remove MXFP8, convert the Engram table to BF16 * deepseek41 : compute the compressed-KV read layers once in the cache * deepseek41 : use get_cells() for the shared-history check, drop llama_kv_cache::seq_pos_tok_le * deepseek41 : gather the Engram rows in-graph like qwen4exp, drop the host gather * deepseek41 : keep the Engram hash constants in hparams.ple_*, one module per Engram layer * deepseek41 : drop the LLAMA_DSV41_KV_MIRROR toggle * deepseek41 : keep il last in build_hc_mixes * deepseek41 : sort the top-k rows before the sparse gather * tests : add deepseek41 to the Metal fusion baseline --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com>
…l-org#30242) * chat : reject JSON nested more than 128 levels before walking it The schema builder, the jinja engine and the decision code walk request JSON with recursion. A request nested deep enough overflows the stack of the HTTP thread and llama-server dies (SIGBUS on macOS, SIGSEGV on Linux): for a tool schema from a few hundred levels on macOS and about 10000 on Linux. Check the depth first and answer 400 past 128 levels: the chat inputs in common_chat_templates_apply (tools, tool-call arguments, chat_template_kwargs, json_schema), the json_schema of /completion, and the whole /v1/systemone body. A gemma4 tool result that is too deep stays text. * fix the root, not the call side --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* opencl: add `mul_mv_q2_k_f32` * opencl: add q2_k SOA mv and tiled mm * opencl: add q2_k noshuffle gemm/gemv for adreno * opencl: add q2_k dp4a gemm for adreno Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com> * opencl: tune q2_k kernels for adreno --------- Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: add topk * opencl: optimize topk * opencl: add fp lightning indexer * opencl: optimize lightning indexer * opencl: refactor * opencl: refactor tiled topk kernel * opencl: adjust format
…ps (ggml-org#31071) * hex-tools: support for --compare option in the profile.py * hex-unary: improve collapse logic to also do splits if nrows is small * hex-colapse: handle all colapsing at the host and make it generic * hex-colapse: remove leftover unary helper
…rg#30106) WM of the small tiles was 32, or 8 for 8-wide subgroups, whatever the tile's BLOCK_SIZE and WARP. With 16-wide subgroups (Mali) the second subgroup of a workgroup wrote into the next tile's columns. With 8-wide subgroups some _id tiles left rows uncomputed. Derive WM so there is one warp tile per subgroup. Tested on my Pixel 9 Android phone. Before the change there were 20 MUL_MAT and 6 MUL_MAT_ID failures. After the change they all pass. Fixes ggml-org#28637 Assisted-by: Claude Opus 5.5
…9920) * vulkan: Add a larger tile size for conv3d (coopmat2 only) This adds a larger tile size, in practice it is only for coopmat2 where we don't need shared memory to store the output (would use more than 64KB). Add some additional representative backend tests. At least, this is what codex came up with from looking at sd.cpp Wan VAE. * only apply large tile size for newer driver versions
…p rows (ggml-org#29948) * cuda: fuse FFN gate/up and GLU in MMQ with interleaved gate/up rows For prompt processing, run the gate and up matmuls of a dense FFN as one mul_mat_q launch and apply the GLU in the write-back. - The loaders take the gate weights as a second pointer and fill the weight tile with up and gate rows interleaved in chunks of 8, so each thread holds an up value and the matching gate value (rows r and r + 8 of an MMA tile). - The MMA loop is unchanged: same tile, warp layout, vec_dot and one sum[] per thread. The fused write-back computes the GLU from registers and writes half the rows. - The fused path uses the regular tiling (no stream-k) with the column tiles innermost so that neighbouring blocks share weight tiles. - Enabled for Q4_K, Q5_K, Q6_K, Q8_0 and IQ4_XS on NVIDIA Turing and newer. * cuda: assert the fused MMQ gate argument and trim redundant comments * cuda: move the fused MMQ hint check to should_fuse and add TODOs * cuda: pass prec_src1 to the fused MMQ write-back * cuda: fuse only with efficient tiling and no fallback, merge the GLU write-back * cuda: require has_fusion explicitly, drop the fusion-only src1 precision check * cuda: launch the fused MMQ kernel through the stream-k kernel with one block per tile * cuda: inline the MMQ tiling-efficiency check * tests: one fused gate/up MMQ case per type and GLU op * cuda: always launch the fused MMQ kernel with one block per tile * tests: shorten the fused gate/up MMQ test comment * cuda: select the fused MMQ loader and write-back in ggml_cuda_mmq_get_util_funcs * cuda: run the fused MMQ kernel on the plain tiling path * cuda: put the column tiles innermost in the MMQ plain tiling path
Co-authored-by: Shreya Agrawal <shreyaagrawal@Shreyas-MacBook-Air.local>
* llama : respect no_alloc in memory caches Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB) * llama : TODO: move no_alloc out of llama_hparams Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB) * llama : TODO: move no_alloc out of llama_hparams Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
…based expert detection (ggml-org#30820) * llama-quant : fix quantizing 4D tensors nrows_total was ne[1]*ne[2], so for a 4D tensor only the first ne[2] slices were quantized and the rest of the rows were dropped. Use ggml_nrows(tensor) to count all rows, including ne[3]. Fixes ggml-org#29916 * llama-quant : detect expert tensors by name and fix imatrix size for 4D tensors The MXFP4_MOE path used ne[2] > 1 to detect MoE expert tensors. This is wrong for non-expert tensors with 3 or more dims, e.g. MLA projections (BailingMoE3) and 4D conv kernels. Use the tensor name instead: all tensors used by GGML_OP_MUL_MAT_ID are named *_exps or *_chexps. This also removes the BailingMoE3 special case. The imatrix holds one vector of ne[0] values per 2D slice, so the expected size is ne[0]*ne[2]*ne[3], not ne[0]*ne[2]. This matches how the imatrix is indexed per row for 4D tensors in llama_tensor_quantize_impl. Assisted-by: Claude Opus 5.5
* qwen: merge tensors on convert * convert: use generator functions * qwen4exp: remove conts and pre-scale
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c3379b02da
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (has_state) { | ||
| state_vec = batch_inp.state; | ||
| } |
There was a problem hiding this comment.
Honor per-token offsets when flattening state embeddings
When a caller sets state embeddings in an order different from token-index order, this copies the states onto the wrong tokens. llama_batch_ext::set_token_state() appends each state in setter-call order and records its location in tokens[idx].state_off, but this assignment ignores those offsets, and ubatch_add() subsequently reads states by token index. For example, setting token 1's state before token 0's silently swaps their hidden states during MTP decoding. Build state_vec in token order using each token's state_off, as the mixed-embedding path does for embd_off.
Useful? React with 👍 / 👎.
| for t in flatten(suite): | ||
| tid = t.id() | ||
| try: | ||
| t.run(res) |
There was a problem hiding this comment.
Run the discovered test suite without bypassing fixtures
For Python agentic tasks whose hidden tests use setUpClass() or setUpModule(), flattening the discovered suite and calling each test's run() directly skips those fixtures, as well as their teardown and cleanup hooks. This incorrectly grades valid implementations as failures and can leak state between tests. I reproduced this with a test that reads an attribute initialized by setUpClass(): standard unittest discovery passes, while this runner reports error. Run the discovered suite through its normal TestSuite.run(res) lifecycle while retaining the custom result collector.
Useful? React with 👍 / 👎.
| (int64_t) (beta->nb[2] / sizeof(float)), | ||
| (int64_t) (beta->nb[3] / sizeof(float)), | ||
| 1.0f / sqrtf((float) S_v), | ||
| g->ne[0] == 1 && ggml_get_op_params_i32(dst, 0) < v->ne[2] && S_v == 128 && q->ne[0] == 128, |
There was a problem hiding this comment.
Exclude snapshot-producing batches from the MMA path
On an MMA-enabled GPU, this condition admits GDN batches with 1 < K < n_tokens when the other shape requirements are met, but gdn_single() only writes the final recurrent state and has no snapshot-count or snapshot-stride argument. For example, an eligible 128-token batch with K=17 leaves snapshot slots 1–16 unwritten. build_recurrent_attn() nevertheless copies all 17 slots into the recurrent cache, so a subsequent partial rollback can restore stale state and silently change inference results. Require K == 1 for this path until the MMA kernel implements the per-token snapshots.
Useful? React with 👍 / 👎.
Syncs master with upstream ggml-org/llama.cpp at b11887 (6050b46), up from b11436 (b9a5a00, 2026-10-06). Must be merged with a merge commit, not squashed, so the sync point stays an ancestor of master (see
scripts/unsloth/upstream-sync.json).Why now: every pin has merged a newer upstream tag than master (#61, #264, #304 now carry b11879). With master this far behind, a pin PR targeting master would show its own change plus ~450 upstream commits, so the pins have been parked on per-pin
base/upstream-<sha>branches. Once master contains b11887, each unslothai pin PR can target master directly and GitHub shows exactly its own change.What the merge does:
git merge --no-ff b11887; the only conflicts were modify/delete.sync_deletes.pykept the fork's deletion for all 19 upstream workflows (the fork owns no upstream CI).AGENTS.mdandCODEOWNERSstay deleted (removed_ok).behavior-check.ymlandbuild-wasi.yml, are dropped for the same reason (the 10-06 sync let 9 of these through and CI: drop stray upstream workflows and waive the deliberate metadata deletions in the sync guard #275 had to remove them).upstream-sync.json: tag b11887, commit 6050b46, synced_at 2026-10-25. That is the only change to.github/orscripts/unsloth/: every workflow and script is byte-identical to master..github/andscripts/unsloth/the tree equals b11887 exactly, apart from the threeremoved_okdeletions.Checks run locally:
verify_upstream_sync.py --merge HEAD --fork master --upstream b11887: history PASS (all 377 fork commits kept), tree PASS, deletions PASS (22, all upstream retiring its own files), ast PASS, c_enums PASS. content flags onlyupstream-sync.json, which the sync has to edit; the 10-06 sync shows the same line.check_workflow_triggers.py: 14 workflows, nopull_request_target, no unjustifiedworkflow_run.test_sync_deletes.py,test_verify_upstream_sync.py(all negative controls caught),test_check_workflow_triggers.py: pass.The nightly and the pin set are unaffected: the prebuilt builds an aged upstream tag plus
pr-set.jsonand only readsscripts/unsloth/from master, which is unchanged.