Repository navigation
Carry what the dropped GLM-5-Next, Qwen MTP and readahead pins had beyond upstream - #241
Open
danielhanchen wants to merge 58 commits into
Open
danielhanchen wants to merge 58 commits into
danielhanchen wants to merge 58 commits into
Conversation
The graph cache is keyed on cgraph->nodes[0] alone, so two evaluations that share a first node but differ in shape collide on one entry. Warmup needs two consecutive calls with unchanged node properties, so a workload whose batch shape varies resets warmup on nearly every call and falls back to eager launch. Speculative decoding is exactly that workload. The qwen4exp verify batch is distributed 2:13 percent, 3:11 percent, 4:75 percent as the accepted count varies, where qwen35 sits at 4:98 percent and is effectively constant. Host launch time for the qwen4exp target decode was 1.52 ms with the draft head disabled and 12.35 ms with it enabled, while GPU time was unchanged, so the regression was entirely host side. The key now mixes the first node, the last node and the node count. This is O(1) rather than a walk over every node: the existing uid early return fires on 127 of 128 decodes, so the hot path must not touch node data. An earlier all-nodes hash reintroduced exactly the per-node walk a CUDA graph exists to avoid. Measured overhead against the previous key is 0.2 to 0.6 percent, with both variants built into one binary to avoid comparing across runs. Capture churn on Qwen3.8-27B UD-Q2_K_XL drops from 52 captures and 50 destroys to 4 and 0, with identical output md5 and an unchanged speculative ratio. Across 14 distinct prefill shapes the cache instantiates 16 entries against 14 before, with no destroys and no growth, and is capped at 64 by LRU on top of the existing sweep. test-backend-ops passes 13646 of 13646 on CUDA0, and Llama-3.2-1B-Instruct Q8_0 is byte identical with no throughput change. (cherry picked from commit 5a08a71)
llama_prefetch_rows only issued madvise(MADV_WILLNEED) on Linux, so the macOS prebuilt read per-layer embedding rows (Gemma 4 PLE, the qwen4exp n-gram table) one page fault at a time. madvise and MADV_WILLNEED exist there with the same meaning. Carries the macOS half of #137.
The glm5-next indexer cache rows hold the compressor gates next to the keys, and the gates feed a softmax. Passing -ctk q8_0 through to that cache quantized them too. Keep it F16 then, as unslothai's earlier GLM-5-Next support did; qwen4exp rows carry no gate and are unchanged.
The NextN block was loaded but build_arch_graph threw for LLM_GRAPH_TYPE_DECODER_MTP. Build it: eh_proj([enorm(e) ; hnorm(h)]), then one DSA layer with a plain residual (the block has no mHC tensors), the MoE + shared expert FFN, and the head norm and head (nextn shared ones when present). It reuses the trunk's k-pool input and DSA layer; the MTP context memory already holds only the NextN block. h_nextn is the post-norm hidden state, the same kind the trunk exports.
|
You have reached your Codex usage limits for security reviews. Please try again later. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b96a713a48
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+2598
to
+2600
| // a captured cuda graph hard-codes the shapes, so a caller that alternates shapes - a | ||
| // speculative verify batch, for example - needs a separate instance per shape. with a | ||
| // single key per split, every shape change resets the warmup and no graph is ever used. |
There was a problem hiding this comment.
These newly added lines hard-wrap sentences across several comment lines solely to fit a column width. Join each sentence onto one line; the repository explicitly prohibits fixed-width wrapping and splitting sentences mid-line because it makes reviews and future edits harder.
AGENTS.md reference: AGENTS.md:L74-L81
Useful? React with 👍 / 👎.
| // NextN draft head: one DSA layer with a plain residual (no mHC), fed by eh_proj([enorm(e) ; hnorm(h)]) | ||
| llama_model_glm5_next::graph_mtp::graph_mtp(const llama_model & model, const llm_graph_params & params) : | ||
| graph(model, params, no_build{}) { | ||
| GGML_ASSERT(hparams.n_layer_nextn == 1 && "GLM5-Next MTP has a single NextN block"); |
There was a problem hiding this comment.
When a GLM5-Next GGUF declares more than one NextN block and the user selects MTP drafting, this assertion terminates the entire process. This is reachable because conversion/glm.py preserves an arbitrary positive num_nextn_predict_layers value and the generic loader only checks that it does not exceed the total block count; reject unsupported counts with a recoverable model/graph error instead of GGML_ASSERT.
Useful? React with 👍 / 👎.
|
|
||
| res->t_embd = cur_out; | ||
|
|
||
| cur = ggml_mul_mat(ctx0, layer.nextn.shared_head_head ? layer.nextn.shared_head_head : model.output, cur_out); |
There was a problem hiding this comment.
When a loaded adapter targets blk.N.nextn.shared_head_head, the adapter loader accepts and associates those weights with this tensor, but the raw ggml_mul_mat bypasses the LoRA lookup, so the draft logits silently ignore that adapter. Use the existing build_lora_mm path, as the other MTP output heads do, so speculative predictions remain consistent with the adapted model.
AGENTS.md reference: AGENTS.md:L80-L80
Useful? React with 👍 / 👎.
Shreesh-Sree
pushed a commit
to Shreesh-Sree/llama.cpp
that referenced
this pull request
Oct 6, 2026
…ed heads (unslothai#242) Drop ggml-org#27754, unslothai#144 and unslothai#137: upstream merged GLM-5-Next (ggml-org#27773), Qwen4Exp MTP (ggml-org#29761) and row prefetch (ggml-org#29599), and the old pins no longer merge. Old-GGUF loading moves to unslothai#239, shared MTP heads to unslothai#240, and what the dropped pins had beyond upstream to unslothai#241. Repin ggml-org#24423, ggml-org#25731, unslothai#61 and unslothai#176 to their conflict-fixed heads.
…n-optimizations # Conflicts: # src/models/models.h
The shape keyed graph cache added a 64 entry LRU cap on top of the existing time based sweep, to bound a workload with many distinct shapes. Tensor split needs about 2 * n_layers + 1 graphs per device for a single shape, one per segment between the cross-device reductions, so every model past ~31 layers exceeds 64. The LRU then evicts graphs that are still in use and every token re-captures instead of replaying, which is as slow as running with GGML_CUDA_DISABLE_GRAPHS=1. Upstream tried the same cap (64, then 128) in ggml-org#21611 and dropped it in review for this reason, keeping only the sweep that evicts graphs unused for 10 s. Do the same here. The shape key stays, so an alternating speculative verify batch keeps one graph per shape. Refs unslothai/unsloth#12468
This was referenced Oct 8, 2026
danielhanchen
changed the base branch from
base/upstream-1fb7ef3e3
to
base/upstream-988190680
October 8, 2026 15:24
EmeraldBitTwizzler
pushed a commit
to EmeraldBitTwizzler/unsloth-llama.cpp
that referenced
this pull request
Oct 9, 2026
…at merge past b11453, drop unslothai#247, pin unslothai#251 Upstream K2 Horizon (ggml-org#29535), the glm5-next gather removal (ggml-org#30042) and the GLM5-Next MTP graph (ggml-org#29928) broke the Inkling, unslothai#243 and unslothai#241 pins. Each branch now carries a merge of upstream master. unslothai#247 is in every tag from b11452 on. unslothai#251 adds --moe-cache-mib auto on top of ggml-org#29887.
EmeraldBitTwizzler
pushed a commit
to EmeraldBitTwizzler/unsloth-llama.cpp
that referenced
this pull request
Oct 9, 2026
unsloth: repin Inkling, unslothai#241 and unslothai#243 past b11453, drop unslothai#247, pin --moe-cache-mib auto (unslothai#251)
EmeraldBitTwizzler
pushed a commit
to EmeraldBitTwizzler/unsloth-llama.cpp
that referenced
this pull request
Oct 9, 2026
…DA graph cache count cap unslothai#253 is unslothai#241 plus one commit that removes the 64 entry cap on the shape keyed CUDA graph cache, which made tensor split re-capture graphs on every token (unslothai/unsloth#12468). Its branch now merges unslothai#241's current head, so pinning it instead of unslothai#241 carries every unslothai#241 item and the fix, and keeps pin_contract from reading the removed cap lines as a lost unslothai#241 change.
EmeraldBitTwizzler
pushed a commit
to EmeraldBitTwizzler/unsloth-llama.cpp
that referenced
this pull request
Oct 9, 2026
unsloth: pin unslothai#253 (no CUDA graph cache count cap) in place of unslothai#241
EmeraldBitTwizzler
pushed a commit
to EmeraldBitTwizzler/unsloth-llama.cpp
that referenced
this pull request
Oct 12, 2026
…cache cap removal The 64 entry cap came from unslothai#241's own shape keyed graph cache commit, so the fix belongs on unslothai#241. Its branch now includes unslothai#253's commit (same head, 20dd977), and the pin goes back to unslothai#241. The built tree is unchanged.
EmeraldBitTwizzler
pushed a commit
to EmeraldBitTwizzler/unsloth-llama.cpp
that referenced
this pull request
Oct 12, 2026
unsloth: pin unslothai#241 with the CUDA graph cache cap removal, repin Inkling and unslothai#61 onto b11529
danielhanchen
added a commit
that referenced
this pull request
Oct 14, 2026
* llama : enable moe amax rescaling for laguna Assisted-by: Devin * llama : declare laguna sm tensor support
* hexagon: add GGML_HEXAGON_VTCM_MB and GGML_HEXAGON_NTHREADS caps Change-Id: Idffe1613c8db6f96aa842d43d1b2efaed45462a8 * hexagon: rename VTCM cap to GGML_HEXAGON_VTCM_SIZE, drop NTHREADS cap Change-Id: If4c5b9ce9fd6b725c3561421c2295d40599d012d * hexagon: make GGML_HEXAGON_VTCM_SIZE an override, accept K/M/G suffixes Change-Id: Ib8eea84c22f24e6d986b37f215e504b046478c2c
…g#31187) Extract the keyboard-shortcut delete confirmation into DialogConversationDelete and reuse it in the sidebar and mobile search page instead of the native window.confirm prompt. Assisted-by: Claude (Opus 5.5 Medium)
…#31176) * ui: open model details in a global drawer without leaving the page The model info actions in the selector used to navigate to the models page to open its details pane, and a phone opened a separate dialog instead. The drawer now lives outside the models table, driven by uiStore, so the info icon opens it over whatever page is showing. The models dropdown and the phone picker stay open under the drawer's backdrop, and keep focus while the drawer opens and closes, so the trigger's tooltip no longer pops over the drawer. Assisted-by: Claude (Opus 5.5 Medium) * ui: let the model details drawer reopen after a drag to dismiss Dragging the drawer down wrote vaul's own open state, which the one-way open prop then kept closed, so the info action could not open it again until a refresh. Bind open both ways through the store instead. Assisted-by: Claude (Opus 5.5 Medium) * ui: close the models selector from the details drawer chat actions Assisted-by: Claude (Opus 5.5 Medium)
* ui: add conversation compaction Add a compaction action that asks the model to summarize the conversation so far and stores the summary in a new compaction node appended to the branch. Later requests slice the branch at the last compaction node, keeping the system prefix and continuing from the summary instead of the full history, which frees up context while the complete history stays visible and navigable in the UI. The slice is applied once at the request funnel, covering the plain send, the agentic flow and the KV cache pre-encode, plus the continue path that bypasses it. The summary request is a one-shot outside the SSE replay machinery, streamed live into the node with a full rollback on abort, error or empty result. The action is exposed as a /compact slash command and a Compact button in the context gauge popup, with an overridable summarization prompt in the settings. * ui: pre-fill KV cache and refresh context gauge after compaction Once the summary is stored, pre-encode the compacted branch and stamp the returned prompt timings on the compaction node: re-encoding the reduced prefix is part of the compaction itself, so the server KV cache and the context gauge reflect the summary right away. The summary stream is registered under a prefixed identity (compact::convId), so the visibility kick and byte-offset resume protect it through tab switches while the reload probe never reattaches it as a chat turn. The stamped node becomes the shared context truth for the gauge and the processing state restore, and the stale live state that kept the gauge full is cleared. * chore: format * ui: allow a dedicated model for conversation compaction An opt-in setting reveals a model selector whose options come from the live router model list. The chosen model writes the summary and is loaded on demand when it is not resident, while the default path keeps resolving to the conversation's own model. * ui: align the compaction model field with project conventions * ui: fix the model option helpers import path The compaction model selector pulls filterModelOptions and groupModelOptions from the models component folder, where they no longer live, so the build fails to resolve the module. Point the import at the navigation folder, which now owns them. Missed during the rebase: the helpers moved in a commit that does not touch any compaction file, so nothing conflicted. * ui: import the model option helpers from models/utils * ui: follow the moved models selector in the compaction model field The models manager moved ModelsSelectorList under ModelsSelector and the model option helpers to /utils, where groupModelOptions takes the options alone. Drop the stale copy the rebase kept and read the helpers from their new home. * ui: drop the dedicated compaction model A summary written by another model loses the prefix the conversation model cached, so the whole history is prefilled again, and the model then resumes from text it did not write. The conversation model always writes its own summary. * ui: keep the last steps of a compacted conversation A compaction now folds the steps the model reads but the last ones, a step being a user message or an assistant message with its tool results, and stores how many it kept on the compaction node. The model then reads the system prefix, the summary, the kept steps and what follows. The summary request is the request the chat sends, with its options, tools and message shaping, cut before the preserved steps: the server reuses the prefix it cached and only the instruction is new. The number of preserved steps is a setting, 10 by default. * ui: run every compaction summary through one path Three callers needed the same thing - the automatic pre-encode pass, the action in the context gauge, and re-running a summary from the block - and each of them had grown its own copy of the request, the streaming and the storing. There is one path now: a summarization owns a node, streams into it, and reports itself through a ledger the rest of the app reads (`isCompacting`, which node belongs to which conversation). Re-running a summary is the same call on an existing node, so the two entry points cannot drift apart again. * ui: give the compaction row its own summary UI The summary was a line of text in a tool-call-shaped collapsible, which is how a compaction reads as a side note rather than as the thing the rest of the conversation now stands on. The row is its own block: a divider that names the boundary, and the summary inside it with the model that wrote it, its statistics, and the same actions an assistant turn has - copy, edit, fork, regenerate and delete all apply to the summary, and it collapses to the divider once a summary is in, because the shrunk context is the headline and the prose is the detail. The divider is a variant of the shared collapsible block, which also needed the toggle and the title split into snippets so a caller can lay them out without copying the markup. The action icons no longer take a role: they render what a message offers, and a compaction offers the same as any other stored message. * ui: report compaction progress on the row The row owns the run, so the row says how it is going: the prompt progress in the header while the context is being read, the phase once the summary starts streaming, and the token count once it is in. Before the first summary token there is nothing to collapse, so the header stands on its own instead of offering an empty block. * ui: keep the summarized turns quiet while a compaction runs The turns being summarized used to keep their own "Processing..." indicator and live reading stats while the summary streamed, so the conversation said the same thing twice in two places. The row reports the run, so the turns it replaces stay silent until it is done. * ui: hand the compact action to the context gauge Compacting is a decision about the context, so the action belongs with the numbers that show the context filling up rather than inside the form: it sits at the bottom of both the hover card and the drawer, under the breakdown it acts on, quiet enough to stay a footnote and disabled - with the row's own spinner - while a summary is already running. * ui: follow the bottom, otherwise leave the view alone The screen scrolled itself with a controller of observers, intervals, direction heuristics and landing timers, and still managed to move the view while the reader was reading: any mutation that grew the branch was treated as a reason to keep following. The page is the scroller, so it follows when the reader sits at the bottom, and stays where it is otherwise. A resize observer plus native scroll anchoring does the rest, and `pointerdown` buys the reader a moment of grace to be caught mid-drag. * ui: expand collapsible bodies with a height transition Expanding a block snapped it open: the body was mounted the moment `open` flipped, so a tall block moved the conversation around it in one frame. The body now goes through the same region the models table uses. The height interpolates and the visibility follows it, so a body is unmounted once the collapse has finished rather than the moment it starts - a closed block still costs nothing while tokens stream past it. * ui: open a compaction summary at the top of the screen Toggling a summary left the reader wherever they were, so a conversation that had just compacted showed the divider at the bottom of the screen and read down from there. A toggle now scrolls the page once and lands the divider a breath under the tabs, both ways: the summary is read from its beginning, and closing it returns to the same boundary instead of wherever the page happens to come to rest. The scroll waits for the body to reach its height, because before that the page is too short to scroll the divider up there and the call would clamp. That wait is the shared collapse duration rather than a transition event: an engine without `interpolate-size` animates the region's grid row instead of its height, so a listener on `height` would never fire there and the page would not move at all. * ui: spin the compaction icon while the summary runs The divider reported the phase in its title while the icon still showed a finished block, so a run in progress and a summary that had landed looked the same in the one place the eye reads first. The icon spins while the summary is running and folds back once it is in. * ui: let the compaction summary speak without the reasoning trimmings The summary is the thing the conversation now stands on, not a note beside it, and the conversation model always writes it, so the body drops the reasoning block's rule and indent (`flushBody` keeps them for the blocks that borrow them) and the model selector under the summary is gone - there is nothing left to swap. * ui: drop the model override from the compaction path The summary row no longer offers a model selector, so nothing hands regenerateCompaction a model anymore. Remove the parameter from the whole path, with the on-demand load of another model it carried: the summary always comes from the model the request funnel resolves. * ui: let the compaction summary think as the chat does The summary request forced thinking off, overriding the reasoning effort the user chose. The summary is the text the conversation resumes from, the task where reasoning helps the most, so the request now carries the very options of the chat. * ui: pre-encode the compacted branch as the next turn reads it The post-compaction pre-encode sent the branch without tools, while the tools render ahead of the messages, so it warmed a prefix the next turn never reads and that turn prefilled the whole compacted context again. The pre-encode now sends the tools, message shaping and reasoning handling of the chat, so the next turn only prefills its new message. * ui: pre-encode the branch the next turn actually reads The per-turn pre-encode sent the messages of the request plus one assistant message holding the streamed text, without tools. With tools on, it replaced the slot the next turn extends with a prefix nobody reads again, and after an agentic turn its history did not match the stored calls and results. Without agentic, it never ran. Both pre-encodes now read the stored branch up to its leaf, cut at the last compaction, and send it with the tools, message shaping and reasoning handling of the chat. The plain completion path triggers it too. * ui: refuse a compaction summary the output limit cut The stream handler read every chunk but dropped the finish reason, so a summary cut by max_tokens folded the conversation into half a text. The handlers now hand the finish reason to onComplete, and a summary that ends on length rolls the compaction back with an error. The summary was framed through a replacement string, so $& or 32560 in it changed the text the model read; it is framed as written now. * ui: wait for the collapse transition in the tool call tests The collapsible body now unmounts once its collapse animation ends rather than as soon as the block closes, so the "folds back" assertions ran before the body was gone. Wait out COLLAPSIBLE_TRANSITION_MS first. Assisted-by: Claude (Opus 5.5 Medium) --------- Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
…tor (ggml-org#31212) On desktop, the reasoning effort toggle now gets a muted background on hover and while its menu is open, matches the model selector's 24px height, uses a 13px icon and sits 5px from the selector. Assisted-by: Claude (Opus 5.5 Medium)
* opencl: add q3_k AOS mv * opencl: add q3_k SOA mv and tiled mm * opencl: add q3_k noshuffle gemm/gemv for adreno * opencl: add q3_k dp4a gemm for adreno Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com> * opencl: tune q3_k kernels for adreno Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com> --------- Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex
The org-less group renders without a header, so when it followed a named org its models looked like they belonged to that org. Put it ahead of the named org groups. Fixes ggml-org#25227 Assisted-by: Claude (Opus 5.5 Medium)
Every caller passes n_pad = 1 and the only two users clamp it to 256. Drop the parameter, the members and the trivially true kv_size assert. Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
…d models (ggml-org#31216) * tests : fix multi-seq split replay against trained models The rollback snapshot is captured by the graph that decoded the rolled-back tail, while the reference context decodes the same prefix with different batch shapes. The two states differ in the last bits and trained models amplify the difference with every replayed token. Compare only the first replayed token of each seq, then restore the rollback state into the reference context before the seq-1-only tail. Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * tests : require bit-exact seq-1 isolation in split replay The seq state save is not always bit-lossless - SWA-masked cells are not saved - so reload the restored state back into the rollback context. That way both contexts enter the seq-1-only tail with byte-identical memory, including the cell layout, and the tail logits must match bit-exact. Also assert that the restored state round-trips bit-identical. Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * tests : drop the n_swa partial-state check from ps The check keys off llama_model_n_swa(), which returns 0 for dsv4 even though its raw KV cache has a 128-cell sliding window. The partial state grows with the tokens there by design, so the flatness check is wrong. Keep the partial <= full invariant. Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* ggml : relax scale to non-contiguous views SCALE requires src0 to be 1D-contiguous. Relax this to dim-0 contiguity with arbitrary row strides (ggml_is_contiguous_rows), same as the unary ops in the Metal backend. - ggml: relax the assert in ggml_scale_impl - cpu: address src0 rows through their byte strides (f32, bf16) - cuda: stride-aware kernel, fastdiv row decomposition - metal: support bf16 scale (guarded unary instantiations) - sycl: keep the gate at contiguous, the kernel is flat f32 Add view variants to test_scale. Verified on CPU, Metal and CUDA. Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB) * cuda : skip the softcap fusion on non-contiguous views The fused softcap kernel reads src and writes dst as dense arrays, so a view reaching SCALE -> TANH -> SCALE now that SCALE accepts non-contiguous rows would be read wrongly. Keep such graphs on the strided kernels until the fusion becomes stride aware. --------- Co-authored-by: Pascal <admin@serveurperso.com>
* glm5-next : QSA without kq_mask * DSA: dsa top-k mask change tested with layers 0,3 of DeepSeek-V3.2
* metal : add GGML_OP_TOP_K_MASK Port the fused top-k mask from the CUDA backend: a grid-stride kernel fills the output with -inf, then one threadgroup per row copies the mask values of the kept cells over it. Function constants drop the score, blocks and extra paths that a given graph does not use. Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * metal : clean up top-k mask kernel formatting Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* model : add MiniCPM-OCR-Jev support The model is a Qwen3.5-VL trunk with a value head read at the last token: it scores an OCR text against an image rendered from it, it does not generate. Add the openbmb-minicpm-ocr-jev decision type, served by POST /v1/systemone. The class scores are the pooled output of the sequence, read with last pooling, so the conversion turns score.weight into cls.output and sets embedding_length_out. The checkpoint has no LM head, so text generation is rejected instead of using the output weights duplicated from the token embeddings. The task is fixed, so its prompt is entirely in the systemone template, the same way h2o-lightning keeps its default there. The processor of the checkpoint allows up to 16384 merged tokens per image, more than the Qwen-VL default of mtmd, so the mmproj keeps the image budget of the preprocessor config. The checkpoint declares MiniCPMOCRForSequenceClassification in its config.json, as other models do. Signed-off-by: tc-mb <tianchi_cai@icloud.com> * fix code Signed-off-by: tc-mb <tianchi_cai@icloud.com> * use gguf meta classifier_output_labels * remove redudant code changes --------- Signed-off-by: tc-mb <tianchi_cai@icloud.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Codex
…ng (ggml-org#21231) * server: add --models-memory-max parameter to allow dynamically unloading models when they exceed a memory size threshold estimate with to-be-loaded model size included use no_alloc to get memory requirements for model load only set model memory_mb if not previously calculated use memory margin instead of total size limit, apply to each device separately add server memory debug logging move llama_context_device_memory function to llama-ext.h fix model count exceeded check improve memory_per_device map naming improve variable naming, fix style also strip models memory margin from child processes cont : clean-up replace device memory map with buft memory map. Use llama_get_memory_breakdown extract duplicated check into helper function move model memory estimation to subprocess precompute name->buft map, map GPU host types to CPU buft cleanup unused variable remove duplicated init calls * adapt to upstream changes * wip * common_memory_reports internface * SERVER_ROLE_MEASURE * server-models-sched * fix build * server : fix memory accounting of measure children and claim spin A measure child runs no_alloc, so the free memory it reports does not include its estimate: subtracting self there erased the memory held by other processes, down to zero with --fit on, and the next load could overshoot the margin. Subtract self only for instance reports. A queued load only checked --models-max before claiming, so when the memory margin blocked it behind a busy model, load() threw and the waiter retried in a tight loop. Refuse the claim while the limits are exceeded and no idle model can be unloaded, so the waiter sleeps on the condition variable until one goes idle. --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: Pascal <admin@serveurperso.com>
…ggml-org#31246) The expand button wrapping the system message card defaulted to inline-block. WebKit takes an inline-block's baseline from its last line even when that line is clipped by the card's max-height, so the row grew to the height of the whole prompt and left a large empty gap below the card. Make the button a block so the row matches the card. Assisted-by: Claude (Opus 5.5 Medium)
…1165) * mtmd-cli: Reduced scope of batch pointer (ggml-org#31155) * mtmd-cli: Remove CLI regression test
…org#31239) * vulkan: allow dim 3 broadcast in MUL_MAT supports_op rejected every MUL_MAT whose src0 and src1 differ in ne3, although mul_mm, mul_mmq and mul_mat_vec already broadcast src0 over dim 3 through broadcast3. Such a matmul then ran on the CPU with a copy of the weight, and the split layout changed with the number of streams in the ubatch. Keep the strict check for MUL_MAT_ID only, and route the broadcast cases away from mul_mat_vec_nc, the one path that requires equal ne3. * tests : harden MUL_MAT with permuted dim 3 broadcast cases Harden the MUL_MAT coverage with a weight broadcast over dim 3 against a permuted per-stream activation, the shape of an MLA output projection with a non unified KV cache, for a single token, several heads and several tokens per stream.
) * tests : speed up tensor data generation and model saving Refactor save_models to create only the model, not a llama_context, which is not needed for saving. Replace the slow std::normal_distribution draw with a splitmix64 + box-muller normal generator. Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB) * tests : use a shared pool of normal draws for tensor data Generate a pool of normal draws once with std::mt19937 and std::normal_distribution, then have every tensor copy a cyclic slice of it instead of running a per-tensor box-muller. The name-keyed slice selection keeps the draw deterministic. Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB) * tests : log the size of saved models Report the produced file size in bytes and MB after each save. Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB) * tests : simplify the model save log Put the model name, target and size on one line per saved model. Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
This was referenced Oct 27, 2026
oobabooga
pushed a commit
that referenced
this pull request
Oct 28, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The nightly pin set drops three pins whose features landed upstream in a different form: ggml-org#27754 (GLM-5-Next, superseded by ggml-org#27773), #144 (Qwen3.8-Flash-Next MTP, superseded by ggml-org#29761) and #137 (lazy-table readahead, superseded by ggml-org#29599). The upstream versions do not carry everything those pins did. This PR brings the missing pieces back as four small commits on b11368 (
base/upstream-1fb7ef3e3), so the pin set can drop the old pins without losing speed.ggml-cuda: key the CUDA graph cache by shape--tensor-spliteach device replays 2 * n_layers + 1 graphs, so a cap below that evicts and recaptures every token (unslothai/unsloth#12468). Upstream has no cap either (ggml-org#21611).llama: prefetch lazily read gather rows on macOS toollama_prefetchonly issuesmadvise(MADV_WILLNEED)on Linux and Windows; #137 also covered macOS. One-line platform guard change.llama: keep the glm5-next indexer cache F16 under a quantized -ctk-ctk q8_0quantized them too; keep that cache F16 as ggml-org#27754 did. qwen4exp is unchanged.glm5-next : NextN (MTP) draft graphLLM_GRAPH_TYPE_DECODER_MTP. This builds the draft graph (eh_proj, one DSA layer, MoE + shared expert, head norm and head), reusing the trunk's k-pool input and DSA layer.6 files, +153/-18. No new flags.
Testing
GLM-5.3-Flash UD-IQ1_S and Qwen3.8-Flash-Next UD-IQ1_S, one exclusive B200, interleaved before/after runs (before = last good nightly, b11160 + the old pins; after = b11368 + the refreshed pin set including this PR):
--spec-type draft-mtp)macos-15runner (M1, Metal) with gemma-4-E2B and-lzm on: greedy output is byte-identical with and without it.additive_merge.py) and builds with-DLLAMA_FATAL_WARNINGS=ON.