Repository navigation
common : add --moe-cache-mib auto to size the MoE expert cache from the fit - #250
danielhanchen wants to merge 28 commits into
Conversation
Two things grew the CUDA compute buffer with n_kv * n_ubatch: - build_kpool_select reshaped the host-side kq_mask (and gather_mask, new_pool_idxs) once per DSA layer. The scheduler copies every view of a host input to the device separately, so each layer held its own [n_kv, n_ubatch] f16 copy of the mask. Build those views once per graph. - The reserve re-pools every pool (kv size / kpool * n_seq_max pools), and the pooling kept several f32 [head_dim, kpool] copies of all of them alive. Pool in chunks of 16384 pools; decode-time graphs still use one chunk. GLM-5.3-Flash UD-IQ1_S, -c 244480 --kv-unified -fa on, CUDA0 compute buffer: -np 4 -ub 512: 4923 -> 1153 MiB, -np 1: 3275 -> 941 MiB, -np 4 -ub 2048: 13954 -> 4463 MiB. Perplexity is unchanged.
…-org#30034) With GGML_BACKEND_DL=ON, backends must be loaded explicitly before creating models. Assisted-by: Codex
* ggml: refactor selective expert copying to user code * tests: enroll two models into selective expert copy test * tests: use deepseek2 as test model * improve comment in ggml-backend.h Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * cont: fix whitespace * cont : better comments Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…gml-org#30020) The speculative MTP init enables NextN extraction on the target and draft contexts after both were created and their schedulers reserved. With unmasked extraction the trunk graph keeps every token through the last layer instead of cropping to the output rows, so the first decode reallocates to that batch's shape and the next, wider batch trips GGML_SCHED_DEBUG_REALLOC. Invalidate the reserve when the flags change so the next compute re-reserves with the new graph shape. Assisted-by: Claude
…0040) * add text_config as fallback * remove redundant llm_config check Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
…#30017) * mimo2 : always emit h_nextn the other nextn-capable models set it unconditionally Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD * models : consolidate nextn row cropping into shared helpers - replace the duplicated crop conditions and the per-model flags (narrow_early, crop_before_ffn, crop_last_layer, emit_h_nextn) with two helpers on llm_graph_context: crop_before_nextn() / crop_after_nextn() - models that only tested embeddings_nextn_masked now share the same condition, so they crop the last layer before the nextn capture whenever extraction is off - t_h_nextn is now set unconditionally in mimo2, qwen4exp and deepseek4 (as in the other nextn-capable models); host-side reads stay gated by cparams.embeddings_nextn Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD
A 1.0 loader (e.g. Android 8.1) has no vkEnumerateInstanceVersion, so backend init called a null pointer. Treat it like any loader under 1.2. Fixes ggml-org#29871. Assisted-by: Claude Opus 5.5
* ggml-cuda: chunk large BF16/FP16 to F32 conversions * Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * ggml-cuda: respect dst stride in chunked cuBLAS matmul --------- Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* ggml: fix CLAMP on non-contiguous views (CPU, CUDA) CUDA clamped ggml_nelements values flat and ignored the view strides. CPU addressed row j as j*nb01 and ignored nb02/nb03. Both now follow the strides of dims 1..3; CUDA supports_op requires contiguous rows, like Metal. test_clamp gains a non-contiguous view case. * cuda: clamp kernel uses fastdiv for the view strides
* rpc: allow -sm tensor * fix flush for apple rdma * move graph_uids to rpc_dispatcher * cont : fix conflict * cont: stop spinning dispatcher thread * remove meta backend change * add TODO to simplify logic * rpc: bump major version --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Assisted-by: Claude
…-org#30042) The gather path attended over the selected latents with a plain matmul and softmax. It only ran with n_ubatch <= 16, and the flash attention backends now skip the masked rows through n_kv_max, so the scatter path covers every case. Drop the gather flag, the gathered attention branch and gather_mla_rows. set_input_kpool always maps padding to the n_kv sentinel, and the slot mask becomes sel_mask since only the scatter reads it.
* model: K2 Horizon gguf conversion code
* model: loading hparams and tensors in k2-horizon.cpp
* model: K2 Horizon compute graph
* model: K2 Horizon compute graph adjustment and registering tokenizers
* model: K2 Horizon chat template and accomodate safetensors naming
* unicode : add the K2-Horizon pre-tokenizer splitter
The K2-Horizon regex had no arm in unicode_regex_split_custom and fell through to the
general std::regex fallback, which fails two ways.
On MSVC std::regex rejects \p{...}, so no K2-Horizon GGUF loads on Windows at all:
llama-quantize, llama-imatrix and llama-perplexity all abort with
regex_error(error_escape) before a token is produced.
Where the fallback does compile it is still wrong. unicode_regex_split collapses each
codepoint to a single byte naming its Unicode category before matching, and U+200C/U+200D
are category Control, which has no entry in k_ucat_cpt, so both become the 0xD0 fallback
byte. The literal and alternatives in K2's regex can then never match and
every ZWNJ or ZWJ ends a letter run.
The splitter is the existing llama3 one with a single rule widened, since K2's regex
differs from llama3's only in that a letter run also takes marks, ZWNJ and ZWJ.
tests/test-unicode.cpp gains a case for this: it fails before the change with
[Amy] [ZWNJ khaham] and passes after with the run intact.
* tests: expand K2 Horizon unicode splitter coverage
* unicode: handle K2 Horizon case folding and empty input
Assisted-by: Codex
* jinja : support sequence indices in selectattr and rejectattr
Assisted-by: Codex
* model : add K2 Horizon dense and MoVA support
Includes the K2 Horizon implementation from ifm-ai/llama.cpp with converter, tensor-parallel and model save/reload fixes.
Assisted-by: Codex
* chat : support K2 Horizon reasoning and tool calls
Assisted-by: Codex
* conversion: remove obsolete K2 Aurora alias
Assisted-by: Codex
* k2-horizon: enforce response schemas and load YaRN betas
Constrain final JSON after reasoning, accept flexible JSON tool envelopes,
enforce XML dialects, and handle repeated or alternate thinking markers.
Load YaRN beta metadata instead of retaining the default values.
Add schema, streaming, continuation, and model reload regressions. Validate
CUDA and CPU builds and 0.9B, 4B, and MoVA conversation/tool round trips.
Assisted-by: Codex
* renaming template fixture
* adressing cisc follows ups
* desloppify the parser / adress aldehir comments
* clean test-chat
* remove fallback : model trained mostly on high anyway
* fix k2 attn_v_exp tn splitting and metal fusion baseline
* k2-horizon : forward expand views before sums
* k2-horizon: copy embds before group norm to fix TP
* disable tesnor parallelism
---------
Co-authored-by: Ryandito Diandaru <ryandito.diandaru@mbzuai.ac.ae>
Co-authored-by: WestWaters <mario.papaleo2013@gmail.com>
Co-authored-by: Natani L. Mayday <71436458+TaskPuppyNatani@users.noreply.github.com>
Co-authored-by: West <100190545+WestWaters@users.noreply.github.com>
Co-authored-by: aaryamonvikram <aaryamonvikram@gmail.com>
Co-authored-by: aaryamonvikram <96529820+aaryamonvikram@users.noreply.github.com>
* vocab : implement PLaMo-3 tokenizer pre-segmentation The PLaMo-3 tokenizer inserts hard boundaries before running the Unigram DP, around <|plamo:...|>-looking text, and around runs of at least 4 identical characters or 2 spaces. Without them llama.cpp tokenizes code indentation and repeated punctuation differently from the reference. Reproduce the two re.sub() passes in llm_tokenizer_plamo2 by encoding each segment independently. * add vocab type "plamo3" * Update src/llama-vocab.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * misc change --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
…-org#28782) * ggml-cuda: use per-thread stream for buffer-init padding memset * ci : re-enable test-backend-ops -j for ROCm
The XIELU CUDA kernel template is already generic over the element type; only the F32/F16 type assertion and the else-if dispatch were missing. Add the nv_bfloat16 branch to the launcher, and drop the temporary supports_op gate in ggml-cuda.cu that rejected BF16+XIELU. test-backend-ops gains two BF16 cases ([10,5,4,3] and [512,16,1,1]). docs/ops/CUDA.csv and docs/ops.md are regenerated; the F32 xIELU row flips from no to yes as well, i.e. the previous record was stale. Tested: - Mac CPU: xIELU F32/F16/BF16, 6/6 - Mac Metal: existing F32/F16, 4/4; BF16 still unsupported - RTX 4090 CUDA: xIELU F32/F16/BF16, 6/6 - RTX 4090 CUDA BF16-only: 2/2 - git diff --check passes
# Conflicts: # src/models/glm5-next.cpp
With --moe-cache-mib auto, --fit sizes the MoE cache when the experts do not fit in device memory: the memory left after the dense weights, the context and the compute buffers goes to the cache, or with --moe-cache-static F a fraction F of it to the experts of whole layers kept in device memory (first layers first). The result is measured again with the cache allocated and the cache shrinks by any overshoot. If the cache would be smaller than n_expert_used + 1 slots per group of cached layers, fit falls back to whole layers without a cache. Only for a MoE model on a single GPU, otherwise no cache is used. llama-fit-params prints the chosen --moe-cache-mib.
- roll back a partially applied MoE cache on errors during the fit - reduce the static layers and re-measure if they overshoot - re-measure up to 4 times while shrinking the cache - require a cache of at least twice the minimum, no cache if no layer is in host memory - do not cache the NextN/MTP layers in the main context
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e5b686c2e3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -0,0 +1,362 @@ | |||
| #include "models.h" | |||
There was a problem hiding this comment.
This commit is presented as the --moe-cache-mib auto change, but this new K2 architecture is bundled with unrelated EmbeddingGemma2 support, PLaMo3 tokenization, decision-model changes, RPC graph caching and collectives, and backend kernel changes. These features do not support the cache implementation and make the change too broad to review, test, or revert safely; split them into independent commits or submissions.
AGENTS.md reference: AGENTS.md:L60-L63
Useful? React with 👍 / 👎.
| GGML_LOG_ERROR("[%s] failed to listen on comm port %u\n", __func__, request.port); | ||
| return true; | ||
| } | ||
| state.peer = srv->accept(); |
There was a problem hiding this comment.
When rank 1 cannot reach the new communication port, rank 1 stops retrying after about five seconds, but rank 0 remains blocked indefinitely in this accept(). The client then waits forever while synchronizing the two initialization requests, so a firewall, routing problem, or failed peer hangs RPC communicator initialization instead of falling back; use a bounded accept or otherwise cancel rank 0 when the peer fails.
Useful? React with 👍 / 👎.
| GGML_ASSERT(hparams.n_value_expert <= LLAMA_MAX_EXPERTS); | ||
| GGML_ASSERT(hparams.n_value_expert_used > 0); | ||
| GGML_ASSERT(hparams.n_value_expert_used <= hparams.n_value_expert); | ||
| } else { | ||
| GGML_ASSERT(hparams.n_value_expert_used == 0); |
There was a problem hiding this comment.
For a malformed or untrusted K2 GGUF whose value-expert counts are zero, inconsistent, or exceed LLAMA_MAX_EXPERTS, these assertions terminate the entire process rather than reporting a model-load error. GGUF metadata is file-controlled, so validate the counts and throw through the loader's normal error path.
Useful? React with 👍 / 👎.
| for (int i = 0; i < n_layer; ++i) { | ||
| auto & layer = layers[i]; | ||
|
|
||
| const bool is_moe_layer = n_expert > 0 && (uint32_t) i >= hparams.n_layer_dense_lead; |
There was a problem hiding this comment.
For K2 checkpoints with decoder_sparse_step greater than one, conversion writes that value as moe_every_n_layers, but this predicate treats every layer after the dense prefix as MoE. Loading then requests expert tensors from the intervening dense layers, and the graph repeats the same incorrect classification; include hparams.moe_every_n_layers when determining whether a layer is routed.
Useful? React with 👍 / 👎.
| params.moe_cache_static = std::stof(value); | ||
| if (params.moe_cache_static < 0.0f || params.moe_cache_static > 1.0f) { |
There was a problem hiding this comment.
With --moe-cache-static nan, std::stof returns NaN and both range comparisons are false, so the invalid value is accepted. The fitter later converts moe_cache->static_frac * surplus to int64_t, which is undefined for NaN; require std::isfinite before accepting the fraction.
Useful? React with 👍 / 👎.
| t_red->src[0] = t_dst; | ||
| t_red->src[1] = t_peer; |
There was a problem hiding this comment.
For F32 partials with at least 32768 elements, each rank sends a BF16-rounded value but this reduction adds the received BF16 value to its own original F32 value. Rank 0 therefore computes local0_f32 + bf16(local1) while rank 1 computes local1_f32 + bf16(local0), so the supposed all-reduce produces different tensors on the two ranks and later tensor-parallel layers consume divergent activations; cast both local and remote operands through the same wire representation before adding.
Useful? React with 👍 / 👎.
|
|
||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" | ||
| LIBS=(libllama libmtmd) |
There was a problem hiding this comment.
Replacing library discovery with this fixed list drops libggml.so and the ggml component libraries that the previous discover_libs implementation found under build/bin. As a result, a release can introduce a binary-incompatible change to the separately versioned public ggml library while check-release-apiabi.sh still passes; include libggml or retain discovery of the libraries common to both builds.
Useful? React with 👍 / 👎.
| } | ||
| } | ||
| ggml_status status = ggml_backend_graph_compute(backends[device], graph); | ||
| ggml_status status = ggml_backend_graph_compute_async(backends[device], graph); |
There was a problem hiding this comment.
When an RPC client disconnects immediately after submitting a graph, this asynchronous call can return while the remote backend is still using the client's buffers. The server loop then destroys rpc_server, whose destructor frees every buffer without synchronizing, so an interrupted or abruptly closed connection can free device memory underneath active kernels; synchronize the backends before teardown.
Useful? React with 👍 / 👎.
… reject a non-numeric --moe-cache-static A context that shares the main model (MTP) runs on its weights wherever the fit places them, so its memory was measured once with its own params and went stale when the fit moved experts to the host. Measure it with the main model params and again whenever their placement changes. std::stof accepted NaN and trailing junk for --moe-cache-static.
|
Superseded by #251, which carries the same change on mainline plus ggml-org#29887 only, without the #243 refresh, so its pin merges independently. |
Adds
--moe-cache-mib autoon top of ggml-org#29887 (GPU LRU cache for MoE experts kept in host memory). Withauto, the fitter sizes the cache itself instead of the user guessing a MiB number.What it does
--moe-cache-mib auto(or-1) only acts when--fitruns on a single GPU, the model has routed experts, and the user did not set-ngl,-tsor-ot:global_surplus_cpu_moe). Withauto, every routed expert goes to host memory and that whole surplus becomes the cache. In other words: free VRAM minus the fit margin minus everything that is not a routed expert (attention, shared experts, dense FFN, output, KV, compute buffers, a draft model).n_expert_used + 1slots for every layout group), fit takes its normal path with no cache.--moe-cache-static F(default 0) keeps a fraction F of that budget as whole expert layers resident on the device instead.The decision is logged in one line:
Other details:
--load-mode none. The cache copies misses from host memory, and pinned memory is 2.5x to 8x faster to copy from than pageable mmap. With mmap, the cache was slower than no cache in our runs.Measurements
All runs used
--load-mode none -ub 2048 -c 32768, single GPU, smallest Unsloth dynamic quants. Values are decode tok/s / prompt tok/s at 2048 tokens. Budgets below the card size were emulated with--fit-target. Eachautoarm ran twice, and the two runs agree within 0.5%.--fit on(no cache)--fit on --moe-cache-mib autoWhy all experts on the CPU instead of keeping some layers resident. At a fixed expert budget we swept the cache share from 0 to 100%, with the rest as whole resident expert layers. That covered 4 models and 8 GPU/host setups (T4, L4, A100 40/80GB, RTX PRO 6000; 4 to 48 vCPUs). Whenever experts spill, a cache covering the whole budget was the best or within about 5% of the best. Its decode gain over resident layers only:
Small caches mixed with resident layers can be slower than no cache, hence the default
--moe-cache-static 0.Tests
llama-fit-params --moe-cache-mib autoon all four models with a capped free-memory report. They are sane, the fallbacks trigger when the dense part does not fit, and the measured device total lands within 1 MiB of the target.llama-serverlaunches stayed inside their budget and answered requests, with cache hit rates around 69%.auto,llama-completion --temp 0 -n 64gives the same tokens as the whole model on the GPU (Qwen3.6, gemma-4).Depends on ggml-org#29887 (pinned at
74db68e04) and includes the #243 refresh it was developed on.