feat(arch): muse-glimmer support with vision and dspark

CHANGELOG.md

@@ -8,6 +8,15 @@ adhere to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ### Added
 
+- Muse Glimmer support (GGUF arch `muse-glimmer`, Meta Muse Glimmer 30B):
+  a vendored text decoder with sandwich norms at two epsilons, an
+  attention output gate, and NoPE on the full-attention layers with RoPE
+  only on the 2048-window sliding ones. Vision rides the `muse-glimmer`
+  mmproj through a vendored ViT and a GGUF-only image processor. The ATEM
+  reasoning channel and its XML tool calls are wired through chat, serve,
+  thinking budgets, and the muse profile family (reasoning strength
+  low/medium/high/xhigh). `--draft-gguf` loads the DFlash drafter for
+  speculative decoding.
 - serve --speculative: a request arriving while one stream decodes with
   MTP no longer waits for it to finish; the stream converts to shared
   batch decode and speculation resumes once the batch drains back under

LICENSE

@@ -27,9 +27,9 @@ please contact asher_feldman@icloud.com.
 
 File-level exception: files that carry an "SPDX-License-Identifier: MIT"
 header (the DSpark draft-model module and the DSpark sidecar converter, the
-mlx-lm-style model modules for kimi-k3, minimax-m3, and hy-v3, and the
-kimi-k3 tests) are licensed under the MIT license (see LICENSE-MIT), not
-under this License.
+mlx-lm-style model modules for kimi-k3, minimax-m3, hy-v3, and muse-glimmer,
+the mlx-vlm-style vision module for muse-glimmer, and the kimi-k3 tests) are
+licensed under the MIT license (see LICENSE-MIT), not under this License.
 
 Notice
 
@@ -110,8 +110,10 @@ was taken:
 - gmlx/server_patches/apc.py and gmlx/thinking_budget.py carry code
   copied or ported from mlx-vlm (MIT, Copyright © Prince Canuma;
   licenses/mlx-vlm-LICENSE).
-- The pre-tokenizer split regex patterns in gmlx/tokenizer.py come from
-  llama.cpp (MIT; licenses/llama.cpp-LICENSE).
+- The pre-tokenizer split regex patterns in gmlx/tokenizer.py, and the
+  Muse Glimmer vision graph, grid index math, and image preprocessing in
+  gmlx/muse_glimmer_vlm_model.py and gmlx/vlm.py, come from llama.cpp
+  (MIT; licenses/llama.cpp-LICENSE).
 
 The test asset tests/e2e/assets/cats.jpg is reproduced from the mlx-vlm
 repository (MIT License, Copyright © Prince Canuma); see

README.md

@@ -265,8 +265,8 @@ Coverage runs across the major open-weight families: Llama and Mistral;
 Qwen 2 through 3.6, dense and MoE, including the gated-DeltaNet hybrids and
 Qwen3-Next; Gemma 1 through 4 plus DiffusionGemma; DeepSeek V3/R1 and
 V4-Flash; GLM 4 through 5.2; gpt-oss; Kimi-K3; MiniMax M2 and M3; Hunyuan
-A13B and Hy3; Granite, including the 4.x hybrids; Nemotron-H; Falcon-H1;
-ERNIE-4.5; Phi-3; Seed-OSS; and SmolLM3. New architectures land regularly,
+A13B and Hy3; Muse Glimmer; Granite, including the 4.x hybrids; Nemotron-H;
+Falcon-H1; ERNIE-4.5; Phi-3; Seed-OSS; and SmolLM3. New architectures land regularly,
 and a family is listed in the generated
 [architecture coverage matrix](https://github.com/asher/gmlx/blob/main/docs/arch-coverage.md)
 only after token-parity certification against llama.cpp at 16k context; the

docs/adding-architectures.md

@@ -4,10 +4,16 @@ What it takes for a new model family to become a supported architecture, and
 the acceptance gate every family clears before its row appears in the
 [coverage matrix](arch-coverage.md).
 
-Precondition: a GGUF arch is only reachable if the installed mlx-lm defines a
-`class Model` for the corresponding `model_type`. If mlx-lm doesn't implement
-the architecture, support is blocked upstream. gmlx never reimplements
-model math.
+Precondition: a GGUF arch needs a model class for its `model_type`. That class
+normally comes from the installed mlx-lm or mlx-vlm, and gmlx supplies only the
+tensor map and the config.
+
+A few families have no upstream class at all (kimi-k3, muse-glimmer). gmlx
+vendors the model math for those, in its own module, grafted into the upstream
+namespace so a later upstream implementation wins. Vendoring is the exception.
+It is worth the cost only when the family is otherwise unreachable, and it adds
+two obligations: numeric parity against llama.cpp, and a collision check that
+reports the graft once upstream ships its own class.
 
 ## The shape of the work
 

docs/arch-coverage.md

@@ -4,7 +4,7 @@
 
 A GGUF loads iff gmlx maps its `general.architecture` to a `model_type` with a model class in one of its installed backend packages (usually mlx-lm; a few archs are backed by mlx-vlm (e.g. DiffusionGemma) or mlx-embeddings (gemma-embedding)) and a config synthesizer exists (or `hf_source` supplies a config.json).
 
-**35 / 36 mapped architectures load end-to-end today** (no `hf_source` needed).
+**36 / 37 mapped architectures load end-to-end today** (no `hf_source` needed).
 
 | GGUF arch | model_type | model class | remap | synth | status | notes |
 |-----------|------------|:-----------:|-------|:-----:|--------|-------|
@@ -32,6 +32,7 @@ A GGUF loads iff gmlx maps its `general.architecture` to a `model_type` with a m
 | `minimax-m2` | `minimax` | yes | yes | yes | loadable | MiniMax-M2 (230B-A10B); every-layer fine-grained sigmoid-gated MoE (no dense layers, no shared expert) with full attention, full-width qk-norm (RMSNorm over head_dim*n_heads), and partial rotary (rotary_dim < head_dim). NEOX rope (no qk-permute); router/experts/correction-bias nested under block_sparse_moe.*; head_dim != hidden//heads (from key_length) |
 | `minimax-m3` | `minimax_m3` | yes | yes | yes | loadable | MiniMax-M3 (428B-A23B); M2's GQA base (partial rotary, head_dim from key_length, NEOX rope) plus gemma-style +1 RMSNorms (unbaked on load), per-head qk-norm, and a DeepSeek-V3-shaped MoE: leading dense layers, sigmoid gating + correction bias, routed weights renormalized x expert_weights_scale, per-layer shared expert (block_sparse_moe.shared_experts). SwiGLU-OAI activation. MSA sparse attention (llama.cpp PR #24908 semantics): a per-GQA-group indexer max-pools scores into 128-token blocks and top-16 blocks (local block forced) bound attention to 2048 KV per query - the form the model is trained with; runs whenever the GGUF carries the blk.N.indexer.* tensors or a `*indexer*.gguf` sidecar sits next to the model (GMLX_INDEXER_SIDECAR overrides discovery, GMLX_MSA_DISABLE=1 forces dense for A/B). Indexless GGUFs fall back to dense with a one-time warning (exact to 2048 tokens, degrades beyond - reasoning loops). Thinking tags are `<mm:think>`/`</mm:think>` (template-detected; the vocab's legacy `</think>` entries are decoys). Model class vendored from mlx-lm PR #1401 (+ gmlx MSA extension) until upstream ships models/minimax_m3.py |
 | `mistral3` | `ministral3` | yes | yes | yes | loadable | llama.cpp 'mistral3' = Ministral-3 / Mistral-Small-3.1, Llama layout |
+| `muse-glimmer` | `muse_glimmer` | yes | yes | yes | loadable | Meta Muse Glimmer 30B (llama.cpp 'muse-glimmer'); dense 52-layer sandwich-norm decoder - pre/post norms around both attention and FFN, at two epsilons (1e-5 pre/final, 1e-8 post) - with an afmoe-shaped attention output gate (sigmoid(x_norm @ W_gate) applied before o_proj), per-head qk-norm whose weights are synthesized at conversion to absorb qk_scale_factor (q_norm a uniform 3.87, k_norm ones), an unweighted RMSNorm on the token embeddings before layer 0, and a logit scale (output_multiplier 0.196) + gemma-style tanh softcap at 20. Attention runs a 3-of-4 sliding window (2048) with RoPE, and full attention with NoPE on every 4th layer - the inverse of the usual arrangement, and the reason the model card says '131072+': the largest positional offset ever resolved is the 2048 window, so there is no rope extrapolation ceiling and past 131k the only limit is KV memory at ~13 KB/token (just the 13 global layers grow; the 39 sliding ones are pinned by their rotating cache). The four per-layer norms carry a baked +1 that a plain RMSNorm consumes as-is (no gemma unbake). NORM rope with Q/K un-permuted at conversion => traditional=True, no qk_permute. Model class vendored (no upstream mlx-lm class; afmoe is the nearest relative). Pairs with the Muse Glimmer mmproj (--mmproj) for vision and with the DFlash drafter GGUF (--draft-gguf) for speculative decoding |
 | `nemotron_h_moe` | `nemotron_h` | yes | yes | yes | loadable | NVIDIA Nemotron-H MoE hybrid: Mamba2 SSM layers + sparse attention layers + MoE MLPs; layer_norm_epsilon (not rms_norm_eps) |
 | `phi3` | `phi3` | yes | yes | yes | loadable | Phi-3 mini/small/medium; fused qkv + gate_up; pass hf_source for 128K long-context (su/longrope) variants |
 | `qwen2` | `qwen2` | yes | yes | yes | loadable | Qwen2 / Qwen2.5 dense; QKV biases, tied embeddings on 0.5B/1.5B |

docs/cli.md

@@ -121,11 +121,12 @@ plumbing at all (`--bench`, `--bench-depths`, `--report-only`, `--stream-cpu`,
 falling back to plain text generation.
 
 A text-only request under `--mmproj` runs through the MTP speculative path
-whenever a drafter is available: a `--draft-gguf` assistant (gemma4) or a
-native `nextn` head in the LLM GGUF (qwen3.5/3.6, no companion needed). The
-verify walk only touches the language model, so a resident VLM gets the decode
-speedup on text turns, token-identical to the same model's text-only MTP. An
-image or audio request uses the plain VLM path; the drafter is idle that turn.
+whenever a drafter is available: a `--draft-gguf` assistant (gemma4,
+muse-glimmer) or a native `nextn` head in the LLM GGUF (qwen3.5/3.6, no
+companion needed). The verify walk only touches the language model, so a
+resident VLM gets the decode speedup on text turns, token-identical to the same
+model's text-only MTP. An image or audio request uses the plain VLM path; the
+drafter is idle that turn.
 
 ### Speculative / MTP
 
@@ -138,9 +139,9 @@ with a warning); `--no-speculative`/`--no-mtp` forces it off.
 
 | Flag | Meaning |
 |------|---------|
-| `--speculative` / `--mtp` | Force MTP speculative decoding on. Native-head models (qwen3.5/3.6 `nextn`) need no companion; gemma4 needs `--draft-gguf`. Native heads are auto-enabled without this. Use it to force the path when a sampler flag would otherwise defer. |
+| `--speculative` / `--mtp` | Force MTP speculative decoding on. Native-head models (qwen3.5/3.6 `nextn`) need no companion; gemma4 and muse-glimmer need `--draft-gguf`. Native heads are auto-enabled without this. Use it to force the path when a sampler flag would otherwise defer. |
 | `--no-speculative` / `--no-mtp` | Disable MTP. Overrides the native-head auto-enable and config `speculative: true`. |
-| `--draft-gguf PATH` | Separate assistant-drafter GGUF (gemma4 two-GGUF MTP shape, or a deepseek4 DSpark/MTP sidecar - gmlx `deepseek4-dspark`, llama.cpp `dflash`, or legacy `deepseek4_mtp_support`); implies `--speculative` (same as `serve`). A sidecar in the target's directory is autodetected without the flag. |
+| `--draft-gguf PATH` | Separate assistant-drafter GGUF (gemma4 two-GGUF MTP shape, a muse-glimmer DFlash drafter, or a deepseek4 DSpark/MTP sidecar - gmlx `deepseek4-dspark`, llama.cpp `dflash`, or legacy `deepseek4_mtp_support`); implies `--speculative` (same as `serve`). A sidecar in the target's directory is autodetected without the flag. |
 | `--draft-block-size N` | Override the MTP draft block size. |
 
 Speculative generation takes only `--temp`/`--top-p`/`--top-k`/`--min-p` plus a
@@ -351,28 +352,28 @@ every command. The terminal is upgraded on top:
   for now.
 - MTP speculative decoding (auto for native heads; `--no-mtp` to disable): a
   native-head model (qwen3.5/3.6 `nextn`) drafts and verifies multiple tokens
-  per step for a decode speedup; gemma4 needs a `--draft-gguf` assistant. The
-  reply streams the same way and ends with the same `tok/s` stat, and the
-  persistent KV cache is reused across turns exactly like the text path. Not
-  combinable with `--adapter` / `--stream-*`. Sampling is
+  per step for a decode speedup; gemma4 and muse-glimmer need a `--draft-gguf`
+  assistant. The reply streams the same way and ends with the same `tok/s`
+  stat, and the persistent KV cache is reused across turns exactly like the
+  text path. Not combinable with `--adapter` / `--stream-*`. Sampling is
   temperature/top-p/top-k/min-p only; the MTP verify walk has no penalty/bias
   hooks, so the other `/` sampling commands don't apply on this path.
   - VLM + MTP: a `--mmproj` VLM with a drafter (a `--draft-gguf` assistant
-    for gemma4, or a native `nextn` head for qwen3.5/3.6) keeps MTP on for
-    text-only turns (the fast path above) while `/image` / `/audio` turns
-    fall back to the plain VLM stream. The first media turn upgrades the
-    session to the VLM path for the rest of the conversation, since the text
-    tokenizer can't render a history that holds image markers. The prior text
-    turns are carried into that re-prefill so nothing is lost.
+    for gemma4 or muse-glimmer, or a native `nextn` head for qwen3.5/3.6)
+    keeps MTP on for text-only turns (the fast path above) while `/image` /
+    `/audio` turns fall back to the plain VLM stream. The first media turn
+    upgrades the session to the VLM path for the rest of the conversation,
+    since the text tokenizer can't render a history that holds image markers.
+    The prior text turns are carried into that re-prefill so nothing is lost.
 - Reasoning display: for thinking models (Qwen3/DeepSeek-R1/GLM `<think>`,
-  gpt-oss harmony channels, Gemma `<|channel>thought`), the chain-of-thought
-  is stripped of its control markers and streamed in the theme's thinking
-  style (italic bright blue under the default `dark` theme) inside a
-  gutter-framed block that closes with a payoff line showing how long the
-  model thought and how many tokens it spent; the final answer follows in
-  normal weight. `--reasoning hide` collapses the reasoning to a single live
-  spinner that resolves to the same payoff, so you see it working without
-  reading it. Ctrl-O toggles expand and collapse live during a reply (and
+  gpt-oss harmony channels, Gemma `<|channel>thought`, Muse Glimmer's ATEM
+  `to=self` channel), the chain-of-thought is stripped of its control markers
+  and streamed in the theme's thinking style (italic bright blue under the
+  default `dark` theme) inside a gutter-framed block that closes with a payoff
+  line showing how long the model thought and how many tokens it spent; the
+  final answer follows in normal weight. `--reasoning hide` collapses the
+  reasoning to a single live spinner that resolves to the same payoff, so you
+  see it working without reading it. Ctrl-O toggles expand and collapse live during a reply (and
   persists as the default for the next). `--reasoning raw` / `/reasoning raw`
   passes everything through verbatim (the old behavior, for when a model's
   markers segment oddly). The stored conversation keeps the raw text in every
@@ -541,7 +542,7 @@ gmlx serve Qwen3.6-27B-Q4_K_S.gguf --speculative
 | `--hf-source REPO` | Processor/config override for a single VLM model (rarely needed). |
 | `--speculative` | Serve a single positional model with MTP (native-head qwen3.5/3.6; gemma4 also needs `--draft-gguf`). |
 | `--draft-gguf PATH` | Companion drafter GGUF for assistant-shape MTP (gemma4); implies `--speculative`. |
-| `--draft-block-size N` | MTP draft tokens per round (analogous to llama-server `--spec-draft-n-max`). Default: the drafter's own block size. Also via `GMLX_DRAFT_BLOCK_SIZE`. |
+| `--draft-block-size N` | MTP draft tokens per round (analogous to llama-server `--spec-draft-n-max`). Default: the drafter's own block size; muse-glimmer defaults to 2 drafts and caps N at the loaded block (raise it with `GMLX_MUSE_DFLASH_BLOCK` at load). Also via `GMLX_DRAFT_BLOCK_SIZE`. |
 | `--adapter PATH` | GGUF LoRA adapter applied live over a single positional model at load (text only, not `--mmproj`/`--speculative`). In config mode set `adapter:` per model instead. |
 | `--stream-cpu` | Run a single positional model entirely on the CPU device: the over-RAM MoE path, same semantics as [`run --stream-cpu`](#loading). In config mode set `stream: cpu` per model instead; see [server-config.md](server-config.md#models). |
 | `--stream-experts` | Routed-expert stacks stream from disk while the every-token layers and KV cache stay on GPU; the decode feeder (default) serves decode from a wired expert arena and makes this the faster placement once warm. Config mode: `stream: experts`. Mutually exclusive with `--stream-cpu`. |

docs/performance.md

@@ -130,8 +130,9 @@ Models that ship a native multi-token-prediction head (Qwen3.5 and Qwen3.6) get
 speculative decoding automatically on `run` and `chat`: the head drafts tokens ahead
 and the base model verifies them. Output is exactly what the base model would
 have produced, just faster when drafts are accepted. `--no-mtp` turns it off.
-gemma-4 models take the two-file shape instead: a small companion drafter GGUF via
-`--draft-gguf`. On the server it is the `speculative:` config key.
+gemma-4 and Muse Glimmer take the two-file shape instead: a small companion
+drafter GGUF via `--draft-gguf`. On the server it is the `speculative:` config
+key.
 
 Gains depend on acceptance rate and context depth. In our serve benchmarks (M5
 Max, the same server with MTP off as the baseline), speculation roughly

docs/server-config.md

@@ -379,6 +379,7 @@ model fully resolved). Values are cited to the primary model cards in
 | `hunyuan` | `hunyuan-moe` | temperature=0.7 top_p=0.8 top_k=20 repetition_penalty=1.05 | - |
 | `hy3` | `hy_v3` | temperature=0.9 thinking_start_token=<think:opensource> thinking_end_token=</think:opensource> | `@reasoning-high`: temperature=0.9 thinking_start_token=<think:opensource> thinking_end_token=</think:opensource> reasoning_effort=high; `@reasoning-low`: temperature=0.9 thinking_start_token=<think:opensource> thinking_end_token=</think:opensource> reasoning_effort=low |
 | `kimi` | `kimi-k3` | temperature=1.0 top_p=0.95 thinking_start_token=<|open|>think<|sep|> thinking_end_token=<|close|>think<|sep|> | `@reasoning-high`: temperature=1.0 top_p=0.95 thinking_start_token=<|open|>think<|sep|> thinking_end_token=<|close|>think<|sep|> thinking_effort=high; `@reasoning-low`: temperature=1.0 top_p=0.95 thinking_start_token=<|open|>think<|sep|> thinking_end_token=<|close|>think<|sep|> thinking_effort=low; `@reasoning-max`: temperature=1.0 top_p=0.95 thinking_start_token=<|open|>think<|sep|> thinking_end_token=<|close|>think<|sep|> thinking_effort=max |
+| `muse` | `muse-glimmer` | temperature=1.0 top_p=0.95 top_k=64 thinking_start_token=<|start|>assistant to=self<|message|> thinking_end_token=<|eom|> | `@reasoning-high`: temperature=1.0 top_p=0.95 top_k=64 thinking_start_token=<|start|>assistant to=self<|message|> thinking_end_token=<|eom|> reasoning_strength=high; `@reasoning-low`: temperature=1.0 top_p=0.95 top_k=64 thinking_start_token=<|start|>assistant to=self<|message|> thinking_end_token=<|eom|> reasoning_strength=low; `@reasoning-medium`: temperature=1.0 top_p=0.95 top_k=64 thinking_start_token=<|start|>assistant to=self<|message|> thinking_end_token=<|eom|> reasoning_strength=medium; `@reasoning-xhigh`: temperature=1.0 top_p=0.95 top_k=64 thinking_start_token=<|start|>assistant to=self<|message|> thinking_end_token=<|eom|> reasoning_strength=xhigh |
 | `llama` | `llama`, `smollm3` | temperature=0.6 top_p=0.9 | - |
 | `mistral` | `mistral3` | temperature=0.15 | - |
 | `default` | *(anything else)* | temperature=0.7 top_p=0.95 | `@coding`: temperature=0.3 top_p=0.95; `@creative`: temperature=1.0 top_p=0.95 min_p=0.05; `@instruct`: temperature=0.7 top_p=0.95 |
@@ -406,6 +407,11 @@ Notes on individual families:
   (`<|open|>think<|sep|>` / `<|close|>think<|sep|>`) are set as the family's
   thinking tokens so open-think detection, thinking budgets, and the stream
   splitter track the model's real section tags.
+- muse: the `@reasoning-*` intents set `reasoning_strength`, the Muse Glimmer
+  template's variable name. It takes `low`/`medium`/`high`/`xhigh` and defaults
+  to `high`. Reasoning is a message channel rather than a tag pair, so the
+  thinking markers are the channel's own delimiters
+  (`<|start|>assistant to=self<|message|>` / `<|eom|>`).
 - qwen3.6 / qwen3: `@instruct` also sets `enable_thinking: false` (the card's
   non-thinking operating point).
 - `default`: the fallback for unknown architectures, the historic scaffold