From b0d97ed0334bc25804e0de1cbea18ec6968f19a8 Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 21:43:22 -0700 Subject: [PATCH 01/19] =?UTF-8?q?docs(readme):=20restructure=20=E2=80=94?= =?UTF-8?q?=20quickstart=20first,=20dedupe=20performance=20claims,=20slim?= =?UTF-8?q?=20arch=20section?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 324 ++++++++++++++++++++++-------------------------------- 1 file changed, 134 insertions(+), 190 deletions(-) diff --git a/README.md b/README.md index e33e7c5..e752b42 100644 --- a/README.md +++ b/README.md @@ -5,86 +5,29 @@ **The fastest way to run GGUF models on Apple Silicon.** -gmlx is a local inference platform for Apple Silicon: chat with an open -model, serve it over OpenAI- and Anthropic-compatible APIs, talk to it by -voice, and fine-tune it. One command, entirely on your Mac. If GGUF files, -quants, and the KV cache are unfamiliar terms, the -[getting-started guide](https://github.com/asher/gmlx/blob/main/docs/getting-started.md) -introduces them alongside the first install, and its -[glossary](https://github.com/asher/gmlx/blob/main/docs/getting-started.md#glossary) -defines the vocabulary used throughout these docs. - -gmlx takes the highest-quality quants available and serves them with the best -performance on Apple Silicon. Today that means the community's K-quant and -IQ-quant builds, which, size for size, keep more of the original model's -accuracy than any other open format, and markedly more than MLX's built-in -affine quantization ([accuracy per byte](#accuracy-per-byte)). Nearly every -open-weight release gets these builds within days, published as GGUF files, -and gmlx runs them exactly as published: nothing converted, nothing -re-quantized, none of that accuracy given back. The companion project +gmlx is a local inference platform: chat with an open model in the terminal +or your browser, serve it over OpenAI- and Anthropic-compatible APIs, +connect your coding agent to it, talk to it by voice, build a local RAG +stack on it, and fine-tune it with LoRA. One command, entirely on your Mac. + +It runs the community's K-quant and IQ-quant GGUF builds — size for size, +the most accurate open quant formats there are — exactly as published: +nothing converted, nothing re-quantized, none of that accuracy given back +([accuracy per byte](#accuracy-per-byte)). The companion project [mlx-kquant](https://github.com/asher/mlx-kquant) supplies the Metal kernels that run these formats natively on Apple's -[MLX](https://github.com/ml-explore/mlx) framework. - -On the same file, gmlx benchmarks faster than llama.cpp: prefill leads on -every model at every depth we've measured, reaching 2-4x past 100k tokens of -context, and with speculative decoding active on both engines, decode runs -1.1-2x ahead, widening with depth -([performance guide](https://github.com/asher/gmlx/blob/main/docs/performance.md)). Coverage runs from Llama, Qwen, -and Gemma through DeepSeek, GLM, and gpt-oss. A family is listed in the -generated [coverage matrix](https://github.com/asher/gmlx/blob/main/docs/arch-coverage.md) only after token-parity -certification against llama.cpp at 16k context. A 100B+ model starts -generating within seconds of launch, and a MoE bigger than RAM still runs, -streaming its experts from disk while everything every token needs stays on -the GPU ([bigger than memory](#bigger-than-memory)). +[MLX](https://github.com/ml-explore/mlx) framework, and on the same file +gmlx benchmarks faster than llama.cpp, with the gap widest at the +50-200k-token contexts where coding agents and long sessions live +([performance](#performance)). A 100B+ model starts generating within +seconds of launch, and a MoE bigger than RAM still runs, streaming its +experts from disk ([bigger than memory](#bigger-than-memory)). gmlx vs llama.cpp: fleet throughput speedup vs KV depth -| Prefill | Decode (speculation on both engines) | MTP lift at depth | -|---|---|---| -| faster on every model at every depth; 2-4x past 100k tokens | 1.1-2x ahead, widening with depth | 1.4-1.8x held from 17k to 110k | - - - - MTP decode lift vs KV depth: gmlx holds its speedup where llama.cpp's decays - - -The depth curve is the point: coding harnesses and agent sessions live at -50-200k tokens of context, and that is where the gap is widest. Full data and -methodology: -[benchmarks.md](https://github.com/asher/gmlx/blob/main/docs/benchmarks.md); -what makes it fast: the -[performance guide](https://github.com/asher/gmlx/blob/main/docs/performance.md). - -The rest of the platform: - -- hands-free voice chat with a built-in assistant that calls MCP tools - mid-turn and keeps long-term memory ([voice chat](#voice-chat)) -- a continuously batched server speaking OpenAI Chat Completions, OpenAI - Responses, and Anthropic Messages on one port, with tool calling, structured - output, logprobs, and vision ([serve an API](#serve-an-api)) -- MTP speculative decoding that holds its speedup deep into long context - ([performance](#performance)) -- run MoE models bigger than RAM, experts streamed from disk with decode - served from a popularity-managed GPU arena - ([bigger than memory](#bigger-than-memory)) -- chat in your browser: one command wires the Open WebUI app to your server - ([getting started](https://github.com/asher/gmlx/blob/main/docs/getting-started.md#chat-in-your-browser)) -- one-command hookups for coding agents and chat clients: pi, opencode, - hermes, Open WebUI, and more - ([connect clients](#connect-coding-agents-and-chat-apps)) -- embeddings, reranking, speech-to-text, and text-to-speech, a fully local RAG - and voice stack ([embeddings and speech](#embeddings-reranking-and-speech)) -- LoRA training directly on the quantized weights, with llama.cpp adapter - interop in both directions ([fine-tune](#fine-tune-with-lora)) -- a macOS menu-bar app to watch and manage the server, with `gmlx service - install` to keep everything running from login ([launch guide](https://github.com/asher/gmlx/blob/main/docs/launch.md)) - -The command is `gmlx`. - @@ -116,80 +59,23 @@ gmlx stop # the server ran detached chat TUI, voice chat, and the MCP assistant. The core alone already carries serving, vision, embeddings, and the menu bar, so `gmlx[chat]` is a smaller install that gives up only voice and the assistant; `gmlx init` adds either -one later. The extras table is in the -[getting-started guide](https://github.com/asher/gmlx/blob/main/docs/getting-started.md). -`uv tool` and pipx put `gmlx` on your PATH in every terminal; `pip install -"gmlx[all]"` into a venv you manage works too, with the command available -only while that venv is active. +one later. `uv tool` and pipx put `gmlx` on your PATH in every terminal; +`pip install "gmlx[all]"` into a venv you manage works too, with the command +available only while that venv is active. A model typically needs roughly its file size in memory, plus the KV cache; the exception is MoE models, which can run -[bigger than memory](#bigger-than-memory). If anything -misbehaves, `gmlx doctor` checks the runtime, config, model paths, and services -in one pass. The full walkthrough, from install to a configured server with a -connected client, is the [getting-started guide](https://github.com/asher/gmlx/blob/main/docs/getting-started.md). - -## Performance - -gmlx and llama.cpp run the same GGUF file, so the comparison is direct: on -an M5 Max (128 GB), gmlx prefills faster on every model in our fleet at -every depth measured, 1.2-1.6x at short context and 2-4x past 100k tokens. -At matched non-speculative baselines, decode starts at parity and wins -fleet-wide from 16k tokens as the KV cache deepens; with speculative -decoding active on both engines, gmlx decodes 1.1-2x ahead at every depth -measured. The speed comes from kernels -built for exactly this work: mlx-kquant's fused K-quant and IQ matmuls, -attention tuned for decode at depth, and a custom MTP verify path built for -this server. Where llama.cpp still leads, the -[performance guide](https://github.com/asher/gmlx/blob/main/docs/performance.md) says so. Reference points on the same -machine at short context, with llama.cpp on the same file alongside: -gemma-4-12B-it (dense, Q6_K) decodes at ~72 tok/s with MTP vs llama.cpp's ~54 -with speculation (1.3x), prefilling at ~850 vs ~730 tok/s; Qwen3.5-9B (dense, -Q6_K) decodes at ~112 vs ~76 tok/s (1.5x), prefilling at ~1600 vs ~1140 -tok/s; the full fleet tables are -in [benchmarks.md](https://github.com/asher/gmlx/blob/main/docs/benchmarks.md). Absolute numbers scale with the machine's memory -bandwidth; measure your own with `gmlx run model.gguf --bench "128,512,2048"`. - -When you want more, the levers are: MTP speculative decoding, which roughly -doubles decode throughput at short context and still delivers 1.4-1.8x from -17k through 110k over the same server with MTP off, automatic on models with a -native draft head (Qwen3.5/3.6) and available to gemma-4 through a small -companion drafter (1.9-2.1x at short context); the prompt cache, which -removes repeated prefill for -agent workloads; KV-cache quantization for long contexts; and disk-streamed -execution for MoE models larger than memory ([below](#bigger-than-memory)). -The file you pick -matters too: a uniform K-quant decodes meaningfully faster than a heavily -mixed one at similar or better quality. When and why, with numbers: the -[performance guide](https://github.com/asher/gmlx/blob/main/docs/performance.md). - -### Bigger than memory - -MoE models whose files exceed RAM still run. `--stream-experts` keeps -attention, the routers, and the KV cache on the GPU and streams the experts -from disk, serving decode from a wired, popularity-managed expert arena sized -to the machine and reading only the misses from the GGUF at SSD queue depth; -`--stream-cpu` instead streams the whole model from disk through the page -cache, running everything on the CPU. Both placements stage prefill straight from -the file into GPU-visible slots, one trip per byte. The arena is also a good -citizen: under system memory pressure it shrinks, keeping its most popular -experts, and regrows once pressure clears, so a long-running model coexists -with a build or a second model. Expect single-digit decode on what the SSD can -deliver: this is a capacity feature that makes a 200B-class MoE usable on a -64 GB machine, not a speed feature. Placements, feeder mechanics, and measured -numbers: the -[streaming guide](https://github.com/asher/gmlx/blob/main/docs/streaming.md). - -### Accuracy per byte - -K-quants are not just fast here; they are more accurate per byte than MLX's -native (affine) quantization, carrying roughly half the KL divergence at the -same bitrate (1.8-2.8x across the models measured). Qwen3.6-27B at a 4-bit -budget: KLD 0.0577 at 4.69 bpw for MLX affine vs 0.0208 at 4.88 bpw for -Q4_K_M, a 2.8x cut. The full table and methodology are in -[mlx-kquant's README](https://github.com/asher/mlx-kquant#why). Converting a -GGUF to MLX-native quantization gives up that margin; running it directly -keeps it. +[bigger than memory](#bigger-than-memory). If anything misbehaves, +`gmlx doctor` checks the runtime, config, model paths, and services in one +pass. + +The [getting-started guide](https://github.com/asher/gmlx/blob/main/docs/getting-started.md) +is the full walkthrough, from install to a configured server with a +connected client, including model picks per machine size and the extras +table. New to GGUF files, quants, or the KV cache? It introduces them as you +go, and its +[glossary](https://github.com/asher/gmlx/blob/main/docs/getting-started.md#glossary) +defines the vocabulary used throughout these docs. ## What you get @@ -253,9 +139,11 @@ config without touching your dotfiles and auto-starting the server if it is down gmlx launch pi --model qwen3.6-27b-q6@coding ``` -Supported: pi, opencode, omp, claude-code, hermes, goose, the aichat and elia chat -clients, and the Open WebUI browser app. A macOS menu-bar app shows what is -resident and offers unload, restart, and logs. +Supported: pi, opencode, omp, claude-code, hermes, goose, the aichat and elia +chat clients, and the Open WebUI browser app — `gmlx launch open-webui` is +chat in your browser on your own server. A macOS menu-bar app shows what is +resident and offers unload, restart, and logs, and `gmlx service install` +keeps the server running from login. Details: the [launch guide](https://github.com/asher/gmlx/blob/main/docs/launch.md). @@ -296,54 +184,106 @@ gmlx run model.gguf --adapter my-lora.gguf --prompt "..." Details: the [LoRA training guide](https://github.com/asher/gmlx/blob/main/docs/lora.md). +## Performance + +gmlx and llama.cpp run the same GGUF file, so the comparison is direct: on an +M5 Max (128 GB), gmlx prefills faster on every model in our fleet at every +depth measured, 1.2-1.6x at short context and 2-4x past 100k tokens. At +matched non-speculative baselines, decode starts at parity and wins +fleet-wide from 16k tokens as the KV cache deepens; with speculative decoding +active on both engines, gmlx decodes 1.1-2x ahead at every depth measured, +and MTP's lift over the same server with it off holds at 1.4-1.8x from 17k +through 110k, where llama.cpp's speculation gain decays with depth. +The depth curve is the point: coding harnesses and agent sessions live at +50-200k tokens of context, and that is where the gap is widest. + +| Prefill | Decode (speculation on both engines) | MTP lift at depth | +|---|---|---| +| faster on every model at every depth; 2-4x past 100k tokens | 1.1-2x ahead, widening with depth | 1.4-1.8x held from 17k to 110k | + + + + MTP decode lift vs KV depth: gmlx holds its speedup where llama.cpp's decays + + +Reference points on the same machine at short context, llama.cpp on the same +file alongside: gemma-4-12B-it (dense, Q6_K) decodes at ~72 tok/s with MTP vs +llama.cpp's ~54 with speculation (1.3x), prefilling at ~850 vs ~730 tok/s; +Qwen3.5-9B (dense, Q6_K) decodes at ~112 vs ~76 tok/s (1.5x), prefilling at +~1600 vs ~1140 tok/s. Absolute numbers scale with the machine's memory +bandwidth; measure your own with `gmlx run model.gguf --bench "128,512,2048"`. + +The speed comes from kernels built for exactly this work: mlx-kquant's fused +K-quant and IQ matmuls, attention tuned for decode at depth, and a custom MTP +verify path built for this server. Where llama.cpp still leads, the +[performance guide](https://github.com/asher/gmlx/blob/main/docs/performance.md) +says so; the full fleet tables, per-model charts, and methodology are in +[benchmarks.md](https://github.com/asher/gmlx/blob/main/docs/benchmarks.md). + +When you want more, the levers are: MTP speculative decoding, which roughly +doubles decode throughput at short context, automatic on models with a native +draft head (Qwen3.5/3.6) and available to gemma-4 through a small companion +drafter (1.9-2.1x at short context); the prompt cache, which removes repeated +prefill for agent workloads; KV-cache quantization for long contexts; and +disk-streamed execution for MoE models larger than memory +([below](#bigger-than-memory)). The file you pick matters too: a uniform +K-quant decodes meaningfully faster than a heavily mixed one at similar or +better quality. When and why, with numbers: the +[performance guide](https://github.com/asher/gmlx/blob/main/docs/performance.md). + +### Bigger than memory + +MoE models whose files exceed RAM still run. `--stream-experts` keeps +attention, the routers, and the KV cache on the GPU and streams the experts +from disk, serving decode from a wired, popularity-managed expert arena sized +to the machine and reading only the misses from the GGUF at SSD queue depth; +`--stream-cpu` instead streams the whole model from disk through the page +cache, running everything on the CPU. Both placements stage prefill straight from +the file into GPU-visible slots, one trip per byte. The arena is also a good +citizen: under system memory pressure it shrinks, keeping its most popular +experts, and regrows once pressure clears, so a long-running model coexists +with a build or a second model. Expect single-digit decode on what the SSD can +deliver: this is a capacity feature that makes a 200B-class MoE usable on a +64 GB machine, not a speed feature. Placements, feeder mechanics, and measured +numbers: the +[streaming guide](https://github.com/asher/gmlx/blob/main/docs/streaming.md). + +### Accuracy per byte + +K-quants are not just fast here; they are more accurate per byte than MLX's +native (affine) quantization, carrying roughly half the KL divergence at the +same bitrate (1.8-2.8x across the models measured). Qwen3.6-27B at a 4-bit +budget: KLD 0.0577 at 4.69 bpw for MLX affine vs 0.0208 at 4.88 bpw for +Q4_K_M, a 2.8x cut. The full table and methodology are in +[mlx-kquant's README](https://github.com/asher/mlx-kquant#why). Converting a +GGUF to MLX-native quantization gives up that margin; running it directly +keeps it. + ## Supported architectures -Every family below loads end to end. New architectures land regularly, so -the authoritative list is the generated -[architecture coverage matrix](https://github.com/asher/gmlx/blob/main/docs/arch-coverage.md), with per-arch parity -notes. +Coverage runs across the major open-weight families: Llama and Mistral; +Qwen 2 through 3.6, dense and MoE, including the gated-DeltaNet hybrids and +Qwen3-Next; Gemma 1 through 4 plus DiffusionGemma; DeepSeek V3/R1 and +V4-Flash; GLM 4 through 5.2; gpt-oss; Kimi-K3; MiniMax M2 and M3; Hunyuan +A13B and Hy3; Granite, including the 4.x hybrids; Nemotron-H; Falcon-H1; +ERNIE-4.5; Phi-3; Seed-OSS; and SmolLM3. New architectures land regularly, +and a family is listed in the generated +[architecture coverage matrix](https://github.com/asher/gmlx/blob/main/docs/arch-coverage.md) +only after token-parity certification against llama.cpp at 16k context; the +matrix carries per-arch notes, including the rare exceptions (DiffusionGemma +has no llama.cpp oracle and is validated by output coherence; gemma-3n is +code-complete but gate-disabled until a correctly converted GGUF exists). A GGUF is loadable when its `general.architecture` is an architecture gmlx -recognizes and can synthesize a config for, or you supply `hf_source`. Preflight runs the architecture gate and checks each tensor's -codec (its GGUF quantization type, like `Q4_K_M`) before any tensor bytes are read. - -| Family | GGUF archs | Notes | -|--------|------------|-------| -| Llama | `llama`, `mistral3` | Llama-2/3, Mistral, Vicuna. Mixtral routes to `mixtral` (stacked experts) | -| Qwen | `qwen2`, `qwen2moe`, `qwen3`, `qwen35`, `qwen3moe`, `qwen35moe`, `qwen3vlmoe`, `qwen3next` | dense + MoE. Qwen3.5/3.6 and Qwen3-Next gated-DeltaNet hybrids (both GGUF layouts); `qwen3vlmoe` is the Qwen3-Omni thinker tower | -| Gemma | `gemma`, `gemma2`, `gemma3`, `gemma3n` [1], `gemma4` | norm bake undone on load; tied embeddings; gemma-3n text tower (AltUp, LAuReL, MatFormer) | -| DiffusionGemma | `diffusion-gemma` [2] | early support: non-autoregressive block-diffusion denoiser on the gemma-4 MoE backbone | -| GLM | `glm4`, `glm4moe`, `glm-dsa` | GLM-4 dense; GLM-4.5/4.6 fine-grained MoE; GLM-5.2 MLA + sparse-attention indexer | -| DeepSeek | `deepseek2`, `deepseek4` | MLA attention + fine-grained MoE (DeepSeek-V3/R1, GLM-4.x MLA conversions); V4-Flash MLA-lite + sparse indexer + MTP drafter (parity reference: the ds4 engine) | -| Phi | `phi3` | mini / small / medium; `hf_source` for 128K long-context variants | -| Nemotron | `nemotron_h_moe` | hybrid Mamba + attention MoE | -| gpt-oss | `gpt-oss` | native MXFP4 experts, attention sinks, YaRN rope, alternating sliding/full attention | -| Seed | `seed_oss` | ByteDance Seed-OSS 36B dense | -| SmolLM | `smollm3` | SmolLM3-3B; Llama backbone with NoPE every 4th layer | -| Granite | `granite`, `granitehybrid` | IBM Granite dense; Granite 4.x hybrid (Mamba2 + attention + MoE) | -| ERNIE | `ernie4_5-moe` | Baidu ERNIE-4.5-MoE (21B-A3B) | -| MiniMax | `minimax-m2`, `minimax-m3` | MiniMax-M2 (230B-A10B) and M3 (428B-A23B); every-layer fine-grained MoE. M3 adds per-head qk-norm + a shared expert; its sparse attention runs dense (exact to 2048 tokens) | -| Hunyuan | `hunyuan-moe`, `hy_v3` | Tencent Hunyuan-A13B; Hy3 (299B-A21B) with its native MTP head | -| Falcon | `falcon-h1` | TII Falcon-H1 (0.5B-34B); parallel attention + Mamba2 in every layer | - -[1] `gemma3n` is implemented but gate-disabled: every published gemma-3n GGUF -carries mis-converted LAuReL weights (an upstream llama.cpp conversion bug at all -quant levels), so the loader refuses them rather than run a degraded model. It -re-enables the moment a correctly converted file exists. - -[2] `diffusion-gemma` is early support and the one architecture with no -token-parity oracle at all (llama.cpp does not implement it). It is -GPU-validated by output coherence instead, currently Q8_0 only, and loads, runs, -chats, and serves through mlx-vlm's denoising engine. - -One further mapped arch, `gemma-embedding`, backs the `/v1/embeddings` endpoint -(EmbeddingGemma from GGUF) rather than chat. Codec coverage spans all 19 -K-quant, legacy, and IQ codecs plus the native-fp pair mxfp4/nvfp4 (run -directly from the file when the model is bigger than RAM, repacked for speed -when it fits - `GMLX_NATIVE_FP` overrides the choice); in -the rare case a file uses a type with no kernel (ternary TQ, for example), -preflight names it and lists what is supported so you can pick another -variant. +recognizes and can synthesize a config for, or you supply `hf_source`. +Preflight runs the architecture gate and checks each tensor's codec (its +GGUF quantization type, like `Q4_K_M`) before any tensor bytes are read. +Codec coverage spans all 19 K-quant, legacy, and IQ codecs plus the +native-fp pair mxfp4/nvfp4 (run directly from the file when the model is +bigger than RAM, repacked for speed when it fits - `GMLX_NATIVE_FP` +overrides the choice); in the rare case a file uses a type with no kernel +(ternary TQ, for example), preflight names it and lists what is supported so +you can pick another variant. Vision-language models load as a K-quant LLM GGUF paired with its float `mmproj` GGUF: supported families and caveats in the [VLM guide](https://github.com/asher/gmlx/blob/main/docs/vlm.md). Want a family @@ -428,6 +368,10 @@ Guides: Open WebUI). - [docs/performance.md](https://github.com/asher/gmlx/blob/main/docs/performance.md): what makes it fast, what the levers cost, and how to measure. +- [docs/streaming.md](https://github.com/asher/gmlx/blob/main/docs/streaming.md): MoE models bigger than RAM, placements + and feeder mechanics. +- [docs/benchmarks.md](https://github.com/asher/gmlx/blob/main/docs/benchmarks.md): the generated fleet scorecard behind + the performance claims. - [docs/troubleshooting.md](https://github.com/asher/gmlx/blob/main/docs/troubleshooting.md): the common failures and their fixes. - [docs/migrating.md](https://github.com/asher/gmlx/blob/main/docs/migrating.md): coming from llama.cpp, Ollama, or From 80d0699e2809ae80e8409b664cf800147adbeecf Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 21:50:47 -0700 Subject: [PATCH 02/19] =?UTF-8?q?docs(readme):=20cut=20restated=20claims?= =?UTF-8?q?=20=E2=80=94=20say=20each=20thing=20once?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 56 ++++++++++++++++++++++--------------------------------- 1 file changed, 22 insertions(+), 34 deletions(-) diff --git a/README.md b/README.md index e752b42..86b50ea 100644 --- a/README.md +++ b/README.md @@ -11,8 +11,7 @@ connect your coding agent to it, talk to it by voice, build a local RAG stack on it, and fine-tune it with LoRA. One command, entirely on your Mac. It runs the community's K-quant and IQ-quant GGUF builds — size for size, -the most accurate open quant formats there are — exactly as published: -nothing converted, nothing re-quantized, none of that accuracy given back +the most accurate open quant formats there are — exactly as published ([accuracy per byte](#accuracy-per-byte)). The companion project [mlx-kquant](https://github.com/asher/mlx-kquant) supplies the Metal kernels that run these formats natively on Apple's @@ -140,9 +139,8 @@ gmlx launch pi --model qwen3.6-27b-q6@coding ``` Supported: pi, opencode, omp, claude-code, hermes, goose, the aichat and elia -chat clients, and the Open WebUI browser app — `gmlx launch open-webui` is -chat in your browser on your own server. A macOS menu-bar app shows what is -resident and offers unload, restart, and logs, and `gmlx service install` +chat clients, and the Open WebUI browser app. A macOS menu-bar app shows what +is resident and offers unload, restart, and logs, and `gmlx service install` keeps the server running from login. Details: the [launch guide](https://github.com/asher/gmlx/blob/main/docs/launch.md). @@ -191,44 +189,37 @@ M5 Max (128 GB), gmlx prefills faster on every model in our fleet at every depth measured, 1.2-1.6x at short context and 2-4x past 100k tokens. At matched non-speculative baselines, decode starts at parity and wins fleet-wide from 16k tokens as the KV cache deepens; with speculative decoding -active on both engines, gmlx decodes 1.1-2x ahead at every depth measured, -and MTP's lift over the same server with it off holds at 1.4-1.8x from 17k -through 110k, where llama.cpp's speculation gain decays with depth. -The depth curve is the point: coding harnesses and agent sessions live at -50-200k tokens of context, and that is where the gap is widest. - -| Prefill | Decode (speculation on both engines) | MTP lift at depth | -|---|---|---| -| faster on every model at every depth; 2-4x past 100k tokens | 1.1-2x ahead, widening with depth | 1.4-1.8x held from 17k to 110k | +active on both engines, gmlx decodes 1.1-2x ahead throughout, and MTP's lift +over the same server with it off holds at 1.4-1.8x from 17k through 110k, +where llama.cpp's speculation gain decays with depth. MTP decode lift vs KV depth: gmlx holds its speedup where llama.cpp's decays -Reference points on the same machine at short context, llama.cpp on the same -file alongside: gemma-4-12B-it (dense, Q6_K) decodes at ~72 tok/s with MTP vs -llama.cpp's ~54 with speculation (1.3x), prefilling at ~850 vs ~730 tok/s; +Reference points at short context: gemma-4-12B-it (dense, Q6_K) decodes at +~72 tok/s with MTP vs llama.cpp's ~54 with speculation (1.3x), prefilling at +~850 vs ~730 tok/s; Qwen3.5-9B (dense, Q6_K) decodes at ~112 vs ~76 tok/s (1.5x), prefilling at ~1600 vs ~1140 tok/s. Absolute numbers scale with the machine's memory bandwidth; measure your own with `gmlx run model.gguf --bench "128,512,2048"`. The speed comes from kernels built for exactly this work: mlx-kquant's fused K-quant and IQ matmuls, attention tuned for decode at depth, and a custom MTP -verify path built for this server. Where llama.cpp still leads, the -[performance guide](https://github.com/asher/gmlx/blob/main/docs/performance.md) -says so; the full fleet tables, per-model charts, and methodology are in +verify path built for this server. The full fleet tables, per-model charts, +and methodology are in [benchmarks.md](https://github.com/asher/gmlx/blob/main/docs/benchmarks.md). When you want more, the levers are: MTP speculative decoding, which roughly doubles decode throughput at short context, automatic on models with a native draft head (Qwen3.5/3.6) and available to gemma-4 through a small companion -drafter (1.9-2.1x at short context); the prompt cache, which removes repeated -prefill for agent workloads; KV-cache quantization for long contexts; and -disk-streamed execution for MoE models larger than memory -([below](#bigger-than-memory)). The file you pick matters too: a uniform -K-quant decodes meaningfully faster than a heavily mixed one at similar or -better quality. When and why, with numbers: the +drafter (1.9-2.1x); the prompt cache, which removes repeated prefill for +agent workloads; KV-cache quantization for long contexts; and disk-streamed +execution for MoE models larger than memory ([below](#bigger-than-memory)). +The file you pick matters too: a uniform K-quant decodes meaningfully faster +than a heavily mixed one at similar or better quality. When each lever pays +off, with numbers — and where llama.cpp still leads — is in the [performance guide](https://github.com/asher/gmlx/blob/main/docs/performance.md). ### Bigger than memory @@ -251,8 +242,8 @@ numbers: the ### Accuracy per byte K-quants are not just fast here; they are more accurate per byte than MLX's -native (affine) quantization, carrying roughly half the KL divergence at the -same bitrate (1.8-2.8x across the models measured). Qwen3.6-27B at a 4-bit +native (affine) quantization, carrying 1.8-2.8x less KL divergence at the +same bitrate across the models measured. Qwen3.6-27B at a 4-bit budget: KLD 0.0577 at 4.69 bpw for MLX affine vs 0.0208 at 4.88 bpw for Q4_K_M, a 2.8x cut. The full table and methodology are in [mlx-kquant's README](https://github.com/asher/mlx-kquant#why). Converting a @@ -287,8 +278,8 @@ you can pick another variant. Vision-language models load as a K-quant LLM GGUF paired with its float `mmproj` GGUF: supported families and caveats in the [VLM guide](https://github.com/asher/gmlx/blob/main/docs/vlm.md). Want a family -that is missing? What it takes, and the acceptance gate every family clears, is -in the [adding-architectures guide](https://github.com/asher/gmlx/blob/main/docs/adding-architectures.md). +that is missing? What it takes is in the +[adding-architectures guide](https://github.com/asher/gmlx/blob/main/docs/adding-architectures.md). ## How it works @@ -323,10 +314,7 @@ flowchart TB serve --> aux["embeddings + rerank
STT + TTS"] ``` -The loader: preflight (shards, codecs, arch gate), mmap the wire bytes, remap GGUF -tensor names to mlx-lm parameter names, synthesize the config and tokenizer from the -GGUF metadata, assemble the model with quantized leaves swapped for `kq.*`-backed -modules, and generate with mlx-lm's sampler and KV cache. Serving-side mechanics: +Serving-side mechanics — engine, batching, and the HTTP layers: [docs/serving-architecture.md](https://github.com/asher/gmlx/blob/main/docs/serving-architecture.md). ## Python API From d4bfdd9cd504b5016c67d0bbb959c54113209448 Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 21:59:07 -0700 Subject: [PATCH 03/19] docs(readme): surface concurrent-request behavior in the serve section --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 86b50ea..8c2aea2 100644 --- a/README.md +++ b/README.md @@ -116,7 +116,11 @@ Details: [picking a model for your Mac](https://github.com/asher/gmlx/blob/main/ `serve` runs a continuously batched, multi-model server speaking three dialects on one port: OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages, all -streaming. It handles tool calling, structured output (`response_format: +streaming. Concurrent requests decode together in one batch, a new prompt's +prefill is paced so in-flight replies keep streaming, and a memory-headroom +gate admits new work only when it fits +([serving concurrent requests](https://github.com/asher/gmlx/blob/main/docs/performance.md#serving-concurrent-requests)). +It handles tool calling, structured output (`response_format: json_schema`, grammar-constrained), logprobs, and vision messages. A YAML config gives named models, reusable sampling profiles, aliases, and directory discovery. Residency is managed (LRU with pinning and idle unload), and repeated prefixes are From 207a36743ec8b5aed5138162006ed12c249d9369 Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 22:04:39 -0700 Subject: [PATCH 04/19] docs(getting-started): dedupe install facts, group extras under a subsection --- docs/getting-started.md | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index e912a6f..1241c51 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -33,7 +33,7 @@ brew install ffmpeg # voice and non-wav audio only That is the whole install. `gmlx` lands on your PATH in every terminal, in an isolated environment, with a suitable Python fetched for it, and `[all]` turns -on every optional feature so nothing else is needed later. +on every optional feature. [uv](https://docs.astral.sh/uv/) itself is one command (`brew install uv`, or `curl -LsSf https://astral.sh/uv/install.sh | sh`); `pipx install "gmlx[all]"` behaves the same way. Upgrade later with @@ -45,8 +45,7 @@ not using voice. To install less, name the extras you want instead of `all` - `gmlx[chat]` is the common choice, giving up only voice and the assistant. `gmlx init` offers -to add the rest later, and issues the correct command for however gmlx was -installed. +to add the rest later ([adding an extra](#the-extras) below). A plain venv you manage yourself works as well: @@ -63,10 +62,16 @@ no such step. The `mlx-kquant` dependency (the Metal kernels) arrives as a prebuilt wheel from PyPI on macOS 26 and newer; on older macOS versions -the install builds it from source, which needs the Xcode Command Line Tools -(`xcode-select --install`) and takes a few minutes. +the install builds it from source - the Command Line Tools requirement from +[What you need](#what-you-need), and a few minutes of compile time. -The extras, all optional: +Tab completion is worth the one line: add `eval "$(gmlx completion zsh)"` to +`~/.zshrc` (there are `bash` and `fish` variants). It completes verbs, flags, your +config's model ids, and the host and port of any running server. + +### The extras + +All optional: | Extra | Adds | Needed for | |-------|------|------------| @@ -93,12 +98,6 @@ grapheme-to-phoneme front-end the default Kokoro voice needs. That front-end also wants a spaCy English pipeline, which gmlx fetches from Hugging Face on first use - no separate install step. -Tab completion is worth the one line: add `eval "$(gmlx completion zsh)"` to -`~/.zshrc` (there are `bash` and `fish` variants). It completes verbs, flags, your -config's model ids, and the host and port of any running server. - -The installed command is `gmlx`. - ## First generation in two minutes Download a small model into the current directory and run it: @@ -172,8 +171,7 @@ with progress, rate, and resume built in. a repo it lists every quant variant as a ready-to-paste ref. K-quant, legacy, and IQ files all load; in the rare case a file uses a codec with no kernel (the ternary TQ types, for instance), the verdict names it so you can pick another -variant. Uniform K-quant files also decode -faster; see [performance.md](performance.md#choosing-a-quant-for-speed). +variant. Uniform K-quant files also decode faster than heavily mixed ones. Set `HF_TOKEN` for gated or private repos. If you already have a model library from LM Studio, it serves as-is (the files are plain GGUFs): From 7000e045d67e7ac3a2ac4746464fe21266b62065 Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 22:05:20 -0700 Subject: [PATCH 05/19] docs(index,migrating): list upstream-upgrades + benchmarks in index, dedupe until-EOS note --- docs/README.md | 5 +++++ docs/migrating.md | 5 ++--- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/README.md b/docs/README.md index 278bbc8..ceea470 100644 --- a/docs/README.md +++ b/docs/README.md @@ -21,6 +21,7 @@ right document for the task at hand. | Run a vision or audio-input model | [vlm.md](vlm.md) | | Fine-tune with LoRA on a quantized base | [lora.md](lora.md) | | Make it faster | [performance.md](performance.md) | +| See the numbers behind the performance claims | [benchmarks.md](benchmarks.md) | | Run a MoE model bigger than RAM | [streaming.md](streaming.md) | | Fix something that broke | run `gmlx doctor`, then [troubleshooting.md](troubleshooting.md) | | Look up a flag or config key | [cli.md](cli.md), [server-config.md](server-config.md) | @@ -118,5 +119,9 @@ as supported. [testing.md](testing.md) describes the test tiers, the end-to-end harnesses, and the manual voice-loop pass. +[upstream-upgrades.md](upstream-upgrades.md) is the maintainer's guide to +bumping the pinned mlx-vlm/mlx-lm/mlx versions: the seam contract, the canary +script, and the qualification steps. + [CONTRIBUTING.md](../CONTRIBUTING.md) has the development setup and expectations for pull requests; [CHANGELOG.md](../CHANGELOG.md) records what shipped when. diff --git a/docs/migrating.md b/docs/migrating.md index cb496f0..069d469 100644 --- a/docs/migrating.md +++ b/docs/migrating.md @@ -7,9 +7,8 @@ has a different name, and what works differently on purpose. ## Coming from llama.cpp / llama-server Your GGUFs work as-is: `gmlx run model.gguf` is the moral equivalent of -`llama-cli -m model.gguf`, and `gmlx serve model.gguf` of `llama-server`. -The default port is the same 8080, and generation runs until the model stops -by default, like `-n -1`. +`llama-cli -m model.gguf`, and `gmlx serve model.gguf` of `llama-server`; +the default port is the same 8080. Common flag equivalents: From 543145d4077e0bd3ded3e914dd84e12412112fbe Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 22:07:29 -0700 Subject: [PATCH 06/19] docs(launch,talk): dedupe per-client dotfile/auth restatements, defer assistant detail to assistant.md --- docs/launch.md | 18 ++++++++---------- docs/talk.md | 36 +++++++++++++++--------------------- 2 files changed, 23 insertions(+), 31 deletions(-) diff --git a/docs/launch.md b/docs/launch.md index f1886ac..389b700 100644 --- a/docs/launch.md +++ b/docs/launch.md @@ -86,9 +86,9 @@ launch carries it into each tool's configuration in that tool's native slot: |--------|--------------------| | opencode | `options.apiKey` in the injected config | | pi | `apiKey` in the merged provider block | -| omp | no API-key slot in its provider registry; launch prints a note to configure auth manually | +| omp | no API-key slot ([omp](#omp-oh-my-pi)) | | hermes | `providers.custom.api_key` in the injected config | -| goose | `OPENAI_API_KEY` in the exec environment only, never written to its `config.yaml` (which may hold a real OpenAI credential) | +| goose | `OPENAI_API_KEY` in the exec environment only ([goose](#goose)) | | claude-code | `ANTHROPIC_AUTH_TOKEN` in the exec environment | | aichat, elia | `api_key` in the injected config | | open-webui | `OPENAI_API_KEY` in the exec environment only | @@ -138,14 +138,12 @@ and prefer a model and machine with strong prefill throughput. ### opencode Injection style: launch writes `~/.config/gmlx/opencode.json` and points opencode -at it via `OPENCODE_CONFIG`. Your own opencode config is untouched. The default model -lands in the top-level `model` key. +at it via `OPENCODE_CONFIG`. The default model lands in the top-level `model` key. ### pi Merge style: launch merges the provider into `~/.pi/agent/models.json` and -`~/.pi/agent/settings.json`, setting `defaultProvider` and `defaultModel`. Other -providers in the files are preserved. +`~/.pi/agent/settings.json`, setting `defaultProvider` and `defaultModel`. ### omp (oh-my-pi) @@ -159,7 +157,7 @@ prints a note). NousResearch hermes-agent. Injection style: launch writes `~/.config/gmlx/hermes-config.yaml` (your `~/.hermes/config.yaml` merged with the gmlx provider block) and injects it via `HERMES_CONFIG` plus `CUSTOM_BASE_URL`. -Your own file is untouched. A model is required (`inference.model`). +A model is required (`inference.model`). hermes refuses models with less than 64k context at startup, so serve it a model whose context window is at least 64k tokens. The window comes from the model's GGUF @@ -179,8 +177,8 @@ OpenAI credential. sigoden/aichat, a chat-focused terminal REPL with tools and agents (not a coding harness). Injection style: launch writes a `config.yaml` with an `openai-compatible` -client under `~/.config/gmlx/aichat/` and injects it via `AICHAT_CONFIG_DIR`; -your own `~/.config/aichat` is untouched. Every served id is flagged +client under `~/.config/gmlx/aichat/` and injects it via +`AICHAT_CONFIG_DIR`. Every served id is flagged `supports_function_calling`, so aichat's tools and agents work against the server's tool-call surface. Actual tool execution still needs aichat's `llm-functions` installed. @@ -189,7 +187,7 @@ installed. darrenburns/elia, a chat TUI (not a coding harness). Injection style: launch writes a fresh `config.toml` under `~/.config/gmlx/elia-xdg` and injects it via -`XDG_CONFIG_HOME`; your own `~/.config/elia` is untouched. Each served id becomes an +`XDG_CONFIG_HOME`. Each served id becomes an OpenAI-compatible litellm model. Requires elia 1.x or newer (older builds ignore custom endpoints); upgrade with `pipx upgrade elia-chat`. diff --git a/docs/talk.md b/docs/talk.md index e5c3566..a120156 100644 --- a/docs/talk.md +++ b/docs/talk.md @@ -65,10 +65,9 @@ that the cat might actually learn it. listening for "hey assistant" ``` -Useful controls while it runs: Space stops the assistant mid-sentence (and is -push-to-talk in `ptt` mode), Esc cancels the current turn, `m` mutes the mic, `q` -quits. Start typing at any time to send a text message instead of speaking; lines -starting with `/` are commands. Try voices live: +Useful while it runs: Space stops the assistant mid-sentence, and typing at any +time sends a text message instead of speaking; the full set is in +[keys and slash commands](#keys-and-slash-commands). Try voices live: ```text /voice # list the server's voices @@ -133,9 +132,8 @@ assistant: enabled: true ``` -Tool calling needs a model that is competent at it. The Qwen3.6-27B class is a good -fit on a 48 GB or larger machine; Qwen3.5-9B is a workable floor on 32 GB, with more -tool fumbles. Then: +Tool calling needs a model that is competent at it; the config's Qwen3.6-27B is +the recommended class ([model picks](assistant.md#the-tool-loop)). Then: ```sh gmlx talk @@ -169,24 +167,20 @@ assistant: Ana's birthday is March 12th. ``` What got stored is a distilled fact ("sister Ana, birthday March 12"), not a -transcript. After each turn, a background request asks the chat model to boil the -exchange down to at most three durable facts, or none, so small talk leaves no -residue. A new fact that restates an existing one replaces it. Extraction runs off -the voice path and adds no latency. +transcript. The distillation runs in the background after each turn, off the +voice path, so it adds no latency; the rules (at most three durable facts, +restatement replaces) are in [assistant.md#memory](assistant.md#memory). -Two honest caveats. Each tool round adds seconds (a full model turn plus the tool -call), so multi-tool answers are noticeably slower than plain chat. And a barge-in +Two honest caveats. Tool rounds cost time - a model turn plus the call, each - +so multi-tool answers are noticeably slower than plain chat. And a barge-in still interrupts cleanly: the loop commits what you heard and never leaves a half-finished tool round in the history. -To inspect or edit memory from inside a session, `/memory` lists the stored -facts with their ids, `/memory forget ID` removes one, and `/memory clear yes` -removes them all. The menu-bar app's voice session exposes the same store: -"Show memory" prints the list into the transcript panel, and "Clear memory" -wipes it after a confirmation dialog. The store itself is a plain sqlite file -at `~/.local/share/gmlx/assistant-memory.db` if you want to look deeper. -It is shared with `gmlx chat --assistant`: a fact taught by voice recalls -in the text REPL, and vice versa. +`/memory` inspects and edits the store from inside a session +([keys and slash commands](#keys-and-slash-commands)); the menu-bar voice +session exposes the same store through its "Show memory" and "Clear memory" +items. The store is shared with `gmlx chat --assistant` - details and the +on-disk location in [assistant.md#memory](assistant.md#memory). ## Modes From 733e662c51532affe038b17e4fa6b894acaf6824 Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 22:07:50 -0700 Subject: [PATCH 07/19] docs(vlm): merge two-sentence restatement --- docs/vlm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/vlm.md b/docs/vlm.md index a2b22b7..51214f2 100644 --- a/docs/vlm.md +++ b/docs/vlm.md @@ -14,7 +14,7 @@ encoders run in float, and the image processor + chat template (including the per-family image/audio marker tokens) are synthesized from the two GGUFs' metadata. `--hf-source` overrides only when a file omits something. -Vision and audio support is included in the base install. No extra is needed. +Vision and audio support is included in the base install; no extra is needed. Where mmproj files come from: llama.cpp-style multimodal GGUF repos ship them as `mmproj-*.gguf` siblings of the LLM GGUF in the same Hugging Face repo. From 99c73b0d7463dc7a1bca977e36ea31556130f237 Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 22:09:44 -0700 Subject: [PATCH 08/19] docs(perf,streaming): reattach stranded decode-feeder measurement, dedupe intro and boilerplate, slim will-it-fit copy --- docs/getting-started.md | 15 +++++---------- docs/performance.md | 5 +++-- docs/streaming.md | 30 ++++++++++++++---------------- 3 files changed, 22 insertions(+), 28 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 1241c51..fc0cc15 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -188,18 +188,13 @@ bytes per token = 2 (K and V) x layers x kv_heads x head_dim x 2 (bf16) ``` An 8B-class model (32 layers, 8 KV heads, head dim 128) uses 128 KB per token of -context, so a 32k-token session adds 4 GB on top of the weights. A 32B-class dense -model (64 layers, same heads) uses 256 KB per token: 8 GB at 32k. The layer and head -counts are in the GGUF metadata, and the model card lists them too. +context, so a 32k-token session adds 4 GB on top of the weights. The layer and +head counts are in the GGUF metadata, and the model card lists them too. If weights plus cache crowd your RAM, quantize the cache: `--kv-bits 8` roughly -halves it at nearly no quality cost, and `--kv-bits 4` roughly quarters it with a -small cost at long range. In server configs the same knob is the `kv_bits` load key. - -Several families are much cheaper than the formula suggests. Sliding-window layers -(gemma) stop growing at the window size, hybrid linear-attention models (Qwen3.5 and -3.6, Falcon-H1, Granite 4.x) keep a small fixed state on most layers, and MLA models -(DeepSeek) store a compressed cache. Worked numbers: +halves it at nearly no quality cost. Several families (sliding-window, +hybrid linear-attention, MLA) are much cheaper than the formula suggests. +More worked numbers, the cheaper families, and the rest of the levers: [performance.md](performance.md#memory-and-the-kv-cache). ## Set up the server diff --git a/docs/performance.md b/docs/performance.md index b8170e5..0309fce 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -335,8 +335,8 @@ batch jobs that only care about aggregate throughput can set `0`. Two interactions to know. Pacing applies to speculative (MTP) serving too, and the two features divide the work: pacing decides how admissions share -GPU time, while the width cap decides which decode mode each batch runs in -(speculative while narrow, plain once it grows past the model's cap). And +GPU time, while the [width cap](#mtp-speculative-decoding) decides which +decode mode each batch runs in. And the prompt cache is the strongest admission lever of all: a warm prefix skips its prefill outright, leaving pacing to govern only the cold suffix. Agent sessions that resend a cached history and add a few thousand tokens @@ -455,6 +455,7 @@ benchmarks should pin it for reproducibility; a negative value (or env policy; `0` disables buffer caching entirely. A bounded cache trades a little allocator churn for a bounded footprint: transients up to the limit are recycled in place, larger ones fall through to fresh allocations. + ## Bigger than memory: MoE offload MoE models whose files exceed what the GPU can wire still run, on one of diff --git a/docs/streaming.md b/docs/streaming.md index dfc6a66..bc20826 100644 --- a/docs/streaming.md +++ b/docs/streaming.md @@ -17,11 +17,10 @@ This is a capacity feature that makes a 200B-class MoE usable on a over-budget case: a model that fits in memory runs several times faster on the normal GPU path. -The disk is the engine here: demand misses are served at the drive's -random-read latency and the feeders read at its queue depth, so disk IO -performance determines inference performance. Keep the GGUF on the -internal NVMe SSD for the best result; an external drive works, but -decode follows its latency and bandwidth down. +Demand misses are served at the drive's random-read latency and the +feeders read at its queue depth, so keep the GGUF on the internal NVMe +SSD for the best result; an external drive works, but decode follows +its latency and bandwidth down. Unless a different machine is named inline, measured numbers in this guide are from a 14-inch M5 Max MacBook Pro (128 GB); the hardware @@ -134,6 +133,12 @@ Streaming models engage two feeder paths by default: model, a build) it shrinks, keeping its most popular experts, and regrows once pressure clears - a long-running model stays a good citizen on a machine that is doing other work (`GMLX_DECODE_PRESSURE=0` pins it instead). + Same model and box as the prefill measurement above: decode went from 2.4 + tok/s on the page-cache path to 4.0 tok/s averaged over a 512-token + generation (~4.7 steady, ~90% arena hits), against 3.0 tok/s for + `--stream-cpu` - so `--stream-experts` now matches `--stream-cpu` on short + generations and pulls ahead roughly 1.5x once the arena warms, before the + KV-cache advantage at depth. - The arena also serves multi-token expert calls whose routed union exceeds its slots - the next chat turn's prefill after a decode, or a wide speculative verify batch - by halving the chunk along the token axis @@ -168,12 +173,6 @@ hit rate. Measured on Kimi-K3 UD-IQ2_XXS (662 GB, 62 GB every-token set, M5 Max 128 GB): decode 0.10 -> 0.38 tok/s, prefill 0.62 -> 0.97 tok/s. The pin is skipped with a printed reason when the every-token set would exceed 60% of RAM. - Same model and - box: decode went from 2.4 tok/s on the page-cache path to 4.0 tok/s averaged - over a 512-token generation (~4.7 steady, ~90% arena hits), against 3.0 - tok/s for `--stream-cpu` - so `--stream-experts` now matches `--stream-cpu` - on short generations and pulls ahead roughly 1.5x once the arena warms, - before the KV-cache advantage at depth. In server configs the placement is the per-model `stream: experts | cpu` key and the feeder opt-outs are `prefill_feeder: false` / @@ -555,9 +554,8 @@ at top-p 0.95, while the same setting sampled untruncated put one wrong-script token into an 11k-token run. The tok/s figures are whole-run averages of these single generations at different lengths, not controlled A/B numbers; the table above is the measured comparison. -Each screenshot links to its generated page, committed beside it in -`docs/assets/perf/` (GitHub shows the page source; download one to -watch the animation). +Each screenshot links to its generated page, as with the samples earlier +in this guide. The sampling interaction also has a constructive reading. The full pair was rerun once on the same prompt and build with cooled sampling, at @@ -576,8 +574,8 @@ win by softening the levers at the card's temperature. The same comparison at the deep end of the space. The four Kimi-K3 settings measured earlier in this guide each ran the same prompt once -to completion at temperature 1.0. Each screenshot links to its -generated page, committed beside it in `docs/assets/perf/`: +to completion at temperature 1.0; screenshots link to the generated +pages as before: | | | |---|---| From 9a91e89e7e156ca3e93de4c12c6e29c0d1ca201f Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 22:10:27 -0700 Subject: [PATCH 09/19] docs(internals): fold diagram-restating component item, trim acceptance-gate echo --- docs/adding-architectures.md | 2 +- docs/serving-architecture.md | 14 ++++++-------- 2 files changed, 7 insertions(+), 9 deletions(-) diff --git a/docs/adding-architectures.md b/docs/adding-architectures.md index 7f23e80..a66a48d 100644 --- a/docs/adding-architectures.md +++ b/docs/adding-architectures.md @@ -48,7 +48,7 @@ load cleanly into wrong weights. ## The acceptance gate (all must pass) -An architecture is not done when it generates fluent text. It's done when: +An architecture is done when: - Strict load: `load_model` builds, swaps, and `load_weights` leaves no parameter unfilled. The loader's unfilled-params warning must be empty. diff --git a/docs/serving-architecture.md b/docs/serving-architecture.md index 7c60d1c..79eaae2 100644 --- a/docs/serving-architecture.md +++ b/docs/serving-architecture.md @@ -89,23 +89,21 @@ flowchart TD ## Components -1. GGUF on disk: a text LLM GGUF (K-quant: Q4_K / Q6_K / MXFP4 / ...). VLM models - add a second `mmproj` GGUF (float or Q8_0) carrying the vision/audio tower. - -2. Loader (`gmlx.load_model`): parses the GGUF wire bytes, remaps +1. Loader (`gmlx.load_model`): parses the GGUF wire bytes (the text LLM GGUF; + a VLM adds a second `mmproj` GGUF carrying the vision/audio tower), remaps GGUF tensor names to HF names, synthesizes the config and tokenizer (including the chat template), builds the stock model class, and swaps the quantized leaves for K-quant modules (`KQuantLinear`, `gather_qmm`, `KQuantMultiLinear`). Output is a `(model, config, tokenizer)` triple: no safetensors round-trip. -3. Model adapter + residency: a text model is wrapped in +2. Model adapter + residency: a text model is wrapped in `mlx_vlm.models.text_only.Model`, which exposes the `get_input_embeddings` / `language_model` interface the engine expects. A `StoppingCriteria` is attached to the tokenizer. VLM models are wrapped in their `mlx-vlm` vision/audio model class instead. Wrapped models are held in a multi-model residency pool (pinned + LRU) that owns a single process-wide `wired_limit`. -4. Engine (`mlx_vlm.generate.ar.BatchGenerator`): continuous (in-flight) +3. Engine (`mlx_vlm.generate.ar.BatchGenerator`): continuous (in-flight) batching over a ragged `BatchKVCache`. The engine is embeds-in: the request path precomputes `inputs_embeds` via `get_input_embeddings` and submits them through `insert(...)`, then drains tokens with `next()`. Prefix reuse is handled by the APC @@ -115,7 +113,7 @@ flowchart TD it under a draft model); see [server-config.md](server-config.md#speculative-decoding--the-prompt-cache). -5. HTTP layer (`mlx-vlm` FastAPI app): exposes OpenAI Chat Completions +4. HTTP layer (`mlx-vlm` FastAPI app): exposes OpenAI Chat Completions (`/v1/chat/completions`), OpenAI Responses (`/v1/responses`), and Anthropic Messages (`/v1/messages`), each with streaming SSE. Tool calls are extracted from the raw token stream by `mlx_lm.tool_parsers`, selected automatically from the @@ -133,6 +131,6 @@ flowchart TD profiles, speculative decoding, and batching apply per round ([assistant.md](assistant.md#served-assistants)). -6. Clients: any OpenAI- or Anthropic-compatible client. Pointing +5. Clients: any OpenAI- or Anthropic-compatible client. Pointing `ANTHROPIC_BASE_URL` at the `/v1/messages` endpoint lets Anthropic-API tools (for example Claude Code) drive a local GGUF model. From 6d7a4a9b0607a53e510071332ae727880b0864c4 Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 22:11:16 -0700 Subject: [PATCH 10/19] docs(cli,launch): single-source the launch flag table in cli.md, drop rename-era intro --- docs/cli.md | 4 ---- docs/launch.md | 18 ++---------------- 2 files changed, 2 insertions(+), 20 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index 1cb5cef..f7b610a 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -23,10 +23,6 @@ | [`gmlx profiles`](#gmlx-profiles) | Show per-family sampling defaults and `@intents`; resolve one model's sampling. | | [`gmlx completion`](#gmlx-completion) | Print a shell completion script (zsh, bash, fish). | -`gmlx` is the installed command; every action is a subcommand (`gmlx run`, -`gmlx serve`, `gmlx init`, and so on). The project, package, and command are -all named `gmlx`. - Every verb operates on the GGUF file itself: the file on disk is the model. Every flag is also visible via `--help`. diff --git a/docs/launch.md b/docs/launch.md index 389b700..600c275 100644 --- a/docs/launch.md +++ b/docs/launch.md @@ -95,22 +95,8 @@ launch carries it into each tool's configuration in that tool's native slot: ## Flags -| Flag | Meaning | -|------|---------| -| `--model ID[@profile]` | Served model (and optional profile) the tool should use. Default: the server's default-marked model. | -| `--base-url URL` | Target an explicit server. Never auto-started. | -| `--host H` / `--port P` | Target host/port for the managed server (default `127.0.0.1:8080`). | -| `--api-key KEY` | Key for a key-protected server, plumbed per the table above. | -| `--provider-id NAME` | Provider id written into the tool's config. | -| `--config-path PATH` | Where the tool config is written (default under `~/.config/gmlx`). For open-webui this picks its `DATA_DIR` instead. | -| `--config-only` | Write the config and print the run command; do not exec. | -| `--no-start` | Never auto-start a server. | -| `--start-timeout S` | Cap the auto-start readiness wait (default `0` = unbounded). | -| `--no-keep` | Do not ask the server to keep `--model` resident. | - -Exit codes: `0` tool launched (or server started and ready); `1` server down with -`--no-start` or `--base-url`, a launchd server mid-restart, or the auto-started -process died; `2` no config or a malformed config; `130` Ctrl-C during the start wait. +The full flag table and exit codes are in the CLI reference: +[cli.md](cli.md#launch-connect-a-coding-agent-or-chat-app). ## The clients From 11abf67e4efd3e0f6c52d6efbae9fed37fd5b355 Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 22:12:23 -0700 Subject: [PATCH 11/19] docs(server-config): cut sequence-diagram narration paragraph --- docs/server-config.md | 9 --------- 1 file changed, 9 deletions(-) diff --git a/docs/server-config.md b/docs/server-config.md index bf2c91d..69ee03d 100644 --- a/docs/server-config.md +++ b/docs/server-config.md @@ -1788,15 +1788,6 @@ sequenceDiagram E-->>C: stream tokens ``` -In prose, a request flows: (1) the client POSTs a chat completion whose -`model` is `id@profile`; (2) the mlx-vlm app asks the residency pool for that -model; (3) the pool has the serving resolver split the id from the profile -and resolve the file path; (4) a resident model is returned as-is, while a -cold one is built by the loader (leaves swapped to `kq.*` modules) and cached -in the pool; (5) the generation arguments are seeded from the active -profile's sampling; (6) the batch generator decodes and streams tokens back -to the client. - For the bridge/residency mechanics (how the path-keyed companion registries and the context-aware runtime proxy work), see [serving-architecture.md](serving-architecture.md). From 18cc88edbd52c7513d91f257c63373b3b55caee0 Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 22:12:40 -0700 Subject: [PATCH 12/19] docs(launch): fold flag-table pointer into the intro, drop the empty Flags heading --- docs/launch.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/docs/launch.md b/docs/launch.md index 600c275..dbc17b9 100644 --- a/docs/launch.md +++ b/docs/launch.md @@ -13,7 +13,8 @@ gmlx launch open-webui # browser chat app on :3000 ``` launch never installs the tool itself. If the binary is not on PATH, it prints an -install hint and exits. +install hint and exits. The full flag table and exit codes are in the +[CLI reference](cli.md#launch-connect-a-coding-agent-or-chat-app). ## How a launch works @@ -93,11 +94,6 @@ launch carries it into each tool's configuration in that tool's native slot: | aichat, elia | `api_key` in the injected config | | open-webui | `OPENAI_API_KEY` in the exec environment only | -## Flags - -The full flag table and exit codes are in the CLI reference: -[cli.md](cli.md#launch-connect-a-coding-agent-or-chat-app). - ## The clients ### claude-code From aa09663b36147e3b6c318607c448744c657a0314 Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 22:32:42 -0700 Subject: [PATCH 13/19] docs(style): de-bridge README, migrating, streaming - split semicolon chains and multi-clause sentences --- README.md | 52 ++++++------ docs/migrating.md | 27 ++++--- docs/streaming.md | 202 +++++++++++++++++++++++----------------------- 3 files changed, 141 insertions(+), 140 deletions(-) diff --git a/README.md b/README.md index 8c2aea2..5747d83 100644 --- a/README.md +++ b/README.md @@ -15,9 +15,9 @@ the most accurate open quant formats there are — exactly as published ([accuracy per byte](#accuracy-per-byte)). The companion project [mlx-kquant](https://github.com/asher/mlx-kquant) supplies the Metal kernels that run these formats natively on Apple's -[MLX](https://github.com/ml-explore/mlx) framework, and on the same file -gmlx benchmarks faster than llama.cpp, with the gap widest at the -50-200k-token contexts where coding agents and long sessions live +[MLX](https://github.com/ml-explore/mlx) framework. On the same file, gmlx +benchmarks faster than llama.cpp, with the gap widest at the 50-200k-token +contexts where coding agents and long sessions live ([performance](#performance)). A 100B+ model starts generating within seconds of launch, and a MoE bigger than RAM still runs, streaming its experts from disk ([bigger than memory](#bigger-than-memory)). @@ -57,13 +57,13 @@ gmlx stop # the server ran detached `gmlx[all]` adds every optional feature to the core platform: the upgraded chat TUI, voice chat, and the MCP assistant. The core alone already carries serving, vision, embeddings, and the menu bar, so `gmlx[chat]` is a smaller -install that gives up only voice and the assistant; `gmlx init` adds either -one later. `uv tool` and pipx put `gmlx` on your PATH in every terminal; +install that gives up only voice and the assistant. `gmlx init` adds either +one later. `uv tool` and pipx put `gmlx` on your PATH in every terminal. `pip install "gmlx[all]"` into a venv you manage works too, with the command available only while that venv is active. -A model typically needs roughly its file size in memory, plus the KV cache; -the exception is MoE models, which can run +A model typically needs roughly its file size in memory, plus the KV cache. +The exception is MoE models, which can run [bigger than memory](#bigger-than-memory). If anything misbehaves, `gmlx doctor` checks the runtime, config, model paths, and services in one pass. @@ -92,9 +92,9 @@ gmlx run model.gguf@creative --prompt "Write a haiku about entropy." gmlx chat qwen3.6-27b-q6 --profile instruct # --profile NAME = the flag form of @NAME ``` -A `.gguf` path works with no setup; a bare id like `qwen3.6-27b-q6` names a -model from your server config - `gmlx list` shows yours (the -[getting-started guide](https://github.com/asher/gmlx/blob/main/docs/getting-started.md) sets one up). +A `.gguf` path works with no setup. A bare id like `qwen3.6-27b-q6` names a +model from your server config - `gmlx list` shows yours, and the +[getting-started guide](https://github.com/asher/gmlx/blob/main/docs/getting-started.md) sets one up. Details: the [CLI reference](https://github.com/asher/gmlx/blob/main/docs/cli.md). @@ -192,10 +192,10 @@ gmlx and llama.cpp run the same GGUF file, so the comparison is direct: on an M5 Max (128 GB), gmlx prefills faster on every model in our fleet at every depth measured, 1.2-1.6x at short context and 2-4x past 100k tokens. At matched non-speculative baselines, decode starts at parity and wins -fleet-wide from 16k tokens as the KV cache deepens; with speculative decoding -active on both engines, gmlx decodes 1.1-2x ahead throughout, and MTP's lift -over the same server with it off holds at 1.4-1.8x from 17k through 110k, -where llama.cpp's speculation gain decays with depth. +fleet-wide from 16k tokens as the KV cache deepens. With speculative decoding +active on both engines, gmlx decodes 1.1-2x ahead throughout. MTP's lift over +the same server with it off holds at 1.4-1.8x from 17k through 110k, where +llama.cpp's speculation gain decays with depth. @@ -207,7 +207,7 @@ Reference points at short context: gemma-4-12B-it (dense, Q6_K) decodes at ~850 vs ~730 tok/s; Qwen3.5-9B (dense, Q6_K) decodes at ~112 vs ~76 tok/s (1.5x), prefilling at ~1600 vs ~1140 tok/s. Absolute numbers scale with the machine's memory -bandwidth; measure your own with `gmlx run model.gguf --bench "128,512,2048"`. +bandwidth. Measure your own with `gmlx run model.gguf --bench "128,512,2048"`. The speed comes from kernels built for exactly this work: mlx-kquant's fused K-quant and IQ matmuls, attention tuned for decode at depth, and a custom MTP @@ -215,16 +215,18 @@ verify path built for this server. The full fleet tables, per-model charts, and methodology are in [benchmarks.md](https://github.com/asher/gmlx/blob/main/docs/benchmarks.md). -When you want more, the levers are: MTP speculative decoding, which roughly -doubles decode throughput at short context, automatic on models with a native -draft head (Qwen3.5/3.6) and available to gemma-4 through a small companion -drafter (1.9-2.1x); the prompt cache, which removes repeated prefill for -agent workloads; KV-cache quantization for long contexts; and disk-streamed -execution for MoE models larger than memory ([below](#bigger-than-memory)). -The file you pick matters too: a uniform K-quant decodes meaningfully faster -than a heavily mixed one at similar or better quality. When each lever pays -off, with numbers — and where llama.cpp still leads — is in the -[performance guide](https://github.com/asher/gmlx/blob/main/docs/performance.md). +When you want more, there are four levers. MTP speculative decoding roughly +doubles decode throughput at short context; it is automatic on models with a +native draft head (Qwen3.5/3.6), and a small companion drafter brings it to +gemma-4 (1.9-2.1x). The prompt cache removes repeated prefill for agent +workloads. KV-cache quantization frees memory at long contexts. And +disk-streamed execution runs MoE models larger than memory +([below](#bigger-than-memory)). The file you pick matters too: a uniform +K-quant decodes meaningfully faster than a heavily mixed one at similar or +better quality. The +[performance guide](https://github.com/asher/gmlx/blob/main/docs/performance.md) +covers when each lever pays off, with numbers, and where llama.cpp still +leads. ### Bigger than memory diff --git a/docs/migrating.md b/docs/migrating.md index 069d469..f11fd82 100644 --- a/docs/migrating.md +++ b/docs/migrating.md @@ -7,8 +7,8 @@ has a different name, and what works differently on purpose. ## Coming from llama.cpp / llama-server Your GGUFs work as-is: `gmlx run model.gguf` is the moral equivalent of -`llama-cli -m model.gguf`, and `gmlx serve model.gguf` of `llama-server`; -the default port is the same 8080. +`llama-cli -m model.gguf`, and `gmlx serve model.gguf` of `llama-server`. +The default port is the same 8080. Common flag equivalents: @@ -29,7 +29,7 @@ Common flag equivalents: | `--lora adapter` | `--adapter adapter.gguf` | llama.cpp-format adapter GGUFs interop in both directions ([lora.md](lora.md)) | `/v1/completions` is served with a minimal surface (single string prompt, -one choice); `/v1/chat/completions` is the primary route, and Anthropic +one choice). `/v1/chat/completions` is the primary route, and Anthropic Messages and OpenAI Responses run on the same port. Per-request details: [server-config.md](server-config.md#api-capabilities). @@ -39,11 +39,11 @@ What carries over: any GGUF you can point at. What does not: Ollama's model store and API. - Ollama's library lives as sha-named blobs, not `.gguf` files, so it cannot - be pointed at directly; re-download the models you use with `gmlx pull` + be pointed at directly. Re-download the models you use with `gmlx pull` (`gmlx validate hf:/` lists every variant first). - gmlx speaks the OpenAI, Anthropic, and OpenAI Responses APIs, not the Ollama API (`/api/generate`, `/api/chat`). Clients configured for an - OpenAI-compatible endpoint work unchanged; Ollama-native integrations need + OpenAI-compatible endpoint work unchanged. Ollama-native integrations need their OpenAI mode, pointed at port 8080 (not 11434). - Modelfile parameters map onto the YAML config: `num_predict` is the server `--max-tokens` default, sampling knobs live per model or in `profiles:` @@ -61,16 +61,17 @@ gmlx init --models-dir ~/.lmstudio/models -r The init wizard also offers the LM Studio directory on its own when it exists. Ids, sampling profiles, and a default model are then yours to adjust -in one YAML file; the local server surface (OpenAI API, `/v1/models`) is the +in one YAML file. The local server surface (OpenAI API, `/v1/models`) is the same shape LM Studio's is, plus Anthropic Messages on the same port. ## Why serve from gmlx Beyond the kernel-level speed on K-quants ([performance.md](performance.md)): -warm config reload (SIGHUP / `POST /v1/reload`) without dropping residents; -residency controls (byte budget, pinning, idle unload, keep tiers); -loopback-by-default binding that refuses a wide bind without a key; -cross-request prompt caching with an optional SSD tier; MTP speculative -decoding on served models; and one-command client hookups -(`gmlx launch claude-code`, `open-webui`, ...) that never touch your -dotfiles ([launch.md](launch.md)). + +- warm config reload (SIGHUP / `POST /v1/reload`) without dropping residents +- residency controls: byte budget, pinning, idle unload, keep tiers +- loopback-by-default binding that refuses a wide bind without a key +- cross-request prompt caching with an optional SSD tier +- MTP speculative decoding on served models +- one-command client hookups (`gmlx launch claude-code`, `open-webui`, ...) + that never touch your dotfiles ([launch.md](launch.md)) diff --git a/docs/streaming.md b/docs/streaming.md index bc20826..1afa451 100644 --- a/docs/streaming.md +++ b/docs/streaming.md @@ -1,7 +1,7 @@ # Streaming: MoE models bigger than memory A MoE model whose file exceeds what the GPU can wire - or exceeds RAM -outright - still runs: gmlx streams the expert weights from disk and +outright - still runs. gmlx streams the expert weights from disk and keeps what every token needs resident. MoE decode is what makes this viable: only the routed experts are read per token, so the per-token working set is a small slice of the file. The levers in this guide @@ -11,19 +11,19 @@ router can spare it. Set expectations first. When experts stream from disk, decode is bound by the SSD and CPU, not the GPU, and single-digit tokens per second is -normal - the feeders raise the constant, not the nature of the bound. +normal. The feeders raise the constant, not the nature of the bound. This is a capacity feature that makes a 200B-class MoE usable on a 64 GB machine, not a speed feature, and it is strictly for the -over-budget case: a model that fits in memory runs several times +over-budget case. A model that fits in memory runs several times faster on the normal GPU path. Demand misses are served at the drive's random-read latency and the feeders read at its queue depth, so keep the GGUF on the internal NVMe -SSD for the best result; an external drive works, but decode follows +SSD for the best result. An external drive works, but decode follows its latency and bandwidth down. Unless a different machine is named inline, measured numbers in this -guide are from a 14-inch M5 Max MacBook Pro (128 GB); the hardware +guide are from a 14-inch M5 Max MacBook Pro (128 GB). The hardware scope note under [Reference numbers](performance.md#reference-numbers) covers scaling to other chips. @@ -55,7 +55,7 @@ The load is a mmap, not a read, so generation starts within seconds whatever the file size. Prefill stages each layer's experts through the prefill feeder. Decode starts at the disk's demand rate and improves over the first few dozen tokens as the expert arena converges on the -model's hot set; the decode feeder's exit stats (printed at `-v` on +model's hot set. The decode feeder's exit stats (printed at `-v` on `run` and `chat`, always in server logs) show the arena hit rate a session settled at. @@ -64,9 +64,9 @@ and speculative entries. On the CLI, MTP composes with `--stream-experts` (not `--stream-cpu`) but defers by default: auto-MTP stays off and an explicit `--speculative` opts in. The lossy `--moe-*` levers below are hard-incompatible with MTP and force plain decoding. -What stays resident - the every-token -layers plus the KV cache - follows the normal fit arithmetic in -[getting-started.md](getting-started.md#will-it-fit), and a quantized +What stays resident (the every-token layers plus the KV cache) follows +the normal fit arithmetic in +[getting-started.md](getting-started.md#will-it-fit). A quantized KV cache (`--kv-bits 8`) is the usual companion at long context. ## What the over-budget case produces @@ -86,9 +86,8 @@ watch the animation). Both pages ran as generated. The larger quant also added scroll and drag controls for the cruising pace that the prompt never asked for. The point of the samples is scale: a model five to seven times the -machine's RAM sustaining a coherent 30k-token single-file program is -what the levers in this guide buy, at the single-digit rates set out -above. +machine's RAM, sustaining a coherent 30k-token single-file program at +the single-digit rates set out above. ## Choosing a placement @@ -96,10 +95,10 @@ Two placements run MoE models whose files exceed what the GPU can wire: - `--stream-experts` keeps the every-token layers (attention, routers, shared experts, KV cache) on the GPU and streams the routed experts, which run on - the CPU stream. Historically slower than `--stream-cpu` at short context - because of the per-layer handoff; the decode feeder (below) reverses that, - and `--stream-experts` keeps its long-context advantage with a quantized KV - cache, where the large KV stays on GPU. + the CPU stream. It was historically slower than `--stream-cpu` at short + context because of the per-layer handoff. The decode feeder (below) + reverses that, and with a quantized KV cache it keeps its long-context + advantage, since the large KV stays on GPU. - `--stream-cpu` runs the whole model on the CPU device, mmap-backed, so the page cache streams weights from disk on demand. Past the wired budget the runtime adds sequential expert prefetch, advising the kernel a couple of layers ahead so @@ -109,7 +108,7 @@ Two placements run MoE models whose files exceed what the GPU can wire: With the decode feeder on, `--stream-experts` is the usual choice: it matches `--stream-cpu` on short generations, pulls ahead once the arena warms (measured below), and keeps the large KV cache on GPU at depth. -`--stream-cpu` keeps everything on one device; in a server config, +`--stream-cpu` keeps everything on one device. In a server config, `stream: cpu` switches the whole process to the CPU device, so it suits a single-model setup rather than mixing with GPU-resident models. @@ -129,33 +128,33 @@ Streaming models engage two feeder paths by default: popularity-managed GPU arena sized to the machine (`GMLX_DECODE_ARENA_GB` overrides) and reads only the misses from the GGUF, at SSD queue depth. The arena starts empty and converges within a few dozen tokens. The arena is - wired, so it also polices itself: under system memory pressure (another + wired, so it also polices itself. Under system memory pressure (another model, a build) it shrinks, keeping its most popular experts, and regrows - once pressure clears - a long-running model stays a good citizen on a - machine that is doing other work (`GMLX_DECODE_PRESSURE=0` pins it instead). + once pressure clears, so a long-running model coexists with a machine + that is doing other work (`GMLX_DECODE_PRESSURE=0` pins it instead). Same model and box as the prefill measurement above: decode went from 2.4 tok/s on the page-cache path to 4.0 tok/s averaged over a 512-token generation (~4.7 steady, ~90% arena hits), against 3.0 tok/s for - `--stream-cpu` - so `--stream-experts` now matches `--stream-cpu` on short - generations and pulls ahead roughly 1.5x once the arena warms, before the - KV-cache advantage at depth. + `--stream-cpu`. `--stream-experts` therefore matches `--stream-cpu` on + short generations and pulls ahead roughly 1.5x once the arena warms, + before the KV-cache advantage at depth. - The arena also serves multi-token expert calls whose routed union exceeds its slots - the next chat turn's prefill after a decode, or a wide speculative verify batch - by halving the chunk along the token axis and recursing until each piece fits (`GMLX_ARENA_SPLIT_MAX_TOKENS`, default 256, caps the size; `0` disables). Without this, those calls fall to a CPU page-cache gather that runs at demand-fault speed while most of RAM is - wired: on Kimi-K3 UD-IQ2_XXS a 48-token second-turn prefill measured 0.25 - tok/s on the fallthrough and 2.13 tok/s through the split (8.5x), with the - post-turn decode dip gone as well, because the reads stay on the arena's - read pool and its popularity accounting. + wired. On Kimi-K3 UD-IQ2_XXS, a 48-token second-turn prefill measured 0.25 + tok/s on the fallthrough and 2.13 tok/s through the split (8.5x). The + post-turn decode dip disappeared as well, because the reads stay on the + arena's read pool and its popularity accounting. - Follow-up turns longer than the split cap go back through the prefill feeder's ring, which was released at first decode so the arena could take its wired budget. Rebuilding the ring on top of a full wired arena would breach the wired cap, so the arena lends the ring its footprint first: every layer shrinks eagerly, keeping its most popular experts, and the next decode releases the ring and regrows the arena layer by layer. Both - feeders on is therefore the right default for chat and serve; the lend + feeders on is therefore the right default for chat and serve. The lend makes long follow-ups safe without giving up warm decode resumes. Streaming installs also pin the every-token weights (`GMLX_PIN_WEIGHTS=0` @@ -163,13 +162,12 @@ disables): every non-expert tensor - attention, routers, shared experts, norms, the lm head - is mlocked so the kernel cannot evict it. Without the pin those weights are plain file-backed mmap pages, and on a box running at the free-page floor the -kernel evicts them between uses; each decode token then re-faults the whole +kernel evicts them between uses. Each decode token then re-faults the whole every-token set from disk, which on a large model saturates the SSD before the experts read a byte. The fault traffic is invisible to the feeder's stall accounting (it appears as compute time), so the symptom is a decode rate stuck near `every_token_bytes / ssd_bandwidth` per token no matter the -arena -hit rate. Measured on Kimi-K3 UD-IQ2_XXS (662 GB, 62 GB every-token set, M5 Max +arena hit rate. Measured on Kimi-K3 UD-IQ2_XXS (662 GB, 62 GB every-token set, M5 Max 128 GB): decode 0.10 -> 0.38 tok/s, prefill 0.62 -> 0.97 tok/s. The pin is skipped with a printed reason when the every-token set would exceed 60% of RAM. @@ -188,10 +186,10 @@ With the decode feeder on, arena misses are also prestaged by lookahead (`GMLX_DECODE_LOOKAHEAD=0` disables): each MoE layer runs the next MoE layer's router on its own input and pre-reads the predicted misses on a small dedicated pool while the current layer computes. The residual changes little between -adjacent sublayers, so the prediction lands: measured recall of the next +adjacent sublayers, so the prediction lands. Measured recall of the next layer's actual top-k is ~78% on GLM-5.2 (@8) and MiniMax-M3 (@4), against ~35% for previous-token routing reuse. Predictions move bytes and nothing -else - routing and outputs are bit-identical - and speculation is kept off +else; routing and outputs are bit-identical. Speculation is kept off the demand path three ways: prestage reads are submitted only after the current layer's demand misses have finished, the read threads run at utility disk-I/O priority so the kernel services demand misses first @@ -199,7 +197,7 @@ utility disk-I/O priority so the kernel services demand misses first layer settles its in-flight prestages before serving. A per-layer rank gate watches how often each prediction rank actually lands and stops submitting ranks that measure below `GMLX_DECODE_LOOKAHEAD_MIN_P` (default -`0.5`); predictions the router then does not route to are cancelled before +`0.5`). Predictions the router then does not route to are cancelled before they reach the disk when their reads have not started (`GMLX_DECODE_LOOKAHEAD_CANCEL=0` disables). Together these keep the wasted-read tax near zero on models where the SSD is the bottleneck. @@ -211,8 +209,8 @@ issuing reads, the check worth running on a new model family. Streamed decode has a work pattern the GPU's power management punishes: sub-millisecond compute bursts separated by host and disk gaps every MoE layer. The GPU races to idle in each gap, its clocks sag, and the next -burst pays the ramp back up - measured as 3-5x inflation of identical -per-layer work (0.3 ms warm vs up to 4+ ms ramp-inflated). The more +burst pays the ramp back up. On identical per-layer work that shows as +3-5x inflation (0.3 ms warm vs up to 4+ ms ramp-inflated). The more per-token host syncs a model's decode path has, the more of its token time is ramp rather than work. @@ -229,10 +227,10 @@ generations: | GLM-5.2 UD-IQ3_XXS (282 GB, 75 streamed layers) | arena capped at 70 GB (`GMLX_DECODE_ARENA_GB=70`) + `--moe-miss-shed 0.85`, lookahead off | 2.51 tok/s | 3.64 tok/s | +45% | | Hunyuan3 IQ4_XS (159 GB, 79 streamed layers) | `--moe-layer-shed 0.10` + `--moe-miss-shed 0.90` | 4.01 tok/s | 5.29 tok/s | +32% | -The diagnostic signature is worth knowing because it says whether a given -model will benefit: stall time and arena hit rate are unchanged by the -heartbeat (the disk is doing the same work), so if a streamed model's -per-token time is dominated by the eval/sync bucket rather than stalls +The diagnostic signature says whether a given model will benefit. Stall +time and arena hit rate are unchanged by the heartbeat, because the disk +is doing the same work. So when a streamed model's per-token time is +dominated by the eval/sync bucket rather than stalls (`GMLX_DECODE_PHASE_STATS=1` prints the split), clock sag is a candidate and keep-warm is the cheap test. Dense in-RAM decode does not have the gap pattern and gains nothing. @@ -240,8 +238,8 @@ gap pattern and gains nothing. The cost is power, and only while decoding: the heartbeat parks (no GPU work) after one second without decode activity and wakes on the next streamed decode call (`GMLX_KEEPWARM_IDLE_S` tunes the window; `0` beats -continuously). An idle server pays nothing; the first token after an idle -gap pays one clock ramp. +continuously). An idle server pays nothing, and the first token after an +idle gap pays one clock ramp. The heartbeat kernel itself is nearly free, and measuring it shows the mechanism plainly (powermetrics, M5 Max, heartbeat alone on an otherwise @@ -252,9 +250,9 @@ own bursts set the clock level; the heartbeat keeps the GPU from going idle in the gaps, so the governor holds that level instead of sagging and re-ramping every layer. The real power cost is therefore holding the decode-level P-state through the gaps, which scales with how hard -the workload drives the clock - the ~0.1 W kernel is noise against it. -It is opt-in because that trade - more watts for the same output, -faster - is yours to make, not a default, especially on battery. +the workload drives the clock; the ~0.1 W kernel is noise against it. +The trade is more watts for the same output, faster. On battery, +`GMLX_GPU_KEEPWARM=0` takes it back. ## The lossy levers @@ -294,11 +292,11 @@ within the 6). The staging levers act at the decode feeder instead of the router. `--moe-miss-shed P` drops routed experts that would demand-miss the arena, lowest scores first, keeping at least share P of the token's gate -mass, so its quality budget is spent exactly where the disk stalls are: -an arena-resident or prestage-inflight expert is never dropped, and a -shed expert earns no popularity credit, so the arena keeps its hot set. +mass. The quality budget is spent exactly where the disk stalls are: an +arena-resident or prestage-inflight expert is never dropped, and a shed +expert earns no popularity credit, so the arena keeps its hot set. It needs the decode feeder and a block that hands router scores to the -expert call; where it engages, it is the most targeted lever per point of +expert call. Where it engages, it is the most targeted lever per point of quality spent, and its payoff scales directly with the miss rate. `--moe-prestage keepers` attacks miss-shed's residual stalls from the @@ -306,7 +304,7 @@ speculative side. It adds no quality knob of its own; it applies the policy miss-shed already defines, one layer earlier. In the default `ranked` mode, lookahead prestages its rank-gated predictions with guess-grade caution (never evicting a more popular resident), which -caps how many demand misses it can absorb; and because an inflight +caps how many demand misses it can absorb. And because an inflight read exempts its expert from the shed, ranked lookahead incidentally rescues some experts the policy would have dropped, at the price of the read. Keeper mode filters each prediction through the policy @@ -315,7 +313,7 @@ at all, and the predicted keepers are staged demand-grade, since if the prediction is right the demand path would do those same reads synchronously one layer later. Where prediction recall is good, this converts demand stalls into reads -that overlap compute; the lookahead exit stats (submitted vs adopted) +that overlap compute. The lookahead exit stats (submitted vs adopted) are the guardrail that the added aggression is landing. Requires `--moe-miss-shed` to define the policy. @@ -328,7 +326,7 @@ arena hit rate is high and misses are rare. In server configs the lossy levers and keeper prestage are the per-model `moe_experts: K` / `moe_expert_mass: P` / `moe_miss_shed: P` / `moe_layer_shed: P` / `moe_prestage: keepers` keys (or the matching -`serve` flags for a single positional model); the probe stays +`serve` flags for a single positional model). The probe stays CLI-only, so size P with a `gmlx run --moe-expert-probe` pass before pinning a value in a config. @@ -337,15 +335,15 @@ and read the decode feeder's exit stats (arena hit rate; printed by `run`/`chat` at `-v`, and always in server logs) from a representative session. The hit rate decides first. A low hit rate - an arena small relative to the model - makes `--moe-miss-shed` the lead lever whatever -the router looks like: it spends only on calls that would stall, so it +the router looks like. It spends only on calls that would stall, so it beats expert-mass on cost at equal reads saved, and a probe-attractive concentration number can still lose to it outright (measured below). At a healthy hit rate, a concentrated router points to `--moe-expert-mass`, -which removes reads and compute together at minimal dropped mass; a flat +which removes reads and compute together at minimal dropped mass. A flat router takes it off the table and leaves the per-layer overhead as the -standing cost, which only `--moe-layer-shed` touches - though a large -share of that overhead is clock ramp, which the lossless keep-warm above -removes first; run it before spending quality here. +standing cost, which only `--moe-layer-shed` touches. A large share of +that overhead is clock ramp, though, which the lossless keep-warm above +removes first. Run it before spending quality here. ### Hy3: flat router, high hit rate @@ -368,21 +366,21 @@ Those are sustained-regime medians. Per the cool-box transient note under [Measuring](performance.md#measuring), a rested 14-inch machine ran the same arms 15-25% faster for its first twenty minutes, baseline at 5.0 tok/s and the pair at 5.6 or better. Note also that softening the pair keeps its quality -margin but not its speed: miss-shed's payoff falls steeply as P rises +margin but not its speed. Miss-shed's payoff falls steeply as P rises (at 0.93 it sheds only a third of the experts it sheds at 0.90), so the softened pair returned a few percent where the full pair returned +13. The soft edge is real. In single long-generation checks at this model card's temperature of 0.9, the pair at 0.09/0.91 emitted a stray token -into code even under top-p 0.97, while 0.07/0.93 ran clean, so the -high-temperature envelope on this model is the softened pair and its -few percent. The larger wins belong to workloads that can run cooler +into code even under top-p 0.97, while 0.07/0.93 ran clean. The +high-temperature envelope on this model is therefore the softened pair +and its few percent. The larger wins belong to workloads that can run cooler sampling or accept an occasional stray. That ordering is this model's, not a law. With a flat router, -expert-mass had nothing cheap to drop; at a 92% hit rate, misses were +expert-mass had nothing cheap to drop. At a 92% hit rate, misses were rare enough that the per-layer overhead was the standing cost, so layer-shed led and the shed pair composed (+8% and +4% multiply to -roughly the observed +13%: they cut disjoint costs). On a +roughly the observed +13% because they cut disjoint costs). On a concentrated-router model with a healthy hit rate the probe will show the inversion - most reads removed for a few percent of mass - before any lossy run needs to be made. @@ -409,52 +407,52 @@ expert-mass did remove reads. But most of the reads it removed were arena hits that cost nothing, and its router-side filtering cost more than the stalls it saved. Miss-shed spends the same budget only where a stall is otherwise certain, which also means its realized cost sits -far below the probe's unconditional number: at P=0.80 the probe -predicts 12% dropped mass, while the residency-aware shed dropped -2.9%, shedding 8% of routed experts across a third of token-layer -calls. Two 10k-token generations at temperature 0.6 / top-p 0.95 ran -clean, producing complete working artifacts with no stray tokens. The -probe sizes expert-mass, but it cannot see residency; when the exit -stats show a low hit rate, reach for miss-shed first. +far below the probe's unconditional number. At P=0.80 the probe +predicts 12% dropped mass; the residency-aware shed dropped 2.9%, +shedding 8% of routed experts across a third of token-layer calls. +Two 10k-token generations at temperature 0.6 / top-p 0.95 ran clean, +producing complete working artifacts with no stray tokens. The probe +sizes expert-mass, but it cannot see residency. When the exit stats +show a low hit rate, reach for miss-shed first. ### GLM-5.2: wider routing A third measured point moves one variable: routing width. GLM-5.2 (282 GB UD-IQ3_XXS, 256 experts top-8, sigmoid gating) streams on the -same machine at a per-expert hit rate near 88% - healthier than M3's - -yet stalls more, because a layer stalls when any of eight routed -experts miss, not four: at hit rate h the stall odds are 1 - h^k, so -k = 8 roughly doubles them at the same h. That amplification works in -both directions - every point of hit rate miss-shed buys back is worth -about twice as much - which is why the same lever measured stronger -here (+16.5% decode at P=0.80, stalls halved; +10.7% at P=0.85, both -even-round alternated 512-token medians) and why arena size, flat on +same machine at a per-expert hit rate near 88%, healthier than M3's, +yet stalls more. A layer stalls when any of eight routed experts miss, +not four: at hit rate h the stall odds are 1 - h^k, so k = 8 roughly +doubles them at the same h. That amplification works in both +directions, since every point of hit rate miss-shed buys back is worth +about twice as much. That is why the same lever measured stronger here +(+16.5% decode at P=0.80 with stalls halved; +10.7% at P=0.85; both +even-round alternated 512-token medians), and why arena size, flat on M3, mattered too (each arena GB bought ~0.2pt of hit rate). Wider routing also concentrates more meaning per expert, and that -moved the quality cliff: P=0.80, clean on M3, broke GLM-5.2 - and +moved the quality cliff. P=0.80, clean on M3, broke GLM-5.2 - and broke it in a way character scans cannot see. A 12k-token one-page-app generation completed with no stray tokens, valid markup, -and working code, but the page it drew was missing its subject (a sky -with no road and no car, on a prompt asking for a car on a road; the -lossless twin at the same seed drew the full scene). P=0.85 drew the -complete scene. Dropped gate mass degrades content before it degrades -form, so a shed level cannot be certified by scanning the output for -corruption: render the artifact and look at it, at deploy sampling -settings, against a lossless twin at the same seed. Miss-shed's safe -range is per-architecture; re-gate it whenever routing width or -gating changes. +and working code, but the page it drew was missing its subject: a sky +with no road and no car, on a prompt asking for a car on a road. The +lossless twin at the same seed drew the full scene, and P=0.85 drew +the complete scene. Dropped gate mass degrades content before it +degrades form, so a shed level cannot be certified by scanning the +output for corruption. Render the artifact and look at it, at deploy +sampling settings, against a lossless twin at the same seed. +Miss-shed's safe range is per-architecture: re-gate it whenever +routing width or gating changes. ### Kimi-K3: far over budget The deepest measured point runs the scale sample from the top of this guide with the levers on: Kimi-K3 UD-Q2_K_XL (861 GB on the same 128 GB machine, 384 experts routed 16 per token, 91 streamed expert -layers). This far over budget the arena holds a sliver of the expert +layers). This far over budget, the arena holds a sliver of the expert set, the lossless hit rate settles near 50%, and demand stalls are -about two thirds of decode wall, so the miss-targeted lever leads by -a wide margin, and each shed arm pairs it with keeper prestage -(`--moe-prestage keepers`, described earlier). One long generation +about two thirds of decode wall. The miss-targeted lever therefore +leads by a wide margin, and each shed arm pairs it with keeper +prestage (`--moe-prestage keepers`, described earlier). One long generation per setting on the same one-shot prompt as the samples above, temperature 1.0, 23-30k tokens each with thinking included. These are whole-run averages, not alternated A/Bs: @@ -477,7 +475,7 @@ from lossless to 0.80 bought little in this sample, while 0.70 and 0.65 returned +16% and +21%. All three shed levels produced complete working pages on this -long-form prompt; what separates them is content drift, compared +long-form prompt. What separates them is content drift, compared side by side in the Kimi-K3 screenshot table later in this guide. One step further down broke form, not just content: at 0.60 a code generation on this model produced a nonfunctional program in one try. @@ -491,7 +489,7 @@ hardened: multi-step arithmetic broke first, well before coherence, formatting, or code. `moe_layer_shed: 0.20` alone dropped arithmetic tasks, and so did `moe_layer_shed: 0.10` combined with `moe_miss_shed: 0.75` even though -each setting is clean alone - stacked levers compound onto the same +each setting is clean alone. Stacked levers compound onto the same cliff, so leave margin on both. On the same battery `moe_miss_shed` alone stayed clean down to 0.75 and `moe_expert_mass` down to 0.70. If a workload leans on chained arithmetic, put that in the test set before @@ -501,11 +499,11 @@ Past the edge, long generations add a second signature: stray token substitutions such as wrong-script digits or a bullet character inside code. The levers widen the low-probability tail that sampling can reach, so their -safe range depends on the sampling regime: untruncated sampling (top-p +safe range depends on the sampling regime. Untruncated sampling (top-p 1.0, which some model cards recommend) exposes the whole perturbed tail that nucleus truncation would mask. Certify at the temperature and top-p -you deploy with - a check run at a lower temperature does not cover a -hotter one. And a short check certifies short answers: a per-token slip +you deploy with; a check run at a lower temperature does not cover a +hotter one. And a short check certifies short answers. A per-token slip rate too small for it to see still accumulates over a 10k-token generation, so size levers softer for long-form code than short-form checks suggest. @@ -553,7 +551,7 @@ shows the sampling interaction from above. Its page was generated clean at top-p 0.95, while the same setting sampled untruncated put one wrong-script token into an 11k-token run. The tok/s figures are whole-run averages of these single generations at different lengths, -not controlled A/B numbers; the table above is the measured comparison. +not controlled A/B numbers. The table above is the measured comparison. Each screenshot links to its generated page, as with the samples earlier in this guide. @@ -594,10 +592,10 @@ landscape are the richest of the four while the car is the most damaged subject, with oversized featureless wheels and a stray light streak across the body. Between 0.65 and 0.80 the flaws differ in kind rather than degree, so one sample per setting cannot rank -adjacent levels; what it can show is that all three sit above the +adjacent levels. What it can show is that all three sit above the cliff, which is one step further down at 0.60, where a code generation broke outright. As on GLM-5.2, dropped mass degraded what -the pages drew long before it corrupted what they wrote: certifying a +the pages drew long before it corrupted what they wrote. Certifying a level means rendering the artifact, and ranking neighboring levels takes more samples than one. @@ -612,6 +610,6 @@ mlx-kquant's fp4 kernels (loads in seconds, streams like any k-quant), `packed` forces the repack, and the default `auto` chooses wire whenever a streaming placement is requested or the file exceeds ~90% of the wired budget. Wire mode is a hair slower than packed when the model fits in RAM (gpt-oss -decode ~5% at depth 0, converging at depth; prefill is at parity or better) -but is what makes the over-RAM case work at all, and it cuts the load time +decode ~5% at depth 0, converging at depth; prefill at parity or better). +It is what makes the over-RAM case work at all, and it cuts the load time from minutes of repack to a mmap. From 7c3e37f53c9bb057eebdb20d48f287ac5f08e472 Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 22:37:52 -0700 Subject: [PATCH 14/19] docs(style): de-bridge performance, talk, assistant --- docs/assistant.md | 18 +++---- docs/performance.md | 120 +++++++++++++++++++++----------------------- docs/talk.md | 51 ++++++++++--------- 3 files changed, 93 insertions(+), 96 deletions(-) diff --git a/docs/assistant.md b/docs/assistant.md index bfe41ac..bcf5120 100644 --- a/docs/assistant.md +++ b/docs/assistant.md @@ -11,7 +11,7 @@ around the server's own chat completions. Configure it once in a top-level server-side for any OpenAI client. It is deliberately a lightweight assistant, not an autonomous agent. Each turn -runs a bounded tool loop and ends when the model answers; nothing keeps working +runs a bounded tool loop and ends when the model answers. Nothing keeps working in the background afterward. It is built for errand-sized tasks: look something up, chain a few tool calls, write a note, remember a fact. The external coding agents `gmlx launch` connects (pi, opencode, and friends) are a different @@ -121,7 +121,7 @@ assistant: ``` A stdio tool server runs with a minimal environment (`HOME`, `PATH`, `SHELL`, -`TERM`, `USER`, `LOGNAME`); `env:` adds to that, and nothing else from this +`TERM`, `USER`, `LOGNAME`). `env:` adds to that, and nothing else from this process's environment is inherited. Pass a secret a server needs explicitly, as above -- an `HF_TOKEN` or `OPENAI_API_KEY` sitting in your shell never reaches third-party tool code. @@ -129,7 +129,7 @@ third-party tool code. **Document retrieval (RAG as a tool).** Qdrant's official server in embedded local mode gives the assistant `qdrant-store` and `qdrant-find` over a vector collection on disk -- no database process to run. Ask the assistant to store -passages and it can retrieve them semantically later; embedding happens +passages and it can retrieve them semantically later. Embedding happens inside the tool server with its own small local model, independent of this server's `/v1/embeddings`: @@ -157,7 +157,7 @@ transient context. They never bloat the rolling chat history. Without warning. What gets stored is a distilled fact ("sister Ana, birthday March 12"), not a -transcript: after each turn, a background request asks the chat model to boil +transcript. After each turn, a background request asks the chat model to boil the exchange down to at most three durable facts, or none, so small talk leaves no residue. A new fact that restates an existing one replaces it. `extract: false` stores raw user and assistant exchanges instead. `ttl_days` @@ -177,7 +177,7 @@ per-id store (`assistant-.db`), never this one. `--assistant` switches the chat REPL's turn engine from a local model load to the assistant on the managed server (auto-started if down, like `talk`). The positional argument is a served model id, or omitted for the server's default -model; a file path is refused, since the server owns the model. Tool activity +model. A file path is refused, since the server owns the model. Tool activity appears as transient status lines while the answer streams. ```sh @@ -189,11 +189,11 @@ The terminal experience is unchanged: markdown rendering, reasoning display, themes, history, sessions and `--resume`, `/system`, `/retry` and `/undo` (a retry or undo rewinds whole tool rounds, so the history never holds a half-finished tool exchange), plus `/memory` as above. Sampling flags and -their `/commands` forward to the server per round once you touch them; -untouched knobs stay on the server's own defaults. Flags that only make sense +their `/commands` forward to the server per round once you touch them. +Untouched knobs stay on the server's own defaults. Flags that only make sense for a local load are rejected (`--adapter`, `--mmproj`, chat-template flags) or ignored with a printed note (loading, KV-cache, speculative/MTP, CPU -placement); `/image`, `/audio` and the thinking budget are not available +placement). `/image`, `/audio` and the thinking budget are not available in this mode. `--base-url`, `--api-key`, `--no-start`, and `--start-timeout` target a remote or already-running server, exactly as in `talk`. @@ -251,7 +251,7 @@ The routing contract, per request to `/v1/chat/completions`: Reported usage sums `completion_tokens` across all rounds; `prompt_tokens` is the final round's. Concurrent assistant turns are capped (currently 4 per -server); a request over the cap gets an immediate 429 rather than queueing. +server), and a request over the cap gets an immediate 429 rather than queueing. Three known limits: a `stop` sequence forwards to every round and can in principle truncate an intermediate tool round; a non-streaming assistant turn cannot be cancelled by client disconnect (it runs its rounds to completion); diff --git a/docs/performance.md b/docs/performance.md index 0309fce..ce49528 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -35,11 +35,11 @@ intend to compare. The cool-box transient runs long, and it is chassis-dependent. A rested 14-inch M5 Max held full boost clocks for roughly twenty minutes of streamed MoE decode before heat-soaking into a sustained rate about 20% -lower; the 16-inch chassis cools better and holds boost longer still. +lower. The 16-inch chassis cools better and holds boost longer still. Both regimes are real. Book sustained numbers for anything long-running, -and size A/B warmup in minutes of decode rather than tokens, but a -chat-length session on a rested machine genuinely runs at the faster -rate the whole time. +and size A/B warmup in minutes of decode rather than tokens. A +chat-length session on a rested machine, though, genuinely runs at the +faster rate the whole time. A note on `pp512`-style numbers: prefill throughput at a 512-token prompt is the conventional benchmark figure, and it is a short-context number. If your real @@ -50,7 +50,7 @@ When a depth number looks wrong, check which attention kernel is actually running before anything else. `GMLX_ROUTE_LOG=1` prints per-route SDPA call counts at process exit; `GMLX_SDPA_DEBUG=1` traces the first deep calls live. Deep decode and speculative verify should land on fused routes -(`gqa_decode`, `fa_decode`, `fa_verify`, `verify_gemm`, `sdpa_vector`); `stock` +(`gqa_decode`, `fa_decode`, `fa_verify`, `verify_gemm`, `sdpa_vector`). `stock` at depth means the shape missed every eligibility gate and is paying for materialized attention scores. A one-shot warning fires automatically when a verify-shaped call does this. These flags work in the server process too, which @@ -67,16 +67,15 @@ medians of repeated runs: | gemma-4-12B-it (dense) | Q6_K | ~44 tok/s | ~72 tok/s | ~54 tok/s | ~850 tok/s | ~730 tok/s | | Qwen3.5-9B (dense) | Q6_K | ~70 tok/s | ~112 tok/s | ~76 tok/s | ~1600 tok/s | ~1140 tok/s | -These are the 512-token-depth medians from the July 2026 fleet round; the +These are the 512-token-depth medians from the July 2026 fleet round. The per-model tables in [benchmarks.md](benchmarks.md) carry the same runs to 200k tokens with run-to-run ranges. Against llama.cpp on the same GGUF, in the July 2026 fleet round: prefill is faster on every model at every depth measured, 1.2-1.6x at short context and -2-4x past 100k tokens; decode is faster in every cell but one at 512 tokens -(0.95x on gpt-oss-20b) and widens as the KV cache -deepens; and with speculative decoding active on both engines, decode runs -1.1-2x ahead at every depth. +2-4x past 100k tokens. Decode is faster in every cell but one at 512 tokens +(0.95x on gpt-oss-20b) and widens as the KV cache deepens. With speculative +decoding active on both engines, decode runs 1.1-2x ahead at every depth. @@ -85,20 +84,19 @@ deepens; and with speculative decoding active on both engines, decode runs Per-model charts, the full tables, methodology, and exact weight provenance are in [benchmarks.md](benchmarks.md). Your absolute numbers scale with your -chip's memory bandwidth: as -a rough guide, a Pro-tier chip has about half the bandwidth of a Max and a base -M-series chip a quarter to a fifth, so scale the table accordingly. The ratios -between models and quants hold. +chip's memory bandwidth. As a rough guide, a Pro-tier chip has about half the +bandwidth of a Max and a base M-series chip a quarter to a fifth, so scale +the table accordingly. The ratios between models and quants hold. A scope note on hardware: every number and every llama.cpp comparison in this guide was measured on an M5 Max (40-core GPU, 128 GB). The kernels are written for the matrix hardware in recent Apple GPU generations (M3 and later), and -tuning is no longer M5-only: MoE prefill and expert-gather batches route +tuning is no longer M5-only. MoE prefill and expert-gather batches route through a kernel tuned and validated on an M3 Max (128 GB), with the M5's tensor units still taking over where the hardware has them. M1 and M2 run the standard kernel paths but have not been a tuning focus, and we have not benchmarked them against llama.cpp. The bandwidth scaling above transfers -gmlx's own numbers between chips; the comparative claims are measured on +gmlx's own numbers between chips. The comparative claims are measured on M5-family hardware. If you run the bench commands on an M1 or M2, an issue with your numbers is welcome. @@ -130,7 +128,7 @@ The practical rules: Models that ship a native multi-token-prediction head (Qwen3.5 and Qwen3.6) get speculative decoding automatically on `run` and `chat`: the head drafts tokens ahead -and the base model verifies them, so output is exactly what the base model would +and the base model verifies them. Output is exactly what the base model would have produced, just faster when drafts are accepted. `--no-mtp` turns it off. gemma-4 models take the two-file shape instead: a small companion drafter GGUF via `--draft-gguf`. On the server it is the `speculative:` config key. @@ -172,16 +170,16 @@ expert layers rather than by a table of architectures, so a new MoE model inherits it on arrival. Two drafters (hy3, deepseek4) can draft only a single sequence, and are capped at 1 for that reason instead. -Per-model `speculative_width_cap` overrides the default; `GMLX_MTP_WIDTH_CAP=0` -turns the cap off for a measurement run. See +Per-model `speculative_width_cap` overrides the default, and +`GMLX_MTP_WIDTH_CAP=0` turns the cap off for a measurement run. See [server-config.md](server-config.md#speculative_width_cap). -A second: quantizing the KV cache (`--kv-bits`) shifts the target model's verify -logits away from the draft head and costs accepted drafts -- about a third fewer -at 4-bit in our measurements (9B hybrid, temperature 0.6), which can outweigh the -memory saved. Keep the KV cache in full precision when speculation is on if you -can; if memory forces quantization, prefer 8-bit, which perturbs the -distribution far less. +A second interaction: quantizing the KV cache (`--kv-bits`) shifts the target +model's verify logits away from the draft head and costs accepted drafts -- +about a third fewer at 4-bit in our measurements (9B hybrid, temperature 0.6), +which can outweigh the memory saved. Keep the KV cache in full precision when +speculation is on if you can. If memory forces quantization, prefer 8-bit, +which perturbs the distribution far less. ### Stochastic acceptance (opt-in) @@ -191,8 +189,8 @@ token-identical. At temperature > 0 it is also a ceiling: however good the draft head, a draft can't match a sampled token more often than the target's own probabilities allow. `--stochastic-mtp` (run/chat/serve) or `stochastic_mtp: true` in the server config lifts that ceiling with rejection -sampling: drafts are sampled and accepted with probability `min(1, p/q)`, -which provably preserves the sampling distribution - output remains a true +sampling: drafts are sampled and accepted with probability `min(1, p/q)`. +This provably preserves the sampling distribution - output remains a true sample from exactly what non-speculative decoding samples from - but tokens are no longer bit-identical to a non-speculative run. Greedy requests are unaffected and stay token-identical. @@ -200,12 +198,11 @@ unaffected and stay token-identical. Measured A/Bs (temp 1.0 unless noted): Qwen3.6-27B coding 73 -> 77% acceptance; Qwen3.6-35B-A3B +2 to +12 points across coding/chat/creative profiles; DeepSeek-V4-Flash at IQ2_XXS is the big winner at 55 -> 69% (+11% -decode throughput) - the lower the trunk precision and the flatter the text, +decode throughput). The lower the trunk precision and the flatter the text, the more exact-match leaves on the table. Long-context chat (ultrachat at 4k depth) gains a smaller +3-5% expected tokens per round. Turn it on when you -sample and want throughput; leave it off and output is exactly what plain -non-MTP decoding produces - MTP does not change output unless stochastic -acceptance is enabled. +sample and want throughput. Leave it off and output is exactly what plain +non-MTP decoding produces. ## The prompt cache @@ -228,7 +225,7 @@ What reuse to expect, per family (tier routing: to the 2048-token checkpoint grid - at a 9k-token history that is ~90% of the prefill. A branch or regenerate restores to the deepest interval boundary below the divergence. Recurrent state cannot rewind, - so the un-restored tail re-prefills; that tail is bounded by the grid, + so the un-restored tail re-prefills. That tail is bounded by the grid, never the whole history. - **Sliding-window models** (gemma-4, gpt-oss; ckpt tier): same shapes as GDN, but turn boundaries are exact rather than grid-snapped once @@ -243,23 +240,23 @@ What reuse to expect, per family (tier routing: Measured on DeepSeek-V4-Flash (87 GB IQ2_XXS, ~8k-token history): cold prefill 44.5 s, identical resend 2.0 s, later turns 1.5-9 s, restart from the SSD tier 1.9 s. The boundary to respect: any edit - to the history is a different sequence and prefills cold - there is - no partial credit for a merely shared prefix on this tier (the same - history with one changed line went back to 43 s). + to the history is a different sequence and prefills cold. There is + no partial credit for a merely shared prefix on this tier - the same + history with one changed line went back to 43 s. - One stated gap: sliding-window models under `--speculative` retain no record of generated tokens (their post-prefill rotating layers decline stores by design), so next-turn reuse there comes from the prefill - boundaries alone; the reply itself re-prefills. + boundaries alone. The reply itself re-prefills. This is the single biggest lever for agent workloads: coding harnesses resend a large, mostly stable system prompt every turn, and multi-turn chat resends the whole history. The optional SSD tier (`gmlx init --disk-cache`, or the `cache:` block in the config) persists entries across restarts and holds more than RAM comfortably -would; entries are evicted by size budget. Configuration keys: +would. Entries are evicted by size budget. Configuration keys: [server-config.md](server-config.md#cache-keys-cache). The cache composes with MTP, and a finished request stores its generated -tokens too: turn N+1 of a conversation warm-starts past the whole of turn N +tokens too. Turn N+1 of a conversation warm-starts past the whole of turn N instead of re-prefilling the previous reply, and a warm hit restores the draft head's state along with the base model's. Details and switches: [server-config.md](server-config.md#speculative-decoding--the-prompt-cache). @@ -279,8 +276,8 @@ existing streams are mid-decode. Prefill runs in 2048-token chunks, and a scheduler that simply alternates one decode step with one chunk lets a long admission starve live streams, because at depth a chunk costs hundreds of decode steps' worth of GPU time. Whether pacing admissions helps is decided -by that same quantity: when a chunk costs a live stream many decode steps -(deep context), stock scheduling starves it and pacing rescues it; when +by that same quantity. When a chunk costs a live stream many decode steps +(deep context), stock scheduling starves it and pacing rescues it. When chunks are cheap (shallow prompts, warm prefix hits), pacing only delays admission, and a delayed admission narrows the decode batch that aggregate throughput comes from. @@ -328,8 +325,8 @@ over-RAM model streams weights from disk. `server.prefill_tick_ms` (default 500, flag `--prefill-tick-ms`, env `GMLX_PREFILL_TICK_MS` read per chunk) bounds that quantum: while streams decode, each chunk is halved until its predicted wall time, taken from the last observed chunk cost, fits the -budget. The two knobs answer different symptoms -- a starved decode batch -needs the ratio, a hitchy stream needs the tick -- and smaller chunks cost +budget. The two knobs answer different symptoms: a starved decode batch +needs the ratio, a hitchy stream needs the tick. Smaller chunks cost a few percent of prefill throughput per halving tier (worst on MoE), so batch jobs that only care about aggregate throughput can set `0`. @@ -346,18 +343,17 @@ Concurrent streams often share a prefix - the same system prompt, or histories restored from the prompt cache. The batch cache holds one copy of that prefix per stream, and a plain batched step re-reads every copy every token. The server detects the sharing from the streams' token ids when a -batch forms or a stream is admitted - cold prompts, cache-restored -histories, and mixes all count - and decodes such batches through a +batch forms or a stream is admitted (cold prompts, cache-restored +histories, and mixes all count). Such batches decode through a shared-prefix cascade kernel that reads the prefix once for the whole -batch, so attention -traffic per step drops from every stream's full context to one prefix plus -each stream's own suffix. Four streams on a 12k-token system prompt decode -about 1.4x faster aggregate; the win grows with prefix length and stream -count, and speculative-verify rounds cascade the same way (their -re-reads are wider, so the saving is larger). On by default and exact -(same numbers as the plain step); `GMLX_CASCADE_SDPA=0` disables it, -`GMLX_CASCADE_MIN_P` (default `1024`) sets the smallest shared prefix -worth routing. +batch, so attention traffic per step drops from every stream's full +context to one prefix plus each stream's own suffix. Four streams on a +12k-token system prompt decode about 1.4x faster aggregate. The win grows +with prefix length and stream count, and speculative-verify rounds +cascade the same way (their re-reads are wider, so the saving is larger). +It is on by default and exact - same numbers as the plain step. +`GMLX_CASCADE_SDPA=0` disables it; `GMLX_CASCADE_MIN_P` (default `1024`) +sets the smallest shared prefix worth routing. ## Sparse attention at depth (opt-in) @@ -371,11 +367,11 @@ are always kept. Attention cost stops growing with depth. This is lossy, which is why it is opt-in. Measured on a Llama-3.1-8B Q6_K at 32k depth with the default 2048-token budget: mean KL divergence against -full attention of 0.008 - the same order as the quantization noise of a Q6 -checkpoint - at 1.40x end-to-end decode single-stream, and 1.8x aggregate -over the exact cascade route at three streams on a 26k shared prompt. The -index is good at finding the pages a query actually needs (needle lookups -deep in the context keep working); the always-resident sink and recency +full attention of 0.008, the same order as the quantization noise of a Q6 +checkpoint, at 1.40x end-to-end single-stream decode. At three streams on a +26k shared prompt it runs 1.8x aggregate over the exact cascade route. The +index is good at finding the pages a query actually needs - needle lookups +deep in the context keep working - and the always-resident sink and recency pages keep the failure mode graceful when it is not. `GMLX_SPARSE_K` sets the budget (default `2048` tokens: larger tracks full @@ -388,7 +384,7 @@ attention. The route only engages on architectures whose quality has been measured, because the property it trades on is architectural: pure full-attention stacks (llama-family) concentrate decode attention into a small key set, -while measured SWA hybrids do not - on gemma-4's global layers even an +while measured SWA hybrids do not. On gemma-4's global layers, even an exact top-k oracle at the default budget lands an order of magnitude outside the acceptable divergence band, so gemma-4 is deliberately excluded and runs full attention regardless of the switch. @@ -441,7 +437,7 @@ GB of headroom. MLX keeps freed GPU buffers in a wired reuse pool (the buffer cache). That is normally free performance, but deep-context serving of a near-RAM-size model retains multi-gigabyte prefill transients in the pool, and the accumulated -wired footprint can exhaust free pages -- the failure is a system freeze, not +wired footprint can exhaust free pages. The failure is a system freeze, not a clean error. The server therefore bounds the cache automatically when the biggest configured model uses more than ~60% of the GPU working set, capping it at a quarter of the remaining slack (clamped to 4-12 GiB) and logging one @@ -449,8 +445,8 @@ it at a quarter of the remaining slack (clamped to 4-12 GiB) and logging one unbounded cache -- the policy never engages there. Override it explicitly when needed: the `server.cache_limit_gb` config key or -the `GMLX_CACHE_LIMIT_GB` env (env wins). A GiB value pins the limit -- -benchmarks should pin it for reproducibility; a negative value (or env +the `GMLX_CACHE_LIMIT_GB` env (env wins). A GiB value pins the limit +(benchmarks should pin it for reproducibility); a negative value (or env `off`/`none`/`unlimited`) forces an unbounded cache and suppresses the auto policy; `0` disables buffer caching entirely. A bounded cache trades a little allocator churn for a bounded footprint: transients up to the limit are diff --git a/docs/talk.md b/docs/talk.md index a120156..12613e3 100644 --- a/docs/talk.md +++ b/docs/talk.md @@ -34,7 +34,7 @@ mode) whenever you configure the two together. If something is missing at startu On first run, two small files download into `~/.cache/gmlx/talk/`: the sherpa-onnx keyword-spotting bundle and the silero voice-activity model, a few MB -together. macOS asks for microphone permission once - and the prompt names +together. macOS asks for microphone permission once, and the prompt names your *terminal* (Terminal, iTerm2, your IDE), not gmlx, because macOS grants the mic to the app you launched from. Allow it. (Voice sessions started from the menu-bar login item are the exception: those prompt as "gmlx" - see @@ -66,7 +66,7 @@ listening for "hey assistant" ``` Useful while it runs: Space stops the assistant mid-sentence, and typing at any -time sends a text message instead of speaking; the full set is in +time sends a text message instead of speaking. The full set is in [keys and slash commands](#keys-and-slash-commands). Try voices live: ```text @@ -87,7 +87,7 @@ than failing. [assistant](assistant.md). The model can call tools mid-turn (the standard OpenAI tool loop, run against this same server), and the conversation gains long-term memory. Tools come from MCP servers you configure; memory is a local store built -on the server's own embeddings. This section is the voice-flavored tour; +on the server's own embeddings. This section is the voice-flavored tour. [assistant.md](assistant.md) is the full reference, and the same engine also drives `gmlx chat --assistant` and served assistant ids. @@ -132,7 +132,7 @@ assistant: enabled: true ``` -Tool calling needs a model that is competent at it; the config's Qwen3.6-27B is +Tool calling needs a model that is competent at it. The config's Qwen3.6-27B is the recommended class ([model picks](assistant.md#the-tool-loop)). Then: ```sh @@ -168,7 +168,7 @@ assistant: Ana's birthday is March 12th. What got stored is a distilled fact ("sister Ana, birthday March 12"), not a transcript. The distillation runs in the background after each turn, off the -voice path, so it adds no latency; the rules (at most three durable facts, +voice path, so it adds no latency. The rules (at most three durable facts, restatement replaces) are in [assistant.md#memory](assistant.md#memory). Two honest caveats. Tool rounds cost time - a model turn plus the call, each - @@ -177,7 +177,7 @@ still interrupts cleanly: the loop commits what you heard and never leaves a half-finished tool round in the history. `/memory` inspects and edits the store from inside a session -([keys and slash commands](#keys-and-slash-commands)); the menu-bar voice +([keys and slash commands](#keys-and-slash-commands)). The menu-bar voice session exposes the same store through its "Show memory" and "Clear memory" items. The store is shared with `gmlx chat --assistant` - details and the on-disk location in [assistant.md#memory](assistant.md#memory). @@ -204,7 +204,7 @@ memories; `forget ID` and `clear` manage them), `/devices`, `/help`, `/quit`. Everything lives in a top-level `talk:` block of the same YAML the server reads. It configures the client, so it is not under `server:`. Most keys have a flag mirror -(`vad.pre_roll_ms` and `push_to_talk_modifier` are config-only); precedence is +(`vad.pre_roll_ms` and `push_to_talk_modifier` are config-only). Precedence is defaults, then YAML, then flags. `--list-devices` and `--list-voices` enumerate the device and voice values. @@ -235,7 +235,7 @@ talk: The assistant itself -- the top-level `assistant:` block with tool servers, memory settings, and their defaults -- is documented in -[assistant.md](assistant.md); it is shared with `gmlx chat --assistant` and +[assistant.md](assistant.md). It is shared with `gmlx chat --assistant` and `server.assistants`, so it does not live under `talk:`. Talk uses it whenever `talk.brain: assistant` is set, and everything there applies as-is: MCP tool-name prefixing, the degrade-to-warning behavior when a tool server or the @@ -247,18 +247,19 @@ tool-name prefixing, the degrade-to-warning behavior when a tool server or the When the tracked server advertises STT and TTS, the macOS menu bar app (`gmlx launch menubar`, raised automatically by a background `serve`) shows a "Talk to " item, named after `talk.model` or the server's default model. -Clicking it starts a voice session inside the menu bar app, no terminal window: the -bar icon changes to show the state (a microphone while listening, a thought bubble -while the model thinks, a speaker while it talks, a muted-speaker while the mic is -off), and the menu offers Stop speaking, Mute mic, and End voice chat. Show -transcript opens a floating panel with the running conversation text. A Volume -slider under the session controls scales the voice (and the chimes) relative -to the system output volume - it applies mid-sentence while dragging, and the -setting persists across sessions. Mic input has no gain control on purpose: -software input gain would shift the endpointing and wake-word thresholds and -clip loud speech; use the macOS Sound settings input level instead. - -All settings come from the YAML `talk:` block; push-to-talk and text modes fall +Clicking it starts a voice session inside the menu bar app, no terminal window. +The bar icon changes to show the state (a microphone while listening, a thought +bubble while the model thinks, a speaker while it talks, a muted-speaker while +the mic is off), and the menu offers Stop speaking, Mute mic, and End voice +chat. Show transcript opens a floating panel with the running conversation +text. A Volume slider under the session controls scales the voice (and the +chimes) relative to the system output volume. It applies mid-sentence while +dragging, and the setting persists across sessions. Mic input has no gain +control on purpose: software input gain would shift the endpointing and +wake-word thresholds and clip loud speech. Use the macOS Sound settings input +level instead. + +All settings come from the YAML `talk:` block. Push-to-talk and text modes fall back to wake mode, since there is no keyboard. A "Talk in a terminal" item opens `gmlx talk` in iTerm2 when it is running, otherwise the default terminal handler. No AppleScript is involved, so there is no automation-permission popup. @@ -301,14 +302,14 @@ inference). Holding Globe as a modifier suppresses macOS's own Globe-key action, so your "Press Globe key to" setting (emoji, dictation, input source) keeps working for bare presses - nothing to reconfigure. -(A bare double-press of Globe was considered and dropped: current macOS +(A bare double-press of Globe was considered and dropped. Current macOS routes the solo Globe press to the system shortcut handler without posting an event that session event taps can see - only raw HID sees it - so double-press stays available for the system's own dictation shortcut.) The choice persists across launches. On startup the app re-arms it only -after a silent permission check - if the grant is missing (denied, or -silently dropped by an app-stub re-sign after an interpreter upgrade) a +after a silent permission check. If the grant is missing (denied, or +silently dropped by an app-stub re-sign after an interpreter upgrade), a "not active - needs permission" note appears under the toggle and nothing prompts until you flip it again. Granting access in System Settings while the bar is running is picked up within a few seconds and the hotkey arms @@ -324,7 +325,7 @@ grants then attach to the terminal app, not gmlx. ## Remote server and scripting `--base-url http://host:8080/v1` (with `--api-key` if the server has one) points the -client at a server elsewhere. STT and TTS then run on that machine; only the mic and +client at a server elsewhere. STT and TTS then run on that machine. Only the mic and speaker are local. Without `--base-url`, `talk` targets the managed local server and starts it when down (`--no-start` disables that). @@ -349,7 +350,7 @@ mind", and the like) acknowledges and goes back to sleep instead of starting a turn -- so ", stop" kills a long reply by voice. Space and Esc do the same from the keyboard, and stop playback within about 150 ms. -Only wake-phrase scoring runs during a reply; full transcription of the open +Only wake-phrase scoring runs during a reply. Full transcription of the open mic stays gated (playback would otherwise be re-transcribed), so `vad` and `ptt` modes remain half-duplex, keyboard-interrupt only. One caveat: there is no protection against the assistant *speaking* the wake phrase -- if a reply From de762ce3b7c9b81ac26897b131667e5bd1caf127 Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 22:40:50 -0700 Subject: [PATCH 15/19] docs(style): de-bridge vlm, getting-started, launch --- docs/getting-started.md | 38 +++++++++++++++--------------- docs/launch.md | 52 +++++++++++++++++++++-------------------- docs/vlm.md | 12 +++++----- 3 files changed, 52 insertions(+), 50 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index fc0cc15..bf0fec6 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -40,7 +40,7 @@ on every optional feature. `uv tool upgrade gmlx` (or `pipx upgrade gmlx`). `ffmpeg` is the one dependency no Python installer can supply. It is needed -only to decode audio uploads and to encode mp3/flac/opus; skip it if you are +only to decode audio uploads and to encode mp3/flac/opus. Skip it if you are not using voice. To install less, name the extras you want instead of `all` - `gmlx[chat]` is @@ -61,8 +61,8 @@ run `source ~/gmlx/.venv/bin/activate` in each new terminal. A no such step. The `mlx-kquant` dependency (the Metal kernels) arrives -as a prebuilt wheel from PyPI on macOS 26 and newer; on older macOS versions -the install builds it from source - the Command Line Tools requirement from +as a prebuilt wheel from PyPI on macOS 26 and newer. On older macOS versions +the install builds it from source: the Command Line Tools requirement from [What you need](#what-you-need), and a few minutes of compile time. Tab completion is worth the one line: add `eval "$(gmlx completion zsh)"` to @@ -86,14 +86,14 @@ Every extra runs on any supported Python, 3.11 through 3.14. Adding one later depends on how gmlx was installed: `pip install 'gmlx[X]'` in a venv, but a `uv tool` or pipx environment is owned by its installer and -takes a different command. These docs write the pip form for brevity; to get +takes a different command. These docs write the pip form for brevity. To get the right one for your machine, run `gmlx init`, which offers to install what it needs, or read it off any "not installed" message - both name the command that works where you are. -`vlm` and `embeddings` exist as empty back-compat extras; multimodal loading and the +`vlm` and `embeddings` exist as empty back-compat extras: multimodal loading and the embeddings endpoint are part of the core install. mlx-audio itself already arrives -with the core install; the `tts` extra pins it and adds the +with the core install. The `tts` extra pins it and adds the grapheme-to-phoneme front-end the default Kokoro voice needs. That front-end also wants a spaCy English pipeline, which gmlx fetches from Hugging Face on first use - no separate install step. @@ -128,7 +128,7 @@ already using the settings the model's authors recommend. `gmlx profiles` prints the table. Two words you will meet for adjusting them: built-in *intents* (`@coding`, `@creative`, ...) work on any model with no config, while *profiles* are your own named setting bundles, defined later in a server config's -`profiles:` block; both are addressed the same way, `model@NAME` or +`profiles:` block. Both are addressed the same way, `model@NAME` or `--profile NAME`. ## Pick a model for your Mac @@ -136,7 +136,7 @@ are your own named setting bundles, defined later in a server config's Suggestions by machine memory, all instruct models that load end to end here. A quick key to the quant names you will see everywhere: the Q-number is roughly bits per weight, so Q4 files are smaller and slightly lossier, Q6/Q8 bigger and -closer to the original; when and why it matters is in +closer to the original. When and why it matters is in [performance.md](performance.md#choosing-a-quant-for-speed). | Mac RAM | Suggestion | Notes | @@ -146,7 +146,7 @@ closer to the original; when and why it matters is in | 64 GB | Qwen3.6-27B (Q6_K, ~23 GB) | strong general model; also the tool-calling pick | | 96+ GB | Qwen3.6-35B-A3B (Q6_K) or gpt-oss-120b (MXFP4, ~63 GB) | MoE models: big-model quality, small-model decode cost | -These sizes leave room for the KV cache at everyday context lengths; +These sizes leave room for the KV cache at everyday context lengths. [Will it fit?](#will-it-fit) below has the per-token arithmetic and the `--kv-bits` lever for long sessions. @@ -160,16 +160,16 @@ gmlx validate hf:unsloth/Qwen3.6-27B-GGUF gmlx pull hf:unsloth/Qwen3.6-27B-GGUF/Qwen3.6-27B-Q6_K.gguf --to ~/models ``` -`--to` says where the file lands; it is required until a config exists (next -section), after which bare `pull` lands files in your model directory, registers -them in the config, and a running server serves them immediately. +`--to` says where the file lands, and it is required only until a config exists +(next section). After that, bare `pull` lands files in your model directory, +registers them in the config, and a running server serves them immediately. `gmlx sync-models` reconciles in bulk after hand-moving or deleting files. These are multi-gigabyte downloads - minutes to an hour depending on your connection - with progress, rate, and resume built in. `validate` accepts a repo, a folder, a pasted browser link, or an exact file. Given a repo it lists every quant variant as a ready-to-paste ref. K-quant, legacy, and -IQ files all load; in the rare case a file uses a codec with no kernel (the +IQ files all load. In the rare case a file uses a codec with no kernel (the ternary TQ types, for instance), the verdict names it so you can pick another variant. Uniform K-quant files also decode faster than heavily mixed ones. @@ -212,9 +212,9 @@ that walks through: 6. Idle unload (how long an unused model stays resident) and a request timeout. 7. Where to write the file, with a preview before anything is saved. -Prefer flags? `gmlx init --models-dir ~/models` scaffolds non-interactively; every -wizard choice has a flag equivalent (`--with-stt`, `--disk-cache`, `--default-model`, -and so on). +Prefer flags? `gmlx init --models-dir ~/models` scaffolds non-interactively, and +every wizard choice has a flag equivalent (`--with-stt`, `--disk-cache`, +`--default-model`, and so on). The config lands at `~/.config/gmlx/gmlx.yaml`. It is one YAML file with a `server:` block (port, model directories, services), a `models:` block (one entry @@ -241,7 +241,7 @@ the resident models, with unload, restart, log, and config-editing controls. The server speaks the OpenAI API (plus Anthropic and OpenAI Responses on the same port). The `model` field is whatever id `init` printed for your file - auto-named -ids carry the quant tag (`qwen3-0.6b-q4`, `qwen3.6-27b-q6`); `gmlx list` shows +ids carry the quant tag (`qwen3-0.6b-q4`, `qwen3.6-27b-q6`). `gmlx list` shows them, and an unknown id gets a 404 listing the valid ones. With the small model from the walkthrough above: @@ -303,7 +303,7 @@ gmlx launch open-webui ``` `launch` starts your gmlx server if needed, wires Open WebUI to it, runs it on -port 3000, and prints the URL to open. Chat works immediately; if your server +port 3000, and prints the URL to open. Chat works immediately. If your server also runs embeddings, speech-to-text, or text-to-speech, document upload and voice light up too. Details, including a no-login single-user setup: [launch.md](launch.md#open-webui). @@ -326,7 +326,7 @@ gmlx service uninstall # remove the login item macOS only. `service install` accepts the same options as `serve`. The menu bar runs as the launchd agent (so its permission prompts attribute to gmlx) -and starts the recorded server once per login; `--no-autostart` leaves the +and starts the recorded server once per login. `--no-autostart` leaves the server to its Start menu item, and `--headless` installs a server-only agent for GUI-less machines (that one restarts on crash and is stopped with `service uninstall`). diff --git a/docs/launch.md b/docs/launch.md index dbc17b9..89097bd 100644 --- a/docs/launch.md +++ b/docs/launch.md @@ -51,7 +51,7 @@ names the model and its on-disk size while it loads, and the model is hot for th tool's first turn. When nothing is preloaded, the server answers in about a second and the model loads lazily on the first request, which makes that first turn slower. -There is no fixed timeout; only the server process dying is a hard failure. Press +There is no fixed timeout. Only the server process dying is a hard failure. Press Ctrl-C to stop waiting (the server keeps starting in the background). With no config anywhere, launch prints `gmlx init` guidance and starts nothing. @@ -67,14 +67,14 @@ anywhere, launch prints `gmlx init` guidance and starts nothing. ## Choosing the model `--model ID` picks which served model the tool uses. Without it, the tool gets the -server's default-marked model. The id half is validated against the served list; an +server's default-marked model. The id half is validated against the served list. An `id@profile` form (`--model qwen3.6-27b@coding`) runs every request from the tool at that profile's sampling, validated by the server. When you pass `--model`, launch also asks the server to keep that model resident through its idle-TTL reaper, so a long coding session's model is not idle-unloaded mid-use (which would force a cold reload on the next turn). This is not a full pin: -under memory pressure the pool can still evict it. The request is fire-and-forget; +under memory pressure the pool can still evict it. The request is fire-and-forget: the server warm-loads in the background while the tool execs. `gmlx ps` shows the model as `kept`, and `POST /unload {model}` releases it. Pass `--no-keep` to opt out. @@ -111,8 +111,8 @@ on PATH. launch exports: Claude Code on local models is prefill-heavy: it sends a very long system prompt (tens of thousands of tokens) and frequently rewrites its request prefix through -context compaction and tool-result injection, so KV-prefix reuse across requests is -limited and turn latency is dominated by prompt processing. Serve with the prompt +context compaction and tool-result injection. KV-prefix reuse across requests is +therefore limited, and turn latency is dominated by prompt processing. Serve with the prompt cache enabled (`cache:` in the config, see [server-config.md](server-config.md#cache-keys-cache)) to soften repeated prefixes, and prefer a model and machine with strong prefill throughput. @@ -143,8 +143,8 @@ A model is required (`inference.model`). hermes refuses models with less than 64k context at startup, so serve it a model whose context window is at least 64k tokens. The window comes from the model's GGUF -metadata and is not configurable server-side; the `max_kv_size` load key only caps -the rolling KV cache, it does not change the advertised window. +metadata and is not configurable server-side. The `max_kv_size` load key caps only +the rolling KV cache, not the advertised window. ### goose @@ -152,7 +152,7 @@ Block's goose. Merge style: launch merges the non-secret pointer keys into `~/.config/goose/config.yaml` (existing keys preserved) and also exports the same values as environment variables at exec, which take precedence in goose. A model is required (`GOOSE_MODEL`). The API key travels as `OPENAI_API_KEY` in the exec -environment only; it is never written into the YAML, where it could clobber a real +environment only. It is never written into the YAML, where it could clobber a real OpenAI credential. ### aichat @@ -171,7 +171,7 @@ darrenburns/elia, a chat TUI (not a coding harness). Injection style: launch wri fresh `config.toml` under `~/.config/gmlx/elia-xdg` and injects it via `XDG_CONFIG_HOME`. Each served id becomes an OpenAI-compatible litellm model. Requires elia 1.x or newer (older builds ignore -custom endpoints); upgrade with `pipx upgrade elia-chat`. +custom endpoints). Upgrade with `pipx upgrade elia-chat`. ### open-webui @@ -181,20 +181,21 @@ service rather than a terminal client. Install it separately: 3.13). Environment style, no config file: launch exports `OPENAI_API_BASE_URL`, -`OPENAI_API_KEY`, `ENABLE_OLLAMA_API=false`, and `DATA_DIR`, runs it on port 3000 -(passed as `serve --port`, since `open-webui serve` ignores the `PORT` variable and -would otherwise collide with the gmlx server on 8080; if the gmlx server itself is -bound to 3000, Open WebUI bumps to 3001), and prints the URL to -open. Chat history and its sqlite database land at `DATA_DIR` on the host filesystem -(default `~/.open-webui`; `--config-path` overrides), not in a Docker volume. +`OPENAI_API_KEY`, `ENABLE_OLLAMA_API=false`, and `DATA_DIR`, runs it on port 3000, +and prints the URL to open. The port is passed as `serve --port`, since +`open-webui serve` ignores the `PORT` variable and would otherwise collide with +the gmlx server on 8080 (if the gmlx server itself is bound to 3000, Open WebUI +bumps to 3001). Chat history and its sqlite database land at `DATA_DIR` on the +host filesystem (default `~/.open-webui`; `--config-path` overrides), not in a +Docker volume. launch points Open WebUI's document-RAG embedder back at the gmlx server (`RAG_EMBEDDING_ENGINE=openai`) instead of the default local HuggingFace download, so -it boots cleanly without fetching or caching an embedder. Chat works immediately; run +it boots cleanly without fetching or caching an embedder. Chat works immediately. Run the server with `--embeddings` and document RAG works too (see [rag.md](rag.md)). If the server also advertises a reranker (started with `--rerank`), launch turns on Open WebUI's hybrid search and points its external reranker at the server's `/v1/rerank` -(`RAG_RERANKING_ENGINE=external`); reranking only runs under hybrid search. +(`RAG_RERANKING_ENGINE=external`). Reranking only runs under hybrid search. Audio is capability-gated: when the server advertises STT and TTS in `/v1/models` (run it with `--stt` / `--tts`), launch also wires Open WebUI's audio engines @@ -227,19 +228,20 @@ and refuses (once) if the file changed on disk while you were editing; Save & Reload validates first, then saves and triggers the running server's config reload in one step. Open in Editor hands the file to your default text editor instead. The item is there whenever the bar knows a config: the tracked -server's own file or, with everything stopped, the default config location - -fixing the config is usually why you are there. +server's own file or, with everything stopped, the default config location. +Fixing the config is usually why you are there. -Like `serve`, it detaches by default; pass `-f` / `--foreground` to run the event -loop in place, and `--stop` to quit a detached monitor from the CLI. One menu bar per machine, deduplicated via a pidfile: a second `serve` -on any port, or a manual `launch menubar`, is a no-op. With no explicit target it -tracks the primary server (the single managed one, else `127.0.0.1:8080`), following -it as servers come and go; pass `--url`, `--host`, or `--port` to pin it to one. +Like `serve`, it detaches by default. Pass `-f` / `--foreground` to run the event +loop in place, and `--stop` to quit a detached monitor from the CLI. One menu bar +per machine, deduplicated via a pidfile: a second `serve` on any port, or a manual +`launch menubar`, is a no-op. With no explicit target it tracks the primary server +(the single managed one, else `127.0.0.1:8080`), following it as servers come and +go. Pass `--url`, `--host`, or `--port` to pin it to one. It reads the API key from the managed server's own `server.api_key`, or takes `--api-key` for a server whose config it cannot see. A key-protected server it has no key for shows as up with a key-required note, never as down. macOS only (it needs -`rumps`, a default dependency there); on Linux or over SSH it prints a one-line +`rumps`, a default dependency there). On Linux or over SSH it prints a one-line notice. `--interval S` sets the poll interval (default 4 seconds). When the tracked server also advertises STT and TTS, the menu gains a voice-chat diff --git a/docs/vlm.md b/docs/vlm.md index 51214f2..be7a338 100644 --- a/docs/vlm.md +++ b/docs/vlm.md @@ -5,7 +5,7 @@ A multimodal model in GGUF is two files: 1. the LLM GGUF, a normal text architecture, K-quantized as usual; and 2. a separate `mmproj` GGUF (`general.architecture = "clip"`) holding the vision (and/or audio) encoder plus the cross-modal projector. It ships float - or Q8_0; a quantized mmproj's encoder matmuls run on the K-quant kernels like + or Q8_0. A quantized mmproj's encoder matmuls run on the K-quant kernels like the text tower, while float weights stay native. `gmlx` pairs them with `--mmproj` and loads an [mlx-vlm](https://github.com/Blaizzy/mlx-vlm) @@ -14,7 +14,7 @@ encoders run in float, and the image processor + chat template (including the per-family image/audio marker tokens) are synthesized from the two GGUFs' metadata. `--hf-source` overrides only when a file omits something. -Vision and audio support is included in the base install; no extra is needed. +Vision and audio support is included in the base install, no extra needed. Where mmproj files come from: llama.cpp-style multimodal GGUF repos ship them as `mmproj-*.gguf` siblings of the LLM GGUF in the same Hugging Face repo. @@ -35,7 +35,7 @@ gmlx serve model.gguf --mmproj mmproj.gguf --port 8080 ``` `--resize-shape N|WxH` resizes images before encoding, setting the soft-token count -that dominates prefill cost. Unset, images encode at native resolution; a +that dominates prefill cost. Unset, images encode at native resolution. A square cap like `448` (or an explicit `672x448`) is a typical choice when prefill cost matters. Audio input (`--audio`, `/audio`) works where the mmproj carries an audio encoder (gemma-4 omni, Qwen3-Omni). Vision-only mmprojs @@ -58,7 +58,7 @@ families that share one. An unsupported pairing fails loudly at load with both n Qwen2-VL / Qwen2.5-VL mmprojs (`qwen2vl_merger`) are not supported yet. The load fails up front with the family named. LLaVA's image processor isn't -synthesized from the GGUF; pass the checkpoint's HF id (e.g. +synthesized from the GGUF. Pass the checkpoint's HF id (e.g. `--hf-source llava-hf/llava-1.5-7b-hf`) so the processor loads from there. On LLaVA the loader reports two unfilled parameters (`vision_tower.[...].post_layernorm.{weight,bias}`). This is expected: llama.cpp's @@ -86,11 +86,11 @@ consumer. shows the same degradation on the same file. This is separate from the already-merged `image_std` mean/std fix (llama.cpp #13208). Current community mmproj files carry that fix. GGUF Pixtral vision quality is capped until a - re-converted mmproj appears; the text tower is unaffected. + re-converted mmproj appears. The text tower is unaffected. ## Caveats -- A multimodal request needs the mmproj at load; the bare LLM GGUF still loads and +- A multimodal request needs the mmproj at load. The bare LLM GGUF still loads and runs as a plain text model (the vision side is simply absent). - Adapters (`--adapter`) don't combine with `--mmproj` yet -- live GGUF LoRA is text-path-only and errors loudly. From dbbb62e3bd7f1132404cb0c4e3eabaf52536f7bc Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 22:43:20 -0700 Subject: [PATCH 16/19] docs(style): de-bridge remaining docs - split prose semicolons, keep reference-table idiom --- docs/lora.md | 4 ++-- docs/python.md | 6 ++--- docs/server-config.md | 44 ++++++++++++++++++------------------ docs/serving-architecture.md | 2 +- docs/troubleshooting.md | 14 ++++++------ 5 files changed, 35 insertions(+), 35 deletions(-) diff --git a/docs/lora.md b/docs/lora.md index a270fae..85ab46e 100644 --- a/docs/lora.md +++ b/docs/lora.md @@ -98,7 +98,7 @@ Pushing further overfits: validation loss climbs and greedy decoding can fall in "me hearty, me hearty..." loops. Turn `--iters` down or add data to taste. `--num-layers` and `--rank` trade capacity for memory: more adapted layers and a higher rank fit more behavior but cost more training memory. The defaults (8 layers, -rank 8) are a sensible start. Note the walkthrough targets dense bases; on a MoE +rank 8) are a sensible start. Note the walkthrough targets dense bases. On a MoE base the default adaptation keys are untested. Full flag table: the [train section of docs/cli.md](cli.md#gmlx-train). @@ -162,5 +162,5 @@ scaling semantics (`delta = (alpha / rank) * B * A`). That buys interop both way - One adapter per resident entry, fixed at load: switching adapters means addressing a different model id (see the config example above), not a per-request parameter. -- The adapter must match the base architecture (checked at load); matching the +- The adapter must match the base architecture (checked at load). Matching the exact base finetune is your responsibility. diff --git a/docs/python.md b/docs/python.md index 433be69..a0ad094 100644 --- a/docs/python.md +++ b/docs/python.md @@ -1,6 +1,6 @@ # Python API -The CLI is the primary interface; this page is for embedding gmlx in your +The CLI is the primary interface. This page is for embedding gmlx in your own Python. The stable surface is exactly what the package root exports (`gmlx.__all__`), and all of it is documented here. @@ -124,7 +124,7 @@ pf.arch, pf.shards, pf.codec_histogram, pf.n_tensors, pf.n_params `preflight(gguf_path, arch=None)` validates a GGUF before committing to a load: it discovers shards, histograms the tensor codecs, refuses unsupported ones by name, and gates on the architecture. It reads only the GGUF header, so -it stays cheap on multi-GB files. `load_model` runs it internally; call it +it stays cheap on multi-GB files. `load_model` runs it internally. Call it yourself to vet a file first (the CLI equivalent is `gmlx validate`). Failures raise one of two exceptions, from `preflight` or `load_model` alike: @@ -176,7 +176,7 @@ loads through `load_model`; non-GGUF paths fall through untouched, so one `mlx_lm.server` process can mix GGUF and ordinary MLX checkpoints. GGUF requests are pinned to mlx-lm's validated sequential path (no batching), and adapters and draft models are not wired on this route. Use it to add GGUF -support to an existing `mlx_lm.server` deployment; `gmlx serve` is the +support to an existing `mlx_lm.server` deployment. `gmlx serve` is the full-featured server. ## Quantized modules diff --git a/docs/server-config.md b/docs/server-config.md index 69ee03d..a0d07b7 100644 --- a/docs/server-config.md +++ b/docs/server-config.md @@ -67,7 +67,7 @@ curl localhost:8080/v1/chat/completions -d '{ }' ``` -Sampling defaults come from the model's family card automatically; add +Sampling defaults come from the model's family card automatically. Add `@coding` / `@instruct` / `@creative` / `@reasoning-low|-medium|-high|-max` to any id to switch operating point (`"model": "qwen3.6-27b@coding"`). `gmlx profiles` prints the table. @@ -160,7 +160,7 @@ Unknown keys in the structural namespaces (top-level, `server`, `profiles`, `models`, `rules`, `discover`) are a hard error: a typo like `pinned:` for `pin:` fails the load instead of silently no-op'ing. To see the whole schema with every key and its effective default, run `gmlx serve --print-config` -(optionally with `--config FILE` / `--models-dir DIR` / a positional GGUF); it +(optionally with `--config FILE` / `--models-dir DIR` / a positional GGUF). It resolves the config for that start mode, prints it as YAML, and exits without starting the server. @@ -217,21 +217,21 @@ server: ``` Paths in `model_dirs` expand `~`/`$VAR`. A relative `path`/`mmproj`/`draft_gguf` -on a model is searched against `model_dirs` in order (first existing wins); a +on a model is searched against `model_dirs` in order (first existing wins). A miss raises, listing the roots searched. One root lets every model entry use a bare filename. `token_queue_timeout_s` bounds how long the request loop waits for the next token. On timeout the server cancels the in-flight generation (freeing the GPU -work) and returns an error to the client; a streaming request gets a final +work) and returns an error to the client. A streaming request gets a final `data: {"error": ...}` event. The failure is recorded as `last_error` in `/v1/metrics` and logged as a `[req] ... FAILED ...` line. Unset, gmlx -defaults it to 1800 seconds (an exported `MLX_VLM_TOKEN_QUEUE_TIMEOUT` wins); +defaults it to 1800 seconds (an exported `MLX_VLM_TOKEN_QUEUE_TIMEOUT` wins). mlx-vlm's own 600-second default is shorter than a deep-context dense prefill. The timeout triggers mainly on a very long prefill that hasn't emitted its -first token yet (a big prompt on a large or over-RAM model); raise it for +first token yet (a big prompt on a large or over-RAM model). Raise it for those, or set `0` to wait indefinitely. The value drives mlx-vlm's -`MLX_VLM_TOKEN_QUEUE_TIMEOUT`; the config is authoritative for a server it +`MLX_VLM_TOKEN_QUEUE_TIMEOUT`. The config is authoritative for a server it starts. `prefill_step_size` sets the prefill chunk size, in tokens, for every model @@ -256,7 +256,7 @@ past a deadline (a prompt already being prefilled is bounded by pacing itself, and time blocked by capacity rather than pacing does not age toward the deadline). A numeric value pins static pacing: a prefill chunk is admitted only after the decode -batch has received that multiple of the chunk's GPU time; live streams then +batch has received that multiple of the chunk's GPU time. Live streams then keep ~half throughput during admissions at `1.0`, while a waiter's time-to-first-token stretch compounds with queue depth (each waiter also waits out the throttled prefill of everyone ahead of it) and delayed @@ -277,7 +277,7 @@ is halved until its predicted wall time -- the last observed chunk cost scaled to the tier -- fits this budget (default 500 ms, floored at `GMLX_PREFILL_MIN_STEP` tokens). Smaller chunks lose some weight amortization, so total prefill throughput under load drops a few percent per -halving tier (worst on MoE); set `0` for batch-job serving where per-stream +halving tier (worst on MoE). Set `0` for batch-job serving where per-stream latency does not matter. Inert whenever nothing is decoding, so single-stream time-to-first-token is untouched. Also available as `--prefill-tick-ms` on `serve` (the flag wins over the config) or an @@ -287,8 +287,8 @@ cycle, the tick sets the stall quantum. `cache_limit_gb` caps MLX's buffer cache (the wired pool of freed GPU buffers kept for reuse). Left `null`, the server bounds it automatically only when -the biggest configured model leaves little GPU working-set slack -- the -deep-context safety case; see +the biggest configured model leaves little GPU working-set slack, the +deep-context safety case. See [performance.md](performance.md#the-mlx-buffer-cache-at-deep-context) for the policy, the `GMLX_CACHE_LIMIT_GB` env override (env wins over this key), and the explicit-unlimited escape. @@ -311,8 +311,8 @@ preflights credential-less by spec, so a browser client holding a key still works, and the actual request authenticates as usual. `server.api_key` is the sole server-side key source. There is no -`serve --api-key` flag and no `GMLX_API_KEY` server fallback; this is a -deliberate simplification, one key in one file, which the lifecycle tools and +`serve --api-key` flag and no `GMLX_API_KEY` server fallback. This is a +deliberate simplification: one key in one file, which the lifecycle tools and the menu bar can also read. The runfile records only whether a key is set, never the key. @@ -362,7 +362,7 @@ gmlx ships those recommendations as data: each model's family is detected from its GGUF header (`general.architecture`) at registration/scan, its base group becomes the lowest sampling layer, and the intents become addressable profiles. `gmlx profiles` prints this table live (add a model id to see one -model fully resolved); values are cited to the primary model cards in +model fully resolved). Values are cited to the primary model cards in `gmlx/profiles.py`: | family | GGUF arches | base (general use) | family intents | @@ -415,7 +415,7 @@ Sampler semantics (matching unpatched mlx_lm / mlx-vlm): a `top_p` or `min_p` of `0` means *disabled* (no filter), not "keep only the argmax" - so `top_p: 0` is a no-op, exactly as on the stock server. When only `top_p` is set (no `top_k`), the nucleus is bounded to the top 1024 candidates so the sort stays -batched; on a very flat distribution the tail past rank 1024 is dropped. +batched. On a very flat distribution the tail past rank 1024 is dropped. Detection reads only the GGUF header (cached across runs in `~/.cache/gmlx/header-meta.json`, keyed by mtime+size). An explicit @@ -466,7 +466,7 @@ profiles: into the tokenizer, so unlike `sampling`/`system` it is load-affecting: two ids on the same GGUF under different templates are distinct resident entries. The value is an inline Jinja string or a path to a `.jinja`/`.txt` file. It -applies to text and native-head/assistant MTP models; a VLM keeps its +applies to text and native-head/assistant MTP models. A VLM keeps its mmproj-synthesized processor template. `chat_template_kwargs` passes extra variables to the template's @@ -477,7 +477,7 @@ by default in those templates but a must-enable for agent / tool-use loops, which depend on the model seeing its own prior reasoning. Unlike `chat_template`, it is applied per request (not baked into the tokenizer), so it is not load-affecting. A client may also send `chat_template_kwargs` on -the request body (OpenAI-extension style); request keys win over the +the request body (OpenAI-extension style). Request keys win over the profile's. ```yaml @@ -586,7 +586,7 @@ only handle one sequence clamps any larger value, since exceeding it raises rather than running slowly. A batch that grows past the cap finishes in plain decode. The drafter stays -loaded and untouched, and the next batch to form re-evaluates; there is no +loaded and untouched, and the next batch to form re-evaluates. There is no mid-flight switch back, so a continuously busy batch keeps decoding plain until it drains. `GMLX_MTP_WIDTH_CAP` overrides every model at once (set it to `0` to measure a model uncapped) and `--speculative-width-cap` does the same @@ -683,8 +683,8 @@ aliases: coder: qwen3.6-27b@qwen-coder # a preset: the 27B with the coder profile baked in ``` -An alias name must not contain `@` and must not collide with a model id; its -target id (and profile, if any) must exist. Validated at load. +An alias name must not contain `@` and must not collide with a model id, and +its target id (and profile, if any) must exist. Validated at load. ### `discover` @@ -866,7 +866,7 @@ sole model, else 400. The same addressing works in the CLI: `gmlx run @coding`, `gmlx run --profile coding`, and identically for `chat`. A bare-path -`run`/`chat` (no config) still gets its family's base defaults; an explicit +`run`/`chat` (no config) still gets its family's base defaults. An explicit sampling flag always wins, and `--no-family-defaults` (or `GMLX_NO_FAMILY_DEFAULTS=1`) opts a run out entirely. @@ -876,7 +876,7 @@ sampling flag always wins, and `--no-family-defaults` (or ### Sampling keys (`sampling:`) -The request fields mlx-vlm already honours; carried verbatim into generation: +The request fields mlx-vlm already honours, carried verbatim into generation: `temperature`, `top_p`, `top_k`, `min_p`, `max_tokens`, `seed`, `repetition_penalty`, `presence_penalty`, `frequency_penalty`, diff --git a/docs/serving-architecture.md b/docs/serving-architecture.md index 79eaae2..336524a 100644 --- a/docs/serving-architecture.md +++ b/docs/serving-architecture.md @@ -110,7 +110,7 @@ flowchart TD manager: exact-snapshot for hybrid / sliding-window caches, block-level for plain attention. Speculative decoding (draft model / MTP) is optional and runs gmlx's own verify round, which keeps APC available (upstream disables - it under a draft model); see + it under a draft model). See [server-config.md](server-config.md#speculative-decoding--the-prompt-cache). 4. HTTP layer (`mlx-vlm` FastAPI app): exposes OpenAI Chat Completions diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 2b7b873..45df235 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -6,7 +6,7 @@ anything it flags ([reference](cli.md#gmlx-doctor)). The rest of this page covers the failures new setups actually hit, with the diagnostic and the fix. Client-launch problems (a tool will not connect or -install) live in [launch.md](launch.md#troubleshooting); everything below is +install) live in [launch.md](launch.md#troubleshooting). Everything below is the runtime and server. ## The install fails compiling the Metal kernels @@ -32,7 +32,7 @@ lives in the Python venv you installed it into, and each new terminal starts with that venv inactive. Run `source /.venv/bin/activate` (the directory from the [install step](getting-started.md#install)) and the command is back. A background server or menu-bar app keeps running either -way; only the terminal command needs the venv. An install via `uv tool +way. Only the terminal command needs the venv. An install via `uv tool install` or pipx stays on PATH in every terminal and never hits this. ## A download was interrupted or the disk filled @@ -42,7 +42,7 @@ Symptom: `gmlx pull` stopped mid-download, or refused to start with Interrupted downloads resume: re-run the same `pull` and it continues from where it stopped (sharded files resume per shard). The disk-space refusal is a -preflight - it names how much the file needs and how much is free; free space, +preflight: it names how much the file needs and how much is free. Free space, pass `--to DIR` on another volume, or `--force` to skip the check. A download that failed mid-write for another reason (network drop, HF hiccup) is also safe to re-run. @@ -62,7 +62,7 @@ downloading. Uniform K-quant files also decode fastest ## A configured model is missing from /v1/models Symptom: an id from your config is not listed, or requesting it returns a 404 -with type `model_file_missing`; `gmlx logs` shows +with type `model_file_missing`. `gmlx logs` shows `[server] skipping model ''` at the last startup or reload. The entry's GGUF is gone from disk (deleted, moved, or renamed), so the server @@ -98,7 +98,7 @@ whether an input device is visible at all. Symptom: `serve` fails to bind, or requests reach some other process. -`gmlx status` shows whether a managed gmlx server already holds the port; if +`gmlx status` shows whether a managed gmlx server already holds the port. If so, `gmlx stop` (or `gmlx restart` after a config change; a launchd-managed server stops with `gmlx service uninstall`). If the holder is not gmlx, `lsof -i :8080` names it. Either free the port or serve elsewhere with @@ -111,7 +111,7 @@ seconds. Nothing was preloaded, so the first request paid the model load. Set `server.defaults.model: ` in the config, pin one, or pass `--model` to -`gmlx launch`; the +`gmlx launch`. The auto-start path then loads the weights before binding the port, and the first turn is hot. Distinct from this: the first turn on a very long prompt is prefill, not loading; see [performance.md](performance.md#the-prompt-cache). @@ -145,7 +145,7 @@ or `--max-models` so residency stops short of the ceiling. ## Where the evidence lives -`gmlx logs -n 100` prints the managed server's log (`-f` follows); the files sit +`gmlx logs -n 100` prints the managed server's log (`-f` follows). The files sit under `~/.cache/gmlx/`. Each completed request logs one line with the model, token counts, and timing, which is usually enough to see what was slow. `gmlx status` reports the process, `gmlx ps` the resident models, and From 03892fb5a7c54262739d91f4f3cd505deaf214fb Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 23:02:32 -0700 Subject: [PATCH 17/19] docs(readme): drop the MTP lift chart - one graph, the fleet ratio, tells the story --- README.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/README.md b/README.md index 5747d83..0b75b5f 100644 --- a/README.md +++ b/README.md @@ -197,11 +197,6 @@ active on both engines, gmlx decodes 1.1-2x ahead throughout. MTP's lift over the same server with it off holds at 1.4-1.8x from 17k through 110k, where llama.cpp's speculation gain decays with depth. - - - MTP decode lift vs KV depth: gmlx holds its speedup where llama.cpp's decays - - Reference points at short context: gemma-4-12B-it (dense, Q6_K) decodes at ~72 tok/s with MTP vs llama.cpp's ~54 with speculation (1.3x), prefilling at ~850 vs ~730 tok/s; From e2fab9e22bf447c7bbee6885bf2b08ab0a7ab2c6 Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 23:05:57 -0700 Subject: [PATCH 18/19] docs(readme): add DeepSeek-V4-Flash IQ2_XXS vs ds4-server chart under the fleet graph --- README.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/README.md b/README.md index 0b75b5f..58dacec 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,15 @@ experts from disk ([bigger than memory](#bigger-than-memory)). gmlx vs llama.cpp: fleet throughput speedup vs KV depth + + + DeepSeek-V4-Flash IQ2_XXS: gmlx vs ds4-server prefill and decode vs KV depth + + +DeepSeek-V4-Flash is measured against dwarfstar's ds4-server, since +llama.cpp has no V4-Flash path +([benchmarks](https://github.com/asher/gmlx/blob/main/docs/benchmarks.md#deepseek-v4-flash-reference-engine-ds4-server)). + From 7187f8c7ec135ef9c766e1e73a32de28c374d481 Mon Sep 17 00:00:00 2001 From: Asher Feldman <59994+asher@users.noreply.github.com> Date: Sun, 9 Aug 2026 23:08:28 -0700 Subject: [PATCH 19/19] docs: ds4 caption without the stale no-llama.cpp-path claim, credit antirez --- README.md | 3 +-- docs/benchmarks.md | 6 +++--- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 58dacec..6115657 100644 --- a/README.md +++ b/README.md @@ -32,8 +32,7 @@ experts from disk ([bigger than memory](#bigger-than-memory)). DeepSeek-V4-Flash IQ2_XXS: gmlx vs ds4-server prefill and decode vs KV depth -DeepSeek-V4-Flash is measured against dwarfstar's ds4-server, since -llama.cpp has no V4-Flash path +DeepSeek-V4-Flash is measured against antirez's ds4 (dwarfstar) ([benchmarks](https://github.com/asher/gmlx/blob/main/docs/benchmarks.md#deepseek-v4-flash-reference-engine-ds4-server)).