Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
b0d97ed
docs(readme): restructure — quickstart first, dedupe performance clai…
asher Aug 10, 2026
80d0699
docs(readme): cut restated claims — say each thing once
asher Aug 10, 2026
d4bfdd9
docs(readme): surface concurrent-request behavior in the serve section
asher Aug 10, 2026
207a367
docs(getting-started): dedupe install facts, group extras under a sub…
asher Aug 10, 2026
7000e04
docs(index,migrating): list upstream-upgrades + benchmarks in index, …
asher Aug 10, 2026
543145d
docs(launch,talk): dedupe per-client dotfile/auth restatements, defer…
asher Aug 10, 2026
733e662
docs(vlm): merge two-sentence restatement
asher Aug 10, 2026
99c73b0
docs(perf,streaming): reattach stranded decode-feeder measurement, de…
asher Aug 10, 2026
9a91e89
docs(internals): fold diagram-restating component item, trim acceptan…
asher Aug 10, 2026
6d7a4a9
docs(cli,launch): single-source the launch flag table in cli.md, drop…
asher Aug 10, 2026
11abf67
docs(server-config): cut sequence-diagram narration paragraph
asher Aug 10, 2026
18cc88e
docs(launch): fold flag-table pointer into the intro, drop the empty …
asher Aug 10, 2026
aa09663
docs(style): de-bridge README, migrating, streaming - split semicolon…
asher Aug 10, 2026
7c3e37f
docs(style): de-bridge performance, talk, assistant
asher Aug 10, 2026
de762ce
docs(style): de-bridge vlm, getting-started, launch
asher Aug 10, 2026
dbbb62e
docs(style): de-bridge remaining docs - split prose semicolons, keep …
asher Aug 10, 2026
03892fb
docs(readme): drop the MTP lift chart - one graph, the fleet ratio, t…
asher Aug 10, 2026
e2fab9e
docs(readme): add DeepSeek-V4-Flash IQ2_XXS vs ds4-server chart under…
asher Aug 10, 2026
7187f8c
docs: ds4 caption without the stale no-llama.cpp-path claim, credit a…
asher Aug 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
341 changes: 141 additions & 200 deletions README.md

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ right document for the task at hand.
| Run a vision or audio-input model | [vlm.md](vlm.md) |
| Fine-tune with LoRA on a quantized base | [lora.md](lora.md) |
| Make it faster | [performance.md](performance.md) |
| See the numbers behind the performance claims | [benchmarks.md](benchmarks.md) |
| Run a MoE model bigger than RAM | [streaming.md](streaming.md) |
| Fix something that broke | run `gmlx doctor`, then [troubleshooting.md](troubleshooting.md) |
| Look up a flag or config key | [cli.md](cli.md), [server-config.md](server-config.md) |
Expand Down Expand Up @@ -118,5 +119,9 @@ as supported.
[testing.md](testing.md) describes the test tiers, the end-to-end harnesses, and
the manual voice-loop pass.

[upstream-upgrades.md](upstream-upgrades.md) is the maintainer's guide to
bumping the pinned mlx-vlm/mlx-lm/mlx versions: the seam contract, the canary
script, and the qualification steps.

[CONTRIBUTING.md](../CONTRIBUTING.md) has the development setup and expectations
for pull requests; [CHANGELOG.md](../CHANGELOG.md) records what shipped when.
2 changes: 1 addition & 1 deletion docs/adding-architectures.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ load cleanly into wrong weights.

## The acceptance gate (all must pass)

An architecture is not done when it generates fluent text. It's done when:
An architecture is done when:

- Strict load: `load_model` builds, swaps, and `load_weights` leaves no
parameter unfilled. The loader's unfilled-params warning must be empty.
Expand Down
18 changes: 9 additions & 9 deletions docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ around the server's own chat completions. Configure it once in a top-level
server-side for any OpenAI client.

It is deliberately a lightweight assistant, not an autonomous agent. Each turn
runs a bounded tool loop and ends when the model answers; nothing keeps working
runs a bounded tool loop and ends when the model answers. Nothing keeps working
in the background afterward. It is built for errand-sized tasks: look something
up, chain a few tool calls, write a note, remember a fact. The external coding
agents `gmlx launch` connects (pi, opencode, and friends) are a different
Expand Down Expand Up @@ -121,15 +121,15 @@ assistant:
```

A stdio tool server runs with a minimal environment (`HOME`, `PATH`, `SHELL`,
`TERM`, `USER`, `LOGNAME`); `env:` adds to that, and nothing else from this
`TERM`, `USER`, `LOGNAME`). `env:` adds to that, and nothing else from this
process's environment is inherited. Pass a secret a server needs explicitly, as
above -- an `HF_TOKEN` or `OPENAI_API_KEY` sitting in your shell never reaches
third-party tool code.

**Document retrieval (RAG as a tool).** Qdrant's official server in embedded
local mode gives the assistant `qdrant-store` and `qdrant-find` over a vector
collection on disk -- no database process to run. Ask the assistant to store
passages and it can retrieve them semantically later; embedding happens
passages and it can retrieve them semantically later. Embedding happens
inside the tool server with its own small local model, independent of this
server's `/v1/embeddings`:

Expand Down Expand Up @@ -157,7 +157,7 @@ transient context. They never bloat the rolling chat history. Without
warning.

What gets stored is a distilled fact ("sister Ana, birthday March 12"), not a
transcript: after each turn, a background request asks the chat model to boil
transcript. After each turn, a background request asks the chat model to boil
the exchange down to at most three durable facts, or none, so small talk
leaves no residue. A new fact that restates an existing one replaces it.
`extract: false` stores raw user and assistant exchanges instead. `ttl_days`
Expand All @@ -177,7 +177,7 @@ per-id store (`assistant-<id>.db`), never this one.
`--assistant` switches the chat REPL's turn engine from a local model load to
the assistant on the managed server (auto-started if down, like `talk`). The
positional argument is a served model id, or omitted for the server's default
model; a file path is refused, since the server owns the model. Tool activity
model. A file path is refused, since the server owns the model. Tool activity
appears as transient status lines while the answer streams.

```sh
Expand All @@ -189,11 +189,11 @@ The terminal experience is unchanged: markdown rendering, reasoning display,
themes, history, sessions and `--resume`, `/system`, `/retry` and `/undo` (a
retry or undo rewinds whole tool rounds, so the history never holds a
half-finished tool exchange), plus `/memory` as above. Sampling flags and
their `/commands` forward to the server per round once you touch them;
untouched knobs stay on the server's own defaults. Flags that only make sense
their `/commands` forward to the server per round once you touch them.
Untouched knobs stay on the server's own defaults. Flags that only make sense
for a local load are rejected (`--adapter`, `--mmproj`, chat-template flags)
or ignored with a printed note (loading, KV-cache, speculative/MTP, CPU
placement); `/image`, `/audio` and the thinking budget are not available
placement). `/image`, `/audio` and the thinking budget are not available
in this mode. `--base-url`, `--api-key`, `--no-start`, and `--start-timeout`
target a remote or already-running server, exactly as in `talk`.

Expand Down Expand Up @@ -251,7 +251,7 @@ The routing contract, per request to `/v1/chat/completions`:

Reported usage sums `completion_tokens` across all rounds; `prompt_tokens` is
the final round's. Concurrent assistant turns are capped (currently 4 per
server); a request over the cap gets an immediate 429 rather than queueing.
server), and a request over the cap gets an immediate 429 rather than queueing.
Three known limits: a `stop` sequence forwards to every round and can in
principle truncate an intermediate tool round; a non-streaming assistant turn
cannot be cancelled by client disconnect (it runs its rounds to completion);
Expand Down
6 changes: 3 additions & 3 deletions docs/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -280,9 +280,9 @@ ladders cell by cell).

## DeepSeek-V4-Flash (reference engine: ds4-server)

This model's comparison engine is **ds4-server** (dwarfstar's
DeepSeek-V4 server, ignore-eos patched), not llama.cpp -- llama.cpp
has no DeepSeek-V4-Flash path. Ratios below are gmlx / ds4-server.
This model's comparison engine is **ds4-server** (antirez's dwarfstar
ds4 server, ignore-eos patched). Ratios below are
gmlx / ds4-server.

### DeepSeek-V4-Flash IQ2_XXS

Expand Down
4 changes: 0 additions & 4 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,6 @@
| [`gmlx profiles`](#gmlx-profiles) | Show per-family sampling defaults and `@intents`; resolve one model's sampling. |
| [`gmlx completion`](#gmlx-completion) | Print a shell completion script (zsh, bash, fish). |

`gmlx` is the installed command; every action is a subcommand (`gmlx run`,
`gmlx serve`, `gmlx init`, and so on). The project, package, and command are
all named `gmlx`.

Every verb operates on the GGUF file itself: the file on disk is the model.
Every flag is also visible via `--help`.

Expand Down
77 changes: 35 additions & 42 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,20 +33,19 @@ brew install ffmpeg # voice and non-wav audio only

That is the whole install. `gmlx` lands on your PATH in every terminal, in an
isolated environment, with a suitable Python fetched for it, and `[all]` turns
on every optional feature so nothing else is needed later.
on every optional feature.
[uv](https://docs.astral.sh/uv/) itself is one command
(`brew install uv`, or `curl -LsSf https://astral.sh/uv/install.sh | sh`);
`pipx install "gmlx[all]"` behaves the same way. Upgrade later with
`uv tool upgrade gmlx` (or `pipx upgrade gmlx`).

`ffmpeg` is the one dependency no Python installer can supply. It is needed
only to decode audio uploads and to encode mp3/flac/opus; skip it if you are
only to decode audio uploads and to encode mp3/flac/opus. Skip it if you are
not using voice.

To install less, name the extras you want instead of `all` - `gmlx[chat]` is
the common choice, giving up only voice and the assistant. `gmlx init` offers
to add the rest later, and issues the correct command for however gmlx was
installed.
to add the rest later ([adding an extra](#the-extras) below).

A plain venv you manage yourself works as well:

Expand All @@ -62,11 +61,17 @@ run `source ~/gmlx/.venv/bin/activate` in each new terminal. A
no such step.

The `mlx-kquant` dependency (the Metal kernels) arrives
as a prebuilt wheel from PyPI on macOS 26 and newer; on older macOS versions
the install builds it from source, which needs the Xcode Command Line Tools
(`xcode-select --install`) and takes a few minutes.
as a prebuilt wheel from PyPI on macOS 26 and newer. On older macOS versions
the install builds it from source: the Command Line Tools requirement from
[What you need](#what-you-need), and a few minutes of compile time.

The extras, all optional:
Tab completion is worth the one line: add `eval "$(gmlx completion zsh)"` to
`~/.zshrc` (there are `bash` and `fish` variants). It completes verbs, flags, your
config's model ids, and the host and port of any running server.

### The extras

All optional:

| Extra | Adds | Needed for |
|-------|------|------------|
Expand All @@ -81,24 +86,18 @@ Every extra runs on any supported Python, 3.11 through 3.14.

Adding one later depends on how gmlx was installed: `pip install 'gmlx[X]'`
in a venv, but a `uv tool` or pipx environment is owned by its installer and
takes a different command. These docs write the pip form for brevity; to get
takes a different command. These docs write the pip form for brevity. To get
the right one for your machine, run `gmlx init`, which offers to install what
it needs, or read it off any "not installed" message - both name the command
that works where you are.

`vlm` and `embeddings` exist as empty back-compat extras; multimodal loading and the
`vlm` and `embeddings` exist as empty back-compat extras: multimodal loading and the
embeddings endpoint are part of the core install. mlx-audio itself already arrives
with the core install; the `tts` extra pins it and adds the
with the core install. The `tts` extra pins it and adds the
grapheme-to-phoneme front-end the default Kokoro voice needs. That front-end
also wants a spaCy English pipeline, which gmlx fetches from Hugging Face on
first use - no separate install step.

Tab completion is worth the one line: add `eval "$(gmlx completion zsh)"` to
`~/.zshrc` (there are `bash` and `fish` variants). It completes verbs, flags, your
config's model ids, and the host and port of any running server.

The installed command is `gmlx`.

## First generation in two minutes

Download a small model into the current directory and run it:
Expand Down Expand Up @@ -129,15 +128,15 @@ already using the settings the model's authors recommend. `gmlx profiles` prints
the table. Two words you will meet for adjusting them: built-in *intents*
(`@coding`, `@creative`, ...) work on any model with no config, while *profiles*
are your own named setting bundles, defined later in a server config's
`profiles:` block; both are addressed the same way, `model@NAME` or
`profiles:` block. Both are addressed the same way, `model@NAME` or
`--profile NAME`.

## Pick a model for your Mac

Suggestions by machine memory, all instruct models that load end to end here.
A quick key to the quant names you will see everywhere: the Q-number is roughly
bits per weight, so Q4 files are smaller and slightly lossier, Q6/Q8 bigger and
closer to the original; when and why it matters is in
closer to the original. When and why it matters is in
[performance.md](performance.md#choosing-a-quant-for-speed).

| Mac RAM | Suggestion | Notes |
Expand All @@ -147,7 +146,7 @@ closer to the original; when and why it matters is in
| 64 GB | Qwen3.6-27B (Q6_K, ~23 GB) | strong general model; also the tool-calling pick |
| 96+ GB | Qwen3.6-35B-A3B (Q6_K) or gpt-oss-120b (MXFP4, ~63 GB) | MoE models: big-model quality, small-model decode cost |

These sizes leave room for the KV cache at everyday context lengths;
These sizes leave room for the KV cache at everyday context lengths.
[Will it fit?](#will-it-fit) below has the per-token arithmetic and the
`--kv-bits` lever for long sessions.

Expand All @@ -161,19 +160,18 @@ gmlx validate hf:unsloth/Qwen3.6-27B-GGUF
gmlx pull hf:unsloth/Qwen3.6-27B-GGUF/Qwen3.6-27B-Q6_K.gguf --to ~/models
```

`--to` says where the file lands; it is required until a config exists (next
section), after which bare `pull` lands files in your model directory, registers
them in the config, and a running server serves them immediately.
`--to` says where the file lands, and it is required only until a config exists
(next section). After that, bare `pull` lands files in your model directory,
registers them in the config, and a running server serves them immediately.
`gmlx sync-models` reconciles in bulk after hand-moving or deleting files. These
are multi-gigabyte downloads - minutes to an hour depending on your connection -
with progress, rate, and resume built in.

`validate` accepts a repo, a folder, a pasted browser link, or an exact file. Given
a repo it lists every quant variant as a ready-to-paste ref. K-quant, legacy, and
IQ files all load; in the rare case a file uses a codec with no kernel (the
IQ files all load. In the rare case a file uses a codec with no kernel (the
ternary TQ types, for instance), the verdict names it so you can pick another
variant. Uniform K-quant files also decode
faster; see [performance.md](performance.md#choosing-a-quant-for-speed).
variant. Uniform K-quant files also decode faster than heavily mixed ones.

Set `HF_TOKEN` for gated or private repos. If you already have a model library
from LM Studio, it serves as-is (the files are plain GGUFs):
Expand All @@ -190,18 +188,13 @@ bytes per token = 2 (K and V) x layers x kv_heads x head_dim x 2 (bf16)
```

An 8B-class model (32 layers, 8 KV heads, head dim 128) uses 128 KB per token of
context, so a 32k-token session adds 4 GB on top of the weights. A 32B-class dense
model (64 layers, same heads) uses 256 KB per token: 8 GB at 32k. The layer and head
counts are in the GGUF metadata, and the model card lists them too.
context, so a 32k-token session adds 4 GB on top of the weights. The layer and
head counts are in the GGUF metadata, and the model card lists them too.

If weights plus cache crowd your RAM, quantize the cache: `--kv-bits 8` roughly
halves it at nearly no quality cost, and `--kv-bits 4` roughly quarters it with a
small cost at long range. In server configs the same knob is the `kv_bits` load key.

Several families are much cheaper than the formula suggests. Sliding-window layers
(gemma) stop growing at the window size, hybrid linear-attention models (Qwen3.5 and
3.6, Falcon-H1, Granite 4.x) keep a small fixed state on most layers, and MLA models
(DeepSeek) store a compressed cache. Worked numbers:
halves it at nearly no quality cost. Several families (sliding-window,
hybrid linear-attention, MLA) are much cheaper than the formula suggests.
More worked numbers, the cheaper families, and the rest of the levers:
[performance.md](performance.md#memory-and-the-kv-cache).

## Set up the server
Expand All @@ -219,9 +212,9 @@ that walks through:
6. Idle unload (how long an unused model stays resident) and a request timeout.
7. Where to write the file, with a preview before anything is saved.

Prefer flags? `gmlx init --models-dir ~/models` scaffolds non-interactively; every
wizard choice has a flag equivalent (`--with-stt`, `--disk-cache`, `--default-model`,
and so on).
Prefer flags? `gmlx init --models-dir ~/models` scaffolds non-interactively, and
every wizard choice has a flag equivalent (`--with-stt`, `--disk-cache`,
`--default-model`, and so on).

The config lands at `~/.config/gmlx/gmlx.yaml`. It is one YAML file with a
`server:` block (port, model directories, services), a `models:` block (one entry
Expand All @@ -248,7 +241,7 @@ the resident models, with unload, restart, log, and config-editing controls.

The server speaks the OpenAI API (plus Anthropic and OpenAI Responses on the same
port). The `model` field is whatever id `init` printed for your file - auto-named
ids carry the quant tag (`qwen3-0.6b-q4`, `qwen3.6-27b-q6`); `gmlx list` shows
ids carry the quant tag (`qwen3-0.6b-q4`, `qwen3.6-27b-q6`). `gmlx list` shows
them, and an unknown id gets a 404 listing the valid ones. With the small model
from the walkthrough above:

Expand Down Expand Up @@ -310,7 +303,7 @@ gmlx launch open-webui
```

`launch` starts your gmlx server if needed, wires Open WebUI to it, runs it on
port 3000, and prints the URL to open. Chat works immediately; if your server
port 3000, and prints the URL to open. Chat works immediately. If your server
also runs embeddings, speech-to-text, or text-to-speech, document upload and
voice light up too. Details, including a no-login single-user setup:
[launch.md](launch.md#open-webui).
Expand All @@ -333,7 +326,7 @@ gmlx service uninstall # remove the login item

macOS only. `service install` accepts the same options as `serve`. The menu
bar runs as the launchd agent (so its permission prompts attribute to gmlx)
and starts the recorded server once per login; `--no-autostart` leaves the
and starts the recorded server once per login. `--no-autostart` leaves the
server to its Start menu item, and `--headless` installs a server-only agent
for GUI-less machines (that one restarts on crash and is stopped with
`service uninstall`).
Expand Down
Loading
Loading