From a85f4a6c158628e651227a3e141ce70087696c7a Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 01:06:22 +0000 Subject: [PATCH 1/2] feat: add oMLX as a provider and a serve engine MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit oMLX (https://omlx.ai) is a local inference server for Apple Silicon built on Apple's MLX framework, exposing an OpenAI-compatible API. Add it to the catalogue so a harness can be pointed at it, and teach `outfit serve` to launch it, so Mac users get the same two-step flow llama.cpp users already have. Serve now picks its engine from PROVIDER, which settles an inconsistency the code already claimed was settled: runnerFor documents itself as "PROVIDER already names the engine — outfit serve starts that engine locally", and the usage text says the same, but cmdServe never read PROVIDER and unconditionally ran llama-server. An Outfit naming any other provider quietly started the wrong server; that is now a loud error naming the providers that can be served. Preset parsing is shared but flag *spelling* is not, so internal/preset grows a Dialect carrying the alias and boolean tables. LlamaCpp keeps the existing tables and remains what the package-level helpers render, so remote.go's isCloudOwned — which compares preset keys through CanonicalKey — is unchanged. OMLX is the zero Dialect: every omlx-cli flag is long-form, and llama.cpp's table would silently rewrite `m` to --model and `c` to --ctx-size. oMLX differs from llama.cpp in loading a whole --model-dir and selecting per request, so it starts with neither a PRESET nor a MODEL, and only BASEURL becomes a launch flag. MODEL and ALIAS keep their usual meaning as the id the harness asks for — load-bearing here rather than a label. serve never passes --api-key. It prints the command it runs, so a resolved secret would land on screen and in the process table. The serve command moves to its own file. Beyond cohesion, this is required: TestCompletionCoversDispatch scans main.go for `case "…":` at one tab, so engineFor's switch there reads as two new commands. remote deploy is deliberately untouched — the cloud side is CUDA/NVIDIA EC2 and oMLX is Apple Silicon, so runnerFor keeps rejecting it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CipwSCMUZRE61zWWiYEcDx --- AGENTS.md | 9 +- CHANGELOG.md | 10 + README.md | 17 +- cmd/outfit/main.go | 175 +---------- cmd/outfit/serve.go | 286 ++++++++++++++++++ cmd/outfit/serve_test.go | 232 +++++++++++++- docs/README.md | 12 +- docs/commands/serve.md | 86 +++++- docs/outfit-file.md | 8 +- examples/omlx/qwen3.6/Outfit | 8 + examples/omlx/qwen3.6/README.md | 152 ++++++++++ examples/omlx/qwen3.6/preset.ini | 18 ++ internal/catalog/catalog_test.go | 72 +++++ internal/catalog/providers.yaml | 17 ++ internal/preset/preset.go | 115 ++++--- internal/preset/preset_test.go | 53 +++- .../changes/add-omlx-provider/.openspec.yaml | 2 + openspec/changes/add-omlx-provider/design.md | 104 +++++++ .../changes/add-omlx-provider/proposal.md | 45 +++ .../specs/local-serving/spec.md | 145 +++++++++ .../specs/provider-catalog/spec.md | 31 ++ openspec/changes/add-omlx-provider/tasks.md | 42 +++ 22 files changed, 1402 insertions(+), 237 deletions(-) create mode 100644 cmd/outfit/serve.go create mode 100644 examples/omlx/qwen3.6/Outfit create mode 100644 examples/omlx/qwen3.6/README.md create mode 100644 examples/omlx/qwen3.6/preset.ini create mode 100644 openspec/changes/add-omlx-provider/.openspec.yaml create mode 100644 openspec/changes/add-omlx-provider/design.md create mode 100644 openspec/changes/add-omlx-provider/proposal.md create mode 100644 openspec/changes/add-omlx-provider/specs/local-serving/spec.md create mode 100644 openspec/changes/add-omlx-provider/specs/provider-catalog/spec.md create mode 100644 openspec/changes/add-omlx-provider/tasks.md diff --git a/AGENTS.md b/AGENTS.md index 363b8386..8b9bb803 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,7 +23,8 @@ Run a single test: `go test -run TestWriteConfig_Idempotent ./...` The binary lives under `cmd/`; the domain logic is split into `internal/` packages so each concern is isolated and independently testable. - `cmd/outfit/main.go` — CLI: command dispatch (`add`/`remove`/`list`/`show`/`apply`/`unapply`/`alias`/`unalias`/`serve`/`export`/`harness`/`init-providers`/`remote`/`completion`, plus the hidden `__complete`), flag parsing (`parseSelection` registers both long and short flags against the same vars, and returns the chosen harness name separately so it never leaks into a `Selection`), and user-facing output. `add` and `apply` share `applySelection`, the core that writes one provider selection; `remove` and `unapply` share `removeSelection`, its inverse. Both resolve the active harness (`harness.Resolve`) and route through it. `show` reports the active harness's current config via `harness.State` (the same read `export` reconstructs an Outfit from). `serve` is harness-agnostic (it launches `llama-server`, not a config write). `harness` launches the active harness (forwarding trailing args), or manages the stored default with `--get`/`--set`; its `--outfit`/`-O` routes through the same `applySelection` before launching, so a launch can dress the harness first. That flag's value is optional (`outfitPathFlag` implements `IsBoolFlag`) — bare it means the default `./Outfit`, so a path has to be attached (`--outfit=`), since positional args belong to the harness; the exception is a *leading* positional that names an Outfit, which is applied and not forwarded (see "Alias registry" below). `alias`/`unalias` manage the registry that lets a name stand in for a path, and `writeAliases` renders it for both `alias --list` and `show`. Tests: `main_test.go` (dispatch/add/remove/list), `apply_test.go` (`apply`/`unapply`/`export`), `harness_test.go` (the `harness` command, `show`, and Pi-routed add/export/remove), `serve_test.go`, `alias_test.go`, `complete_test.go`. -- `cmd/outfit/complete.go` + `completion.bash` — tab completion, and the only reason the CLI is no longer a single source file: `//go:embed` needs the script beside it. `outfit completion bash` prints the script; the script calls the hidden `outfit __complete `, which prints one candidate per line and a final `:nofile`/`:file` directive. The `commands` table there mirrors `run`'s switch and each `cmdX`'s `FlagSet` (the `flag` package cannot enumerate a `FlagSet`); `TestCompletionCoversDispatch` scans `main.go` and fails when a new command is not completable. +- `cmd/outfit/serve.go` — the `serve` command: `engineFor` and the `serveEngine` table, the binary lookups (`llamaServerBinary`, `resolveOMLXBinary` — PATH then the `/Applications/oMLX.app` bundle), the Outfit→flags mappings (`llamacppServeParams`/`omlxServeParams`/`bindAddressParams`), and the helpers `remote.go` shares (`resolvePresetPath`, `isModelPath`, `hostPortFromURL`). It is a separate file for a second reason beyond cohesion: `TestCompletionCoversDispatch` scans `main.go` for `case "…":` at one tab, so any other switch on string literals living there is mistaken for a new command. +- `cmd/outfit/complete.go` + `completion.bash` — tab completion, and one of the reasons the CLI is no longer a single source file: `//go:embed` needs the script beside it. `outfit completion bash` prints the script; the script calls the hidden `outfit __complete `, which prints one candidate per line and a final `:nofile`/`:file` directive. The `commands` table there mirrors `run`'s switch and each `cmdX`'s `FlagSet` (the `flag` package cannot enumerate a `FlagSet`); `TestCompletionCoversDispatch` scans `main.go` and fails when a new command is not completable. - `internal/config` — outfit's own config file, `${XDG_CONFIG_HOME:-~/.config}/outfit/config.json`: the default-harness preference and the Outfit alias registry. A **leaf package** — stdlib imports only, and it must stay that way (in particular it must never import `internal/outfit` to parse an Outfit), because that is what lets `readOutfit` resolve a name without dragging in the harness abstraction. Every write goes through `Update`, a read-modify-write of the whole document; unknown top-level keys round-trip through `File.extra`. - `internal/outfit` — the `Outfit` file format and the shared `Selection` type: a flat, Dockerfile-style description of one provider selection (`PROVIDER`/`MODEL`/`ALIAS`/`CONTEXT`/`OUTPUT`/`BASEURL`/`PRESET`/`REMOTE`). `Parse` reads it into a `Selection` (keywords case-insensitive via `canonicalKeyword`, UPPERCASE canonical, `#` comments); `Format` renders one back out for `export`. `apply` defaults to `./Outfit` (`outfit.DefaultFile`). The harness is deliberately **not** a keyword — it is a runtime choice, so an Outfit stays portable. - `internal/harness` — the harness abstraction. The `Harness` interface (`Name`/`Command`/`ConfigPath`/`Apply`/`Remove`/`State`), a `registry` of the opencode and pi adapters (`adapters.go`, which wrap `catalog` + `opencode`/`pi` + `contextsize`), runtime selection via `Resolve` with precedence **`--harness`/`-H` flag > `OUTFIT_HARNESS` env > stored preference > opencode**, and preference load/save (`LoadPreference`/`SavePreference`), which delegate to `internal/config` — that package, not this one, owns the config file. **Start here when adding a third harness:** implement the interface and register it. @@ -33,7 +34,7 @@ The binary lives under `cmd/`; the domain logic is split into `internal/` packag - `cmd/outfit/remote.go` — the `remote` command group (`start`/`stop`/`status`/`deploy`), the CLI's only nested subcommand: each subcommand is a thin wrapper over `internal/remote`. `deploy` is the exception that carries logic: `deployConfigFor` turns an Outfit (plus its preset) into a `remote.DeployConfig` — `PROVIDER` picks the runner (`llamacpp`/`vllm`; other providers are not self-hosted engines, so there is nothing to deploy), `MODEL` or the preset's `hf` names the weights as `repo:QUANT`, `CONTEXT` or the preset's `ctx-size` the window, `ALIAS` the served name, and the preset's remaining flags become `serveArgs`. The `[*]` globals are layered under the section (so `ngl`/`jinja` written there survive), and `cloudOwnedFlags` drops what the cloud sets itself (host/port/model/hf/api-key/ctx-size/alias/metrics), matched on `preset.CanonicalKey` so short aliases count. Config discovery (`resolveRemoteConfig`): an explicit `[path]` argument must name an Outfit with a `REMOTE` instruction; with no argument, `./Outfit` is consulted when present (its `REMOTE` resolved relative to the Outfit, like `PRESET`), falling back to the per-user `remote.json`. Tests: `remote_test.go` (dispatch plus full flows against an `httptest.Server`, with `stubAWSEnv` pinning the AWS credential chain to static env credentials). - `internal/remote` — controls the scale-to-zero GPU inference instance defined in [`remote/`](remote/) by calling its start/stop/deploy Lambda Function URLs (IAM auth, so requests are SigV4-signed via `aws-sdk-go-v2` — the repo's only AWS/network dependency). `Deploy` is the only call with a body, and SigV4 covers the payload hash, so `call`/`sign` take the bytes and hash exactly what is sent. `DeployConfig` deliberately carries no weights prefix: the Lambda derives the S3 layout and seeds the weights when they are absent, replying `{seeding, seedInstanceId}`. Config in `${XDG_CONFIG_HOME:-~/.config}/outfit/remote.json` (`start_url`/`stop_url`/`deploy_url`/`region`), overridable via `OUTFIT_REMOTE_*` env vars; `deploy_url` is optional so configs predating deploy still work for start/stop, and only `Deploy` complains when it is missing; the region falls back to `AWS_REGION`, then to the region embedded in the Function URL host. `Start` retries while the endpoint reports 503/starting, honouring `retry_after_seconds`. - `internal/contextsize` — parses human-friendly sizes (`128k`, `1.5m`, `200000`); `Apply` sets `limit.context` and `limit.output` on opencode model blocks; `DefaultOutput` returns a quarter of the context (the fallback when `OUTPUT` is unset). -- `internal/preset` — parses llama.cpp [preset `.ini`](https://github.com/ggml-org/llama.cpp/blob/master/docs/preset.md) files into named sections of flags, and renders a `llama-server` command for `serve` (`Parse`/`Flags`/`FormatCommand`). +- `internal/preset` — parses [preset `.ini`](https://github.com/ggml-org/llama.cpp/blob/master/docs/preset.md) files into named sections of flags, and renders a server command for `serve` (`Parse`/`Flags`/`FormatCommand`). Parsing is engine-neutral; only the *spelling* of flags differs, which a `Dialect` carries (`LlamaCpp`, `OMLX`). The package-level `Flags`/`CanonicalKey`/`Preset.Args`/`Preset.Command` render LlamaCpp, so pre-dialect callers — notably `remote.go`'s `isCloudOwned` — keep their behaviour; `Preset.CommandIn` takes a dialect plus the engine's subcommand. - `internal/catalog/providers.yaml` — externalised provider plumbing (URLs, key env vars, npm packages, and a `pi:` block per Pi-capable provider) — no model ids. **Add providers here, not in Go.** Embedded at build time but kept external for maintenance. - `examples/` — runnable guides, each a directory with a README and an `Outfit`. - `remote/` — the cloud GPU deployment `outfit remote` drives: a TypeScript CDK project (AWS Lambdas, EC2 Image Builder, S3 weights) with its own `package.json`, tests and docs, built and tested by pnpm rather than Go. It is the only non-Go part of the repository; CI runs its checks separately. **This repository is public, so nothing identifying a deployment — account ids, ARNs, Function URL hosts, allocated addresses, bucket names, resource ids — may be committed here or anywhere else in the repo.** `scripts/check-no-cloud-identifiers.sh` enforces that in CI; deployment state belongs in the gitignored files under `remote/` (`.env`, `Outfit`, `remote.json`, `cdk-outputs.json`). @@ -57,7 +58,7 @@ The binary lives under `cmd/`; the domain logic is split into `internal/` packag **Key/option resolution.** `catalog.BuildProviderBlock` sets `options.apiKey` (opencode) to the provider's `apiKeyEnv` as an `{env:VAR}` reference — never the secret — validating `apiKeyPrefix` and erroring when `apiKeyRequired` against the value it resolves. Resolution is `opencode.EnvResolver(dir)`: a `.env` **beside the Outfit being applied**, then the environment — `dir` is empty for a flags-only `add`, which falls back to the working directory. It deliberately is not relative to the binary or the source file (the old `runtime.Caller` form resolved to the build machine's path, so an installed binary never found one). The option is omitted only for a provider with no key variable, or an `apiKeyOptional` one at a local endpoint. Both harnesses therefore rely on the variable being set when the agent runs, which `harnessEnv` in the CLI arranges at launch. `optionsFromEnv` injects other options (e.g. `baseURL`, `region`) when their env vars are set. The opencode top-level `model` is `/`. Base-URL precedence for both builders: `--base-url`/`-u` flag > `OUTFIT_BASE_URL` > the catalogue's `pi.baseUrl`/`options.baseURL`. -**`serve`.** `outfit serve` launches `llama-server` for an Outfit without touching any harness config. With a `PRESET` it flattens the matching `.ini` section (plus `[*]` defaults) into flags via `internal/preset`; otherwise it derives a command from `MODEL`/`ALIAS`/`CONTEXT`/`BASEURL`. Anything the Outfit states overrides the preset. `--dry-run`/`-n` prints the command without running it. +**`serve`.** `outfit serve` launches a local inference server for an Outfit without touching any harness config. The engine comes from `PROVIDER` via `engineFor` (`cmd/outfit/serve.go`), the local twin of `runnerFor`: `llamacpp` runs `llama-server`, `omlx` runs the Apple-Silicon oMLX CLI, and anything else is an error. A `serveEngine` carries the binary, any subcommand, the preset dialect, the Outfit→flags mapping, and the install hint. With a `PRESET` it flattens the matching `.ini` section (plus `[*]` defaults) into flags in that engine's dialect; otherwise it derives a command from the Outfit alone. Anything the Outfit states overrides the preset. `--dry-run`/`-n` prints the command without running it. oMLX serves a whole `--model-dir` and picks per request, so only `BASEURL` becomes a launch flag for it, and it starts happily with neither `PRESET` nor `MODEL`. `serve` never resolves or passes an API key — it prints the command it runs. Invariants to keep: opencode merges stay idempotent and never drop unrelated config or comments; Pi merges preserve sibling providers and unknown fields; output stays `0600`; the harness never appears in an Outfit, and neither does an alias — both are machine-local, so an Outfit stays portable; outfit's own `config.json` is only ever written read-modify-write, so a new setting cannot clobber an existing one; `__complete` never returns an error and never writes to stderr, whatever the state of the config or catalogue. @@ -71,6 +72,8 @@ These are mistakes already made here; each was silent rather than loud, which is **Match preset flags by canonical name.** Preset keys have short aliases (`ngl` is `n-gpu-layers`, `c` is `ctx-size`, `hf` is `hf-repo`, `a` is `alias`). Comparing raw keys against a list of flag names therefore matches only the spelling you happened to think of. Use `preset.CanonicalKey`, which is exported for exactly this; `Flags` already dedupes layers by canonical name, last layer winning. +**A preset dialect is not interchangeable.** `internal/preset` parses any INI the same way, so an oMLX preset fed through llama.cpp's dialect parses cleanly and produces a *wrong* command: the alias table rewrites `m` to `--model` and `c` to `--ctx-size`, and the boolean table drops a `key = 0` entirely. Nothing errors — the server just receives flags it does not accept, or silently loses a setting. The dialect always comes from the engine `PROVIDER` names, never from the file. + **`contextsize.Parse` is decimal.** `128k` is 128000, not 131072 — a `CONTEXT` written that way is not the power-of-two window it looks like. It also *overrides* a preset's `ctx-size` (both in `serve` and in `remote deploy`), so the Outfit, not the preset, decides the window whenever it states one. ## Reference diff --git a/CHANGELOG.md b/CHANGELOG.md index b1a2c8b2..65ac522d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,16 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] +### Added +- feat: add an omlx provider to the catalogue +- feat: teach outfit serve to launch oMLX on Apple Silicon + +### Changed +- feat!: outfit serve picks its engine from PROVIDER, and rejects providers that + are not self-hosted engines (it previously ran llama-server whatever the + Outfit said) + ## [1.9.0] - 2026-07-29 ### Added - feat: add a vllm provider to the catalogue diff --git a/README.md b/README.md index 611a8411..7d9c44b6 100644 --- a/README.md +++ b/README.md @@ -47,8 +47,8 @@ model refs and quirks, and none of it is written down where you need it. it dresses the agent for you: - **One command, any model.** Pick from a built-in catalogue — OpenRouter, - Bedrock, Ollama, llama.cpp, or any OpenAI-compatible endpoint. No URLs or model - ids to look up. + Bedrock, Ollama, llama.cpp, vLLM, oMLX, or any OpenAI-compatible endpoint. No + URLs or model ids to look up. - **Your config survives.** Settings are merged *into* what you already have. Other providers, your theme, even your comments stay exactly where you left them. - **Keys stay where they belong.** Secrets are read from a local `.env` and never @@ -262,9 +262,10 @@ the bash and zsh completions for you. ## Serving a local model -Running a model with llama.cpp? `outfit serve` reads an `Outfit` and launches -`llama-server` for it — so the same file that points opencode at a model can -start it too. The simple case needs no preset: +Running a model locally? `outfit serve` reads an `Outfit` and launches the +inference server its `PROVIDER` names — `llamacpp` runs `llama-server`, `omlx` +runs [oMLX](https://omlx.ai) on Apple Silicon — so the same file that points +opencode at a model can start it too. The simple case needs no preset: ```dockerfile # Outfit @@ -320,7 +321,8 @@ minutes while the instance boots and loads the model; `--timeout` (default Each provider declares which environment variable holds its key (`outfit list` shows them). Values are looked up in a `.env` beside the `Outfit` first, then -your shell environment. Local providers like Ollama and llama.cpp need no key; +your shell environment. Local providers like Ollama, llama.cpp and oMLX need no +key; Bedrock authenticates through your AWS credentials. Base URLs default to the usual local ports. Override the endpoint for **any** @@ -334,7 +336,7 @@ OUTFIT_BASE_URL=https://gateway/v1 outfit add -p openai-compatible -m my-model The flag wins over the env var, and either wins over the catalogue's defaults and the per-provider variables (`OLLAMA_BASE_URL`, `LLAMACPP_BASE_URL`, -`OPENAI_BASE_URL`). +`OMLX_BASE_URL`, `VLLM_BASE_URL`, `OPENAI_BASE_URL`). ## Guides @@ -343,6 +345,7 @@ with a ready-to-apply `Outfit`: - [Qwen3.6-35B-A3B on llama.cpp](examples/llamacpp/qwen3.6/README.md) - [Gemma-4-12B-IT on llama.cpp](examples/llamacpp/gemma4/README.md) +- [Qwen3.6-35B-A3B on oMLX (Apple Silicon)](examples/omlx/qwen3.6/README.md) ## Adding providers and models diff --git a/cmd/outfit/main.go b/cmd/outfit/main.go index 91493d23..368103da 100644 --- a/cmd/outfit/main.go +++ b/cmd/outfit/main.go @@ -19,7 +19,7 @@ // outfit unapply [path] # remove what an Outfit file selects // outfit alias [path] [-n ] # name an Outfit; -l lists them // outfit unalias # drop a registered name -// outfit serve [path] # run llama-server for the Outfit (PRESET or MODEL) +// outfit serve [path] # run the PROVIDER's server (llama.cpp or oMLX) // outfit export [-p name] # print the current config as an Outfit // outfit init-providers [path] # write the embedded providers.yaml out // outfit harness [-H name] [-O[=path]] # launch the harness, optionally applying an @@ -50,7 +50,6 @@ import ( "errors" "flag" "fmt" - "net/url" "os" "os/exec" "path/filepath" @@ -64,7 +63,6 @@ import ( "github.com/lucinate-ai/outfit/internal/harness" "github.com/lucinate-ai/outfit/internal/opencode" "github.com/lucinate-ai/outfit/internal/outfit" - "github.com/lucinate-ai/outfit/internal/preset" ) // version is the binary's version. It defaults to "dev" and is overridden at @@ -141,7 +139,7 @@ Usage: outfit unapply [path] (remove what an Outfit selects) outfit alias [path] [-n ] [-l] (name an Outfit; -l lists them) outfit unalias (drop a registered name) - outfit serve [path] [--dry-run] (run llama-server from the PRESET) + outfit serve [path] [--dry-run] (run the PROVIDER's inference server) outfit export [--provider ] outfit init-providers [path] (defaults to ./providers.yaml) outfit harness [] [-H ] [--outfit[=]] [args...] @@ -156,7 +154,8 @@ Flags: -p, --provider provider name (see `+"`outfit list`"+`) -m, --model model id to set as default / to add or remove -a, --alias friendly name for the model (the harness key); for - llama.cpp the server's reported model name under serve + llama.cpp the server's reported model name under serve, + and the preset section serve selects -c, --context context window size for the added model(s); accepts human suffixes (128k, 1m) or an absolute count (200000) -o, --output max output tokens for the added model(s); same format as @@ -187,10 +186,12 @@ alias: registers an Outfit under a short name, which then stands in wherever an name already registered, and --list/-l shows them all. A path on disk always wins over a name, so registering one changes nothing that works. unalias: drops a registered name. The Outfit file itself is left alone. -serve: runs llama-server for the Outfit. With a PRESET (a llama.cpp .ini) it - turns the matching section into the command; otherwise it derives one - from MODEL/ALIAS/CONTEXT/BASEURL. Prints the command before running it; - --dry-run/-n prints without launching the server. +serve: runs the inference server the Outfit's PROVIDER names — llamacpp + (llama-server) or omlx (Apple Silicon). With a PRESET it turns the + matching section into the command, reading it in that engine's flag + vocabulary; otherwise it derives one from the Outfit's own instructions. + Prints the command before running it; --dry-run/-n prints without + launching the server. export: prints the active harness's config as an Outfit (outfit export > Outfit). init-providers: writes the binary's built-in providers.yaml to the working directory (or [path]) so you can customise the catalogue and point @@ -632,162 +633,6 @@ func writeAliases(b *strings.Builder, header bool) error { return nil } -// llamaServerBinary is the llama.cpp server executable that `serve` launches. -// It is a package var so tests can point it at a stub instead of a real build. -var llamaServerBinary = "llama-server" - -// cmdServe reads an Outfit and runs llama-server for it. With a PRESET it turns -// the matching preset section into the command; without one it derives the -// command from the Outfit's own MODEL/ALIAS/CONTEXT/BASEURL. Either way it -// prints the command before running it. The Outfit path defaults to ./Outfit. -func cmdServe(args []string) error { - fs := flag.NewFlagSet("serve", flag.ContinueOnError) - var dryRun bool - fs.BoolVar(&dryRun, "dry-run", false, "print the llama-server command without running it") - fs.BoolVar(&dryRun, "n", false, "print the command without running it (shorthand)") - if err := fs.Parse(args); err != nil { - return err - } - - var path string - if rest := fs.Args(); len(rest) > 0 { - path = rest[0] - } - sel, outfitPath, err := readOutfit("outfit serve ", path) - if err != nil { - return err - } - - var argv []string - if sel.Preset != "" { - presetPath := resolvePresetPath(sel.Preset, outfitPath) - data, err := os.ReadFile(presetPath) - if err != nil { - return fmt.Errorf("reading preset %s: %w", presetPath, err) - } - pre, err := preset.Parse(data) - if err != nil { - return fmt.Errorf("%s: %w", presetPath, err) - } - // The preset's sections are named by the friendly ALIAS, not the - // provider-native MODEL, so that is what selects one. - sec, err := pre.Select(sel.Alias) - if err != nil { - return fmt.Errorf("%s: %w", presetPath, err) - } - // Anything the Outfit states (CONTEXT, BASEURL, ALIAS, MODEL) overrides - // the preset's own values. - overrides, err := outfitServeParams(sel) - if err != nil { - return err - } - argv = pre.Command(llamaServerBinary, sec, overrides) - fmt.Printf("Using preset %s (model %s)\n\n", presetPath, sec.Name) - } else { - if sel.Model == "" { - return fmt.Errorf("serve needs a PRESET or a MODEL (an HF repo like org/model:quant, or a path to a .gguf)") - } - params, err := outfitServeParams(sel) - if err != nil { - return err - } - argv = append([]string{llamaServerBinary}, preset.Flags(params)...) - fmt.Printf("Serving %s from %s\n\n", sel.Model, outfitPath) - } - - fmt.Printf("%s\n\n", preset.FormatCommand(argv)) - if dryRun { - return nil - } - - cmd := exec.Command(argv[0], argv[1:]...) - cmd.Stdin = os.Stdin - cmd.Stdout = os.Stdout - cmd.Stderr = os.Stderr - if err := cmd.Run(); err != nil { - if errors.Is(err, exec.ErrNotFound) || errors.Is(err, os.ErrNotExist) { - return fmt.Errorf("%s not found — install llama.cpp (e.g. brew install llama.cpp) or check the path", argv[0]) - } - return err - } - return nil -} - -// resolvePresetPath resolves an Outfit's PRESET value: a relative one is taken -// against the Outfit's own directory, so an Outfit and its preset travel -// together (the same rule REMOTE uses). -func resolvePresetPath(presetValue, outfitPath string) string { - if filepath.IsAbs(presetValue) { - return presetValue - } - return filepath.Join(filepath.Dir(outfitPath), presetValue) -} - -// outfitServeParams turns the llama-server settings an Outfit states into preset -// params: the provider-native MODEL supplies the model source (hf for a Hugging -// Face repo, model for a .gguf path); ALIAS, CONTEXT, and BASEURL fill in the -// rest. They seed a preset-less command and, with a preset, override its values. -func outfitServeParams(sel outfit.Selection) ([]preset.Param, error) { - var params []preset.Param - if sel.Model != "" { - if isModelPath(sel.Model) { - params = append(params, preset.Param{Key: "model", Value: sel.Model}) - } else { - params = append(params, preset.Param{Key: "hf", Value: sel.Model}) - } - } - if sel.Alias != "" { - params = append(params, preset.Param{Key: "alias", Value: sel.Alias}) - } - if sel.Context != "" { - n, err := contextsize.Parse(sel.Context) - if err != nil { - return nil, err - } - params = append(params, preset.Param{Key: "ctx-size", Value: strconv.Itoa(n)}) - } - if sel.BaseURL != "" { - host, port, err := hostPortFromURL(sel.BaseURL) - if err != nil { - return nil, err - } - if host != "" { - params = append(params, preset.Param{Key: "host", Value: host}) - } - if port != "" { - params = append(params, preset.Param{Key: "port", Value: port}) - } - } - return params, nil -} - -// isModelPath reports whether a MODEL value is a local file rather than a -// Hugging Face repo: an absolute or explicitly-relative path, a home-relative -// path, or anything ending in .gguf. Everything else is treated as org/model. -func isModelPath(model string) bool { - if strings.HasSuffix(strings.ToLower(model), ".gguf") { - return true - } - return strings.HasPrefix(model, "/") || - strings.HasPrefix(model, "./") || - strings.HasPrefix(model, "../") || - strings.HasPrefix(model, "~") -} - -// hostPortFromURL extracts the host and port from a BASEURL so serve can bind -// llama-server to the same endpoint the harness will call. A bare host:port -// with no scheme is accepted too. -func hostPortFromURL(raw string) (host, port string, err error) { - if !strings.Contains(raw, "://") { - raw = "http://" + raw - } - u, err := url.Parse(raw) - if err != nil { - return "", "", fmt.Errorf("invalid BASEURL %q: %w", raw, err) - } - return u.Hostname(), u.Port(), nil -} - // cmdExport reconstructs an Outfit from the active harness's config and prints // it to stdout, so an existing setup can be captured (outfit export > Outfit). func cmdExport(args []string) error { diff --git a/cmd/outfit/serve.go b/cmd/outfit/serve.go new file mode 100644 index 00000000..624db39c --- /dev/null +++ b/cmd/outfit/serve.go @@ -0,0 +1,286 @@ +// Serve: launching a local inference server for an Outfit. The engine is chosen +// by the Outfit's PROVIDER, the same way `outfit remote deploy` picks a cloud +// runner, so one file describes both what dresses the harness and what serves +// it. Kept out of main.go so the dispatch-coverage scan in complete_test.go only +// ever sees run()'s own switch. + +package main + +import ( + "errors" + "flag" + "fmt" + "net/url" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + + "github.com/lucinate-ai/outfit/internal/contextsize" + "github.com/lucinate-ai/outfit/internal/outfit" + "github.com/lucinate-ai/outfit/internal/preset" +) + +// llamaServerBinary is the llama.cpp server executable that `serve` launches. +// It is a package var so tests can point it at a stub instead of a real build. +var llamaServerBinary = "llama-server" + +// omlxBinary is the oMLX executable that `serve` launches. Empty means "look it +// up"; tests set it to a stub, exactly as they do for llamaServerBinary. +var omlxBinary = "" + +// omlxBundleBinary is where the macOS app installs its CLI. oMLX ships as a +// signed app rather than a PATH install, so a user who has only ever launched +// it from the menu bar still has this and nothing on their PATH. +const omlxBundleBinary = "/Applications/oMLX.app/Contents/MacOS/omlx-cli" + +// resolveOMLXBinary finds the oMLX CLI: an explicit override wins, then the +// PATH, then the app bundle. The bundle path is returned unchecked when nothing +// else matches, so the failure surfaces as serve's usual not-found hint naming +// a concrete path rather than as a bare "omlx-cli". +func resolveOMLXBinary() string { + if omlxBinary != "" { + return omlxBinary + } + if p, err := exec.LookPath("omlx-cli"); err == nil { + return p + } + return omlxBundleBinary +} + +// serveEngine is a local inference server `outfit serve` can launch: which +// binary to run, the subcommand it needs, the dialect its preset is written in, +// and how the Outfit's own instructions turn into its flags. +type serveEngine struct { + // binary is resolved late so tests can stub it after engineFor has run. + binary func() string + subcommand []string + dialect preset.Dialect + // params turns the Outfit's instructions into this engine's flags. + params func(outfit.Selection) ([]preset.Param, error) + // needsModel marks an engine that cannot start without being told what to + // load. oMLX serves a whole model directory and picks per request, so it + // starts happily with neither a PRESET nor a MODEL. + needsModel bool + // installHint completes " not found — ...". + installHint string +} + +// engineFor maps an Outfit's PROVIDER to the engine `serve` launches locally. +// It is the local twin of runnerFor: PROVIDER already names the engine, so no +// separate keyword is needed. Providers that are not self-hosted engines have +// nothing to launch. +func engineFor(provider string) (serveEngine, error) { + switch provider { + case "llamacpp": + return serveEngine{ + binary: func() string { return llamaServerBinary }, + dialect: preset.LlamaCpp, + params: llamacppServeParams, + needsModel: true, + installHint: "install llama.cpp (e.g. brew install llama.cpp) or check the path", + }, nil + case "omlx": + return serveEngine{ + binary: resolveOMLXBinary, + subcommand: []string{"serve"}, + dialect: preset.OMLX, + params: omlxServeParams, + installHint: "install oMLX (https://omlx.ai) or check the path", + }, nil + default: + return serveEngine{}, fmt.Errorf( + "PROVIDER %q cannot be served locally: serve runs a self-hosted engine, so use llamacpp or omlx", + provider) + } +} + +// cmdServe reads an Outfit and runs the inference server its PROVIDER names. +// With a PRESET it turns the matching preset section into the command; without +// one it derives the command from the Outfit's own instructions. Either way it +// prints the command before running it. The Outfit path defaults to ./Outfit. +func cmdServe(args []string) error { + fs := flag.NewFlagSet("serve", flag.ContinueOnError) + var dryRun bool + fs.BoolVar(&dryRun, "dry-run", false, "print the server command without running it") + fs.BoolVar(&dryRun, "n", false, "print the command without running it (shorthand)") + if err := fs.Parse(args); err != nil { + return err + } + + var path string + if rest := fs.Args(); len(rest) > 0 { + path = rest[0] + } + sel, outfitPath, err := readOutfit("outfit serve ", path) + if err != nil { + return err + } + engine, err := engineFor(sel.Provider) + if err != nil { + return err + } + + // Anything the Outfit states overrides the preset's own values. + params, err := engine.params(sel) + if err != nil { + return err + } + + binary := engine.binary() + var argv []string + if sel.Preset != "" { + presetPath := resolvePresetPath(sel.Preset, outfitPath) + data, err := os.ReadFile(presetPath) + if err != nil { + return fmt.Errorf("reading preset %s: %w", presetPath, err) + } + pre, err := preset.Parse(data) + if err != nil { + return fmt.Errorf("%s: %w", presetPath, err) + } + // The preset's sections are named by the friendly ALIAS, not the + // provider-native MODEL, so that is what selects one. + sec, err := pre.Select(sel.Alias) + if err != nil { + return fmt.Errorf("%s: %w", presetPath, err) + } + argv = pre.CommandIn(engine.dialect, binary, engine.subcommand, sec, params) + fmt.Printf("Using preset %s (model %s)\n\n", presetPath, sec.Name) + } else { + if engine.needsModel && sel.Model == "" { + return fmt.Errorf("serve needs a PRESET or a MODEL (an HF repo like org/model:quant, or a path to a .gguf)") + } + argv = append([]string{binary}, engine.subcommand...) + argv = append(argv, engine.dialect.Flags(params)...) + if sel.Model != "" { + fmt.Printf("Serving %s from %s\n\n", sel.Model, outfitPath) + } else { + // An engine that needs no model to start (oMLX serves a whole + // directory) has nothing to name but itself. + fmt.Printf("Starting %s from %s\n\n", sel.Provider, outfitPath) + } + } + + fmt.Printf("%s\n\n", preset.FormatCommand(argv)) + if dryRun { + return nil + } + + cmd := exec.Command(argv[0], argv[1:]...) + cmd.Stdin = os.Stdin + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + if err := cmd.Run(); err != nil { + if errors.Is(err, exec.ErrNotFound) || errors.Is(err, os.ErrNotExist) { + return fmt.Errorf("%s not found — %s", argv[0], engine.installHint) + } + return err + } + return nil +} + +// resolvePresetPath resolves an Outfit's PRESET value: a relative one is taken +// against the Outfit's own directory, so an Outfit and its preset travel +// together (the same rule REMOTE uses). +func resolvePresetPath(presetValue, outfitPath string) string { + if filepath.IsAbs(presetValue) { + return presetValue + } + return filepath.Join(filepath.Dir(outfitPath), presetValue) +} + +// omlxServeParams turns the oMLX settings an Outfit states into preset params. +// Only the bind address maps: oMLX serves a whole --model-dir and picks the +// model per request, so MODEL and ALIAS keep their usual job of naming what the +// harness asks for, and CONTEXT sizes the harness's window rather than the +// server (oMLX has no context flag). Everything else — the model directory, +// memory guard, SSD cache — comes from the PRESET or from oMLX's own settings. +// +// The API key is deliberately not passed: serve prints the command it runs, and +// oMLX takes its key as a command-line flag, so passing one would put the secret +// on screen and in the process table. Auth belongs in oMLX's own settings. +func omlxServeParams(sel outfit.Selection) ([]preset.Param, error) { + return bindAddressParams(sel) +} + +// llamacppServeParams turns the llama-server settings an Outfit states into +// preset params: the provider-native MODEL supplies the model source (hf for a +// Hugging Face repo, model for a .gguf path); ALIAS, CONTEXT, and BASEURL fill +// in the rest. They seed a preset-less command and, with a preset, override its +// values. +func llamacppServeParams(sel outfit.Selection) ([]preset.Param, error) { + var params []preset.Param + if sel.Model != "" { + if isModelPath(sel.Model) { + params = append(params, preset.Param{Key: "model", Value: sel.Model}) + } else { + params = append(params, preset.Param{Key: "hf", Value: sel.Model}) + } + } + if sel.Alias != "" { + params = append(params, preset.Param{Key: "alias", Value: sel.Alias}) + } + if sel.Context != "" { + n, err := contextsize.Parse(sel.Context) + if err != nil { + return nil, err + } + params = append(params, preset.Param{Key: "ctx-size", Value: strconv.Itoa(n)}) + } + bind, err := bindAddressParams(sel) + if err != nil { + return nil, err + } + return append(params, bind...), nil +} + +// bindAddressParams turns an Outfit's BASEURL into the host and port flags that +// bind a server to the endpoint the harness will call. Both engines spell these +// the same way. With no BASEURL it yields nothing, so the server's own defaults +// stand. +func bindAddressParams(sel outfit.Selection) ([]preset.Param, error) { + if sel.BaseURL == "" { + return nil, nil + } + host, port, err := hostPortFromURL(sel.BaseURL) + if err != nil { + return nil, err + } + var params []preset.Param + if host != "" { + params = append(params, preset.Param{Key: "host", Value: host}) + } + if port != "" { + params = append(params, preset.Param{Key: "port", Value: port}) + } + return params, nil +} + +// isModelPath reports whether a MODEL value is a local file rather than a +// Hugging Face repo: an absolute or explicitly-relative path, a home-relative +// path, or anything ending in .gguf. Everything else is treated as org/model. +func isModelPath(model string) bool { + if strings.HasSuffix(strings.ToLower(model), ".gguf") { + return true + } + return strings.HasPrefix(model, "/") || + strings.HasPrefix(model, "./") || + strings.HasPrefix(model, "../") || + strings.HasPrefix(model, "~") +} + +// hostPortFromURL extracts the host and port from a BASEURL so serve can bind +// llama-server to the same endpoint the harness will call. A bare host:port +// with no scheme is accepted too. +func hostPortFromURL(raw string) (host, port string, err error) { + if !strings.Contains(raw, "://") { + raw = "http://" + raw + } + u, err := url.Parse(raw) + if err != nil { + return "", "", fmt.Errorf("invalid BASEURL %q: %w", raw, err) + } + return u.Hostname(), u.Port(), nil +} diff --git a/cmd/outfit/serve_test.go b/cmd/outfit/serve_test.go index 5c41ca7b..c9978437 100644 --- a/cmd/outfit/serve_test.go +++ b/cmd/outfit/serve_test.go @@ -28,18 +28,56 @@ func writePresetOutfit(t *testing.T, outfitBody string) string { return outfitPath } -// stubLlamaServer points llamaServerBinary at a script that records its argv to -// argsFile, and restores the original binary afterwards. -func stubLlamaServer(t *testing.T, argsFile string) { +// stubServer writes a script named binName that records its argv to argsFile, +// points *target at it, and restores the original afterwards. +func stubServer(t *testing.T, target *string, binName, argsFile string) { t.Helper() - script := filepath.Join(t.TempDir(), "llama-server") + script := filepath.Join(t.TempDir(), binName) body := "#!/bin/sh\nprintf '%s\\n' \"$@\" > " + argsFile + "\n" if err := os.WriteFile(script, []byte(body), 0o755); err != nil { t.Fatal(err) } - orig := llamaServerBinary - llamaServerBinary = script - t.Cleanup(func() { llamaServerBinary = orig }) + orig := *target + *target = script + t.Cleanup(func() { *target = orig }) +} + +// stubLlamaServer points llamaServerBinary at a script that records its argv to +// argsFile, and restores the original binary afterwards. +func stubLlamaServer(t *testing.T, argsFile string) { + t.Helper() + stubServer(t, &llamaServerBinary, "llama-server", argsFile) +} + +// stubOMLX does the same for the oMLX CLI, so no Apple Silicon (or oMLX +// install) is needed to pin the argv serve builds for it. +func stubOMLX(t *testing.T, argsFile string) { + t.Helper() + stubServer(t, &omlxBinary, "omlx-cli", argsFile) +} + +// omlxPreset is a preset written in oMLX's own flag vocabulary. `m` is here +// deliberately: llama.cpp's dialect would rewrite it to --model, and this preset +// must not be read that way. +const omlxPreset = `[*] +model-dir = /Users/me/models +host = 127.0.0.1 +port = 8000 + +[qwen] +memory-guard = safe +m = should-stay-literal +` + +// writeOMLXOutfit writes an oMLX preset.ini and an Outfit referencing it into a +// fresh temp dir, and returns the Outfit's path. +func writeOMLXOutfit(t *testing.T, outfitBody string) string { + t.Helper() + dir := t.TempDir() + mustWrite(t, filepath.Join(dir, "preset.ini"), omlxPreset) + outfitPath := filepath.Join(dir, "Outfit") + mustWrite(t, outfitPath, outfitBody) + return outfitPath } func TestCmdServe_PresetDryRun(t *testing.T) { @@ -271,3 +309,183 @@ func TestCmdServe_RelativePresetResolvesToOutfitDir(t *testing.T) { t.Errorf("preset not resolved relative to the Outfit dir:\n%s", out) } } + +// TestCmdServe_OMLXDryRun pins the whole oMLX shape: the `serve` subcommand, and +// a bind address taken from BASEURL. +func TestCmdServe_OMLXDryRun(t *testing.T) { + dir := t.TempDir() + outfitPath := filepath.Join(dir, "Outfit") + mustWrite(t, outfitPath, "PROVIDER omlx\nBASEURL http://127.0.0.1:9100/v1\n") + + out := captureStdout(t, func() { + if err := cmdServe([]string{"--dry-run", outfitPath}); err != nil { + t.Fatalf("cmdServe: %v", err) + } + }) + + for _, want := range []string{"serve", "--host 127.0.0.1", "--port 9100"} { + if !strings.Contains(out, want) { + t.Errorf("command missing %q:\n%s", want, out) + } + } + // llama.cpp's vocabulary must not leak into another engine's command. + for _, unwanted := range []string{"llama-server", "--hf-repo", "--ctx-size"} { + if strings.Contains(out, unwanted) { + t.Errorf("command should not contain %q:\n%s", unwanted, out) + } + } +} + +// TestCmdServe_OMLXNeedsNoModel checks that oMLX starts with neither a PRESET +// nor a MODEL: it serves a whole model directory and picks per request, so +// llama.cpp's "needs a PRESET or a MODEL" rule must not apply to it. +func TestCmdServe_OMLXNeedsNoModel(t *testing.T) { + dir := t.TempDir() + outfitPath := filepath.Join(dir, "Outfit") + mustWrite(t, outfitPath, "PROVIDER omlx\n") + + out := captureStdout(t, func() { + if err := cmdServe([]string{"--dry-run", outfitPath}); err != nil { + t.Fatalf("cmdServe: %v", err) + } + }) + + if !strings.Contains(out, "serve") { + t.Errorf("expected a serve command:\n%s", out) + } + // With no BASEURL, oMLX's own defaults stand. + if strings.Contains(out, "--host") || strings.Contains(out, "--port") { + t.Errorf("no BASEURL should mean no bind flags:\n%s", out) + } +} + +// TestCmdServe_OMLXPresetKeysAreNotAliased is the guard on the dialect split: an +// oMLX preset is read in oMLX's vocabulary, so a key llama.cpp would rewrite is +// left exactly as written. +func TestCmdServe_OMLXPresetKeysAreNotAliased(t *testing.T) { + outfitPath := writeOMLXOutfit(t, "PROVIDER omlx\nALIAS qwen\nPRESET preset.ini\n") + + out := captureStdout(t, func() { + if err := cmdServe([]string{"--dry-run", outfitPath}); err != nil { + t.Fatalf("cmdServe: %v", err) + } + }) + + for _, want := range []string{"--model-dir /Users/me/models", "--memory-guard safe", "-m should-stay-literal"} { + if !strings.Contains(out, want) { + t.Errorf("command missing %q:\n%s", want, out) + } + } + if strings.Contains(out, "--model should-stay-literal") { + t.Errorf("llama.cpp aliasing leaked into the oMLX dialect:\n%s", out) + } +} + +// TestCmdServe_OMLXOutfitOverridesPreset checks the Outfit still wins over its +// preset for the settings it states, as it does for llama.cpp. +func TestCmdServe_OMLXOutfitOverridesPreset(t *testing.T) { + outfitPath := writeOMLXOutfit(t, "PROVIDER omlx\nALIAS qwen\nPRESET preset.ini\nBASEURL http://0.0.0.0:9999/v1\n") + + out := captureStdout(t, func() { + if err := cmdServe([]string{"--dry-run", outfitPath}); err != nil { + t.Fatalf("cmdServe: %v", err) + } + }) + + if !strings.Contains(out, "--host 0.0.0.0") || !strings.Contains(out, "--port 9999") { + t.Errorf("Outfit should override the preset's bind address:\n%s", out) + } + if strings.Contains(out, "--port 8000") { + t.Errorf("preset port should have been overridden:\n%s", out) + } +} + +// TestCmdServe_OMLXNeverPassesAPIKey is a security guard: serve prints the +// command it runs, and oMLX takes its key on the command line, so a resolved +// secret must never reach either. +func TestCmdServe_OMLXNeverPassesAPIKey(t *testing.T) { + t.Setenv("OPENAI_API_KEY", "sk-must-not-appear") + dir := t.TempDir() + outfitPath := filepath.Join(dir, "Outfit") + mustWrite(t, outfitPath, "PROVIDER omlx\nBASEURL http://127.0.0.1:9100/v1\n") + + out := captureStdout(t, func() { + if err := cmdServe([]string{"--dry-run", outfitPath}); err != nil { + t.Fatalf("cmdServe: %v", err) + } + }) + + if strings.Contains(out, "sk-must-not-appear") || strings.Contains(out, "--api-key") { + t.Errorf("serve must not pass or print an API key:\n%s", out) + } +} + +// TestCmdServe_OMLXRuns checks the stubbed binary is actually executed with the +// subcommand first. +func TestCmdServe_OMLXRuns(t *testing.T) { + argsFile := filepath.Join(t.TempDir(), "args") + stubOMLX(t, argsFile) + dir := t.TempDir() + outfitPath := filepath.Join(dir, "Outfit") + mustWrite(t, outfitPath, "PROVIDER omlx\nBASEURL http://127.0.0.1:9100/v1\n") + + captureStdout(t, func() { + if err := cmdServe([]string{outfitPath}); err != nil { + t.Fatalf("cmdServe: %v", err) + } + }) + + got, err := os.ReadFile(argsFile) + if err != nil { + t.Fatalf("reading recorded args: %v", err) + } + args := strings.Fields(string(got)) + if len(args) == 0 || args[0] != "serve" { + t.Errorf("expected the serve subcommand first, got %v", args) + } + if !strings.Contains(string(got), "9100") { + t.Errorf("recorded args missing the port:\n%s", got) + } +} + +// TestCmdServe_OMLXNotFound checks the install hint names oMLX rather than +// llama.cpp. +func TestCmdServe_OMLXNotFound(t *testing.T) { + orig := omlxBinary + omlxBinary = filepath.Join(t.TempDir(), "definitely-not-installed") + t.Cleanup(func() { omlxBinary = orig }) + dir := t.TempDir() + outfitPath := filepath.Join(dir, "Outfit") + mustWrite(t, outfitPath, "PROVIDER omlx\n") + + var err error + captureStdout(t, func() { err = cmdServe([]string{outfitPath}) }) + if err == nil || !strings.Contains(err.Error(), "not found") { + t.Fatalf("expected a not-found error, got %v", err) + } + if !strings.Contains(err.Error(), "oMLX") { + t.Errorf("hint should name oMLX, got %v", err) + } + if strings.Contains(err.Error(), "llama.cpp") { + t.Errorf("hint should not name llama.cpp, got %v", err) + } +} + +// TestCmdServe_UnsupportedProvider pins the behaviour change: serve used to run +// llama-server whatever the PROVIDER said, which quietly served the wrong engine. +func TestCmdServe_UnsupportedProvider(t *testing.T) { + dir := t.TempDir() + outfitPath := filepath.Join(dir, "Outfit") + mustWrite(t, outfitPath, "PROVIDER ollama\nMODEL llama3.2\n") + + var err error + captureStdout(t, func() { err = cmdServe([]string{"--dry-run", outfitPath}) }) + if err == nil { + t.Fatal("expected an error for a provider that is not a local engine") + } + for _, want := range []string{"ollama", "llamacpp", "omlx"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error should mention %q, got %v", want, err) + } + } +} diff --git a/docs/README.md b/docs/README.md index 306dd105..d30225f3 100644 --- a/docs/README.md +++ b/docs/README.md @@ -15,8 +15,8 @@ Four words carry the whole tool: Pi is also supported. Chosen at runtime, so the same selection works for either. See [`outfit harness`](commands/harness.md). - **Provider** — what `outfit` can configure, from a built-in - catalogue: OpenRouter, AWS Bedrock, Ollama, llama.cpp, vLLM, or any - OpenAI-compatible endpoint. See + catalogue: OpenRouter, AWS Bedrock, Ollama, llama.cpp, vLLM, oMLX (Apple + Silicon), or any OpenAI-compatible endpoint. See [`outfit list`](commands/list.md). - **Outfit file** — a small, declarative file (like a `Dockerfile`, but for your agent's model) that captures one selection so you can commit it and @@ -46,7 +46,7 @@ Four words carry the whole tool: | [`outfit unapply`](commands/unapply.md) | Remove what an `Outfit` file selects | | [`outfit alias`](commands/alias.md) | Name an `Outfit` so the name works anywhere a path does | | [`outfit unalias`](commands/unalias.md) | Drop a registered name | -| [`outfit serve`](commands/serve.md) | Run `llama-server` for the model an `Outfit` names | +| [`outfit serve`](commands/serve.md) | Run the inference server for the model an `Outfit` names | | [`outfit remote`](commands/remote.md) | Run the model on a cloud GPU that stops when you do | | [`outfit export`](commands/export.md) | Capture the current setup as an `Outfit` | | [`outfit harness`](commands/harness.md) | Launch the agent, optionally dressing it first | @@ -63,7 +63,7 @@ Four words carry the whole tool: | `OUTFIT_PROVIDERS` | Path to a custom provider catalogue (`--providers` beats it) | | `OUTFIT_BASE_URL` | Overrides any provider's API base URL (`--base-url`/`-u` beats it) | | `DEEPSEEK_API_KEY`, `OPENAI_API_KEY`, … | Provider API keys — `outfit list` shows which each provider reads | -| `OLLAMA_BASE_URL`, `LLAMACPP_BASE_URL`, `OPENAI_BASE_URL` | Per-provider endpoint overrides | +| `OLLAMA_BASE_URL`, `LLAMACPP_BASE_URL`, `OMLX_BASE_URL`, `VLLM_BASE_URL`, `OPENAI_BASE_URL` | Per-provider endpoint overrides | | `AWS_REGION` | Region for AWS Bedrock | Keys are looked up in a `.env` file **beside the `Outfit` being applied** first @@ -72,5 +72,5 @@ shell environment — so a project keeps its own key next to the file that needs it, the same way `PRESET` and `REMOTE` travel with an Outfit. They are **never written into the agent's config** — outfit writes a reference the agent resolves when it runs, and `outfit harness` passes the keys it can resolve to the agent it launches. If you start the agent yourself, -set the variable in your own environment. Local providers (Ollama, llama.cpp on -localhost) need no key; Bedrock uses your AWS credentials. +set the variable in your own environment. Local providers (Ollama, llama.cpp and +oMLX on localhost) need no key; Bedrock uses your AWS credentials. diff --git a/docs/commands/serve.md b/docs/commands/serve.md index a3a48b31..f682f4a9 100644 --- a/docs/commands/serve.md +++ b/docs/commands/serve.md @@ -1,11 +1,11 @@ # outfit serve -Run `llama-server` for the model an [`Outfit` file](../outfit-file.md) names — -so the same file that points your agent at a local model can also start the -server behind it. +Run the inference server for the model an [`Outfit` file](../outfit-file.md) +names — so the same file that points your agent at a local model can also start +the server behind it. ```sh -outfit serve # reads ./Outfit and runs llama-server +outfit serve # reads ./Outfit and runs its PROVIDER's server outfit serve path/to/Outfit outfit serve qwen3.6-27b # a name registered with `outfit alias` outfit serve --dry-run # print the command without launching the server @@ -15,7 +15,27 @@ It prints the command before running it, and never touches your agent's config — pair it with [`outfit apply`](apply.md) to point the agent at the server. -## Simple case — straight from the Outfit +## The engine comes from `PROVIDER` + +`PROVIDER` already names the engine, so `serve` needs no keyword of its own — +the same way [`outfit remote deploy`](remote.md) picks the engine for a cloud +GPU: + +| `PROVIDER` | `serve` runs | +| ---------- | ------------ | +| `llamacpp` | `llama-server` | +| `omlx` | [oMLX](https://omlx.ai) on Apple Silicon | + +Any other provider is an error: `serve` launches a self-hosted engine, and the +rest of the catalogue names endpoints somebody else runs. + +Each engine reads its `PRESET` in its **own** flag vocabulary. That matters: +llama.cpp's short aliases would rewrite another engine's keys (`m` to `--model`, +`c` to `--ctx-size`), so a preset is never portable between engines. + +## llama.cpp + +### Simple case — straight from the Outfit With no `PRESET`, `serve` builds the command from the Outfit itself: @@ -30,7 +50,7 @@ BASEURL http://127.0.0.1:8080/v1 # llama-server --host/--port `MODEL` becomes `-hf` (a Hugging Face repo) or `-m` (anything that looks like a path or ends in `.gguf`); `ALIAS`, `CONTEXT`, and `BASEURL` fill in the rest. -## Full control — a llama.cpp preset +### Full control — a llama.cpp preset For flags an Outfit doesn't model — `-ngl`, `--jinja`, KV-cache types, draft models — point at a llama.cpp @@ -62,17 +82,63 @@ Keys map straight to flags — `ctx-size = 262144` becomes `--ctx-size 262144`, A relative `PRESET` path resolves against the Outfit's own directory, so the pair can travel together. +## oMLX + +[oMLX](https://omlx.ai) serves MLX models on Apple Silicon. It differs from +llama.cpp in one way that shapes everything else: it loads a whole **model +directory** and picks the model per request, rather than being launched with one +model. So a bare Outfit is enough to start it: + +```dockerfile +PROVIDER omlx +BASEURL http://127.0.0.1:8000/v1 # omlx-cli serve --host/--port +``` + +- `BASEURL` sets the bind address. With none, oMLX's own defaults stand. +- `MODEL` and `ALIAS` keep their usual job of naming what the *agent* asks for; + they are not launch flags. +- `CONTEXT` sizes the *harness's* window — oMLX has no context flag. + +Everything else — the model directory, the memory guard, the SSD cache — comes +from a `PRESET` written in oMLX's own flags, or from oMLX's settings +(`~/.omlx/settings.json`, and its admin panel): + +```ini +[*] +model-dir = /Users/you/models + +[default] # selected by the Outfit's ALIAS +memory-guard = safe +paged-ssd-cache-dir = /Users/you/.omlx/cache +max-concurrent-requests = 16 +``` + +Preset values are passed to the server verbatim, so `~` is not expanded — write +paths out in full. + +`serve` never passes `--api-key`. It prints the command it runs, and oMLX takes +its key on the command line, so passing one would put the secret on your screen +and in the process table. If you want auth on the server, configure it in oMLX; +`outfit add`/`apply` still picks up `OPENAI_API_KEY` for the agent's own config. + +### Finding the binary + +oMLX ships as a macOS app, so `serve` looks for `omlx-cli` on your `PATH` first +and falls back to `/Applications/oMLX.app/Contents/MacOS/omlx-cli`. If you've +only ever launched it from the menu bar, the fallback is the one that finds it. + ## Flags | Flag | Meaning | | ---- | ------- | -| `-n`, `--dry-run` | Print the `llama-server` command without running it | +| `-n`, `--dry-run` | Print the server command without running it | ## Notes -- `serve` needs `llama-server` on your `PATH` (e.g. - `brew install llama.cpp`). -- Without a `PRESET`, the Outfit must name a `MODEL`. +- `serve` needs the engine installed: `llama-server` on your `PATH` (e.g. + `brew install llama.cpp`), or [oMLX](https://omlx.ai). +- For llama.cpp, an Outfit with no `PRESET` must name a `MODEL`. oMLX needs + neither. ## See also diff --git a/docs/outfit-file.md b/docs/outfit-file.md index 5e9a6d17..1f62b76c 100644 --- a/docs/outfit-file.md +++ b/docs/outfit-file.md @@ -13,7 +13,7 @@ ALIAS deepseek # optional; friendly name for the model CONTEXT 128k # optional; context window OUTPUT 32k # optional; max output tokens BASEURL https://gateway/v1 # optional; API base URL override -PRESET ./preset.ini # optional; llama.cpp preset for `outfit serve` +PRESET ./preset.ini # optional; engine preset for `outfit serve` ``` Applying it is the same as running the equivalent @@ -103,9 +103,11 @@ Rules: - `BASEURL` overrides the provider's API base URL — handy for a gateway or a llama.cpp server on a non-default port. `URL`, `BASE-URL`, and `BASE_URL` are accepted as aliases. -- `PRESET` points at a llama.cpp preset `.ini`, used only by +- `PRESET` points at a preset `.ini`, used only by [`outfit serve`](commands/serve.md); `apply` ignores it. A relative path is - resolved against the Outfit's own directory. + resolved against the Outfit's own directory. The file is read in the flag + vocabulary of the engine `PROVIDER` names, so a preset written for llama.cpp + is not portable to oMLX and vice versa. - Keywords are **case-insensitive** — `provider`, `Provider`, and `PROVIDER` are all accepted — but **UPPERCASE is canonical** and is what `outfit export` writes. diff --git a/examples/omlx/qwen3.6/Outfit b/examples/omlx/qwen3.6/Outfit new file mode 100644 index 00000000..4b1357f7 --- /dev/null +++ b/examples/omlx/qwen3.6/Outfit @@ -0,0 +1,8 @@ +# Qwen3.6-35B-A3B served locally by oMLX on Apple Silicon. +# oMLX loads a whole model directory and picks per request, so MODEL names what +# opencode asks for — it is not a launch flag. +PROVIDER omlx +MODEL Qwen3.6-35B-A3B-4bit # the directory name under the preset's model-dir +CONTEXT 32768 # opencode's context window +PRESET ./preset.ini # `outfit serve` launches omlx-cli from this +# BASEURL http://127.0.0.1:8000/v1 # uncomment for a non-default host/port diff --git a/examples/omlx/qwen3.6/README.md b/examples/omlx/qwen3.6/README.md new file mode 100644 index 00000000..77ea27a2 --- /dev/null +++ b/examples/omlx/qwen3.6/README.md @@ -0,0 +1,152 @@ +# Qwen3.6-35B-A3B on oMLX (Apple Silicon) + +Run an MLX build of Qwen3.6-35B-A3B on your Mac with [oMLX](https://omlx.ai), +then point opencode at it with the [`Outfit`](Outfit) in this directory. + +`A3B` means it's a mixture-of-experts model: ~35B total parameters but only ~3B +active per token, so it's far lighter to run than its size suggests — which is +what makes it practical on a laptop's unified memory. + +oMLX is built on [MLX](https://github.com/ml-explore/mlx), Apple's array +framework, so it runs on the GPU and Neural Engine rather than through a CUDA +path. Its headline feature is **paged SSD caching**: KV-cache blocks are +persisted to disk, so returning to a long prompt restores instead of recomputing +it. For a coding agent, which re-sends a large and mostly-unchanged context on +every turn, that is the difference between a usable and an unusable setup. + +## Prerequisites + +- An **Apple Silicon** Mac (M1 or later). oMLX is Apple Silicon only — there is + no Intel or Linux build, and no cloud equivalent, so + [`outfit remote`](../../../docs/commands/remote.md) cannot deploy it. +- [oMLX](https://omlx.ai), installed from its DMG or from source. +- Enough unified memory for the weights. The 4-bit build is roughly 20 GB, so + plan for a 32 GB machine or better. + +## 1. Pull the model + +oMLX's admin panel (`http://localhost:8000/admin`) has a Hugging Face browser +that downloads a model in one click, which is the easiest route. + +To do it from the shell instead, put the model in the directory oMLX serves +from: + +```sh +hf download mlx-community/Qwen3.6-35B-A3B-4bit \ + --local-dir ~/models/Qwen3.6-35B-A3B-4bit +``` + +The **directory name** is what the model is called over the API — +`Qwen3.6-35B-A3B-4bit` here — unless you set an alias for it in the admin panel. +That is the name the `Outfit` uses as its `MODEL`. + +## 2. Start oMLX + +```sh +omlx-cli serve \ + --model-dir ~/models \ + --memory-guard safe \ + --paged-ssd-cache-dir ~/.omlx/cache \ + --max-concurrent-requests 8 \ + --host 127.0.0.1 --port 8000 +``` + +What the flags do: + +- `--model-dir` — the directory of models to serve. oMLX loads *all* of them and + picks per request, which is why there's no "the model" flag here. +- `--memory-guard safe` — keep memory headroom for the rest of the machine. + `balanced` gives the model more; `--memory-guard-gb` sets a ceiling directly. +- `--paged-ssd-cache-dir` — persist KV-cache blocks to SSD. This is the setting + that makes long agent contexts fast on repeat turns. +- `--max-concurrent-requests` — continuous batching width (default 8). +- `--host`/`--port` — the OpenAI-compatible API is served at + `http://127.0.0.1:8000/v1`. + +Installed from the DMG and haven't put it on your `PATH`? The binary lives at +`/Applications/oMLX.app/Contents/MacOS/omlx-cli`. + +Rather than remember those flags, this directory keeps them in a +[`preset.ini`](preset.ini) and lets `outfit` build and run the command (with the +paths spelled out in full — preset values reach the server verbatim, so `~` is +not expanded): + +```sh +outfit serve # from this directory; reads ./Outfit and its PRESET +outfit serve --dry-run # print the omlx-cli command without running it +``` + +`serve` finds `omlx-cli` on your `PATH`, falling back to the app bundle — so +this works even if you've only ever launched oMLX from the menu bar. + +Note the preset holds `omlx-cli` flags, **not** llama.cpp's. Each engine's +preset is read in its own vocabulary, so this file and the ones under +[`examples/llamacpp/`](../../llamacpp/) are not interchangeable. + +### Check it's up + +```sh +curl http://127.0.0.1:8000/v1/models +``` + +This is also how to confirm what your model is actually called — the response +lists the names requests must use. + +## 3. Point opencode at it + +oMLX speaks the OpenAI-compatible API, which is what the `omlx` provider targets +(default base URL `http://localhost:8000/v1`). Apply the [`Outfit`](Outfit) in +this directory: + +```sh +outfit apply examples/omlx/qwen3.6/Outfit +# or, from this directory: +outfit apply +``` + +The Outfit is: + +```dockerfile +PROVIDER omlx +MODEL Qwen3.6-35B-A3B-4bit # the directory name under the preset's model-dir +CONTEXT 32768 +PRESET ./preset.ini +``` + +`MODEL` matters more here than it does for a single-model llama.cpp server: +oMLX serves everything in its model directory, so the name is what actually +selects the model per request. It must match the directory name (or the alias +you set in the admin panel). + +`CONTEXT` sets opencode's context window. Unlike llama.cpp there is no +`--ctx-size` to keep it in step with — oMLX sizes its cache dynamically — so +this is purely about what opencode will send. + +Running on a different host or port? Add a `BASEURL` line (the file ships one +commented out): + +```dockerfile +BASEURL http://127.0.0.1:9100/v1 +``` + +That sets both what opencode calls and what `serve` binds to. + +## A note on API keys + +A local oMLX server needs no key, and `outfit` writes none: the `omlx` provider +is marked `apiKeyOptional`, so a localhost endpoint gets no `apiKey` at all. + +Serving to another machine on your network? Start oMLX with `--api-key`, export +the same value as `OPENAI_API_KEY`, and point `OMLX_BASE_URL` at the Mac — +`outfit` then writes an environment *reference* into the agent's config, never +the secret itself. + +`outfit serve` never passes `--api-key`: it prints the command it runs, so a key +there would land on your screen and in the process table. Configure auth in oMLX +itself. + +## See also + +- [`outfit serve`](../../../docs/commands/serve.md) — the full command reference +- [The `Outfit` file](../../../docs/outfit-file.md) — full syntax +- The same model on llama.cpp: [`examples/llamacpp/qwen3.6`](../../llamacpp/qwen3.6/README.md) diff --git a/examples/omlx/qwen3.6/preset.ini b/examples/omlx/qwen3.6/preset.ini new file mode 100644 index 00000000..6aac6f95 --- /dev/null +++ b/examples/omlx/qwen3.6/preset.ini @@ -0,0 +1,18 @@ +# An oMLX preset, mirroring the omlx-cli command in this directory's README. +# `outfit serve` turns it into that command and runs it. +# +# These are `omlx-cli serve` flags with their leading dashes stripped. Note that +# they are NOT llama.cpp's: each engine's preset is read in its own vocabulary, +# so this file is not interchangeable with the ones under examples/llamacpp/. + +[*] +host = 127.0.0.1 +port = 8000 + +# Paths are passed to the server verbatim, so `~` is not expanded — spell them +# out in full. +[default] # served with no ALIAS: the only section +model-dir = /Users/you/models +memory-guard = safe # leave headroom for the rest of the machine +paged-ssd-cache-dir = /Users/you/.omlx/cache # persist KV blocks across requests +max-concurrent-requests = 8 diff --git a/internal/catalog/catalog_test.go b/internal/catalog/catalog_test.go index db51feb1..3cc2df47 100644 --- a/internal/catalog/catalog_test.go +++ b/internal/catalog/catalog_test.go @@ -673,3 +673,75 @@ func TestBuildPiProvider_RemoteOptionalKeyKeepsReference(t *testing.T) { t.Errorf("apiKey = %q, want the placeholder for a local server", prov.APIKey) } } + +// TestBuildProviderBlock_OMLXKeylessLocal checks that a local oMLX server gets +// no apiKey option at all: the var points nowhere useful for a server that needs +// no auth, and writing it would make opencode fail to resolve it. +func TestBuildProviderBlock_OMLXKeylessLocal(t *testing.T) { + cat, _ := Load() + p := cat.Providers["omlx"] + if p == nil { + t.Fatal("omlx missing from the catalogue") + } + block, defaultModel, err := BuildProviderBlock("omlx", p, "my-model", "", noEnv) + if err != nil { + t.Fatalf("BuildProviderBlock: %v", err) + } + if defaultModel != "omlx/my-model" { + t.Errorf("defaultModel = %q, want omlx/my-model", defaultModel) + } + options := block["options"].(map[string]any) + if _, ok := options["apiKey"]; ok { + t.Errorf("a keyless local server should get no apiKey: %v", options) + } + if options["baseURL"] != "http://localhost:8000/v1" { + t.Errorf("baseURL = %v", options["baseURL"]) + } +} + +// TestBuildProviderBlock_OMLXReferencesKeyWhenSet checks the other half of +// apiKeyOptional: a key that is set is written as an env reference, never as the +// resolved secret. +func TestBuildProviderBlock_OMLXReferencesKeyWhenSet(t *testing.T) { + cat, _ := Load() + block, _, err := BuildProviderBlock("omlx", cat.Providers["omlx"], "my-model", "", + envMap(map[string]string{"OPENAI_API_KEY": "sk-secret"})) + if err != nil { + t.Fatalf("BuildProviderBlock: %v", err) + } + options := block["options"].(map[string]any) + if options["apiKey"] != EnvRef("OPENAI_API_KEY") { + t.Errorf("apiKey = %v, want an env reference", options["apiKey"]) + } + if options["apiKey"] == "sk-secret" { + t.Error("the resolved secret must never reach the config") + } +} + +// TestBuildPiProvider_OMLXPlaceholderThenReference pins both sides of the Pi +// rule for oMLX: a keyless local server gets the literal placeholder (Pi hides a +// provider's models until some auth is configured), while a remote endpoint gets +// the $VAR reference Pi resolves at run time. +func TestBuildPiProvider_OMLXPlaceholderThenReference(t *testing.T) { + cat, _ := Load() + p := cat.Providers["omlx"] + + local, _, err := BuildPiProvider("omlx", p, "my-model", "", noEnv) + if err != nil { + t.Fatalf("BuildPiProvider: %v", err) + } + if local.APIKey != piPlaceholderAPIKey { + t.Errorf("local apiKey = %q, want %q", local.APIKey, piPlaceholderAPIKey) + } + if local.API != "openai-completions" { + t.Errorf("api = %q, want openai-completions", local.API) + } + + remote, _, err := BuildPiProvider("omlx", p, "my-model", "http://mac-studio.local:8000/v1", noEnv) + if err != nil { + t.Fatalf("BuildPiProvider: %v", err) + } + if remote.APIKey != "$OPENAI_API_KEY" { + t.Errorf("remote apiKey = %q, want $OPENAI_API_KEY", remote.APIKey) + } +} diff --git a/internal/catalog/providers.yaml b/internal/catalog/providers.yaml index 2bbcbe55..c75644a9 100644 --- a/internal/catalog/providers.yaml +++ b/internal/catalog/providers.yaml @@ -107,6 +107,23 @@ providers: pi: api: openai-completions + omlx: + description: oMLX server on Apple Silicon (OpenAI-compatible API) + name: oMLX + npm: "@ai-sdk/openai-compatible" + # Like llama.cpp, a local oMLX server usually needs no key and none is + # injected when the var is unset; oMLX can be started with --api-key, so a + # key is picked up when there is one (e.g. a Mac on the LAN named by + # OMLX_BASE_URL). + apiKeyEnv: OPENAI_API_KEY + apiKeyOptional: true + options: + baseURL: http://localhost:8000/v1 + optionsFromEnv: + baseURL: OMLX_BASE_URL + pi: + api: openai-completions + vllm: description: vLLM server (OpenAI-compatible API) name: vLLM diff --git a/internal/preset/preset.go b/internal/preset/preset.go index 407de260..db528dbc 100644 --- a/internal/preset/preset.go +++ b/internal/preset/preset.go @@ -1,16 +1,21 @@ -// Package preset reads llama.cpp INI preset files and turns one model section -// into a concrete `llama-server` command. +// Package preset reads INI preset files and turns one section into a concrete +// inference-server command. // -// llama.cpp presets (https://github.com/ggml-org/llama.cpp/blob/master/docs/preset.md) -// are INI files where each `[section]` is a model and every key is a -// `llama-server` argument with its leading dashes stripped — `ctx-size = 4096` -// is `--ctx-size 4096`. A leading `[*]` (or `[global]`) section holds defaults +// The format is llama.cpp's (https://github.com/ggml-org/llama.cpp/blob/master/docs/preset.md): +// INI files where each `[section]` is a model and every key is a server +// argument with its leading dashes stripped — `ctx-size = 4096` is +// `--ctx-size 4096`. A leading `[*]` (or `[global]`) section holds defaults // shared by every model; a per-model section overrides them. // -// Presets are designed for the server's router (multi-model) mode, loaded with -// `--models-preset`. There is no equivalent for plain single-model serving, so -// this package flattens a chosen section back into the explicit flags you would -// have typed by hand. +// llama.cpp presets are designed for the server's router (multi-model) mode, +// loaded with `--models-preset`. There is no equivalent for plain single-model +// serving, so this package flattens a chosen section back into the explicit +// flags you would have typed by hand. +// +// Parsing is engine-neutral; only the *spelling* of flags differs between +// engines, which is what a Dialect carries. The package-level Flags, +// CanonicalKey, Preset.Args and Preset.Command render in the LlamaCpp dialect, +// so callers that predate dialects keep their behaviour. package preset import ( @@ -152,20 +157,46 @@ func (p Preset) Select(model string) (Section, error) { case len(p.Sections) == 1: return p.Sections[0], nil default: - return Section{}, fmt.Errorf("preset defines %d models (%s); set MODEL in the Outfit to choose one", len(p.Sections), strings.Join(p.SectionNames(), ", ")) + return Section{}, fmt.Errorf("preset defines %d models (%s); set ALIAS in the Outfit to choose one", len(p.Sections), strings.Join(p.SectionNames(), ", ")) } } -// Flags merges ordered layers of params into `llama-server` flag tokens. Later -// layers override earlier ones by canonical flag name, in place, so the first -// layer fixes the order: pass globals, then the section, then any overrides. It -// does not include the binary name; see Command. -func Flags(layers ...[]Param) []string { +// Dialect describes how one inference engine spells its flags: the short +// aliases it accepts, and the flags it takes bare with no value. The zero +// Dialect passes every key through unchanged, which is what an engine with only +// long-form flags wants. +// +// A dialect is never guessed from the preset's contents — it comes from the +// engine the Outfit's PROVIDER names — because the tables are not +// interchangeable: llama.cpp's would rewrite another engine's `m` to `--model` +// and `c` to `--ctx-size`. +type Dialect struct { + // Aliases maps a short flag alias, as it appears in a preset key (without + // the leading dash), to its canonical long-form name. + Aliases map[string]string + // Boolean lists canonical flag names the engine accepts with no value. + Boolean map[string]bool +} + +// LlamaCpp is the dialect `llama-server` speaks, and the one the package-level +// rendering helpers use. +var LlamaCpp = Dialect{Aliases: canonical, Boolean: boolean} + +// OMLX is the dialect `omlx-cli serve` speaks. It spells every flag in long +// form (`--model-dir`, `--memory-guard`, `--paged-ssd-cache-dir`), so no +// aliasing is applied and keys pass through as written. +var OMLX = Dialect{} + +// Flags merges ordered layers of params into flag tokens. Later layers override +// earlier ones by canonical flag name, in place, so the first layer fixes the +// order: pass globals, then the section, then any overrides. It does not +// include the binary name; see Command. +func (d Dialect) Flags(layers ...[]Param) []string { var ordered []Param at := map[string]int{} // canonical flag name -> index in ordered for _, layer := range layers { for _, kv := range layer { - ck := canonicalName(kv.Key) + ck := d.CanonicalKey(kv.Key) if i, ok := at[ck]; ok { ordered[i] = kv continue @@ -177,22 +208,37 @@ func Flags(layers ...[]Param) []string { var args []string for _, kv := range ordered { - args = append(args, flagFor(kv.Key, kv.Value)...) + args = append(args, d.flagFor(kv.Key, kv.Value)...) } return args } +// Flags merges ordered layers of params into `llama-server` flag tokens, in the +// LlamaCpp dialect. +func Flags(layers ...[]Param) []string { + return LlamaCpp.Flags(layers...) +} + // Args flattens the global defaults and a section into `llama-server` flags, // with the section's values overriding the globals'. func (p Preset) Args(sec Section) []string { return Flags(p.Global, sec.Params) } -// Command builds the full argv (binary first) for serving a section. Any -// override layers (e.g. values an Outfit specifies) win over the preset. +// Command builds the full argv (binary first) for serving a section in the +// LlamaCpp dialect. Any override layers (e.g. values an Outfit specifies) win +// over the preset. func (p Preset) Command(binary string, sec Section, overrides ...[]Param) []string { + return p.CommandIn(LlamaCpp, binary, nil, sec, overrides...) +} + +// CommandIn builds the full argv for serving a section with a given engine: the +// binary, then any subcommand the engine needs (`omlx-cli serve` takes one, +// `llama-server` does not), then the flags rendered in that engine's dialect. +func (p Preset) CommandIn(d Dialect, binary string, subcommand []string, sec Section, overrides ...[]Param) []string { layers := append([][]Param{p.Global, sec.Params}, overrides...) - return append([]string{binary}, Flags(layers...)...) + argv := append([]string{binary}, subcommand...) + return append(argv, d.Flags(layers...)...) } // canonical maps llama.cpp short flag aliases — as they appear in preset keys, @@ -225,35 +271,34 @@ var boolean = map[string]bool{ "cpu-moe": true, "color": true, } -// canonicalName lower-cases a preset key and resolves any short alias to its -// canonical long-form flag name, so the same flag written different ways (c, -// ctx-size, env-var forms) collapses to one when merging layers. -func canonicalName(key string) string { +// CanonicalKey resolves a preset key to the canonical long-form flag name that +// Flags would emit, so the same setting written different ways (`ngl` and +// `n-gpu-layers`, `c` and `ctx-size`) compares equal. Callers that need to +// match specific flags rather than render them should compare canonical names. +// A dialect with no aliases only lower-cases the key. +func (d Dialect) CanonicalKey(key string) string { name := strings.ToLower(strings.TrimSpace(key)) - if c, ok := canonical[name]; ok { + if c, ok := d.Aliases[name]; ok { name = c } return name } -// CanonicalKey resolves a preset key to the canonical long-form flag name that -// Flags would emit, so the same setting written different ways (`ngl` and -// `n-gpu-layers`, `c` and `ctx-size`) compares equal. Callers that need to -// match specific flags rather than render them should compare canonical names. +// CanonicalKey resolves a preset key in the LlamaCpp dialect. func CanonicalKey(key string) string { - return canonicalName(key) + return LlamaCpp.CanonicalKey(key) } -// flagFor turns one preset key/value into its `llama-server` flag tokens. -func flagFor(key, value string) []string { - name := canonicalName(key) +// flagFor turns one preset key/value into its flag tokens. +func (d Dialect) flagFor(key, value string) []string { + name := d.CanonicalKey(key) flag := "--" + name if len(name) == 1 { // an unknown single-character key is a short flag flag = "-" + name } value = strings.TrimSpace(value) - if boolean[name] { + if d.Boolean[name] { if isFalsy(value) { return nil } diff --git a/internal/preset/preset_test.go b/internal/preset/preset_test.go index e94c9815..a92ae824 100644 --- a/internal/preset/preset_test.go +++ b/internal/preset/preset_test.go @@ -192,7 +192,7 @@ func TestFlagFor(t *testing.T) { {"unknown-long", "", []string{"--unknown-long"}}, // valueless passthrough } for _, tc := range cases { - got := flagFor(tc.key, tc.value) + got := LlamaCpp.flagFor(tc.key, tc.value) if strings.Join(got, " ") != strings.Join(tc.want, " ") { t.Errorf("flagFor(%q,%q) = %v, want %v", tc.key, tc.value, got, tc.want) } @@ -219,3 +219,54 @@ func TestCommand(t *testing.T) { t.Errorf("Command args = %v", argv[1:]) } } + +// TestDialectOMLXPassesKeysThrough is the point of having dialects at all: oMLX +// spells every flag in long form, so llama.cpp's alias table must not touch its +// keys. `m` and `c` are the dangerous ones — llama.cpp would silently rewrite +// them to --model and --ctx-size, neither of which oMLX accepts. +func TestDialectOMLXPassesKeysThrough(t *testing.T) { + params := []Param{ + {Key: "model-dir", Value: "/models"}, + {Key: "memory-guard", Value: "safe"}, + {Key: "c", Value: "literal"}, + } + got := strings.Join(OMLX.Flags(params), " ") + want := "--model-dir /models --memory-guard safe -c literal" + if got != want { + t.Errorf("OMLX.Flags =\n %s\nwant\n %s", got, want) + } +} + +// TestDialectOMLXHasNoBareBooleans checks the boolean table is llama.cpp's +// alone: a key oMLX does not treat as a bare flag keeps its value. +func TestDialectOMLXHasNoBareBooleans(t *testing.T) { + got := strings.Join(OMLX.Flags([]Param{{Key: "jinja", Value: "1"}}), " ") + if got != "--jinja 1" { + t.Errorf("OMLX.Flags = %q, want %q", got, "--jinja 1") + } +} + +// TestPackageFlagsStillLlamaCpp guards the callers that predate dialects: the +// package-level helpers, and remote.go's CanonicalKey, must keep rendering +// llama.cpp. +func TestPackageFlagsStillLlamaCpp(t *testing.T) { + got := strings.Join(Flags([]Param{{Key: "hf", Value: "org/m"}, {Key: "mmap", Value: "1"}}), " ") + if got != "--hf-repo org/m --mmap" { + t.Errorf("Flags = %q, want %q", got, "--hf-repo org/m --mmap") + } + if CanonicalKey("ngl") != "n-gpu-layers" { + t.Errorf("CanonicalKey(ngl) = %q, want n-gpu-layers", CanonicalKey("ngl")) + } +} + +// TestCommandInAddsSubcommand checks an engine that needs a subcommand gets it +// between the binary and the flags. +func TestCommandInAddsSubcommand(t *testing.T) { + p, _ := Parse([]byte("[m]\nmodel-dir = /models\n")) + sec, _ := p.Select("m") + argv := p.CommandIn(OMLX, "omlx-cli", []string{"serve"}, sec) + want := "omlx-cli serve --model-dir /models" + if strings.Join(argv, " ") != want { + t.Errorf("CommandIn = %v, want %s", argv, want) + } +} diff --git a/openspec/changes/add-omlx-provider/.openspec.yaml b/openspec/changes/add-omlx-provider/.openspec.yaml new file mode 100644 index 00000000..5849c2db --- /dev/null +++ b/openspec/changes/add-omlx-provider/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-08-01 diff --git a/openspec/changes/add-omlx-provider/design.md b/openspec/changes/add-omlx-provider/design.md new file mode 100644 index 00000000..a7d8c8ab --- /dev/null +++ b/openspec/changes/add-omlx-provider/design.md @@ -0,0 +1,104 @@ +## Context + +Two engines now have to coexist behind one command. They differ in three ways that the +design has to absorb, and one way it deliberately does not. + +- **Binary**: `llama-server` is a `PATH` install; oMLX ships as a signed macOS app whose + CLI lives at `/Applications/oMLX.app/Contents/MacOS/omlx-cli`. +- **Shape**: `llama-server` takes flags directly; `omlx-cli` takes a `serve` subcommand + first. +- **Model**: `llama-server` is launched with one model. oMLX loads a whole `--model-dir` + and selects per request, so it has nothing to be told at launch and no context flag. +- **Preset format**: identical. Both are INI files of `key = value` under `[section]` + headers. Only the *vocabulary* differs. + +## Goals / Non-Goals + +- Goals: one engine-selection rule shared with the cloud path; no silent misrendering of + one engine's preset in another's vocabulary; existing llama.cpp behaviour byte-identical. +- Non-goals: a plugin system for engines; a generic engine descriptor in the catalogue; + cloud deployment of oMLX. + +## Decisions + +### Engine selection comes from PROVIDER + +The alternatives were a new `ENGINE` Outfit keyword or a `--engine` flag on `serve`. + +`PROVIDER` wins because the codebase already committed to it: `runnerFor` maps `PROVIDER` +to a cloud runner and documents that `serve` does the same locally. Adding a keyword would +create a second source of truth for a question already answered, and `internal/outfit` +errors on unknown keywords, so every Outfit stating `ENGINE` would break on older +binaries. A flag would be worse still — it is per-invocation, so it could not travel with +the Outfit. + +The cost is the breaking change: `serve` now rejects providers that are not local engines. +That is a fix, not a regression. The previous behaviour ran `llama-server` for an Outfit +saying `PROVIDER ollama`, which is a command the Outfit never asked for. + +`engineFor` mirrors `runnerFor` deliberately: a hard-coded switch in Go, not a catalogue +field. Putting engine metadata in `providers.yaml` would be a single source of truth +across `serve`/`deploy`/`list`, but it would also mean a user-supplied `--providers` file +could name an arbitrary binary for `outfit serve` to execute. The allow-list stays in the +binary. + +### Dialects, not a second parser + +`internal/preset` splits cleanly: `Parse`, `Select`, `SectionNames`, the layer-merge loop +and `FormatCommand` never look at key names, while `canonical`, `boolean` and `flagFor` +are entirely llama.cpp's. So the engine-specific part becomes a value — `Dialect{Aliases, +Boolean}` — rather than a fork of the package. + +The zero `Dialect` passes keys through unchanged, which is exactly right for oMLX: every +`omlx-cli serve` flag is long-form, so there is nothing to alias. + +The package-level `Flags`, `CanonicalKey`, `Preset.Args` and `Preset.Command` are kept as +`LlamaCpp` wrappers rather than removed. This is not politeness to callers: `remote.go`'s +`isCloudOwned` compares preset keys through `preset.CanonicalKey` to decide which flags +the cloud owns, so changing that function's meaning would silently change what +`outfit remote deploy` strips from a preset. + +**The trap this closes**: an oMLX preset read in llama.cpp's dialect parses without error +and produces a wrong command — `m` becomes `--model`, `c` becomes `--ctx-size`, and a +`key = 0` for a name in llama.cpp's boolean table vanishes. The dialect therefore comes +from the engine `PROVIDER` names, never from inspecting the file. + +### What an Outfit means for oMLX + +Only `BASEURL` becomes a launch flag, via the already engine-neutral `hostPortFromURL`. +`MODEL` and `ALIAS` keep their existing meaning — the id the harness requests — because +that is what they mean for every other provider, and for oMLX it is load-bearing rather +than a label: the server really does dispatch on it. `CONTEXT` sizes the harness's window; +oMLX has no context flag to keep in step with. + +`isModelPath`'s `.gguf` heuristic is not reached on the oMLX path. MLX models are +directories of safetensors, so the predicate would be wrong for them. + +`serveEngine.needsModel` keeps llama.cpp's "needs a PRESET or a MODEL" rule from applying +to an engine that needs neither. + +### No API key to the server + +`serve` prints the command it runs, and oMLX takes `--api-key` on the command line. Passing +a resolved key would put it on the terminal and in `ps` output — a sharper failure than +the config-file secrets the rest of the codebase is careful to avoid, since those at least +stay `0600` on disk. Auth on the server is configured in oMLX. The catalogue path is +unaffected: `outfit add`/`apply` still writes an `{env:OPENAI_API_KEY}` reference for the +harness when a key is set. + +### Serve moves to its own file + +`cmd/outfit/serve.go` is partly cohesion and partly a constraint: +`TestCompletionCoversDispatch` scans `main.go` for `case "…":` at one tab of indentation to +prove every dispatched command is completable. `engineFor`'s switch matches that shape, so +leaving it in `main.go` reports `llamacpp` and `omlx` as uncompletable commands. + +## Risks / Trade-offs + +- **Breaking `serve` for non-engine providers** — mitigated by an error that names the + valid values, and by the fact that the previous behaviour was already wrong. +- **oMLX cannot be verified in CI** — it is Apple Silicon only. Mitigated by the stubbed + binary seam the llama.cpp tests already use, which pins the whole argv without running + anything real. +- **Port 8000 collides with the `vllm` default** — harmless (nothing assumes base URLs are + unique), and documented so anyone running both knows to set `OMLX_BASE_URL`. diff --git a/openspec/changes/add-omlx-provider/proposal.md b/openspec/changes/add-omlx-provider/proposal.md new file mode 100644 index 00000000..e5667a4c --- /dev/null +++ b/openspec/changes/add-omlx-provider/proposal.md @@ -0,0 +1,45 @@ +## Why + +[oMLX](https://omlx.ai) is a local inference server for Apple Silicon, built on Apple's +MLX framework, exposing an OpenAI-compatible API. `outfit` today knows two local engines +it can point a harness at (`llamacpp`, `vllm`) and exactly one it can *launch* +(`llama-server`), so Mac users get half the flow: they can dress the agent, but not start +the server behind it from the same file. + +Adding oMLX also settles an inconsistency the code already claims is settled. `runnerFor` +documents itself as "PROVIDER already names the engine — `outfit serve` starts that engine +locally", and the CLI usage says "PROVIDER picks the engine, just as it does for serve". +Neither is true: `cmdServe` never reads `PROVIDER` and unconditionally runs +`llama-server`, so an Outfit naming any other provider silently starts the wrong server. + +## What Changes + +- Add an `omlx` provider to the catalogue: an OpenAI-compatible endpoint defaulting to + `http://localhost:8000/v1`, overridable with `OMLX_BASE_URL`, keyless on localhost and + key-bearing when pointed elsewhere (`apiKeyOptional`, like `llamacpp`), and Pi-capable + via `openai-completions`. +- **BREAKING**: `outfit serve` selects its engine from `PROVIDER` rather than always + running `llama-server`. `llamacpp` and `omlx` are served; every other provider is now a + loud error instead of a silently wrong command. +- Teach `internal/preset` that flag *spelling* is per-engine. Parsing stays shared; a + `Dialect` carries the alias and boolean tables, with `LlamaCpp` the default so existing + callers are unaffected, and `OMLX` passing long-form keys through untouched. +- Give `serve` a per-engine install hint, and find the oMLX CLI on `PATH` or in the macOS + app bundle, since oMLX ships as a signed app rather than a `PATH` install. +- `serve` never passes an API key to oMLX. It prints the command it runs, and oMLX takes + its key as a flag, so a resolved secret would land on screen and in the process table. + +Non-goals: + +- `outfit remote deploy` stays closed to oMLX. The cloud side is CUDA/NVIDIA EC2 and oMLX + is Apple Silicon only, so `runnerFor` keeps rejecting it — correctly. +- No new Outfit keyword. `PROVIDER` names the engine, as the codebase already intends. + +## Impact + +- Affected specs: `provider-catalog`, `local-serving` +- Affected code: `internal/catalog/providers.yaml`, `internal/preset/preset.go`, + `cmd/outfit/serve.go` (extracted from `main.go`), `cmd/outfit/main.go` (usage text) +- Migration: an Outfit that named a non-engine `PROVIDER` and relied on `serve` running + `llama-server` anyway must change its `PROVIDER` to `llamacpp`. That configuration was + already producing a command for an engine the Outfit did not name. diff --git a/openspec/changes/add-omlx-provider/specs/local-serving/spec.md b/openspec/changes/add-omlx-provider/specs/local-serving/spec.md new file mode 100644 index 00000000..7d426297 --- /dev/null +++ b/openspec/changes/add-omlx-provider/specs/local-serving/spec.md @@ -0,0 +1,145 @@ +## ADDED Requirements + +### Requirement: Choosing the engine + +`outfit serve` SHALL launch the inference engine the Outfit's `PROVIDER` names, +the local counterpart of the runner `outfit remote deploy` selects from the same +instruction. `llamacpp` SHALL run `llama-server`; `omlx` SHALL run the oMLX CLI. +There SHALL be no default: a `PROVIDER` that is not a self-hosted engine SHALL +fail, naming the providers that can be served, rather than launching an engine +the Outfit did not ask for. + +An engine whose executable is normally installed outside the `PATH` SHALL also be +looked for at its conventional install location, so a user who has never put it +on their `PATH` can still serve. + +#### Scenario: Provider selects the engine + +- **WHEN** the Outfit says `PROVIDER omlx` +- **THEN** the printed command runs the oMLX CLI, not `llama-server` + +#### Scenario: A provider that is not a local engine + +- **WHEN** the Outfit names a hosted provider such as `ollama` or `openrouter` +- **THEN** the command fails, listing the providers that can be served locally + +#### Scenario: Engine installed outside the PATH + +- **WHEN** the oMLX CLI is not on the `PATH` but is present in its macOS app + bundle +- **THEN** `serve` uses the bundled executable + +### Requirement: Serving a multi-model engine + +An engine that loads a directory of models and selects one per request SHALL be +servable with neither a `PRESET` nor a `MODEL`, since it needs nothing at launch +to know what to load. For such an engine the Outfit's `MODEL` and `ALIAS` SHALL +keep their usual meaning — the id the harness requests — rather than becoming +launch flags, and `CONTEXT` SHALL size the harness's window only. `BASEURL`'s +host and port SHALL still set the server's bind address, and SHALL be omitted +entirely when the Outfit states no `BASEURL`, so the engine's own defaults stand. + +#### Scenario: Bare Outfit starts the server + +- **WHEN** the Outfit states only `PROVIDER omlx` +- **THEN** the server is launched with no bind flags and no model flags, rather + than failing for want of a model + +#### Scenario: Bind address from BASEURL + +- **WHEN** the Outfit adds `BASEURL http://127.0.0.1:9100/v1` +- **THEN** the command binds the server to that host and port + +### Requirement: Secrets never reach the server command + +`serve` SHALL NOT resolve or pass an API key to the engine it launches. Because +the command is printed before it runs, and an engine may accept its key as a +command-line flag, passing one would expose the secret on screen and to any local +process. Configuring authentication on the server is the engine's own concern. + +#### Scenario: A set key is not passed through + +- **WHEN** the provider's API key variable is set in the environment +- **THEN** neither the printed command nor the launched process carries the key + +## MODIFIED Requirements + +### Requirement: Serve basics + +`outfit serve [path]` SHALL read an Outfit (default `./Outfit`, aliases and +directories accepted like every Outfit command), build the command for the +engine its `PROVIDER` names, print it in copy-pasteable shell form, and run it +with stdio forwarded. `--dry-run`/`-n` SHALL print the command without +launching. A missing binary SHALL produce an install hint naming **that** engine +rather than a raw exec error. + +#### Scenario: Dry run + +- **WHEN** the user runs `outfit serve --dry-run` +- **THEN** the resolved command is printed and no server starts + +#### Scenario: Engine not installed + +- **WHEN** the selected engine's binary cannot be found +- **THEN** the error suggests installing that engine, not another one + +### Requirement: Preset-based serving + +With a `PRESET`, the referenced `.ini` SHALL supply the command: a relative +preset path resolves against the Outfit's own directory, so the pair can travel +together. The preset's `[*]`/`[global]` section holds shared defaults; each named +section is one model whose keys are server arguments with dashes stripped. The +served section is chosen by the Outfit's `ALIAS`, matched case-insensitively; a +preset with exactly one section is always served; no sections is an error; +several sections with no matching name SHALL fail listing the available +sections. Values the Outfit itself states SHALL override the preset's. + +A preset SHALL be read in the flag vocabulary of the engine the `PROVIDER` names, +never in a vocabulary inferred from the file. A preset written for one engine is +therefore not portable to another: read in the wrong vocabulary it would parse +cleanly and produce a command with silently rewritten or dropped flags. + +#### Scenario: Outfit overrides the preset + +- **WHEN** the preset section sets `ctx-size = 4096` and the Outfit says + `CONTEXT 32768` +- **THEN** the command carries a context size of 32768 + +#### Scenario: Ambiguous preset + +- **WHEN** the preset defines several sections and the Outfit's `ALIAS` matches + none +- **THEN** the command fails listing the section names to choose from + +#### Scenario: Another engine's preset keys are left alone + +- **WHEN** an oMLX preset contains a key that llama.cpp would treat as a short + alias +- **THEN** the key is rendered as written, not rewritten to llama.cpp's + long-form flag + +### Requirement: Flag rendering + +Preset keys SHALL render as the engine's flags. Where an engine defines short +aliases, they SHALL be canonicalised to their long form so the same flag written +different ways collapses to one when layers merge, later layers overriding +earlier ones in place; an engine that defines none SHALL pass its keys through +unchanged. Flags the engine accepts bare SHALL render bare when truthy and be +dropped when falsy (`0`, `false`, `off`, `no`). Printed commands SHALL quote only +the tokens that need it, and SHALL pass values to the server verbatim. + +#### Scenario: Layer override by canonical name + +- **WHEN** the global section sets `c = 4096` and the model section sets + `ctx-size = 8192` for llama.cpp +- **THEN** the command carries a single context-size flag with value 8192 + +#### Scenario: Boolean flags + +- **WHEN** a llama.cpp preset sets `jinja = 1` and `mmap = 0` +- **THEN** the command includes a bare `--jinja` and no mmap flag + +#### Scenario: An engine with no short aliases + +- **WHEN** an oMLX preset sets `model-dir` and `memory-guard` +- **THEN** both render as their long-form flags with their values unchanged diff --git a/openspec/changes/add-omlx-provider/specs/provider-catalog/spec.md b/openspec/changes/add-omlx-provider/specs/provider-catalog/spec.md new file mode 100644 index 00000000..82524470 --- /dev/null +++ b/openspec/changes/add-omlx-provider/specs/provider-catalog/spec.md @@ -0,0 +1,31 @@ +## ADDED Requirements + +### Requirement: Apple Silicon local provider + +The catalogue SHALL include an `omlx` provider describing a local +[oMLX](https://omlx.ai) server: an OpenAI-compatible endpoint defaulting to +`http://localhost:8000/v1`, overridable per-provider with `OMLX_BASE_URL`, and +Pi-capable through the `openai-completions` API. + +Like the llama.cpp provider it SHALL be optionally authenticated: a local server +needs no key, while the same provider may name an oMLX server started with +`--api-key` on another machine. An unset key at a local endpoint therefore +yields no `apiKey` option for opencode and the keyless placeholder for Pi, while +a set key — or any non-local endpoint — yields the environment reference the +harness resolves at run time. + +Its default port coinciding with another provider's is not a conflict: the +catalogue places no uniqueness requirement on base URLs. + +#### Scenario: Local server needs no key + +- **WHEN** a selection names `omlx` with the default localhost base URL and no + API key set +- **THEN** the opencode provider block carries no `apiKey` option, and the Pi + entry carries the keyless placeholder so its models stay selectable + +#### Scenario: Remote server keeps the key reference + +- **WHEN** the base URL is overridden to a non-local host +- **THEN** the API key is written as an environment reference, never as the + resolved secret diff --git a/openspec/changes/add-omlx-provider/tasks.md b/openspec/changes/add-omlx-provider/tasks.md new file mode 100644 index 00000000..e78176ee --- /dev/null +++ b/openspec/changes/add-omlx-provider/tasks.md @@ -0,0 +1,42 @@ +## 1. Catalogue + +- [x] 1.1 Add an `omlx` provider to `internal/catalog/providers.yaml`: `npm` `@ai-sdk/openai-compatible`, `options.baseURL` `http://localhost:8000/v1`, `optionsFromEnv.baseURL` `OMLX_BASE_URL`, `apiKeyEnv` `OPENAI_API_KEY` with `apiKeyOptional: true`, and a `pi` block using `openai-completions`. + +## 2. Preset dialects + +- [x] 2.1 Add `Dialect{Aliases, Boolean}` to `internal/preset`, with `LlamaCpp` wrapping the existing tables and `OMLX` as the zero value (long-form keys pass through). +- [x] 2.2 Move `canonicalName`/`flagFor`/`Flags` onto `Dialect`; keep package-level `Flags`, `CanonicalKey`, `Preset.Args` and `Preset.Command` as `LlamaCpp` wrappers so `remote.go`'s `isCloudOwned` is unchanged. +- [x] 2.3 Add `Preset.CommandIn(dialect, binary, subcommand, sec, overrides...)` for engines that need a subcommand. +- [x] 2.4 Update the package doc; fix `Select`'s error text, which told the user to set `MODEL` when `serve` selects a section by `ALIAS`. + +## 3. Serve + +- [x] 3.1 Extract the serve command from `cmd/outfit/main.go` into `cmd/outfit/serve.go` (required: `TestCompletionCoversDispatch` scans `main.go` for `case "…":` at one tab, so `engineFor`'s switch there reads as new commands). +- [x] 3.2 Add `serveEngine` (binary, subcommand, dialect, params, needsModel, installHint) and `engineFor`, the local twin of `runnerFor`; error for any provider that is not a local engine. +- [x] 3.3 Add `resolveOMLXBinary`: the `omlxBinary` override, then `PATH`, then `/Applications/oMLX.app/Contents/MacOS/omlx-cli`. +- [x] 3.4 Split `outfitServeParams` into `llamacppServeParams` and `omlxServeParams` over a shared `bindAddressParams`; oMLX maps only `BASEURL`. +- [x] 3.5 Route `cmdServe` through the engine: per-engine install hint, subcommand in the argv, and `needsModel` gating the "needs a PRESET or a MODEL" error. + +## 4. Tests + +- [x] 4.1 `internal/preset`: dialect isolation (`OMLX` does not alias `m`/`c`, has no bare booleans), the LlamaCpp wrappers still render `--hf-repo`/`--mmap` and `CanonicalKey`, and `CommandIn` places the subcommand. +- [x] 4.2 `internal/catalog`: `omlx` keyless on localhost, an env reference when the key is set, and the Pi placeholder/reference pair. +- [x] 4.3 `cmd/outfit`: generalise the binary stub, then cover the oMLX argv, no-model start, preset pass-through, Outfit-over-preset, never-passes-a-key, execution, the oMLX install hint, and the now-rejected non-engine provider. +- [x] 4.4 `go test ./... -race -cover` green, every package ≥ 80%. + +## 5. Docs + +- [x] 5.1 Rewrite `docs/commands/serve.md` around per-engine selection; document the oMLX flags, the binary lookup, and that presets are not portable between engines. +- [x] 5.2 Update `docs/README.md` (provider list, base-URL env row, keyless-local sentence), `docs/outfit-file.md` (`PRESET` is engine-specific), and `README.md` (provider list, serve section, env vars, guides). +- [x] 5.3 Update the CLI usage text in `cmd/outfit/main.go`. +- [x] 5.4 Update `AGENTS.md`: the layout (new `serve.go`), the `internal/preset` and `serve` notes, and a new trap for cross-dialect presets. +- [x] 5.5 Add `examples/omlx/qwen3.6/` (README, Outfit, preset.ini) and link it from `README.md`. +- [x] 5.6 Add a CHANGELOG entry, including the breaking `serve` change. + +## 6. Verify + +- [x] 6.1 `gofmt -l` clean, `go vet ./...`, `go test ./... -race -cover` green. +- [x] 6.2 `outfit serve --dry-run` on the oMLX example, on a bare `PROVIDER omlx`, and on a non-engine provider (which must now error). +- [x] 6.3 Regression: `outfit serve --dry-run examples/llamacpp/qwen3.6/Outfit` produces the same command as before the change. +- [x] 6.4 `outfit add -p omlx` under both harnesses: keyless on localhost, `{env:…}`/`$VAR` when remote or keyed. +- [ ] 6.5 `openspec validate add-omlx-provider` — the CLI is not installed in this environment; artifacts were hand-written to the archived change's shape. From cd0379fa34148334c4bac5422a5e5fab90519611 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 16:12:05 +0100 Subject: [PATCH 2/2] docs: clarify oMLX API-key setup and add a Gemma-4-E2B example MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Real testing surfaced two things the oMLX docs got wrong. oMLX can require an API key even on localhost (an admin-panel setting), yet the docs claimed a local server never needs one; and because the provider is apiKeyOptional, outfit only writes the key reference when OPENAI_API_KEY is set at apply time, so setting it afterwards silently does nothing. Spell both out, in the example and in serve.md. Also make the example's MODEL substitution unmissable: oMLX serves by the model's local directory name, so a copied Outfit fails with "model not found" unless the id matches what you downloaded. Add a Gemma-4-E2B example alongside the Qwen3.6 one — a small model that loads in seconds, a gentler first run than the 27B. --- README.md | 1 + docs/README.md | 5 +- docs/commands/serve.md | 6 ++ examples/omlx/gemma-4-e2b/Outfit | 13 +++ examples/omlx/gemma-4-e2b/README.md | 132 +++++++++++++++++++++++++++ examples/omlx/gemma-4-e2b/preset.ini | 18 ++++ examples/omlx/qwen3.6/Outfit | 9 +- examples/omlx/qwen3.6/README.md | 43 +++++++-- 8 files changed, 214 insertions(+), 13 deletions(-) create mode 100644 examples/omlx/gemma-4-e2b/Outfit create mode 100644 examples/omlx/gemma-4-e2b/README.md create mode 100644 examples/omlx/gemma-4-e2b/preset.ini diff --git a/README.md b/README.md index 7d9c44b6..3433d09b 100644 --- a/README.md +++ b/README.md @@ -346,6 +346,7 @@ with a ready-to-apply `Outfit`: - [Qwen3.6-35B-A3B on llama.cpp](examples/llamacpp/qwen3.6/README.md) - [Gemma-4-12B-IT on llama.cpp](examples/llamacpp/gemma4/README.md) - [Qwen3.6-35B-A3B on oMLX (Apple Silicon)](examples/omlx/qwen3.6/README.md) +- [Gemma-4-E2B on oMLX (Apple Silicon)](examples/omlx/gemma-4-e2b/README.md) ## Adding providers and models diff --git a/docs/README.md b/docs/README.md index d30225f3..4dfada1f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -72,5 +72,6 @@ shell environment — so a project keeps its own key next to the file that needs it, the same way `PRESET` and `REMOTE` travel with an Outfit. They are **never written into the agent's config** — outfit writes a reference the agent resolves when it runs, and `outfit harness` passes the keys it can resolve to the agent it launches. If you start the agent yourself, -set the variable in your own environment. Local providers (Ollama, llama.cpp and -oMLX on localhost) need no key; Bedrock uses your AWS credentials. +set the variable in your own environment. Local providers on localhost (Ollama, +llama.cpp) need no key; Bedrock uses your AWS credentials. oMLX needs one only if +you enabled its API-key auth — set `OPENAI_API_KEY` before applying if you did. diff --git a/docs/commands/serve.md b/docs/commands/serve.md index f682f4a9..3bd68e29 100644 --- a/docs/commands/serve.md +++ b/docs/commands/serve.md @@ -121,6 +121,12 @@ its key on the command line, so passing one would put the secret on your screen and in the process table. If you want auth on the server, configure it in oMLX; `outfit add`/`apply` still picks up `OPENAI_API_KEY` for the agent's own config. +Note that oMLX can require a key even on localhost (it is an admin-panel +setting). Because the `omlx` provider is `apiKeyOptional`, `outfit` only writes +the key reference when `OPENAI_API_KEY` is set **at apply time** — so if your +oMLX needs a key, set it before `add`/`apply`, not just before launching the +agent. + ### Finding the binary oMLX ships as a macOS app, so `serve` looks for `omlx-cli` on your `PATH` first diff --git a/examples/omlx/gemma-4-e2b/Outfit b/examples/omlx/gemma-4-e2b/Outfit new file mode 100644 index 00000000..e1de6176 --- /dev/null +++ b/examples/omlx/gemma-4-e2b/Outfit @@ -0,0 +1,13 @@ +# Gemma-4-E2B (instruction-tuned) served locally by oMLX on Apple Silicon — a +# small model that fits comfortably in memory. +# oMLX loads a whole model directory and picks per request, so MODEL names what +# opencode asks for — it is not a launch flag. +PROVIDER omlx +# MODEL must match a model your oMLX actually serves — its directory name or an +# alias. If you downloaded something else, change this: list what yours exposes +# with `curl -H "Authorization: Bearer $OPENAI_API_KEY" localhost:8000/v1/models`. +# A name oMLX does not have is rejected with "model not found". +MODEL gemma-4-E2B-it-MLX-6bit +CONTEXT 32768 # opencode's context window +PRESET ./preset.ini # `outfit serve` launches omlx-cli from this +# BASEURL http://127.0.0.1:8000/v1 # uncomment for a non-default host/port diff --git a/examples/omlx/gemma-4-e2b/README.md b/examples/omlx/gemma-4-e2b/README.md new file mode 100644 index 00000000..c30429e7 --- /dev/null +++ b/examples/omlx/gemma-4-e2b/README.md @@ -0,0 +1,132 @@ +# Gemma-4-E2B on oMLX (Apple Silicon) + +Run LM Studio's MLX build of Gemma-4-E2B-it on your Mac with +[oMLX](https://omlx.ai), then point opencode at it with the [`Outfit`](Outfit) in +this directory. + +`E2B` is the small member of the Gemma-4 family: an *effective* ~2B parameters, +so it loads in seconds and leaves plenty of memory headroom — a good first model +to prove the setup end to end before reaching for something larger (see the +[Qwen3.6-27B example](../qwen3.6/README.md) for a heavier one). + +## Prerequisites + +- An **Apple Silicon** Mac (M1 or later). oMLX is Apple Silicon only, so + [`outfit remote`](../../../docs/commands/remote.md) cannot deploy it. +- [oMLX](https://omlx.ai), installed from its DMG or from source. +- The 6-bit build is only a few GB, so memory is not a concern here. + +## 1. Pull the model + +oMLX's admin panel (`http://localhost:8000/admin`) has a Hugging Face browser +that downloads a model in one click, which is the easiest route. + +To do it from the shell instead, put the model in the directory oMLX serves +from: + +```sh +hf download lmstudio-community/gemma-4-E2B-it-MLX-6bit \ + --local-dir ~/models/gemma-4-E2B-it-MLX-6bit +``` + +The **directory name** is what the model is called over the API — +`gemma-4-E2B-it-MLX-6bit` here — unless you set an alias for it in the admin +panel. That is the name the `Outfit` uses as its `MODEL`. + +## 2. Start oMLX + +```sh +omlx-cli serve \ + --model-dir ~/models \ + --memory-guard safe \ + --max-concurrent-requests 8 \ + --host 127.0.0.1 --port 8000 +``` + +- `--model-dir` — the directory of models to serve. oMLX loads what it needs and + picks per request, so there is no "the model" flag here. +- `--host`/`--port` — the OpenAI-compatible API is served at + `http://127.0.0.1:8000/v1`. + +Installed from the DMG and haven't put it on your `PATH`? The binary lives at +`/Applications/oMLX.app/Contents/MacOS/omlx-cli`. + +Rather than remember those flags, this directory keeps them in a +[`preset.ini`](preset.ini) (with the paths spelled out in full — preset values +reach the server verbatim, so `~` is not expanded) and lets `outfit` build and +run the command: + +```sh +outfit serve # from this directory; reads ./Outfit and its PRESET +outfit serve --dry-run # print the omlx-cli command without running it +``` + +`serve` finds `omlx-cli` on your `PATH`, falling back to the app bundle. + +### Check it's up + +```sh +curl -H "Authorization: Bearer $OPENAI_API_KEY" http://localhost:8000/v1/models +``` + +This is also how to confirm what your model is actually called — the response +lists the names requests must use. (Drop the header if you have not enabled +oMLX's API-key auth.) + +## 3. Point opencode at it + +oMLX speaks the OpenAI-compatible API, which is what the `omlx` provider targets +(default base URL `http://localhost:8000/v1`). Apply the [`Outfit`](Outfit) in +this directory: + +```sh +outfit apply examples/omlx/gemma-4-e2b/Outfit +# or, from this directory: +outfit apply +``` + +The Outfit is: + +```dockerfile +PROVIDER omlx +MODEL gemma-4-E2B-it-MLX-6bit +CONTEXT 32768 +PRESET ./preset.ini +``` + +`MODEL` must match a model oMLX actually has — its directory name, or the alias +you set in the admin panel. This Outfit names the model downloaded above; **if +you downloaded something else, change `MODEL` to match** (list yours with the +`/v1/models` call above). A name oMLX does not have comes back as +`model not found`. + +`CONTEXT` sets opencode's context window. Unlike llama.cpp there is no +`--ctx-size` to keep it in step with — oMLX sizes its cache dynamically — so +this is purely about what opencode will send. + +## A note on API keys + +By default `outfit` writes no key for a localhost oMLX: the `omlx` provider is +marked `apiKeyOptional`, so a local endpoint with no `OPENAI_API_KEY` set gets no +`apiKey` field at all. + +**But oMLX can require a key even on localhost** — it is a per-install setting in +the admin panel, and some builds turn it on by default. If yours does, set +`OPENAI_API_KEY` **before** you apply the Outfit: + +```sh +OPENAI_API_KEY=your-omlx-key outfit apply # writes an {env:OPENAI_API_KEY} reference +OPENAI_API_KEY=your-omlx-key opencode # or `outfit harness`, which forwards it +``` + +The *before* matters: with the key unset at apply time, `outfit` omits the +`apiKey` field, and exporting the key afterwards does nothing — there is no +reference in the config to resolve, so you would re-run `apply`. See the +[Qwen3.6 example](../qwen3.6/README.md#a-note-on-api-keys) for the full +explanation. + +## See also + +- [`outfit serve`](../../../docs/commands/serve.md) — the full command reference +- [The `Outfit` file](../../../docs/outfit-file.md) — full syntax +- A heavier model on oMLX: [`examples/omlx/qwen3.6`](../qwen3.6/README.md) diff --git a/examples/omlx/gemma-4-e2b/preset.ini b/examples/omlx/gemma-4-e2b/preset.ini new file mode 100644 index 00000000..c569738b --- /dev/null +++ b/examples/omlx/gemma-4-e2b/preset.ini @@ -0,0 +1,18 @@ +# An oMLX preset for Gemma-4-E2B, mirroring the omlx-cli command in this +# directory's README. `outfit serve` turns it into that command and runs it. +# +# These are `omlx-cli serve` flags with their leading dashes stripped. Note that +# they are NOT llama.cpp's: each engine's preset is read in its own vocabulary, +# so this file is not interchangeable with the ones under examples/llamacpp/. +# +# Paths are passed to the server verbatim, so `~` is not expanded — spell them +# out in full. + +[*] +host = 127.0.0.1 +port = 8000 + +[default] # served with no ALIAS: the only section +model-dir = /Users/you/models +memory-guard = safe # this model is small; safe leaves ample room +max-concurrent-requests = 8 diff --git a/examples/omlx/qwen3.6/Outfit b/examples/omlx/qwen3.6/Outfit index 4b1357f7..47a3fb70 100644 --- a/examples/omlx/qwen3.6/Outfit +++ b/examples/omlx/qwen3.6/Outfit @@ -1,8 +1,13 @@ -# Qwen3.6-35B-A3B served locally by oMLX on Apple Silicon. +# Qwen3.6-35B-A3B served locally by oMLX on Apple Silicon (the model this +# directory's README walks through downloading). # oMLX loads a whole model directory and picks per request, so MODEL names what # opencode asks for — it is not a launch flag. PROVIDER omlx -MODEL Qwen3.6-35B-A3B-4bit # the directory name under the preset's model-dir +# MODEL must match a model your oMLX actually serves — its directory name or an +# alias. If you downloaded something else, change this: list what yours exposes +# with `curl -H "Authorization: Bearer $OPENAI_API_KEY" localhost:8000/v1/models`. +# A name oMLX does not have is rejected with "model not found". +MODEL Qwen3.6-35B-A3B-4bit CONTEXT 32768 # opencode's context window PRESET ./preset.ini # `outfit serve` launches omlx-cli from this # BASEURL http://127.0.0.1:8000/v1 # uncomment for a non-default host/port diff --git a/examples/omlx/qwen3.6/README.md b/examples/omlx/qwen3.6/README.md index 77ea27a2..b4a7be93 100644 --- a/examples/omlx/qwen3.6/README.md +++ b/examples/omlx/qwen3.6/README.md @@ -114,9 +114,19 @@ PRESET ./preset.ini ``` `MODEL` matters more here than it does for a single-model llama.cpp server: -oMLX serves everything in its model directory, so the name is what actually -selects the model per request. It must match the directory name (or the alias -you set in the admin panel). +oMLX serves everything in its model directory and dispatches on the requested +name, so it must match a model oMLX actually has — its directory name, or the +alias you set in the admin panel. This Outfit names the model downloaded above; +**if you downloaded something else, change `MODEL` to match.** List what yours +exposes with: + +```sh +curl -H "Authorization: Bearer $OPENAI_API_KEY" http://localhost:8000/v1/models +``` + +A name oMLX does not have comes back as `model not found`. And unlike the +llama.cpp examples — where `outfit serve` can fetch the weights from Hugging +Face — oMLX only serves what you have already downloaded. `CONTEXT` sets opencode's context window. Unlike llama.cpp there is no `--ctx-size` to keep it in step with — oMLX sizes its cache dynamically — so @@ -133,13 +143,28 @@ That sets both what opencode calls and what `serve` binds to. ## A note on API keys -A local oMLX server needs no key, and `outfit` writes none: the `omlx` provider -is marked `apiKeyOptional`, so a localhost endpoint gets no `apiKey` at all. +By default `outfit` writes no key for a localhost oMLX: the `omlx` provider is +marked `apiKeyOptional`, so a local endpoint with no `OPENAI_API_KEY` set gets no +`apiKey` field at all. + +**But oMLX can require a key even on localhost** — it is a per-install setting in +the admin panel, and some builds turn it on by default. If yours does, requests +without a key come back as `API key required`. To use it, set `OPENAI_API_KEY` +**before** you apply the Outfit: + +```sh +OPENAI_API_KEY=your-omlx-key outfit apply # writes an {env:OPENAI_API_KEY} reference +OPENAI_API_KEY=your-omlx-key opencode # or `outfit harness`, which forwards it +``` + +The *before* matters: with the key set at apply time, `outfit` writes an +`{env:OPENAI_API_KEY}` reference into the config (never the secret). With it +unset, `outfit` omits the `apiKey` field entirely — and exporting the key later +does nothing, because there is no reference in the config to resolve. If you hit +`API key required` after the fact, set the variable and re-run `apply`. -Serving to another machine on your network? Start oMLX with `--api-key`, export -the same value as `OPENAI_API_KEY`, and point `OMLX_BASE_URL` at the Mac — -`outfit` then writes an environment *reference* into the agent's config, never -the secret itself. +Reaching an oMLX on another machine works the same way: start it with +`--api-key`, export `OPENAI_API_KEY`, and point `OMLX_BASE_URL` at the Mac. `outfit serve` never passes `--api-key`: it prints the command it runs, so a key there would land on your screen and in the process table. Configure auth in oMLX