diff --git a/AGENTS.md b/AGENTS.md index 363b8386..8b9bb803 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,7 +23,8 @@ Run a single test: `go test -run TestWriteConfig_Idempotent ./...` The binary lives under `cmd/`; the domain logic is split into `internal/` packages so each concern is isolated and independently testable. - `cmd/outfit/main.go` — CLI: command dispatch (`add`/`remove`/`list`/`show`/`apply`/`unapply`/`alias`/`unalias`/`serve`/`export`/`harness`/`init-providers`/`remote`/`completion`, plus the hidden `__complete`), flag parsing (`parseSelection` registers both long and short flags against the same vars, and returns the chosen harness name separately so it never leaks into a `Selection`), and user-facing output. `add` and `apply` share `applySelection`, the core that writes one provider selection; `remove` and `unapply` share `removeSelection`, its inverse. Both resolve the active harness (`harness.Resolve`) and route through it. `show` reports the active harness's current config via `harness.State` (the same read `export` reconstructs an Outfit from). `serve` is harness-agnostic (it launches `llama-server`, not a config write). `harness` launches the active harness (forwarding trailing args), or manages the stored default with `--get`/`--set`; its `--outfit`/`-O` routes through the same `applySelection` before launching, so a launch can dress the harness first. That flag's value is optional (`outfitPathFlag` implements `IsBoolFlag`) — bare it means the default `./Outfit`, so a path has to be attached (`--outfit=`), since positional args belong to the harness; the exception is a *leading* positional that names an Outfit, which is applied and not forwarded (see "Alias registry" below). `alias`/`unalias` manage the registry that lets a name stand in for a path, and `writeAliases` renders it for both `alias --list` and `show`. Tests: `main_test.go` (dispatch/add/remove/list), `apply_test.go` (`apply`/`unapply`/`export`), `harness_test.go` (the `harness` command, `show`, and Pi-routed add/export/remove), `serve_test.go`, `alias_test.go`, `complete_test.go`. -- `cmd/outfit/complete.go` + `completion.bash` — tab completion, and the only reason the CLI is no longer a single source file: `//go:embed` needs the script beside it. `outfit completion bash` prints the script; the script calls the hidden `outfit __complete `, which prints one candidate per line and a final `:nofile`/`:file` directive. The `commands` table there mirrors `run`'s switch and each `cmdX`'s `FlagSet` (the `flag` package cannot enumerate a `FlagSet`); `TestCompletionCoversDispatch` scans `main.go` and fails when a new command is not completable. +- `cmd/outfit/serve.go` — the `serve` command: `engineFor` and the `serveEngine` table, the binary lookups (`llamaServerBinary`, `resolveOMLXBinary` — PATH then the `/Applications/oMLX.app` bundle), the Outfit→flags mappings (`llamacppServeParams`/`omlxServeParams`/`bindAddressParams`), and the helpers `remote.go` shares (`resolvePresetPath`, `isModelPath`, `hostPortFromURL`). It is a separate file for a second reason beyond cohesion: `TestCompletionCoversDispatch` scans `main.go` for `case "…":` at one tab, so any other switch on string literals living there is mistaken for a new command. +- `cmd/outfit/complete.go` + `completion.bash` — tab completion, and one of the reasons the CLI is no longer a single source file: `//go:embed` needs the script beside it. `outfit completion bash` prints the script; the script calls the hidden `outfit __complete `, which prints one candidate per line and a final `:nofile`/`:file` directive. The `commands` table there mirrors `run`'s switch and each `cmdX`'s `FlagSet` (the `flag` package cannot enumerate a `FlagSet`); `TestCompletionCoversDispatch` scans `main.go` and fails when a new command is not completable. - `internal/config` — outfit's own config file, `${XDG_CONFIG_HOME:-~/.config}/outfit/config.json`: the default-harness preference and the Outfit alias registry. A **leaf package** — stdlib imports only, and it must stay that way (in particular it must never import `internal/outfit` to parse an Outfit), because that is what lets `readOutfit` resolve a name without dragging in the harness abstraction. Every write goes through `Update`, a read-modify-write of the whole document; unknown top-level keys round-trip through `File.extra`. - `internal/outfit` — the `Outfit` file format and the shared `Selection` type: a flat, Dockerfile-style description of one provider selection (`PROVIDER`/`MODEL`/`ALIAS`/`CONTEXT`/`OUTPUT`/`BASEURL`/`PRESET`/`REMOTE`). `Parse` reads it into a `Selection` (keywords case-insensitive via `canonicalKeyword`, UPPERCASE canonical, `#` comments); `Format` renders one back out for `export`. `apply` defaults to `./Outfit` (`outfit.DefaultFile`). The harness is deliberately **not** a keyword — it is a runtime choice, so an Outfit stays portable. - `internal/harness` — the harness abstraction. The `Harness` interface (`Name`/`Command`/`ConfigPath`/`Apply`/`Remove`/`State`), a `registry` of the opencode and pi adapters (`adapters.go`, which wrap `catalog` + `opencode`/`pi` + `contextsize`), runtime selection via `Resolve` with precedence **`--harness`/`-H` flag > `OUTFIT_HARNESS` env > stored preference > opencode**, and preference load/save (`LoadPreference`/`SavePreference`), which delegate to `internal/config` — that package, not this one, owns the config file. **Start here when adding a third harness:** implement the interface and register it. @@ -33,7 +34,7 @@ The binary lives under `cmd/`; the domain logic is split into `internal/` packag - `cmd/outfit/remote.go` — the `remote` command group (`start`/`stop`/`status`/`deploy`), the CLI's only nested subcommand: each subcommand is a thin wrapper over `internal/remote`. `deploy` is the exception that carries logic: `deployConfigFor` turns an Outfit (plus its preset) into a `remote.DeployConfig` — `PROVIDER` picks the runner (`llamacpp`/`vllm`; other providers are not self-hosted engines, so there is nothing to deploy), `MODEL` or the preset's `hf` names the weights as `repo:QUANT`, `CONTEXT` or the preset's `ctx-size` the window, `ALIAS` the served name, and the preset's remaining flags become `serveArgs`. The `[*]` globals are layered under the section (so `ngl`/`jinja` written there survive), and `cloudOwnedFlags` drops what the cloud sets itself (host/port/model/hf/api-key/ctx-size/alias/metrics), matched on `preset.CanonicalKey` so short aliases count. Config discovery (`resolveRemoteConfig`): an explicit `[path]` argument must name an Outfit with a `REMOTE` instruction; with no argument, `./Outfit` is consulted when present (its `REMOTE` resolved relative to the Outfit, like `PRESET`), falling back to the per-user `remote.json`. Tests: `remote_test.go` (dispatch plus full flows against an `httptest.Server`, with `stubAWSEnv` pinning the AWS credential chain to static env credentials). - `internal/remote` — controls the scale-to-zero GPU inference instance defined in [`remote/`](remote/) by calling its start/stop/deploy Lambda Function URLs (IAM auth, so requests are SigV4-signed via `aws-sdk-go-v2` — the repo's only AWS/network dependency). `Deploy` is the only call with a body, and SigV4 covers the payload hash, so `call`/`sign` take the bytes and hash exactly what is sent. `DeployConfig` deliberately carries no weights prefix: the Lambda derives the S3 layout and seeds the weights when they are absent, replying `{seeding, seedInstanceId}`. Config in `${XDG_CONFIG_HOME:-~/.config}/outfit/remote.json` (`start_url`/`stop_url`/`deploy_url`/`region`), overridable via `OUTFIT_REMOTE_*` env vars; `deploy_url` is optional so configs predating deploy still work for start/stop, and only `Deploy` complains when it is missing; the region falls back to `AWS_REGION`, then to the region embedded in the Function URL host. `Start` retries while the endpoint reports 503/starting, honouring `retry_after_seconds`. - `internal/contextsize` — parses human-friendly sizes (`128k`, `1.5m`, `200000`); `Apply` sets `limit.context` and `limit.output` on opencode model blocks; `DefaultOutput` returns a quarter of the context (the fallback when `OUTPUT` is unset). -- `internal/preset` — parses llama.cpp [preset `.ini`](https://github.com/ggml-org/llama.cpp/blob/master/docs/preset.md) files into named sections of flags, and renders a `llama-server` command for `serve` (`Parse`/`Flags`/`FormatCommand`). +- `internal/preset` — parses [preset `.ini`](https://github.com/ggml-org/llama.cpp/blob/master/docs/preset.md) files into named sections of flags, and renders a server command for `serve` (`Parse`/`Flags`/`FormatCommand`). Parsing is engine-neutral; only the *spelling* of flags differs, which a `Dialect` carries (`LlamaCpp`, `OMLX`). The package-level `Flags`/`CanonicalKey`/`Preset.Args`/`Preset.Command` render LlamaCpp, so pre-dialect callers — notably `remote.go`'s `isCloudOwned` — keep their behaviour; `Preset.CommandIn` takes a dialect plus the engine's subcommand. - `internal/catalog/providers.yaml` — externalised provider plumbing (URLs, key env vars, npm packages, and a `pi:` block per Pi-capable provider) — no model ids. **Add providers here, not in Go.** Embedded at build time but kept external for maintenance. - `examples/` — runnable guides, each a directory with a README and an `Outfit`. - `remote/` — the cloud GPU deployment `outfit remote` drives: a TypeScript CDK project (AWS Lambdas, EC2 Image Builder, S3 weights) with its own `package.json`, tests and docs, built and tested by pnpm rather than Go. It is the only non-Go part of the repository; CI runs its checks separately. **This repository is public, so nothing identifying a deployment — account ids, ARNs, Function URL hosts, allocated addresses, bucket names, resource ids — may be committed here or anywhere else in the repo.** `scripts/check-no-cloud-identifiers.sh` enforces that in CI; deployment state belongs in the gitignored files under `remote/` (`.env`, `Outfit`, `remote.json`, `cdk-outputs.json`). @@ -57,7 +58,7 @@ The binary lives under `cmd/`; the domain logic is split into `internal/` packag **Key/option resolution.** `catalog.BuildProviderBlock` sets `options.apiKey` (opencode) to the provider's `apiKeyEnv` as an `{env:VAR}` reference — never the secret — validating `apiKeyPrefix` and erroring when `apiKeyRequired` against the value it resolves. Resolution is `opencode.EnvResolver(dir)`: a `.env` **beside the Outfit being applied**, then the environment — `dir` is empty for a flags-only `add`, which falls back to the working directory. It deliberately is not relative to the binary or the source file (the old `runtime.Caller` form resolved to the build machine's path, so an installed binary never found one). The option is omitted only for a provider with no key variable, or an `apiKeyOptional` one at a local endpoint. Both harnesses therefore rely on the variable being set when the agent runs, which `harnessEnv` in the CLI arranges at launch. `optionsFromEnv` injects other options (e.g. `baseURL`, `region`) when their env vars are set. The opencode top-level `model` is `/`. Base-URL precedence for both builders: `--base-url`/`-u` flag > `OUTFIT_BASE_URL` > the catalogue's `pi.baseUrl`/`options.baseURL`. -**`serve`.** `outfit serve` launches `llama-server` for an Outfit without touching any harness config. With a `PRESET` it flattens the matching `.ini` section (plus `[*]` defaults) into flags via `internal/preset`; otherwise it derives a command from `MODEL`/`ALIAS`/`CONTEXT`/`BASEURL`. Anything the Outfit states overrides the preset. `--dry-run`/`-n` prints the command without running it. +**`serve`.** `outfit serve` launches a local inference server for an Outfit without touching any harness config. The engine comes from `PROVIDER` via `engineFor` (`cmd/outfit/serve.go`), the local twin of `runnerFor`: `llamacpp` runs `llama-server`, `omlx` runs the Apple-Silicon oMLX CLI, and anything else is an error. A `serveEngine` carries the binary, any subcommand, the preset dialect, the Outfit→flags mapping, and the install hint. With a `PRESET` it flattens the matching `.ini` section (plus `[*]` defaults) into flags in that engine's dialect; otherwise it derives a command from the Outfit alone. Anything the Outfit states overrides the preset. `--dry-run`/`-n` prints the command without running it. oMLX serves a whole `--model-dir` and picks per request, so only `BASEURL` becomes a launch flag for it, and it starts happily with neither `PRESET` nor `MODEL`. `serve` never resolves or passes an API key — it prints the command it runs. Invariants to keep: opencode merges stay idempotent and never drop unrelated config or comments; Pi merges preserve sibling providers and unknown fields; output stays `0600`; the harness never appears in an Outfit, and neither does an alias — both are machine-local, so an Outfit stays portable; outfit's own `config.json` is only ever written read-modify-write, so a new setting cannot clobber an existing one; `__complete` never returns an error and never writes to stderr, whatever the state of the config or catalogue. @@ -71,6 +72,8 @@ These are mistakes already made here; each was silent rather than loud, which is **Match preset flags by canonical name.** Preset keys have short aliases (`ngl` is `n-gpu-layers`, `c` is `ctx-size`, `hf` is `hf-repo`, `a` is `alias`). Comparing raw keys against a list of flag names therefore matches only the spelling you happened to think of. Use `preset.CanonicalKey`, which is exported for exactly this; `Flags` already dedupes layers by canonical name, last layer winning. +**A preset dialect is not interchangeable.** `internal/preset` parses any INI the same way, so an oMLX preset fed through llama.cpp's dialect parses cleanly and produces a *wrong* command: the alias table rewrites `m` to `--model` and `c` to `--ctx-size`, and the boolean table drops a `key = 0` entirely. Nothing errors — the server just receives flags it does not accept, or silently loses a setting. The dialect always comes from the engine `PROVIDER` names, never from the file. + **`contextsize.Parse` is decimal.** `128k` is 128000, not 131072 — a `CONTEXT` written that way is not the power-of-two window it looks like. It also *overrides* a preset's `ctx-size` (both in `serve` and in `remote deploy`), so the Outfit, not the preset, decides the window whenever it states one. ## Reference diff --git a/CHANGELOG.md b/CHANGELOG.md index b1a2c8b2..65ac522d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,16 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] +### Added +- feat: add an omlx provider to the catalogue +- feat: teach outfit serve to launch oMLX on Apple Silicon + +### Changed +- feat!: outfit serve picks its engine from PROVIDER, and rejects providers that + are not self-hosted engines (it previously ran llama-server whatever the + Outfit said) + ## [1.9.0] - 2026-07-29 ### Added - feat: add a vllm provider to the catalogue diff --git a/README.md b/README.md index 611a8411..3433d09b 100644 --- a/README.md +++ b/README.md @@ -47,8 +47,8 @@ model refs and quirks, and none of it is written down where you need it. it dresses the agent for you: - **One command, any model.** Pick from a built-in catalogue — OpenRouter, - Bedrock, Ollama, llama.cpp, or any OpenAI-compatible endpoint. No URLs or model - ids to look up. + Bedrock, Ollama, llama.cpp, vLLM, oMLX, or any OpenAI-compatible endpoint. No + URLs or model ids to look up. - **Your config survives.** Settings are merged *into* what you already have. Other providers, your theme, even your comments stay exactly where you left them. - **Keys stay where they belong.** Secrets are read from a local `.env` and never @@ -262,9 +262,10 @@ the bash and zsh completions for you. ## Serving a local model -Running a model with llama.cpp? `outfit serve` reads an `Outfit` and launches -`llama-server` for it — so the same file that points opencode at a model can -start it too. The simple case needs no preset: +Running a model locally? `outfit serve` reads an `Outfit` and launches the +inference server its `PROVIDER` names — `llamacpp` runs `llama-server`, `omlx` +runs [oMLX](https://omlx.ai) on Apple Silicon — so the same file that points +opencode at a model can start it too. The simple case needs no preset: ```dockerfile # Outfit @@ -320,7 +321,8 @@ minutes while the instance boots and loads the model; `--timeout` (default Each provider declares which environment variable holds its key (`outfit list` shows them). Values are looked up in a `.env` beside the `Outfit` first, then -your shell environment. Local providers like Ollama and llama.cpp need no key; +your shell environment. Local providers like Ollama, llama.cpp and oMLX need no +key; Bedrock authenticates through your AWS credentials. Base URLs default to the usual local ports. Override the endpoint for **any** @@ -334,7 +336,7 @@ OUTFIT_BASE_URL=https://gateway/v1 outfit add -p openai-compatible -m my-model The flag wins over the env var, and either wins over the catalogue's defaults and the per-provider variables (`OLLAMA_BASE_URL`, `LLAMACPP_BASE_URL`, -`OPENAI_BASE_URL`). +`OMLX_BASE_URL`, `VLLM_BASE_URL`, `OPENAI_BASE_URL`). ## Guides @@ -343,6 +345,8 @@ with a ready-to-apply `Outfit`: - [Qwen3.6-35B-A3B on llama.cpp](examples/llamacpp/qwen3.6/README.md) - [Gemma-4-12B-IT on llama.cpp](examples/llamacpp/gemma4/README.md) +- [Qwen3.6-35B-A3B on oMLX (Apple Silicon)](examples/omlx/qwen3.6/README.md) +- [Gemma-4-E2B on oMLX (Apple Silicon)](examples/omlx/gemma-4-e2b/README.md) ## Adding providers and models diff --git a/cmd/outfit/main.go b/cmd/outfit/main.go index 91493d23..368103da 100644 --- a/cmd/outfit/main.go +++ b/cmd/outfit/main.go @@ -19,7 +19,7 @@ // outfit unapply [path] # remove what an Outfit file selects // outfit alias [path] [-n ] # name an Outfit; -l lists them // outfit unalias # drop a registered name -// outfit serve [path] # run llama-server for the Outfit (PRESET or MODEL) +// outfit serve [path] # run the PROVIDER's server (llama.cpp or oMLX) // outfit export [-p name] # print the current config as an Outfit // outfit init-providers [path] # write the embedded providers.yaml out // outfit harness [-H name] [-O[=path]] # launch the harness, optionally applying an @@ -50,7 +50,6 @@ import ( "errors" "flag" "fmt" - "net/url" "os" "os/exec" "path/filepath" @@ -64,7 +63,6 @@ import ( "github.com/lucinate-ai/outfit/internal/harness" "github.com/lucinate-ai/outfit/internal/opencode" "github.com/lucinate-ai/outfit/internal/outfit" - "github.com/lucinate-ai/outfit/internal/preset" ) // version is the binary's version. It defaults to "dev" and is overridden at @@ -141,7 +139,7 @@ Usage: outfit unapply [path] (remove what an Outfit selects) outfit alias [path] [-n ] [-l] (name an Outfit; -l lists them) outfit unalias (drop a registered name) - outfit serve [path] [--dry-run] (run llama-server from the PRESET) + outfit serve [path] [--dry-run] (run the PROVIDER's inference server) outfit export [--provider ] outfit init-providers [path] (defaults to ./providers.yaml) outfit harness [] [-H ] [--outfit[=]] [args...] @@ -156,7 +154,8 @@ Flags: -p, --provider provider name (see `+"`outfit list`"+`) -m, --model model id to set as default / to add or remove -a, --alias friendly name for the model (the harness key); for - llama.cpp the server's reported model name under serve + llama.cpp the server's reported model name under serve, + and the preset section serve selects -c, --context context window size for the added model(s); accepts human suffixes (128k, 1m) or an absolute count (200000) -o, --output max output tokens for the added model(s); same format as @@ -187,10 +186,12 @@ alias: registers an Outfit under a short name, which then stands in wherever an name already registered, and --list/-l shows them all. A path on disk always wins over a name, so registering one changes nothing that works. unalias: drops a registered name. The Outfit file itself is left alone. -serve: runs llama-server for the Outfit. With a PRESET (a llama.cpp .ini) it - turns the matching section into the command; otherwise it derives one - from MODEL/ALIAS/CONTEXT/BASEURL. Prints the command before running it; - --dry-run/-n prints without launching the server. +serve: runs the inference server the Outfit's PROVIDER names — llamacpp + (llama-server) or omlx (Apple Silicon). With a PRESET it turns the + matching section into the command, reading it in that engine's flag + vocabulary; otherwise it derives one from the Outfit's own instructions. + Prints the command before running it; --dry-run/-n prints without + launching the server. export: prints the active harness's config as an Outfit (outfit export > Outfit). init-providers: writes the binary's built-in providers.yaml to the working directory (or [path]) so you can customise the catalogue and point @@ -632,162 +633,6 @@ func writeAliases(b *strings.Builder, header bool) error { return nil } -// llamaServerBinary is the llama.cpp server executable that `serve` launches. -// It is a package var so tests can point it at a stub instead of a real build. -var llamaServerBinary = "llama-server" - -// cmdServe reads an Outfit and runs llama-server for it. With a PRESET it turns -// the matching preset section into the command; without one it derives the -// command from the Outfit's own MODEL/ALIAS/CONTEXT/BASEURL. Either way it -// prints the command before running it. The Outfit path defaults to ./Outfit. -func cmdServe(args []string) error { - fs := flag.NewFlagSet("serve", flag.ContinueOnError) - var dryRun bool - fs.BoolVar(&dryRun, "dry-run", false, "print the llama-server command without running it") - fs.BoolVar(&dryRun, "n", false, "print the command without running it (shorthand)") - if err := fs.Parse(args); err != nil { - return err - } - - var path string - if rest := fs.Args(); len(rest) > 0 { - path = rest[0] - } - sel, outfitPath, err := readOutfit("outfit serve ", path) - if err != nil { - return err - } - - var argv []string - if sel.Preset != "" { - presetPath := resolvePresetPath(sel.Preset, outfitPath) - data, err := os.ReadFile(presetPath) - if err != nil { - return fmt.Errorf("reading preset %s: %w", presetPath, err) - } - pre, err := preset.Parse(data) - if err != nil { - return fmt.Errorf("%s: %w", presetPath, err) - } - // The preset's sections are named by the friendly ALIAS, not the - // provider-native MODEL, so that is what selects one. - sec, err := pre.Select(sel.Alias) - if err != nil { - return fmt.Errorf("%s: %w", presetPath, err) - } - // Anything the Outfit states (CONTEXT, BASEURL, ALIAS, MODEL) overrides - // the preset's own values. - overrides, err := outfitServeParams(sel) - if err != nil { - return err - } - argv = pre.Command(llamaServerBinary, sec, overrides) - fmt.Printf("Using preset %s (model %s)\n\n", presetPath, sec.Name) - } else { - if sel.Model == "" { - return fmt.Errorf("serve needs a PRESET or a MODEL (an HF repo like org/model:quant, or a path to a .gguf)") - } - params, err := outfitServeParams(sel) - if err != nil { - return err - } - argv = append([]string{llamaServerBinary}, preset.Flags(params)...) - fmt.Printf("Serving %s from %s\n\n", sel.Model, outfitPath) - } - - fmt.Printf("%s\n\n", preset.FormatCommand(argv)) - if dryRun { - return nil - } - - cmd := exec.Command(argv[0], argv[1:]...) - cmd.Stdin = os.Stdin - cmd.Stdout = os.Stdout - cmd.Stderr = os.Stderr - if err := cmd.Run(); err != nil { - if errors.Is(err, exec.ErrNotFound) || errors.Is(err, os.ErrNotExist) { - return fmt.Errorf("%s not found — install llama.cpp (e.g. brew install llama.cpp) or check the path", argv[0]) - } - return err - } - return nil -} - -// resolvePresetPath resolves an Outfit's PRESET value: a relative one is taken -// against the Outfit's own directory, so an Outfit and its preset travel -// together (the same rule REMOTE uses). -func resolvePresetPath(presetValue, outfitPath string) string { - if filepath.IsAbs(presetValue) { - return presetValue - } - return filepath.Join(filepath.Dir(outfitPath), presetValue) -} - -// outfitServeParams turns the llama-server settings an Outfit states into preset -// params: the provider-native MODEL supplies the model source (hf for a Hugging -// Face repo, model for a .gguf path); ALIAS, CONTEXT, and BASEURL fill in the -// rest. They seed a preset-less command and, with a preset, override its values. -func outfitServeParams(sel outfit.Selection) ([]preset.Param, error) { - var params []preset.Param - if sel.Model != "" { - if isModelPath(sel.Model) { - params = append(params, preset.Param{Key: "model", Value: sel.Model}) - } else { - params = append(params, preset.Param{Key: "hf", Value: sel.Model}) - } - } - if sel.Alias != "" { - params = append(params, preset.Param{Key: "alias", Value: sel.Alias}) - } - if sel.Context != "" { - n, err := contextsize.Parse(sel.Context) - if err != nil { - return nil, err - } - params = append(params, preset.Param{Key: "ctx-size", Value: strconv.Itoa(n)}) - } - if sel.BaseURL != "" { - host, port, err := hostPortFromURL(sel.BaseURL) - if err != nil { - return nil, err - } - if host != "" { - params = append(params, preset.Param{Key: "host", Value: host}) - } - if port != "" { - params = append(params, preset.Param{Key: "port", Value: port}) - } - } - return params, nil -} - -// isModelPath reports whether a MODEL value is a local file rather than a -// Hugging Face repo: an absolute or explicitly-relative path, a home-relative -// path, or anything ending in .gguf. Everything else is treated as org/model. -func isModelPath(model string) bool { - if strings.HasSuffix(strings.ToLower(model), ".gguf") { - return true - } - return strings.HasPrefix(model, "/") || - strings.HasPrefix(model, "./") || - strings.HasPrefix(model, "../") || - strings.HasPrefix(model, "~") -} - -// hostPortFromURL extracts the host and port from a BASEURL so serve can bind -// llama-server to the same endpoint the harness will call. A bare host:port -// with no scheme is accepted too. -func hostPortFromURL(raw string) (host, port string, err error) { - if !strings.Contains(raw, "://") { - raw = "http://" + raw - } - u, err := url.Parse(raw) - if err != nil { - return "", "", fmt.Errorf("invalid BASEURL %q: %w", raw, err) - } - return u.Hostname(), u.Port(), nil -} - // cmdExport reconstructs an Outfit from the active harness's config and prints // it to stdout, so an existing setup can be captured (outfit export > Outfit). func cmdExport(args []string) error { diff --git a/cmd/outfit/serve.go b/cmd/outfit/serve.go new file mode 100644 index 00000000..624db39c --- /dev/null +++ b/cmd/outfit/serve.go @@ -0,0 +1,286 @@ +// Serve: launching a local inference server for an Outfit. The engine is chosen +// by the Outfit's PROVIDER, the same way `outfit remote deploy` picks a cloud +// runner, so one file describes both what dresses the harness and what serves +// it. Kept out of main.go so the dispatch-coverage scan in complete_test.go only +// ever sees run()'s own switch. + +package main + +import ( + "errors" + "flag" + "fmt" + "net/url" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + + "github.com/lucinate-ai/outfit/internal/contextsize" + "github.com/lucinate-ai/outfit/internal/outfit" + "github.com/lucinate-ai/outfit/internal/preset" +) + +// llamaServerBinary is the llama.cpp server executable that `serve` launches. +// It is a package var so tests can point it at a stub instead of a real build. +var llamaServerBinary = "llama-server" + +// omlxBinary is the oMLX executable that `serve` launches. Empty means "look it +// up"; tests set it to a stub, exactly as they do for llamaServerBinary. +var omlxBinary = "" + +// omlxBundleBinary is where the macOS app installs its CLI. oMLX ships as a +// signed app rather than a PATH install, so a user who has only ever launched +// it from the menu bar still has this and nothing on their PATH. +const omlxBundleBinary = "/Applications/oMLX.app/Contents/MacOS/omlx-cli" + +// resolveOMLXBinary finds the oMLX CLI: an explicit override wins, then the +// PATH, then the app bundle. The bundle path is returned unchecked when nothing +// else matches, so the failure surfaces as serve's usual not-found hint naming +// a concrete path rather than as a bare "omlx-cli". +func resolveOMLXBinary() string { + if omlxBinary != "" { + return omlxBinary + } + if p, err := exec.LookPath("omlx-cli"); err == nil { + return p + } + return omlxBundleBinary +} + +// serveEngine is a local inference server `outfit serve` can launch: which +// binary to run, the subcommand it needs, the dialect its preset is written in, +// and how the Outfit's own instructions turn into its flags. +type serveEngine struct { + // binary is resolved late so tests can stub it after engineFor has run. + binary func() string + subcommand []string + dialect preset.Dialect + // params turns the Outfit's instructions into this engine's flags. + params func(outfit.Selection) ([]preset.Param, error) + // needsModel marks an engine that cannot start without being told what to + // load. oMLX serves a whole model directory and picks per request, so it + // starts happily with neither a PRESET nor a MODEL. + needsModel bool + // installHint completes " not found — ...". + installHint string +} + +// engineFor maps an Outfit's PROVIDER to the engine `serve` launches locally. +// It is the local twin of runnerFor: PROVIDER already names the engine, so no +// separate keyword is needed. Providers that are not self-hosted engines have +// nothing to launch. +func engineFor(provider string) (serveEngine, error) { + switch provider { + case "llamacpp": + return serveEngine{ + binary: func() string { return llamaServerBinary }, + dialect: preset.LlamaCpp, + params: llamacppServeParams, + needsModel: true, + installHint: "install llama.cpp (e.g. brew install llama.cpp) or check the path", + }, nil + case "omlx": + return serveEngine{ + binary: resolveOMLXBinary, + subcommand: []string{"serve"}, + dialect: preset.OMLX, + params: omlxServeParams, + installHint: "install oMLX (https://omlx.ai) or check the path", + }, nil + default: + return serveEngine{}, fmt.Errorf( + "PROVIDER %q cannot be served locally: serve runs a self-hosted engine, so use llamacpp or omlx", + provider) + } +} + +// cmdServe reads an Outfit and runs the inference server its PROVIDER names. +// With a PRESET it turns the matching preset section into the command; without +// one it derives the command from the Outfit's own instructions. Either way it +// prints the command before running it. The Outfit path defaults to ./Outfit. +func cmdServe(args []string) error { + fs := flag.NewFlagSet("serve", flag.ContinueOnError) + var dryRun bool + fs.BoolVar(&dryRun, "dry-run", false, "print the server command without running it") + fs.BoolVar(&dryRun, "n", false, "print the command without running it (shorthand)") + if err := fs.Parse(args); err != nil { + return err + } + + var path string + if rest := fs.Args(); len(rest) > 0 { + path = rest[0] + } + sel, outfitPath, err := readOutfit("outfit serve ", path) + if err != nil { + return err + } + engine, err := engineFor(sel.Provider) + if err != nil { + return err + } + + // Anything the Outfit states overrides the preset's own values. + params, err := engine.params(sel) + if err != nil { + return err + } + + binary := engine.binary() + var argv []string + if sel.Preset != "" { + presetPath := resolvePresetPath(sel.Preset, outfitPath) + data, err := os.ReadFile(presetPath) + if err != nil { + return fmt.Errorf("reading preset %s: %w", presetPath, err) + } + pre, err := preset.Parse(data) + if err != nil { + return fmt.Errorf("%s: %w", presetPath, err) + } + // The preset's sections are named by the friendly ALIAS, not the + // provider-native MODEL, so that is what selects one. + sec, err := pre.Select(sel.Alias) + if err != nil { + return fmt.Errorf("%s: %w", presetPath, err) + } + argv = pre.CommandIn(engine.dialect, binary, engine.subcommand, sec, params) + fmt.Printf("Using preset %s (model %s)\n\n", presetPath, sec.Name) + } else { + if engine.needsModel && sel.Model == "" { + return fmt.Errorf("serve needs a PRESET or a MODEL (an HF repo like org/model:quant, or a path to a .gguf)") + } + argv = append([]string{binary}, engine.subcommand...) + argv = append(argv, engine.dialect.Flags(params)...) + if sel.Model != "" { + fmt.Printf("Serving %s from %s\n\n", sel.Model, outfitPath) + } else { + // An engine that needs no model to start (oMLX serves a whole + // directory) has nothing to name but itself. + fmt.Printf("Starting %s from %s\n\n", sel.Provider, outfitPath) + } + } + + fmt.Printf("%s\n\n", preset.FormatCommand(argv)) + if dryRun { + return nil + } + + cmd := exec.Command(argv[0], argv[1:]...) + cmd.Stdin = os.Stdin + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + if err := cmd.Run(); err != nil { + if errors.Is(err, exec.ErrNotFound) || errors.Is(err, os.ErrNotExist) { + return fmt.Errorf("%s not found — %s", argv[0], engine.installHint) + } + return err + } + return nil +} + +// resolvePresetPath resolves an Outfit's PRESET value: a relative one is taken +// against the Outfit's own directory, so an Outfit and its preset travel +// together (the same rule REMOTE uses). +func resolvePresetPath(presetValue, outfitPath string) string { + if filepath.IsAbs(presetValue) { + return presetValue + } + return filepath.Join(filepath.Dir(outfitPath), presetValue) +} + +// omlxServeParams turns the oMLX settings an Outfit states into preset params. +// Only the bind address maps: oMLX serves a whole --model-dir and picks the +// model per request, so MODEL and ALIAS keep their usual job of naming what the +// harness asks for, and CONTEXT sizes the harness's window rather than the +// server (oMLX has no context flag). Everything else — the model directory, +// memory guard, SSD cache — comes from the PRESET or from oMLX's own settings. +// +// The API key is deliberately not passed: serve prints the command it runs, and +// oMLX takes its key as a command-line flag, so passing one would put the secret +// on screen and in the process table. Auth belongs in oMLX's own settings. +func omlxServeParams(sel outfit.Selection) ([]preset.Param, error) { + return bindAddressParams(sel) +} + +// llamacppServeParams turns the llama-server settings an Outfit states into +// preset params: the provider-native MODEL supplies the model source (hf for a +// Hugging Face repo, model for a .gguf path); ALIAS, CONTEXT, and BASEURL fill +// in the rest. They seed a preset-less command and, with a preset, override its +// values. +func llamacppServeParams(sel outfit.Selection) ([]preset.Param, error) { + var params []preset.Param + if sel.Model != "" { + if isModelPath(sel.Model) { + params = append(params, preset.Param{Key: "model", Value: sel.Model}) + } else { + params = append(params, preset.Param{Key: "hf", Value: sel.Model}) + } + } + if sel.Alias != "" { + params = append(params, preset.Param{Key: "alias", Value: sel.Alias}) + } + if sel.Context != "" { + n, err := contextsize.Parse(sel.Context) + if err != nil { + return nil, err + } + params = append(params, preset.Param{Key: "ctx-size", Value: strconv.Itoa(n)}) + } + bind, err := bindAddressParams(sel) + if err != nil { + return nil, err + } + return append(params, bind...), nil +} + +// bindAddressParams turns an Outfit's BASEURL into the host and port flags that +// bind a server to the endpoint the harness will call. Both engines spell these +// the same way. With no BASEURL it yields nothing, so the server's own defaults +// stand. +func bindAddressParams(sel outfit.Selection) ([]preset.Param, error) { + if sel.BaseURL == "" { + return nil, nil + } + host, port, err := hostPortFromURL(sel.BaseURL) + if err != nil { + return nil, err + } + var params []preset.Param + if host != "" { + params = append(params, preset.Param{Key: "host", Value: host}) + } + if port != "" { + params = append(params, preset.Param{Key: "port", Value: port}) + } + return params, nil +} + +// isModelPath reports whether a MODEL value is a local file rather than a +// Hugging Face repo: an absolute or explicitly-relative path, a home-relative +// path, or anything ending in .gguf. Everything else is treated as org/model. +func isModelPath(model string) bool { + if strings.HasSuffix(strings.ToLower(model), ".gguf") { + return true + } + return strings.HasPrefix(model, "/") || + strings.HasPrefix(model, "./") || + strings.HasPrefix(model, "../") || + strings.HasPrefix(model, "~") +} + +// hostPortFromURL extracts the host and port from a BASEURL so serve can bind +// llama-server to the same endpoint the harness will call. A bare host:port +// with no scheme is accepted too. +func hostPortFromURL(raw string) (host, port string, err error) { + if !strings.Contains(raw, "://") { + raw = "http://" + raw + } + u, err := url.Parse(raw) + if err != nil { + return "", "", fmt.Errorf("invalid BASEURL %q: %w", raw, err) + } + return u.Hostname(), u.Port(), nil +} diff --git a/cmd/outfit/serve_test.go b/cmd/outfit/serve_test.go index 5c41ca7b..c9978437 100644 --- a/cmd/outfit/serve_test.go +++ b/cmd/outfit/serve_test.go @@ -28,18 +28,56 @@ func writePresetOutfit(t *testing.T, outfitBody string) string { return outfitPath } -// stubLlamaServer points llamaServerBinary at a script that records its argv to -// argsFile, and restores the original binary afterwards. -func stubLlamaServer(t *testing.T, argsFile string) { +// stubServer writes a script named binName that records its argv to argsFile, +// points *target at it, and restores the original afterwards. +func stubServer(t *testing.T, target *string, binName, argsFile string) { t.Helper() - script := filepath.Join(t.TempDir(), "llama-server") + script := filepath.Join(t.TempDir(), binName) body := "#!/bin/sh\nprintf '%s\\n' \"$@\" > " + argsFile + "\n" if err := os.WriteFile(script, []byte(body), 0o755); err != nil { t.Fatal(err) } - orig := llamaServerBinary - llamaServerBinary = script - t.Cleanup(func() { llamaServerBinary = orig }) + orig := *target + *target = script + t.Cleanup(func() { *target = orig }) +} + +// stubLlamaServer points llamaServerBinary at a script that records its argv to +// argsFile, and restores the original binary afterwards. +func stubLlamaServer(t *testing.T, argsFile string) { + t.Helper() + stubServer(t, &llamaServerBinary, "llama-server", argsFile) +} + +// stubOMLX does the same for the oMLX CLI, so no Apple Silicon (or oMLX +// install) is needed to pin the argv serve builds for it. +func stubOMLX(t *testing.T, argsFile string) { + t.Helper() + stubServer(t, &omlxBinary, "omlx-cli", argsFile) +} + +// omlxPreset is a preset written in oMLX's own flag vocabulary. `m` is here +// deliberately: llama.cpp's dialect would rewrite it to --model, and this preset +// must not be read that way. +const omlxPreset = `[*] +model-dir = /Users/me/models +host = 127.0.0.1 +port = 8000 + +[qwen] +memory-guard = safe +m = should-stay-literal +` + +// writeOMLXOutfit writes an oMLX preset.ini and an Outfit referencing it into a +// fresh temp dir, and returns the Outfit's path. +func writeOMLXOutfit(t *testing.T, outfitBody string) string { + t.Helper() + dir := t.TempDir() + mustWrite(t, filepath.Join(dir, "preset.ini"), omlxPreset) + outfitPath := filepath.Join(dir, "Outfit") + mustWrite(t, outfitPath, outfitBody) + return outfitPath } func TestCmdServe_PresetDryRun(t *testing.T) { @@ -271,3 +309,183 @@ func TestCmdServe_RelativePresetResolvesToOutfitDir(t *testing.T) { t.Errorf("preset not resolved relative to the Outfit dir:\n%s", out) } } + +// TestCmdServe_OMLXDryRun pins the whole oMLX shape: the `serve` subcommand, and +// a bind address taken from BASEURL. +func TestCmdServe_OMLXDryRun(t *testing.T) { + dir := t.TempDir() + outfitPath := filepath.Join(dir, "Outfit") + mustWrite(t, outfitPath, "PROVIDER omlx\nBASEURL http://127.0.0.1:9100/v1\n") + + out := captureStdout(t, func() { + if err := cmdServe([]string{"--dry-run", outfitPath}); err != nil { + t.Fatalf("cmdServe: %v", err) + } + }) + + for _, want := range []string{"serve", "--host 127.0.0.1", "--port 9100"} { + if !strings.Contains(out, want) { + t.Errorf("command missing %q:\n%s", want, out) + } + } + // llama.cpp's vocabulary must not leak into another engine's command. + for _, unwanted := range []string{"llama-server", "--hf-repo", "--ctx-size"} { + if strings.Contains(out, unwanted) { + t.Errorf("command should not contain %q:\n%s", unwanted, out) + } + } +} + +// TestCmdServe_OMLXNeedsNoModel checks that oMLX starts with neither a PRESET +// nor a MODEL: it serves a whole model directory and picks per request, so +// llama.cpp's "needs a PRESET or a MODEL" rule must not apply to it. +func TestCmdServe_OMLXNeedsNoModel(t *testing.T) { + dir := t.TempDir() + outfitPath := filepath.Join(dir, "Outfit") + mustWrite(t, outfitPath, "PROVIDER omlx\n") + + out := captureStdout(t, func() { + if err := cmdServe([]string{"--dry-run", outfitPath}); err != nil { + t.Fatalf("cmdServe: %v", err) + } + }) + + if !strings.Contains(out, "serve") { + t.Errorf("expected a serve command:\n%s", out) + } + // With no BASEURL, oMLX's own defaults stand. + if strings.Contains(out, "--host") || strings.Contains(out, "--port") { + t.Errorf("no BASEURL should mean no bind flags:\n%s", out) + } +} + +// TestCmdServe_OMLXPresetKeysAreNotAliased is the guard on the dialect split: an +// oMLX preset is read in oMLX's vocabulary, so a key llama.cpp would rewrite is +// left exactly as written. +func TestCmdServe_OMLXPresetKeysAreNotAliased(t *testing.T) { + outfitPath := writeOMLXOutfit(t, "PROVIDER omlx\nALIAS qwen\nPRESET preset.ini\n") + + out := captureStdout(t, func() { + if err := cmdServe([]string{"--dry-run", outfitPath}); err != nil { + t.Fatalf("cmdServe: %v", err) + } + }) + + for _, want := range []string{"--model-dir /Users/me/models", "--memory-guard safe", "-m should-stay-literal"} { + if !strings.Contains(out, want) { + t.Errorf("command missing %q:\n%s", want, out) + } + } + if strings.Contains(out, "--model should-stay-literal") { + t.Errorf("llama.cpp aliasing leaked into the oMLX dialect:\n%s", out) + } +} + +// TestCmdServe_OMLXOutfitOverridesPreset checks the Outfit still wins over its +// preset for the settings it states, as it does for llama.cpp. +func TestCmdServe_OMLXOutfitOverridesPreset(t *testing.T) { + outfitPath := writeOMLXOutfit(t, "PROVIDER omlx\nALIAS qwen\nPRESET preset.ini\nBASEURL http://0.0.0.0:9999/v1\n") + + out := captureStdout(t, func() { + if err := cmdServe([]string{"--dry-run", outfitPath}); err != nil { + t.Fatalf("cmdServe: %v", err) + } + }) + + if !strings.Contains(out, "--host 0.0.0.0") || !strings.Contains(out, "--port 9999") { + t.Errorf("Outfit should override the preset's bind address:\n%s", out) + } + if strings.Contains(out, "--port 8000") { + t.Errorf("preset port should have been overridden:\n%s", out) + } +} + +// TestCmdServe_OMLXNeverPassesAPIKey is a security guard: serve prints the +// command it runs, and oMLX takes its key on the command line, so a resolved +// secret must never reach either. +func TestCmdServe_OMLXNeverPassesAPIKey(t *testing.T) { + t.Setenv("OPENAI_API_KEY", "sk-must-not-appear") + dir := t.TempDir() + outfitPath := filepath.Join(dir, "Outfit") + mustWrite(t, outfitPath, "PROVIDER omlx\nBASEURL http://127.0.0.1:9100/v1\n") + + out := captureStdout(t, func() { + if err := cmdServe([]string{"--dry-run", outfitPath}); err != nil { + t.Fatalf("cmdServe: %v", err) + } + }) + + if strings.Contains(out, "sk-must-not-appear") || strings.Contains(out, "--api-key") { + t.Errorf("serve must not pass or print an API key:\n%s", out) + } +} + +// TestCmdServe_OMLXRuns checks the stubbed binary is actually executed with the +// subcommand first. +func TestCmdServe_OMLXRuns(t *testing.T) { + argsFile := filepath.Join(t.TempDir(), "args") + stubOMLX(t, argsFile) + dir := t.TempDir() + outfitPath := filepath.Join(dir, "Outfit") + mustWrite(t, outfitPath, "PROVIDER omlx\nBASEURL http://127.0.0.1:9100/v1\n") + + captureStdout(t, func() { + if err := cmdServe([]string{outfitPath}); err != nil { + t.Fatalf("cmdServe: %v", err) + } + }) + + got, err := os.ReadFile(argsFile) + if err != nil { + t.Fatalf("reading recorded args: %v", err) + } + args := strings.Fields(string(got)) + if len(args) == 0 || args[0] != "serve" { + t.Errorf("expected the serve subcommand first, got %v", args) + } + if !strings.Contains(string(got), "9100") { + t.Errorf("recorded args missing the port:\n%s", got) + } +} + +// TestCmdServe_OMLXNotFound checks the install hint names oMLX rather than +// llama.cpp. +func TestCmdServe_OMLXNotFound(t *testing.T) { + orig := omlxBinary + omlxBinary = filepath.Join(t.TempDir(), "definitely-not-installed") + t.Cleanup(func() { omlxBinary = orig }) + dir := t.TempDir() + outfitPath := filepath.Join(dir, "Outfit") + mustWrite(t, outfitPath, "PROVIDER omlx\n") + + var err error + captureStdout(t, func() { err = cmdServe([]string{outfitPath}) }) + if err == nil || !strings.Contains(err.Error(), "not found") { + t.Fatalf("expected a not-found error, got %v", err) + } + if !strings.Contains(err.Error(), "oMLX") { + t.Errorf("hint should name oMLX, got %v", err) + } + if strings.Contains(err.Error(), "llama.cpp") { + t.Errorf("hint should not name llama.cpp, got %v", err) + } +} + +// TestCmdServe_UnsupportedProvider pins the behaviour change: serve used to run +// llama-server whatever the PROVIDER said, which quietly served the wrong engine. +func TestCmdServe_UnsupportedProvider(t *testing.T) { + dir := t.TempDir() + outfitPath := filepath.Join(dir, "Outfit") + mustWrite(t, outfitPath, "PROVIDER ollama\nMODEL llama3.2\n") + + var err error + captureStdout(t, func() { err = cmdServe([]string{"--dry-run", outfitPath}) }) + if err == nil { + t.Fatal("expected an error for a provider that is not a local engine") + } + for _, want := range []string{"ollama", "llamacpp", "omlx"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error should mention %q, got %v", want, err) + } + } +} diff --git a/docs/README.md b/docs/README.md index 306dd105..4dfada1f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -15,8 +15,8 @@ Four words carry the whole tool: Pi is also supported. Chosen at runtime, so the same selection works for either. See [`outfit harness`](commands/harness.md). - **Provider** — what `outfit` can configure, from a built-in - catalogue: OpenRouter, AWS Bedrock, Ollama, llama.cpp, vLLM, or any - OpenAI-compatible endpoint. See + catalogue: OpenRouter, AWS Bedrock, Ollama, llama.cpp, vLLM, oMLX (Apple + Silicon), or any OpenAI-compatible endpoint. See [`outfit list`](commands/list.md). - **Outfit file** — a small, declarative file (like a `Dockerfile`, but for your agent's model) that captures one selection so you can commit it and @@ -46,7 +46,7 @@ Four words carry the whole tool: | [`outfit unapply`](commands/unapply.md) | Remove what an `Outfit` file selects | | [`outfit alias`](commands/alias.md) | Name an `Outfit` so the name works anywhere a path does | | [`outfit unalias`](commands/unalias.md) | Drop a registered name | -| [`outfit serve`](commands/serve.md) | Run `llama-server` for the model an `Outfit` names | +| [`outfit serve`](commands/serve.md) | Run the inference server for the model an `Outfit` names | | [`outfit remote`](commands/remote.md) | Run the model on a cloud GPU that stops when you do | | [`outfit export`](commands/export.md) | Capture the current setup as an `Outfit` | | [`outfit harness`](commands/harness.md) | Launch the agent, optionally dressing it first | @@ -63,7 +63,7 @@ Four words carry the whole tool: | `OUTFIT_PROVIDERS` | Path to a custom provider catalogue (`--providers` beats it) | | `OUTFIT_BASE_URL` | Overrides any provider's API base URL (`--base-url`/`-u` beats it) | | `DEEPSEEK_API_KEY`, `OPENAI_API_KEY`, … | Provider API keys — `outfit list` shows which each provider reads | -| `OLLAMA_BASE_URL`, `LLAMACPP_BASE_URL`, `OPENAI_BASE_URL` | Per-provider endpoint overrides | +| `OLLAMA_BASE_URL`, `LLAMACPP_BASE_URL`, `OMLX_BASE_URL`, `VLLM_BASE_URL`, `OPENAI_BASE_URL` | Per-provider endpoint overrides | | `AWS_REGION` | Region for AWS Bedrock | Keys are looked up in a `.env` file **beside the `Outfit` being applied** first @@ -72,5 +72,6 @@ shell environment — so a project keeps its own key next to the file that needs it, the same way `PRESET` and `REMOTE` travel with an Outfit. They are **never written into the agent's config** — outfit writes a reference the agent resolves when it runs, and `outfit harness` passes the keys it can resolve to the agent it launches. If you start the agent yourself, -set the variable in your own environment. Local providers (Ollama, llama.cpp on -localhost) need no key; Bedrock uses your AWS credentials. +set the variable in your own environment. Local providers on localhost (Ollama, +llama.cpp) need no key; Bedrock uses your AWS credentials. oMLX needs one only if +you enabled its API-key auth — set `OPENAI_API_KEY` before applying if you did. diff --git a/docs/commands/serve.md b/docs/commands/serve.md index a3a48b31..3bd68e29 100644 --- a/docs/commands/serve.md +++ b/docs/commands/serve.md @@ -1,11 +1,11 @@ # outfit serve -Run `llama-server` for the model an [`Outfit` file](../outfit-file.md) names — -so the same file that points your agent at a local model can also start the -server behind it. +Run the inference server for the model an [`Outfit` file](../outfit-file.md) +names — so the same file that points your agent at a local model can also start +the server behind it. ```sh -outfit serve # reads ./Outfit and runs llama-server +outfit serve # reads ./Outfit and runs its PROVIDER's server outfit serve path/to/Outfit outfit serve qwen3.6-27b # a name registered with `outfit alias` outfit serve --dry-run # print the command without launching the server @@ -15,7 +15,27 @@ It prints the command before running it, and never touches your agent's config — pair it with [`outfit apply`](apply.md) to point the agent at the server. -## Simple case — straight from the Outfit +## The engine comes from `PROVIDER` + +`PROVIDER` already names the engine, so `serve` needs no keyword of its own — +the same way [`outfit remote deploy`](remote.md) picks the engine for a cloud +GPU: + +| `PROVIDER` | `serve` runs | +| ---------- | ------------ | +| `llamacpp` | `llama-server` | +| `omlx` | [oMLX](https://omlx.ai) on Apple Silicon | + +Any other provider is an error: `serve` launches a self-hosted engine, and the +rest of the catalogue names endpoints somebody else runs. + +Each engine reads its `PRESET` in its **own** flag vocabulary. That matters: +llama.cpp's short aliases would rewrite another engine's keys (`m` to `--model`, +`c` to `--ctx-size`), so a preset is never portable between engines. + +## llama.cpp + +### Simple case — straight from the Outfit With no `PRESET`, `serve` builds the command from the Outfit itself: @@ -30,7 +50,7 @@ BASEURL http://127.0.0.1:8080/v1 # llama-server --host/--port `MODEL` becomes `-hf` (a Hugging Face repo) or `-m` (anything that looks like a path or ends in `.gguf`); `ALIAS`, `CONTEXT`, and `BASEURL` fill in the rest. -## Full control — a llama.cpp preset +### Full control — a llama.cpp preset For flags an Outfit doesn't model — `-ngl`, `--jinja`, KV-cache types, draft models — point at a llama.cpp @@ -62,17 +82,69 @@ Keys map straight to flags — `ctx-size = 262144` becomes `--ctx-size 262144`, A relative `PRESET` path resolves against the Outfit's own directory, so the pair can travel together. +## oMLX + +[oMLX](https://omlx.ai) serves MLX models on Apple Silicon. It differs from +llama.cpp in one way that shapes everything else: it loads a whole **model +directory** and picks the model per request, rather than being launched with one +model. So a bare Outfit is enough to start it: + +```dockerfile +PROVIDER omlx +BASEURL http://127.0.0.1:8000/v1 # omlx-cli serve --host/--port +``` + +- `BASEURL` sets the bind address. With none, oMLX's own defaults stand. +- `MODEL` and `ALIAS` keep their usual job of naming what the *agent* asks for; + they are not launch flags. +- `CONTEXT` sizes the *harness's* window — oMLX has no context flag. + +Everything else — the model directory, the memory guard, the SSD cache — comes +from a `PRESET` written in oMLX's own flags, or from oMLX's settings +(`~/.omlx/settings.json`, and its admin panel): + +```ini +[*] +model-dir = /Users/you/models + +[default] # selected by the Outfit's ALIAS +memory-guard = safe +paged-ssd-cache-dir = /Users/you/.omlx/cache +max-concurrent-requests = 16 +``` + +Preset values are passed to the server verbatim, so `~` is not expanded — write +paths out in full. + +`serve` never passes `--api-key`. It prints the command it runs, and oMLX takes +its key on the command line, so passing one would put the secret on your screen +and in the process table. If you want auth on the server, configure it in oMLX; +`outfit add`/`apply` still picks up `OPENAI_API_KEY` for the agent's own config. + +Note that oMLX can require a key even on localhost (it is an admin-panel +setting). Because the `omlx` provider is `apiKeyOptional`, `outfit` only writes +the key reference when `OPENAI_API_KEY` is set **at apply time** — so if your +oMLX needs a key, set it before `add`/`apply`, not just before launching the +agent. + +### Finding the binary + +oMLX ships as a macOS app, so `serve` looks for `omlx-cli` on your `PATH` first +and falls back to `/Applications/oMLX.app/Contents/MacOS/omlx-cli`. If you've +only ever launched it from the menu bar, the fallback is the one that finds it. + ## Flags | Flag | Meaning | | ---- | ------- | -| `-n`, `--dry-run` | Print the `llama-server` command without running it | +| `-n`, `--dry-run` | Print the server command without running it | ## Notes -- `serve` needs `llama-server` on your `PATH` (e.g. - `brew install llama.cpp`). -- Without a `PRESET`, the Outfit must name a `MODEL`. +- `serve` needs the engine installed: `llama-server` on your `PATH` (e.g. + `brew install llama.cpp`), or [oMLX](https://omlx.ai). +- For llama.cpp, an Outfit with no `PRESET` must name a `MODEL`. oMLX needs + neither. ## See also diff --git a/docs/outfit-file.md b/docs/outfit-file.md index 5e9a6d17..1f62b76c 100644 --- a/docs/outfit-file.md +++ b/docs/outfit-file.md @@ -13,7 +13,7 @@ ALIAS deepseek # optional; friendly name for the model CONTEXT 128k # optional; context window OUTPUT 32k # optional; max output tokens BASEURL https://gateway/v1 # optional; API base URL override -PRESET ./preset.ini # optional; llama.cpp preset for `outfit serve` +PRESET ./preset.ini # optional; engine preset for `outfit serve` ``` Applying it is the same as running the equivalent @@ -103,9 +103,11 @@ Rules: - `BASEURL` overrides the provider's API base URL — handy for a gateway or a llama.cpp server on a non-default port. `URL`, `BASE-URL`, and `BASE_URL` are accepted as aliases. -- `PRESET` points at a llama.cpp preset `.ini`, used only by +- `PRESET` points at a preset `.ini`, used only by [`outfit serve`](commands/serve.md); `apply` ignores it. A relative path is - resolved against the Outfit's own directory. + resolved against the Outfit's own directory. The file is read in the flag + vocabulary of the engine `PROVIDER` names, so a preset written for llama.cpp + is not portable to oMLX and vice versa. - Keywords are **case-insensitive** — `provider`, `Provider`, and `PROVIDER` are all accepted — but **UPPERCASE is canonical** and is what `outfit export` writes. diff --git a/examples/omlx/gemma-4-e2b/Outfit b/examples/omlx/gemma-4-e2b/Outfit new file mode 100644 index 00000000..e1de6176 --- /dev/null +++ b/examples/omlx/gemma-4-e2b/Outfit @@ -0,0 +1,13 @@ +# Gemma-4-E2B (instruction-tuned) served locally by oMLX on Apple Silicon — a +# small model that fits comfortably in memory. +# oMLX loads a whole model directory and picks per request, so MODEL names what +# opencode asks for — it is not a launch flag. +PROVIDER omlx +# MODEL must match a model your oMLX actually serves — its directory name or an +# alias. If you downloaded something else, change this: list what yours exposes +# with `curl -H "Authorization: Bearer $OPENAI_API_KEY" localhost:8000/v1/models`. +# A name oMLX does not have is rejected with "model not found". +MODEL gemma-4-E2B-it-MLX-6bit +CONTEXT 32768 # opencode's context window +PRESET ./preset.ini # `outfit serve` launches omlx-cli from this +# BASEURL http://127.0.0.1:8000/v1 # uncomment for a non-default host/port diff --git a/examples/omlx/gemma-4-e2b/README.md b/examples/omlx/gemma-4-e2b/README.md new file mode 100644 index 00000000..c30429e7 --- /dev/null +++ b/examples/omlx/gemma-4-e2b/README.md @@ -0,0 +1,132 @@ +# Gemma-4-E2B on oMLX (Apple Silicon) + +Run LM Studio's MLX build of Gemma-4-E2B-it on your Mac with +[oMLX](https://omlx.ai), then point opencode at it with the [`Outfit`](Outfit) in +this directory. + +`E2B` is the small member of the Gemma-4 family: an *effective* ~2B parameters, +so it loads in seconds and leaves plenty of memory headroom — a good first model +to prove the setup end to end before reaching for something larger (see the +[Qwen3.6-27B example](../qwen3.6/README.md) for a heavier one). + +## Prerequisites + +- An **Apple Silicon** Mac (M1 or later). oMLX is Apple Silicon only, so + [`outfit remote`](../../../docs/commands/remote.md) cannot deploy it. +- [oMLX](https://omlx.ai), installed from its DMG or from source. +- The 6-bit build is only a few GB, so memory is not a concern here. + +## 1. Pull the model + +oMLX's admin panel (`http://localhost:8000/admin`) has a Hugging Face browser +that downloads a model in one click, which is the easiest route. + +To do it from the shell instead, put the model in the directory oMLX serves +from: + +```sh +hf download lmstudio-community/gemma-4-E2B-it-MLX-6bit \ + --local-dir ~/models/gemma-4-E2B-it-MLX-6bit +``` + +The **directory name** is what the model is called over the API — +`gemma-4-E2B-it-MLX-6bit` here — unless you set an alias for it in the admin +panel. That is the name the `Outfit` uses as its `MODEL`. + +## 2. Start oMLX + +```sh +omlx-cli serve \ + --model-dir ~/models \ + --memory-guard safe \ + --max-concurrent-requests 8 \ + --host 127.0.0.1 --port 8000 +``` + +- `--model-dir` — the directory of models to serve. oMLX loads what it needs and + picks per request, so there is no "the model" flag here. +- `--host`/`--port` — the OpenAI-compatible API is served at + `http://127.0.0.1:8000/v1`. + +Installed from the DMG and haven't put it on your `PATH`? The binary lives at +`/Applications/oMLX.app/Contents/MacOS/omlx-cli`. + +Rather than remember those flags, this directory keeps them in a +[`preset.ini`](preset.ini) (with the paths spelled out in full — preset values +reach the server verbatim, so `~` is not expanded) and lets `outfit` build and +run the command: + +```sh +outfit serve # from this directory; reads ./Outfit and its PRESET +outfit serve --dry-run # print the omlx-cli command without running it +``` + +`serve` finds `omlx-cli` on your `PATH`, falling back to the app bundle. + +### Check it's up + +```sh +curl -H "Authorization: Bearer $OPENAI_API_KEY" http://localhost:8000/v1/models +``` + +This is also how to confirm what your model is actually called — the response +lists the names requests must use. (Drop the header if you have not enabled +oMLX's API-key auth.) + +## 3. Point opencode at it + +oMLX speaks the OpenAI-compatible API, which is what the `omlx` provider targets +(default base URL `http://localhost:8000/v1`). Apply the [`Outfit`](Outfit) in +this directory: + +```sh +outfit apply examples/omlx/gemma-4-e2b/Outfit +# or, from this directory: +outfit apply +``` + +The Outfit is: + +```dockerfile +PROVIDER omlx +MODEL gemma-4-E2B-it-MLX-6bit +CONTEXT 32768 +PRESET ./preset.ini +``` + +`MODEL` must match a model oMLX actually has — its directory name, or the alias +you set in the admin panel. This Outfit names the model downloaded above; **if +you downloaded something else, change `MODEL` to match** (list yours with the +`/v1/models` call above). A name oMLX does not have comes back as +`model not found`. + +`CONTEXT` sets opencode's context window. Unlike llama.cpp there is no +`--ctx-size` to keep it in step with — oMLX sizes its cache dynamically — so +this is purely about what opencode will send. + +## A note on API keys + +By default `outfit` writes no key for a localhost oMLX: the `omlx` provider is +marked `apiKeyOptional`, so a local endpoint with no `OPENAI_API_KEY` set gets no +`apiKey` field at all. + +**But oMLX can require a key even on localhost** — it is a per-install setting in +the admin panel, and some builds turn it on by default. If yours does, set +`OPENAI_API_KEY` **before** you apply the Outfit: + +```sh +OPENAI_API_KEY=your-omlx-key outfit apply # writes an {env:OPENAI_API_KEY} reference +OPENAI_API_KEY=your-omlx-key opencode # or `outfit harness`, which forwards it +``` + +The *before* matters: with the key unset at apply time, `outfit` omits the +`apiKey` field, and exporting the key afterwards does nothing — there is no +reference in the config to resolve, so you would re-run `apply`. See the +[Qwen3.6 example](../qwen3.6/README.md#a-note-on-api-keys) for the full +explanation. + +## See also + +- [`outfit serve`](../../../docs/commands/serve.md) — the full command reference +- [The `Outfit` file](../../../docs/outfit-file.md) — full syntax +- A heavier model on oMLX: [`examples/omlx/qwen3.6`](../qwen3.6/README.md) diff --git a/examples/omlx/gemma-4-e2b/preset.ini b/examples/omlx/gemma-4-e2b/preset.ini new file mode 100644 index 00000000..c569738b --- /dev/null +++ b/examples/omlx/gemma-4-e2b/preset.ini @@ -0,0 +1,18 @@ +# An oMLX preset for Gemma-4-E2B, mirroring the omlx-cli command in this +# directory's README. `outfit serve` turns it into that command and runs it. +# +# These are `omlx-cli serve` flags with their leading dashes stripped. Note that +# they are NOT llama.cpp's: each engine's preset is read in its own vocabulary, +# so this file is not interchangeable with the ones under examples/llamacpp/. +# +# Paths are passed to the server verbatim, so `~` is not expanded — spell them +# out in full. + +[*] +host = 127.0.0.1 +port = 8000 + +[default] # served with no ALIAS: the only section +model-dir = /Users/you/models +memory-guard = safe # this model is small; safe leaves ample room +max-concurrent-requests = 8 diff --git a/examples/omlx/qwen3.6/Outfit b/examples/omlx/qwen3.6/Outfit new file mode 100644 index 00000000..47a3fb70 --- /dev/null +++ b/examples/omlx/qwen3.6/Outfit @@ -0,0 +1,13 @@ +# Qwen3.6-35B-A3B served locally by oMLX on Apple Silicon (the model this +# directory's README walks through downloading). +# oMLX loads a whole model directory and picks per request, so MODEL names what +# opencode asks for — it is not a launch flag. +PROVIDER omlx +# MODEL must match a model your oMLX actually serves — its directory name or an +# alias. If you downloaded something else, change this: list what yours exposes +# with `curl -H "Authorization: Bearer $OPENAI_API_KEY" localhost:8000/v1/models`. +# A name oMLX does not have is rejected with "model not found". +MODEL Qwen3.6-35B-A3B-4bit +CONTEXT 32768 # opencode's context window +PRESET ./preset.ini # `outfit serve` launches omlx-cli from this +# BASEURL http://127.0.0.1:8000/v1 # uncomment for a non-default host/port diff --git a/examples/omlx/qwen3.6/README.md b/examples/omlx/qwen3.6/README.md new file mode 100644 index 00000000..b4a7be93 --- /dev/null +++ b/examples/omlx/qwen3.6/README.md @@ -0,0 +1,177 @@ +# Qwen3.6-35B-A3B on oMLX (Apple Silicon) + +Run an MLX build of Qwen3.6-35B-A3B on your Mac with [oMLX](https://omlx.ai), +then point opencode at it with the [`Outfit`](Outfit) in this directory. + +`A3B` means it's a mixture-of-experts model: ~35B total parameters but only ~3B +active per token, so it's far lighter to run than its size suggests — which is +what makes it practical on a laptop's unified memory. + +oMLX is built on [MLX](https://github.com/ml-explore/mlx), Apple's array +framework, so it runs on the GPU and Neural Engine rather than through a CUDA +path. Its headline feature is **paged SSD caching**: KV-cache blocks are +persisted to disk, so returning to a long prompt restores instead of recomputing +it. For a coding agent, which re-sends a large and mostly-unchanged context on +every turn, that is the difference between a usable and an unusable setup. + +## Prerequisites + +- An **Apple Silicon** Mac (M1 or later). oMLX is Apple Silicon only — there is + no Intel or Linux build, and no cloud equivalent, so + [`outfit remote`](../../../docs/commands/remote.md) cannot deploy it. +- [oMLX](https://omlx.ai), installed from its DMG or from source. +- Enough unified memory for the weights. The 4-bit build is roughly 20 GB, so + plan for a 32 GB machine or better. + +## 1. Pull the model + +oMLX's admin panel (`http://localhost:8000/admin`) has a Hugging Face browser +that downloads a model in one click, which is the easiest route. + +To do it from the shell instead, put the model in the directory oMLX serves +from: + +```sh +hf download mlx-community/Qwen3.6-35B-A3B-4bit \ + --local-dir ~/models/Qwen3.6-35B-A3B-4bit +``` + +The **directory name** is what the model is called over the API — +`Qwen3.6-35B-A3B-4bit` here — unless you set an alias for it in the admin panel. +That is the name the `Outfit` uses as its `MODEL`. + +## 2. Start oMLX + +```sh +omlx-cli serve \ + --model-dir ~/models \ + --memory-guard safe \ + --paged-ssd-cache-dir ~/.omlx/cache \ + --max-concurrent-requests 8 \ + --host 127.0.0.1 --port 8000 +``` + +What the flags do: + +- `--model-dir` — the directory of models to serve. oMLX loads *all* of them and + picks per request, which is why there's no "the model" flag here. +- `--memory-guard safe` — keep memory headroom for the rest of the machine. + `balanced` gives the model more; `--memory-guard-gb` sets a ceiling directly. +- `--paged-ssd-cache-dir` — persist KV-cache blocks to SSD. This is the setting + that makes long agent contexts fast on repeat turns. +- `--max-concurrent-requests` — continuous batching width (default 8). +- `--host`/`--port` — the OpenAI-compatible API is served at + `http://127.0.0.1:8000/v1`. + +Installed from the DMG and haven't put it on your `PATH`? The binary lives at +`/Applications/oMLX.app/Contents/MacOS/omlx-cli`. + +Rather than remember those flags, this directory keeps them in a +[`preset.ini`](preset.ini) and lets `outfit` build and run the command (with the +paths spelled out in full — preset values reach the server verbatim, so `~` is +not expanded): + +```sh +outfit serve # from this directory; reads ./Outfit and its PRESET +outfit serve --dry-run # print the omlx-cli command without running it +``` + +`serve` finds `omlx-cli` on your `PATH`, falling back to the app bundle — so +this works even if you've only ever launched oMLX from the menu bar. + +Note the preset holds `omlx-cli` flags, **not** llama.cpp's. Each engine's +preset is read in its own vocabulary, so this file and the ones under +[`examples/llamacpp/`](../../llamacpp/) are not interchangeable. + +### Check it's up + +```sh +curl http://127.0.0.1:8000/v1/models +``` + +This is also how to confirm what your model is actually called — the response +lists the names requests must use. + +## 3. Point opencode at it + +oMLX speaks the OpenAI-compatible API, which is what the `omlx` provider targets +(default base URL `http://localhost:8000/v1`). Apply the [`Outfit`](Outfit) in +this directory: + +```sh +outfit apply examples/omlx/qwen3.6/Outfit +# or, from this directory: +outfit apply +``` + +The Outfit is: + +```dockerfile +PROVIDER omlx +MODEL Qwen3.6-35B-A3B-4bit # the directory name under the preset's model-dir +CONTEXT 32768 +PRESET ./preset.ini +``` + +`MODEL` matters more here than it does for a single-model llama.cpp server: +oMLX serves everything in its model directory and dispatches on the requested +name, so it must match a model oMLX actually has — its directory name, or the +alias you set in the admin panel. This Outfit names the model downloaded above; +**if you downloaded something else, change `MODEL` to match.** List what yours +exposes with: + +```sh +curl -H "Authorization: Bearer $OPENAI_API_KEY" http://localhost:8000/v1/models +``` + +A name oMLX does not have comes back as `model not found`. And unlike the +llama.cpp examples — where `outfit serve` can fetch the weights from Hugging +Face — oMLX only serves what you have already downloaded. + +`CONTEXT` sets opencode's context window. Unlike llama.cpp there is no +`--ctx-size` to keep it in step with — oMLX sizes its cache dynamically — so +this is purely about what opencode will send. + +Running on a different host or port? Add a `BASEURL` line (the file ships one +commented out): + +```dockerfile +BASEURL http://127.0.0.1:9100/v1 +``` + +That sets both what opencode calls and what `serve` binds to. + +## A note on API keys + +By default `outfit` writes no key for a localhost oMLX: the `omlx` provider is +marked `apiKeyOptional`, so a local endpoint with no `OPENAI_API_KEY` set gets no +`apiKey` field at all. + +**But oMLX can require a key even on localhost** — it is a per-install setting in +the admin panel, and some builds turn it on by default. If yours does, requests +without a key come back as `API key required`. To use it, set `OPENAI_API_KEY` +**before** you apply the Outfit: + +```sh +OPENAI_API_KEY=your-omlx-key outfit apply # writes an {env:OPENAI_API_KEY} reference +OPENAI_API_KEY=your-omlx-key opencode # or `outfit harness`, which forwards it +``` + +The *before* matters: with the key set at apply time, `outfit` writes an +`{env:OPENAI_API_KEY}` reference into the config (never the secret). With it +unset, `outfit` omits the `apiKey` field entirely — and exporting the key later +does nothing, because there is no reference in the config to resolve. If you hit +`API key required` after the fact, set the variable and re-run `apply`. + +Reaching an oMLX on another machine works the same way: start it with +`--api-key`, export `OPENAI_API_KEY`, and point `OMLX_BASE_URL` at the Mac. + +`outfit serve` never passes `--api-key`: it prints the command it runs, so a key +there would land on your screen and in the process table. Configure auth in oMLX +itself. + +## See also + +- [`outfit serve`](../../../docs/commands/serve.md) — the full command reference +- [The `Outfit` file](../../../docs/outfit-file.md) — full syntax +- The same model on llama.cpp: [`examples/llamacpp/qwen3.6`](../../llamacpp/qwen3.6/README.md) diff --git a/examples/omlx/qwen3.6/preset.ini b/examples/omlx/qwen3.6/preset.ini new file mode 100644 index 00000000..6aac6f95 --- /dev/null +++ b/examples/omlx/qwen3.6/preset.ini @@ -0,0 +1,18 @@ +# An oMLX preset, mirroring the omlx-cli command in this directory's README. +# `outfit serve` turns it into that command and runs it. +# +# These are `omlx-cli serve` flags with their leading dashes stripped. Note that +# they are NOT llama.cpp's: each engine's preset is read in its own vocabulary, +# so this file is not interchangeable with the ones under examples/llamacpp/. + +[*] +host = 127.0.0.1 +port = 8000 + +# Paths are passed to the server verbatim, so `~` is not expanded — spell them +# out in full. +[default] # served with no ALIAS: the only section +model-dir = /Users/you/models +memory-guard = safe # leave headroom for the rest of the machine +paged-ssd-cache-dir = /Users/you/.omlx/cache # persist KV blocks across requests +max-concurrent-requests = 8 diff --git a/internal/catalog/catalog_test.go b/internal/catalog/catalog_test.go index db51feb1..3cc2df47 100644 --- a/internal/catalog/catalog_test.go +++ b/internal/catalog/catalog_test.go @@ -673,3 +673,75 @@ func TestBuildPiProvider_RemoteOptionalKeyKeepsReference(t *testing.T) { t.Errorf("apiKey = %q, want the placeholder for a local server", prov.APIKey) } } + +// TestBuildProviderBlock_OMLXKeylessLocal checks that a local oMLX server gets +// no apiKey option at all: the var points nowhere useful for a server that needs +// no auth, and writing it would make opencode fail to resolve it. +func TestBuildProviderBlock_OMLXKeylessLocal(t *testing.T) { + cat, _ := Load() + p := cat.Providers["omlx"] + if p == nil { + t.Fatal("omlx missing from the catalogue") + } + block, defaultModel, err := BuildProviderBlock("omlx", p, "my-model", "", noEnv) + if err != nil { + t.Fatalf("BuildProviderBlock: %v", err) + } + if defaultModel != "omlx/my-model" { + t.Errorf("defaultModel = %q, want omlx/my-model", defaultModel) + } + options := block["options"].(map[string]any) + if _, ok := options["apiKey"]; ok { + t.Errorf("a keyless local server should get no apiKey: %v", options) + } + if options["baseURL"] != "http://localhost:8000/v1" { + t.Errorf("baseURL = %v", options["baseURL"]) + } +} + +// TestBuildProviderBlock_OMLXReferencesKeyWhenSet checks the other half of +// apiKeyOptional: a key that is set is written as an env reference, never as the +// resolved secret. +func TestBuildProviderBlock_OMLXReferencesKeyWhenSet(t *testing.T) { + cat, _ := Load() + block, _, err := BuildProviderBlock("omlx", cat.Providers["omlx"], "my-model", "", + envMap(map[string]string{"OPENAI_API_KEY": "sk-secret"})) + if err != nil { + t.Fatalf("BuildProviderBlock: %v", err) + } + options := block["options"].(map[string]any) + if options["apiKey"] != EnvRef("OPENAI_API_KEY") { + t.Errorf("apiKey = %v, want an env reference", options["apiKey"]) + } + if options["apiKey"] == "sk-secret" { + t.Error("the resolved secret must never reach the config") + } +} + +// TestBuildPiProvider_OMLXPlaceholderThenReference pins both sides of the Pi +// rule for oMLX: a keyless local server gets the literal placeholder (Pi hides a +// provider's models until some auth is configured), while a remote endpoint gets +// the $VAR reference Pi resolves at run time. +func TestBuildPiProvider_OMLXPlaceholderThenReference(t *testing.T) { + cat, _ := Load() + p := cat.Providers["omlx"] + + local, _, err := BuildPiProvider("omlx", p, "my-model", "", noEnv) + if err != nil { + t.Fatalf("BuildPiProvider: %v", err) + } + if local.APIKey != piPlaceholderAPIKey { + t.Errorf("local apiKey = %q, want %q", local.APIKey, piPlaceholderAPIKey) + } + if local.API != "openai-completions" { + t.Errorf("api = %q, want openai-completions", local.API) + } + + remote, _, err := BuildPiProvider("omlx", p, "my-model", "http://mac-studio.local:8000/v1", noEnv) + if err != nil { + t.Fatalf("BuildPiProvider: %v", err) + } + if remote.APIKey != "$OPENAI_API_KEY" { + t.Errorf("remote apiKey = %q, want $OPENAI_API_KEY", remote.APIKey) + } +} diff --git a/internal/catalog/providers.yaml b/internal/catalog/providers.yaml index 2bbcbe55..c75644a9 100644 --- a/internal/catalog/providers.yaml +++ b/internal/catalog/providers.yaml @@ -107,6 +107,23 @@ providers: pi: api: openai-completions + omlx: + description: oMLX server on Apple Silicon (OpenAI-compatible API) + name: oMLX + npm: "@ai-sdk/openai-compatible" + # Like llama.cpp, a local oMLX server usually needs no key and none is + # injected when the var is unset; oMLX can be started with --api-key, so a + # key is picked up when there is one (e.g. a Mac on the LAN named by + # OMLX_BASE_URL). + apiKeyEnv: OPENAI_API_KEY + apiKeyOptional: true + options: + baseURL: http://localhost:8000/v1 + optionsFromEnv: + baseURL: OMLX_BASE_URL + pi: + api: openai-completions + vllm: description: vLLM server (OpenAI-compatible API) name: vLLM diff --git a/internal/preset/preset.go b/internal/preset/preset.go index 407de260..db528dbc 100644 --- a/internal/preset/preset.go +++ b/internal/preset/preset.go @@ -1,16 +1,21 @@ -// Package preset reads llama.cpp INI preset files and turns one model section -// into a concrete `llama-server` command. +// Package preset reads INI preset files and turns one section into a concrete +// inference-server command. // -// llama.cpp presets (https://github.com/ggml-org/llama.cpp/blob/master/docs/preset.md) -// are INI files where each `[section]` is a model and every key is a -// `llama-server` argument with its leading dashes stripped — `ctx-size = 4096` -// is `--ctx-size 4096`. A leading `[*]` (or `[global]`) section holds defaults +// The format is llama.cpp's (https://github.com/ggml-org/llama.cpp/blob/master/docs/preset.md): +// INI files where each `[section]` is a model and every key is a server +// argument with its leading dashes stripped — `ctx-size = 4096` is +// `--ctx-size 4096`. A leading `[*]` (or `[global]`) section holds defaults // shared by every model; a per-model section overrides them. // -// Presets are designed for the server's router (multi-model) mode, loaded with -// `--models-preset`. There is no equivalent for plain single-model serving, so -// this package flattens a chosen section back into the explicit flags you would -// have typed by hand. +// llama.cpp presets are designed for the server's router (multi-model) mode, +// loaded with `--models-preset`. There is no equivalent for plain single-model +// serving, so this package flattens a chosen section back into the explicit +// flags you would have typed by hand. +// +// Parsing is engine-neutral; only the *spelling* of flags differs between +// engines, which is what a Dialect carries. The package-level Flags, +// CanonicalKey, Preset.Args and Preset.Command render in the LlamaCpp dialect, +// so callers that predate dialects keep their behaviour. package preset import ( @@ -152,20 +157,46 @@ func (p Preset) Select(model string) (Section, error) { case len(p.Sections) == 1: return p.Sections[0], nil default: - return Section{}, fmt.Errorf("preset defines %d models (%s); set MODEL in the Outfit to choose one", len(p.Sections), strings.Join(p.SectionNames(), ", ")) + return Section{}, fmt.Errorf("preset defines %d models (%s); set ALIAS in the Outfit to choose one", len(p.Sections), strings.Join(p.SectionNames(), ", ")) } } -// Flags merges ordered layers of params into `llama-server` flag tokens. Later -// layers override earlier ones by canonical flag name, in place, so the first -// layer fixes the order: pass globals, then the section, then any overrides. It -// does not include the binary name; see Command. -func Flags(layers ...[]Param) []string { +// Dialect describes how one inference engine spells its flags: the short +// aliases it accepts, and the flags it takes bare with no value. The zero +// Dialect passes every key through unchanged, which is what an engine with only +// long-form flags wants. +// +// A dialect is never guessed from the preset's contents — it comes from the +// engine the Outfit's PROVIDER names — because the tables are not +// interchangeable: llama.cpp's would rewrite another engine's `m` to `--model` +// and `c` to `--ctx-size`. +type Dialect struct { + // Aliases maps a short flag alias, as it appears in a preset key (without + // the leading dash), to its canonical long-form name. + Aliases map[string]string + // Boolean lists canonical flag names the engine accepts with no value. + Boolean map[string]bool +} + +// LlamaCpp is the dialect `llama-server` speaks, and the one the package-level +// rendering helpers use. +var LlamaCpp = Dialect{Aliases: canonical, Boolean: boolean} + +// OMLX is the dialect `omlx-cli serve` speaks. It spells every flag in long +// form (`--model-dir`, `--memory-guard`, `--paged-ssd-cache-dir`), so no +// aliasing is applied and keys pass through as written. +var OMLX = Dialect{} + +// Flags merges ordered layers of params into flag tokens. Later layers override +// earlier ones by canonical flag name, in place, so the first layer fixes the +// order: pass globals, then the section, then any overrides. It does not +// include the binary name; see Command. +func (d Dialect) Flags(layers ...[]Param) []string { var ordered []Param at := map[string]int{} // canonical flag name -> index in ordered for _, layer := range layers { for _, kv := range layer { - ck := canonicalName(kv.Key) + ck := d.CanonicalKey(kv.Key) if i, ok := at[ck]; ok { ordered[i] = kv continue @@ -177,22 +208,37 @@ func Flags(layers ...[]Param) []string { var args []string for _, kv := range ordered { - args = append(args, flagFor(kv.Key, kv.Value)...) + args = append(args, d.flagFor(kv.Key, kv.Value)...) } return args } +// Flags merges ordered layers of params into `llama-server` flag tokens, in the +// LlamaCpp dialect. +func Flags(layers ...[]Param) []string { + return LlamaCpp.Flags(layers...) +} + // Args flattens the global defaults and a section into `llama-server` flags, // with the section's values overriding the globals'. func (p Preset) Args(sec Section) []string { return Flags(p.Global, sec.Params) } -// Command builds the full argv (binary first) for serving a section. Any -// override layers (e.g. values an Outfit specifies) win over the preset. +// Command builds the full argv (binary first) for serving a section in the +// LlamaCpp dialect. Any override layers (e.g. values an Outfit specifies) win +// over the preset. func (p Preset) Command(binary string, sec Section, overrides ...[]Param) []string { + return p.CommandIn(LlamaCpp, binary, nil, sec, overrides...) +} + +// CommandIn builds the full argv for serving a section with a given engine: the +// binary, then any subcommand the engine needs (`omlx-cli serve` takes one, +// `llama-server` does not), then the flags rendered in that engine's dialect. +func (p Preset) CommandIn(d Dialect, binary string, subcommand []string, sec Section, overrides ...[]Param) []string { layers := append([][]Param{p.Global, sec.Params}, overrides...) - return append([]string{binary}, Flags(layers...)...) + argv := append([]string{binary}, subcommand...) + return append(argv, d.Flags(layers...)...) } // canonical maps llama.cpp short flag aliases — as they appear in preset keys, @@ -225,35 +271,34 @@ var boolean = map[string]bool{ "cpu-moe": true, "color": true, } -// canonicalName lower-cases a preset key and resolves any short alias to its -// canonical long-form flag name, so the same flag written different ways (c, -// ctx-size, env-var forms) collapses to one when merging layers. -func canonicalName(key string) string { +// CanonicalKey resolves a preset key to the canonical long-form flag name that +// Flags would emit, so the same setting written different ways (`ngl` and +// `n-gpu-layers`, `c` and `ctx-size`) compares equal. Callers that need to +// match specific flags rather than render them should compare canonical names. +// A dialect with no aliases only lower-cases the key. +func (d Dialect) CanonicalKey(key string) string { name := strings.ToLower(strings.TrimSpace(key)) - if c, ok := canonical[name]; ok { + if c, ok := d.Aliases[name]; ok { name = c } return name } -// CanonicalKey resolves a preset key to the canonical long-form flag name that -// Flags would emit, so the same setting written different ways (`ngl` and -// `n-gpu-layers`, `c` and `ctx-size`) compares equal. Callers that need to -// match specific flags rather than render them should compare canonical names. +// CanonicalKey resolves a preset key in the LlamaCpp dialect. func CanonicalKey(key string) string { - return canonicalName(key) + return LlamaCpp.CanonicalKey(key) } -// flagFor turns one preset key/value into its `llama-server` flag tokens. -func flagFor(key, value string) []string { - name := canonicalName(key) +// flagFor turns one preset key/value into its flag tokens. +func (d Dialect) flagFor(key, value string) []string { + name := d.CanonicalKey(key) flag := "--" + name if len(name) == 1 { // an unknown single-character key is a short flag flag = "-" + name } value = strings.TrimSpace(value) - if boolean[name] { + if d.Boolean[name] { if isFalsy(value) { return nil } diff --git a/internal/preset/preset_test.go b/internal/preset/preset_test.go index e94c9815..a92ae824 100644 --- a/internal/preset/preset_test.go +++ b/internal/preset/preset_test.go @@ -192,7 +192,7 @@ func TestFlagFor(t *testing.T) { {"unknown-long", "", []string{"--unknown-long"}}, // valueless passthrough } for _, tc := range cases { - got := flagFor(tc.key, tc.value) + got := LlamaCpp.flagFor(tc.key, tc.value) if strings.Join(got, " ") != strings.Join(tc.want, " ") { t.Errorf("flagFor(%q,%q) = %v, want %v", tc.key, tc.value, got, tc.want) } @@ -219,3 +219,54 @@ func TestCommand(t *testing.T) { t.Errorf("Command args = %v", argv[1:]) } } + +// TestDialectOMLXPassesKeysThrough is the point of having dialects at all: oMLX +// spells every flag in long form, so llama.cpp's alias table must not touch its +// keys. `m` and `c` are the dangerous ones — llama.cpp would silently rewrite +// them to --model and --ctx-size, neither of which oMLX accepts. +func TestDialectOMLXPassesKeysThrough(t *testing.T) { + params := []Param{ + {Key: "model-dir", Value: "/models"}, + {Key: "memory-guard", Value: "safe"}, + {Key: "c", Value: "literal"}, + } + got := strings.Join(OMLX.Flags(params), " ") + want := "--model-dir /models --memory-guard safe -c literal" + if got != want { + t.Errorf("OMLX.Flags =\n %s\nwant\n %s", got, want) + } +} + +// TestDialectOMLXHasNoBareBooleans checks the boolean table is llama.cpp's +// alone: a key oMLX does not treat as a bare flag keeps its value. +func TestDialectOMLXHasNoBareBooleans(t *testing.T) { + got := strings.Join(OMLX.Flags([]Param{{Key: "jinja", Value: "1"}}), " ") + if got != "--jinja 1" { + t.Errorf("OMLX.Flags = %q, want %q", got, "--jinja 1") + } +} + +// TestPackageFlagsStillLlamaCpp guards the callers that predate dialects: the +// package-level helpers, and remote.go's CanonicalKey, must keep rendering +// llama.cpp. +func TestPackageFlagsStillLlamaCpp(t *testing.T) { + got := strings.Join(Flags([]Param{{Key: "hf", Value: "org/m"}, {Key: "mmap", Value: "1"}}), " ") + if got != "--hf-repo org/m --mmap" { + t.Errorf("Flags = %q, want %q", got, "--hf-repo org/m --mmap") + } + if CanonicalKey("ngl") != "n-gpu-layers" { + t.Errorf("CanonicalKey(ngl) = %q, want n-gpu-layers", CanonicalKey("ngl")) + } +} + +// TestCommandInAddsSubcommand checks an engine that needs a subcommand gets it +// between the binary and the flags. +func TestCommandInAddsSubcommand(t *testing.T) { + p, _ := Parse([]byte("[m]\nmodel-dir = /models\n")) + sec, _ := p.Select("m") + argv := p.CommandIn(OMLX, "omlx-cli", []string{"serve"}, sec) + want := "omlx-cli serve --model-dir /models" + if strings.Join(argv, " ") != want { + t.Errorf("CommandIn = %v, want %s", argv, want) + } +} diff --git a/openspec/changes/add-omlx-provider/.openspec.yaml b/openspec/changes/add-omlx-provider/.openspec.yaml new file mode 100644 index 00000000..5849c2db --- /dev/null +++ b/openspec/changes/add-omlx-provider/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-08-01 diff --git a/openspec/changes/add-omlx-provider/design.md b/openspec/changes/add-omlx-provider/design.md new file mode 100644 index 00000000..a7d8c8ab --- /dev/null +++ b/openspec/changes/add-omlx-provider/design.md @@ -0,0 +1,104 @@ +## Context + +Two engines now have to coexist behind one command. They differ in three ways that the +design has to absorb, and one way it deliberately does not. + +- **Binary**: `llama-server` is a `PATH` install; oMLX ships as a signed macOS app whose + CLI lives at `/Applications/oMLX.app/Contents/MacOS/omlx-cli`. +- **Shape**: `llama-server` takes flags directly; `omlx-cli` takes a `serve` subcommand + first. +- **Model**: `llama-server` is launched with one model. oMLX loads a whole `--model-dir` + and selects per request, so it has nothing to be told at launch and no context flag. +- **Preset format**: identical. Both are INI files of `key = value` under `[section]` + headers. Only the *vocabulary* differs. + +## Goals / Non-Goals + +- Goals: one engine-selection rule shared with the cloud path; no silent misrendering of + one engine's preset in another's vocabulary; existing llama.cpp behaviour byte-identical. +- Non-goals: a plugin system for engines; a generic engine descriptor in the catalogue; + cloud deployment of oMLX. + +## Decisions + +### Engine selection comes from PROVIDER + +The alternatives were a new `ENGINE` Outfit keyword or a `--engine` flag on `serve`. + +`PROVIDER` wins because the codebase already committed to it: `runnerFor` maps `PROVIDER` +to a cloud runner and documents that `serve` does the same locally. Adding a keyword would +create a second source of truth for a question already answered, and `internal/outfit` +errors on unknown keywords, so every Outfit stating `ENGINE` would break on older +binaries. A flag would be worse still — it is per-invocation, so it could not travel with +the Outfit. + +The cost is the breaking change: `serve` now rejects providers that are not local engines. +That is a fix, not a regression. The previous behaviour ran `llama-server` for an Outfit +saying `PROVIDER ollama`, which is a command the Outfit never asked for. + +`engineFor` mirrors `runnerFor` deliberately: a hard-coded switch in Go, not a catalogue +field. Putting engine metadata in `providers.yaml` would be a single source of truth +across `serve`/`deploy`/`list`, but it would also mean a user-supplied `--providers` file +could name an arbitrary binary for `outfit serve` to execute. The allow-list stays in the +binary. + +### Dialects, not a second parser + +`internal/preset` splits cleanly: `Parse`, `Select`, `SectionNames`, the layer-merge loop +and `FormatCommand` never look at key names, while `canonical`, `boolean` and `flagFor` +are entirely llama.cpp's. So the engine-specific part becomes a value — `Dialect{Aliases, +Boolean}` — rather than a fork of the package. + +The zero `Dialect` passes keys through unchanged, which is exactly right for oMLX: every +`omlx-cli serve` flag is long-form, so there is nothing to alias. + +The package-level `Flags`, `CanonicalKey`, `Preset.Args` and `Preset.Command` are kept as +`LlamaCpp` wrappers rather than removed. This is not politeness to callers: `remote.go`'s +`isCloudOwned` compares preset keys through `preset.CanonicalKey` to decide which flags +the cloud owns, so changing that function's meaning would silently change what +`outfit remote deploy` strips from a preset. + +**The trap this closes**: an oMLX preset read in llama.cpp's dialect parses without error +and produces a wrong command — `m` becomes `--model`, `c` becomes `--ctx-size`, and a +`key = 0` for a name in llama.cpp's boolean table vanishes. The dialect therefore comes +from the engine `PROVIDER` names, never from inspecting the file. + +### What an Outfit means for oMLX + +Only `BASEURL` becomes a launch flag, via the already engine-neutral `hostPortFromURL`. +`MODEL` and `ALIAS` keep their existing meaning — the id the harness requests — because +that is what they mean for every other provider, and for oMLX it is load-bearing rather +than a label: the server really does dispatch on it. `CONTEXT` sizes the harness's window; +oMLX has no context flag to keep in step with. + +`isModelPath`'s `.gguf` heuristic is not reached on the oMLX path. MLX models are +directories of safetensors, so the predicate would be wrong for them. + +`serveEngine.needsModel` keeps llama.cpp's "needs a PRESET or a MODEL" rule from applying +to an engine that needs neither. + +### No API key to the server + +`serve` prints the command it runs, and oMLX takes `--api-key` on the command line. Passing +a resolved key would put it on the terminal and in `ps` output — a sharper failure than +the config-file secrets the rest of the codebase is careful to avoid, since those at least +stay `0600` on disk. Auth on the server is configured in oMLX. The catalogue path is +unaffected: `outfit add`/`apply` still writes an `{env:OPENAI_API_KEY}` reference for the +harness when a key is set. + +### Serve moves to its own file + +`cmd/outfit/serve.go` is partly cohesion and partly a constraint: +`TestCompletionCoversDispatch` scans `main.go` for `case "…":` at one tab of indentation to +prove every dispatched command is completable. `engineFor`'s switch matches that shape, so +leaving it in `main.go` reports `llamacpp` and `omlx` as uncompletable commands. + +## Risks / Trade-offs + +- **Breaking `serve` for non-engine providers** — mitigated by an error that names the + valid values, and by the fact that the previous behaviour was already wrong. +- **oMLX cannot be verified in CI** — it is Apple Silicon only. Mitigated by the stubbed + binary seam the llama.cpp tests already use, which pins the whole argv without running + anything real. +- **Port 8000 collides with the `vllm` default** — harmless (nothing assumes base URLs are + unique), and documented so anyone running both knows to set `OMLX_BASE_URL`. diff --git a/openspec/changes/add-omlx-provider/proposal.md b/openspec/changes/add-omlx-provider/proposal.md new file mode 100644 index 00000000..e5667a4c --- /dev/null +++ b/openspec/changes/add-omlx-provider/proposal.md @@ -0,0 +1,45 @@ +## Why + +[oMLX](https://omlx.ai) is a local inference server for Apple Silicon, built on Apple's +MLX framework, exposing an OpenAI-compatible API. `outfit` today knows two local engines +it can point a harness at (`llamacpp`, `vllm`) and exactly one it can *launch* +(`llama-server`), so Mac users get half the flow: they can dress the agent, but not start +the server behind it from the same file. + +Adding oMLX also settles an inconsistency the code already claims is settled. `runnerFor` +documents itself as "PROVIDER already names the engine — `outfit serve` starts that engine +locally", and the CLI usage says "PROVIDER picks the engine, just as it does for serve". +Neither is true: `cmdServe` never reads `PROVIDER` and unconditionally runs +`llama-server`, so an Outfit naming any other provider silently starts the wrong server. + +## What Changes + +- Add an `omlx` provider to the catalogue: an OpenAI-compatible endpoint defaulting to + `http://localhost:8000/v1`, overridable with `OMLX_BASE_URL`, keyless on localhost and + key-bearing when pointed elsewhere (`apiKeyOptional`, like `llamacpp`), and Pi-capable + via `openai-completions`. +- **BREAKING**: `outfit serve` selects its engine from `PROVIDER` rather than always + running `llama-server`. `llamacpp` and `omlx` are served; every other provider is now a + loud error instead of a silently wrong command. +- Teach `internal/preset` that flag *spelling* is per-engine. Parsing stays shared; a + `Dialect` carries the alias and boolean tables, with `LlamaCpp` the default so existing + callers are unaffected, and `OMLX` passing long-form keys through untouched. +- Give `serve` a per-engine install hint, and find the oMLX CLI on `PATH` or in the macOS + app bundle, since oMLX ships as a signed app rather than a `PATH` install. +- `serve` never passes an API key to oMLX. It prints the command it runs, and oMLX takes + its key as a flag, so a resolved secret would land on screen and in the process table. + +Non-goals: + +- `outfit remote deploy` stays closed to oMLX. The cloud side is CUDA/NVIDIA EC2 and oMLX + is Apple Silicon only, so `runnerFor` keeps rejecting it — correctly. +- No new Outfit keyword. `PROVIDER` names the engine, as the codebase already intends. + +## Impact + +- Affected specs: `provider-catalog`, `local-serving` +- Affected code: `internal/catalog/providers.yaml`, `internal/preset/preset.go`, + `cmd/outfit/serve.go` (extracted from `main.go`), `cmd/outfit/main.go` (usage text) +- Migration: an Outfit that named a non-engine `PROVIDER` and relied on `serve` running + `llama-server` anyway must change its `PROVIDER` to `llamacpp`. That configuration was + already producing a command for an engine the Outfit did not name. diff --git a/openspec/changes/add-omlx-provider/specs/local-serving/spec.md b/openspec/changes/add-omlx-provider/specs/local-serving/spec.md new file mode 100644 index 00000000..7d426297 --- /dev/null +++ b/openspec/changes/add-omlx-provider/specs/local-serving/spec.md @@ -0,0 +1,145 @@ +## ADDED Requirements + +### Requirement: Choosing the engine + +`outfit serve` SHALL launch the inference engine the Outfit's `PROVIDER` names, +the local counterpart of the runner `outfit remote deploy` selects from the same +instruction. `llamacpp` SHALL run `llama-server`; `omlx` SHALL run the oMLX CLI. +There SHALL be no default: a `PROVIDER` that is not a self-hosted engine SHALL +fail, naming the providers that can be served, rather than launching an engine +the Outfit did not ask for. + +An engine whose executable is normally installed outside the `PATH` SHALL also be +looked for at its conventional install location, so a user who has never put it +on their `PATH` can still serve. + +#### Scenario: Provider selects the engine + +- **WHEN** the Outfit says `PROVIDER omlx` +- **THEN** the printed command runs the oMLX CLI, not `llama-server` + +#### Scenario: A provider that is not a local engine + +- **WHEN** the Outfit names a hosted provider such as `ollama` or `openrouter` +- **THEN** the command fails, listing the providers that can be served locally + +#### Scenario: Engine installed outside the PATH + +- **WHEN** the oMLX CLI is not on the `PATH` but is present in its macOS app + bundle +- **THEN** `serve` uses the bundled executable + +### Requirement: Serving a multi-model engine + +An engine that loads a directory of models and selects one per request SHALL be +servable with neither a `PRESET` nor a `MODEL`, since it needs nothing at launch +to know what to load. For such an engine the Outfit's `MODEL` and `ALIAS` SHALL +keep their usual meaning — the id the harness requests — rather than becoming +launch flags, and `CONTEXT` SHALL size the harness's window only. `BASEURL`'s +host and port SHALL still set the server's bind address, and SHALL be omitted +entirely when the Outfit states no `BASEURL`, so the engine's own defaults stand. + +#### Scenario: Bare Outfit starts the server + +- **WHEN** the Outfit states only `PROVIDER omlx` +- **THEN** the server is launched with no bind flags and no model flags, rather + than failing for want of a model + +#### Scenario: Bind address from BASEURL + +- **WHEN** the Outfit adds `BASEURL http://127.0.0.1:9100/v1` +- **THEN** the command binds the server to that host and port + +### Requirement: Secrets never reach the server command + +`serve` SHALL NOT resolve or pass an API key to the engine it launches. Because +the command is printed before it runs, and an engine may accept its key as a +command-line flag, passing one would expose the secret on screen and to any local +process. Configuring authentication on the server is the engine's own concern. + +#### Scenario: A set key is not passed through + +- **WHEN** the provider's API key variable is set in the environment +- **THEN** neither the printed command nor the launched process carries the key + +## MODIFIED Requirements + +### Requirement: Serve basics + +`outfit serve [path]` SHALL read an Outfit (default `./Outfit`, aliases and +directories accepted like every Outfit command), build the command for the +engine its `PROVIDER` names, print it in copy-pasteable shell form, and run it +with stdio forwarded. `--dry-run`/`-n` SHALL print the command without +launching. A missing binary SHALL produce an install hint naming **that** engine +rather than a raw exec error. + +#### Scenario: Dry run + +- **WHEN** the user runs `outfit serve --dry-run` +- **THEN** the resolved command is printed and no server starts + +#### Scenario: Engine not installed + +- **WHEN** the selected engine's binary cannot be found +- **THEN** the error suggests installing that engine, not another one + +### Requirement: Preset-based serving + +With a `PRESET`, the referenced `.ini` SHALL supply the command: a relative +preset path resolves against the Outfit's own directory, so the pair can travel +together. The preset's `[*]`/`[global]` section holds shared defaults; each named +section is one model whose keys are server arguments with dashes stripped. The +served section is chosen by the Outfit's `ALIAS`, matched case-insensitively; a +preset with exactly one section is always served; no sections is an error; +several sections with no matching name SHALL fail listing the available +sections. Values the Outfit itself states SHALL override the preset's. + +A preset SHALL be read in the flag vocabulary of the engine the `PROVIDER` names, +never in a vocabulary inferred from the file. A preset written for one engine is +therefore not portable to another: read in the wrong vocabulary it would parse +cleanly and produce a command with silently rewritten or dropped flags. + +#### Scenario: Outfit overrides the preset + +- **WHEN** the preset section sets `ctx-size = 4096` and the Outfit says + `CONTEXT 32768` +- **THEN** the command carries a context size of 32768 + +#### Scenario: Ambiguous preset + +- **WHEN** the preset defines several sections and the Outfit's `ALIAS` matches + none +- **THEN** the command fails listing the section names to choose from + +#### Scenario: Another engine's preset keys are left alone + +- **WHEN** an oMLX preset contains a key that llama.cpp would treat as a short + alias +- **THEN** the key is rendered as written, not rewritten to llama.cpp's + long-form flag + +### Requirement: Flag rendering + +Preset keys SHALL render as the engine's flags. Where an engine defines short +aliases, they SHALL be canonicalised to their long form so the same flag written +different ways collapses to one when layers merge, later layers overriding +earlier ones in place; an engine that defines none SHALL pass its keys through +unchanged. Flags the engine accepts bare SHALL render bare when truthy and be +dropped when falsy (`0`, `false`, `off`, `no`). Printed commands SHALL quote only +the tokens that need it, and SHALL pass values to the server verbatim. + +#### Scenario: Layer override by canonical name + +- **WHEN** the global section sets `c = 4096` and the model section sets + `ctx-size = 8192` for llama.cpp +- **THEN** the command carries a single context-size flag with value 8192 + +#### Scenario: Boolean flags + +- **WHEN** a llama.cpp preset sets `jinja = 1` and `mmap = 0` +- **THEN** the command includes a bare `--jinja` and no mmap flag + +#### Scenario: An engine with no short aliases + +- **WHEN** an oMLX preset sets `model-dir` and `memory-guard` +- **THEN** both render as their long-form flags with their values unchanged diff --git a/openspec/changes/add-omlx-provider/specs/provider-catalog/spec.md b/openspec/changes/add-omlx-provider/specs/provider-catalog/spec.md new file mode 100644 index 00000000..82524470 --- /dev/null +++ b/openspec/changes/add-omlx-provider/specs/provider-catalog/spec.md @@ -0,0 +1,31 @@ +## ADDED Requirements + +### Requirement: Apple Silicon local provider + +The catalogue SHALL include an `omlx` provider describing a local +[oMLX](https://omlx.ai) server: an OpenAI-compatible endpoint defaulting to +`http://localhost:8000/v1`, overridable per-provider with `OMLX_BASE_URL`, and +Pi-capable through the `openai-completions` API. + +Like the llama.cpp provider it SHALL be optionally authenticated: a local server +needs no key, while the same provider may name an oMLX server started with +`--api-key` on another machine. An unset key at a local endpoint therefore +yields no `apiKey` option for opencode and the keyless placeholder for Pi, while +a set key — or any non-local endpoint — yields the environment reference the +harness resolves at run time. + +Its default port coinciding with another provider's is not a conflict: the +catalogue places no uniqueness requirement on base URLs. + +#### Scenario: Local server needs no key + +- **WHEN** a selection names `omlx` with the default localhost base URL and no + API key set +- **THEN** the opencode provider block carries no `apiKey` option, and the Pi + entry carries the keyless placeholder so its models stay selectable + +#### Scenario: Remote server keeps the key reference + +- **WHEN** the base URL is overridden to a non-local host +- **THEN** the API key is written as an environment reference, never as the + resolved secret diff --git a/openspec/changes/add-omlx-provider/tasks.md b/openspec/changes/add-omlx-provider/tasks.md new file mode 100644 index 00000000..e78176ee --- /dev/null +++ b/openspec/changes/add-omlx-provider/tasks.md @@ -0,0 +1,42 @@ +## 1. Catalogue + +- [x] 1.1 Add an `omlx` provider to `internal/catalog/providers.yaml`: `npm` `@ai-sdk/openai-compatible`, `options.baseURL` `http://localhost:8000/v1`, `optionsFromEnv.baseURL` `OMLX_BASE_URL`, `apiKeyEnv` `OPENAI_API_KEY` with `apiKeyOptional: true`, and a `pi` block using `openai-completions`. + +## 2. Preset dialects + +- [x] 2.1 Add `Dialect{Aliases, Boolean}` to `internal/preset`, with `LlamaCpp` wrapping the existing tables and `OMLX` as the zero value (long-form keys pass through). +- [x] 2.2 Move `canonicalName`/`flagFor`/`Flags` onto `Dialect`; keep package-level `Flags`, `CanonicalKey`, `Preset.Args` and `Preset.Command` as `LlamaCpp` wrappers so `remote.go`'s `isCloudOwned` is unchanged. +- [x] 2.3 Add `Preset.CommandIn(dialect, binary, subcommand, sec, overrides...)` for engines that need a subcommand. +- [x] 2.4 Update the package doc; fix `Select`'s error text, which told the user to set `MODEL` when `serve` selects a section by `ALIAS`. + +## 3. Serve + +- [x] 3.1 Extract the serve command from `cmd/outfit/main.go` into `cmd/outfit/serve.go` (required: `TestCompletionCoversDispatch` scans `main.go` for `case "…":` at one tab, so `engineFor`'s switch there reads as new commands). +- [x] 3.2 Add `serveEngine` (binary, subcommand, dialect, params, needsModel, installHint) and `engineFor`, the local twin of `runnerFor`; error for any provider that is not a local engine. +- [x] 3.3 Add `resolveOMLXBinary`: the `omlxBinary` override, then `PATH`, then `/Applications/oMLX.app/Contents/MacOS/omlx-cli`. +- [x] 3.4 Split `outfitServeParams` into `llamacppServeParams` and `omlxServeParams` over a shared `bindAddressParams`; oMLX maps only `BASEURL`. +- [x] 3.5 Route `cmdServe` through the engine: per-engine install hint, subcommand in the argv, and `needsModel` gating the "needs a PRESET or a MODEL" error. + +## 4. Tests + +- [x] 4.1 `internal/preset`: dialect isolation (`OMLX` does not alias `m`/`c`, has no bare booleans), the LlamaCpp wrappers still render `--hf-repo`/`--mmap` and `CanonicalKey`, and `CommandIn` places the subcommand. +- [x] 4.2 `internal/catalog`: `omlx` keyless on localhost, an env reference when the key is set, and the Pi placeholder/reference pair. +- [x] 4.3 `cmd/outfit`: generalise the binary stub, then cover the oMLX argv, no-model start, preset pass-through, Outfit-over-preset, never-passes-a-key, execution, the oMLX install hint, and the now-rejected non-engine provider. +- [x] 4.4 `go test ./... -race -cover` green, every package ≥ 80%. + +## 5. Docs + +- [x] 5.1 Rewrite `docs/commands/serve.md` around per-engine selection; document the oMLX flags, the binary lookup, and that presets are not portable between engines. +- [x] 5.2 Update `docs/README.md` (provider list, base-URL env row, keyless-local sentence), `docs/outfit-file.md` (`PRESET` is engine-specific), and `README.md` (provider list, serve section, env vars, guides). +- [x] 5.3 Update the CLI usage text in `cmd/outfit/main.go`. +- [x] 5.4 Update `AGENTS.md`: the layout (new `serve.go`), the `internal/preset` and `serve` notes, and a new trap for cross-dialect presets. +- [x] 5.5 Add `examples/omlx/qwen3.6/` (README, Outfit, preset.ini) and link it from `README.md`. +- [x] 5.6 Add a CHANGELOG entry, including the breaking `serve` change. + +## 6. Verify + +- [x] 6.1 `gofmt -l` clean, `go vet ./...`, `go test ./... -race -cover` green. +- [x] 6.2 `outfit serve --dry-run` on the oMLX example, on a bare `PROVIDER omlx`, and on a non-engine provider (which must now error). +- [x] 6.3 Regression: `outfit serve --dry-run examples/llamacpp/qwen3.6/Outfit` produces the same command as before the change. +- [x] 6.4 `outfit add -p omlx` under both harnesses: keyless on localhost, `{env:…}`/`$VAR` when remote or keyed. +- [ ] 6.5 `openspec validate add-omlx-provider` — the CLI is not installed in this environment; artifacts were hand-written to the archived change's shape.