Skip to content

feat: add oMLX as a provider and a serve engine - #24

Merged
outofcoffee merged 2 commits into
mainfrom
claude/omlx-provider-support-7h0com
Aug 1, 2026
Merged

outofcoffee merged 2 commits into
mainfrom
claude/omlx-provider-support-7h0com

Conversation

@outofcoffee

@outofcoffee outofcoffee commented Aug 1, 2026 •

Copy link
Copy Markdown
Collaborator

Adds oMLX, a local inference server for Apple Silicon, both as a catalogue provider and as an engine outfit serve can launch.

The Pi option-resolution fix that was originally part of this branch now lives in #25, stacked on top of this one.

Summary

  • Add an omlx provider to the catalogue: OpenAI-compatible at http://localhost:8000/v1, overridable with OMLX_BASE_URL, keyless on localhost and key-bearing when pointed elsewhere (apiKeyOptional, as llamacpp already is), and usable from both harnesses.
  • Breaking: outfit serve now picks its engine from PROVIDER — llamacpp runs llama-server, omlx runs the oMLX CLI, and any other provider is a loud error. It previously ran llama-server whatever the Outfit said, so PROVIDER ollama silently started the wrong server.
  • Teach internal/preset that flag spelling is per-engine, via a Dialect carrying the alias and boolean tables. Parsing itself stays shared.
  • Look for the oMLX CLI on PATH and then in its macOS app bundle, with a per-engine install hint on failure.
  • Add a worked examples/omlx/qwen3.6/ guide, update the docs, and record an OpenSpec change with local-serving and provider-catalog deltas.

Implementation details

PROVIDER selects the engine. This settles an inconsistency the code already claimed was settled: runnerFor documents itself as "PROVIDER already names the engine — outfit serve starts that engine locally", and the CLI usage says the same, but cmdServe never read sel.Provider. engineFor mirrors runnerFor deliberately — a hard-coded switch rather than a catalogue field, so a user-supplied --providers file cannot name an arbitrary binary for serve to execute. The alternatives (a new ENGINE keyword, a --engine flag) would each add a second source of truth for a question PROVIDER already answers.

Preset dialects. internal/preset split cleanly: Parse, Select and FormatCommand never look at key names, while canonical, boolean and flagFor are entirely llama.cpp's. So the engine-specific part becomes a value rather than a fork of the package, and the zero Dialect is exactly right for oMLX, whose flags are all long-form. The package-level Flags/CanonicalKey/Preset.Args/Preset.Command are kept as LlamaCpp wrappers — not for politeness, but because remote.go's isCloudOwned compares preset keys through CanonicalKey to decide which flags the cloud owns. The trap this closes: an oMLX preset read in llama.cpp's dialect parses without error and produces a wrong command, rewriting m to --model and dropping a falsy boolean. The dialect therefore comes from the engine PROVIDER names, never from inspecting the file.

What an Outfit means for oMLX. oMLX loads a whole --model-dir and dispatches per request, so only BASEURL becomes a launch flag; MODEL and ALIAS keep their usual meaning as the id the harness requests, which is load-bearing here rather than a label. It follows that oMLX starts with neither a PRESET nor a MODEL, so llama.cpp's "needs one of them" rule is gated behind a needsModel flag.

serve never passes an API key. It prints the command it runs, and oMLX takes --api-key on the command line, so a resolved key would land on the terminal and in the process table. Auth on the server is configured in oMLX; the catalogue path still writes an {env:…} reference for the agent.

Serve moves to its own file. Partly cohesion, partly a constraint: TestCompletionCoversDispatch scans main.go for case "…": at one tab of indentation to prove every dispatched command is completable, and engineFor's switch matches that shape.

outfit remote deploy is deliberately untouched — the cloud side is CUDA/NVIDIA EC2 and oMLX is Apple Silicon, so runnerFor keeps rejecting it.

One known gap while these two PRs are separate: this branch's docs say OMLX_BASE_URL redirects the endpoint, which holds for opencode but not for the Pi harness until #25 lands.

oMLX (https://omlx.ai) is a local inference server for Apple Silicon built on
Apple's MLX framework, exposing an OpenAI-compatible API. Add it to the
catalogue so a harness can be pointed at it, and teach `outfit serve` to launch
it, so Mac users get the same two-step flow llama.cpp users already have.

Serve now picks its engine from PROVIDER, which settles an inconsistency the
code already claimed was settled: runnerFor documents itself as "PROVIDER
already names the engine — outfit serve starts that engine locally", and the
usage text says the same, but cmdServe never read PROVIDER and unconditionally
ran llama-server. An Outfit naming any other provider quietly started the wrong
server; that is now a loud error naming the providers that can be served.

Preset parsing is shared but flag *spelling* is not, so internal/preset grows a
Dialect carrying the alias and boolean tables. LlamaCpp keeps the existing
tables and remains what the package-level helpers render, so remote.go's
isCloudOwned — which compares preset keys through CanonicalKey — is unchanged.
OMLX is the zero Dialect: every omlx-cli flag is long-form, and llama.cpp's
table would silently rewrite `m` to --model and `c` to --ctx-size.

oMLX differs from llama.cpp in loading a whole --model-dir and selecting per
request, so it starts with neither a PRESET nor a MODEL, and only BASEURL
becomes a launch flag. MODEL and ALIAS keep their usual meaning as the id the
harness asks for — load-bearing here rather than a label.

serve never passes --api-key. It prints the command it runs, so a resolved
secret would land on screen and in the process table.

The serve command moves to its own file. Beyond cohesion, this is required:
TestCompletionCoversDispatch scans main.go for `case "…":` at one tab, so
engineFor's switch there reads as two new commands.

remote deploy is deliberately untouched — the cloud side is CUDA/NVIDIA EC2 and
oMLX is Apple Silicon, so runnerFor keeps rejecting it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CipwSCMUZRE61zWWiYEcDx
@outofcoffee outofcoffee added the enhancement New feature or request label Aug 1, 2026
@outofcoffee
outofcoffee force-pushed the claude/omlx-provider-support-7h0com branch from 8af1f07 to a85f4a6 Compare August 1, 2026 01:18
Real testing surfaced two things the oMLX docs got wrong. oMLX can require an
API key even on localhost (an admin-panel setting), yet the docs claimed a
local server never needs one; and because the provider is apiKeyOptional,
outfit only writes the key reference when OPENAI_API_KEY is set at apply time,
so setting it afterwards silently does nothing. Spell both out, in the example
and in serve.md.

Also make the example's MODEL substitution unmissable: oMLX serves by the
model's local directory name, so a copied Outfit fails with "model not found"
unless the id matches what you downloaded.

Add a Gemma-4-E2B example alongside the Qwen3.6 one — a small model that loads
in seconds, a gentler first run than the 27B.
@outofcoffee
outofcoffee merged commit cfb55c7 into main Aug 1, 2026
1 check passed
@outofcoffee
outofcoffee deleted the claude/omlx-provider-support-7h0com branch August 1, 2026 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants