Repository navigation
feat: add oMLX as a provider and a serve engine - #24
Merged
Merged
Conversation
oMLX (https://omlx.ai) is a local inference server for Apple Silicon built on Apple's MLX framework, exposing an OpenAI-compatible API. Add it to the catalogue so a harness can be pointed at it, and teach `outfit serve` to launch it, so Mac users get the same two-step flow llama.cpp users already have. Serve now picks its engine from PROVIDER, which settles an inconsistency the code already claimed was settled: runnerFor documents itself as "PROVIDER already names the engine — outfit serve starts that engine locally", and the usage text says the same, but cmdServe never read PROVIDER and unconditionally ran llama-server. An Outfit naming any other provider quietly started the wrong server; that is now a loud error naming the providers that can be served. Preset parsing is shared but flag *spelling* is not, so internal/preset grows a Dialect carrying the alias and boolean tables. LlamaCpp keeps the existing tables and remains what the package-level helpers render, so remote.go's isCloudOwned — which compares preset keys through CanonicalKey — is unchanged. OMLX is the zero Dialect: every omlx-cli flag is long-form, and llama.cpp's table would silently rewrite `m` to --model and `c` to --ctx-size. oMLX differs from llama.cpp in loading a whole --model-dir and selecting per request, so it starts with neither a PRESET nor a MODEL, and only BASEURL becomes a launch flag. MODEL and ALIAS keep their usual meaning as the id the harness asks for — load-bearing here rather than a label. serve never passes --api-key. It prints the command it runs, so a resolved secret would land on screen and in the process table. The serve command moves to its own file. Beyond cohesion, this is required: TestCompletionCoversDispatch scans main.go for `case "…":` at one tab, so engineFor's switch there reads as two new commands. remote deploy is deliberately untouched — the cloud side is CUDA/NVIDIA EC2 and oMLX is Apple Silicon, so runnerFor keeps rejecting it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CipwSCMUZRE61zWWiYEcDx
outofcoffee
force-pushed
the
claude/omlx-provider-support-7h0com
branch
from
August 1, 2026 01:18
8af1f07 to
a85f4a6
Compare
Real testing surfaced two things the oMLX docs got wrong. oMLX can require an API key even on localhost (an admin-panel setting), yet the docs claimed a local server never needs one; and because the provider is apiKeyOptional, outfit only writes the key reference when OPENAI_API_KEY is set at apply time, so setting it afterwards silently does nothing. Spell both out, in the example and in serve.md. Also make the example's MODEL substitution unmissable: oMLX serves by the model's local directory name, so a copied Outfit fails with "model not found" unless the id matches what you downloaded. Add a Gemma-4-E2B example alongside the Qwen3.6 one — a small model that loads in seconds, a gentler first run than the 27B.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds oMLX, a local inference server for Apple Silicon, both as a catalogue provider and as an engine
outfit servecan launch.Summary
omlxprovider to the catalogue: OpenAI-compatible athttp://localhost:8000/v1, overridable withOMLX_BASE_URL, keyless on localhost and key-bearing when pointed elsewhere (apiKeyOptional, asllamacppalready is), and usable from both harnesses.outfit servenow picks its engine fromPROVIDER—llamacpprunsllama-server,omlxruns the oMLX CLI, and any other provider is a loud error. It previously ranllama-serverwhatever the Outfit said, soPROVIDER ollamasilently started the wrong server.internal/presetthat flag spelling is per-engine, via aDialectcarrying the alias and boolean tables. Parsing itself stays shared.PATHand then in its macOS app bundle, with a per-engine install hint on failure.examples/omlx/qwen3.6/guide, update the docs, and record an OpenSpec change withlocal-servingandprovider-catalogdeltas.Implementation details
PROVIDERselects the engine. This settles an inconsistency the code already claimed was settled:runnerFordocuments itself as "PROVIDER already names the engine —outfit servestarts that engine locally", and the CLI usage says the same, butcmdServenever readsel.Provider.engineFormirrorsrunnerFordeliberately — a hard-coded switch rather than a catalogue field, so a user-supplied--providersfile cannot name an arbitrary binary forserveto execute. The alternatives (a newENGINEkeyword, a--engineflag) would each add a second source of truth for a questionPROVIDERalready answers.Preset dialects.
internal/presetsplit cleanly:Parse,SelectandFormatCommandnever look at key names, whilecanonical,booleanandflagForare entirely llama.cpp's. So the engine-specific part becomes a value rather than a fork of the package, and the zeroDialectis exactly right for oMLX, whose flags are all long-form. The package-levelFlags/CanonicalKey/Preset.Args/Preset.Commandare kept asLlamaCppwrappers — not for politeness, but becauseremote.go'sisCloudOwnedcompares preset keys throughCanonicalKeyto decide which flags the cloud owns. The trap this closes: an oMLX preset read in llama.cpp's dialect parses without error and produces a wrong command, rewritingmto--modeland dropping a falsy boolean. The dialect therefore comes from the enginePROVIDERnames, never from inspecting the file.What an Outfit means for oMLX. oMLX loads a whole
--model-dirand dispatches per request, so onlyBASEURLbecomes a launch flag;MODELandALIASkeep their usual meaning as the id the harness requests, which is load-bearing here rather than a label. It follows that oMLX starts with neither aPRESETnor aMODEL, so llama.cpp's "needs one of them" rule is gated behind aneedsModelflag.servenever passes an API key. It prints the command it runs, and oMLX takes--api-keyon the command line, so a resolved key would land on the terminal and in the process table. Auth on the server is configured in oMLX; the catalogue path still writes an{env:…}reference for the agent.Serve moves to its own file. Partly cohesion, partly a constraint:
TestCompletionCoversDispatchscansmain.goforcase "…":at one tab of indentation to prove every dispatched command is completable, andengineFor's switch matches that shape.outfit remote deployis deliberately untouched — the cloud side is CUDA/NVIDIA EC2 and oMLX is Apple Silicon, sorunnerForkeeps rejecting it.One known gap while these two PRs are separate: this branch's docs say
OMLX_BASE_URLredirects the endpoint, which holds for opencode but not for the Pi harness until #25 lands.