Repository navigation
feat: run the model on a cloud GPU with outfit remote - #20
Merged
Merged
Conversation
outfit remote start|stop|status drives the scale-to-zero inference instance from the cloud-vm-llm stack, via SigV4-signed calls to its Lambda Function URLs using the standard AWS credential chain.
An Outfit can name its remote-instance config with `REMOTE <file>`, resolved relative to the Outfit like PRESET. outfit remote start|stop|status prefer it over the per-user remote.json, so a project can carry its own endpoint config.
PROVIDER carries the runner, so a vLLM deployment needs its own entry rather than borrowing openai-compatible — that keeps the llama.cpp/vLLM cut-back symmetric and lets the provider name say which runner is meant.
PROVIDER already names the engine, so an Outfit that says llamacpp asks for llama.cpp — locally under `outfit serve`, in the cloud under `outfit remote deploy`. The Outfit and its preset are the single source of truth: MODEL or the preset's hf gives the weights as repo:QUANT, CONTEXT or ctx-size the window, ALIAS the served name, and the preset's remaining flags become the runner's serveArgs. Flags the cloud sets itself are dropped, matched on canonical names so short aliases count, and the preset's [*] globals are layered under the section so settings written there (ngl, jinja) survive. Deploy is the first call with a body; SigV4 covers the payload hash, so the bytes are hashed and signed rather than assumed empty.
PROVIDER llamacpp now names the remote deployment as well as a local server, and the remote one is started with --api-key — but the catalogue entry had no apiKeyEnv, so applying such an Outfit produced a keyless config that 401s. Adding the key env alone would have regressed local use under Pi, which writes a $VAR reference resolved at run time: a local server needs no key, so the reference would resolve to nothing and Pi hides models with unusable auth. apiKeyOptional marks that case, keeping Pi's keyless placeholder when the var is unset while every other provider keeps its reference.
The docs directory is a user manual, so the remote command needed a page of its own alongside the others, and the Outfit guide's REMOTE example still showed the shape from before PROVIDER selected the engine. The OpenSpec change records the specification delta this branch introduced: a new remote-endpoint capability, and the requirement changes it forced on outfit files (the REMOTE keyword), the provider catalogue (an optional API key), Pi integration (when the keyless placeholder applies), and shell completion (subcommands).
The start endpoint blocks until the model is serving, so the only progress callback — the one for a 503 retry — never fired on the path that actually takes minutes: a cold start sat silent and looked hung. It now says what it is doing up front and repeats with the elapsed time while it waits. Progress goes to stderr and stdout carries only the exports, so the output stays pipeable and `eval "$(outfit remote start)"` needs no filtering.
Neither harness needs the secret in its config: opencode substitutes
{env:VAR} and Pi resolves $VAR, both when they run. Writing the resolved
key into opencode's config was justified by a global config being unable
to rely on a project-local .env — so the agent outfit launches is now
given the keys outfit can resolve, and that justification goes with it.
A .env is read beside the Outfit being applied, the same rule PRESET and
REMOTE follow, falling back to the working directory for a selection
made from flags alone. It previously resolved through runtime.Caller,
which records the path of the machine that built the binary, so an
installed outfit could never find one.
Applying a config for a remote endpoint with no resolvable key used to
succeed and then fail on the first request; it now says so, naming the
variable. A provider whose key is optional keeps Pi's keyless
placeholder only for a local endpoint, since a placeholder written for a
remote one could not be repaired by setting the variable afterwards.
The deployment outfit remote drives lived in a separate private repository, which meant the command and the thing it commands were versioned, reviewed and released apart. It is now a first-class part of this project: a TypeScript CDK subproject under remote/, with its own tests run by CI. This repository is public, unlike the one it came from, so nothing identifying a deployment may be committed. scripts/check-no-cloud-identifiers.sh fails the build on an account id, ARN, Function URL host, public address, bucket name or resource id, and runs before anything else in CI. Deployment state stays in the gitignored files under remote/. Logical stack names, tag keys and SSM paths are kept as they are: they are chosen in source and identify the deployed resources, so changing them would orphan the running stack.
The CDK project's assertions, typecheck and a full synth now run on their own, triggered only when remote/ changes rather than on every Go pull request. The synth covers what the assertions cannot — bin/app.ts wiring and esbuild bundling each Lambda — and needs no credentials, since the stacks take their availability zones from config rather than looking them up. The change's OpenSpec artifacts are brought back in line with what the branch became: start reporting progress, the missing-key warning, keys referenced rather than embedded, and the deployment moving into this repository under the public-repo identifier rule.
outofcoffee
marked this pull request as ready for review
July 29, 2026 01:34
The endpoint, its engines and its weights were described only by the code that implements them, so the rules that matter most — when an instance is torn down, what bounds a running one, where weights are looked for — were whatever the tests happened to assert. They are now requirements: an idle policy that could be changed plausibly and wrongly is the difference between minutes and hours of GPU time. The change's deltas are applied to the main specs and the change is archived; four capabilities are new, the rest gain the requirements this work altered.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Run a model too big for your laptop on a cloud GPU that only exists while you are using it, driven by the same
Outfitfile you would use locally.Summary
outfit remotecommand group —start,stop,statusanddeploy— targeting the scale-to-zero endpoint the companioncloud-vm-llmproject deploys. It drives that endpoint; it does not create it.REMOTEOutfit instruction naming the file holding the endpoint's control URLs, resolved relative to the Outfit likePRESET, so a project can carry its own endpoint. Without one, a per-user config is used, so the commands still work outside a project.outfit remote deployderives what to serve from the Outfit and its preset:PROVIDERpicks the engine,MODELor the preset'shfnames the weights,CONTEXTthe window,ALIASthe served name, and the preset's remaining flags the engine's arguments.vllmprovider to the catalogue, so both self-hostable engines can be named byPROVIDERand switching between them is a one-line edit.apiKeyOptionalcatalogue flag for a provider that works both unauthenticated and authenticated, and marksllamacppwith it — without which applying a remote llama.cpp Outfit produced a keyless config that 401s.outfit remote <TAB>offers its subcommands.add-remote-endpoint-control.Additive throughout: an Outfit without
REMOTEbehaves exactly as before, and no existing command changes behaviour.Implementation details
PROVIDERnames the engine; there is deliberately noRUNNERkeyword. An Outfit already saysllamacpp, andoutfit servealready runs llama.cpp from it — the remote case is the same statement pointed at another machine. A separate keyword would allowPROVIDER llamacppalongsideRUNNER vllm, a contradiction worth making unrepresentable. The cost is that a hosted provider cannot be deployed, which is reported as an error rather than left as a gap.The deploy payload describes intent, not storage. It carries the engine, model, quantisation, context, served name and flags — and deliberately no storage location. The endpoint derives its own layout and fetches missing weights itself, so a change to that layout never becomes an outfit release.
One preset serves locally and deploys unchanged. Rather than a second list of remote flags, the existing preset is reused with the settings the endpoint owns (host, port, model location, API key, context, alias, metrics) removed. Two subtleties this forced: flags must be matched after alias resolution, since
nglandn-gpu-layersare the same setting, and a preset's[*]defaults are a separate layer that must be merged — missing it silently droppednglandjinja, which would have served the model on CPU with no tool calling.internal/remoteis the only package that reaches the network or touches AWS. It brings the first non-stdlib runtime dependency outside YAML parsing, and is reached only by this command group; every other command still runs offline with no credentials. Requests are signed with the caller's own AWS chain and need onlylambda:InvokeFunctionUrl.deployis the first request with a body, so the signature covers the payload hash andContent-Lengthcomputed from the exact bytes sent.deploy_urlis optional. A config written beforedeployexisted still works for the other three subcommands; onlydeploycomplains, and it says what to add.apiKeyOptionalexists because adding the key variable alone regressed local use. The Pi harness writes a$VARreference it resolves at run time, so a keyless local server would have received a reference to an unset variable — which makes Pi hide the provider's models. Falling back to the placeholder globally was not an option either, since for other providers that reference is deliberately resolved later. The flag marks the one case where an unset variable genuinely means "no key".Also in this PR
remote/: the CDK application thatoutfit remotedrives — an endpoint holding no instance until asked for one, launching into whichever zone has GPU capacity, and terminating itself once unused. Weights live in object storage at a location derived from engine, model and quantisation, fetched automatically when absent.startreports progress. The endpoint blocks until the model is serving, so a cold start used to sit silent for minutes. Progress goes to stderr and only the exports to stdout, soeval "$(outfit remote start)"needs no filtering.{env:VAR}reference rather than the secret, matching how Pi has always worked, andoutfit harnesspasses the keys it can resolve to the agent it launches. A.envis read beside the Outfit being applied — it previously resolved through a build-machine path an installed binary could never find.scripts/check-no-cloud-identifiers.shfails the build on account ids, ARNs, endpoint hostnames, public addresses, bucket names and resource ids, and runs first in CI.Remote deploymentworkflow typechecks, tests and synthesizesremote/when it changes.remote-endpoint,endpoint-lifecycle,inference-runners,model-weights) plus the requirements this work altered across seven existing ones.