This is maintainer reference, not a user guide (see the documentation
index for that) and not a behavior spec (see openspec/specs/
for that — the
requirements below are already covered there). It's the gotchas and
cross-cutting rationale worth knowing before you touch the code, that don't
fit either of those.
These are mistakes already made here; each was silent rather than loud, which is what makes them worth writing down.
The palette's two colour groups are not swappable. The brand colours (brandAccent, brandInk, brandInkDim) carry the same values as the web repo's --accent, --ink and --ink-2 tokens, written as hex and downsampled by lipgloss, and the accent is used only where nothing about a node is being reported — the title bar's product name and the selected panel's border. The state colours (ansiGreen and friends) are raw ANSI and report what an engine is doing: the resource bars and the health glyph draw from them. spinnerFrames is one cycle for the whole tool, shared by fleet deploy's progress lines and the dashboard's in-flight tiles.
The catalogue is embedded at build time. providers.yaml is //go:embed-ed, so a previously-built spinloop binary keeps applying the old catalogue no matter what the file says. Rebuild before testing any catalogue change, or you will "verify" a fix that is not in the binary you ran. (--providers/SPINLOOP_PROVIDERS reads a file at run time and sidesteps this.)
A preset section is not the whole preset. Preset.Select returns only the named [section]; the [*] defaults live separately in Preset.Global. Anything that consumes a section's Params directly — rather than going through Args/Command, which layer both — silently drops whatever the user put in [*]. That is usually the settings they consider obvious enough to write once, like ngl and jinja, so the failure surfaces later as a model running on CPU or refusing tool calls.
Match preset flags by canonical name. Preset keys have short aliases (ngl is n-gpu-layers, c is ctx-size, hf is hf-repo, a is alias). Comparing raw keys against a list of flag names therefore matches only the spelling you happened to think of. Use preset.CanonicalKey, which is exported for exactly this; Flags already dedupes layers by canonical name, last layer winning.
A preset dialect is not interchangeable. internal/preset parses any INI the same way, so an oMLX preset fed through llama.cpp's dialect parses cleanly and produces a wrong command: the alias table rewrites m to --model and c to --ctx-size, and the boolean table drops a key = 0 entirely. Nothing errors — the server just receives flags it does not accept, or silently loses a setting. The dialect always comes from the engine PROVIDER names, never from the file.
A busy engine does not answer its own metrics endpoint. llama.cpp serves /metrics from the same queue it serves inference from, so a scrape taken while a prompt is being processed waits for that prompt to finish — tens of seconds on a long context. /v1/metrics used to scrape inline, so the handler blocked for as long as the engine had work, and spinloop metrics (5s client timeout, against a handler whose scrape timeout was also 5s) could not win that race: the view went blank exactly when there was something worth watching, reporting the node as unreachable. The counters now come from the background sampler's last reading (engineSample), which the daemon takes every 15s regardless of who is asking — so the handler never waits on the engine, and staleness is bounded by the sample interval. Three things to preserve if you touch this: the cached scrape error is still reported, because silently omitting the token block is what once hid a scraper pointed at the wrong port; the sample is forgotten on start, so one engine's counters are never reported against the next; and the sampler retries at catchUpInterval (1s) until a reading lands, dropping to the full interval only afterwards — without that, a freshly started engine reports no counters for up to 15s, which is exactly the window someone watching a node they just started is looking at.
An exported-but-empty variable is a gap, not a choice. setEnvIfAbsent keys on the variable being present, so an OPENAI_BASE_URL= in the environment counts as set and suppresses the value routing meant to supply — leaving the agent pointed at nothing. setEnvIfBlank is the one to use for an address or a key, matching what harnessEnv already does for the cloud endpoint's values; the distinction only shows up when something exports an empty string, which shells do more often than you would think.
A pre-warm that races the engine's faults cannot win. The cloud's model load is I/O-bound and the engine arrives first: it maps its weights and faults pages in as it copies them to the GPU, so from the first second of a load the volume serves its per-page faults, and a provisioned gp3 root hands out at most 4,000 IOPS × ~68 KB ≈ 260 MB/s whatever else wants the disk. A sequential pre-warm reader that starts with the engine spends its first seconds ahead of the faults, then goes flat — the daemon's read_bytes did exactly that on a live instance (~1.5 GB of real EBS reads in the first seconds, then nothing for the whole load), because the engine's faults consume the budget and its readahead turns every later "read" into a cache hit. And the shape's 32 GB of RAM cannot hold a ~30 GB model in the page cache at all, so even a finished pre-warm would only shift the faults to a different second. Live-checked 2026-08-23: pre-warm on cost double-reads and saved no time (the ~30 GB model loaded in ~115 s either way). The feature was removed after that check; the provisioned gp3 throughput and IOPS stay, because the S3 sync is the one reader whose limit is the volume's.
contextsize.Parse is decimal. 128k is 128000, not 131072 — a CONTEXT written that way is not the power-of-two window it looks like. It also overrides a preset's ctx-size (both in serve and in cloud deploy), so the Spinloop, not the preset, decides the window whenever it states one.
up dispatches by directory and reuses both branches. cmd/spinloop/up.go routes a working-directory fleet.yaml to the fleet start path — runFleetDrive over the named nodes, or over every node when none are given, since a bare fleet start lists and does nothing — and everything else to runServe's own body, so up and serve resolve and word things identically by construction. The completion slot is the only CWD-dependent one: upSlot offers the fleet's node names where ./fleet.yaml parses, the Spinloop slot elsewhere, and nothing where a fleet file is present but unreadable — __complete never errors, whatever the directory holds.
A freshly issued access key is not instantly resolvable. STS lags the iam:CreateAccessKey response by seconds — on one account the store's verification needed six attempts before the key resolved, at roughly 8–10 s propagation. The verify probe in cmd/spinloop/cloud_auth.go (verifyNewKey) retries only when the error message contains InvalidClientTokenId, on an exponential backoff from 1 s to a 16 s cap (a ~31 s window over six attempts), and the stderr line says which retry is running. Two things to preserve: the match is a string match because the pinned service/sts SDK version has no typed error for that code — if the SDK is upgraded, switch to the typed check — and the retry is exclusive to that code. Any other verification failure, a key that resolves to a different account or lacks a permission, fails at once and deletes the key it created; waiting it out would only delay the deletion. verifyProbeBackoff is a seam the tests zero.
An IAM user's inline policies are capped at 2,048 characters in aggregate. The control plane's eight functions each take a grantInvokeUrl pair — two actions, the auth-type conditions, the function's ARN — and with the log-reading, stack-discovery, pricing and self-service statements the document far exceeds that; the first deploy of the RemoteCliUser inline policy failed with ServiceLimitExceeded, which CDK does not warn about ahead of time. It is now a stack-owned AWS::IAM::ManagedPolicy (RemoteCliPolicy, 6,144 cap; the deployed document measures ~2 KB, so the grant list has room to grow). Keep it managed rather than re-inlining it, and keep the iam self-service ARN built from the AWS::Partition/AWS::AccountId pseudo parameters instead of the user's Arn: the policy attaches to that user, so referencing the user from inside it is a dependency cycle.
Schedules live in SSM, and the EventBridge Scheduler schedules are a copy of that list. The schedule Lambda writes an environment's list to /cloud-vm-llm/<env>/schedules, then creates, updates and deletes Scheduler schedules in the stack's schedule group until they match it. Everything that can be refused (cron, zone, action) is checked before the write, so a bad request changes nothing; a Scheduler failure after the write leaves the stored list as intended and a repeat of the same request converges. Scheduler schedule names are e-<16 hex of sha256(env)>-<n>, not the environment name: names are capped at 64 characters, an environment name can be that long alone, and the prefix has to be exact so one environment's reconcile never lists another's. The environment travels in each schedule's JSON input instead. Users write five-field cron; Scheduler wants six fields, ? in the unused day field, and its own weekday numbering (Sunday is 1), so toSchedulerCron expands weekdays to names (MON,TUE) and refuses an expression that sets both day fields, which classic cron would treat as "either" and Scheduler cannot express. "Next run" is computed by nextRun in lambda/shared/schedules.ts, not read from Scheduler: it walks calendar days in the schedule's zone, skips a wall-clock time the clocks jump over, and takes the first of a time they repeat. The start and stop Lambdas tell a schedule firing from the idle sweep and from a Function URL call by the event's source (spinloop.schedule, aws.events, none); keep the check for the schedule's source ahead of the others.
A start holds a per-environment lock for its whole run, and it is an SSM parameter, not a conditional write. wake() in the start Lambda takes /cloud-vm-llm/<env>/wake-lock (created with Overwrite: false) after the read-only checks and before the weights check, and releases it in a finally. It has to cover the whole poll and not only the launch, because the tag lookup that decides whether to launch lags a launch: a second start that got in after the lock was released could still miss the first one's instance. The value is {owner, expiresAt} with the expiry set to the invocation's remaining time plus 30 seconds, so a start killed before its finally blocks the environment for no longer than it could have run. A refused start replies 503 starting with 15 seconds to retry, which the CLI and gateway already retry. Taking over an expired lock is read, re-read, delete, create-if-absent; SSM has no compare-and-set, so two starts taking over the same abandoned lock within milliseconds can still both proceed. That needs a killed start plus a near-simultaneous pair, and a DynamoDB conditional write would remove it. Releasing needs ssm:DeleteParameter, granted on parameter/cloud-vm-llm/*/wake-lock only. Stops do not take the lock: a lock held for up to 15 minutes would refuse the stop that gets you out of a hung start, and the stop Lambda's 120 second limit could not wait for it. Instead each polling loop in a start checks the instance state and ends the start, releasing the lock, if the instance is stopped, stopping or terminated. The status read reports the lock without taking it: starting when the lock is held and the instance is absent, stopped or pending, and start_in_progress in every reply. A start waiting for capacity has released its lock between attempts, so it is not shown.
The file credential store's index is non-secret by design. OS keystores offer no way to list entries, so the file store — used where no keystore is reachable or SPINLOOP_CLOUD_KEYSTORE=file — keeps a plain-text index of the stored regions beside the 0600 per-region files under <config>/keystore/. The report (spinloop cloud auth) reads the index, so a file added, removed or renamed by hand is reported wrong until the index matches; and a corrupt index is reported, not silently reset, because a report that misleads about what is stored misleads about which access keys exist on the AWS side.
The two local model caches are separate. A model llama-server downloaded sits in llama.cpp's cache ($LLAMA_CACHE, else the platform's user cache directory) as flat filenames; one fetched with the hub's tools sits in the Hugging Face hub cache ($HF_HUB_CACHE, else $HF_HOME/hub, else ~/.cache/huggingface/hub) as models--<owner>--<name>/snapshots/<sha>/ of symlinks into a content-addressed blob store. Neither tool looks in the other's, so a model already on the machine is "already on the machine" in only one of the two senses. spinloop hf therefore resolves both roots up front (hf.ResolveRoots) and checks both before touching the network; a cache-aware lookup that consults only one side re-downloads what is already there. The hub-cache shape has two more traps: a snapshot entry whose symlink dangles is an interrupted or half-finished download and must count as absent, and refs/<revision> holds a commit sha, so a revision name is only a snapshot once it has been resolved through refs/.
opencode run takes its working directory from the PWD variable, not its own cwd. opencode run with no directory flag sets the session's directory — and with it the directory every tool the agent runs — from process.env.PWD where that variable is set, falling back to the process's cwd only where it is absent (as of v1.18.30). The dispatcher gave the child the item's directory with cmd.Dir, but the environment it inherited carried the PWD of wherever the orchestrator was started, so the session and the agent's tools ran in the orchestrator's directory while the process's cwd sat in the item's: an item's dir: was silently ignored, and nothing failed. startChild therefore rewrites the child's PWD to the item's directory (resolved with filepath.Abs) before the exec, on top of cmd.Dir carrying it. The variable then holds the unresolved path — the /var/... form on macOS — which is why the stub agent's record takes the physical directory with pwd -P and the variable's own value with printenv PWD.
internal/daemon must not depend on a cloud package. What an engine should serve is inference.DeployConfig, in internal/inference — a leaf that imports only the standard library. Both the daemon and the cloud control plane speak it, so neither has to import the other to be described: a daemon is handed one over its control API, and internal/cloud persists one against an environment. The dependency used to run the other way, internal/daemon importing internal/cloud for the type and for a config-directory helper that only forwarded to internal/config.Dir, which made the package that knows nothing about AWS depend on the package that is nothing but AWS. Keep new shared vocabulary in internal/inference only where it describes an engine's workload and needs nothing of ours to express; anything cloud-shaped — IsInstanceType and the rest of the EC2 vocabulary — stays in internal/cloud.
A captured engine's stdout is a pseudo-terminal, and the normaliser above it is a line model, not a screen emulator. llama.cpp's download bar prints nothing when its stdout is not a terminal, and the download happens in-process before the HTTP listener comes up — there is no status API to scrape — so the capture (the daemon and the serve view) presents the engine's stdout as a PTY: the only way the progress reaches the engine log at all. ptylog.go turns the terminal stream into the log's lines as a column, and three of its rules are easy to break by "simplifying". The engine's up-down cursor dance is part of each redraw — the bar's state is drawn between a cursor-up and a cursor-back-down, the cursor parked on an anchor line — so a cursor move must move the drawing between the column's lines and never end a line: committing on a move records every state as a "final" one and resets the dedup, which is how the first design put the whole download, state for state, in the log. A state's replacement is lazy — a carriage return settles the state drawn before it, and the line's content is cleared only by the next text byte — so S\r\r\n, the PTY's ONLCR turning a lone \n into a CRLF, commits S intact. And the tick at the interval's half-rate runs on every line of the column, because a bar's last state outlives the engine's move off its line: the download finishes, the engine goes quiet on stdout, and the log still owes it. Because the engine's stdout is now a terminal, the capture branch sets NO_COLOR=1 in the engine's environment: llama.cpp routes its log lines to stderr and colours them by whether it sees a terminal — stdout among them — and the normaliser never sees the stderr path, so without it escapes would reach the log file. The bar draws no colour, so nothing is lost; the forwarding paths (piped serve, off-terminal serve --api) set nothing and must stay byte-for-byte what the engine wrote — a test pins the raw \r surviving. Where no pseudo-terminal can be opened — Windows, a restricted environment — the fallback writes stdout to the log file directly, no worse than the capture did before, and the engine is told NO_COLOR the same way; a test stands the opening in for with a failure to keep that branch alive. The pump always drains — the throttle delays the log's writes, never the reads, so the engine can never wedge on a full terminal — and the log file closes only after the pump's final record, never under it. The master is not closed when the engine exits: closing it discards bytes the engine wrote just before exiting that the pump has not read yet (on Linux the read returns EIO once every slave fd is closed, and an early close loses the tail — the cause of the TestSupervisorPTYCapture flake on CI). The supervisor waits for the pump to reach end of stream, and closes the master itself only after ptyDrainTimeout, which covers a child of the engine that keeps the slave open.
The generation speed is read where the capture already assembles lines, and the wiring order is the whole feature. llama.cpp prints its instantaneous rate into stderr's print_timing lines; the supervisor wraps its log-file sink in a lineTee (tee.go) that writes every byte to the file untouched and hands each complete line to Supervisor.OnOutput, so both capture paths — stderr straight through, pty stdout through the normaliser — feed the parser with no second reader of the log. OnOutput is read as Supervisor.Start builds the sink, so a daemon must set it before starting: wired afterwards, the tee is simply never there and every line is silently lost (this was the first implementation's bug). The parser reads tg = <n> t/s only — tg\s*= cannot match tg_3s =, so the three-second average and prompt rates never read as the current rate — and a tracker with two clocks splits the answer: the sampler's tick folds a history point for the figures its interval saw (strictly after the previous tick: a point is an observation, never a continuation), while a metrics read reports the figure only until one interval of quiet ages it out. Both start engines reset tracker, baseline and history together, so one engine's rate never rides the next engine's report.
Calling prog.Send from inside Update deadlocks the program. The event loop reads its own channel on the goroutine that calls Update, so a command that hands a message back through prog.Send while Update is on the stack waits for a reader that is waiting for it. The board hung on exactly this: pressing Enter on the form's last field, or a/x on a card, and the whole TUI went still — no spinner, no keystrokes, Ctrl-C ignored. Every message from work that starts inside Update must return through the loop instead: tea.Batch(workCmd, spinCmd) delivers each command's messages normally, and a repaint chain re-arms through its own returned tea.Tick. The tests guard this by landing commands through the real program loop, not by calling Update and discarding the command.
A few Bubble Tea/lipgloss specifics that are easy to break by "simplifying":
- The
tea.Programholds the model by pointer: Bubble Tea never reads a value model'sInitback, so the first round's mutations (its deadline spend — real cloud calls, for cloud environments) would be silently discarded. - The tick reschedules itself whenever it fires; a one-shot
tea.Tickwithout the reschedule leaves the board still after the second round. A second, faster chain (dashSpinTickMsg) runs only while an action is in flight, so the spinner and the elapsed time beside a verb advance; it stops on the first tick that finds nothing in flight. - Every reading carries the time its own call returned (
fleet.NodeResult.At, set in the fan-out), and the board draws a reading only when it was taken later than the one already on screen. Reads run concurrently and take differing times, so a reading can land after one taken later than it — including a round issued before an action finished and landing after it, which would otherwise repaint the node's pre-action state. - A start reports a
fleet.StartPhase— what it is doing, when that began, when the next attempt is due — rather than a line of text, andfleet.RenderPhase(phase, now)is the only place it becomes text. A wait therefore counts down and a boot counts up on every repaint, and a situation the start has moved on from cannot be left on the tile: each phase replaces the one before it.spinloop cloud startrenders the same phases to stderr, so the tile and the CLI cannot word one situation differently. - One function,
dashNodeView, produces both a panel's lines and its health tier, from the reading, the action, the current time, and how old a reading of that node may be. Nothing in it reads a clock, so every pairing of a start's phase against a reading can be enumerated in a test. - A tile's first line is a header bar drawn in raw ANSI — the body is one plain string under a single lipgloss style, so per-character colour cannot be lipgloss's. The board's own title bar (
dashTitleBar) uses lipgloss instead, and the two share one surface index (barSurface) because they are set through different mechanisms and would otherwise drift. - A grid row joins the corresponding lines of the tiles it places, not the tile blocks — joining whole blocks glues the second tile's top border to the first tile's bottom border and shifts its body down a line.
- A tile's content is exactly the lines
metricsprints for the node in the board's current format (renderStatBars/renderStatGauges/renderTokenLinesare shared, not reimplemented), so the panel andmetricscan never disagree on a number. The format is board-wide, indashModel.gauge, toggled byg, opening in gauge; the tile draws the sparkline atdashBarLineW, chosen so a full row (label, glyphs, trailing percentage) fits the tile's width exactly. - The log panes (the detail view's and the serve view's) count rows at draw time and lines everywhere else:
dashLogRowsexpands the window's lines to wrapped rows (dashWrapLinesbreaks at exact display columns —ansi.CutWcwith a non-zeroleftmisbehaves on wide runes in the pinnedcharmbracelet/x/ansi, which is why the wrapper cuts heads withTruncateWcand tails withTruncateLeftWc), the pane takes the lastavailrows, and retention budgets (logBudget,detailLogCapacity) stay in lines. Scrolling stays line-based (logBehind) so a wrapped line moves whole; do not "tidy" one granularity into the other. Metrics lines keep clipping —dashClip's cut is what keeps the grid and the bar lines the shape of the frame.
Behavior (panel contents, refresh cadence, start/stop/abort semantics, the detail view) is specified in openspec/specs/fleet-client/spec.md.
- The bar format's data lives in the daemon, not the client:
systemHistory(internal/daemon/history.go) is appended on each sampler tick while an engine runs, survives a stop, and clears on the next start alongside the counter baseline'ssample.forget(). Every client — one-shot, watch, dashboard, the cloud relay — draws the same window from the one read. - The history samples' JSON field names are one letter each (
t/c/m/s/g,i/u/m) because the reply crosses SSM on the cloud relay, and SSM command output truncates at 4KB. The window is 10 minutes at the 15s cadence (40 samples) and is capped athistoryLimitsamples regardless of cadence — the catch-up ticks run at 1s, and an engine with no scrape target never leaves that cadence. If the window or the sample shape grows, cap the size in the daemon, not the client: a truncated reply is corrupt JSON, andparseDaemonMetricsturns that into "daemon unreachable" for the whole metrics call.
- opencode config schema: https://opencode.ai/docs/config/. The catalogue follows it:
amazon-bedrockis the Bedrock provider id, custom providers (ollama,llamacpp,openai-compatible) carry annpmpackage plusoptions.baseURL. The key is written as opencode's{env:VAR}substitution rather than the resolved secret, so no secret lands on disk;spinloop harness openpasses the keys it can resolve to the agent it launches, which is what makes a config spinloop wrote usable without exporting anything by hand. - Pi custom-models schema: https://github.com/earendil-works/pi (
packages/coding-agent/docs/models.md).apiis one ofopenai-completions/openai-responses/anthropic-messages/google-generative-ai;apiKeysupports$ENV_VARinterpolation. Not every provider maps to Pi — those without api:block (e.g.amazon-bedrock) error under the pi harness. - lucinate connections store: https://github.com/lucinate-ai/lucinate. An OpenAI-compatible connection is
{id, name, type: "openai", url, defaultModel}; the key comes from lucinate's secrets store or, when unset, theLUCINATE_OPENAI_API_KEYenv var (which is how spinloop configures it — no secret on disk). Only providers with alucinate:marker map; the rest (e.g.amazon-bedrock, the Vertex providers) error under the lucinate harness.