Skip to content

Project server: one supervisor per project, many agents #103

Open

Description

@androidand
Proposal

Revision 2026-10-25 (opencode 2 pivot): upstream ≥ v2 moved this direction already —
opencode service runs a server independent of the TUI. The build-out below is now
verify-first, gap-fill second (charter §7 status note): the §7.6 ten lifecycle
assertions are driven against opencode service (task owned by
adopt-upstream-acp-and-service 1.1) before any of the "What Changes" build items are
implemented; items upstream already covers are dropped from this change with the
evidence row cited, and this change keeps only what the verification shows missing.
The DurableAgentExecution record (§7.4 amendment) remains the contract for
plugin-owned durable loop state (port-loop-crew-plugin), independent of who owns
the server.

Parent: autonomous-swarm (foundation; replaces the "depend on Go skein" assumption).

Why

skein's pattern was one supervisor, many runners. opencode-skein inverted it: every
opencode launch starts its own server inside the process (cli/cmd/tui.ts spawns a
worker per TUI; opencode serve exists but is opt-in and nothing discovers it). The
consequences are the root of several swarm problems:

  • All coordination state is per process and in memory: the loop registry (loop.ts
    Ref<Map>, not persisted), PeerInbox, pending reply correlation, the repeat guard,
    placement slot reservations, recentPlacements. Two sessions in two processes cannot see
    each other's state, so crew-loop has to bolt on a database table, a "one queue per
    directory" refusal, and a foreign-turn guard.
  • A loop dies when its TUI closes. Agents are tied to terminal windows, which is the
    "one iTerm session per agent" cost.
  • A2A between opencode sessions needs sockets, a registry directory and a sidecar per
    session even when both live on the same machine and project.
  • There is nowhere to run a coordinator that is not itself somebody's TUI.

Already true and reusable: the server loads project instances per request via the
x-opencode-directory header, so one server can serve every worktree of a repo;
opencode attach <url> [--session] already connects a TUI to a running server;
opencode run --attach does the same for CLI runs.

What Changes

  1. One server per repository (identity = git common dir, so every worktree shares it),
    discovered through a registry record <state>/servers/<repo-key>.json
    {url, pid, startedAt, authFile}; liveness = pid alive + health ping. Bound to loopback
    or a unix socket, with a generated password in a 0600 file. Never unsecured.
  2. Auto-attach. opencode in a repo looks up the record; healthy → attach as a
    client; absent or dead → start a detached opencode serve and attach. A TUI close
    detaches; it does not stop the server or the agents in it. Opt-out --standalone
    (today's behaviour). Ships behind experimental.project_server.
  3. Agents are sessions + loops owned by the server, not processes. An agent can be
    observed or driven from a TUI client, a CLI (opencode run --attach), or by no client
    at all (headless). opencode agent spawn --role <persona> [--change <slug>] creates a
    worktree-backed session with a loop in the server; opencode agent list|attach|stop.
    The human can attach a TUI to any agent at any time — "talk to every individual"
    remains, without a window per agent.
  4. Durable loops. Loop records move from the in-memory Ref to the shared database
    so a server restart resumes running agents instead of losing them (precondition for
    trusting a long-lived server).
  5. In-server A2A short-circuit. Sessions in the same server exchange messages
    through the server's own inbox and status (no socket), keeping the envelope and the
    trust framing identical. Sidecars stay for Claude Code and cross-repo peers, so
    nothing in peer-conversation-reliability is thrown away.
  6. Cross-instance aggregation. Session status is InstanceState, i.e. per directory,
    not per process. The server exposes one aggregated view over all worktree instances
    (the roster the coordinator reads).
  7. Supervision of the supervisor. Crash containment: heavy work stays in child
    processes (shell tools already are); a watchdog restarts a wedged server; the first
    client to notice a dead record respawns it; recovery replays durable loops. State that
    matters lives in the database and files, never only in the server's memory (the old
    skein post-mortem's failure).

Non-goals

  • No machine-wide singleton; the unit is the repo. Cross-repo coordination stays on A2A
    sockets.
  • No removal of standalone mode or of the sidecar/Claude Code path.
  • No new wire protocol: the existing HTTP API, SSE events and attach.

Risks

  1. Single point of failure. One server crash stops every agent in the repo. Mitigation:
    durable loops, restart on death, --standalone escape hatch, a soak with a deliberate
    kill (Phase 5).
  2. Per-instance assumptions. Code that treats module-level or InstanceState maps as
    "the process" may mis-scope when many worktrees share a server. Mitigation: audit list
    in design.md; cross-worktree tests.
  3. Security. A long-lived local server is an attack surface. Mitigation: loopback or
    unix socket only, mandatory password, same-user file permissions.
  4. Memory growth in a long-lived process (a known concern: long-session-memory-retention
    change). Mitigation: that change becomes a dependency for default-on.

Impact

cli/cmd/tui.ts, cli/cmd/serve.ts, cli/cmd/attach.ts, cli/cmd/run.ts, new
server/registry.ts and cli/cmd/agent.ts, loop/loop.ts (persistence), peer/*
(in-server delivery), session/status.ts (aggregation), tests.

Design notes

Design: project server

Topology

          human                                   Claude Code lead / peers
   ┌──────┴───────┐                                        │  (sidecar sockets, unchanged)
 TUI client     CLI client                                  │
   │  attach        │  run --attach                          │
   └──────┬────────┘                                         │
          ▼                                                  ▼
 ┌────────────────────── project server (one per repo) ──────────────────────┐
 │ instances: <repo>, <repo-worktree-A>, <repo-worktree-B> … (x-opencode-dir) │
 │ agents  = session + loop (headless unless a client attaches)               │
 │ in-server A2A inbox/status      coordinator (design: swarm-coordinator)    │
 └───────────────┬────────────────────────────────────────────────────────────┘
                 ▼  shared SQLite (sessions, durable loops, claims, grants) + files

The server is the supervisor; agents are sessions it owns; clients are optional views.

D1. Identity and discovery

repo-key = hash(realpath(git common dir)). Registry <state>/servers/<repo-key>.json:
{version, url, pid, procStart, startedAt, authFile}. A client is valid only if pid is
alive, procStart matches (guards pid reuse, same trick as the sidecar registry) and
GET /health answers. Anything else is a stale record, removed by the client that finds it.
Start race: creating the record uses an exclusive-create lock file; the loser attaches to
the winner.

Not in a git repo: fall back to the directory itself as the key.

D2. Auto-attach flow

opencode [dir]:
  key = repoKey(dir)
  rec = readRecord(key)
  if rec healthy: attach(rec, dir)
  else: spawn detached `opencode serve --repo-key key` ; wait for record; attach

Detached means the server is not a child of the TUI (own session/process group), so closing
the terminal does not kill agents. opencode server stop|status|restart for the human.
Version skew: record carries the build id; a client with a different build refuses to attach
silently and offers --standalone or server restart.

D3. Agents without processes

An agent is {sessionID, directory(worktree), persona/model, loop record, claim}.
agent spawn = create worktree via the existing Worktree service → create session in that
instance → start a loop (queue mode for a slug, or prompt). No new OS process. The
peers/roster view labels it headless until a client attaches. agent attach <id> is
opencode attach <url> --session <id> --dir <worktree>.

Capacity model unchanged: an agent still needs an inference slot; the project server just
sees all of them in one place, so the in-memory slot reservations stop being per-process lies
(for agents in this server).

D4. Durable loops

Loop records written to the shared DB on every patch; on server start, records in
running/paused/idle-watch are re-adopted: iteration continues from the recorded state
with a fresh child session (the design already runs each iteration in a child session, so
there is no in-flight turn to recover). A turn that was mid-flight at the crash is
abandoned, and its claim heartbeat lapses and is resumed per crew-loop abandoned-claim
rules. Precondition for default-on.

D5. In-server A2A

send_peer_message resolves the target first against the server's own sessions. Hit →
deliver through the in-process path (PeerInbox/SessionPrompt), same envelope, same
formatPeerMessage framing, same repeat guard. Miss → today's sidecar route. Sender identity
for in-server delivery is the session id the server itself attributes (not self-asserted),
which is strictly stronger than the socket check lead-authority uses for foreign peers. The
inbox and reply correlation maps stop being per-process for these sessions.

D6. Aggregation across worktrees

Session status/Loop/Inbox state is scoped by InstanceState (directory). The server adds a
repo-level service Roster that walks live instances and returns one list
{sessionID, directory, branch, status, loop, claim, headless}. The coordinator and peers
read only this. Audit list (each must be checked and tested with two worktrees in one server):
session/status.ts, peer/inbox.ts, peer/envelope.ts pendingReplies,
peer/repeat-guard.ts instances, peer/delegate.ts pending, local/placement.ts
reservations and recentPlacements, loop/loop.ts registry, worktree/*.

D7. Failure model

failure effect recovery
TUI client closes none for agents reattach
server process dies agents stop next client respawns (D2); durable loops re-adopted (D4)
server wedged (event loop stalled) agents hang watchdog heartbeat file; client or launchd restarts
one agent runaway (tool output, memory) risk to all tools already run as child processes; per-session limits; kill agent not server
port/record stale attach fails stale record removal (D1)

D8. Security

Loopback or unix socket only. Password mandatory for the auto-started server (random, in a
0600 authFile the client reads); serve run by hand keeps today's warning. The server
refuses non-loopback binds without an explicit flag. Peer authority from lead-authority is
unaffected: grants are verified, never trusted from the transport.

Open questions

  • Scope when the operator runs sessions in several unrelated repos: one server each is
    accepted; is there demand for a machine-wide "overseer" that talks to all project servers
    (that is the cross-repo lead — today a Claude Code session over A2A)?
  • Should the default flip on, and when? Gate: soak with deliberate kill, memory retention
    change landed, and no regression of the 3-process crew spike.

Tasks

Phase 0: Spikes (answers decide the rest)

  • 0.1 (works; shared projectID — see findings.md) Two worktrees of this repo in ONE opencode serve via x-opencode-directory: create
    a session in each, run a prompt in each concurrently. Record what breaks.
  • 0.2 (NOT TESTED: needs a terminal) opencode attach <url> --session <id> against a session whose loop is running with
    no other client: does the TUI show live loop state; does closing the TUI leave the loop
    running (expected yes, the loop lives in the server)?
  • 0.3 (sessions survive, loops do not — findings.md) Kill the server mid-loop. Record exactly what is lost (loop registry is an in-memory
    Ref; expect: everything). This is the baseline D4 must beat.
  • 0.4 (only per-directory InstanceState needs aggregating — findings.md) Audit the D6 list: for each map, state its scope (process / instance / session) and
    whether two worktrees in one server would collide or fail to see each other.
  • 0.5 (yes, at creation; one ~172 MB sidecar process each — findings.md) Do headless sessions register sidecars and appear in peers for Claude Code peers?
  • [~] 0.6 (partial: single-run numbers in findings.md; the 4-hour run is not done) Memory over a 4-hour, 3-agent run in one server vs three TUIs. Baseline for the
    overhead argument.

Phase 1: Registry and auto-attach (behind experimental.project_server)

  • 1.1 server/registry.ts: repo key, record schema (closed, validated at read), stale
    detection by pid + procStart + health, exclusive-create start lock. Observed-red tests:
    dead pid, reused pid, wrong schema type, concurrent start.
  • 1.2 Detached spawn with mandatory generated password in 0600 file; loopback/unix only;
    refuse non-loopback.
  • 1.3 opencode and opencode run auto-attach per D2; --standalone; server stop|status|restart; build-id mismatch handling.
  • 1.4 TUI close detaches without stopping the server (verify against 0.2).

Phase 2: Durable loops

  • 2.1 Loop record table and write-through on every patch.
  • 2.2 Re-adopt on server start; fresh child session per iteration; abandoned-turn handling.
  • 2.3 Test: kill -9 the server mid-iteration; restart; loop continues; no duplicate
    iteration; claim heartbeat lapses then resumes. Seen red against current code first.
  • 2.4 DurableAgentExecution record (charter §7.4): table + write-through on every
    transition (executionId, sessionId, projectId, workItemId, status, currentStep,
    attempt, heartbeatAt, leaseExpiresAt, updatedAt). Recovery reads this record plus the
    actual repository state — never the chat transcript — and does not blindly rerun a
    non-idempotent operation whose outcome is unknown.

Phase 3: Agents without processes

  • 3.1 agent spawn|list|attach|stop over worktree + session + loop.
  • 3.2 headless flag in the roster; reattach shows full history.
  • 3.3 Cleanup: stopping an agent releases claim and optionally the worktree.

Phase 4: In-server A2A and aggregation

  • 4.0 (from the 0.5/0.6 findings) Host the per-session unix sockets inside the server, or in one multiplexing sidecar, instead of one 200 MB child process per registered session; (sessions already register at creation.)

  • 4.1 Roster service across instances (D6) with tests over two worktrees.

  • 4.2 In-server delivery path in send_peer_message and peers; sender identity
    server-attributed; same framing and repeat guard.

  • 4.3 Fix each D6-audit finding that mis-scopes (one commit per map, each with a test).

  • 4.4 Relax the "one queue loop per directory" refusal for crew mode, in coordination with
    crew-loop 1.3 (shared claims make it safe).

Phase 5: Supervision and soak

  • 5.1 Watchdog heartbeat and client-side respawn.
  • 5.2 Soak: three agents, deliberate kill of the server every 30 minutes for 4 hours;
    assert no lost work and no duplicated commits.
  • 5.3 Memory comparison against 0.6; decide whether long-session-memory-retention must
    land before default-on.
  • 5.4 Decide default-on for this fork.

Plan changes

4 done · 1 dropped

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions