Repository navigation
Swarm coordinator: skein's conductor ideas, inside the project server #104
Description
Proposal
Parent: autonomous-swarm. Depends on project-server (hosts it) and lead-authority
(who may direct). Replaces the plan to depend on the Go skein for any of this.
Why
The Go skein's useful ideas are mechanical, not LLM: a stage/gate graph with evidence, stall
rules, send-once nudges with backoff, claim reaping, a roster, a spawn path, a board. Its
failure was a resident process that owned its own agency and its own state. With a project
server that already owns every agent session, the same ideas fit as a module that reads
server state and the database and does five things only: send one message, write one
record, raise one pending decision, spawn or stop an agent, start or stop a loop. It never
runs a model turn of its own, and every judgement call goes to the lead.
What Changes (ideas absorbed from skein, by source)
| from skein | becomes |
|---|---|
pipeline_stages.go stage graph |
coordinator/gates.ts: worktree → implement → tests → review → merge → push → [deploy] → [live-test], each with named evidence (checked tasks, gate run outcome, PR/commit, .skein/review.json verdict, merge SHA). Reuses loop/spec-queue/gates.ts. |
| stall rules / watchdog thresholds | coordinator/rules.ts: IdleWithReadyWork, ClaimHeldNoProgress, RepliedThenIdle, OrphanedClaim, evaluated from the roster and claims only |
| nudge protocol | coordinator/nudge.ts: dedupe key (slug, agent, gate, reason), backoff tick×2^n, budget 3 then escalate to the lead once; ledger in the DB |
| claims / board | crew-loop claims table and board, unchanged; the coordinator reaps and reassigns |
| roster + adapters | in-server Roster (from project-server) plus foreign adapters for Claude Code over the sidecar; one AgentAdapter interface (prompt, steer, interrupt, capabilities) |
| runner / spawn | agent spawn (from project-server) |
skein_conduct MCP |
the coordinator's tools, available to the lead session: swarm_status, swarm_assign, swarm_nudge, swarm_stop; for a Claude Code lead, a thin MCP shim that proxies to the project server (spike) |
| pending decisions | board decide items, rendered from .skein/blocker.md (escalate-before-idle) |
| provider chains / model choice | existing placement + persona models; no new config |
| fleet YAML | optional spawn templates only (role, model, change filter); presence comes from discovery |
The coordinator is a plain tick (default 30 s) that is idempotent and writes only through
the ledger, so it can run in whichever server instance holds the coordinator lease (database
row with TTL and heartbeat) and move on failure. With a shared project server there is
normally exactly one holder.
Trust and authority
The coordinator acts for the lead. It holds no authority of its own: the lead-authority
grant names it as a delegate with scope nudge (and spawn if the operator adds it).
Messages it sends carry the lead frame because the server attributes the sender, not because
the text says so. It cannot widen permissions or publish; standing-publish-authority's
merge driver is a separate, policy-gated action it may request but not perform.
Non-goals
- No LLM in the coordinator; no scope, plan or priority decisions.
- No replacement for the lead: "what to build next" is the lead's, from specs.
- No dependency on the Go skein. Interop with skein over A2A may remain, optional.
Risks
- Rebuilding the resident-process failure. Mitigation: all coordinator state in the DB;
lease-based; restart replays; negative-control soak asserts an idle coordinator writes
nothing. - Nudge storms. Mitigation: dedupe key, budget, positive-control test.
- Coordinator misreads a healthy long turn as stalled. Mitigation: per-gate
StallAge
thresholds and the roster's busy signal (exact, not windowed) gate every rule.
Impact
New coordinator/* (gates, rules, nudge, lease, tools), crew-loop board hooks,
session/peers.ts roster consumers, tool registry, TUI view, tests.
Design notes
Design: swarm coordinator
D1. Shape
tick(): # idempotent, bounded, no model calls
require lease(coordinator) # DB row {holder, ttl, heartbeat}
grant = lead.grant(); if none: record `no-lead` decision; pause nudging; return
view = roster() ⨝ claims ⨝ changes ⨝ gate evidence
for cond in rules(view): act(cond) # act ∈ {message once, write record,
# decision, spawn/stop agent, start/stop loop}
Everything it does is recorded in coordinator_ledger(key, at, action, outcome); a restart
rebuilds from the ledger and the DB.
D2. Rules (from skein D7, kept)
| condition | signal | action |
|---|---|---|
IdleWithReadyWork |
agent idle and attended; board has an unclaimed change in a ready column matching its role | message: claim <slug> at gate <g> (or start a queue loop for it on an in-server agent) |
ClaimHeldNoProgress |
claim held, no evidence delta for the gate's StallAge |
nudge with gate state and next step |
RepliedThenIdle |
claim held, busy→idle, no evidence delta | nudge: continue at gate <g> |
OrphanedClaim |
heartbeat stale, no live agent | release to board; branch and worktree preserved |
BlockedUnanswered |
blocker rung deadline passed | rung L2 request to lead, once (escalate-before-idle) |
ReviewDue |
done without a review at head SHA | trigger review-on-done |
Dedupe (slug, agent, gate, reason); backoff tick×2^n; after 3, stop and escalate once.
"Send once" is a tested invariant with a positive control.
D3. Gates and evidence
Gate list and evidence kinds per proposal. coordinator/gates.ts is data, validated at load
(closed schema). complete for a change fails closed and returns the missing evidence: a
completion token alone is never sufficient. Optional gates (deploy, live-test) per project
config. Review evidence is .skein/review.json; merge evidence is the merge driver's record.
D4. Lead and delegation
The lead directs; the coordinator mechanises. A directive "take change X" becomes: claim X
for the target agent, start a queue loop for X on it (verified completion), expect an ack,
send nothing further until the budget says so. Completion arrives as the one notice from
loop-done-handoff.
D5. Tools for the lead
swarm_status (board, roster, claims, decisions), swarm_assign(agent, slug),
swarm_nudge(agent, text) (one message, deduped), swarm_stop(agent). All read the
server's state; none writes authority. Available to in-server sessions as tools; for a Claude
Code lead through a small MCP shim to the project server's HTTP API (spike 0.3).
D6. Why in the server, not a separate process
The server already owns sessions, loops, the roster, the inbox and the DB; the coordinator
needs exactly that and nothing else, so a second process would re-create the split state
that hurt before. It stays small and replaceable because its only inputs and outputs are the
DB and the server API.
D7. Standalone mode
Without a project server (standalone processes), the coordinator runs in whichever instance
holds the lease and reads the roster through the sidecar registry. Degraded: no headless
spawn, no in-process delivery. This is the migration path and keeps --standalone useful.
Tasks
Phase 0: Decide scope against what exists
- 0.1 Read the skein conductor tasks (
live-session-conductorslices 1–5) and mark each
as absorbed here, deferred, or dropped. Record in this change'sfindings.md. - 0.2 Inventory
loop/spec-queue/gates.tsandcrew-loopboard code; the coordinator
must reuse, not fork, them. - 0.3 Spike: Claude Code lead → project server via a minimal MCP shim. Does a
tools/list
round trip work with the server's password auth?
Phase 1: Core
- 1.1 Coordinator lease (DB row, TTL, heartbeat); two-holder test; takeover after kill.
- 1.2 Ledger table and replay.
- 1.3
coordinator/gates.tsdata + closed-schema validation at load (observed-red on a
wrong-typed file). - 1.4 Completion fails closed with missing-evidence list; token-only test rejected.
Phase 2: Rules and nudges
- 2.1 Rules table per D2, each with a table test over fixture roster/claims.
- 2.2 Nudge dedupe, backoff and budget; positive control (identical second condition
does not resend); negative control (idle coordinator with no work writes nothing). - 2.3 Escalation: one lead request, one
decideitem.
Phase 3: Lead tools
- 3.1
swarm_status|assign|nudge|stoptools; reject calls from sessions without a lead
grant (lead-authority). - 3.2
swarm_assignend to end: claim → queue loop on target agent → ack → notice. - 3.3 MCP shim per spike 0.3 (if viable).
Phase 4: Standalone fallback
- 4.1 Lease-holder coordinator over the sidecar registry; document the degraded features.
Phase 5: Live
- 5.1 Lead + three headless agents, one real backlog, one evening; record human
interventions, nudges sent, escalations, merges. - 5.2 Kill the server during the run; coordinator resumes from the ledger.
Activity