Skip to content

Swarm coordinator: skein's conductor ideas, inside the project server #104

Open

Description

@androidand
Proposal

Parent: autonomous-swarm. Depends on project-server (hosts it) and lead-authority
(who may direct). Replaces the plan to depend on the Go skein for any of this.

Why

The Go skein's useful ideas are mechanical, not LLM: a stage/gate graph with evidence, stall
rules, send-once nudges with backoff, claim reaping, a roster, a spawn path, a board. Its
failure was a resident process that owned its own agency and its own state. With a project
server that already owns every agent session, the same ideas fit as a module that reads
server state and the database and does five things only: send one message, write one
record, raise one pending decision, spawn or stop an agent, start or stop a loop. It never
runs a model turn of its own, and every judgement call goes to the lead.

What Changes (ideas absorbed from skein, by source)

from skein becomes
pipeline_stages.go stage graph coordinator/gates.ts: worktree → implement → tests → review → merge → push → [deploy] → [live-test], each with named evidence (checked tasks, gate run outcome, PR/commit, .skein/review.json verdict, merge SHA). Reuses loop/spec-queue/gates.ts.
stall rules / watchdog thresholds coordinator/rules.ts: IdleWithReadyWork, ClaimHeldNoProgress, RepliedThenIdle, OrphanedClaim, evaluated from the roster and claims only
nudge protocol coordinator/nudge.ts: dedupe key (slug, agent, gate, reason), backoff tick×2^n, budget 3 then escalate to the lead once; ledger in the DB
claims / board crew-loop claims table and board, unchanged; the coordinator reaps and reassigns
roster + adapters in-server Roster (from project-server) plus foreign adapters for Claude Code over the sidecar; one AgentAdapter interface (prompt, steer, interrupt, capabilities)
runner / spawn agent spawn (from project-server)
skein_conduct MCP the coordinator's tools, available to the lead session: swarm_status, swarm_assign, swarm_nudge, swarm_stop; for a Claude Code lead, a thin MCP shim that proxies to the project server (spike)
pending decisions board decide items, rendered from .skein/blocker.md (escalate-before-idle)
provider chains / model choice existing placement + persona models; no new config
fleet YAML optional spawn templates only (role, model, change filter); presence comes from discovery

The coordinator is a plain tick (default 30 s) that is idempotent and writes only through
the ledger, so it can run in whichever server instance holds the coordinator lease (database
row with TTL and heartbeat) and move on failure. With a shared project server there is
normally exactly one holder.

Trust and authority

The coordinator acts for the lead. It holds no authority of its own: the lead-authority
grant names it as a delegate with scope nudge (and spawn if the operator adds it).
Messages it sends carry the lead frame because the server attributes the sender, not because
the text says so. It cannot widen permissions or publish; standing-publish-authority's
merge driver is a separate, policy-gated action it may request but not perform.

Non-goals

  • No LLM in the coordinator; no scope, plan or priority decisions.
  • No replacement for the lead: "what to build next" is the lead's, from specs.
  • No dependency on the Go skein. Interop with skein over A2A may remain, optional.

Risks

  1. Rebuilding the resident-process failure. Mitigation: all coordinator state in the DB;
    lease-based; restart replays; negative-control soak asserts an idle coordinator writes
    nothing.
  2. Nudge storms. Mitigation: dedupe key, budget, positive-control test.
  3. Coordinator misreads a healthy long turn as stalled. Mitigation: per-gate StallAge
    thresholds and the roster's busy signal (exact, not windowed) gate every rule.

Impact

New coordinator/* (gates, rules, nudge, lease, tools), crew-loop board hooks,
session/peers.ts roster consumers, tool registry, TUI view, tests.

Design notes

Design: swarm coordinator

D1. Shape

tick():                                  # idempotent, bounded, no model calls
  require lease(coordinator)             # DB row {holder, ttl, heartbeat}
  grant = lead.grant(); if none: record `no-lead` decision; pause nudging; return
  view  = roster() ⨝ claims ⨝ changes ⨝ gate evidence
  for cond in rules(view): act(cond)     # act ∈ {message once, write record,
                                         #        decision, spawn/stop agent, start/stop loop}

Everything it does is recorded in coordinator_ledger(key, at, action, outcome); a restart
rebuilds from the ledger and the DB.

D2. Rules (from skein D7, kept)

condition signal action
IdleWithReadyWork agent idle and attended; board has an unclaimed change in a ready column matching its role message: claim <slug> at gate <g> (or start a queue loop for it on an in-server agent)
ClaimHeldNoProgress claim held, no evidence delta for the gate's StallAge nudge with gate state and next step
RepliedThenIdle claim held, busy→idle, no evidence delta nudge: continue at gate <g>
OrphanedClaim heartbeat stale, no live agent release to board; branch and worktree preserved
BlockedUnanswered blocker rung deadline passed rung L2 request to lead, once (escalate-before-idle)
ReviewDue done without a review at head SHA trigger review-on-done

Dedupe (slug, agent, gate, reason); backoff tick×2^n; after 3, stop and escalate once.
"Send once" is a tested invariant with a positive control.

D3. Gates and evidence

Gate list and evidence kinds per proposal. coordinator/gates.ts is data, validated at load
(closed schema). complete for a change fails closed and returns the missing evidence: a
completion token alone is never sufficient. Optional gates (deploy, live-test) per project
config. Review evidence is .skein/review.json; merge evidence is the merge driver's record.

D4. Lead and delegation

The lead directs; the coordinator mechanises. A directive "take change X" becomes: claim X
for the target agent, start a queue loop for X on it (verified completion), expect an ack,
send nothing further until the budget says so. Completion arrives as the one notice from
loop-done-handoff.

D5. Tools for the lead

swarm_status (board, roster, claims, decisions), swarm_assign(agent, slug),
swarm_nudge(agent, text) (one message, deduped), swarm_stop(agent). All read the
server's state; none writes authority. Available to in-server sessions as tools; for a Claude
Code lead through a small MCP shim to the project server's HTTP API (spike 0.3).

D6. Why in the server, not a separate process

The server already owns sessions, loops, the roster, the inbox and the DB; the coordinator
needs exactly that and nothing else, so a second process would re-create the split state
that hurt before. It stays small and replaceable because its only inputs and outputs are the
DB and the server API.

D7. Standalone mode

Without a project server (standalone processes), the coordinator runs in whichever instance
holds the lease and reads the roster through the sidecar registry. Degraded: no headless
spawn, no in-process delivery. This is the migration path and keeps --standalone useful.

Tasks

Phase 0: Decide scope against what exists

  • 0.1 Read the skein conductor tasks (live-session-conductor slices 1–5) and mark each
    as absorbed here, deferred, or dropped. Record in this change's findings.md.
  • 0.2 Inventory loop/spec-queue/gates.ts and crew-loop board code; the coordinator
    must reuse, not fork, them.
  • 0.3 Spike: Claude Code lead → project server via a minimal MCP shim. Does a tools/list
    round trip work with the server's password auth?

Phase 1: Core

  • 1.1 Coordinator lease (DB row, TTL, heartbeat); two-holder test; takeover after kill.
  • 1.2 Ledger table and replay.
  • 1.3 coordinator/gates.ts data + closed-schema validation at load (observed-red on a
    wrong-typed file).
  • 1.4 Completion fails closed with missing-evidence list; token-only test rejected.

Phase 2: Rules and nudges

  • 2.1 Rules table per D2, each with a table test over fixture roster/claims.
  • 2.2 Nudge dedupe, backoff and budget; positive control (identical second condition
    does not resend); negative control (idle coordinator with no work writes nothing).
  • 2.3 Escalation: one lead request, one decide item.

Phase 3: Lead tools

  • 3.1 swarm_status|assign|nudge|stop tools; reject calls from sessions without a lead
    grant (lead-authority).
  • 3.2 swarm_assign end to end: claim → queue loop on target agent → ack → notice.
  • 3.3 MCP shim per spike 0.3 (if viable).

Phase 4: Standalone fallback

  • 4.1 Lease-holder coordinator over the sidecar registry; document the degraded features.

Phase 5: Live

  • 5.1 Lead + three headless agents, one real backlog, one evening; record human
    interventions, nudges sent, escalations, merges.
  • 5.2 Kill the server during the run; coordinator resumes from the ledger.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions