Repository navigation
Multi-agent teams: progress, task channel, cross-peer handoff, workstreams, generic runners - #41
Merged
Merged
Conversation
cmesh's destination is a multi-agent team: agents from different vendors on different machines working one problem under one policy boundary. This records the model and the sequenced plan before any of it is built. The load-bearing decision is the conductor model. Free-form agent-to-agent chat is rejected explicitly -- one agent's output becoming another agent's instructions is a prompt-injection amplifier and destroys the audit story. Work products move as branches and envelopes; the team conversation is the origin session. That posture is what lets agents from vendors with no reason to trust each other collaborate at all. The design insight that sets the order: the task channel splits in two. Worker-to-origin messaging needs no harness streaming, because a worker task already runs with a task-scoped MCP gateway and a scoped token -- a question is just another tool call over infrastructure that exists. Only inbound live steering needs the exec rework, so it is deferred and separable. Everything unbuilt is marked planned so no user-facing doc cites a feature that does not ship yet.
…lt to query Coherence cleanup ahead of the teams work (docs/teams-plan.md, Phase 0). Shrinks the surface before anything new lands. No tool is added or removed: both MCP surfaces still expose 31 and 22 tools. - Remove the REST POST /tools/<name> subset from the gateway, along with the 21 handler wrappers and two helpers it was the only caller of. It was a second, partial entry path to tools that JSON-RPC already serves, with no test and no CLI consumer -- surface area to keep hardened for no benefit. - Consolidate register_gateway (stdio wizard) and register_harness (gateway) onto internal/mcp/harnessreg. They had drifted: only the wizard supported Codex, only the wizard removed the stdio registration it was replacing, and the gateway failed the call where the wizard degraded to manual instructions. Which behavior a user got depended on which surface their agent happened to be on. Both tool names stay, now identical. The wizard keeps requiring an explicit harness, since guessing wrong mid-onboarding strands the user; the gateway keeps defaulting to claude-code. - Default an unspecified profile to query rather than the legacy dev, which is high-risk under the current ladder and the opposite of the safe default every other surface uses. - Docs: canonical profile names only in the ladder (CLAUDE.md, concepts.md, configuration.md). network and git_push are capabilities, not rungs; dev/readonly/interactive/network/git_push are documented as compatibility aliases in exactly one place, verified against CanonicalProfile. Tests cover the new package's security-relevant behavior: previews never echo the host token on either harness, the Codex writer replaces only its own TOML table at 0600, and an unknown harness degrades to manual rather than failing.
Follow-up to the Phase 0 commit, which changed only the policy engine's fallback and left three siblings disagreeing with it. The important one: handleCreateTask recorded ProfileRequested from defaultProfileForTask, which returned the legacy high-risk dev for any non-query task. With policy now resolving query, a profile-less request produced a record claiming the caller asked for dev while the worker enforced query -- the same requested-vs-effective divergence closed for runners last session, reopened on the profile axis. prepareGit builds the branch slug from ProfileRequested, so it reached the git history too. A test now asserts the two agree rather than trusting they do. Also: - gateway defaultProfile's last-resort dev -> query. - runnerFromProfile's dev lookup -> query, and its final fallback now iterates profiles in sorted order instead of relying on map iteration, which was nondeterministic across runs. - register_harness advertised harness values said "claude-code or manual"; manual now falls into the unsupported branch, so a model reading the schema would pass a value that no longer registers. Corrected to claude-code/codex. - profile_unavailable for a profile-less request said the fallback profile was not configured, naming a profile the caller never asked for. It now says no profile was requested and what to do about it.
A conductor could only poll terminal state, so a running task was opaque
until it finished. task_events surfaces the progress this mesh already
records -- state transitions, policy decisions, dispatch -- so an agent can
report what a task is doing while it runs. No new trust surface: this reads a
table written on every task transition, and it is daemon-recorded evidence,
not worker prose.
Store (migration v3): task_events gains a monotonic seq column, the cursor
for incremental reads. The existing (task_id, timestamp) index could not
serve as one -- timestamps are not unique, so a poller resuming from one
would replay or skip events written in the same instant. SQLite cannot add an
AUTOINCREMENT column to an existing table, so seq is allocated from a counter
table inside the same transaction as the insert, which also keeps two
concurrent recorders from sharing a value. Existing rows are backfilled in
write order; a test seeds pre-v3 rows and asserts the backfill preserves
order and the counter continues past them.
Mesh RPC: GET /v1/tasks/{id}/events?after=&limit=&wait=. It enforces
requireSchema and the shared ownership check in its own right rather than
relying on the gateway having done so, and is added to the test that
enumerates every task-scoped endpoint's ownership refusal. Long-polling stops
early once a task is terminal, since nothing further will be recorded.
Gateway: task_events routed through fetchOwnedTask, so a per-task token sees
only its own delegation's events -- covered by extending the existing
cross-session scoping test. 32 tools advertised, 32 dispatched.
…s_more Review follow-ups to the Phase 1 commit. Two were blocking. The v3 backfill used a correlated COUNT(*) per row to order by (timestamp, rowid) -- a full scan per row. Measured at 61s for 20k rows versus 10ms for UPDATE task_events SET seq = rowid, and it is quadratic, so ~100k rows would be roughly 25 minutes. task_events is append-only and grows with every state transition on every task, migrations hold the write lock, and Migrate runs before the daemon binds a listener -- so on a real worker this would have read as "cmesh up hangs forever after upgrade". The table is insert-only, so rowid is already write order; the existing backfill test passes unchanged, which was the point. It now selects statements by content rather than slice position. The seq allocation claimed concurrent recorders could not share a cursor, but nothing tested it and the daemon records audit events from per-task goroutines. Added a -race test spawning 20 concurrent recorders that asserts 20 distinct seqs, and set SetMaxOpenConns(1): SQLite has one writer, so without it the pool opens connections that contend for the WAL lock and wait out busy_timeout instead of queueing in-process -- and an audit write failure now fails the task. task_events was worker-local before this phase; now a delegating peer reads it. The handler passed the whole stored audit record through, including error_detail and a free-form extra map that can carry worker-side filesystem paths. schema.TaskEvent is now an explicit projection -- type, time, runner, profile, error code -- with named fields so widening it stays deliberate. A test asserts the local detail does not cross the mesh, and fails if the passthrough is reintroduced. Also: has_more said "the page was full" rather than "the page was truncated", so a page that exactly drained the tail claimed more. It now fetches one past the limit and trims.
A delegated task could only speak once, at the end. If it hit an ambiguity it had to guess or die: needs_human was terminal, so asking a question meant ending the task. This gives a running task a conversation with the operator who delegated it, without opening any path between workers. Two ends, two credentials, and confusing them would break the design: task_post / task_ask run INSIDE a worker task. They authenticate with the per-task token cmesh already injects into that runner and resolve the task from the token's context file -- a request-supplied task id is only ever checked against it, never used to reach another task. task_send / task_messages run on the ORIGIN and require task ownership. A task token is refused on task_send. A worker that could answer its own question would manufacture apparent owner approval, which is precisely what asking is meant to prevent; both refusals are asserted in tests. Approval requests are typed. One naming a capability outside the profile that task was actually granted is auto-denied -- policy would deny it anyway -- and everything else, including free prose touching authority, goes to the human. The fallback is always ask, never allow. The origin mirror now records the granted profile so that check has data instead of silently degrading to always-ask. awaiting_input is non-terminal and swept on restart exactly like running, so a task blocked on a question fails visibly rather than being stranded; a test asserts it by failing when the state is dropped from the sweep. It can also transition straight to a terminal state, because the runner keeps executing while the task displays awaiting_input and an agent that stops waiting must not have its result rejected as an invalid transition. task_ask returns unanswered rather than hanging, so a harness MCP timeout cannot strand the agent, and reusing client_message_id resumes the same question instead of asking twice. Waiting is a poll loop over short queries: the store serializes writes on one connection, so blocking inside a transaction would stall every task on the node. Bodies cap at 16 KiB and each channel at 200 messages, both explicit errors. Every message is mirrored to the audit log, but the body is not: it is worker-authored text already stored in task_messages, and copying it into the evidence record would duplicate untrusted content there. Store migration v4 reuses the task_event_seq counter pattern rather than generalizing it -- two tables with one obvious pattern beats one clever one. 36 tools advertised, 36 dispatched.
… refusals Review follow-ups to Phase 2a. Three were blocking; I confirmed each by probing the running code rather than reading for it. A question that timed out unanswered left the task displaying awaiting_input for the rest of its life: askCall cleared the state only on the answered path, and the resume path skipped it entirely. Since the state is what an operator reads to understand why a task looks idle, a wrong one is a real misreport. It is now cleared on every return, and an answer arriving at the mesh handler unblocks the task directly so the state is right even when the worker's ask already gave up. AppendTaskMessage checked client_message_id before opening its transaction, so two concurrent retries could both miss and race to the unique index -- surfacing a raw constraint error instead of the idempotent replay the field exists to provide. The lookup moved inside the transaction, the same fix applied to seq allocation last phase. A worker's runner sees the whole tool list on its own gateway, and the origin-side task tools answered it with "store: not found" from the peer mirror -- which reads as a missing task rather than the wrong credential. They now refuse with peer_unauthorized and say which end of the channel the tool belongs to. Nothing forbidden ever succeeded; the message was wrong. Also: task_messages gains a foreign key to tasks, matching idempotency_keys, so an orphan message cannot be written and a task's channel drops with the task; waitForTaskStatus returns early on awaiting_input rather than burning a whole Wait budget on a task that cannot progress until someone answers; and delegation-guidance documents the state so an agent seeing it knows to read task_messages and reply.
The core team motion: Claude writes the fix on the Linux box, Codex verifies it natively on Windows. followup_task_id deliberately pins to the same peer and bundles only travel worker->origin, so this previously meant a human fetching a bundle and re-seeding by hand. handoff carries the source branch as a git bundle in the CreateTask body rather than a separate upload. That was a real design fork: peer B has no task to authorize an upload against until POST /v1/tasks creates one, but create launches a detached goroutine that reaches prepareGit immediately -- so a create-then-upload flow would race the runner. Carrying the seed in the request keeps one authorization path and guarantees the branch is present before git preparation. It is a wrapper over delegateCall, not a second dispatcher, so it grants no new authority: outbound policy, denied profiles, profile ceilings, and the receiving peer's inbound policy all apply exactly as to any task. The prompt is conductor-authored and required. Only structured refs cross -- branch, base, head, changed files. Seeding the receiving agent with the source worker's summary would turn one agent's claims into another agent's instructions, and a test asserts the prose does not travel. cmesh bundles are thin (Bundle appends ^<base>), so the worker verifies before fetching and a missing prerequisite becomes the typed handoff_base_missing, whose fix is specific: a shared remote, or a push from the source peer. Every other way a handoff cannot proceed is its own explicit refusal -- no result yet, no branch, no artifact, over the size cap, or targeting the peer that already holds the work. Lineage lands in tasks.handoff_from, the audit event, and a cmesh-handoff-from commit trailer, so provenance is answerable from the git object without cmesh's database. gitx gains VerifyBundle and FetchBundle with a real round-trip test across two repositories. Writing it caught a fixture bug worth noting: two repos seeded with identical content, author, and timestamp produce the SAME commit hash, so my first 'unrelated' repository genuinely held the prerequisite and the negative test passed for the wrong reason. 37 tools.
origin_session_id already grouped a task with its siblings and its subdelegated children, and this session made it the ownership key for task-token reads -- but it was invisible plumbing. A workstream is a name for it, so a conductor can ask "where does the auth refactor stand across all three machines?" instead of polling tasks one at a time. The security consequence is the reason this needed care rather than a parameter. A workstream name becomes an origin session, and that session decides which tasks a per-task token may read. So only a host-token caller may choose one: the name is honored on the host-token branch of session derivation only, while a task token's session still comes from its stored context. A worker that could name a workstream could name its way into another team's tasks. A test dispatches under a task token asking for a named workstream and asserts the child never lands there. Names are validated and namespaced as ws:<slug>, so a workstream can never collide with or impersonate a harness-supplied session id, and an invalid name is refused rather than silently ignored. workstream_status reports from this node's mirror rather than fetching each peer live: it is a summary of what this node dispatched, and one peer being briefly unreachable should not fail the overview. It says so, and points at task_status for an authoritative single task. Deliberately not built: cross-node runner session resumption. The setup wizard's "there is no cross-node session" rule is right -- the portable state is the branch. 38 tools.
Two hardcoded adapters capped the multi-vendor story at two vendors, which undercuts the whole premise: a team of agents from different vendors needs a way for a third vendor to join without cmesh shipping code for it. What the built-in adapters actually enforce turns out to be small and portable -- prompt on stdin, profile authority mapped to flags, structured output parsed only when promised, capabilities declared honestly -- so the generic adapter takes those four as config. Three constraints are deliberate, and each is enforced rather than requested: manifest_path is required. A runner cmesh has never seen must declare what it enforces, including its known_gaps. A runner that will not declare itself does not run. enforcement_confidence: high is impossible for a generic runner, checked in New as well as ValidateSemantics so a manifest file cannot smuggle the claim past config validation. The prompt is always on stdin -- there is no opt-out, because cmesh cannot verify a third-party CLI's argument handling. The shared runnertest.AssertPromptViaStdin helper asserts it, the same one the built-in adapters use. An unrecognized runner name is still an error; generic is opt-in via type = "generic", so a typo cannot silently become a runner. Verified end to end rather than by construction: a config-declared runner registers through buildRunners, dropping manifest_path is refused, and cmesh doctor runner passes every check against it including the behavioral stdin smoke -- honesty demonstrated, not asserted. Also named the previously-magic 8000-byte summary cap as runner.MaxSummaryBytes, now shared by the generic adapter.
…docs Three finishing items. config.Validate now rejects a type=generic runner with no manifest_path, an unrecognized type, and runner_args on a built-in runner. The first two previously failed at daemon preflight rather than at load, unlike every other config-shape error, so validate_config and profile_validate reported success on a config that could not start. The third was silently dropped config, which reads as "I set the flag and nothing happened". Phase 2b lands as an honest manifest declaration rather than an exec.go rework. mid_task_input says how each runner receives messages sent to a running task: checkpoint for the agent-backed runners, which pick up queued messages when the agent calls task_ask through its task-scoped MCP gateway, and none for probe and shell, which have no agent to talk to. No runner declares stream, and a test guards the value set so that claim cannot be added without the code to back it. That is a deliberate scope call. Live streaming needs a second interactive path through RunProgram -- the single spawn chokepoint every runner shares -- for a capability whose useful 80% already ships via the channel. Declaring what each runner actually supports is what this repo's manifest philosophy asks for; overclaiming it would not be. It stays on the roadmap. Docs: the README and concepts reframe the plan deferred until the primitives existed, plus a new docs/teams.md covering the conductor model, the five primitives, a worked Linux-to-Windows example, and the rules that keep it safe -- including what is not built. Roadmap updated to match reality.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds out multi-agent teams: agents on different machines, from different vendors, working one problem under one policy boundary with a durable record of who did what.
The model is deliberate — one conductor, many instruments. Your session delegates and decides; workers return evidence. Agents never message each other, and work moves between machines as git branches rather than as one agent's prose becoming another agent's instructions. That posture is what lets agents from vendors with no reason to trust each other collaborate at all.
docs/teams-plan.mdis the plan of record;docs/teams.mdis the user-facing picture.What ships
querytask_events— follow a running task from daemon-recorded evidence, with a monotonic cursorhandoff— continue one peer's branch on another (implement on Linux, verify natively on Windows)Gateway tools: 31 → 38. Store migrations v3–v5.
Security decisions worth review
task_post/task_askrun inside a worker task and resolve their task from the token's context file — a request-supplied task id is only ever checked against it.task_send/task_messagesrequire task ownership. A task token is refused ontask_send: a worker answering its own question would manufacture apparent owner approval, which is exactly what asking prevents. Both refusals are tested in both directions.delegateCall, so outbound policy, denied profiles, ceilings, and the receiving peer's inbound policy all apply. The prompt is conductor-authored and required; a test asserts the source worker's prose never travels.awaiting_inputis non-terminal and swept on restart exactly likerunning, so a task blocked on a question fails visibly rather than being stranded.Bugs caught during the work
Each was confirmed by probing the running code, not by reading:
task_eventspassed the whole audit record across the mesh, includingerror_detailand a free-form map that can hold worker-local paths. Now an explicit projection, with a test that fails if the passthrough returns.awaiting_inputpermanently — the state an operator reads to understand why a task looks idle.seqallocation a phase earlier.gitxtest that passed for the wrong reason: two repos seeded with identical content, author, and timestamp produce the same commit hash, so the "unrelated" repository genuinely held the prerequisite under test.Deliberately not built
task_askcheckpoint. Each runner declares what it supports asmid_task_input; no runner claimsstream, and a test guards the value set so it can't be claimed without the code. Building it needs a second interactive path throughRunProgram— the single spawn chokepoint every runner shares — for a capability whose useful 80% already ships. Stays on the roadmap.ROADMAP.md.Verification
make testpasses exceptvuln, which fails identically onmain(Go stdlib 1.26.4 → 1.26.5, a toolchain bump). Race detector clean on store, gateway, mesh RPC, and all runners. Migrations verified against real databases across reopens; the generic runner verified end-to-end throughbuildRunnersandcmesh doctor runner.