From 39e5a9e6df3646ad356b31d8e811e938243cae56 Mon Sep 17 00:00:00 2001 From: AndreasS Date: Fri, 2 Oct 2026 21:37:37 +0200 Subject: [PATCH 1/2] plan(project-server): Phase 0 spike results and a design change they forced One server hosts both worktrees under a shared projectID; sessions survive a hard kill but loops do not; only per-directory state needs a roster; headless sessions register as peers only after their first status, and each costs a 204 MB sidecar process, so in-server delivery alone does not deliver the memory argument and the sockets must be hosted in the server. Co-Authored-By: Claude Sonnet 5.5 --- openspec/changes/project-server/findings.md | 78 +++++++++++++++++++++ openspec/changes/project-server/tasks.md | 14 ++-- 2 files changed, 86 insertions(+), 6 deletions(-) create mode 100644 openspec/changes/project-server/findings.md diff --git a/openspec/changes/project-server/findings.md b/openspec/changes/project-server/findings.md new file mode 100644 index 000000000000..bc783c4296bc --- /dev/null +++ b/openspec/changes/project-server/findings.md @@ -0,0 +1,78 @@ +# Findings: project-server Phase 0 spikes (2026-10-02) + +Method: a throwaway `opencode serve` (build b7d95c8427) on loopback with a password, in an isolated +data, state, config and Claude-registry directory (so nothing appeared as a peer to live sessions), +serving one git repo and one of its worktrees. No model provider was configured, so everything below +is about hosting, state and lifecycle, not inference. Numbers are single runs on one machine. + +## 0.1 Two worktrees in one server: works + +- A session can be created in either directory through the `x-opencode-directory` header or a + `?directory=` query parameter. +- Both worktrees resolve to the SAME `projectID` (git common-dir identity). The design's repo key + (D1) can therefore be the existing `projectID`; no new hashing is needed. +- `GET /session` from either directory returns every session of the project, across both + worktrees. Cross-worktree session listing already exists; what does not is cross-worktree live + status (see 0.4). + +## 0.2 Attach: NOT TESTED + +`opencode attach` needs a terminal; a non-TTY run is not meaningful. Needs the operator's terminal: +attach to a session whose loop is running with no other client, close the TUI, check the loop +survives. + +## 0.3 Kill mid-run: sessions survive, loops do not (baseline for durable loops) + +A queue loop with `queueWatch` stays alive without a model, so it stood in for a long-running agent. +After `kill -9` and a restart on the same data: the session was still listed (the database holds it); +`GET /loop` returned `[]`. The loop that was driving the session is gone with no record that it ever +existed. This is the exact behaviour D4 (durable loops) has to replace, and it confirms the loop +registry is memory-only. + +## 0.4 Audit of the D6 maps + +| state | scope | in a shared server | +| --- | --- | --- | +| `peer/inbox.ts` pending deliveries | process, keyed by session id | works unchanged; becomes coherent across worktrees | +| `peer/envelope.ts` pendingReplies, `peer/delegate.ts` pending | process | same | +| `peer/claude/sidecar-manager.ts` active | process, keyed by session id | same | +| `session/unattended.ts` policies, `session/compaction.ts` loop-break state | process, keyed by session id | same | +| `local/placement.ts` reservations, recentPlacements | process, keyed by provider | becomes correct for every agent in the server (was per terminal) | +| `loop/loop.ts` registry | one `Ref` per process | global to the server, already aggregates all directories | +| `session/status.ts`, `permission/index.ts`, `session/run-state.ts`, `session/instruction.ts` | `InstanceState`, per directory | the only genuinely per-worktree state: a roster must aggregate it | + +Result: nothing collides. Only per-directory state needs an aggregating roster. + +## 0.5 Headless sessions as peers: yes, but late and expensive + +- A session created over HTTP is NOT registered as a peer at creation. After its first status report + (a prompt, even one that fails) the server registered it: `managedBy: opencode-skein`, status + idle, correct cwd. A freshly spawned agent is therefore invisible to other agents until it first + runs. Cheap fix: ensure the sidecar at creation, or have the coordinator start its loop immediately. +- Each registered session gets its OWN sidecar process, `opencode debug claude-sidecar-entry`, which + is a full copy of the binary: about 204 MB resident each, about 1% CPU idle. + +## 0.6 Cost + +| thing | resident | CPU idle | +| --- | --- | --- | +| headless server, 2 worktree instances, 4 sessions, no provider | 318 MB | about 0.5% | +| same server after ONE prompt attempt (provider path loaded) | 1,091 MB | 0% | +| one sidecar (per registered session) | 204 MB | about 1% | +| for comparison, an interactive session on the operator's machine | about 1.1 GB | 40-54% | + +The first request to a fresh instance spiked to 92% CPU once (setup), then settled. Idle cost lives in +the per-terminal process, not in hosting sessions. Caveat: no providers, LSP or MCP were loaded. + +## Consequences for the design + +1. D1 stands, with `projectID` as the key. +2. D4 (durable loops) is confirmed necessary, with a measured baseline. +3. D5 (in-server delivery) is NOT enough on its own. Registration for Claude Code and cross-repo + peers still costs one 200 MB process per agent, so ten agents would spend 2 GB on sidecars alone, + which defeats the memory argument for a shared server. NEW REQUIREMENT: the project server hosts + the per-session unix sockets itself (in process, or one multiplexing sidecar for the whole + server) instead of one child process per session. This becomes its own task in Phase 4. +4. Register at creation (or immediately on spawn), so a spawned agent is addressable at once. +5. The CPU figure for interactive sessions is not explained by hosting. Profile one before blaming a + feature (the OTLP collector approach in the operator's notes). diff --git a/openspec/changes/project-server/tasks.md b/openspec/changes/project-server/tasks.md index 0ca0d0583058..d99bc34e3afb 100644 --- a/openspec/changes/project-server/tasks.md +++ b/openspec/changes/project-server/tasks.md @@ -2,17 +2,17 @@ ## Phase 0: Spikes (answers decide the rest) -- [ ] 0.1 Two worktrees of this repo in ONE `opencode serve` via `x-opencode-directory`: create +- [x] 0.1 (works; shared projectID — see findings.md) Two worktrees of this repo in ONE `opencode serve` via `x-opencode-directory`: create a session in each, run a prompt in each concurrently. Record what breaks. -- [ ] 0.2 `opencode attach --session ` against a session whose loop is running with +- [ ] 0.2 (NOT TESTED: needs a terminal) `opencode attach --session ` against a session whose loop is running with no other client: does the TUI show live loop state; does closing the TUI leave the loop running (expected yes, the loop lives in the server)? -- [ ] 0.3 Kill the server mid-loop. Record exactly what is lost (loop registry is an in-memory +- [x] 0.3 (sessions survive, loops do not — findings.md) Kill the server mid-loop. Record exactly what is lost (loop registry is an in-memory `Ref`; expect: everything). This is the baseline D4 must beat. -- [ ] 0.4 Audit the D6 list: for each map, state its scope (process / instance / session) and +- [x] 0.4 (only per-directory InstanceState needs aggregating — findings.md) Audit the D6 list: for each map, state its scope (process / instance / session) and whether two worktrees in one server would collide or fail to see each other. -- [ ] 0.5 Do headless sessions register sidecars and appear in `peers` for Claude Code peers? -- [ ] 0.6 Memory over a 4-hour, 3-agent run in one server vs three TUIs. Baseline for the +- [x] 0.5 (yes after first status; one 204 MB sidecar process each — findings.md) Do headless sessions register sidecars and appear in `peers` for Claude Code peers? +- [~] 0.6 (partial: single-run numbers in findings.md; the 4-hour run is not done) Memory over a 4-hour, 3-agent run in one server vs three TUIs. Baseline for the overhead argument. ## Phase 1: Registry and auto-attach (behind `experimental.project_server`) @@ -41,6 +41,8 @@ ## Phase 4: In-server A2A and aggregation +- [ ] 4.0 (from the 0.5/0.6 findings) Host the per-session unix sockets inside the server, or in one multiplexing sidecar, instead of one 200 MB child process per registered session; register a session at creation, not at first status. + - [ ] 4.1 `Roster` service across instances (D6) with tests over two worktrees. - [ ] 4.2 In-server delivery path in `send_peer_message` and `peers`; sender identity server-attributed; same framing and repeat guard. From 8982341a9d5a96faf57600cdc5ada25bc8e2da41 Mon Sep 17 00:00:00 2001 From: AndreasS Date: Fri, 2 Oct 2026 22:35:58 +0200 Subject: [PATCH 2/2] plan(project-server): correct the registration finding; state the cost caveat up front Review found the claim that sessions register only after their first status contradicted the lifecycle code. Re-tested: a session created in a running server registers within ~4s; the earlier observation was the resume-from- history recovery path. Sidecar cost remeasured with three fresh sessions (~172 MB each). The no-provider caveat moves to the top. Co-Authored-By: Claude Sonnet 5.5 --- openspec/changes/project-server/findings.md | 41 ++++++++++++++------- openspec/changes/project-server/tasks.md | 4 +- 2 files changed, 29 insertions(+), 16 deletions(-) diff --git a/openspec/changes/project-server/findings.md b/openspec/changes/project-server/findings.md index bc783c4296bc..f637cfcde854 100644 --- a/openspec/changes/project-server/findings.md +++ b/openspec/changes/project-server/findings.md @@ -2,8 +2,11 @@ Method: a throwaway `opencode serve` (build b7d95c8427) on loopback with a password, in an isolated data, state, config and Claude-registry directory (so nothing appeared as a peer to live sessions), -serving one git repo and one of its worktrees. No model provider was configured, so everything below -is about hosting, state and lifecycle, not inference. Numbers are single runs on one machine. +serving one git repo and one of its worktrees. + +**Read the numbers with this in mind:** no model provider, LSP or MCP server was configured, so the +server was as light as it can be, and every figure below is a single run on one machine. They show +what HOSTING sessions costs, not what a loaded agent costs. Treat them as a floor, not a baseline. ## 0.1 Two worktrees in one server: works @@ -43,14 +46,22 @@ registry is memory-only. Result: nothing collides. Only per-directory state needs an aggregating roster. -## 0.5 Headless sessions as peers: yes, but late and expensive - -- A session created over HTTP is NOT registered as a peer at creation. After its first status report - (a prompt, even one that fails) the server registered it: `managedBy: opencode-skein`, status - idle, correct cwd. A freshly spawned agent is therefore invisible to other agents until it first - runs. Cheap fix: ensure the sidecar at creation, or have the coordinator start its loop immediately. -- Each registered session gets its OWN sidecar process, `opencode debug claude-sidecar-entry`, which - is a full copy of the binary: about 204 MB resident each, about 1% CPU idle. +## 0.5 Headless sessions as peers: yes, at creation, and each costs a process + +- A session created over HTTP in a RUNNING server registers as a peer within about 4 seconds + (`managedBy: opencode-skein`, status idle, correct cwd). Three fresh sessions gave three + registrations. The sidecar is spawned asynchronously, so the registry is empty at the instant of + creation. +- CORRECTION: an earlier draft of this note said a session registers only after its first status + report. That was wrong. It came from observing a session that had been created in an earlier + server process and was RESUMED from the database after a restart: that is the recovery path the + lifecycle code describes (it registers on first status), not the normal path. Found in review. +- Each registered session gets its OWN sidecar process, `opencode debug claude-sidecar-entry`, a full + copy of the binary. Measured at about 172 MB resident and 0.2% CPU idle for a fresh session, 204 MB + after the session had run a prompt attempt. +- The sidecar is intentionally a separate OS process (it detects parent death and shuts down, and is + swept if orphaned), so hosting the sockets in the server is a design change with a robustness cost, + not a free refactor. Phase 4 task 4.0 has to weigh that. ## 0.6 Cost @@ -58,7 +69,9 @@ Result: nothing collides. Only per-directory state needs an aggregating roster. | --- | --- | --- | | headless server, 2 worktree instances, 4 sessions, no provider | 318 MB | about 0.5% | | same server after ONE prompt attempt (provider path loaded) | 1,091 MB | 0% | -| one sidecar (per registered session) | 204 MB | about 1% | +| one sidecar per registered session, fresh | about 172 MB | about 0.2% | +| one sidecar after a prompt attempt | about 204 MB | about 1% | +| server with 3 fresh sessions registered | 387 MB (+ 3 x 172 MB sidecars = 517 MB) | about 0.2% | | for comparison, an interactive session on the operator's machine | about 1.1 GB | 40-54% | The first request to a fresh instance spiked to 92% CPU once (setup), then settled. Idle cost lives in @@ -69,10 +82,10 @@ the per-terminal process, not in hosting sessions. Caveat: no providers, LSP or 1. D1 stands, with `projectID` as the key. 2. D4 (durable loops) is confirmed necessary, with a measured baseline. 3. D5 (in-server delivery) is NOT enough on its own. Registration for Claude Code and cross-repo - peers still costs one 200 MB process per agent, so ten agents would spend 2 GB on sidecars alone, - which defeats the memory argument for a shared server. NEW REQUIREMENT: the project server hosts + peers still costs one 170-200 MB process per agent, so ten agents would spend about 2 GB on + sidecars alone, which defeats the memory argument for a shared server. NEW REQUIREMENT: the project server hosts the per-session unix sockets itself (in process, or one multiplexing sidecar for the whole server) instead of one child process per session. This becomes its own task in Phase 4. -4. Register at creation (or immediately on spawn), so a spawned agent is addressable at once. +4. Registration already happens at creation (within seconds); nothing to change there. A spawned agent is addressable almost at once. 5. The CPU figure for interactive sessions is not explained by hosting. Profile one before blaming a feature (the OTLP collector approach in the operator's notes). diff --git a/openspec/changes/project-server/tasks.md b/openspec/changes/project-server/tasks.md index d99bc34e3afb..bcc26353aee7 100644 --- a/openspec/changes/project-server/tasks.md +++ b/openspec/changes/project-server/tasks.md @@ -11,7 +11,7 @@ `Ref`; expect: everything). This is the baseline D4 must beat. - [x] 0.4 (only per-directory InstanceState needs aggregating — findings.md) Audit the D6 list: for each map, state its scope (process / instance / session) and whether two worktrees in one server would collide or fail to see each other. -- [x] 0.5 (yes after first status; one 204 MB sidecar process each — findings.md) Do headless sessions register sidecars and appear in `peers` for Claude Code peers? +- [x] 0.5 (yes, at creation; one ~172 MB sidecar process each — findings.md) Do headless sessions register sidecars and appear in `peers` for Claude Code peers? - [~] 0.6 (partial: single-run numbers in findings.md; the 4-hour run is not done) Memory over a 4-hour, 3-agent run in one server vs three TUIs. Baseline for the overhead argument. @@ -41,7 +41,7 @@ ## Phase 4: In-server A2A and aggregation -- [ ] 4.0 (from the 0.5/0.6 findings) Host the per-session unix sockets inside the server, or in one multiplexing sidecar, instead of one 200 MB child process per registered session; register a session at creation, not at first status. +- [ ] 4.0 (from the 0.5/0.6 findings) Host the per-session unix sockets inside the server, or in one multiplexing sidecar, instead of one 200 MB child process per registered session; (sessions already register at creation.) - [ ] 4.1 `Roster` service across instances (D6) with tests over two worktrees. - [ ] 4.2 In-server delivery path in `send_peer_message` and `peers`; sender identity