Skip to content

fix(server): full shell reads leave the main thread, thread usage reads one thread - #339

Merged
bk-agent-01 merged 3 commits into
stagefrom
fix/bkt3-main-thread-read-guards
Oct 21, 2026
Merged

bk-agent-01 merged 3 commits into
stagefrom
fix/bkt3-main-thread-read-guards

Conversation

@bk-agent-01

@bk-agent-01 bk-agent-01 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

After #338, some code paths still read all ~1,800 bkt3 threads on the server's main thread. A review of stage 4d6aa7488f against the production trace found:

  1. ProjectionStore.getShellSnapshot runs on the writer connection, on the event loop. On production, one full read costs about 1.0 s (active threads) to 2.2 s (all threads). Callers that do not add NodeSqliteClient.readOnly:
    • ThreadLaunchService branch rename, after each new worktree launch. My estimate is 1–2 per hour, which matches the old baseline of 1–3 stalls per hour.
    • mcp/toolkits/agent/context.ts loadVisibleSessions (whoami, inbox, find_sessions, brief, start_session, wait). Measured 1.0–1.06 s each.
    • manager.sessions, CheckpointRestoreSafety, AgentAwarenessRelay, SessionActivityReactor, startup.
  2. threadUsage.get reads the full active shell to find one thread. Each open thread view on each client calls it up to once a minute. Production showed 14 calls in 549 s, with p50 4.4–7.5 s and max 17.7 s end to end.

Fix

  1. getShellSnapshot wraps its transaction in NodeSqliteClient.readOnly, so the SQL runs on the reader worker from fix(server): shell reads run on a SQLite reader worker and no longer freeze the server #328. Inside an open write transaction, the read stays on the writer (nodeSqliteClient.ts), so callers that read in a transaction keep the same consistency. Only the decode stays on the main thread. On stage, decode costs about 0.14 ms per row, so my estimate is 110–260 ms on production.
  2. threadUsage.get uses getThreadShellById. To keep the old result for archived threads, the handler still returns no session and no start time for them.
  3. AGENTS.md (also loaded through CLAUDE.md) gets a fork section, "Server code must scale with bkt3's database". It has six rules from the incidents in fix(server): reaper no longer freezes the event loop with unindexed event scans #164, fix(server): pull-request sweep and plan-flag reads no longer freeze the server #325/fix(server): promote the pull-request sweep and plan-flag freeze fix to bkt3 #326, fix(server): shell reads run on a SQLite reader worker and no longer freeze the server #328 and fix(server): account policies no longer freeze the server reading the shell #338.

No schema changes, no migrations, no new events. The fork markers are in place.

How I tested

This host runs no tests (the dev-server compute policy). CI runs these on the head:

  • New wsForkHandlers.threadUsage.test.ts: the handler reads one thread shell and never calls getShellSnapshot. Archived and missing threads keep the empty result.
  • The existing ProjectionStore and orchestration tests cover the getShellSnapshot result. I did not add a reader-routing test, because the routing is the shared readOnly helper that nodeSqliteClient.test.ts already covers.

After the merge into stage, I will check stagebkt3 (stall warnings and probe latency).

Risks

  • The JSON decode of every row still runs on the main thread. My estimate is 110–260 ms on production. I did not measure it.
  • More callers now share the one reader connection with the shell stream and the PR sync. A slow full read can delay those reads. That is better than a block on the writer and the event loop.

Follow-up found in review (older bug, not changed here)

  • legacyThreadShell (legacyProjection.expbkt3.ts:75) always sets session.providerThreadId to null. So threadUsage.get always sends sessionIds: [], and thread usage is empty in production, both before and after this PR.

Left for later (measured, not worth a change now)

  • The PR sync full pass every minute already uses the reader (about 110 ms of decode per minute, an estimate).
  • The Linear bridge GET /api/orchestration/shell already uses the reader. The real fix is BRIDGE_T3_EVENT_FEED in the bridge.
  • About 4.6 one-thread shell reads per second (p50 2.7 ms, about 1.3% of the main thread).

🤖 Generated with Claude Code

…ds one thread

- ProjectionStore.getShellSnapshot runs on the SQLite reader worker, so the
  remaining writer-path callers (worktree branch rename, MCP agent session
  lists, manager.sessions, startup) no longer block the event loop for ~2 s.
- threadUsage.get reads only the requested thread's shell row instead of the
  full active shell on every open thread view each minute.
- AGENTS.md: rules for server code that must scale with bkt3's database.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:M labels Oct 21, 2026
bk-agent-01 and others added 2 commits October 21, 2026 16:03
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ute and listLive fallback

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bk-agent-01
bk-agent-01 marked this pull request as ready for review October 21, 2026 16:14
@bk-agent-01
bk-agent-01 merged commit aebd05c into stage Oct 21, 2026
32 of 33 checks passed
@bk-agent-01

Copy link
Copy Markdown
Collaborator Author

Merged as bk-agent-01, pinned to the approved head 7d035d7eb2. Merge commit: aebd05cc31.

  • Go: Tushar answered "Merge when approved" in the session, and @tusharbhardwaj-bk approved 7d035d7eb2 on GitHub.
  • Checks on the head: all passed or were skipped. The one cancelled "Native fingerprint diff" run is an older run, and the newer run of that check passed.

Next: I check stagebkt3 after the deploy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants