Skip to content

fix(server): promote shell reader worker, subagent archive and stuck-subagent fixes to bkt3 - #335

Merged
bk-agent-01 merged 15 commits into
bkmainfrom
promote/stage-2026-10-20
Oct 20, 2026
Merged

fix(server): promote shell reader worker, subagent archive and stuck-subagent fixes to bkt3#335
bk-agent-01 merged 15 commits into
bkmainfrom
promote/stage-2026-10-20

Conversation

@bk-agent-01

@bk-agent-01 bk-agent-01 commented Oct 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

What this promotes

This PR promotes three stage fixes to bkt3 production. The files are the same as stage at 3dc90ea742. The merge of bkmain (#324) into this branch changes no file.

PR Fix Effect on bkt3
#328 Full shell reads (HTTP shell and the websocket shell snapshot) run on a SQLite reader worker. The server no longer freezes while it reads the shell. Production logged 37 freezes in 24 h. The 6.7 s freeze at 11:10 UTC today had two shell reads on the main thread (1.43 s and 1.0 s).
#330 Finished provider subagent threads go into the archive with their parent, with a paced startup pass for old ones. The shell gets smaller. On production, 944 of 1,228 shell rows are finished subagents.
#334 A stopped session marks its open subagents as cancelled. Account rebalance and mode-change restarts wait for up to 60 min while background work runs. Subagents no longer show "Working" for hours after the account service stops their session (124 stops in 7 days, 29 of them hit open subagents).

How I tested

Merge effect

Merging this PR deploys bkt3 and restarts t3-nextbkt3.service. The restart stops every live session on bkt3. Merge at a quiet time.

🤖 Generated with Claude Code

t3dotgg and others added 15 commits October 19, 2026 15:08
@t3dotgg @claude @bk-agent-01
fix(server): thread list reads no longer freeze the server on large d…
b5fae5a 
…atabases (pingdotgg#19226)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Alex Southwell <saphid@gmail.com>
(cherry picked from commit 8659fcf)

Conflict resolution: ShellStream.ts keeps the fork side. This base has no
loadShellSnapshotParts (upstream pingdotgg#17141, 1f31906, is not on stage), so
the reader wrap for the shell snapshot routes lands in the follow-up fork
commit at the routes' own transactions in http.ts and ws.ts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bk-agent-01 @claude
fix(server): fork shell reads run on the SQLite reader worker
bdf129e 
The cherry-picked reader worker (upstream pingdotgg#19226) only moves reads marked
NodeSqliteClient.readOnly. Upstream marks its shell snapshot routes inside
loadShellSnapshotParts, which stage does not have, so this marks the
fork's equivalents and the fork-only shell readers that froze bkt3:

- http.orchestration.loadShellSnapshot (GET /api/orchestration/shell, the
  Linear bridge) and ws.orchestrationV2.loadShellSnapshot (shell
  subscription): the whole read transaction, enrichment after it.
- ws getArchivedShellSnapshot: the whole read transaction.
- ws.orchestrationV2.projectShellItems: the per-thread shell read. Events
  are published after their commit, so the reader already sees them.
- Fork ProjectionSnapshotQuery shell() and threadShell(): only the
  ProjectionStore read, not the provider name lookup. Covers
  threadUsage.get, claudeAccounts.setThreadMode (access guard and owner
  lookup) and the other compatibility callers.

Every marked read is a pure read of committed state. Inside an open
transaction readOnly is ignored and the read stays on the writer. On the
reader a deferred BEGIN does not wait on another process's write lock,
where the writer's BEGIN IMMEDIATE blocks the event loop for up to
busy_timeout (the 5 s setThreadMode stall).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bk-agent-01 @claude
chore(fork): mark the cherry-picked reader worker for the upstream merge
c692f1e 
The fork marker check treats a line as upstream's only when it matches
upstream main. Upstream changed the reader worker after pingdotgg#19226 (pingdotgg#19305,
pingdotgg#19324, pingdotgg#19417, pingdotgg#19425, pingdotgg#19559), so the cherry-picked lines differ from
upstream main even though they are upstream's code.

- nodeSqliteClient.ts: BEGIN/END markers around the three cherry-picked
  regions, saying the next upstream merge takes upstream's side. The code
  stays byte-identical to 8659fcf.
- ThreadPullRequestService.ts: drop the two comment lines that upstream
  itself removed in pingdotgg#19559, so the wrap matches upstream main exactly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bk-agent-01 @claude
fix(server): keep one-thread shell reads on the writer, as upstream does
605efd6 
A review of the reader-worker port found that the fork-only wraps of
projectShellItems' per-event getThreadShell and ProjectionSnapshotQuery's
threadShell() put frequent small reads on the single reader worker, where
each waits behind multi-second full snapshots. After a restart that can
fill the 1,000-event live buffers and make shell streams resubscribe.
Upstream keeps these reads on the writer; only full snapshots move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bk-agent-01 @claude
fix(server): stop the branch rename before cancelling worktree setup (p…
ee73689 
…ingdotgg#18443)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: SAPHID <saphid@users.noreply.github.com>
(cherry picked from commit 62c74a9)

Conflict resolution: ThreadLaunchService.ts keeps the fork's cleanup
condition (`cancelled || !workspaceRecorded`, terminal close inside it).
Stage has no `runReleased` flag, so only the rename fiber handle and its
interrupt are ported. The test applies unchanged.

Why now: "cancels tracked setup before provider work is released" failed
on this branch because the background rename's thread.metadata.update
(worktreePath: worktreeCwd) landed after the cancel's update to null.
The race was already in the code; the reader-worker client adds a few
effect steps per SQL statement, which moved the fiber interleaving.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bk-agent-01 @claude
fix(server): provider subagent threads follow their parent into the a…
dd604f2 
…rchive

Archiving or deleting a thread never archived the provider-native subagent
threads (Claude Task/Agent children and the like) mirrored under it. Every
client hides those threads, but they stayed in every active shell read: on
bkt3 on 2026-10-19, 318 of 1,144 active threads were finished subagents of an
archived parent.

A fork-owned reactor (SubagentArchiveCascade.expbkt3.ts) now:
- archives a parent's provider-native subagent children on thread.archived
  and thread.deleted, and unarchives them on thread.unarchived;
- once per start, after activation, archives every provider-native subagent
  whose parent is archived, deleted or missing, and logs the count.

delegate_task children, forks and children with an unfinished run are left
alone. Candidates come from one index-backed SQL read; commands go out four
at a time. server.ts gains one marked import and one marked layer line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bk-agent-01 @claude
fix(server): pace the startup pass that archives orphaned subagents
17cec6b 
Every archive also starts a history export that runs git in the worktree, so
archiving bkt3's ~318 orphaned subagents at once would burst ~1,300-1,600 git
processes right after a deploy.

The startup pass now queues one orphan per pause (default 2 seconds, set with
`make({ startupPause })`), so the archives and their exports spread over ~11
minutes. A separate fiber does the pacing and feeds the same worker that runs
live cascades. Live cascades therefore run between orphans and are not held
for the whole pass, and nothing races them on the same child. Each orphan is
re-checked when its turn comes and skipped if its parent came back. The pass
logs progress once per 50 orphans, then one summary line.

Tests pass a zero pause, and a new test advances TestClock to show one archive
per pause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bk-agent-01 @claude
fix(server): guard subagent archives on the subagent record and pace …
63fd610 
…live cascades

The "unfinished run" guard could never fire: a provider-native subagent
thread has no run (0 of 777 on bkt3). Its liveness lives in its parent's
subagent record (orchestration_v2_projection_subagents, one per child
thread). An archive now skips a child whose record is pending, running or
waiting, using the child_thread_id index. When a provider-native record
stops being live, its thread is re-checked and follows an archived parent,
so archiving a parent mid-turn no longer leaves its subagents behind until
the next restart. Unarchive no longer applies the guard.

Live cascades are now paced too: a live archive or unarchive waits until the
previous live move is `cascadePause` old (default 500 ms, an option to
`make`). A bulk clean-out of parents with 40-60 subagents each therefore no
longer bursts history exports. Startup orphans still come one per
`startupPause` from the pacing fiber, and they interleave with live work in
the shared worker. Each child is re-read just before it moves.

Tests seed a live subagent record instead of a run. New tests cover a
subagent that finishes after its parent was archived, and a cascade paced on
the TestClock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bk-agent-01
Merge pull request #330 from iamtushar324/fix/bkt3-archive-orphan-sub…
76ca555 
…agents

fix(server): provider subagent threads follow their parent into the archive
@bk-agent-01
Merge pull request #328 from iamtushar324/fix/bkt3-shell-reads-reader…
8772bc8 
…-worker

fix(server): shell reads run on a SQLite reader worker and no longer freeze the server
@bk-agent-01 @claude
fix(server): subagents no longer stay Running after an account restar…
168bd3c 
…t stops their session

The Claude account service stops a thread's provider session between turns
(rebalance, mode change, hard limit). Its idle check looked only at the
parent turn, so it also stopped sessions whose background subagents were
still working, and the session release recorded nothing for them: the
subagent rows, the parent's subagent cards and the child's root turn stayed
running until the parent got another message (84 minutes on 2026-10-20).

- A released provider session now settles the provider-native subagents it
  leaves open (cancelled), for every release reason except server shutdown,
  which startup recovery already handles. A cheap subagent-row check first
  keeps idle releases from paying for the full recovery read.
- Rebalance and mode-change restarts now wait for the session's background
  work, re-checked on the 15 s status poll and capped at 60 minutes.
  Access-revoked and hard-limit stops are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bk-agent-01 @claude
fix(server): lint and test fake for the released-session subagent check
c2876b3 
Use Effect.catchTags (t3code/prefer-catch-tags), give the test store's
getThreadRecords the projection's subagents, and cover an idle release
with no open subagent skipping the full recovery read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bk-agent-01 @claude
fix(server): review follow-ups for released-session subagents and res…
ffa176f 
…tart waits

- Skip a subagent whose provider thread the recovery read did not load: it
  may still run in another shared session.
- A completePendingRestart fiber that lost the race no longer leaves a
  stale background-wait start behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bk-agent-01
Merge pull request #334 from iamtushar324/fix/bkt3-stuck-subagents-on…
3dc90ea 
…-session-stop

fix(server): subagents no longer stay Running after an account restart stops their session
@bk-agent-01 @claude
Merge bkmain into the 2026-10-20 stage promotion
22800b1 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL labels Oct 20, 2026
@bk-agent-01
bk-agent-01 merged commit 806b639 into bkmain Oct 20, 2026
26 of 27 checks passed
@bk-agent-01

bk-agent-01 commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator Author

Merged as bk-agent-01, pinned to the approved head 22800b123310aa6e16e11174092225777e1296c4. Merge commit: 806b6390cc.

  • Go: Tushar answered "Merge now after my approval" in the session, and @tusharbhardwaj-bk approved 22800b1233 on GitHub (2026-10-20 13:32:56 UTC).
  • Checks on the head: all passed or were skipped. "Transfer report artifact" stays queued on the Blacksmith runner, and this job never runs on this repository.
  • This deploy restarts bkt3. A detached watcher measures bkt3 after the restart: freezes, reply times, forks per minute, reader-worker errors and stale subagent rows. I will post the result here.

@bk-agent-01

bk-agent-01 commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator Author

Post-deploy check on bkt3

bkt3 runs 806b6390cc since 13:41:54 UTC (deployed-sha matched at 13:42:17). Health returned 200.

Measurement Before (bkmain b244dbc) After this PR
"event loop stalled" warnings 37 in 24 h; 2 (3.3 s, 6.7 s) in the 10:58–11:13 window 0 from 13:41:54 to 14:09
Reply probe, 4 per second 10:58–11:13: p99 717 ms, max 8,347 ms, 1.22% > 500 ms 13:44–13:54: p50 2 ms, p99 217 ms, max 1,154 ms, 0.05% > 500 ms
"SQLite reader worker unavailable" / errors — 0 / 0
Subagent rows stuck "running" for > 60 min (live threads) 9 (10:47 UTC) 1, an old row from 2026-10-14 whose card is "interrupted" (not shown as running)
Subagent cards that show "running" on live threads — 0
Live threads (not archived) shell had 1,228 rows on 2026-10-19 783
Forks per minute (server process) 380–425 after #326 1,118–6,681 in the first 12 min after the restart (restart recovery and the archive pass), 300 at 14:09

The #330 startup pass archived 165 of 168 orphaned subagent threads between 13:42 and 13:47 UTC. No session was released with open subagents yet, so the new release-subagents log has no entries so far. The freeze-free window is 27 minutes, which is too short to prove the full effect. The 24-hour freeze count will show it.

@bk-agent-01

bk-agent-01 commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator Author

24-hour check on bkt3 (2026-10-20 13:41 → 2026-10-21 14:15 UTC): the result does not hold

Measurement Before #335 24 h after #335
"event loop stalled" warnings 37 in 24 h 98 in 24 h: 1–5 per hour until 10:00 UTC today, then 12–22 per hour. Longest 8.5 s.
5-minute reply probe p99 717 ms, max 8,347 ms p50 2 ms, p90 611 ms, p99 1,071 ms, max 2,972 ms, 12.5% > 500 ms
Subagent cards stuck "running" (60+ min) 9 9, all in one session ("Outbound System Review and Design")
release-subagents-cancelled / -failed — 6 / 0
Desktop connects (ConnectionDriver.connect, 3 in the trace) median 2.3 s 6.6–7.0 s

Cause of the freezes. The trace gaps on the main thread match full shell reads of 2.1–2.2 s that have no parent span (for example, 14:04:57.7, 14:09:27.4, 14:13:27.4 UTC). ForkProviderSessions.listSessions makes them. It reads the full shell (active and archived) on the main thread, then runs one query per thread. ClaudeAccountsService.maybeRebalance calls it at least twice at each turn end of a thread whose account is full or not placeable, before it checks whether another account has room. The freezes rose when the Claude accounts hit their limits after 10:00 UTC (38 hard-limit events, 31 place-none). The reads that #328 moved to the reader worker (HTTP shell, websocket shell, threadUsage.get, PR sweep) do not block the main thread.

Cause of the 9 stuck cards. The hard-limit path opened a new session for the same thread at 12:42:49 UTC (claude.account.placed), before the old session finished its stop at 12:42:53. Fix B skips a thread that a live session serves, so it settled nothing. This is review finding 2 of #334.

This was referenced Oct 21, 2026
Merged
Merged
Merged
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tusharbhardwaj-bk tusharbhardwaj-bk tusharbhardwaj-bk approved these changes

Assignees

No one assigned

Labels

size:XXL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@bk-agent-01 @tusharbhardwaj-bk @t3dotgg