Skip to content

fix: writer-connection starvation (index the finalization ancestry read, retry a failed interrupt poll) - #432

Merged
PHPCraftdream merged 4 commits into
mainfrom
fix/jobbatch-writer-starvation-20261116
Nov 16, 2026
Merged

PHPCraftdream merged 4 commits into
mainfrom
fix/jobbatch-writer-starvation-20261116

Conversation

@PHPCraftdream

Copy link
Copy Markdown
Owner

Writer-connection starvation: index the finalization ancestry read, retry a failed interrupt poll

Long rush run processes died with exit code 70 ("batch runtime fault: jobbatch: commit failed: timeout") and with "rush run: pending interrupt: context deadline exceeded": seven deaths in one day, including a process alone on its own private database. The fault log showed the commit hook waiting the whole 45s budget on the single writer connection (pool_max_open 1, pool_in_use 1, thousands of pool waits, 1566-2306 s cumulative wait), once spending 39.7 s already inside the transaction (phase statements).

Root cause (static reading of the commit, admission and loop paths plus a goroutine dump): one commit acquires the single writer connection many times, and the per-ancestor finalization read SELECT ... FROM batch_nodes WHERE child_session_id=? ORDER BY node_id, run during admission and again inside the commit transaction, had no index: every lookup was a full scan of batch_nodes and its wide payload column. Separately, nextStep ended the whole run on any interrupt-poll error, although the scope read beside it already retries transient store errors.

  • fix(db): partial index idx_batch_nodes_child_session ON batch_nodes (child_session_id, node_id) WHERE child_session_id IS NOT NULL (new forward migration, mirrored into the store_v1 fixture and the finalization fencing schema copy). A plan test pins the SEARCH; without the migration it reports SCAN batch_nodes USING INDEX sqlite_autoindex_batch_nodes_1.
  • fix(run): a failed pending-interrupt poll retries with the same pause and overall bound as the scope read and ends the run only when the bound is exhausted. Tests for a transient and a persistent poll failure; both fail on the previous code.
  • refactor(run): both retries share retryTransientRead, which keeps app_run_async.go under its line ratchet; the plan test lives in an existing file to keep the internal/session file ratchet.

Not in this PR (planned separately, they change snapshot consistency or the watchdog contract): moving the admission pre-read to the reader pool, a priority gate for the commit on the writer, a watchdog budget counted from the first durable write.

Full fork CI run 43917537918 on 742c7d6b8 is green.

🤖 Generated with Claude Code

PHPCraftdream and others added 4 commits November 16, 2026 17:58
…stry read

The per-ancestor finalization read (SELECT ... FROM batch_nodes WHERE
child_session_id=? ORDER BY node_id) runs on the single writer connection
during commit admission and again inside the commit transaction. With no
index on child_session_id it scanned the whole table and its wide payload
column on every lookup; under a growing store that held the writer long
enough to exhaust the 45s jobbatch commit budget, and the process exited
with code 70 (commit_phase "statements", tens of minutes of cumulative pool
waits in the fault log).

A partial index on (child_session_id, node_id) turns each lookup into a
SEARCH. The migration is mirrored into the store_v1 fixture and the
finalization fencing schema copy; a plan test pins the SEARCH (it reports
"SCAN batch_nodes USING INDEX sqlite_autoindex_batch_nodes_1" without the
migration).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… run

nextStep ended the whole `rush run` on any PollExternalInterrupt error
("rush run: pending interrupt: context deadline exceeded"), although the
scope read right below already treats a store error as transient. When the
store is busy past the 10s poll deadline, long agent runs died this way.

A poll error now retries with the same pause and inside the same overall
bound as the scope read (cliDBErrorRetryOverallLimit), visible on stderr;
the run ends only when the bound is exhausted or its own context is done.
Tests: a transient poll failure is retried and the loop goes on; a
persistent one is retried and then ends the run with the poll error. Both
fail on the previous code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…st.go (internal/session file ratchet)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ads (keeps app_run_async.go under its line ratchet)

The scope read and the pending-interrupt poll share retryTransientRead in
app_run_gates.go: same pause, same overall bound, same give-up message shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@PHPCraftdream
PHPCraftdream merged commit c93ccac into main Nov 16, 2026
1 check passed
@PHPCraftdream
PHPCraftdream deleted the fix/jobbatch-writer-starvation-20261116 branch November 16, 2026 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant