Repository navigation
fix: writer-connection starvation (index the finalization ancestry read, retry a failed interrupt poll) - #432
Merged
Conversation
…stry read The per-ancestor finalization read (SELECT ... FROM batch_nodes WHERE child_session_id=? ORDER BY node_id) runs on the single writer connection during commit admission and again inside the commit transaction. With no index on child_session_id it scanned the whole table and its wide payload column on every lookup; under a growing store that held the writer long enough to exhaust the 45s jobbatch commit budget, and the process exited with code 70 (commit_phase "statements", tens of minutes of cumulative pool waits in the fault log). A partial index on (child_session_id, node_id) turns each lookup into a SEARCH. The migration is mirrored into the store_v1 fixture and the finalization fencing schema copy; a plan test pins the SEARCH (it reports "SCAN batch_nodes USING INDEX sqlite_autoindex_batch_nodes_1" without the migration). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… run
nextStep ended the whole `rush run` on any PollExternalInterrupt error
("rush run: pending interrupt: context deadline exceeded"), although the
scope read right below already treats a store error as transient. When the
store is busy past the 10s poll deadline, long agent runs died this way.
A poll error now retries with the same pause and inside the same overall
bound as the scope read (cliDBErrorRetryOverallLimit), visible on stderr;
the run ends only when the bound is exhausted or its own context is done.
Tests: a transient poll failure is retried and the loop goes on; a
persistent one is retried and then ends the run with the poll error. Both
fail on the previous code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…st.go (internal/session file ratchet) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ads (keeps app_run_async.go under its line ratchet) The scope read and the pending-interrupt poll share retryTransientRead in app_run_gates.go: same pause, same overall bound, same give-up message shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Writer-connection starvation: index the finalization ancestry read, retry a failed interrupt poll
Long
rush runprocesses died with exit code 70 ("batch runtime fault: jobbatch: commit failed: timeout") and with "rush run: pending interrupt: context deadline exceeded": seven deaths in one day, including a process alone on its own private database. The fault log showed the commit hook waiting the whole 45s budget on the single writer connection (pool_max_open 1, pool_in_use 1, thousands of pool waits, 1566-2306 s cumulative wait), once spending 39.7 s already inside the transaction (phasestatements).Root cause (static reading of the commit, admission and loop paths plus a goroutine dump): one commit acquires the single writer connection many times, and the per-ancestor finalization read
SELECT ... FROM batch_nodes WHERE child_session_id=? ORDER BY node_id, run during admission and again inside the commit transaction, had no index: every lookup was a full scan ofbatch_nodesand its wide payload column. Separately,nextStepended the whole run on any interrupt-poll error, although the scope read beside it already retries transient store errors.fix(db): partial indexidx_batch_nodes_child_session ON batch_nodes (child_session_id, node_id) WHERE child_session_id IS NOT NULL(new forward migration, mirrored into the store_v1 fixture and the finalization fencing schema copy). A plan test pins the SEARCH; without the migration it reportsSCAN batch_nodes USING INDEX sqlite_autoindex_batch_nodes_1.fix(run): a failed pending-interrupt poll retries with the same pause and overall bound as the scope read and ends the run only when the bound is exhausted. Tests for a transient and a persistent poll failure; both fail on the previous code.refactor(run): both retries shareretryTransientRead, which keepsapp_run_async.gounder its line ratchet; the plan test lives in an existing file to keep theinternal/sessionfile ratchet.Not in this PR (planned separately, they change snapshot consistency or the watchdog contract): moving the admission pre-read to the reader pool, a priority gate for the commit on the writer, a watchdog budget counted from the first durable write.
Full fork CI run 43917537918 on
742c7d6b8is green.🤖 Generated with Claude Code