Repository navigation
fix: detect lost merge owners and stop revoked work promptly - #298
Merged
Merged
Conversation
beinan
marked this pull request as ready for review
October 3, 2026 18:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An owned merge could keep holding resources until its long idle deadline after its worker died or its execution was revoked. The master watched only stored progress, and a worker checked ownership only when publishing a new processing checkpoint.
Probe the executor's process identity every 10 seconds, with a two-second request timeout. A changed incarnation or three consecutive failed probes without new progress enters the existing storage-fencing recovery path. Successful probes or completed work reset the failure count. Workers also check ownership every two seconds while queued or executing, so revocation cancels a blocked merge even if the cancel HTTP request never arrives.
Ownership remains protected until admitted storage writes drain or are fenced. Worker fan-out remains serial, retry budgets remain durable, and cancellation targets the individual merge rather than restarting the worker. Legacy targets and the default 600-second idle deadline are unchanged; this does not add pod provisioning.
Validation: all 16 targeted tests passed against an isolated local etcd (10 master, 6 worker), including lost/restarted owners, revoked queue/lock waits, replacement after a barrier, completion racing recovery, progress despite failed HTTP probes, serial fan-out and durable retry budgets. Worker tests were rerun after boxing the supervised future to address Clippy's async layout recursion limit. Formatting, diff checks and local server Clippy pass. All 10 GitHub CI checks passed on
1e6cdcc453708517eaebedd2ce9d6258a3f6319d, including Rust tests, etcd-backed recovery tests, coverage, Clippy, all three platform builds, and Python tests.The live-but-stalled case still uses the configurable 600-second idle deadline. Probe timing bounds detection under a responsive coordinator; they do not bound storage-fence completion. Production activation requires the existing per-table ownership handoff, and process-level termination remains an external watchdog responsibility.