Skip to content

fix: detect lost merge owners and stop revoked work promptly - #298

Merged
beinan merged 3 commits into
lance-format:mainfrom
beinan:codex/prompt-merge-takeover
Oct 3, 2026
Merged

beinan merged 3 commits into
lance-format:mainfrom
beinan:codex/prompt-merge-takeover

Conversation

@beinan

@beinan beinan commented Oct 3, 2026 •

Copy link
Copy Markdown
Collaborator

An owned merge could keep holding resources until its long idle deadline after its worker died or its execution was revoked. The master watched only stored progress, and a worker checked ownership only when publishing a new processing checkpoint.

Probe the executor's process identity every 10 seconds, with a two-second request timeout. A changed incarnation or three consecutive failed probes without new progress enters the existing storage-fencing recovery path. Successful probes or completed work reset the failure count. Workers also check ownership every two seconds while queued or executing, so revocation cancels a blocked merge even if the cancel HTTP request never arrives.

Ownership remains protected until admitted storage writes drain or are fenced. Worker fan-out remains serial, retry budgets remain durable, and cancellation targets the individual merge rather than restarting the worker. Legacy targets and the default 600-second idle deadline are unchanged; this does not add pod provisioning.

Validation: all 16 targeted tests passed against an isolated local etcd (10 master, 6 worker), including lost/restarted owners, revoked queue/lock waits, replacement after a barrier, completion racing recovery, progress despite failed HTTP probes, serial fan-out and durable retry budgets. Worker tests were rerun after boxing the supervised future to address Clippy's async layout recursion limit. Formatting, diff checks and local server Clippy pass. All 10 GitHub CI checks passed on 1e6cdcc453708517eaebedd2ce9d6258a3f6319d, including Rust tests, etcd-backed recovery tests, coverage, Clippy, all three platform builds, and Python tests.

The live-but-stalled case still uses the configurable 600-second idle deadline. Probe timing bounds detection under a responsive coordinator; they do not bound storage-fence completion. Production activation requires the existing per-table ownership handoff, and process-level termination remains an external watchdog responsibility.

@beinan
beinan marked this pull request as ready for review October 3, 2026 18:03
@beinan
beinan merged commit db35d9c into lance-format:main Oct 3, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant