Repository navigation
Expose legacy merge progress and uncertain writes to the master - #418
Merged
Merged
Conversation
beinan
marked this pull request as ready for review
October 20, 2026 22:37
This was referenced Oct 20, 2026
beinan
added a commit
that referenced
this pull request
Oct 20, 2026
…419) Masters currently bypass execution fencing for compact/index/repair on tables outside `MERGE_OWNED_TARGETS`. A stuck operation on a newly hot legacy table can therefore retain an untracked write even though the native recovery machinery exists. Add an opt-in `MASTER_FENCE_LEGACY_MAINTENANCE=true` mode that uses the existing durable admission, real-progress watchdog, commit permits and storage recovery barrier for new rollout/generic master maintenance executions. Prepared compaction/index work keeps its separate preparation phase. The execution scope also observes successful file IO so long scans need not finish an entire phase to demonstrate progress. Recovery scans continue resolving recorded protocol-2 maintenance executions after flag rollback. This does not change worker merge admission, enroll legacy tables into native merging, fence pre-upgrade writes, change datagen behavior, add Pods, or stop ingestion. It is the master-write prerequisite for safe automatic handoff, not a claim that handoff is complete. The flag defaults off for a measured canary rollout. Validation: all 13 maintenance execution tests passed against isolated local etcd 3.7.0, including successful real file reads across multiple idle windows, all three stalled legacy maintenance kinds, version fencing before successor execution, flag rollback with an unresolved barrier, and preserved retry budgets. The source hashes matched the tested commit at completion. Rebased onto main after #418 merged; final-head CI is running. --------- Co-authored-by: Beinan Wang <beinanwang@microsoft.com>
beinan
added a commit
that referenced
this pull request
Oct 21, 2026
) The worker rollout compaction endpoint currently accepts untracked base-table rewrites even when the table is draining or configured for owned maintenance. That bypass can race the master’s fenced recovery and handoff. Apply the existing legacy admission check before opening the table or acquiring its write lock. Owned/draining tables use the master’s fenced maintenance path. The regression test holds a live reader while checking immediate rejection for both protected states, and exercises successful compaction for an ordinary legacy table. Validation: formatting and diff checks passed. The first Clippy run found unused response values in the regression test; the final commit now checks the create status and compaction result explicitly. Rebased onto main after #418 merged. The focused local regression passed on the initial functional patch (1 passed, 0 failed); response assertions were tightened afterward. Final-head CI must pass before merge. This closes an HTTP compaction bypass; automatic hotspot enrollment and durable legacy worker admission remain separate unfinished work. --------- Co-authored-by: Beinan Wang <beinanwang@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Legacy merge requests can time out while a worker is waiting for a slot, still making progress, or has an uncertain storage commit. Today the master records only the HTTP failure, leaving operators unable to identify the execution behind the stall.
Instrument legacy worker HTTP merges and self-merge passes with process-local execution IDs, slot/running/returned/interrupted states, actual work counters, and outstanding/uncertain commit observations. Master obtains a bounded snapshot after an ambiguous legacy RPC failure and puts a small typed summary in the task error. The read-only endpoint does not open stores or contact etcd. Registry/history/response limits explicitly report lost coverage and do not reject work.
The legacy scope is observation-only: it preserves existing cancellation and slot lifetimes instead of spawning detached storage work. Caller cancellation during an admitted commit records uncertainty. These observations never authorize takeover and do not change serial fan-out, fencing, migration admission, or automatic table enrollment. Other processes, old binaries, and evicted history remain outside their coverage.
Validation: five local server tests passed, including sustained concurrent healthy/frozen execution isolation, interrupted/slot-wait states, bounded registry/history, etcd-independent status access, and a real three-generation self-merge proving reclamation without attributing ingestion flushes to maintenance. Final-commit CI test compilation and all three wheel builds passed. Master legacy fan-out, core cancellation/uncertainty, and the remaining CI suites are still running; this is not a deployment report.