Repository navigation
fix: preserve merge demand across running passes and racing flushes - #295
Merged
Merged
Conversation
beinan
marked this pull request as ready for review
October 2, 2026 07:10
beinan
force-pushed
the
codex/durable-merge-demand
branch
from
October 2, 2026 07:25
55d82b6 to
f035e0e
Compare
This was referenced Oct 2, 2026
beinan
added a commit
that referenced
this pull request
Oct 2, 2026
A stalled IndexId, Compact or Repair can retain the table lock needed by WAL merging indefinitely, even when WAL merge recovery is enabled. Give local maintenance on explicitly owned targets bounded execution and the same durable admission/version fence as WAL merges, so a lost master or ambiguous storage write is reconciled before another writer takes over. - Share the exclusive reconciler lease across all table writers. Keep unresolved ownership after task failure; scan durable executions for recovery even without a failure ledger or stats sweep. - Cancel preparation on total/no-progress deadlines, drain manifest commits, and fence uncertain writes before handoff. Keep unfinished leaves supervised when storage recovery fails. - Preserve per-kind retry budgets and backoff through task recreation. Bind only ephemeral maintenance handles to their execution; cached worker handles keep using the current scope. - Keep serial worker fan-out and memory/concurrency budgets. Legacy targets retain their current path; old readers cannot mutate the new local execution records. Defaults: 3600 s total, 600 s without observed completion, 30 s commit drain grace. Long index/compaction steps may have no intermediate progress signal; configure deadlines using actual large-table timings. Existing unfenced writers still require the existing protocol drain before owned activation. No production activation is included. Validation of the combined tree with #294, #295 and #275: all 97 master, 92 worker, 15 coordinator, 20 registry, 2 commit-guard and 4 write-scope tests passed, including ignored etcd fault tests; all-target Clippy passed. Staging validation used an isolated etcd 3.5.16 Pod and the combined Rust test binaries with temporary local Lance datasets. All 27 selected tests passed: stalled maintenance/worker cancellation while other tables progress, lost-master takeover with a real Lance manifest barrier, unavailable storage retaining ownership, persistent backoff, demand retention, and stale admission rejection. This is staging-backed integration/fault coverage, not a deployed-service soak or a large-table Azure storage benchmark. CI is running on the current PR head. Self-review also caught and fixed the interaction with automatic missing-fragment repair: only a committed repair clears the corresponding missing-file failure, allowing the dependent task to resume. Fresh tasks, no-op repairs and unrelated failures retain their budget; regression tests cover both immediate resumption and unrelated budget preservation. Reviewed claim/CAS exclusion, ambiguous admission and finish, expired leases, commit drain/fence ordering, delayed child commits, old-reader compatibility, and recovery inventory pagination. See docs/owned-maintenance-recovery.md for behavior and limitations. --------- Co-authored-by: Beinan Wang <>
beinan
added a commit
that referenced
this pull request
Oct 2, 2026
Store-directory lookups currently fetch Lance manifests on worker cache misses, and registry writes contend on shared Lance datasets. This PR adds an opt-in etcd registry for rollout, generic and datagen stores. Payload storage and the default Lance backend are unchanged. ## Implementation - Introduce `StoreRegistry`, Lance and etcd implementations, and shared etcd connection configuration. Preserve existing worker caches, Lance registry maintenance, and #292's merge recovery and serial fan-out. - Support Lance authority with an etcd migration mirror. Master startup, maintenance, backfill and diff cover all three registry kinds, including datagen. - Reconcile complete Lance snapshots: repair missed deletions and changed URIs, preserve original timestamps, and compare full entries rather than names alone. Per-entry source versions, retained tombstones and a CAS-protected snapshot floor reject delayed older writes; interrupted passes can resume. - Validate all entries before first etcd activation and seal subsequent mirror writes. Refuse direct authoritative writes to an unsealed migration mirror. Reverse mirroring is rejected at startup. ## Migration constraints Deploying this PR does not select etcd. The supported sequence is Lance → Lance with etcd mirror → authoritative etcd without a mirror. Before switching authority, quiesce and drain **all registry writers**, including discovery and retirement in auxiliary masters, reconcile all three kinds and verify empty diffs. The application does not implement a fleet-wide quiescence barrier; an empty live diff does not authorize a rolling backend flip. After activation, Lance is a historical snapshot, not a live rollback copy. Returning to Lance requires a separate verified export/import while writers are quiescent. Retain migration state/tombstones and do not recreate the source dataset at the same URI. See `docs/src/design/registry-etcd.md` for the procedure and partial-activation recovery. ## Validation Regression coverage includes missed mirror deletions, changed URIs and timestamps, stale snapshot/point-write ordering, interrupted reconciliation, cutover refusal/sealing, tombstone replacement by authoritative discovery, reverse-mirror rejection and datagen startup/migration routes. Validated in the combined tree with #294, #295 and #296 using isolated local etcd: - 20 core registry, 97 master, 92 worker, 15 coordinator, 2 commit-guard and 4 write-scope tests passed, including etcd-backed cases. - All-target Clippy, formatting and diff checks passed. - CI runs the core etcd registry regressions and includes them in coverage. - Adapted the new no-op compaction regressions to the registry/config APIs after rebasing on `211ad24`. No registry backend has been enabled or deployed by this PR update. CI is running on the current PR head. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Beinan Wang <>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A worker can request a merge after a running pass has already visited its shard. The scheduler currently deduplicates against that running task and deletes the demand, leaving the WAL waiting for another flush or a coarse stats sweep. An unconditional acknowledgement can also delete a newer flush racing the enqueue.
Retain demand while the deduplicated task is Running. A queued future pass may acknowledge only the exact etcd revision it observed; each new demand advances that revision. Storage remains one coalesced key per target and retry budgets are untouched. Worker fan-out remains serial. The guarantee requires updated demand producers and all scheduler consumers; mixed old consumers retain their old acknowledgement behavior.
Regression tests exercise demand during a running pass with stats sweeps disabled, a later flush racing an old acknowledgement, and preservation of failure backoff. Self-review checked the enqueue/start race, revision acknowledgement, persistent failure budget, pagination, and mixed-version limitations.
Validation of the combined tree with #294, #275 and #296: 97 master, 92 worker, 15 coordinator, 20 registry, 2 commit-guard and 4 write-scope tests passed, including ignored etcd tests; all-target Clippy passed. Staging validation used an isolated etcd 3.5.16 Pod and the combined Rust test binaries, with temporary local Lance datasets: 27 selected tests passed, including demand retention, stalled maintenance/worker recovery, fencing before takeover, and persistent failure budgets. This is staging-backed integration/fault coverage, not a deployed-service soak or large-table Azure storage benchmark. All 10 GitHub CI checks and their workflow runs passed before merging the reviewed head.