Skip to content

Recover stalled local maintenance before resuming WAL merges - #296

Merged
beinan merged 3 commits into
lance-format:mainfrom
beinan:codex/owned-maintenance
Oct 2, 2026
Merged

beinan merged 3 commits into
lance-format:mainfrom
beinan:codex/owned-maintenance

Conversation

@beinan

@beinan beinan commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

A stalled IndexId, Compact or Repair can retain the table lock needed by WAL merging indefinitely, even when WAL merge recovery is enabled. Give local maintenance on explicitly owned targets bounded execution and the same durable admission/version fence as WAL merges, so a lost master or ambiguous storage write is reconciled before another writer takes over.

  • Share the exclusive reconciler lease across all table writers. Keep unresolved ownership after task failure; scan durable executions for recovery even without a failure ledger or stats sweep.
  • Cancel preparation on total/no-progress deadlines, drain manifest commits, and fence uncertain writes before handoff. Keep unfinished leaves supervised when storage recovery fails.
  • Preserve per-kind retry budgets and backoff through task recreation. Bind only ephemeral maintenance handles to their execution; cached worker handles keep using the current scope.
  • Keep serial worker fan-out and memory/concurrency budgets. Legacy targets retain their current path; old readers cannot mutate the new local execution records.

Defaults: 3600 s total, 600 s without observed completion, 30 s commit drain grace. Long index/compaction steps may have no intermediate progress signal; configure deadlines using actual large-table timings. Existing unfenced writers still require the existing protocol drain before owned activation. No production activation is included.

Validation of the combined tree with #294, #295 and #275: all 97 master, 92 worker, 15 coordinator, 20 registry, 2 commit-guard and 4 write-scope tests passed, including ignored etcd fault tests; all-target Clippy passed. Staging validation used an isolated etcd 3.5.16 Pod and the combined Rust test binaries with temporary local Lance datasets. All 27 selected tests passed: stalled maintenance/worker cancellation while other tables progress, lost-master takeover with a real Lance manifest barrier, unavailable storage retaining ownership, persistent backoff, demand retention, and stale admission rejection. This is staging-backed integration/fault coverage, not a deployed-service soak or a large-table Azure storage benchmark. All 10 GitHub CI checks and their workflow runs passed before merging the reviewed head.

Self-review also caught and fixed the interaction with automatic missing-fragment repair: only a committed repair clears the corresponding missing-file failure, allowing the dependent task to resume. Fresh tasks, no-op repairs and unrelated failures retain their budget; regression tests cover both immediate resumption and unrelated budget preservation. Reviewed claim/CAS exclusion, ambiguous admission and finish, expired leases, commit drain/fence ordering, delayed child commits, old-reader compatibility, and recovery inventory pagination. See docs/owned-maintenance-recovery.md for behavior and limitations.

@beinan
beinan marked this pull request as ready for review October 2, 2026 07:12
@beinan
beinan force-pushed the codex/owned-maintenance branch from 593fef2 to 3de3604 Compare October 2, 2026 07:26
beinan added a commit that referenced this pull request Oct 2, 2026
…295)

A worker can request a merge after a running pass has already visited
its shard. The scheduler currently deduplicates against that running
task and deletes the demand, leaving the WAL waiting for another flush
or a coarse stats sweep. An unconditional acknowledgement can also
delete a newer flush racing the enqueue.

Retain demand while the deduplicated task is Running. A queued future
pass may acknowledge only the exact etcd revision it observed; each new
demand advances that revision. Storage remains one coalesced key per
target and retry budgets are untouched. Worker fan-out remains serial.
The guarantee requires updated demand producers and all scheduler
consumers; mixed old consumers retain their old acknowledgement
behavior.

Regression tests exercise demand during a running pass with stats sweeps
disabled, a later flush racing an old acknowledgement, and preservation
of failure backoff. Self-review checked the enqueue/start race, revision
acknowledgement, persistent failure budget, pagination, and
mixed-version limitations.

Validation of the combined tree with #294, #275 and #296: 97 master, 92
worker, 15 coordinator, 20 registry, 2 commit-guard and 4 write-scope
tests passed, including ignored etcd tests; all-target Clippy passed.
Staging validation used an isolated etcd 3.5.16 Pod and the combined
Rust test binaries, with temporary local Lance datasets: 27 selected
tests passed, including demand retention, stalled maintenance/worker
recovery, fencing before takeover, and persistent failure budgets. This
is staging-backed integration/fault coverage, not a deployed-service
soak or large-table Azure storage benchmark. CI is running on the
current PR head.

Co-authored-by: Beinan Wang <>
@beinan
beinan merged commit 34db26f into lance-format:main Oct 2, 2026
10 checks passed
beinan added a commit that referenced this pull request Oct 2, 2026
Store-directory lookups currently fetch Lance manifests on worker cache
misses, and registry writes contend on shared Lance datasets. This PR
adds an opt-in etcd registry for rollout, generic and datagen stores.
Payload storage and the default Lance backend are unchanged.

## Implementation

- Introduce `StoreRegistry`, Lance and etcd implementations, and shared
etcd connection configuration. Preserve existing worker caches, Lance
registry maintenance, and #292's merge recovery and serial fan-out.
- Support Lance authority with an etcd migration mirror. Master startup,
maintenance, backfill and diff cover all three registry kinds, including
datagen.
- Reconcile complete Lance snapshots: repair missed deletions and
changed URIs, preserve original timestamps, and compare full entries
rather than names alone. Per-entry source versions, retained tombstones
and a CAS-protected snapshot floor reject delayed older writes;
interrupted passes can resume.
- Validate all entries before first etcd activation and seal subsequent
mirror writes. Refuse direct authoritative writes to an unsealed
migration mirror. Reverse mirroring is rejected at startup.

## Migration constraints

Deploying this PR does not select etcd. The supported sequence is Lance
→ Lance with etcd mirror → authoritative etcd without a mirror. Before
switching authority, quiesce and drain **all registry writers**,
including discovery and retirement in auxiliary masters, reconcile all
three kinds and verify empty diffs. The application does not implement a
fleet-wide quiescence barrier; an empty live diff does not authorize a
rolling backend flip.

After activation, Lance is a historical snapshot, not a live rollback
copy. Returning to Lance requires a separate verified export/import
while writers are quiescent. Retain migration state/tombstones and do
not recreate the source dataset at the same URI. See
`docs/src/design/registry-etcd.md` for the procedure and
partial-activation recovery.

## Validation

Regression coverage includes missed mirror deletions, changed URIs and
timestamps, stale snapshot/point-write ordering, interrupted
reconciliation, cutover refusal/sealing, tombstone replacement by
authoritative discovery, reverse-mirror rejection and datagen
startup/migration routes. Validated in the combined tree with #294, #295
and #296 using isolated local etcd:

- 20 core registry, 97 master, 92 worker, 15 coordinator, 2 commit-guard
and 4 write-scope tests passed, including etcd-backed cases.
- All-target Clippy, formatting and diff checks passed.
- CI runs the core etcd registry regressions and includes them in
coverage.
- Adapted the new no-op compaction regressions to the registry/config
APIs after rebasing on `211ad24`.

No registry backend has been enabled or deployed by this PR update. CI
is running on the current PR head.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Beinan Wang <>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant