Repository navigation
feat: add etcd store registries with versioned migration - #275
Merged
Merged
Conversation
beinan
force-pushed
the
feat/registry-etcd
branch
2 times, most recently
from
October 2, 2026 03:48
df05f52 to
0d61086
Compare
beinan
changed the title
… a mirrored migration path The store registries (rollout, generic, datagen) are a name -> uri directory every worker consults on a cache miss and writes on every create/delete. As a Lance table each read is a manifest fetch from object storage and each write is two contended commits; even after lance-format#274 keeps it compact, a create costs seconds and throttling turns it into failures. Introduce a StoreRegistry trait (&self, so the RwLock every request went through goes away) with three impls: LanceRegistry (today's table), EtcdRegistry (one key per store under <ETCD_PREFIX>/registry/<kind>/, put-if-absent backfill), and MirroredRegistry (read one, write both, mirror best-effort). REGISTRY_BACKEND / REGISTRY_MIRROR select them; defaults keep Lance with no mirror, so this change is behaviour-neutral until the flags are set. The master seeds the mirror from the primary at startup and heals it every maintenance round, and exposes GET /registry/diff and POST /registry/backfill for verifying a migration step. ETCD_* flags move to core (EtcdConfig) so the server can share them; the master's etcd prefix is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
beinan
force-pushed
the
feat/registry-etcd
branch
from
October 2, 2026 07:26
aeffc85 to
29fa598
Compare
This was referenced Oct 2, 2026
beinan
added a commit
that referenced
this pull request
Oct 2, 2026
…295) A worker can request a merge after a running pass has already visited its shard. The scheduler currently deduplicates against that running task and deletes the demand, leaving the WAL waiting for another flush or a coarse stats sweep. An unconditional acknowledgement can also delete a newer flush racing the enqueue. Retain demand while the deduplicated task is Running. A queued future pass may acknowledge only the exact etcd revision it observed; each new demand advances that revision. Storage remains one coalesced key per target and retry budgets are untouched. Worker fan-out remains serial. The guarantee requires updated demand producers and all scheduler consumers; mixed old consumers retain their old acknowledgement behavior. Regression tests exercise demand during a running pass with stats sweeps disabled, a later flush racing an old acknowledgement, and preservation of failure backoff. Self-review checked the enqueue/start race, revision acknowledgement, persistent failure budget, pagination, and mixed-version limitations. Validation of the combined tree with #294, #275 and #296: 97 master, 92 worker, 15 coordinator, 20 registry, 2 commit-guard and 4 write-scope tests passed, including ignored etcd tests; all-target Clippy passed. Staging validation used an isolated etcd 3.5.16 Pod and the combined Rust test binaries, with temporary local Lance datasets: 27 selected tests passed, including demand retention, stalled maintenance/worker recovery, fencing before takeover, and persistent failure budgets. This is staging-backed integration/fault coverage, not a deployed-service soak or large-table Azure storage benchmark. CI is running on the current PR head. Co-authored-by: Beinan Wang <>
beinan
added a commit
that referenced
this pull request
Oct 2, 2026
A stalled IndexId, Compact or Repair can retain the table lock needed by WAL merging indefinitely, even when WAL merge recovery is enabled. Give local maintenance on explicitly owned targets bounded execution and the same durable admission/version fence as WAL merges, so a lost master or ambiguous storage write is reconciled before another writer takes over. - Share the exclusive reconciler lease across all table writers. Keep unresolved ownership after task failure; scan durable executions for recovery even without a failure ledger or stats sweep. - Cancel preparation on total/no-progress deadlines, drain manifest commits, and fence uncertain writes before handoff. Keep unfinished leaves supervised when storage recovery fails. - Preserve per-kind retry budgets and backoff through task recreation. Bind only ephemeral maintenance handles to their execution; cached worker handles keep using the current scope. - Keep serial worker fan-out and memory/concurrency budgets. Legacy targets retain their current path; old readers cannot mutate the new local execution records. Defaults: 3600 s total, 600 s without observed completion, 30 s commit drain grace. Long index/compaction steps may have no intermediate progress signal; configure deadlines using actual large-table timings. Existing unfenced writers still require the existing protocol drain before owned activation. No production activation is included. Validation of the combined tree with #294, #295 and #275: all 97 master, 92 worker, 15 coordinator, 20 registry, 2 commit-guard and 4 write-scope tests passed, including ignored etcd fault tests; all-target Clippy passed. Staging validation used an isolated etcd 3.5.16 Pod and the combined Rust test binaries with temporary local Lance datasets. All 27 selected tests passed: stalled maintenance/worker cancellation while other tables progress, lost-master takeover with a real Lance manifest barrier, unavailable storage retaining ownership, persistent backoff, demand retention, and stale admission rejection. This is staging-backed integration/fault coverage, not a deployed-service soak or a large-table Azure storage benchmark. CI is running on the current PR head. Self-review also caught and fixed the interaction with automatic missing-fragment repair: only a committed repair clears the corresponding missing-file failure, allowing the dependent task to resume. Fresh tasks, no-op repairs and unrelated failures retain their budget; regression tests cover both immediate resumption and unrelated budget preservation. Reviewed claim/CAS exclusion, ambiguous admission and finish, expired leases, commit drain/fence ordering, delayed child commits, old-reader compatibility, and recovery inventory pagination. See docs/owned-maintenance-recovery.md for behavior and limitations. --------- Co-authored-by: Beinan Wang <>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Store-directory lookups currently fetch Lance manifests on worker cache misses, and registry writes contend on shared Lance datasets. This PR adds an opt-in etcd registry for rollout, generic and datagen stores. Payload storage and the default Lance backend are unchanged.
Implementation
StoreRegistry, Lance and etcd implementations, and shared etcd connection configuration. Preserve existing worker caches, Lance registry maintenance, and fix: opt-in recovery for stalled WAL merges with progress deadlines #292's merge recovery and serial fan-out.Migration constraints
Deploying this PR does not select etcd. The supported sequence is Lance → Lance with etcd mirror → authoritative etcd without a mirror. Before switching authority, quiesce and drain all registry writers, including discovery and retirement in auxiliary masters, reconcile all three kinds and verify empty diffs. The application does not implement a fleet-wide quiescence barrier; an empty live diff does not authorize a rolling backend flip.
After activation, Lance is a historical snapshot, not a live rollback copy. Returning to Lance requires a separate verified export/import while writers are quiescent. Retain migration state/tombstones and do not recreate the source dataset at the same URI. See
docs/src/design/registry-etcd.mdfor the procedure and partial-activation recovery.Validation
Regression coverage includes missed mirror deletions, changed URIs and timestamps, stale snapshot/point-write ordering, interrupted reconciliation, cutover refusal/sealing, tombstone replacement by authoritative discovery, reverse-mirror rejection and datagen startup/migration routes. Validated in the combined tree with #294, #295 and #296 using isolated local etcd:
211ad24.No registry backend has been enabled or deployed by this PR update. All 15 GitHub CI checks and their workflow runs passed before merging the reviewed head. The final main tree matches the locally validated combined integration tree exactly.