Skip to content

feat(wal): retire bounded manifest history - #428

Merged
beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-manifest-retention-main-v147
Oct 21, 2026
Merged

beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-manifest-retention-main-v147

Conversation

@beinan

@beinan beinan commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

The managed catalog retains a record for each selected Lance manifest and eventually reaches its bounded history limit. Add explicit retirement that keeps the bootstrap anchor and a contiguous suffix, while protecting the current head, pending plans, confirmed publications and the latest durable advancement receipt. This allows further publications without increasing the index capacity.

Retired version queries and reservation retries return a history boundary, including retries from an old publisher. That response does not authorize deleting the supplied immutable object. This retires catalog metadata only; object cleanup and production reader integration remain separate. All replicas must support retirement before it is enabled because older readers may reject the sparse history.

Refs #378.

Testing

  • All-feature WAL unit tests: 108 passed, including metadata-retirement protection and query/retry boundaries.
  • Actual TCP/Lance process tests: 11 passed. The new case appends through version 162, keeps history at 16 records or fewer, loses a committed reply before restarting all six processes, changes publisher ownership, and verifies every latest payload plus rejection of a retired old-owner retry without deleting its object.
  • Snapshot integration, feature Clippy and no-default compilation passed.
  • Full repository helper: 628 Rust and 229 Python tests passed (2 skipped, 1 expected failure), plus formatting, workspace Clippy, Ruff and Pyright.
  • Workspace packaging and final source/binary integrity checks passed.

@beinan
beinan merged commit 5f4f77c into lance-format:main Oct 21, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant