Skip to content

feat(wal): advance catalog cursors after source closure - #424

Merged
beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-catalog-advance-v115
Oct 21, 2026
Merged

beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-catalog-advance-v115

Conversation

@beinan

@beinan beinan commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

After a managed Lance publication closes its source pins, the catalog still retains the pending plan and prepared descriptors, preventing the next publication. Add AdvancePublication to atomically advance each selected source's checkpoint after verifying all signed source-quorum closure proofs, then release those dependencies.

A bounded completion receipt preserves exact retries after publisher takeover, a newer pending plan, and checkpoint/restart. Table checkpoint frames preserve their legacy bytes before the first advancement and use a tagged extension afterward. Upgrade every replica before submitting the new command; advanced directories and the extended public Table query encoding do not support older readers. This change performs no physical file or object deletion.

Refs #378.

Testing

  • All-feature WAL unit tests: 100 passed, 1 ignored.
  • Actual TCP/Lance process tests: 10 passed, 1 child fixture ignored, including lost advancement replies followed by checkpointing and restarting all six replicas.
  • Feature Clippy and no-default compilation passed.
  • Full repository helper: 628 Rust tests and 229 Python tests passed (2 skipped, 1 expected failure), plus formatting, workspace Clippy, Ruff and Pyright.
  • Workspace packaging with all features passed.

@beinan
beinan merged commit 73c731c into lance-format:main Oct 21, 2026
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant