Skip to content

feat(wal): retire closed source payloads while retaining retry receipts - #416

Merged
beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-source-retirement-final
Oct 20, 2026
Merged

beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-source-retirement-final

Conversation

@beinan

@beinan beinan commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

After a verified publication closes its selected source prefix, release those accepted-record slots and omit the retired bodies from new WAL checkpoints. Keep each producer lane's latest receipt and retirement boundary so an exact retry after restart returns its original ACK. Other tables and unpublished suffixes remain retained.

Checkpoint format v9 validates retained sequence continuity and encodes each retirement boundary in a fixed 52-byte frame. Previously captured checkpoints keep their payload ownership. Physical log/image collection and catalog advancement remain separate work.

Refs #378.

Testing

  • 100 focused Rust tests, including real Lance multi-process recovery, retirement, full restart, original ACK retries and malformed checkpoint boundaries.
  • Full repository helper: 605 workspace Rust tests, 229 Python tests, style/type checks, Clippy and package checks.
  • The 16 changed files are byte-identical to that qualified implementation. Inherited main changes are audited separately; final-head GitHub CI validates the rebased integration.

@beinan
beinan merged commit 02af9c1 into lance-format:main Oct 20, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant