Skip to content

feat(wal): reclaim obsolete physical log segments - #427

Merged
beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-log-maintenance-main-v139
Oct 21, 2026
Merged

beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-log-maintenance-main-v139

Conversation

@beinan

@beinan beinan commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

Logical checkpointing leaves obsolete native log segments on disk. Add an explicit node-wide maintenance handle that rewrites retained entries and metadata before reclaiming those segments, plus configurable rotation and purge thresholds.

Maintenance drains writes through a separate gate and syncs a nonempty marker before native reclamation; uncertain errors fence every group sharing the engine. Ordinary reads remain concurrent with ordinary fsyncs. The native maintenance pass temporarily excludes payload reads and persistence and has no per-call byte or time limit; scheduling, admission quotas and application snapshot collection remain separate.

Refs #378.

Testing

  • All-feature WAL unit tests: 105 passed, 1 ignored.
  • Actual TCP/Lance process tests: 10 passed, 1 child fixture ignored; native segments are removed before lagging-replica snapshot transfer, interrupted transfer and all-replica restart.
  • Repeated real-file append/checkpoint/purge tests verify removed paths, retained suffixes and an uncheckpointed sibling after reopen; injected rewrite-sync failure fences every shared group. A blocked ordinary fsync does not block a sibling durable payload read.
  • Feature Clippy and no-default compilation passed.
  • Full repository helper: 628 Rust tests and 229 Python tests passed (2 skipped, 1 expected failure), plus formatting, workspace Clippy, Ruff and Pyright.
  • Workspace packaging with all features and final source/binary integrity checks passed.

@beinan
beinan merged commit 052bc0a into lance-format:main Oct 21, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant