Skip to content

feat(wal): collect obsolete application snapshot files - #426

Merged
beinan merged 3 commits into
lance-format:mainfrom
beinan:codex/wal-image-collection-v118
Oct 21, 2026
Merged

beinan merged 3 commits into
lance-format:mainfrom
beinan:codex/wal-image-collection-v118

Conversation

@beinan

@beinan beinan commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Obsolete and abandoned application snapshot files remain on disk after checkpoint replacement or an interrupted transfer. Add a resumable, bounded directory scan that removes unreferenced snapshot files while preserving installed recovery images and captured readers across all groups sharing a node.

Image readers and collection cursors retain the engine's exclusive directory lock, so callers must release them before reopening the node. Ordinary Raft appends can continue during collection. This API does not reclaim Raft-engine segments or Lance objects, schedule maintenance, or enforce a disk quota.

The coverage run also exposed a directory-lock failure in a parallel snapshot reopen test. Isolate independent fork/reopen cases in that test binary and add a deterministic regression showing that an unrelated pre-exec child retains the native directory lock until exec. The original unmodified suite passed 128 local repetitions, so the regression demonstrates the mechanism without claiming to have reproduced the original CI failure. Concurrency and recovery assertions inside each original case remain intact.

Refs #378.

Testing

The following checks passed on the combined current-main tree:

  • All-feature WAL unit tests: 102 passed, 1 ignored.
  • Actual TCP/Lance process tests: 10 passed, 1 child fixture ignored; includes interrupted transfers, physical snapshot-file removal after publication, and complete payload readback after all-replica restart.
  • Snapshot integration suite: 4 passed, 2 subprocess fixtures ignored.
  • Feature Clippy and no-default compilation passed.
  • Full repository helper: 628 Rust tests and 229 Python tests passed (2 skipped, 1 expected failure), plus formatting, workspace Clippy, Ruff and Pyright.
  • Workspace packaging with all features and final source/binary integrity checks passed.

The unchanged WAL source also passed 64 repetitions of the four-test snapshot integration suite before combining with the latest main changes.

@beinan
beinan merged commit 3dacdfe into lance-format:main Oct 21, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant