Skip to content

feat(wal): recover replicated state through application snapshots - #404

Merged
beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-application-checkpoint
Oct 20, 2026
Merged

beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-application-checkpoint

Conversation

@beinan

@beinan beinan commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

A lagging retained replica needs complete application state before Raft can accept a compacted prefix. Add explicit application checkpoints and bounded snapshot transfer to the optional Rust replication component. Checkpoints preserve opaque accepted payloads and exact producer retry state; reopen restores the selected image and replays the committed suffix.

Builders and receivers run outside the Raft owner under one shared physical-node permit. Immutable files are length/hash validated and synced before their descriptor can cross the Raft storage boundary. An interrupted transfer leaves unselected files; it cannot expose partial state or release a snapshot response.

This version retains all accepted records and copies retained payloads into full images. Publication-aware retirement, physical collection, dynamic voter replacement and production transport/admission remain separate work. It does not change the existing ingestion path or enable periodic compaction.

Refs #378.

Testing

  • 46 focused Rust tests passed, including corrupted/truncated images, exact producer tails, append progress after a captured cut and shared sync-failure fencing.
  • Real Rust TCP child processes exercise a kill after file sync but before install, interrupted follower transfer, retry with fresh file identities, complete process restart, exact payload readback and receipt deduplication.
  • All-feature WAL Clippy passed.
  • Complete repository CI helper passed: workspace Rust formatting, Clippy and tests; Python tests (229 passed, 2 skipped, 1 expected failure), Ruff and Pyright. Server build and workspace packaging passed; validated source and server checksums remained unchanged.

@beinan
beinan merged commit 91c98c3 into lance-format:main Oct 20, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant