Skip to content

feat(wal): add shared durable Raft log storage - #399

Merged
beinan merged 2 commits into
lance-format:mainfrom
beinan:codex/wal-durable-log
Oct 20, 2026
Merged

beinan merged 2 commits into
lance-format:mainfrom
beinan:codex/wal-durable-log

Conversation

@beinan

@beinan beinan commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Add an opt-in raft-log storage boundary: one physical engine per node and cloneable raft-rs Storage handles for its static session-sharded groups. Persist entries, term ranges and HardState atomically, then wait for a shared physical sync barrier. Bind the layout and node identity before returning any handles.
  • Reject committed-prefix replacement and term/vote/commit regressions. Look up terms without decoding payloads, and bound each engine range-read request before applying Raft's byte limit. A disk-write error or unwinding sync panic permanently fences every group handle until the physical log is reopened.
  • Cover real files, independent-engine cache isolation, concurrent sync/error paths, Linux child-process write failures and SIGKILL recovery, and the RawNode persistence/Ready contract. Use the vendored cache correction from fix(wal): isolate raft-engine entry caches by engine #398 and run the optional storage tests in CI.

A successful local persist is not a quorum ACK. This increment retains the full log and static membership; transport, snapshots/reclamation, producer deduplication and Lance publication remain follow-up work. It does not switch production ingestion.

Refs #378.

Testing

  • cargo test --locked -p lance-context-wal --features raft-log --lib --test raft_log --test raft_log_cache_identity: 18 passed; the subprocess worker is invoked by its parent tests and ignored as a standalone test.
  • The three unchanged facade cache tests returned exact wrong payloads on the original engine and passed with the vendored correction.
  • cargo clippy --locked -p lance-context-wal --all-features --all-targets -- -D warnings: passed.
  • Full ci-pr-helper checks passed: workspace Rust tests, Python (229 passed, 2 skipped, 1 expected failure), Ruff, Pyright, formatting and Clippy. Workspace package metadata and spelling checks passed.
  • The final term-decoding iteration also passes Rust 1.99 Clippy and all 18 durable-storage tests; the truncated-index check and on-disk representation are unchanged.

@beinan
beinan merged commit 42827bf into lance-format:main Oct 20, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant