Skip to content

fix: reclaim client journal rewrite bytes before fencing - #430

Merged
beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-journal-byte-pressure
Oct 21, 2026
Merged

beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-journal-byte-pressure

Conversation

@beinan

@beinan beinan commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

Large value replacements among many small cold keys can fill the client journal even when most retained bytes are obsolete: raft-engine's normal rewrite trigger counts stale entries rather than their bytes. Before fencing a batch for insufficient disk headroom, attempt one synchronized rewrite-queue reclamation using the existing reserve, measure the directory again, and retain the fence if space is still insufficient or maintenance fails.

The binding, file format, configured budgets and durable completion barrier stay unchanged. Add a pressure-rewrite counter and regressions for continuous mixed-size replacements, sync failure, and SIGKILL immediately before and after the rewritten-data sync. The fallback is synchronous and can add latency to the triggering batch.

Refs #378.

Testing

  • Full repository ci-pr-helper checks: Rust formatting, workspace Clippy/tests, Python tests, Ruff and Pyright.
  • Client-journal feature tests and Clippy, including the final fault-hook changes.
  • The mixed-size regression completes 40 rounds without reopening; the previous implementation fenced in round 7. Recovery compares every acknowledged latest value.

@beinan
beinan merged commit e94b707 into lance-format:main Oct 21, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant