Skip to content

fix(wal): refuse writes after a WAL write fails instead of hanging - #1400

Draft
zahariash wants to merge 3 commits into
LadybugDB:mainfrom
zahariash:fix/wal-write-failure
Draft

zahariash wants to merge 3 commits into
LadybugDB:mainfrom
zahariash:fix/wal-write-failure

Conversation

@zahariash

@zahariash zahariash commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

This work was produced with the help of language models.

Fixes #1397.

On a full disk, a write whose WAL append failed reported IO exception: Cannot write to file. ... Error: File too large, and the WAL was unusable after it: the next commit never returned, and closing the database aborted the process. WAL::waitForDurabilityNoLock flushed the writer before entering the try that poisons the WAL when a sync fails, so the failed flush left the WAL unpoisoned and syncInProgress set, and the next commit waited for that sync forever. The bytes stayed in BufferedFileWriter's buffer, and its destructor wrote them when the database closed: a commit the client saw fail could reach the WAL, and when that write failed too, it threw out of the destructor.

Fix.

  • A failed write while appending a commit, flushing it, or flushing a checkpoint record poisons the WAL, as a failed sync does: WAL write failed; database is in a panic state and refuses further writes until restart. Poisoning also clears syncInProgress and wakes the waiting commits. A failed write of the frozen WAL's CHECKPOINT record, which already poisoned the WAL, is now reported the same way instead of as a failed sync.
  • BufferedFileWriter::flush drops its bytes even when the write fails, so neither a later flush nor the destructor writes them. Its other users, the frozen WAL's writer and the shadow file's record trailer, never flush the same writer again after a failure. It also no longer zeroes the whole buffer after every write.
  • As after a failed sync, recovery on the restart (with the default throwOnWalReplayFailure off) drops a torn tail and keeps every commit whose records were written in full. When one flush carries several commits and the write stops partway, an earlier one of them can be kept although its client got the error.

Tests. Five WalTest cases (test/transaction/wal_test.cpp) make the test file system's writes fail. In WALWriteFailurePoisonsWAL the error on main is a bare IO exception and the next commit would wait forever; WALAppendFailurePoisonsWAL fails a commit larger than the writer's buffer while it is appended; in FailedWALWriteIsNotWrittenOnClose closing on main writes the failed commit's 81 bytes into the WAL; CheckpointRecordWriteFailurePoisonsWAL covers the checkpoint record and FrozenCheckpointRecordWriteFailurePoisonsWAL the frozen WAL's. With #1397's script the second write now fails at once, the close finishes, and the database reopens without the failed write. Full ctest: 3191/3191.

Touches WAL::logCommittedWAL, as #1380 does; merging the two keeps both changes.

WAL::waitForDurabilityNoLock flushed the WAL writer before entering the try that poisons
the WAL when a sync fails. On a full disk the flush threw without poisoning the WAL, so
syncInProgress stayed set and the next commit waited for that sync forever. The bytes also
stayed in BufferedFileWriter's buffer: its destructor wrote them when the database closed,
so a commit reported as failed could reach the WAL, and when that write failed too, it
threw out of the destructor and aborted the process.

- A failed write while appending a commit, flushing it or flushing a checkpoint record now
  poisons the WAL, as a failed sync does; poisoning also wakes the commits waiting for the
  sync.
- BufferedFileWriter::flush drops its bytes even when the write fails, so neither a later
  flush nor the destructor writes the bytes of a commit reported as failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zahariash and others added 2 commits October 25, 2026 11:43
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
logAndFlushCheckpointToFrozen poisoned the WAL when the flush of its CHECKPOINT record failed,
but reported a sync failure. That flush now goes through failWriteNoLock like the active WAL's
writes. BufferedFileWriter::flush also stops zeroing its whole buffer after every write: only
the bytes before bufferOffset are written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: after a WAL write fails on a full disk, the next commit hangs and close aborts

1 participant