Tracking: commit, checkpoint and WAL recovery fixes found by crash and stress testing #1376

This work was produced with the help of language models.

This collects the open fixes for commit, checkpoint and WAL recovery bugs, so they can be reviewed as a set. Each is its own draft PR against main with its own tests; they stay drafts, since one may change without notice to stay compatible with the others. This is a broad experiment aimed at improving durability under pressure. Backward compatibility is out of scope until a working solution is found.

Process. Bugs come from a workload simulator and the stress loop below. Each gets an issue with a deterministic repro, checked on main and the last release, and a fix PR whose tests fail on main. Because one bug can hide another (a segfault after a refused BEGIN ends a workload before a lost commit shows), the open PRs are also merged onto one main commit and stressed against main alone, with the same seeds.

Merged from the same testing: #1195, #1333, #1338, #1355.

Related, outside this set: #1389 fixes #1386, a segfault after a write or checkpoint is refused at max_db_size. #1361 turns that refused automatic checkpoint into a warning, so with #1361 the crash becomes the normal path; #1389 should land with or before it.

Merge order. Each PR is a fix of its own, but some depend on others or change the same lines:

• #1389 with or before #1361: #1361 turns a checkpoint refused at max_db_size into a warning, so without #1389 the next statement segfaults (#1386).
• #1378 before #1379, which is stacked on it and needs a rebase once #1378 is in.
• #1380 after #1378: both change WALReplayer::dryReplay and the replay loops.
• ladybug-python#75 after #1358 is in the engine the driver builds against: with the driver's new lenient default and without #1358, opening with the other checksum setting deletes the WAL.
• #1375, #1358, #1399, #1400 and #1408 at any point.
• #1358, #1378, #1379 and #1380 all change the WAL replay code and conflict in places, and #1400 conflicts with #1380 in WAL::logCommittedWAL; whichever lands second resolves it, as the merged branch of the runs below does.

A safe sequence: #1375, #1358, #1389, #1361, #1399, #1378, #1379, #1380, #1400, #1408, then ladybug-python#75.

Compatibility. Every fix except #1377's keeps the WAL and data-file formats. #1378 changes only the content of the pinned-checkpoint marker file, which older builds check just for existence. #1377's fix adds an ABORT WAL record that older builds cannot read; it lasts only until the next checkpoint retires the WAL.

Stress loop. A C program on the C API: writer threads add relationships tagged with unique ids, with a checkpoint after every commit, and rounds add a full 64 MB buffer pool, COPY, several writers, a write transaction held past the 5 s checkpoint wait, or a file-size limit that fails checkpoint writes. Each round ends with SIGKILL or a clean close; a fresh process then checks that every committed write is there exactly once.

Run 1, main 7bcc47d:

Run 2, #1378 on main 7bcc47d + #1358 + #1361 + #1375, 300 rounds with the same seeds: 2,884 committed writes lost in 42 rounds without it, none with it.

Run 3, #1380 on main 7bcc47d + #1358 + #1361 + #1375 + #1378, 300 rounds with the same seeds: rounds where a write reported as failed was there after reopening went from 4 to none, and no committed write was lost.

Run 4, all six PRs at their current heads merged on main 7bcc47d: 300 stress rounds without a violation, and the workload simulator (sessions, heavy, default and races, 40 workloads of 60 steps each) passed with the rules for these bugs enabled.

Run 5, the workload simulator with a rule for each of these bugs: a kill during the checkpoint after a pinned one, a file-size limit (a full disk), a max_db_size cap, a COMMIT on a full buffer pool followed by a crash, a reopen with the other checksum setting, and a named graph. Each rule fails on a build without its fix: #1316, #1357, #1199, #1396 and #1397 on main, #1304 without #1378, the named-graph case without #1379, #1386 without #1389. The exception is the full-pool COMMIT, which never failed in the simulator, so #1380 rests on its own tests and Run 3. With all eight PRs and #1389 at their current heads merged on main 4ca2237, the five simulator suites (sessions, heavy, default, races, recovery; 40 workloads of 60 steps) passed with every rule enabled, apart from #1406, an older bug (also in 0.21.2) that the recovery suite found. With its fix, #1408, merged too, the recovery suite passed with that rule enabled as well.

#1304: #1378 and #1379.

• A transaction committed after a pinned CHECKPOINT record proves that checkpoint completed. Recovery now skips through the last such record and discards any leftover shadow.
• The pinned-checkpoint marker now means "every CHECKPOINT record in the active WAL is applied". That covers a crash during the next checkpoint and both crash points from the #1304 comment.
• fix(storage): reopen named graphs after a crash around a pinned checkpoint #1379, on top of fix(storage): recover pinned checkpoints that a crash interrupts #1378, covers named graphs, which still failed to reopen (graph shadow file db.g.shadow is missing.). Under the marker they load without their shadows. A pinned checkpoint now also retires graph WALs, since a named graph's commits go to the main WAL.
• Implement WAL based replication #1366's standby replica would rely on the same pinned-WAL state.

#1377: approach. When applying a transaction fails after its COMMIT record is durable (a full buffer pool, or a named graph dropped by a concurrent DROP GRAPH):

• the fix appends and syncs an ABORT record naming that transaction's block in the WAL;
• recovery skips aborted blocks, so a failed COMMIT is never applied, and the database stays writable;
• only if the ABORT record itself cannot be written does the WAL refuse writes until restart.

#1377: rejected alternatives.

• Refuse writes until restart and report the transaction as committed, so the next reopen applies it once. Simpler and needs no new WAL record, but:
  • a transient full buffer pool would stop every writer until a restart;
  • the restart would apply a transaction the client saw fail, and readers would see the rolled-back state until then;
  • it breaks the existing DROP GRAPH race test, whose failed commit recovery already skips.
• Refuse writes only when recovery would apply the failed transaction: not safe, since one transaction can write to both a dropped graph and main.
• Reserve the memory before writing the COMMIT record: not possible; the amount is only known while applying.
• Let the buffer pool overcommit while a commit is applied: removes the out-of-memory cause, but changes pool policy and doesn't cover the DROP GRAPH race. Left as a possible follow-up.

Candidates, not fixed yet:

• Bug: a CHECKPOINT while a read-only transaction is open writes an update to the wrong row #1303: a CHECKPOINT while a read-only transaction is open writes an update to the wrong row; the same pinned-checkpoint code that fix(storage): recover pinned checkpoints that a crash interrupts #1378 changes.
• Bug: dropping the built-in primary-key index is not rolled back and is lost after a crash #1317: dropping the built-in primary-key index is not rolled back and is lost after a crash.
• Bug: Rows replayed from the WAL after a crash are missing from the FTS index and cannot be deleted #1241: rows replayed from the WAL after a crash are missing from the FTS index and cannot be deleted.
• Bug: a partitioned table with a primary key segfaults on a later checkpoint #1407: a partitioned table with a primary key segfaults on a later checkpoint, and after a failed checkpoint accepts duplicate keys; partition children are never finalized after a checkpoint.

To do once these PRs settle:

• Update the user docs' transactions page (ladybug-docs cypher/transaction.md):
  • its Checkpoint section still says a checkpoint needs no active transactions and that CHECKPOINT fails otherwise;
  • it should say that a failed automatic checkpoint no longer fails the commit;
  • it should say what happens when the checkpoint COPY requires fails.
• Release notes:
  • fix(txn): don't fail a committed write when its automatic checkpoint fails #1361: a failed automatic checkpoint no longer fails the commit; it prints a warning on stderr, and a later commit retries it.
  • fix(wal): throw on a checksum-setting mismatch instead of dropping the WAL #1358: opening a database with the other enableChecksums setting is refused and keeps the WAL, where it used to delete it.
  • fix(txn): log an ABORT record when a committed transaction cannot be applied #1380: the ABORT record lasts until the next checkpoint, and older builds cannot read it, so checkpoint on the new build before downgrading.