Verify a backup can be read before restore_backup replaces the database

[cb1kenobi]

⊙ Problem

restore_backup purged the destination database before anything had checked that this build could read the backup (#2965). A backup with a corrupt file, a table format_version from a newer binding, or a transaction log with an unsupported header destroyed the very database it was meant to restore. Reproduced on main: one corrupted file in a managed backup makes backups.restore(… purgeAllFiles) fail on its checksum after the purge, leaving the database without CURRENT. Both the online operation and the offline CLI took that path.

❓ Your call: Is a full stage-then-swap warranted rather than a cheaper preflight? I think yes. backups.verify before the purge would catch only the checksum case; an unsupported table format or log header passes it, and a mid-copy ENOSPC would still destroy the database. The cost is disk for one extra engine copy during a restore. Two of the calls below follow from that cost.

💡 Solution

Restore now stages, proves, then publishes:

1. Check there is room for the copy (assertRoomToStage), then restore the engine into `restore`/<key>.staging, beside the database (stageRestore). Online, this happens while the database is still serving.
2. Validate every staged transaction-log store, then open and stamp the staged copy with this build.
3. Only then replace the database with two renames: the database moves to <key>.replaced, and staging moves into its place.

The restore marker's one-way door moves from the purge to the first rename. Any refusal or failure before it — an unreadable backup, too little space, a symlinked or mount-point database directory, or a .replaced that no marker accounts for — leaves the database untouched, and the error says so (… was not modified). If a restore is interrupted between the two renames, .replaced holds the database and survives reruns until a new copy publishes. If the second rename fails, the database is moved back. The online and offline paths share one implementation.

⚠️ Look hardest: the recovery rules. A rerun treats an existing .replaced as proof that the database path holds only a disposable candidate. That holds only because a finished restore renames .replaced away while its marker still stands. What counts as "nothing destroyed" — the callers' marker branch — rests on publishStagedRestore setting destroyed correctly.

⚖️ Alternatives

• backups.verify(…, { verifyWithChecksum: true }) before the existing purge. Rejected: it can't detect a checksum-valid table in an unsupported format, which takes an open, and it doesn't protect against a mid-copy failure after the purge.
• Restore into scratch, open it, delete it, then purge and restore as before. Rejected: it proves the same thing but copies every restore twice, inside the downtime window, and reads the source a second time after the proof. It would avoid the space refusal below, at the cost of reopening the post-purge failure window.
• Gating on rocksdb_js_version in the archive manifest. Rejected: Identify a get_backup archive with a capability manifest as its first entry #2648 deliberately declined version comparison, and a version gate can't catch corruption or backups written before the manifest carried producer information.
• A non-destructive restore mode in rocksdb-js. Rejected: the binding doesn't own the marker, the blob fence or the generation stamp, so Harper would still have to sequence the swap.

❓ Your call: An online restore now stages while the database keeps serving, so the close comes only after a copy that can take minutes. Writes made during staging succeed and are then replaced by the restore. Before, the database closed first and those writes failed with DatabaseClosingError. Staging after the close instead turns the whole copy into downtime; reordering is a few lines.

❓ Your call: The marker is still written when the restore claims the database, before staging. A crash or restart during a long staging copy therefore leaves a database that was never touched refused at startup as an incomplete restore until someone reruns the restore; the rerun's refusal now says the database may be incomplete rather than that it is. Before, that window was only the close and the 3 s closure poll. Recording a phase in the marker (staging, then publishing) would let startup clear a staging-phase marker that has no .replaced; I left that for a follow-up rather than widen the marker format here.

❓ Your call: A staging copy that would not leave headroom — the larger of 256 MiB and a tenth of the copy — is refused with a 507 before it starts. This holds offline too, so a database larger than about half its volume can no longer be restored in place without freeing space first. The check is also per restore: concurrent restores of different databases on one filesystem are not reserved against each other. An operator override is possible later.

❓ Your call: The proof is a bounded open. With the default table cache, RocksDB loads only part of a large database's tables at open, so an unsupported table outside that set surfaces later, on a cold read. Opening every table (maxOpenFiles: -1) costs one file descriptor per table, which a large database can exhaust and so turn into a refused restore. DESIGN.md states the bound. Tightening it later, for example with a footer scan, would be additive.

❓ Your call: Transaction logs are validated for readability, not strictly. A torn tail passes, because open-time recovery truncates it and the operator has no override to restore past a refusal. Making it strict is one flag.

❓ Your call: A database directory that is itself a symlink or a mount point is now refused before staging, because the swap renames the directory itself. Restores of those layouts that worked before now fail until the configured path points at a real directory on its parent's filesystem. A same-device bind mount still passes the check and fails only at the rename, with nothing lost.

❓ Your call: Blob roots are not staged. They can span filesystems, and the archive's capability tokens already gate their encodings. A blob-copy failure after publication still requires a rerun, as it does today.

🔧 Changes

Product and architecture tour

Where the point of no return moved

What does a failed restore leave behind now?

Before	After
backups.restore(… purgeAllFiles) empties the database directory, then copies. A corrupt or unreadable backup fails after the purge, so the database is gone and the marker demands a rerun that hits the same backup.	The backup is restored into staging and opened by this build first. Only after that does the database move aside. A backup that fails to stage, validate or open, or that will not fit, leaves the database as it was, with a fresh marker cleared.

The database directory is not touched until a staged copy of the backup has opened in this build and every staged transaction-log store has validated.
Once anything is published, the marker stays on every later failure until a rerun succeeds.
.replaced exists only while a publication is unfinished.

Restore states and what a failure leaves

Failures before the first rename clear a fresh marker. Failures after it keep the marker, and .replaced keeps the pre-restore database until a later attempt publishes.

stateDiagram-v2
    [*] --> Marked: lock and marker
    Marked --> Staged: space check, restore into staging
    Staged --> Proven: validate logs, open and stamp
    Proven --> Closed: close broadcast (online) or LOCK probe (offline)
    Closed --> Displaced: database to .replaced
    Displaced --> Published: staging to database path
    Published --> Done: blobs, .replaced to .discarded, clear marker
    Marked --> Untouched: any refusal or failure
    Staged --> NeedsRerun: crash during the copy (database untouched)
    Staged --> Untouched: any failure
    Proven --> Untouched: any failure
    Displaced --> Untouched: second rename fails, rollback fsynced
    Displaced --> NeedsRerun: crash or rollback fails
    Published --> NeedsRerun: blob, fsync or retirement failure
    Untouched --> [*]
    NeedsRerun --> Marked: rerun keeps .replaced, drops the candidate
    Done --> [*]

Loading

• restoreBlocksLoad — a rescan during online staging keeps the database this thread is serving loaded, so the close broadcast can still close it
• prepareRestoreStaging — refuses symlink and mount-point layouts; staging and .discarded are always disposable; under .replaced the database path is a candidate
• publishStagedRestore — moves the database aside, publishes staging, and marks destruction once anything is published
• rollBackPublication — "nothing destroyed" only after both directories are fsynced
• discardReplaced — retires .replaced atomically while the marker still stands

What operators see

Which outcomes does each kind of bad backup or interruption produce?

Example: a backup written by a newer binding

The backup's newest table declares a format_version this build cannot read. Restore copies it into staging; the staged open fails with Corrupt or unsupported format_version.

Outcome: The restore fails with Backup <id> could not be staged and verified, so <dir> was not modified: …, and the database stays loaded and writable, with its subscriptions running.

Example: the process dies between the two renames

The database path is empty and .replaced holds the database; the marker blocks loading. A rerun with an unreadable backup fails and keeps both, and a rerun with a good backup publishes and then retires .replaced.

Outcome: The only copy of the database is never deleted before a replacement has been published.

What this does not prove

Where does the guarantee stop?

The proof is an ordinary bounded open, so a large database's tables beyond RocksDB's initial load set are not read until later. Transaction logs are checked for readability, not completeness: a torn tail passes. Blob roots are not staged, so a blob-copy failure after publication still needs a rerun, as before. The space check is per restore. Each of these is a ❓ Your call above.

Files:

• dataLayer/restoreStaging.ts (new) holds the protocol: preparation, including the symlink and mount-point refusals the refusal of a .replaced no marker accounts for, and clearing .discarded; the space check and its headroom; staging with the mode taken from the database or, during recovery, from .replaced; the refusal message, which only claims "not modified" when that is true; publication; rollback; and cleanup of .replaced and staging.
• dataLayer/rocksdbBackup.ts routes online restoreBackup and offline restoreBackupOffline through it:
  • Both retire .replaced inside the protected section, keep the marker whenever publication happened, and drop staging on failure.
  • Online, a rerun of an unfinished publication verifies closure before preparation drops the candidate it left.
  • A rerun-required failure is wrapped with its status code kept.
  • Offline, the RocksDB LOCK probe is extracted into assertNotOpenElsewhere and runs before staging and again before publication.
  • The generated backup README and the docblock describe the new order.
• dataLayer/restoreMarker.ts adds the staging, replaced and discarded paths inside the existing reserved metadata directory, which the startup scan already skips, and documents them in the module header.
• resources/databases.ts: online staging holds the marker for the whole copy, and a schema rescan in that window used to unload the marked database without closing its handle, which left the close broadcast nothing to close and made the closure check refuse with a misleading 409. Both scan loops now go through restoreBlocksLoad (main directory, configured paths), which keeps a marked root this thread already has open; one it has not opened is still skipped. The "in progress; not loading it" warning is skipped for a database that is still serving.
• server/itc/serverHandlers.js updates a comment that described the purge.
• .github/workflows/review-coverage.yml lists the new module in framing_paths, as the review-coverage policy test requires for every production module that imports a storage binding.
• dataLayer/DESIGN.md records the invariant and recovery rules, the bound on the proof, the space check and its limits, the symlink and mount-point refusals, the rescan rule during staging, and why .replaced is retired by rename. It also updates the bullets that described the purge: stamping, closure, drop and the offline probe.

✅ Verification

End-to-end route: a new integration test, plus unit tests that drive the real engine with real damaged backups.

• unitTests/dataLayer/rocksdbBackup.test.js adds a "restore staging" suite (21 tests): npx mocha unitTests/dataLayer/rocksdbBackup.test.js -g "restore staging" → 21 passing.
  • Online and offline, the restore refuses a corrupt backup, a checksum-valid SST with format_version 99, and a transaction log with an unsupported header version, with the destination intact and no marker. These fail on main, where the database is destroyed or the unreadable log is restored.
  • Rollback when the second rename fails, online and offline, and an online restore of a valid backup.
  • Refusals before staging: a copy that would not fit (507, online and offline), a mount point and a symlink.
  • Subprocess SIGKILL between the renames, followed by a failed rerun (.replaced and the marker are kept) and a good rerun (.replaced is retired, mode kept).
  • Online: a schema rescan during staging still lets the restore close and publish (409 before the fix), and a rerun refuses to drop a candidate something holds open. A .replaced no marker accounts for is refused with the message that names it, online and offline.
  • The marker is kept when a restore into a new target_database fails after publication and when .replaced cannot be retired. A rerun after a failed publication drops the candidate before the space check. The directory mode survives the swap.
• Guards checked by deleting them from dist/: the rollback, keeping .replaced on rerun, marking destruction after publication, the mode from .replaced, the mount-point and space refusals, dropping the candidate, retiring .replaced by rename, the closure check before a rerun drops its candidate, the preexisting-marker gate on that check, and the refusal of an unaccounted .replaced. Each deletion fails exactly the test that covers it; the rescan test failed before its fix.
• integrationTests/database/restore-backup-staging.test.ts (new, 3 workers) → 3/3 passing at this head; on main, 3/3 fail. It covers a refused online Operations API job, a refused offline CLI restore that exits non-zero, and an offline CLI restore through the swap that survives a restart. The success path runs offline because of a separate, pre-existing bug found here: #3120. A runtime schema change leaks RocksDB handles, so an online restore_backup of a database whose schema changed at runtime is always refused while the server runs. The unit suite covers online success.
• A probe on this branch confirmed that a restore refused before the close leaves subscriptions running: the listener received a write made after the refusal.
• Full gates on the final code, before the merge from main:
  • test:unit:backup: 176 passing.
  • restoreGeneration: 3 passing.
  • test:unit:main, with applicationSpawn excluded because it wedges on main too: 6788 passing and 17 failing. That is the same 17 failing tests as with main's restore sources, in unrelated suites (module tracking, preload resolution).
  • test:unit:resources: 4058 passing and 4 failing. Three are a flush-timing premise test that also fails with main's sources. The fourth, sourceTxnStreams, timed out under machine load and passes 11/11 when run alone.
  • lint, format:check, typecheck and check:design-docs are clean.
• After merging main: test:unit:backup, restoreGeneration and the integration test pass again, and lint, format and typecheck are clean. The review-coverage action tests pass 143/143 locally after the framing_paths addition.
• unitTests/resources/databases.test.js: the startup-scan test for an empty restoring marker now closes its database first, as a real startup has nothing open; with the database still open, restoreBlocksLoad keeps it loaded by design. 36 passing.
• After the latest review rounds: test:unit:backup 179 passing; test:unit:resources 4071 passing, 0 failing; lint, format, typecheck and check:design-docs clean. test:unit:main was not rerun after them.
• Not run: the TypeStrip test mode, which is broken locally on main.

Docs: HarperFast/documentation#721 Document that restore_backup checks a backup before replacing the database covers staging, the space requirement, the symlink and mount-point refusals, and the 5.3 release note.

_{🤖 Generated by Anthropic Claude (Opus); posted via @cb1kenobi.}

Related PRs: #2649 overlaps (this narrows its scan-level restore skip for a root this thread already has open), 2 others independent
Complexity: complicated

_{Review-Coverage: authored=claude; ran=gemini,codex,cursor-composer,cursor-muse,cursor-grok; adjudicated=domain; declined=cursor-kimi; rounds=12; full=6 @ bf1e140}

_{Review-Attention: deep ~60m (critical: restoreMarker.ts, rocksdbBackup.ts +1; sensitive: review-coverage.yml; decisions: do-less-alternative, stage-while-serving, keep-loaded-under-any-marker, refuse-symlink-and-mount-point, space-headroom-policy; raised: degraded review) @ bf1e140}

[github-actions]

Release cherry-pick v5.3: failed

The cherry-pick job for v5.3 did not complete. It is unclear whether v5.3 was updated — check before re-running.

Run: https://github.com/HarperFast/harper/actions/runs/37890866919

Re-run once the cause is addressed (workflow_dispatch, pr_number=3121).