Repository navigation
fix(scanner): write the state file atomically; never silently replace one that cannot be restored (SNB-654) - #31
Merged
Conversation
…SNB-541) The official vectors only reached Scanner::scan_transaction_full, which has no production caller. The daemon receives through scan_short_block_data -> scan_transaction_short (k = 0, labels enumerated by hand, 8-byte x-only prefixes) and, on a probable match, apply_block_relevant on the external indexer. A false negative in the short step means the block is never fetched: silent loss. - Serve every receiving vector the way the oracle does (taproot outputs' 8-byte x-only prefixes, the 33-byte tweak, the display-order txid) and assert a probable match wherever the vector expects >= 1 output. Pins which k = 0 branch did the work: unlabelled P_0, labels of both output parities, and the change label m = 0. False positives are counted and reported (zero on these vectors). - Drive the real post-match step for the same vectors: the vector tx in a full block through apply_block_relevant, asserting the stored outputs and priv_key_tweaks equal expected.outputs. - Extract the post-match step (txid mapping + apply_block_relevant + stage merge) from scan_block_range into Scanner::apply_matched_block, unchanged, so the test exercises the production code rather than a copy. - K_max through the live path is SNB-540: an ignored test records that it stores 2324 outputs where BIP-352 allows 2323. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The scanner's `owned_outputs` set, which the per-block spent-output check matches the oracle's 8-byte `spent_outputs` prefixes against, had a writer (`add_owned_output`) that no call site ever invoked. The set stayed empty, the spent check never fired, and a block whose only wallet-relevant transaction was a spend (no Silent Payments output back to the wallet) was never fetched. The spending transaction never reached the wallet graph, so blindbit-cli and friglet reported a balance that still included spent coins and Electrum clients never saw the spend. Spends were only noticed by accident, when the spending transaction also paid the wallet (e.g. change). - `Scanner::sync_owned_outputs` rebuilds the owned-output records from the indexer after every applied block and on restore: outpoint, full x-only key, amount, height, and the BIP-352 label `m` the output matched. - A record is marked spent only when a confirmed transaction in the wallet graph spends exactly that outpoint. The 8-byte prefixes, matched through a prefix -> unspent-outpoint map, only decide which blocks to fetch; a foreign key sharing the prefix fetches the block but marks nothing. - The state file's `owned_outputs` array now holds these records (label and spend included). Legacy bare-hex entries are accepted and dropped; the records are re-derived from the indexer on load. - Restore registered labels only after re-indexing the stored transactions, and `load_from_file` blanks the persisted label table, so every labelled output (change included, m = 0) silently dropped out of the index, the balance and spend detection after each restart. Labels are now registered before the changeset is applied. - `apply_matched_block` is extracted from `scan_block_range` (same shape as in #17) so the tests drive the real per-block path without P2P. Tests (in-process, `owned_outputs_tests.rs`): receive then spend in a later block marks the output spent and the balance drops; a colliding foreign prefix marks nothing; records, labels and spends survive save/load/restore and spend detection keeps working after it; a labelled receive records the right m. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… messages Adds an in-process seam (BlockScanDataStream) between scan_block_range and the scan loop, with no behaviour change, and tests that drive a real Scanner through it. They fail on this commit: an empty-hash block mid-range is treated as an empty block (payments in it are lost and the scan reports success), height gaps and short streams are accepted, and any error status panics the process via unwrap(). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ipping it An oracle answers a height it has not indexed with an OK message carrying an empty block hash and no data. scan_block_range took that as a block with no payments: it warned, advanced the scan height past it and reported success, so any payment in that block was never found (proven by the testkit mutation check unindexed-ok-empty: 80000 sats reported instead of 120600). Separately, every stream message was unwrap()ed, so an error status (NOT_FOUND, UNAVAILABLE) panicked the process, as did failing to open the stream. Now each message must be the next expected height and carry a valid 32-byte, non-zero block hash; otherwise, and on any error status or a stream that ends before the requested end, the scan returns an error naming the height to rescan from. The scan height stays at the last fully processed block, so watch_chain retries from there on its next poll (every 10 s, no spin) and blindbit-cli exits non-zero after saving that height. The example is updated the same way. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ix/reorg-rollback
…point The live scanner had no reorg handling: after a reorganisation it kept every output and spend learned from disconnected blocks. An evicted payment stayed in the balance as a phantom output, a payment that re-confirmed at another height kept its old height, and an output whose spend was disconnected stayed marked spent. The scanner now remembers the block hash of each scanned height for the last REORG_LOOKBACK (144) heights, persisted in the state file as `scanned_block_hashes`. A range scan that continues the scanned chain streams one block early and compares it; watch_chain also runs that check when the oracle has no new block, which catches a same-height tip replacement. On a mismatch the scan walks up from the oldest remembered hash to find the fork point. It rolls back everything above it: graph transactions and anchors from disconnected blocks with their tweaks, the owned-output records and spent marks derived from them, checkpoints, remembered hashes, and Electrum history/SP history/headers. It persists that state and scans the new branch from the fork. A fork at or below the oldest remembered hash is a ReorgTooDeep error. That error changes nothing and repeats on every attempt, and it tells the user to rescan from the wallet birthday. friglet drops its snapshotted checkpoint hashes and cached headers above the fork (and rewrites the header sidecar) so it never serves a header of a disconnected block. blindbit-cli goes through scan_block_range and gets the same check. Tests drive the stream seam with an in-process oracle, mirroring the testkit T2 reorg scenario (fork at 104, 2 disconnected, 3 mined): eviction, re-confirmation at a new height, a disconnected spend, a same-height tip replacement, persistence plus restart mid-rescan, and a reorg deeper than the lookback. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…d-labels
Resolve scanning.rs: keep sync_owned_outputs() after apply_matched_block,
keep both test-module lines, and drop the dead
`if probable_match.spent {}` block that #17's extraction carried over.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s below it; report stalls A wallet whose start height lies below the oracle's first indexed block (the hosted oracle has nothing below 800000 on mainnet and 100000 on signet) stopped at its first height on every poll, forever, with only an error-level log line. Since #20 refuses to scan past an unindexed height, that is the only thing such a wallet ever did. - When `watch_chain`'s scan stops at the wallet's start height because the oracle has not indexed it, the scanner locates the oracle's lowest indexed height with GetBlockHashByHeight lookups and starts there, logs a warning, persists the adjustment (`oracle_floor_start`, optional in the state file) and publishes it in the scan health. - Only heights below the oracle's floor are skipped. A height above the first scanned block, or one with an indexed height below it (a gap in the oracle's index), keeps #20's stop-and-retry. The search is a binary search verified by exponential back-off probes below the boundary, so a gap above the floor does not move the floor up. NOT_FOUND and an empty or zero hash both mean "not indexed". - A scan that cannot get past a height (or cannot reach the oracle) is published as a stall: height, reason, and the time of the first failed attempt at that height. It clears when the scan makes progress. - Stream stops are a typed `ScanStopped` error (same message text) so the loop can tell "not indexed" from other failures. - Health lives in the wallet's Electrum index, which status readers lock without waiting on the scan task. friglet's GetStatus reports it as `StatusInfo::scan_health` (serde default, so old trays and daemons interoperate), and the tray's status window shows a stall line and a note for an adjusted start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd-labels #20 lands before this PR, so this branch carries it to merge cleanly after it. changeset.rs: keep both imports (#20's OracleFloorStart and this branch's OwnedOutputRecord); the serde_hex module #20 still had is the one this branch replaced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ends it missed A state file written before owned outputs were recorded still loads, but its wallet never fetched blocks whose only wallet-relevant transaction was a spend, so those spends stay missing and nothing said so. - The state file now carries `format_version` (STATE_FORMAT_VERSION = 1; absent = 0, written before the owned-output records). Every save writes the current version. - Restoring a format-0 state rewinds the scan to just below its oldest unspent output, the lowest height a missed spend can be at. The next scan revisits those blocks and, with the outputs now recorded, fetches the ones that spend them. Re-applying blocks already in the graph is idempotent. With no unspent output there is nothing to re-check and only the version is bumped. - The rewind is logged and published in the scan health (`StateRescan`), shown in friglet's status and the tray, and cleared once the scan is back where the state had been. While it runs, the Electrum tip stays at that height, so clients never see wallet history above the tip. - Test fixture `testdata/state_before_owned_outputs.json` was written by the pre-fix code (integration 93f6a92): payment + change at 100, scanned to 300, spend at 200 missing. The test loads it as written and with legacy bare-hex `owned_outputs` entries, checks the restored records and labels, the rewind to 99, the held Electrum tip, that the rescan finds the spend, and that the upgraded file restores without another rewind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fix/reorg-rollback Keeps this branch a superset of the stack below it. - changeset.rs / scanner.rs: keep both sides' new fields (scanned_block_hashes; format_version, oracle_floor_start). - scanning.rs: keep this branch's scan_range_from / BlockStreamSource; OracleStreams::open now reports a refused stream as #20's typed ScanStopped. watch_chain combines both: the no-new-block hash check from here, and #20's floor start (only for a real range) and stall reporting, with the ReorgTooDeep log kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ing blocks; catch mid-stream oracle reorgs The reorg check streamed the tip block again on every idle poll (~45 KB of compute index every 10 s on mainnet, ~385 MB a day per friglet and the same egress on the oracle), streamed the last scanned block again in front of every new block, and located a fork by streaming the whole 144-block lookback window. - The block below a range is checked with one GetBlockHashByHeight lookup against the remembered hash. An idle poll streams nothing; a new block is streamed once. An empty hash or NOT_FOUND means "cannot check yet", not a reorganisation. - A disagreement is located with a binary search over the remembered heights (lookups only), and the rollback stream starts at the fork point. The oracle disagreeing at the oldest remembered height is ReorgTooDeep without streaming anything, so a halted wallet costs two lookups per poll. - The short stream carries no previous-block hash, so an oracle that switches branches while a stream runs could hand over blocks of both and nothing noticed. When a stream ends, its newest blocks (POST_STREAM_CHECK = 6) and the block below them are checked again; a disagreement is rolled back like any other reorganisation (bounded to 3 restarts per scan). - The per-poll body of watch_chain is `watch_step`, generic over the oracle, so the tests drive the real loop: a ReorgTooDeep is published as a stall (reason says to rescan from the wallet birthday), like any other scan that cannot proceed. - record_scanned_block_hash no longer copies the 144-hash window into the staged changeset on every block (save_to_file writes the current window), and trims the window in place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n attribute) The status window's error, stall and scan-notes blocks use class="col", whose display: flex beats the user-agent [hidden] rule, so an empty "Last error" label was always shown and the new stall/notes blocks would be too. Seen in a screenshot against a live daemon. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd-labels Picks up the tray CSS fix for hidden status blocks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Picks up #20's tray CSS fix for hidden status blocks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… one that cannot be restored
save_to_file wrote the state in place, so a crash or power loss during a
save could leave a truncated file. load_scanner then treated any file it
could not parse or restore as absent: it logged a warning, created a
fresh scanner, and that scanner's first save overwrote the old file.
- save_to_file writes `<state>.tmp` (created 0600: the file holds the
scan secret), fsyncs it, renames it over the state file and fsyncs the
directory. A failed write leaves the previous file untouched and
removes the temporary file; a temporary file left by a crash is
replaced. A new state file is no longer created world-readable until
friglet's next start tightens it.
- load_scanner (via restore_or_create):
- no state file: new scanner, as before;
- a state file that cannot be read (permissions, I/O): error, nothing
is moved or created;
- a state file that cannot be restored (invalid JSON, missing keys, or a
format_version newer than this build knows): moved to
`<state>.unreadable-<unix time>`, never overwritten; a new scan starts
from the configured start height; the error is logged and published as
ScanHealth::state_file_reset, shown in friglet's GetStatus
(`scan_health.state_reset`) and as a note in the tray.
- The oracle connection is opened once for both paths (previously each
path connected on its own; failure behaviour is unchanged).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger
added a commit
that referenced
this pull request
Oct 1, 2026
…ood/preview Brings in the #17/#20/#19/#21/#31 stack. One conflict, in friglet/src/electrum/mod.rs, resolved keeping both sides: - #26's ChainSource (p2p/network/oracle) and pending-broadcast store, - #21's reorg handling: checkpoints behind a mutex, reorg_rx feeding forget_blocks_above; its test now builds the server state with #26's test helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since ScanStopped became a struct (22c50bf), blindbit-cli printed a stopped scan as `Error: ScanStopped { height: 104, reason: "...", not_indexed: true }`, the Debug form main uses for a returned error. The message the type renders for the user, "scan stopped at height 104: ...; nothing from height 104 on was scanned, rescan from height 104", no longer appeared. The testkit's mutation checks match that message and went red for stale-view, truncated-stream, close-mid-stream and unindexed-ok-empty. main now runs the command and prints the error with Display. The exit status of a failed command is still 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd-labels Brings in the blindbit-cli error display fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in the blindbit-cli error display fix from #20. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in the blindbit-cli error display fix from #20. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Stacked on #21 (which contains #17, #20 and #19). Land #17 → #20 → #19 → #21 first; after that this PR's diff is only its own commit. It cannot sit on
integrationalone: the error is reported through #20's scan-health status, andsave_to_file/ the state format come from #19 and #21.Implements SNB-654.
What was broken
Scanner::save_to_filewrote the state file in place (std::fs::write). A crash or power loss during a save can leave it truncated.load_scannertreated a state file it could not parse or restore as absent. It logged awarn, built a fresh scanner, and that scanner's first save overwrote the file. The wallet state was gone, with nothing in the status. The same path is taken when an older build reads a newer-format file (see fix: record owned outputs so spends are detected; persist labels #19's downgrade note).-rw-rw-r--in this PR's smoke runs), although it holds the scan secret. friglet only tightened it at its next start or onsave_state.Fix
Atomic save (
scanner/state_file.rs). The state is written to<state>.tmp, created0600, thenfsynced and renamed over the state file, and the directory isfsynced (Unix, best effort). If any step fails, the previous file is untouched and the temporary file is removed. A temporary file left behind by a crash is replaced.Load (
load_scanner→restore_or_create):from_changesetfails<state>.unreadable-<unix time>(never overwritten), new scan from the start height,error!log,ScanHealth::state_file_resetformat_versionnewer than this buildVisible. friglet's
GetStatuscarries the reset asscan_health.state_reset(backup_path,error). The tray's status window shows a note: "The state file could not be read (…). It was moved to …, and the wallet is being scanned again from its start height."The oracle connection is opened once for both paths. Before, each path connected on its own. A connect failure is still an error, as before.
Cost
An
fsyncper save. Measured on this VM's disk, average of 40 runs each:For scale, the signet smoke states were 1.7–13 KB. Saves happen once per range scan (per new block), per block with a probable match, and every 1000 blocks during catch-up. Idle polls save nothing. A 224 500-block catch-up (the signet floor run) makes about 225 saves, so roughly 1.3 s on top of a scan of several minutes. A synced wallet adds about 144 × 6 ms ≈ 1 s a day.
Tests (
blindbit-lib/src/scanner/state_file_tests.rs)save_replaces_the_file_atomically_and_owner_only0644file and a stale.tmpfrom a crash: the save leaves exactly one0600file that parsesfailed_save_leaves_the_previous_file_intactsave_to_fileerrors, the old file is byte-identicalunparseable_state_file_is_moved_aside_and_reportedstate_file_from_a_newer_build_is_moved_asideformat_versionabove the build'sunreadable_state_path_is_an_error_and_nothing_movesvalid_and_missing_state_files_behave_as_beforefriglet
status_reports_a_stalled_scan_and_an_adjusted_startalso checks the newstate_resetfield, and friglet-ipc's round trip covers it.Red checks (each reverted):
Verification
cargo test --workspaceat d3200b8, built in a private target dir: all suitesok. blindbit-lib 55 passed + 1 ignored, friglet 27, friglet-ipc 8+1, friglet-tray 26+9.cargo test -p blindbit-lib --libwithoutserde: 45 passed, no new warnings.cargo clippy --no-deps -p friglet -p friglet-ipc -p friglet-tray --all-targets -- -D warnings: clean.cargo clippy -p blindbit-lib --all-targets --features serde: 6 warnings, the same as fix(scanner): detect chain reorganisations and roll back to the fork point #21.cargo fmt -p friglet -p friglet-ipc -p friglet-tray --checkis clean.