Skip to content

fix(scanner): stop at an unscannable oracle block instead of silently skipping it - #20

Merged
setavenger merged 5 commits into
integrationfrom
fix/stream-unindexed-height
Oct 1, 2026
Merged

setavenger merged 5 commits into
integrationfrom
fix/stream-unindexed-height

Conversation

@setavenger

@setavenger setavenger commented Sep 24, 2026 •

Copy link
Copy Markdown
Owner

Important

Deploy order (re-checked against the oracle PRs' diffs):

  • blindbit-oracle#65 (streams end with NOT_FOUND at an unindexed height) deploys last, only once clients with this PR are out. Older clients unwrap() every stream message; with panic = "abort" in the release profile the whole friglet process dies, and any wallet whose start height is below the oracle's first block hits that on its first poll.
  • This PR is safe against the hosted oracle as it is today: a start height below the oracle's first indexed block now starts at that block (see below) instead of stopping there forever.
  • blindbit-oracle#66 only matters for freshly synced oracles (it indexes the configured start height on an empty database). It is no longer a precondition for this client: on an oracle without it, a wallet starting at or below the never-indexed start height starts one block above it. (The testkit's T2 run had found a height-1 friglet looping on oracle 1b0e113, which lacks blindbit-lib tests: close peers after the block request (SNB-692) #66; that case now starts at height 2.)
  • blindbit-oracle#67 (reorg cleanup, stacked on blindbit-lib tests: close peers after the block request (SNB-692) #66) and friglet: count restored wallet outputs at startup (SNB-689) #64 (unary NOT_FOUND) have no ordering constraint. NOT_FOUND from GetBlockHashByHeight is read as "not indexed".
  • fix: record owned outputs so spends are detected; persist labels #19 (state format) has no oracle constraint.

Bug: silent fund loss

  • An oracle answers a height it has not indexed with an OK message. The message has an empty block_hash, an empty compute index and no spent outputs.
  • scan_block_range handled that message in blindbit-lib/src/scanner/scanning.rs at origin/integration:
    • scan_short_block_data logged WARN block has malformed block_hash; spent-output notifications will be skipped (:627-631) and skipped only the spent-output check.
    • It read the empty compute index as "no payments".
    • It moved the scan height past the block (:130-131) and reported success at the tip.
  • Proof: the blindbit-testkit mutation check unindexed-ok-empty gets 80000 sats instead of 120600, so two outputs are lost silently. friglet runs the same scan loop.

A second bug is in the same loop. stream.message().await.unwrap() (:95) and the stream-opening .await.unwrap() (:70) panic the process on any error status, such as NOT_FOUND or UNAVAILABLE. Because of this, the oracle could not start returning errors either.

Fix (blindbit-lib/src/scanner/scanning.rs)

Every block message must now be the next expected height and carry a valid block hash, meaning 32 bytes and not all zero. The scan stops with an error, instead of skipping, in each of these cases:

  • a bad or missing block hash
  • a skipped or repeated height
  • an error status mid-stream
  • failing to open the stream
  • a stream that ends before end

The error names the height to rescan from, for example:

scan stopped at height 102: the oracle sent no valid block hash (0 bytes); the height is probably not indexed; nothing from height 102 on was scanned, rescan from height 102
scan stopped at height 104: oracle stream error NotFound: height 104 is not indexed ...

last_scanned_block_height is never moved past the last fully processed block. The old guard that "advanced progress and moved on" for a malformed hash is removed. There is no retry inside the stream, so the scan stops at the bad height and the caller resumes there:

Consumer Behaviour now
friglet (watch_chain) Logs scan_block_range failed, will retry on next poll at error level, sleeps 10 s, and rescans from last_scanned + 1. No spin and no crash loop; the task no longer dies from a panic.
blindbit-cli Saves state, with the height at the last full block, then exits 1 with the error above.
examples/stream_oracle.rs Same checks: ? instead of unwrap(), and an empty hash is rejected.

To make this testable in-process, the scan loop now reads messages through a small BlockScanDataStream trait, implemented for tonic::Streaming. scan_block_range opens the stream and calls scan_block_stream. The loop body is unchanged apart from the checks.

Tests (blindbit-lib/src/scanner/stream_safety_tests.rs)

Each test drives a real Scanner through scan_block_stream, using in-process messages and real silent-payment outputs. Only the P2P block fetch is replaced, by a #[cfg(test)] hook that serves the full block.

Test Proves
empty_hash_block_mid_range_stops_scan_and_next_scan_finds_its_outputs Payments at 1100, 1101 and 1102, with 1101 served as an empty-hash block. The scan errors naming 1101, the height stays at 1100, and only 1 output is found. The next scan from last_scanned + 1 then finds all 3.
malformed_block_hash_stops_scan A 31-byte hash and an all-zero hash each stop the scan, and the height stays put.
skipped_height_stops_scan / stream_starting_at_wrong_height_stops_scan A height gap stops the scan, and the block after the gap is not applied.
error_status_mid_stream_is_a_clean_error NOT_FOUND and UNAVAILABLE mid-stream: no panic, the error names the height and the code, and the height stays at the last full block.
stream_ending_before_requested_end_is_an_error A short stream is not reported as success.
unreachable_oracle_is_a_clean_error Failing to open the stream is an error, not a panic.

Red without the fix: commit ad80ea4 adds the seam and the tests with the old behaviour. On that commit all 7 tests fail. The three panic cases hit called Result::unwrap() on an Err value: Status { code: NotFound, message: "height 1501 is not indexed" } (and the equivalent for Unavailable). The rest fail with a block without a valid hash must stop the scan: (), meaning the old code returned Ok. With the fix all 7 pass.

Wallet start below the oracle's first block; stall reporting (commit 22c50bf)

The hosted oracle has no data below 800 000 on mainnet and 100 000 on signet (both probed: GetBlockHashByHeight(99 999) on signet answers an empty hash, 100 000 a real one). friglet's config accepts any start height ≥ 1, so with the stop-and-retry above, a wallet starting lower stopped at its first height on every poll, forever, with only a log line.

  • Floor start. When watch_chain's scan stops at the wallet's start height because the oracle has not indexed it, the scanner finds the oracle's lowest indexed height with GetBlockHashByHeight lookups (find_oracle_floor, scanner/health.rs) and starts there. It logs a warning, persists the adjustment as oracle_floor_start in the state file (optional field, omitted when unset; older binaries ignore it), and publishes it in the scan health.
  • Never skips a gap. Only heights below the oracle's floor are skipped. The move happens only at the wallet's start height (sp_start_height, the configured start). Once a block was scanned, an unindexed height after it keeps the stop-and-retry. A start height with any indexed height below it (a gap in the index) is not moved either.
  • Robust to a non-contiguous index. A plain binary search can land in a gap above the floor and report the gap's end. The search is verified by probes at exponentially growing distances below the boundary, and any indexed hit restarts it lower. The same back-off below the start height tells a gap from the region below the floor. Cost: about 50 lookups (~100 bytes each) once per wallet on signet, at most 400.
  • Stalls are visible. A scan that cannot get past a height, or cannot reach the oracle, is published as a stall: height, reason, and the time of the first failed attempt at that height. It clears when the scan makes progress. friglet's GetStatus carries it as StatusInfo::scan_health (stall, start_adjusted; serde defaults, so old trays and old daemons still talk to each other). The tray's status window shows a "Scan stalled" line and a note for an adjusted start.
  • Stream stops are now a typed ScanStopped error (message text unchanged), so the loop can tell "not indexed" from other failures. The health is kept in the wallet's Electrum index, which status readers lock without waiting on the scan task (the scan task holds the scanner lock).

Tests (blindbit-lib/src/scanner/oracle_floor_tests.rs, plus friglet status_reports_a_stalled_scan_and_an_adjusted_start):

Test Proves
start_below_a_contiguous_index_finds_its_first_block signet and mainnet layouts: the floor is found in ≤ 60 / ≤ 140 lookups
oracle_missing_its_own_start_height_starts_one_above the pre-#66 oracle layout (start height never indexed)
gap_above_the_floor_does_not_move_the_floor_up a gap where a plain binary search lands is not taken as the floor
start_inside_a_gap_above_the_floor_is_never_skipped / randomised_gappy_indexes 2 000 random layouts: the floor is always found, and no start inside a gap is moved
wallet_start_below_the_floor_starts_at_the_floor_and_says_so the stream stops at the unindexed start, the start moves to the floor, the payment at the floor is found, and the note is in the status and survives a restart
unindexed_height_after_the_wallet_start_is_never_skipped / only_a_not_indexed_stop_at_the_start_moves_the_start gaps, other stream errors and other heights never move the start
a_stall_keeps_its_start_time_until_the_height_changes_and_clears_on_progress stall bookkeeping

Red checks (each reverted): without the back-off verification, the gap test and the randomised test fail. Without the below-start probes, the gap-start tests fail. Without the wallet-start condition, unindexed_height_after_the_wallet_start_is_never_skipped fails.

Verification

  • cargo test --workspace at 22c50bf: all suites ok. blindbit-lib 30 passed, friglet 26, friglet-ipc 8+1, friglet-tray 26+9.
  • CI lint, cargo clippy --no-deps -p friglet -p friglet-ipc -p friglet-tray --all-targets -- -D warnings: clean. cargo clippy -p blindbit-lib --all-targets --features serde: 10 warnings, the same count as before this PR, none on changed lines.
  • cargo build -p blindbit-lib --examples builds. cargo fmt -p friglet -p friglet-ipc -p friglet-tray --check is clean; the new blindbit-lib files were formatted on their own (AGENTS.md: no workspace-wide fmt).
  • Merges cleanly onto integration + test: BIP-352 vectors through the live short receive path (SNB-541) #17.
  • Live smoke run against the hosted signet oracle, built from the stack's tip (fix(scanner): detect chain reorganisations and roll back to the fork point #21 at 9740d66). Details are in fix(scanner): detect chain reorganisations and roll back to the fork point #21.
    • Start height 99 000: 46 lookups found the floor, the log said requested_height=99000 oracle_floor=100000 skipped_blocks=1000, scanning went on from 100 000, and GetStatus showed start_adjusted.
    • A height the proxy reported as NOT_FOUND (324 470, above the wallet's start) was never skipped. Each poll stopped there, GetStatus showed stall {height: 324470, since_unix: 1790863242} with the reason, and it cleared on the first poll after the fault was removed.
    • The tray's status window showed both the stall line and the start note (screenshots taken under Xvfb). The screenshot exposed that .col blocks ignored hidden, which also left an empty "Last error" label on screen; 5492765 fixes it with .col[hidden] { display: none; }.

End to end with blindbit-testkit (testkit mutate, master at the #18 merge, release build of blindbit-cli from this branch):

  • unindexed-ok-empty is no longer a silent MISSING [big-k1 dust-k0]. blindbit-cli now exits 1 with scan stopped at height 102: the oracle sent no valid block hash ..., so the run is unjudgeable, not wrong.
  • stale-view and close-mid-stream used to pin the panic (signal: aborted ... panicked at ... Status { code: NotFound|Unavailable). They now end in a clean exit status 1 naming the code.
  • truncated-stream used to be caught only by the harness's tip check. It is now refused by the client itself.
  • The testkit catalog pins the old client behaviour for these four entries and needs updating. That is a separate change; clean runs pass for every case.

Merge note

Merges cleanly after #17. #19 is stacked on this PR (it contains #17 and #20), and #21 on #19, so the planned order is #17 → #20 → #19 → #21.

🤖 Generated with Claude Code

setavenger and others added 2 commits September 24, 2026 11:20
… messages

Adds an in-process seam (BlockScanDataStream) between scan_block_range and
the scan loop, with no behaviour change, and tests that drive a real Scanner
through it. They fail on this commit: an empty-hash block mid-range is
treated as an empty block (payments in it are lost and the scan reports
success), height gaps and short streams are accepted, and any error status
panics the process via unwrap().

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ipping it

An oracle answers a height it has not indexed with an OK message carrying an
empty block hash and no data. scan_block_range took that as a block with no
payments: it warned, advanced the scan height past it and reported success,
so any payment in that block was never found (proven by the testkit mutation
check unindexed-ok-empty: 80000 sats reported instead of 120600).
Separately, every stream message was unwrap()ed, so an error status
(NOT_FOUND, UNAVAILABLE) panicked the process, as did failing to open the
stream.

Now each message must be the next expected height and carry a valid
32-byte, non-zero block hash; otherwise, and on any error status or a stream
that ends before the requested end, the scan returns an error naming the
height to rescan from. The scan height stays at the last fully processed
block, so watch_chain retries from there on its next poll (every 10 s, no
spin) and blindbit-cli exits non-zero after saving that height. The example
is updated the same way.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s below it; report stalls

A wallet whose start height lies below the oracle's first indexed block
(the hosted oracle has nothing below 800000 on mainnet and 100000 on
signet) stopped at its first height on every poll, forever, with only an
error-level log line. Since #20 refuses to scan past an unindexed height,
that is the only thing such a wallet ever did.

- When `watch_chain`'s scan stops at the wallet's start height because
  the oracle has not indexed it, the scanner locates the oracle's lowest
  indexed height with GetBlockHashByHeight lookups and starts there,
  logs a warning, persists the adjustment (`oracle_floor_start`, optional
  in the state file) and publishes it in the scan health.
- Only heights below the oracle's floor are skipped. A height above the
  first scanned block, or one with an indexed height below it (a gap in
  the oracle's index), keeps #20's stop-and-retry. The search is a binary
  search verified by exponential back-off probes below the boundary, so a
  gap above the floor does not move the floor up. NOT_FOUND and an empty
  or zero hash both mean "not indexed".
- A scan that cannot get past a height (or cannot reach the oracle) is
  published as a stall: height, reason, and the time of the first failed
  attempt at that height. It clears when the scan makes progress.
- Stream stops are a typed `ScanStopped` error (same message text) so the
  loop can tell "not indexed" from other failures.
- Health lives in the wallet's Electrum index, which status readers lock
  without waiting on the scan task. friglet's GetStatus reports it as
  `StatusInfo::scan_health` (serde default, so old trays and daemons
  interoperate), and the tray's status window shows a stall line and a
  note for an adjusted start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 1, 2026
…nd-labels

#20 lands before this PR, so this branch carries it to merge cleanly
after it. changeset.rs: keep both imports (#20's OracleFloorStart and
this branch's OwnedOutputRecord); the serde_hex module #20 still had is
the one this branch replaced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n attribute)

The status window's error, stall and scan-notes blocks use class="col",
whose display: flex beats the user-agent [hidden] rule, so an empty
"Last error" label was always shown and the new stall/notes blocks
would be too. Seen in a screenshot against a live daemon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 1, 2026
…nd-labels

Picks up the tray CSS fix for hidden status blocks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 1, 2026
…fix/reorg-rollback

Keeps this branch a superset of the stack below it.

- changeset.rs / scanner.rs: keep both sides' new fields
  (scanned_block_hashes; format_version, oracle_floor_start).
- scanning.rs: keep this branch's scan_range_from / BlockStreamSource;
  OracleStreams::open now reports a refused stream as #20's typed
  ScanStopped. watch_chain combines both: the no-new-block hash check
  from here, and #20's floor start (only for a real range) and stall
  reporting, with the ReorgTooDeep log kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 1, 2026
Picks up #20's tray CSS fix for hidden status blocks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 1, 2026
…ood/preview

Brings in the #17/#20/#19/#21/#31 stack. One conflict, in
friglet/src/electrum/mod.rs, resolved keeping both sides:
- #26's ChainSource (p2p/network/oracle) and pending-broadcast store,
- #21's reorg handling: checkpoints behind a mutex, reorg_rx feeding
  forget_blocks_above; its test now builds the server state with #26's
  test helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 1, 2026
Brings in the fix stack (#17, #20, #19, #21) so the T0 harness runs
against the receive path that records owned outputs.

scanning.rs: both sides extracted apply_matched_block; kept #21's
sync_owned_outputs() call after it in the scan loop and its doc line,
and kept both test modules (owned_outputs_tests from #21, testkit_t0
from this branch).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 1, 2026
The T0 harness replayed each block through a hand copy of the scan loop's
per-block steps, and that copy had already fallen behind: the loop on the
fix stack records owned outputs (sync_owned_outputs) after
apply_matched_block and the copy did not, so nine spends were carried as
KNOWN_SPEND_GAP.

process_block now drives Scanner::scan_block_stream, the per-block loop
scan_block_range and watch_chain run, with the oracle's message streamed
in-process and the full block served in place of the P2P fetch (the
stream-safety test hooks from #20/#21). There is no copy left to drift: a
step added to the loop is exercised here. State files the loop now saves
are removed when a scenario ends.

KNOWN_SPEND_GAP and its assertions are gone; all nine scenarios must pass
with their spends. Dropping the sync_owned_outputs() call from the loop
turns exactly those nine spends red again (k-gap, labelled-multi,
spent-outputs, dust-boundary, rescan).

Fixture provenance now names testkit 68f9f1c, the merge of testkit PR #25
into master; the fixtures are byte-identical to the export at b1f962e.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since ScanStopped became a struct (22c50bf), blindbit-cli printed a
stopped scan as `Error: ScanStopped { height: 104, reason: "...",
not_indexed: true }`, the Debug form main uses for a returned error. The
message the type renders for the user, "scan stopped at height 104:
...; nothing from height 104 on was scanned, rescan from height 104", no
longer appeared. The testkit's mutation checks match that message and
went red for stale-view, truncated-stream, close-mid-stream and
unindexed-ok-empty.

main now runs the command and prints the error with Display. The exit
status of a failed command is still 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 1, 2026
Brings in the blindbit-cli error display fix from #20.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 1, 2026
Brings in the blindbit-cli error display fix from #20.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 1, 2026
Brings in the blindbit-cli error display fix from #20.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@setavenger

Copy link
Copy Markdown
Owner Author

Added d989e53 fix(cli): print a failed scan's error with Display, not Debug.

Problem. Since ScanStopped became a struct in 22c50bf, blindbit-cli has printed a stopped scan as Error: ScanStopped { height: 104, reason: "...", not_indexed: true }. That is the Debug form main uses for a returned error. The user-facing message, scan stopped at height 104: ...; nothing from height 104 on was scanned, rescan from height 104, no longer appeared.

How it was found. The testkit's mutation checks match on that message. With the pin moved to #31's previous head d3200b8, four of them went red: stale-view, truncated-stream, close-mid-stream and unindexed-ok-empty (testkit run 36878809374).

Fix. main now runs the command and prints the error with Display. A failed command still exits with status 1.

Propagation. Merged up the stack (merges only):

PR New head
#19 b7b4f59
#21 824613f
#31 1a9e29e
#22 c9c8fa1

Verification. Built blindbit-cli at 1a9e29e and ran the testkit mutation checks: 28/28 caught on both backends, and every weakened run went red. The T1 replay tests pass.

@setavenger
setavenger merged commit b7b4f59 into integration Oct 1, 2026
1 check passed
@setavenger
setavenger deleted the fix/stream-unindexed-height branch October 1, 2026 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant