Skip to content

blindbit-lib: one code path in the scan loop (SNB-669) - #52

Closed
setavenger wants to merge 4 commits into
integrationfrom
snb-669-one-scan-loop-path
Closed

setavenger wants to merge 4 commits into
integrationfrom
snb-669-one-scan-loop-path

Conversation

@setavenger

@setavenger setavenger commented Oct 3, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #47 (itself on #44); review only the commits after it.

Linear: SNB-669

Net: 249 insertions, 366 deletions (−117 lines). Most of the scan-loop diff is re-indentation; with whitespace hidden it is 92 / 209. Please review with "Hide whitespace".

What was merged or removed, and why

  • One path through the scan loop. The no-match arm and the matched arm each advanced the scanned height, recorded the block hash and sent the Electrum progress notification. Now the matched-block work is an if let Some(..) step, and the shared tail runs once, in the same order as before:
    • every block: notify, then advance height, then record hash;
    • matched blocks still save inside their step (before the height advance), as before;
    • the every-1000-blocks checkpoint still runs only for blocks without a match.
  • The oracle block hash is decoded once. check_block_identifier now returns the hash, decoded by health.rs's indexed_block_hash. Its rules are the ones the check used: 32 bytes and not all zero, with the same error and message. The P2P fetch uses that hash instead of reversing the bytes again. block_hash_from_oracle (reorg.rs) is gone.
  • One balance computation. balance_at(tip) replaces the two copies in the loop (the per-block "balance" log and the "balance after scan" log; both texts are unchanged) and the third copy in owned_outputs_tests.rs.
  • Duplicate scan_progress write dropped. The matched arm set idx.scan_progress, and the progress notification right after it wrote the same value.
  • Debug loop dropped. It logged every graph transaction after each matched block (debug level only).
  • Matched-txid filter. apply_matched_block computed tx.compute_txid() once per transaction per matched txid, and converted each matched txid twice. It now builds a Txid → tweak map once and hashes each block transaction once. The result is the same map: only matched txids found in the block, and for a repeated txid the last tweak wins, as before. A block fetched only for a possible spend has no matched txids; then the filter is skipped and nothing is hashed (72104e0, after review).
  • OracleStreams newtype removed. It only delegated to OracleServiceClient, which now implements BlockStreamSource itself; it already implemented OracleProbe in health.rs. The identity match in probabilistic_match is also gone.
  • Stale comments:
    • the todos in mod.rs and in scan_short_block_data (the notifications they mention were removed in blindbit-lib: remove dead scanner API and P2P leftovers (SNB-663) #44);
    • the "config with defaults" TODO on Scanner::new (the talk benchmark keeps that signature);
    • the comments in byte_array_to_txid and on the balance test helper;
    • the WalletElectrumIndex::txs / headers docs, which missed restore, broadcast and friglet's header writers.

Why behaviour is unchanged

  • Same order of side effects per block (notify → height → hash record → periodic save), same errors, same log messages. friglet-ui's strings ("scan progress", "balance after scan", "scan_block_range failed", …) are untouched.
  • The only lost output is the debug-level graph dump.
  • The live scans below wrote byte-identical state files with the old and the new binary.

Measurement in release (after review)

blindbit-cli --release (workspace release profile: LTO, one codegen unit) built at origin/integration (a9ff4d6) and at this branch's head 72104e0. Same signet range, peer, keys and method as below: 274010..=276010, fresh state file per run, /usr/bin/time -v, 6 runs each, interleaved.

user+sys CPU (s) mean ± sd wall (s) wall mean ± sd
integration a9ff4d6 8.24 8.59 8.51 8.22 8.13 8.47 8.36 ± 0.19 8.47 8.92 8.77 8.38 9.85 8.64 8.84 ± 0.53 (median 8.71)
this PR (72104e0) 8.39 8.43 8.44 8.32 8.21 8.43 8.37 ± 0.09 8.69 8.94 8.76 8.59 8.46 8.89 8.72 ± 0.18 (median 8.72)
  • CPU differs by +0.01 s (0.1 %, t ≈ 0.1): no measurable change on this range. Max RSS is 11 MB for both.
  • All 12 runs wrote the same state file. It also matches the debug runs below.

Reviewer's micro-benchmark of apply_matched_block's txid filter (per block):

case before #52 #52 before 72104e0 #52 now
spend-only match (no matched txid) 27 ns 2.09 ms nothing hashed (as before #52)
1 matched txid ≈ 2 ms ≈ 2 ms ≈ 2 ms
10 matched txids 20.97 ms 2.06 ms 2.06 ms

Measurement in debug (hot path)

blindbit-cli debug build (dev profile; libsecp256k1 at -O3 as configured), signet oracle https://signet.oracle.setor.dev, peer 152.53.151.148:38333, smoke keys (secret …01, pubkey G), blocks 274010..=276010 (2,001 blocks), fresh state file each run, /usr/bin/time -v. Runs were interleaved old/new to cancel network drift; there were 8 of each (the 3 asked for, plus 5 more because the first 3 were close).

user+sys CPU (s), 8 runs mean ± sd wall mean ± sd max RSS
before (#47 head) 9.65 9.81 9.84 9.59 9.73 9.60 9.58 9.79 9.70 ± 0.11 9.47 ± 0.10 s 19 MB
after (this PR) 9.84 9.84 9.87 9.81 9.71 9.56 9.57 9.84 9.75 ± 0.13 9.49 ± 0.12 s 20 MB
  • The difference (+0.5 % CPU, +0.2 % wall) is well inside one standard deviation (t ≈ 0.8). The first 3 pairs alone gave 9.77 vs 9.85 s CPU and 9.53 vs 9.57 s wall.
  • All 16 runs wrote the same state file (one sha256).
  • These keys find nothing, so this range exercises the no-match path. The matched path (the txid filter and the balance helper) is exercised by the stream_safety, reorg, witness and owned_outputs tests.

Verification

  • cargo check --workspace --all-targets --all-features → 0 warnings.
  • cargo check -p blindbit-lib --no-default-features --all-targets → 5 warnings (baseline 5).
  • cargo clippy --no-deps -p friglet -p friglet-ipc -p friglet-tray --all-targets -- -D warnings → clean.
  • cargo fmt -p friglet -p friglet-ipc -p friglet-tray -- --check → clean.
  • cargo test --workspace → 254 passed, 1 ignored, 0 failed. This includes the stream_safety, reorg, witness, cancel and owned_outputs suites.
  • Re-run on 72104e0: the same results (0 warnings; 5 with --no-default-features; clippy and fmt clean; 254 passed, 1 ignored, 0 failed).

Left out

  • ReorgTooDeep decided in two places (scan_range_from's _ arm and scan_block_stream_from's first-block check). The issue says to do this only if the reorg suite proves equivalence.
    • I temporarily instrumented both decision points and the ReorgBelowStreamStart return and ran every blindbit-lib test. None of the three is reached by any test; the only ReorgTooDeep test goes through locate_fork.
    • So the suite cannot prove the change, and it is left as is.

setavenger and others added 3 commits October 3, 2026 08:43
Remove public items nothing calls, in this workspace or in the other
repos that use blindbit-lib (blindbit-node-apps, testkit, the talk
benchmark):

- broadcast_tx: its only caller, friglet's Electrum broadcast, uses
  friglet's own relay since #26.
- Scanner::from_config, get_last_scanned_block_height_rescan,
  staged_mut, take_staged and the commented-out get_outputs.
- watch_chain: friglet calls watch_chain_until; its doc moves there.
- BlockIdentifierDisplay (only .0.block_height was read), OwnedOutput
  (never built), ProbableMatch::new, _match_short_pubkey_bytes and the
  commented-out ScannerConfig Default. electrum_index becomes a private
  module; its items stay re-exported.
- The spent-outpoint and probabilistic-match channels and the code that
  only fed them. They had no subscribers and never sent anything: the
  sends were guarded by Sender::is_empty(), which means "no queued
  value", so they were always skipped. The malformed-hash branch that
  guarded them cannot run: check_block_identifier rejects such a block
  before scan_short_block_data sees it. The "possible spend of owned
  output" log line stays.
- save_to_file's back-fill of keys and max_label_num: stage holds them
  from construction on and only take_staged could clear them. A test
  run with assertions in place of the back-fill passed every workspace
  test.
- P2P: TimeoutParams setters the handshake never reads (it only uses
  the ping interval; the socket timeouts are set directly), history
  notes about deleted code, and a retry doc that predates the
  witness-stripped and bad-block retries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scanner::new and from_changeset each wrote out the whole struct; both
now hand their staged changeset to one private constructor,
from_stage, which derives the rest (checkpoints with genesis, owned
outputs, remembered hashes, Electrum index) from it.

- "New indexer, labels 0..=max, then apply_changeset" was written in
  from_changeset and in the reorg rollback; it is restored_indexer now,
  with the reason the labels must come first.
- The genesis checkpoint hash was built in three places; genesis_hash().
- new() merged add_label(0..=max) into the stage a second time after
  initial_changeset() already carried those labels (label_lookup is a
  set, so the merge changed nothing); dropped.
- The remembered-hash lookup in reorg.rs is one helper.
- Scanner.last_scanned_block_height_rescan was only restored, lowered on
  rollback and copied into every save. The stage already holds the same
  value, so the field goes; the ChangeSet field and the state format
  stay as they are.

A probe (not committed) dumped stage, saved file and full scanner
state for fresh scanners, a test-produced wallet, the old-format
fixture, restores and rollbacks of each, before and after this change:
all 42 outputs identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The scan loop's two arms (no match / matched block) each advanced
  the scanned height, recorded the block hash and notified Electrum;
  that now happens once after the optional matched-block step, in the
  same order as before. The periodic checkpoint still runs only for
  blocks without a match (a matched block is saved earlier).
- The oracle's display-order block hash is decoded once:
  check_block_identifier returns it (via health.rs's
  indexed_block_hash, the same length and all-zero rules), and the P2P
  fetch uses it instead of decoding the bytes again.
  block_hash_from_oracle goes.
- The balance over the sparse checkpoints is one helper, balance_at,
  used for the per-block "balance" log, the "balance after scan" log
  and the owned-output tests (which had a third copy).
- The matched block's scan_progress write is dropped; the progress
  notification right after it writes the same value.
- The debug loop that logged every graph transaction after each
  matched block is dropped.
- apply_matched_block computed every transaction's txid once per
  matched txid; it now builds a txid map and hashes each transaction
  once.
- OracleStreams only delegated to the oracle client: the client now
  implements BlockStreamSource itself (it already implemented
  OracleProbe). An identity match in probabilistic_match goes.
- Stale TODOs and comments (notifications that no longer exist, a
  config-constructor TODO the talk benchmark rules out, Electrum index
  docs that missed the restore and broadcast writers).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A block fetched only because an owned output may be spent carries no
matched txids. The txid filter still hashed every transaction of it
(about 2 ms per block); the code before #52 did nothing there. Skip
the filter when there are no matched txids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 3, 2026
…NB-666)

Brings #52's spend-only fix (skip the txid filter when no txid
matched) into this stacked branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 3, 2026
…nfirmed (SNB-687)

Picks up #57's review nits and #52's follow-up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 3, 2026
…688)

Picks up #57's review nits and #52's follow-up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@setavenger

Copy link
Copy Markdown
Owner Author

Merged into integration through #68 on 2026-10-11 (the user live-tested the combined stack first). GitHub's stacked-PR merge rebased the stack, so this PR's commits landed with new IDs and GitHub can't mark it merged itself. The code is identical: integration now has the same tree as the tested branch friglet-deslop-live-test (08bf11d), which contains this PR's head. Closing as merged via #68.

@setavenger setavenger closed this Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant