Skip to content

blindbit-lib, friglet: share the Electrum-index and oracle-hash code (SNB-671) - #76

Merged
setavenger merged 6 commits into
integrationfrom
snb-671-shared-electrum-index
Oct 30, 2026
Merged

setavenger merged 6 commits into
integrationfrom
snb-671-shared-electrum-index

Conversation

@setavenger

@setavenger setavenger commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #75 (SNB-662), which sits on #74 (SNB-668) and #72 (snb-661-stack-06-docs, top of the Friglet cleanup stack #67–#72); review only the commits after #75 (the merge commit only brings #75/#74 review follow-ups in). Retargets to integration once #67–#72 land.

Implements SNB-671. Non-test code: net −80 lines (180 added, 260 removed); tests: +1 characterisation test of the Electrum index and +1 unit test of the shared insert.

What was duplicated

The per-transaction update of the wallet's Electrum index (what Sparrow reads through blockchain.scripthash.get_history, blockchain.transaction.get and silentpayments.subscribe) was written twice in blindbit-lib and a third time, in part, in friglet:

live scan (scanning.rs, matched block) restore (rebuild_electrum_index_from_graph) friglet broadcast handler
owned outpoint set + outpoint→script map built the same way built the same way —
receive / spend history entries insert-or-promote, push + sort per insert push-if-absent, one sort at the end push-if-absent at height 0, push + sort
raw tx bytes wallet txs only wallet txs only (serialised eagerly) always
SP history separate pass, push-if-absent + sort push-if-absent, one sort at the end —

friglet also had its own copy of convert_network (wallet_network) and of the oracle's block-hash decoding.

What it is now

WalletElectrumIndex owns the logic:

  • add_history(scripthash, tx_hash, height) -> bool (was the free fn upsert_history_entry): lists a tx in a scripthash's history, sorted by height with unconfirmed entries first. A listed tx keeps its entry, except that an unconfirmed one is promoted when it confirms. Returns whether it added an entry. friglet's broadcast handler calls it at height 0, where it is exactly the old push-if-absent + sort, and counts the trues as before.
  • index_confirmed_tx(tx, txid, height, &WalletView): receive entries, spend entries, raw bytes if the tx touched the wallet, and the SP-history entry if it paid the wallet. The live scan calls it once per transaction of a matched block, the restore once per confirmed graph transaction. The txid is formatted only for wallet transactions, as before.
  • WalletView::of(&indexer): what an update needs to know about the wallet (owned outpoints, their scripts, the SP tweak map), built outside the index lock and after apply_matched_block/sync_owned_outputs, so an output received and spent in the same block counts.
  • New entries are inserted where pushing and stably sorting would put them (after every entry at the same or a lower height), instead of pushing and re-sorting the whole list. Every writer keeps these lists sorted, so the result is identical.
  • The restore feeds transactions in height order (a stable sort, so same-height transactions keep the wallet graph's order, as before) and each entry is appended.

friglet calls blindbit-lib's convert_network (now public) instead of its identical wallet_network, and its oracle block-hash decoding calls the new blindbit_lib::scanner::oracle_block_hash (length check + byte order). The two decoders were not identical: blindbit-lib's indexed_block_hash treats an all-zero hash as "not indexed", friglet's returns it and has its own wrong-length error text that reaches Electrum JSON-RPC errors. Both keep exactly that: indexed_block_hash is oracle_block_hash plus the zero rule, and friglet's block_hash_from_oracle keeps its signature and message.

Why behaviour is unchanged

  • Live scan: same inserts, same order per list (SP history and scripthash histories are independent lists, so doing a transaction's SP entry right after its scripthash entries instead of in a second pass changes neither), same raw bytes, same pending-promotion pass after it, same lock scope, no await under the lock. None of the log lines friglet-ui parses are in this code; the pending-promotion debug log stays in scanning.rs.
  • Restore: the old code pushed in graph order and stably sorted each list once; the new one stably sorts the transactions by height and appends. Both give "by height, ties in graph order". (That tie order is random per process, before and after; filed as SNB-781.) The only path where they could differ is a restore onto an index that already holds unconfirmed entries; the one production caller restores once at startup onto a fresh index.
  • friglet: add_history(.., 0) is the old loop body; convert_network is the same match; the decoder keeps the same results and messages.

Measurements

Electrum index before/after on identical state files (SNB-671's verify step). A throwaway harness (not committed) restored each state file on base 08bf11d and on this branch and dumped every scripthash history, its Electrum status, the SP history (with tweaks), the served txids with a hash of the raw bytes, and the tip:

state file entries content
regtest fixture (state_before_owned_outputs.json, format 0, 2 receives at one height) 10 lines identical; the two same-height SP entries come out in either order on base itself across runs (SNB-781)
synthetic, 200 payments, receive+spend in one block every 10th, later spends every 3rd 882 lines identical, byte for byte
synthetic, 3,000 payments over 30 blocks, spends 12,437 lines identical

Restore time (rebuild_electrum_index_from_graph, best of 16, 3 interleaved rounds): 3,000-payment wallet base 79–83 ms, this PR 72–73 ms; 200-payment wallet 3.8 ms → 3.2 ms. (The first version of this PR re-sorted the wallet-wide SP history after every insert and took 128 ms; review caught it, and the height-ordered restore fixed it.)

Scan cost, with PR #52's method: release blindbit-cli, signet blocks 274010..=276010, scan key 1 / spend key G, state deleted before each run, 6 runs per build in rotating order, /usr/bin/time -v:

build CPU (user+sys) per scan wall per scan max RSS
base 08bf11d 3.23 ± 0.05 s 3.40 s 10.5 MiB
#75 (SNB-662) 3.24 ± 0.02 s 3.42 s 10.5 MiB
this PR 3.23 ± 0.04 s 3.43 s 10.5 MiB

Re-run at the end on the combined branch friglet-deslop-live-test-2 (ce0d417, all four PRs) against base, 8 runs each, interleaved: base 3.23 ± 0.07 s CPU / 3.45 s median wall, combined 3.24 ± 0.04 s CPU / 3.46 s median wall. All runs, 34 in total, end in the same state file (sha256 8c7aec9071b0…, as in #52). The VM is shared (load average about 2 from other work during the runs), which is why the runs were interleaved. These keys never match a block, so this measures the common no-match path; the changed code runs only for matched blocks and at startup, which the restore timing and the testkit runs below cover.

BlindBit testkit, baseline 08bf11d vs friglet-deslop-live-test-2 (all four PRs):

  • T1 replay (fake oracle over the real gRPC contract, P2P block peer, the two release blindbit-cli builds): 8 scenarios × 3 index configurations. No found-set delta and no cost-counter delta in any cell (every counter equal). 23 cells pass; rescan × cut-through-dust fails its judgement identically on both sides (a known scenario/client mismatch the testkit's own tests skip).
  • T0 (testkit vectors through blindbit-lib, the port of PR test: run the testkit scenarios at T0 through the live receive path (SNB-517) #22's testkit_t0, applied locally on both sides): 10/10 pass on both.
  • testkit mutate against the candidate blindbit-cli: 28 of 28 mutants caught, all 28 weakened checks go red (same as baseline).
  • T2 regtest (real bitcoind 30.3, blindbit-oracle 31b32a5 and friglet built from each commit in Docker; found sets read through friglet's Electrum server): 10 scenarios × 3 index configurations = 30 diffs. No found-set delta and no cost-counter delta in any of them. 24 pass on both sides; 6 fail their judgement identically on both sides, all under the two filtering configurations, because the oracle at this pin still serves outputs those configurations expect to be dust-filtered or cut through (an oracle-side gap, independent of this stack). input-types is lib-only by design and refused at T2 (it passed at T0).

Not changed on purpose

  • WalletElectrumIndex's fields stay pub: friglet reads and its tests seed them about 100 times; filed as SNB-783.

Reviews

  • gstack review: ship with fixes. Applied: the restore cost (see above), the test now also checks the SP tweaks and the raw bytes. Kept: the test borrows the in-process oracle from reorg_tests (widening seven test items to pub(super)) rather than moving it to test_support.
  • thermo-nuclear-review: ship, no findings; a focused re-review of the follow-up commit (insert position, height-ordered restore, SP entry folded into index_confirmed_tx): ship.
  • codebase-design: ship. Applied its option (a): the SP entry moved into index_confirmed_tx (one call per transaction for both paths, one compute_txid less per transaction of a matched block). Deferred: moving the pending-promotion pass behind a method and making the fields private (SNB-783).

Tests

  • live_scan_and_restore_build_the_same_electrum_index (first commit; passes on the code before the refactor, including the stronger checks): receives at 101–103 (the 103 output spent in its own block) and a spend at 104; asserts every scripthash's exact (tx, height) list, the SP history with tweaks and the served raw transactions, after the live scan and after save → restore → rebuild. A mutant that drops the spend entries fails it.
  • add_history_keeps_one_entry_per_tx_and_promotes_unconfirmed_ones: sort order, repeated tx, promotion, never back to unconfirmed, return value.

Verification

setavenger and others added 6 commits October 11, 2026 07:51
…e build (SNB-671)

Nothing compared the Electrum index a restarted daemon rebuilds from its
state file with the one the live scan writes, and the existing tests only
check that an entry exists. The new test scans receives at 101-103 (the 103
output spent in its own block) and a later spend at 104, then asserts the
exact (tx, height) list of every scripthash, the SP history and the served
txids, live and after save, restore and rebuild. Same-height entries are
compared by txid after a restore, which walks the wallet graph in hash order.

It passes on the code as it is, ahead of the refactor that shares this code
between the two paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…(SNB-671)

The per-transaction Electrum index update was written twice in blindbit-lib,
once in the live scan and once in the restore from the wallet graph, and the
history insert a third time in friglet's broadcast handler. It now lives in
WalletElectrumIndex:

- `add_history` (was `upsert_history_entry`): lists a tx in a scripthash's
  history, sorted by height; a listed tx keeps its entry unless an
  unconfirmed one confirms. friglet's broadcast handler calls it at height 0,
  where it is exactly the old push-if-absent and sort.
- `index_confirmed_tx`: receive and spend entries plus the raw tx, for the
  live scan's pass over a matched block and for the restore. The txid is
  formatted only for wallet transactions, as before.
- `add_sp_history`: the SP history entry, for both.
- `OwnedScripts::of`: the owned-outpoint set and outpoint-to-script map both
  paths built identically, still built outside the index lock and after the
  block was applied.

The restore used push-if-absent with one sort at the end; it now uses the same
insert-and-sort as the live scan. On the fresh index a restore runs on, the
result is identical: a stable sort after every insert leaves the same list as
one stable sort at the end, and an unconfirmed entry to promote cannot exist
yet. The live scan keeps its three passes over the block in the same order.

friglet uses blindbit-lib's `convert_network` (now public) instead of its
identical `wallet_network`, and its oracle block-hash decoding calls the new
`blindbit_lib::scanner::oracle_block_hash` (length check and byte order),
keeping its own wrong-length message. blindbit-lib's `indexed_block_hash` is
that function plus its "all zeros means not indexed" rule, unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndex

Brings in the SNB-662 and SNB-668 review follow-ups (a live vout test, the
opt-level comment, the README note on http_addr). No conflicts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sition (SNB-671)

Review follow-ups.

- index_confirmed_tx also adds the SP history entry, so the live scan and
  the restore no longer each look up the tweak; WalletView (was
  OwnedScripts) carries the indexer's tweak map. The live scan loses its
  separate SP pass and one compute_txid per transaction of a matched block.
  The two lists are independent, so their contents and order are unchanged.
- add_history and the SP history insert a new entry where pushing it and
  stably sorting would put it (after every entry at the same or a lower
  height) instead of pushing and re-sorting the whole list.
- The restore feeds transactions in height order (stable, so same-height
  transactions keep the graph's order, as before), so each entry is
  appended. Re-sorting the wallet-wide SP history after each insert, in the
  graph's hash order, made a 3,000-payment restore 128 ms instead of 78 ms;
  it now takes 73 ms.
- The characterisation test also checks the SP tweaks and the served raw
  bytes (it passes on the code before the refactor too).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review nit: the outpoints and tweaks come from the SP index, only the
scripts from the graph.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndex

Brings in integration (the merged deslop stack #67-#72) via snb-662-remove-full-receive-path.
No code change: this branch's tree stays the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from snb-662-remove-full-receive-path to integration October 30, 2026 06:33
@setavenger
setavenger merged commit 78f16a1 into integration Oct 30, 2026
2 checks passed
@setavenger
setavenger deleted the snb-671-shared-electrum-index branch October 30, 2026 06:33
setavenger added a commit that referenced this pull request Oct 30, 2026
Brings #18 up to date with master after the Friglet Deslop cleanup (#67-#76).
Conflicts resolved as described in #75:

- blindbit-lib/Cargo.toml, friglet/Cargo.toml: keep #18's bdk-sp rev pin and
  comments on top of the reduced manifest lines from SNB-670.
- scanning.rs: BIP352_K_MAX loses its #[expect(dead_code)] (the compile-time
  assert against bdk_sp::K_MAX now uses it); its doc no longer mentions the
  removed scan_transaction_full, which stays deleted.
- bip352_vectors.rs: the full-path K_max boundary test stays deleted; the live
  K_max test takes the label set and imports BIP352_K_MAX; its doc no longer
  links the deleted test.
- Cargo.toml: the secp256k1-sys comment no longer says the live K_max test is
  ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 30, 2026
Brings #22 up to date with master (Friglet Deslop #67-#76, #18, #80, #81).
No textual conflicts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant