Skip to content

fix: record owned outputs so spends are detected; persist labels - #19

Merged
setavenger merged 12 commits into
integrationfrom
fix/cli-spent-and-labels
Oct 1, 2026
Merged

setavenger merged 12 commits into
integrationfrom
fix/cli-spent-and-labels

Conversation

@setavenger

@setavenger setavenger commented Sep 24, 2026 •

Copy link
Copy Markdown
Owner

Important

Contains #17 and #20 — land #17 and #20 first. This branch has snb-541-short-path-vectors (#17) and fix/stream-unindexed-height (#20) merged in, so it merges cleanly after them (planned order #17 → #20 → #19 → #21). Once they are in integration, this PR's diff is only its own commits. The #17 merge resolves scanning.rs as announced below: sync_owned_outputs() stays after apply_matched_block, both test-module lines stay, and the dead if probable_match.spent {} block is dropped.

Warning

Downgrade break. The state file's owned_outputs array now holds records (objects), and the file carries format_version: 1. A build older than this PR cannot read such a file once it lists an owned output: load_from_file fails with Failed to deserialize ChangeSet. Error: invalid type: map, expected a string (checked with the integration 93f6a92 code). Its load_scanner then logs failed to load state file, creating new scanner, starts a fresh wallet from the configured start height, and overwrites the file on its next save. Nothing is lost for good, because the rescan rebuilds it, but the downgraded wallet rescans from its birthday. A file with an empty owned_outputs still loads in old builds. There is no CHANGELOG or release-notes file in this repo, so this note is the record.

What was broken

Spends of owned outputs were almost never detected, so blindbit-cli and friglet reported balances that still included spent coins.

  • Scanner::owned_outputs (blindbit-lib/src/scanner/scanner.rs:64, persisted via changeset.rs:64) is the set the per-block spent check matches the oracle's 8-byte spent_outputs prefixes against (scanning.rs:667-685). Its only writer, add_owned_output (scanner.rs:399), had no call site. The set was always empty, so the check never fired.
  • The full block is fetched only on a probable match (scanning.rs:119-174). A block whose only wallet-relevant transaction was a spend was never fetched, so the spending tx never reached the wallet graph. The balance (graph.balance, scanning.rs:247, :447) kept the coin as unspent. Friglet's Electrum index never recorded the spend, so Sparrow saw it as unspent too.
  • A spend was detected only by accident: when the same tx also paid the wallet (for example SP change), the block was fetched for the receive.
  • Friglet runs the same scan_block_range from watch_chain, so it has the same bug. Its outputs_found counter, seeded from the always-empty owned_outputs array, also started at 0.

Labelled outputs dropped out of the wallet after every restart.

Scanner::load_from_file blanks the persisted label_lookup. from_changeset then rebuilt the indexer with SpIndexerV2::try_from, which re-runs the BIP-352 scan over every stored tx, and only registered the labels afterwards. So every labelled output (change included, m = 0) was missing from the restored index:

  • it was missing from the balance;
  • a later spend of it was never detected.

This affects both the CLI and friglet.

No label was persisted per found output. The state file had nothing a user or a restore could use to tell labelled payments apart.

Fix

  • Scanner::sync_owned_outputs runs after every applied block and on restore. It derives one OwnedOutputRecord per output the indexer matched: outpoint, full x-only key, amount, height, and label m (None means unlabelled, Some(0) means change).
  • Spend confirmation is by exact outpoint. A record is marked spent_by/spent_height only when a confirmed tx in the wallet graph has an input whose previous_output is exactly that outpoint. The 8-byte prefixes go through a prefix → unspent-outpoint map and only decide which blocks to fetch. A foreign key with the same prefix causes one extra block fetch and marks nothing. Once fetched, apply_block_relevant puts the spending tx into the graph, so the balance and the Electrum history pick it up.
  • Persistence. The owned_outputs array in the state file now holds these records:
    {"outpoint":"<txid>:<vout>","pubkey":"<x-only hex>","amount_sat":50000,"height":100,"label":null,"spent_by":"<txid>","spent_height":200}
    Legacy bare-hex entries still deserialize and are dropped (no writer ever produced them); records are re-derived from the indexer on load. Friglet's owned_outputs_count still just counts the array.
  • Restore. Labels are registered before the persisted changeset is applied.
  • apply_matched_block is extracted from scan_block_range with the same shape as in test: BIP-352 vectors through the live short receive path (SNB-541) #17, so the tests can drive the real per-block path.

State files written before this fix (commit 04d57ad)

Such a file loads, but its wallet never fetched blocks whose only wallet-relevant transaction was a spend, so those spends stay missing. Before this commit nothing said so.

  • Explicit format version. ChangeSet::format_version (STATE_FORMAT_VERSION = 1). It is absent (0) in every file written before this PR, and every save writes 1. A heuristic such as "owned_outputs is empty" would misfire on a current wallet that simply has no outputs.

  • One automatic rescan. Restoring a format-0 state rewinds last_scanned_block_height to (oldest output the graph still shows unspent) − 1. That is the lowest height a missed spend can be at. The next scan revisits those blocks and, with the outputs now recorded, fetches the blocks that spend them. Blocks already in the graph re-apply idempotently, and the Electrum history is upserted. If no output is unspent, nothing is rewound and only the version is bumped. The rescan runs once: the next save stores the lower height with format 1.

  • Visible. A warning is logged (state file predates owned-output records ... rescanning from the oldest unspent output). The rewind is published as ScanHealth::state_rescan (from fix(scanner): stop at an unscannable oracle block instead of silently skipping it #20's health mechanism), reported in friglet's GetStatus as scan_health.state_rescan, and shown as a note in the tray's status window. It clears once the scan is back at the old height. Meanwhile the Electrum tip stays at the old height, so Sparrow never sees wallet history above the tip.

  • Old-format test. testdata/state_before_owned_outputs.json was written by the pre-fix code (integration 93f6a92, generated with its own Scanner/save_to_file): a payment and an m = 0 change output at 100, scanned to 300, and a spend at 200 that the old code never fetched. state_file_from_before_owned_outputs_rescans_for_missed_spends loads it as written and with legacy bare-hex owned_outputs entries. It checks:

    • both records, including the change label (the restore-order fix);
    • the rewind to 99, the StateRescan { 100, 300 } note and the Electrum tip held at 300;
    • that rescanning block 200 marks the spend (balance 70 000 → 20 000);
    • that the upgraded file restores without another rewind.

    ..._without_unspent_outputs_is_not_rescanned covers the no-output case. Red checks (each reverted): without the rewind call, the first test fails with left: 300, right: 99; without the tip hold, it fails with left: Some(99), right: Some(300).

Verification

New tests are in blindbit-lib/src/scanner/owned_outputs_tests.rs. They drive scan_short_block_data → apply_matched_block → sync_owned_outputs exactly as scan_block_range does, with only the P2P fetch replaced:

Test Checks
spend_in_later_block_marks_output_spent_and_balance_drops receive, then a spend in a later block with no output back to the wallet: marked spent, balance 50 000 → 0; an unrelated block is not fetched
foreign_output_with_colliding_prefix_marks_nothing our prefix in spent_outputs, but the tx spends a different outpoint: block fetched, nothing marked, balance unchanged
owned_outputs_labels_and_spends_survive_restart save → load → from_changeset: records, labels and spends identical, balance unchanged, spend detection still works after the restart
labelled_receive_records_its_label unlabelled / m = 0 / m = 3 receives record None / Some(0) / Some(3)

Each test fails when its fix is reverted:

  • sync_owned_outputs records nothing (the old orphaned-writer state): all 4 fail.
  • The spend is marked on the prefix alone: the collision test fails.
  • Labels are registered after apply_changeset (the old restore order): the restart test fails with restored balance.

Notes / out of scope

🤖 Generated with Claude Code

setavenger and others added 4 commits September 24, 2026 10:26
…SNB-541)

The official vectors only reached Scanner::scan_transaction_full, which has
no production caller. The daemon receives through scan_short_block_data ->
scan_transaction_short (k = 0, labels enumerated by hand, 8-byte x-only
prefixes) and, on a probable match, apply_block_relevant on the external
indexer. A false negative in the short step means the block is never
fetched: silent loss.

- Serve every receiving vector the way the oracle does (taproot outputs'
  8-byte x-only prefixes, the 33-byte tweak, the display-order txid) and
  assert a probable match wherever the vector expects >= 1 output. Pins
  which k = 0 branch did the work: unlabelled P_0, labels of both output
  parities, and the change label m = 0. False positives are counted and
  reported (zero on these vectors).
- Drive the real post-match step for the same vectors: the vector tx in a
  full block through apply_block_relevant, asserting the stored outputs and
  priv_key_tweaks equal expected.outputs.
- Extract the post-match step (txid mapping + apply_block_relevant + stage
  merge) from scan_block_range into Scanner::apply_matched_block, unchanged,
  so the test exercises the production code rather than a copy.
- K_max through the live path is SNB-540: an ignored test records that it
  stores 2324 outputs where BIP-352 allows 2323.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The scanner's `owned_outputs` set, which the per-block spent-output check
matches the oracle's 8-byte `spent_outputs` prefixes against, had a writer
(`add_owned_output`) that no call site ever invoked. The set stayed empty,
the spent check never fired, and a block whose only wallet-relevant
transaction was a spend (no Silent Payments output back to the wallet) was
never fetched. The spending transaction never reached the wallet graph, so
blindbit-cli and friglet reported a balance that still included spent coins
and Electrum clients never saw the spend. Spends were only noticed by
accident, when the spending transaction also paid the wallet (e.g. change).

- `Scanner::sync_owned_outputs` rebuilds the owned-output records from the
  indexer after every applied block and on restore: outpoint, full x-only
  key, amount, height, and the BIP-352 label `m` the output matched.
- A record is marked spent only when a confirmed transaction in the wallet
  graph spends exactly that outpoint. The 8-byte prefixes, matched through
  a prefix -> unspent-outpoint map, only decide which blocks to fetch; a
  foreign key sharing the prefix fetches the block but marks nothing.
- The state file's `owned_outputs` array now holds these records (label
  and spend included). Legacy bare-hex entries are accepted and dropped;
  the records are re-derived from the indexer on load.
- Restore registered labels only after re-indexing the stored
  transactions, and `load_from_file` blanks the persisted label table, so
  every labelled output (change included, m = 0) silently dropped out of
  the index, the balance and spend detection after each restart. Labels
  are now registered before the changeset is applied.
- `apply_matched_block` is extracted from `scan_block_range` (same shape
  as in #17) so the tests drive the real per-block path without P2P.

Tests (in-process, `owned_outputs_tests.rs`): receive then spend in a later
block marks the output spent and the balance drops; a colliding foreign
prefix marks nothing; records, labels and spends survive save/load/restore
and spend detection keeps working after it; a labelled receive records the
right m.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… messages

Adds an in-process seam (BlockScanDataStream) between scan_block_range and
the scan loop, with no behaviour change, and tests that drive a real Scanner
through it. They fail on this commit: an empty-hash block mid-range is
treated as an empty block (payments in it are lost and the scan reports
success), height gaps and short streams are accepted, and any error status
panics the process via unwrap().

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ipping it

An oracle answers a height it has not indexed with an OK message carrying an
empty block hash and no data. scan_block_range took that as a block with no
payments: it warned, advanced the scan height past it and reported success,
so any payment in that block was never found (proven by the testkit mutation
check unindexed-ok-empty: 80000 sats reported instead of 120600).
Separately, every stream message was unwrap()ed, so an error status
(NOT_FOUND, UNAVAILABLE) panicked the process, as did failing to open the
stream.

Now each message must be the next expected height and carry a valid
32-byte, non-zero block hash; otherwise, and on any error status or a stream
that ends before the requested end, the scan returns an error naming the
height to rescan from. The scan height stays at the last fully processed
block, so watch_chain retries from there on its next poll (every 10 s, no
spin) and blindbit-cli exits non-zero after saving that height. The example
is updated the same way.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
setavenger and others added 2 commits October 1, 2026 12:53
…d-labels

Resolve scanning.rs: keep sync_owned_outputs() after apply_matched_block,
keep both test-module lines, and drop the dead
`if probable_match.spent {}` block that #17's extraction carried over.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s below it; report stalls

A wallet whose start height lies below the oracle's first indexed block
(the hosted oracle has nothing below 800000 on mainnet and 100000 on
signet) stopped at its first height on every poll, forever, with only an
error-level log line. Since #20 refuses to scan past an unindexed height,
that is the only thing such a wallet ever did.

- When `watch_chain`'s scan stops at the wallet's start height because
  the oracle has not indexed it, the scanner locates the oracle's lowest
  indexed height with GetBlockHashByHeight lookups and starts there,
  logs a warning, persists the adjustment (`oracle_floor_start`, optional
  in the state file) and publishes it in the scan health.
- Only heights below the oracle's floor are skipped. A height above the
  first scanned block, or one with an indexed height below it (a gap in
  the oracle's index), keeps #20's stop-and-retry. The search is a binary
  search verified by exponential back-off probes below the boundary, so a
  gap above the floor does not move the floor up. NOT_FOUND and an empty
  or zero hash both mean "not indexed".
- A scan that cannot get past a height (or cannot reach the oracle) is
  published as a stall: height, reason, and the time of the first failed
  attempt at that height. It clears when the scan makes progress.
- Stream stops are a typed `ScanStopped` error (same message text) so the
  loop can tell "not indexed" from other failures.
- Health lives in the wallet's Electrum index, which status readers lock
  without waiting on the scan task. friglet's GetStatus reports it as
  `StatusInfo::scan_health` (serde default, so old trays and daemons
  interoperate), and the tray's status window shows a stall line and a
  note for an adjusted start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd-labels

#20 lands before this PR, so this branch carries it to merge cleanly
after it. changeset.rs: keep both imports (#20's OracleFloorStart and
this branch's OwnedOutputRecord); the serde_hex module #20 still had is
the one this branch replaced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ends it missed

A state file written before owned outputs were recorded still loads, but
its wallet never fetched blocks whose only wallet-relevant transaction
was a spend, so those spends stay missing and nothing said so.

- The state file now carries `format_version` (STATE_FORMAT_VERSION = 1;
  absent = 0, written before the owned-output records). Every save writes
  the current version.
- Restoring a format-0 state rewinds the scan to just below its oldest
  unspent output, the lowest height a missed spend can be at. The next
  scan revisits those blocks and, with the outputs now recorded, fetches
  the ones that spend them. Re-applying blocks already in the graph is
  idempotent. With no unspent output there is nothing to re-check and
  only the version is bumped.
- The rewind is logged and published in the scan health
  (`StateRescan`), shown in friglet's status and the tray, and cleared
  once the scan is back where the state had been. While it runs, the
  Electrum tip stays at that height, so clients never see wallet history
  above the tip.
- Test fixture `testdata/state_before_owned_outputs.json` was written by
  the pre-fix code (integration 93f6a92): payment + change at 100,
  scanned to 300, spend at 200 missing. The test loads it as written and
  with legacy bare-hex `owned_outputs` entries, checks the restored
  records and labels, the rewind to 99, the held Electrum tip, that the
  rescan finds the spend, and that the upgraded file restores without
  another rewind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger and others added 2 commits October 1, 2026 14:02
…n attribute)

The status window's error, stall and scan-notes blocks use class="col",
whose display: flex beats the user-agent [hidden] rule, so an empty
"Last error" label was always shown and the new stall/notes blocks
would be too. Seen in a screenshot against a live daemon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd-labels

Picks up the tray CSS fix for hidden status blocks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 1, 2026
…fix/reorg-rollback

Keeps this branch a superset of the stack below it.

- changeset.rs / scanner.rs: keep both sides' new fields
  (scanned_block_hashes; format_version, oracle_floor_start).
- scanning.rs: keep this branch's scan_range_from / BlockStreamSource;
  OracleStreams::open now reports a refused stream as #20's typed
  ScanStopped. watch_chain combines both: the no-new-block hash check
  from here, and #20's floor start (only for a real range) and stall
  reporting, with the ReorgTooDeep log kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 1, 2026
Picks up #20's tray CSS fix for hidden status blocks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 1, 2026
…ood/preview

Brings in the #17/#20/#19/#21/#31 stack. One conflict, in
friglet/src/electrum/mod.rs, resolved keeping both sides:
- #26's ChainSource (p2p/network/oracle) and pending-broadcast store,
- #21's reorg handling: checkpoints behind a mutex, reorg_rx feeding
  forget_blocks_above; its test now builds the server state with #26's
  test helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 1, 2026
Brings in the fix stack (#17, #20, #19, #21) so the T0 harness runs
against the receive path that records owned outputs.

scanning.rs: both sides extracted apply_matched_block; kept #21's
sync_owned_outputs() call after it in the scan loop and its doc line,
and kept both test modules (owned_outputs_tests from #21, testkit_t0
from this branch).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger and others added 2 commits October 1, 2026 15:03
Since ScanStopped became a struct (22c50bf), blindbit-cli printed a
stopped scan as `Error: ScanStopped { height: 104, reason: "...",
not_indexed: true }`, the Debug form main uses for a returned error. The
message the type renders for the user, "scan stopped at height 104:
...; nothing from height 104 on was scanned, rescan from height 104", no
longer appeared. The testkit's mutation checks match that message and
went red for stale-view, truncated-stream, close-mid-stream and
unindexed-ok-empty.

main now runs the command and prints the error with Display. The exit
status of a failed command is still 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd-labels

Brings in the blindbit-cli error display fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@setavenger
setavenger merged commit 824613f into integration Oct 1, 2026
1 check passed
@setavenger
setavenger deleted the fix/cli-spent-and-labels branch October 1, 2026 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant