Skip to content

fix(dash-spv): speed up the rescan for derived scripts - #1368

Merged
ZocoLini merged 2 commits into
devfrom
fix/rescan-derived-scripts
Nov 9, 2026
Merged

ZocoLini merged 2 commits into
devfrom
fix/rescan-derived-scripts

Conversation

@xdustinface

@xdustinface xdustinface commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

After #1344, two costs still slowed the rescan for newly derived scripts.

The wallet manager checked each transaction of a block once. A transaction paying an address that a later transaction of the same block derives was missed, so the rescan applied that block again for every round of derivation. A testnet sync applied one CoinJoin block six times in two seconds. WalletManager::process_block_for_wallets now checks the whole block again for the wallets that derived addresses, until none derives more. A transaction found again is recorded once. The new WalletInterface::checks_derived_addresses tells BlockMatchTracker that the deriving block does not need to be applied again. It defaults to false, which keeps the old behaviour for other implementations.

Every block that derived scripts also made the next tick rescan all scanned batches, up to 16 of them. The lowest batch, which the commit frontier waits on, queued its block requests behind the others. New scripts are now deferred to each scanned batch that has not matched them. The lowest batch is rescanned right away, each tick rescans the lowest batch that still has deferred scripts, and a batch matches its deferred scripts before it commits.

PR Hygiene · 00d20ff

  • Bots — coderabbitai skipped after its own rate limit
  • Self-review — posted; again after any push
  • Build green
  • Approvals — you own every area touched; none needed

When every merge requirement is met, the PR Hygiene check passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.

Two rescan costs remained once blocks that already saw the new scripts were skipped.

- `WalletManager::process_block_for_wallets` checked each transaction once, so a transaction paying an address that a later transaction of the same block derives was missed, and the rescan applied the block again for every round of derivation. A testnet sync applied one CoinJoin block six times in two seconds this way. The wallets that derived addresses now check the whole block again until none derives more. A transaction found again is listed and recorded once, and a record corrected by a later pass is reported as inserted. `WalletInterface::checks_derived_addresses` says whether an implementation does this. `WalletManager` returns `true`, so `BlockMatchTracker` no longer re-applies the deriving block. The default is `false`, which keeps re-applying it.
- Each `BlockProcessed` with new scripts made the next tick, and every commit attempt, rescan all up to 16 scanned batches, so the lowest batch's `BlocksNeeded` waited behind the others while the commit frontier waited on exactly those blocks. New scripts are now deferred to each scanned batch that has not matched them, with the pass that derived them. The lowest batch is rescanned while the `BlockProcessed` is handled, and each tick rescans the lowest batch that still has deferred scripts. A batch matches its deferred scripts before it commits, and reconciliation still tests whatever else the wallet watches.
@coderabbitai

coderabbitai Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 11 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between c7bb2ce and 00d20ff.

Files selected for processing: 9

  • dash-spv/src/sync/filters/batch.rs
  • dash-spv/src/sync/filters/block_match_tracker.rs
  • dash-spv/src/sync/filters/coinjoin_gap_discovery_tests.rs
  • dash-spv/src/sync/filters/manager.rs
  • dash-spv/src/sync/filters/sync_manager.rs
  • key-wallet-manager/src/process_block.rs
  • key-wallet-manager/src/test_utils/coinjoin.rs
  • key-wallet-manager/src/test_utils/mod.rs
  • key-wallet-manager/src/wallet_interface.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.88372% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.52%. Comparing base (c7bb2ce) to head (00d20ff).
⚠️ Report is 2 commits behind head on dev.

Files with missing lines Patch % Lines
key-wallet-manager/src/process_block.rs 92.75% 5 Missing ⚠️
dash-spv/src/sync/filters/manager.rs 97.11% 3 Missing ⚠️
key-wallet-manager/src/wallet_interface.rs 0.00% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##              dev    #1368      +/-   ##
==========================================
+ Coverage   85.46%   85.52%   +0.06%     
==========================================
  Files         268      268              
  Lines       69420    69593     +173     
==========================================
+ Hits        59329    59519     +190     
+ Misses      10091    10074      -17     
Flag Coverage Δ
core 82.63% <ø> (ø)
rpc 52.10% <ø> (ø)
spv 94.28% <97.90%> (+0.01%) ⬆️
wallet 83.51% <88.88%> (+0.12%) ⬆️
Files with missing lines Coverage Δ
dash-spv/src/sync/filters/batch.rs 97.91% <100.00%> (+0.27%) ⬆️
dash-spv/src/sync/filters/block_match_tracker.rs 99.54% <100.00%> (+0.03%) ⬆️
dash-spv/src/sync/filters/sync_manager.rs 100.00% <ø> (ø)
dash-spv/src/sync/filters/manager.rs 97.92% <97.11%> (-0.06%) ⬇️
key-wallet-manager/src/wallet_interface.rs 7.89% <0.00%> (-0.68%) ⬇️
key-wallet-manager/src/process_block.rs 93.54% <92.75%> (-0.10%) ⬇️

... and 5 files with indirect coverage changes

The helper that derives CoinJoin account 0 External addresses beyond the wallet's watch window moved to `key_wallet_manager::test_utils::coinjoin_external_addresses`, and both the `process_block` tests and the dash-spv CoinJoin gap discovery tests now use it instead of their own copies.
@xdustinface
xdustinface marked this pull request as ready for review November 9, 2026 14:17
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Waiting for bot review — coderabbitai not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added the waiting-bots Waiting for the review bots to report on this head label Nov 9, 2026
@xdustinface
xdustinface requested a review from ZocoLini November 9, 2026 14:21
@xdustinface

Copy link
Copy Markdown
Collaborator Author

/self-reviewed

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Policy satisfied — this can merge.
Full checklist in the description.

@github-actions github-actions Bot added bot-review-skipped A required review bot did not report; it was skipped by the window or by a person. and removed waiting-bots Waiting for the review bots to report on this head labels Nov 9, 2026
@ZocoLini
ZocoLini merged commit e191fb7 into dev Nov 9, 2026
38 checks passed
@ZocoLini
ZocoLini deleted the fix/rescan-derived-scripts branch November 9, 2026 15:50
PastaPastaPasta added a commit that referenced this pull request Nov 13, 2026
…passes

Since #1368, process_block_for_wallets checks a block again until no
wallet derives more, so a block that derives its own addresses is
credited by one call and the filter rescan no longer delivers it again.
The rescan still delivers an earlier block again when a later block
derives an address it pays, which is the path #1326 fixed.

Both tests now live in process_block.rs and use that cross-block shape
on BIP44 account 0; dev's part of the credits test keeps the shared
CoinJoin helper from test_utils:
- test_process_block_credits_addresses_the_block_derives also delivers
  a block whose far output only the next block derives, applies a
  ChainLock, and delivers it again: the coins, the balance and the last
  emitted record must own every output.
- test_rescan_after_an_account_reload_keeps_late_outputs_spent spends
  both outputs in the block after, prunes observed spends at a
  checkpoint, reloads the account through serde, and delivers the
  blocks again: both coins must end spent.

rescan_tests.rs, its rescan helper and MAX_PASSES are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ZocoLini pushed a commit that referenced this pull request Nov 14, 2026
* test(key-wallet-manager): cover rescans over chainlocked blocks

#1326 fixed crediting outputs that a rescan newly matches on a known or finalized transaction, with tests at the `check_core_transaction` level. These tests drive `WalletManager::process_block_for_wallets` the way the filter rescan does: the block is processed again while a pass derives new scripts, with the ChainLock applied before the first pass, between passes, or not at all.

For a transaction paying a run of `gap + gap/2` addresses, and for a transaction whose far output only a later transaction's gap extension reaches, they check the coins held, the balance, and the last record the `BlockProcessed` events carried, which is what a store built from the events keeps. Another set checks that outputs spent in the next block end spent after the rescan, including after a checkpoint prunes the observed spends and after the account is reloaded through serde.

The `ManagedAccountRefMut::confirm_transaction` doc said it returns a record only when the confirmation status changes. It also returns one that gains outputs derived since it was recorded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(key-wallet-manager): reuse the shared block and tx helpers

The rescan tests built their blocks with a local `block_at`, a near copy of `Block::dummy`. They now use `Block::dummy` and `TX_AMOUNT`, and their multi-input, multi-output `transaction` builder moves to `test_helpers` as `create_tx`, which `create_tx_paying_to` now builds on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(key-wallet-manager): trim the rescan tests to two

The rescan tests ran ten cases over three ChainLock timings. Under mutants of the #1326 code, the case where the ChainLock lands before the first pass caught exactly what the case where it lands between passes caught, the gap-limit run and the spent-output cases caught nothing the others did not, and the unchainlocked path is caught by #1326's own wallet_checker tests. Two tests remain: a later payment's derived output with the ChainLock between passes, and the spend after an account reload.

One `rescan` helper now delivers the blocks with their heights, then re-delivers them until a pass derives nothing, with a hook after the first pass. The payments come from `create_tx_paying`, which takes a list of addresses and which `create_tx_paying_to` now builds on; the spender is the one local builder left. The `confirm_transaction` doc is shortened.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(key-wallet-manager): consolidate rescan coverage on dev's block passes

Since #1368, process_block_for_wallets checks a block again until no
wallet derives more, so a block that derives its own addresses is
credited by one call and the filter rescan no longer delivers it again.
The rescan still delivers an earlier block again when a later block
derives an address it pays, which is the path #1326 fixed.

Both tests now live in process_block.rs and use that cross-block shape
on BIP44 account 0; dev's part of the credits test keeps the shared
CoinJoin helper from test_utils:
- test_process_block_credits_addresses_the_block_derives also delivers
  a block whose far output only the next block derives, applies a
  ChainLock, and delivers it again: the coins, the balance and the last
  emitted record must own every output.
- test_rescan_after_an_account_reload_keeps_late_outputs_spent spends
  both outputs in the block after, prunes observed spends at a
  checkpoint, reloads the account through serde, and delivers the
  blocks again: both coins must end spent.

rescan_tests.rs, its rescan helper and MAX_PASSES are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(key-wallet-manager): assert the reloaded rescan credits the late coin

With keep-finalized-transactions the reloaded account still holds the
spender's record, so the parent's re-delivery credits nothing. Under
the default features finalized records are pruned, and the re-delivery
must credit the far output before the spender's re-delivery spends it
again. Checking that keeps the reload test from passing when the
crediting itself stops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(key-wallet-manager): share the standalone external address helper

The rescan tests carried a private copy of the shared CoinJoin helper
for BIP44 account 0. The shared helper now takes the account type, as
`external_addresses`, and `coinjoin_external_addresses` calls it for
CoinJoin account 0, so its callers are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(key-wallet-manager): read the BIP44 gap limit from the account

late_and_extending only needs the external gap limit, so take it from
`external_gap_limit()` and drop the `external_pool` clone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(key-wallet-manager): build spend_first_output_of on spending

Both helpers built a version 2 spender with default inputs; let
spend_first_output_of start from `spending` and set the output it
needs. The transactions it builds are byte-identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot-review-skipped A required review bot did not report; it was skipped by the window or by a person.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants