Skip to content

fix(dash-spv): settle filter requests on network arrival - #1370

Draft
xdustinface wants to merge 9 commits into
fix/shared-stall-kicksfrom
fix/cfilters-settle-on-arrival
Draft

xdustinface wants to merge 9 commits into
fix/shared-stall-kicksfrom
fix/cfilters-settle-on-arrival

Conversation

@xdustinface

@xdustinface xdustinface commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

A CFilters or CfHeaders request stayed on the wire until its serial manager processed the reply, which could be a million messages later. A kicked peer then had its delivered batches sent again, and a busy manager looked like a stalled peer.

The network pump now settles a filter request as soon as its reply arrives. CFilters requests are keyed by stop hash, so the last cfilter of a reply identifies its request without a scan. Every path that drops a delivered batch releases it, and a batch rejected by verification or lost to a storage error is requested again instead of waited on forever. The lowest batch not yet delivered gets a second copy from the peer with the shortest queue once its peer has answered nothing for HEDGE_AFTER, and dropping a peer clears the hedges it held.

It is stacked on #1345, which sits on #1388 and #1387 below it.

Stacked on #1345, and will be retargeted to dev when that one merges.

@coderabbitai

coderabbitai Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor
Pre-merge checks: Passed 1 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Title check Pending This assessment has not completed. The check will be retried on the next pre-merge check run.
Passed Passed checks: 1 passed

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.36000% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.75%. Comparing base (568d586) to head (bb5a0d0).
⚠️ Report is 19 commits behind head on fix/shared-stall-kicks.

Files with missing lines Patch % Lines
dash-spv/src/sync/filters/manager.rs 97.02% 3 Missing ⚠️
dash-spv/src/network/manager.rs 99.63% 1 Missing ⚠️
Additional details and impacted files
@@                    Coverage Diff                     @@
##           fix/shared-stall-kicks    #1370      +/-   ##
==========================================================
+ Coverage                   85.67%   85.75%   +0.07%     
==========================================================
  Files                         268      268              
  Lines                       70326    70790     +464     
==========================================================
+ Hits                        60254    60706     +452     
- Misses                      10072    10084      +12     
Flag Coverage Δ
core 82.80% <ø> (ø)
rpc 52.10% <ø> (ø)
spv 94.45% <99.36%> (+0.05%) ⬆️
wallet 83.53% <ø> (ø)
Files with missing lines Coverage Δ
dash-spv/src/network/broker.rs 98.50% <100.00%> (+0.16%) ⬆️
dash-spv/src/network/peer.rs 91.17% <100.00%> (+0.68%) ⬆️
dash-spv/src/sync/filter_headers/manager.rs 96.79% <100.00%> (+0.08%) ⬆️
dash-spv/src/sync/filter_headers/sync_manager.rs 100.00% <ø> (ø)
dash-spv/src/sync/filters/pipeline.rs 98.55% <100.00%> (+0.03%) ⬆️
dash-spv/src/sync/filters/sync_manager.rs 100.00% <ø> (ø)
dash-spv/src/network/manager.rs 91.48% <99.63%> (+1.88%) ⬆️
dash-spv/src/sync/filters/manager.rs 97.84% <97.02%> (+0.01%) ⬆️

... and 2 files with indirect coverage changes

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

This PR has merge conflicts with the base branch. Please rebase or merge the base branch into your branch to resolve them.

@github-actions github-actions Bot added the merge-conflict The PR conflicts with the target branch. label Nov 9, 2026
@xdustinface
xdustinface force-pushed the fix/cfilters-settle-on-arrival branch from 3a69ba0 to 73f9869 Compare November 10, 2026 06:07
@github-actions github-actions Bot removed the merge-conflict The PR conflicts with the target branch. label Nov 10, 2026
@xdustinface
xdustinface force-pushed the fix/cfilters-settle-on-arrival branch 2 times, most recently from 60cb181 to 810e7de Compare November 10, 2026 06:11
The timeout monitor re-queued a culprit's requests and woke the router before it removed the culprit from the connected set, so the router could send the freed work straight back to the peer being closed. Remove and close the culprits first, then re-queue.

The router also works from a snapshot of the set that can predate the kick, so `Broker::route` now skips a peer whose connection is already closed.
@xdustinface
xdustinface force-pushed the fix/shared-stall-kicks branch from 80d80bf to ca66de7 Compare November 10, 2026 14:19
@github-actions github-actions Bot added the merge-conflict The PR conflicts with the target branch. label Nov 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This PR has merge conflicts with the base branch. Please rebase or merge the base branch into your branch to resolve them.

@xdustinface
xdustinface force-pushed the fix/cfilters-settle-on-arrival branch from 810e7de to 74f9283 Compare November 10, 2026 14:27
@github-actions github-actions Bot removed the merge-conflict The PR conflicts with the target branch. label Nov 10, 2026
The flow controller hands `Broker::hedge` a snapshot of the peer handles, so a peer the timeout monitor kicks right after the snapshot is still listed. `PeerHandle::send` does not check the closed token and the snapshot keeps the write half alive, so the send to such a peer succeeds. The hedge was then recorded against a dead peer, and the lowest block got no working second copy until its original request timed out. `hedge` now skips a closed peer, as `Broker::route` already does.
A slow peer streaming one large reply, such as a `headers2` of up to 8000 headers (about 344 KB), was kicked by `Broker::expire` because only a completed message counted as progress on its request, although its bytes kept arriving.

The reader now records whether bytes of an unfinished message sit in its buffer, which works for v1 and v2 framing alike without knowing the command. Each tick the timeout monitor collects the peers whose byte counter grew while such a message is arriving and refreshes their requests with one `Broker::receiving` call. Small unrelated messages such as pings, `inv`s and `addr`s arrive whole, so a peer that ignores a request while relaying them still stalls after one timeout.

A request also stalls once its peer completed no answer for `ANSWER_TIMEOUTS` (6) timeouts, so a peer trickling bytes into a message that never finishes cannot keep its request alive forever. The bound counts from the last answer, not from send time: peers answer in order, and a slow peer's queue can hold a request well past a minute while every answer ahead of it arrives.

`Broker::hedge` waits on answers only, as before, so a block still arriving slowly can be hedged.
When our own link stalls, every peer goes quiet at once. The timeout monitor judged each peer alone, so peers whose timeouts started a second apart crossed them one at a time and each looked like a lone stall. A testnet run lost four healthy peers in 14 s during a 40 s outage, and with one configured peer a 30 s outage kicked it and banned it for 10 minutes.

The monitor now hands its decision to a `StallJudge`. A silent peer is blamed only once our link provably works: some peer, this one included, received bytes more than a timeout into its silence. Silence that no peer witnessed is held as a possible stall of our link, for at most `LINK_STALL_TIMEOUTS` (6) request timeouts, 60 s with the 10 s base timeout, enough for the 40 s outages seen on testnet. Past the bound the set is more likely dead than our link, so the peer is kicked anyway. A new peer's first byte counter proves nothing, since its handshake may predate the stall.

Silent peers are kicked one per timeout, the one silent the longest first, so a stall of our link costs at most a single peer whatever share of the set it is. A replacement joining right after a kick does not unpace the next one.

When the first peer receives again after a silence nobody witnessed, our link counts as back from that moment. Both the witness and the hold bound then count from the later of a peer's silence and that return, so a peer still recovering from TCP backoff gets a full timeout instead of being kicked as soon as another peer speaks. Bytes from an idle peer later on cannot restart a dead peer's timeout.

`Broker::expire` reports when each culprit's stalled request went silent, so its silence is measured the same way as the owed-bytes check. A request past `ANSWER_TIMEOUTS` is blamed from its last answer, so the peer's own recent bytes witness our link and it is kicked rather than held.

While the judge holds a peer, `NetworkState` flags our link as stalled. The flow controller then skips the tick in a peer's serving rate, which otherwise stayed dragged down for 30 s after the stall, and neither the flow controller nor the supervisor names an underserving peer from rates the stall skewed.
A dashd test freezes dashd with `SIGSTOP` for 30 s in the middle of a sync, three request timeouts but under the bound for holding a silent peer, then resumes it with `SIGCONT`. Its socket stays open, so the client sees a silent connection, as when our own link stalls. The test asserts the client never drops its only peer and that sync completes. Against the old timeout monitor the peer is kicked about 10 s into the freeze.

`DashCoreNode::set_paused` sends the signals. The test is unix only, since Windows has no equivalent.
`StallJudge::held` collected the peers held for a possible stall of our link, but it was only read to check whether it was empty and to count the spared culprits. The judge now builds the kick set directly, from the culprits that already left plus the peer silent the longest when a kick is due, counts the held peers while judging, and keeps a `link_stalled` flag instead of the set. The kicks and the flag are unchanged.
`StallJudge::progress` holds each peer's `Liveness`, and "progress" already means request progress in the broker and the monitor. The field is now `liveness`.
The doc read "until the first second and again from the second", which mixes the ordinal with the time unit. It now names both fields as times in seconds.
A `CFilters` or `CfHeaders` key stayed on the wire until its serial manager processed the reply, up to a million messages later. A kicked peer then had its delivered batches sent again, and a busy manager read as a stalled peer.

- `CFilters` requests are keyed by stop hash, so the last `cfilter` of a reply identifies its request without a scan.
- The pump settles the key when the reply arrives. Every way a manager drops a delivered batch releases it: a reply that leaves its batch incomplete, a filter that resolves to no stored header, a batch thrown away by a rescan and a `cfheaders` reply that matches nothing.
- A batch rejected by verification or lost to a storage error is requested again instead of waited on forever.
- Filters are stored in height order, so the lowest batch not yet delivered gets a second copy from the peer with the shortest queue once its peer answered nothing for `HEDGE_AFTER`, at most once per request and past the peers' caps.
- Dropping a peer clears the hedges it held, so a request whose second copy was lost with its peer can be hedged again.
@xdustinface
xdustinface force-pushed the fix/shared-stall-kicks branch from ca66de7 to e89874d Compare November 11, 2026 01:21
@xdustinface
xdustinface force-pushed the fix/cfilters-settle-on-arrival branch from 74f9283 to bb5a0d0 Compare November 11, 2026 01:21
@xdustinface
xdustinface force-pushed the fix/shared-stall-kicks branch from e89874d to 5cc69f8 Compare November 12, 2026 22:33
@github-actions github-actions Bot added the merge-conflict The PR conflicts with the target branch. label Nov 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This PR has merge conflicts with the base branch. Please rebase or merge the base branch into your branch to resolve them.

@xdustinface
xdustinface force-pushed the fix/shared-stall-kicks branch from 5cc69f8 to 1095e19 Compare November 12, 2026 22:46
@xdustinface
xdustinface force-pushed the fix/shared-stall-kicks branch 2 times, most recently from 3b3971d to a37e9b9 Compare November 13, 2026 02:57
@xdustinface
xdustinface force-pushed the fix/shared-stall-kicks branch 6 times, most recently from 4d4641f to 46beb07 Compare November 13, 2026 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-conflict The PR conflicts with the target branch.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant