Repository navigation
fix(dash-spv): settle filter requests on network arrival - #1370
Draft
xdustinface wants to merge 9 commits into
Draft
xdustinface wants to merge 9 commits into
xdustinface wants to merge 9 commits into
Conversation
Contributor
Pre-merge checks:
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## fix/shared-stall-kicks #1370 +/- ##
==========================================================
+ Coverage 85.67% 85.75% +0.07%
==========================================================
Files 268 268
Lines 70326 70790 +464
==========================================================
+ Hits 60254 60706 +452
- Misses 10072 10084 +12
|
Contributor
|
This PR has merge conflicts with the base branch. Please rebase or merge the base branch into your branch to resolve them. |
xdustinface
force-pushed
the
fix/cfilters-settle-on-arrival
branch
from
November 10, 2026 06:07
3a69ba0 to
73f9869
Compare
xdustinface
force-pushed
the
fix/cfilters-settle-on-arrival
branch
2 times, most recently
from
November 10, 2026 06:11
60cb181 to
810e7de
Compare
The timeout monitor re-queued a culprit's requests and woke the router before it removed the culprit from the connected set, so the router could send the freed work straight back to the peer being closed. Remove and close the culprits first, then re-queue. The router also works from a snapshot of the set that can predate the kick, so `Broker::route` now skips a peer whose connection is already closed.
xdustinface
force-pushed
the
fix/shared-stall-kicks
branch
from
November 10, 2026 14:19
80d80bf to
ca66de7
Compare
Contributor
|
This PR has merge conflicts with the base branch. Please rebase or merge the base branch into your branch to resolve them. |
xdustinface
force-pushed
the
fix/cfilters-settle-on-arrival
branch
from
November 10, 2026 14:27
810e7de to
74f9283
Compare
The flow controller hands `Broker::hedge` a snapshot of the peer handles, so a peer the timeout monitor kicks right after the snapshot is still listed. `PeerHandle::send` does not check the closed token and the snapshot keeps the write half alive, so the send to such a peer succeeds. The hedge was then recorded against a dead peer, and the lowest block got no working second copy until its original request timed out. `hedge` now skips a closed peer, as `Broker::route` already does.
A slow peer streaming one large reply, such as a `headers2` of up to 8000 headers (about 344 KB), was kicked by `Broker::expire` because only a completed message counted as progress on its request, although its bytes kept arriving. The reader now records whether bytes of an unfinished message sit in its buffer, which works for v1 and v2 framing alike without knowing the command. Each tick the timeout monitor collects the peers whose byte counter grew while such a message is arriving and refreshes their requests with one `Broker::receiving` call. Small unrelated messages such as pings, `inv`s and `addr`s arrive whole, so a peer that ignores a request while relaying them still stalls after one timeout. A request also stalls once its peer completed no answer for `ANSWER_TIMEOUTS` (6) timeouts, so a peer trickling bytes into a message that never finishes cannot keep its request alive forever. The bound counts from the last answer, not from send time: peers answer in order, and a slow peer's queue can hold a request well past a minute while every answer ahead of it arrives. `Broker::hedge` waits on answers only, as before, so a block still arriving slowly can be hedged.
When our own link stalls, every peer goes quiet at once. The timeout monitor judged each peer alone, so peers whose timeouts started a second apart crossed them one at a time and each looked like a lone stall. A testnet run lost four healthy peers in 14 s during a 40 s outage, and with one configured peer a 30 s outage kicked it and banned it for 10 minutes. The monitor now hands its decision to a `StallJudge`. A silent peer is blamed only once our link provably works: some peer, this one included, received bytes more than a timeout into its silence. Silence that no peer witnessed is held as a possible stall of our link, for at most `LINK_STALL_TIMEOUTS` (6) request timeouts, 60 s with the 10 s base timeout, enough for the 40 s outages seen on testnet. Past the bound the set is more likely dead than our link, so the peer is kicked anyway. A new peer's first byte counter proves nothing, since its handshake may predate the stall. Silent peers are kicked one per timeout, the one silent the longest first, so a stall of our link costs at most a single peer whatever share of the set it is. A replacement joining right after a kick does not unpace the next one. When the first peer receives again after a silence nobody witnessed, our link counts as back from that moment. Both the witness and the hold bound then count from the later of a peer's silence and that return, so a peer still recovering from TCP backoff gets a full timeout instead of being kicked as soon as another peer speaks. Bytes from an idle peer later on cannot restart a dead peer's timeout. `Broker::expire` reports when each culprit's stalled request went silent, so its silence is measured the same way as the owed-bytes check. A request past `ANSWER_TIMEOUTS` is blamed from its last answer, so the peer's own recent bytes witness our link and it is kicked rather than held. While the judge holds a peer, `NetworkState` flags our link as stalled. The flow controller then skips the tick in a peer's serving rate, which otherwise stayed dragged down for 30 s after the stall, and neither the flow controller nor the supervisor names an underserving peer from rates the stall skewed.
A dashd test freezes dashd with `SIGSTOP` for 30 s in the middle of a sync, three request timeouts but under the bound for holding a silent peer, then resumes it with `SIGCONT`. Its socket stays open, so the client sees a silent connection, as when our own link stalls. The test asserts the client never drops its only peer and that sync completes. Against the old timeout monitor the peer is kicked about 10 s into the freeze. `DashCoreNode::set_paused` sends the signals. The test is unix only, since Windows has no equivalent.
`StallJudge::held` collected the peers held for a possible stall of our link, but it was only read to check whether it was empty and to count the spared culprits. The judge now builds the kick set directly, from the culprits that already left plus the peer silent the longest when a kick is due, counts the held peers while judging, and keeps a `link_stalled` flag instead of the set. The kicks and the flag are unchanged.
`StallJudge::progress` holds each peer's `Liveness`, and "progress" already means request progress in the broker and the monitor. The field is now `liveness`.
The doc read "until the first second and again from the second", which mixes the ordinal with the time unit. It now names both fields as times in seconds.
A `CFilters` or `CfHeaders` key stayed on the wire until its serial manager processed the reply, up to a million messages later. A kicked peer then had its delivered batches sent again, and a busy manager read as a stalled peer. - `CFilters` requests are keyed by stop hash, so the last `cfilter` of a reply identifies its request without a scan. - The pump settles the key when the reply arrives. Every way a manager drops a delivered batch releases it: a reply that leaves its batch incomplete, a filter that resolves to no stored header, a batch thrown away by a rescan and a `cfheaders` reply that matches nothing. - A batch rejected by verification or lost to a storage error is requested again instead of waited on forever. - Filters are stored in height order, so the lowest batch not yet delivered gets a second copy from the peer with the shortest queue once its peer answered nothing for `HEDGE_AFTER`, at most once per request and past the peers' caps. - Dropping a peer clears the hedges it held, so a request whose second copy was lost with its peer can be hedged again.
xdustinface
force-pushed
the
fix/shared-stall-kicks
branch
from
November 11, 2026 01:21
ca66de7 to
e89874d
Compare
xdustinface
force-pushed
the
fix/cfilters-settle-on-arrival
branch
from
November 11, 2026 01:21
74f9283 to
bb5a0d0
Compare
1 of 4 tasks
xdustinface
force-pushed
the
fix/shared-stall-kicks
branch
from
November 12, 2026 22:33
e89874d to
5cc69f8
Compare
Contributor
|
This PR has merge conflicts with the base branch. Please rebase or merge the base branch into your branch to resolve them. |
xdustinface
force-pushed
the
fix/shared-stall-kicks
branch
from
November 12, 2026 22:46
5cc69f8 to
1095e19
Compare
xdustinface
force-pushed
the
fix/shared-stall-kicks
branch
2 times, most recently
from
November 13, 2026 02:57
3b3971d to
a37e9b9
Compare
2 of 4 tasks
xdustinface
force-pushed
the
fix/shared-stall-kicks
branch
6 times, most recently
from
November 13, 2026 11:52
4d4641f to
46beb07
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A
CFiltersorCfHeadersrequest stayed on the wire until its serial manager processed the reply, which could be a million messages later. A kicked peer then had its delivered batches sent again, and a busy manager looked like a stalled peer.The network pump now settles a filter request as soon as its reply arrives.
CFiltersrequests are keyed by stop hash, so the lastcfilterof a reply identifies its request without a scan. Every path that drops a delivered batch releases it, and a batch rejected by verification or lost to a storage error is requested again instead of waited on forever. The lowest batch not yet delivered gets a second copy from the peer with the shortest queue once its peer has answered nothing forHEDGE_AFTER, and dropping a peer clears the hedges it held.It is stacked on #1345, which sits on #1388 and #1387 below it.
Stacked on #1345, and will be retargeted to
devwhen that one merges.