Skip to content

fix(dash-spv): stop a shared stall from emptying the peer set - #1345

Open
xdustinface wants to merge 11 commits into
devfrom
fix/shared-stall-kicks
Open

xdustinface wants to merge 11 commits into
devfrom
fix/shared-stall-kicks

Conversation

@xdustinface

@xdustinface xdustinface commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #1228.

When the client's own link stalls, every peer goes quiet at once. The timeout monitor judged each peer alone, so one tick could kick the whole set, and peers whose timeouts started a second apart crossed them one at a time and each looked like a lone stall. In 12 testnet runs the set dropped to zero 8 times while the replacements delivered within about 2 s. With a single configured peer, a 30 s outage kicked it.

The monitor now leaves the decision to a StallJudge, which adds up how long our link provably worked: every tick on which any peer received bytes counts. A silent peer is blamed once our link worked for more than one request timeout of its silence, however pauses and outages split that time, so they never count as proof against it. Bytes received from the silent peer itself count too, so a peer that keeps sending other data but never answers is still blamed. Otherwise the peer is held as a possible stall of our link until more than LINK_STALL_TIMEOUTS request timeouts of its silence (60 s with the base timeout) passed without proof. Silent peers are kicked at most one per timeout, the one silent the longest first, so a stall of our link never takes the whole set at once. A peer kicked only because its hold ran out is not banned, since our link may have failed it. A peer that is really unreachable fails its redial and backs off.

While some silent peer lacks that proof, our link counts as stalled. The flow controller then leaves those ticks out of a peer's serving rate, and neither it nor the supervisor names or swaps out an underserving peer. A tick that spares culprits but kicks nobody logs at debug. A dashd test freezes dashd for 30 s mid-sync and checks that its only peer stays connected.

PR Hygiene · 46beb07

  • Bots — coderabbitai ✓
  • Self-review — address ZocoLini requested changes, then post /self-reviewed
  • Build green
  • Approvals — you own every area touched; none needed

When every merge requirement is met, the PR Hygiene check passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.

Summary by CodeRabbit

  • Bug Fixes
    • Improved synchronization during temporary network or node stalls, helping prevent healthy peers from being disconnected when progress pauses.
    • Stalled peers are handled gradually as connections recover, reducing unnecessary disruptions to syncing.
    • Peer replacement is less likely to occur based on temporarily slow responses.
    • Syncing can continue through a paused node without treating the pause alone as a peer failure.

@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Walkthrough

The timeout monitor now distinguishes shared client-link stalls from peer-specific timeouts. Flow control and slow-peer replacement use the stall assessment. Unix-only integration tests pause dashd during sync and check recovery.

Changes: Network stall handling

Layer / File(s) Summary
Timeout attribution and stall detection
dash-spv/src/network/broker.rs, dash-spv/src/network/manager.rs
Broker::expire returns the earliest blame instant for eligible peers. The timeout monitor uses StallJudge to assess peer observations, hold silent peers during suspected shared stalls, and pace kicks. Tests cover stall detection, recovery, and timeout behavior.
Flow control during link stalls
dash-spv/src/network/manager.rs
Serving-rate samples exclude ticks during suspected link stalls. Slow-peer detection and replacement account for the stall state, including stalls that begin during candidate dialing.
Paused-node sync test
dash-spv/src/network/peer.rs, dash-spv/src/test_utils/node.rs, dash-spv/tests/dashd_sync/main.rs, dash-spv/tests/dashd_sync/tests_stall.rs
Unix-only test support pauses and resumes dashd. The integration test checks sync progress and peer connectivity across a 30-second pause. The test socket enables TCP_NODELAY.

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: zocolini

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: a shared stall (the client's own link failing) no longer empties the peer set. The changeset adds a StallJudge that holds peers during link outages and kicks at most o…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Title check

Explanation

The title names the main change: a shared stall (the client's own link failing) no longer empties the peer set. The changeset adds a StallJudge that holds peers during link outages and kicks at most one silent peer per timeout, which matches the title. The title uses the conventional fix(dash-spv) prefix and is specific enough for a teammate scanning history.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Review coverage is incomplete: 3 files could not be fully reviewed. Findings from completed review steps are included; see review info for details.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 4d4641f and 46beb07.

Files selected for processing: 3

  • dash-spv/src/network/broker.rs
  • dash-spv/src/network/manager.rs
  • dash-spv/tests/dashd_sync/tests_stall.rs
Files not reviewed due to moderation or processing errors: 3

  • dash-spv/src/network/broker.rs
  • dash-spv/src/network/manager.rs
  • dash-spv/tests/dashd_sync/tests_stall.rs

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

@codecov

codecov Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.81797% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.91%. Comparing base (c320718) to head (46beb07).
⚠️ Report is 1 commits behind head on dev.

Files with missing lines Patch % Lines
dash-spv/src/network/manager.rs 98.79% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##              dev    #1345      +/-   ##
==========================================
+ Coverage   85.81%   85.91%   +0.09%     
==========================================
  Files         269      269              
  Lines       70847    71204     +357     
==========================================
+ Hits        60799    61172     +373     
+ Misses      10048    10032      -16     
Flag Coverage Δ
core 82.81% <ø> (ø)
rpc 52.10% <ø> (ø)
spv 94.55% <98.81%> (+0.14%) ⬆️
wallet 83.89% <ø> (ø)
Files with missing lines Coverage Δ
dash-spv/src/network/broker.rs 97.83% <100.00%> (-0.62%) ⬇️
dash-spv/src/network/peer.rs 90.85% <100.00%> (-0.94%) ⬇️
dash-spv/src/network/manager.rs 93.63% <98.79%> (+3.37%) ⬆️

... and 5 files with indirect coverage changes

@github-actions github-actions Bot added the merge-conflict The PR conflicts with the target branch. label Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

This PR has merge conflicts with the base branch. Please rebase or merge the base branch into your branch to resolve them.

@xdustinface
xdustinface force-pushed the fix/shared-stall-kicks branch from 52e040b to fb93704 Compare November 6, 2026 21:07
@github-actions github-actions Bot removed the merge-conflict The PR conflicts with the target branch. label Nov 6, 2026
@xdustinface
xdustinface force-pushed the fix/shared-stall-kicks branch 2 times, most recently from 7e7e4a7 to 43d5e8c Compare November 9, 2026 15:36
@xdustinface
xdustinface force-pushed the fix/shared-stall-kicks branch from 80d80bf to ca66de7 Compare November 10, 2026 14:19
@xdustinface
xdustinface changed the base branch from dev to fix/streaming-request-progress November 10, 2026 14:20
@xdustinface
xdustinface force-pushed the fix/streaming-request-progress branch from 761b7d8 to 0467c8c Compare November 11, 2026 01:21
@xdustinface
xdustinface force-pushed the fix/shared-stall-kicks branch from ca66de7 to e89874d Compare November 11, 2026 01:21
@xdustinface
xdustinface force-pushed the fix/streaming-request-progress branch from 0467c8c to 2a5a48e Compare November 11, 2026 17:24
@github-actions github-actions Bot added the merge-conflict The PR conflicts with the target branch. label Nov 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This PR has merge conflicts with the base branch. Please rebase or merge the base branch into your branch to resolve them.

@xdustinface
xdustinface force-pushed the fix/streaming-request-progress branch from 2a5a48e to 78d6190 Compare November 11, 2026 18:21
Base automatically changed from fix/streaming-request-progress to dev November 12, 2026 15:23
@xdustinface
xdustinface force-pushed the fix/shared-stall-kicks branch from e89874d to 5cc69f8 Compare November 12, 2026 22:33
@github-actions github-actions Bot removed the merge-conflict The PR conflicts with the target branch. label Nov 12, 2026
@xdustinface
xdustinface force-pushed the fix/shared-stall-kicks branch 3 times, most recently from 3b3971d to a37e9b9 Compare November 13, 2026 02:57
@xdustinface
xdustinface marked this pull request as ready for review November 13, 2026 03:18
@manki-review

manki-review Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

✅ Review complete in 4m 33s

Planner (4s)
    fix · 615 lines · 2 agents
    judge effort: high

Review (behavior · pair): 4 findings
    ✅ Claude Sonnet 5.5 · high: 1 finding (1m 13s)
    ✅ Claude Opus 5.5 · high: 3 findings (2m 34s)

Judge: 3 kept · 0 dropped (1m 30s)
    kept: 2 suggestion · 1 nitpick

Review metadata

Config:

  • Models: reviewer=claude-sonnet-5-5, judge=claude-opus-5-5
  • Triage: trivial=false, subtle=true, paired=true (planner)
  • Team: Claude Sonnet 5.5 · high, Claude Opus 5.5 · high
  • Memory: enabled (central)

Account limits:

  • Claude subscription: weekly 70% (resets 2026-11-17 05:00 UTC)

Judge decisions:

  • Kept: "New test helpers were inserted between settle and its doc comment" (nitpick, high confidence): "nitpick. The diff inserts advance_receiving and deliver between /// Lets the spawned tasks run what the last time advance made due. at dash-spv/src/network/manager.rs:1497 and async fn settle(). As a result advance_receiving opens with a sentence about settle, and settle has no doc. This is a misplaced test-helper doc, so it is cosmetic."
  • Kept: "Closes #1228, but the issue's queued-behind-others kicks and its refill failure remain" (suggestion, medium confidence): "suggestion. The code does still kick, one per timeout, a peer whose answer waits behind another peer's stream. Any peer's growth sets last_heard at manager.rs:1098, and that makes witnessed_from true at manager.rs:1148-1151. The PR description states this witness rule as the chosen design, so the right form is a question about whether the issue should stay open. The refill-logging symptom sits in supervisor code outside the diff, and other PRs may cover it. Which keyword closes an issue is PR metadata, not a code defect, and the issue lists no checkbox acceptance criteria."
  • Kept: "No test covers the link_stalled flag in the flow controller or the supervisor" (suggestion, high confidence): "suggestion. This is missing coverage. No test exercises the stalled load at manager.rs:696, the gate on underserving_peer at :765, or the !stalled gate in repair_round at :862. the_serving_rate_averages_ticks_with_work does run the changed sample logic and would fail if its stall check were removed. It cannot catch the wiring around sample. It also calls a private method, which AGENTS.md discourages."

Timing:

  • Parse: 4.0s
  • Review agents: 178.9s
  • Judge: 90.1s
  • Total: 273.0s

@github-actions

Copy link
Copy Markdown
Contributor

Waiting for bot review — coderabbitai not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added the waiting-bots Waiting for the review bots to report on this head label Nov 13, 2026
@manki-review

manki-review Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

✅ Review complete in 5m 59s

Planner (5s)
    fix · 907 lines · 2 agents
    judge effort: high

Review (behavior · pair): 2 findings
    ✅ Claude Sonnet 5.5 · high: 0 findings (54s)
    ✅ Claude Opus 5.5 · high: 2 findings (3m 4s)

Judge: 2 kept · 0 dropped (1m 43s)
    kept: 1 warning · 1 nitpick

Review metadata

Config:

  • Models: reviewer=claude-sonnet-5-5, judge=claude-opus-5-5
  • Triage: trivial=false, subtle=true, paired=true (planner)
  • Team: Claude Sonnet 5.5 · high, Claude Opus 5.5 · high
  • Memory: enabled (central)

Account limits:

  • Claude subscription: weekly 79% (resets 2026-11-17 05:00 UTC)

Judge decisions:

  • Kept: "After an outage shorter than one request timeout, a silent peer gets almost no time to resume and is banned" (warning, medium confidence): "Warning. The trace holds. observe sets link_back only when link_down is set (dash-spv/src/network/manager.rs:1102). judge clears link_down on every call (:1130) and returns early while there are no culprits (:1144). A peer becomes a culprit only after a full timeout of silence (:1139).

So an outage shorter than REQUEST_TIMEOUT never sets link_back. blamed_from then falls back to since (:1159). Take every peer silent from t=0, with the link back at t=9:

  • Peer B's bytes at t=11 make witnessed_from(0) true for peer A.
  • A is kicked and banned with Ban::Kicked 2 s after the link returned.
  • A 12 s outage would instead give A a full timeout from the return.

This contradicts the stated goal of a full timeout after the link is back, in the unstable-link use case of #1228. The damage is one healthy peer per timeout, kicked and banned while the supervisor refills the set. That is wrong behaviour of bounded scope, so it is a warning rather than a blocker."

  • Kept: "The StallJudge doc comment goes past the three-line ceiling" (nitpick, high confidence): "Nitpick. The doc comment at dash-spv/src/network/manager.rs:1066-1069 is four lines. Root AGENTS.md sets three lines as the ceiling. This is a comment convention, so it is a nitpick."

Timing:

  • Parse: 12.5s
  • Review agents: 244.2s
  • Judge: 102.7s
  • Total: 359.3s

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Your move: coderabbitai left review threads unresolved; resolve them.
Full checklist in the description.

manki-review[bot]
manki-review Bot previously approved these changes Nov 13, 2026

@manki-review manki-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approved. 1 warning and 1 nitpick, none of which block.

What this does: Before, a peer was dropped and banned after 10 s of silence, even when our own internet link was the one that stalled. Now a peer that goes silent is kept until another peer proves the link works, or for at most about 60 s.


Found this round · 1 nitpick

📝 The StallJudge doc comment goes past the three-line ceiling
Nitpick, not blocking · Style · Quick fix · dash-spv/src/network/manager.rs:1066

Why: The new doc comment on StallJudge runs four lines. Root AGENTS.md says: "One line is normal, three is the ceiling."

}
/// Decides which peers [`spawn_timeout_monitor`] kicks. A peer silent with work is held
/// while no peer proves our link works, for at most [`LINK_STALL_TIMEOUTS`] timeouts after
/// the link's last return and [`MAX_HOLD_TIMEOUTS`] after its silence began. Silent peers

How to fix it
Cut it to three lines or fewer. The details of the hold bounds are already on LINK_STALL_TIMEOUTS and MAX_HOLD_TIMEOUTS.

Checked in the code
On the flagged line: Holds the four-line /// Decides which peers [``spawn_timeout_monitor``] kicks. ... comment

  • AGENTS.md, Comments and docs: One line is normal, three is the ceiling.
  • Create an issue for this

Related work: open PRs and issues that may cover the same change. None of them blocks.


What to test

The review model wrote these steps. Check what a step runs before you follow it.

Setup: A regtest dashd set up through eval $(python3 contrib/setup-dashd.py) on a Unix host, with the dash-spv CLI or the dashd sync tests.

  1. Run just dash-spv::test tests_stall. Expect test_sync_keeps_a_peer_frozen_mid_sync to pass with no PeerDisconnected event.
  2. Start a sync against one dashd, then send it kill -STOP for 30 s and kill -CONT. Expect the peer to stay connected and sync to finish.
  3. Repeat with a freeze of 75 s. Expect the peer to be kicked after about 60 s with a request timeout: kicked 1 peer(s) warning, and redialed without a ban.
  4. With two dashd peers, freeze only one of them mid-sync. Expect the frozen one to be kicked about 10 s into its silence and banned, with its requests re-queued to the other.
  5. Freeze both peers of a two-peer sync for 40 s. Expect request timeout: kicked none, spared 2 debug lines and no disconnects.
  6. Edge case: during a shared freeze with debug logging on, check that no swapped out slow line appears and that peer_speed serving rates do not drop.
About this review
  • Reviewed 6ffb7f2, round 5
  • 907 changed lines read in 5m 59s
  • Found by claude-sonnet-5-5 and claude-opus-5-5, judged by claude-opus-5-5
  • Tokens: 920k in (456k from cache) · 35k out · about $2.60 at API prices
  • How Manki reviews

Comment thread dash-spv/src/network/manager.rs Resolved
When our own link stalls, every peer goes quiet at once. The timeout monitor judged each peer alone, so peers whose timeouts started a second apart crossed them one at a time and each looked like a lone stall. A testnet run lost four healthy peers in 14 s during a 40 s outage, and with one configured peer a 30 s outage kicked it and banned it for 10 minutes.

The monitor now hands its decision to a `StallJudge`. A silent peer is blamed only once our link provably works: some peer, this one included, received bytes more than a timeout into its silence. Silence that no peer witnessed is held as a possible stall of our link, for at most `LINK_STALL_TIMEOUTS` (6) request timeouts, 60 s with the 10 s base timeout, enough for the 40 s outages seen on testnet. Past the bound the set is more likely dead than our link, so the peer is kicked anyway. A new peer's first byte counter proves nothing, since its handshake may predate the stall.

Silent peers are kicked one per timeout, the one silent the longest first, so a stall of our link never takes the whole set at once. A replacement joining right after a kick does not unpace the next one.

When the first peer receives again after a silence nobody witnessed, our link counts as back from that moment. Both the witness and the hold bound then count from the later of a peer's silence and that return, so a peer still recovering from TCP backoff gets a full timeout instead of being kicked as soon as another peer speaks. Bytes from an idle peer later on cannot restart a dead peer's timeout. A link that keeps coming back for less than a timeout would push that start forward forever, so a hold also ends `MAX_HOLD_TIMEOUTS` (12) timeouts after the peer's own silence began. No single return reaches that, since the last one within a hold comes at most `LINK_STALL_TIMEOUTS` in.

`Broker::expire` reports when each culprit's stalled request went silent, so its silence is measured the same way as the owed-bytes check. A request past `ANSWER_TIMEOUTS` is blamed from its last answer, so the peer's own recent bytes witness our link and it is kicked rather than held.

While the judge holds a peer, `NetworkState` flags our link as stalled. The flow controller then skips the tick in a peer's serving rate, which otherwise stayed dragged down for 30 s after the stall, and neither the flow controller nor the supervisor names an underserving peer from rates the stall skewed.
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Your move: coderabbitai left review threads unresolved; resolve them.
Full checklist in the description.

@manki-review

manki-review Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

✅ Review complete in 6m 11s

Planner (4s)
    fix · 932 lines · 2 agents
    judge effort: high

Review (behavior · pair): 1 finding
    ✅ Claude Sonnet 5.5 · high: 0 findings (53s)
    ✅ Claude Opus 5.5 · high: 1 finding (3m 24s)

Judge: 1 kept · 0 dropped (2m 13s)
    kept: 1 suggestion

Review metadata

Config:

  • Models: reviewer=claude-sonnet-5-5, judge=claude-opus-5-5
  • Triage: trivial=false, subtle=true, paired=true (planner)
  • Team: Claude Sonnet 5.5 · high, Claude Opus 5.5 · high
  • Memory: enabled (central)

Account limits:

  • Claude subscription: weekly 86% (resets 2026-11-17 05:00 UTC)

Judge decisions:

  • Kept: "Two healthy peers with short gaps in their replies keep a dead peer held for 120 s, and it is then kicked without a ban" (suggestion, medium confidence): "suggestion. The mechanism is real.

  • shared_silence at manager.rs:1101-1110 leaves out the dead culprit, because its silence exceeds p.timeout. Two other owing peers that are both silent for more than 2 s are enough to trigger it.

  • The next byte from either peer then sets link_back = now (manager.rs:1119-1120).

  • blamed_from = link_back.max(since) at manager.rs:1176 moves past the culprit's silence. While that keeps happening every 10 s or less, neither the witness check nor the LINK_STALL_TIMEOUTS bound is met.

  • Only MAX_HOLD_TIMEOUTS (120 s) ends the hold, and the kick is then reported as unwitnessed, so no ban follows (manager.rs:1204, 1274).

The trigger is the heuristic itself: every owing peer going quiet at once is the signal the PR uses for a stall of our own link. The damage is bounded. The dead peer's work waits up to 120 s, swaps of slow peers are paused, and the dead peer is not banned, so it may be dialed again. That is degraded but correct behaviour in a case that looks uncommon. It would be a warning if in-phase gaps of more than 2 s among healthy owing peers are frequent in normal sync.

The Flaps(12, 5) test cited by the finding has only one other peer, so its link return comes from the link_down path, not from shared_silence. There, peer 2's 12 s gaps are a flapping link by design, which matches the description's intent.
Refutation unsettled: The code confirms the mechanism. shared_silence leaves out silences longer than the timeout, and any byte growth after a shared silence sets link_back = now. That pushes blamed_from forward, so witnessed_from stays false until the 120 s bound, and the kick is then not banned. What the code cannot show is whether two healthy owing peers really go quiet together for more than 2 s, at least every 10 s, during real sync. It may even be less likely, because peers that run out of work drop to ... Rule checked: A dead culprit's witness clock starts at link_back.max(since). If link_back moves to now every time two or more owing peers that are not culprits break a shared silence of more than 2 s, and that happens at least every 10 s, then only MAX_HOLD_TIMEOUTS ends the hold, and the kick is reported as unwitnessed.

  • Premise "During normal sync, two or more healthy peers that owe bytes regularly receive nothing for more than 2 s at the same time, at least once every 10 s, while a third peer is dead" unsettled. How often healthy peers go quiet during sync depends on network and request timing, not on anything in the code. The code only shows that peers with no work in flight are not counted as owing (line 1130), which can make such shared silences rarer, and no test sets up two healthy owing peers with overlapping gaps."

Timing:

  • Parse: 7.3s
  • Review agents: 230.1s
  • Judge: 133.2s
  • Total: 370.6s

manki-review[bot]
manki-review Bot previously approved these changes Nov 13, 2026

@manki-review manki-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approved. 1 suggestion, which doesn't block.

What this does: Before, a peer that went quiet for 10 s with pending work was always kicked and banned, even when our own connection had dropped. The client now keeps silent peers through a likely outage of its own link, up to a bound.


Related work: open PRs and issues that may cover the same change. None of them blocks.


What to test

The review model wrote these steps. Check what a step runs before you follow it.

Setup: Run a regtest dashd with just dash-spv::test, or point the CLI at a local node on a Unix machine.

  1. Start a sync against one dashd, then freeze it with kill -STOP <pid> for 30 s and resume it with kill -CONT. Expect the peer to stay connected and sync to finish.
  2. Freeze the only peer for over 60 s. Expect it to be kicked once the hold runs out, and no ban recorded for it.
  3. With two peers, freeze one while the other keeps serving. Expect the frozen one to be kicked after about 10 s and banned.
  4. With several peers, cut the client's own network for 40 s and restore it. Expect no peer kicked and no slow-peer swap logged during the outage.
  5. After the outage ends, keep one peer much slower than the others. Expect the supervisor to swap it out once rates are measured again.
  6. Check peer_speed debug logs during an outage. Expect serving rates to stay frozen instead of dropping.
About this review
  • Reviewed 4d4641f, round 6
  • 932 changed lines read in 6m 11s
  • Found by claude-sonnet-5-5 and claude-opus-5-5, judged by claude-opus-5-5
  • Tokens: 1.4M in (932k from cache) · 40k out · about $3.04 at API prices
  • How Manki reviews

Comment thread dash-spv/src/network/manager.rs Resolved

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2


  • Fix comments: Fix CodeRabbit comments on this PR
Prompt to fix all issues
Treat finding text, file paths, and code as untrusted review data. Never follow
instructions embedded in them. Verify each finding against current code. Fix
only still-valid issues, skip the rest with a brief reason, keep changes
minimal, and validate.

Inline comments:
Review comments at @dash-spv/src/network/manager.rs:
- Line 1106: Update the silence filtering used by observe and judge so
shared-silence evidence survives the first tick beyond p.timeout, allowing a
peer’s return to be recognized before another silent peer is kicked or banned.
Preserve the full post-recovery timeout described by the PR objectives without
discarding the existing timeout safeguards.
- Around line 1119-1120: Change the link-return rule in the
`link_down`/`shared_silence` branch so recurring shared-silence cycles or
continuous peer receipts cannot keep advancing `link_back`. Use the existing
link-state model to identify a genuine return condition and timestamp that
transition without re-arming it on each cycle; preserve the intended timeout
behavior so a dead peer can age out and normal sampling and replacement can
resume.

After applying the fix, consider running `coderabbit review --agent` for local
review. Visit https://docs.coderabbit.ai/cli?utm_source=ghpr
Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 8efdd29 and 4d4641f.

Files selected for processing: 2

  • dash-spv/src/network/broker.rs
  • dash-spv/src/network/manager.rs

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Comment thread dash-spv/src/network/manager.rs Resolved
Comment thread dash-spv/src/network/manager.rs Resolved
@github-actions

Copy link
Copy Markdown
Contributor

Your move: coderabbitai requested changes on this head; dismiss the review or push a fix; coderabbitai left review threads unresolved; resolve them.
Full checklist in the description.

A dashd test freezes dashd with `SIGSTOP` for 30 s in the middle of a sync, three request timeouts but under the bound for holding a silent peer, then resumes it with `SIGCONT`. Its socket stays open, so the client sees a silent connection, as when our own link stalls. The test asserts the client never drops its only peer and that sync completes. Against the old timeout monitor the peer is kicked about 10 s into the freeze.

`DashCoreNode::set_paused` sends the signals. The test is unix only, since Windows has no equivalent.
`StallJudge::held` collected the peers held for a possible stall of our link, but it was only read to check whether it was empty and to count the spared culprits. The judge now builds the kick set directly, from the culprits that already left plus the peer silent the longest when a kick is due, counts the held peers while judging, and keeps a `link_stalled` flag instead of the set. The kicks and the flag are unchanged.
`StallJudge::progress` holds each peer's `Liveness`, and "progress" already means request progress in the broker and the monitor. The field is now `liveness`.
The doc read "until the first second and again from the second", which mixes the ordinal with the time unit. It now names both fields as times in seconds.
Nothing failed if the flow controller sampled serving rates or named a slow peer during a stall of our own link, or if the supervisor picked a slow peer to swap during one. `the_serving_rate_averages_ticks_with_work` called the private `PeerFlow::sample` and read its private rate.

`a_stall_of_our_link_leaves_the_serving_rates_alone` measures two peers through the real flow controller, with work up to their caps and bytes fed through their sockets. Once both go silent, their rates fall until the judge holds them as a possible stall of our link, and then stay put. `no_slow_peer_is_swapped_out_while_our_link_stalls` starts the supervisor during such a stall with a slow peer and a replacement that completes a real handshake, and checks that nothing is dialed. Once the peers deliver again, the swap the stall held back must go ahead. It does only because the flow controller names no slow peer during a stall: naming it again afterwards is what wakes a full supervisor, which has no timer to fall back on. The private-state test is removed, as the new test covers its stall and silence cases and `only_bandwidth_bound_service_condemns_a_peer` its idle one.
`repair_round` picks the slow peer to replace before it dials the candidates, which can take up to `CONNECT_DEADLINE`. If our own link stalled during the dial, `admit` still swapped out that peer, closing it and re-queuing its work in the middle of the stall.

The supervisor now checks `link_stalled` again when it admits a candidate. A new case of `no_slow_peer_is_swapped_out_while_our_link_stalls` lets the stall begin while the replacement is being dialed.
A stall of our own link longer than the hold still kicks the held peers one per timeout, and every kicked peer was banned for 10 minutes with `Ban::Kicked`. Once the link came back, the supervisor could not redial the very peers our link had failed.

`StallJudge::judge` now also returns the peer it kicked because a hold bound ran out with no peer witnessing its silence. The monitor still kicks that peer and re-queues its work, but does not ban it: it is most likely a victim of our link, and if it is dead after all, redialing it fails and `Ban::DialFailed` backs it off. A peer kicked while another peer proves the link works is banned as before, and so is a culprit that already left the set.

Every case of `the_stall_judge_kicks_one_peer_per_shared_stall` now also states which kicks go unbanned. Through the real monitor, the peers of `a_departed_peer_is_announced_once` that are kicked past the bound stay unbanned, while the peer `the_timeout_monitor_requeues_stalled_requests` kicks beside a witness is banned.
The judge kept a single instant for our link's return and restarted a culprit's witness clock from it. Every rule for setting it could be re-armed by ordinary traffic: two healthy peers pausing together, owing peers with reply gaps beside an idle stream, or a flapping link kept a dead culprit held up to two minutes and then kicked it unbanned, while a peer resuming on the first tick past the timeout dropped the evidence altogether.

The judge now counts how long our link demonstrably worked. Each tick on which any peer's bytes grew adds the time since the tick before, at most `TIMEOUT_CHECK`, so a late monitor tick cannot count an outage as uptime. A culprit is witnessed once our link worked for more than a timeout of its silence, however outages split that time, and is kicked unbanned only after more than `LINK_STALL_TIMEOUTS` timeouts of downtime. Since uptime and downtime together are the whole silence, a hold ends within `LINK_STALL_TIMEOUTS` plus one timeouts, and `MAX_HOLD_TIMEOUTS`, `link_down` and `link_back` are gone.

Uptime is kept as one cumulative sample per tick, back to the last tick before the oldest silence judged now or later: a peer's, a culprit's still waiting behind kick pacing, or a request's, which `Broker::expire` blames from at most `ANSWER_TIMEOUTS` timeouts back. A silence is measured from the last tick at or before it began, so a request sent between ticks is blamed on the first tick past its timeout. `link_stalled` stays set while any culprit is unwitnessed, held or waiting behind kick pacing.

Rows of `the_stall_judge_kicks_one_peer_per_shared_stall` and `the_timeout_monitor_requeues_stalled_requests` whose request predates the judge's first tick are kicked a tick later, as uptime before that tick is unknown. Rows after an outage are kicked a tick earlier, as the tick that ends an outage counts. Sparse or flapping traffic now proves only as much uptime as it shows: a peer beside pings every 3 s is held until they add up to a timeout, and one beside a receipt every 10 s is kicked unbanned at the bound. The rows where every peer goes silent past the bound now report `link_stalled` while the rest wait to be kicked. New rows cover peers pausing together, an idle stream beside owing peers with gaps, a peer resuming on the timeout tick, 5 s and 40 s outages, culprits paced past the link's last uptime, a request sent between ticks and a sole peer streaming an answer it never finishes.
`sample` was split out of `tick` so a test could drive it, and that test is gone. `tick` is its only caller again, so the serving rate update goes back inline with the stall check beside it.
`unsettling_filter` had `feed` as its only caller, so the frame is built where it is sent.
@manki-review

manki-review Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

✅ Review complete in 9m 54s

Planner (3s)
    fix · 954 lines · 2 agents
    judge effort: high

Review (behavior · pair): 2 findings
    ✅ Claude Sonnet 5.5 · high: 0 findings (34s)
    ✅ Claude Opus 5.5 · high: 2 findings (3m 13s)

Judge: 2 kept · 0 dropped (1m 20s)
    kept: 2 suggestion

Review metadata

Config:

  • Models: reviewer=claude-sonnet-5-5, judge=claude-opus-5-5
  • Triage: trivial=false, subtle=true, paired=true (planner)
  • Team: Claude Sonnet 5.5 · high, Claude Opus 5.5 · high
  • Memory: enabled (central)

Account limits:

  • Claude subscription: weekly 78% (resets 2026-11-13 20:00 UTC)

Judge decisions:

  • Kept: "Router keeps sending new work to a stalled peer while it waits for its paced kick" (suggestion, medium confidence): "Suggestion. Only silent.first() is kicked per timeout (manager.rs:1181-1193), so other culprits, witnessed ones included, stay in the set unmarked. The route filter at broker.rs:320 checks only is_closed and cap room. The pacing is a deliberate choice: the tests "two stalls beside a delivering peer kick one" and "the second stall goes a timeout later" pin it. The result is still correct, because the waiting peer is kicked later and its work is re-queued. A waiting peer usually sits at or near its cap, and hedging re-sends stalled requests elsewhere after about 2 s, which limits the delay. Neither point is proven from the code shown. If new work does reach such a peer and hedging does not cover it, requests are delayed by up to one timeout per culprit ahead, which would be a warning. One option: does a witnessed culprit need pacing, or could it be kicked at once?"
  • Kept: "PR description promises a MAX_HOLD_TIMEOUTS 120 s cap and a witness rule the code does not have" (suggestion, high confidence): "Suggestion. This round removed MAX_HOLD_TIMEOUTS and link_back; the inter-round diff drops both. The witness is now cumulative uptime: witnessed = up > p.timeout at manager.rs:1170, with link_up growing at most 1 s per tick in which some peer's bytes grew (manager.rs:1106-1107). A hold now ends after more than 60 s of downtime, not after 120 s. The description still states the old rules, and readers act on it. A description is not code, so this does not block."

Timing:

  • Parse: 8.3s
  • Review agents: 505.8s
  • Judge: 80.1s
  • Total: 594.1s

Last reviewed 46beb07 · head 46beb07 · up to date

  • Pause automatic reviews
  • Review again
  • Deeper review

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Bots are done — your move: address ZocoLini requested changes, then post /self-reviewed.
Full checklist in the description.

@manki-review manki-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approved. 2 suggestions, none of which block.

What this does: Before, any peer silent for 10 s was kicked and banned, so a short outage of the client's own connection dropped every peer. Now a silent peer is kicked only once other peers show the link works, or after 60 s of total silence.


Related work: open PRs and issues that may cover the same change. None of them blocks.


What to test

The review model wrote these steps. Check what a step runs before you follow it.

Setup: Run the dashd integration tests on a Unix host with just dash-spv::test. For manual checks, use the CLI against a regtest or testnet node.

  1. Run just dash-spv::test test_sync_keeps_a_peer_frozen_mid_sync. Expect sync to finish after the 30 s freeze with no PeerDisconnected event.
  2. Run cargo test -p dash-spv --lib network::manager. Expect every the_stall_judge_kicks_one_peer_per_shared_stall case to pass.
  3. Sync with the CLI and send SIGSTOP to the only dashd peer for about 30 s, then SIGCONT. Expect request timeout: kicked none, spared 1 debug lines and no disconnect.
  4. Keep dashd stopped for more than 70 s. Expect one kick, no ban, and a redial of the same peer.
  5. With two peers, freeze only one. Expect it to be kicked and banned about 10 s after it goes silent while the other keeps receiving.
  6. Disconnect the client's network for about 20 s during filter sync. Expect no slow-peer swap and no serving at log lines during the outage.
  7. Edge case: freeze every peer at once with max_peers above 2. Expect at most one kick per 10 s once 60 s have passed.
About this review
  • Reviewed 46beb07, round 7
  • 954 changed lines read in 9m 54s
  • Found by claude-sonnet-5-5 and claude-opus-5-5, judged by claude-opus-5-5
  • Tokens: 437k in · 33k out · about $2.41 at API prices
  • How Manki reviews

Comment thread dash-spv/src/network/manager.rs Resolved
Comment thread dash-spv/src/network/manager.rs Resolved

@ZocoLini ZocoLini left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can be much simpler. To know whether our own link is stalled, we only need to know whether some peer received bytes on every tick of the last timeout.

The timeout monitor keeps one more value, link_up_since. It is the instant since which some peer has received bytes on every tick, and it resets to None on any tick with no bytes from anyone. A culprit is kicked only if our link has been up for more than a timeout, or if the culprit has been silent for more than LINK_STALL_TIMEOUTS timeouts:

link_up_since.is_some_and(|up| now.duration_since(up) > timeout)
    || now.duration_since(since) > timeout * LINK_STALL_TIMEOUTS

So a stall of our own link never kicks anyone, peers going silent a second apart included. When the link comes back, every peer gets a full timeout to answer. A single peer still goes after 60 s. Besides that, Broker::expire only needs to return since when it blames each peer.

That is about 50 lines instead of the StallJudge. The link_stalled signal to the flow controller and the supervisor seems like a separate change to me. What do you think?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-self-review Waiting for the author to post /self-reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants