Repository navigation
fix(consensus): periodically re-announce round state to peers - #1598
Claudius-Maginificent wants to merge 9 commits into
Conversation
A peer learns this node's height, round and step only from NewRoundStep, sent once on connect and once per step change, never acknowledged. One lost after the channel accepted it (router queue overflow, the peer's state-channel allowance) leaves the peer gossiping votes and proposals for the round it last heard of until this node changes step — which may never happen when that step is waiting on exactly the votes or proposal the peer is holding back, stalling the round until a reconnect. Add a per-peer gossip handler that re-sends the current round step on the PeerQueryMaj23SleepDuration cadence (one cost-1 state message per peer every 2s at the default). A repeat is a no-op for a peer whose picture is current, so the healthy path is unchanged beyond the message. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Decouples the handler from p2pMsgSender and matches the shape the pending round-skip gossip handlers use. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The per-peer gossip handlers registered at this cadence run back to back when it is zero; the round-step refresh sends unconditionally, so a zero value would flood every peer's state channel and burn this node's allowance there. Reject zero in ValidateBasic and document that the setting also bounds round-step re-announcement latency. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
|
|
✅ Final review complete — no blockers (commit 68dba53) · triage: critical |
…ate-refresh # Conflicts: # internal/consensus/gossip_peer_worker.go
The per-peer refresh re-sent only NewRoundStep, while resendRoundState treats a round picture as NewRoundStep plus NewValidBlock whenever the round has block parts. A lost NewValidBlock left the peer without this node's part-set header and holdings, and nothing repeated it. The handler now sends both from a shared roundStateMessages helper, also used by resendRoundState; a failed send ends the pass and the next one retries. The explicit NewRoundStep send in peerUp is removed: the worker starts after readySignal, where WaitSync is already false, and its first pass sends the announcement at once, so the two paths were redundant. The handler test moves next to its sibling handler tests, uses the same capturing-send pattern, and covers the send-failure branch; the P2P spec notes the periodic re-announcement. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Rejecting peer-query-maj23-sleep-duration <= 0 in ValidateBasic was a breaking config change that still left the loop unguarded: any tiny positive value spun the handlers the same way, peer-gossip-sleep-duration = 0 was still accepted, and ValidateBasic runs only on the CLI path, so an embedder calling node.New got no check at all. The per-peer gossip worker now runs no handler faster than minGossipSleepDuration (1ms, the same floor as incoming-connection-window) whatever either setting says, warning once per worker when it clamps. The <= 0 rejection and its test are reverted, so a 0s config starts again, and the config comments and the maj23AnswerTTLFor zero branch describe the floor rather than a validation rule. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A peer's periodic NewValidBlock landing before we send it the round's proposal makes PeerState.SetHasProposal return early, so the proposal's POL round is never recorded for the peer. The test fails on adbc822. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SetHasProposal returned before recording ProposalPOLRound when the peer's NewValidBlock had already set its block parts, so a re-proposal's POL round stayed at -1 and the peer was never sent the POL prevotes or the POL majority claim. The periodic round-state refresh makes that ordering routine. The POL fields are now recorded before the block-parts early return; the reviewer's repro test moves next to the other peer-state tests. ApplyNewValidBlockMessage keeps replacing the header and holdings with no ordering guard, now documented: a stale copy is corrected within one pass, whereas a grow-only guard would make a wrong bit permanent. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The floor was applied inside newPeerGossipWorker, so the maj23 answer TTL still read the raw config value and the warning repeated on every peerUp. NewReactor now derives a gossipCadence from the config once, warning once per raised setting, and both the per-peer workers and maj23AnswerTTLFor read it from there. The floor's comment and the config docs now say what it does: it stops a busy loop and nothing more; well below the defaults the handlers still offer peers more than their state-channel allowance admits. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
thepastaclaw
left a comment
There was a problem hiding this comment.
Final validation — Phase 1 + Phase 2
The cadence floor is correctly derived once in the reactor and used by peer workers and majority-answer TTLs. One shutdown retry path still reads the raw configuration, so zero or sub-millisecond settings bypass the stated 1 ms floor and can retry immediately; the worker-level concern is speculative because the constructor is package-private and all current callers pass the reactor-derived cadence.
🟡 1 suggestion(s)
Approved: nothing here blocks the merge. The non-blocking findings are optional: fix them forward, in this PR or a follow-up, without waiting for another review round.
1 finding(s) not shown inline (the lines are not part of this PR's diff)
🟡 Suggestion: Apply the cached floor to peer-up retry backoff
internal/consensus/reactor.go:956
This retry path still sleeps on the raw PeerGossipSleepDuration, while the reactor-level floor is used by every peer gossip handler and the majority-answer TTL. When the accepted configuration is zero or below 1 ms, a peer-up observed during shutdown retries immediately rather than respecting the floor, creating an avoidable burst of retry and peer-admission work. Use the already-derived cadence here so all consumers share the same bounded delay.
time.Sleep(r.cadence.gossip)
source: muse-spark-1.3-contributor (phase1-reviewer: general); gpt-6.1-sol (phase2-reviewer: general, tenderdash-consensus-security)
Review provenance
Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)
- Triage:
criticalbygpt-6.1-sol(effort low) — The change is large and intricate and directly modifies consensus round-state propagation, peer-facing consensus message handling, and recovery behavior in internal/consensus/gossip_handlers.go, gossip_peer_worker.go, peer_state.go, and reactor.go. - Phase 1 reviewers:
muse-spark-1.3-contributor— general (completed, effort xhigh); agentphase1-reviewer - Phase 1 model:
muse-spark-1.3-contributor— not quota-gated; passed overgemini-3.8-flash-high(antigravity quota lookup failed),glm-5.3-flash(not used above high effort; tier asks max) - Single stage: Phase 1 and Phase 2 reviewed this head side by side, with no blocker gate between them (triage tier)
- Fresh verifier:
gpt-6.1-sol— final-verifier; agentsol-verifier - Phase 2 reviewers:
gpt-6.1-sol— general (completed, effort xhigh); agentphase2-reviewer,gpt-6.1-sol— tenderdash-consensus-security (completed, effort xhigh); agentphase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions don't block the merge, but most are worth fixing in this PR; judge each on its merits.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.
In `internal/consensus/reactor.go`:
- [SUGGESTION] internal/consensus/reactor.go:956: Apply the cached floor to peer-up retry backoff
This retry path still sleeps on the raw `PeerGossipSleepDuration`, while the reactor-level floor is used by every peer gossip handler and the majority-answer TTL. When the accepted configuration is zero or below 1 ms, a peer-up observed during shutdown retries immediately rather than respecting the floor, creating an avoidable burst of retry and peer-admission work. Use the already-derived cadence here so all consumers share the same bounded delay.
|
Ready for review — files with no dedicated owner: lklimek or shumkov. |
|
Declining the peer-up retry backoff suggestion. Dashmate's mainnet preset inherits With a custom value below 🤖 Co-authored by Claudius the Magnificent AI Agent |
TL;DR: Validators now re-announce their current round to each peer every couple of seconds, so one lost announcement can no longer leave the network stuck.
User story
As a validator operator, I want the network to recover by itself when a few consensus messages are lost during congestion, so that blocks keep being produced without manual restarts.
Scenario
Base flow
Validators move through consensus rounds and tell their peers which round and step they are in, and which block they are working on. Peers use that to decide which votes, proposals and block pieces to send.
Actual behavior
A node announces its round only once, at the moment it changes. If that single announcement is dropped (for example by an overloaded outgoing queue or by the receiver's rate limit), the peer keeps believing the node is still in the previous round. It then sends the wrong round's data, or nothing at all. If validators are waiting in a step that has no timer, the round can stall until a node reconnects or restarts.
Expected behavior
A lost announcement is repaired within a few seconds. The peer learns the node's real round again, and votes, proposals and block pieces start flowing.
Detailed discussion
What was done
PeerState.PRS(height/round/step) mostly advances viaNewRoundStepMessage. That message is sent on step changes and when a peer connects, and is never repeated or acknowledged.internal/p2p/rqueue.godrops envelopes abovemaxSize;allowStateChannelMessage) drops messages over budget.prs.Round, and proposal/block-part gossip requires equal rounds. A view that lags behind therefore blocks current-round data in both directions.create_empty_blocks=false.setHasCommitcan also advancePRS. The refresh repairs a view that lags behind, not one that is ahead.roundStateGossipHandler,gossip_handlers.go). Registered innewPeerGossipWorkeratPeerQueryMaj23SleepDuration(2 s by default).roundStateMessages(rs):NewRoundStep, plusNewValidBlockwhile the round holds block parts. It sends them unconditionally, because the stale view is the peer's view of us, which we cannot observe.resendRoundStatenow uses the same helper, andp2pMsgSendercan sendNewValidBlockon the state channel.NewRoundSteprepeats that do not advance (CompareHRS <= 0).NewValidBlockreplaces the peer's record of our block-part bit array. Replacement is intentional: a stale copy is corrected on the next pass, whereas a grow-only guard would make a wrong "already has part X" bit permanent.peerUpno longer sends its ownNewRoundStep. The worker runs every handler once before its first sleep, so the first refresh pass announces the round immediately.PeerState.SetHasProposal):NewValidBlockhas already initialised the peer's block-part header.ProposalPOLRound, so POL prevotes and POL maj23 claims were never sent to that peer.NewValidBlockmakes it routine.flooredGossipCadence,gossip_peer_worker.go):NewReactorfloorspeer-gossip-sleep-durationandpeer-query-maj23-sleep-durationto 1 ms once and logs one warning per raised setting.0sis still accepted).docs/nodes/configuration.mdandspec/p2p/messages/consensus.mddescribe the periodic re-announcement and the floor.Testing
New tests:
TestReactorRefreshesRoundStepAfterLostAnnouncementPRSto match the sender's H/R/S with no further step change on the sender. Fails ondev(peer still sees 0/-1/RoundStepUnknown).TestRoundStateGossipHandlerNewRoundStep+NewValidBlock), and send failure.TestPeerStateSetHasProposalKeepsPOLRoundAfterNewValidBlockSetHasProposalfix.TestFlooredGossipCadence,TestReactorFloorsGossipCadenceThe touched and neighbouring consensus tests pass with
-race, as dogo test ./config, andmake lintreports 0 issues.A full independent review (security, design/reuse, correctness) and a follow-up review of the fixes found only low-severity items. They are addressed here, except two structural follow-ups:
Breaking changes
None.
Prior work
🤖 Generated with Claude Code
https://claude.ai/code/session_01WunYYgcoPDNh3uebhUCwig
🤖 Co-authored by Claudius the Magnificent AI Agent
PR Hygiene ·
68dba53config/config.go,config/toml.go,docs/nodes/configuration.mdand 11 more) — lklimek or shumkovWhen every merge requirement is met, the
PR Hygienecheck passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.