Skip to content

fix(consensus): periodically re-announce round state to peers - #1598

Open
Claudius-Maginificent wants to merge 9 commits into
devfrom
fix/consensus-round-state-refresh
Open

Claudius-Maginificent wants to merge 9 commits into
devfrom
fix/consensus-round-state-refresh

Conversation

@Claudius-Maginificent

@Claudius-Maginificent Claudius-Maginificent commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

TL;DR: Validators now re-announce their current round to each peer every couple of seconds, so one lost announcement can no longer leave the network stuck.

User story

As a validator operator, I want the network to recover by itself when a few consensus messages are lost during congestion, so that blocks keep being produced without manual restarts.

Scenario

Base flow

Validators move through consensus rounds and tell their peers which round and step they are in, and which block they are working on. Peers use that to decide which votes, proposals and block pieces to send.

Actual behavior

A node announces its round only once, at the moment it changes. If that single announcement is dropped (for example by an overloaded outgoing queue or by the receiver's rate limit), the peer keeps believing the node is still in the previous round. It then sends the wrong round's data, or nothing at all. If validators are waiting in a step that has no timer, the round can stall until a node reconnects or restarts.

Expected behavior

A lost announcement is repaired within a few seconds. The peer learns the node's real round again, and votes, proposals and block pieces start flowing.

Detailed discussion

What was done

  • Problem. PeerState.PRS (height/round/step) mostly advances via NewRoundStepMessage. That message is sent on step changes and when a peer connects, and is never repeated or acknowledged.
    • Delivery can still fail after the channel has accepted the message:
      • internal/p2p/rqueue.go drops envelopes above maxSize;
      • the receiver's per-peer state allowance (allowStateChannelMessage) drops messages over budget.
    • Vote gossip selects votes for prs.Round, and proposal/block-part gossip requires equal rounds. A view that lags behind therefore blocks current-round data in both directions.
    • Nodes waiting in a step without a timer stall: Prevote/Precommit without +2/3-any, or a no-tx wait with create_empty_blocks=false.
    • setHasCommit can also advance PRS. The refresh repairs a view that lags behind, not one that is ahead.
  • Periodic round-state refresh (roundStateGossipHandler, gossip_handlers.go). Registered in newPeerGossipWorker at PeerQueryMaj23SleepDuration (2 s by default).
    • It sends the messages returned by roundStateMessages(rs): NewRoundStep, plus NewValidBlock while the round holds block parts. It sends them unconditionally, because the stale view is the peer's view of us, which we cannot observe.
    • feat(consensus)!: hand a silent proposer's idle round to the next proposer #1572's resendRoundState now uses the same helper, and p2pMsgSender can send NewValidBlock on the state channel.
    • The receiver ignores NewRoundStep repeats that do not advance (CompareHRS <= 0).
    • A repeated NewValidBlock replaces the peer's record of our block-part bit array. Replacement is intentional: a stale copy is corrected on the next pass, whereas a grow-only guard would make a wrong "already has part X" bit permanent.
    • Cost: 1–2 small messages per peer per interval, against a 5000/s per-peer state allowance.
    • The handler does not run during block sync or state sync, and stops with the peer.
  • Peer connect. peerUp no longer sends its own NewRoundStep. The worker runs every handler once before its first sleep, so the first refresh pass announces the round immediately.
  • Fix needed by the refresh (PeerState.SetHasProposal):
    • The proposal's POL round is now recorded even when a NewValidBlock has already initialised the peer's block-part header.
    • Before, the early return skipped ProposalPOLRound, so POL prevotes and POL maj23 claims were never sent to that peer.
    • The bug already existed, but the periodic NewValidBlock makes it routine.
  • Gossip interval floor (flooredGossipCadence, gossip_peer_worker.go):
    • NewReactor floors peer-gossip-sleep-duration and peer-query-maj23-sleep-duration to 1 ms once and logs one warning per raised setting.
    • The result drives every per-peer gossip handler and the maj23 answer TTL.
    • Zero or tiny values previously made the handlers spin without sleeping.
    • The floor only prevents a busy loop. Well below the defaults, peers still drop this node's state updates that exceed their per-peer allowance.
    • Config validation is unchanged (0s is still accepted).
  • Docs. The config comments, the TOML template, docs/nodes/configuration.md and spec/p2p/messages/consensus.md describe the periodic re-announcement and the floor.

Testing

  • New tests:

    Test Kind
    TestReactorRefreshesRoundStepAfterLostAnnouncement Discards everything sent during a loss window, then requires the peer's PRS to match the sender's H/R/S with no further step change on the sender. Fails on dev (peer still sees 0/-1/RoundStepUnknown).
    TestRoundStateGossipHandler Without block parts, with block parts (NewRoundStep + NewValidBlock), and send failure.
    TestPeerStateSetHasProposalKeepsPOLRoundAfterNewValidBlock Fails without the SetHasProposal fix.
    TestFlooredGossipCadence, TestReactorFloorsGossipCadence Floor applied once at the reactor, for both settings and all handlers.
  • The touched and neighbouring consensus tests pass with -race, as do go test ./config, and make lint reports 0 issues.

  • A full independent review (security, design/reuse, correctness) and a follow-up review of the fixes found only low-severity items. They are addressed here, except two structural follow-ups:

Breaking changes

None.

Prior work

🤖 Generated with Claude Code

https://claude.ai/code/session_01WunYYgcoPDNh3uebhUCwig

🤖 Co-authored by Claudius the Magnificent AI Agent

PR Hygiene · 68dba53

  • Bots — thepastaclaw ✓
  • Self-review — not asked of a bot author
  • Build green
  • Approvals
    • files with no dedicated owner (config/config.go, config/toml.go, docs/nodes/configuration.md and 11 more) — lklimek or shumkov

When every merge requirement is met, the PR Hygiene check passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.

lklimek and others added 3 commits November 9, 2026 11:16
A peer learns this node's height, round and step only from NewRoundStep,
sent once on connect and once per step change, never acknowledged. One
lost after the channel accepted it (router queue overflow, the peer's
state-channel allowance) leaves the peer gossiping votes and proposals
for the round it last heard of until this node changes step — which may
never happen when that step is waiting on exactly the votes or proposal
the peer is holding back, stalling the round until a reconnect.

Add a per-peer gossip handler that re-sends the current round step on
the PeerQueryMaj23SleepDuration cadence (one cost-1 state message per
peer every 2s at the default). A repeat is a no-op for a peer whose
picture is current, so the healthy path is unchanged beyond the message.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Decouples the handler from p2pMsgSender and matches the shape the
pending round-skip gossip handlers use.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The per-peer gossip handlers registered at this cadence run back to back
when it is zero; the round-step refresh sends unconditionally, so a zero
value would flood every peer's state channel and burn this node's
allowance there. Reject zero in ValidateBasic and document that the
setting also bounds round-step re-announcement latency.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@thepastaclaw

thepastaclaw commented Nov 9, 2026 •

Copy link
Copy Markdown

✅ Final review complete — no blockers (commit 68dba53) · triage: critical

…ate-refresh

# Conflicts:
#	internal/consensus/gossip_peer_worker.go
lklimek and others added 5 commits November 9, 2026 14:06
The per-peer refresh re-sent only NewRoundStep, while resendRoundState
treats a round picture as NewRoundStep plus NewValidBlock whenever the
round has block parts. A lost NewValidBlock left the peer without this
node's part-set header and holdings, and nothing repeated it. The handler
now sends both from a shared roundStateMessages helper, also used by
resendRoundState; a failed send ends the pass and the next one retries.

The explicit NewRoundStep send in peerUp is removed: the worker starts
after readySignal, where WaitSync is already false, and its first pass
sends the announcement at once, so the two paths were redundant.

The handler test moves next to its sibling handler tests, uses the same
capturing-send pattern, and covers the send-failure branch; the P2P spec
notes the periodic re-announcement.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Rejecting peer-query-maj23-sleep-duration <= 0 in ValidateBasic was a
breaking config change that still left the loop unguarded: any tiny
positive value spun the handlers the same way, peer-gossip-sleep-duration
= 0 was still accepted, and ValidateBasic runs only on the CLI path, so an
embedder calling node.New got no check at all.

The per-peer gossip worker now runs no handler faster than
minGossipSleepDuration (1ms, the same floor as incoming-connection-window)
whatever either setting says, warning once per worker when it clamps. The
<= 0 rejection and its test are reverted, so a 0s config starts again,
and the config comments and the maj23AnswerTTLFor zero branch describe
the floor rather than a validation rule.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A peer's periodic NewValidBlock landing before we send it the round's
proposal makes PeerState.SetHasProposal return early, so the proposal's
POL round is never recorded for the peer. The test fails on adbc822.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SetHasProposal returned before recording ProposalPOLRound when the peer's
NewValidBlock had already set its block parts, so a re-proposal's POL
round stayed at -1 and the peer was never sent the POL prevotes or the POL
majority claim. The periodic round-state refresh makes that ordering
routine. The POL fields are now recorded before the block-parts early
return; the reviewer's repro test moves next to the other peer-state
tests.

ApplyNewValidBlockMessage keeps replacing the header and holdings with no
ordering guard, now documented: a stale copy is corrected within one
pass, whereas a grow-only guard would make a wrong bit permanent.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The floor was applied inside newPeerGossipWorker, so the maj23 answer TTL
still read the raw config value and the warning repeated on every peerUp.
NewReactor now derives a gossipCadence from the config once, warning once
per raised setting, and both the per-peer workers and maj23AnswerTTLFor
read it from there.

The floor's comment and the config docs now say what it does: it stops a
busy loop and nothing more; well below the defaults the handlers still
offer peers more than their state-channel allowance admits.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Claudius-Maginificent Claudius-Maginificent changed the title fix(consensus)!: periodically re-announce round step to peers fix(consensus): periodically re-announce round state to peers Nov 9, 2026
@lklimek lklimek added this to the v2.1 milestone Nov 16, 2026

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final validation — Phase 1 + Phase 2

The cadence floor is correctly derived once in the reactor and used by peer workers and majority-answer TTLs. One shutdown retry path still reads the raw configuration, so zero or sub-millisecond settings bypass the stated 1 ms floor and can retry immediately; the worker-level concern is speculative because the constructor is package-private and all current callers pass the reactor-derived cadence.

🟡 1 suggestion(s)

Approved: nothing here blocks the merge. The non-blocking findings are optional: fix them forward, in this PR or a follow-up, without waiting for another review round.

1 finding(s) not shown inline (the lines are not part of this PR's diff)

🟡 Suggestion: Apply the cached floor to peer-up retry backoff
internal/consensus/reactor.go:956

This retry path still sleeps on the raw PeerGossipSleepDuration, while the reactor-level floor is used by every peer gossip handler and the majority-answer TTL. When the accepted configuration is zero or below 1 ms, a peer-up observed during shutdown retries immediately rather than respecting the floor, creating an avoidable burst of retry and peer-admission work. Use the already-derived cadence here so all consumers share the same bounded delay.

time.Sleep(r.cadence.gossip)

source: muse-spark-1.3-contributor (phase1-reviewer: general); gpt-6.1-sol (phase2-reviewer: general, tenderdash-consensus-security)

Review provenance

Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: critical by gpt-6.1-sol (effort low) — The change is large and intricate and directly modifies consensus round-state propagation, peer-facing consensus message handling, and recovery behavior in internal/consensus/gossip_handlers.go, gossip_peer_worker.go, peer_state.go, and reactor.go.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Single stage: Phase 1 and Phase 2 reviewed this head side by side, with no blocker gate between them (triage tier)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort xhigh); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions don't block the merge, but most are worth fixing in this PR; judge each on its merits.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.

In `internal/consensus/reactor.go`:
- [SUGGESTION] internal/consensus/reactor.go:956: Apply the cached floor to peer-up retry backoff
  This retry path still sleeps on the raw `PeerGossipSleepDuration`, while the reactor-level floor is used by every peer gossip handler and the majority-answer TTL. When the accepted configuration is zero or below 1 ms, a peer-up observed during shutdown retries immediately rather than respecting the floor, creating an avoidable burst of retry and peer-admission work. Use the already-derived cadence here so all consumers share the same bounded delay.

@lklimek
lklimek marked this pull request as ready for review November 16, 2026 11:03
@github-actions

Copy link
Copy Markdown

Ready for review — files with no dedicated owner: lklimek or shumkov.
Full checklist in the description.

@github-actions github-actions Bot added the ready-for-human PR Hygiene: bots done, author self-reviewed, awaiting human review label Nov 16, 2026
@github-actions
github-actions Bot requested review from lklimek and shumkov November 16, 2026 11:04
@Claudius-Maginificent

Copy link
Copy Markdown
Collaborator Author

Declining the peer-up retry backoff suggestion. Dashmate's mainnet preset inherits gossipSleepDuration: '100ms' from getBaseConfigFactory.js, and config.toml.dot writes it directly as peer-gossip-sleep-duration. At this setting, the raw and floored delays are identical, so the reported case is unreachable with our standard mainnet configuration.

With a custom value below 1ms, the consequence is only a small, bounded amount of additional retry/admission work during shutdown (peerUp starts with a retry budget of 3), with no consensus correctness impact. We accept this edge case and will leave the code unchanged.

🤖 Co-authored by Claudius the Magnificent AI Agent

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human PR Hygiene: bots done, author self-reviewed, awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants