From f0bf55d58211ed0f2b394a72856ad4ac2b89e2b7 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 10:42:50 +0000 Subject: [PATCH 01/36] fix(blocksync): hand over to consensus at the tip despite unserved peer height claims Block sync considered the node caught up only once it reached the highest height any peer reported. A peer that reported a much higher height and did not deliver blocks kept that claim counted: the existing stall checks apply only when no peer advertises the next height or the node is within 10 blocks of the claim, blocks arriving from other peers kept resetting the stall timer, and a peer dropped after failed requests was re-added with a clean history on its next status. A restarted node could then stay in block sync instead of returning to consensus. Record, per peer ID, the lowest height the peer failed to serve (timeout or other failed request, NoBlockResponse, or a block that fails to apply). The record survives dropping and re-adding the peer, is cleared when a block that peer supplied at or above that height is applied, and expires 10 minutes after the last failure. Claims at or above it stop counting. Judge whether the node is at the tip from its last applied block time, which the validators commit to: once that time has stayed within five minutes of the local clock (and no further ahead than the synchrony bounds allow) for two minutes, and no counted claim covers the next height, hand over. If the last block is older, hand over after 10 minutes without progress when no counted claim covers the next height. The consensus catch-up target leaves out discounted claims. Bounds and remaining cases are documented in docs/tenderdash-core/block-sync/reactor.md. Co-Authored-By: Claude Opus 5.5 --- docs/tenderdash-core/block-sync/reactor.md | 56 ++ internal/blocksync/peer_service_test.go | 9 +- internal/blocksync/peer_store.go | 69 +- internal/blocksync/peer_store_test.go | 48 ++ internal/blocksync/reactor.go | 16 + internal/blocksync/synchronizer.go | 184 ++++- internal/blocksync/synchronizer_test.go | 816 +++++++++++++++++++-- 7 files changed, 1092 insertions(+), 106 deletions(-) diff --git a/docs/tenderdash-core/block-sync/reactor.md b/docs/tenderdash-core/block-sync/reactor.md index 5a31d7545..5774ac1a2 100644 --- a/docs/tenderdash-core/block-sync/reactor.md +++ b/docs/tenderdash-core/block-sync/reactor.md @@ -53,6 +53,62 @@ and hard failure policy are unchanged. The 60-second request deadline can still reject a large response on a slow connection even if its eventual service rate would meet the scheduling allowance. +Block sync hands over to consensus when it reaches the highest height any peer +claims; when no peer advertises the next height and no block has been applied +for 60 seconds; or when no block has been applied for 10 minutes while the node +is within 10 blocks of the highest claim that covers the next height. + +Peer height claims are unverified, so a claim stops counting toward that +decision once its peer fails a request: a timeout or other failure, a +`NoBlockResponse`, or a block that fails to apply. The claim is discounted from +the failed height up until a block that peer supplied at or above it is +applied. Unlike service history, this record survives dropping and re-adding the +peer; it is forgotten 10 minutes after the peer's last failure. + +Whether the node is at the tip is judged from the time of its last applied +block, which the validators commit to and a peer cannot change: a node behind +the network applies old blocks. Once that time has stayed no more than five +minutes behind the local clock, and no further ahead of it than the synchrony +bounds allow, for two minutes, and no peer whose claim still counts advertises +the next height, block sync hands over even though blocks keep arriving. Five +minutes covers the three-minute empty-block interval Dash +Platform runs with, the default synchrony bounds and a margin for extra rounds +and clock skew. If the last block is older than that, because the chain has +stopped producing blocks or does not produce empty ones, block sync hands over +once no block has been applied for 10 minutes and no counted claim covers the +next height. + +A peer that claims a far height and does not serve it therefore delays the +handover until two minutes after the node applies a block less than five +minutes old, or until its first request to that peer times out 60 seconds +later, whichever comes last. On a chain whose last block is older than five +minutes it takes 10 minutes. On this handover the consensus catch-up target leaves out discounted +claims, so they do not delay the node's own proposals. A claim whose range +starts above the node's height is never requested and stays in the target, so +consensus may withhold proposals for up to 10 minutes while it is above the node. + +Some cases remain open: + +- A peer that answers every block request within the 60-second request timeout + never fails, so its claim is never discounted and keeps the node in block sync + while it claims the next height. +- A node whose clock runs more than about two minutes ahead of the validators' + sees even the newest blocks as old, so on a live chain it does not hand over + past an unserved claim. +- A block time further ahead of the local clock than the synchrony bounds + (precision plus message delay) allow does not count as recent, since the + node's clock is then behind the validators' and cannot tell how old the + block is. Such a node, too, hands over past an unserved claim only once no + block has been applied for 10 minutes. +- Within five minutes of chain time of the tip, if the only peers that advertise + the next height have failed a request and serve nothing at or above it for two + minutes, block sync hands over even though those peers may be slow rather + than wrong. On a chain that commits many blocks a minute this can leave the + node more blocks behind than consensus catches up quickly. +- A node any distance behind whose only peers that advertise the next height + fail every request for 10 minutes hands over, as it would if none advertised + it. + The historical Blocksync reactor is organised as a set of concurrent tasks: - Receive routine of Blocksync Reactor diff --git a/internal/blocksync/peer_service_test.go b/internal/blocksync/peer_service_test.go index 0dfd42812..3c044f301 100644 --- a/internal/blocksync/peer_service_test.go +++ b/internal/blocksync/peer_service_test.go @@ -215,10 +215,11 @@ func TestServiceProbationRemainsServableBelowTip(t *testing.T) { endServiceRequest(syncer.peerStore, peer, request, 100) _, found := syncer.peerStore.FindPeer(1) require.False(t, found, "peer is cooling") - _, _, servable, _, hasPeers, maxServable := syncer.stallSnapshot() - require.True(t, servable) - require.True(t, hasPeers) - require.EqualValues(t, 1000, maxServable) + st := syncer.stallSnapshot() + require.True(t, st.servable) + require.True(t, st.hasPeers) + require.EqualValues(t, 1000, st.maxServableHeight) + require.EqualValues(t, 1000, st.maxCredibleHeight, "cooling is not a failure to serve") } func TestServiceSavedCreditCannotHideNewSlowService(t *testing.T) { diff --git a/internal/blocksync/peer_store.go b/internal/blocksync/peer_store.go index 3c3d2d000..2787006a3 100644 --- a/internal/blocksync/peer_store.go +++ b/internal/blocksync/peer_store.go @@ -1,7 +1,11 @@ package blocksync import ( + "maps" + "time" + "github.com/jonboulle/clockwork" + sync "github.com/sasha-s/go-deadlock" "golang.org/x/exp/constraints" "github.com/dashpay/tenderdash/libs/store" @@ -17,6 +21,18 @@ type ( // a concurrent write is lost. Nothing here substitutes another store. store *store.InMemStore[types.NodeID, PeerData] clock clockwork.Clock + + unservedMtx sync.Mutex + // unserved holds the peers whose claims are discounted. It is kept + // outside PeerData so that neither dropping the peer nor its next status + // erases it. + unserved map[types.NodeID]unservedRecord + } + // unservedRecord discounts a peer's claim from the lowest height it failed to serve. + unservedRecord struct { + from int64 + // failedAt is when the peer last failed, for pruning. + failedAt time.Time } // PeerData uses to keep peer related data like base height and the current height etc PeerData struct { @@ -40,8 +56,9 @@ type ( // NewInMemPeerStore creates a new in-memory peer store func NewInMemPeerStore(peers ...PeerData) *InMemPeerStore { mem := &InMemPeerStore{ - store: store.NewInMemStore[types.NodeID, PeerData](), - clock: clockwork.NewRealClock(), + store: store.NewInMemStore[types.NodeID, PeerData](), + clock: clockwork.NewRealClock(), + unserved: map[types.NodeID]unservedRecord{}, } for _, peer := range peers { mem.Put(peer.peerID, peer) @@ -191,6 +208,54 @@ func (p *InMemPeerStore) RecordMissingBlock(peerID types.NodeID, height int64) { }) } +// RecordUnserved notes that the peer failed to serve a height it was asked for. +func (p *InMemPeerStore) RecordUnserved(peerID types.NodeID, height int64) { + p.unservedMtx.Lock() + defer p.unservedMtx.Unlock() + now := p.pruneUnservedLocked() + record, ok := p.unserved[peerID] + if !ok || height < record.from { + record.from = height + } + record.failedAt = now + p.unserved[peerID] = record +} + +// RecordServed clears the peer's record once a block it supplied at or above +// the failed height has been applied. Delivery alone is not enough: anyone can +// send a block that fails verification. +func (p *InMemPeerStore) RecordServed(peerID types.NodeID, height int64) { + p.unservedMtx.Lock() + defer p.unservedMtx.Unlock() + if record, ok := p.unserved[peerID]; ok && height >= record.from { + delete(p.unserved, peerID) + } +} + +// unservedFrom returns the lowest unserved height of each of peers that has a record. +func (p *InMemPeerStore) unservedFrom(peers []PeerData) map[types.NodeID]int64 { + p.unservedMtx.Lock() + defer p.unservedMtx.Unlock() + p.pruneUnservedLocked() + from := make(map[types.NodeID]int64) + for _, peer := range peers { + if record, ok := p.unserved[peer.peerID]; ok { + from[peer.peerID] = record.from + } + } + return from +} + +// pruneUnservedLocked drops records of peers that have not failed for +// unservedRecordTTL and returns the time it pruned at. +func (p *InMemPeerStore) pruneUnservedLocked() time.Time { + now := p.clock.Now() + maps.DeleteFunc(p.unserved, func(_ types.NodeID, record unservedRecord) bool { + return now.Sub(record.failedAt) > unservedRecordTTL + }) + return now +} + // All returns all stored peers in the store func (p *InMemPeerStore) All() []PeerData { return p.store.All() diff --git a/internal/blocksync/peer_store_test.go b/internal/blocksync/peer_store_test.go index 833eb6867..416472210 100644 --- a/internal/blocksync/peer_store_test.go +++ b/internal/blocksync/peer_store_test.go @@ -4,7 +4,9 @@ import ( "fmt" "sync" "testing" + "time" + "github.com/jonboulle/clockwork" "github.com/stretchr/testify/require" "github.com/dashpay/tenderdash/types" @@ -315,3 +317,49 @@ func TestAddFailureClearsPending(t *testing.T) { require.EqualValues(t, 0, peer.numPending) require.EqualValues(t, 2, peer.numFailures) } + +// TestUnservedRecord checks what discounts a peer's claim and what makes it count +// again, and that records of peers that stop failing are forgotten. +func TestUnservedRecord(t *testing.T) { + clock := clockwork.NewFakeClock() + peers := NewInMemPeerStore() + peers.clock = clock + from := func(peerID types.NodeID) (int64, bool) { + height, ok := peers.unservedFrom([]PeerData{{peerID: peerID}})[peerID] + return height, ok + } + + peers.RecordUnserved("peer", 100) + peers.RecordUnserved("peer", 90) + height, ok := from("peer") + require.True(t, ok) + require.EqualValues(t, 90, height, "the lowest failed height is the one discounted from") + + peers.RecordServed("peer", 89) + _, ok = from("peer") + require.True(t, ok, "a block below the failed height does not restore the claim") + peers.RecordServed("peer", 90) + _, ok = from("peer") + require.False(t, ok, "an applied block at the failed height restores the claim") + + peers.RecordUnserved("gone", 5) + clock.Advance(unservedRecordTTL + time.Second) + peers.RecordUnserved("fresh", 7) + _, ok = from("gone") + require.False(t, ok, "a record not refreshed within unservedRecordTTL is dropped") + _, ok = from("fresh") + require.True(t, ok) +} + +// TestUnservedRecordExpiresWithoutAnotherFailure checks the documented expiry: a +// record is forgotten unservedRecordTTL after the peer's last failure, whether or +// not some other peer fails in the meantime. +func TestUnservedRecordExpiresWithoutAnotherFailure(t *testing.T) { + clock := clockwork.NewFakeClock() + peers := NewInMemPeerStore() + peers.clock = clock + peers.RecordUnserved("peer", 5) + clock.Advance(unservedRecordTTL + time.Second) + _, ok := peers.unservedFrom([]PeerData{{peerID: "peer"}})["peer"] + require.False(t, ok, "the record outlived unservedRecordTTL") +} diff --git a/internal/blocksync/reactor.go b/internal/blocksync/reactor.go index 43fb6501d..44e80726e 100644 --- a/internal/blocksync/reactor.go +++ b/internal/blocksync/reactor.go @@ -47,6 +47,22 @@ const ( // up on it puts a validator at heights the network committed long ago // (dashpay/tenderdash#1413). maxCatchupGap int64 = 10 + + // tipRecency is how recent the last applied block must be for the node to + // count as at the tip. It covers the three-minute empty-block interval Dash + // Platform runs with, the default synchrony bounds (12 s message delay, 505 ms + // precision) and a margin for extra rounds and clock skew. + tipRecency = 5 * time.Minute + + // unservedClaimGrace is how long the node must have been at the tip before + // claims that went unserved stop keeping it in block sync. It leaves a slow + // peer that timed out a cooldown and a full probe to serve again. + unservedClaimGrace = 2 * syncTimeout + + // unservedRecordTTL is how long a peer's claim stays discounted after its + // last failure. A peer still asked for blocks it cannot serve keeps failing + // well within it, so this only forgets peers that went away. + unservedRecordTTL = maxSyncStall ) type ReactorOption func(*Reactor) diff --git a/internal/blocksync/synchronizer.go b/internal/blocksync/synchronizer.go index 1c214d051..81840aea5 100644 --- a/internal/blocksync/synchronizer.go +++ b/internal/blocksync/synchronizer.go @@ -131,6 +131,11 @@ type ( logger log.Logger lastAdvance time.Time + // lastBlockTime is the time of the last applied block, guarded by mtx. + lastBlockTime time.Time + // synchrony bounds how far ahead of the local clock a block time can be + // for a clock in step with the validators'. Set before WaitForSync runs. + synchrony types.SynchronyParams mtx sync.RWMutex @@ -206,6 +211,7 @@ func NewSynchronizer(start int64, client client.BlockClient, blockExec *blockApp monitorInterval: defaultSyncRateIntervalBlocks, workerPool: workerpool.New(poolWorkerSize, workerpool.WithLogger(logger)), pendingToApply: map[int64]BlockResponse{}, + synchrony: types.DefaultSynchronyParams(), } for _, opt := range opts { opt(bp) @@ -244,6 +250,11 @@ func (s *Synchronizer) OnStart(ctx context.Context) error { } s.lastAdvance = s.clock.Now() s.lastMonitorUpdate = s.lastAdvance + if s.applier != nil { + state := s.applier.State() + s.lastBlockTime = state.LastBlockTime + s.synchrony = state.ConsensusParams.Synchrony.SynchronyParamsOrDefaults() + } s.ctx, s.cancel = context.WithCancel(ctx) applicationCtx, ok := ctx.Value(applicationContextKey{}).(context.Context) if !ok { @@ -324,6 +335,7 @@ func (s *Synchronizer) consumeJobResult(ctx, applyCtx context.Context) error { if !errors.As(res.Err, &bfErr) { return nil } + s.peerStore.RecordUnserved(bfErr.peerID, bfErr.height) var missing *client.ErrBlockNotFound if errors.As(bfErr.err, &missing) && missing.Height == bfErr.height { s.peerStore.RecordMissingBlock(bfErr.peerID, bfErr.height) @@ -383,6 +395,7 @@ func (s *Synchronizer) consumeJobResult(ctx, applyCtx context.Context) error { "height", failed.Block.Height, "peer", failed.PeerID, "error", err.Error()) + s.peerStore.RecordUnserved(failed.PeerID, failed.Block.Height) s.RemovePeer(failed.PeerID) _ = s.client.Send(ctx, p2p.PeerError{NodeID: failed.PeerID, Err: err, ConnID: failed.ConnID}) } @@ -444,65 +457,121 @@ func (s *Synchronizer) IsCaughtUp() bool { // block, or once the stall has outlasted maxSyncStall within maxCatchupGap of // the tip, so that a wedged synchronizer can still hand over rather than // blocking forever without handing over a node consensus cannot catch up. +// +// A height claim costs a peer nothing, so claims alone cannot keep the node in +// block sync once the node is at the tip. The time of the last applied block is +// what tells it so: the validators commit to it, and a node far behind applies +// old blocks. Once that time has stayed within tipRecency of the local clock for +// unservedClaimGrace, and no peer that has not failed to serve it holds the next +// block, the node hands over even though blocks arriving meanwhile keep it from +// stalling. A chain that stops producing blocks ages that time out, and then +// only a stall of maxSyncStall with no such peer ends block sync. func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHeight int64) { ticker := s.clock.NewTicker(switchToConsensusIntervalSeconds * time.Second) defer ticker.Stop() + atTipSince := s.clock.Now() for { select { case <-ctx.Done(): - height, _, _, peerHeight, hasPeers, _ := s.stallSnapshot() - return hasPeers && height >= peerHeight, max(targetHeight, peerHeight) + st := s.stallSnapshot() + return st.hasPeers && st.height >= st.maxPeerHeight, max(targetHeight, st.maxPeerHeight) case <-ticker.Chan(): - height, stalledFor, servable, maxPeerHeight, hasPeers, maxServableHeight := s.stallSnapshot() - targetHeight = max(targetHeight, maxPeerHeight) - if hasPeers && height >= maxPeerHeight { + st := s.stallSnapshot() + targetHeight = max(targetHeight, st.maxPeerHeight) + if st.hasPeers && st.height >= st.maxPeerHeight { return true, targetHeight } - switch stallVerdictFor(servable, maxServableHeight-height, stalledFor) { + now := s.clock.Now() + if !s.isRecent(st.lastBlockTime, now) { + atTipSince = now + } + atTipFor := now.Sub(atTipSince) + verdict := stallVerdictFor(st.servable, st.maxServableHeight-st.height, st.stalledFor, + st.maxCredibleHeight >= st.height, atTipFor) + switch verdict { + case stopUnservedClaims: + s.logger.Error( + "higher peer claims went unserved, handing over to consensus", + "height", st.height, + "max_peer_height", st.maxPeerHeight, + "max_counted_height", st.maxCountedHeight, + "last_block_time", st.lastBlockTime, + "stalled_for", st.stalledFor, + ) + // Claims that went unserved must not hold back our own proposals. + return false, st.maxCountedHeight case stopNothingToFetch: - if maxPeerHeight > height { + if st.maxPeerHeight > st.height { // Peers claim to be ahead yet none of them holds the block we // need. Worth saying: the node stops while apparently behind, // which otherwise looks like it gave up for no reason. s.logger.Error( "no peer can serve the next block, handing over to consensus", - "height", height, - "max_peer_height", maxPeerHeight, - "stalled_for", stalledFor, + "height", st.height, + "max_peer_height", st.maxPeerHeight, + "stalled_for", st.stalledFor, ) } return false, targetHeight case stopStalledTooLong: s.logger.Error( "block sync stalled for too long, handing over to consensus while still behind", - "height", height, - "max_peer_height", maxPeerHeight, - "max_servable_height", maxServableHeight, - "stalled_for", stalledFor, + "height", st.height, + "max_peer_height", st.maxPeerHeight, + "max_servable_height", st.maxServableHeight, + "stalled_for", st.stalledFor, ) return false, targetHeight } - if stalledFor > syncTimeout { + if st.stalledFor > syncTimeout { s.logger.Error( "block sync has stalled but a peer still holds the block we need, still trying", - "height", height, - "max_peer_height", maxPeerHeight, - "stalled_for", stalledFor, - "max_servable_height", maxServableHeight, - "behind_by", maxServableHeight-height, + "height", st.height, + "max_peer_height", st.maxPeerHeight, + "stalled_for", st.stalledFor, + "max_servable_height", st.maxServableHeight, + "behind_by", st.maxServableHeight-st.height, ) continue } s.logger.Info( "not caught up yet", - "height", height, - "max_peer_height", maxPeerHeight, - "timeout_in", syncTimeout-stalledFor, + "height", st.height, + "max_peer_height", st.maxPeerHeight, + "timeout_in", syncTimeout-st.stalledFor, ) } } } +// isRecent reports whether a block with blockTime puts the node at the tip. A +// time further ahead of the local clock than the synchrony bounds allow means +// the clock is behind the validators', so how old the block is cannot be told. +func (s *Synchronizer) isRecent(blockTime, now time.Time) bool { + age := now.Sub(blockTime) + return age <= tipRecency && -age <= s.synchrony.Precision+s.synchrony.MessageDelay +} + +// stallState is one observation of what the stall verdict is formed from. +type stallState struct { + // height is the next height to apply, waiting for stalledFor. + height int64 + stalledFor time.Duration + hasPeers bool + // servable says some peer advertises height; maxServableHeight is the + // highest such claim and maxPeerHeight the highest claim of any peer. + servable bool + maxServableHeight int64 + maxPeerHeight int64 + // maxCredibleHeight is the highest servable claim still counted, capped below + // the height its peer failed; 0 when none covers height. + maxCredibleHeight int64 + // maxCountedHeight is the highest claim of any peer, capped the same way. + maxCountedHeight int64 + // lastBlockTime is the time of the last applied block. + lastBlockTime time.Time +} + // stallSnapshot reads everything the stall verdict is formed from as one // observation: the height block sync is waiting for, how long it has been // waiting for it, and the highest advertised and servable peer heights. @@ -513,31 +582,45 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe // - a peer whose blocks start above us has nothing we can use however high it // claims to be. Only servable peers determine the stall gap; the overall maximum // remains the handover target so pruning cannot erase evidence of a higher tip. +// A claim is counted only below the lowest height its peer failed to serve. // -// They are read under one lock because advance() stamps the height and the -// advance time together under that same lock. A block applied concurrently -// therefore lands either wholly inside the snapshot or wholly outside it, and -// the verdict can never pair a height with a staleness or a servability -// measured against a different one. Read separately, it could: a height read -// before an advance, checked for servability after one, reports nothing to -// fetch while the height we had by then moved on to is served. Ending block -// sync is a one-way door, so a stop assembled from two inconsistent readings -// leaves the node in consensus catch-up it cannot leave. -func (s *Synchronizer) stallSnapshot() ( - height int64, stalledFor time.Duration, servable bool, maxPeerHeight int64, hasPeers bool, maxServableHeight int64, -) { +// They are read under one lock because advance() stamps the height, the advance +// time, the block time and the supplier's record together under that same lock. A block applied +// concurrently therefore lands either wholly inside the snapshot or wholly +// outside it, and the verdict can never pair a height with a staleness or a +// servability measured against a different one. Read separately, it could: a +// height read before an advance, checked for servability after one, reports +// nothing to fetch while the height we had by then moved on to is served. Ending +// block sync is a one-way door, so a stop assembled from two inconsistent +// readings leaves the node in consensus catch-up it cannot leave. +func (s *Synchronizer) stallSnapshot() stallState { s.mtx.RLock() defer s.mtx.RUnlock() peers := s.peerStore.All() + unserved := s.peerStore.unservedFrom(peers) + st := stallState{ + height: s.height, + stalledFor: s.clock.Since(s.lastAdvance), + hasPeers: len(peers) > 0, + lastBlockTime: s.lastBlockTime, + } hasHeight := heightBetweenPeerHeightRange(s.height) for _, peer := range peers { - maxPeerHeight = max(maxPeerHeight, peer.height) + st.maxPeerHeight = max(st.maxPeerHeight, peer.height) + counted := peer.height + if from, failed := unserved[peer.peerID]; failed { + counted = min(counted, from-1) + } + st.maxCountedHeight = max(st.maxCountedHeight, counted) if hasHeight(peer.peerID, peer) { - servable = true - maxServableHeight = max(maxServableHeight, peer.height) + st.servable = true + st.maxServableHeight = max(st.maxServableHeight, peer.height) + if counted >= s.height { + st.maxCredibleHeight = max(st.maxCredibleHeight, counted) + } } } - return s.height, s.clock.Since(s.lastAdvance), servable, maxPeerHeight, len(peers) > 0, maxServableHeight + return st } // stallVerdict says what a lack of progress in block sync means. @@ -550,6 +633,8 @@ const ( stopNothingToFetch // stopStalledTooLong means a peer still holds it but we have to hand over anyway stopStalledTooLong + // stopUnservedClaims means the claims above us went unserved for too long + stopUnservedClaims ) // stallVerdictFor decides what to do when block sync has made no progress for @@ -569,12 +654,25 @@ const ( // block arrives anyway - but only within maxCatchupGap of the tip. Further back // the backstop buys nothing and costs what dashpay/tenderdash#1413 describes, so // block sync keeps retrying and says so every interval. -func stallVerdictFor(servable bool, behindBy int64, stalledFor time.Duration) stallVerdict { +// +// credibleServable tells such a peer apart once it has failed a request: it says +// whether a peer that has not failed to serve the block holds it. atTipFor is how +// long the last applied block has been recent. Unlike stalledFor it is not reset +// by applying blocks, so a node following its peers at the tip is not kept in +// block sync by a claim nobody serves. A node whose last block is not recent +// relies on a stall of maxSyncStall with no such peer holding the block instead. +func stallVerdictFor( + servable bool, behindBy int64, stalledFor time.Duration, credibleServable bool, atTipFor time.Duration, +) stallVerdict { switch { + case !credibleServable && atTipFor > unservedClaimGrace: + return stopUnservedClaims case stalledFor <= syncTimeout: return keepSyncing case !servable: return stopNothingToFetch + case !credibleServable && stalledFor > maxSyncStall: + return stopUnservedClaims case stalledFor > maxSyncStall && behindBy <= maxCatchupGap: return stopStalledTooLong default: @@ -656,6 +754,12 @@ func (s *Synchronizer) getPendingResponse() (BlockResponse, bool) { func (s *Synchronizer) advance() { s.mtx.Lock() defer s.mtx.Unlock() + if resp, ok := s.pendingToApply[s.height]; ok { + s.peerStore.RecordServed(resp.PeerID, s.height) + if resp.Block != nil { + s.lastBlockTime = resp.Block.Time + } + } delete(s.pendingToApply, s.height) s.height++ s.lastAdvance = s.clock.Now() diff --git a/internal/blocksync/synchronizer_test.go b/internal/blocksync/synchronizer_test.go index fb0a92363..3d4c91ba4 100644 --- a/internal/blocksync/synchronizer_test.go +++ b/internal/blocksync/synchronizer_test.go @@ -982,6 +982,9 @@ func (suite *SynchronizerTestSuite) TestApplyFailurePunishesSupplyingPeer() { suite.Require().NotContains(pool.pendingToApply, poisonH1.Block.Height) _, found := pool.peerStore.Get(poisonPeerID) suite.Require().False(found) + suite.Require().Equal(poisonH1.Block.Height, + pool.peerStore.unservedFrom([]PeerData{{peerID: poisonPeerID}})[poisonPeerID], + "a block that fails to apply leaves the claim to that height unserved") // The honest peer keeps both its response and its place in the peer store. suite.Require().Contains(pool.pendingToApply, honestH2.Block.Height) @@ -1326,6 +1329,20 @@ func (suite *SynchronizerTestSuite) TestNoBlockResponseForWrongHeightPreservesRa suite.EqualValues(1, peer.numFailures) } +// TestAdvanceCreditsTheSupplier checks that applying a block clears the record of +// the peer that supplied it and stamps its time in the same step that moves the +// height, so a stall snapshot never pairs the new height with old values. +func (suite *SynchronizerTestSuite) TestAdvanceCreditsTheSupplier() { + const peerID = types.NodeID("supplier") + pool := NewSynchronizer(100, suite.client, nil) + pool.peerStore.RecordUnserved(peerID, 100) + blockTime := time.Unix(1_700_000_000, 0) + pool.pendingToApply[100] = BlockResponse{PeerID: peerID, Block: &types.Block{Header: types.Header{Time: blockTime}}} + pool.advance() + suite.NotContains(pool.peerStore.unserved, peerID) + suite.Equal(blockTime, pool.lastBlockTime) +} + // TestStatusRefreshUpdatesAdvertisedRange checks that a status response still does // the one thing it is for: moving the range of blocks the peer claims to serve. It // is the guard on the tests above, which a synchronizer that ignored status @@ -1405,17 +1422,20 @@ func makePeers(numPeers int, minHeight, maxHeight int64) map[types.NodeID]PeerDa // is so far behind that consensus catch-up could never close the gap. func TestStallVerdictFor(t *testing.T) { testCases := []struct { - name string - servable bool - behindBy int64 - stalledFor time.Duration - want stallVerdict + name string + servable bool + behindBy int64 + stalledFor time.Duration + credibleServable bool + atTipFor time.Duration + want stallVerdict }{ { - name: "progressing with the block available", - servable: true, - stalledFor: syncTimeout / 2, - want: keepSyncing, + name: "progressing with the block available", + servable: true, + stalledFor: syncTimeout / 2, + credibleServable: true, + want: keepSyncing, }, { name: "progressing with nobody holding the block", @@ -1430,30 +1450,34 @@ func TestStallVerdictFor(t *testing.T) { want: stopNothingToFetch, }, { - name: "stalled but a peer still holds the block", - servable: true, - stalledFor: syncTimeout + time.Second, - want: keepSyncing, + name: "stalled but a peer still holds the block", + servable: true, + stalledFor: syncTimeout + time.Second, + credibleServable: true, + want: keepSyncing, }, { - name: "stalled just short of the wedge limit", - servable: true, - stalledFor: maxSyncStall, - want: keepSyncing, + name: "stalled just short of the wedge limit", + servable: true, + stalledFor: maxSyncStall, + credibleServable: true, + want: keepSyncing, }, { - name: "stalled past the wedge limit within reach of the tip", - servable: true, - behindBy: maxCatchupGap, - stalledFor: maxSyncStall + time.Second, - want: stopStalledTooLong, + name: "stalled past the wedge limit within reach of the tip", + servable: true, + behindBy: maxCatchupGap, + stalledFor: maxSyncStall + time.Second, + credibleServable: true, + want: stopStalledTooLong, }, { - name: "stalled past the wedge limit while far behind", - servable: true, - behindBy: maxCatchupGap + 1, - stalledFor: maxSyncStall + time.Second, - want: keepSyncing, + name: "stalled past the wedge limit while far behind", + servable: true, + behindBy: maxCatchupGap + 1, + stalledFor: maxSyncStall + time.Second, + credibleServable: true, + want: keepSyncing, }, { name: "the wedge limit does not override nothing to fetch", @@ -1461,10 +1485,41 @@ func TestStallVerdictFor(t *testing.T) { stalledFor: maxSyncStall + time.Second, want: stopNothingToFetch, }, + { + name: "only unserved claims stalled past the wedge limit", + servable: true, + behindBy: maxCatchupGap + 1, + stalledFor: maxSyncStall + time.Second, + want: stopUnservedClaims, + }, + { + name: "only unserved claims while at the tip", + servable: true, + behindBy: maxCatchupGap + 1, + atTipFor: unservedClaimGrace + time.Second, + want: stopUnservedClaims, + }, + { + name: "only unserved claims within the grace", + servable: true, + behindBy: maxCatchupGap + 1, + atTipFor: unservedClaimGrace, + want: keepSyncing, + }, + { + name: "a served claim outlasts the grace", + servable: true, + behindBy: maxCatchupGap + 1, + stalledFor: syncTimeout + time.Second, + credibleServable: true, + atTipFor: unservedClaimGrace + time.Second, + want: keepSyncing, + }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { - require.Equal(t, tc.want, stallVerdictFor(tc.servable, tc.behindBy, tc.stalledFor)) + require.Equal(t, tc.want, + stallVerdictFor(tc.servable, tc.behindBy, tc.stalledFor, tc.credibleServable, tc.atTipFor)) }) } } @@ -1473,9 +1528,14 @@ func TestStallVerdictFor(t *testing.T) { // The synchronizer never advances a height, so the only thing that changes the // verdict is how far the clock is moved. type waitForSyncHarness struct { - clock *clockwork.FakeClock - done chan bool - cancel context.CancelFunc + suite *SynchronizerTestSuite + sync *Synchronizer + resultCh chan workerpool.Result + clock *clockwork.FakeClock + done chan bool + cancel context.CancelFunc + // target is the handover target WaitForSync returned; read it only after done. + target int64 } // newWaitForSyncHarness starts WaitForSync at the given height with the given @@ -1484,22 +1544,35 @@ type waitForSyncHarness struct { func (suite *SynchronizerTestSuite) newWaitForSyncHarness(height int64, peers ...PeerData) *waitForSyncHarness { clock := clockwork.NewFakeClock() applier := newBlockApplier(suite.blockExec, suite.store, applierWithState(suite.initialState)) - sync := NewSynchronizer(height, suite.client, applier, WithClock(clock)) + resultCh := make(chan workerpool.Result, 1) + sync := NewSynchronizer(height, suite.client, applier, WithClock(clock), + WithWorkerPool(workerpool.New(0, workerpool.WithResultCh(resultCh)))) // Deliberately not started: WaitForSync reads only the height, the peer store // and lastAdvance, and leaving the job pipeline out keeps the ticker the sole // waiter on the fake clock. sync.lastAdvance = clock.Now() + // The node starts at the tip: its last block is recent. + sync.lastBlockTime = clock.Now() for _, peer := range peers { sync.AddPeer(peer) } + h := suite.startWaitForSync(sync, clock) + h.resultCh = resultCh + return h +} + +// startWaitForSync runs WaitForSync for sync on its own goroutine and returns once +// its ticker is registered with clock. +func (suite *SynchronizerTestSuite) startWaitForSync(sync *Synchronizer, clock *clockwork.FakeClock) *waitForSyncHarness { ctx, cancel := context.WithCancel(context.Background()) - done := make(chan bool, 1) + h := &waitForSyncHarness{suite: suite, sync: sync, clock: clock, done: make(chan bool, 1), cancel: cancel} go func() { - caughtUp, _ := sync.WaitForSync(ctx) - done <- caughtUp + caughtUp, target := sync.WaitForSync(ctx) + h.target = target + h.done <- caughtUp }() suite.Require().NoError(clock.BlockUntilContext(ctx, 1)) - return &waitForSyncHarness{clock: clock, done: done, cancel: cancel} + return h } // result waits up to timeout of real time for WaitForSync to return. ended @@ -1590,23 +1663,42 @@ func (suite *SynchronizerTestSuite) TestStallSnapshotServableHeight() { busyPeer := newPeerData("busy", 1, 30) busyPeer.numPending = maxPendingRequestsPerPeer testCases := []struct { - name string - peers []PeerData - wantMax int64 - wantUsable int64 + name string + peers []PeerData + unserved map[types.NodeID]int64 + wantMax int64 + wantUsable int64 + wantCredible int64 }{ {name: "no peers"}, { - name: "available ranges", - peers: []PeerData{newPeerData("archive", 1, 20), newPeerData("recent", 5, 30)}, - wantMax: 30, - wantUsable: 30, + name: "available ranges", + peers: []PeerData{newPeerData("archive", 1, 20), newPeerData("recent", 5, 30)}, + wantMax: 30, + wantUsable: 30, + wantCredible: 30, + }, + { + name: "pruned range", + peers: []PeerData{newPeerData("archive", 1, 20), newPeerData("pruned", 25, 30)}, + wantMax: 30, + wantUsable: 20, + wantCredible: 20, + }, + { + name: "claim failed above our height", + peers: []PeerData{newPeerData("archive", 1, 20), newPeerData("recent", 5, 30)}, + unserved: map[types.NodeID]int64{"recent": 25}, + wantMax: 30, + wantUsable: 30, + wantCredible: 24, }, { - name: "pruned range", - peers: []PeerData{newPeerData("archive", 1, 20), newPeerData("pruned", 25, 30)}, + name: "claims failed at our height", + peers: []PeerData{newPeerData("archive", 1, 20), newPeerData("recent", 5, 30)}, + unserved: map[types.NodeID]int64{"archive": height, "recent": height}, wantMax: 30, - wantUsable: 20, + wantUsable: 30, }, { name: "no overlapping range", @@ -1614,10 +1706,11 @@ func (suite *SynchronizerTestSuite) TestStallSnapshotServableHeight() { wantMax: 30, }, { - name: "busy peer remains servable", - peers: []PeerData{busyPeer}, - wantMax: 30, - wantUsable: 30, + name: "busy peer remains servable", + peers: []PeerData{busyPeer}, + wantMax: 30, + wantUsable: 30, + wantCredible: 30, }, } for _, tc := range testCases { @@ -1627,11 +1720,15 @@ func (suite *SynchronizerTestSuite) TestStallSnapshotServableHeight() { for _, peer := range tc.peers { sync.AddPeer(peer) } - _, _, servable, maxHeight, hasPeers, maxServableHeight := sync.stallSnapshot() - suite.Require().Equal(tc.wantMax, maxHeight) - suite.Require().Equal(tc.wantUsable, maxServableHeight) - suite.Require().Equal(tc.wantUsable > 0, servable) - suite.Require().Equal(len(tc.peers) > 0, hasPeers) + for peerID, from := range tc.unserved { + sync.peerStore.RecordUnserved(peerID, from) + } + st := sync.stallSnapshot() + suite.Require().Equal(tc.wantMax, st.maxPeerHeight) + suite.Require().Equal(tc.wantUsable, st.maxServableHeight) + suite.Require().Equal(tc.wantCredible, st.maxCredibleHeight) + suite.Require().Equal(tc.wantUsable > 0, st.servable) + suite.Require().Equal(len(tc.peers) > 0, st.hasPeers) }) } } @@ -1661,7 +1758,9 @@ func (suite *SynchronizerTestSuite) TestStallSnapshotIsOneObservation() { }() runtime.Gosched() close(ready) - height, stalled, servable, maxPeerHeight, hasPeers, maxServableHeight := sync.stallSnapshot() + st := sync.stallSnapshot() + height, stalled, servable, maxPeerHeight, hasPeers, maxServableHeight := + st.height, st.stalledFor, st.servable, st.maxPeerHeight, st.hasPeers, st.maxServableHeight <-applied suite.Require().True(hasPeers) suite.Require().Equal(int64(1000), maxPeerHeight, "the retained handover target") @@ -1697,6 +1796,601 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncStopsWhenThereIsNothingToFetc suite.Require().False(caughtUp) } +// fail hands the consumer a request to peerID for height that failed with err, as +// the worker pool would. +func (h *waitForSyncHarness) fail(peerID types.NodeID, height int64, err error) { + h.resultCh <- errorResult(PeerData{peerID: peerID}, height, err) + h.suite.Require().NoError(h.sync.consumeJobResult(context.Background(), context.Background())) +} + +// run moves the fake clock a second at a time for up to limit, calling each +// before every step. It returns how long WaitForSync took to hand over, its +// verdict, and whether it did by limit. +func (h *waitForSyncHarness) run(limit time.Duration, each func(elapsed time.Duration)) ( + elapsed time.Duration, caughtUp bool, ended bool, +) { + for ; elapsed <= limit; elapsed += time.Second { + select { + case caughtUp = <-h.done: + return elapsed, caughtUp, true + default: + } + if each != nil { + each(elapsed) + } + h.clock.Advance(time.Second) + // let WaitForSync read this tick before the next one moves the clock again + time.Sleep(time.Millisecond) + } + caughtUp, ended = h.result(2 * time.Second) + return elapsed, caughtUp, ended +} + +// tipBlocks returns a step for run that, every interval, lets the tip peer +// advertise the height the node needs and applies it, as for a node level with +// the network when it commits a block. Blocks arrive well within syncTimeout, so +// block sync never sees a stall. +func (h *waitForSyncHarness) tipBlocks(tipPeer types.NodeID, interval time.Duration) func(time.Duration) { + return func(elapsed time.Duration) { + if elapsed == 0 || elapsed%interval != 0 { + return + } + height, _ := h.sync.GetStatus() + h.sync.AddPeer(newPeerData(tipPeer, 1, height)) + h.suite.Require().NoError(h.sync.addBlock(BlockResponse{ + PeerID: tipPeer, + Block: &types.Block{Header: types.Header{Height: height, Time: h.clock.Now()}}, + })) + h.sync.advance() + } +} + +const ( + // tipBlockInterval is shorter than syncTimeout, so chain growth alone + // keeps every stall-based handover from firing. + tipBlockInterval = 30 * time.Second + // unservedHandoverSlack covers ticks WaitForSync reads late. + unservedHandoverSlack = 10 * time.Second +) + +// TestWaitForSyncHandsOverPastUnservedClaims checks that a node level with its +// the other peers hands over to consensus even while another peer claims heights +// far above and never serves them, and while new blocks keep arriving faster +// than a stall can be declared. Each variant costs the peer nothing to send. +func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastUnservedClaims() { + const ( + height = int64(100) + tipPeer = types.NodeID("tip peer") + unserving = types.NodeID("unserving peer") + ) + testCases := []struct { + name string + claimBase int64 + evidence func(h *waitForSyncHarness) + }{ + { + name: "claims our height and never answers", + claimBase: 1, + evidence: func(h *waitForSyncHarness) { h.fail(unserving, height, client.ErrPeerNotResponded) }, + }, + { + name: "claims our height and answers it has no block", + claimBase: 1, + evidence: func(h *waitForSyncHarness) { h.fail(unserving, height, &client.ErrBlockNotFound{Height: height}) }, + }, + { + // The base stays above every height new blocks take us to here, + // so the peer is never asked for anything and never fails. + name: "claims a range starting above ours", + claimBase: height + 1000, + evidence: func(*waitForSyncHarness) {}, + }, + } + for _, tc := range testCases { + suite.Run(tc.name, func() { + h := suite.newWaitForSyncHarness(height, + newPeerData(tipPeer, 1, height-1), + newPeerData(unserving, tc.claimBase, maxPlausiblePeerHeight)) + defer h.cancel() + tc.evidence(h) + + elapsed, caughtUp, ended := h.run(unservedClaimGrace+time.Minute, h.tipBlocks(tipPeer, tipBlockInterval)) + suite.Require().True(ended, "block sync kept following the tip behind a claim nobody serves") + suite.Require().False(caughtUp, "an unserved claim was not reached, so this is not catching up") + suite.Require().LessOrEqual(elapsed, unservedClaimGrace+unservedHandoverSlack) + }) + } +} + +// TestWaitForSyncUnservedClaimSurvivesReAdd checks that a peer dropped for failing +// the heights it claims does not regain its standing by sending its status again. +// The node polls every peer for status and a peer may also push one unsolicited, +// so without this the peer is back with a clean record seconds after each drop. +func (suite *SynchronizerTestSuite) TestWaitForSyncUnservedClaimSurvivesReAdd() { + const ( + height = int64(100) + tipPeer = types.NodeID("tip peer") + unserving = types.NodeID("unserving peer") + ) + h := suite.newWaitForSyncHarness(height, + newPeerData(tipPeer, 1, height-1), + newPeerData(unserving, 1, maxPlausiblePeerHeight)) + defer h.cancel() + + suite.client. + On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")). + Once(). + Return(nil) + for i := int64(0); i < int64(maxConsecutiveFailures); i++ { + h.fail(unserving, height+i, client.ErrPeerNotResponded) + } + _, found := h.sync.peerStore.Get(unserving) + suite.Require().False(found, "the peer must be dropped after %d failed requests", maxConsecutiveFailures) + h.sync.AddPeer(newPeerData(unserving, 1, maxPlausiblePeerHeight)) + + elapsed, caughtUp, ended := h.run(unservedClaimGrace+time.Minute, h.tipBlocks(tipPeer, tipBlockInterval)) + suite.Require().True(ended, "a fresh status restored the claim the peer had failed to serve") + suite.Require().False(caughtUp) + suite.Require().LessOrEqual(elapsed, unservedClaimGrace+unservedHandoverSlack) +} + +// TestWaitForSyncHandsOverPastTwoUnservedClaims checks that a second peer joining +// later only delays the handover until it, too, fails to serve what it claims. +func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastTwoUnservedClaims() { + const ( + height = int64(100) + tipPeer = types.NodeID("tip peer") + first = types.NodeID("first unserving peer") + second = types.NodeID("second unserving peer") + joins = 30 * time.Second + timeout = joins + 60*time.Second + ) + h := suite.newWaitForSyncHarness(height, + newPeerData(tipPeer, 1, height-1), + newPeerData(first, 1, maxPlausiblePeerHeight)) + defer h.cancel() + h.fail(first, height, client.ErrPeerNotResponded) + + traffic := h.tipBlocks(tipPeer, tipBlockInterval) + elapsed, caughtUp, ended := h.run(timeout+unservedClaimGrace+time.Minute, func(elapsed time.Duration) { + traffic(elapsed) + switch elapsed { + case joins: + h.sync.AddPeer(newPeerData(second, 1, maxPlausiblePeerHeight)) + case timeout: + height, _ := h.sync.GetStatus() + h.fail(second, height, client.ErrPeerNotResponded) + } + }) + suite.Require().True(ended, "block sync kept following the tip behind two unserved claims") + suite.Require().False(caughtUp) + suite.Require().Greater(elapsed, timeout, "handed over while the second peer's claim was still untested") + suite.Require().LessOrEqual(elapsed, timeout+unservedClaimGrace+unservedHandoverSlack) +} + +// TestWaitForSyncKeepsSyncingBehindAServedClaim is the control for the tests +// above: a peer that failed a request does not end block sync while another +// peer still has far more to give. +func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingBehindAServedClaim() { + const ( + height = int64(100) + unserving = types.NodeID("unserving peer") + ) + h := suite.newWaitForSyncHarness(height, + newPeerData("peer far ahead", 1, height+1000), + newPeerData(unserving, 1, maxPlausiblePeerHeight)) + defer h.cancel() + h.fail(unserving, height, client.ErrPeerNotResponded) + + _, _, ended := h.run(3*unservedClaimGrace, nil) + suite.Require().False(ended, "block sync gave up while another peer still held 1000 more blocks") +} + +// TestWaitForSyncKeepsSyncingWhenASlowPeerRecovers checks that a peer far ahead +// which once failed the height we need, and then serves a run of blocks from it, +// counts again: one timeout from a slow peer does not make its claim false, and +// handing over while thousands of blocks behind leaves the node in consensus +// catch-up it cannot finish (dashpay/tenderdash#1413). A second peer level with +// the node is what lets the unserved-claims rule apply at all. +func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingWhenASlowPeerRecovers() { + ctx := context.Background() + b := suite.newBacklogHarness() + clock := clockwork.NewFakeClock() + b.pool.clock = clock + b.pool.lastAdvance = clock.Now() + for _, peer := range b.pool.peerStore.All() { + b.pool.peerStore.Delete(peer.peerID) + } + const slow = types.NodeID("slow peer") + b.pool.AddPeer(newPeerData(slow, 1, 10000)) + b.pool.AddPeer(newPeerData("level", 0, 0)) + + const run = maxCatchupGap + 1 + first := b.produce(ctx) + suite.Require().NotNil(first) + suite.Require().EqualValues(1, first.Block.Height) + b.timeout(ctx, slow, 1) + responses := map[int64]*BlockResponse{} + for len(responses) < int(run) { + resp := b.produce(ctx) + suite.Require().NotNil(resp) + responses[resp.Block.Height] = resp + } + + h := suite.startWaitForSync(b.pool, clock) + defer h.cancel() + _, _, ended := h.run(3*unservedClaimGrace, func(elapsed time.Duration) { + if elapsed == unservedClaimGrace/2 { + for height := int64(1); height <= run; height++ { + b.deliver(ctx, responses[height]) + } + b.pool.AddPeer(newPeerData("level", 0, run)) + } + }) + suite.Require().False(ended, "block sync gave up on a peer that served the heights it had failed") + height, _ := b.pool.GetStatus() + suite.Require().EqualValues(run+1, height, "the recovered blocks must have been applied") +} + +// TestWaitForSyncKeepsSyncingBehindASoleSlowPeerThatFailsTwice checks that a sole +// peer thousands of blocks ahead, which times out the height we need and then +// times out the recovery probe for it too, does not make block sync hand over +// while the node is still far behind. Two timeouts are well short of the +// failures that drop a peer, and the peer serves the height afterwards. +func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingBehindASoleSlowPeerThatFailsTwice() { + ctx := context.Background() + b := suite.newBacklogHarness() + fake := clockwork.NewFakeClock() + b.pool.clock = fake + b.pool.lastAdvance = fake.Now() + for _, peer := range b.pool.peerStore.All() { + b.pool.peerStore.Delete(peer.peerID) + } + const slow = types.NodeID("slow peer") + b.pool.AddPeer(newPeerData(slow, 1, 10000)) + + resp := b.produce(ctx) + suite.Require().NotNil(resp) + suite.Require().EqualValues(1, resp.Block.Height) + b.timeout(ctx, slow, 1) + + h := suite.startWaitForSync(b.pool, fake) + defer h.cancel() + // first timeout at 0; probe 30s after it times out 60s later; served afterwards + const ( + probeTimedOut = 90 * time.Second + served = unservedClaimGrace + 30*time.Second + ) + elapsed, _, ended := h.run(3*unservedClaimGrace, func(elapsed time.Duration) { + switch elapsed { + case probeTimedOut: + b.timeout(ctx, slow, 1) + case served: + b.deliver(ctx, resp) + } + }) + suite.Require().False(ended, + "block sync handed over after %s while 9999 blocks behind a peer that failed only twice", elapsed) +} + +// TestWaitForSyncHandsOverPastAClaimServedOnlyAtTheTip checks the handover bound +// against a peer that claims a far height but supplies the newest block of the +// chain once in a while, failing the height after it in between. Serving a block +// every other peer will also hold says nothing about the far claim. +func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAClaimServedOnlyAtTheTip() { + const ( + height = int64(100) + tipPeer = types.NodeID("tip peer") + relay = types.NodeID("relay") + // the network commits a block every serveEvery; the relay supplies it and + // times out the next height's request syncTimeout later + serveEvery = 90 * time.Second + ) + h := suite.newWaitForSyncHarness(height, + newPeerData(tipPeer, 1, height-1), + newPeerData(relay, 1, maxPlausiblePeerHeight)) + defer h.cancel() + h.fail(relay, height, client.ErrPeerNotResponded) + + limit := 10 * time.Minute + elapsed, caughtUp, ended := h.run(limit, func(elapsed time.Duration) { + if elapsed == 0 { + return + } + switch elapsed % serveEvery { + case 0: + next, _ := h.sync.GetStatus() + // what applyBlock does for a block the relay supplied + h.sync.advance() + h.sync.peerStore.RecordServed(relay, next) + h.sync.peerStore.Update(relay, ResetFailures()) + h.sync.AddPeer(newPeerData(tipPeer, 1, next)) + case syncTimeout: + next, _ := h.sync.GetStatus() + h.fail(relay, next, client.ErrPeerNotResponded) + } + }) + suite.Require().True(ended, + "block sync still following the tip after %s behind a claim nobody serves", limit) + suite.Require().False(caughtUp) + suite.Require().LessOrEqual(elapsed, syncTimeout+unservedClaimGrace+unservedHandoverSlack) +} + +// TestWaitForSyncKeepsSyncingBehindASlowPeerWithASyncingPeerAlongside is the sole +// slow peer case with one more peer that is itself still syncing at our height, +// as when several nodes start together. That peer has nothing to offer, so it +// must not make the node give up on the slow peer thousands of blocks ahead. +func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingBehindASlowPeerWithASyncingPeerAlongside() { + ctx := context.Background() + b := suite.newBacklogHarness() + fake := clockwork.NewFakeClock() + b.pool.clock = fake + b.pool.lastAdvance = fake.Now() + for _, peer := range b.pool.peerStore.All() { + b.pool.peerStore.Delete(peer.peerID) + } + const slow = types.NodeID("slow peer") + b.pool.AddPeer(newPeerData(slow, 1, 10000)) + b.pool.AddPeer(newPeerData("syncing peer", 0, 0)) + + resp := b.produce(ctx) + suite.Require().NotNil(resp) + suite.Require().EqualValues(1, resp.Block.Height) + b.timeout(ctx, slow, 1) + + h := suite.startWaitForSync(b.pool, fake) + defer h.cancel() + const ( + probeTimedOut = 90 * time.Second + served = unservedClaimGrace + 30*time.Second + ) + elapsed, _, ended := h.run(3*unservedClaimGrace, func(elapsed time.Duration) { + switch elapsed { + case probeTimedOut: + b.timeout(ctx, slow, 1) + case served: + b.deliver(ctx, resp) + } + }) + suite.Require().False(ended, + "block sync handed over after %s while 9999 blocks behind, because a syncing peer sat at our height", elapsed) +} + +// TestWaitForSyncHandsOverPastAnUnservedClaimOnAFastChain checks the handover bound +// when the chain commits more than maxCatchupGap blocks between two status rounds, +// so each status briefly shows the other peers more than maxCatchupGap ahead +// until the node applies what they report a second later. +func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAnUnservedClaimOnAFastChain() { + const ( + height = int64(100) + tip = types.NodeID("tip peer") + unserving = types.NodeID("unserving peer") + perRound = maxCatchupGap + 2 + ) + h := suite.newWaitForSyncHarness(height, + newPeerData(tip, 1, height-1), + newPeerData(unserving, 1, maxPlausiblePeerHeight)) + defer h.cancel() + h.fail(unserving, height, client.ErrPeerNotResponded) + + limit := 10 * time.Minute + elapsed, caughtUp, ended := h.run(limit, func(elapsed time.Duration) { + if elapsed == 0 { + return + } + next, _ := h.sync.GetStatus() + switch elapsed % (10 * time.Second) { + case 0: + // status round: the tip peer reports perRound new blocks + h.sync.AddPeer(newPeerData(tip, 1, next-1+perRound)) + case 2 * time.Second: + for i := int64(0); i < perRound; i++ { + h.sync.advance() + } + } + }) + suite.Require().True(ended, "block sync still following the tip after %s behind a claim nobody serves", limit) + suite.Require().False(caughtUp) + suite.Require().LessOrEqual(elapsed, syncTimeout+unservedClaimGrace+unservedHandoverSlack) +} + +// TestWaitForSyncHandsOverPastAnUnservedClaimOnAHaltedChain checks that a chain +// that stopped producing blocks, as one waiting for this node's vote may, does not +// leave the node in block sync behind a claim nobody serves. Its last block is +// too old to say the node is at the tip, so the wedge limit decides. +func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAnUnservedClaimOnAHaltedChain() { + const ( + height = int64(100) + tipPeer = types.NodeID("tip peer") + unserving = types.NodeID("unserving peer") + ) + h := suite.newWaitForSyncHarness(height, + newPeerData(tipPeer, 1, height-1), + newPeerData(unserving, 1, maxPlausiblePeerHeight)) + defer h.cancel() + h.sync.mtx.Lock() + h.sync.lastBlockTime = h.clock.Now().Add(-tipRecency) + h.sync.mtx.Unlock() + h.fail(unserving, height, client.ErrPeerNotResponded) + + // the peer keeps failing the probes it is sent for the height, and reports + // its status again whenever it is dropped for that + suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) + const probeEvery = 90 * time.Second + elapsed, caughtUp, ended := h.run(maxSyncStall+time.Minute, func(elapsed time.Duration) { + if elapsed > 0 && elapsed%probeEvery == 0 { + h.fail(unserving, height, client.ErrPeerNotResponded) + h.sync.AddPeer(newPeerData(unserving, 1, maxPlausiblePeerHeight)) + } + }) + suite.Require().True(ended, "block sync kept waiting on a halted chain behind a claim nobody serves") + suite.Require().False(caughtUp) + suite.Require().Greater(elapsed, maxSyncStall, "handed over with no recent block before the wedge limit") + suite.Require().LessOrEqual(elapsed, maxSyncStall+unservedHandoverSlack) +} + +// flakySource returns a step for run in which source, the only peer far ahead, +// supplies the next block every deliverEvery and times out one request every +// failEvery, the way a loaded peer does. The node keeps making progress. +func (h *waitForSyncHarness) flakySource(source types.NodeID, deliverEvery, failEvery time.Duration) func(time.Duration) { + return func(elapsed time.Duration) { + if elapsed == 0 { + return + } + next, _ := h.sync.GetStatus() + if elapsed%failEvery == 0 { + h.fail(source, next, client.ErrPeerNotResponded) + return + } + if elapsed%deliverEvery == 0 { + h.suite.Require().NoError(h.sync.addBlock(BlockResponse{ + PeerID: source, + Block: &types.Block{Header: types.Header{Height: next}}, + })) + h.sync.advance() + h.sync.peerStore.Update(source, ResetFailures()) + } + } +} + +// TestWaitForSyncKeepsSyncingFromAFlakySourceWithAPeerSyncingBehind checks that a +// node far behind, making steady progress from its only source ahead, does not +// hand over because that source times out a request now and then while another +// node syncs behind it. +func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingFromAFlakySourceWithAPeerSyncingBehind() { + const ( + height = int64(100) + source = types.NodeID("source") + behind = types.NodeID("peer syncing behind") + limit = 3 * unservedClaimGrace + deliver = 5 * time.Second + failure = 50 * time.Second + ) + h := suite.newWaitForSyncHarness(height, + newPeerData(source, 1, height+10000), + newPeerData(behind, 1, height-5)) + defer h.cancel() + h.fail(source, height, client.ErrPeerNotResponded) + + flaky := h.flakySource(source, deliver, failure) + elapsed, _, ended := h.run(limit, func(elapsed time.Duration) { + flaky(elapsed) + if elapsed%deliver == 0 { + next, _ := h.sync.GetStatus() + h.sync.AddPeer(newPeerData(behind, 1, next-3)) + } + }) + applied, _ := h.sync.GetStatus() + suite.Require().False(ended, + "handed over after %s at height %d, about 10000 blocks behind a source still delivering", elapsed, applied) +} + +// TestWaitForSyncKeepsSyncingFromAFlakySourceThroughADropAndReAdd checks the same +// for a sole source that once fails enough requests in a row to be dropped and +// comes back with its next status. +func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingFromAFlakySourceThroughADropAndReAdd() { + const ( + height = int64(100) + source = types.NodeID("source") + limit = 3 * unservedClaimGrace + deliver = 5 * time.Second + failure = 50 * time.Second + dropAt = unservedClaimGrace + 7*time.Second + ) + h := suite.newWaitForSyncHarness(height, newPeerData(source, 1, height+10000)) + defer h.cancel() + suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) + h.fail(source, height, client.ErrPeerNotResponded) + + flaky := h.flakySource(source, deliver, failure) + elapsed, _, ended := h.run(limit, func(elapsed time.Duration) { + switch elapsed { + case dropAt: + next, _ := h.sync.GetStatus() + for i := int64(0); i < int64(maxConsecutiveFailures); i++ { + h.fail(source, next+i, client.ErrPeerNotResponded) + } + return + case dropAt + 5*time.Second: + h.sync.AddPeer(newPeerData(source, 1, height+10000)) + return + } + if _, present := h.sync.peerStore.Get(source); present { + flaky(elapsed) + } + }) + applied, _ := h.sync.GetStatus() + suite.Require().False(ended, + "handed over after %s at height %d, about 10000 blocks behind a source still delivering", elapsed, applied) +} + +// TestWaitForSyncKeepsSyncingWhenTheLastBlockLooksNewerThanTheLocalClock checks +// that a node whose clock runs behind the validators', so that a block it applied +// long ago carries a time ahead of its clock, does not take that block for one at +// the tip and give up on a slow peer thousands of blocks ahead. +func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingWhenTheLastBlockLooksNewerThanTheLocalClock() { + const ( + height = int64(100) + slow = types.NodeID("slow peer") + ) + h := suite.newWaitForSyncHarness(height, newPeerData(slow, 1, height+10000)) + defer h.cancel() + h.sync.mtx.Lock() + h.sync.lastBlockTime = h.clock.Now().Add(30 * time.Minute) + h.sync.mtx.Unlock() + h.fail(slow, height, client.ErrPeerNotResponded) + + elapsed, _, ended := h.run(3*unservedClaimGrace, nil) + suite.Require().False(ended, + "handed over after %s, 10000 blocks behind, on a block time ahead of the local clock", elapsed) +} + +// TestIsRecent checks which last block times put the node at the tip: old ones +// do not, and neither do ones further ahead of the local clock than the +// synchrony bounds allow a clock in step with the validators' to see. +func TestIsRecent(t *testing.T) { + sync := NewSynchronizer(1, nil, nil) + now := time.Unix(1_700_000_000, 0) + ahead := sync.synchrony.Precision + sync.synchrony.MessageDelay + for _, tc := range []struct { + name string + age time.Duration + want bool + }{ + {name: "just applied", age: 0, want: true}, + {name: "as old as tipRecency", age: tipRecency, want: true}, + {name: "older than tipRecency", age: tipRecency + time.Second}, + {name: "ahead within the synchrony bounds", age: -ahead, want: true}, + {name: "ahead beyond the synchrony bounds", age: -ahead - time.Second}, + } { + t.Run(tc.name, func(t *testing.T) { + require.Equal(t, tc.want, sync.isRecent(now.Add(-tc.age), now)) + }) + } +} + +// TestWaitForSyncUnservedHandoverTargetsCountedClaims checks that handing over past +// claims that went unserved does not pass them on as the consensus catch-up +// target, which would hold back the node's own proposals for that long again. +func (suite *SynchronizerTestSuite) TestWaitForSyncUnservedHandoverTargetsCountedClaims() { + const ( + height = int64(100) + tipPeer = types.NodeID("tip peer") + unserving = types.NodeID("unserving peer") + ) + h := suite.newWaitForSyncHarness(height, + newPeerData(tipPeer, 1, height-1), + newPeerData(unserving, 1, maxPlausiblePeerHeight)) + defer h.cancel() + h.fail(unserving, height, client.ErrPeerNotResponded) + + _, caughtUp, ended := h.run(unservedClaimGrace+time.Minute, h.tipBlocks(tipPeer, tipBlockInterval)) + suite.Require().True(ended) + suite.Require().False(caughtUp) + applied, _ := h.sync.GetStatus() + suite.Require().Equal(applied-1, h.target, "the unserved claim must not become the catch-up target") +} + // backlogChainLen is how many heights the backlog tests have blocks for. It is // wider than the pending window so that a run which ignores the window keeps // going well past it instead of stopping for want of a block to serve. @@ -2320,9 +3014,11 @@ func (suite *SynchronizerTestSuite) TestSmallBlocksAfterSlowApplyKeepPeersUsable } fakeClock.Advance(syncTimeout + time.Second) - height, stalled, servable, tip, _, servableTip := h.pool.stallSnapshot() - suite.Equal(keepSyncing, stallVerdictFor(servable, servableTip-height, stalled), - "healthy peers must not trigger consensus handoff at height %d below tip %d", height, tip) + st := h.pool.stallSnapshot() + height := st.height + suite.Equal(keepSyncing, + stallVerdictFor(st.servable, st.maxServableHeight-height, st.stalledFor, st.maxCredibleHeight >= height, 0), + "healthy peers must not trigger consensus handoff at height %d below tip %d", height, st.maxPeerHeight) _, found := h.pool.peerStore.FindPeer(height) suite.True(found, "healthy peers must accept the next block request after the batch drains") if found { From 3a77810b5703d7187324c41d5e71459e01a89e45 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 11:34:30 +0000 Subject: [PATCH 02/36] fix(blocksync): judge the tip by block time alone and require backed claims Node IDs cost nothing, so a per-peer record of unserved heights cannot keep a far height claim from holding a node in block sync: a claim made under a new ID every minute, or by a peer that supplies each block as the network commits it, never stopped counting. Base the handover on what peers cannot fake instead: - At the tip: the node is at the tip while its last applied block is no older than the chain took to commit its last maxCatchupGap blocks (at least 70 s, at most 5 min) and no further ahead of the local clock than the chain's current synchrony precision. After two minutes there it hands over whatever peers claim; the catch-up target leaves out claims more than maxCatchupGap above it. - 10-minute backstop: far behind, only a peer that supplied a block the node applied, and has not failed a request since, keeps the node syncing. A new peer has no such credit. The block time comes from the applier state rather than the block response. The unserved-height map and its expiry are removed. Co-Authored-By: Claude Opus 5.5 --- docs/tenderdash-core/block-sync/reactor.md | 89 ++-- internal/blocksync/peer_service_test.go | 1 - internal/blocksync/peer_store.go | 74 +-- internal/blocksync/peer_store_test.go | 66 +-- internal/blocksync/reactor.go | 19 +- internal/blocksync/synchronizer.go | 170 ++++--- internal/blocksync/synchronizer_test.go | 566 +++++++++++---------- 7 files changed, 469 insertions(+), 516 deletions(-) diff --git a/docs/tenderdash-core/block-sync/reactor.md b/docs/tenderdash-core/block-sync/reactor.md index 5774ac1a2..2885c7b15 100644 --- a/docs/tenderdash-core/block-sync/reactor.md +++ b/docs/tenderdash-core/block-sync/reactor.md @@ -55,59 +55,54 @@ would meet the scheduling allowance. Block sync hands over to consensus when it reaches the highest height any peer claims; when no peer advertises the next height and no block has been applied -for 60 seconds; or when no block has been applied for 10 minutes while the node -is within 10 blocks of the highest claim that covers the next height. - -Peer height claims are unverified, so a claim stops counting toward that -decision once its peer fails a request: a timeout or other failure, a -`NoBlockResponse`, or a block that fails to apply. The claim is discounted from -the failed height up until a block that peer supplied at or above it is -applied. Unlike service history, this record survives dropping and re-adding the -peer; it is forgotten 10 minutes after the peer's last failure. +for 60 seconds; when the node has been at the tip of the chain for two minutes; +or when no block has been applied for 10 minutes and either the node is within +10 blocks of the highest claim that covers the next height, or no peer that +advertises the next height has backed its claim. Whether the node is at the tip is judged from the time of its last applied block, which the validators commit to and a peer cannot change: a node behind -the network applies old blocks. Once that time has stayed no more than five -minutes behind the local clock, and no further ahead of it than the synchrony -bounds allow, for two minutes, and no peer whose claim still counts advertises -the next height, block sync hands over even though blocks keep arriving. Five -minutes covers the three-minute empty-block interval Dash -Platform runs with, the default synchrony bounds and a margin for extra rounds -and clock skew. If the last block is older than that, because the chain has -stopped producing blocks or does not produce empty ones, block sync hands over -once no block has been applied for 10 minutes and no counted claim covers the -next height. - -A peer that claims a far height and does not serve it therefore delays the -handover until two minutes after the node applies a block less than five -minutes old, or until its first request to that peer times out 60 seconds -later, whichever comes last. On a chain whose last block is older than five -minutes it takes 10 minutes. On this handover the consensus catch-up target leaves out discounted -claims, so they do not delay the node's own proposals. A claim whose range -starts above the node's height is never requested and stays in the target, so -consensus may withhold proposals for up to 10 minutes while it is above the node. +the network applies old blocks. The node is at the tip while that block is no +older than the chain took to commit its last 10 blocks, or than 70 seconds (a +status round and a request timeout) where that is longer, and in any case no +older than five minutes. Five minutes covers the three-minute empty-block +interval Dash Platform runs with and a margin for extra rounds and clock skew. A +block time further ahead of the local clock than the synchrony precision does +not count, since the clock is then behind the validators' and cannot tell how +old the block is. Peer height claims play no part in this rule: a claim costs +nothing, and neither does the node ID it comes with. On this handover the +consensus catch-up target leaves out claims more than 10 blocks above the node, +so they do not delay its own proposals. + +A peer backs its claim by supplying a block the node applies, and loses that +credit when it fails a request (a timeout or other failure, or a +`NoBlockResponse`) or is dropped; a newly connected peer has none. Within a +10-minute stall every peer asked for the next block has failed it, so only a +peer not asked since it last supplied a block can hold the node in block sync. + +A peer that claims a far height therefore cannot keep a node at the tip in block +sync for more than two minutes after its last block becomes recent, however it +answers requests and whatever node IDs it uses. On a chain whose last block is +not recent, for instance one that has stopped producing blocks, the handover +takes 10 minutes, or until each peer that backed its claim earlier has been +asked for the next block and failed it, whichever is later. Some cases remain open: -- A peer that answers every block request within the 60-second request timeout - never fails, so its claim is never discounted and keeps the node in block sync - while it claims the next height. -- A node whose clock runs more than about two minutes ahead of the validators' - sees even the newest blocks as old, so on a live chain it does not hand over - past an unserved claim. -- A block time further ahead of the local clock than the synchrony bounds - (precision plus message delay) allow does not count as recent, since the - node's clock is then behind the validators' and cannot tell how old the - block is. Such a node, too, hands over past an unserved claim only once no - block has been applied for 10 minutes. -- Within five minutes of chain time of the tip, if the only peers that advertise - the next height have failed a request and serve nothing at or above it for two - minutes, block sync hands over even though those peers may be slow rather - than wrong. On a chain that commits many blocks a minute this can leave the - node more blocks behind than consensus catches up quickly. -- A node any distance behind whose only peers that advertise the next height - fail every request for 10 minutes hands over, as it would if none advertised - it. +- A node whose clock runs ahead of the validators' by more than its at-tip + window leaves (about two minutes on a chain committing a block every three + minutes, less on a faster one), or behind them by more than the synchrony + precision, does not see itself at the tip; while blocks keep arriving, a far + claim keeps it in block sync. +- On a chain that commits more than 10 blocks in 70 seconds, a peer that answers + each request just before it times out holds the node more than 70 seconds + behind, and so in block sync, for as long as it does so. Such a node can also + hand over at the tip up to 70 seconds of blocks behind, which consensus + catch-up then closes. +- A node any distance behind whose peers that advertise the next height fail + every request for 10 minutes hands over, as it would if none advertised it. +- After a handover on a stall, the catch-up target includes every claim, so + consensus may withhold the node's own proposals for up to 10 minutes. The historical Blocksync reactor is organised as a set of concurrent tasks: diff --git a/internal/blocksync/peer_service_test.go b/internal/blocksync/peer_service_test.go index 3c044f301..9834d91cf 100644 --- a/internal/blocksync/peer_service_test.go +++ b/internal/blocksync/peer_service_test.go @@ -219,7 +219,6 @@ func TestServiceProbationRemainsServableBelowTip(t *testing.T) { require.True(t, st.servable) require.True(t, st.hasPeers) require.EqualValues(t, 1000, st.maxServableHeight) - require.EqualValues(t, 1000, st.maxCredibleHeight, "cooling is not a failure to serve") } func TestServiceSavedCreditCannotHideNewSlowService(t *testing.T) { diff --git a/internal/blocksync/peer_store.go b/internal/blocksync/peer_store.go index 2787006a3..20a404062 100644 --- a/internal/blocksync/peer_store.go +++ b/internal/blocksync/peer_store.go @@ -1,11 +1,7 @@ package blocksync import ( - "maps" - "time" - "github.com/jonboulle/clockwork" - sync "github.com/sasha-s/go-deadlock" "golang.org/x/exp/constraints" "github.com/dashpay/tenderdash/libs/store" @@ -21,18 +17,6 @@ type ( // a concurrent write is lost. Nothing here substitutes another store. store *store.InMemStore[types.NodeID, PeerData] clock clockwork.Clock - - unservedMtx sync.Mutex - // unserved holds the peers whose claims are discounted. It is kept - // outside PeerData so that neither dropping the peer nor its next status - // erases it. - unserved map[types.NodeID]unservedRecord - } - // unservedRecord discounts a peer's claim from the lowest height it failed to serve. - unservedRecord struct { - from int64 - // failedAt is when the peer last failed, for pruning. - failedAt time.Time } // PeerData uses to keep peer related data like base height and the current height etc PeerData struct { @@ -50,15 +34,17 @@ type ( // status on. Requests go out by node ID, so a reservation's copy of it is // the only generation known for a request that never gets an answer. connID uint64 + // supplied says a block this peer sent has been applied since it last + // failed a request. A new peer has supplied nothing. + supplied bool } ) // NewInMemPeerStore creates a new in-memory peer store func NewInMemPeerStore(peers ...PeerData) *InMemPeerStore { mem := &InMemPeerStore{ - store: store.NewInMemStore[types.NodeID, PeerData](), - clock: clockwork.NewRealClock(), - unserved: map[types.NodeID]unservedRecord{}, + store: store.NewInMemStore[types.NodeID, PeerData](), + clock: clockwork.NewRealClock(), } for _, peer := range peers { mem.Put(peer.peerID, peer) @@ -195,6 +181,7 @@ func (p *InMemPeerStore) AddFailure(peerID types.NodeID, maxFailures int32) bool p.store.Update(peerID, func(_ types.NodeID, peer *PeerData) { peer.addPending(-1) peer.numFailures++ + peer.supplied = false tooMany = peer.numFailures >= maxFailures }) return tooMany @@ -205,57 +192,10 @@ func (p *InMemPeerStore) RecordMissingBlock(peerID types.NodeID, height int64) { p.store.Update(peerID, func(_ types.NodeID, peer *PeerData) { peer.addPending(-1) peer.missingThrough = max(peer.missingThrough, height) + peer.supplied = false }) } -// RecordUnserved notes that the peer failed to serve a height it was asked for. -func (p *InMemPeerStore) RecordUnserved(peerID types.NodeID, height int64) { - p.unservedMtx.Lock() - defer p.unservedMtx.Unlock() - now := p.pruneUnservedLocked() - record, ok := p.unserved[peerID] - if !ok || height < record.from { - record.from = height - } - record.failedAt = now - p.unserved[peerID] = record -} - -// RecordServed clears the peer's record once a block it supplied at or above -// the failed height has been applied. Delivery alone is not enough: anyone can -// send a block that fails verification. -func (p *InMemPeerStore) RecordServed(peerID types.NodeID, height int64) { - p.unservedMtx.Lock() - defer p.unservedMtx.Unlock() - if record, ok := p.unserved[peerID]; ok && height >= record.from { - delete(p.unserved, peerID) - } -} - -// unservedFrom returns the lowest unserved height of each of peers that has a record. -func (p *InMemPeerStore) unservedFrom(peers []PeerData) map[types.NodeID]int64 { - p.unservedMtx.Lock() - defer p.unservedMtx.Unlock() - p.pruneUnservedLocked() - from := make(map[types.NodeID]int64) - for _, peer := range peers { - if record, ok := p.unserved[peer.peerID]; ok { - from[peer.peerID] = record.from - } - } - return from -} - -// pruneUnservedLocked drops records of peers that have not failed for -// unservedRecordTTL and returns the time it pruned at. -func (p *InMemPeerStore) pruneUnservedLocked() time.Time { - now := p.clock.Now() - maps.DeleteFunc(p.unserved, func(_ types.NodeID, record unservedRecord) bool { - return now.Sub(record.failedAt) > unservedRecordTTL - }) - return now -} - // All returns all stored peers in the store func (p *InMemPeerStore) All() []PeerData { return p.store.All() diff --git a/internal/blocksync/peer_store_test.go b/internal/blocksync/peer_store_test.go index 416472210..2467044a9 100644 --- a/internal/blocksync/peer_store_test.go +++ b/internal/blocksync/peer_store_test.go @@ -4,9 +4,7 @@ import ( "fmt" "sync" "testing" - "time" - "github.com/jonboulle/clockwork" "github.com/stretchr/testify/require" "github.com/dashpay/tenderdash/types" @@ -318,48 +316,26 @@ func TestAddFailureClearsPending(t *testing.T) { require.EqualValues(t, 2, peer.numFailures) } -// TestUnservedRecord checks what discounts a peer's claim and what makes it count -// again, and that records of peers that stop failing are forgotten. -func TestUnservedRecord(t *testing.T) { - clock := clockwork.NewFakeClock() - peers := NewInMemPeerStore() - peers.clock = clock - from := func(peerID types.NodeID) (int64, bool) { - height, ok := peers.unservedFrom([]PeerData{{peerID: peerID}})[peerID] - return height, ok +// TestFailureEndsSuppliedCredit checks that a failed request or an answer that +// the block is missing ends the credit a peer earned by supplying blocks, and +// that a fresh status for the peer does not. +func TestFailureEndsSuppliedCredit(t *testing.T) { + peerID := types.NodeID("peer1") + for name, fail := range map[string]func(*InMemPeerStore){ + "failed request": func(p *InMemPeerStore) { p.AddFailure(peerID, 10) }, + "missing block": func(p *InMemPeerStore) { p.RecordMissingBlock(peerID, 5) }, + } { + t.Run(name, func(t *testing.T) { + supplier := newPeerData(peerID, 1, 100) + supplier.supplied = true + inmem := NewInMemPeerStore(supplier) + inmem.Upsert(newPeerData(peerID, 1, 200)) + peer, _ := inmem.Get(peerID) + require.True(t, peer.supplied, "a status update keeps the credit") + + fail(inmem) + peer, _ = inmem.Get(peerID) + require.False(t, peer.supplied) + }) } - - peers.RecordUnserved("peer", 100) - peers.RecordUnserved("peer", 90) - height, ok := from("peer") - require.True(t, ok) - require.EqualValues(t, 90, height, "the lowest failed height is the one discounted from") - - peers.RecordServed("peer", 89) - _, ok = from("peer") - require.True(t, ok, "a block below the failed height does not restore the claim") - peers.RecordServed("peer", 90) - _, ok = from("peer") - require.False(t, ok, "an applied block at the failed height restores the claim") - - peers.RecordUnserved("gone", 5) - clock.Advance(unservedRecordTTL + time.Second) - peers.RecordUnserved("fresh", 7) - _, ok = from("gone") - require.False(t, ok, "a record not refreshed within unservedRecordTTL is dropped") - _, ok = from("fresh") - require.True(t, ok) -} - -// TestUnservedRecordExpiresWithoutAnotherFailure checks the documented expiry: a -// record is forgotten unservedRecordTTL after the peer's last failure, whether or -// not some other peer fails in the meantime. -func TestUnservedRecordExpiresWithoutAnotherFailure(t *testing.T) { - clock := clockwork.NewFakeClock() - peers := NewInMemPeerStore() - peers.clock = clock - peers.RecordUnserved("peer", 5) - clock.Advance(unservedRecordTTL + time.Second) - _, ok := peers.unservedFrom([]PeerData{{peerID: "peer"}})["peer"] - require.False(t, ok, "the record outlived unservedRecordTTL") } diff --git a/internal/blocksync/reactor.go b/internal/blocksync/reactor.go index 44e80726e..8f116aec5 100644 --- a/internal/blocksync/reactor.go +++ b/internal/blocksync/reactor.go @@ -48,21 +48,18 @@ const ( // (dashpay/tenderdash#1413). maxCatchupGap int64 = 10 - // tipRecency is how recent the last applied block must be for the node to + // tipRecency is the oldest the last applied block may be for the node to // count as at the tip. It covers the three-minute empty-block interval Dash - // Platform runs with, the default synchrony bounds (12 s message delay, 505 ms - // precision) and a margin for extra rounds and clock skew. + // Platform runs with and a margin for extra rounds and clock skew. tipRecency = 5 * time.Minute - // unservedClaimGrace is how long the node must have been at the tip before - // claims that went unserved stop keeping it in block sync. It leaves a slow - // peer that timed out a cooldown and a full probe to serve again. - unservedClaimGrace = 2 * syncTimeout + // tipLag is how far behind the network block sync runs at the tip: a status + // round to learn of a new block and a request timeout to fetch it. + tipLag = statusUpdateIntervalSeconds*time.Second + syncTimeout - // unservedRecordTTL is how long a peer's claim stays discounted after its - // last failure. A peer still asked for blocks it cannot serve keeps failing - // well within it, so this only forgets peers that went away. - unservedRecordTTL = maxSyncStall + // tipGrace is how long the node must stay at the tip before higher height + // claims stop keeping it in block sync. + tipGrace = 2 * syncTimeout ) type ReactorOption func(*Reactor) diff --git a/internal/blocksync/synchronizer.go b/internal/blocksync/synchronizer.go index 81840aea5..93069907d 100644 --- a/internal/blocksync/synchronizer.go +++ b/internal/blocksync/synchronizer.go @@ -131,11 +131,9 @@ type ( logger log.Logger lastAdvance time.Time - // lastBlockTime is the time of the last applied block, guarded by mtx. - lastBlockTime time.Time - // synchrony bounds how far ahead of the local clock a block time can be - // for a clock in step with the validators'. Set before WaitForSync runs. - synchrony types.SynchronyParams + // blockTimes holds the times of the last maxCatchupGap+1 applied blocks, + // oldest first. Guarded by mtx. + blockTimes []time.Time mtx sync.RWMutex @@ -211,7 +209,6 @@ func NewSynchronizer(start int64, client client.BlockClient, blockExec *blockApp monitorInterval: defaultSyncRateIntervalBlocks, workerPool: workerpool.New(poolWorkerSize, workerpool.WithLogger(logger)), pendingToApply: map[int64]BlockResponse{}, - synchrony: types.DefaultSynchronyParams(), } for _, opt := range opts { opt(bp) @@ -251,9 +248,7 @@ func (s *Synchronizer) OnStart(ctx context.Context) error { s.lastAdvance = s.clock.Now() s.lastMonitorUpdate = s.lastAdvance if s.applier != nil { - state := s.applier.State() - s.lastBlockTime = state.LastBlockTime - s.synchrony = state.ConsensusParams.Synchrony.SynchronyParamsOrDefaults() + s.blockTimes = []time.Time{s.applier.State().LastBlockTime} } s.ctx, s.cancel = context.WithCancel(ctx) applicationCtx, ok := ctx.Value(applicationContextKey{}).(context.Context) @@ -335,7 +330,6 @@ func (s *Synchronizer) consumeJobResult(ctx, applyCtx context.Context) error { if !errors.As(res.Err, &bfErr) { return nil } - s.peerStore.RecordUnserved(bfErr.peerID, bfErr.height) var missing *client.ErrBlockNotFound if errors.As(bfErr.err, &missing) && missing.Height == bfErr.height { s.peerStore.RecordMissingBlock(bfErr.peerID, bfErr.height) @@ -395,7 +389,6 @@ func (s *Synchronizer) consumeJobResult(ctx, applyCtx context.Context) error { "height", failed.Block.Height, "peer", failed.PeerID, "error", err.Error()) - s.peerStore.RecordUnserved(failed.PeerID, failed.Block.Height) s.RemovePeer(failed.PeerID) _ = s.client.Send(ctx, p2p.PeerError{NodeID: failed.PeerID, Err: err, ConnID: failed.ConnID}) } @@ -455,17 +448,16 @@ func (s *Synchronizer) IsCaughtUp() bool { // peer holds the block we are waiting for there is something to retry, so keep // going and say so loudly. Give up on the stall only once no peer can serve that // block, or once the stall has outlasted maxSyncStall within maxCatchupGap of -// the tip, so that a wedged synchronizer can still hand over rather than -// blocking forever without handing over a node consensus cannot catch up. +// the tip or with no peer holding the block that has backed its claim, so that +// a wedged synchronizer can still hand over rather than blocking forever +// without handing over a node consensus cannot catch up. // -// A height claim costs a peer nothing, so claims alone cannot keep the node in -// block sync once the node is at the tip. The time of the last applied block is -// what tells it so: the validators commit to it, and a node far behind applies -// old blocks. Once that time has stayed within tipRecency of the local clock for -// unservedClaimGrace, and no peer that has not failed to serve it holds the next -// block, the node hands over even though blocks arriving meanwhile keep it from -// stalling. A chain that stops producing blocks ages that time out, and then -// only a stall of maxSyncStall with no such peer ends block sync. +// A height claim costs a peer nothing, and a node ID costs nothing either, so +// what a peer claims cannot keep the node in block sync once it is at the tip. +// The time of the last applied block tells it so: the validators commit to it, +// and a node far behind applies old blocks. Once the node has stayed at the tip +// by that time for tipGrace, it hands over whatever peers claim, even though +// blocks arriving meanwhile keep it from stalling. func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHeight int64) { ticker := s.clock.NewTicker(switchToConsensusIntervalSeconds * time.Second) defer ticker.Stop() @@ -482,24 +474,21 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe return true, targetHeight } now := s.clock.Now() - if !s.isRecent(st.lastBlockTime, now) { + if !atTip(st.lastBlockTime, st.tipWindow, s.precision(), now) { atTipSince = now } - atTipFor := now.Sub(atTipSince) verdict := stallVerdictFor(st.servable, st.maxServableHeight-st.height, st.stalledFor, - st.maxCredibleHeight >= st.height, atTipFor) + st.suppliedServable, now.Sub(atTipSince)) switch verdict { - case stopUnservedClaims: - s.logger.Error( - "higher peer claims went unserved, handing over to consensus", + case stopAtTip: + s.logger.Warn( + "at the tip by block time, handing over to consensus below the highest peer claim", "height", st.height, "max_peer_height", st.maxPeerHeight, - "max_counted_height", st.maxCountedHeight, "last_block_time", st.lastBlockTime, - "stalled_for", st.stalledFor, ) - // Claims that went unserved must not hold back our own proposals. - return false, st.maxCountedHeight + // Claims out of reach of the tip must not hold back our own proposals. + return false, st.maxNearHeight case stopNothingToFetch: if st.maxPeerHeight > st.height { // Peers claim to be ahead yet none of them holds the block we @@ -544,12 +533,22 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe } } -// isRecent reports whether a block with blockTime puts the node at the tip. A -// time further ahead of the local clock than the synchrony bounds allow means -// the clock is behind the validators', so how old the block is cannot be told. -func (s *Synchronizer) isRecent(blockTime, now time.Time) bool { +// atTip reports whether a last applied block with blockTime puts the node at +// the tip: it is at most window old, and no further ahead of the local clock +// than precision, the bound validators hold block times to. Further ahead, the +// local clock is behind the validators' and cannot tell how old the block is. +func atTip(blockTime time.Time, window, precision time.Duration, now time.Time) bool { age := now.Sub(blockTime) - return age <= tipRecency && -age <= s.synchrony.Precision+s.synchrony.MessageDelay + return age <= window && age >= -precision +} + +// precision is the synchrony precision of the chain's current parameters. +func (s *Synchronizer) precision() time.Duration { + var params types.SynchronyParams + if s.applier != nil { + params = s.applier.State().ConsensusParams.Synchrony + } + return params.SynchronyParamsOrDefaults().Precision } // stallState is one observation of what the stall verdict is formed from. @@ -563,13 +562,16 @@ type stallState struct { servable bool maxServableHeight int64 maxPeerHeight int64 - // maxCredibleHeight is the highest servable claim still counted, capped below - // the height its peer failed; 0 when none covers height. - maxCredibleHeight int64 - // maxCountedHeight is the highest claim of any peer, capped the same way. - maxCountedHeight int64 - // lastBlockTime is the time of the last applied block. + // suppliedServable says a peer advertising height has supplied a block we + // applied since it last failed a request. + suppliedServable bool + // maxNearHeight is the highest claim within maxCatchupGap of the last + // applied block. + maxNearHeight int64 + // lastBlockTime is the time of the last applied block; within tipWindow of + // the local clock the node is at the tip. lastBlockTime time.Time + tipWindow time.Duration } // stallSnapshot reads everything the stall verdict is formed from as one @@ -582,14 +584,17 @@ type stallState struct { // - a peer whose blocks start above us has nothing we can use however high it // claims to be. Only servable peers determine the stall gap; the overall maximum // remains the handover target so pruning cannot erase evidence of a higher tip. -// A claim is counted only below the lowest height its peer failed to serve. +// +// The node is at the tip when its last block is no older than the chain took +// to commit the last maxCatchupGap blocks, or than tipLag where that is longer, +// and in any case no older than tipRecency. // // They are read under one lock because advance() stamps the height, the advance -// time, the block time and the supplier's record together under that same lock. A block applied -// concurrently therefore lands either wholly inside the snapshot or wholly -// outside it, and the verdict can never pair a height with a staleness or a -// servability measured against a different one. Read separately, it could: a -// height read before an advance, checked for servability after one, reports +// time, the block time and the supplier's credit together under that same lock. +// A block applied concurrently therefore lands either wholly inside the snapshot +// or wholly outside it, and the verdict can never pair a height with a staleness +// or a servability measured against a different one. Read separately, it could: +// a height read before an advance, checked for servability after one, reports // nothing to fetch while the height we had by then moved on to is served. Ending // block sync is a one-way door, so a stop assembled from two inconsistent // readings leaves the node in consensus catch-up it cannot leave. @@ -597,27 +602,28 @@ func (s *Synchronizer) stallSnapshot() stallState { s.mtx.RLock() defer s.mtx.RUnlock() peers := s.peerStore.All() - unserved := s.peerStore.unservedFrom(peers) st := stallState{ - height: s.height, - stalledFor: s.clock.Since(s.lastAdvance), - hasPeers: len(peers) > 0, - lastBlockTime: s.lastBlockTime, + height: s.height, + stalledFor: s.clock.Since(s.lastAdvance), + hasPeers: len(peers) > 0, + tipWindow: tipRecency, + } + if last := len(s.blockTimes) - 1; last >= 0 { + st.lastBlockTime = s.blockTimes[last] + if last == int(maxCatchupGap) { + st.tipWindow = min(tipRecency, max(tipLag, s.blockTimes[last].Sub(s.blockTimes[0]))) + } } hasHeight := heightBetweenPeerHeightRange(s.height) for _, peer := range peers { st.maxPeerHeight = max(st.maxPeerHeight, peer.height) - counted := peer.height - if from, failed := unserved[peer.peerID]; failed { - counted = min(counted, from-1) + if peer.height < s.height+maxCatchupGap { + st.maxNearHeight = max(st.maxNearHeight, peer.height) } - st.maxCountedHeight = max(st.maxCountedHeight, counted) if hasHeight(peer.peerID, peer) { st.servable = true st.maxServableHeight = max(st.maxServableHeight, peer.height) - if counted >= s.height { - st.maxCredibleHeight = max(st.maxCredibleHeight, counted) - } + st.suppliedServable = st.suppliedServable || peer.supplied } } return st @@ -633,8 +639,8 @@ const ( stopNothingToFetch // stopStalledTooLong means a peer still holds it but we have to hand over anyway stopStalledTooLong - // stopUnservedClaims means the claims above us went unserved for too long - stopUnservedClaims + // stopAtTip means the node has stayed at the tip for tipGrace + stopAtTip ) // stallVerdictFor decides what to do when block sync has made no progress for @@ -655,25 +661,26 @@ const ( // the backstop buys nothing and costs what dashpay/tenderdash#1413 describes, so // block sync keeps retrying and says so every interval. // -// credibleServable tells such a peer apart once it has failed a request: it says -// whether a peer that has not failed to serve the block holds it. atTipFor is how -// long the last applied block has been recent. Unlike stalledFor it is not reset -// by applying blocks, so a node following its peers at the tip is not kept in -// block sync by a claim nobody serves. A node whose last block is not recent -// relies on a stall of maxSyncStall with no such peer holding the block instead. +// Advertising costs nothing, so far behind the backstop counts only a peer that +// has backed its claim: suppliedServable says a peer holding the block supplied +// one we applied since it last failed a request. A stall of maxSyncStall means +// every peer asked for the block meanwhile failed, so only one never asked keeps +// that credit, and a new peer has none. +// +// atTipFor is how long the node has been at the tip by the time of its last +// block. Unlike stalledFor it is not reset by applying blocks, so a node +// following the network is not kept in block sync by a claim nobody can reach. func stallVerdictFor( - servable bool, behindBy int64, stalledFor time.Duration, credibleServable bool, atTipFor time.Duration, + servable bool, behindBy int64, stalledFor time.Duration, suppliedServable bool, atTipFor time.Duration, ) stallVerdict { switch { - case !credibleServable && atTipFor > unservedClaimGrace: - return stopUnservedClaims + case atTipFor > tipGrace: + return stopAtTip case stalledFor <= syncTimeout: return keepSyncing case !servable: return stopNothingToFetch - case !credibleServable && stalledFor > maxSyncStall: - return stopUnservedClaims - case stalledFor > maxSyncStall && behindBy <= maxCatchupGap: + case stalledFor > maxSyncStall && (behindBy <= maxCatchupGap || !suppliedServable): return stopStalledTooLong default: return keepSyncing @@ -739,7 +746,7 @@ func (s *Synchronizer) applyBlock(ctx context.Context) (BlockResponse, error) { if err != nil { return resp, fmt.Errorf("cannot apply block: %w", err) } - s.advance() + s.advance(s.applier.State().LastBlockTime) s.updateMonitor() } } @@ -751,14 +758,17 @@ func (s *Synchronizer) getPendingResponse() (BlockResponse, bool) { return resp, ok } -func (s *Synchronizer) advance() { +// advance moves past the block just applied, which carries blockTime, and +// credits the peer that supplied it. +func (s *Synchronizer) advance(blockTime time.Time) { s.mtx.Lock() defer s.mtx.Unlock() if resp, ok := s.pendingToApply[s.height]; ok { - s.peerStore.RecordServed(resp.PeerID, s.height) - if resp.Block != nil { - s.lastBlockTime = resp.Block.Time - } + s.peerStore.Update(resp.PeerID, func(_ types.NodeID, peer *PeerData) { peer.supplied = true }) + } + s.blockTimes = append(s.blockTimes, blockTime) + if len(s.blockTimes) > int(maxCatchupGap)+1 { + s.blockTimes = s.blockTimes[1:] } delete(s.pendingToApply, s.height) s.height++ diff --git a/internal/blocksync/synchronizer_test.go b/internal/blocksync/synchronizer_test.go index 3d4c91ba4..f8697b88e 100644 --- a/internal/blocksync/synchronizer_test.go +++ b/internal/blocksync/synchronizer_test.go @@ -982,9 +982,6 @@ func (suite *SynchronizerTestSuite) TestApplyFailurePunishesSupplyingPeer() { suite.Require().NotContains(pool.pendingToApply, poisonH1.Block.Height) _, found := pool.peerStore.Get(poisonPeerID) suite.Require().False(found) - suite.Require().Equal(poisonH1.Block.Height, - pool.peerStore.unservedFrom([]PeerData{{peerID: poisonPeerID}})[poisonPeerID], - "a block that fails to apply leaves the claim to that height unserved") // The honest peer keeps both its response and its place in the peer store. suite.Require().Contains(pool.pendingToApply, honestH2.Block.Height) @@ -1329,18 +1326,25 @@ func (suite *SynchronizerTestSuite) TestNoBlockResponseForWrongHeightPreservesRa suite.EqualValues(1, peer.numFailures) } -// TestAdvanceCreditsTheSupplier checks that applying a block clears the record of -// the peer that supplied it and stamps its time in the same step that moves the -// height, so a stall snapshot never pairs the new height with old values. +// TestAdvanceCreditsTheSupplier checks that applying a block credits the peer +// that supplied it and stamps its time in the same step that moves the height, +// so a stall snapshot never pairs the new height with old values. func (suite *SynchronizerTestSuite) TestAdvanceCreditsTheSupplier() { const peerID = types.NodeID("supplier") pool := NewSynchronizer(100, suite.client, nil) - pool.peerStore.RecordUnserved(peerID, 100) + pool.AddPeer(newPeerData(peerID, 1, 200)) + pool.AddPeer(newPeerData("bystander", 1, 200)) blockTime := time.Unix(1_700_000_000, 0) - pool.pendingToApply[100] = BlockResponse{PeerID: peerID, Block: &types.Block{Header: types.Header{Time: blockTime}}} - pool.advance() - suite.NotContains(pool.peerStore.unserved, peerID) - suite.Equal(blockTime, pool.lastBlockTime) + pool.pendingToApply[100] = BlockResponse{PeerID: peerID} + pool.advance(blockTime) + supplier, _ := pool.peerStore.Get(peerID) + suite.True(supplier.supplied) + bystander, _ := pool.peerStore.Get("bystander") + suite.False(bystander.supplied, "only the peer whose block was applied is credited") + st := pool.stallSnapshot() + suite.EqualValues(101, st.height) + suite.Equal(blockTime, st.lastBlockTime) + suite.True(st.suppliedServable) } // TestStatusRefreshUpdatesAdvertisedRange checks that a status response still does @@ -1426,7 +1430,7 @@ func TestStallVerdictFor(t *testing.T) { servable bool behindBy int64 stalledFor time.Duration - credibleServable bool + suppliedServable bool atTipFor time.Duration want stallVerdict }{ @@ -1434,7 +1438,7 @@ func TestStallVerdictFor(t *testing.T) { name: "progressing with the block available", servable: true, stalledFor: syncTimeout / 2, - credibleServable: true, + suppliedServable: true, want: keepSyncing, }, { @@ -1453,14 +1457,14 @@ func TestStallVerdictFor(t *testing.T) { name: "stalled but a peer still holds the block", servable: true, stalledFor: syncTimeout + time.Second, - credibleServable: true, + suppliedServable: true, want: keepSyncing, }, { name: "stalled just short of the wedge limit", servable: true, stalledFor: maxSyncStall, - credibleServable: true, + suppliedServable: true, want: keepSyncing, }, { @@ -1468,7 +1472,7 @@ func TestStallVerdictFor(t *testing.T) { servable: true, behindBy: maxCatchupGap, stalledFor: maxSyncStall + time.Second, - credibleServable: true, + suppliedServable: true, want: stopStalledTooLong, }, { @@ -1476,7 +1480,7 @@ func TestStallVerdictFor(t *testing.T) { servable: true, behindBy: maxCatchupGap + 1, stalledFor: maxSyncStall + time.Second, - credibleServable: true, + suppliedServable: true, want: keepSyncing, }, { @@ -1486,40 +1490,32 @@ func TestStallVerdictFor(t *testing.T) { want: stopNothingToFetch, }, { - name: "only unserved claims stalled past the wedge limit", + name: "far behind a claim no peer has backed, stalled past the wedge limit", servable: true, behindBy: maxCatchupGap + 1, stalledFor: maxSyncStall + time.Second, - want: stopUnservedClaims, + want: stopStalledTooLong, }, { - name: "only unserved claims while at the tip", - servable: true, - behindBy: maxCatchupGap + 1, - atTipFor: unservedClaimGrace + time.Second, - want: stopUnservedClaims, + name: "at the tip past the grace, behind a backed claim", + servable: true, + behindBy: maxCatchupGap + 1, + suppliedServable: true, + atTipFor: tipGrace + time.Second, + want: stopAtTip, }, { - name: "only unserved claims within the grace", + name: "at the tip within the grace", servable: true, behindBy: maxCatchupGap + 1, - atTipFor: unservedClaimGrace, + atTipFor: tipGrace, want: keepSyncing, }, - { - name: "a served claim outlasts the grace", - servable: true, - behindBy: maxCatchupGap + 1, - stalledFor: syncTimeout + time.Second, - credibleServable: true, - atTipFor: unservedClaimGrace + time.Second, - want: keepSyncing, - }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { require.Equal(t, tc.want, - stallVerdictFor(tc.servable, tc.behindBy, tc.stalledFor, tc.credibleServable, tc.atTipFor)) + stallVerdictFor(tc.servable, tc.behindBy, tc.stalledFor, tc.suppliedServable, tc.atTipFor)) }) } } @@ -1552,7 +1548,7 @@ func (suite *SynchronizerTestSuite) newWaitForSyncHarness(height int64, peers .. // waiter on the fake clock. sync.lastAdvance = clock.Now() // The node starts at the tip: its last block is recent. - sync.lastBlockTime = clock.Now() + sync.blockTimes = []time.Time{clock.Now()} for _, peer := range peers { sync.AddPeer(peer) } @@ -1645,10 +1641,12 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverAfterMaxSyncStall() // backstop must not hand over a node that is thousands of blocks behind. It // would arrive in consensus as a validator at a height the network committed // long ago (dashpay/tenderdash#1413), and consensus catch-up moves about one -// block per gossip cycle, so it would never reach the tip that way either. +// block per gossip cycle, so it would never reach the tip that way either. The +// peer far ahead has backed its claim by supplying blocks the node applied. func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingWhileFarBehind() { const height = int64(100) peer := newPeerData("peer far ahead", 1, height+maxCatchupGap+1) + peer.supplied = true h := suite.newWaitForSyncHarness(height, peer) defer h.cancel() @@ -1662,55 +1660,63 @@ func (suite *SynchronizerTestSuite) TestStallSnapshotServableHeight() { const height = int64(10) busyPeer := newPeerData("busy", 1, 30) busyPeer.numPending = maxPendingRequestsPerPeer + supplied := func(peer PeerData) PeerData { + peer.supplied = true + return peer + } testCases := []struct { name string peers []PeerData - unserved map[types.NodeID]int64 wantMax int64 wantUsable int64 - wantCredible int64 + wantSupplied bool + wantNear int64 }{ {name: "no peers"}, { - name: "available ranges", - peers: []PeerData{newPeerData("archive", 1, 20), newPeerData("recent", 5, 30)}, - wantMax: 30, - wantUsable: 30, - wantCredible: 30, + name: "available ranges", + peers: []PeerData{newPeerData("archive", 1, 20), newPeerData("recent", 5, 30)}, + wantMax: 30, + wantUsable: 30, }, { - name: "pruned range", - peers: []PeerData{newPeerData("archive", 1, 20), newPeerData("pruned", 25, 30)}, - wantMax: 30, - wantUsable: 20, - wantCredible: 20, + name: "pruned range", + peers: []PeerData{newPeerData("archive", 1, 20), newPeerData("pruned", 25, 30)}, + wantMax: 30, + wantUsable: 20, }, { - name: "claim failed above our height", - peers: []PeerData{newPeerData("archive", 1, 20), newPeerData("recent", 5, 30)}, - unserved: map[types.NodeID]int64{"recent": 25}, - wantMax: 30, - wantUsable: 30, - wantCredible: 24, + name: "no overlapping range", + peers: []PeerData{newPeerData("older", 1, 9), newPeerData("recent", 25, 30)}, + wantMax: 30, + wantNear: 9, }, { - name: "claims failed at our height", - peers: []PeerData{newPeerData("archive", 1, 20), newPeerData("recent", 5, 30)}, - unserved: map[types.NodeID]int64{"archive": height, "recent": height}, + name: "busy peer remains servable", + peers: []PeerData{busyPeer}, wantMax: 30, wantUsable: 30, }, { - name: "no overlapping range", - peers: []PeerData{newPeerData("older", 1, 9), newPeerData("recent", 25, 30)}, - wantMax: 30, + name: "claims within reach of the node", + peers: []PeerData{newPeerData("near", 1, height+maxCatchupGap-1), newPeerData("far", 1, 30)}, + wantMax: 30, + wantUsable: 30, + wantNear: height + maxCatchupGap - 1, }, { - name: "busy peer remains servable", - peers: []PeerData{busyPeer}, + name: "a servable peer has supplied a block", + peers: []PeerData{newPeerData("archive", 1, 20), supplied(newPeerData("recent", 5, 30))}, wantMax: 30, wantUsable: 30, - wantCredible: 30, + wantSupplied: true, + }, + { + name: "only a peer that cannot serve the height has supplied a block", + peers: []PeerData{supplied(newPeerData("older", 1, 9)), newPeerData("recent", 5, 30)}, + wantMax: 30, + wantUsable: 30, + wantNear: 9, }, } for _, tc := range testCases { @@ -1720,19 +1726,44 @@ func (suite *SynchronizerTestSuite) TestStallSnapshotServableHeight() { for _, peer := range tc.peers { sync.AddPeer(peer) } - for peerID, from := range tc.unserved { - sync.peerStore.RecordUnserved(peerID, from) - } st := sync.stallSnapshot() suite.Require().Equal(tc.wantMax, st.maxPeerHeight) suite.Require().Equal(tc.wantUsable, st.maxServableHeight) - suite.Require().Equal(tc.wantCredible, st.maxCredibleHeight) suite.Require().Equal(tc.wantUsable > 0, st.servable) + suite.Require().Equal(tc.wantSupplied, st.suppliedServable) + suite.Require().Equal(tc.wantNear, st.maxNearHeight) suite.Require().Equal(len(tc.peers) > 0, st.hasPeers) }) } } +// TestStallSnapshotTipWindow checks how old the last block may be for the node +// to be at the tip: as old as the chain took to commit the last maxCatchupGap +// blocks, but never less than tipLag nor more than tipRecency. +func TestStallSnapshotTipWindow(t *testing.T) { + start := time.Unix(1_700_000_000, 0) + for _, tc := range []struct { + name string + blocks int + interval time.Duration + want time.Duration + }{ + {name: "no block applied", want: tipRecency}, + {name: "fewer blocks than the gap", blocks: int(maxCatchupGap), interval: time.Second, want: tipRecency}, + {name: "a fast chain", blocks: 100, interval: time.Second, want: tipLag}, + {name: "a moderate chain", blocks: 100, interval: 15 * time.Second, want: 150 * time.Second}, + {name: "a slow chain", blocks: 100, interval: 3 * time.Minute, want: tipRecency}, + } { + t.Run(tc.name, func(t *testing.T) { + sync := NewSynchronizer(1, nil, nil) + for i := 0; i < tc.blocks; i++ { + sync.advance(start.Add(time.Duration(i) * tc.interval)) + } + require.Equal(t, tc.want, sync.stallSnapshot().tipWindow) + }) + } +} + // Advancing the height changes servability and its maximum together with the // stall timestamp; the snapshot must not mix values from opposite sides. func (suite *SynchronizerTestSuite) TestStallSnapshotIsOneObservation() { @@ -1753,7 +1784,7 @@ func (suite *SynchronizerTestSuite) TestStallSnapshotIsOneObservation() { applied := make(chan struct{}) go func() { <-ready - sync.advance() + sync.advance(time.Time{}) close(applied) }() runtime.Gosched() @@ -1839,12 +1870,19 @@ func (h *waitForSyncHarness) tipBlocks(tipPeer types.NodeID, interval time.Durat h.sync.AddPeer(newPeerData(tipPeer, 1, height)) h.suite.Require().NoError(h.sync.addBlock(BlockResponse{ PeerID: tipPeer, - Block: &types.Block{Header: types.Header{Height: height, Time: h.clock.Now()}}, + Block: &types.Block{Header: types.Header{Height: height}}, })) - h.sync.advance() + h.sync.advance(h.clock.Now()) } } +// setLastBlockTime makes blockTime the time of the last block the node applied. +func (h *waitForSyncHarness) setLastBlockTime(blockTime time.Time) { + h.sync.mtx.Lock() + defer h.sync.mtx.Unlock() + h.sync.blockTimes = []time.Time{blockTime} +} + const ( // tipBlockInterval is shorter than syncTimeout, so chain growth alone // keeps every stall-based handover from firing. @@ -1853,8 +1891,8 @@ const ( unservedHandoverSlack = 10 * time.Second ) -// TestWaitForSyncHandsOverPastUnservedClaims checks that a node level with its -// the other peers hands over to consensus even while another peer claims heights +// TestWaitForSyncHandsOverPastUnservedClaims checks that a node level with the +// other peers hands over to consensus even while another peer claims heights // far above and never serves them, and while new blocks keep arriving faster // than a stall can be declared. Each variant costs the peer nothing to send. func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastUnservedClaims() { @@ -1894,104 +1932,19 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastUnservedClaims() defer h.cancel() tc.evidence(h) - elapsed, caughtUp, ended := h.run(unservedClaimGrace+time.Minute, h.tipBlocks(tipPeer, tipBlockInterval)) + elapsed, caughtUp, ended := h.run(tipGrace+time.Minute, h.tipBlocks(tipPeer, tipBlockInterval)) suite.Require().True(ended, "block sync kept following the tip behind a claim nobody serves") suite.Require().False(caughtUp, "an unserved claim was not reached, so this is not catching up") - suite.Require().LessOrEqual(elapsed, unservedClaimGrace+unservedHandoverSlack) + suite.Require().LessOrEqual(elapsed, tipGrace+unservedHandoverSlack) }) } } -// TestWaitForSyncUnservedClaimSurvivesReAdd checks that a peer dropped for failing -// the heights it claims does not regain its standing by sending its status again. -// The node polls every peer for status and a peer may also push one unsolicited, -// so without this the peer is back with a clean record seconds after each drop. -func (suite *SynchronizerTestSuite) TestWaitForSyncUnservedClaimSurvivesReAdd() { - const ( - height = int64(100) - tipPeer = types.NodeID("tip peer") - unserving = types.NodeID("unserving peer") - ) - h := suite.newWaitForSyncHarness(height, - newPeerData(tipPeer, 1, height-1), - newPeerData(unserving, 1, maxPlausiblePeerHeight)) - defer h.cancel() - - suite.client. - On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")). - Once(). - Return(nil) - for i := int64(0); i < int64(maxConsecutiveFailures); i++ { - h.fail(unserving, height+i, client.ErrPeerNotResponded) - } - _, found := h.sync.peerStore.Get(unserving) - suite.Require().False(found, "the peer must be dropped after %d failed requests", maxConsecutiveFailures) - h.sync.AddPeer(newPeerData(unserving, 1, maxPlausiblePeerHeight)) - - elapsed, caughtUp, ended := h.run(unservedClaimGrace+time.Minute, h.tipBlocks(tipPeer, tipBlockInterval)) - suite.Require().True(ended, "a fresh status restored the claim the peer had failed to serve") - suite.Require().False(caughtUp) - suite.Require().LessOrEqual(elapsed, unservedClaimGrace+unservedHandoverSlack) -} - -// TestWaitForSyncHandsOverPastTwoUnservedClaims checks that a second peer joining -// later only delays the handover until it, too, fails to serve what it claims. -func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastTwoUnservedClaims() { - const ( - height = int64(100) - tipPeer = types.NodeID("tip peer") - first = types.NodeID("first unserving peer") - second = types.NodeID("second unserving peer") - joins = 30 * time.Second - timeout = joins + 60*time.Second - ) - h := suite.newWaitForSyncHarness(height, - newPeerData(tipPeer, 1, height-1), - newPeerData(first, 1, maxPlausiblePeerHeight)) - defer h.cancel() - h.fail(first, height, client.ErrPeerNotResponded) - - traffic := h.tipBlocks(tipPeer, tipBlockInterval) - elapsed, caughtUp, ended := h.run(timeout+unservedClaimGrace+time.Minute, func(elapsed time.Duration) { - traffic(elapsed) - switch elapsed { - case joins: - h.sync.AddPeer(newPeerData(second, 1, maxPlausiblePeerHeight)) - case timeout: - height, _ := h.sync.GetStatus() - h.fail(second, height, client.ErrPeerNotResponded) - } - }) - suite.Require().True(ended, "block sync kept following the tip behind two unserved claims") - suite.Require().False(caughtUp) - suite.Require().Greater(elapsed, timeout, "handed over while the second peer's claim was still untested") - suite.Require().LessOrEqual(elapsed, timeout+unservedClaimGrace+unservedHandoverSlack) -} - -// TestWaitForSyncKeepsSyncingBehindAServedClaim is the control for the tests -// above: a peer that failed a request does not end block sync while another -// peer still has far more to give. -func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingBehindAServedClaim() { - const ( - height = int64(100) - unserving = types.NodeID("unserving peer") - ) - h := suite.newWaitForSyncHarness(height, - newPeerData("peer far ahead", 1, height+1000), - newPeerData(unserving, 1, maxPlausiblePeerHeight)) - defer h.cancel() - h.fail(unserving, height, client.ErrPeerNotResponded) - - _, _, ended := h.run(3*unservedClaimGrace, nil) - suite.Require().False(ended, "block sync gave up while another peer still held 1000 more blocks") -} - // TestWaitForSyncKeepsSyncingWhenASlowPeerRecovers checks that a peer far ahead // which once failed the height we need, and then serves a run of blocks from it, -// counts again: one timeout from a slow peer does not make its claim false, and -// handing over while thousands of blocks behind leaves the node in consensus -// catch-up it cannot finish (dashpay/tenderdash#1413). A second peer level with -// the node is what lets the unserved-claims rule apply at all. +// keeps block sync going: one timeout from a slow peer does not make its claim +// false, and handing over while thousands of blocks behind leaves the node in +// consensus catch-up it cannot finish (dashpay/tenderdash#1413). func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingWhenASlowPeerRecovers() { ctx := context.Background() b := suite.newBacklogHarness() @@ -2019,8 +1972,8 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingWhenASlowPeerReco h := suite.startWaitForSync(b.pool, clock) defer h.cancel() - _, _, ended := h.run(3*unservedClaimGrace, func(elapsed time.Duration) { - if elapsed == unservedClaimGrace/2 { + _, _, ended := h.run(3*tipGrace, func(elapsed time.Duration) { + if elapsed == tipGrace/2 { for height := int64(1); height <= run; height++ { b.deliver(ctx, responses[height]) } @@ -2059,9 +2012,9 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingBehindASoleSlowPe // first timeout at 0; probe 30s after it times out 60s later; served afterwards const ( probeTimedOut = 90 * time.Second - served = unservedClaimGrace + 30*time.Second + served = tipGrace + 30*time.Second ) - elapsed, _, ended := h.run(3*unservedClaimGrace, func(elapsed time.Duration) { + elapsed, _, ended := h.run(3*tipGrace, func(elapsed time.Duration) { switch elapsed { case probeTimedOut: b.timeout(ctx, slow, 1) @@ -2073,49 +2026,6 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingBehindASoleSlowPe "block sync handed over after %s while 9999 blocks behind a peer that failed only twice", elapsed) } -// TestWaitForSyncHandsOverPastAClaimServedOnlyAtTheTip checks the handover bound -// against a peer that claims a far height but supplies the newest block of the -// chain once in a while, failing the height after it in between. Serving a block -// every other peer will also hold says nothing about the far claim. -func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAClaimServedOnlyAtTheTip() { - const ( - height = int64(100) - tipPeer = types.NodeID("tip peer") - relay = types.NodeID("relay") - // the network commits a block every serveEvery; the relay supplies it and - // times out the next height's request syncTimeout later - serveEvery = 90 * time.Second - ) - h := suite.newWaitForSyncHarness(height, - newPeerData(tipPeer, 1, height-1), - newPeerData(relay, 1, maxPlausiblePeerHeight)) - defer h.cancel() - h.fail(relay, height, client.ErrPeerNotResponded) - - limit := 10 * time.Minute - elapsed, caughtUp, ended := h.run(limit, func(elapsed time.Duration) { - if elapsed == 0 { - return - } - switch elapsed % serveEvery { - case 0: - next, _ := h.sync.GetStatus() - // what applyBlock does for a block the relay supplied - h.sync.advance() - h.sync.peerStore.RecordServed(relay, next) - h.sync.peerStore.Update(relay, ResetFailures()) - h.sync.AddPeer(newPeerData(tipPeer, 1, next)) - case syncTimeout: - next, _ := h.sync.GetStatus() - h.fail(relay, next, client.ErrPeerNotResponded) - } - }) - suite.Require().True(ended, - "block sync still following the tip after %s behind a claim nobody serves", limit) - suite.Require().False(caughtUp) - suite.Require().LessOrEqual(elapsed, syncTimeout+unservedClaimGrace+unservedHandoverSlack) -} - // TestWaitForSyncKeepsSyncingBehindASlowPeerWithASyncingPeerAlongside is the sole // slow peer case with one more peer that is itself still syncing at our height, // as when several nodes start together. That peer has nothing to offer, so it @@ -2142,9 +2052,9 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingBehindASlowPeerWi defer h.cancel() const ( probeTimedOut = 90 * time.Second - served = unservedClaimGrace + 30*time.Second + served = tipGrace + 30*time.Second ) - elapsed, _, ended := h.run(3*unservedClaimGrace, func(elapsed time.Duration) { + elapsed, _, ended := h.run(3*tipGrace, func(elapsed time.Duration) { switch elapsed { case probeTimedOut: b.timeout(ctx, slow, 1) @@ -2185,48 +2095,50 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAnUnservedClaimO h.sync.AddPeer(newPeerData(tip, 1, next-1+perRound)) case 2 * time.Second: for i := int64(0); i < perRound; i++ { - h.sync.advance() + h.sync.advance(h.clock.Now()) } } }) suite.Require().True(ended, "block sync still following the tip after %s behind a claim nobody serves", limit) suite.Require().False(caughtUp) - suite.Require().LessOrEqual(elapsed, syncTimeout+unservedClaimGrace+unservedHandoverSlack) + suite.Require().LessOrEqual(elapsed, syncTimeout+tipGrace+unservedHandoverSlack) } // TestWaitForSyncHandsOverPastAnUnservedClaimOnAHaltedChain checks that a chain // that stopped producing blocks, as one waiting for this node's vote may, does not // leave the node in block sync behind a claim nobody serves. Its last block is -// too old to say the node is at the tip, so the wedge limit decides. +// too old to say the node is at the tip, so the wedge limit decides, and a peer +// that supplied blocks before the chain stopped loses that credit by failing. func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAnUnservedClaimOnAHaltedChain() { const ( height = int64(100) tipPeer = types.NodeID("tip peer") unserving = types.NodeID("unserving peer") ) - h := suite.newWaitForSyncHarness(height, - newPeerData(tipPeer, 1, height-1), - newPeerData(unserving, 1, maxPlausiblePeerHeight)) - defer h.cancel() - h.sync.mtx.Lock() - h.sync.lastBlockTime = h.clock.Now().Add(-tipRecency) - h.sync.mtx.Unlock() - h.fail(unserving, height, client.ErrPeerNotResponded) - - // the peer keeps failing the probes it is sent for the height, and reports - // its status again whenever it is dropped for that - suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) - const probeEvery = 90 * time.Second - elapsed, caughtUp, ended := h.run(maxSyncStall+time.Minute, func(elapsed time.Duration) { - if elapsed > 0 && elapsed%probeEvery == 0 { - h.fail(unserving, height, client.ErrPeerNotResponded) - h.sync.AddPeer(newPeerData(unserving, 1, maxPlausiblePeerHeight)) - } - }) - suite.Require().True(ended, "block sync kept waiting on a halted chain behind a claim nobody serves") - suite.Require().False(caughtUp) - suite.Require().Greater(elapsed, maxSyncStall, "handed over with no recent block before the wedge limit") - suite.Require().LessOrEqual(elapsed, maxSyncStall+unservedHandoverSlack) + for _, supplied := range []bool{false, true} { + suite.Run(fmt.Sprintf("supplied before %t", supplied), func() { + claimant := newPeerData(unserving, 1, maxPlausiblePeerHeight) + claimant.supplied = supplied + h := suite.newWaitForSyncHarness(height, newPeerData(tipPeer, 1, height-1), claimant) + defer h.cancel() + h.setLastBlockTime(h.clock.Now().Add(-tipRecency)) + + // the peer fails the probes it is sent for the height, too seldom to + // be dropped for it, and reports its status again after each + suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) + const probeEvery = 150 * time.Second + elapsed, caughtUp, ended := h.run(maxSyncStall+time.Minute, func(elapsed time.Duration) { + if elapsed > 0 && elapsed%probeEvery == 0 { + h.fail(unserving, height, client.ErrPeerNotResponded) + h.sync.AddPeer(newPeerData(unserving, 1, maxPlausiblePeerHeight)) + } + }) + suite.Require().True(ended, "block sync kept waiting on a halted chain behind a claim nobody serves") + suite.Require().False(caughtUp) + suite.Require().Greater(elapsed, maxSyncStall, "handed over with no recent block before the wedge limit") + suite.Require().LessOrEqual(elapsed, maxSyncStall+unservedHandoverSlack) + }) + } } // flakySource returns a step for run in which source, the only peer far ahead, @@ -2247,7 +2159,8 @@ func (h *waitForSyncHarness) flakySource(source types.NodeID, deliverEvery, fail PeerID: source, Block: &types.Block{Header: types.Header{Height: next}}, })) - h.sync.advance() + // a block the network committed long ago + h.sync.advance(time.Time{}) h.sync.peerStore.Update(source, ResetFailures()) } } @@ -2262,7 +2175,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingFromAFlakySourceW height = int64(100) source = types.NodeID("source") behind = types.NodeID("peer syncing behind") - limit = 3 * unservedClaimGrace + limit = 3 * tipGrace deliver = 5 * time.Second failure = 50 * time.Second ) @@ -2292,10 +2205,10 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingFromAFlakySourceT const ( height = int64(100) source = types.NodeID("source") - limit = 3 * unservedClaimGrace + limit = 3 * tipGrace deliver = 5 * time.Second failure = 50 * time.Second - dropAt = unservedClaimGrace + 7*time.Second + dropAt = tipGrace + 7*time.Second ) h := suite.newWaitForSyncHarness(height, newPeerData(source, 1, height+10000)) defer h.cancel() @@ -2335,44 +2248,45 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingWhenTheLastBlockL ) h := suite.newWaitForSyncHarness(height, newPeerData(slow, 1, height+10000)) defer h.cancel() - h.sync.mtx.Lock() - h.sync.lastBlockTime = h.clock.Now().Add(30 * time.Minute) - h.sync.mtx.Unlock() + h.setLastBlockTime(h.clock.Now().Add(30 * time.Minute)) h.fail(slow, height, client.ErrPeerNotResponded) - elapsed, _, ended := h.run(3*unservedClaimGrace, nil) + elapsed, _, ended := h.run(3*tipGrace, nil) suite.Require().False(ended, "handed over after %s, 10000 blocks behind, on a block time ahead of the local clock", elapsed) } -// TestIsRecent checks which last block times put the node at the tip: old ones -// do not, and neither do ones further ahead of the local clock than the -// synchrony bounds allow a clock in step with the validators' to see. -func TestIsRecent(t *testing.T) { - sync := NewSynchronizer(1, nil, nil) +// TestAtTip checks which last block times put the node at the tip: none older +// than the window, and none further ahead of the local clock than the +// precision validators hold block times to. +func TestAtTip(t *testing.T) { + const ( + window = time.Minute + precision = time.Second + ) now := time.Unix(1_700_000_000, 0) - ahead := sync.synchrony.Precision + sync.synchrony.MessageDelay for _, tc := range []struct { name string age time.Duration want bool }{ {name: "just applied", age: 0, want: true}, - {name: "as old as tipRecency", age: tipRecency, want: true}, - {name: "older than tipRecency", age: tipRecency + time.Second}, - {name: "ahead within the synchrony bounds", age: -ahead, want: true}, - {name: "ahead beyond the synchrony bounds", age: -ahead - time.Second}, + {name: "as old as the window", age: window, want: true}, + {name: "older than the window", age: window + time.Millisecond}, + {name: "ahead within the precision", age: -precision, want: true}, + {name: "ahead beyond the precision", age: -precision - time.Millisecond}, } { t.Run(tc.name, func(t *testing.T) { - require.Equal(t, tc.want, sync.isRecent(now.Add(-tc.age), now)) + require.Equal(t, tc.want, atTip(now.Add(-tc.age), window, precision, now)) }) } + require.False(t, atTip(time.Time{}, window, precision, now), "no block time") } -// TestWaitForSyncUnservedHandoverTargetsCountedClaims checks that handing over past -// claims that went unserved does not pass them on as the consensus catch-up +// TestWaitForSyncAtTipHandoverTargetsNearClaims checks that handing over at the +// tip does not pass on a claim out of reach of the tip as the consensus catch-up // target, which would hold back the node's own proposals for that long again. -func (suite *SynchronizerTestSuite) TestWaitForSyncUnservedHandoverTargetsCountedClaims() { +func (suite *SynchronizerTestSuite) TestWaitForSyncAtTipHandoverTargetsNearClaims() { const ( height = int64(100) tipPeer = types.NodeID("tip peer") @@ -2382,13 +2296,135 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncUnservedHandoverTargetsCounte newPeerData(tipPeer, 1, height-1), newPeerData(unserving, 1, maxPlausiblePeerHeight)) defer h.cancel() - h.fail(unserving, height, client.ErrPeerNotResponded) - _, caughtUp, ended := h.run(unservedClaimGrace+time.Minute, h.tipBlocks(tipPeer, tipBlockInterval)) + _, caughtUp, ended := h.run(tipGrace+time.Minute, h.tipBlocks(tipPeer, tipBlockInterval)) suite.Require().True(ended) suite.Require().False(caughtUp) applied, _ := h.sync.GetStatus() - suite.Require().Equal(applied-1, h.target, "the unserved claim must not become the catch-up target") + suite.Require().Equal(applied-1, h.target, "a claim out of reach of the tip must not become the catch-up target") +} + +// newPeerEveryMinute returns a step for run that, every minute, drops the peer +// claiming a far height and connects a new one making the same claim. Node IDs +// cost nothing, so no new peer has a history to judge it by. +func (h *waitForSyncHarness) newPeerEveryMinute() func(time.Duration) { + var current types.NodeID + return func(elapsed time.Duration) { + if elapsed%time.Minute != 0 { + return + } + if current != "" { + h.sync.RemovePeer(current) + } + current = types.NodeID(fmt.Sprintf("claimant %d", elapsed/time.Minute)) + h.sync.AddPeer(newPeerData(current, 1, maxPlausiblePeerHeight)) + } +} + +// TestWaitForSyncHandsOverPastAFarClaimFromNewPeers checks that a far claim made +// by a new peer every minute does not keep a node level with the network in +// block sync while the chain keeps committing blocks. +func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAFarClaimFromNewPeers() { + const ( + height = int64(100) + tipPeer = types.NodeID("tip peer") + ) + h := suite.newWaitForSyncHarness(height, newPeerData(tipPeer, 1, height-1)) + defer h.cancel() + + traffic := h.tipBlocks(tipPeer, tipBlockInterval) + claims := h.newPeerEveryMinute() + elapsed, caughtUp, ended := h.run(tipGrace+time.Minute, func(elapsed time.Duration) { + claims(elapsed) + traffic(elapsed) + }) + suite.Require().True(ended, "block sync kept following the tip behind a claim moved to a new peer every minute") + suite.Require().False(caughtUp) + suite.Require().LessOrEqual(elapsed, tipGrace+unservedHandoverSlack) +} + +// TestWaitForSyncHandsOverPastAFarClaimFromNewPeersOnAHaltedChain checks the same +// on a chain that stopped producing blocks: the stall backstop still fires. +func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAFarClaimFromNewPeersOnAHaltedChain() { + const ( + height = int64(100) + tipPeer = types.NodeID("tip peer") + ) + h := suite.newWaitForSyncHarness(height, newPeerData(tipPeer, 1, height-1)) + defer h.cancel() + h.setLastBlockTime(h.clock.Now().Add(-time.Hour)) + + elapsed, caughtUp, ended := h.run(maxSyncStall+time.Minute, h.newPeerEveryMinute()) + suite.Require().True(ended, "block sync kept waiting on a halted chain behind a claim moved to a new peer every minute") + suite.Require().False(caughtUp) + suite.Require().Greater(elapsed, maxSyncStall) + suite.Require().LessOrEqual(elapsed, maxSyncStall+unservedHandoverSlack) +} + +// TestWaitForSyncHandsOverPastAFarClaimFromAPeerSupplyingTheTip checks that a peer +// claiming a far height does not keep the node in block sync by supplying each +// new block as the network commits it. It never fails a request, yet its blocks +// only ever take the node to the network's head. +func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAFarClaimFromAPeerSupplyingTheTip() { + const ( + height = int64(100) + tipPeer = types.NodeID("tip peer") + relay = types.NodeID("relay") + ) + h := suite.newWaitForSyncHarness(height, + newPeerData(tipPeer, 1, height-1), + newPeerData(relay, 1, maxPlausiblePeerHeight)) + defer h.cancel() + + elapsed, caughtUp, ended := h.run(tipGrace+time.Minute, func(elapsed time.Duration) { + if elapsed == 0 || elapsed%tipBlockInterval != 0 { + return + } + next, _ := h.sync.GetStatus() + h.sync.AddPeer(newPeerData(tipPeer, 1, next)) + h.suite.Require().NoError(h.sync.addBlock(BlockResponse{ + PeerID: relay, + Block: &types.Block{Header: types.Header{Height: next}}, + })) + h.sync.advance(h.clock.Now()) + h.sync.peerStore.Update(relay, ResetFailures()) + }) + suite.Require().True(ended, "block sync kept following the tip behind a claim of a peer supplying the tip") + suite.Require().False(caughtUp) + suite.Require().LessOrEqual(elapsed, tipGrace+unservedHandoverSlack) +} + +// TestWaitForSyncKeepsSyncingOnAFastChainWhileOutOfReach checks that on a chain +// committing two blocks a second, a node 100 seconds behind - 200 blocks, far +// more than consensus catch-up closes - is not taken for one at the tip because +// its last block is only minutes old, whatever another peer claims. +func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingOnAFastChainWhileOutOfReach() { + const ( + height = int64(100) + source = types.NodeID("source") + behind = 100 * time.Second + interval = 500 * time.Millisecond + ) + h := suite.newWaitForSyncHarness(height, + newPeerData(source, 1, height+10000), + newPeerData("claimant", 1, maxPlausiblePeerHeight)) + defer h.cancel() + start := h.clock.Now() + h.setLastBlockTime(start.Add(-behind)) + + // the source supplies blocks at the pace the chain commits them, so the node + // stays the same distance behind + elapsed, _, ended := h.run(maxSyncStall/2, func(time.Duration) { + for i := 0; i < int(time.Second/interval); i++ { + next, _ := h.sync.GetStatus() + h.suite.Require().NoError(h.sync.addBlock(BlockResponse{ + PeerID: source, + Block: &types.Block{Header: types.Header{Height: next}}, + })) + h.sync.advance(start.Add(-behind + time.Duration(next-height+1)*interval)) + } + }) + suite.Require().False(ended, "handed over after %s while %s of blocks behind on a fast chain", elapsed, behind) } // backlogChainLen is how many heights the backlog tests have blocks for. It is @@ -3017,7 +3053,7 @@ func (suite *SynchronizerTestSuite) TestSmallBlocksAfterSlowApplyKeepPeersUsable st := h.pool.stallSnapshot() height := st.height suite.Equal(keepSyncing, - stallVerdictFor(st.servable, st.maxServableHeight-height, st.stalledFor, st.maxCredibleHeight >= height, 0), + stallVerdictFor(st.servable, st.maxServableHeight-height, st.stalledFor, st.suppliedServable, 0), "healthy peers must not trigger consensus handoff at height %d below tip %d", height, st.maxPeerHeight) _, found := h.pool.peerStore.FindPeer(height) suite.True(found, "healthy peers must accept the next block request after the batch drains") From 304db7b1381cd4c241b2a11ed02b84cd756cea75 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 11:03:25 +0000 Subject: [PATCH 03/36] test(blocksync): reproduce block sync held at the tip by far claims under rotating peer IDs Fails on f0bf55d58: a fresh peer ID whose far claim has not yet failed counts as credible, so a peer rotating IDs faster than syncTimeout keeps every handover path (tip grace, 10-minute backstop) from firing. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/synchronizer_test.go | 33 +++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/internal/blocksync/synchronizer_test.go b/internal/blocksync/synchronizer_test.go index f8697b88e..6374fc815 100644 --- a/internal/blocksync/synchronizer_test.go +++ b/internal/blocksync/synchronizer_test.go @@ -1940,6 +1940,39 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastUnservedClaims() } } +// TestWaitForSyncHandsOverPastRotatingUnservedClaims checks that unserved far +// claims arriving under a fresh peer ID before the previous one has failed do +// not keep a node at the tip in block sync indefinitely. A node ID costs a peer +// nothing, so a claim record kept per ID must not be the only way out. +func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastRotatingUnservedClaims() { + const ( + height = int64(100) + tipPeer = types.NodeID("tip peer") + // a new ID joins every rotateEvery; each fails syncTimeout after joining + rotateEvery = 30 * time.Second + ) + h := suite.newWaitForSyncHarness(height, newPeerData(tipPeer, 1, height-1)) + defer h.cancel() + + traffic := h.tipBlocks(tipPeer, tipBlockInterval) + limit := maxSyncStall + tipGrace + elapsed, caughtUp, ended := h.run(limit, func(elapsed time.Duration) { + traffic(elapsed) + if elapsed%rotateEvery == 0 { + id := types.NodeID(fmt.Sprintf("rotating peer %d", elapsed/rotateEvery)) + h.sync.AddPeer(newPeerData(id, 1, maxPlausiblePeerHeight)) + } + if elapsed >= syncTimeout && (elapsed-syncTimeout)%rotateEvery == 0 { + id := types.NodeID(fmt.Sprintf("rotating peer %d", (elapsed-syncTimeout)/rotateEvery)) + next, _ := h.sync.GetStatus() + h.fail(id, next, client.ErrPeerNotResponded) + } + }) + suite.Require().True(ended, + "block sync still following the tip after %s behind far claims under rotating peer IDs", elapsed) + suite.Require().False(caughtUp) +} + // TestWaitForSyncKeepsSyncingWhenASlowPeerRecovers checks that a peer far ahead // which once failed the height we need, and then serves a run of blocks from it, // keeps block sync going: one timeout from a slow peer does not make its claim From 7451f76d049dabaa9ed6dcea73fac2af4ba23d9d Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 11:41:01 +0000 Subject: [PATCH 04/36] fix(blocksync): estimate the chain's pace from the blocks applied so far Until block sync has applied maxCatchupGap blocks since it started, the at-tip window fell back to tipRecency, so a node restarting minutes behind a fast chain was taken for one at the tip when its source supplied fewer than that many blocks within the grace period. Scale the span of the blocks applied so far up to maxCatchupGap instead; with no block applied yet, tipRecency alone still applies. Also test that starting block sync seeds the last block time from the stored state and that the synchrony precision follows the current state. Co-Authored-By: Claude Opus 5.5 --- docs/tenderdash-core/block-sync/reactor.md | 8 +- internal/blocksync/synchronizer.go | 9 +- internal/blocksync/synchronizer_test.go | 119 ++++++++++++++++----- 3 files changed, 107 insertions(+), 29 deletions(-) diff --git a/docs/tenderdash-core/block-sync/reactor.md b/docs/tenderdash-core/block-sync/reactor.md index 2885c7b15..e0dc760eb 100644 --- a/docs/tenderdash-core/block-sync/reactor.md +++ b/docs/tenderdash-core/block-sync/reactor.md @@ -65,7 +65,9 @@ block, which the validators commit to and a peer cannot change: a node behind the network applies old blocks. The node is at the tip while that block is no older than the chain took to commit its last 10 blocks, or than 70 seconds (a status round and a request timeout) where that is longer, and in any case no -older than five minutes. Five minutes covers the three-minute empty-block +older than five minutes. Until the node has applied 10 blocks since it started, +the pace of those it has applied stands in for that; before the first, only +the five-minute limit applies. Five minutes covers the three-minute empty-block interval Dash Platform runs with and a margin for extra rounds and clock skew. A block time further ahead of the local clock than the synchrony precision does not count, since the clock is then behind the validators' and cannot tell how @@ -99,6 +101,10 @@ Some cases remain open: behind, and so in block sync, for as long as it does so. Such a node can also hand over at the tip up to 70 seconds of blocks behind, which consensus catch-up then closes. +- A node that restarts with a last block less than five minutes old and then + applies no block for two minutes hands over at the tip; on a chain that + commits more than 10 blocks in 70 seconds it may then be up to five minutes + of blocks behind. - A node any distance behind whose peers that advertise the next height fail every request for 10 minutes hands over, as it would if none advertised it. - After a handover on a stall, the catch-up target includes every claim, so diff --git a/internal/blocksync/synchronizer.go b/internal/blocksync/synchronizer.go index 93069907d..7de4c04e3 100644 --- a/internal/blocksync/synchronizer.go +++ b/internal/blocksync/synchronizer.go @@ -587,7 +587,8 @@ type stallState struct { // // The node is at the tip when its last block is no older than the chain took // to commit the last maxCatchupGap blocks, or than tipLag where that is longer, -// and in any case no older than tipRecency. +// and in any case no older than tipRecency. Until the node has seen that many +// blocks, the pace of those it has seen stands in; with none, tipRecency alone. // // They are read under one lock because advance() stamps the height, the advance // time, the block time and the supplier's credit together under that same lock. @@ -610,8 +611,10 @@ func (s *Synchronizer) stallSnapshot() stallState { } if last := len(s.blockTimes) - 1; last >= 0 { st.lastBlockTime = s.blockTimes[last] - if last == int(maxCatchupGap) { - st.tipWindow = min(tipRecency, max(tipLag, s.blockTimes[last].Sub(s.blockTimes[0]))) + if last > 0 { + // Fewer than maxCatchupGap intervals since start are scaled up to it. + span := s.blockTimes[last].Sub(s.blockTimes[0]) / time.Duration(last) * time.Duration(maxCatchupGap) + st.tipWindow = min(tipRecency, max(tipLag, span)) } } hasHeight := heightBetweenPeerHeightRange(s.height) diff --git a/internal/blocksync/synchronizer_test.go b/internal/blocksync/synchronizer_test.go index 6374fc815..4549544fd 100644 --- a/internal/blocksync/synchronizer_test.go +++ b/internal/blocksync/synchronizer_test.go @@ -1347,6 +1347,31 @@ func (suite *SynchronizerTestSuite) TestAdvanceCreditsTheSupplier() { suite.True(st.suppliedServable) } +// TestStartSeedsTheLastBlockTime checks that block sync judges the tip from the +// last block the node already has until it applies another, so a node restarting +// after long downtime is not taken for one at the tip. +func (suite *SynchronizerTestSuite) TestStartSeedsTheLastBlockTime() { + state := suite.initialState.Copy() + state.LastBlockTime = time.Unix(1_700_000_000, 0) + applier := newBlockApplier(suite.blockExec, suite.store, applierWithState(state)) + pool := NewSynchronizer(1, suite.client, applier, WithWorkerPool(workerpool.New(0))) + suite.Require().NoError(pool.Start(context.Background())) + defer pool.Stop() + suite.Equal(state.LastBlockTime, pool.stallSnapshot().lastBlockTime) +} + +// TestPrecisionFollowsTheCurrentState checks that the synchrony precision is read +// from the chain's current parameters, not those it had when block sync started. +func (suite *SynchronizerTestSuite) TestPrecisionFollowsTheCurrentState() { + applier := newBlockApplier(suite.blockExec, suite.store, applierWithState(suite.initialState)) + pool := NewSynchronizer(1, suite.client, applier) + suite.Equal(types.DefaultSynchronyParams().Precision, pool.precision()) + state := suite.initialState.Copy() + state.ConsensusParams.Synchrony.Precision = 2 * time.Second + applier.UpdateState(state) + suite.Equal(2*time.Second, pool.precision()) +} + // TestStatusRefreshUpdatesAdvertisedRange checks that a status response still does // the one thing it is for: moving the range of blocks the peer claims to serve. It // is the guard on the tests above, which a synchronizer that ignored status @@ -1749,7 +1774,9 @@ func TestStallSnapshotTipWindow(t *testing.T) { want time.Duration }{ {name: "no block applied", want: tipRecency}, - {name: "fewer blocks than the gap", blocks: int(maxCatchupGap), interval: time.Second, want: tipRecency}, + {name: "one block applied", blocks: 1, interval: time.Second, want: tipRecency}, + {name: "fewer blocks than the gap, fast", blocks: 3, interval: time.Second, want: tipLag}, + {name: "fewer blocks than the gap, moderate", blocks: 3, interval: 15 * time.Second, want: 150 * time.Second}, {name: "a fast chain", blocks: 100, interval: time.Second, want: tipLag}, {name: "a moderate chain", blocks: 100, interval: 15 * time.Second, want: 150 * time.Second}, {name: "a slow chain", blocks: 100, interval: 3 * time.Minute, want: tipRecency}, @@ -2428,36 +2455,78 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAFarClaimFromAPe } // TestWaitForSyncKeepsSyncingOnAFastChainWhileOutOfReach checks that on a chain -// committing two blocks a second, a node 100 seconds behind - 200 blocks, far -// more than consensus catch-up closes - is not taken for one at the tip because -// its last block is only minutes old, whatever another peer claims. +// committing two blocks a second, a node minutes behind - hundreds of blocks, +// far more than consensus catch-up closes - is not taken for one at the tip +// because its last block is less than tipRecency old, whatever another peer +// claims: not while it applies blocks at the chain's pace, not when it has +// applied too few since it started to span maxCatchupGap blocks, and not when +// the blocks before it were committed minutes apart. func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingOnAFastChainWhileOutOfReach() { const ( height = int64(100) source = types.NodeID("source") - behind = 100 * time.Second interval = 500 * time.Millisecond ) - h := suite.newWaitForSyncHarness(height, - newPeerData(source, 1, height+10000), - newPeerData("claimant", 1, maxPlausiblePeerHeight)) - defer h.cancel() - start := h.clock.Now() - h.setLastBlockTime(start.Add(-behind)) - - // the source supplies blocks at the pace the chain commits them, so the node - // stays the same distance behind - elapsed, _, ended := h.run(maxSyncStall/2, func(time.Duration) { - for i := 0; i < int(time.Second/interval); i++ { - next, _ := h.sync.GetStatus() - h.suite.Require().NoError(h.sync.addBlock(BlockResponse{ - PeerID: source, - Block: &types.Block{Header: types.Header{Height: next}}, - })) - h.sync.advance(start.Add(-behind + time.Duration(next-height+1)*interval)) - } - }) - suite.Require().False(ended, "handed over after %s while %s of blocks behind on a fast chain", elapsed, behind) + testCases := []struct { + name string + // behind is how old the node's last block is when block sync starts + behind time.Duration + // quietBefore, when set, is how far apart the blocks up to the node's + // last one were committed + quietBefore time.Duration + // supplyEvery is how often the source supplies the next block + supplyEvery time.Duration + }{ + {name: "applying at the chain's pace", behind: 100 * time.Second, supplyEvery: interval}, + {name: "a slow source after a restart", behind: 100 * time.Second, supplyEvery: 15 * time.Second}, + { + name: "a busy chain after a quiet one", + behind: 4 * time.Minute, + quietBefore: 3 * time.Minute, + supplyEvery: interval, + }, + } + for _, tc := range testCases { + suite.Run(tc.name, func() { + h := suite.newWaitForSyncHarness(height, + newPeerData(source, 1, height+10000), + newPeerData("claimant", 1, maxPlausiblePeerHeight)) + defer h.cancel() + start := h.clock.Now() + last := start.Add(-tc.behind) + h.setLastBlockTime(last) + if tc.quietBefore > 0 { + h.sync.mtx.Lock() + h.sync.blockTimes = nil + for i := maxCatchupGap; i >= 0; i-- { + h.sync.blockTimes = append(h.sync.blockTimes, last.Add(-time.Duration(i)*tc.quietBefore)) + } + h.sync.mtx.Unlock() + // the chain turns busy a minute after the node's last block + last = last.Add(time.Minute - interval) + } + supply := func() { + next, _ := h.sync.GetStatus() + h.suite.Require().NoError(h.sync.addBlock(BlockResponse{ + PeerID: source, + Block: &types.Block{Header: types.Header{Height: next}}, + })) + h.sync.advance(last.Add(time.Duration(next-height+1) * interval)) + } + elapsed, _, ended := h.run(maxSyncStall/2, func(elapsed time.Duration) { + if tc.supplyEvery < time.Second { + for i := 0; i < int(time.Second/tc.supplyEvery); i++ { + supply() + } + } else if elapsed%tc.supplyEvery == 0 { + supply() + } + }) + applied, _ := h.sync.GetStatus() + suite.Require().False(ended, "handed over after %s at height %d, minutes of blocks behind on a fast chain", + elapsed, applied) + }) + } } // backlogChainLen is how many heights the backlog tests have blocks for. It is From f55738800391272fb5244fc613ba084a724b9a65 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 11:46:49 +0000 Subject: [PATCH 05/36] test(blocksync): reproduce stall backstop and producer wait issues found in review Both tests are expected to fail on 7451f76d0: - a far claim from a peer that backed nothing keeps a stalled node in block sync while another, near peer has backed its claim - a pushed-back height an idle peer serves waits behind a higher height only a busy peer claims Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/handover_review_test.go | 60 ++++++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100644 internal/blocksync/handover_review_test.go diff --git a/internal/blocksync/handover_review_test.go b/internal/blocksync/handover_review_test.go new file mode 100644 index 000000000..a27dd700d --- /dev/null +++ b/internal/blocksync/handover_review_test.go @@ -0,0 +1,60 @@ +package blocksync + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/require" + + "github.com/dashpay/tenderdash/libs/log" + "github.com/dashpay/tenderdash/types" +) + +// TestWaitForSyncStallBackstopIgnoresAFarClaimNoPeerBacked checks the stall +// backstop against a far claim from a peer that has supplied nothing, made while +// another peer that has backed its claim is within maxCatchupGap. Only a backed +// claim may keep a stalled node in block sync, so the distance that decides it +// must come from backed claims too. +func (suite *SynchronizerTestSuite) TestWaitForSyncStallBackstopIgnoresAFarClaimNoPeerBacked() { + const height = int64(100) + near := newPeerData("near peer", 1, height+maxCatchupGap/2) + near.supplied = true + h := suite.newWaitForSyncHarness(height, near, newPeerData("claimant", 1, maxPlausiblePeerHeight)) + defer h.cancel() + h.setLastBlockTime(h.clock.Now().Add(-time.Hour)) + + elapsed, caughtUp, ended := h.run(maxSyncStall+time.Minute, nil) + suite.Require().True(ended, "a far claim nobody backed kept a stalled node in block sync") + suite.Require().False(caughtUp) + suite.Require().Greater(elapsed, maxSyncStall) +} + +// TestNextJobDoesNotWaitBehindAHeightNoIdlePeerServes checks that a height +// pushed back for a retry, which an idle peer advertises, is requested while the +// producer waits for a peer to serve a higher height only a busy peer claims. +func TestNextJobDoesNotWaitBehindAHeightNoIdlePeerServes(t *testing.T) { + const retry = int64(100) + busy := newPeerData("busy claimant", 1, 1000) + busy.numPending = maxPendingRequestsPerPeer + store := NewInMemPeerStore(newPeerData("idle peer", 1, retry+5), busy) + gen := newJobGenerator(retry+10, log.NewNopLogger(), nil, store) + + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + done := make(chan error, 1) + go func() { + _, err := gen.nextJob(ctx) + done <- err + }() + time.Sleep(50 * time.Millisecond) + gen.pushBack(retry) + + select { + case err := <-done: + require.NoError(t, err) + case <-ctx.Done(): + require.Fail(t, "producer still waiting for a peer for a height only a busy peer claims", + "height %d stays queued although %s can serve it", retry, types.NodeID("idle peer")) + } +} From 34a9adeec8b905b798c161d94859b2f6179ddc06 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 11:52:12 +0000 Subject: [PATCH 06/36] fix(blocksync): measure the stall backstop's distance from backed claims only The backstop took how far behind the node is from any servable claim and the credit from any peer that backed its claim, so a far claim nobody backed, next to a near peer that had, kept a stalled node in block sync. Take the distance from claims of peers that supplied a block applied since they last failed a request; with none, the node is not known to be behind. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/synchronizer.go | 40 ++++----- internal/blocksync/synchronizer_test.go | 107 +++++++++++------------- 2 files changed, 71 insertions(+), 76 deletions(-) diff --git a/internal/blocksync/synchronizer.go b/internal/blocksync/synchronizer.go index 7de4c04e3..cb3dd4e56 100644 --- a/internal/blocksync/synchronizer.go +++ b/internal/blocksync/synchronizer.go @@ -448,9 +448,9 @@ func (s *Synchronizer) IsCaughtUp() bool { // peer holds the block we are waiting for there is something to retry, so keep // going and say so loudly. Give up on the stall only once no peer can serve that // block, or once the stall has outlasted maxSyncStall within maxCatchupGap of -// the tip or with no peer holding the block that has backed its claim, so that -// a wedged synchronizer can still hand over rather than blocking forever -// without handing over a node consensus cannot catch up. +// the highest claim a peer holding the block has backed, so that a wedged +// synchronizer can still hand over rather than blocking forever without handing +// over a node consensus cannot catch up. // // A height claim costs a peer nothing, and a node ID costs nothing either, so // what a peer claims cannot keep the node in block sync once it is at the tip. @@ -477,8 +477,8 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe if !atTip(st.lastBlockTime, st.tipWindow, s.precision(), now) { atTipSince = now } - verdict := stallVerdictFor(st.servable, st.maxServableHeight-st.height, st.stalledFor, - st.suppliedServable, now.Sub(atTipSince)) + verdict := stallVerdictFor(st.servable, st.maxSuppliedHeight-st.height, st.stalledFor, + now.Sub(atTipSince)) switch verdict { case stopAtTip: s.logger.Warn( @@ -508,6 +508,7 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe "height", st.height, "max_peer_height", st.maxPeerHeight, "max_servable_height", st.maxServableHeight, + "max_backed_height", st.maxSuppliedHeight, "stalled_for", st.stalledFor, ) return false, targetHeight @@ -562,9 +563,9 @@ type stallState struct { servable bool maxServableHeight int64 maxPeerHeight int64 - // suppliedServable says a peer advertising height has supplied a block we - // applied since it last failed a request. - suppliedServable bool + // maxSuppliedHeight is the highest claim of a peer advertising height that + // has supplied a block we applied since it last failed a request; 0 if none. + maxSuppliedHeight int64 // maxNearHeight is the highest claim within maxCatchupGap of the last // applied block. maxNearHeight int64 @@ -626,7 +627,9 @@ func (s *Synchronizer) stallSnapshot() stallState { if hasHeight(peer.peerID, peer) { st.servable = true st.maxServableHeight = max(st.maxServableHeight, peer.height) - st.suppliedServable = st.suppliedServable || peer.supplied + if peer.supplied { + st.maxSuppliedHeight = max(st.maxSuppliedHeight, peer.height) + } } } return st @@ -648,7 +651,8 @@ const ( // stallVerdictFor decides what to do when block sync has made no progress for // stalledFor, given whether any peer holds the block we are waiting for and how -// many blocks behind the highest height a servable peer claims we are. +// many blocks behind the highest height a servable peer that backed its claim +// says we are. // // Waiting on a block someone has is a reason to keep retrying, not to stop: // handing over to consensus is effectively irreversible, so stopping while @@ -664,18 +668,16 @@ const ( // the backstop buys nothing and costs what dashpay/tenderdash#1413 describes, so // block sync keeps retrying and says so every interval. // -// Advertising costs nothing, so far behind the backstop counts only a peer that -// has backed its claim: suppliedServable says a peer holding the block supplied -// one we applied since it last failed a request. A stall of maxSyncStall means -// every peer asked for the block meanwhile failed, so only one never asked keeps -// that credit, and a new peer has none. +// Advertising costs nothing, so behindBy counts only the claims of peers that +// backed them by supplying a block we applied since they last failed a request; +// with no such peer the node is not known to be behind at all. A stall of +// maxSyncStall means every peer asked for the block meanwhile failed, so only +// one never asked keeps that credit, and a new peer has none. // // atTipFor is how long the node has been at the tip by the time of its last // block. Unlike stalledFor it is not reset by applying blocks, so a node // following the network is not kept in block sync by a claim nobody can reach. -func stallVerdictFor( - servable bool, behindBy int64, stalledFor time.Duration, suppliedServable bool, atTipFor time.Duration, -) stallVerdict { +func stallVerdictFor(servable bool, behindBy int64, stalledFor, atTipFor time.Duration) stallVerdict { switch { case atTipFor > tipGrace: return stopAtTip @@ -683,7 +685,7 @@ func stallVerdictFor( return keepSyncing case !servable: return stopNothingToFetch - case stalledFor > maxSyncStall && (behindBy <= maxCatchupGap || !suppliedServable): + case stalledFor > maxSyncStall && behindBy <= maxCatchupGap: return stopStalledTooLong default: return keepSyncing diff --git a/internal/blocksync/synchronizer_test.go b/internal/blocksync/synchronizer_test.go index 4549544fd..9cd41c8ab 100644 --- a/internal/blocksync/synchronizer_test.go +++ b/internal/blocksync/synchronizer_test.go @@ -1344,7 +1344,7 @@ func (suite *SynchronizerTestSuite) TestAdvanceCreditsTheSupplier() { st := pool.stallSnapshot() suite.EqualValues(101, st.height) suite.Equal(blockTime, st.lastBlockTime) - suite.True(st.suppliedServable) + suite.EqualValues(200, st.maxSuppliedHeight) } // TestStartSeedsTheLastBlockTime checks that block sync judges the tip from the @@ -1451,20 +1451,18 @@ func makePeers(numPeers int, minHeight, maxHeight int64) map[types.NodeID]PeerDa // is so far behind that consensus catch-up could never close the gap. func TestStallVerdictFor(t *testing.T) { testCases := []struct { - name string - servable bool - behindBy int64 - stalledFor time.Duration - suppliedServable bool - atTipFor time.Duration - want stallVerdict + name string + servable bool + behindBy int64 + stalledFor time.Duration + atTipFor time.Duration + want stallVerdict }{ { - name: "progressing with the block available", - servable: true, - stalledFor: syncTimeout / 2, - suppliedServable: true, - want: keepSyncing, + name: "progressing with the block available", + servable: true, + stalledFor: syncTimeout / 2, + want: keepSyncing, }, { name: "progressing with nobody holding the block", @@ -1479,34 +1477,30 @@ func TestStallVerdictFor(t *testing.T) { want: stopNothingToFetch, }, { - name: "stalled but a peer still holds the block", - servable: true, - stalledFor: syncTimeout + time.Second, - suppliedServable: true, - want: keepSyncing, + name: "stalled but a peer still holds the block", + servable: true, + stalledFor: syncTimeout + time.Second, + want: keepSyncing, }, { - name: "stalled just short of the wedge limit", - servable: true, - stalledFor: maxSyncStall, - suppliedServable: true, - want: keepSyncing, + name: "stalled just short of the wedge limit", + servable: true, + stalledFor: maxSyncStall, + want: keepSyncing, }, { - name: "stalled past the wedge limit within reach of the tip", - servable: true, - behindBy: maxCatchupGap, - stalledFor: maxSyncStall + time.Second, - suppliedServable: true, - want: stopStalledTooLong, + name: "stalled past the wedge limit within reach of the tip", + servable: true, + behindBy: maxCatchupGap, + stalledFor: maxSyncStall + time.Second, + want: stopStalledTooLong, }, { - name: "stalled past the wedge limit while far behind", - servable: true, - behindBy: maxCatchupGap + 1, - stalledFor: maxSyncStall + time.Second, - suppliedServable: true, - want: keepSyncing, + name: "stalled past the wedge limit while far behind", + servable: true, + behindBy: maxCatchupGap + 1, + stalledFor: maxSyncStall + time.Second, + want: keepSyncing, }, { name: "the wedge limit does not override nothing to fetch", @@ -1515,19 +1509,18 @@ func TestStallVerdictFor(t *testing.T) { want: stopNothingToFetch, }, { - name: "far behind a claim no peer has backed, stalled past the wedge limit", + name: "stalled past the wedge limit with no claim backed", servable: true, - behindBy: maxCatchupGap + 1, + behindBy: -100, stalledFor: maxSyncStall + time.Second, want: stopStalledTooLong, }, { - name: "at the tip past the grace, behind a backed claim", - servable: true, - behindBy: maxCatchupGap + 1, - suppliedServable: true, - atTipFor: tipGrace + time.Second, - want: stopAtTip, + name: "at the tip past the grace, behind a backed claim", + servable: true, + behindBy: maxCatchupGap + 1, + atTipFor: tipGrace + time.Second, + want: stopAtTip, }, { name: "at the tip within the grace", @@ -1540,7 +1533,7 @@ func TestStallVerdictFor(t *testing.T) { for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { require.Equal(t, tc.want, - stallVerdictFor(tc.servable, tc.behindBy, tc.stalledFor, tc.suppliedServable, tc.atTipFor)) + stallVerdictFor(tc.servable, tc.behindBy, tc.stalledFor, tc.atTipFor)) }) } } @@ -1690,12 +1683,12 @@ func (suite *SynchronizerTestSuite) TestStallSnapshotServableHeight() { return peer } testCases := []struct { - name string - peers []PeerData - wantMax int64 - wantUsable int64 - wantSupplied bool - wantNear int64 + name string + peers []PeerData + wantMax int64 + wantUsable int64 + wantBacked int64 + wantNear int64 }{ {name: "no peers"}, { @@ -1730,11 +1723,11 @@ func (suite *SynchronizerTestSuite) TestStallSnapshotServableHeight() { wantNear: height + maxCatchupGap - 1, }, { - name: "a servable peer has supplied a block", - peers: []PeerData{newPeerData("archive", 1, 20), supplied(newPeerData("recent", 5, 30))}, - wantMax: 30, - wantUsable: 30, - wantSupplied: true, + name: "a servable peer has supplied a block", + peers: []PeerData{newPeerData("archive", 1, 20), supplied(newPeerData("recent", 5, 30))}, + wantMax: 30, + wantUsable: 30, + wantBacked: 30, }, { name: "only a peer that cannot serve the height has supplied a block", @@ -1755,7 +1748,7 @@ func (suite *SynchronizerTestSuite) TestStallSnapshotServableHeight() { suite.Require().Equal(tc.wantMax, st.maxPeerHeight) suite.Require().Equal(tc.wantUsable, st.maxServableHeight) suite.Require().Equal(tc.wantUsable > 0, st.servable) - suite.Require().Equal(tc.wantSupplied, st.suppliedServable) + suite.Require().Equal(tc.wantBacked, st.maxSuppliedHeight) suite.Require().Equal(tc.wantNear, st.maxNearHeight) suite.Require().Equal(len(tc.peers) > 0, st.hasPeers) }) @@ -3155,7 +3148,7 @@ func (suite *SynchronizerTestSuite) TestSmallBlocksAfterSlowApplyKeepPeersUsable st := h.pool.stallSnapshot() height := st.height suite.Equal(keepSyncing, - stallVerdictFor(st.servable, st.maxServableHeight-height, st.stalledFor, st.suppliedServable, 0), + stallVerdictFor(st.servable, st.maxSuppliedHeight-height, st.stalledFor, 0), "healthy peers must not trigger consensus handoff at height %d below tip %d", height, st.maxPeerHeight) _, found := h.pool.peerStore.FindPeer(height) suite.True(found, "healthy peers must accept the next block request after the batch drains") From 624641e99ba9a03cd6e7b4840cb7ef465009bf64 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 11:53:12 +0000 Subject: [PATCH 07/36] fix(blocksync): request a lower retry height before waiting on a higher one The job producer took the next height and waited until some peer could take it. When only busy or cooling peers claimed that height, a lower height queued for a retry - typically the one block sync waits to apply - was not requested, even with an idle peer advertising it. Put the higher height back and request the lower one first. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/block_fetch_job.go | 31 ++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/internal/blocksync/block_fetch_job.go b/internal/blocksync/block_fetch_job.go index 7c58f5255..3d23dd2b5 100644 --- a/internal/blocksync/block_fetch_job.go +++ b/internal/blocksync/block_fetch_job.go @@ -123,14 +123,24 @@ func (p *jobGenerator) nextHeight() int64 { } func (p *jobGenerator) nextJob(ctx context.Context) (*workerpool.Job, error) { - height := p.nextHeight() - peer, err := p.getPeer(ctx, height) - if err != nil { - return nil, err + for { + height := p.nextHeight() + peer, err := p.getPeer(ctx, height) + if errors.Is(err, errLowerHeightQueued) { + p.pushBack(height) + continue + } + if err != nil { + return nil, err + } + return workerpool.NewJob(blockFetchJobHandler(p.client, peer, height)), nil } - return workerpool.NewJob(blockFetchJobHandler(p.client, peer, height)), nil } +// errLowerHeightQueued reports that a height below the one waiting for a peer +// was queued for a retry meanwhile. +var errLowerHeightQueued = errors.New("a lower height is queued") + // pushBack schedules heights to be fetched again, skipping those already queued so // a queued height is not fetched twice. Heights already handed to a worker are not // tracked here, so an in-flight height can still be queued and fetched again. @@ -164,6 +174,12 @@ func (p *jobGenerator) pushBack(heights ...int64) { slices.Sort(p.pushedBack) } +func (p *jobGenerator) lowerQueued(height int64) bool { + p.mtx.RLock() + defer p.mtx.RUnlock() + return len(p.pushedBack) > 0 && p.pushedBack[0] < height +} + func (p *jobGenerator) getPeer(ctx context.Context, height int64) (PeerData, error) { for { if ctx.Err() != nil { @@ -173,6 +189,11 @@ func (p *jobGenerator) getPeer(ctx context.Context, height int64) (PeerData, err if found { return peer, nil } + // Blocks are applied in order, so waiting on a higher height for a free + // peer must not hold back a lower one that has to be fetched again. + if p.lowerQueued(height) { + return PeerData{}, errLowerHeightQueued + } // This is preferable to using a timer because the request // interval is so small. Larger request intervals may // necessitate using a timer/ticker. From 38cfc34cc2bdebd5717013bc1acd62833f8a2808 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 11:54:17 +0000 Subject: [PATCH 08/36] fix(blocksync): ask peers that backed their claims before those that did not Peer selection took a due recovery probe first and otherwise the first eligible peer, so new peers, which come with no history, could take turns holding the height block sync waits for, each until its request timed out. Ask a peer that supplied a block applied since it last failed a request before any other; among the rest the order is unchanged. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/peer_service_test.go | 30 +++++++++++++++++++++++++ internal/blocksync/peer_store.go | 10 +++++++-- 2 files changed, 38 insertions(+), 2 deletions(-) diff --git a/internal/blocksync/peer_service_test.go b/internal/blocksync/peer_service_test.go index 9834d91cf..94b1f10c8 100644 --- a/internal/blocksync/peer_service_test.go +++ b/internal/blocksync/peer_service_test.go @@ -234,3 +234,33 @@ func TestServiceSavedCreditCannotHideNewSlowService(t *testing.T) { require.True(t, found) require.Equal(t, types.NodeID("second"), selected.peerID) } + +// TestFindPeerPrefersBackedPeers checks that a peer that supplied a block we +// applied since it last failed a request is asked before peers that did not, +// a due recovery probe included. Otherwise new peers, which come with no +// history, could take turns holding the height block sync waits for. +func TestFindPeerPrefersBackedPeers(t *testing.T) { + back := func(peers *InMemPeerStore, peerID types.NodeID) { + peers.Update(peerID, func(_ types.NodeID, peer *PeerData) { peer.supplied = true }) + } + t.Run("over an ordinary peer listed first", func(t *testing.T) { + peers, _ := servicePeers() + back(peers, "second") + selected, found := peers.FindPeer(2) + require.True(t, found) + require.Equal(t, types.NodeID("second"), selected.peerID) + }) + t.Run("over a due probe", func(t *testing.T) { + peers, clock := servicePeers() + peer, request := beginServiceRequest(t, peers, 1) + clock.Advance(59 * time.Second) + endServiceRequest(peers, peer, request, 100) + clock.Advance(30 * time.Second) + selected, _ := peers.FindPeer(2) + require.Equal(t, peer.peerID, selected.peerID, "a due probe goes first among peers with no record") + back(peers, "second") + selected, found := peers.FindPeer(2) + require.True(t, found) + require.Equal(t, types.NodeID("second"), selected.peerID) + }) +} diff --git a/internal/blocksync/peer_store.go b/internal/blocksync/peer_store.go index 20a404062..46e6ff438 100644 --- a/internal/blocksync/peer_store.go +++ b/internal/blocksync/peer_store.go @@ -128,14 +128,20 @@ func (p *InMemPeerStore) Query(spec store.QueryFunc[types.NodeID, PeerData], lim return p.store.Query(spec, limit) } -// FindPeer prefers ordinary service except when a bounded recovery probe is due. +// FindPeer prefers a peer that supplied a block we applied since it last failed +// a request, then a due recovery probe, then ordinary service. func (p *InMemPeerStore) FindPeer(height int64) (PeerData, bool) { peers := p.Query(store.AndX(peerNumPendingCond(maxPendingRequestsPerPeer, "<"), heightBetweenPeerHeightRange(height)), 0) var selected PeerData priority := 0 for _, peer := range peers { - if next := peer.service.priority(peer.numPending, false); next > priority { + next := peer.service.priority(peer.numPending, false) + if next > 0 && peer.supplied { + // A peer that has backed its claim goes before any that has not. + next += 2 + } + if next > priority { selected, priority = peer, next } } From 050fe873f45396031100d71667d244a93c9b499a Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 11:55:49 +0000 Subject: [PATCH 09/36] fix(blocksync): keep the handover loop off the applier lock The handover loop read the synchrony precision from the applier every second, and the applier holds its lock for the whole time a block is executed, so a slow block kept the loop from noticing cancellation. Record the precision with the block time when a block is applied, seed it at start, and read it from the stall snapshot. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/synchronizer.go | 35 ++++++------- internal/blocksync/synchronizer_test.go | 68 +++++++++++++++++-------- 2 files changed, 66 insertions(+), 37 deletions(-) diff --git a/internal/blocksync/synchronizer.go b/internal/blocksync/synchronizer.go index cb3dd4e56..2048d2ff6 100644 --- a/internal/blocksync/synchronizer.go +++ b/internal/blocksync/synchronizer.go @@ -132,8 +132,10 @@ type ( lastAdvance time.Time // blockTimes holds the times of the last maxCatchupGap+1 applied blocks, - // oldest first. Guarded by mtx. + // oldest first, and precision the synchrony precision of the state they + // left. Guarded by mtx. blockTimes []time.Time + precision time.Duration mtx sync.RWMutex @@ -209,6 +211,7 @@ func NewSynchronizer(start int64, client client.BlockClient, blockExec *blockApp monitorInterval: defaultSyncRateIntervalBlocks, workerPool: workerpool.New(poolWorkerSize, workerpool.WithLogger(logger)), pendingToApply: map[int64]BlockResponse{}, + precision: types.DefaultSynchronyParams().Precision, } for _, opt := range opts { opt(bp) @@ -248,7 +251,9 @@ func (s *Synchronizer) OnStart(ctx context.Context) error { s.lastAdvance = s.clock.Now() s.lastMonitorUpdate = s.lastAdvance if s.applier != nil { - s.blockTimes = []time.Time{s.applier.State().LastBlockTime} + state := s.applier.State() + s.blockTimes = []time.Time{state.LastBlockTime} + s.precision = state.ConsensusParams.Synchrony.SynchronyParamsOrDefaults().Precision } s.ctx, s.cancel = context.WithCancel(ctx) applicationCtx, ok := ctx.Value(applicationContextKey{}).(context.Context) @@ -474,7 +479,7 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe return true, targetHeight } now := s.clock.Now() - if !atTip(st.lastBlockTime, st.tipWindow, s.precision(), now) { + if !atTip(st.lastBlockTime, st.tipWindow, st.precision, now) { atTipSince = now } verdict := stallVerdictFor(st.servable, st.maxSuppliedHeight-st.height, st.stalledFor, @@ -543,15 +548,6 @@ func atTip(blockTime time.Time, window, precision time.Duration, now time.Time) return age <= window && age >= -precision } -// precision is the synchrony precision of the chain's current parameters. -func (s *Synchronizer) precision() time.Duration { - var params types.SynchronyParams - if s.applier != nil { - params = s.applier.State().ConsensusParams.Synchrony - } - return params.SynchronyParamsOrDefaults().Precision -} - // stallState is one observation of what the stall verdict is formed from. type stallState struct { // height is the next height to apply, waiting for stalledFor. @@ -570,9 +566,11 @@ type stallState struct { // applied block. maxNearHeight int64 // lastBlockTime is the time of the last applied block; within tipWindow of - // the local clock the node is at the tip. + // the local clock, and no further than precision ahead of it, the node is at + // the tip. lastBlockTime time.Time tipWindow time.Duration + precision time.Duration } // stallSnapshot reads everything the stall verdict is formed from as one @@ -609,6 +607,7 @@ func (s *Synchronizer) stallSnapshot() stallState { stalledFor: s.clock.Since(s.lastAdvance), hasPeers: len(peers) > 0, tipWindow: tipRecency, + precision: s.precision, } if last := len(s.blockTimes) - 1; last >= 0 { st.lastBlockTime = s.blockTimes[last] @@ -751,7 +750,8 @@ func (s *Synchronizer) applyBlock(ctx context.Context) (BlockResponse, error) { if err != nil { return resp, fmt.Errorf("cannot apply block: %w", err) } - s.advance(s.applier.State().LastBlockTime) + state := s.applier.State() + s.advance(state.LastBlockTime, state.ConsensusParams.Synchrony.SynchronyParamsOrDefaults().Precision) s.updateMonitor() } } @@ -763,11 +763,12 @@ func (s *Synchronizer) getPendingResponse() (BlockResponse, bool) { return resp, ok } -// advance moves past the block just applied, which carries blockTime, and -// credits the peer that supplied it. -func (s *Synchronizer) advance(blockTime time.Time) { +// advance moves past the block just applied, which carries blockTime and left +// the chain with synchrony precision, and credits the peer that supplied it. +func (s *Synchronizer) advance(blockTime time.Time, precision time.Duration) { s.mtx.Lock() defer s.mtx.Unlock() + s.precision = precision if resp, ok := s.pendingToApply[s.height]; ok { s.peerStore.Update(resp.PeerID, func(_ types.NodeID, peer *PeerData) { peer.supplied = true }) } diff --git a/internal/blocksync/synchronizer_test.go b/internal/blocksync/synchronizer_test.go index 9cd41c8ab..796a0ce92 100644 --- a/internal/blocksync/synchronizer_test.go +++ b/internal/blocksync/synchronizer_test.go @@ -1336,7 +1336,7 @@ func (suite *SynchronizerTestSuite) TestAdvanceCreditsTheSupplier() { pool.AddPeer(newPeerData("bystander", 1, 200)) blockTime := time.Unix(1_700_000_000, 0) pool.pendingToApply[100] = BlockResponse{PeerID: peerID} - pool.advance(blockTime) + pool.advance(blockTime, defaultPrecision) supplier, _ := pool.peerStore.Get(peerID) suite.True(supplier.supplied) bystander, _ := pool.peerStore.Get("bystander") @@ -1347,29 +1347,57 @@ func (suite *SynchronizerTestSuite) TestAdvanceCreditsTheSupplier() { suite.EqualValues(200, st.maxSuppliedHeight) } +// defaultPrecision is the synchrony precision of a chain on default parameters. +var defaultPrecision = types.DefaultSynchronyParams().Precision + // TestStartSeedsTheLastBlockTime checks that block sync judges the tip from the -// last block the node already has until it applies another, so a node restarting -// after long downtime is not taken for one at the tip. +// last block the node already has, and the synchrony precision of its stored +// state, until it applies another block, so a node restarting after long +// downtime is not taken for one at the tip. func (suite *SynchronizerTestSuite) TestStartSeedsTheLastBlockTime() { state := suite.initialState.Copy() state.LastBlockTime = time.Unix(1_700_000_000, 0) + state.ConsensusParams.Synchrony.Precision = 2 * time.Second applier := newBlockApplier(suite.blockExec, suite.store, applierWithState(state)) pool := NewSynchronizer(1, suite.client, applier, WithWorkerPool(workerpool.New(0))) suite.Require().NoError(pool.Start(context.Background())) defer pool.Stop() - suite.Equal(state.LastBlockTime, pool.stallSnapshot().lastBlockTime) + st := pool.stallSnapshot() + suite.Equal(state.LastBlockTime, st.lastBlockTime) + suite.Equal(2*time.Second, st.precision) } -// TestPrecisionFollowsTheCurrentState checks that the synchrony precision is read -// from the chain's current parameters, not those it had when block sync started. -func (suite *SynchronizerTestSuite) TestPrecisionFollowsTheCurrentState() { - applier := newBlockApplier(suite.blockExec, suite.store, applierWithState(suite.initialState)) - pool := NewSynchronizer(1, suite.client, applier) - suite.Equal(types.DefaultSynchronyParams().Precision, pool.precision()) +// TestApplyingABlockTakesThePrecisionOfTheNewState checks that the synchrony +// precision follows the chain's parameters as blocks are applied, not those it +// had when block sync started. +func (suite *SynchronizerTestSuite) TestApplyingABlockTakesThePrecisionOfTheNewState() { + ctx := context.Background() + b := suite.newBacklogHarness() + suite.Equal(defaultPrecision, b.pool.stallSnapshot().precision) state := suite.initialState.Copy() - state.ConsensusParams.Synchrony.Precision = 2 * time.Second - applier.UpdateState(state) - suite.Equal(2*time.Second, pool.precision()) + state.ConsensusParams.Synchrony.Precision = 3 * time.Second + b.pool.applier.UpdateState(state) + resp := b.produce(ctx) + suite.Require().NotNil(resp) + b.deliver(ctx, resp) + st := b.pool.stallSnapshot() + suite.EqualValues(2, st.height) + suite.Equal(3*time.Second, st.precision) +} + +// TestWaitForSyncDoesNotWaitForABlockBeingApplied checks that the handover +// loop does not wait on the applier, which is held for the whole time a block +// is executed, so it notices cancellation while a slow block is applied. +func (suite *SynchronizerTestSuite) TestWaitForSyncDoesNotWaitForABlockBeingApplied() { + const height = int64(100) + h := suite.newWaitForSyncHarness(height, newPeerData("peer", 1, height+1000)) + h.sync.applier.mtx.Lock() + defer h.sync.applier.mtx.Unlock() + h.clock.Advance(time.Second) + time.Sleep(10 * time.Millisecond) + h.cancel() + _, ended := h.result(time.Second) + suite.Require().True(ended, "the handover loop waited for the block being applied") } // TestStatusRefreshUpdatesAdvertisedRange checks that a status response still does @@ -1777,7 +1805,7 @@ func TestStallSnapshotTipWindow(t *testing.T) { t.Run(tc.name, func(t *testing.T) { sync := NewSynchronizer(1, nil, nil) for i := 0; i < tc.blocks; i++ { - sync.advance(start.Add(time.Duration(i) * tc.interval)) + sync.advance(start.Add(time.Duration(i)*tc.interval), defaultPrecision) } require.Equal(t, tc.want, sync.stallSnapshot().tipWindow) }) @@ -1804,7 +1832,7 @@ func (suite *SynchronizerTestSuite) TestStallSnapshotIsOneObservation() { applied := make(chan struct{}) go func() { <-ready - sync.advance(time.Time{}) + sync.advance(time.Time{}, defaultPrecision) close(applied) }() runtime.Gosched() @@ -1892,7 +1920,7 @@ func (h *waitForSyncHarness) tipBlocks(tipPeer types.NodeID, interval time.Durat PeerID: tipPeer, Block: &types.Block{Header: types.Header{Height: height}}, })) - h.sync.advance(h.clock.Now()) + h.sync.advance(h.clock.Now(), defaultPrecision) } } @@ -2148,7 +2176,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAnUnservedClaimO h.sync.AddPeer(newPeerData(tip, 1, next-1+perRound)) case 2 * time.Second: for i := int64(0); i < perRound; i++ { - h.sync.advance(h.clock.Now()) + h.sync.advance(h.clock.Now(), defaultPrecision) } } }) @@ -2213,7 +2241,7 @@ func (h *waitForSyncHarness) flakySource(source types.NodeID, deliverEvery, fail Block: &types.Block{Header: types.Header{Height: next}}, })) // a block the network committed long ago - h.sync.advance(time.Time{}) + h.sync.advance(time.Time{}, defaultPrecision) h.sync.peerStore.Update(source, ResetFailures()) } } @@ -2439,7 +2467,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAFarClaimFromAPe PeerID: relay, Block: &types.Block{Header: types.Header{Height: next}}, })) - h.sync.advance(h.clock.Now()) + h.sync.advance(h.clock.Now(), defaultPrecision) h.sync.peerStore.Update(relay, ResetFailures()) }) suite.Require().True(ended, "block sync kept following the tip behind a claim of a peer supplying the tip") @@ -2504,7 +2532,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingOnAFastChainWhile PeerID: source, Block: &types.Block{Header: types.Header{Height: next}}, })) - h.sync.advance(last.Add(time.Duration(next-height+1) * interval)) + h.sync.advance(last.Add(time.Duration(next-height+1)*interval), defaultPrecision) } elapsed, _, ended := h.run(maxSyncStall/2, func(elapsed time.Duration) { if tc.supplyEvery < time.Second { From 4c57d66f5cf80e3cfacf8c0d3ac4a22433924421 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 11:57:59 +0000 Subject: [PATCH 10/36] docs(blocksync): state when the at-tip handover bound holds The two-minute bound holds once the node has reached the tip and while the next block keeps reaching it within its at-tip window. Describe the request order, how a peer delaying the next block keeps the node from the tip, with one node ID or several, and how peers that report their status first can cause the restart case. Co-Authored-By: Claude Opus 5.5 --- docs/tenderdash-core/block-sync/reactor.md | 49 +++++++++++++--------- 1 file changed, 30 insertions(+), 19 deletions(-) diff --git a/docs/tenderdash-core/block-sync/reactor.md b/docs/tenderdash-core/block-sync/reactor.md index e0dc760eb..5dbdb9bfd 100644 --- a/docs/tenderdash-core/block-sync/reactor.md +++ b/docs/tenderdash-core/block-sync/reactor.md @@ -56,9 +56,9 @@ would meet the scheduling allowance. Block sync hands over to consensus when it reaches the highest height any peer claims; when no peer advertises the next height and no block has been applied for 60 seconds; when the node has been at the tip of the chain for two minutes; -or when no block has been applied for 10 minutes and either the node is within -10 blocks of the highest claim that covers the next height, or no peer that -advertises the next height has backed its claim. +or when no block has been applied for 10 minutes and the node is within 10 +blocks of every claim that covers the next height and comes from a peer that +has backed it. Whether the node is at the tip is judged from the time of its last applied block, which the validators commit to and a peer cannot change: a node behind @@ -78,16 +78,20 @@ so they do not delay its own proposals. A peer backs its claim by supplying a block the node applies, and loses that credit when it fails a request (a timeout or other failure, or a -`NoBlockResponse`) or is dropped; a newly connected peer has none. Within a -10-minute stall every peer asked for the next block has failed it, so only a -peer not asked since it last supplied a block can hold the node in block sync. - -A peer that claims a far height therefore cannot keep a node at the tip in block -sync for more than two minutes after its last block becomes recent, however it -answers requests and whatever node IDs it uses. On a chain whose last block is -not recent, for instance one that has stopped producing blocks, the handover -takes 10 minutes, or until each peer that backed its claim earlier has been -asked for the next block and failed it, whichever is later. +`NoBlockResponse`) or is dropped; a newly connected peer has none. Block +requests go to peers that have backed their claims before any other, and a +height that has to be fetched again is requested before the node waits for a +peer to take a higher one. Within a 10-minute stall every peer asked for the +next block has failed it, so only a peer not asked since it last supplied a +block can hold the node in block sync. + +Once the node has reached the tip, a peer that claims a far height therefore +cannot keep it in block sync for more than two minutes, whatever node IDs it +uses, as long as the next block keeps reaching the node within its at-tip +window. On a chain whose last block is not recent, for instance one that has +stopped producing blocks, the handover takes 10 minutes, or until each peer +that backed its claim earlier has been asked for the next block and failed it, +whichever is later. Some cases remain open: @@ -96,15 +100,22 @@ Some cases remain open: minutes, less on a faster one), or behind them by more than the synchrony precision, does not see itself at the tip; while blocks keep arriving, a far claim keeps it in block sync. -- On a chain that commits more than 10 blocks in 70 seconds, a peer that answers - each request just before it times out holds the node more than 70 seconds - behind, and so in block sync, for as long as it does so. Such a node can also - hand over at the tip up to 70 seconds of blocks behind, which consensus - catch-up then closes. +- A peer that delays the next block beyond the node's at-tip window keeps the + node from reaching the tip, and so in block sync, for as long as it does so. + The block is first asked of a peer that advertises it, which before other + peers report it may be only the one claiming a far height; that costs up to + a request timeout, enough on a chain that commits more than 10 blocks in 70 + seconds. Several node IDs can each take a request for it in turn while no + peer that has backed its claim advertises the block - after a restart, or + while every such peer is busy - which is enough on slower chains too. +- On a chain that commits more than 10 blocks in 70 seconds, the node can hand + over at the tip up to 70 seconds of blocks behind, which consensus catch-up + then closes. - A node that restarts with a last block less than five minutes old and then applies no block for two minutes hands over at the tip; on a chain that commits more than 10 blocks in 70 seconds it may then be up to five minutes - of blocks behind. + of blocks behind. Besides a lack of peers, this can follow from peers that + report their status first and then hold the first requests. - A node any distance behind whose peers that advertise the next height fail every request for 10 minutes hands over, as it would if none advertised it. - After a handover on a stall, the catch-up target includes every claim, so From 3584fe70d61acf53776fcee79ab6e2156a7e8315 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 12:19:30 +0000 Subject: [PATCH 11/36] fix(blocksync): ask every peer for the height to apply once a request for it fails A retry of the height block sync waits to apply went to one peer at a time, the best ranked first, so peers listed first - or due a recovery probe - could each hold it until its request timed out while a peer able to serve it waited behind them. Nothing was applied meanwhile, no peer earned credit, and after 10 minutes the stall backstop handed a node far behind over to consensus. Once a request for that height fails, ask each other peer advertising it once, alongside the retry. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/block_fetch_job.go | 66 ++++++++++++++++++++-- internal/blocksync/block_fetch_job_test.go | 44 +++++++++++++++ internal/blocksync/peer_store.go | 21 ++++++- 3 files changed, 123 insertions(+), 8 deletions(-) diff --git a/internal/blocksync/block_fetch_job.go b/internal/blocksync/block_fetch_job.go index 3d23dd2b5..bc1a307b4 100644 --- a/internal/blocksync/block_fetch_job.go +++ b/internal/blocksync/block_fetch_job.go @@ -97,6 +97,12 @@ type jobGenerator struct { peerStore *InMemPeerStore height int64 pushedBack []int64 + // applyHeight is the height block sync waits to apply. Once a request for it + // fails, every peer that advertises it is asked for it; asked holds the + // peers asked since, and spreading says some may remain. + applyHeight int64 + asked map[types.NodeID]struct{} + spreading bool } func newJobGenerator(height int64, logger log.Logger, client client.BlockClient, peerStore *InMemPeerStore) *jobGenerator { @@ -110,21 +116,30 @@ func newJobGenerator(height int64, logger log.Logger, client client.BlockClient, } func (p *jobGenerator) nextHeight() int64 { + height, _ := p.next() + return height +} + +// next returns the next height to request and whether it is a retry. +func (p *jobGenerator) next() (int64, bool) { p.mtx.Lock() defer p.mtx.Unlock() if len(p.pushedBack) > 0 { height := p.pushedBack[0] p.pushedBack = p.pushedBack[1:] - return height + return height, true } height := p.height p.height++ - return height + return height, false } func (p *jobGenerator) nextJob(ctx context.Context) (*workerpool.Job, error) { + if job := p.spreadRetry(); job != nil { + return job, nil + } for { - height := p.nextHeight() + height, retry := p.next() peer, err := p.getPeer(ctx, height) if errors.Is(err, errLowerHeightQueued) { p.pushBack(height) @@ -133,10 +148,46 @@ func (p *jobGenerator) nextJob(ctx context.Context) (*workerpool.Job, error) { if err != nil { return nil, err } + if retry { + p.markAsked(height, peer.peerID) + } return workerpool.NewJob(blockFetchJobHandler(p.client, peer, height)), nil } } +// markAsked records that the height block sync waits to apply was asked of +// peerID again, and has it asked of every other peer that advertises it. One +// peer after another, peers listed first could each hold it until it times out. +func (p *jobGenerator) markAsked(height int64, peerID types.NodeID) { + p.mtx.Lock() + defer p.mtx.Unlock() + if height != p.applyHeight { + return + } + if p.asked == nil { + p.asked = map[types.NodeID]struct{}{} + } + p.asked[peerID] = struct{}{} + p.spreading = true +} + +// spreadRetry returns a job asking one more peer, not asked yet, for the +// height block sync waits to apply, or nil once there is none. +func (p *jobGenerator) spreadRetry() *workerpool.Job { + p.mtx.Lock() + defer p.mtx.Unlock() + if !p.spreading { + return nil + } + peer, found := p.peerStore.reservePeerExcept(p.applyHeight, p.asked) + if !found { + p.spreading = false + return nil + } + p.asked[peer.peerID] = struct{}{} + return workerpool.NewJob(blockFetchJobHandler(p.client, peer, p.applyHeight)) +} + // errLowerHeightQueued reports that a height below the one waiting for a peer // was queued for a retry meanwhile. var errLowerHeightQueued = errors.New("a lower height is queued") @@ -227,9 +278,12 @@ func (p *jobGenerator) getPeer(ctx context.Context, height int64) (PeerData, err // typically the very height everything else is waiting for, so holding those // back is what would turn a bound into a permanent stall. func (p *jobGenerator) shouldJobBeGenerated(applyHeight int64, pendingBytes int) bool { - p.mtx.RLock() - defer p.mtx.RUnlock() - if len(p.pushedBack) > 0 { + p.mtx.Lock() + defer p.mtx.Unlock() + if applyHeight != p.applyHeight { + p.applyHeight, p.asked, p.spreading = applyHeight, nil, false + } + if len(p.pushedBack) > 0 || p.spreading { return true } if pendingBytes >= maxPendingApplyBytes || p.height-applyHeight >= maxOutstandingHeights { diff --git a/internal/blocksync/block_fetch_job_test.go b/internal/blocksync/block_fetch_job_test.go index 17bbb2385..0d47bae1c 100644 --- a/internal/blocksync/block_fetch_job_test.go +++ b/internal/blocksync/block_fetch_job_test.go @@ -471,3 +471,47 @@ func (suite *BlockFetchJobTestSuite) TestSlowSuccessPrefersAnotherPeer() { suite.Require().True(syncer.peerStore.HasPeerForHeight(2)) suite.Require().Equal(2, syncer.peerStore.Len()) } + +// TestRetryOfTheHeightToApplyAsksEveryPeer checks that once a request for the +// height block sync waits to apply fails, that height is asked of every peer +// that advertises it at once, not of one peer after another. Peers listed first +// could otherwise each hold it until its request times out while a peer able to +// serve it waits behind them. +func TestRetryOfTheHeightToApplyAsksEveryPeer(t *testing.T) { + const height = int64(100) + peerIDs := []types.NodeID{"first", "second", "third", "fourth", "serving"} + store := NewInMemPeerStore() + for _, peerID := range peerIDs { + store.Upsert(newPeerData(peerID, 1, height+1000)) + } + asked := map[types.NodeID]int{} + blockClient := mocks.NewBlockClient(t) + blockClient. + On("GetBlock", mock.Anything, mock.Anything, mock.Anything). + Return(func(_ context.Context, h int64, peerID types.NodeID) *promise.Promise[*bcproto.BlockResponse] { + if h == height { + asked[peerID]++ + } + return promise.New(func(_ func(*bcproto.BlockResponse), reject func(error)) { + reject(client.ErrPeerNotResponded) + }) + }, nil) + gen := newJobGenerator(height+1, log.NewNopLogger(), blockClient, store) + require.True(t, gen.shouldJobBeGenerated(height, 0)) + + // the first request for height went to the first peer and failed + store.Update("first", AddNumPending(1)) + store.AddFailure("first", maxConsecutiveFailures) + gen.pushBack(height) + + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + for range 2 * len(peerIDs) { + job, err := gen.nextJob(ctx) + require.NoError(t, err) + job.Execute(ctx) + } + for _, peerID := range peerIDs { + require.Equal(t, 1, asked[peerID], "%s asked for the height %d times", peerID, asked[peerID]) + } +} diff --git a/internal/blocksync/peer_store.go b/internal/blocksync/peer_store.go index 46e6ff438..2df8490ae 100644 --- a/internal/blocksync/peer_store.go +++ b/internal/blocksync/peer_store.go @@ -131,8 +131,13 @@ func (p *InMemPeerStore) Query(spec store.QueryFunc[types.NodeID, PeerData], lim // FindPeer prefers a peer that supplied a block we applied since it last failed // a request, then a due recovery probe, then ordinary service. func (p *InMemPeerStore) FindPeer(height int64) (PeerData, bool) { + return p.findPeer(height, nil) +} + +// findPeer is FindPeer among peers not in except. +func (p *InMemPeerStore) findPeer(height int64, except map[types.NodeID]struct{}) (PeerData, bool) { peers := p.Query(store.AndX(peerNumPendingCond(maxPendingRequestsPerPeer, "<"), - heightBetweenPeerHeightRange(height)), 0) + heightBetweenPeerHeightRange(height), notIn(except)), 0) var selected PeerData priority := 0 for _, peer := range peers { @@ -151,7 +156,12 @@ func (p *InMemPeerStore) FindPeer(height int64) (PeerData, bool) { // reservePeer checks and reserves both pending capacity and the single probe // under the store lock. Concurrent producers cannot reserve the same probe. func (p *InMemPeerStore) reservePeer(height int64) (PeerData, bool) { - peer, found := p.FindPeer(height) + return p.reservePeerExcept(height, nil) +} + +// reservePeerExcept is reservePeer among peers not in except. +func (p *InMemPeerStore) reservePeerExcept(height int64, except map[types.NodeID]struct{}) (PeerData, bool) { + peer, found := p.findPeer(height, except) if !found { return PeerData{}, false } @@ -230,6 +240,13 @@ func peerNumPendingCond(val int32, op string) store.QueryFunc[types.NodeID, Peer } } +func notIn(except map[types.NodeID]struct{}) store.QueryFunc[types.NodeID, PeerData] { + return func(peerID types.NodeID, _ PeerData) bool { + _, found := except[peerID] + return !found + } +} + func heightBetweenPeerHeightRange(height int64) store.QueryFunc[types.NodeID, PeerData] { return func(peerID types.NodeID, peer PeerData) bool { return height >= peer.base && height <= peer.height && height > peer.missingThrough From 7a346dba6d839571f8cabab275c50464fe564ae7 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 12:21:04 +0000 Subject: [PATCH 12/36] fix(blocksync): credit the supplier of an applied block removed meanwhile The synchronizer lock is released while a block is applied, so the peer that supplied it can be removed, and its pending response dropped, in the meantime. The credit for the block was looked up from that response and so was lost. Pass the supplier of the applied response to advance. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/synchronizer.go | 12 +++--- internal/blocksync/synchronizer_test.go | 55 +++++++++++++++++++++---- 2 files changed, 53 insertions(+), 14 deletions(-) diff --git a/internal/blocksync/synchronizer.go b/internal/blocksync/synchronizer.go index 2048d2ff6..74e1a668b 100644 --- a/internal/blocksync/synchronizer.go +++ b/internal/blocksync/synchronizer.go @@ -751,7 +751,7 @@ func (s *Synchronizer) applyBlock(ctx context.Context) (BlockResponse, error) { return resp, fmt.Errorf("cannot apply block: %w", err) } state := s.applier.State() - s.advance(state.LastBlockTime, state.ConsensusParams.Synchrony.SynchronyParamsOrDefaults().Precision) + s.advance(resp.PeerID, state.LastBlockTime, state.ConsensusParams.Synchrony.SynchronyParamsOrDefaults().Precision) s.updateMonitor() } } @@ -764,14 +764,14 @@ func (s *Synchronizer) getPendingResponse() (BlockResponse, bool) { } // advance moves past the block just applied, which carries blockTime and left -// the chain with synchrony precision, and credits the peer that supplied it. -func (s *Synchronizer) advance(blockTime time.Time, precision time.Duration) { +// the chain with synchrony precision, and credits supplier, the peer that sent +// it. The pending response may be gone by now: the peer can be removed while +// its block is applied. +func (s *Synchronizer) advance(supplier types.NodeID, blockTime time.Time, precision time.Duration) { s.mtx.Lock() defer s.mtx.Unlock() s.precision = precision - if resp, ok := s.pendingToApply[s.height]; ok { - s.peerStore.Update(resp.PeerID, func(_ types.NodeID, peer *PeerData) { peer.supplied = true }) - } + s.peerStore.Update(supplier, func(_ types.NodeID, peer *PeerData) { peer.supplied = true }) s.blockTimes = append(s.blockTimes, blockTime) if len(s.blockTimes) > int(maxCatchupGap)+1 { s.blockTimes = s.blockTimes[1:] diff --git a/internal/blocksync/synchronizer_test.go b/internal/blocksync/synchronizer_test.go index 796a0ce92..85baa83af 100644 --- a/internal/blocksync/synchronizer_test.go +++ b/internal/blocksync/synchronizer_test.go @@ -1336,7 +1336,7 @@ func (suite *SynchronizerTestSuite) TestAdvanceCreditsTheSupplier() { pool.AddPeer(newPeerData("bystander", 1, 200)) blockTime := time.Unix(1_700_000_000, 0) pool.pendingToApply[100] = BlockResponse{PeerID: peerID} - pool.advance(blockTime, defaultPrecision) + pool.advance(peerID, blockTime, defaultPrecision) supplier, _ := pool.peerStore.Get(peerID) suite.True(supplier.supplied) bystander, _ := pool.peerStore.Get("bystander") @@ -1400,6 +1400,45 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncDoesNotWaitForABlockBeingAppl suite.Require().True(ended, "the handover loop waited for the block being applied") } +// TestApplyCreditsASupplierRemovedMeanwhile checks that a peer whose block is +// applied is credited for it, and the block's time recorded, even when the peer +// is removed and its pending response dropped while the block is being applied. +func (suite *SynchronizerTestSuite) TestApplyCreditsASupplierRemovedMeanwhile() { + const supplier = types.NodeID("supplier") + decoded, err := BlockResponseFromProto(suite.responses[0], supplier) + suite.Require().NoError(err) + resp := *decoded + state := suite.initialState.Copy() + state.LastBlockTime = resp.Block.Time + applier := newBlockApplier(suite.blockExec, suite.store, applierWithState(state)) + pool := NewSynchronizer(1, suite.client, applier) + pool.AddPeer(newPeerData(supplier, 1, 100)) + suite.blockExec. + On("ValidateBlock", mock.Anything, mock.Anything, mock.Anything, mock.Anything). + Return(nil) + suite.blockExec. + On("VerifyCommit", mock.Anything, mock.Anything, mock.Anything, mock.Anything). + Return(types.VerifiedCommit{}, nil) + expectApply(suite.blockExec, func(call *mock.Call) { call.Once() }) + suite.store. + On("SaveBlock", mock.Anything, mock.Anything, mock.Anything). + Run(func(mock.Arguments) { + // the peer is dropped, and reports its status again, mid-apply + pool.RemovePeer(supplier) + pool.AddPeer(newPeerData(supplier, 1, 100)) + }). + Once() + suite.Require().NoError(pool.addBlock(resp)) + + _, err = pool.applyBlock(context.Background()) + suite.Require().NoError(err) + peer, _ := pool.peerStore.Get(supplier) + suite.True(peer.supplied, "the peer whose block was applied must be credited") + st := pool.stallSnapshot() + suite.EqualValues(2, st.height) + suite.Equal(resp.Block.Time, st.lastBlockTime) +} + // TestStatusRefreshUpdatesAdvertisedRange checks that a status response still does // the one thing it is for: moving the range of blocks the peer claims to serve. It // is the guard on the tests above, which a synchronizer that ignored status @@ -1805,7 +1844,7 @@ func TestStallSnapshotTipWindow(t *testing.T) { t.Run(tc.name, func(t *testing.T) { sync := NewSynchronizer(1, nil, nil) for i := 0; i < tc.blocks; i++ { - sync.advance(start.Add(time.Duration(i)*tc.interval), defaultPrecision) + sync.advance("", start.Add(time.Duration(i)*tc.interval), defaultPrecision) } require.Equal(t, tc.want, sync.stallSnapshot().tipWindow) }) @@ -1832,7 +1871,7 @@ func (suite *SynchronizerTestSuite) TestStallSnapshotIsOneObservation() { applied := make(chan struct{}) go func() { <-ready - sync.advance(time.Time{}, defaultPrecision) + sync.advance("", time.Time{}, defaultPrecision) close(applied) }() runtime.Gosched() @@ -1920,7 +1959,7 @@ func (h *waitForSyncHarness) tipBlocks(tipPeer types.NodeID, interval time.Durat PeerID: tipPeer, Block: &types.Block{Header: types.Header{Height: height}}, })) - h.sync.advance(h.clock.Now(), defaultPrecision) + h.sync.advance(tipPeer, h.clock.Now(), defaultPrecision) } } @@ -2176,7 +2215,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAnUnservedClaimO h.sync.AddPeer(newPeerData(tip, 1, next-1+perRound)) case 2 * time.Second: for i := int64(0); i < perRound; i++ { - h.sync.advance(h.clock.Now(), defaultPrecision) + h.sync.advance("", h.clock.Now(), defaultPrecision) } } }) @@ -2241,7 +2280,7 @@ func (h *waitForSyncHarness) flakySource(source types.NodeID, deliverEvery, fail Block: &types.Block{Header: types.Header{Height: next}}, })) // a block the network committed long ago - h.sync.advance(time.Time{}, defaultPrecision) + h.sync.advance(source, time.Time{}, defaultPrecision) h.sync.peerStore.Update(source, ResetFailures()) } } @@ -2467,7 +2506,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAFarClaimFromAPe PeerID: relay, Block: &types.Block{Header: types.Header{Height: next}}, })) - h.sync.advance(h.clock.Now(), defaultPrecision) + h.sync.advance(relay, h.clock.Now(), defaultPrecision) h.sync.peerStore.Update(relay, ResetFailures()) }) suite.Require().True(ended, "block sync kept following the tip behind a claim of a peer supplying the tip") @@ -2532,7 +2571,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingOnAFastChainWhile PeerID: source, Block: &types.Block{Header: types.Header{Height: next}}, })) - h.sync.advance(last.Add(time.Duration(next-height+1)*interval), defaultPrecision) + h.sync.advance(source, last.Add(time.Duration(next-height+1)*interval), defaultPrecision) } elapsed, _, ended := h.run(maxSyncStall/2, func(elapsed time.Duration) { if tc.supplyEvery < time.Second { From 79c4d07c4229d073fb9fae1ea14695e959f8cfab Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 12:22:37 +0000 Subject: [PATCH 13/36] fix(blocksync): use one consensus catch-up target for every handover at the tip Only the at-tip handover left claims out of reach of the tip out of the target, and only those of peers still connected. A node at the tip handing over because no peer could serve the next block, or because the far claimant had gone, passed the far claim on and held back its own proposals for up to 10 minutes, while a near peer that left was dropped from the target. At the tip, every handover now targets the highest claim seen within maxCatchupGap of the node. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/synchronizer.go | 26 ++++++--- internal/blocksync/synchronizer_test.go | 76 +++++++++++++++++++++++++ 2 files changed, 93 insertions(+), 9 deletions(-) diff --git a/internal/blocksync/synchronizer.go b/internal/blocksync/synchronizer.go index 74e1a668b..7e0353ad8 100644 --- a/internal/blocksync/synchronizer.go +++ b/internal/blocksync/synchronizer.go @@ -444,7 +444,8 @@ func (s *Synchronizer) IsCaughtUp() bool { } // WaitForSync blocks until block sync is finished, and reports whether the node -// actually caught up and the highest peer target observed while waiting. +// actually caught up and the highest peer target observed while waiting; for a +// node at the tip, the highest within maxCatchupGap of it. // // A stall on its own is not a reason to stop. Handing over to consensus is a // one-way door - only the state sync path ever switches back to block sync - and @@ -467,6 +468,8 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe ticker := s.clock.NewTicker(switchToConsensusIntervalSeconds * time.Second) defer ticker.Stop() atTipSince := s.clock.Now() + // nearTarget is the highest claim seen within maxCatchupGap of the node. + var nearTarget int64 for { select { case <-ctx.Done(): @@ -475,13 +478,19 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe case <-ticker.Chan(): st := s.stallSnapshot() targetHeight = max(targetHeight, st.maxPeerHeight) - if st.hasPeers && st.height >= st.maxPeerHeight { - return true, targetHeight - } + nearTarget = max(nearTarget, st.maxNearHeight) now := s.clock.Now() - if !atTip(st.lastBlockTime, st.tipWindow, st.precision, now) { + // At the tip, claims out of reach of it must not hold back our own + // proposals, whichever way block sync ends. + target := targetHeight + if atTip(st.lastBlockTime, st.tipWindow, st.precision, now) { + target = nearTarget + } else { atTipSince = now } + if st.hasPeers && st.height >= st.maxPeerHeight { + return true, target + } verdict := stallVerdictFor(st.servable, st.maxSuppliedHeight-st.height, st.stalledFor, now.Sub(atTipSince)) switch verdict { @@ -492,8 +501,7 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe "max_peer_height", st.maxPeerHeight, "last_block_time", st.lastBlockTime, ) - // Claims out of reach of the tip must not hold back our own proposals. - return false, st.maxNearHeight + return false, target case stopNothingToFetch: if st.maxPeerHeight > st.height { // Peers claim to be ahead yet none of them holds the block we @@ -506,7 +514,7 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe "stalled_for", st.stalledFor, ) } - return false, targetHeight + return false, target case stopStalledTooLong: s.logger.Error( "block sync stalled for too long, handing over to consensus while still behind", @@ -516,7 +524,7 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe "max_backed_height", st.maxSuppliedHeight, "stalled_for", st.stalledFor, ) - return false, targetHeight + return false, target } if st.stalledFor > syncTimeout { s.logger.Error( diff --git a/internal/blocksync/synchronizer_test.go b/internal/blocksync/synchronizer_test.go index 85baa83af..f0827914b 100644 --- a/internal/blocksync/synchronizer_test.go +++ b/internal/blocksync/synchronizer_test.go @@ -2424,6 +2424,82 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncAtTipHandoverTargetsNearClaim suite.Require().Equal(applied-1, h.target, "a claim out of reach of the tip must not become the catch-up target") } +// TestWaitForSyncHandoverTargetAtTheTip checks that whichever way a node at the +// tip hands over, the consensus catch-up target holds the claims within reach +// of the tip that were seen while waiting, including those of peers gone since, +// and leaves out claims beyond it. +func (suite *SynchronizerTestSuite) TestWaitForSyncHandoverTargetAtTheTip() { + const ( + height = int64(100) + tipPeer = types.NodeID("tip peer") + claimant = types.NodeID("claimant") + nearPeer = types.NodeID("near peer") + nearClaim = height + 3 + ) + testCases := []struct { + name string + peers []PeerData + setup func(h *waitForSyncHarness) + step func(h *waitForSyncHarness, elapsed time.Duration) + want int64 + // caughtUp is set where no claim above the node remains + caughtUp bool + }{ + { + name: "the far claimant has no block for the next height", + peers: []PeerData{newPeerData(tipPeer, 1, height-1), newPeerData(claimant, 1, maxPlausiblePeerHeight)}, + setup: func(h *waitForSyncHarness) { + h.fail(claimant, height, &client.ErrBlockNotFound{Height: height}) + }, + want: height - 1, + }, + { + name: "the far claimant is removed for failing", + peers: []PeerData{newPeerData(tipPeer, 1, height-1), newPeerData(claimant, 1, maxPlausiblePeerHeight)}, + step: func(h *waitForSyncHarness, elapsed time.Duration) { + if elapsed == 5*time.Second { + for i := int64(0); i < int64(maxConsecutiveFailures); i++ { + h.fail(claimant, height+i, client.ErrPeerNotResponded) + } + } + }, + want: height - 1, + caughtUp: true, + }, + { + name: "a near peer leaves without failing", + peers: []PeerData{ + newPeerData(nearPeer, 1, nearClaim), + newPeerData(claimant, 1, maxPlausiblePeerHeight), + }, + step: func(h *waitForSyncHarness, elapsed time.Duration) { + if elapsed == 5*time.Second { + h.sync.RemovePeer(nearPeer) + } + }, + want: nearClaim, + }, + } + for _, tc := range testCases { + suite.Run(tc.name, func() { + suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) + h := suite.newWaitForSyncHarness(height, tc.peers...) + defer h.cancel() + if tc.setup != nil { + tc.setup(h) + } + _, caughtUp, ended := h.run(tipGrace+time.Minute, func(elapsed time.Duration) { + if tc.step != nil { + tc.step(h, elapsed) + } + }) + suite.Require().True(ended) + suite.Require().Equal(tc.caughtUp, caughtUp) + suite.Require().Equal(tc.want, h.target) + }) + } +} + // newPeerEveryMinute returns a step for run that, every minute, drops the peer // claiming a far height and connects a new one making the same claim. Node IDs // cost nothing, so no new peer has a history to judge it by. From bebe0ca8861d33db5276d1e3f6c3dae865c5c943 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 12:23:36 +0000 Subject: [PATCH 14/36] test(blocksync): keep syncing behind a sole source that fails now and then A node minutes behind a fast chain, with recent block times, does not hand over when its only source times out a request now and then: the at-tip window follows the chain's pace, not the source's credit. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/synchronizer_test.go | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/internal/blocksync/synchronizer_test.go b/internal/blocksync/synchronizer_test.go index f0827914b..d7574a89e 100644 --- a/internal/blocksync/synchronizer_test.go +++ b/internal/blocksync/synchronizer_test.go @@ -2612,6 +2612,9 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingOnAFastChainWhile quietBefore time.Duration // supplyEvery is how often the source supplies the next block supplyEvery time.Duration + // failEvery, when set, is how often the source, then the only peer, + // fails a request + failEvery time.Duration }{ {name: "applying at the chain's pace", behind: 100 * time.Second, supplyEvery: interval}, {name: "a slow source after a restart", behind: 100 * time.Second, supplyEvery: 15 * time.Second}, @@ -2621,12 +2624,20 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingOnAFastChainWhile quietBefore: 3 * time.Minute, supplyEvery: interval, }, + { + name: "a sole source that fails now and then", + behind: 3 * time.Minute, + supplyEvery: interval, + failEvery: 50 * time.Second, + }, } for _, tc := range testCases { suite.Run(tc.name, func() { - h := suite.newWaitForSyncHarness(height, - newPeerData(source, 1, height+10000), - newPeerData("claimant", 1, maxPlausiblePeerHeight)) + peers := []PeerData{newPeerData(source, 1, height+10000)} + if tc.failEvery == 0 { + peers = append(peers, newPeerData("claimant", 1, maxPlausiblePeerHeight)) + } + h := suite.newWaitForSyncHarness(height, peers...) defer h.cancel() start := h.clock.Now() last := start.Add(-tc.behind) @@ -2648,8 +2659,14 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingOnAFastChainWhile Block: &types.Block{Header: types.Header{Height: next}}, })) h.sync.advance(source, last.Add(time.Duration(next-height+1)*interval), defaultPrecision) + h.sync.peerStore.Update(source, ResetFailures()) } elapsed, _, ended := h.run(maxSyncStall/2, func(elapsed time.Duration) { + if tc.failEvery > 0 && elapsed > 0 && elapsed%tc.failEvery == 0 { + next, _ := h.sync.GetStatus() + h.fail(source, next, client.ErrPeerNotResponded) + return + } if tc.supplyEvery < time.Second { for i := 0; i < int(time.Second/tc.supplyEvery); i++ { supply() From b92bc638ed58ca266fa368e543beaeb051079c36 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 12:26:30 +0000 Subject: [PATCH 15/36] docs(blocksync): describe asking every peer for the next block and the target at the tip Co-Authored-By: Claude Opus 5.5 --- docs/tenderdash-core/block-sync/reactor.md | 36 ++++++++++++---------- 1 file changed, 20 insertions(+), 16 deletions(-) diff --git a/docs/tenderdash-core/block-sync/reactor.md b/docs/tenderdash-core/block-sync/reactor.md index 5dbdb9bfd..39d37eb0d 100644 --- a/docs/tenderdash-core/block-sync/reactor.md +++ b/docs/tenderdash-core/block-sync/reactor.md @@ -72,18 +72,20 @@ interval Dash Platform runs with and a margin for extra rounds and clock skew. A block time further ahead of the local clock than the synchrony precision does not count, since the clock is then behind the validators' and cannot tell how old the block is. Peer height claims play no part in this rule: a claim costs -nothing, and neither does the node ID it comes with. On this handover the -consensus catch-up target leaves out claims more than 10 blocks above the node, -so they do not delay its own proposals. +nothing, and neither does the node ID it comes with. Whenever a node at the tip +hands over, the consensus catch-up target is the highest claim seen within 10 +blocks of it, so claims further above do not delay its own proposals. A peer backs its claim by supplying a block the node applies, and loses that credit when it fails a request (a timeout or other failure, or a `NoBlockResponse`) or is dropped; a newly connected peer has none. Block requests go to peers that have backed their claims before any other, and a height that has to be fetched again is requested before the node waits for a -peer to take a higher one. Within a 10-minute stall every peer asked for the -next block has failed it, so only a peer not asked since it last supplied a -block can hold the node in block sync. +peer to take a higher one. Once a request for the next block fails, every +other peer that advertises it is asked for it as well, so peers listed or +ranked first cannot hold it one after another. Within a 10-minute stall every +peer asked for the next block has failed it, so only a peer not asked since it +last supplied a block can hold the node in block sync. Once the node has reached the tip, a peer that claims a far height therefore cannot keep it in block sync for more than two minutes, whatever node IDs it @@ -102,24 +104,26 @@ Some cases remain open: claim keeps it in block sync. - A peer that delays the next block beyond the node's at-tip window keeps the node from reaching the tip, and so in block sync, for as long as it does so. - The block is first asked of a peer that advertises it, which before other - peers report it may be only the one claiming a far height; that costs up to - a request timeout, enough on a chain that commits more than 10 blocks in 70 - seconds. Several node IDs can each take a request for it in turn while no - peer that has backed its claim advertises the block - after a restart, or - while every such peer is busy - which is enough on slower chains too. + The block is first asked of one peer that advertises it, which before other + peers report it may be only the one claiming a far height, or one that + earned credit by supplying earlier blocks; until that request times out and + the block is asked of every peer, up to about 70 seconds pass. That is + enough on a chain that commits a block more often than about every eight + seconds, and each peer that earns credit again can repeat it once. - On a chain that commits more than 10 blocks in 70 seconds, the node can hand over at the tip up to 70 seconds of blocks behind, which consensus catch-up then closes. - A node that restarts with a last block less than five minutes old and then applies no block for two minutes hands over at the tip; on a chain that commits more than 10 blocks in 70 seconds it may then be up to five minutes - of blocks behind. Besides a lack of peers, this can follow from peers that - report their status first and then hold the first requests. + of blocks behind. Peers that report their status first and hold the first + requests can delay the first block by about a request timeout, which alone + does not reach two minutes. - A node any distance behind whose peers that advertise the next height fail every request for 10 minutes hands over, as it would if none advertised it. -- After a handover on a stall, the catch-up target includes every claim, so - consensus may withhold the node's own proposals for up to 10 minutes. +- When a node not at the tip hands over on a stall, the catch-up target + includes every claim seen, so consensus may withhold the node's own + proposals for up to 10 minutes. The historical Blocksync reactor is organised as a set of concurrent tasks: From 85a905c52c20bd614d6b1372dd19e58bda99de14 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 12:25:37 +0000 Subject: [PATCH 16/36] test(blocksync): reproduce handover gaps behind late suppliers and failing peers Two tests that state the intended behaviour and fail on 4c57d66f5: - TestWaitForSyncHandsOverBehindPeersSupplyingEachBlockLate: on a chain committing a block every three minutes, peers that report first after a restart and supply each block about three minutes late are asked for every block, the prompt peer never, and the node stays in block sync. - TestWaitForSyncKeepsSyncingFarBehindPeersThatFailEveryRequest: a node far behind whose peers fail every request for ten minutes hands over to consensus (dashpay/tenderdash#1413). Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_01WunYYgi2uMsQDcBcKAwLdt --- .../blocksync/handover_late_supply_test.go | 225 ++++++++++++++++++ 1 file changed, 225 insertions(+) create mode 100644 internal/blocksync/handover_late_supply_test.go diff --git a/internal/blocksync/handover_late_supply_test.go b/internal/blocksync/handover_late_supply_test.go new file mode 100644 index 000000000..b321a7493 --- /dev/null +++ b/internal/blocksync/handover_late_supply_test.go @@ -0,0 +1,225 @@ +package blocksync + +import ( + "fmt" + "time" + + "github.com/stretchr/testify/mock" + + "github.com/dashpay/tenderdash/internal/p2p/client" + "github.com/dashpay/tenderdash/types" +) + +// lateSupplyRun drives a node that restarted a few blocks behind a chain +// committing a block every three minutes. Requests for the height the node waits +// for are handed out by the peer store, as the job generator does. The peers +// named late answer a request only once the block is lateBy old, or answer just +// inside the request deadline that they have no such block when that is further +// off; the peer named prompt answers at once. Every peer reports each block as soon as it is committed, and one more +// peer reports a far range it is never asked for. +type lateSupplyRun struct { + h *waitForSyncHarness + prompt types.NodeID + lateBy time.Duration + interval time.Duration + // blockTimes maps a committed height to its block time. + blockTimes map[int64]time.Time + tip int64 + nextCommit time.Time + // outstanding is the request for the height the node waits for, if any. + outstanding *lateRequest + // servedByPrompt counts the blocks the prompt peer was asked for. + servedByPrompt int + served int +} + +type lateRequest struct { + peer PeerData + request *serviceRequest + height int64 + endAt time.Time + deliver bool +} + +func (r *lateSupplyRun) ids() []types.NodeID { + var ids []types.NodeID + for _, peer := range r.h.sync.peerStore.All() { + if peer.base == 1 { + ids = append(ids, peer.peerID) + } + } + return ids +} + +func (r *lateSupplyRun) commit(blockTime time.Time) { + r.tip++ + r.blockTimes[r.tip] = blockTime + for _, id := range r.ids() { + r.h.sync.AddPeer(newPeerData(id, 1, r.tip)) + } +} + +func (r *lateSupplyRun) apply(peer PeerData, request *serviceRequest, height int64) { + sync := r.h.sync + request.Finished() + request.complete(1000) + sync.peerStore.Update(peer.peerID, AddNumPending(-1), ResetFailures()) + r.h.suite.Require().NoError(sync.addBlock(BlockResponse{ + PeerID: peer.peerID, + Block: &types.Block{Header: types.Header{Height: height}}, + })) + sync.advance(r.blockTimes[height], defaultPrecision) + r.served++ +} + +func (r *lateSupplyRun) step(time.Duration) { + sync := r.h.sync + now := r.h.clock.Now() + for !now.Before(r.nextCommit) { + r.commit(r.nextCommit) + r.nextCommit = r.nextCommit.Add(r.interval) + } + if out := r.outstanding; out != nil && !now.Before(out.endAt) { + r.outstanding = nil + if out.deliver { + r.apply(out.peer, out.request, out.height) + } else { + out.request.Finished() + out.request.complete(0) + r.h.fail(out.peer.peerID, out.height, &client.ErrBlockNotFound{Height: out.height}) + } + } + for r.outstanding == nil { + height, _ := sync.GetStatus() + if height > r.tip { + return + } + peer, found := sync.peerStore.reservePeer(height) + if !found { + return + } + request := peer.service.request() + request.Started() + if peer.peerID == r.prompt { + r.servedByPrompt++ + r.apply(peer, request, height) + continue + } + deliverAt := r.blockTimes[height].Add(r.lateBy) + if !deliverAt.After(now) { + r.apply(peer, request, height) + continue + } + out := &lateRequest{peer: peer, request: request, height: height} + if deliverAt.Sub(now) < syncTimeout { + out.endAt, out.deliver = deliverAt, true + } else { + out.endAt = now.Add(syncTimeout - time.Second) + } + r.outstanding = out + } +} + +// TestWaitForSyncHandsOverBehindPeersSupplyingEachBlockLate checks the handover +// of a node that restarts three blocks behind a chain committing a block every +// three minutes, with a prompt peer connected, while a far range is reported by +// a peer nobody asks. The node receives every block within its at-tip window. +// +// When the prompt peer reports first it supplies every block and the node hands +// over two minutes after reaching the tip. When a few peers that supply each +// block about three minutes late report first, they are asked for every block: +// before any block is applied no peer has backed its claim, a due recovery probe +// goes before a peer with no record, and each block then credits the late peer +// that supplied it. The prompt peer is never asked, and the node must still +// hand over. +func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverBehindPeersSupplyingEachBlockLate() { + const ( + height = int64(100) + prompt = types.NodeID("prompt peer") + interval = 3 * time.Minute + // late enough that less than tipGrace of the at-tip window is left + lateBy = tipRecency - tipGrace + 5*time.Second + limit = time.Hour + ) + for _, tc := range []struct { + name string + promptFirst bool + }{ + {name: "prompt peer reports first", promptFirst: true}, + {name: "late peers report first"}, + } { + suite.Run(tc.name, func() { + suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) + h := suite.newWaitForSyncHarness(height) + defer h.cancel() + start := h.clock.Now() + // the node was down for ten minutes + h.setLastBlockTime(start.Add(-10 * time.Minute)) + + r := &lateSupplyRun{ + h: h, + prompt: prompt, + lateBy: lateBy, + interval: interval, + blockTimes: map[int64]time.Time{}, + tip: height - 1, + } + if tc.promptFirst { + h.sync.AddPeer(newPeerData(prompt, 1, r.tip)) + } + for i := 0; i < 6; i++ { + h.sync.AddPeer(newPeerData(types.NodeID(fmt.Sprintf("late peer %d", i)), 1, r.tip)) + } + if !tc.promptFirst { + h.sync.AddPeer(newPeerData(prompt, 1, r.tip)) + } + h.sync.AddPeer(newPeerData("far range", 1_000_000_000, maxPlausiblePeerHeight)) + // the three blocks committed while the node was down + for _, age := range []time.Duration{7 * time.Minute, 4 * time.Minute, time.Minute} { + r.commit(start.Add(-age)) + } + r.nextCommit = start.Add(2 * time.Minute) + + elapsed, caughtUp, ended := h.run(limit, r.step) + suite.Require().True(ended, + "still in block sync after %s: %d blocks applied, each within the at-tip window, %d of them from the prompt peer", + elapsed, r.served, r.servedByPrompt) + suite.Require().False(caughtUp) + }) + } +} + +// TestWaitForSyncKeepsSyncingFarBehindPeersThatFailEveryRequest checks that a +// node far behind the network, by the time of its last block and by every +// claim, is not handed over to consensus because its peers, which earlier +// supplied blocks, fail every request for longer than the stall backstop. +// Consensus catch-up does not close such a gap, and block sync resumes as soon +// as a peer serves again (dashpay/tenderdash#1413). +func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingFarBehindPeersThatFailEveryRequest() { + const height = int64(24175) + suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) + var peers []PeerData + for i := 0; i < 5; i++ { + peer := newPeerData(types.NodeID(fmt.Sprintf("peer %d", i)), 1, 300_000) + // each supplied blocks the node applied before the stall + peer.supplied = true + peers = append(peers, peer) + } + h := suite.newWaitForSyncHarness(height, peers...) + defer h.cancel() + // the node is re-syncing history committed long ago + h.setLastBlockTime(h.clock.Now().Add(-365 * 24 * time.Hour)) + + elapsed, _, ended := h.run(maxSyncStall+2*time.Minute, func(elapsed time.Duration) { + if elapsed == 0 || elapsed%syncTimeout != 0 { + return + } + // every peer times out the request it holds and reports its range again + for _, peer := range peers { + h.fail(peer.peerID, height, client.ErrPeerNotResponded) + h.sync.AddPeer(newPeerData(peer.peerID, 1, 300_000)) + } + }) + suite.Require().False(ended, + "handed over after %s, 275825 blocks behind, although five peers still report the next height", elapsed) +} From f23a65fe4b1e4d89464602600aac0fb23e0eae21 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 12:37:15 +0000 Subject: [PATCH 17/36] fix(blocksync): count every claim in the stall backstop when the last block is old Taking the backstop's distance from backed claims alone let a node far back in the chain's history hand over to consensus once its peers had failed every request for 10 minutes (dashpay/tenderdash#1413). When the last applied block is more than an hour old, any servable claim within reach counts again, as before; the distance from backed claims applies to a node restarted while its chain is stopped. Also adapt the late-supply reproduction to the current advance signature. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/handover_late_supply_test.go | 2 +- internal/blocksync/handover_review_test.go | 2 +- internal/blocksync/reactor.go | 5 +++++ internal/blocksync/synchronizer.go | 9 +++++++-- internal/blocksync/synchronizer_test.go | 2 +- 5 files changed, 15 insertions(+), 5 deletions(-) diff --git a/internal/blocksync/handover_late_supply_test.go b/internal/blocksync/handover_late_supply_test.go index b321a7493..1f0640043 100644 --- a/internal/blocksync/handover_late_supply_test.go +++ b/internal/blocksync/handover_late_supply_test.go @@ -68,7 +68,7 @@ func (r *lateSupplyRun) apply(peer PeerData, request *serviceRequest, height int PeerID: peer.peerID, Block: &types.Block{Header: types.Header{Height: height}}, })) - sync.advance(r.blockTimes[height], defaultPrecision) + sync.advance(peer.peerID, r.blockTimes[height], defaultPrecision) r.served++ } diff --git a/internal/blocksync/handover_review_test.go b/internal/blocksync/handover_review_test.go index a27dd700d..1e30a6692 100644 --- a/internal/blocksync/handover_review_test.go +++ b/internal/blocksync/handover_review_test.go @@ -22,7 +22,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncStallBackstopIgnoresAFarClaim near.supplied = true h := suite.newWaitForSyncHarness(height, near, newPeerData("claimant", 1, maxPlausiblePeerHeight)) defer h.cancel() - h.setLastBlockTime(h.clock.Now().Add(-time.Hour)) + h.setLastBlockTime(h.clock.Now().Add(-staleBlockAge / 2)) elapsed, caughtUp, ended := h.run(maxSyncStall+time.Minute, nil) suite.Require().True(ended, "a far claim nobody backed kept a stalled node in block sync") diff --git a/internal/blocksync/reactor.go b/internal/blocksync/reactor.go index 8f116aec5..79ebae5af 100644 --- a/internal/blocksync/reactor.go +++ b/internal/blocksync/reactor.go @@ -60,6 +60,11 @@ const ( // tipGrace is how long the node must stay at the tip before higher height // claims stop keeping it in block sync. tipGrace = 2 * syncTimeout + + // staleBlockAge is how old the last applied block must be for every claim, + // backed or not, to count in the stall backstop. A node restarted while its + // chain is stopped, as during an upgrade, sees a younger one. + staleBlockAge = time.Hour ) type ReactorOption func(*Reactor) diff --git a/internal/blocksync/synchronizer.go b/internal/blocksync/synchronizer.go index 7e0353ad8..9ab2594ec 100644 --- a/internal/blocksync/synchronizer.go +++ b/internal/blocksync/synchronizer.go @@ -491,8 +491,13 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe if st.hasPeers && st.height >= st.maxPeerHeight { return true, target } - verdict := stallVerdictFor(st.servable, st.maxSuppliedHeight-st.height, st.stalledFor, - now.Sub(atTipSince)) + // Far back in the chain's history any claim counts, as nothing then + // suggests a chain that stopped at the node's height. + behindBy := st.maxSuppliedHeight - st.height + if now.Sub(st.lastBlockTime) > staleBlockAge { + behindBy = st.maxServableHeight - st.height + } + verdict := stallVerdictFor(st.servable, behindBy, st.stalledFor, now.Sub(atTipSince)) switch verdict { case stopAtTip: s.logger.Warn( diff --git a/internal/blocksync/synchronizer_test.go b/internal/blocksync/synchronizer_test.go index d7574a89e..f98188aa3 100644 --- a/internal/blocksync/synchronizer_test.go +++ b/internal/blocksync/synchronizer_test.go @@ -2548,7 +2548,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAFarClaimFromNew ) h := suite.newWaitForSyncHarness(height, newPeerData(tipPeer, 1, height-1)) defer h.cancel() - h.setLastBlockTime(h.clock.Now().Add(-time.Hour)) + h.setLastBlockTime(h.clock.Now().Add(-staleBlockAge / 2)) elapsed, caughtUp, ended := h.run(maxSyncStall+time.Minute, h.newPeerEveryMinute()) suite.Require().True(ended, "block sync kept waiting on a halted chain behind a claim moved to a new peer every minute") From 55b5c9e8ffdbeccf3f1b2bf76859146cab4cc5c9 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 12:37:44 +0000 Subject: [PATCH 18/36] test(blocksync): drive the stall backstop tests tick by tick from an old block Both tests moved the clock past the backstop in one jump from a recent last block, which skips the at-tip rule they would otherwise meet first. Start them from an old last block and step the clock a second at a time. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/synchronizer_test.go | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/internal/blocksync/synchronizer_test.go b/internal/blocksync/synchronizer_test.go index f98188aa3..b463d05c0 100644 --- a/internal/blocksync/synchronizer_test.go +++ b/internal/blocksync/synchronizer_test.go @@ -1708,37 +1708,35 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsGoingWhileAPeerCanServeU // that claims our height and never answers keeps the node in block sync // indefinitely unless a stall long enough to look like a wedge ends it. Within // reach of the tip consensus catch-up can cover what is left, so the backstop -// applies. +// applies. The last block is old, so the node is not at the tip. func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverAfterMaxSyncStall() { const height = int64(100) peer := newPeerData("peer in range", 1, height+maxCatchupGap) h := suite.newWaitForSyncHarness(height, peer) defer h.cancel() + h.setLastBlockTime(h.clock.Now().Add(-2 * staleBlockAge)) - h.clock.Advance(maxSyncStall + time.Second) - - caughtUp, ended := h.result(2 * time.Second) + elapsed, caughtUp, ended := h.run(maxSyncStall+time.Minute, nil) suite.Require().True(ended, "no progress for %s did not hand over to consensus", maxSyncStall) suite.Require().False(caughtUp, "handing over after a stall is not catching up") + suite.Require().Greater(elapsed, maxSyncStall) + suite.Require().LessOrEqual(elapsed, maxSyncStall+unservedHandoverSlack) } // TestWaitForSyncKeepsSyncingWhileFarBehind is the counterpart: the wall-clock // backstop must not hand over a node that is thousands of blocks behind. It // would arrive in consensus as a validator at a height the network committed // long ago (dashpay/tenderdash#1413), and consensus catch-up moves about one -// block per gossip cycle, so it would never reach the tip that way either. The -// peer far ahead has backed its claim by supplying blocks the node applied. +// block per gossip cycle, so it would never reach the tip that way either. func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingWhileFarBehind() { const height = int64(100) peer := newPeerData("peer far ahead", 1, height+maxCatchupGap+1) - peer.supplied = true h := suite.newWaitForSyncHarness(height, peer) defer h.cancel() + h.setLastBlockTime(h.clock.Now().Add(-2 * staleBlockAge)) - h.clock.Advance(maxSyncStall + time.Second) - - _, ended := h.result(200 * time.Millisecond) - suite.Require().False(ended, "block sync gave up while still %d blocks behind", maxCatchupGap+1) + elapsed, _, ended := h.run(maxSyncStall+time.Minute, nil) + suite.Require().False(ended, "block sync gave up after %s while still %d blocks behind", elapsed, maxCatchupGap+1) } func (suite *SynchronizerTestSuite) TestStallSnapshotServableHeight() { From 6331866f2531e027506f1124cb4b620f7de7cdc3 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 12:42:52 +0000 Subject: [PATCH 19/36] fix(blocksync): stop asking more peers for the height to apply within the window Once every peer advertising the height to apply had been asked for it, the producer was still told a job was due and took a new height past the outstanding window instead. Report a job only while a peer is left to ask. Drive the late-supply reproduction through the generator's retry handling, as the producer does: a request that fails has the block asked of every peer advertising it, so peers that report first and supply each block late no longer keep the node in block sync. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/block_fetch_job.go | 32 ++++--- internal/blocksync/block_fetch_job_test.go | 13 +++ .../blocksync/handover_late_supply_test.go | 85 +++++++++++++------ 3 files changed, 90 insertions(+), 40 deletions(-) diff --git a/internal/blocksync/block_fetch_job.go b/internal/blocksync/block_fetch_job.go index bc1a307b4..f802aef02 100644 --- a/internal/blocksync/block_fetch_job.go +++ b/internal/blocksync/block_fetch_job.go @@ -135,8 +135,8 @@ func (p *jobGenerator) next() (int64, bool) { } func (p *jobGenerator) nextJob(ctx context.Context) (*workerpool.Job, error) { - if job := p.spreadRetry(); job != nil { - return job, nil + if peer, height, found := p.nextSpread(); found { + return workerpool.NewJob(blockFetchJobHandler(p.client, peer, height)), nil } for { height, retry := p.next() @@ -149,16 +149,16 @@ func (p *jobGenerator) nextJob(ctx context.Context) (*workerpool.Job, error) { return nil, err } if retry { - p.markAsked(height, peer.peerID) + p.markRetried(height, peer.peerID) } return workerpool.NewJob(blockFetchJobHandler(p.client, peer, height)), nil } } -// markAsked records that the height block sync waits to apply was asked of +// markRetried records that the height block sync waits to apply was asked of // peerID again, and has it asked of every other peer that advertises it. One // peer after another, peers listed first could each hold it until it times out. -func (p *jobGenerator) markAsked(height int64, peerID types.NodeID) { +func (p *jobGenerator) markRetried(height int64, peerID types.NodeID) { p.mtx.Lock() defer p.mtx.Unlock() if height != p.applyHeight { @@ -171,21 +171,21 @@ func (p *jobGenerator) markAsked(height int64, peerID types.NodeID) { p.spreading = true } -// spreadRetry returns a job asking one more peer, not asked yet, for the -// height block sync waits to apply, or nil once there is none. -func (p *jobGenerator) spreadRetry() *workerpool.Job { +// nextSpread reserves one more peer, not asked yet, for the height block sync +// waits to apply, and reports false once there is none. +func (p *jobGenerator) nextSpread() (PeerData, int64, bool) { p.mtx.Lock() defer p.mtx.Unlock() if !p.spreading { - return nil + return PeerData{}, 0, false } peer, found := p.peerStore.reservePeerExcept(p.applyHeight, p.asked) if !found { p.spreading = false - return nil + return PeerData{}, 0, false } p.asked[peer.peerID] = struct{}{} - return workerpool.NewJob(blockFetchJobHandler(p.client, peer, p.applyHeight)) + return peer, p.applyHeight, true } // errLowerHeightQueued reports that a height below the one waiting for a peer @@ -283,7 +283,15 @@ func (p *jobGenerator) shouldJobBeGenerated(applyHeight int64, pendingBytes int) if applyHeight != p.applyHeight { p.applyHeight, p.asked, p.spreading = applyHeight, nil, false } - if len(p.pushedBack) > 0 || p.spreading { + // Asking one more peer is a job only while one is left to ask; otherwise + // nextJob would take a new height past the limits below. + if p.spreading { + if _, found := p.peerStore.findPeer(applyHeight, p.asked); found { + return true + } + p.spreading = false + } + if len(p.pushedBack) > 0 { return true } if pendingBytes >= maxPendingApplyBytes || p.height-applyHeight >= maxOutstandingHeights { diff --git a/internal/blocksync/block_fetch_job_test.go b/internal/blocksync/block_fetch_job_test.go index 0d47bae1c..87721883f 100644 --- a/internal/blocksync/block_fetch_job_test.go +++ b/internal/blocksync/block_fetch_job_test.go @@ -515,3 +515,16 @@ func TestRetryOfTheHeightToApplyAsksEveryPeer(t *testing.T) { require.Equal(t, 1, asked[peerID], "%s asked for the height %d times", peerID, asked[peerID]) } } + +// TestAskingEveryPeerEndsWithinTheWindow checks that once every peer has been +// asked for the height to apply, that is no reason to request another height: +// the request would be a new one beyond the outstanding window. +func TestAskingEveryPeerEndsWithinTheWindow(t *testing.T) { + const height = int64(100) + store := NewInMemPeerStore(newPeerData("only peer", 1, height+1000)) + gen := newJobGenerator(height+maxOutstandingHeights, log.NewNopLogger(), nil, store) + require.False(t, gen.shouldJobBeGenerated(height, 0), "the window is full") + + gen.markRetried(height, "only peer") + require.False(t, gen.shouldJobBeGenerated(height, 0), "no peer is left to ask") +} diff --git a/internal/blocksync/handover_late_supply_test.go b/internal/blocksync/handover_late_supply_test.go index 1f0640043..47b118e21 100644 --- a/internal/blocksync/handover_late_supply_test.go +++ b/internal/blocksync/handover_late_supply_test.go @@ -12,7 +12,8 @@ import ( // lateSupplyRun drives a node that restarted a few blocks behind a chain // committing a block every three minutes. Requests for the height the node waits -// for are handed out by the peer store, as the job generator does. The peers +// for are handed out as the job generator does: the first by the peer store, +// further ones by the generator once one fails or goes unanswered. The peers // named late answer a request only once the block is lateBy old, or answer just // inside the request deadline that they have no such block when that is further // off; the peer named prompt answers at once. Every peer reports each block as soon as it is committed, and one more @@ -26,8 +27,10 @@ type lateSupplyRun struct { blockTimes map[int64]time.Time tip int64 nextCommit time.Time - // outstanding is the request for the height the node waits for, if any. - outstanding *lateRequest + // outstanding are the requests for the height the node waits for. + outstanding []*lateRequest + // failed records the heights a request failed for. + failed map[int64]bool // servedByPrompt counts the blocks the prompt peer was asked for. servedByPrompt int served int @@ -64,6 +67,9 @@ func (r *lateSupplyRun) apply(peer PeerData, request *serviceRequest, height int request.Finished() request.complete(1000) sync.peerStore.Update(peer.peerID, AddNumPending(-1), ResetFailures()) + if applied, _ := sync.GetStatus(); height < applied { + return + } r.h.suite.Require().NoError(sync.addBlock(BlockResponse{ PeerID: peer.peerID, Block: &types.Block{Header: types.Header{Height: height}}, @@ -79,45 +85,67 @@ func (r *lateSupplyRun) step(time.Duration) { r.commit(r.nextCommit) r.nextCommit = r.nextCommit.Add(r.interval) } - if out := r.outstanding; out != nil && !now.Before(out.endAt) { - r.outstanding = nil - if out.deliver { + var waiting []*lateRequest + for _, out := range r.outstanding { + switch { + case now.Before(out.endAt): + waiting = append(waiting, out) + case out.deliver: r.apply(out.peer, out.request, out.height) - } else { + default: out.request.Finished() out.request.complete(0) + r.failed[out.height] = true r.h.fail(out.peer.peerID, out.height, &client.ErrBlockNotFound{Height: out.height}) } } - for r.outstanding == nil { + r.outstanding = waiting + for { height, _ := sync.GetStatus() if height > r.tip { return } - peer, found := sync.peerStore.reservePeer(height) - if !found { + sync.jobGen.shouldJobBeGenerated(height, 0) + peer, spreadHeight, found := sync.jobGen.nextSpread() + if found && spreadHeight != height { return } - request := peer.service.request() - request.Started() - if peer.peerID == r.prompt { - r.servedByPrompt++ - r.apply(peer, request, height) - continue - } - deliverAt := r.blockTimes[height].Add(r.lateBy) - if !deliverAt.After(now) { - r.apply(peer, request, height) - continue - } - out := &lateRequest{peer: peer, request: request, height: height} - if deliverAt.Sub(now) < syncTimeout { - out.endAt, out.deliver = deliverAt, true - } else { - out.endAt = now.Add(syncTimeout - time.Second) + if !found { + if len(r.outstanding) > 0 { + return + } + if peer, found = sync.peerStore.reservePeer(height); !found { + return + } + if r.failed[height] { + sync.jobGen.markRetried(height, peer.peerID) + } } - r.outstanding = out + r.ask(peer, height, now) + } +} + +// ask has peer answer a request for height as its kind does. +func (r *lateSupplyRun) ask(peer PeerData, height int64, now time.Time) { + request := peer.service.request() + request.Started() + if peer.peerID == r.prompt { + r.servedByPrompt++ + r.apply(peer, request, height) + return + } + deliverAt := r.blockTimes[height].Add(r.lateBy) + if !deliverAt.After(now) { + r.apply(peer, request, height) + return + } + out := &lateRequest{peer: peer, request: request, height: height} + if deliverAt.Sub(now) < syncTimeout { + out.endAt, out.deliver = deliverAt, true + } else { + out.endAt = now.Add(syncTimeout - time.Second) } + r.outstanding = append(r.outstanding, out) } // TestWaitForSyncHandsOverBehindPeersSupplyingEachBlockLate checks the handover @@ -162,6 +190,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverBehindPeersSupplying lateBy: lateBy, interval: interval, blockTimes: map[int64]time.Time{}, + failed: map[int64]bool{}, tip: height - 1, } if tc.promptFirst { From c1e3a20ce62956a2a73c884bb5d469b2bf6064c1 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 12:44:26 +0000 Subject: [PATCH 20/36] fix(blocksync): learn the chain's pace from the block store at start Until the first block was applied the at-tip window defaulted to five minutes, so a node restarting minutes behind a fast chain counted as at the tip. Seed the recent block times from the stored block metas below the last block, up to maxCatchupGap of them, stopping at the first one missing. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/synchronizer.go | 20 ++++++++++++- internal/blocksync/synchronizer_test.go | 39 +++++++++++++++++++++++++ 2 files changed, 58 insertions(+), 1 deletion(-) diff --git a/internal/blocksync/synchronizer.go b/internal/blocksync/synchronizer.go index 9ab2594ec..e88aa6b8a 100644 --- a/internal/blocksync/synchronizer.go +++ b/internal/blocksync/synchronizer.go @@ -4,6 +4,7 @@ import ( "context" "errors" "fmt" + "slices" "sync/atomic" "time" @@ -12,6 +13,7 @@ import ( "github.com/dashpay/tenderdash/internal/p2p" "github.com/dashpay/tenderdash/internal/p2p/client" + sm "github.com/dashpay/tenderdash/internal/state" "github.com/dashpay/tenderdash/libs/log" "github.com/dashpay/tenderdash/libs/service" "github.com/dashpay/tenderdash/libs/workerpool" @@ -252,7 +254,7 @@ func (s *Synchronizer) OnStart(ctx context.Context) error { s.lastMonitorUpdate = s.lastAdvance if s.applier != nil { state := s.applier.State() - s.blockTimes = []time.Time{state.LastBlockTime} + s.blockTimes = storedBlockTimes(s.applier.store, state.LastBlockHeight, state.LastBlockTime) s.precision = state.ConsensusParams.Synchrony.SynchronyParamsOrDefaults().Precision } s.ctx, s.cancel = context.WithCancel(ctx) @@ -273,6 +275,22 @@ func (s *Synchronizer) OnStart(ctx context.Context) error { return nil } +// storedBlockTimes returns the times of the blocks the store holds right below +// height, up to maxCatchupGap of them and oldest first, followed by lastTime, +// the time of the block at height. +func storedBlockTimes(store sm.BlockStore, height int64, lastTime time.Time) []time.Time { + times := []time.Time{lastTime} + for h := height - 1; store != nil && h >= 1 && h >= height-maxCatchupGap; h-- { + meta := store.LoadBlockMeta(h) + if meta == nil { + break + } + times = append(times, meta.Header.Time) + } + slices.Reverse(times) + return times +} + func (s *Synchronizer) OnStop() { s.cancel() s.workerPool.Stop(context.Background()) diff --git a/internal/blocksync/synchronizer_test.go b/internal/blocksync/synchronizer_test.go index b463d05c0..d8f44bae8 100644 --- a/internal/blocksync/synchronizer_test.go +++ b/internal/blocksync/synchronizer_test.go @@ -1367,6 +1367,45 @@ func (suite *SynchronizerTestSuite) TestStartSeedsTheLastBlockTime() { suite.Equal(2*time.Second, st.precision) } +// TestStartLearnsTheChainsPaceFromTheStore checks that block sync starts out +// knowing how fast the chain commits blocks from the blocks the store holds, so +// a node restarting minutes behind a fast chain is not taken for one at the +// tip. Only an unbroken run of stored blocks counts. +func (suite *SynchronizerTestSuite) TestStartLearnsTheChainsPaceFromTheStore() { + const lastHeight = int64(50) + lastTime := time.Unix(1_700_000_000, 0) + for _, tc := range []struct { + name string + missing int64 + want time.Duration + }{ + {name: "a fast chain", want: tipLag}, + {name: "the block before the last is not stored", missing: lastHeight - 1, want: tipRecency}, + } { + suite.Run(tc.name, func() { + store := mocks.NewBlockStore(suite.T()) + store.On("LoadBlockMeta", mock.Anything).Maybe().Return(func(height int64) *types.BlockMeta { + if height == tc.missing { + return nil + } + return &types.BlockMeta{Header: types.Header{ + Height: height, + Time: lastTime.Add(-time.Duration(lastHeight-height) * time.Second), + }} + }) + state := suite.initialState.Copy() + state.LastBlockHeight, state.LastBlockTime = lastHeight, lastTime + applier := newBlockApplier(suite.blockExec, store, applierWithState(state)) + pool := NewSynchronizer(lastHeight+1, suite.client, applier, WithWorkerPool(workerpool.New(0))) + suite.Require().NoError(pool.Start(context.Background())) + defer pool.Stop() + st := pool.stallSnapshot() + suite.Equal(lastTime, st.lastBlockTime) + suite.Equal(tc.want, st.tipWindow) + }) + } +} + // TestApplyingABlockTakesThePrecisionOfTheNewState checks that the synchrony // precision follows the chain's parameters as blocks are applied, not those it // had when block sync started. From 6b25314d2ca611909db4313ba8b31347fe14fcda Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 12:46:00 +0000 Subject: [PATCH 21/36] docs(blocksync): describe the current stall backstop rules and tie tipLag to the request deadline Update the comments on maxSyncStall, maxCatchupGap and stallVerdictFor to the backed-claim distance and the old-block exception, and build tipLag from the block client's request timeout, exported as client.RequestTimeout. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/reactor.go | 9 ++++++--- internal/blocksync/synchronizer.go | 11 ++++++----- internal/p2p/client/client.go | 3 +++ 3 files changed, 15 insertions(+), 8 deletions(-) diff --git a/internal/blocksync/reactor.go b/internal/blocksync/reactor.go index 79ebae5af..81a7235a3 100644 --- a/internal/blocksync/reactor.go +++ b/internal/blocksync/reactor.go @@ -35,10 +35,13 @@ const ( // hand over to consensus after this duration of inactivity, even when peers // still report higher blocks, so that a wedged synchronizer cannot keep the // node out of consensus forever - but only within maxCatchupGap of the tip + // as stallVerdictFor measures it maxSyncStall = 10 * time.Minute - // maxCatchupGap is how far behind the highest height any peer claims the node - // may be and still hand over to consensus on the stall backstop above. + // maxCatchupGap is how far behind the node may be and still hand over to + // consensus on the stall backstop above: behind the highest claim of a peer + // that supplied a block, or of any peer once the last block is older than + // staleBlockAge. It also bounds the claims a node at the tip targets. // // What is left after the handover is covered by consensus catch-up, which // moves about one block per gossip cycle: it closes a gap of a few blocks in @@ -55,7 +58,7 @@ const ( // tipLag is how far behind the network block sync runs at the tip: a status // round to learn of a new block and a request timeout to fetch it. - tipLag = statusUpdateIntervalSeconds*time.Second + syncTimeout + tipLag = statusUpdateIntervalSeconds*time.Second + client.RequestTimeout // tipGrace is how long the node must stay at the tip before higher height // claims stop keeping it in block sync. diff --git a/internal/blocksync/synchronizer.go b/internal/blocksync/synchronizer.go index e88aa6b8a..25de1816b 100644 --- a/internal/blocksync/synchronizer.go +++ b/internal/blocksync/synchronizer.go @@ -698,11 +698,12 @@ const ( // the backstop buys nothing and costs what dashpay/tenderdash#1413 describes, so // block sync keeps retrying and says so every interval. // -// Advertising costs nothing, so behindBy counts only the claims of peers that -// backed them by supplying a block we applied since they last failed a request; -// with no such peer the node is not known to be behind at all. A stall of -// maxSyncStall means every peer asked for the block meanwhile failed, so only -// one never asked keeps that credit, and a new peer has none. +// Advertising costs nothing, so near the tip behindBy counts only the claims of +// peers that backed them by supplying a block we applied since they last failed +// a request; with no such peer the node is not known to be behind at all. A +// stall of maxSyncStall means every peer asked for the block meanwhile failed, +// so only one never asked keeps that credit, and a new peer has none. With a +// last block older than staleBlockAge every claim counts. // // atTipFor is how long the node has been at the tip by the time of its last // block. Unlike stalledFor it is not reset by applying blocks, so a node diff --git a/internal/p2p/client/client.go b/internal/p2p/client/client.go index 6be04c25c..3049e9800 100644 --- a/internal/p2p/client/client.go +++ b/internal/p2p/client/client.go @@ -32,6 +32,9 @@ const ( ResponseIDAttribute = "ResponseID" ) +// RequestTimeout is how long a request waits for the peer's response. +const RequestTimeout = peerTimeout + const ( // peerTimeout leaves ample room for bounded stall detection and cleanup. peerTimeout = 60 * time.Second From 1ac0acae14ffecbc918945f6381cd6821beb3a3d Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 12:46:00 +0000 Subject: [PATCH 22/36] docs(blocksync): state the handover behaviour operators can expect Replace the per-mechanism list of open cases with what an operator sees: when the node hands over, how far behind it may be, and what to keep in order. Correct the at-tip bound to blocks arriving within the window minus two minutes, and describe the old-block rule and the pace taken from the block store. Co-Authored-By: Claude Opus 5.5 --- docs/tenderdash-core/block-sync/reactor.md | 90 +++++++++------------- 1 file changed, 35 insertions(+), 55 deletions(-) diff --git a/docs/tenderdash-core/block-sync/reactor.md b/docs/tenderdash-core/block-sync/reactor.md index 39d37eb0d..21c256583 100644 --- a/docs/tenderdash-core/block-sync/reactor.md +++ b/docs/tenderdash-core/block-sync/reactor.md @@ -58,72 +58,52 @@ claims; when no peer advertises the next height and no block has been applied for 60 seconds; when the node has been at the tip of the chain for two minutes; or when no block has been applied for 10 minutes and the node is within 10 blocks of every claim that covers the next height and comes from a peer that -has backed it. +has backed it. When the node's last block is more than an hour old, that last +rule counts every claim that covers the next height, backed or not, so a node +far behind keeps syncing while its peers fail requests and resumes as soon as +one serves again. Whether the node is at the tip is judged from the time of its last applied block, which the validators commit to and a peer cannot change: a node behind the network applies old blocks. The node is at the tip while that block is no older than the chain took to commit its last 10 blocks, or than 70 seconds (a status round and a request timeout) where that is longer, and in any case no -older than five minutes. Until the node has applied 10 blocks since it started, -the pace of those it has applied stands in for that; before the first, only -the five-minute limit applies. Five minutes covers the three-minute empty-block -interval Dash Platform runs with and a margin for extra rounds and clock skew. A -block time further ahead of the local clock than the synchrony precision does -not count, since the clock is then behind the validators' and cannot tell how -old the block is. Peer height claims play no part in this rule: a claim costs -nothing, and neither does the node ID it comes with. Whenever a node at the tip -hands over, the consensus catch-up target is the highest claim seen within 10 -blocks of it, so claims further above do not delay its own proposals. +older than five minutes. The pace of the chain is taken from the blocks in the +block store at start and from those applied since; with no earlier block +stored, only the five-minute limit applies until a block is applied. Five +minutes covers the three-minute empty-block interval Dash Platform runs with +and a margin for extra rounds and clock skew. A block time further ahead of the +local clock than the synchrony precision does not count. Peer height claims +play no part in this rule. Whenever a node at the tip hands over, the +consensus catch-up target is the highest claim seen within 10 blocks of it. A peer backs its claim by supplying a block the node applies, and loses that credit when it fails a request (a timeout or other failure, or a `NoBlockResponse`) or is dropped; a newly connected peer has none. Block -requests go to peers that have backed their claims before any other, and a -height that has to be fetched again is requested before the node waits for a -peer to take a higher one. Once a request for the next block fails, every -other peer that advertises it is asked for it as well, so peers listed or -ranked first cannot hold it one after another. Within a 10-minute stall every -peer asked for the next block has failed it, so only a peer not asked since it -last supplied a block can hold the node in block sync. - -Once the node has reached the tip, a peer that claims a far height therefore -cannot keep it in block sync for more than two minutes, whatever node IDs it -uses, as long as the next block keeps reaching the node within its at-tip -window. On a chain whose last block is not recent, for instance one that has -stopped producing blocks, the handover takes 10 minutes, or until each peer -that backed its claim earlier has been asked for the next block and failed it, -whichever is later. - -Some cases remain open: - +requests go to peers that have backed their claims first. Once a request for +the next block fails, every other peer that advertises it is asked for it as +well. + +What operators can expect: + +- A node that keeps receiving each next block while that block is less than + its at-tip window minus two minutes old hands over within two minutes of + reaching the tip, whatever heights peers claim. +- On a chain that has stopped producing blocks, a restarted node hands over + after 10 minutes without a block, or once every peer that supplied blocks + earlier has failed a request for the next one, whichever is later. +- A node may hand over a few blocks behind the network, up to about 70 + seconds of blocks on a fast chain, or up to five minutes of blocks after a + restart with no earlier block stored; consensus catch-up closes the rest. - A node whose clock runs ahead of the validators' by more than its at-tip - window leaves (about two minutes on a chain committing a block every three - minutes, less on a faster one), or behind them by more than the synchrony - precision, does not see itself at the tip; while blocks keep arriving, a far - claim keeps it in block sync. -- A peer that delays the next block beyond the node's at-tip window keeps the - node from reaching the tip, and so in block sync, for as long as it does so. - The block is first asked of one peer that advertises it, which before other - peers report it may be only the one claiming a far height, or one that - earned credit by supplying earlier blocks; until that request times out and - the block is asked of every peer, up to about 70 seconds pass. That is - enough on a chain that commits a block more often than about every eight - seconds, and each peer that earns credit again can repeat it once. -- On a chain that commits more than 10 blocks in 70 seconds, the node can hand - over at the tip up to 70 seconds of blocks behind, which consensus catch-up - then closes. -- A node that restarts with a last block less than five minutes old and then - applies no block for two minutes hands over at the tip; on a chain that - commits more than 10 blocks in 70 seconds it may then be up to five minutes - of blocks behind. Peers that report their status first and hold the first - requests can delay the first block by about a request timeout, which alone - does not reach two minutes. -- A node any distance behind whose peers that advertise the next height fail - every request for 10 minutes hands over, as it would if none advertised it. -- When a node not at the tip hands over on a stall, the catch-up target - includes every claim seen, so consensus may withhold the node's own - proposals for up to 10 minutes. + window, or behind them by more than the synchrony precision, does not see + itself at the tip and leaves block sync only by the other rules. Keep the + clock synchronised. +- After a handover on a stall away from the tip, the node may hold back its + own proposals for up to 10 minutes while consensus catches up. +- A node that stays in block sync while its blocks are recent, or hands over + far behind, is worth reporting with its logs: a handover below the highest + peer claim is logged at warning or error level with its reason. The historical Blocksync reactor is organised as a set of concurrent tasks: From 9aef4253ec6e3b9f9149f188a98a902190430e70 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 13:08:47 +0000 Subject: [PATCH 23/36] test(blocksync): check the handover behind a peer answering inside the deadline and on a long-halted chain Three tests that state the intended behaviour and fail on 1ac0acae1, and one that passes: - TestWaitForSyncHandsOverBehindAPeerAnsweringJustInsideTheDeadline: on a chain committing a block every five seconds, a peer that reports first after a restart and answers every request with the block one second inside the request deadline keeps the node in block sync; on a chain committing a block every three minutes the node hands over. Jobs are handed out through shouldJobBeGenerated, nextSpread, next, reservePeer and markRetried as produceJob and nextJob do. - TestWaitForSyncHandsOverPastAFarClaimFromNewPeersOnAChainHaltedForLong: a far claim moved to a new peer every minute keeps a node in block sync once the last block is about 50 minutes old or older. - TestRetryOfTheHeightToApplyAsksEveryPeerWhileTheProducerWaits: a failed request for the height to apply is not asked of the other peers when that height became the one to apply while the producer was waiting in nextJob for a higher one. - TestWaitForSyncFetchesEachBlockAtTheTipFromOnePeer (passes): a peer that reports a far height and has no block does not have every new block fetched from all peers. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_01WunYYgi2uMsQDcBcKAwLdt --- .../block_fetch_spread_review_test.go | 94 ++++++ internal/blocksync/handover_deadline_test.go | 310 ++++++++++++++++++ 2 files changed, 404 insertions(+) create mode 100644 internal/blocksync/block_fetch_spread_review_test.go create mode 100644 internal/blocksync/handover_deadline_test.go diff --git a/internal/blocksync/block_fetch_spread_review_test.go b/internal/blocksync/block_fetch_spread_review_test.go new file mode 100644 index 000000000..1f2fcb34f --- /dev/null +++ b/internal/blocksync/block_fetch_spread_review_test.go @@ -0,0 +1,94 @@ +package blocksync + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/mock" + "github.com/stretchr/testify/require" + + "github.com/dashpay/tenderdash/internal/p2p/client" + "github.com/dashpay/tenderdash/internal/p2p/client/mocks" + "github.com/dashpay/tenderdash/libs/log" + "github.com/dashpay/tenderdash/libs/promise" + "github.com/dashpay/tenderdash/libs/workerpool" + bcproto "github.com/dashpay/tenderdash/proto/tendermint/blocksync" + "github.com/dashpay/tenderdash/types" +) + +// TestRetryOfTheHeightToApplyAsksEveryPeerWhileTheProducerWaits checks that a +// failed request for the height block sync waits to apply is asked of every +// peer that advertises it when that height became the one to apply while the +// producer was waiting, inside nextJob, for a peer to advertise a higher one. +// That is where the producer of a node following the network spends its time +// while some peer reports a range further up. +func TestRetryOfTheHeightToApplyAsksEveryPeerWhileTheProducerWaits(t *testing.T) { + const ( + applying = int64(100) + next = applying + 1 + ) + peerIDs := []types.NodeID{"first", "second", "third"} + store := NewInMemPeerStore() + for _, peerID := range peerIDs { + store.Upsert(newPeerData(peerID, 1, next)) + } + store.Upsert(newPeerData("far range", 1_000_000_000, maxPlausiblePeerHeight)) + asked := map[types.NodeID]int{} + blockClient := mocks.NewBlockClient(t) + blockClient. + On("GetBlock", mock.Anything, mock.Anything, mock.Anything). + Return(func(_ context.Context, h int64, peerID types.NodeID) *promise.Promise[*bcproto.BlockResponse] { + if h == next { + asked[peerID]++ + } + return promise.New(func(_ func(*bcproto.BlockResponse), reject func(error)) { + reject(client.ErrPeerNotResponded) + }) + }, nil) + // applying and next are requested; the producer finds another job due while + // block sync waits for applying, and waits for a peer to advertise next+1 + gen := newJobGenerator(next+1, log.NewNopLogger(), blockClient, store) + require.True(t, gen.shouldJobBeGenerated(applying, 0)) + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + jobs := make(chan *jobOrErr, 1) + go func() { + job, err := gen.nextJob(ctx) + jobs <- &jobOrErr{job: job, err: err} + }() + time.Sleep(50 * time.Millisecond) + + // applying is applied meanwhile, and the request for next, sent to the + // first peer, fails + store.Update("first", AddNumPending(1)) + store.AddFailure("first", maxConsecutiveFailures) + gen.pushBack(next) + + // the producer goes on as produceJob does + res := <-jobs + require.NoError(t, res.err) + res.job.Execute(ctx) + for range 2 * len(peerIDs) { + if !gen.shouldJobBeGenerated(next, 0) { + break + } + jobCtx, jobCancel := context.WithTimeout(ctx, 200*time.Millisecond) + job, err := gen.nextJob(jobCtx) + jobCancel() + if err != nil { + // waiting for a peer to advertise next+1 again + break + } + job.Execute(ctx) + } + for _, peerID := range peerIDs[1:] { + require.Equal(t, 1, asked[peerID], "%s asked for the height to apply %d times (requests: %v)", + peerID, asked[peerID], asked) + } +} + +type jobOrErr struct { + job *workerpool.Job + err error +} diff --git a/internal/blocksync/handover_deadline_test.go b/internal/blocksync/handover_deadline_test.go new file mode 100644 index 000000000..cb9629666 --- /dev/null +++ b/internal/blocksync/handover_deadline_test.go @@ -0,0 +1,310 @@ +package blocksync + +import ( + "errors" + "fmt" + "time" + + "github.com/stretchr/testify/mock" + + "github.com/dashpay/tenderdash/internal/p2p/client" + "github.com/dashpay/tenderdash/types" +) + +// tipRun drives the producer and the consumer of a node following a chain that +// commits a block every interval. Jobs are handed out as produceJob and nextJob +// do: shouldJobBeGenerated admits them, nextSpread or next picks the height, and +// reservePeer picks the peer; a height no peer advertises yet is held until one +// does, as getPeer holds it. Responses are queued and applied in height order as +// consumeJobResult and applyBlock do. A peer listed in declining answers at +// once that it has no such block; any other answers with the block, a peer +// listed in hold that long after it is asked and the rest at once. The peers in +// reporting report the chain's height every status round. +type tipRun struct { + h *waitForSyncHarness + interval time.Duration + hold map[types.NodeID]time.Duration + declining map[types.NodeID]bool + // reporting are the peers that report the chain's height, in the order + // their reports arrive. + reporting []types.NodeID + + blockTimes map[int64]time.Time + tip int64 + nextCommit time.Time + inFlight []*tipRequest + // waiting is the height the producer holds until a peer advertises it. + waiting int64 + waitingRetry bool + + applied int + suppliedBy map[types.NodeID]int + // blocksSent counts the requests answered with a block. + blocksSent int + // oldest is the greatest age the last applied block reached. + oldest time.Duration +} + +type tipRequest struct { + peer PeerData + request *serviceRequest + height int64 + endAt time.Time +} + +func (r *tipRun) step(elapsed time.Duration) { + now := r.h.clock.Now() + for !now.Before(r.nextCommit) { + r.tip++ + r.blockTimes[r.tip] = r.nextCommit + r.nextCommit = r.nextCommit.Add(r.interval) + } + if elapsed%(statusUpdateIntervalSeconds*time.Second) == 0 { + for _, id := range r.reporting { + r.h.sync.AddPeer(newPeerData(id, 1, r.tip)) + } + } + for r.consume(now) || r.produce(now) { + } + st := r.h.sync.stallSnapshot() + // the age the next tick reads + r.oldest = max(r.oldest, now.Add(time.Second).Sub(st.lastBlockTime)) +} + +// consume hands the consumer every response due by now and applies what can be +// applied, and reports whether there was any. +func (r *tipRun) consume(now time.Time) bool { + sync := r.h.sync + var waiting []*tipRequest + consumed := false + for _, req := range r.inFlight { + if now.Before(req.endAt) { + waiting = append(waiting, req) + continue + } + consumed = true + r.blocksSent++ + req.request.Finished() + req.request.complete(1000) + sync.peerStore.Update(req.peer.peerID, AddNumPending(-1), ResetFailures()) + err := sync.addBlock(BlockResponse{ + PeerID: req.peer.peerID, + Block: &types.Block{Header: types.Header{Height: req.height}}, + }) + if err != nil && !errors.Is(err, errDuplicateBlock) { + r.h.suite.Require().NoError(err) + } + for { + resp, ok := sync.getPendingResponse() + if !ok { + break + } + sync.advance(resp.PeerID, r.blockTimes[resp.Block.Height], defaultPrecision) + r.applied++ + r.suppliedBy[resp.PeerID]++ + } + } + r.inFlight = waiting + return consumed +} + +// produce hands out every job due now and reports whether there was any. +func (r *tipRun) produce(now time.Time) bool { + gen := r.h.sync.jobGen + produced := false + for { + var ( + peer PeerData + height int64 + found bool + retry bool + ) + if r.waiting != 0 { + height, retry, r.waiting = r.waiting, r.waitingRetry, 0 + } else { + if !gen.shouldJobBeGenerated(r.h.sync.backlog()) { + return produced + } + if peer, height, found = gen.nextSpread(); !found { + height, retry = gen.next() + } + } + if !found { + if peer, found = r.h.sync.peerStore.reservePeer(height); !found { + if gen.lowerQueued(height) { + gen.pushBack(height) + continue + } + r.waiting, r.waitingRetry = height, retry + return produced + } + if retry { + gen.markRetried(height, peer.peerID) + } + } + request := peer.service.request() + request.Started() + produced = true + if r.declining[peer.peerID] { + request.Finished() + request.complete(0) + r.h.fail(peer.peerID, height, &client.ErrBlockNotFound{Height: height}) + continue + } + r.inFlight = append(r.inFlight, &tipRequest{ + peer: peer, + request: request, + height: height, + endAt: now.Add(r.hold[peer.peerID]), + }) + } +} + +// TestWaitForSyncHandsOverBehindAPeerAnsweringJustInsideTheDeadline checks the +// handover of a node that restarts three blocks behind the network, with a +// prompt peer connected, while a far range is reported by a peer nobody asks. +// One more peer answers every request with the block, one second inside the +// request deadline. It never fails a request, so no block is asked of every +// peer, and each block it supplies renews its credit. +// +// On a chain committing a block every three minutes the node hands over within +// minutes whichever peer reports first. On one committing a block every five +// seconds it must as well. +func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverBehindAPeerAnsweringJustInsideTheDeadline() { + const ( + height = int64(100) + prompt = types.NodeID("prompt peer") + slow = types.NodeID("peer answering inside the deadline") + behind = 3 + limit = time.Hour + ) + for _, tc := range []struct { + name string + interval time.Duration + reporting []types.NodeID + }{ + {name: "a block every 3m, the prompt peer reports first", interval: 3 * time.Minute, reporting: []types.NodeID{prompt, slow}}, + {name: "a block every 3m, the other peer reports first", interval: 3 * time.Minute, reporting: []types.NodeID{slow, prompt}}, + {name: "a block every 5s, the prompt peer reports first", interval: 5 * time.Second, reporting: []types.NodeID{prompt, slow}}, + {name: "a block every 5s, the other peer reports first", interval: 5 * time.Second, reporting: []types.NodeID{slow, prompt}}, + } { + suite.Run(tc.name, func() { + suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) + h := suite.newWaitForSyncHarness(height) + defer h.cancel() + start := h.clock.Now() + r := &tipRun{ + h: h, + interval: tc.interval, + hold: map[types.NodeID]time.Duration{slow: client.RequestTimeout - time.Second}, + reporting: tc.reporting, + blockTimes: map[int64]time.Time{}, + suppliedBy: map[types.NodeID]int{}, + tip: height - 1 + behind, + } + // the last committed block is two seconds old + for i := int64(0); i < behind; i++ { + r.blockTimes[r.tip-i] = start.Add(-2*time.Second - time.Duration(i)*tc.interval) + } + r.nextCommit = r.blockTimes[r.tip].Add(tc.interval) + // the block store holds the blocks the node applied before it stopped + var stored []time.Time + for i := int64(maxCatchupGap); i >= 0; i-- { + stored = append(stored, r.blockTimes[height].Add(-time.Duration(i+1)*tc.interval)) + } + h.sync.mtx.Lock() + h.sync.blockTimes = stored + h.sync.mtx.Unlock() + h.sync.AddPeer(newPeerData("far range", 1_000_000_000, maxPlausiblePeerHeight)) + + elapsed, caughtUp, ended := h.run(limit, r.step) + st := h.sync.stallSnapshot() + suite.Require().True(ended, + "still in block sync after %s: %d blocks applied (%s), the last applied block never older than %s, at-tip window %s", + elapsed, r.applied, fmt.Sprint(r.suppliedBy), r.oldest, st.tipWindow) + suite.Require().False(caughtUp) + suite.T().Logf("handed over after %s: %d blocks applied (%s), the last applied block never older than %s, at-tip window %s", + elapsed, r.applied, fmt.Sprint(r.suppliedBy), r.oldest, st.tipWindow) + }) + } +} + +// TestWaitForSyncFetchesEachBlockAtTheTipFromOnePeer checks how many times a +// node following the network fetches each new block while it is still in block +// sync. Five peers hold every block and answer at once. One more peer reports a +// far height and answers every request that it has no such block; it is the +// only peer advertising the next height until the others report it, so it is +// asked for it first. Its answer must not have every block fetched from all +// five peers. +// +// This holds only as long as the retry is taken before the height becomes the +// one to apply: the generator then does not ask the other peers (see +// TestRetryOfTheHeightToApplyAsksEveryPeerWhileTheProducerWaits). The first +// block, retried while it is the one to apply, is fetched five times. +func (suite *SynchronizerTestSuite) TestWaitForSyncFetchesEachBlockAtTheTipFromOnePeer() { + const ( + height = int64(100) + interval = 5 * time.Second + claimant = types.NodeID("claimant") + holders = 5 + ) + suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) + h := suite.newWaitForSyncHarness(height) + defer h.cancel() + start := h.clock.Now() + r := &tipRun{ + h: h, + interval: interval, + declining: map[types.NodeID]bool{claimant: true}, + blockTimes: map[int64]time.Time{height: start.Add(-2 * time.Second)}, + suppliedBy: map[types.NodeID]int{}, + tip: height, + nextCommit: start.Add(interval - 2*time.Second), + } + for i := 0; i < holders; i++ { + r.reporting = append(r.reporting, types.NodeID(fmt.Sprintf("holder %d", i))) + } + h.sync.AddPeer(newPeerData(claimant, 1, maxPlausiblePeerHeight)) + + elapsed, _, ended := h.run(tipGrace+time.Minute, r.step) + suite.Require().True(ended) + suite.T().Logf("%d blocks applied in %s, %d block responses fetched (%s)", r.applied, elapsed, r.blocksSent, fmt.Sprint(r.suppliedBy)) + suite.Require().Greater(r.applied, 10) + suite.Require().LessOrEqual(r.blocksSent, 2*r.applied, + "%d blocks applied in %s, %d block responses fetched from %d peers", + r.applied, elapsed, r.blocksSent, holders) +} + +// TestWaitForSyncHandsOverPastAFarClaimFromNewPeersOnAChainHaltedForLong checks +// that a far claim made by a new peer every minute does not keep a node in block +// sync on a chain that stopped producing blocks, however long ago it stopped. +// Validators restarted to bring such a chain back reach consensus only through +// this handover. +func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAFarClaimFromNewPeersOnAChainHaltedForLong() { + const ( + height = int64(100) + tipPeer = types.NodeID("tip peer") + ) + for _, tc := range []struct { + name string + // haltedFor is the age of the last block when the node starts + haltedFor time.Duration + }{ + {name: "half an hour", haltedFor: staleBlockAge / 2}, + {name: "55 minutes", haltedFor: staleBlockAge - maxSyncStall/2}, + {name: "two hours", haltedFor: 2 * staleBlockAge}, + } { + suite.Run(tc.name, func() { + suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) + h := suite.newWaitForSyncHarness(height, newPeerData(tipPeer, 1, height-1)) + defer h.cancel() + h.setLastBlockTime(h.clock.Now().Add(-tc.haltedFor)) + + elapsed, caughtUp, ended := h.run(maxSyncStall+time.Minute, h.newPeerEveryMinute()) + suite.Require().True(ended, + "still in block sync after %s on a chain halted for %s, behind a claim moved to a new peer every minute", + elapsed, tc.haltedFor) + suite.Require().False(caughtUp) + }) + } +} From ed01f131e57031d7a1891a8678b620c2793b7015 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 13:16:20 +0000 Subject: [PATCH 24/36] fix(blocksync): judge the old-block rule by a day, at the start of the stall With a one-hour limit read on every tick, a node restarted on a chain stopped for about 50 minutes or more counted every far claim in the stall backstop and stayed in block sync. Raise staleBlockAge to 24 hours and compare it with the age of the last block when the stall began, so the stall itself does not carry a stopped chain past it. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/handover_deadline_test.go | 9 +++++---- internal/blocksync/reactor.go | 8 ++++---- internal/blocksync/synchronizer.go | 6 +++--- 3 files changed, 12 insertions(+), 11 deletions(-) diff --git a/internal/blocksync/handover_deadline_test.go b/internal/blocksync/handover_deadline_test.go index cb9629666..71203e498 100644 --- a/internal/blocksync/handover_deadline_test.go +++ b/internal/blocksync/handover_deadline_test.go @@ -277,7 +277,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncFetchesEachBlockAtTheTipFromO // TestWaitForSyncHandsOverPastAFarClaimFromNewPeersOnAChainHaltedForLong checks // that a far claim made by a new peer every minute does not keep a node in block -// sync on a chain that stopped producing blocks, however long ago it stopped. +// sync on a chain that stopped producing blocks, up to staleBlockAge ago. // Validators restarted to bring such a chain back reach consensus only through // this handover. func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAFarClaimFromNewPeersOnAChainHaltedForLong() { @@ -290,9 +290,10 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAFarClaimFromNew // haltedFor is the age of the last block when the node starts haltedFor time.Duration }{ - {name: "half an hour", haltedFor: staleBlockAge / 2}, - {name: "55 minutes", haltedFor: staleBlockAge - maxSyncStall/2}, - {name: "two hours", haltedFor: 2 * staleBlockAge}, + {name: "half an hour", haltedFor: 30 * time.Minute}, + {name: "55 minutes", haltedFor: 55 * time.Minute}, + {name: "two hours", haltedFor: 2 * time.Hour}, + {name: "just under the stale age", haltedFor: staleBlockAge - time.Minute}, } { suite.Run(tc.name, func() { suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) diff --git a/internal/blocksync/reactor.go b/internal/blocksync/reactor.go index 81a7235a3..6daf43204 100644 --- a/internal/blocksync/reactor.go +++ b/internal/blocksync/reactor.go @@ -64,10 +64,10 @@ const ( // claims stop keeping it in block sync. tipGrace = 2 * syncTimeout - // staleBlockAge is how old the last applied block must be for every claim, - // backed or not, to count in the stall backstop. A node restarted while its - // chain is stopped, as during an upgrade, sees a younger one. - staleBlockAge = time.Hour + // staleBlockAge is how old the last applied block must be, when a stall + // begins, for every claim, backed or not, to count in the stall backstop. + // It is longer than any halt a restarted node is expected to help end. + staleBlockAge = 24 * time.Hour ) type ReactorOption func(*Reactor) diff --git a/internal/blocksync/synchronizer.go b/internal/blocksync/synchronizer.go index 25de1816b..39eaa480a 100644 --- a/internal/blocksync/synchronizer.go +++ b/internal/blocksync/synchronizer.go @@ -509,10 +509,10 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe if st.hasPeers && st.height >= st.maxPeerHeight { return true, target } - // Far back in the chain's history any claim counts, as nothing then - // suggests a chain that stopped at the node's height. + // Far back in the chain's history any claim counts. The age is taken + // when the stall began, so a stall does not age a stopped chain into it. behindBy := st.maxSuppliedHeight - st.height - if now.Sub(st.lastBlockTime) > staleBlockAge { + if now.Add(-st.stalledFor).Sub(st.lastBlockTime) > staleBlockAge { behindBy = st.maxServableHeight - st.height } verdict := stallVerdictFor(st.servable, behindBy, st.stalledFor, now.Sub(atTipSince)) From 3b2eb25f2541943ac5a1854650005ae8e0c88031 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 13:25:26 +0000 Subject: [PATCH 25/36] fix(blocksync): ask every peer for the height to apply once it goes unanswered A peer answering each request just inside the deadline could hold the height to apply for up to a minute without failing, which on a chain committing a block every few seconds kept the node from ever counting as at the tip. The generator now records, per height, when it was first requested, of whom, and whether a request for it failed. Once the height to apply has failed, or has gone unanswered for serviceGrace, it is asked of every peer that advertises it. The generator reads the height to apply from the synchronizer also while it waits for a peer inside nextJob, so a height that became the one to apply meanwhile is covered. A height nextJob took and could not request returns as the next new height rather than as a retry, a peer answering that it has no such block has the height asked of one other peer only, and a failed request for a block applied meanwhile is not queued again. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/block_fetch_job.go | 162 ++++++++++++++---- internal/blocksync/block_fetch_job_test.go | 8 +- internal/blocksync/handover_deadline_test.go | 14 +- .../blocksync/handover_late_supply_test.go | 8 +- internal/blocksync/synchronizer.go | 22 ++- 5 files changed, 159 insertions(+), 55 deletions(-) diff --git a/internal/blocksync/block_fetch_job.go b/internal/blocksync/block_fetch_job.go index f802aef02..c726b3d35 100644 --- a/internal/blocksync/block_fetch_job.go +++ b/internal/blocksync/block_fetch_job.go @@ -98,11 +98,22 @@ type jobGenerator struct { height int64 pushedBack []int64 // applyHeight is the height block sync waits to apply. Once a request for it - // fails, every peer that advertises it is asked for it; asked holds the - // peers asked since, and spreading says some may remain. + // fails, or goes unanswered for serviceGrace, every peer that advertises it + // is asked for it; spreading says some peer is left to ask. requests holds + // what was asked of whom for each height from applyHeight up. applyHeight int64 - asked map[types.NodeID]struct{} spreading bool + requests map[int64]*heightRequests + // applyState, when set, reports the height to apply and whether its block + // has arrived, so that a wait for a peer inside nextJob sees them. + applyState func() (height int64, received bool) +} + +type heightRequests struct { + // since is when the height was first requested. + since time.Time + asked map[types.NodeID]struct{} + failed bool } func newJobGenerator(height int64, logger log.Logger, client client.BlockClient, peerStore *InMemPeerStore) *jobGenerator { @@ -135,56 +146,100 @@ func (p *jobGenerator) next() (int64, bool) { } func (p *jobGenerator) nextJob(ctx context.Context) (*workerpool.Job, error) { - if peer, height, found := p.nextSpread(); found { - return workerpool.NewJob(blockFetchJobHandler(p.client, peer, height)), nil - } for { + if peer, height, found := p.nextSpread(); found { + return workerpool.NewJob(blockFetchJobHandler(p.client, peer, height)), nil + } height, retry := p.next() peer, err := p.getPeer(ctx, height) if errors.Is(err, errLowerHeightQueued) { - p.pushBack(height) + p.giveBack(height, retry) continue } if err != nil { return nil, err } - if retry { - p.markRetried(height, peer.peerID) - } + p.markRequested(height, peer.peerID) return workerpool.NewJob(blockFetchJobHandler(p.client, peer, height)), nil } } -// markRetried records that the height block sync waits to apply was asked of -// peerID again, and has it asked of every other peer that advertises it. One -// peer after another, peers listed first could each hold it until it times out. -func (p *jobGenerator) markRetried(height int64, peerID types.NodeID) { +// giveBack returns a height taken by next and not requested, as a retry if it +// was one and otherwise as the next new height. +func (p *jobGenerator) giveBack(height int64, retry bool) { p.mtx.Lock() defer p.mtx.Unlock() - if height != p.applyHeight { + if !retry && p.height == height+1 { + p.height = height return } - if p.asked == nil { - p.asked = map[types.NodeID]struct{}{} + p.queue(height) +} + +// markRequested records that height was asked of peerID. +func (p *jobGenerator) markRequested(height int64, peerID types.NodeID) { + p.mtx.Lock() + defer p.mtx.Unlock() + if rec := p.requestsFor(height); rec != nil { + rec.asked[peerID] = struct{}{} + } +} + +// requestsFor returns the record of requests for height, created if missing, +// or nil for a height below the one to apply. The caller holds p.mtx. +func (p *jobGenerator) requestsFor(height int64) *heightRequests { + if height < p.applyHeight { + return nil + } + rec, found := p.requests[height] + if !found { + if p.requests == nil { + p.requests = map[int64]*heightRequests{} + } + rec = &heightRequests{since: p.peerStore.clock.Now(), asked: map[types.NodeID]struct{}{}} + p.requests[height] = rec + } + return rec +} + +// observe takes note of the height to apply and whether its block has arrived, +// and decides whether to ask every peer that advertises it: once a request for +// it failed or has gone unanswered for serviceGrace. One peer after another, +// peers ranked first could each hold it until it times out. The caller holds +// p.mtx. +func (p *jobGenerator) observe(applyHeight int64, received bool) { + if applyHeight > p.applyHeight { + p.applyHeight = applyHeight + for height := range p.requests { + if height < applyHeight { + delete(p.requests, height) + } + } + } + rec := p.requests[p.applyHeight] + p.spreading = false + if rec == nil || received || !rec.failed && p.peerStore.clock.Since(rec.since) <= serviceGrace { + return } - p.asked[peerID] = struct{}{} - p.spreading = true + _, p.spreading = p.peerStore.findPeer(p.applyHeight, rec.asked) } // nextSpread reserves one more peer, not asked yet, for the height block sync -// waits to apply, and reports false once there is none. +// waits to apply, and reports false once there is none. A retry of that height +// queued after a failure goes first. func (p *jobGenerator) nextSpread() (PeerData, int64, bool) { p.mtx.Lock() defer p.mtx.Unlock() - if !p.spreading { + if !p.spreading || len(p.pushedBack) > 0 && p.pushedBack[0] == p.applyHeight { return PeerData{}, 0, false } - peer, found := p.peerStore.reservePeerExcept(p.applyHeight, p.asked) + rec := p.requestsFor(p.applyHeight) + peer, found := p.peerStore.reservePeerExcept(p.applyHeight, rec.asked) if !found { p.spreading = false return PeerData{}, 0, false } - p.asked[peer.peerID] = struct{}{} + rec.asked[peer.peerID] = struct{}{} return peer, p.applyHeight, true } @@ -192,9 +247,10 @@ func (p *jobGenerator) nextSpread() (PeerData, int64, bool) { // was queued for a retry meanwhile. var errLowerHeightQueued = errors.New("a lower height is queued") -// pushBack schedules heights to be fetched again, skipping those already queued so -// a queued height is not fetched twice. Heights already handed to a worker are not -// tracked here, so an in-flight height can still be queued and fetched again. +// pushBack schedules heights whose requests failed to be fetched again, skipping +// those already queued so a queued height is not fetched twice. Heights already +// handed to a worker are not tracked here, so an in-flight height can still be +// queued and fetched again. // // The queue is kept sorted because nextHeight pops from the front and callers hand // us heights in arbitrary order - dropPeer collects them by ranging a map. Only the @@ -209,7 +265,24 @@ func (p *jobGenerator) pushBack(heights ...int64) { } p.mtx.Lock() defer p.mtx.Unlock() + for _, height := range heights { + if rec := p.requestsFor(height); rec != nil { + rec.failed = true + } + } + p.queue(heights...) +} + +// requeue schedules height to be fetched again after a peer answered that it +// has no such block. Another peer is asked; that alone is no reason to ask all. +func (p *jobGenerator) requeue(height int64) { + p.mtx.Lock() + defer p.mtx.Unlock() + p.queue(height) +} +// queue adds heights to the retry queue. The caller holds p.mtx. +func (p *jobGenerator) queue(heights ...int64) { queued := make(map[int64]struct{}, len(p.pushedBack)+len(heights)) for _, height := range p.pushedBack { queued[height] = struct{}{} @@ -225,10 +298,23 @@ func (p *jobGenerator) pushBack(heights ...int64) { slices.Sort(p.pushedBack) } +// lowerQueued reports whether a height below height is due to be requested: a +// retry, or the height to apply to be asked of more peers. func (p *jobGenerator) lowerQueued(height int64) bool { - p.mtx.RLock() - defer p.mtx.RUnlock() - return len(p.pushedBack) > 0 && p.pushedBack[0] < height + var ( + applyHeight int64 + received bool + ) + if p.applyState != nil { + applyHeight, received = p.applyState() + } + p.mtx.Lock() + defer p.mtx.Unlock() + if p.applyState != nil { + p.observe(applyHeight, received) + } + return len(p.pushedBack) > 0 && p.pushedBack[0] < height || + p.spreading && p.applyHeight < height } func (p *jobGenerator) getPeer(ctx context.Context, height int64) (PeerData, error) { @@ -254,7 +340,7 @@ func (p *jobGenerator) getPeer(ctx context.Context, height int64) (PeerData, err // shouldJobBeGenerated reports whether another height may be requested now. // applyHeight is the height block sync is waiting to apply and pendingBytes is -// the size of the responses already held above it. +// the size of the responses already held. // // Blocks are applied strictly in order, so a response that arrives before the // height below it has been applied is held until that one does, and nothing @@ -278,18 +364,18 @@ func (p *jobGenerator) getPeer(ctx context.Context, height int64) (PeerData, err // typically the very height everything else is waiting for, so holding those // back is what would turn a bound into a permanent stall. func (p *jobGenerator) shouldJobBeGenerated(applyHeight int64, pendingBytes int) bool { + received := false + if p.applyState != nil { + if height, arrived := p.applyState(); height == applyHeight { + received = arrived + } + } p.mtx.Lock() defer p.mtx.Unlock() - if applyHeight != p.applyHeight { - p.applyHeight, p.asked, p.spreading = applyHeight, nil, false - } // Asking one more peer is a job only while one is left to ask; otherwise // nextJob would take a new height past the limits below. - if p.spreading { - if _, found := p.peerStore.findPeer(applyHeight, p.asked); found { - return true - } - p.spreading = false + if p.observe(applyHeight, received); p.spreading { + return true } if len(p.pushedBack) > 0 { return true diff --git a/internal/blocksync/block_fetch_job_test.go b/internal/blocksync/block_fetch_job_test.go index 87721883f..4ce129760 100644 --- a/internal/blocksync/block_fetch_job_test.go +++ b/internal/blocksync/block_fetch_job_test.go @@ -507,6 +507,9 @@ func TestRetryOfTheHeightToApplyAsksEveryPeer(t *testing.T) { ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) defer cancel() for range 2 * len(peerIDs) { + if !gen.shouldJobBeGenerated(height, 0) { + break + } job, err := gen.nextJob(ctx) require.NoError(t, err) job.Execute(ctx) @@ -522,9 +525,12 @@ func TestRetryOfTheHeightToApplyAsksEveryPeer(t *testing.T) { func TestAskingEveryPeerEndsWithinTheWindow(t *testing.T) { const height = int64(100) store := NewInMemPeerStore(newPeerData("only peer", 1, height+1000)) + clock := clockwork.NewFakeClock() + store.clock = clock gen := newJobGenerator(height+maxOutstandingHeights, log.NewNopLogger(), nil, store) require.False(t, gen.shouldJobBeGenerated(height, 0), "the window is full") - gen.markRetried(height, "only peer") + gen.markRequested(height, "only peer") + clock.Advance(serviceGrace + time.Second) require.False(t, gen.shouldJobBeGenerated(height, 0), "no peer is left to ask") } diff --git a/internal/blocksync/handover_deadline_test.go b/internal/blocksync/handover_deadline_test.go index 71203e498..139b16d75 100644 --- a/internal/blocksync/handover_deadline_test.go +++ b/internal/blocksync/handover_deadline_test.go @@ -132,15 +132,13 @@ func (r *tipRun) produce(now time.Time) bool { if !found { if peer, found = r.h.sync.peerStore.reservePeer(height); !found { if gen.lowerQueued(height) { - gen.pushBack(height) + gen.giveBack(height, retry) continue } r.waiting, r.waitingRetry = height, retry return produced } - if retry { - gen.markRetried(height, peer.peerID) - } + gen.markRequested(height, peer.peerID) } request := peer.service.request() request.Started() @@ -165,10 +163,10 @@ func (r *tipRun) produce(now time.Time) bool { // prompt peer connected, while a far range is reported by a peer nobody asks. // One more peer answers every request with the block, one second inside the // request deadline. It never fails a request, so no block is asked of every -// peer, and each block it supplies renews its credit. +// peer at once, and each block it supplies renews its credit. // // On a chain committing a block every three minutes the node hands over within -// minutes whichever peer reports first. On one committing a block every five +// minutes whichever peer reports first. On one committing a block every few // seconds it must as well. func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverBehindAPeerAnsweringJustInsideTheDeadline() { const ( @@ -187,6 +185,8 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverBehindAPeerAnswering {name: "a block every 3m, the other peer reports first", interval: 3 * time.Minute, reporting: []types.NodeID{slow, prompt}}, {name: "a block every 5s, the prompt peer reports first", interval: 5 * time.Second, reporting: []types.NodeID{prompt, slow}}, {name: "a block every 5s, the other peer reports first", interval: 5 * time.Second, reporting: []types.NodeID{slow, prompt}}, + {name: "a block every 2s, the other peer reports first", interval: 2 * time.Second, reporting: []types.NodeID{slow, prompt}}, + {name: "a block every second, the other peer reports first", interval: time.Second, reporting: []types.NodeID{slow, prompt}}, } { suite.Run(tc.name, func() { suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) @@ -209,7 +209,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverBehindAPeerAnswering r.nextCommit = r.blockTimes[r.tip].Add(tc.interval) // the block store holds the blocks the node applied before it stopped var stored []time.Time - for i := int64(maxCatchupGap); i >= 0; i-- { + for i := maxCatchupGap; i >= 0; i-- { stored = append(stored, r.blockTimes[height].Add(-time.Duration(i+1)*tc.interval)) } h.sync.mtx.Lock() diff --git a/internal/blocksync/handover_late_supply_test.go b/internal/blocksync/handover_late_supply_test.go index 47b118e21..519ab97e7 100644 --- a/internal/blocksync/handover_late_supply_test.go +++ b/internal/blocksync/handover_late_supply_test.go @@ -29,8 +29,6 @@ type lateSupplyRun struct { nextCommit time.Time // outstanding are the requests for the height the node waits for. outstanding []*lateRequest - // failed records the heights a request failed for. - failed map[int64]bool // servedByPrompt counts the blocks the prompt peer was asked for. servedByPrompt int served int @@ -95,7 +93,6 @@ func (r *lateSupplyRun) step(time.Duration) { default: out.request.Finished() out.request.complete(0) - r.failed[out.height] = true r.h.fail(out.peer.peerID, out.height, &client.ErrBlockNotFound{Height: out.height}) } } @@ -117,9 +114,7 @@ func (r *lateSupplyRun) step(time.Duration) { if peer, found = sync.peerStore.reservePeer(height); !found { return } - if r.failed[height] { - sync.jobGen.markRetried(height, peer.peerID) - } + sync.jobGen.markRequested(height, peer.peerID) } r.ask(peer, height, now) } @@ -190,7 +185,6 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverBehindPeersSupplying lateBy: lateBy, interval: interval, blockTimes: map[int64]time.Time{}, - failed: map[int64]bool{}, tip: height - 1, } if tc.promptFirst { diff --git a/internal/blocksync/synchronizer.go b/internal/blocksync/synchronizer.go index 39eaa480a..15879ad70 100644 --- a/internal/blocksync/synchronizer.go +++ b/internal/blocksync/synchronizer.go @@ -219,6 +219,7 @@ func NewSynchronizer(start int64, client client.BlockClient, blockExec *blockApp opt(bp) } bp.peerStore.clock = bp.clock + bp.jobGen.applyState = bp.applyState bp.BaseService = *service.NewBaseService(logger, "Synchronizer", bp) return bp } @@ -353,13 +354,21 @@ func (s *Synchronizer) consumeJobResult(ctx, applyCtx context.Context) error { if !errors.As(res.Err, &bfErr) { return nil } + // A block applied meanwhile, typically from another peer asked for it + // as well, is not fetched again. + applyHeight, _ := s.GetStatus() + pending := bfErr.height >= applyHeight var missing *client.ErrBlockNotFound if errors.As(bfErr.err, &missing) && missing.Height == bfErr.height { s.peerStore.RecordMissingBlock(bfErr.peerID, bfErr.height) - s.jobGen.pushBack(bfErr.height) + if pending { + s.jobGen.requeue(bfErr.height) + } return nil } - s.jobGen.pushBack(bfErr.height) + if pending { + s.jobGen.pushBack(bfErr.height) + } // One failed request is usually a timeout under load, not a bad peer. // Dropping the peer also fails its other in-flight requests, each of // which drops another peer in turn, so react only to a sustained run of @@ -442,6 +451,15 @@ func (s *Synchronizer) backlog() (applyHeight int64, pendingBytes int) { } // GetStatus returns synchronizer's height, count of in progress requests +// applyState reports the height block sync waits to apply and whether its block +// has arrived. +func (s *Synchronizer) applyState() (int64, bool) { + s.mtx.RLock() + defer s.mtx.RUnlock() + _, received := s.pendingToApply[s.height] + return s.height, received +} + func (s *Synchronizer) GetStatus() (int64, int32) { cnt := s.jobProgressCounter.Load() s.mtx.RLock() From ea2e7ea5c46bb3e5d7a2d3def29ed01763c7303e Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 13:25:32 +0000 Subject: [PATCH 26/36] fix(blocksync): narrow the at-tip lag to the wait before every peer is asked With the height to apply asked of every peer after serviceGrace, block sync at the tip no longer lags by a whole request deadline. Measured with one peer answering just inside the deadline and a prompt one, the last applied block reached 21-25 s of age on chains committing a block every 1-4 s. Set tipLag to a status round plus three times serviceGrace (40 s) instead of 70 s. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/reactor.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/internal/blocksync/reactor.go b/internal/blocksync/reactor.go index 6daf43204..8dede39ed 100644 --- a/internal/blocksync/reactor.go +++ b/internal/blocksync/reactor.go @@ -57,8 +57,9 @@ const ( tipRecency = 5 * time.Minute // tipLag is how far behind the network block sync runs at the tip: a status - // round to learn of a new block and a request timeout to fetch it. - tipLag = statusUpdateIntervalSeconds*time.Second + client.RequestTimeout + // round to learn of a new block, serviceGrace before every peer that has it + // is asked for it, and serviceGrace each for an answer and a margin. + tipLag = statusUpdateIntervalSeconds*time.Second + 3*serviceGrace // tipGrace is how long the node must stay at the tip before higher height // claims stop keeping it in block sync. From 42801001fbbebbaa981063e87b90b486639239ea Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 13:26:04 +0000 Subject: [PATCH 27/36] docs(blocksync): match the operator notes to the current handover rules State the at-tip condition as applying each block before the previous one leaves the at-tip window, the 40-second floor of that window, the 10-second wait before every peer is asked for the next block, and the one-day limit past which a stopped chain's restarted node keeps waiting behind far claims. Co-Authored-By: Claude Opus 5.5 --- docs/tenderdash-core/block-sync/reactor.md | 35 +++++++++++++--------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/docs/tenderdash-core/block-sync/reactor.md b/docs/tenderdash-core/block-sync/reactor.md index 21c256583..001e65103 100644 --- a/docs/tenderdash-core/block-sync/reactor.md +++ b/docs/tenderdash-core/block-sync/reactor.md @@ -58,17 +58,18 @@ claims; when no peer advertises the next height and no block has been applied for 60 seconds; when the node has been at the tip of the chain for two minutes; or when no block has been applied for 10 minutes and the node is within 10 blocks of every claim that covers the next height and comes from a peer that -has backed it. When the node's last block is more than an hour old, that last -rule counts every claim that covers the next height, backed or not, so a node -far behind keeps syncing while its peers fail requests and resumes as soon as -one serves again. +has backed it. When the node's last block was more than a day old as the stall +began, that last rule counts every claim that covers the next height, backed or +not, so a node far behind keeps syncing while its peers fail requests and +resumes as soon as one serves again. Whether the node is at the tip is judged from the time of its last applied block, which the validators commit to and a peer cannot change: a node behind the network applies old blocks. The node is at the tip while that block is no -older than the chain took to commit its last 10 blocks, or than 70 seconds (a -status round and a request timeout) where that is longer, and in any case no -older than five minutes. The pace of the chain is taken from the blocks in the +older than the chain took to commit its last 10 blocks, or than 40 seconds (a +status round, the 10-second wait before every peer is asked for the next +block, and time for an answer) where that is longer, and in any case no older +than five minutes. The pace of the chain is taken from the blocks in the block store at start and from those applied since; with no earlier block stored, only the five-minute limit applies until a block is applied. Five minutes covers the three-minute empty-block interval Dash Platform runs with @@ -81,18 +82,24 @@ A peer backs its claim by supplying a block the node applies, and loses that credit when it fails a request (a timeout or other failure, or a `NoBlockResponse`) or is dropped; a newly connected peer has none. Block requests go to peers that have backed their claims first. Once a request for -the next block fails, every other peer that advertises it is asked for it as -well. +the next block fails or goes unanswered for 10 seconds, every other peer that +advertises it is asked for it as well. What operators can expect: -- A node that keeps receiving each next block while that block is less than - its at-tip window minus two minutes old hands over within two minutes of - reaching the tip, whatever heights peers claim. +- A node that has reached the tip hands over two minutes later, whatever + heights peers claim, as long as it applies each new block before the one + before it is older than the at-tip window: within about two minutes of its + commit on a chain committing a block every three minutes, and within about + 35 seconds on a chain committing one every few seconds. A single slow peer + delays a block by about 10 seconds. - On a chain that has stopped producing blocks, a restarted node hands over after 10 minutes without a block, or once every peer that supplied blocks - earlier has failed a request for the next one, whichever is later. -- A node may hand over a few blocks behind the network, up to about 70 + earlier has failed a request for the next one, whichever is later. If the + chain had been stopped for more than a day when the node started waiting, + the node keeps waiting while a peer claims a height more than 10 blocks + above it and advertises the next block. +- A node may hand over a few blocks behind the network, up to about 40 seconds of blocks on a fast chain, or up to five minutes of blocks after a restart with no earlier block stored; consensus catch-up closes the rest. - A node whose clock runs ahead of the validators' by more than its at-tip From 5d5d674132d83ba3caa688d7337e0ae2e4eb1a82 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 13:43:47 +0000 Subject: [PATCH 28/36] test(blocksync): check the fetch volume far behind and the handover on a chain committing in bursts Three tests that state the intended behaviour and fail on 42801001f; all pass with the generator and constants of 1ac0acae1: - TestBlockSyncFarBehindFetchesEachBlockOnce: twelve peers that each need a second to send a block, so a full queue of twenty takes twenty seconds. 600 blocks take 6m5s and 4273 block responses instead of 3m21s and 600, because a height first requested more than serviceGrace ago is asked of every peer as soon as it becomes the one to apply. - TestBlockSyncFarBehindOnASharedDownlink: the same with a downlink of four blocks a second in all: 6m53s and 1062 responses instead of 3m21s and 600. - TestWaitForSyncHandsOverAtTheTipOfAChainCommittingInBursts: with prompt peers and a far range nobody is asked for, a chain committing 12 blocks three seconds apart and then none for 50 seconds keeps the node in block sync: the pause is longer than the 40-second at-tip window and shorter than the 60 seconds after which a node with nothing to fetch hands over. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_01WunYYgi2uMsQDcBcKAwLdt --- internal/blocksync/handover_bursts_test.go | 83 +++++++ .../blocksync/handover_fetch_volume_test.go | 227 ++++++++++++++++++ 2 files changed, 310 insertions(+) create mode 100644 internal/blocksync/handover_bursts_test.go create mode 100644 internal/blocksync/handover_fetch_volume_test.go diff --git a/internal/blocksync/handover_bursts_test.go b/internal/blocksync/handover_bursts_test.go new file mode 100644 index 000000000..5e5c6bcfc --- /dev/null +++ b/internal/blocksync/handover_bursts_test.go @@ -0,0 +1,83 @@ +package blocksync + +import ( + "fmt" + "time" + + "github.com/stretchr/testify/mock" + + "github.com/dashpay/tenderdash/types" +) + +// TestWaitForSyncHandsOverAtTheTipOfAChainCommittingInBursts checks the handover +// of a node level with a chain that commits blocks only while there are +// transactions: a run of blocks three seconds apart, then a pause until the +// next run. Three peers hold every block and answer at once, so the node +// applies each block within a status round of its commit, and a far range is +// reported by a peer nobody asks. A pause in block production is not the node +// falling behind, so it must hand over as it does on a chain committing a block +// every three seconds without pauses. +func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverAtTheTipOfAChainCommittingInBursts() { + const ( + height = int64(100) + interval = 3 * time.Second + limit = time.Hour + ) + for _, tc := range []struct { + name string + // run is the number of blocks committed interval apart before each pause + run int64 + pause time.Duration + }{ + {name: "no pauses"}, + {name: "12 blocks, then 20 seconds without one", run: 12, pause: 20 * time.Second}, + {name: "12 blocks, then 50 seconds without one", run: 12, pause: 50 * time.Second}, + {name: "12 blocks, then 80 seconds without one", run: 12, pause: 80 * time.Second}, + } { + suite.Run(tc.name, func() { + suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) + h := suite.newWaitForSyncHarness(height) + defer h.cancel() + start := h.clock.Now() + r := &tipRun{ + h: h, + interval: interval, + blockTimes: map[int64]time.Time{}, + suppliedBy: map[types.NodeID]int{}, + tip: height - 1, + nextCommit: start, + } + for i := 0; i < 3; i++ { + r.reporting = append(r.reporting, types.NodeID(fmt.Sprintf("holder %d", i))) + } + // the block store holds the blocks the node applied before it stopped + var stored []time.Time + for i := maxCatchupGap; i >= 0; i-- { + stored = append(stored, start.Add(-time.Duration(i+1)*interval)) + } + h.sync.mtx.Lock() + h.sync.blockTimes = stored + h.sync.mtx.Unlock() + h.sync.AddPeer(newPeerData("far range", 1_000_000_000, maxPlausiblePeerHeight)) + + breaks := 0 + elapsed, caughtUp, ended := h.run(limit, func(elapsed time.Duration) { + // the gap after the block about to be committed + r.interval = interval + if next := r.tip + 1 - height; tc.run > 0 && next%tc.run == tc.run-1 { + r.interval = tc.pause + } + r.step(elapsed) + st := h.sync.stallSnapshot() + if !atTip(st.lastBlockTime, st.tipWindow, st.precision, h.clock.Now().Add(time.Second)) { + breaks++ + } + }) + suite.Require().True(ended, + "still in block sync after %s: %d blocks applied, each within a status round of its commit; the node counted as away from the tip at %d ticks", + elapsed, r.applied, breaks) + suite.Require().False(caughtUp) + suite.T().Logf("handed over after %s: %d blocks applied, away from the tip at %d ticks", elapsed, r.applied, breaks) + }) + } +} diff --git a/internal/blocksync/handover_fetch_volume_test.go b/internal/blocksync/handover_fetch_volume_test.go new file mode 100644 index 000000000..93083648b --- /dev/null +++ b/internal/blocksync/handover_fetch_volume_test.go @@ -0,0 +1,227 @@ +package blocksync + +import ( + "fmt" + "time" + + "github.com/stretchr/testify/mock" + + "github.com/dashpay/tenderdash/internal/p2p/client" + "github.com/dashpay/tenderdash/types" +) + +// linkRun drives tipRun for a node far behind whose peers each send one block +// at a time over their connection, taking perBlock for each, as a peer serving +// its response queue over one rate-limited connection does. Every request is +// answered with the block, in the order the peer was asked. +// +// With downlink set, the node receives at most that many blocks a second over +// all its connections, shared between the peers in turn, each peer sending at +// most one a second, and a request not answered within the request deadline +// fails as it does in the block client. +type linkRun struct { + *tipRun + perBlock time.Duration + downlink int + // busyUntil is when a peer finishes sending what it was asked for so far. + busyUntil map[types.NodeID]time.Time + scheduled map[*tipRequest]bool + // slowest is the longest a request took to be answered. + slowest time.Duration + + // peers is the order the downlink is shared in; queues and askedAt hold + // what each peer has still to send and when it was asked. + peers []types.NodeID + queues map[types.NodeID][]*tipRequest + askedAt map[*tipRequest]time.Time + turn int + timeouts int +} + +// never is when a response waiting for the downlink is due. +var never = time.Unix(1<<40, 0) + +func (l *linkRun) step(time.Duration) { + if l.downlink > 0 { + l.sharedStep() + return + } + now := l.h.clock.Now() + for { + consumed := l.consume(now) + produced := l.produce(now) + for _, req := range l.inFlight { + if l.scheduled[req] { + continue + } + l.scheduled[req] = true + start := now + if busy := l.busyUntil[req.peer.peerID]; busy.After(start) { + start = busy + } + req.endAt = start.Add(l.perBlock) + l.busyUntil[req.peer.peerID] = req.endAt + l.slowest = max(l.slowest, req.endAt.Sub(now)) + } + if !consumed && !produced { + return + } + } +} + +// TestBlockSyncFarBehindFetchesEachBlockOnce checks how many block responses a +// node far behind downloads for the blocks it applies. Twelve peers hold every +// block and none fails or delays a request beyond its place in the peer's +// queue. When a peer needs a second to send a block, a full queue of twenty +// takes twenty seconds, well inside the 60-second request deadline; the node +// must still fetch each block about once, as it does when blocks arrive at +// once. +func (suite *SynchronizerTestSuite) TestBlockSyncFarBehindFetchesEachBlockOnce() { + const ( + height = int64(100) + blocks = int64(600) + peers = 12 + ) + for _, tc := range []struct { + name string + perBlock time.Duration + }{ + {name: "blocks arrive at once"}, + {name: "a peer sends a block a second", perBlock: time.Second}, + } { + suite.Run(tc.name, func() { + suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) + h := suite.newWaitForSyncHarness(height) + defer h.cancel() + // the node is re-syncing history committed long ago + h.setLastBlockTime(h.clock.Now().Add(-365 * 24 * time.Hour)) + l := &linkRun{ + tipRun: &tipRun{ + h: h, + blockTimes: map[int64]time.Time{}, + suppliedBy: map[types.NodeID]int{}, + }, + perBlock: tc.perBlock, + busyUntil: map[types.NodeID]time.Time{}, + scheduled: map[*tipRequest]bool{}, + } + for i := 0; i < peers; i++ { + h.sync.AddPeer(newPeerData(types.NodeID(fmt.Sprintf("peer %d", i)), 1, height+blocks-1)) + } + + elapsed, caughtUp, ended := h.run(30*time.Minute, l.step) + suite.Require().True(ended, "not caught up after %s: %d blocks applied", elapsed, l.applied) + suite.Require().True(caughtUp) + // the handover comes with the last block still to apply + suite.Require().GreaterOrEqual(int64(l.applied), blocks-1) + suite.T().Logf("%d blocks applied in %s, %d block responses downloaded, slowest answer %s", + l.applied, elapsed, l.blocksSent, l.slowest) + suite.Require().LessOrEqual(l.blocksSent, 2*l.applied, + "%d blocks applied in %s, %d block responses downloaded from %d peers, slowest answer %s", + l.applied, elapsed, l.blocksSent, peers, l.slowest) + }) + } +} + +// sharedStep is step for a node with a limited downlink. +func (l *linkRun) sharedStep() { + now := l.h.clock.Now() + // requests past the deadline fail + for _, peerID := range l.peers { + var kept []*tipRequest + for _, req := range l.queues[peerID] { + if now.Sub(l.askedAt[req]) < client.RequestTimeout { + kept = append(kept, req) + continue + } + l.timeouts++ + l.drop(req) + req.request.Finished() + req.request.complete(0) + l.h.fail(peerID, req.height, client.ErrPeerNotResponded) + } + l.queues[peerID] = kept + } + // the downlink carries the next block of this many peers, in turn + for sent, tried := 0, 0; sent < l.downlink && tried < len(l.peers); tried++ { + peerID := l.peers[l.turn%len(l.peers)] + l.turn++ + if queue := l.queues[peerID]; len(queue) > 0 { + queue[0].endAt = now + l.slowest = max(l.slowest, now.Sub(l.askedAt[queue[0]])) + l.queues[peerID] = queue[1:] + sent++ + } + } + for { + consumed := l.consume(now) + produced := l.produce(now) + for _, req := range l.inFlight { + if _, queued := l.askedAt[req]; queued { + continue + } + l.askedAt[req] = now + req.endAt = never + l.queues[req.peer.peerID] = append(l.queues[req.peer.peerID], req) + } + if !consumed && !produced { + return + } + } +} + +// drop takes req out of the requests in flight. +func (l *linkRun) drop(req *tipRequest) { + for i, other := range l.inFlight { + if other == req { + l.inFlight = append(l.inFlight[:i], l.inFlight[i+1:]...) + return + } + } +} + +// TestBlockSyncFarBehindOnASharedDownlink checks the same for a node whose +// downlink carries four blocks a second in all, as much as the four peers a +// window of requests is spread over can send. Each block must still be fetched +// about once, and no request may run into the request deadline. +func (suite *SynchronizerTestSuite) TestBlockSyncFarBehindOnASharedDownlink() { + const ( + height = int64(100) + blocks = int64(600) + peers = 12 + ) + suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) + h := suite.newWaitForSyncHarness(height) + defer h.cancel() + // the node is re-syncing history committed long ago + h.setLastBlockTime(h.clock.Now().Add(-365 * 24 * time.Hour)) + l := &linkRun{ + tipRun: &tipRun{ + h: h, + blockTimes: map[int64]time.Time{}, + suppliedBy: map[types.NodeID]int{}, + }, + downlink: 4, + queues: map[types.NodeID][]*tipRequest{}, + askedAt: map[*tipRequest]time.Time{}, + } + for i := 0; i < peers; i++ { + peerID := types.NodeID(fmt.Sprintf("peer %d", i)) + l.peers = append(l.peers, peerID) + h.sync.AddPeer(newPeerData(peerID, 1, height+blocks-1)) + } + + elapsed, _, ended := h.run(2*time.Hour, l.step) + suite.T().Logf("%d blocks applied in %s, %d block responses downloaded, %d requests timed out, %d of %d peers left, slowest answer %s", + l.applied, elapsed, l.blocksSent, l.timeouts, h.sync.peerStore.Len(), peers, l.slowest) + suite.Require().True(ended, "not caught up after %s: %d blocks applied, %d requests timed out, %d of %d peers left", + elapsed, l.applied, l.timeouts, h.sync.peerStore.Len(), peers) + suite.Require().LessOrEqual(elapsed, 5*time.Minute, + "%d blocks applied in %s, %d block responses downloaded, slowest answer %s; four peers alone send them in under four minutes", + l.applied, elapsed, l.blocksSent, l.slowest) + suite.Require().Zero(l.timeouts, + "%d blocks applied in %s, %d block responses downloaded, %d requests timed out, %d of %d peers left", + l.applied, elapsed, l.blocksSent, l.timeouts, h.sync.peerStore.Len(), peers) + suite.Require().LessOrEqual(l.blocksSent, 2*l.applied, + "%d blocks applied in %s, %d block responses downloaded from %d peers", l.applied, elapsed, l.blocksSent, peers) +} From aaaf2df38a3dd7085031a2e5959cb46c2d26f355 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 13:52:23 +0000 Subject: [PATCH 29/36] fix(blocksync): keep the at-tip window floor above the nothing-to-fetch stall With a 40-second floor, a pause in block production of 40 to 60 seconds after a run of fast blocks ended the two minutes at the tip before the rule for a node with nothing to fetch could apply, so a far claim kept a node at the tip in block sync. Derive the floor from that stall plus a status round (70 s), so a pause ends block sync by one rule or the other. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/reactor.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/internal/blocksync/reactor.go b/internal/blocksync/reactor.go index 8dede39ed..5b33dc7fa 100644 --- a/internal/blocksync/reactor.go +++ b/internal/blocksync/reactor.go @@ -56,10 +56,10 @@ const ( // Platform runs with and a margin for extra rounds and clock skew. tipRecency = 5 * time.Minute - // tipLag is how far behind the network block sync runs at the tip: a status - // round to learn of a new block, serviceGrace before every peer that has it - // is asked for it, and serviceGrace each for an answer and a margin. - tipLag = statusUpdateIntervalSeconds*time.Second + 3*serviceGrace + // tipLag is the shortest at-tip window: the stall after which a node with + // nothing to fetch hands over, and a status round to learn there is nothing. + // A pause in block production then ends block sync by one rule or the other. + tipLag = syncTimeout + statusUpdateIntervalSeconds*time.Second // tipGrace is how long the node must stay at the tip before higher height // claims stop keeping it in block sync. From 2e781562ba1218dfa75c740e9b21c62b7f21b1d2 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 13:52:23 +0000 Subject: [PATCH 30/36] fix(blocksync): count the wait for the height to apply from when it becomes one Heights requested ahead wait behind others in a peer's queue, so counting the 10-second wait from their first request had a node far behind ask every peer for each block when a peer needed more than half a second per block. Count it from the last request for the height or from when it became the one to apply, whichever is later. Ask the others at once when every connected peer asked for it is on probation for holding requests too long, so a slow peer holding several heights in a row delays each by no extra wait. Co-Authored-By: Claude Opus 5.5 --- internal/blocksync/block_fetch_job.go | 35 ++++++++++++++++++---- internal/blocksync/block_fetch_job_test.go | 29 ++++++++++++++++++ internal/blocksync/peer_service.go | 8 +++++ 3 files changed, 67 insertions(+), 5 deletions(-) diff --git a/internal/blocksync/block_fetch_job.go b/internal/blocksync/block_fetch_job.go index c726b3d35..808df11f3 100644 --- a/internal/blocksync/block_fetch_job.go +++ b/internal/blocksync/block_fetch_job.go @@ -110,7 +110,8 @@ type jobGenerator struct { } type heightRequests struct { - // since is when the height was first requested. + // since is when the height was last requested, or became the one to apply + // if that was later. since time.Time asked map[types.NodeID]struct{} failed bool @@ -182,6 +183,7 @@ func (p *jobGenerator) markRequested(height int64, peerID types.NodeID) { defer p.mtx.Unlock() if rec := p.requestsFor(height); rec != nil { rec.asked[peerID] = struct{}{} + rec.since = p.peerStore.clock.Now() } } @@ -204,9 +206,10 @@ func (p *jobGenerator) requestsFor(height int64) *heightRequests { // observe takes note of the height to apply and whether its block has arrived, // and decides whether to ask every peer that advertises it: once a request for -// it failed or has gone unanswered for serviceGrace. One peer after another, -// peers ranked first could each hold it until it times out. The caller holds -// p.mtx. +// it failed, every peer asked for it is on probation, or serviceGrace passed +// since it was last requested or became the one to apply, whichever is later, +// without its block. One peer after another, peers ranked first could each +// hold it until it times out. The caller holds p.mtx. func (p *jobGenerator) observe(applyHeight int64, received bool) { if applyHeight > p.applyHeight { p.applyHeight = applyHeight @@ -215,15 +218,37 @@ func (p *jobGenerator) observe(applyHeight int64, received bool) { delete(p.requests, height) } } + // A height requested ahead waits behind others in a peer's queue. + if rec := p.requests[applyHeight]; rec != nil { + rec.since = p.peerStore.clock.Now() + } } rec := p.requests[p.applyHeight] p.spreading = false - if rec == nil || received || !rec.failed && p.peerStore.clock.Since(rec.since) <= serviceGrace { + if rec == nil || received || + !rec.failed && p.peerStore.clock.Since(rec.since) <= serviceGrace && !p.allSlow(rec.asked) { return } _, p.spreading = p.peerStore.findPeer(p.applyHeight, rec.asked) } +// allSlow reports whether every peer in asked that is still connected, and at +// least one, is on probation. +func (p *jobGenerator) allSlow(asked map[types.NodeID]struct{}) bool { + slow := false + for peerID := range asked { + peer, found := p.peerStore.Get(peerID) + if !found { + continue + } + if !peer.service.slow() { + return false + } + slow = true + } + return slow +} + // nextSpread reserves one more peer, not asked yet, for the height block sync // waits to apply, and reports false once there is none. A retry of that height // queued after a failure goes first. diff --git a/internal/blocksync/block_fetch_job_test.go b/internal/blocksync/block_fetch_job_test.go index 4ce129760..568bc7090 100644 --- a/internal/blocksync/block_fetch_job_test.go +++ b/internal/blocksync/block_fetch_job_test.go @@ -534,3 +534,32 @@ func TestAskingEveryPeerEndsWithinTheWindow(t *testing.T) { clock.Advance(serviceGrace + time.Second) require.False(t, gen.shouldJobBeGenerated(height, 0), "no peer is left to ask") } + +// TestHeightToApplyHeldBySlowPeersIsAskedOfOthersAtOnce checks that the height +// block sync waits to apply is asked of every other peer that advertises it as +// soon as it becomes the one to apply, without the usual wait, when the only +// peer asked for it is on probation for holding requests too long. +func TestHeightToApplyHeldBySlowPeersIsAskedOfOthersAtOnce(t *testing.T) { + const height = int64(100) + clock := clockwork.NewFakeClock() + slow, prompt := newPeerData("slow", 1, height+10), newPeerData("prompt", 1, height+10) + store := NewInMemPeerStore() + store.clock = clock + store.Put(slow.peerID, slow) + store.Put(prompt.peerID, prompt) + gen := newJobGenerator(height+2, log.NewNopLogger(), nil, store) + gen.shouldJobBeGenerated(height, 0) + + // height+1 was requested of the slow peer, which holds its requests + gen.markRequested(height+1, slow.peerID) + slow.service.request().Started() + clock.Advance(serviceGrace + time.Second) + require.True(t, slow.service.slow()) + + // height is applied from another peer, and height+1 becomes the one to apply + require.True(t, gen.shouldJobBeGenerated(height+1, 0), "the prompt peer is not asked") + peer, spreadHeight, found := gen.nextSpread() + require.True(t, found) + require.Equal(t, height+1, spreadHeight) + require.Equal(t, prompt.peerID, peer.peerID) +} diff --git a/internal/blocksync/peer_service.go b/internal/blocksync/peer_service.go index d1b61d147..993202475 100644 --- a/internal/blocksync/peer_service.go +++ b/internal/blocksync/peer_service.go @@ -79,6 +79,14 @@ func (s *peerService) priority(pending int32, reserve bool) int { return 2 } +// slow reports whether the peer is on probation for holding requests too long. +func (s *peerService) slow() bool { + s.mtx.Lock() + defer s.mtx.Unlock() + s.classify(s.advance()) + return s.probation +} + func (s *peerService) request() *serviceRequest { s.mtx.Lock() defer s.mtx.Unlock() From b09fcadfcdd3f450f2f1247e1c1112ee11dc594a Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 13:52:36 +0000 Subject: [PATCH 31/36] docs(blocksync): state when the next block is asked of every peer Describe the 70-second floor of the at-tip window and why, what makes the next block go to every peer that advertises it, from when the 10-second wait counts, and that a "no such block" answer sends it to one other peer only. Restore the GetStatus comment and update a test comment that described the earlier generator. Co-Authored-By: Claude Opus 5.5 --- docs/tenderdash-core/block-sync/reactor.md | 24 ++++++++++++-------- internal/blocksync/handover_deadline_test.go | 7 +----- internal/blocksync/synchronizer.go | 2 +- 3 files changed, 16 insertions(+), 17 deletions(-) diff --git a/docs/tenderdash-core/block-sync/reactor.md b/docs/tenderdash-core/block-sync/reactor.md index 001e65103..32853752d 100644 --- a/docs/tenderdash-core/block-sync/reactor.md +++ b/docs/tenderdash-core/block-sync/reactor.md @@ -66,10 +66,11 @@ resumes as soon as one serves again. Whether the node is at the tip is judged from the time of its last applied block, which the validators commit to and a peer cannot change: a node behind the network applies old blocks. The node is at the tip while that block is no -older than the chain took to commit its last 10 blocks, or than 40 seconds (a -status round, the 10-second wait before every peer is asked for the next -block, and time for an answer) where that is longer, and in any case no older -than five minutes. The pace of the chain is taken from the blocks in the +older than the chain took to commit its last 10 blocks, or than 70 seconds +where that is longer, and in any case no older than five minutes. Seventy +seconds is the 60-second stall after which a node with nothing to fetch hands +over plus a status round, so a pause in block production ends block sync by +one rule or the other. The pace of the chain is taken from the blocks in the block store at start and from those applied since; with no earlier block stored, only the five-minute limit applies until a block is applied. Five minutes covers the three-minute empty-block interval Dash Platform runs with @@ -81,9 +82,12 @@ consensus catch-up target is the highest claim seen within 10 blocks of it. A peer backs its claim by supplying a block the node applies, and loses that credit when it fails a request (a timeout or other failure, or a `NoBlockResponse`) or is dropped; a newly connected peer has none. Block -requests go to peers that have backed their claims first. Once a request for -the next block fails or goes unanswered for 10 seconds, every other peer that -advertises it is asked for it as well. +requests go to peers that have backed their claims first. The next block is +asked of every other peer that advertises it once a request for it times out +or fails otherwise, once every peer asked for it is on probation for holding +requests too long, or once 10 seconds pass without it, counted from its latest +request or from when it became the next block, whichever is later. A peer +answering that it has no such block has it asked of one other peer instead. What operators can expect: @@ -91,15 +95,15 @@ What operators can expect: heights peers claim, as long as it applies each new block before the one before it is older than the at-tip window: within about two minutes of its commit on a chain committing a block every three minutes, and within about - 35 seconds on a chain committing one every few seconds. A single slow peer - delays a block by about 10 seconds. + a minute on a chain committing one every few seconds. A single slow peer + delays a block by up to about 10 seconds. - On a chain that has stopped producing blocks, a restarted node hands over after 10 minutes without a block, or once every peer that supplied blocks earlier has failed a request for the next one, whichever is later. If the chain had been stopped for more than a day when the node started waiting, the node keeps waiting while a peer claims a height more than 10 blocks above it and advertises the next block. -- A node may hand over a few blocks behind the network, up to about 40 +- A node may hand over a few blocks behind the network, up to about 70 seconds of blocks on a fast chain, or up to five minutes of blocks after a restart with no earlier block stored; consensus catch-up closes the rest. - A node whose clock runs ahead of the validators' by more than its at-tip diff --git a/internal/blocksync/handover_deadline_test.go b/internal/blocksync/handover_deadline_test.go index 139b16d75..c2351a7e7 100644 --- a/internal/blocksync/handover_deadline_test.go +++ b/internal/blocksync/handover_deadline_test.go @@ -235,12 +235,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverBehindAPeerAnswering // far height and answers every request that it has no such block; it is the // only peer advertising the next height until the others report it, so it is // asked for it first. Its answer must not have every block fetched from all -// five peers. -// -// This holds only as long as the retry is taken before the height becomes the -// one to apply: the generator then does not ask the other peers (see -// TestRetryOfTheHeightToApplyAsksEveryPeerWhileTheProducerWaits). The first -// block, retried while it is the one to apply, is fetched five times. +// five peers: a "no such block" answer has the block asked of one other peer. func (suite *SynchronizerTestSuite) TestWaitForSyncFetchesEachBlockAtTheTipFromOnePeer() { const ( height = int64(100) diff --git a/internal/blocksync/synchronizer.go b/internal/blocksync/synchronizer.go index 15879ad70..5ab1c8be8 100644 --- a/internal/blocksync/synchronizer.go +++ b/internal/blocksync/synchronizer.go @@ -450,7 +450,6 @@ func (s *Synchronizer) backlog() (applyHeight int64, pendingBytes int) { return s.height, pendingBytes } -// GetStatus returns synchronizer's height, count of in progress requests // applyState reports the height block sync waits to apply and whether its block // has arrived. func (s *Synchronizer) applyState() (int64, bool) { @@ -460,6 +459,7 @@ func (s *Synchronizer) applyState() (int64, bool) { return s.height, received } +// GetStatus returns synchronizer's height, count of in progress requests func (s *Synchronizer) GetStatus() (int64, int32) { cnt := s.jobProgressCounter.Load() s.mtx.RLock() From dc12b76548f2c4c1bda846ef80c3a4abe9733cff Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 15:00:56 +0000 Subject: [PATCH 32/36] refactor(blocksync): judge the handover by block time and ask every peer for an overdue block Replace the per-peer bookkeeping the handover rules had grown with two rules that use nothing a peer reports about itself: - the last applied block has been at most tipRecency (5 min) old for tipGrace (2 min): hand over whatever heights peers report; - no block for maxSyncStall (10 min) on a block less than staleBlockAge (a day) old when the stall began: hand over as well. Further back the distance to servable peer heights decides, as on dev (#1413). To keep the first rule from depending on which peer holds the request for the next block, a block 10 seconds overdue is requested from every peer that reports it, once each, by the synchronizer itself. Removed: the supplied-block credit and the peer ranking built on it, the per-height request records in the job generator, the at-tip window that followed the chain's pace, and client.RequestTimeout. The job generator, the peer service and the p2p client are as on dev again. A node may now hand over up to five minutes of blocks behind on any chain; TestWaitForSyncOnAFastChainBehindTheTip pins that bound. Co-Authored-By: Claude Fable 5.2 --- internal/blocksync/ask_every_peer_test.go | 131 ++++++++ internal/blocksync/block_fetch_job.go | 220 +------------ internal/blocksync/block_fetch_job_test.go | 92 ------ .../block_fetch_spread_review_test.go | 94 ------ internal/blocksync/handover_bursts_test.go | 10 +- internal/blocksync/handover_deadline_test.go | 116 ++++--- .../blocksync/handover_fetch_volume_test.go | 2 +- .../blocksync/handover_late_supply_test.go | 50 ++- internal/blocksync/handover_review_test.go | 50 +-- internal/blocksync/peer_service.go | 8 - internal/blocksync/peer_service_test.go | 30 -- internal/blocksync/peer_store.go | 41 +-- internal/blocksync/peer_store_test.go | 24 -- internal/blocksync/reactor.go | 24 +- internal/blocksync/synchronizer.go | 218 ++++++------- internal/blocksync/synchronizer_test.go | 290 ++++-------------- internal/p2p/client/client.go | 3 - 17 files changed, 418 insertions(+), 985 deletions(-) create mode 100644 internal/blocksync/ask_every_peer_test.go delete mode 100644 internal/blocksync/block_fetch_spread_review_test.go diff --git a/internal/blocksync/ask_every_peer_test.go b/internal/blocksync/ask_every_peer_test.go new file mode 100644 index 000000000..31a08dc5f --- /dev/null +++ b/internal/blocksync/ask_every_peer_test.go @@ -0,0 +1,131 @@ +package blocksync + +import ( + "context" + "sync/atomic" + "testing" + "time" + + "github.com/jonboulle/clockwork" + "github.com/stretchr/testify/mock" + "github.com/stretchr/testify/require" + + "github.com/dashpay/tenderdash/internal/p2p/client" + "github.com/dashpay/tenderdash/internal/p2p/client/mocks" + "github.com/dashpay/tenderdash/libs/promise" + "github.com/dashpay/tenderdash/libs/workerpool" + bcproto "github.com/dashpay/tenderdash/proto/tendermint/blocksync" + "github.com/dashpay/tenderdash/types" +) + +func reservedIDs(peers []PeerData) []types.NodeID { + ids := make([]types.NodeID, 0, len(peers)) + for _, peer := range peers { + ids = append(ids, peer.peerID) + } + return ids +} + +// TestReserveNextBlockPeers checks who is asked for the height block sync waits +// to apply, and when: nobody until its block is serviceGrace overdue, then every +// peer that advertises it, each once and a second after it is first seen +// advertising it, until the block arrives or the height is applied. +func TestReserveNextBlockPeers(t *testing.T) { + const height = int64(100) + clock := clockwork.NewFakeClock() + sync := NewSynchronizer(height, nil, nil, WithClock(clock), WithWorkerPool(workerpool.New(0))) + sync.lastAdvance = clock.Now() + sync.AddPeer(newPeerData("first", 1, height)) + sync.AddPeer(newPeerData("second", 1, height+1000)) + sync.AddPeer(newPeerData("above", height+1, height+1000)) + busy := newPeerData("busy", 1, height) + busy.numPending = maxPendingRequestsPerPeer + sync.AddPeer(busy) + next := func() []types.NodeID { + clock.Advance(time.Second) + reservedFor, peers := sync.reserveNextBlockPeers() + applying, _ := sync.GetStatus() + require.Equal(t, applying, reservedFor) + return reservedIDs(peers) + } + + clock.Advance(serviceGrace - 2*time.Second) + require.Empty(t, next(), "the block is not overdue yet") + require.Empty(t, next(), "the block is not overdue yet") + require.Empty(t, next(), "peers advertising the height are given a second first") + require.ElementsMatch(t, []types.NodeID{"first", "second"}, next()) + for _, peerID := range []types.NodeID{"first", "second"} { + peer, _ := sync.peerStore.Get(peerID) + require.EqualValues(t, 1, peer.numPending, "the request counts against the peer's limit") + } + require.Empty(t, next(), "each peer is asked once") + + // a peer that left is not kept on record + sync.RemovePeer("first") + require.NotContains(t, sync.asked, types.NodeID("first")) + sync.AddPeer(newPeerData("first", 1, height)) + require.Empty(t, next()) + require.ElementsMatch(t, []types.NodeID{"first"}, next()) + + // a peer that starts advertising the height, and one that gets free + sync.AddPeer(newPeerData("above", 1, height+1000)) + sync.peerStore.Update("busy", AddNumPending(-1)) + require.ElementsMatch(t, []types.NodeID{"busy"}, next()) + require.ElementsMatch(t, []types.NodeID{"above"}, next()) + + // nobody else is asked once the block is here + sync.AddPeer(newPeerData("late", 1, height)) + require.NoError(t, sync.addBlock(BlockResponse{ + PeerID: "first", + Block: &types.Block{Header: types.Header{Height: height}}, + })) + require.Empty(t, next()) + require.Empty(t, next()) + + // the next height starts over + sync.advance(clock.Now(), defaultPrecision) + clock.Advance(serviceGrace - time.Second) + require.Empty(t, next(), "the next block is not overdue yet") + require.Empty(t, next()) + require.ElementsMatch(t, []types.NodeID{"second", "above"}, next()) +} + +// TestNextBlockIsFetchedWhileTheProducerWaits checks that the height block sync +// waits to apply is requested again after its only request failed, although the +// job producer is waiting for a peer to serve a higher height that only a peer +// with no room for another request reports. +func TestNextBlockIsFetchedWhileTheProducerWaits(t *testing.T) { + const height = int64(100) + var asked atomic.Int32 + // fail is closed once the producer has moved on to the next height + fail := make(chan struct{}) + blockClient := mocks.NewBlockClient(t) + blockClient. + On("GetBlock", mock.Anything, height, types.NodeID("idle peer")). + Return(func(context.Context, int64, types.NodeID) *promise.Promise[*bcproto.BlockResponse] { + asked.Add(1) + return promise.New(func(_ func(*bcproto.BlockResponse), reject func(error)) { + <-fail + reject(client.ErrPeerNotResponded) + }) + }, nil) + clock := clockwork.NewFakeClock() + sync := NewSynchronizer(height, blockClient, nil, WithClock(clock), WithWorkerPool(workerpool.New(2))) + sync.AddPeer(newPeerData("idle peer", 1, height)) + busy := newPeerData("busy claimant", height+1, maxPlausiblePeerHeight) + busy.numPending = maxPendingRequestsPerPeer + sync.AddPeer(busy) + + require.NoError(t, sync.Start(context.Background())) + defer sync.Stop() + require.Eventually(t, func() bool { return asked.Load() == 1 }, 5*time.Second, time.Millisecond, + "the producer requests the height first") + time.Sleep(50 * time.Millisecond) + close(fail) + + require.Eventually(t, func() bool { + clock.Advance(time.Second) + return asked.Load() >= 2 + }, 10*time.Second, 5*time.Millisecond, + "height %d was not requested again while the producer waits for a peer for height %d", height, height+1) +} diff --git a/internal/blocksync/block_fetch_job.go b/internal/blocksync/block_fetch_job.go index 808df11f3..7c58f5255 100644 --- a/internal/blocksync/block_fetch_job.go +++ b/internal/blocksync/block_fetch_job.go @@ -97,24 +97,6 @@ type jobGenerator struct { peerStore *InMemPeerStore height int64 pushedBack []int64 - // applyHeight is the height block sync waits to apply. Once a request for it - // fails, or goes unanswered for serviceGrace, every peer that advertises it - // is asked for it; spreading says some peer is left to ask. requests holds - // what was asked of whom for each height from applyHeight up. - applyHeight int64 - spreading bool - requests map[int64]*heightRequests - // applyState, when set, reports the height to apply and whether its block - // has arrived, so that a wait for a peer inside nextJob sees them. - applyState func() (height int64, received bool) -} - -type heightRequests struct { - // since is when the height was last requested, or became the one to apply - // if that was later. - since time.Time - asked map[types.NodeID]struct{} - failed bool } func newJobGenerator(height int64, logger log.Logger, client client.BlockClient, peerStore *InMemPeerStore) *jobGenerator { @@ -128,154 +110,30 @@ func newJobGenerator(height int64, logger log.Logger, client client.BlockClient, } func (p *jobGenerator) nextHeight() int64 { - height, _ := p.next() - return height -} - -// next returns the next height to request and whether it is a retry. -func (p *jobGenerator) next() (int64, bool) { p.mtx.Lock() defer p.mtx.Unlock() if len(p.pushedBack) > 0 { height := p.pushedBack[0] p.pushedBack = p.pushedBack[1:] - return height, true + return height } height := p.height p.height++ - return height, false + return height } func (p *jobGenerator) nextJob(ctx context.Context) (*workerpool.Job, error) { - for { - if peer, height, found := p.nextSpread(); found { - return workerpool.NewJob(blockFetchJobHandler(p.client, peer, height)), nil - } - height, retry := p.next() - peer, err := p.getPeer(ctx, height) - if errors.Is(err, errLowerHeightQueued) { - p.giveBack(height, retry) - continue - } - if err != nil { - return nil, err - } - p.markRequested(height, peer.peerID) - return workerpool.NewJob(blockFetchJobHandler(p.client, peer, height)), nil - } -} - -// giveBack returns a height taken by next and not requested, as a retry if it -// was one and otherwise as the next new height. -func (p *jobGenerator) giveBack(height int64, retry bool) { - p.mtx.Lock() - defer p.mtx.Unlock() - if !retry && p.height == height+1 { - p.height = height - return - } - p.queue(height) -} - -// markRequested records that height was asked of peerID. -func (p *jobGenerator) markRequested(height int64, peerID types.NodeID) { - p.mtx.Lock() - defer p.mtx.Unlock() - if rec := p.requestsFor(height); rec != nil { - rec.asked[peerID] = struct{}{} - rec.since = p.peerStore.clock.Now() - } -} - -// requestsFor returns the record of requests for height, created if missing, -// or nil for a height below the one to apply. The caller holds p.mtx. -func (p *jobGenerator) requestsFor(height int64) *heightRequests { - if height < p.applyHeight { - return nil - } - rec, found := p.requests[height] - if !found { - if p.requests == nil { - p.requests = map[int64]*heightRequests{} - } - rec = &heightRequests{since: p.peerStore.clock.Now(), asked: map[types.NodeID]struct{}{}} - p.requests[height] = rec - } - return rec -} - -// observe takes note of the height to apply and whether its block has arrived, -// and decides whether to ask every peer that advertises it: once a request for -// it failed, every peer asked for it is on probation, or serviceGrace passed -// since it was last requested or became the one to apply, whichever is later, -// without its block. One peer after another, peers ranked first could each -// hold it until it times out. The caller holds p.mtx. -func (p *jobGenerator) observe(applyHeight int64, received bool) { - if applyHeight > p.applyHeight { - p.applyHeight = applyHeight - for height := range p.requests { - if height < applyHeight { - delete(p.requests, height) - } - } - // A height requested ahead waits behind others in a peer's queue. - if rec := p.requests[applyHeight]; rec != nil { - rec.since = p.peerStore.clock.Now() - } - } - rec := p.requests[p.applyHeight] - p.spreading = false - if rec == nil || received || - !rec.failed && p.peerStore.clock.Since(rec.since) <= serviceGrace && !p.allSlow(rec.asked) { - return + height := p.nextHeight() + peer, err := p.getPeer(ctx, height) + if err != nil { + return nil, err } - _, p.spreading = p.peerStore.findPeer(p.applyHeight, rec.asked) + return workerpool.NewJob(blockFetchJobHandler(p.client, peer, height)), nil } -// allSlow reports whether every peer in asked that is still connected, and at -// least one, is on probation. -func (p *jobGenerator) allSlow(asked map[types.NodeID]struct{}) bool { - slow := false - for peerID := range asked { - peer, found := p.peerStore.Get(peerID) - if !found { - continue - } - if !peer.service.slow() { - return false - } - slow = true - } - return slow -} - -// nextSpread reserves one more peer, not asked yet, for the height block sync -// waits to apply, and reports false once there is none. A retry of that height -// queued after a failure goes first. -func (p *jobGenerator) nextSpread() (PeerData, int64, bool) { - p.mtx.Lock() - defer p.mtx.Unlock() - if !p.spreading || len(p.pushedBack) > 0 && p.pushedBack[0] == p.applyHeight { - return PeerData{}, 0, false - } - rec := p.requestsFor(p.applyHeight) - peer, found := p.peerStore.reservePeerExcept(p.applyHeight, rec.asked) - if !found { - p.spreading = false - return PeerData{}, 0, false - } - rec.asked[peer.peerID] = struct{}{} - return peer, p.applyHeight, true -} - -// errLowerHeightQueued reports that a height below the one waiting for a peer -// was queued for a retry meanwhile. -var errLowerHeightQueued = errors.New("a lower height is queued") - -// pushBack schedules heights whose requests failed to be fetched again, skipping -// those already queued so a queued height is not fetched twice. Heights already -// handed to a worker are not tracked here, so an in-flight height can still be -// queued and fetched again. +// pushBack schedules heights to be fetched again, skipping those already queued so +// a queued height is not fetched twice. Heights already handed to a worker are not +// tracked here, so an in-flight height can still be queued and fetched again. // // The queue is kept sorted because nextHeight pops from the front and callers hand // us heights in arbitrary order - dropPeer collects them by ranging a map. Only the @@ -290,24 +148,7 @@ func (p *jobGenerator) pushBack(heights ...int64) { } p.mtx.Lock() defer p.mtx.Unlock() - for _, height := range heights { - if rec := p.requestsFor(height); rec != nil { - rec.failed = true - } - } - p.queue(heights...) -} - -// requeue schedules height to be fetched again after a peer answered that it -// has no such block. Another peer is asked; that alone is no reason to ask all. -func (p *jobGenerator) requeue(height int64) { - p.mtx.Lock() - defer p.mtx.Unlock() - p.queue(height) -} -// queue adds heights to the retry queue. The caller holds p.mtx. -func (p *jobGenerator) queue(heights ...int64) { queued := make(map[int64]struct{}, len(p.pushedBack)+len(heights)) for _, height := range p.pushedBack { queued[height] = struct{}{} @@ -323,25 +164,6 @@ func (p *jobGenerator) queue(heights ...int64) { slices.Sort(p.pushedBack) } -// lowerQueued reports whether a height below height is due to be requested: a -// retry, or the height to apply to be asked of more peers. -func (p *jobGenerator) lowerQueued(height int64) bool { - var ( - applyHeight int64 - received bool - ) - if p.applyState != nil { - applyHeight, received = p.applyState() - } - p.mtx.Lock() - defer p.mtx.Unlock() - if p.applyState != nil { - p.observe(applyHeight, received) - } - return len(p.pushedBack) > 0 && p.pushedBack[0] < height || - p.spreading && p.applyHeight < height -} - func (p *jobGenerator) getPeer(ctx context.Context, height int64) (PeerData, error) { for { if ctx.Err() != nil { @@ -351,11 +173,6 @@ func (p *jobGenerator) getPeer(ctx context.Context, height int64) (PeerData, err if found { return peer, nil } - // Blocks are applied in order, so waiting on a higher height for a free - // peer must not hold back a lower one that has to be fetched again. - if p.lowerQueued(height) { - return PeerData{}, errLowerHeightQueued - } // This is preferable to using a timer because the request // interval is so small. Larger request intervals may // necessitate using a timer/ticker. @@ -365,7 +182,7 @@ func (p *jobGenerator) getPeer(ctx context.Context, height int64) (PeerData, err // shouldJobBeGenerated reports whether another height may be requested now. // applyHeight is the height block sync is waiting to apply and pendingBytes is -// the size of the responses already held. +// the size of the responses already held above it. // // Blocks are applied strictly in order, so a response that arrives before the // height below it has been applied is held until that one does, and nothing @@ -389,19 +206,8 @@ func (p *jobGenerator) getPeer(ctx context.Context, height int64) (PeerData, err // typically the very height everything else is waiting for, so holding those // back is what would turn a bound into a permanent stall. func (p *jobGenerator) shouldJobBeGenerated(applyHeight int64, pendingBytes int) bool { - received := false - if p.applyState != nil { - if height, arrived := p.applyState(); height == applyHeight { - received = arrived - } - } - p.mtx.Lock() - defer p.mtx.Unlock() - // Asking one more peer is a job only while one is left to ask; otherwise - // nextJob would take a new height past the limits below. - if p.observe(applyHeight, received); p.spreading { - return true - } + p.mtx.RLock() + defer p.mtx.RUnlock() if len(p.pushedBack) > 0 { return true } diff --git a/internal/blocksync/block_fetch_job_test.go b/internal/blocksync/block_fetch_job_test.go index 568bc7090..17bbb2385 100644 --- a/internal/blocksync/block_fetch_job_test.go +++ b/internal/blocksync/block_fetch_job_test.go @@ -471,95 +471,3 @@ func (suite *BlockFetchJobTestSuite) TestSlowSuccessPrefersAnotherPeer() { suite.Require().True(syncer.peerStore.HasPeerForHeight(2)) suite.Require().Equal(2, syncer.peerStore.Len()) } - -// TestRetryOfTheHeightToApplyAsksEveryPeer checks that once a request for the -// height block sync waits to apply fails, that height is asked of every peer -// that advertises it at once, not of one peer after another. Peers listed first -// could otherwise each hold it until its request times out while a peer able to -// serve it waits behind them. -func TestRetryOfTheHeightToApplyAsksEveryPeer(t *testing.T) { - const height = int64(100) - peerIDs := []types.NodeID{"first", "second", "third", "fourth", "serving"} - store := NewInMemPeerStore() - for _, peerID := range peerIDs { - store.Upsert(newPeerData(peerID, 1, height+1000)) - } - asked := map[types.NodeID]int{} - blockClient := mocks.NewBlockClient(t) - blockClient. - On("GetBlock", mock.Anything, mock.Anything, mock.Anything). - Return(func(_ context.Context, h int64, peerID types.NodeID) *promise.Promise[*bcproto.BlockResponse] { - if h == height { - asked[peerID]++ - } - return promise.New(func(_ func(*bcproto.BlockResponse), reject func(error)) { - reject(client.ErrPeerNotResponded) - }) - }, nil) - gen := newJobGenerator(height+1, log.NewNopLogger(), blockClient, store) - require.True(t, gen.shouldJobBeGenerated(height, 0)) - - // the first request for height went to the first peer and failed - store.Update("first", AddNumPending(1)) - store.AddFailure("first", maxConsecutiveFailures) - gen.pushBack(height) - - ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) - defer cancel() - for range 2 * len(peerIDs) { - if !gen.shouldJobBeGenerated(height, 0) { - break - } - job, err := gen.nextJob(ctx) - require.NoError(t, err) - job.Execute(ctx) - } - for _, peerID := range peerIDs { - require.Equal(t, 1, asked[peerID], "%s asked for the height %d times", peerID, asked[peerID]) - } -} - -// TestAskingEveryPeerEndsWithinTheWindow checks that once every peer has been -// asked for the height to apply, that is no reason to request another height: -// the request would be a new one beyond the outstanding window. -func TestAskingEveryPeerEndsWithinTheWindow(t *testing.T) { - const height = int64(100) - store := NewInMemPeerStore(newPeerData("only peer", 1, height+1000)) - clock := clockwork.NewFakeClock() - store.clock = clock - gen := newJobGenerator(height+maxOutstandingHeights, log.NewNopLogger(), nil, store) - require.False(t, gen.shouldJobBeGenerated(height, 0), "the window is full") - - gen.markRequested(height, "only peer") - clock.Advance(serviceGrace + time.Second) - require.False(t, gen.shouldJobBeGenerated(height, 0), "no peer is left to ask") -} - -// TestHeightToApplyHeldBySlowPeersIsAskedOfOthersAtOnce checks that the height -// block sync waits to apply is asked of every other peer that advertises it as -// soon as it becomes the one to apply, without the usual wait, when the only -// peer asked for it is on probation for holding requests too long. -func TestHeightToApplyHeldBySlowPeersIsAskedOfOthersAtOnce(t *testing.T) { - const height = int64(100) - clock := clockwork.NewFakeClock() - slow, prompt := newPeerData("slow", 1, height+10), newPeerData("prompt", 1, height+10) - store := NewInMemPeerStore() - store.clock = clock - store.Put(slow.peerID, slow) - store.Put(prompt.peerID, prompt) - gen := newJobGenerator(height+2, log.NewNopLogger(), nil, store) - gen.shouldJobBeGenerated(height, 0) - - // height+1 was requested of the slow peer, which holds its requests - gen.markRequested(height+1, slow.peerID) - slow.service.request().Started() - clock.Advance(serviceGrace + time.Second) - require.True(t, slow.service.slow()) - - // height is applied from another peer, and height+1 becomes the one to apply - require.True(t, gen.shouldJobBeGenerated(height+1, 0), "the prompt peer is not asked") - peer, spreadHeight, found := gen.nextSpread() - require.True(t, found) - require.Equal(t, height+1, spreadHeight) - require.Equal(t, prompt.peerID, peer.peerID) -} diff --git a/internal/blocksync/block_fetch_spread_review_test.go b/internal/blocksync/block_fetch_spread_review_test.go deleted file mode 100644 index 1f2fcb34f..000000000 --- a/internal/blocksync/block_fetch_spread_review_test.go +++ /dev/null @@ -1,94 +0,0 @@ -package blocksync - -import ( - "context" - "testing" - "time" - - "github.com/stretchr/testify/mock" - "github.com/stretchr/testify/require" - - "github.com/dashpay/tenderdash/internal/p2p/client" - "github.com/dashpay/tenderdash/internal/p2p/client/mocks" - "github.com/dashpay/tenderdash/libs/log" - "github.com/dashpay/tenderdash/libs/promise" - "github.com/dashpay/tenderdash/libs/workerpool" - bcproto "github.com/dashpay/tenderdash/proto/tendermint/blocksync" - "github.com/dashpay/tenderdash/types" -) - -// TestRetryOfTheHeightToApplyAsksEveryPeerWhileTheProducerWaits checks that a -// failed request for the height block sync waits to apply is asked of every -// peer that advertises it when that height became the one to apply while the -// producer was waiting, inside nextJob, for a peer to advertise a higher one. -// That is where the producer of a node following the network spends its time -// while some peer reports a range further up. -func TestRetryOfTheHeightToApplyAsksEveryPeerWhileTheProducerWaits(t *testing.T) { - const ( - applying = int64(100) - next = applying + 1 - ) - peerIDs := []types.NodeID{"first", "second", "third"} - store := NewInMemPeerStore() - for _, peerID := range peerIDs { - store.Upsert(newPeerData(peerID, 1, next)) - } - store.Upsert(newPeerData("far range", 1_000_000_000, maxPlausiblePeerHeight)) - asked := map[types.NodeID]int{} - blockClient := mocks.NewBlockClient(t) - blockClient. - On("GetBlock", mock.Anything, mock.Anything, mock.Anything). - Return(func(_ context.Context, h int64, peerID types.NodeID) *promise.Promise[*bcproto.BlockResponse] { - if h == next { - asked[peerID]++ - } - return promise.New(func(_ func(*bcproto.BlockResponse), reject func(error)) { - reject(client.ErrPeerNotResponded) - }) - }, nil) - // applying and next are requested; the producer finds another job due while - // block sync waits for applying, and waits for a peer to advertise next+1 - gen := newJobGenerator(next+1, log.NewNopLogger(), blockClient, store) - require.True(t, gen.shouldJobBeGenerated(applying, 0)) - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - jobs := make(chan *jobOrErr, 1) - go func() { - job, err := gen.nextJob(ctx) - jobs <- &jobOrErr{job: job, err: err} - }() - time.Sleep(50 * time.Millisecond) - - // applying is applied meanwhile, and the request for next, sent to the - // first peer, fails - store.Update("first", AddNumPending(1)) - store.AddFailure("first", maxConsecutiveFailures) - gen.pushBack(next) - - // the producer goes on as produceJob does - res := <-jobs - require.NoError(t, res.err) - res.job.Execute(ctx) - for range 2 * len(peerIDs) { - if !gen.shouldJobBeGenerated(next, 0) { - break - } - jobCtx, jobCancel := context.WithTimeout(ctx, 200*time.Millisecond) - job, err := gen.nextJob(jobCtx) - jobCancel() - if err != nil { - // waiting for a peer to advertise next+1 again - break - } - job.Execute(ctx) - } - for _, peerID := range peerIDs[1:] { - require.Equal(t, 1, asked[peerID], "%s asked for the height to apply %d times (requests: %v)", - peerID, asked[peerID], asked) - } -} - -type jobOrErr struct { - job *workerpool.Job - err error -} diff --git a/internal/blocksync/handover_bursts_test.go b/internal/blocksync/handover_bursts_test.go index 5e5c6bcfc..553f7c419 100644 --- a/internal/blocksync/handover_bursts_test.go +++ b/internal/blocksync/handover_bursts_test.go @@ -50,14 +50,6 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverAtTheTipOfAChainComm for i := 0; i < 3; i++ { r.reporting = append(r.reporting, types.NodeID(fmt.Sprintf("holder %d", i))) } - // the block store holds the blocks the node applied before it stopped - var stored []time.Time - for i := maxCatchupGap; i >= 0; i-- { - stored = append(stored, start.Add(-time.Duration(i+1)*interval)) - } - h.sync.mtx.Lock() - h.sync.blockTimes = stored - h.sync.mtx.Unlock() h.sync.AddPeer(newPeerData("far range", 1_000_000_000, maxPlausiblePeerHeight)) breaks := 0 @@ -69,7 +61,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverAtTheTipOfAChainComm } r.step(elapsed) st := h.sync.stallSnapshot() - if !atTip(st.lastBlockTime, st.tipWindow, st.precision, h.clock.Now().Add(time.Second)) { + if !isRecent(st.lastBlockTime, tipRecency, st.precision, h.clock.Now().Add(time.Second)) { breaks++ } }) diff --git a/internal/blocksync/handover_deadline_test.go b/internal/blocksync/handover_deadline_test.go index c2351a7e7..32f674a51 100644 --- a/internal/blocksync/handover_deadline_test.go +++ b/internal/blocksync/handover_deadline_test.go @@ -11,15 +11,20 @@ import ( "github.com/dashpay/tenderdash/types" ) +// requestDeadline is how long the block client waits for a peer's response. +const requestDeadline = 60 * time.Second + // tipRun drives the producer and the consumer of a node following a chain that // commits a block every interval. Jobs are handed out as produceJob and nextJob -// do: shouldJobBeGenerated admits them, nextSpread or next picks the height, and -// reservePeer picks the peer; a height no peer advertises yet is held until one -// does, as getPeer holds it. Responses are queued and applied in height order as -// consumeJobResult and applyBlock do. A peer listed in declining answers at -// once that it has no such block; any other answers with the block, a peer -// listed in hold that long after it is asked and the rest at once. The peers in -// reporting report the chain's height every status round. +// do: shouldJobBeGenerated admits them, nextHeight picks the height and +// reservePeer the peer; a height no peer advertises yet is held until one does, +// as getPeer holds it. Once a second, as askEveryPeer does, the peers +// reserveNextBlockPeers returns are asked for the height to apply. Responses are +// queued and applied in height order as consumeJobResult and applyBlock do. A +// peer listed in declining answers at once that it has no such block; any other +// answers with the block, a peer listed in hold that long after it is asked and +// the rest at once. The peers in reporting report the chain's height every +// status round. type tipRun struct { h *waitForSyncHarness interval time.Duration @@ -34,8 +39,7 @@ type tipRun struct { nextCommit time.Time inFlight []*tipRequest // waiting is the height the producer holds until a peer advertises it. - waiting int64 - waitingRetry bool + waiting int64 applied int suppliedBy map[types.NodeID]int @@ -66,6 +70,12 @@ func (r *tipRun) step(elapsed time.Duration) { } for r.consume(now) || r.produce(now) { } + height, peers := r.h.sync.reserveNextBlockPeers() + for _, peer := range peers { + r.request(peer, height, now) + } + for r.consume(now) || r.produce(now) { + } st := r.h.sync.stallSnapshot() // the age the next tick reads r.oldest = max(r.oldest, now.Add(time.Second).Sub(st.lastBlockTime)) @@ -99,7 +109,7 @@ func (r *tipRun) consume(now time.Time) bool { if !ok { break } - sync.advance(resp.PeerID, r.blockTimes[resp.Block.Height], defaultPrecision) + sync.advance(r.blockTimes[resp.Block.Height], defaultPrecision) r.applied++ r.suppliedBy[resp.PeerID]++ } @@ -113,57 +123,47 @@ func (r *tipRun) produce(now time.Time) bool { gen := r.h.sync.jobGen produced := false for { - var ( - peer PeerData - height int64 - found bool - retry bool - ) - if r.waiting != 0 { - height, retry, r.waiting = r.waiting, r.waitingRetry, 0 - } else { + height := r.waiting + if height == 0 { if !gen.shouldJobBeGenerated(r.h.sync.backlog()) { return produced } - if peer, height, found = gen.nextSpread(); !found { - height, retry = gen.next() - } + height = gen.nextHeight() } + peer, found := r.h.sync.peerStore.reservePeer(height) if !found { - if peer, found = r.h.sync.peerStore.reservePeer(height); !found { - if gen.lowerQueued(height) { - gen.giveBack(height, retry) - continue - } - r.waiting, r.waitingRetry = height, retry - return produced - } - gen.markRequested(height, peer.peerID) + r.waiting = height + return produced } - request := peer.service.request() - request.Started() + r.waiting = 0 + r.request(peer, height, now) produced = true - if r.declining[peer.peerID] { - request.Finished() - request.complete(0) - r.h.fail(peer.peerID, height, &client.ErrBlockNotFound{Height: height}) - continue - } - r.inFlight = append(r.inFlight, &tipRequest{ - peer: peer, - request: request, - height: height, - endAt: now.Add(r.hold[peer.peerID]), - }) } } +// request has peer answer a request for height as its kind does. +func (r *tipRun) request(peer PeerData, height int64, now time.Time) { + request := peer.service.request() + request.Started() + if r.declining[peer.peerID] { + request.Finished() + request.complete(0) + r.h.fail(peer.peerID, height, &client.ErrBlockNotFound{Height: height}) + return + } + r.inFlight = append(r.inFlight, &tipRequest{ + peer: peer, + request: request, + height: height, + endAt: now.Add(r.hold[peer.peerID]), + }) +} + // TestWaitForSyncHandsOverBehindAPeerAnsweringJustInsideTheDeadline checks the // handover of a node that restarts three blocks behind the network, with a // prompt peer connected, while a far range is reported by a peer nobody asks. // One more peer answers every request with the block, one second inside the -// request deadline. It never fails a request, so no block is asked of every -// peer at once, and each block it supplies renews its credit. +// request deadline, and never fails one. // // On a chain committing a block every three minutes the node hands over within // minutes whichever peer reports first. On one committing a block every few @@ -196,7 +196,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverBehindAPeerAnswering r := &tipRun{ h: h, interval: tc.interval, - hold: map[types.NodeID]time.Duration{slow: client.RequestTimeout - time.Second}, + hold: map[types.NodeID]time.Duration{slow: requestDeadline - time.Second}, reporting: tc.reporting, blockTimes: map[int64]time.Time{}, suppliedBy: map[types.NodeID]int{}, @@ -207,24 +207,15 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverBehindAPeerAnswering r.blockTimes[r.tip-i] = start.Add(-2*time.Second - time.Duration(i)*tc.interval) } r.nextCommit = r.blockTimes[r.tip].Add(tc.interval) - // the block store holds the blocks the node applied before it stopped - var stored []time.Time - for i := maxCatchupGap; i >= 0; i-- { - stored = append(stored, r.blockTimes[height].Add(-time.Duration(i+1)*tc.interval)) - } - h.sync.mtx.Lock() - h.sync.blockTimes = stored - h.sync.mtx.Unlock() h.sync.AddPeer(newPeerData("far range", 1_000_000_000, maxPlausiblePeerHeight)) elapsed, caughtUp, ended := h.run(limit, r.step) - st := h.sync.stallSnapshot() suite.Require().True(ended, - "still in block sync after %s: %d blocks applied (%s), the last applied block never older than %s, at-tip window %s", - elapsed, r.applied, fmt.Sprint(r.suppliedBy), r.oldest, st.tipWindow) + "still in block sync after %s: %d blocks applied (%s), the last applied block never older than %s", + elapsed, r.applied, fmt.Sprint(r.suppliedBy), r.oldest) suite.Require().False(caughtUp) - suite.T().Logf("handed over after %s: %d blocks applied (%s), the last applied block never older than %s, at-tip window %s", - elapsed, r.applied, fmt.Sprint(r.suppliedBy), r.oldest, st.tipWindow) + suite.T().Logf("handed over after %s: %d blocks applied (%s), the last applied block never older than %s", + elapsed, r.applied, fmt.Sprint(r.suppliedBy), r.oldest) }) } } @@ -235,7 +226,8 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverBehindAPeerAnswering // far height and answers every request that it has no such block; it is the // only peer advertising the next height until the others report it, so it is // asked for it first. Its answer must not have every block fetched from all -// five peers: a "no such block" answer has the block asked of one other peer. +// five peers: the request the producer sends once a peer reports the block +// brings it before the others are asked. func (suite *SynchronizerTestSuite) TestWaitForSyncFetchesEachBlockAtTheTipFromOnePeer() { const ( height = int64(100) diff --git a/internal/blocksync/handover_fetch_volume_test.go b/internal/blocksync/handover_fetch_volume_test.go index 93083648b..96cb36bbe 100644 --- a/internal/blocksync/handover_fetch_volume_test.go +++ b/internal/blocksync/handover_fetch_volume_test.go @@ -130,7 +130,7 @@ func (l *linkRun) sharedStep() { for _, peerID := range l.peers { var kept []*tipRequest for _, req := range l.queues[peerID] { - if now.Sub(l.askedAt[req]) < client.RequestTimeout { + if now.Sub(l.askedAt[req]) < requestDeadline { kept = append(kept, req) continue } diff --git a/internal/blocksync/handover_late_supply_test.go b/internal/blocksync/handover_late_supply_test.go index 519ab97e7..326d9368e 100644 --- a/internal/blocksync/handover_late_supply_test.go +++ b/internal/blocksync/handover_late_supply_test.go @@ -12,12 +12,13 @@ import ( // lateSupplyRun drives a node that restarted a few blocks behind a chain // committing a block every three minutes. Requests for the height the node waits -// for are handed out as the job generator does: the first by the peer store, -// further ones by the generator once one fails or goes unanswered. The peers -// named late answer a request only once the block is lateBy old, or answer just +// for are handed out as block sync does: one at a time by the peer store, and +// once a second to the peers reserveNextBlockPeers returns. The peers named +// late answer a request only once the block is lateBy old, or answer just // inside the request deadline that they have no such block when that is further -// off; the peer named prompt answers at once. Every peer reports each block as soon as it is committed, and one more -// peer reports a far range it is never asked for. +// off; the peer named prompt answers at once. Every peer reports each block as +// soon as it is committed, and one more peer reports a far range it is never +// asked for. type lateSupplyRun struct { h *waitForSyncHarness prompt types.NodeID @@ -72,7 +73,7 @@ func (r *lateSupplyRun) apply(peer PeerData, request *serviceRequest, height int PeerID: peer.peerID, Block: &types.Block{Header: types.Header{Height: height}}, })) - sync.advance(peer.peerID, r.blockTimes[height], defaultPrecision) + sync.advance(r.blockTimes[height], defaultPrecision) r.served++ } @@ -97,24 +98,20 @@ func (r *lateSupplyRun) step(time.Duration) { } } r.outstanding = waiting - for { + for len(r.outstanding) == 0 { height, _ := sync.GetStatus() - if height > r.tip { - return - } - sync.jobGen.shouldJobBeGenerated(height, 0) - peer, spreadHeight, found := sync.jobGen.nextSpread() - if found && spreadHeight != height { - return + peer, found := sync.peerStore.reservePeer(height) + if height > r.tip || !found { + break } - if !found { - if len(r.outstanding) > 0 { - return - } - if peer, found = sync.peerStore.reservePeer(height); !found { - return - } - sync.jobGen.markRequested(height, peer.peerID) + r.ask(peer, height, now) + } + height, peers := sync.reserveNextBlockPeers() + for _, peer := range peers { + if applying, _ := sync.GetStatus(); applying != height { + // applied meanwhile; the request would bring a duplicate + sync.peerStore.Update(peer.peerID, AddNumPending(-1)) + continue } r.ask(peer, height, now) } @@ -150,11 +147,8 @@ func (r *lateSupplyRun) ask(peer PeerData, height int64, now time.Time) { // // When the prompt peer reports first it supplies every block and the node hands // over two minutes after reaching the tip. When a few peers that supply each -// block about three minutes late report first, they are asked for every block: -// before any block is applied no peer has backed its claim, a due recovery probe -// goes before a peer with no record, and each block then credits the late peer -// that supplied it. The prompt peer is never asked, and the node must still -// hand over. +// block about three minutes late report first, they are asked for every block +// first. The prompt peer must be asked as well, and the node must hand over. func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverBehindPeersSupplyingEachBlockLate() { const ( height = int64(100) @@ -224,8 +218,6 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingFarBehindPeersTha var peers []PeerData for i := 0; i < 5; i++ { peer := newPeerData(types.NodeID(fmt.Sprintf("peer %d", i)), 1, 300_000) - // each supplied blocks the node applied before the stall - peer.supplied = true peers = append(peers, peer) } h := suite.newWaitForSyncHarness(height, peers...) diff --git a/internal/blocksync/handover_review_test.go b/internal/blocksync/handover_review_test.go index 1e30a6692..6dd692484 100644 --- a/internal/blocksync/handover_review_test.go +++ b/internal/blocksync/handover_review_test.go @@ -1,60 +1,22 @@ package blocksync import ( - "context" - "testing" "time" - - "github.com/stretchr/testify/require" - - "github.com/dashpay/tenderdash/libs/log" - "github.com/dashpay/tenderdash/types" ) -// TestWaitForSyncStallBackstopIgnoresAFarClaimNoPeerBacked checks the stall -// backstop against a far claim from a peer that has supplied nothing, made while -// another peer that has backed its claim is within maxCatchupGap. Only a backed -// claim may keep a stalled node in block sync, so the distance that decides it -// must come from backed claims too. -func (suite *SynchronizerTestSuite) TestWaitForSyncStallBackstopIgnoresAFarClaimNoPeerBacked() { +// TestWaitForSyncStallBackstopIgnoresAFarClaimOnARecentBlock checks the stall +// backstop against a far claim made while another peer is within maxCatchupGap +// and the node's last block is less than staleBlockAge old. The claim is not +// verified, so it must not keep the stalled node in block sync. +func (suite *SynchronizerTestSuite) TestWaitForSyncStallBackstopIgnoresAFarClaimOnARecentBlock() { const height = int64(100) near := newPeerData("near peer", 1, height+maxCatchupGap/2) - near.supplied = true h := suite.newWaitForSyncHarness(height, near, newPeerData("claimant", 1, maxPlausiblePeerHeight)) defer h.cancel() h.setLastBlockTime(h.clock.Now().Add(-staleBlockAge / 2)) elapsed, caughtUp, ended := h.run(maxSyncStall+time.Minute, nil) - suite.Require().True(ended, "a far claim nobody backed kept a stalled node in block sync") + suite.Require().True(ended, "a far claim kept a node stalled on a recent block in block sync") suite.Require().False(caughtUp) suite.Require().Greater(elapsed, maxSyncStall) } - -// TestNextJobDoesNotWaitBehindAHeightNoIdlePeerServes checks that a height -// pushed back for a retry, which an idle peer advertises, is requested while the -// producer waits for a peer to serve a higher height only a busy peer claims. -func TestNextJobDoesNotWaitBehindAHeightNoIdlePeerServes(t *testing.T) { - const retry = int64(100) - busy := newPeerData("busy claimant", 1, 1000) - busy.numPending = maxPendingRequestsPerPeer - store := NewInMemPeerStore(newPeerData("idle peer", 1, retry+5), busy) - gen := newJobGenerator(retry+10, log.NewNopLogger(), nil, store) - - ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) - defer cancel() - done := make(chan error, 1) - go func() { - _, err := gen.nextJob(ctx) - done <- err - }() - time.Sleep(50 * time.Millisecond) - gen.pushBack(retry) - - select { - case err := <-done: - require.NoError(t, err) - case <-ctx.Done(): - require.Fail(t, "producer still waiting for a peer for a height only a busy peer claims", - "height %d stays queued although %s can serve it", retry, types.NodeID("idle peer")) - } -} diff --git a/internal/blocksync/peer_service.go b/internal/blocksync/peer_service.go index 993202475..d1b61d147 100644 --- a/internal/blocksync/peer_service.go +++ b/internal/blocksync/peer_service.go @@ -79,14 +79,6 @@ func (s *peerService) priority(pending int32, reserve bool) int { return 2 } -// slow reports whether the peer is on probation for holding requests too long. -func (s *peerService) slow() bool { - s.mtx.Lock() - defer s.mtx.Unlock() - s.classify(s.advance()) - return s.probation -} - func (s *peerService) request() *serviceRequest { s.mtx.Lock() defer s.mtx.Unlock() diff --git a/internal/blocksync/peer_service_test.go b/internal/blocksync/peer_service_test.go index 94b1f10c8..9834d91cf 100644 --- a/internal/blocksync/peer_service_test.go +++ b/internal/blocksync/peer_service_test.go @@ -234,33 +234,3 @@ func TestServiceSavedCreditCannotHideNewSlowService(t *testing.T) { require.True(t, found) require.Equal(t, types.NodeID("second"), selected.peerID) } - -// TestFindPeerPrefersBackedPeers checks that a peer that supplied a block we -// applied since it last failed a request is asked before peers that did not, -// a due recovery probe included. Otherwise new peers, which come with no -// history, could take turns holding the height block sync waits for. -func TestFindPeerPrefersBackedPeers(t *testing.T) { - back := func(peers *InMemPeerStore, peerID types.NodeID) { - peers.Update(peerID, func(_ types.NodeID, peer *PeerData) { peer.supplied = true }) - } - t.Run("over an ordinary peer listed first", func(t *testing.T) { - peers, _ := servicePeers() - back(peers, "second") - selected, found := peers.FindPeer(2) - require.True(t, found) - require.Equal(t, types.NodeID("second"), selected.peerID) - }) - t.Run("over a due probe", func(t *testing.T) { - peers, clock := servicePeers() - peer, request := beginServiceRequest(t, peers, 1) - clock.Advance(59 * time.Second) - endServiceRequest(peers, peer, request, 100) - clock.Advance(30 * time.Second) - selected, _ := peers.FindPeer(2) - require.Equal(t, peer.peerID, selected.peerID, "a due probe goes first among peers with no record") - back(peers, "second") - selected, found := peers.FindPeer(2) - require.True(t, found) - require.Equal(t, types.NodeID("second"), selected.peerID) - }) -} diff --git a/internal/blocksync/peer_store.go b/internal/blocksync/peer_store.go index 2df8490ae..64bc1a1ae 100644 --- a/internal/blocksync/peer_store.go +++ b/internal/blocksync/peer_store.go @@ -34,9 +34,6 @@ type ( // status on. Requests go out by node ID, so a reservation's copy of it is // the only generation known for a request that never gets an answer. connID uint64 - // supplied says a block this peer sent has been applied since it last - // failed a request. A new peer has supplied nothing. - supplied bool } ) @@ -128,25 +125,14 @@ func (p *InMemPeerStore) Query(spec store.QueryFunc[types.NodeID, PeerData], lim return p.store.Query(spec, limit) } -// FindPeer prefers a peer that supplied a block we applied since it last failed -// a request, then a due recovery probe, then ordinary service. +// FindPeer prefers ordinary service except when a bounded recovery probe is due. func (p *InMemPeerStore) FindPeer(height int64) (PeerData, bool) { - return p.findPeer(height, nil) -} - -// findPeer is FindPeer among peers not in except. -func (p *InMemPeerStore) findPeer(height int64, except map[types.NodeID]struct{}) (PeerData, bool) { peers := p.Query(store.AndX(peerNumPendingCond(maxPendingRequestsPerPeer, "<"), - heightBetweenPeerHeightRange(height), notIn(except)), 0) + heightBetweenPeerHeightRange(height)), 0) var selected PeerData priority := 0 for _, peer := range peers { - next := peer.service.priority(peer.numPending, false) - if next > 0 && peer.supplied { - // A peer that has backed its claim goes before any that has not. - next += 2 - } - if next > priority { + if next := peer.service.priority(peer.numPending, false); next > priority { selected, priority = peer, next } } @@ -156,15 +142,15 @@ func (p *InMemPeerStore) findPeer(height int64, except map[types.NodeID]struct{} // reservePeer checks and reserves both pending capacity and the single probe // under the store lock. Concurrent producers cannot reserve the same probe. func (p *InMemPeerStore) reservePeer(height int64) (PeerData, bool) { - return p.reservePeerExcept(height, nil) -} - -// reservePeerExcept is reservePeer among peers not in except. -func (p *InMemPeerStore) reservePeerExcept(height int64, except map[types.NodeID]struct{}) (PeerData, bool) { - peer, found := p.findPeer(height, except) + peer, found := p.FindPeer(height) if !found { return PeerData{}, false } + return p.reserve(peer, height) +} + +// reserve is reservePeer for a peer the caller selected. +func (p *InMemPeerStore) reserve(peer PeerData, height int64) (PeerData, bool) { reserved := false p.store.Update(peer.peerID, func(_ types.NodeID, current *PeerData) { if current.service != peer.service || current.numPending >= maxPendingRequestsPerPeer || @@ -197,7 +183,6 @@ func (p *InMemPeerStore) AddFailure(peerID types.NodeID, maxFailures int32) bool p.store.Update(peerID, func(_ types.NodeID, peer *PeerData) { peer.addPending(-1) peer.numFailures++ - peer.supplied = false tooMany = peer.numFailures >= maxFailures }) return tooMany @@ -208,7 +193,6 @@ func (p *InMemPeerStore) RecordMissingBlock(peerID types.NodeID, height int64) { p.store.Update(peerID, func(_ types.NodeID, peer *PeerData) { peer.addPending(-1) peer.missingThrough = max(peer.missingThrough, height) - peer.supplied = false }) } @@ -240,13 +224,6 @@ func peerNumPendingCond(val int32, op string) store.QueryFunc[types.NodeID, Peer } } -func notIn(except map[types.NodeID]struct{}) store.QueryFunc[types.NodeID, PeerData] { - return func(peerID types.NodeID, _ PeerData) bool { - _, found := except[peerID] - return !found - } -} - func heightBetweenPeerHeightRange(height int64) store.QueryFunc[types.NodeID, PeerData] { return func(peerID types.NodeID, peer PeerData) bool { return height >= peer.base && height <= peer.height && height > peer.missingThrough diff --git a/internal/blocksync/peer_store_test.go b/internal/blocksync/peer_store_test.go index 2467044a9..833eb6867 100644 --- a/internal/blocksync/peer_store_test.go +++ b/internal/blocksync/peer_store_test.go @@ -315,27 +315,3 @@ func TestAddFailureClearsPending(t *testing.T) { require.EqualValues(t, 0, peer.numPending) require.EqualValues(t, 2, peer.numFailures) } - -// TestFailureEndsSuppliedCredit checks that a failed request or an answer that -// the block is missing ends the credit a peer earned by supplying blocks, and -// that a fresh status for the peer does not. -func TestFailureEndsSuppliedCredit(t *testing.T) { - peerID := types.NodeID("peer1") - for name, fail := range map[string]func(*InMemPeerStore){ - "failed request": func(p *InMemPeerStore) { p.AddFailure(peerID, 10) }, - "missing block": func(p *InMemPeerStore) { p.RecordMissingBlock(peerID, 5) }, - } { - t.Run(name, func(t *testing.T) { - supplier := newPeerData(peerID, 1, 100) - supplier.supplied = true - inmem := NewInMemPeerStore(supplier) - inmem.Upsert(newPeerData(peerID, 1, 200)) - peer, _ := inmem.Get(peerID) - require.True(t, peer.supplied, "a status update keeps the credit") - - fail(inmem) - peer, _ = inmem.Get(peerID) - require.False(t, peer.supplied) - }) - } -} diff --git a/internal/blocksync/reactor.go b/internal/blocksync/reactor.go index 5b33dc7fa..bb9018432 100644 --- a/internal/blocksync/reactor.go +++ b/internal/blocksync/reactor.go @@ -34,14 +34,14 @@ const ( // hand over to consensus after this duration of inactivity, even when peers // still report higher blocks, so that a wedged synchronizer cannot keep the - // node out of consensus forever - but only within maxCatchupGap of the tip - // as stallVerdictFor measures it + // node out of consensus forever - but only within maxCatchupGap of the tip, + // or when the inactivity began on a block less than staleBlockAge old maxSyncStall = 10 * time.Minute - // maxCatchupGap is how far behind the node may be and still hand over to - // consensus on the stall backstop above: behind the highest claim of a peer - // that supplied a block, or of any peer once the last block is older than - // staleBlockAge. It also bounds the claims a node at the tip targets. + // maxCatchupGap is how far behind the highest height any peer claims the node + // may be and still hand over to consensus on the stall backstop above when + // its last block is older than staleBlockAge. It also bounds the claims a + // node with a recent last block passes on as the consensus catch-up target. // // What is left after the handover is covered by consensus catch-up, which // moves about one block per gossip cycle: it closes a gap of a few blocks in @@ -56,18 +56,14 @@ const ( // Platform runs with and a margin for extra rounds and clock skew. tipRecency = 5 * time.Minute - // tipLag is the shortest at-tip window: the stall after which a node with - // nothing to fetch hands over, and a status round to learn there is nothing. - // A pause in block production then ends block sync by one rule or the other. - tipLag = syncTimeout + statusUpdateIntervalSeconds*time.Second - // tipGrace is how long the node must stay at the tip before higher height - // claims stop keeping it in block sync. + // claims stop keeping it in block sync. A node still fetching blocks reaches + // the highest claim in that time when its peers serve them. tipGrace = 2 * syncTimeout // staleBlockAge is how old the last applied block must be, when a stall - // begins, for every claim, backed or not, to count in the stall backstop. - // It is longer than any halt a restarted node is expected to help end. + // begins, for peer height claims to count in the stall backstop. It is + // longer than any halt a restarted node is expected to help end. staleBlockAge = 24 * time.Hour ) diff --git a/internal/blocksync/synchronizer.go b/internal/blocksync/synchronizer.go index 5ab1c8be8..3278fecba 100644 --- a/internal/blocksync/synchronizer.go +++ b/internal/blocksync/synchronizer.go @@ -4,7 +4,6 @@ import ( "context" "errors" "fmt" - "slices" "sync/atomic" "time" @@ -13,7 +12,6 @@ import ( "github.com/dashpay/tenderdash/internal/p2p" "github.com/dashpay/tenderdash/internal/p2p/client" - sm "github.com/dashpay/tenderdash/internal/state" "github.com/dashpay/tenderdash/libs/log" "github.com/dashpay/tenderdash/libs/service" "github.com/dashpay/tenderdash/libs/workerpool" @@ -133,11 +131,13 @@ type ( logger log.Logger lastAdvance time.Time - // blockTimes holds the times of the last maxCatchupGap+1 applied blocks, - // oldest first, and precision the synchrony precision of the state they - // left. Guarded by mtx. - blockTimes []time.Time - precision time.Duration + // lastBlockTime is the time of the last applied block and precision the + // synchrony precision of the state it left. Guarded by mtx. + lastBlockTime time.Time + precision time.Duration + // asked holds, for the height to apply, the peers askEveryPeer has seen + // advertise it, and whether it asked them for it yet. Guarded by mtx. + asked map[types.NodeID]bool mtx sync.RWMutex @@ -214,12 +214,12 @@ func NewSynchronizer(start int64, client client.BlockClient, blockExec *blockApp workerPool: workerpool.New(poolWorkerSize, workerpool.WithLogger(logger)), pendingToApply: map[int64]BlockResponse{}, precision: types.DefaultSynchronyParams().Precision, + asked: map[types.NodeID]bool{}, } for _, opt := range opts { opt(bp) } bp.peerStore.clock = bp.clock - bp.jobGen.applyState = bp.applyState bp.BaseService = *service.NewBaseService(logger, "Synchronizer", bp) return bp } @@ -255,7 +255,7 @@ func (s *Synchronizer) OnStart(ctx context.Context) error { s.lastMonitorUpdate = s.lastAdvance if s.applier != nil { state := s.applier.State() - s.blockTimes = storedBlockTimes(s.applier.store, state.LastBlockHeight, state.LastBlockTime) + s.lastBlockTime = state.LastBlockTime s.precision = state.ConsensusParams.Synchrony.SynchronyParamsOrDefaults().Precision } s.ctx, s.cancel = context.WithCancel(ctx) @@ -266,6 +266,7 @@ func (s *Synchronizer) OnStart(ctx context.Context) error { s.consumerDone = make(chan struct{}) s.workerPool.Run(s.ctx) go s.runHandler(s.ctx, s.produceJob) + go s.runHandler(s.ctx, s.askEveryPeer) go func() { defer close(s.consumerDone) s.runHandler(s.ctx, func(handlerCtx context.Context) error { @@ -276,22 +277,6 @@ func (s *Synchronizer) OnStart(ctx context.Context) error { return nil } -// storedBlockTimes returns the times of the blocks the store holds right below -// height, up to maxCatchupGap of them and oldest first, followed by lastTime, -// the time of the block at height. -func storedBlockTimes(store sm.BlockStore, height int64, lastTime time.Time) []time.Time { - times := []time.Time{lastTime} - for h := height - 1; store != nil && h >= 1 && h >= height-maxCatchupGap; h-- { - meta := store.LoadBlockMeta(h) - if meta == nil { - break - } - times = append(times, meta.Header.Time) - } - slices.Reverse(times) - return times -} - func (s *Synchronizer) OnStop() { s.cancel() s.workerPool.Stop(context.Background()) @@ -362,7 +347,7 @@ func (s *Synchronizer) consumeJobResult(ctx, applyCtx context.Context) error { if errors.As(bfErr.err, &missing) && missing.Height == bfErr.height { s.peerStore.RecordMissingBlock(bfErr.peerID, bfErr.height) if pending { - s.jobGen.requeue(bfErr.height) + s.jobGen.pushBack(bfErr.height) } return nil } @@ -450,13 +435,56 @@ func (s *Synchronizer) backlog() (applyHeight int64, pendingBytes int) { return s.height, pendingBytes } -// applyState reports the height block sync waits to apply and whether its block -// has arrived. -func (s *Synchronizer) applyState() (int64, bool) { - s.mtx.RLock() - defer s.mtx.RUnlock() - _, received := s.pendingToApply[s.height] - return s.height, received +// askEveryPeer asks, once a second, every peer reserveNextBlockPeers returns for +// the height block sync waits to apply. +func (s *Synchronizer) askEveryPeer(ctx context.Context) error { + select { + case <-ctx.Done(): + return ctx.Err() + case <-s.clock.After(time.Second): + } + height, peers := s.reserveNextBlockPeers() + for _, peer := range peers { + s.jobProgressCounter.Add(1) + if err := s.workerPool.Send(ctx, workerpool.NewJob(blockFetchJobHandler(s.client, peer, height))); err != nil { + s.jobProgressCounter.Add(-1) + return err + } + } + return nil +} + +// reserveNextBlockPeers returns the height block sync waits to apply and, once +// its block is serviceGrace overdue, every peer to ask for it: each peer that +// advertises it, once. Blocks are applied in order, so whichever peers hold the +// requests made for that height so far must not decide how long the others, +// which may serve it at once, go unasked. +// +// A peer is asked on the second call that finds it advertising the height, so +// the request the job producer sends when a peer first reports a new block gets +// a second to bring it, and a block served promptly is fetched once. +func (s *Synchronizer) reserveNextBlockPeers() (int64, []PeerData) { + s.mtx.Lock() + defer s.mtx.Unlock() + if _, received := s.pendingToApply[s.height]; received || s.clock.Since(s.lastAdvance) <= serviceGrace { + return s.height, nil + } + var peers []PeerData + for _, peer := range s.peerStore.Query(heightBetweenPeerHeightRange(s.height), 0) { + asked, seen := s.asked[peer.peerID] + if !seen { + s.asked[peer.peerID] = false + continue + } + if asked { + continue + } + if reserved, ok := s.peerStore.reserve(peer, s.height); ok { + s.asked[peer.peerID] = true + peers = append(peers, reserved) + } + } + return s.height, peers } // GetStatus returns synchronizer's height, count of in progress requests @@ -490,16 +518,16 @@ func (s *Synchronizer) IsCaughtUp() bool { // peer holds the block we are waiting for there is something to retry, so keep // going and say so loudly. Give up on the stall only once no peer can serve that // block, or once the stall has outlasted maxSyncStall within maxCatchupGap of -// the highest claim a peer holding the block has backed, so that a wedged -// synchronizer can still hand over rather than blocking forever without handing -// over a node consensus cannot catch up. +// the tip, so that a wedged synchronizer can still hand over rather than +// blocking forever without handing over a node consensus cannot catch up. // -// A height claim costs a peer nothing, and a node ID costs nothing either, so -// what a peer claims cannot keep the node in block sync once it is at the tip. -// The time of the last applied block tells it so: the validators commit to it, -// and a node far behind applies old blocks. Once the node has stayed at the tip -// by that time for tipGrace, it hands over whatever peers claim, even though -// blocks arriving meanwhile keep it from stalling. +// What peers report about themselves is not verified, so it cannot be what +// keeps the node in block sync once it has caught up. The time of the last +// applied block is: the validators commit to it, and a node far behind applies +// old blocks. Once that time has been within tipRecency of the local clock for +// tipGrace, the node hands over whatever peers report, even though blocks +// arriving meanwhile keep it from stalling. A stall of maxSyncStall that began +// on a block less than staleBlockAge old ends block sync the same way. func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHeight int64) { ticker := s.clock.NewTicker(switchToConsensusIntervalSeconds * time.Second) defer ticker.Stop() @@ -519,7 +547,7 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe // At the tip, claims out of reach of it must not hold back our own // proposals, whichever way block sync ends. target := targetHeight - if atTip(st.lastBlockTime, st.tipWindow, st.precision, now) { + if isRecent(st.lastBlockTime, tipRecency, st.precision, now) { target = nearTarget } else { atTipSince = now @@ -527,11 +555,12 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe if st.hasPeers && st.height >= st.maxPeerHeight { return true, target } - // Far back in the chain's history any claim counts. The age is taken - // when the stall began, so a stall does not age a stopped chain into it. - behindBy := st.maxSuppliedHeight - st.height - if now.Add(-st.stalledFor).Sub(st.lastBlockTime) > staleBlockAge { - behindBy = st.maxServableHeight - st.height + // Only far back in the chain's history does the distance to what peers + // report keep a stalled node in block sync. The age is taken when the + // stall began, so a stall does not age a stopped chain into it. + behindBy := st.maxServableHeight - st.height + if isRecent(st.lastBlockTime, staleBlockAge, st.precision, now.Add(-st.stalledFor)) { + behindBy = 0 } verdict := stallVerdictFor(st.servable, behindBy, st.stalledFor, now.Sub(atTipSince)) switch verdict { @@ -562,7 +591,7 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe "height", st.height, "max_peer_height", st.maxPeerHeight, "max_servable_height", st.maxServableHeight, - "max_backed_height", st.maxSuppliedHeight, + "last_block_time", st.lastBlockTime, "stalled_for", st.stalledFor, ) return false, target @@ -588,11 +617,11 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe } } -// atTip reports whether a last applied block with blockTime puts the node at -// the tip: it is at most window old, and no further ahead of the local clock -// than precision, the bound validators hold block times to. Further ahead, the -// local clock is behind the validators' and cannot tell how old the block is. -func atTip(blockTime time.Time, window, precision time.Duration, now time.Time) bool { +// isRecent reports whether blockTime, the time of the last applied block, is +// at most window old at now, and no further ahead of it than precision, the +// bound validators hold block times to. Further ahead, the local clock is behind +// the validators' and cannot tell how old the block is. +func isRecent(blockTime time.Time, window, precision time.Duration, now time.Time) bool { age := now.Sub(blockTime) return age <= window && age >= -precision } @@ -608,17 +637,12 @@ type stallState struct { servable bool maxServableHeight int64 maxPeerHeight int64 - // maxSuppliedHeight is the highest claim of a peer advertising height that - // has supplied a block we applied since it last failed a request; 0 if none. - maxSuppliedHeight int64 // maxNearHeight is the highest claim within maxCatchupGap of the last // applied block. maxNearHeight int64 - // lastBlockTime is the time of the last applied block; within tipWindow of - // the local clock, and no further than precision ahead of it, the node is at - // the tip. + // lastBlockTime is the time of the last applied block and precision the + // synchrony precision of the state it left. lastBlockTime time.Time - tipWindow time.Duration precision time.Duration } @@ -633,17 +657,12 @@ type stallState struct { // claims to be. Only servable peers determine the stall gap; the overall maximum // remains the handover target so pruning cannot erase evidence of a higher tip. // -// The node is at the tip when its last block is no older than the chain took -// to commit the last maxCatchupGap blocks, or than tipLag where that is longer, -// and in any case no older than tipRecency. Until the node has seen that many -// blocks, the pace of those it has seen stands in; with none, tipRecency alone. -// // They are read under one lock because advance() stamps the height, the advance -// time, the block time and the supplier's credit together under that same lock. -// A block applied concurrently therefore lands either wholly inside the snapshot -// or wholly outside it, and the verdict can never pair a height with a staleness -// or a servability measured against a different one. Read separately, it could: -// a height read before an advance, checked for servability after one, reports +// time and the block time together under that same lock. A block applied +// concurrently therefore lands either wholly inside the snapshot or wholly +// outside it, and the verdict can never pair a height with a staleness or a +// servability measured against a different one. Read separately, it could: a +// height read before an advance, checked for servability after one, reports // nothing to fetch while the height we had by then moved on to is served. Ending // block sync is a one-way door, so a stop assembled from two inconsistent // readings leaves the node in consensus catch-up it cannot leave. @@ -652,19 +671,11 @@ func (s *Synchronizer) stallSnapshot() stallState { defer s.mtx.RUnlock() peers := s.peerStore.All() st := stallState{ - height: s.height, - stalledFor: s.clock.Since(s.lastAdvance), - hasPeers: len(peers) > 0, - tipWindow: tipRecency, - precision: s.precision, - } - if last := len(s.blockTimes) - 1; last >= 0 { - st.lastBlockTime = s.blockTimes[last] - if last > 0 { - // Fewer than maxCatchupGap intervals since start are scaled up to it. - span := s.blockTimes[last].Sub(s.blockTimes[0]) / time.Duration(last) * time.Duration(maxCatchupGap) - st.tipWindow = min(tipRecency, max(tipLag, span)) - } + height: s.height, + stalledFor: s.clock.Since(s.lastAdvance), + hasPeers: len(peers) > 0, + lastBlockTime: s.lastBlockTime, + precision: s.precision, } hasHeight := heightBetweenPeerHeightRange(s.height) for _, peer := range peers { @@ -675,9 +686,6 @@ func (s *Synchronizer) stallSnapshot() stallState { if hasHeight(peer.peerID, peer) { st.servable = true st.maxServableHeight = max(st.maxServableHeight, peer.height) - if peer.supplied { - st.maxSuppliedHeight = max(st.maxSuppliedHeight, peer.height) - } } } return st @@ -693,14 +701,13 @@ const ( stopNothingToFetch // stopStalledTooLong means a peer still holds it but we have to hand over anyway stopStalledTooLong - // stopAtTip means the node has stayed at the tip for tipGrace + // stopAtTip means the last applied block has stayed recent for tipGrace stopAtTip ) // stallVerdictFor decides what to do when block sync has made no progress for // stalledFor, given whether any peer holds the block we are waiting for and how -// many blocks behind the highest height a servable peer that backed its claim -// says we are. +// many blocks behind the highest height a servable peer claims we are. // // Waiting on a block someone has is a reason to keep retrying, not to stop: // handing over to consensus is effectively irreversible, so stopping while @@ -716,16 +723,13 @@ const ( // the backstop buys nothing and costs what dashpay/tenderdash#1413 describes, so // block sync keeps retrying and says so every interval. // -// Advertising costs nothing, so near the tip behindBy counts only the claims of -// peers that backed them by supplying a block we applied since they last failed -// a request; with no such peer the node is not known to be behind at all. A -// stall of maxSyncStall means every peer asked for the block meanwhile failed, -// so only one never asked keeps that credit, and a new peer has none. With a -// last block older than staleBlockAge every claim counts. +// The caller passes no distance for a stall that began on a recent block, so +// such a stall ends at maxSyncStall whatever peers report: askEveryPeer has by +// then asked every peer advertising the block for it. // -// atTipFor is how long the node has been at the tip by the time of its last -// block. Unlike stalledFor it is not reset by applying blocks, so a node -// following the network is not kept in block sync by a claim nobody can reach. +// atTipFor is how long the last applied block has been recent. Unlike +// stalledFor it is not reset by applying blocks, so a node following the +// network is not kept in block sync by a height nobody serves. func stallVerdictFor(servable bool, behindBy int64, stalledFor, atTipFor time.Duration) stallVerdict { switch { case atTipFor > tipGrace: @@ -784,6 +788,7 @@ func (s *Synchronizer) dropPeer(peerID types.NodeID) []int64 { } } s.peerStore.Delete(peerID) + delete(s.asked, peerID) return heights } @@ -801,7 +806,7 @@ func (s *Synchronizer) applyBlock(ctx context.Context) (BlockResponse, error) { return resp, fmt.Errorf("cannot apply block: %w", err) } state := s.applier.State() - s.advance(resp.PeerID, state.LastBlockTime, state.ConsensusParams.Synchrony.SynchronyParamsOrDefaults().Precision) + s.advance(state.LastBlockTime, state.ConsensusParams.Synchrony.SynchronyParamsOrDefaults().Precision) s.updateMonitor() } } @@ -814,18 +819,13 @@ func (s *Synchronizer) getPendingResponse() (BlockResponse, bool) { } // advance moves past the block just applied, which carries blockTime and left -// the chain with synchrony precision, and credits supplier, the peer that sent -// it. The pending response may be gone by now: the peer can be removed while -// its block is applied. -func (s *Synchronizer) advance(supplier types.NodeID, blockTime time.Time, precision time.Duration) { +// the chain with synchrony precision. +func (s *Synchronizer) advance(blockTime time.Time, precision time.Duration) { s.mtx.Lock() defer s.mtx.Unlock() + s.lastBlockTime = blockTime s.precision = precision - s.peerStore.Update(supplier, func(_ types.NodeID, peer *PeerData) { peer.supplied = true }) - s.blockTimes = append(s.blockTimes, blockTime) - if len(s.blockTimes) > int(maxCatchupGap)+1 { - s.blockTimes = s.blockTimes[1:] - } + clear(s.asked) delete(s.pendingToApply, s.height) s.height++ s.lastAdvance = s.clock.Now() diff --git a/internal/blocksync/synchronizer_test.go b/internal/blocksync/synchronizer_test.go index d8f44bae8..4b084b2f9 100644 --- a/internal/blocksync/synchronizer_test.go +++ b/internal/blocksync/synchronizer_test.go @@ -1326,27 +1326,6 @@ func (suite *SynchronizerTestSuite) TestNoBlockResponseForWrongHeightPreservesRa suite.EqualValues(1, peer.numFailures) } -// TestAdvanceCreditsTheSupplier checks that applying a block credits the peer -// that supplied it and stamps its time in the same step that moves the height, -// so a stall snapshot never pairs the new height with old values. -func (suite *SynchronizerTestSuite) TestAdvanceCreditsTheSupplier() { - const peerID = types.NodeID("supplier") - pool := NewSynchronizer(100, suite.client, nil) - pool.AddPeer(newPeerData(peerID, 1, 200)) - pool.AddPeer(newPeerData("bystander", 1, 200)) - blockTime := time.Unix(1_700_000_000, 0) - pool.pendingToApply[100] = BlockResponse{PeerID: peerID} - pool.advance(peerID, blockTime, defaultPrecision) - supplier, _ := pool.peerStore.Get(peerID) - suite.True(supplier.supplied) - bystander, _ := pool.peerStore.Get("bystander") - suite.False(bystander.supplied, "only the peer whose block was applied is credited") - st := pool.stallSnapshot() - suite.EqualValues(101, st.height) - suite.Equal(blockTime, st.lastBlockTime) - suite.EqualValues(200, st.maxSuppliedHeight) -} - // defaultPrecision is the synchrony precision of a chain on default parameters. var defaultPrecision = types.DefaultSynchronyParams().Precision @@ -1367,45 +1346,6 @@ func (suite *SynchronizerTestSuite) TestStartSeedsTheLastBlockTime() { suite.Equal(2*time.Second, st.precision) } -// TestStartLearnsTheChainsPaceFromTheStore checks that block sync starts out -// knowing how fast the chain commits blocks from the blocks the store holds, so -// a node restarting minutes behind a fast chain is not taken for one at the -// tip. Only an unbroken run of stored blocks counts. -func (suite *SynchronizerTestSuite) TestStartLearnsTheChainsPaceFromTheStore() { - const lastHeight = int64(50) - lastTime := time.Unix(1_700_000_000, 0) - for _, tc := range []struct { - name string - missing int64 - want time.Duration - }{ - {name: "a fast chain", want: tipLag}, - {name: "the block before the last is not stored", missing: lastHeight - 1, want: tipRecency}, - } { - suite.Run(tc.name, func() { - store := mocks.NewBlockStore(suite.T()) - store.On("LoadBlockMeta", mock.Anything).Maybe().Return(func(height int64) *types.BlockMeta { - if height == tc.missing { - return nil - } - return &types.BlockMeta{Header: types.Header{ - Height: height, - Time: lastTime.Add(-time.Duration(lastHeight-height) * time.Second), - }} - }) - state := suite.initialState.Copy() - state.LastBlockHeight, state.LastBlockTime = lastHeight, lastTime - applier := newBlockApplier(suite.blockExec, store, applierWithState(state)) - pool := NewSynchronizer(lastHeight+1, suite.client, applier, WithWorkerPool(workerpool.New(0))) - suite.Require().NoError(pool.Start(context.Background())) - defer pool.Stop() - st := pool.stallSnapshot() - suite.Equal(lastTime, st.lastBlockTime) - suite.Equal(tc.want, st.tipWindow) - }) - } -} - // TestApplyingABlockTakesThePrecisionOfTheNewState checks that the synchrony // precision follows the chain's parameters as blocks are applied, not those it // had when block sync started. @@ -1439,45 +1379,6 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncDoesNotWaitForABlockBeingAppl suite.Require().True(ended, "the handover loop waited for the block being applied") } -// TestApplyCreditsASupplierRemovedMeanwhile checks that a peer whose block is -// applied is credited for it, and the block's time recorded, even when the peer -// is removed and its pending response dropped while the block is being applied. -func (suite *SynchronizerTestSuite) TestApplyCreditsASupplierRemovedMeanwhile() { - const supplier = types.NodeID("supplier") - decoded, err := BlockResponseFromProto(suite.responses[0], supplier) - suite.Require().NoError(err) - resp := *decoded - state := suite.initialState.Copy() - state.LastBlockTime = resp.Block.Time - applier := newBlockApplier(suite.blockExec, suite.store, applierWithState(state)) - pool := NewSynchronizer(1, suite.client, applier) - pool.AddPeer(newPeerData(supplier, 1, 100)) - suite.blockExec. - On("ValidateBlock", mock.Anything, mock.Anything, mock.Anything, mock.Anything). - Return(nil) - suite.blockExec. - On("VerifyCommit", mock.Anything, mock.Anything, mock.Anything, mock.Anything). - Return(types.VerifiedCommit{}, nil) - expectApply(suite.blockExec, func(call *mock.Call) { call.Once() }) - suite.store. - On("SaveBlock", mock.Anything, mock.Anything, mock.Anything). - Run(func(mock.Arguments) { - // the peer is dropped, and reports its status again, mid-apply - pool.RemovePeer(supplier) - pool.AddPeer(newPeerData(supplier, 1, 100)) - }). - Once() - suite.Require().NoError(pool.addBlock(resp)) - - _, err = pool.applyBlock(context.Background()) - suite.Require().NoError(err) - peer, _ := pool.peerStore.Get(supplier) - suite.True(peer.supplied, "the peer whose block was applied must be credited") - st := pool.stallSnapshot() - suite.EqualValues(2, st.height) - suite.Equal(resp.Block.Time, st.lastBlockTime) -} - // TestStatusRefreshUpdatesAdvertisedRange checks that a status response still does // the one thing it is for: moving the range of blocks the peer claims to serve. It // is the guard on the tests above, which a synchronizer that ignored status @@ -1622,7 +1523,7 @@ func TestStallVerdictFor(t *testing.T) { want: stopStalledTooLong, }, { - name: "at the tip past the grace, behind a backed claim", + name: "at the tip past the grace, behind a far claim", servable: true, behindBy: maxCatchupGap + 1, atTipFor: tipGrace + time.Second, @@ -1672,7 +1573,7 @@ func (suite *SynchronizerTestSuite) newWaitForSyncHarness(height int64, peers .. // waiter on the fake clock. sync.lastAdvance = clock.Now() // The node starts at the tip: its last block is recent. - sync.blockTimes = []time.Time{clock.Now()} + sync.lastBlockTime = clock.Now() for _, peer := range peers { sync.AddPeer(peer) } @@ -1782,16 +1683,11 @@ func (suite *SynchronizerTestSuite) TestStallSnapshotServableHeight() { const height = int64(10) busyPeer := newPeerData("busy", 1, 30) busyPeer.numPending = maxPendingRequestsPerPeer - supplied := func(peer PeerData) PeerData { - peer.supplied = true - return peer - } testCases := []struct { name string peers []PeerData wantMax int64 wantUsable int64 - wantBacked int64 wantNear int64 }{ {name: "no peers"}, @@ -1826,20 +1722,6 @@ func (suite *SynchronizerTestSuite) TestStallSnapshotServableHeight() { wantUsable: 30, wantNear: height + maxCatchupGap - 1, }, - { - name: "a servable peer has supplied a block", - peers: []PeerData{newPeerData("archive", 1, 20), supplied(newPeerData("recent", 5, 30))}, - wantMax: 30, - wantUsable: 30, - wantBacked: 30, - }, - { - name: "only a peer that cannot serve the height has supplied a block", - peers: []PeerData{supplied(newPeerData("older", 1, 9)), newPeerData("recent", 5, 30)}, - wantMax: 30, - wantUsable: 30, - wantNear: 9, - }, } for _, tc := range testCases { suite.Run(tc.name, func() { @@ -1852,42 +1734,12 @@ func (suite *SynchronizerTestSuite) TestStallSnapshotServableHeight() { suite.Require().Equal(tc.wantMax, st.maxPeerHeight) suite.Require().Equal(tc.wantUsable, st.maxServableHeight) suite.Require().Equal(tc.wantUsable > 0, st.servable) - suite.Require().Equal(tc.wantBacked, st.maxSuppliedHeight) suite.Require().Equal(tc.wantNear, st.maxNearHeight) suite.Require().Equal(len(tc.peers) > 0, st.hasPeers) }) } } -// TestStallSnapshotTipWindow checks how old the last block may be for the node -// to be at the tip: as old as the chain took to commit the last maxCatchupGap -// blocks, but never less than tipLag nor more than tipRecency. -func TestStallSnapshotTipWindow(t *testing.T) { - start := time.Unix(1_700_000_000, 0) - for _, tc := range []struct { - name string - blocks int - interval time.Duration - want time.Duration - }{ - {name: "no block applied", want: tipRecency}, - {name: "one block applied", blocks: 1, interval: time.Second, want: tipRecency}, - {name: "fewer blocks than the gap, fast", blocks: 3, interval: time.Second, want: tipLag}, - {name: "fewer blocks than the gap, moderate", blocks: 3, interval: 15 * time.Second, want: 150 * time.Second}, - {name: "a fast chain", blocks: 100, interval: time.Second, want: tipLag}, - {name: "a moderate chain", blocks: 100, interval: 15 * time.Second, want: 150 * time.Second}, - {name: "a slow chain", blocks: 100, interval: 3 * time.Minute, want: tipRecency}, - } { - t.Run(tc.name, func(t *testing.T) { - sync := NewSynchronizer(1, nil, nil) - for i := 0; i < tc.blocks; i++ { - sync.advance("", start.Add(time.Duration(i)*tc.interval), defaultPrecision) - } - require.Equal(t, tc.want, sync.stallSnapshot().tipWindow) - }) - } -} - // Advancing the height changes servability and its maximum together with the // stall timestamp; the snapshot must not mix values from opposite sides. func (suite *SynchronizerTestSuite) TestStallSnapshotIsOneObservation() { @@ -1908,7 +1760,7 @@ func (suite *SynchronizerTestSuite) TestStallSnapshotIsOneObservation() { applied := make(chan struct{}) go func() { <-ready - sync.advance("", time.Time{}, defaultPrecision) + sync.advance(time.Time{}, defaultPrecision) close(applied) }() runtime.Gosched() @@ -1996,7 +1848,7 @@ func (h *waitForSyncHarness) tipBlocks(tipPeer types.NodeID, interval time.Durat PeerID: tipPeer, Block: &types.Block{Header: types.Header{Height: height}}, })) - h.sync.advance(tipPeer, h.clock.Now(), defaultPrecision) + h.sync.advance(h.clock.Now(), defaultPrecision) } } @@ -2004,7 +1856,7 @@ func (h *waitForSyncHarness) tipBlocks(tipPeer types.NodeID, interval time.Durat func (h *waitForSyncHarness) setLastBlockTime(blockTime time.Time) { h.sync.mtx.Lock() defer h.sync.mtx.Unlock() - h.sync.blockTimes = []time.Time{blockTime} + h.sync.lastBlockTime = blockTime } const ( @@ -2066,7 +1918,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastUnservedClaims() // TestWaitForSyncHandsOverPastRotatingUnservedClaims checks that unserved far // claims arriving under a fresh peer ID before the previous one has failed do -// not keep a node at the tip in block sync indefinitely. A node ID costs a peer +// not keep a node at the tip in block sync for good. A node ID costs a peer // nothing, so a claim record kept per ID must not be the only way out. func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastRotatingUnservedClaims() { const ( @@ -2252,7 +2104,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAnUnservedClaimO h.sync.AddPeer(newPeerData(tip, 1, next-1+perRound)) case 2 * time.Second: for i := int64(0); i < perRound; i++ { - h.sync.advance("", h.clock.Now(), defaultPrecision) + h.sync.advance(h.clock.Now(), defaultPrecision) } } }) @@ -2264,38 +2116,33 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAnUnservedClaimO // TestWaitForSyncHandsOverPastAnUnservedClaimOnAHaltedChain checks that a chain // that stopped producing blocks, as one waiting for this node's vote may, does not // leave the node in block sync behind a claim nobody serves. Its last block is -// too old to say the node is at the tip, so the wedge limit decides, and a peer -// that supplied blocks before the chain stopped loses that credit by failing. +// too old to say the node is at the tip, so the wedge limit decides. func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAnUnservedClaimOnAHaltedChain() { const ( height = int64(100) tipPeer = types.NodeID("tip peer") unserving = types.NodeID("unserving peer") ) - for _, supplied := range []bool{false, true} { - suite.Run(fmt.Sprintf("supplied before %t", supplied), func() { - claimant := newPeerData(unserving, 1, maxPlausiblePeerHeight) - claimant.supplied = supplied - h := suite.newWaitForSyncHarness(height, newPeerData(tipPeer, 1, height-1), claimant) - defer h.cancel() - h.setLastBlockTime(h.clock.Now().Add(-tipRecency)) + h := suite.newWaitForSyncHarness(height, + newPeerData(tipPeer, 1, height-1), + newPeerData(unserving, 1, maxPlausiblePeerHeight)) + defer h.cancel() + h.setLastBlockTime(h.clock.Now().Add(-tipRecency)) - // the peer fails the probes it is sent for the height, too seldom to - // be dropped for it, and reports its status again after each - suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) - const probeEvery = 150 * time.Second - elapsed, caughtUp, ended := h.run(maxSyncStall+time.Minute, func(elapsed time.Duration) { - if elapsed > 0 && elapsed%probeEvery == 0 { - h.fail(unserving, height, client.ErrPeerNotResponded) - h.sync.AddPeer(newPeerData(unserving, 1, maxPlausiblePeerHeight)) - } - }) - suite.Require().True(ended, "block sync kept waiting on a halted chain behind a claim nobody serves") - suite.Require().False(caughtUp) - suite.Require().Greater(elapsed, maxSyncStall, "handed over with no recent block before the wedge limit") - suite.Require().LessOrEqual(elapsed, maxSyncStall+unservedHandoverSlack) - }) - } + // the peer fails the probes it is sent for the height, too seldom to + // be dropped for it, and reports its status again after each + suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) + const probeEvery = 150 * time.Second + elapsed, caughtUp, ended := h.run(maxSyncStall+time.Minute, func(elapsed time.Duration) { + if elapsed > 0 && elapsed%probeEvery == 0 { + h.fail(unserving, height, client.ErrPeerNotResponded) + h.sync.AddPeer(newPeerData(unserving, 1, maxPlausiblePeerHeight)) + } + }) + suite.Require().True(ended, "block sync kept waiting on a halted chain behind a claim nobody serves") + suite.Require().False(caughtUp) + suite.Require().Greater(elapsed, maxSyncStall, "handed over with no recent block before the wedge limit") + suite.Require().LessOrEqual(elapsed, maxSyncStall+unservedHandoverSlack) } // flakySource returns a step for run in which source, the only peer far ahead, @@ -2317,7 +2164,7 @@ func (h *waitForSyncHarness) flakySource(source types.NodeID, deliverEvery, fail Block: &types.Block{Header: types.Header{Height: next}}, })) // a block the network committed long ago - h.sync.advance(source, time.Time{}, defaultPrecision) + h.sync.advance(time.Time{}, defaultPrecision) h.sync.peerStore.Update(source, ResetFailures()) } } @@ -2413,10 +2260,10 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingWhenTheLastBlockL "handed over after %s, 10000 blocks behind, on a block time ahead of the local clock", elapsed) } -// TestAtTip checks which last block times put the node at the tip: none older -// than the window, and none further ahead of the local clock than the -// precision validators hold block times to. -func TestAtTip(t *testing.T) { +// TestIsRecent checks which last block times count as recent: none older than +// the window, and none further ahead of the local clock than the precision +// validators hold block times to. +func TestIsRecent(t *testing.T) { const ( window = time.Minute precision = time.Second @@ -2434,10 +2281,10 @@ func TestAtTip(t *testing.T) { {name: "ahead beyond the precision", age: -precision - time.Millisecond}, } { t.Run(tc.name, func(t *testing.T) { - require.Equal(t, tc.want, atTip(now.Add(-tc.age), window, precision, now)) + require.Equal(t, tc.want, isRecent(now.Add(-tc.age), window, precision, now)) }) } - require.False(t, atTip(time.Time{}, window, precision, now), "no block time") + require.False(t, isRecent(time.Time{}, window, precision, now), "no block time") } // TestWaitForSyncAtTipHandoverTargetsNearClaims checks that handing over at the @@ -2619,7 +2466,7 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAFarClaimFromAPe PeerID: relay, Block: &types.Block{Header: types.Header{Height: next}}, })) - h.sync.advance(relay, h.clock.Now(), defaultPrecision) + h.sync.advance(h.clock.Now(), defaultPrecision) h.sync.peerStore.Update(relay, ResetFailures()) }) suite.Require().True(ended, "block sync kept following the tip behind a claim of a peer supplying the tip") @@ -2627,14 +2474,14 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncHandsOverPastAFarClaimFromAPe suite.Require().LessOrEqual(elapsed, tipGrace+unservedHandoverSlack) } -// TestWaitForSyncKeepsSyncingOnAFastChainWhileOutOfReach checks that on a chain -// committing two blocks a second, a node minutes behind - hundreds of blocks, -// far more than consensus catch-up closes - is not taken for one at the tip -// because its last block is less than tipRecency old, whatever another peer -// claims: not while it applies blocks at the chain's pace, not when it has -// applied too few since it started to span maxCatchupGap blocks, and not when -// the blocks before it were committed minutes apart. -func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingOnAFastChainWhileOutOfReach() { +// TestWaitForSyncOnAFastChainBehindTheTip checks where the at-tip rule draws +// the line on a chain committing two blocks a second, for a node behind a +// source that supplies blocks no faster than the chain commits them, whatever +// another peer claims. While its last block stays older than tipRecency the +// node keeps syncing. Once that block has been within tipRecency for tipGrace +// the node hands over, up to tipRecency of blocks behind: no peer serves it +// faster than the chain moves, so block sync would not close the gap either. +func (suite *SynchronizerTestSuite) TestWaitForSyncOnAFastChainBehindTheTip() { const ( height = int64(100) source = types.NodeID("source") @@ -2644,29 +2491,22 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingOnAFastChainWhile name string // behind is how old the node's last block is when block sync starts behind time.Duration - // quietBefore, when set, is how far apart the blocks up to the node's - // last one were committed - quietBefore time.Duration // supplyEvery is how often the source supplies the next block supplyEvery time.Duration // failEvery, when set, is how often the source, then the only peer, // fails a request - failEvery time.Duration + failEvery time.Duration + wantHandover bool }{ - {name: "applying at the chain's pace", behind: 100 * time.Second, supplyEvery: interval}, - {name: "a slow source after a restart", behind: 100 * time.Second, supplyEvery: 15 * time.Second}, + {name: "at the chain's pace, out of the window", behind: tipRecency + time.Minute, supplyEvery: interval}, + {name: "falling behind a slow source", behind: tipRecency - time.Minute, supplyEvery: 15 * time.Second}, { - name: "a busy chain after a quiet one", - behind: 4 * time.Minute, - quietBefore: 3 * time.Minute, - supplyEvery: interval, - }, - { - name: "a sole source that fails now and then", - behind: 3 * time.Minute, + name: "a sole source that fails now and then, out of the window", + behind: tipRecency + time.Minute, supplyEvery: interval, failEvery: 50 * time.Second, }, + {name: "at the chain's pace, within the window", behind: 100 * time.Second, supplyEvery: interval, wantHandover: true}, } for _, tc := range testCases { suite.Run(tc.name, func() { @@ -2676,29 +2516,18 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingOnAFastChainWhile } h := suite.newWaitForSyncHarness(height, peers...) defer h.cancel() - start := h.clock.Now() - last := start.Add(-tc.behind) + last := h.clock.Now().Add(-tc.behind) h.setLastBlockTime(last) - if tc.quietBefore > 0 { - h.sync.mtx.Lock() - h.sync.blockTimes = nil - for i := maxCatchupGap; i >= 0; i-- { - h.sync.blockTimes = append(h.sync.blockTimes, last.Add(-time.Duration(i)*tc.quietBefore)) - } - h.sync.mtx.Unlock() - // the chain turns busy a minute after the node's last block - last = last.Add(time.Minute - interval) - } supply := func() { next, _ := h.sync.GetStatus() h.suite.Require().NoError(h.sync.addBlock(BlockResponse{ PeerID: source, Block: &types.Block{Header: types.Header{Height: next}}, })) - h.sync.advance(source, last.Add(time.Duration(next-height+1)*interval), defaultPrecision) + h.sync.advance(last.Add(time.Duration(next-height+1)*interval), defaultPrecision) h.sync.peerStore.Update(source, ResetFailures()) } - elapsed, _, ended := h.run(maxSyncStall/2, func(elapsed time.Duration) { + elapsed, caughtUp, ended := h.run(maxSyncStall/2, func(elapsed time.Duration) { if tc.failEvery > 0 && elapsed > 0 && elapsed%tc.failEvery == 0 { next, _ := h.sync.GetStatus() h.fail(source, next, client.ErrPeerNotResponded) @@ -2713,8 +2542,15 @@ func (suite *SynchronizerTestSuite) TestWaitForSyncKeepsSyncingOnAFastChainWhile } }) applied, _ := h.sync.GetStatus() - suite.Require().False(ended, "handed over after %s at height %d, minutes of blocks behind on a fast chain", - elapsed, applied) + if !tc.wantHandover { + suite.Require().False(ended, "handed over after %s at height %d, more than %s of blocks behind", + elapsed, applied, tipRecency) + return + } + suite.Require().True(ended, "still in block sync after %s with a last block %s old", elapsed, tc.behind) + suite.Require().False(caughtUp) + suite.Require().Greater(elapsed, tipGrace) + suite.Require().LessOrEqual(elapsed, tipGrace+unservedHandoverSlack) }) } } @@ -3345,7 +3181,7 @@ func (suite *SynchronizerTestSuite) TestSmallBlocksAfterSlowApplyKeepPeersUsable st := h.pool.stallSnapshot() height := st.height suite.Equal(keepSyncing, - stallVerdictFor(st.servable, st.maxSuppliedHeight-height, st.stalledFor, 0), + stallVerdictFor(st.servable, st.maxServableHeight-height, st.stalledFor, 0), "healthy peers must not trigger consensus handoff at height %d below tip %d", height, st.maxPeerHeight) _, found := h.pool.peerStore.FindPeer(height) suite.True(found, "healthy peers must accept the next block request after the batch drains") diff --git a/internal/p2p/client/client.go b/internal/p2p/client/client.go index 3049e9800..6be04c25c 100644 --- a/internal/p2p/client/client.go +++ b/internal/p2p/client/client.go @@ -32,9 +32,6 @@ const ( ResponseIDAttribute = "ResponseID" ) -// RequestTimeout is how long a request waits for the peer's response. -const RequestTimeout = peerTimeout - const ( // peerTimeout leaves ample room for bounded stall detection and cleanup. peerTimeout = 60 * time.Second From 6e2726b905ea34f0f6ad71db49c26c8841eaaaf9 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 15:00:56 +0000 Subject: [PATCH 33/36] docs(blocksync): describe the handover rules and what operators can expect Co-Authored-By: Claude Fable 5.2 --- docs/tenderdash-core/block-sync/reactor.md | 106 ++++++++++----------- 1 file changed, 51 insertions(+), 55 deletions(-) diff --git a/docs/tenderdash-core/block-sync/reactor.md b/docs/tenderdash-core/block-sync/reactor.md index 32853752d..83a709dde 100644 --- a/docs/tenderdash-core/block-sync/reactor.md +++ b/docs/tenderdash-core/block-sync/reactor.md @@ -53,68 +53,64 @@ and hard failure policy are unchanged. The 60-second request deadline can still reject a large response on a slow connection even if its eventual service rate would meet the scheduling allowance. -Block sync hands over to consensus when it reaches the highest height any peer -claims; when no peer advertises the next height and no block has been applied -for 60 seconds; when the node has been at the tip of the chain for two minutes; -or when no block has been applied for 10 minutes and the node is within 10 -blocks of every claim that covers the next height and comes from a peer that -has backed it. When the node's last block was more than a day old as the stall -began, that last rule counts every claim that covers the next height, backed or -not, so a node far behind keeps syncing while its peers fail requests and -resumes as soon as one serves again. - -Whether the node is at the tip is judged from the time of its last applied -block, which the validators commit to and a peer cannot change: a node behind -the network applies old blocks. The node is at the tip while that block is no -older than the chain took to commit its last 10 blocks, or than 70 seconds -where that is longer, and in any case no older than five minutes. Seventy -seconds is the 60-second stall after which a node with nothing to fetch hands -over plus a status round, so a pause in block production ends block sync by -one rule or the other. The pace of the chain is taken from the blocks in the -block store at start and from those applied since; with no earlier block -stored, only the five-minute limit applies until a block is applied. Five -minutes covers the three-minute empty-block interval Dash Platform runs with -and a margin for extra rounds and clock skew. A block time further ahead of the -local clock than the synchrony precision does not count. Peer height claims -play no part in this rule. Whenever a node at the tip hands over, the -consensus catch-up target is the highest claim seen within 10 blocks of it. - -A peer backs its claim by supplying a block the node applies, and loses that -credit when it fails a request (a timeout or other failure, or a -`NoBlockResponse`) or is dropped; a newly connected peer has none. Block -requests go to peers that have backed their claims first. The next block is -asked of every other peer that advertises it once a request for it times out -or fails otherwise, once every peer asked for it is on probation for holding -requests too long, or once 10 seconds pass without it, counted from its latest -request or from when it became the next block, whichever is later. A peer -answering that it has no such block has it asked of one other peer instead. +Block sync hands over to consensus when one of these holds: + +- the node reaches the highest height any peer reports; +- no peer reports the next height and no block has been applied for 60 + seconds; +- the node's last applied block has been recent, at most five minutes old by + the local clock, for two minutes without a break; +- no block has been applied for 10 minutes, and either that wait began on a + block less than a day old, or the node is within 10 blocks of every peer + that reports the next height. + +The heights peers report are not verified, so the last two rules rest on the +time of the last applied block instead, which the validators commit to: a node +far behind the network applies old blocks. Five minutes covers the three-minute +empty-block interval Dash Platform runs with and a margin for extra rounds and +clock skew; the two minutes let a node that is still fetching blocks finish. A +block time further ahead of the local clock than the synchrony precision does +not count as recent. When a node with a recent last block hands over, the +consensus catch-up target is the highest height reported within 10 blocks of +it. Only a node whose last block was more than a day old when it stopped +receiving blocks keeps waiting for as long as a peer reports the next height +and a height more than 10 blocks above the node, and resumes as soon as a peer +serves again. + +Blocks are applied in order, so the next block is the one that matters. When +it has not arrived 10 seconds after the previous one was applied, it is +requested from every peer that reports it, once each, starting a second after +the peer first reports it. A block that a peer serves promptly is therefore +fetched once; a block that takes longer is fetched from each peer that reports +it, which on a node whose peers all need more than 10 seconds per block +multiplies the download by the number of those peers. What operators can expect: -- A node that has reached the tip hands over two minutes later, whatever - heights peers claim, as long as it applies each new block before the one - before it is older than the at-tip window: within about two minutes of its - commit on a chain committing a block every three minutes, and within about - a minute on a chain committing one every few seconds. A single slow peer - delays a block by up to about 10 seconds. +- A node that has caught up hands over within about two minutes, whatever + heights its peers report, provided a connected peer serves each new block + when asked. - On a chain that has stopped producing blocks, a restarted node hands over - after 10 minutes without a block, or once every peer that supplied blocks - earlier has failed a request for the next one, whichever is later. If the - chain had been stopped for more than a day when the node started waiting, - the node keeps waiting while a peer claims a height more than 10 blocks - above it and advertises the next block. -- A node may hand over a few blocks behind the network, up to about 70 - seconds of blocks on a fast chain, or up to five minutes of blocks after a - restart with no earlier block stored; consensus catch-up closes the rest. -- A node whose clock runs ahead of the validators' by more than its at-tip - window, or behind them by more than the synchrony precision, does not see - itself at the tip and leaves block sync only by the other rules. Keep the + after 10 minutes without a block. If the chain had been stopped for more + than a day when the node started waiting, the node keeps waiting while a peer + reports the next height and a height more than 10 blocks above the node. +- A node may hand over behind the network by up to five minutes of blocks: + one or two blocks on a chain committing every three minutes, a few hundred on + one committing every second. Consensus catch-up closes the rest. On a chain + that keeps committing a block every second or two this can take long, and it + happens only when no peer served blocks faster than the chain produced them. +- A node less than a day behind the network whose peers all fail to deliver + the next block for 10 minutes hands over where it is. Restart it once its + peers serve blocks again if consensus catch-up does not close the gap. +- A node whose clock runs minutes ahead of the validators', or + behind them by more than the synchrony precision, does not see its last + block as recent and leaves block sync only by the other rules. Keep the clock synchronised. -- After a handover on a stall away from the tip, the node may hold back its - own proposals for up to 10 minutes while consensus catches up. +- After a handover on a 10-minute wait, the node may hold back its own + proposals for up to 10 minutes while consensus catches up. - A node that stays in block sync while its blocks are recent, or hands over far behind, is worth reporting with its logs: a handover below the highest - peer claim is logged at warning or error level with its reason. + height a peer reports is logged at warning or error level with its reason. The historical Blocksync reactor is organised as a set of concurrent tasks: From 9961ff1531210d09fc12c9d452ff75a1bd9142f4 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 15:35:37 +0000 Subject: [PATCH 34/36] fix(blocksync): go on to the tip while closing the gap and ask every peer only after a stall The two minutes at the tip start over each time the last applied block has got tipGain (30 s) younger, so a node applying blocks at least a quarter faster than the chain commits them is not handed over on its way. The next block is asked of every peer once nothing has been applied for syncTimeout (60 s), the time a request may take, not after 10 seconds: on a link carrying a block in more than 10 seconds every block was fetched from every peer, and the requests queued behind them ran into their deadline. The fetch-volume tests now ask the peers reserveNextBlockPeers returns, and cover a slow downlink and peers faster than the chain. Co-Authored-By: Claude Fable 5.2 --- internal/blocksync/ask_every_peer_test.go | 6 +- .../blocksync/handover_fetch_volume_test.go | 160 +++++++++++++++++- internal/blocksync/reactor.go | 9 +- internal/blocksync/synchronizer.go | 25 ++- 4 files changed, 182 insertions(+), 18 deletions(-) diff --git a/internal/blocksync/ask_every_peer_test.go b/internal/blocksync/ask_every_peer_test.go index 31a08dc5f..76f984169 100644 --- a/internal/blocksync/ask_every_peer_test.go +++ b/internal/blocksync/ask_every_peer_test.go @@ -27,7 +27,7 @@ func reservedIDs(peers []PeerData) []types.NodeID { } // TestReserveNextBlockPeers checks who is asked for the height block sync waits -// to apply, and when: nobody until its block is serviceGrace overdue, then every +// to apply, and when: nobody until its block is syncTimeout overdue, then every // peer that advertises it, each once and a second after it is first seen // advertising it, until the block arrives or the height is applied. func TestReserveNextBlockPeers(t *testing.T) { @@ -49,7 +49,7 @@ func TestReserveNextBlockPeers(t *testing.T) { return reservedIDs(peers) } - clock.Advance(serviceGrace - 2*time.Second) + clock.Advance(syncTimeout - 2*time.Second) require.Empty(t, next(), "the block is not overdue yet") require.Empty(t, next(), "the block is not overdue yet") require.Empty(t, next(), "peers advertising the height are given a second first") @@ -84,7 +84,7 @@ func TestReserveNextBlockPeers(t *testing.T) { // the next height starts over sync.advance(clock.Now(), defaultPrecision) - clock.Advance(serviceGrace - time.Second) + clock.Advance(syncTimeout - time.Second) require.Empty(t, next(), "the next block is not overdue yet") require.Empty(t, next()) require.ElementsMatch(t, []types.NodeID{"second", "above"}, next()) diff --git a/internal/blocksync/handover_fetch_volume_test.go b/internal/blocksync/handover_fetch_volume_test.go index 96cb36bbe..bb79946f2 100644 --- a/internal/blocksync/handover_fetch_volume_test.go +++ b/internal/blocksync/handover_fetch_volume_test.go @@ -18,11 +18,18 @@ import ( // With downlink set, the node receives at most that many blocks a second over // all its connections, shared between the peers in turn, each peer sending at // most one a second, and a request not answered within the request deadline -// fails as it does in the block client. +// fails as it does in the block client. With every set, the downlink carries +// one block every that long instead. +// +// Once a second, before the link moves, the peers reserveNextBlockPeers returns +// are asked for the height to apply, as askEveryPeer does. type linkRun struct { *tipRun perBlock time.Duration downlink int + every time.Duration + // asked counts the requests made as askEveryPeer makes them. + asked int // busyUntil is when a peer finishes sending what it was asked for so far. busyUntil map[types.NodeID]time.Time scheduled map[*tipRequest]bool @@ -41,12 +48,23 @@ type linkRun struct { // never is when a response waiting for the downlink is due. var never = time.Unix(1<<40, 0) -func (l *linkRun) step(time.Duration) { - if l.downlink > 0 { +func (l *linkRun) step(elapsed time.Duration) { + now := l.h.clock.Now() + height, peers := l.h.sync.reserveNextBlockPeers() + for _, peer := range peers { + l.asked++ + l.request(peer, height, now) + } + if l.every > 0 { + l.downlink = 0 + if elapsed%l.every == 0 { + l.downlink = 1 + } + } + if l.downlink > 0 || l.every > 0 { l.sharedStep() return } - now := l.h.clock.Now() for { consumed := l.consume(now) produced := l.produce(now) @@ -225,3 +243,137 @@ func (suite *SynchronizerTestSuite) TestBlockSyncFarBehindOnASharedDownlink() { suite.Require().LessOrEqual(l.blocksSent, 2*l.applied, "%d blocks applied in %s, %d block responses downloaded from %d peers", l.applied, elapsed, l.blocksSent, peers) } + +// TestBlockSyncFarBehindOnASlowDownlink checks a node behind the network by +// hours over a downlink that carries one block in 12 or 20 seconds. +// Twelve peers hold every block and never fail a request on their own. The node +// must reach the tip and fetch each block about once: a block that takes its +// time within the request deadline is not a reason to ask every peer for it. +func (suite *SynchronizerTestSuite) TestBlockSyncFarBehindOnASlowDownlink() { + const ( + height = int64(100) + blocks = int64(200) + peers = 12 + ) + for _, tc := range []struct { + name string + every time.Duration + // reconnect has every peer report its range every status round, so a + // dropped peer comes back as a reconnecting one would. + reconnect bool + }{ + {name: "a block every 12s", every: 12 * time.Second}, + {name: "a block every 20s", every: 20 * time.Second}, + {name: "a block every 12s, dropped peers reconnect", every: 12 * time.Second, reconnect: true}, + } { + suite.Run(tc.name, func() { + suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) + h := suite.newWaitForSyncHarness(height) + defer h.cancel() + blockTime := h.clock.Now().Add(-staleBlockAge / 2) + h.setLastBlockTime(blockTime) + l := &linkRun{ + tipRun: &tipRun{ + h: h, + blockTimes: map[int64]time.Time{}, + suppliedBy: map[types.NodeID]int{}, + }, + every: tc.every, + queues: map[types.NodeID][]*tipRequest{}, + askedAt: map[*tipRequest]time.Time{}, + } + top := height + blocks - 1 + for i := 0; i < peers; i++ { + l.peers = append(l.peers, types.NodeID(fmt.Sprintf("peer %d", i))) + } + for applying := height; applying <= top; applying++ { + l.blockTimes[applying] = blockTime + } + report := func() { + for _, id := range l.peers { + h.sync.AddPeer(newPeerData(id, 1, top)) + } + } + report() + + elapsed, caughtUp, ended := h.run(6*time.Hour, func(elapsed time.Duration) { + if tc.reconnect && elapsed%(statusUpdateIntervalSeconds*time.Second) == 0 { + report() + } + l.step(elapsed) + }) + summary := fmt.Sprintf("caught up %v after %s: %d of %d blocks applied, %d responses downloaded, "+ + "%d asked of every peer, %d requests timed out, %d of %d peers left", + caughtUp, elapsed, l.applied, blocks, l.blocksSent, l.asked, l.timeouts, h.sync.peerStore.Len(), peers) + suite.T().Log(summary) + suite.Require().True(ended, summary) + suite.Require().True(caughtUp, "handed over behind: "+summary) + suite.Require().LessOrEqual(l.blocksSent, 2*l.applied, summary) + }) + } +} + +// TestWaitForSyncGoesOnToTheTipBehindPeersFasterThanTheChain checks the at-tip +// rule on a chain committing a block every second, with every peer serving what +// it reports. The node restarts ten minutes behind and its peers together +// deliver more blocks a second than the chain commits, so block sync closes the +// gap. The node must not hand over on its way: by the time it does, its last +// block is within tipGain of the chain's. +func (suite *SynchronizerTestSuite) TestWaitForSyncGoesOnToTheTipBehindPeersFasterThanTheChain() { + const ( + height = int64(1000) + behind = int64(600) + peers = 5 + ) + for _, downlink := range []int{2, 3, 4} { + suite.Run(fmt.Sprintf("peers deliver %d blocks a second", downlink), func() { + suite.client.On("Send", mock.Anything, mock.AnythingOfType("p2p.PeerError")).Maybe().Return(nil) + h := suite.newWaitForSyncHarness(height) + defer h.cancel() + start := h.clock.Now() + l := &linkRun{ + tipRun: &tipRun{ + h: h, + interval: time.Second, + blockTimes: map[int64]time.Time{}, + suppliedBy: map[types.NodeID]int{}, + tip: height - 1 + behind, + nextCommit: start.Add(time.Second), + }, + downlink: downlink, + queues: map[types.NodeID][]*tipRequest{}, + askedAt: map[*tipRequest]time.Time{}, + } + for i := 0; i < peers; i++ { + l.peers = append(l.peers, types.NodeID(fmt.Sprintf("peer %d", i))) + } + for committed := int64(1); committed <= l.tip; committed++ { + l.blockTimes[committed] = start.Add(-time.Duration(l.tip-committed) * time.Second) + } + h.setLastBlockTime(l.blockTimes[height-1]) + + elapsed, caughtUp, ended := h.run(time.Hour, func(elapsed time.Duration) { + now := h.clock.Now() + for !now.Before(l.nextCommit) { + l.tip++ + l.blockTimes[l.tip] = l.nextCommit + l.nextCommit = l.nextCommit.Add(l.interval) + } + if elapsed%(statusUpdateIntervalSeconds*time.Second) == 0 { + for _, id := range l.peers { + h.sync.AddPeer(newPeerData(id, 1, l.tip)) + } + } + l.step(elapsed) + }) + applying, _ := h.sync.GetStatus() + summary := fmt.Sprintf("caught up %v after %s: applying %d, chain tip %d, %d responses, %d timeouts", + caughtUp, elapsed, applying, l.tip, l.blocksSent, l.timeouts) + suite.T().Log(summary) + suite.Require().True(ended, summary) + suite.Require().LessOrEqual(l.tip-applying+1, int64(tipGain/time.Second), + "handed over %d blocks behind peers delivering %d blocks a second on a chain committing one: %s", + l.tip-applying+1, downlink, summary) + }) + } +} diff --git a/internal/blocksync/reactor.go b/internal/blocksync/reactor.go index bb9018432..3da6aa586 100644 --- a/internal/blocksync/reactor.go +++ b/internal/blocksync/reactor.go @@ -57,10 +57,15 @@ const ( tipRecency = 5 * time.Minute // tipGrace is how long the node must stay at the tip before higher height - // claims stop keeping it in block sync. A node still fetching blocks reaches - // the highest claim in that time when its peers serve them. + // claims stop keeping it in block sync. tipGrace = 2 * syncTimeout + // tipGain is how much younger the last applied block must get while the + // node counts as at the tip for tipGrace to start over: the node is still + // closing the gap, as it does when it applies blocks a quarter faster than + // the chain commits them. One block of a slow chain does that too, once. + tipGain = 30 * time.Second + // staleBlockAge is how old the last applied block must be, when a stall // begins, for peer height claims to count in the stall backstop. It is // longer than any halt a restarted node is expected to help end. diff --git a/internal/blocksync/synchronizer.go b/internal/blocksync/synchronizer.go index 3278fecba..d2d992336 100644 --- a/internal/blocksync/synchronizer.go +++ b/internal/blocksync/synchronizer.go @@ -455,10 +455,12 @@ func (s *Synchronizer) askEveryPeer(ctx context.Context) error { } // reserveNextBlockPeers returns the height block sync waits to apply and, once -// its block is serviceGrace overdue, every peer to ask for it: each peer that -// advertises it, once. Blocks are applied in order, so whichever peers hold the -// requests made for that height so far must not decide how long the others, -// which may serve it at once, go unasked. +// block sync has stalled on it for syncTimeout, every peer to ask for it: each +// peer that advertises it, once. Blocks are applied in order, so whichever peers +// hold the requests made for that height so far must not decide how long the +// others, which may serve it at once, go unasked. The wait is as long as a +// request may take, so a link that carries blocks slowly but within their +// request deadline does not have each of them fetched from every peer. // // A peer is asked on the second call that finds it advertising the height, so // the request the job producer sends when a peer first reports a new block gets @@ -466,7 +468,7 @@ func (s *Synchronizer) askEveryPeer(ctx context.Context) error { func (s *Synchronizer) reserveNextBlockPeers() (int64, []PeerData) { s.mtx.Lock() defer s.mtx.Unlock() - if _, received := s.pendingToApply[s.height]; received || s.clock.Since(s.lastAdvance) <= serviceGrace { + if _, received := s.pendingToApply[s.height]; received || s.clock.Since(s.lastAdvance) <= syncTimeout { return s.height, nil } var peers []PeerData @@ -526,12 +528,15 @@ func (s *Synchronizer) IsCaughtUp() bool { // applied block is: the validators commit to it, and a node far behind applies // old blocks. Once that time has been within tipRecency of the local clock for // tipGrace, the node hands over whatever peers report, even though blocks -// arriving meanwhile keep it from stalling. A stall of maxSyncStall that began +// arriving meanwhile keep it from stalling. The wait starts over each time the +// last block has got tipGain younger: the node is still closing the gap. A stall of maxSyncStall that began // on a block less than staleBlockAge old ends block sync the same way. func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHeight int64) { ticker := s.clock.NewTicker(switchToConsensusIntervalSeconds * time.Second) defer ticker.Stop() + // tipAge is the age of the last block at atTipSince. atTipSince := s.clock.Now() + tipAge := atTipSince.Sub(s.stallSnapshot().lastBlockTime) // nearTarget is the highest claim seen within maxCatchupGap of the node. var nearTarget int64 for { @@ -547,10 +552,12 @@ func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHe // At the tip, claims out of reach of it must not hold back our own // proposals, whichever way block sync ends. target := targetHeight - if isRecent(st.lastBlockTime, tipRecency, st.precision, now) { + recent := isRecent(st.lastBlockTime, tipRecency, st.precision, now) + if recent { target = nearTarget - } else { - atTipSince = now + } + if age := now.Sub(st.lastBlockTime); !recent || tipAge-age > tipGain { + atTipSince, tipAge = now, age } if st.hasPeers && st.height >= st.maxPeerHeight { return true, target From 554703c940c644d112a8b84f1014936bc04ff999 Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 15:35:37 +0000 Subject: [PATCH 35/36] docs(blocksync): state when the at-tip wait starts over and which handovers hold back proposals Co-Authored-By: Claude Fable 5.2 --- docs/tenderdash-core/block-sync/reactor.md | 34 +++++++++++++--------- 1 file changed, 20 insertions(+), 14 deletions(-) diff --git a/docs/tenderdash-core/block-sync/reactor.md b/docs/tenderdash-core/block-sync/reactor.md index 83a709dde..38465eed8 100644 --- a/docs/tenderdash-core/block-sync/reactor.md +++ b/docs/tenderdash-core/block-sync/reactor.md @@ -59,7 +59,8 @@ Block sync hands over to consensus when one of these holds: - no peer reports the next height and no block has been applied for 60 seconds; - the node's last applied block has been recent, at most five minutes old by - the local clock, for two minutes without a break; + the local clock, for two minutes without a break and without getting more + than 30 seconds younger meanwhile; - no block has been applied for 10 minutes, and either that wait began on a block less than a day old, or the node is within 10 blocks of every peer that reports the next height. @@ -68,8 +69,11 @@ The heights peers report are not verified, so the last two rules rest on the time of the last applied block instead, which the validators commit to: a node far behind the network applies old blocks. Five minutes covers the three-minute empty-block interval Dash Platform runs with and a margin for extra rounds and -clock skew; the two minutes let a node that is still fetching blocks finish. A -block time further ahead of the local clock than the synchrony precision does +clock skew. A last block that keeps getting younger means the node is still +closing the gap, so the two minutes start over each time it has gained 30 +seconds: a node applying blocks at least a quarter faster than the chain +commits them goes on until its last block is within 30 seconds of the chain's. +A block time further ahead of the local clock than the synchrony precision does not count as recent. When a node with a recent last block hands over, the consensus catch-up target is the highest height reported within 10 blocks of it. Only a node whose last block was more than a day old when it stopped @@ -78,18 +82,18 @@ and a height more than 10 blocks above the node, and resumes as soon as a peer serves again. Blocks are applied in order, so the next block is the one that matters. When -it has not arrived 10 seconds after the previous one was applied, it is -requested from every peer that reports it, once each, starting a second after -the peer first reports it. A block that a peer serves promptly is therefore -fetched once; a block that takes longer is fetched from each peer that reports -it, which on a node whose peers all need more than 10 seconds per block -multiplies the download by the number of those peers. +no block has been applied for 60 seconds and the next one has not arrived, it +is requested from every peer that reports it, once each, starting a second +after the peer first reports it. Sixty seconds is also how long a request may +take, so a block that arrives within its request deadline is fetched once, +however slow the link. What operators can expect: -- A node that has caught up hands over within about two minutes, whatever +- A node that has caught up hands over within two to four minutes, whatever heights its peers report, provided a connected peer serves each new block - when asked. + when asked. Peers that delay blocks can stretch this to about 20 minutes, + and a single block by about a minute. - On a chain that has stopped producing blocks, a restarted node hands over after 10 minutes without a block. If the chain had been stopped for more than a day when the node started waiting, the node keeps waiting while a peer @@ -98,7 +102,8 @@ What operators can expect: one or two blocks on a chain committing every three minutes, a few hundred on one committing every second. Consensus catch-up closes the rest. On a chain that keeps committing a block every second or two this can take long, and it - happens only when no peer served blocks faster than the chain produced them. + happens only when the node received blocks less than a quarter faster than + the chain produced them. - A node less than a day behind the network whose peers all fail to deliver the next block for 10 minutes hands over where it is. Restart it once its peers serve blocks again if consensus catch-up does not close the gap. @@ -106,8 +111,9 @@ What operators can expect: behind them by more than the synchrony precision, does not see its last block as recent and leaves block sync only by the other rules. Keep the clock synchronised. -- After a handover on a 10-minute wait, the node may hold back its own - proposals for up to 10 minutes while consensus catches up. +- After a handover below a height a peer reported, the node may hold back its + own proposals for up to 10 minutes while consensus catches up. With a recent + last block only heights reported within 10 blocks of the node count. - A node that stays in block sync while its blocks are recent, or hands over far behind, is worth reporting with its logs: a handover below the highest height a peer reports is logged at warning or error level with its reason. From eaed8f075491237525be68eecf8df181fbd53b4c Mon Sep 17 00:00:00 2001 From: Lukasz Klimek <842586+lklimek@users.noreply.github.com> Date: Wed, 11 Nov 2026 15:46:54 +0000 Subject: [PATCH 36/36] docs(blocksync): note the slower catch-up with few serving peers and wrap a comment Co-Authored-By: Claude Fable 5.2 --- docs/tenderdash-core/block-sync/reactor.md | 3 +++ internal/blocksync/synchronizer.go | 5 +++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/tenderdash-core/block-sync/reactor.md b/docs/tenderdash-core/block-sync/reactor.md index 38465eed8..384533064 100644 --- a/docs/tenderdash-core/block-sync/reactor.md +++ b/docs/tenderdash-core/block-sync/reactor.md @@ -107,6 +107,9 @@ What operators can expect: - A node less than a day behind the network whose peers all fail to deliver the next block for 10 minutes hands over where it is. Restart it once its peers serve blocks again if consensus catch-up does not close the gap. +- With only a few peers serving blocks, a node restarting far behind a chain + that commits a block every second or two may need much longer to reach the + tip. More peers that serve blocks help. - A node whose clock runs minutes ahead of the validators', or behind them by more than the synchrony precision, does not see its last block as recent and leaves block sync only by the other rules. Keep the diff --git a/internal/blocksync/synchronizer.go b/internal/blocksync/synchronizer.go index d2d992336..32886ac27 100644 --- a/internal/blocksync/synchronizer.go +++ b/internal/blocksync/synchronizer.go @@ -529,8 +529,9 @@ func (s *Synchronizer) IsCaughtUp() bool { // old blocks. Once that time has been within tipRecency of the local clock for // tipGrace, the node hands over whatever peers report, even though blocks // arriving meanwhile keep it from stalling. The wait starts over each time the -// last block has got tipGain younger: the node is still closing the gap. A stall of maxSyncStall that began -// on a block less than staleBlockAge old ends block sync the same way. +// last block has got tipGain younger: the node is still closing the gap. A +// stall of maxSyncStall that began on a block less than staleBlockAge old ends +// block sync the same way. func (s *Synchronizer) WaitForSync(ctx context.Context) (caughtUp bool, targetHeight int64) { ticker := s.clock.NewTicker(switchToConsensusIntervalSeconds * time.Second) defer ticker.Stop()