diff --git a/abci/types/types.pb.go b/abci/types/types.pb.go index 73467302f..3f535c820 100644 --- a/abci/types/types.pb.go +++ b/abci/types/types.pb.go @@ -3193,9 +3193,10 @@ type ResponseFinalizeBlock struct { // Blocks below this height may be removed. Defaults to `0` (retain all). RetainHeight int64 `protobuf:"varint,2,opt,name=retain_height,json=retainHeight,proto3" json:"retain_height,omitempty"` // When `true`, this node does not wait for transactions (`create-empty-blocks-interval`) - // before proposing round 0 of the next height: the application has block-driven work - // pending, for example withdrawal transactions to sign. A hint for the local node only, - // consumed once; a node that restarts in between falls back to the configured wait. + // before proposing at the next height, in round 0 or, on consensus version 2, in a round + // entered after a silent proposer's round was skipped: the application has block-driven + // work pending, for example withdrawal transactions to sign. A hint for the local node + // only, consumed once; a node that restarts in between falls back to the configured wait. // Defaults to `false`. ProposeNextBlockImmediately bool `protobuf:"varint,3,opt,name=propose_next_block_immediately,json=proposeNextBlockImmediately,proto3" json:"propose_next_block_immediately,omitempty"` } diff --git a/book/SUMMARY.md b/book/SUMMARY.md index ee85dc841..5af563c2e 100644 --- a/book/SUMMARY.md +++ b/book/SUMMARY.md @@ -77,6 +77,7 @@ - [Primary flow of a height](spec/consensus/consensus-happy-path.md) - [Transitions and decision rules](spec/consensus/consensus-transitions.md) - [Alternative flows and edge cases](spec/consensus/consensus-alternative-flows.md) + - [Proposer liveness and round skipping](spec/consensus/consensus-proposer-liveness.md) - [Commit handling](spec/consensus/consensus-commit.md) - [Recovery and synchronisation](spec/consensus/consensus-recovery.md) - [Known issues and discrepancies](spec/consensus/consensus-known-issues.md) diff --git a/docs/nodes/configuration.md b/docs/nodes/configuration.md index 8c5090939..c2a95ccad 100644 --- a/docs/nodes/configuration.md +++ b/docs/nodes/configuration.md @@ -487,7 +487,12 @@ namespace = "tendermint" `create-empty-blocks = false` makes round 0 wait for transactions above the initial height. A positive `create-empty-blocks-interval` starts a proposal after that interval even without transactions. Later rounds do not wait for -transactions. +transactions, except on a chain with consensus version 2, where a round entered +because the validators skipped a silent proposer's waiting round may wait too, +unless the node holds a locked or valid block, the mempool has already reported +transactions at this height, a polka exists at the height, or a prevote of that +round was recorded; its interval counts from the start of the height. See +[Proposer liveness and round skipping](../../spec/consensus/consensus-proposer-liveness.md). The application can bypass that wait for the next height by setting `ResponseFinalizeBlock.propose_next_block_immediately`. The hint is local, diff --git a/docs/nodes/metrics.md b/docs/nodes/metrics.md index 564ed762a..b439137b1 100644 --- a/docs/nodes/metrics.md +++ b/docs/nodes/metrics.md @@ -62,6 +62,10 @@ The following metrics are available: | consensus_verification_budget_saturation | Gauge | | Fraction of the node-wide verification budget still available at a peer-message check (1.0 full/idle, 0.0 drained/under pressure) | | consensus_peer_lane_active_count | Gauge | | Number of per-peer lanes currently holding queued messages and taking turns in the scheduler | | consensus_peer_lane_max_depth | Gauge | | Largest number of messages queued in any single peer lane | +| consensus_proposer_heartbeats_sent | Counter | | Proposer heartbeats this node signed and broadcast while waiting for transactions as proposer (consensus version 2) | +| consensus_proposer_heartbeats_received | Counter | | New, verified proposer heartbeats accepted for the round this node waits in | +| consensus_proposer_silences | Counter | | Rounds in which the proposer went silent while this validator waited for transactions | +| consensus_round_skips | Counter | | Rounds this node moved past on a round skip certificate from +2/3 of the validators (by default; configuration can change the threshold) | | evidence_pool_num_evidence | Gauge | | Number of evidence in the evidence pool | | evidence_pool_dropped_evidence | Counter | reason | Inbound gossip evidence refused before verification, labeled by reason (structural, oversized, already_known, height_window, node_budget, peer_budget) | | p2p_peers | Gauge | | Number of peers node's connected to | diff --git a/docs/nodes/remote-signer.md b/docs/nodes/remote-signer.md index 10758e479..ad3a11818 100644 --- a/docs/nodes/remote-signer.md +++ b/docs/nodes/remote-signer.md @@ -8,6 +8,8 @@ Tenderdash provides a remote signer option for validators. A remote signer enabl The remote signer protocol implements a [client and server architecture](https://en.wikipedia.org/wiki/Client%E2%80%93server_model). When Tenderdash requires the public key or signature for a proposal or vote it requests it from the remote signer. +The remote signer protocol does not cover proposer heartbeats or round skip votes. On a chain with consensus version 2, a validator using a remote signer sends neither, so validators skip its rounds that wait for transactions once its heartbeats are missed. See [Proposer liveness and round skipping](../../spec/consensus/consensus-proposer-liveness.md#signers). + To run a secure validator and remote signer system it is recommended to use a VPC (virtual private cloud) or a private connection. There are two different configurations that can be used: Raw or gRPC. diff --git a/docs/tenderdash-core/using-tenderdash.md b/docs/tenderdash-core/using-tenderdash.md index 23c38966a..de2b167b4 100644 --- a/docs/tenderdash-core/using-tenderdash.md +++ b/docs/tenderdash-core/using-tenderdash.md @@ -209,7 +209,8 @@ Some fields from the config file can be overwritten with flags. ## No Empty Blocks Configure `create-empty-blocks` and `create-empty-blocks-interval` in -`config.toml` to control waiting for transactions at round 0. The application +`config.toml` to control waiting for transactions at round 0 (and, on consensus +version 2, in a round entered after a silent proposer's round was skipped). The application can request an immediate next block through `ResponseFinalizeBlock`. An app hash change does not itself require an extra block: Tenderdash uses same-block execution. diff --git a/proto/tendermint/abci/types.proto b/proto/tendermint/abci/types.proto index 4099f1966..27931d24f 100644 --- a/proto/tendermint/abci/types.proto +++ b/proto/tendermint/abci/types.proto @@ -600,9 +600,10 @@ message ResponseFinalizeBlock { // Blocks below this height may be removed. Defaults to `0` (retain all). int64 retain_height = 2; // When `true`, this node does not wait for transactions (`create-empty-blocks-interval`) - // before proposing round 0 of the next height: the application has block-driven work - // pending, for example withdrawal transactions to sign. A hint for the local node only, - // consumed once; a node that restarts in between falls back to the configured wait. + // before proposing at the next height, in round 0 or, on consensus version 2, in a round + // entered after a silent proposer's round was skipped: the application has block-driven + // work pending, for example withdrawal transactions to sign. A hint for the local node + // only, consumed once; a node that restarts in between falls back to the configured wait. // Defaults to `false`. bool propose_next_block_immediately = 3; } diff --git a/spec/abci/api.md b/spec/abci/api.md index 13eb3f0c6..343197cb3 100644 --- a/spec/abci/api.md +++ b/spec/abci/api.md @@ -793,7 +793,7 @@ nondeterministic | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | retain_height | [int64](#int64) | | Blocks below this height may be removed. Defaults to `0` (retain all). | -| propose_next_block_immediately | [bool](#bool) | | When `true`, this node does not wait for transactions (`create-empty-blocks-interval`) before proposing round 0 of the next height: the application has block-driven work pending, for example withdrawal transactions to sign. A hint for the local node only, consumed once; a node that restarts in between falls back to the configured wait. Defaults to `false`. | +| propose_next_block_immediately | [bool](#bool) | | When `true`, this node does not wait for transactions (`create-empty-blocks-interval`) before proposing at the next height, in round 0 or, on consensus version 2, in a round entered after a silent proposer's round was skipped: the application has block-driven work pending, for example withdrawal transactions to sign. A hint for the local node only, consumed once; a node that restarts in between falls back to the configured wait. Defaults to `false`. | diff --git a/spec/abci/basic-concepts.md b/spec/abci/basic-concepts.md index f0819d71e..350239cf0 100644 --- a/spec/abci/basic-concepts.md +++ b/spec/abci/basic-concepts.md @@ -112,7 +112,7 @@ In summary: ### Other methods Additionally, there is a [**Flush**](methods.md#flush) method that is called on every connection, -and an [**Echo**](methods.md#echo) method that is just for debugging. +and an [**Echo**](methods.md#echo) method for debugging, which a waiting proposer also uses as a readiness probe. More details on managing state across connections can be found in the section on [ABCI Applications](app-requirements.md). diff --git a/spec/abci/expected-behavior.md b/spec/abci/expected-behavior.md index f47377f01..edb1b2f17 100644 --- a/spec/abci/expected-behavior.md +++ b/spec/abci/expected-behavior.md @@ -62,7 +62,9 @@ ordinary voting rounds. `CheckTx` controls mempool admission separately from the block execution sequence. `Query`, `ListSnapshots` and `LoadSnapshotChunk` read application state or existing -snapshots. `Echo` and `Flush` serve client/debugging purposes. `Info` can also serve +snapshots. `Echo` and `Flush` serve client/debugging purposes; from consensus version 2 a +waiting proposer also uses `Echo` as a readiness probe +([Echo](./methods.md#echo)). `Info` can also serve RPC queries independently of startup. These methods are not bounded by the normal height's consensus callback count. diff --git a/spec/abci/methods.md b/spec/abci/methods.md index c79983316..880c016ac 100644 --- a/spec/abci/methods.md +++ b/spec/abci/methods.md @@ -15,6 +15,12 @@ title: Methods * `Message (string)`: The input string * **Usage**: * Echo a string to test an abci client/server implementation +* From consensus version 2, the proposer of a round that waits for transactions probes + the backends that receive `PrepareProposal` with `Echo` before each heartbeat, once it + is not held back after block sync. At most one probe is in flight; an unanswered probe + is reused by later heartbeats rather than repeated. A heartbeat is sent only if the + backends answer within the heartbeat interval. See + [Proposer liveness](../consensus/consensus-proposer-liveness.md#heartbeats). ### Flush @@ -577,7 +583,8 @@ contract above and [Commit handling](../consensus/consensus-commit.md). * Set `ResponseFinalizeBlock.propose_next_block_immediately` when the Application has block-driven work pending that the next block would carry out, for example withdrawal transactions waiting to be signed. The node then does not wait for transactions (`create-empty-blocks-interval`) before - proposing round 0 of the next height. This is a hint to the local node, not part of consensus: it is + proposing at the next height, in round 0 or, on consensus version 2, in a round entered after a + silent proposer's round was skipped. This is a hint to the local node, not part of consensus: it is not compared between validators, it only matters on the node that proposes the next height, and it is consumed once; a node that restarts in between falls back to the configured wait. * Just as `ProcessProposal`, the implementation of `FinalizeBlock` MUST be deterministic, since it is diff --git a/spec/consensus/consensus-alternative-flows.md b/spec/consensus/consensus-alternative-flows.md index 4e1dc8aae..53408e7a7 100644 --- a/spec/consensus/consensus-alternative-flows.md +++ b/spec/consensus/consensus-alternative-flows.md @@ -41,6 +41,11 @@ sequenceDiagram V->>V: EnterPrecommit (no-op) → EnterNewRound(h,r+1) → EnterPropose(h,r+1) ``` +A proposer that is silent while its round waits for transactions is handled +before the propose step on `CONSENSUS_VERSION_2` chains: validators that stop +hearing its heartbeats skip the round with a round skip certificate +([Proposer liveness and round skipping](./consensus-proposer-liveness.md)). + ### Proposal arrives after the node prevoted `Proposaler.Set` has no step check, so the proposal is still stored and parts @@ -195,6 +200,11 @@ on another block prevotes nil for it; see If the block is absent, signature verification can park the commit and advance to its round before the later application check. See [Commit handling](./consensus-commit.md#commit-for-a-later-round). +- **Round skip certificate** (`CONSENSUS_VERSION_2`). A verified certificate + for round `r' ≥ r` → `EnterNewRound(r' + 1)`, unless the node is at + `ApplyCommit`; one for `r' < r` is only kept and gossiped + (`AddRoundSkipCertificateAction`). See + [Proposer liveness and round skipping](./consensus-proposer-liveness.md). - **Round bookkeeping.** `HeightVoteSet` keeps vote sets for rounds `0 .. Round + 1` (`SetRound(round + 1)` in `EnterNewRoundAction`) plus at most two "catch-up" rounds per validator pro-tx-hash @@ -290,17 +300,25 @@ and `addVoteErrorMw` (`internal/consensus/state_add_vote.go`). `EnterNewRoundAction.Execute` waits for transactions when `WaitForTxs() = ¬CreateEmptyBlocks ∨ CreateEmptyBlocksInterval > 0`, at -round 0, above `InitialHeight`, unless `ProposeNextBlockImmediately` +round 0 or in a round entered on a round skip certificate that satisfies +`StateData.waitsAfterRoundSkip` +([The round after a skip](./consensus-proposer-liveness.md#the-round-after-a-skip)), +above `InitialHeight`, unless `ProposeNextBlockImmediately` (set from `ResponseFinalizeBlock.propose_next_block_immediately` of the previous height by `ApplyCommitAction.Execute`; consumed by the next `updateToState`; not persisted, so a restart falls back to the configured wait). -`State.handleTxsAvailable` acts only at round 0: at step `NewHeight` (and -not at `InitialHeight`) it schedules a `NewRound` tock for +`State.handleTxsAvailable` records the report in `TxsReported` (the mempool +reports once per height) and acts only at round 0 or in a round that may wait +after a round skip (`StateData.mayWaitAfterRoundSkip`): at step `NewHeight` +(and not at `InitialHeight`) it schedules a `NewRound` tock for `StartTime − now + 1 ms`; at step `NewRound` it dispatches `EnterPropose`. With `CreateEmptyBlocksInterval > 0` the interval tock calls `EnterPropose` -even without transactions (empty block). Rounds `≥ 1` never wait. +even without transactions (empty block); in a round after a skip the interval +counts from `StartTime`. Other rounds `≥ 1` never wait. On +`CONSENSUS_VERSION_2` chains a proposal or prevotes can also end the wait +([When it applies](./consensus-proposer-liveness.md#when-it-applies)). ## Validator-set and parameter changes @@ -319,7 +337,8 @@ even without transactions (empty block). Rounds `≥ 1` never wait. version) proposed in block `h` take effect at `h + 1` (`LastHeightConsensusParamsChanged`). - Proposer selection version is `ConsensusParams.Version.ConsensusVersion` - ([Proposer selection](./proposer-selection.md)). + ([Proposer selection](./proposer-selection.md)); version 2 also enables + [proposer liveness](./consensus-proposer-liveness.md). ## Core chain locks @@ -356,7 +375,15 @@ even without transactions (empty block). Rounds `≥ 1` never wait. `(h, r)` names the same block (`Proposaler.verifyProposalForNonValidatorSet`). - **Signing failure** (`PrivValidator.SignVote`/`SignProposal` error or deadline): logged; no vote/proposal and **no retry** within the step - (`voteSigner.signAddVote`, `Proposaler.Create`). + (`voteSigner.signAddVote`, `Proposaler.Create`). A failed round skip vote is + retried after another silence timeout. +- **Remote signers** (socket, gRPC) cannot sign proposer heartbeats or round + skip votes (`RemoteSignerError` 501, `types.ErrSigningNotSupported`): a + validator using one sends neither, so its waiting rounds are skipped + ([Signers](./consensus-proposer-liveness.md#signers)). +- **Observers and liveness messages:** a node without the validators' key + shares neither verifies nor relays heartbeats and skip votes; it verifies + and applies round skip certificates against the quorum public key. - **Block sync just finished**: proposing is suppressed until the node has passed the handover target height or 10 minutes elapsed (`catchupTracker.mayPropose`, `maxCatchupSuppression`); voting is not. @@ -408,6 +435,11 @@ faults, unless stated). For operational monitoring, see evicts a peer (`isPeerFloodableError`). - **Block-part proof budget** (`block_part_budget.go`): per-peer allowance for parts whose Merkle proof fails. +- **Liveness messages** (`CONSENSUS_VERSION_2`): heartbeats, round skip votes + and certificates pay extra state-channel tokens only when new, go through + the peer lanes and the verification budget, and are sent through a bounded + reactor outbox; heartbeats are paced per round + ([Message admission and budgets](./consensus-proposer-liveness.md#message-admission-and-budgets)). - **Bounded memory per height:** commit candidate slots (one per peer), `commitExpectations`, `MaxVoteExtensions`, `HeightVoteSet` catch-up rounds, `maxPeerMaj23s` claims per vote set. diff --git a/spec/consensus/consensus-happy-path.md b/spec/consensus/consensus-happy-path.md index 5cae36939..6290ef32d 100644 --- a/spec/consensus/consensus-happy-path.md +++ b/spec/consensus/consensus-happy-path.md @@ -97,11 +97,16 @@ tock to `EnterNewRoundEvent{Height}`. `EnterNewRoundAction.Execute` - `Votes.SetRound(1)` so that round 1 votes are tracked for round skipping; - `TriggeredTimeoutPrecommit = false`; - decides whether to wait for transactions: - `waitForTxs = config.WaitForTxs() && round == 0 && Height != InitialHeight`, - overridden to `false` by `ProposeNextBlockImmediately`. **⇢** + `waitForTxs = config.WaitForTxs() && (round == 0 || waitsAfterRoundSkip(round)) && Height != InitialHeight`, + overridden to `false` by `ProposeNextBlockImmediately` and, on + `CONSENSUS_VERSION_2` chains, by a proposal already held for the round + (`roundSkipping && Proposal != nil`). **⇢** [Empty blocks](./consensus-alternative-flows.md#mempool-and-empty-blocks). With the default configuration (`create-empty-blocks = true`, interval 0) it does not wait and dispatches `EnterProposeEvent{h, 0}`. + On `CONSENSUS_VERSION_2` chains the proposer of a waiting round sends + heartbeats while it waits. **⇢** + [Proposer liveness](./consensus-proposer-liveness.md). ### 3. Propose diff --git a/spec/consensus/consensus-known-issues.md b/spec/consensus/consensus-known-issues.md index 1cec3f764..8ae0833e6 100644 --- a/spec/consensus/consensus-known-issues.md +++ b/spec/consensus/consensus-known-issues.md @@ -58,8 +58,10 @@ the last tick after it fires. When `EnterPropose` itself was triggered by a else moves the node out of `NewRound` for that round. The node recovers only when the other validators' nil precommits reach +2/3-any (`addVoteDispatchPrecommitMw` → `EnterPrecommitWait` → `EnterNewRound(r + 1)`), -costing the network one round. Preconditions: `WaitForTxs()` true, round 0, -the node is the proposer, and its clock is behind the previous block time by +costing the network one round. Preconditions: `WaitForTxs()` true, round 0 +(on `CONSENSUS_VERSION_2` chains also a round entered on a round skip +certificate that waits, whose empty-block interval schedules the same +`NewRound` tock), the node is the proposer, and its clock is behind the previous block time by more than the time since that block was produced. (Inferred; not reproduced.) ## Precommit requires this round's proposal even when locked diff --git a/spec/consensus/consensus-proposer-liveness.md b/spec/consensus/consensus-proposer-liveness.md new file mode 100644 index 000000000..8f2f54783 --- /dev/null +++ b/spec/consensus/consensus-proposer-liveness.md @@ -0,0 +1,323 @@ +--- +order: 10 +--- + +# Proposer liveness and round skipping + +A round that waits for transactions has no timeout running unless +`create-empty-blocks-interval` is set, so a proposer that is offline while the +chain is idle is only noticed once a transaction arrives, and that transaction +then waits out the propose timeout. From `CONSENSUS_VERSION_2` the proposer of +a waiting round announces itself with signed heartbeats, and validators that +stop hearing them hand the round to the next proposer with a threshold-signed +round skip certificate. + +Wire formats are in [Consensus channel messages](../p2p/messages/consensus.md); +sign bytes and request IDs in +[Signing](./signing.md#proposer-heartbeats-and-round-skip-votes). + +## When it applies + +`roundSkipping` (`internal/consensus/proposer_liveness.go`) gates everything on +this page. It holds when both: + +- `ConsensusParams.Version.ConsensusVersion ≥ 2` + (`VersionParams.ProposerLivenessEnabled`, `types/params.go`), and +- `types.RoundSkipsProvable(Validators)`: `T ≤ R`, i.e. a + `VotingPowerThreshold` above the recovery threshold disables round skipping, + because a certificate only proves that shares worth `R` signed + (`types/round_skip.go`). + +Otherwise no heartbeat is sent or accepted, no skip vote is signed or +collected, every certificate is dropped, and the node behaves as on version 0 +or 1. + +A node **waits for transactions** in its round (`waitingForTxs`) while +`roundSkipping ∧ step == NewRound ∧ Commit == nil`; liveness is tracked while +it also holds no proposal for the round (`waitingRound`). Whether a round waits +at all is decided by `EnterNewRoundAction.Execute` +([Mempool and empty blocks](./consensus-alternative-flows.md#mempool-and-empty-blocks)). +On a version-2 chain a wait also ends, with `Dispatch(EnterPropose{h, r})`, +when: + +- a proposal for the round is accepted (`proposalMessageHandler`, + `internal/consensus/msg_handlers.go`); a round-0 proposal accepted before the + round starts keeps the round from waiting at all; +- a prevote is added while the node waits and the added prevote's round (this + round or an earlier one) has a polka, or the prevote is for the current round + and `r > 0` (`addVoteDispatchPrevoteMw`). + +A proposal or polka shows that a block is under way; from then on the +propose timeout, not the heartbeats, moves the round on. + +## Heartbeats + +`State.followLiveness` (`internal/consensus/state.go`) runs once before the +receive routine's first input and after every input. It calls +`proposerLiveness.reconcile`, which, while `waitingRound` holds, either +**announces** (this node is `proposer(h, r)`) or **watches** (any other node); +otherwise it stops both. + +Sending (`proposerLiveness.sendHeartbeats`, `sendHeartbeat`): + +- one heartbeat at once and then every `ProposerHeartbeat`, until the node + leaves the round; +- only while the node could propose (`State.proposerReady`): the propose step + would not suppress it after a block-sync handover (`catchupTracker.mayPropose`) + and the application answers ABCI `Echo` on every backend that receives + `PrepareProposal` (`BlockExecutor.AppReady` in `internal/state/execution.go`, + `abciclient.EchoRoute`) within the interval. At most one probe is in flight + (`appReadiness`, `internal/consensus/app_readiness.go`); +- `Sequence = max(last + 1, signing time in Unix ms)` (`heartbeatSequence`); +- one heartbeat is signed at a time, whatever its round; a heartbeat whose + round the node left while signing is dropped. A signer error wrapping + `types.ErrSigningNotSupported` stops heartbeats for the rest of the process + (`signerUnsupported`). + +Receiving: the reactor screens a heartbeat without a signature check +(`proposerLiveness.wanted` → `screen`) and queues it in the sender's lane; the +consensus goroutine verifies it (`proposerLiveness.heard`, called from +`State.handleLivenessMessage`). A heartbeat is accepted iff: + +1. its height is the node's and its round is within the relay window + `[max(r − 4, 0), r + 4]` (`relayWindow`, `maxRoundSkipRoundsAhead`); +2. it names `proposer(h, round)`, otherwise `errProposerHeartbeatWrongProposer`; +3. it was not seen before, identified by `(height, round, sequence, timestamp)` + (`heartbeatID`; at most `maxSeenHeartbeats = 256` remembered); +4. its timestamp is within the silence timeout of local time, in either + direction; +5. the node holds the proposer's public key share (validators only); +6. the round's pace has a token: one per half heartbeat interval, burst 2 + (`heartbeatPaceBurst`); +7. the verification budget allows one check and `ProposerHeartbeat.Verify` + succeeds (`types/proposer_heartbeat.go`). + +An accepted heartbeat for the round the node watches re-arms its silence timer; +one for a nearby round is only relayed. Every accepted heartbeat is relayed +once. Heartbeats go only to validator peers at the same height whose round is +within the relay window of the heartbeat's round, never back to the sender +(`Reactor.sendHeartbeat`, `wantsHeartbeat` in `internal/consensus/reactor.go`). +A heartbeat changes no consensus state and is not written to the WAL. + +## Silence and skip votes + +The watch arms a timer of `ProposerSilenceTimeout(r) = ProposerSilence + +ProposerHeartbeat · r`, saturating at the maximum duration +(`TimeoutParams.ProposerSilenceTimeout`, `types/params.go`). When it fires, +`State.handleProposerSilence` acts only if the report is still current (same +height and round, still waiting, no heartbeat re-armed the timer since) and +the node is a validator. `ProposerSilences` counts the first report of a round. + +`State.voteToSkipRound` then: + +- if this validator's vote for the round is already counted, broadcasts it + again; +- otherwise has `RoundSkipVote{h, r, own pro-tx-hash, own index}` signed off + the consensus goroutine (`roundSkipSigner`, + `internal/consensus/round_skip_signer.go`), one vote at a time, with the + silence timeout as the signing deadline. If a vote is still being signed, + this one is retried after another silence timeout. + +`State.handleSignedRoundSkipVote` drops the result if the node left the round +or a heartbeat arrived meanwhile. `ErrSigningNotSupported` ends the attempt; +any other signing error retries after another silence timeout. A signed vote +is counted, broadcast, and broadcast again every silence timeout while the +round stays silent. + +A skip vote is **not** a prevote: it locks nothing, and the validator still +prevotes and precommits the round's proposal if one arrives. Every validator +signs the same bytes for `(h, r)`, so a validator can make only one skip vote +per round. + +## Collecting votes and recovering the certificate + +`roundSkipCollector` (`internal/consensus/round_skip_collector.go`) holds the +skip votes of the current height and quorum for rounds `[r, r + 4]`. Votes of +rounds below `r` are dropped when the node moves on. A vote from a peer is +counted iff its index and pro-tx-hash name the same validator (otherwise +`errRoundSkipVoteWrongValidator`), it is not already counted, the round is not +already certified, the node holds the validator's key share, the verification +budget allows one check, and `RoundSkipVote.Verify` succeeds. + +A vote is relayed when first counted. A copy of a counted vote that arrives at +least half a silence timeout after the last relay has the counted vote relayed +again, until the round is certified (`roundSkipCollector.relayAgain`). Skip +votes are not written to the WAL. + +Once the counted power reaches +`RoundSkipThresholdPower = max(T, R)` (`types/round_skip.go`; equal to `R` +whenever `roundSkipping` holds; +2/3 on mainnet, whose `LLMQ_100_67` quorums +recover from 67 of 100 shares and set no `VotingPowerThreshold`), +`RecoverRoundSkipCertificate` recovers the +quorum threshold signature over the round's sign bytes from the shares and +verifies it against `ValidatorSet.ThresholdPublicKey`. A failed recovery is +retried at once with any votes counted meanwhile, otherwise with the next vote. +The recovered certificate is dispatched immediately on the consensus goroutine +(`State.applyRecoveredCertificate`). + +## Applying a certificate + +`AddRoundSkipCertificateAction.Execute` +(`internal/consensus/state_add_round_skip.go`): + +- **Guard** (`roundSkipCertificateWanted`, also checked by the reactor before + queuing): `roundSkipping ∧ cert.Height == h ∧ (RoundSkip == nil ∨ cert.Round > RoundSkip.Round) ∧ step < ApplyCommit`. +- **Actions:** + 1. Unless this node recovered it: charge the verification budget and + `RoundSkipCertificate.Verify` (quorum hash must equal the validator + set's, else `ErrRoundSkipCertificateWrongQuorum`; threshold signature + against `ThresholdPublicKey`). + 2. Write the message it came in to the WAL (`Write`, no fsync); not on + replay. + 3. `RoundSkip = cert`; broadcast it. + 4. If `cert.Round < r`: stop (the certificate is kept for peers still in + that round). Else count `RoundSkips` and + `Dispatch(EnterNewRound{h, cert.Round + 1})`. + +The certificate does not touch locks, valid blocks or vote sets; it only moves +the node to a higher round, which the locking rules make safe. + +```mermaid +sequenceDiagram + participant P as Proposer of round r + participant V as Validators + participant N as Proposer of round r+1 + P-->>V: ProposerHeartbeat every proposer_heartbeat + Note over P: goes offline + V->>V: no heartbeat for ProposerSilenceTimeout(r) + V-->>V: RoundSkipVote(h, r) + V->>V: +2/3 skip votes → RoundSkipCertificate(h, r) + V->>V: EnterNewRound(h, r+1): waits for txs + N-->>V: ProposerHeartbeat for round r+1 +``` + +## The round after a skip + +`EnterNewRoundAction.Execute` lets round `R > 0` wait for transactions, as +round 0 does, when `StateData.waitsAfterRoundSkip(R)` holds +(`internal/consensus/state_data.go`): + +- `RoundSkip.Round == R − 1`; +- `LockedRound == -1 ∧ ValidRound == -1`; +- the mempool has not reported transactions at this height (`TxsReported`; the + mempool reports once per height, and `State.handleTxsAvailable` records the + report in any round); +- no polka at the height (`HeightVoteSet.POLInfo` round `-1`) and no prevote + recorded for round `R`. + +The other wait conditions are unchanged (`WaitForTxs()`, `h ≠ InitialHeight`, +no `ProposeNextBlockImmediately`). A waiting round `R > 0` publishes its +`NewRound` step, so peers send it the round's proposal. The empty-block tock is +scheduled after `max(CreateEmptyBlocksInterval − (now − StartTime), 0)`: the +interval counts from the start of the height, not from the skipped round. + +## Gossip + +Two gossip handlers run per peer at the `PeerQueryMaj23SleepDuration` cadence +(`internal/consensus/gossip_handlers.go`, wired in `gossip_peer_worker.go`): + +- `roundSkipGossipHandler` sends the held `RoundSkip` to every peer at the + height that is not committing and has not sent a certificate for that round + or a later one (`PeerState.HasRoundSkip`). Nothing else moves a peer out of a + skipped round: it has no votes to follow and the next proposal is sent only + to peers already in the next round. +- `roundVoteResendGossipHandler` (`roundSkipping` only, validator peers) sends + one random vote per pass: the round's prevotes to a peer waiting in this + node's round `r > 0`; the prevotes and precommits of the peer's round, and of + the latest later round with +2/3-any (`laterRoundVotes`), to a peer in a round + this node has left. + +## Recovery + +- Certificates are WAL `msgInfo` records; replay applies them again without + charging the verification budget. Replay never skips rounds on a + `roundSkipping` chain (below). +- On a `roundSkipping` chain `WalSkipRoundsToLast` is ignored + (`State.replaySkipsRounds`, `internal/consensus/replay.go`): whether a round + after a certificate waits depends on polkas of earlier rounds. +- The mempool report is not in the WAL. On a `roundSkipping` chain a replayed + `Propose` step event for the current height enters the recorded round, and + enters the propose step if replay left the node waiting + (`State.readReplayMessage`). +- After a restart into a waiting round the heartbeat or watch starts before the + first input (`State.followLiveness`). + +## Message admission and budgets + +- **State channel.** `ProposerHeartbeat`, `RoundSkipVote` and + `RoundSkipCertificate` travel on the state channel (`p2p.ResolveChannelID`). + Each pays the state-channel cost of 1 on arrival and `livenessTokenCost` + (100) more only once the checks that need no signature find it new + (`Reactor.chargeLivenessNews`); it then joins the sender's peer lane. +- **Verification budget.** Each costs `baseMessageCost` (1) to verify + (`peerMessageCost`, `budgetedMessageCost` in + `internal/consensus/message_cost.go`). One that cannot be afforded is dropped; + it is sent again. +- **Peer errors.** All rejections below are logged at Debug + (`isPeerFloodableError`): + - a liveness message that does not decode or fails its basic checks + (`MsgFromProto`, `errMalformedLivenessMessage`) is reported as + `p2p.ErrInvalidData`, on which the router disconnects the sender; + - a heartbeat from the wrong proposer or a skip vote whose index and + pro-tx-hash disagree, found by the reactor's screening, is a non-fatal + peer error (`PeerStatusBad`); + - a heartbeat, skip vote or certificate whose signature fails against the + key the node holds is reported as a fatal peer error, which disconnects + the sender (`forgedLivenessMiddleware`); + - a certificate for another quorum (`ErrRoundSkipCertificateWrongQuorum`) + is only logged. +- **Outboxes.** Liveness messages are sent from a reactor goroutine through a + bounded outbox (`livenessOutboxSize = 256`), each send bounded by + `voteSetResponseTimeout`; a message that finds the outbox full is dropped and + sent again on its own schedule. Round-step, valid-block, `HasVote` and + `HasCommit` broadcasts go through a separate outbox + (`stateOutboxSize = 1024`); after a drop the sender rebroadcasts the current + round step (`Reactor.sendState`). Neither blocks the consensus goroutine. + +## Signers + +`FilePV` (`privval/file.go`) and `DashCoreSignerClient` +(`privval/dash_core_signer_client.go`) sign heartbeats and skip votes +(`SignProposerHeartbeat`, `SignRoundSkipVote` of the `PrivValidator` +interface, `types/priv_validator.go`). +The socket and gRPC remote signers (`privval/signer_client.go`, +`privval/grpc/client.go`) return `RemoteSignerError` code 501, which matches +`types.ErrSigningNotSupported`. A validator behind such a signer sends no +heartbeats, so validators skip its waiting rounds after the silence timeout, +and it signs no skip votes; it still counts others' votes and applies +certificates. + +## Parameters + +`TimeoutParams.ProposerHeartbeat` (field 7) and `ProposerSilence` (field 8) +come from the committed state's consensus parameters, like the other timeouts. +`TimeoutParamsOrDefaults` derives the values in effect from what is stored: + +| Stored heartbeat | Stored silence | Heartbeat in effect | Silence in effect | +| --- | --- | --- | --- | +| unset | unset | 1 s | 5 s | +| unset | `s` | `min(1 s, s / 5)` | `s` | +| `b` | unset | `b` | `max(5 s, 5·b)` | +| `b` | `s` | `b` | `s` | + +- **Validation** (`ConsensusParams.ValidateConsensusParams`): neither negative; + the derived heartbeat is positive and shorter than the derived silence. +- **Updates** (`ConsensusParams.UpdateConsensusParams`): an update that names + either field, or that raises `ConsensusVersion` from below 2 to 2 or more, + clears both and sets only those it names; the other is derived again. +- **Encoding** (`ConsensusParams.ToProto`): unset fields are omitted. +- **Hashing** (`ConsensusParams.HashConsensusParams`): from version 2, + `HashedParams.proposer_heartbeat_ns` (field 4) and `proposer_silence_ns` + (field 5) carry the values in effect, in nanoseconds; earlier versions leave + them unset, so their hash is unchanged. + +## Metrics + +| Metric | Counts | +| --- | --- | +| `consensus_proposer_heartbeats_sent` | Heartbeats this node signed and broadcast | +| `consensus_proposer_heartbeats_received` | New, verified heartbeats for the round this node waits in | +| `consensus_proposer_silences` | Rounds in which this validator stopped hearing from the proposer while waiting | +| `consensus_round_skips` | Rounds this node moved past on a certificate | + +Defined in `internal/consensus/metrics.go`. diff --git a/spec/consensus/consensus-recovery.md b/spec/consensus/consensus-recovery.md index 07929987b..d91731563 100644 --- a/spec/consensus/consensus-recovery.md +++ b/spec/consensus/consensus-recovery.md @@ -12,12 +12,14 @@ The consensus WAL (`internal/consensus/wal.go`; file layout in | Record | Written by | Durability | | -------- | ------------ | ------------ | | `msgInfo` — proposal, block part, vote or commit, with the peer id and receive time | `walMiddleware` (`internal/consensus/msg_handlers.go`) | peer messages `Write` (buffered); the node's own messages `WriteSync` (fsync), **panic** on failure | +| `msgInfo` — round skip certificate, once verified and news | `AddRoundSkipCertificateAction.Execute` | `Write`; a failure is only logged | | `timeoutInfo` — every tock | `State.receiveRoutine` | `Write` | | `EventDataRoundState` — every step change | `EventPublisher.PublishNewRoundStepEvent` | `Write` | | `EndHeightMessage{h}` | `ApplyCommitAction.Execute`, after `SaveBlock` and before `FinalizeBlock` | `WriteSync`, **panic** on failure | `verificationBudgetMiddleware` wraps `walMiddleware`, so a message dropped -for budget reasons leaves no WAL record. Before signing a proposal or a vote +for budget reasons leaves no WAL record. Proposer heartbeats and round skip +votes are never written (`livenessMessage`). Before signing a proposal or a vote the WAL is flushed and fsynced (`EnterProposeAction.Execute`, `voteSigner.signVote`) so that replay reproduces the same message for the private validator's double-sign protection. A failed sync stops the vote @@ -59,10 +61,13 @@ ticker and, unless `doWALCatchup` was disabled by block sync, runs - the WAL must **not** contain `#ENDHEIGHT h` for the current height; - it must contain `#ENDHEIGHT h − 1` (`0` at `InitialHeight`); - every record after it is replayed through `readReplayMessage`: step - events are only logged, `msgInfo`s are dispatched with `fromReplay` (no + events are only logged (except as below), `msgInfo`s are dispatched with `fromReplay` (no peer budgets, no evictions), tocks go through `handleTimeout`. With `WalSkipRoundsToLast` a replayed proposal for a later round advances the - round directly; + round directly; on `CONSENSUS_VERSION_2` chains that skip rounds the setting + is ignored, and a replayed `Propose` step event re-enters the recorded round + and propose step (`State.replaySkipsRounds`, `State.readReplayMessage`; + [Recovery](./consensus-proposer-liveness.md#recovery)); - replay mode: the node does not propose (`EnterProposeAction.Execute`) and skips timeliness checks (`isValidForPrevote`), but it may still attempt to sign votes: replayed messages drive the normal actions, and neither diff --git a/spec/consensus/consensus-transitions.md b/spec/consensus/consensus-transitions.md index a0ed60fdc..ba7971c43 100644 --- a/spec/consensus/consensus-transitions.md +++ b/spec/consensus/consensus-transitions.md @@ -25,11 +25,15 @@ Notation: `h = rs.Height`, `r = rs.Round`, `step = rs.Step`; | Tock `(H, R, Propose)` | `State.handleTimeout` | publish `TimeoutPropose`; `EnterPrevote{H, R}` | | Tock `(H, R, PrevoteWait)` | `State.handleTimeout` | publish `TimeoutWait`; `EnterPrecommit{H, R}` | | Tock `(H, R, PrecommitWait)` | `State.handleTimeout` | publish `TimeoutWait`; `EnterPrecommit{H, R}`; `EnterNewRound{H, R + 1}` | -| `TxsAvailable` | `State.handleTxsAvailable` | only at `r == 0`: at step `NewHeight` (and `h ≠ InitialHeight`) schedule tock `(h, 0, NewRound)` after `StartTime − now + 1 ms`; at step `NewRound` dispatch `EnterPropose{h, 0}` | -| `ProposalMessage` | `proposalMessageHandler` → `Proposaler.Set` | see [Proposal acceptance](#proposal-acceptance-proposalerset) | +| `TxsAvailable` | `State.handleTxsAvailable` | sets `TxsReported`; acts only at `r == 0` or in a round that may wait after a round skip (`mayWaitAfterRoundSkip`): at step `NewHeight` (and `h ≠ InitialHeight`) schedule tock `(h, 0, NewRound)` after `StartTime − now + 1 ms`; at step `NewRound` dispatch `EnterPropose{h, r}` | +| `ProposalMessage` | `proposalMessageHandler` → `Proposaler.Set` | see [Proposal acceptance](#proposal-acceptance-proposalerset); if accepted while `waitingForTxs`, `EnterPropose{h, r}` | | `BlockPartMessage` | `blockPartMessageHandler` | `AddProposalBlockPart`; on completion `ProposalCompleted` or `AddCommit` | | `VoteMessage` | `voteMessageHandler` | `AddVote` → [vote-driven dispatch](#vote-driven-dispatch) | | `CommitMessage` | `commitMessageHandler` | `TryAddCommit` ([Commit handling](./consensus-commit.md)) | +| `RoundSkipCertificateMessage` | `roundSkipCertificateMessageHandler` | [`AddRoundSkipCertificate`](#addroundskipcertificatecert) | +| `ProposerHeartbeatMessage`, `RoundSkipVoteMessage` | `State.handleLivenessMessage` | none, or `AddRoundSkipCertificate` for a certificate the vote completed ([Proposer liveness](./consensus-proposer-liveness.md)) | +| Proposer-silence report | `State.handleProposerSilence` | none; signs a round skip vote off the consensus goroutine | +| Signed round skip vote | `State.handleSignedRoundSkipVote` | none, or `AddRoundSkipCertificate` for a certificate the vote completed | Every tock is written to the WAL before handling; `handleTimeout` drops a tock when `ti.Height ≠ h`, `ti.Round < r`, or `ti.Round == r && ti.Step < step` @@ -71,11 +75,16 @@ The receive routine handles the tock only if it still matches 3. `Votes.SetRound(R + 1)`; `TriggeredTimeoutPrecommit = false`; save. 4. Publish `NewRound`; if a commit is parked also publish the step and a `ValidBlock` event so peers keep serving the committed block's parts. - 5. `waitForTxs = config.WaitForTxs() ∧ R == 0 ∧ h ≠ InitialHeight ∧ ¬ProposeNextBlockImmediately` + 5. `waitForTxs = config.WaitForTxs() ∧ (R == 0 ∨ waitsAfterRoundSkip(R)) ∧ h ≠ InitialHeight ∧ ¬ProposeNextBlockImmediately ∧ ¬(roundSkipping ∧ Proposal ≠ nil)` where `WaitForTxs = ¬CreateEmptyBlocks ∨ CreateEmptyBlocksInterval > 0` - (`config/config.go`). + (`config/config.go`); `roundSkipping` is defined in + [When it applies](./consensus-proposer-liveness.md#when-it-applies) and + `waitsAfterRoundSkip` in + [The round after a skip](./consensus-proposer-liveness.md#the-round-after-a-skip). + - `waitForTxs ∧ R > 0 ∧ Commit == nil`: publish the `NewRound` step. - `waitForTxs ∧ CreateEmptyBlocksInterval > 0`: schedule tock - `(h, R, NewRound)` after the interval; stay in `NewRound`. + `(h, R, NewRound)` after the interval (for `R > 0`, the interval + minus `now − StartTime`, at least 0); stay in `NewRound`. - `waitForTxs ∧ CreateEmptyBlocksInterval == 0`: stay in `NewRound` until `TxsAvailable`. - otherwise (unless the test-only `DontAutoPropose`): `Dispatch(EnterPropose{h, R})`. @@ -240,6 +249,19 @@ Properties: `CommitTime = now`, publish step, `Dispatch(TryFinalizeCommit{h})`. - **Postcondition:** `step = ApplyCommit`, `CommitRound = CR`. +### AddRoundSkipCertificate(cert) + +`AddRoundSkipCertificateAction.Execute` (`internal/consensus/state_add_round_skip.go`). + +- **Guard:** `roundSkipping ∧ cert.Height == h ∧ (RoundSkip == nil ∨ cert.Round > RoundSkip.Round) ∧ step < ApplyCommit` + (`roundSkipCertificateWanted`). +- **Actions:** verify the certificate unless this node recovered it (budget, + quorum hash, threshold signature); write its message to the WAL (not on replay); + `RoundSkip = cert`; broadcast it; if `cert.Round ≥ r`, + `Dispatch(EnterNewRound{h, cert.Round + 1})`. +- **Postcondition:** `RoundSkip = cert`; `r > cert.Round`, or `r` unchanged + when the certificate is for an earlier round. + ### TryFinalizeCommit(H) / ApplyCommit(commit) See [Commit handling](./consensus-commit.md#applying-a-commit-built-from-local-precommits). @@ -298,6 +320,7 @@ unless it is the node's own key. | `r == vote.Round ∧ step ≥ Prevote ∧ maj23(P) = B ∧ (complete() ∨ B = nil)` | `EnterPrecommit{h, r}` | | `r == vote.Round ∧ step ≥ Prevote ∧ any23(P)` (otherwise) | `EnterPrevoteWait{h, r}` | | `Proposal ≠ nil ∧ Proposal.POLRound == vote.Round ≥ 0 ∧ complete()` (and neither of the above) | `EnterPrevote{h, r}` (POL votes completed a proposal with `POLRound`) | +| `waitingForTxs ∧ (maj23(P) exists ∨ (vote.Round == r ∧ r > 0))` (and none of the above) | `EnterPropose{h, r}` (ends a wait for transactions; `CONSENSUS_VERSION_2`) | ### After a precommit is added diff --git a/spec/consensus/consensus.md b/spec/consensus/consensus.md index dd50d52b9..ce3226aed 100644 --- a/spec/consensus/consensus.md +++ b/spec/consensus/consensus.md @@ -23,6 +23,7 @@ and the ABCI interface ([ABCI spec](../abci/README.md)). | [Primary flow of a height](./consensus-happy-path.md) | The happy path from `NewHeight` to the next `NewHeight`, with the ABCI calls it makes | | [Transitions and decision rules](./consensus-transitions.md) | Formal transition table: guards, actions and postconditions of every step action; timeout semantics; vote-driven dispatch; prevote, precommit and proposal-validity decision procedures | | [Alternative flows and edge cases](./consensus-alternative-flows.md) | Missing or invalid proposals, locking, valid-block bookkeeping, round skipping, nil rounds, equivocation, vote extensions, timestamps, empty blocks, non-validators, panics, DoS budgets | +| [Proposer liveness and round skipping](./consensus-proposer-liveness.md) | From `CONSENSUS_VERSION_2`: proposer heartbeats while a round waits for transactions, skip votes, round skip certificates, the round after a skip, their parameters and budgets | | [Commit handling](./consensus-commit.md) | Commits built from local precommits and commits received from peers: verification, parking, candidates, application of a commit, gossip | | [Recovery and synchronisation](./consensus-recovery.md) | WAL, crash recovery, ABCI handshake replay, block-sync handover | | [Known issues and discrepancies](./consensus-known-issues.md) | Behaviour that looks like a bug or diverges from the intended algorithm | @@ -48,7 +49,7 @@ and the complete DoS-defence design. For operational monitoring, see [DoS observability](../../docs/dos-observability/README.md). Source of truth: the code at the revision this document was written against -(branch `v1.8-dev`). Citations use `path/file.go` and a function name; line +(branch `dev`). Citations use `path/file.go` and a function name; line numbers are omitted because they drift. ## Terminology @@ -72,6 +73,7 @@ numbers are omitted because they drift. | **Locked block / round** | The block the node precommitted most recently and the round it did so; `-1`/nil when none. | `RoundState.LockedRound/LockedBlock` | | **Valid block / round** | The most recent block (and round) for which the node saw a polka in a round `≥` any previous one. | `RoundState.ValidRound/ValidBlock` | | **CurrentRoundState** | The application's answer to the latest `PrepareProposal`/`ProcessProposal` for the block held in `ProposalBlock`, tagged with the round it was processed for. | `RoundState.CurrentRoundState`, `RoundState.ProcessedBlockID`; `processedFor` in `internal/consensus/block_executor.go` | +| **Round skip certificate** | A quorum threshold signature recovered from the skip votes of validators holding `R`, proving that they stopped hearing from the proposer of `(h, r)` while waiting for transactions; moves a node to `r + 1`. `CONSENSUS_VERSION_2` only. | `types.RoundSkipCertificate` in `types/round_skip.go`; [Proposer liveness](./consensus-proposer-liveness.md) | | **Parked commit** | A verified commit for the current height whose block has not yet been assembled (`RoundState.Commit != nil`). | `StateData.adoptCommit` in `internal/consensus/state_data.go` | | **Timely** | A proposal whose receive time lies within `[ts − Precision, ts + MessageDelay·2^⌊round/10⌋ + Precision]` of its timestamp `ts`. | `checkTimely` in `types/validation.go` | @@ -81,11 +83,13 @@ Throughout, `rs` denotes the node's `RoundState` and `state` the committed ## System model as enforced by the code - **Single writer.** `State.receiveRoutine` (`internal/consensus/state.go`) is - the only goroutine that mutates `RoundState`. It `select`s over three inputs: + the only goroutine that mutates `RoundState`. It `select`s over five inputs: transactions becoming available in the mempool (`txNotifier.TxsAvailable()`), messages (`msgInfoQueue.read()`: the node's own proposal/votes on an internal - queue, peers' messages on per-peer lanes), and timer expiries - (`timeoutTicker.Chan()`). Go's `select` picks randomly among ready cases, so + queue, peers' messages on per-peer lanes), timer expiries + (`timeoutTicker.Chan()`), proposer-silence reports + (`proposerLiveness.silent()`) and signed round skip votes + (`roundSkipSigner.signed()`). Go's `select` picks randomly among ready cases, so there is **no ordering guarantee** between a timeout and a message that are both ready. - **Actions and guards.** Every transition is an `ActionHandler` registered in @@ -132,13 +136,15 @@ Throughout, `rs` denotes the node's `RoundState` and `state` the committed | `Votes` | `HeightVoteSet`: prevote and precommit `VoteSet`s for rounds `0..Round+1` plus up to two catch-up rounds per validator. | | `LastCommit` | The commit of height `h − 1`, included in the next proposed block. There is no `LastCommit` vote set: late precommits for `h − 1` are dropped. | | `TriggeredTimeoutPrecommit` | Whether the precommit-wait timeout was already scheduled for this round. | +| `RoundSkip` | Round skip certificate for the highest skipped round of this height; kept for peers still in that round even after the node moved past it. | +| `TxsReported` | Whether the mempool reported transactions at this height (it does so once per height). | | `ProposeNextBlockImmediately` | Hint from `ResponseFinalizeBlock.propose_next_block_immediately` of the previous height; consumed once, not persisted. | | `CurrentRoundState, ProcessedBlockID` | Latest application result for `ProposalBlock` (see Terminology). | `StateData.updateToState` re-initialises `Height`, `Validators`, `LastValidators`, `LastCommit`, `ProposerSelector` and `StartTime` for the -new height and resets every proposal, lock, valid-block, commit, vote and -flag field. `CurrentRoundState`/`ProcessedBlockID` are left untouched; they +new height and resets every proposal, lock, valid-block, commit, +round-skip, vote and flag field. `CurrentRoundState`/`ProcessedBlockID` are left untouched; they are compared against the new height's block by `processedFor`. Within a height `LockedRound/Block` and `ValidRound/Block` are never reset; `Commit` is cleared only when a parked commit is rejected (`selectParkedCommit`, @@ -165,7 +171,7 @@ stateDiagram-v2 [*] --> NewHeight : updateToState (height h, round 0) NewHeight --> NewRound : tock NewHeight / EnterNewRound(h,0) NewRound --> NewRound : proposer waits for clock > LastBlockTime (NewRound tock rescheduled) - NewRound --> Propose : EnterPropose (immediately, or after txs available / empty-block interval) + NewRound --> Propose : EnterPropose (immediately, or after txs available / empty-block interval / proposal or prevotes while waiting) Propose --> Prevote : proposal + block (+ POL) complete, or tock Propose Prevote --> PrevoteWait : prevotes reach +2/3 any without a usable majority Prevote --> Precommit : polka (block held or nil) @@ -175,6 +181,7 @@ stateDiagram-v2 Precommit --> ApplyCommit : +2/3 precommits for a block (threshold signature recovered) state "any step" as Any Any --> NewRound : +2/3 any prevotes or precommits at round r' > r (round skip) + Any --> NewRound : RoundSkipCertificate for round r' ≥ r, before ApplyCommit / EnterNewRound(h, r'+1) Any --> ApplyCommit : verified CommitMessage for height h ApplyCommit --> NewHeight : block applied (height h+1) ApplyCommit --> ApplyCommit : waiting for block parts, or application rejected own commit @@ -279,6 +286,8 @@ proposed in block `h` apply from `h + 1`, `CurrentRoundState` in | `VoteDelta` | 500 ms | | | `Synchrony.Precision` | 505 ms | Timeliness window | | `Synchrony.MessageDelay` | 12 s | Timeliness window; doubled every 10 rounds | +| `ProposerHeartbeat` | 1 s, derived when only `ProposerSilence` is set | Heartbeat interval of the proposer of a waiting round (`CONSENSUS_VERSION_2`) | +| `ProposerSilence` | 5 s, derived when only `ProposerHeartbeat` is set | `ProposerSilenceTimeout(r) = ProposerSilence + ProposerHeartbeat·r`; see [Parameters](./consensus-proposer-liveness.md#parameters) | Per-node `Unsafe*TimeoutOverride` settings in `config.ConsensusConfig` replace the corresponding parameter locally (`StateData.proposeTimeout`, diff --git a/spec/consensus/proposer-selection.md b/spec/consensus/proposer-selection.md index 0818ec573..01443b12d 100644 --- a/spec/consensus/proposer-selection.md +++ b/spec/consensus/proposer-selection.md @@ -47,7 +47,7 @@ strategy by `ConsensusParams.Version.ConsensusVersion`: | `ConsensusVersion` | Strategy | File | | --- | --- | --- | | 0 (default) | **Height-based**: the index advances once per height; the round offset is derived without being persisted | `height_proposer.go` (`heightProposerSelector`) | -| 1 | **Height-and-round-based**: the index advances once per height and once per round | `height_round_proposer.go` (`heightRoundProposerSelector`) | +| 1, 2 | **Height-and-round-based**: the index advances once per height and once per round. Version 2 keeps this strategy and adds [proposer liveness](./consensus-proposer-liveness.md) | `height_round_proposer.go` (`heightRoundProposerSelector`) | Changing the version is a consensus-parameter update and takes effect at the next height ([Validator-set and parameter changes](./consensus-alternative-flows.md#validator-set-and-parameter-changes)). diff --git a/spec/consensus/readme.md b/spec/consensus/readme.md index 036a711a3..3f4668520 100644 --- a/spec/consensus/readme.md +++ b/spec/consensus/readme.md @@ -17,6 +17,7 @@ implementation in `internal/consensus`. - [Primary flow of a height](./consensus-happy-path.md) - [Transitions and decision rules](./consensus-transitions.md) - [Alternative flows and edge cases](./consensus-alternative-flows.md) + - [Proposer liveness and round skipping](./consensus-proposer-liveness.md) - [Commit handling](./consensus-commit.md) - [Recovery and synchronisation](./consensus-recovery.md) - [Known issues and discrepancies](./consensus-known-issues.md) diff --git a/spec/consensus/wal.md b/spec/consensus/wal.md index 6a0756544..37b4e1530 100644 --- a/spec/consensus/wal.md +++ b/spec/consensus/wal.md @@ -1,6 +1,8 @@ # WAL -Consensus module writes every message to the WAL (write-ahead log). Which +Consensus module writes every message to the WAL (write-ahead log), except +proposer heartbeats and round skip votes; round skip certificates are written +once verified. Which records are written, with which durability, and how they are replayed is specified in [Recovery and synchronisation](./consensus-recovery.md). diff --git a/spec/core/data_structures.md b/spec/core/data_structures.md index 3a996016d..624af59a1 100644 --- a/spec/core/data_structures.md +++ b/spec/core/data_structures.md @@ -252,6 +252,10 @@ enum SignedMsgType { // Proposal SIGNED_MSG_TYPE_PROPOSAL = 32; + + // Proposer liveness, from CONSENSUS_VERSION_2 + SIGNED_MSG_TYPE_PROPOSER_HEARTBEAT = 48; + SIGNED_MSG_TYPE_ROUND_SKIP = 49; } ``` @@ -378,6 +382,8 @@ This override does not lower the separate BLS recovery threshold `R` returned by `QuorumTypeThresholdVotingPower`. Prevotes expose a majority at `T`; precommits also require successful recovery and verification of threshold signatures. A low vote-counting threshold alone does not let one signature share produce a commit. +A threshold above `R` disables round skipping +([Proposer liveness](../consensus/consensus-proposer-liveness.md#when-it-applies)). See [Threshold recovery](../consensus/consensus.md#threshold-recovery) for the canonical extension-group and recovery rules, and [Commit threshold is configurable below the BFT bound](../consensus/consensus-known-issues.md#commit-threshold-is-configurable-below-the-bft-bound) @@ -390,7 +396,7 @@ parameter is absent from a particular header-hash subset. | Name | Type | Description | Field Number | |-------------|--------|-------------------------------|--------------| | app_version | uint64 | The ABCI application version. | 1 | -| consensus_version | `ConsensusVersion` | Version of the consensus protocol, selecting the [proposer strategy](../consensus/proposer-selection.md). | 2 | +| consensus_version | `ConsensusVersion` | Version of the consensus protocol, selecting the [proposer strategy](../consensus/proposer-selection.md). `CONSENSUS_VERSION_2` also enables [proposer liveness](../consensus/consensus-proposer-liveness.md). | 2 | ### SynchronyParams @@ -407,6 +413,8 @@ parameter is absent from a particular header-hash subset. | propose_delta | [google.protobuf.Duration](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration) | Parameter that, along with propose, configures the timeout for the propose step of the consensus algorithm. | 2 | | vote | [google.protobuf.Duration](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration) | Parameter that, along with vote_delta, configures the timeout for the prevote and precommit step of the consensus algorithm. | 3 | | vote_delta | [google.protobuf.Duration](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration) | Parameter that, along with vote, configures the timeout for the prevote and precommit step of the consensus algorithm. | 4 | +| proposer_heartbeat | [google.protobuf.Duration](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration) | How often the proposer of a round that waits for transactions sends a heartbeat. Used from `CONSENSUS_VERSION_2`; unset means derived (see [Parameters](../consensus/consensus-proposer-liveness.md#parameters)). | 7 | +| proposer_silence | [google.protobuf.Duration](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration) | How long a validator waits without a heartbeat before it votes to skip the round; grows by `proposer_heartbeat` per round. Used from `CONSENSUS_VERSION_2`; unset means derived. | 8 | ### ABCIParams diff --git a/spec/core/genesis.md b/spec/core/genesis.md index 250f608d8..405edb1cf 100644 --- a/spec/core/genesis.md +++ b/spec/core/genesis.md @@ -19,6 +19,7 @@ The genesis file is the starting point of a chain. An application will populate - `pub_key_types`: Lists the validator consensus key types allowed by the chain. Recognized values are `ed25519`, `secp256k1`, and `bls12381`; the default allows only `bls12381` (BLS12-381). - `version` - `app_version`: The version of the application. This is set by the application and is used to identify which version of the app a user should be using in order to operate a node. + - `consensus`: The consensus protocol version (`CONSENSUS_VERSION_*`). It selects the [proposer strategy](../consensus/proposer-selection.md); from version 2 it also enables [proposer liveness and round skipping](../consensus/consensus-proposer-liveness.md). - `synchrony` - `message_delay`: A bound on how long a proposal message may take to reach all validators on a network and still be considered valid. - `precision`: A bound on how skewed the proposer's clock may be from any validator on the network while still producing valid proposals. @@ -27,6 +28,8 @@ The genesis file is the starting point of a chain. An application will populate - `propose_delta`: How much the propose timeout increase with each round. - `vote`: How long the consensus engine will wait after receiving +2/3 votes in a round. - `vote_delta`: How much the vote timeout increases with each round. + - `proposer_heartbeat`: From consensus version 2, how often the proposer of a round that waits for transactions sends a heartbeat. Optional; see [Parameters](../consensus/consensus-proposer-liveness.md#parameters) for the defaults. + - `proposer_silence`: From consensus version 2, how long a validator waits without a heartbeat before voting to skip the round. Optional; see [Parameters](../consensus/consensus-proposer-liveness.md#parameters) for the defaults. - `validators` - This is an array of validators. This validator set is used as the starting validator set of the chain. This field can be empty, if the application sets the validator set in `InitChain`. - `app_hash`: The applications state root hash. This field does not need to be populated at the start of the chain, the application may provide the needed information via `Initchain`. diff --git a/spec/core/state.md b/spec/core/state.md index 858324b1b..a168d18d1 100644 --- a/spec/core/state.md +++ b/spec/core/state.md @@ -40,7 +40,9 @@ see [Recovery and synchronisation](../consensus/consensus-recovery.md). [Version](./data_structures.md#version) distinguishes block protocol version from application version. `ConsensusParams.Version.ConsensusVersion` additionally -selects the [proposer-selection strategy](../consensus/proposer-selection.md). +selects the [proposer-selection strategy](../consensus/proposer-selection.md) +and, from version 2, enables +[proposer liveness](../consensus/consensus-proposer-liveness.md). ## Block diff --git a/spec/p2p/messages/consensus.md b/spec/p2p/messages/consensus.md index 40590d548..6bb5ad62b 100644 --- a/spec/p2p/messages/consensus.md +++ b/spec/p2p/messages/consensus.md @@ -151,6 +151,75 @@ the votes a process has. | block_id | [BlockID](../../core/data_structures.md#blockid) | | 4 | | votes | BitArray | Votes the process has for the BlockID. | 5 | +### ProposerHeartbeat + +ProposerHeartbeat is sent on the StateChannel by the proposer of a round that +waits for transactions, and relayed once by validators that verify it +(`CONSENSUS_VERSION_2`; see +[Proposer liveness](../../consensus/consensus-proposer-liveness.md#heartbeats)). + +| Name | Type | Description | Field Number | +|-----------|--------------------------------------------------|-----------------------|--------------| +| heartbeat | [ProposerHeartbeat](#proposerheartbeat-contents) | The signed heartbeat. | 1 | + +#### ProposerHeartbeat contents + +`tendermint.types.ProposerHeartbeat`: + +| Name | Type | Description | Field Number | +|----------------------|---------------------------|----------------------------------------------------------------------------------------------------------------------------------|--------------| +| height | int64 | Height of the round. | 1 | +| round | int32 | Round the sender proposes in. | 2 | +| sequence | uint64 | Grows with every heartbeat of the round, counting from the signing time in milliseconds. | 3 | +| proposer_pro_tx_hash | bytes | ProTxHash of the proposer. | 4 | +| timestamp | google.protobuf.Timestamp | Signing time. | 5 | +| signature | bytes | Signature with the proposer's quorum key share ([Signing](../../consensus/signing.md#proposer-heartbeats-and-round-skip-votes)). | 6 | + +### RoundSkipVote + +RoundSkipVote is sent on the StateChannel by a validator that heard no +heartbeat from the proposer of its waiting round for the silence timeout, and +relayed by validators when they first count it and again on a later copy until +the round is certified (`CONSENSUS_VERSION_2`; see +[Collecting votes](../../consensus/consensus-proposer-liveness.md#collecting-votes-and-recovering-the-certificate)). + +| Name | Type | Description | Field Number | +|------|------------------------------------------|-----------------------|--------------| +| vote | [RoundSkipVote](#roundskipvote-contents) | The signed skip vote. | 1 | + +#### RoundSkipVote contents + +`tendermint.types.RoundSkipVote`: + +| Name | Type | Description | Field Number | +|-----------------------|-------|---------------------------------------------------------------------------------|--------------| +| height | int64 | Height of the round to skip. | 1 | +| round | int32 | Round to skip. | 2 | +| validator_pro_tx_hash | bytes | ProTxHash of the voting validator. | 3 | +| validator_index | int32 | Index of the voting validator in the validator set. | 4 | +| signature | bytes | Signature share over the round's skip sign bytes, the same for every validator. | 5 | + +### RoundSkipCertificate + +RoundSkipCertificate is sent on the StateChannel by a node that recovered or +received a valid certificate, and resent by gossip to peers at its height that +have not sent one for that round or a later one (`CONSENSUS_VERSION_2`). + +| Name | Type | Description | Field Number | +|-------------|--------------------------------------------------------|-----------------------------|--------------| +| certificate | [RoundSkipCertificate](#roundskipcertificate-contents) | The round skip certificate. | 1 | + +#### RoundSkipCertificate contents + +`tendermint.types.RoundSkipCertificate`: + +| Name | Type | Description | Field Number | +|---------------------|-------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------| +| height | int64 | Height of the skipped round. | 1 | +| round | int32 | Skipped round. | 2 | +| quorum_hash | bytes | Quorum that signed. | 3 | +| threshold_signature | bytes | Quorum threshold signature recovered from the skip votes of +2/3 of the validator set (the mainnet default; the quorum type and `VotingPowerThreshold` can change it). | 4 | + ### Message Message is a [`oneof` protobuf type](https://developers.google.com/protocol-buffers/docs/proto#oneof). @@ -168,3 +237,6 @@ Message is a [`oneof` protobuf type](https://developers.google.com/protocol-buff | vote_set_bits | [VoteSetBits](#votesetbits) | | 9 | | commit | [Commit](#commit) | | 10 | | has_commit | [HasCommit](#hascommit) | | 11 | +| proposer_heartbeat | [ProposerHeartbeat](#proposerheartbeat) | | 12 | +| round_skip_vote | [RoundSkipVote](#roundskipvote) | | 13 | +| round_skip_certificate | [RoundSkipCertificate](#roundskipcertificate) | | 14 |