Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions abci/types/types.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions book/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@
- [Primary flow of a height](spec/consensus/consensus-happy-path.md)
- [Transitions and decision rules](spec/consensus/consensus-transitions.md)
- [Alternative flows and edge cases](spec/consensus/consensus-alternative-flows.md)
- [Proposer liveness and round skipping](spec/consensus/consensus-proposer-liveness.md)
- [Commit handling](spec/consensus/consensus-commit.md)
- [Recovery and synchronisation](spec/consensus/consensus-recovery.md)
- [Known issues and discrepancies](spec/consensus/consensus-known-issues.md)
Expand Down
7 changes: 6 additions & 1 deletion docs/nodes/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -487,7 +487,12 @@ namespace = "tendermint"
`create-empty-blocks = false` makes round 0 wait for transactions above the
initial height. A positive `create-empty-blocks-interval` starts a proposal after
that interval even without transactions. Later rounds do not wait for
transactions.
transactions, except on a chain with consensus version 2, where a round entered
because the validators skipped a silent proposer's waiting round may wait too,
unless the node holds a locked or valid block, the mempool has already reported
transactions at this height, a polka exists at the height, or a prevote of that
round was recorded; its interval counts from the start of the height. See
[Proposer liveness and round skipping](../../spec/consensus/consensus-proposer-liveness.md).

The application can bypass that wait for the next height by setting
`ResponseFinalizeBlock.propose_next_block_immediately`. The hint is local,
Expand Down
4 changes: 4 additions & 0 deletions docs/nodes/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,10 @@ The following metrics are available:
| consensus_verification_budget_saturation | Gauge | | Fraction of the node-wide verification budget still available at a peer-message check (1.0 full/idle, 0.0 drained/under pressure) |
| consensus_peer_lane_active_count | Gauge | | Number of per-peer lanes currently holding queued messages and taking turns in the scheduler |
| consensus_peer_lane_max_depth | Gauge | | Largest number of messages queued in any single peer lane |
| consensus_proposer_heartbeats_sent | Counter | | Proposer heartbeats this node signed and broadcast while waiting for transactions as proposer (consensus version 2) |
| consensus_proposer_heartbeats_received | Counter | | New, verified proposer heartbeats accepted for the round this node waits in |
| consensus_proposer_silences | Counter | | Rounds in which the proposer went silent while this validator waited for transactions |
| consensus_round_skips | Counter | | Rounds this node moved past on a round skip certificate from +2/3 of the validators (by default; configuration can change the threshold) |
| evidence_pool_num_evidence | Gauge | | Number of evidence in the evidence pool |
| evidence_pool_dropped_evidence | Counter | reason | Inbound gossip evidence refused before verification, labeled by reason (structural, oversized, already_known, height_window, node_budget, peer_budget) |
| p2p_peers | Gauge | | Number of peers node's connected to |
Expand Down
2 changes: 2 additions & 0 deletions docs/nodes/remote-signer.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ Tenderdash provides a remote signer option for validators. A remote signer enabl

The remote signer protocol implements a [client and server architecture](https://en.wikipedia.org/wiki/Client%E2%80%93server_model). When Tenderdash requires the public key or signature for a proposal or vote it requests it from the remote signer.

The remote signer protocol does not cover proposer heartbeats or round skip votes. On a chain with consensus version 2, a validator using a remote signer sends neither, so validators skip its rounds that wait for transactions once its heartbeats are missed. See [Proposer liveness and round skipping](../../spec/consensus/consensus-proposer-liveness.md#signers).

To run a secure validator and remote signer system it is recommended to use a VPC (virtual private cloud) or a private connection.

There are two different configurations that can be used: Raw or gRPC.
Expand Down
3 changes: 2 additions & 1 deletion docs/tenderdash-core/using-tenderdash.md
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,8 @@ Some fields from the config file can be overwritten with flags.
## No Empty Blocks

Configure `create-empty-blocks` and `create-empty-blocks-interval` in
`config.toml` to control waiting for transactions at round 0. The application
`config.toml` to control waiting for transactions at round 0 (and, on consensus
version 2, in a round entered after a silent proposer's round was skipped). The application
can request an immediate next block through `ResponseFinalizeBlock`.
An app hash change does not itself require an extra block: Tenderdash uses
same-block execution.
Expand Down
7 changes: 4 additions & 3 deletions proto/tendermint/abci/types.proto
Original file line number Diff line number Diff line change
Expand Up @@ -600,9 +600,10 @@ message ResponseFinalizeBlock {
// Blocks below this height may be removed. Defaults to `0` (retain all).
int64 retain_height = 2;
// When `true`, this node does not wait for transactions (`create-empty-blocks-interval`)
// before proposing round 0 of the next height: the application has block-driven work
// pending, for example withdrawal transactions to sign. A hint for the local node only,
// consumed once; a node that restarts in between falls back to the configured wait.
// before proposing at the next height, in round 0 or, on consensus version 2, in a round
// entered after a silent proposer's round was skipped: the application has block-driven
// work pending, for example withdrawal transactions to sign. A hint for the local node
// only, consumed once; a node that restarts in between falls back to the configured wait.
// Defaults to `false`.
bool propose_next_block_immediately = 3;
}
Expand Down
2 changes: 1 addition & 1 deletion spec/abci/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -793,7 +793,7 @@ nondeterministic
| Field | Type | Label | Description |
| ----- | ---- | ----- | ----------- |
| retain_height | [int64](#int64) | | Blocks below this height may be removed. Defaults to `0` (retain all). |
| propose_next_block_immediately | [bool](#bool) | | When `true`, this node does not wait for transactions (`create-empty-blocks-interval`) before proposing round 0 of the next height: the application has block-driven work pending, for example withdrawal transactions to sign. A hint for the local node only, consumed once; a node that restarts in between falls back to the configured wait. Defaults to `false`. |
| propose_next_block_immediately | [bool](#bool) | | When `true`, this node does not wait for transactions (`create-empty-blocks-interval`) before proposing at the next height, in round 0 or, on consensus version 2, in a round entered after a silent proposer's round was skipped: the application has block-driven work pending, for example withdrawal transactions to sign. A hint for the local node only, consumed once; a node that restarts in between falls back to the configured wait. Defaults to `false`. |



Expand Down
2 changes: 1 addition & 1 deletion spec/abci/basic-concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ In summary:
### Other methods

Additionally, there is a [**Flush**](methods.md#flush) method that is called on every connection,
and an [**Echo**](methods.md#echo) method that is just for debugging.
and an [**Echo**](methods.md#echo) method for debugging, which a waiting proposer also uses as a readiness probe.

More details on managing state across connections can be found in the section on
[ABCI Applications](app-requirements.md).
Expand Down
4 changes: 3 additions & 1 deletion spec/abci/expected-behavior.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,9 @@ ordinary voting rounds.

`CheckTx` controls mempool admission separately from the block execution sequence.
`Query`, `ListSnapshots` and `LoadSnapshotChunk` read application state or existing
snapshots. `Echo` and `Flush` serve client/debugging purposes. `Info` can also serve
snapshots. `Echo` and `Flush` serve client/debugging purposes; from consensus version 2 a
waiting proposer also uses `Echo` as a readiness probe
([Echo](./methods.md#echo)). `Info` can also serve
RPC queries independently of startup. These methods are not bounded by the normal
height's consensus callback count.

Expand Down
9 changes: 8 additions & 1 deletion spec/abci/methods.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,12 @@ title: Methods
* `Message (string)`: The input string
* **Usage**:
* Echo a string to test an abci client/server implementation
* From consensus version 2, the proposer of a round that waits for transactions probes
the backends that receive `PrepareProposal` with `Echo` before each heartbeat, once it
is not held back after block sync. At most one probe is in flight; an unanswered probe
is reused by later heartbeats rather than repeated. A heartbeat is sent only if the
backends answer within the heartbeat interval. See
[Proposer liveness](../consensus/consensus-proposer-liveness.md#heartbeats).

### Flush

Expand Down Expand Up @@ -577,7 +583,8 @@ contract above and [Commit handling](../consensus/consensus-commit.md).
* Set `ResponseFinalizeBlock.propose_next_block_immediately` when the Application has block-driven
work pending that the next block would carry out, for example withdrawal transactions waiting to be
signed. The node then does not wait for transactions (`create-empty-blocks-interval`) before
proposing round 0 of the next height. This is a hint to the local node, not part of consensus: it is
proposing at the next height, in round 0 or, on consensus version 2, in a round entered after a
silent proposer's round was skipped. This is a hint to the local node, not part of consensus: it is
not compared between validators, it only matters on the node that proposes the next height, and it
is consumed once; a node that restarts in between falls back to the configured wait.
* Just as `ProcessProposal`, the implementation of `FinalizeBlock` MUST be deterministic, since it is
Expand Down
44 changes: 38 additions & 6 deletions spec/consensus/consensus-alternative-flows.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@ sequenceDiagram
V->>V: EnterPrecommit (no-op) → EnterNewRound(h,r+1) → EnterPropose(h,r+1)
```

A proposer that is silent while its round waits for transactions is handled
before the propose step on `CONSENSUS_VERSION_2` chains: validators that stop
hearing its heartbeats skip the round with a round skip certificate
([Proposer liveness and round skipping](./consensus-proposer-liveness.md)).

### Proposal arrives after the node prevoted

`Proposaler.Set` has no step check, so the proposal is still stored and parts
Expand Down Expand Up @@ -195,6 +200,11 @@ on another block prevotes nil for it; see
If the block is absent, signature verification can park the commit and advance
to its round before the later application check. See
[Commit handling](./consensus-commit.md#commit-for-a-later-round).
- **Round skip certificate** (`CONSENSUS_VERSION_2`). A verified certificate
for round `r' ≥ r` → `EnterNewRound(r' + 1)`, unless the node is at
`ApplyCommit`; one for `r' < r` is only kept and gossiped
(`AddRoundSkipCertificateAction`). See
[Proposer liveness and round skipping](./consensus-proposer-liveness.md).
- **Round bookkeeping.** `HeightVoteSet` keeps vote sets for rounds
`0 .. Round + 1` (`SetRound(round + 1)` in `EnterNewRoundAction`) plus at
most two "catch-up" rounds per validator pro-tx-hash
Expand Down Expand Up @@ -290,17 +300,25 @@ and `addVoteErrorMw` (`internal/consensus/state_add_vote.go`).

`EnterNewRoundAction.Execute` waits for transactions when
`WaitForTxs() = ¬CreateEmptyBlocks ∨ CreateEmptyBlocksInterval > 0`, at
round 0, above `InitialHeight`, unless `ProposeNextBlockImmediately`
round 0 or in a round entered on a round skip certificate that satisfies
`StateData.waitsAfterRoundSkip`
([The round after a skip](./consensus-proposer-liveness.md#the-round-after-a-skip)),
above `InitialHeight`, unless `ProposeNextBlockImmediately`
(set from `ResponseFinalizeBlock.propose_next_block_immediately` of the
previous height by `ApplyCommitAction.Execute`; consumed by the next
`updateToState`; not persisted, so a restart falls back to the configured
wait).

`State.handleTxsAvailable` acts only at round 0: at step `NewHeight` (and
not at `InitialHeight`) it schedules a `NewRound` tock for
`State.handleTxsAvailable` records the report in `TxsReported` (the mempool
reports once per height) and acts only at round 0 or in a round that may wait
after a round skip (`StateData.mayWaitAfterRoundSkip`): at step `NewHeight`
(and not at `InitialHeight`) it schedules a `NewRound` tock for
`StartTime − now + 1 ms`; at step `NewRound` it dispatches `EnterPropose`.
With `CreateEmptyBlocksInterval > 0` the interval tock calls `EnterPropose`
even without transactions (empty block). Rounds `≥ 1` never wait.
even without transactions (empty block); in a round after a skip the interval
counts from `StartTime`. Other rounds `≥ 1` never wait. On
`CONSENSUS_VERSION_2` chains a proposal or prevotes can also end the wait
([When it applies](./consensus-proposer-liveness.md#when-it-applies)).

## Validator-set and parameter changes

Expand All @@ -319,7 +337,8 @@ even without transactions (empty block). Rounds `≥ 1` never wait.
version) proposed in block `h` take effect at `h + 1`
(`LastHeightConsensusParamsChanged`).
- Proposer selection version is `ConsensusParams.Version.ConsensusVersion`
([Proposer selection](./proposer-selection.md)).
([Proposer selection](./proposer-selection.md)); version 2 also enables
[proposer liveness](./consensus-proposer-liveness.md).

## Core chain locks

Expand Down Expand Up @@ -356,7 +375,15 @@ even without transactions (empty block). Rounds `≥ 1` never wait.
`(h, r)` names the same block (`Proposaler.verifyProposalForNonValidatorSet`).
- **Signing failure** (`PrivValidator.SignVote`/`SignProposal` error or
deadline): logged; no vote/proposal and **no retry** within the step
(`voteSigner.signAddVote`, `Proposaler.Create`).
(`voteSigner.signAddVote`, `Proposaler.Create`). A failed round skip vote is
retried after another silence timeout.
- **Remote signers** (socket, gRPC) cannot sign proposer heartbeats or round
skip votes (`RemoteSignerError` 501, `types.ErrSigningNotSupported`): a
validator using one sends neither, so its waiting rounds are skipped
([Signers](./consensus-proposer-liveness.md#signers)).
- **Observers and liveness messages:** a node without the validators' key
shares neither verifies nor relays heartbeats and skip votes; it verifies
and applies round skip certificates against the quorum public key.
- **Block sync just finished**: proposing is suppressed until the node has
passed the handover target height or 10 minutes elapsed
(`catchupTracker.mayPropose`, `maxCatchupSuppression`); voting is not.
Expand Down Expand Up @@ -408,6 +435,11 @@ faults, unless stated). For operational monitoring, see
evicts a peer (`isPeerFloodableError`).
- **Block-part proof budget** (`block_part_budget.go`): per-peer allowance
for parts whose Merkle proof fails.
- **Liveness messages** (`CONSENSUS_VERSION_2`): heartbeats, round skip votes
and certificates pay extra state-channel tokens only when new, go through
the peer lanes and the verification budget, and are sent through a bounded
reactor outbox; heartbeats are paced per round
([Message admission and budgets](./consensus-proposer-liveness.md#message-admission-and-budgets)).
- **Bounded memory per height:** commit candidate slots (one per peer),
`commitExpectations`, `MaxVoteExtensions`, `HeightVoteSet` catch-up rounds,
`maxPeerMaj23s` claims per vote set.
Expand Down
9 changes: 7 additions & 2 deletions spec/consensus/consensus-happy-path.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,11 +97,16 @@ tock to `EnterNewRoundEvent{Height}`. `EnterNewRoundAction.Execute`
- `Votes.SetRound(1)` so that round 1 votes are tracked for round skipping;
- `TriggeredTimeoutPrecommit = false`;
- decides whether to wait for transactions:
`waitForTxs = config.WaitForTxs() && round == 0 && Height != InitialHeight`,
overridden to `false` by `ProposeNextBlockImmediately`. **⇢**
`waitForTxs = config.WaitForTxs() && (round == 0 || waitsAfterRoundSkip(round)) && Height != InitialHeight`,
overridden to `false` by `ProposeNextBlockImmediately` and, on
`CONSENSUS_VERSION_2` chains, by a proposal already held for the round
(`roundSkipping && Proposal != nil`). **⇢**
[Empty blocks](./consensus-alternative-flows.md#mempool-and-empty-blocks).
With the default configuration (`create-empty-blocks = true`,
interval 0) it does not wait and dispatches `EnterProposeEvent{h, 0}`.
On `CONSENSUS_VERSION_2` chains the proposer of a waiting round sends
heartbeats while it waits. **⇢**
[Proposer liveness](./consensus-proposer-liveness.md).

### 3. Propose

Expand Down
6 changes: 4 additions & 2 deletions spec/consensus/consensus-known-issues.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,10 @@ the last tick after it fires. When `EnterPropose` itself was triggered by a
else moves the node out of `NewRound` for that round. The node recovers only
when the other validators' nil precommits reach +2/3-any
(`addVoteDispatchPrecommitMw` → `EnterPrecommitWait` → `EnterNewRound(r + 1)`),
costing the network one round. Preconditions: `WaitForTxs()` true, round 0,
the node is the proposer, and its clock is behind the previous block time by
costing the network one round. Preconditions: `WaitForTxs()` true, round 0
(on `CONSENSUS_VERSION_2` chains also a round entered on a round skip
certificate that waits, whose empty-block interval schedules the same
`NewRound` tock), the node is the proposer, and its clock is behind the previous block time by
more than the time since that block was produced. (Inferred; not reproduced.)

## Precommit requires this round's proposal even when locked
Expand Down
Loading
Loading