Repository navigation
fix(consensus): validate a proposed block against state before ProcessProposal - #1616
Conversation
…sProposal ProcessProposal sent a proposed block to the application before checking it against the committed state; ValidateBlock ran only after the application answered. A block that does not continue the chain (for example a header with a past height, another chain ID or a different last block ID) therefore reached the application, and an application that returns an error instead of REJECT for such input stopped the consensus routine, also when the proposal was read back from the WAL on the next start. With verify, ProcessProposal now runs ValidateBlock first and returns ErrInvalidBlock without calling the application, so consensus prevotes nil. Checks that depend on the application's answer (AppHash, ResultsHash, next validators and params, chain lock) stay after the call. The pre-application validation is timed as its own process_proposal_prevalidate stage. ValidateBlock's cache is keyed on the block hash alone; a hit now counts only when the block still continues the given state (version, chain ID, height, last block ID). Also report expected and actual heights in the right order in the initial-height error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pre-merge checks:
|
|
✅ Final review complete — no blockers (commit 0302edb) · triage: critical · Phase 2 only (queue backlog) |
|
Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
Replay skipped the timeliness check, so a proposal that a node prevoted nil on when it received it reached ProcessProposal when the node replayed its WAL. Replay now dates a message by its WAL entry, the closest record of when it was received and never earlier, and runs the check: a proposal received too early, or a first-height block without the genesis time, gets a nil prevote again without an application call. Too late is not judged in replay, since the WAL entry can lag the receipt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y timeliness Align the prevote flowchart, the happy-path sequence diagram and failure paragraph, and the P-Timely rule with the current order: ValidateBlock runs against the committed state before the application call, and replay judges timeliness by the WAL entry time. Update the ValidateBlockWithRoundState comment on what a ValidateBlock cache hit re-checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…I error A block that failed ValidateBlock before ProcessProposal was logged as "ProcessProposal abci method: ..." although the application was never called. BlockExecutor.ProcessProposal now labels the pre-application failure "validate block before ProcessProposal" and the application call error "ProcessProposal abci method"; ensureProcess wraps both as "process proposal". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The replay tests checked only the round step. They now read the node's own prevote from its queue and require it to be for nil. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ative flows Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-proposal-before-app
…eight At the first height a proposal is judged by the genesis time alone, so the replay test there never reached the receive-time window. Commit a block first and replay a proposal whose WAL entry is too early for its timestamp (nil prevote, no application call) and one whose entry is timely (reaches the application). Keep the first-height case under a name that says what it checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ensureProcess already prefixes its errors with "process proposal", so the prevoter panic and the commit path no longer add their own prefix on top. A failed ProcessProposal call now panics as "process proposal: ProcessProposal abci method: ..." instead of "ProcessProposal: process proposal: ProcessProposal abci method: ...". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
thepastaclaw
left a comment
There was a problem hiding this comment.
Final validation — Phase 1 + Phase 2
Static verification of head 472a6ff found no actionable in-scope defects: committed-state validation now precedes ABCI execution, preserves response-dependent checks, and routes invalid proposals through existing nil-prevote handling. Current CI confirms passing builds, test shards, end-to-end tests, and lint; govulncheck fails on GO-2026-6859 in the unchanged Go 1.27.3 toolchain, not an infrastructure error or a regression from this change. Review and PR Hygiene checks remain pending, and the four intervening base commits neither overlap the changed files nor introduce reported merge conflicts.
Review provenance
Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 5: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)
- Triage:
normalbygpt-6.1-sol(effort low) — The diff changes consensus validation ordering and cache-hit checks in ProcessProposal and ValidateBlock, but the production changes are small and contained rather than large or intricate enough to qualify as critical. - Phase 1 reviewers:
muse-spark-1.3-contributor— general (completed, effort xhigh); agentphase1-reviewer - Phase 1 model:
muse-spark-1.3-contributor— not quota-gated; passed overgemini-3.8-flash-high(antigravity quota lookup failed),glm-5.3-flash(not used above high effort; tier asks max) - Second Phase-2 pass: on this first full review every Phase-2 reviewer also ran a second, independent time, and the verifier weighed both sets
- Fresh verifier:
gpt-6.1-sol— final-verifier; agentsol-verifier - Phase 2 reviewers:
gpt-6.1-sol— general (completed, effort high); agentphase2-reviewer,gpt-6.1-sol— tenderdash-consensus-security (completed, effort high); agentphase2-reviewer,gpt-6.1-sol— general (completed, effort high); agentphase2-reviewer,gpt-6.1-sol— tenderdash-consensus-security (completed, effort high); agentphase2-reviewer
Out-of-scope follow-up suggestions (1)
These are valid observations, but they are outside this PR's scope and should be handled in separate issues or author/maintainer-requested PRs rather than blocking this review.
- Update the Go toolchain for the HTTP server vulnerability reported by CI — The govulncheck job reports GO-2026-6859, an HTTP/1 server crash caused by a MaxBytesReader header race, affecting Go 1.27.3 and fixed in Go 1.27.4. The unchanged workflow pins Go 1.27.3, and rpc/jsonrpc/server/http_server.go directly uses http.MaxBytesReader, making this an important availability follow-up for exposed HTTP services. Exploitability was not independently reproduced.
- Follow-up: Open a separate toolchain-update PR adopting Go 1.27.4 or a later patched release across applicable CI and production build configurations, then rerun govulncheck and the RPC tests.
…-proposal-before-app
TestProcessProposalValidatesChainLockHeightBeforeApp states the intended behaviour and fails on 0523cf0: with verify, a block whose header core chain locked height does not follow the committed state reaches the application, and its failure is returned instead of ErrInvalidBlock. The check (validateCoreChainLock) needs nothing from the application; only the prevote path runs it first, through ValidateBlockChainLock. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WunYYgi2uMsQDcBcKAwLdt
validateCoreChainLock needs only the block and the committed state, yet it ran after the application call, in ValidateBlockWithRoundState. Run it in validateBlock instead, so ProcessProposal with verify and the block sync applier refuse such a block before the application sees it. The header hash does not cover the block's chain lock, so a ValidateBlock cache hit re-checks it as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The WAL time bounds the receipt from above, so replay refuses only proposals the node refused when it received them. It also dates a block made valid during replay, which decides whether the node re-proposes it after a restart, and feeds the proposal timestamp metric. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
thepastaclaw
left a comment
There was a problem hiding this comment.
Re-review — Final validation — Phase 1 + Phase 2
At f304093, the new pre-application validation fixes the direct chain-position checks, but a header-cache hit still lets an invalid LastCommit reach ABCI; the recovery specification also contradicts the new replay behavior. Build and e2e checks passed; tests (04)/(05), go vet, golangci-lint, govulncheck and Super linter failed during dependency installation with Ubuntu DNS errors, while tests (00)–(03) were pending when checked. This verification was static; no local builds or tests were run.
🔴 1 blocking
Follow-up candidates (optional) (1)
New non-blocking observations on code unchanged since the last full review. No inline threads are opened for them and later reviews do not repeat them: fix forward or track separately if they are worth it.
🟡 Suggestion: Update the recovery specification's replay timeliness rule
spec/consensus/consensus-recovery.md:71-73
The recovery specification still says replay skips timeliness checks, but isValidForPrevote now calls proposalIsTimely during replay. For proposals not already represented by ValidBlock, replay enforces the initial-height genesis timestamp and rejects above-initial-height proposals whose WAL timestamp is too early; only the too-late verdict is exempted. Update this bullet to describe those rules and clarify that the WAL timestamp approximates, rather than preserves, the original receive time.
source: gpt-6.1-sol (phase2-reviewer: general)
Review provenance
Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)
- Triage:
criticalbygpt-6.1-sol(effort low) — The change is intricate and alters consensus-critical behavior in BlockExecutor.ProcessProposal and ValidateBlock, including state-dependent cache validation, plus WAL replay timeliness checks in internal/consensus/state_data.go that affect proposal voting after restart. - Phase 1 reviewers:
muse-spark-1.3-contributor— general (completed, effort xhigh); agentphase1-reviewer - Phase 1 model:
muse-spark-1.3-contributor— not quota-gated; passed overgemini-3.8-flash-high(antigravity quota lookup failed),glm-5.3-flash(not used above high effort; tier asks max) - Single stage: Phase 1 and Phase 2 reviewed this head side by side, with no blocker gate between them (triage tier)
- Code unchanged since the last final review (
472a6ff9): 1 new suggestion(s) listed as follow-up candidates, not posted inline - Fresh verifier:
gpt-6.1-sol— final-verifier; agentsol-verifier - Phase 2 reviewers:
gpt-6.1-sol— general (completed, effort xhigh); agentphase2-reviewer,gpt-6.1-sol— tenderdash-consensus-security (completed, effort xhigh); agentphase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions don't block the merge, but most are worth fixing in this PR; judge each on its merits.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.
In `internal/state/execution.go`:
- [BLOCKING] internal/state/execution.go:464-466: Verify the actual LastCommit before accepting a pre-application cache hit
This PR now relies on ValidateBlock as the pre-ABCI gate, but its cache does not authenticate all of LastCommit. Commit.Hash() hashes only ThresholdBlockSignature. After a legitimate block above InitialHeight populates the cache, a later-round proposer can retain its header and signature while replacing LastCommit.QuorumHash, rebuild the parts, and sign the resulting proposal. BlockFromProto accepts this change because Commit.ValidateBasic does not validate the quorum hash, and both cache guards still pass. The changed part-set header prevents processedFor from reusing the previous application result; POLRound >= 0 bypasses the original-block-proposer check, and ensureProcess runs before the prevoter checks the POL. Consequently, buildLastCommitInfo sends the invalid quorum to ABCI. The explicit verifyLastCommit in ValidateBlockWithRoundState runs only after a successful application response, so an application error on that invalid CommitInfo still reaches the fatal prevoter.handleError path this PR is intended to prevent. Verify LastCommit before accepting the cache hit, or bind cache reuse to its complete authenticated content and verification parameters. Add a cached-then-altered-LastCommit regression asserting ErrInvalidBlock without an application call.
…nged The header hash covers the LastCommit only by its threshold signature and does not cover the chain lock. A block validated once and then presented with a different LastCommit quorum hash, round or vote extensions, or a different chain lock, must be refused as invalid without an application call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cache was keyed on the block hash, which covers the LastCommit only by its threshold signature and leaves out the chain lock, so a block that differed there from a validated one passed ValidateBlock unchecked and reached ProcessProposal. Key it on the block hash plus a checksum of the LastCommit and the chain lock. The chain lock no longer needs re-checking on a hit, and an unchanged block still hits, so the honest path performs no extra commit verification. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ocked height changed The header hash omits CoreChainLockedHeight, so the validation cache key does not cover it. Adds a RED case for that, and a chain lock case that is RED against the pre-fix cache so it pins the chain lock part of the key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WunYYgi2uMsQDcBcKAwLdt
Header.Hash() leaves out CoreChainLockedHeight, so a block differing from a validated one only there kept the same key and passed ValidateBlock unchecked. Checksum the full header proto along with the LastCommit and the chain lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
thepastaclaw
left a comment
There was a problem hiding this comment.
Re-review — Final validation — Phase 2 only (queue backlog)
The complete diff at the exact review head addresses the prior LastCommit cache-bypass issue by binding cache reuse to the full header, LastCommit, and chain-lock content while retaining state-position checks. Static review found no remaining in-scope defects; supplied build, test, lint, vet, vulnerability, and e2e checks passed, with application, ABCI CLI, and link checks still queued.
🔴 0 blocking | 🟡 0 suggestion(s) | 💬 0 nitpick(s)
Review provenance
Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)
- Triage:
criticalbygpt-6.1-sol(effort low) — The diff makes intricate, interdependent changes to consensus-critical proposal acceptance in internal/state/execution.go (ProcessProposal and ValidateBlock caching) and replay-time voting behavior in internal/consensus/state_data.go. - Phase 1 reviewers: not run (skipped for throughput: 20 PRs queued, above the 10 limit)
- Fresh final gate: an independent Phase-2 review ran after iterative findings were reconciled
- Fresh verifier:
gpt-6.1-sol— final-verifier; agentsol-verifier - Phase 2 reviewers:
gpt-6.1-sol— general (completed, effort xhigh); agentphase2-reviewer,gpt-6.1-sol— tenderdash-consensus-security (completed, effort xhigh); agentphase2-reviewer,gpt-6.1-sol— general (completed, effort xhigh); agentphase2-reviewer,gpt-6.1-sol— tenderdash-consensus-security (completed, effort xhigh); agentphase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify the current code and confirm that no unresolved issues remain.
No unresolved findings remain from the prior review on this head.
|
Ready for review — files with no dedicated owner: lklimek or shumkov. |
|
Policy satisfied — this can merge. |
TL;DR: A node now checks that a proposed block continues its own chain before handing it to the application, so a block that does not fit is simply voted down instead of stopping the node.
User story
As a node operator, I want my node to turn down a proposed block that does not continue the chain without involving the application, to keep the node running and voting normally.
Scenario
Base flow
A validator receives a proposed block for the current height and asks the application whether it accepts it before voting.
Actual behavior
The node passes the block to the application first and only afterwards checks that the block belongs at this point of the chain (height, network, previous block). If the application answers such a block with an error instead of a refusal, the node stops. Because the proposal is kept in the node's recovery log, the node stops again each time it is started, until the log is removed by hand.
Expected behavior
A block that does not continue the chain is refused by the node itself, before the application sees it. The node votes against it and keeps running, also when it reads the proposal back from its recovery log after a restart.
Detailed discussion
What was done
BlockExecutor.ProcessProposal(internal/state/execution.go): withverify, runValidateBlockagainst the committed state before calling the application and returnErrInvalidBlockon failure. Consensus already turnsErrInvalidBlockinto a nil prevote (prevoter.handleError). The check includes the block's core chain lock, which depends only on the committed state. Checks that depend on the application's answer (AppHash, ResultsHash, NextValidatorsHash, NextConsensusHash) stay after the call inValidateBlockWithRoundState.ValidateBlockcache (internal/state/execution.go): the cache was keyed on the block hash alone, which covers neither theLastCommitfields sent to the application (round, quorum hash, vote extensions;Commit.Hash()covers only the threshold signature) nor the chain lock, andBlockExecutor.Copy()shares the map. The key is now the block hash plus a checksum of the full header (the header hash omits the core chain locked height),LastCommitand chain lock, and a hit counts only if the newvalidateBlockPosition(version, chain ID, height,LastBlockID; moved unchanged out ofvalidateBlock) still passes against the current state.internal/consensus/replay.go,state_data.go): a replayed proposal's receive time is the time its WAL entry was written, and replay checks proposal timeliness against it instead of skipping the check. A proposal received too early is prevoted nil in replay as it was live; a "too late" verdict is still accepted in replay, because the WAL entry is written after the message is received. The same time becomes the replayed valid block's receive time, used when deciding whether to propose it again after a restart.process proposal: validate block before ProcessProposal: …, an application errorprocess proposal: ProcessProposal abci method: …, without repeated prefixes.process_proposal_prevalidatestage ofBlockApplyStageDuration; the existingprocess_proposal_validatestage keeps its meaning.spec/abci/methods.md,spec/consensus/consensus-happy-path.mdandspec/consensus/consensus-transitions.mddescribe the new order.Other callers, traced: block sync already runs
ValidateBlockinverify(), so the new call is a cache hit;ApplyBlockand the replayer handshake useverify=falseand are unchanged; the commit path (verifyProcessedCommit) andmustEnsureProcessbehave as before for valid blocks; the proposer's own block passes. The number of BLS verifications on the firstProcessProposalis unchanged (two).Testing
Tests written first and confirmed failing before the change:
internal/state/execution_test.goTestProcessProposalValidatesBlockBeforeApp: wrong height, chain ID,LastBlockID, and a cached block at the next height; each expectsErrInvalidBlockand noProcessProposalcall to the application.internal/consensus/proposal_state_validation_test.go:TestStatePrevotesNilOnBlockNotFollowingState(live state machine: nil prevote, no application call) andTestReplayOfBlockNotFollowingStatePrevotesNil(replayed messages: node keeps running, no application call), each for height, chain ID andLastBlockID.TestReplayJudgesProposalTimelinessByWALTime(above the first height): a proposal whose WAL entry predates its timestamp is prevoted nil without an application call; one written in time reaches the application.TestReplayOfFirstBlockWithoutGenesisTimePrevotesNilcovers the first-height rule.TestDoPanicsOnProcessProposalFailurepins the panic text.TestProcessProposalRevalidatesUncoveredPartsOfCachedBlock: a cached block whoseLastCommitquorum hash, round or vote extensions, header core chain locked height, or chain lock were changed is refused before the application call.TestProcessProposalValidatesChainLockHeightBeforeApp: a block whose chain lock does not match the state is refused before the application call.Passing locally with
-tags=deadlock: the new tests,./internal/state/,./internal/blocksync/, and the consensus subsetTestHandshake|TestWALCrash|TestSimulateValidatorsChange|TestStateFullRound|TestStateBadProposal|TestStateLock|TestStateProposal|TestEnsureProcess|TestPrevot|TestReplay|TestBlockExecutor|TestVerifyProcessedCommit|TestStateOversizedBlock. golangci-lint on changed packages: 0 issues. Full suite left to CI.Breaking changes
None. Blocks that previously reached the application and were then found invalid now get the same nil prevote without the application call.
Attribution
🤖 Co-authored by Claudius the Magnificent AI Agent
🤖 Generated with Claude Code
PR Hygiene ·
0302edbinternal/consensus/block_executor.go,internal/consensus/proposal_state_validation_test.go,internal/consensus/replay.goand 14 more) — approved by lklimekWhen every merge requirement is met, the
PR Hygienecheck passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.