Skip to content

fix(statesync): restore LastAppHash of the snapshot height, not the next one - #1584

Open
QuantumExplorer wants to merge 2 commits into
devfrom
fix/statesync-last-app-hash
Open

QuantumExplorer wants to merge 2 commits into
devfrom
fix/statesync-last-app-hash

Conversation

@QuantumExplorer

@QuantumExplorer QuantumExplorer commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Basic explanation

State sync and tenderdash rollback both build a node state at some height H. That state's LastAppHash must be the app hash after block H, because that is what the application reports when the node restarts. Both took it from the header of block H+1 instead.

The habit comes from Tendermint, which executed each block one height late, so header H+1 carried the result of block H. Tenderdash executes a block in its own height: header H already carries the app hash after block H, and header H+1 carries the one after block H+1. LastResultsHash, on the next line, had the same off-by-one.

Before, for a state restored or rolled back to height H:

state.LastBlockHeight = H
state.LastAppHash     = header(H+1).AppHash      app state after block H+1
state.LastResultsHash = header(H+1).ResultsHash
app reports             header(H).AppHash        app state after block H

After:

state.LastBlockHeight = H
state.LastAppHash     = header(H).AppHash        app state after block H
state.LastResultsHash = header(H).ResultsHash
app reports             header(H).AppHash

The wrong value rarely shows, because committing block H+1 overwrites both hashes. It matters when the node restarts at exactly height H, before it commits the next block. The handshake then compares the app's hash with state.LastAppHash, and the node stops with state.AppHash does not match AppHash after replay. That can happen after a state sync, and it always happens after tenderdash rollback --store, which also rolls the block store back to H.

TL;DR: State sync and rollback now set LastAppHash and LastResultsHash from the header at the state's own height, so a node restarted right after either one passes the handshake.

User story

As a masternode operator, I want a node I just state-synced or rolled back to start again after a restart, so that I don't have to wipe its data and sync it again.

Scenario

Base flow

A node restores a snapshot at height H with state sync, or an operator runs tenderdash rollback --store to go back to height H. The node restarts before it commits block H+1.

Actual behavior

The handshake fails. The app reports the app hash after block H, the state holds the one after block H+1, and the node does not start.

Expected behavior

The state holds the app hash after block H, the handshake passes, and the node starts at height H.

Detailed discussion

What was done

  • internal/statesync/stateprovider.go: stateProviderP2P.State and stateProviderRPC.State take LastAppHash and LastResultsHash from the light block at the snapshot height (lastLightBlock) instead of the next one. AppHash(H) already returned header H's app hash; its outdated comment about heights h+1 and h+2 is corrected.
  • internal/state/rollback.go: Rollback takes both hashes from the header at the height it rolls back to. It no longer loads the latest block, which it only read for those hashes.

Block contents do not change: the proposer always replaces the header hashes that MakeBlock copies from the state with the ones PrepareProposal returns, and block validation compares headers with the ProcessProposal result, not with state.LastAppHash.

Testing

Targeted tests only. Each one below fails before the fix and passes after it.

  • TestReactor_StateProviderP2P and TestReactor_Sync (internal/statesync): the mock light blocks had an all-zero app hash at every height, which hid the off-by-one. mockLB now lets factory.MakeHeader pick a random app hash per height. Both tests check LastAppHash and LastResultsHash against the snapshot block's header. The mock app in TestReactor_Sync now reports the snapshot height's app hash. It used to report the next height's, which only passed because every hash was zero.
  • TestReactor_StateProviderRPC (new): the RPC provider's State had no test. Its light client gets light blocks from peers, and a small JSON-RPC server answers consensus_params.
  • TestRollback (internal/state): each block header now carries its own block's hashes. The test used to put the rolled back state's app hash in the next block's header.
  • TestRollbackIntegration (cmd/tenderdash/commands): a second case also rolls back the block store, then restarts the node. Before the fix it stops with state.AppHash does not match AppHash after replay. The single validator then refuses to sign heights it already signed, so this case checks the height and app hash the node starts at instead of waiting for new blocks.

I also ran the other statesync tests that use the mock light blocks (TestReactor_*, TestBlockQueue*, backfill and dispatcher tests), ran the provider and rollback tests with -race, and ran golangci-lint on the changed packages (no new issues).

Breaking changes

None. The wrong values only lasted until the node committed its next block.

Prior work

Refs dashpay/platform#4682

🤖 Generated with Claude Code

PR Hygiene · 23c1443

  • Bots — thepastaclaw ✓
  • Self-review — post /self-reviewed
  • Build green
  • Approvals
    • files with no dedicated owner (cmd/tenderdash/commands/rollback_test.go, internal/state/rollback.go, internal/state/rollback_test.go and 2 more) — approved by lklimek

When every merge requirement is met, the PR Hygiene check passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.

QuantumExplorer and others added 2 commits November 4, 2026 02:33
…ext one

State sync and Rollback took LastAppHash and LastResultsHash from the header
one height above the state they build. That comes from Tendermint's next-block
execution, where header H+1 carried the results of block H. Tenderdash executes
same-block: header H already carries them, and header H+1 holds the app state
after block H+1.

Nothing reads LastAppHash before the next block commits, except the handshake
after a restart at exactly the restored height, which then fails with
"state.AppHash does not match AppHash after replay".

Both state providers and Rollback now take the hashes from the header at the
state's height. Rollback no longer loads the block it rolls back. The mock
light blocks in the statesync tests get a random app hash per height, so the
tests see a hash taken from the wrong height. A new test covers the RPC state
provider's State.

Refs dashpay/platform#4682

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With `tenderdash rollback --store`, the restarted node finds the app, the
state and the block store at the same height, so the handshake compares the
app's app hash with the rolled back state's LastAppHash. Before the previous
commit it stopped with "state.AppHash does not match AppHash after replay".

The only validator already signed the next heights and refuses to sign them
again, so this case checks the height and app hash the node starts at instead
of waiting for new blocks.

Refs dashpay/platform#4682

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

@thepastaclaw

thepastaclaw commented Nov 3, 2026 •

Copy link
Copy Markdown

✅ Final review complete — no blockers (commit 23c1443) · triage: low

@QuantumExplorer
QuantumExplorer marked this pull request as ready for review November 3, 2026 20:07
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final validation — Phase 1 + Phase 2

Verified the changes at 23c1443: both state providers and rollback now restore execution hashes from the state's own height, consistent with same-block execution and restart handshake checks. The regression tests cover both provider implementations and rollback with and without block-store removal; the sole proposed documentation nitpick is not actionable because the height mapping remains correct. Validation was static only; current CI shows all build, test, lint, and vet checks passing, with only PR Hygiene pending.

Review provenance

Source: reviewer 1: glm-5.3-flash (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: low by gpt-6.1-sol (effort low) — The production diff is a small, contained correction to which block header supplies restored-state hashes, with targeted regression tests, rather than a large or intricate change to consensus rules.
  • Phase 1 reviewers: glm-5.3-flash — general (completed, effort high); agent phase1-reviewer
  • Phase 1 model: glm-5.3-flash — zai quota: 5h 89% left, weekly 76% left; passed over gemini-3.8-flash-high (antigravity quota lookup failed)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort medium); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort medium); agent phase2-reviewer

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Nov 3, 2026
@github-actions github-actions Bot removed the waiting-self-review Waiting for the author to post /self-reviewed label Nov 5, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Bots are done — your move: post /self-reviewed.
Full checklist in the description.

@github-actions github-actions Bot added the waiting-self-review Waiting for the author to post /self-reviewed label Nov 6, 2026
@lklimek lklimek added this to the v3.0 milestone Nov 6, 2026
@lklimek
lklimek enabled auto-merge (squash) November 12, 2026 07:47

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-self-review Waiting for the author to post /self-reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants