Skip to content

fix(blocksync): serve blocks as the bytes they were committed with - #1595

Merged
lklimek merged 8 commits into
devfrom
fix/blocksync-serve-original-parts
Nov 12, 2026
Merged

lklimek merged 8 commits into
devfrom
fix/blocksync-serve-original-parts

Conversation

@QuantumExplorer

@QuantumExplorer QuantumExplorer commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Basic explanation

A block is gossiped and committed as parts: the proposer serializes it, cuts the bytes into 64kB parts, and the commit's block ID carries the Merkle root over those parts (the part set header). The block store keeps those exact parts.

Block sync did not serve them. It loaded the block, decoded it, and sent a re-encoding. The syncing node then cut that re-encoding into parts and checked the result against the commit. When the committed bytes are not what a re-encoding gives (an unknown field, or a non-canonical encoding, both of which decoding throws away), the part set headers differ, and the commit check fails with invalid commit -- wrong block ID. The syncing node drops the peer and stalls at that height, because every honest peer serves the same re-encoding.

Before:

serving node   LoadBlock(h) -> decode -> re-encode -> BlockResponse.block
syncing node   decode -> re-encode -> parts of 64kB -> part set header != commit's

After:

serving node   stored parts of h -> their bytes -> BlockResponse.block
syncing node   those bytes -> parts of 64kB -> part set header == commit's

TL;DR: Block sync now serves a block as the bytes it was committed with, and the syncing node cuts those bytes into 64kB parts and computes the part set root itself, as the proposer did. A block whose committed bytes are not what a re-encoding gives now syncs.

User story

As a node operator, I want a node to block-sync every committed block from honest peers, so that a block whose bytes a proposer encoded oddly cannot stop new or lagging nodes from catching up.

Scenario

Base flow

A proposer proposes a block whose bytes carry an unknown field. Validators accept the parts, since they verify against the proposal's part set header, and commit the block. Later a node syncs that height with block sync.

Actual behavior

The serving peer sends a re-encoding of the block. The syncing node computes a different part set header, VerifyCommit rejects the commit with invalid commit -- wrong block ID, and the node drops the peer and retries forever. Every honest peer sends the same re-encoding.

Expected behavior

The syncing node gets the committed bytes, computes the committed part set header from them, verifies the commit and applies the block. It stores the same parts as the serving node.

Detailed discussion

What was done

  • BlockResponse.block (proto field 1) is now bytes: the block's committed bytes, the stored parts concatenated. A length-delimited bytes field and an embedded message are the same on the wire, so this is wire-compatible with older releases (see Breaking changes).
  • Serving (prepareBlockResponse): reads the block meta and stored parts with LoadBlockMeta/LoadBlockPart (loadBlockBytes) instead of LoadBlock, so the block is no longer decoded and re-encoded on the serving side. A header stored without its block (state sync) still gets a NoBlockResponse.
  • Receiving (BlockResponseFromProto): decodes the block from those bytes, so a block that does not decode is refused before it is queued, then keeps only the block's height and the bytes. A pending response so holds one copy of its block, the one the backlog charges it for. A block buffer left oversized by a repeated block field is copied into a right-sized allocation before it is kept.
  • Applying (blockApplier.ApplyBytes): decodes the block again, under the applier lock and timed as the new decode stage, and cuts the bytes into 64kB parts with types.NewPartSetFromData. That part set is what VerifyCommit checks and what SaveBlock stores. The commit check is unchanged.
  • ABCI: RequestFinalizeBlock.block_id and the replay events now use the commit's block ID instead of block.BlockID(nil). That call re-encoded the block, so for these blocks it gave the application an ID that did not match the commit. This also corrects the same mismatch on the consensus path.
  • Spec: spec/p2p/messages/block-sync.md documents the fields and how the receiver rebuilds the part set.

Only the standard 64kB cut is rebuilt. An honest proposer always cuts a block that way (block.MakePartSet(types.BlockPartSizeBytes) in createProposalBlock), and lklimek notes this has held since the mainnet launch on 1.1.0, so this covers the whole history. Consensus still accepts parts cut to other sizes, and proofs whose total differs from the header's. A block committed that way is served, but the syncing node's part set header will not match the commit, and it rejects the block. As lklimek suggested in review, the place to close that gap is consensus, by rejecting non-standard cuts behind a consensus version, not block sync. This PR does not change consensus validity.

Compatibility between versions:

Sender Receiver Result
new new Every block committed with the standard cut syncs, including blocks whose bytes a re-encoding does not reproduce.
new old Old receivers read field 1 as an embedded Block and re-encode it, so they behave as today.
old new Old senders send a re-encoding, which the new receiver treats like today's code does.

The apply path decodes the block instead of serializing it, as it did before, and the partset stage now measures only the Merkle tree. The backlog holds no more than before: a pending response holds its block only as bytes, which cost less than the decoded block it held until now.

Relation to #1582: that PR's types.UnmarshalBlockProto drops LastCommit fields 7/8 when they have a non-bytes wire type. Before this change, block sync received the block already decoded inside the p2p envelope, so a block-sync node could neither apply that fallback nor serve such a block's original bytes. With this change the receiver decodes the raw bytes itself (BlockResponseFromProto), and that is where #1582 should call UnmarshalBlockProto after a rebase.

Testing

  • TestReactor_SyncBlocksAsCommitted (regression, internal/blocksync/reactor_test.go): a node commits the block at height 3 with an unknown field appended to its bytes. A second node block-syncs from it over the p2p test network, and must get past height 3 and store the same block ID and bytes. With the applier reverted to cutting a re-encoding, it stalls at height 3 with invalid commit -- wrong block ID, as origin/dev does.
  • TestServeBlockAsCommitted: with a real block store, a standard block, an unknown-field block and a block cut into 100-byte parts are each served as their stored bytes. The standard cut of the bytes rebuilds the stored part set for the first two, and not for the 100-byte cut, which is left to consensus. A header saved without its block gets NoBlockResponse.
  • TestBlockResponseWireCompatible: the new BlockResponse encodes byte for byte like the old one, with field 1 as an embedded Block, and the old encoding decodes into the new type.
  • TestBlockResponseDoesNotRetainOverwrittenBlockField: a repeated block field does not leave its larger allocation retained.
  • TestBlockResponseRetainsOnlyItsBytes: 16 decoded responses of a block of small transactions retain about 10 KB beyond their 17 MB of bytes. With the decoded block kept as well, they retain 23 MB more.
  • TestFinalizeBlockGetsCommittedBlockID: the application gets the commit's block ID when applying and when replaying the unknown-field block. It fails if execBlock goes back to block.BlockID(nil).
  • Ran with CGO/BLS, -tags deadlock:
    • go test -race ./internal/blocksync/
    • go test ./types/ ./proto/... ./internal/store/ ./internal/p2p/client/
    • golangci-lint run --new-from-rev=fece5eed4 on internal/blocksync, internal/state and internal/consensus
  • buf generate changed only proto/tendermint/blocksync/types.pb.go.

Breaking changes

None on the wire: older and newer nodes exchange block sync messages as before.

For Go code, bcproto.BlockResponse.Block changes from *tmproto.Block to []byte. Code that builds or reads that message must now marshal or unmarshal the block itself. In this repository, that is only block sync and its tests. make proto-check-breaking (buf FILE rules) reports the type change as FIELD_SAME_TYPE, although bytes and an embedded message share the wire format. CI does not run buf breaking.

Prior work

Attribution

Investigated and prepared with Claude Code.

🤖 Generated with Claude Code

PR Hygiene · abfa42b

  • Bots — thepastaclaw ✓
  • Self-review — post /self-reviewed
  • Build green
  • Approvals
    • files with no dedicated owner (docs/nodes/metrics.md, internal/blocksync/applier.go, internal/blocksync/applier_test.go and 14 more) — approved by lklimek

When every merge requirement is met, the PR Hygiene check passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.

Block sync served LoadBlock(h).ToProto(), a re-encoding of the decoded
block, and the syncing node rebuilt the part set from that re-encoding.
When the committed bytes carry an unknown field or a non-canonical
encoding, or the proposer cut them into parts shorter than 64kB, the
rebuilt part set header differs from the one in the commit's block ID.
The commit check then fails with "wrong block ID", so no honest peer can
serve the block.

BlockResponse.block is now bytes, the stored parts concatenated, which
is the same wire format as the embedded Block it replaces. A new
part_sizes field carries the part sizes when they are not the standard
64kB cut. The syncing node decodes the block from those bytes and builds
the part set it verifies and stores from them, with the new
types.NewPartSetFromPartSizes for a non-standard cut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@thepastaclaw

thepastaclaw commented Nov 5, 2026 •

Copy link
Copy Markdown

✅ Final review complete — no blockers (commit abfa42b) · triage: critical

@lklimek lklimek added this to the v2.0 milestone Nov 6, 2026
@QuantumExplorer
QuantumExplorer marked this pull request as ready for review November 9, 2026 11:40
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix; thepastaclaw left review threads unresolved; resolve them.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed bot-review-skipped A required review bot did not report; it was skipped by the window or by a person. labels Nov 9, 2026
@QuantumExplorer

Copy link
Copy Markdown
Member Author

Reviewed the concept, seems correct, but needs a second opinion. @lklimek

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final validation — Phase 1 + Phase 2

At head cf9ef92, the raw-byte transport and explicit part sizes address the reported re-encoding and partitioning failures, but four blocking gaps remain in admission bounds, ABCI block-ID propagation, proof preservation, and transport capacity. This was a static review; current build, test, and lint checks pass, while the review and PR Hygiene checks remain pending. The five base-branch commits do not overlap or supersede this implementation, and the supplied merge evidence reports no conflicts.

🔴 4 blocking

1 finding(s) not shown inline (the lines are not part of this PR's diff)

🔴 Blocking: Preserve the authenticated block ID through ABCI finalization
internal/blocksync/applier.go:101-105

The reconstructed block ID now reaches VerifyCommit, SaveBlock, and BlockExecutor.FinalizeBlock, but FinalizeBlock ultimately calls execBlock in internal/state/execution.go, which computes block.BlockID(nil) at line 838. That re-encodes the decoded block and rebuilds standard parts, so RequestFinalizeBlock.BlockID differs from commit.BlockID for the representations this PR fixes. Applications that verify the supplied commit against that ID, as kvstore.WithCommitVerification does through RequestFinalizeBlock.ToCanonicalVote, reject the valid commit; Apply then panics after saving the block. The new reactor regression uses BaseApplication and does not check the ABCI request ID. Update the owning state-execution path to use the authenticated commit ID or the supplied blockID, and assert the request carries the committed ID for both unusual representations and replay. Deferring this missed sibling requires a linked tracking issue or follow-up PR.

source: gpt-6.1-sol (phase2-reviewer: general, tenderdash-consensus-security)

Review provenance

Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); reviewer 3: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: critical by gpt-6.1-sol (effort low) — The change is intricate and directly changes peer-facing network deserialization in internal/blocksync/synchronizer.go:BlockResponseFromProto and protobuf BlockResponse decoding, alongside reconstruction and validation of peer-supplied part boundaries in types/part_set.go:NewPartSetFromPartSizes.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Single stage: Phase 1 and Phase 2 reviewed this head side by side, with no blocker gate between them (triage tier)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort xhigh); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.

In `internal/blocksync/synchronizer.go`:
- [BLOCKING] internal/blocksync/synchronizer.go:832-838: Bound part-size metadata before retaining pending responses
  PartSizes is retained directly from the peer, BlockResponse.Validate does not inspect it, and backlog() charges only Size, which is len(resp.Block). A serving peer can withhold the next required height while answering higher requests with small, structurally valid blocks and commits plus enormous packed lists of zero sizes. An 8 MiB list decodes into a 32 MiB uint32 slice, yet that allocation is excluded from the 256 MiB backlog budget. The count check in NewPartSetFromPartSizes runs only during ordered application, so it cannot reject those higher responses while the lower height is missing. Reject lists exceeding MaxBlockPartsCount during response conversion or fetch-time validation, and validate individual sizes and byte coverage before queueing. Add an out-of-order-response regression with a missing lower height and oversized metadata on a later response.

In `internal/blocksync/applier.go`:
- [BLOCKING] internal/blocksync/applier.go:101-105: Preserve the authenticated block ID through ABCI finalization
  The reconstructed block ID now reaches VerifyCommit, SaveBlock, and BlockExecutor.FinalizeBlock, but FinalizeBlock ultimately calls execBlock in internal/state/execution.go, which computes block.BlockID(nil) at line 838. That re-encodes the decoded block and rebuilds standard parts, so RequestFinalizeBlock.BlockID differs from commit.BlockID for the representations this PR fixes. Applications that verify the supplied commit against that ID, as kvstore.WithCommitVerification does through RequestFinalizeBlock.ToCanonicalVote, reject the valid commit; Apply then panics after saving the block. The new reactor regression uses BaseApplication and does not check the ABCI request ID. Update the owning state-execution path to use the authenticated commit ID or the supplied blockID, and assert the request carries the committed ID for both unusual representations and replay. Deferring this missed sibling requires a linked tracking issue or follow-up PR.
- [BLOCKING] internal/blocksync/applier.go:226-230: Preserve accepted part proofs that cuts alone cannot reconstruct
  Current consensus accepts part sets whose root cannot be regenerated from their collected bytes and sizes. PartSet.AddPart verifies inclusion against the advertised root but does not require part.Proof.Total to equal the part-set header's Total. For a block fitting in one part, a proposer can advertise Total=1 with the Merkle root of [encodedBlock, extraLeaf] and send part 0 with its genuine two-leaf proof. The part passes validation, completes the receiving set, and is decoded and voted on using the advertised header. SaveBlock preserves that proof, but loadCommittedBlock discards it, and this helper generates a one-leaf root instead. VerifyCommit therefore still rejects that committed height from every honest serving peer. Preserve enough stored proof information to reconstruct this accepted representation and add a regression. This falls within the stated goal of syncing every committed block; a future consensus rejection alone cannot repair existing heights, and deferral requires a linked tracking issue or follow-up PR.

In `internal/blocksync/p2p_msg_handler.go`:
- [BLOCKING] internal/blocksync/p2p_msg_handler.go:171-175: Include required part-size metadata in the transport capacity
  The response now includes PartSizes, but BlockSyncChannel.RecvMessageCapacity remains MaxBlockSizeBytes + 5 in internal/p2p/channel_params.go. With the permitted 100 MiB block limit, consensus accepts MaxBlockSizeBytes - 4096 raw bytes partitioned into 1,601 legal nonstandard parts: two 32,768-byte parts, 1,598 65,536-byte parts, and one 61,440-byte part. The packed size field adds 4,806 bytes, so this response exceeds the receive capacity even before adding the commit and envelope overhead. Unknown-field padding can produce such a valid raw block without changing its decoded contents. recvPacketMsg rejects every honest peer's response before commit verification, leaving this committed height unsyncable despite the new reconstruction logic. Increase the channel capacity to cover bounded part-size metadata and the remaining response/envelope overhead, and add a boundary regression that exercises the real transport limit.

Comment thread internal/blocksync/synchronizer.go Resolved
Comment thread internal/blocksync/applier.go Resolved
Comment thread internal/blocksync/p2p_msg_handler.go Resolved
@github-actions github-actions Bot removed the bot-review-skipped A required review bot did not report; it was skipped by the window or by a person. label Nov 9, 2026
Follow-up to review of the committed-parts change:

- PartSet.AddPart checks a proof against the root but not the header's
  total, so a part set can be committed whose root cannot be recomputed
  from its parts. BlockResponse now carries the stored proofs in that
  case, and the receiver verifies them against the commit's header.
- FinalizeBlock and replay events used block.BlockID(nil), a re-encoding
  that differs from the committed ID for these blocks. They now use the
  commit's block ID.
- Part sizes and proofs from a peer were held unchecked while waiting
  for a lower height. They are validated when the response is decoded
  and charged to the backlog.
- The block sync channel had no room for part metadata or the commit on
  top of a maximum size block. Its capacity now covers both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Your move: thepastaclaw left review threads unresolved; resolve them.
Full checklist in the description.

Covers a later response with oversized part metadata while a lower
height is missing, through the synchronizer rather than the fetch job
alone. The same response without the metadata is held.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed waiting-self-review Waiting for the author to post /self-reviewed labels Nov 9, 2026

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

At bf7abf6, all four prior findings are fixed, but two newly introduced P2P memory-exhaustion paths remain blocking. This was a static review; the supplied CI snapshot and current checks show successful build, tests, lint, vet, and protobuf validation, with review and hygiene checks pending. The overlapping base-branch commit neither supersedes these changes nor conflicts with them, consistent with the supplied clean-merge evidence.

🔴 2 blocking

Review provenance

Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); reviewer 3: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: critical by gpt-6.1-sol (effort low) — The intricate, cross-cutting change directly modifies peer-facing network deserialization and validation in internal/blocksync/committed_parts.go, including reconstruction of committed part sets from untrusted sizes and Merkle proofs, and changes P2P message size limits.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Single stage: Phase 1 and Phase 2 reviewed this head side by side, with no blocker gate between them (triage tier)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort xhigh); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.

In `proto/tendermint/blocksync/types.proto`:
- [BLOCKING] proto/tendermint/blocksync/types.proto:39: Bound proof metadata before protobuf materialization
  The metadata checks run after the allocation they need to prevent. Router.receivePeer calls proto.Unmarshal on the envelope before request matching or BlockResponseFromProto, and the generated BlockResponse.Unmarshal appends a crypto.Proof for every field-4 occurrence. A connected peer can send an unsolicited response containing approximately 100 MiB of repeated `0x22 0x00` pairs, each encoding an empty proof. This fits BlockSyncMaxMsgSize but produces roughly 52 million proof structs: about 3.1 GiB for their 64-byte entries alone on a 64-bit node, excluding transient slice-growth allocations. Neither a valid block nor a commit is needed to reach this allocation. The base decoder skipped this field, so the new representation introduces this amplification. Enforce cumulative metadata limits before general protobuf decoding materializes the collections, using a bounded wire preflight or decoder. Cover proof counts, nested aunt counts and lengths, and part_sizes, and add a raw-wire regression that exercises the actual envelope-decoding boundary.

In `internal/blocksync/committed_parts.go`:
- [BLOCKING] internal/blocksync/committed_parts.go:122-125: Account for oversized backing storage retained by raw block bytes
  This accounting charges len(resp.Block), but committedPartsFromProto retains resp.Block directly, including its backing allocation. The generated bytes-field decoder assigns `m.Block = append(m.Block[:0], ...)`, so duplicate field-1 occurrences can leave a large allocation behind a small final value. A response can contain an approximately 100 MiB first block field followed by a small, structurally valid block field and commit at the requested height. The first field is treated as opaque bytes; only the final value is decoded and validated, while its slice still retains the approximately 100 MiB backing array. When a lower height is missing, pendingToApply retains that allocation but charges only the small final value, allowing further responses to accumulate multi-gigabyte backing storage without spending the intended 256 MiB backlog budget. The outstanding-height limit caps the number of responses but does not restore the byte accounting. Before retaining the bytes, copy disproportionately oversized backing storage into a right-sized allocation, or charge its retained capacity to the budget. Add a duplicate-field raw-wire regression that checks retained capacity or backlog accounting.

Comment thread proto/tendermint/blocksync/types.proto Resolved
Comment thread internal/blocksync/committed_parts.go Resolved
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix; thepastaclaw left review threads unresolved; resolve them.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Nov 9, 2026
Every received message is decoded before it is matched to a request, and
protobuf decoding allocates for each occurrence of a repeated field. A
peer could send an unsolicited response of repeated empty proofs that
expanded about thirty times in memory.

The part sizes and proofs move into one opaque part_layout bytes field,
a serialized BlockPartLayout, decoded only for a response to a request
and after its encoding passes bounds checks on the number of sizes,
proofs and aunts and the length of hashes.

A repeated block field also left the allocation of a larger occurrence
held behind the small last value, uncharged to the backlog. Oversized
backing storage is now copied into a right-sized allocation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix; thepastaclaw left review threads unresolved; resolve them.
Full checklist in the description.

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

At head 0f343d4, five prior findings are fixed, but the proof-metadata allocation blocker remains: malformed packed fields can bypass the new bounds scanner and trigger excessive allocation in the generated decoder. This verification was static; current CI reports passing build, test, lint, and protobuf checks, with PastaClaw review and PR Hygiene pending. The overlapping base-branch changes neither supersede this implementation nor conflict with it according to the supplied merge evidence.

🔴 1 blocking

Review provenance

Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); reviewer 3: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: critical by gpt-6.1-sol (effort low) — The large, intricate diff changes peer-facing network deserialization in internal/blocksync/committed_parts.go and proto/tendermint/blocksync/types.proto, including reconstruction and validation of attacker-controlled part layouts and Merkle proofs.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Single stage: Phase 1 and Phase 2 reviewed this head side by side, with no blocker gate between them (triage tier)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort xhigh); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.

In `internal/blocksync/committed_parts.go`:
- [BLOCKING] internal/blocksync/committed_parts.go:166-171: Bound proof metadata before protobuf materialization
  The scanner counts terminating bytes but accepts an unterminated varint inside a packed sizes field. The generated BlockPartLayout.Unmarshal then checks the whole message length rather than the packed field's postIndex while decoding that varint (types.pb.go:1095–1111), allowing it to consume the following tag and interpret fields differently from the scanner. Consequently, proofs hidden inside an unknown field bypass both the proof-count and nested-proof checks.

  For example, the prefix `0a 01 80 1a 82 80 80 01 00 18 80 00` followed by `12 00` repeated 1,048,575 times occupies 2,097,162 bytes, below BlockPartLayoutMaxBytes. The scanner counts zero sizes and proofs and skips the length-delimited unknown field 3. The generated decoder instead consumes `80 1a` as a size, skips a different unknown field, and appends 1,048,575 empty proof structs—approximately 64 MiB before slice-growth overhead—before decoded-size validation rejects the response. Repeated 23-byte chunks consisting of `0a 01 80 1a 12 11 08 80 01` followed by seven `22 00` pairs similarly hide proofs with seven aunt entries each.

  A peer answering block-sync requests can reach this allocation with any nonempty block field and no valid commit, because layout decoding precedes block and commit validation. The opaque envelope and backlog accounting therefore do not close this transient denial-of-service path. Decode each packed varint within its declared field slice and reject truncation before protobuf materialization; add regression coverage for this scanner/decoder disagreement.

Comment thread internal/blocksync/committed_parts.go Resolved
The bounds check counted packed sizes by terminating bytes, but the
generated decoder lets a varint in a packed field run past the end of
that field. A truncated varint made the two split the layout into
different fields, so proofs and aunts could be hidden from the check
inside what it took for an unknown field.

The check now decodes each packed varint within its field and accepts
only the fields of a layout, well formed, which also removes the
decoder's own rules for skipping unknown fields from the comparison.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Your move: thepastaclaw left review threads unresolved; resolve them.
Full checklist in the description.

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

Static verification of the complete diff at eeff796 found no actionable in-scope defects, and all seven prior findings are fixed. Committed bytes and necessary proofs remain authenticated through commit verification, metadata is bounded before materialization and retention, and ABCI/replay use the committed block ID. No local builds or tests were run; current CI shows code checks passing with review and PR Hygiene pending, and the seven-commit base drift has no reported merge conflicts or superseding implementation.

🔴 0 blocking | 🟡 0 suggestion(s) | 💬 0 nitpick(s)

Review provenance

Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); reviewer 3: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 5: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: critical by gpt-6.1-sol (effort low) — The intricate changes in internal/blocksync/committed_parts.go introduce peer-facing protobuf metadata scanning, bounded decoding, and Merkle-proof-based reconstruction of committed block parts, directly changing a critical network deserialization surface.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Single stage: Phase 1 and Phase 2 reviewed this head side by side, with no blocker gate between them (triage tier)
  • Fresh final gate: an independent Phase-2 review ran after iterative findings were reconciled
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — general (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort xhigh); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify the current code and confirm that no unresolved issues remain.

No unresolved findings remain from the prior review on this head.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Bots are done — your move: address lklimek left a review thread unresolved, then post /self-reviewed.
Full checklist in the description.

Comment thread internal/blocksync/committed_parts.go Resolved
Block sync now has a single way to rebuild a committed part set. The
serving node sends the stored parts' bytes as BlockResponse.block, and
the receiver cuts them into parts of 64kB and computes the part set root
itself, as the proposer did. Honest Tenderdash proposers have always
used that cut, so every block they produced syncs, including blocks
whose bytes a re-encoding does not reproduce. A block committed with a
different cut, or with proofs of a larger tree, is no longer rebuilt:
consensus should reject those, gated by consensus version.

This removes BlockResponse.part_layout and BlockPartLayout,
the wire-level bounds checks they needed,
types.NewPartSetFromPartSizes and ValidatePartSizes, and the larger
block sync channel capacity. The commit's block ID still goes to
FinalizeBlock and the replay events, and a block buffer left oversized
by a repeated block field is still copied before it is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed waiting-self-review Waiting for the author to post /self-reviewed labels Nov 10, 2026
@QuantumExplorer QuantumExplorer changed the title fix(blocksync): serve blocks as the parts they were committed with fix(blocksync): serve blocks as the bytes they were committed with Nov 10, 2026

@lklimek lklimek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CommittedParts.partSet supports three types of part set: bytes only, bytes + sizes , bytes + sizes + proofs. I see no need to support all of them altogether; let's only support one mechanism. See comment.

Note current Proof implementation is not correct, it has some weakness as it doesn't check if the number of parts matches, so if you decide to go with proofs, you need to improve it.

@lklimek
lklimek self-requested a review November 10, 2026 15:52

@Claudius-Maginificent Claudius-Maginificent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 847dce5 after the single-path refactor: no blocking issues. The earlier body-substitution concern is gone with the proof path, and the commit BlockID reaches FinalizeBlock on every caller. Three low-severity notes: memory accounting and spec wording inline, stale comments below.

3 finding(s) posted: 2 inline, 1 in this body.

Findings outside the diff or deferred

  • DOC-002 · LOW — Three comments still attribute the part set stage to block serialization — internal/blocksync/applier.go:147-148

Nothing is serialized here any more; ByteSize is the length of the received bytes, and this stage builds the Merkle tree over them. Same wording remains at applier.go:241-242 and synchronizer.go:681-683 ("serialization, signature verification, disk or the ABCI application"). Reword all three the way applyTimings.PartSet already is.

🤖 Co-authored by Claudius the Magnificent AI Agent

Comment thread internal/blocksync/synchronizer.go Resolved
Comment thread spec/p2p/messages/block-sync.md Resolved
…cost

Block sync rebuilds only the standard 64kB cut, while consensus still
accepts other part sizes and proofs of a larger tree. The spec, the
proto comment and the applier now say so: a block committed with
another cut fails the commit check whichever peer serves it.

The backlog ceiling is in serialized bytes, and a pending response
holds its block twice, as received bytes and decoded. The comment now
says the heap is at least twice the charged bytes, about 14 GB at the
ceiling. Charging both copies would only change the budget term, not
the in-flight term that dominates.

The part set stage no longer serializes the block, so three comments
that said it did are reworded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

The current implementation still permits a malicious peer set to fill the in-order backlog with near-MaxBlockSize responses while the lowest height is missing. Each response retains both the decoded block and committed wire bytes, but the byte budget is checked only before issuing requests, leaving the already-issued window capable of consuming roughly 14 GB before commit verification rejects the blocks. The supplied CI snapshot reports successful build, test, lint, and protobuf checks; PastaClaw review and PR Hygiene remain pending.

🔴 1 blocking

Review provenance

Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); reviewer 3: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: critical by gpt-6.1-sol (effort low) — This is a large cross-cutting change to peer-facing block-sync deserialization and consensus/ABCI block application, including BlockResponse handling, commit verification, storage, and block IDs in internal/blocksync/p2p_msg_handler.go and internal/state/execution.go.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Single stage: Phase 1 and Phase 2 reviewed this head side by side, with no blocker gate between them (triage tier)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort xhigh); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions are optional: fix them forward in a later PR rather than pushing another round just for them.
- Address every current finding in one push: self-review the full diff first, and don't push piecemeal.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.

In `internal/blocksync/synchronizer.go`:
- [BLOCKING] internal/blocksync/synchronizer.go:789-797: Bound raw and decoded block copies in the in-order backlog
  Block responses are decoded and inserted into pendingToApply before VerifyCommit runs, so a peer can return structurally valid but unauthenticated near-MaxBlockSize blocks for higher heights while withholding the next height. The producer checks maxPendingApplyBytes only before issuing more requests, and maxOutstandingHeights still allows up to 64 requests already in flight; when those responses arrive, they can all be retained behind the missing height. Each response now keeps both BlockBytes and the decoded Block, while Size charges only len(BlockBytes), so the documented approximately 6.98-GB serialized ceiling corresponds to roughly 14 GB of block data before commits, maps, queues, and other allocations. Reserve memory for the in-flight window or otherwise enforce a bound that covers both retained representations before accepting more responses; documenting the multiplier does not prevent an attacker from exhausting memory.
Out-of-scope follow-up suggestions (1)

These are valid observations, but they are outside this PR's scope and should be handled in separate issues or author/maintainer-requested PRs rather than blocking this review.

  • Include commit and envelope overhead in the block-sync receive limit — BlockSyncChannel.RecvMessageCapacity is derived from MaxBlockSizeBytes plus only the block-response framing prefix and field key, although a BlockResponse also carries a Commit and the outer P2P envelope. A valid block close to the hard maximum can therefore exceed the receive capacity and be rejected before block sync handles it. This predates the PR and the bytes-versus-embedded-message change preserves the same length-delimited payload size, so it is an adjacent hardening issue rather than a defect introduced here.
    • Follow-up: Open a follow-up hardening PR that derives the block-sync receive capacity from the complete serialized envelope, including maximum Commit and envelope overhead.

Comment thread internal/blocksync/synchronizer.go Resolved
@github-actions

Copy link
Copy Markdown

Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix; thepastaclaw left review threads unresolved; resolve them.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Nov 10, 2026
A block response kept both the committed bytes and the decoded block
while it waited behind a missing lower height, but was charged to the
backlog for its bytes only. That doubled what the in-order backlog can
retain, to about 14 GB at its ceiling, against what dev held.

A response now keeps the block's height and bytes. The block is still
decoded on arrival, so a block that does not decode is refused before it
is queued, and is decoded again when it is applied: the new
blockApplier.ApplyBytes decodes it under the applier lock and times it
as the "decode" stage, so decoding does not count as waiting. A pending
response so holds one copy of its block, the one it is charged for.

TestBlockResponseRetainsOnlyItsBytes holds 16 responses of a block of
small transactions and checks what they retain beyond their bytes. With
the decoded block kept as well it retains 23 MB more, against about
10 KB without.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Your move: thepastaclaw left review threads unresolved; resolve them.
Full checklist in the description.

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

At abfa42b, pending block data has a single retained representation, while arrival-time validation and application from authenticated committed bytes remain intact; no new in-scope defects were found. This was a static review of the complete PR diff and the incremental change, with no local builds or tests run. Current GitHub checks pass for build, tests, end-to-end tests, lint, vet and protobuf validation; PastaClaw review and PR Hygiene remain pending.

🔴 0 blocking | 🟡 0 suggestion(s) | 💬 0 nitpick(s)

Review provenance

Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); reviewer 3: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 5: gpt-6.1-sol (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: critical by gpt-6.1-sol (effort low) — This large, intricate change modifies peer-facing block-response deserialization and validation in BlockResponseFromProto, block application and commit verification in blockApplier.ApplyBytes, and consensus/ABCI block-ID handling in internal/state/execution.go and internal/consensus/replayer.go.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Single stage: Phase 1 and Phase 2 reviewed this head side by side, with no blocker gate between them (triage tier)
  • Fresh final gate: an independent Phase-2 review ran after iterative findings were reconciled
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — general (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — tenderdash-consensus-security (completed, effort xhigh); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify the current code and confirm that no unresolved issues remain.

No unresolved findings remain from the prior review on this head.
Out-of-scope follow-up suggestions (1)

These are valid observations, but they are outside this PR's scope and should be handled in separate issues or author/maintainer-requested PRs rather than blocking this review.

  • Reserve aggregate memory for unauthenticated in-flight block responses — The pre-existing admission logic in internal/blocksync/block_fetch_job.go checks the 256 MiB backlog budget before issuing requests but permits a 64-height window. Malicious peers can withhold the blocking height and return structurally valid, near-100 MiB responses for higher requested heights, retaining several gigabytes before commit verification. Removing the additional decoded representation fixes the PR's amplification but leaves this existing denial-of-service exposure.
    • Follow-up: Create a separate issue or follow-up PR for aggregate memory reservations covering pending and in-flight responses, while preserving capacity to retry the height that unblocks application.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Bots are done — your move: post /self-reviewed.
Full checklist in the description.

@lklimek
lklimek merged commit 0bdad33 into dev Nov 12, 2026
36 of 37 checks passed
@lklimek
lklimek deleted the fix/blocksync-serve-original-parts branch November 12, 2026 19:05
lklimek added a commit that referenced this pull request Nov 12, 2026
…1621)

#1595 replaced the decoded block in the synchronizer's BlockResponse with
its committed bytes and a Height field. The handover tests from #1618,
merged after it, still set and read BlockResponse.Block, so the
blocksync test package no longer compiled on dev. They only need the
height, which addBlock reads from Height.

<sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub>


Claude-Session: https://claude.ai/code/session_01WunYYgi2uMsQDcBcKAwLdt

Co-authored-by: Lukasz Klimek <842586+lklimek@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-self-review Waiting for the author to post /self-reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants