Skip to content

Decide the fate of merk's multi-chunk API (ChunkOp / MultiChunk): wire it into state sync or delete it #976

Open

Description

@QuantumExplorer

This is Claude, filing on behalf of @QuantumExplorer after working on #975.

Summary

Merk ships a public multi-chunk API that nothing uses. It should either be adopted by state sync while state sync is still being built out, or deleted. Leaving it as unowned public surface keeps costing audit and review time.

The surface in question:

  • ChunkOp and its ed::Encode / ed::Decode impls — merk/src/proofs/chunk/chunk_op.rs
  • MultiChunk, ChunkProducer::multi_chunk_with_limit, ChunkProducer::multi_chunk_with_limit_and_index — merk/src/merk/chunks.rs
  • Restorer::process_multi_chunk / validate_multi_chunk — merk/src/merk/restore.rs
  • ChunkError::InvalidMultiChunk and friends — merk/src/proofs/chunk/error.rs

It has never had a caller

  • In this repo the only callers are the #[cfg(test)] modules in chunks.rs and restore.rs. Nothing under grovedb/ references ChunkOp, MultiChunk or process_multi_chunk.
  • This is not a regression from a protocol change. The API arrived in feat: db state sync by merk chunking #292 ("db state sync by merk chunking", May 2024), and in that same commit grovedb/src/replication.rs already used the single-chunk path (chunk_producer.chunk(chunk_id) → restorer.process_chunk(chunk_id, ops)). git log -S finds no commit where grovedb/ ever referenced the multi-chunk API.
  • Dash Platform (v4.2-dev) has no references either. Other downstream users of the published grovedb-merk crate are unknown.

What it costs to keep

Audit findings filed against code no production path reaches:

Each one needs triage, a fix or a written rationale, tests and review. #913 is next in line unless this is settled first.

What it could offer

The live protocol is receiver-driven and count-bounded: the receiver names up to CONST_GROUP_PACKING_SIZE global chunk ids and the source answers with one op list per id, packed with pack_nested_bytes.

Multi-chunk is source-driven and byte-bounded: "start at chunk X, fill up to N bytes", returning interleaved ChunkId / Chunk ops plus a next_index resume cursor. That is a better fit when the transport has a hard byte ceiling (ABCI snapshot chunks do) and chunk sizes vary widely, where count-based packing either wastes space or overshoots.

What adopting it would actually take

Options

  1. Use it — decide that byte-budgeted batching is wanted for state sync, and rework this API (or replace it with a GroveDB-layer equivalent) as part of that work. Fix [audit][C016] Multichunk encoding can exceed its requested byte budget #913 as part of it.
  2. Delete it — remove the surface listed above plus its tests, close [audit][C016] Multichunk encoding can exceed its requested byte budget #913 as moot. The shared helpers (chunk_index_from_traversal_instruction, generate_traversal_instruction, the [audit][S10] Chunk identifier recovery exceeds the valid tree-depth work bound #883 depth bound) stay; they serve the live path. The idea is one sentence and the code is recoverable from feat: db state sync by merk chunking #292. This is a breaking change to the public grovedb-merk API, so it belongs in a release that can carry one.

Ask

Pick one while state sync is still in flux, rather than continuing to patch the API finding by finding. Until then, #913 should probably wait on this decision.

Activity

  1. added
    area:state-syncS groups: state restoration, synchronization and append-client transactions.
    on Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:state-syncS groups: state restoration, synchronization and append-client transactions.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions