Repository navigation
Description
This is Claude, filing on behalf of @QuantumExplorer after working on #975.
Summary
Merk ships a public multi-chunk API that nothing uses. It should either be adopted by state sync while state sync is still being built out, or deleted. Leaving it as unowned public surface keeps costing audit and review time.
The surface in question:
ChunkOpand itsed::Encode/ed::Decodeimpls —merk/src/proofs/chunk/chunk_op.rsMultiChunk,ChunkProducer::multi_chunk_with_limit,ChunkProducer::multi_chunk_with_limit_and_index—merk/src/merk/chunks.rsRestorer::process_multi_chunk/validate_multi_chunk—merk/src/merk/restore.rsChunkError::InvalidMultiChunkand friends —merk/src/proofs/chunk/error.rs
It has never had a caller
- In this repo the only callers are the
#[cfg(test)]modules inchunks.rsandrestore.rs. Nothing undergrovedb/referencesChunkOp,MultiChunkorprocess_multi_chunk. - This is not a regression from a protocol change. The API arrived in feat: db state sync by merk chunking #292 ("db state sync by merk chunking", May 2024), and in that same commit
grovedb/src/replication.rsalready used the single-chunk path (chunk_producer.chunk(chunk_id)→restorer.process_chunk(chunk_id, ops)).git log -Sfinds no commit wheregrovedb/ever referenced the multi-chunk API. - Dash Platform (
v4.2-dev) has no references either. Other downstream users of the publishedgrovedb-merkcrate are unknown.
What it costs to keep
Audit findings filed against code no production path reaches:
- [audit][medium] Merk chunk-op decoding canonicalizes invalid traversal bytes #704 — ChunkOp decoder canonicalized non-binary traversal bytes (fixed in fix(merk): validate ChunkOp bytes before trusting them #975)
- [audit][medium] Multi-chunk restore accepts dangling ChunkId #705 — multi-chunk restore accepted a dangling
ChunkId - [audit][C003] Public ChunkOp decoding allocates before validating resource limits #904 — ChunkOp decoder allocated from declared lengths before validating them (fixed in fix(merk): validate ChunkOp bytes before trusting them #975)
- [audit][C016] Multichunk encoding can exceed its requested byte budget #913 — multichunk encoding can exceed its requested byte budget (still open)
Each one needs triage, a fix or a written rationale, tests and review. #913 is next in line unless this is settled first.
What it could offer
The live protocol is receiver-driven and count-bounded: the receiver names up to CONST_GROUP_PACKING_SIZE global chunk ids and the source answers with one op list per id, packed with pack_nested_bytes.
Multi-chunk is source-driven and byte-bounded: "start at chunk X, fill up to N bytes", returning interleaved ChunkId / Chunk ops plus a next_index resume cursor. That is a better fit when the transport has a hard byte ceiling (ABCI snapshot chunks do) and chunk sizes vary widely, where count-based packing either wastes space or overshoots.
What adopting it would actually take
- It operates on a single Merk. State sync spans subtrees (global chunk ids, prefixes, tree types), so a byte-budgeted batch would have to be built at the GroveDB layer, most likely next to
pack_nested_bytesrather than on top ofChunkOp. - It is unfinished: the producer still carries
// TODO: what happens if the vec is filled? we need to have some kind of hardhoc limit value if none is supplied, andlimit: Nonematerializes the whole tree. - Its byte accounting is wrong ([audit][C016] Multichunk encoding can exceed its requested byte budget #913).
- It has not followed the live path: no support for the append-only tree family (feat(replication): state sync support for the append-only tree family (fixes #785) #788), none of the session hardening from feat(replication)!: complete V1 state sync and bounded-memory restore #840, no commit modes.
- Source-driven streaming changes the trust model. Today the receiver knows exactly which ids it asked for; with multi-chunk it must validate a peer-chosen sequence of chunk ids (the [audit][medium] Multi-chunk restore accepts dangling ChunkId #705 class of bug).
Options
- Use it — decide that byte-budgeted batching is wanted for state sync, and rework this API (or replace it with a GroveDB-layer equivalent) as part of that work. Fix [audit][C016] Multichunk encoding can exceed its requested byte budget #913 as part of it.
- Delete it — remove the surface listed above plus its tests, close [audit][C016] Multichunk encoding can exceed its requested byte budget #913 as moot. The shared helpers (
chunk_index_from_traversal_instruction,generate_traversal_instruction, the [audit][S10] Chunk identifier recovery exceeds the valid tree-depth work bound #883 depth bound) stay; they serve the live path. The idea is one sentence and the code is recoverable from feat: db state sync by merk chunking #292. This is a breaking change to the publicgrovedb-merkAPI, so it belongs in a release that can carry one.
Ask
Pick one while state sync is still in flux, rather than continuing to patch the API finding by finding. Until then, #913 should probably wait on this decision.
Activity