Repository navigation
fix(batch)!: commit same-batch sum-item references and replacements to the bytes stored (GROVE_V4) - #1021
Conversation
…o the bytes stored (GROVE_V4) Two existing bugs in how a batch accounts for a write over a stored value, both behind new GROVE_V4 version slots (0 in V1..V3, 1 in V4): - apply_batch.same_batch_reference_target_prediction: a reference written or refreshed in a batch that also updates its target item predicts the bytes the update stores. Version 0 (the previous body, frozen) assumed a sum item keeps its stored flags, while the apply stores the flags the flags update merges, so an owned item updated in a later epoch or an unowned one gaining an owner left the reference committed to bytes that are never stored. This is the batch Drive writes for a document of a summable type. Version 1 hashes the bytes stored for a target the pass has already written, and otherwise runs Merk's own just-in-time value update with the put the apply performs, once per target. The sum item to item refusal goes with it. - merk_versions.tree.just_in_time_value_update: a replacement whose flags update leaves the flags unchanged kept the measurement taken with the old value's flags, so new flags of another length failed the commit with a storage cost mismatch. Version 1 measures it again from the bytes it stores and restores the put's value-defined cost whenever it restores the put's value. Version 0 is the previous body, frozen. Merk's put functions that run the update take a &GroveVersion, and TreeNode::provided_value_hash_put_final_value becomes predict_put_final_value, which predicts any put kind. BREAKING CHANGE: GroveDBApplyBatchVersions and MerkTreeVersions gain public fields; TreeNode::put_value_with_fixed_cost, the two layered put functions and their Walker counterparts take a &GroveVersion; provided_value_hash_put_final_value is renamed predict_put_final_value with new arguments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 27 seconds. View limit details
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
|
Waiting for bot review — coderabbitai ✓ · thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
|
⏹️ Not reviewed — PR is closed (merged) (commit 160c61d) |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #1021 +/- ##
=========================================
Coverage 93.20% 93.21%
=========================================
Files 352 356 +4
Lines 108428 108953 +525
=========================================
+ Hits 101062 101558 +496
- Misses 7366 7395 +29
🚀 New features to boost your workflow:
|
thepastaclaw
left a comment
There was a problem hiding this comment.
Final validation — Phase 2 only (no Phase 1 for this repository)
The two supplied blocking findings describe one confirmed mismatch between prediction and application for newly inserted targets retained across partial-batch segments. The V4 predictor can commit a reference to callback-modified flags while the live target stores the supplied flags without running that callback. Verification was static at the exact head; the supplied CI snapshot shows all Ubuntu test shards, formatting, linting, and security audit passing, with PR Hygiene pending.
🔴 1 blocking
Review provenance
Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: architecture-layering); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: platform-versioning); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: rust-quality); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)
- Triage:
criticalbygpt-6.1-sol(effort low) — The intricate cross-layer changes directly alter authenticated-state commitments in grovedb/src/batch/just_in_time_reference_update/v1.rs::process_old_element_flags_v1 by changing which serialized target bytes are hashed into same-batch references. - Phase 1 reviewers: not run (disabled for this repository)
- Fresh verifier:
gpt-6.1-sol— final-verifier; agentsol-verifier - Phase 2 reviewers:
gpt-6.1-sol— general (completed, effort xhigh); agentphase2-reviewer,gpt-6.1-sol— architecture-layering (completed, effort xhigh); agentphase2-reviewer,gpt-6.1-sol— platform-versioning (completed, effort xhigh); agentphase2-reviewer,gpt-6.1-sol— rust-quality (completed, effort xhigh); agentphase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `grovedb/src/batch/just_in_time_reference_update/v1.rs`:
- [BLOCKING] grovedb/src/batch/just_in_time_reference_update/v1.rs:73-85: Preserve the live node's replacement eligibility when predicting
A flagged target first inserted by a partial batch's initial segment remains in the retained Merk with `old_value: None`: the insertion uses `TreeNode::new`, and neither `TreeNode::commit` nor `MerkCommitter::write` advances that field. Rewriting this item in the add-on is allowed by the cross-segment gate. When the add-on also writes a deeper reference, `continue_partial_apply_body` clears the target's written marker, so this call predicts the pending rewrite from its cached bytes.
The detached node in `TreeNode::predict_put_final_value` unconditionally sets `old_value: Some(old_value)` and runs the flags updater. The actual `put_value_with_fixed_cost` on the retained node skips that updater because its `old_value` is still `None`. A deterministic callback that changes flag contents without changing their length therefore makes prediction hash modified flags while application stores the supplied flags. Ordinary same-batch references have no post-apply comparison to reject this disagreement, leaving the reference committed to bytes its target does not hold.
This is a V4 regression for sum targets when the insertion and add-on supply identical flags: the legacy sum-item shortcut hashes those supplied flags and matches application. Carry the live node's updater eligibility into prediction, or synchronize its mutation-history state under the V4 gate. Extend the continuation regression to cover a target first inserted by the initial segment; the existing rewrite test seeds an already-stored predecessor through `grove_with`.
|
Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix; thepastaclaw left review threads unresolved; resolve them. |
…rite Review fixes for the GROVE_V4 same-batch reference prediction: - After a pass writes a target that a same-batch reference was committed to by prediction, the batch checks that the write stored exactly the predicted bytes and is refused otherwise, as it already is for a backward-references item. The check reads the node just written and is not charged. - The reference arm decides whether to predict with the same predicate that records written targets, instead of a second copy tied by a debug_assert. - The prediction always loads the stored node's value-defined cost for an ordinary put and lets the versioned install decide whether to keep it; TreeNode::ordinary_put_keeps_value_defined_cost is gone, so the put_value semantics live in the install helpers only. - Tests: Drive's shapes (a flagged ReferenceWithSumItem in a sum tree under SumTree, CountSumTree, ProvableSumTree and ProvableCountProvableSumTree primaries, a NonCounted target, a two-epoch target shrunk in a later epoch), a proof through the reference (verifies on V4, not on V3), prediction v1 with Merk put_value 0, and the refusal of a write that stores other bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Waiting for bot review — coderabbitai not yet · thepastaclaw 1 thread unresolved — resolve it. Wait for the missing reviews, or a writer can post |
…s given A partial batch's initial segment can insert a flagged item at a new key and its add-on rewrite it alongside a deeper reference to it. The Merk still holds the inserted node with no value to replace, so the add-on's write stores its bytes as given without running the flags update, while the version 1 prediction ran Merk's just-in-time value update over the inserted bytes. A flags update that changes the flags' contents (the merging one keeping epoch-0 flags over same-size epoch-2 ones) left the reference committed to bytes the item does not hold, which the post-apply check now refuses. Merk::replacement_runs_value_update says, from the nodes already in memory, whether a write over a key runs the update; the prediction hashes the supplied bytes when it does not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Waiting for bot review — coderabbitai not yet · thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
…tinuation-dd8070 develop's #1021 landed this branch's two GROVE_V4 fixes (same-batch reference prediction and the Unchanged re-measure) with later review fixes of its own, so those now exist once, in develop's form. What this branch adds over develop is the settlement mode: - the prediction keeps develop's runs_value_update and post-apply check; a settling batch uses version 1 on every grove version - CHANGELOG: the fixes stay under develop's entries in 6.1.0, and the settlement entries move to Unreleased - same_batch_reference_prediction_tests is develop's file with the settling modes added, its helpers shared with settle_test_support - the option-off estimate pins are retaken from develop at 0ea327d: 18 of the 66 latest-version rows moved with #1018..#1020 and none of the 66 grove v3 rows did - a settled tree's lookup comment says why it is still needed after #1018 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Basic explanation
What this does: It fixes two existing bugs in how a batch accounts for a write over a value that is already stored. Both fixes apply from
GROVE_V4only;GROVE_V1toGROVE_V3behave exactly as before.SumItem,ItemWithSumItem) keeps its stored flags. The apply actually stores the flags the flags callback merges. So whenever the merge changes them, the reference commits to a hash of bytes that never land, andverify_grovedbreports it.storage_cost cost mismatch.Value: Dash Platform writes the first batch shape for every document of a summable type (protocol version 14). When such a document is updated in a later epoch and changes size, its index reference is refreshed in the same batch as its primary-storage
ItemWithSumItem. Without the fix, that reference commits to the wrong hash, so proofs of reads through that index entry fail to verify. Landing the fix inGROVE_V4, before Platform's protocol version 14 activates, keeps such entries out of state from the start.Risks: Low to medium. Only
GROVE_V4changes behaviour. The flags callback still answersbool, andBatchApplyOptionsis unchanged. Two edits touch code thatGROVE_V1toGROVE_V3also run; both are refactors with identical output (see "In-place changes to shipped generations"). Some Merk put functions now take a&GroveVersion(see "Breaking Changes").These fixes were found during review of #1001 and are split out of it so they can ship without the opt-in settlement of storage owner changes, which is now planned for a later Platform release. #1001 will be rebased on this PR.
Issue being fixed or feature implemented
1. Stale same-batch reference to a sum item
A batch that updates an item and also writes or refreshes a reference to it must commit the reference to the item's final bytes, flags update included. The old prediction got this right for plain items but assumed a sum item keeps its stored flags. Two cases make the merged flags differ:
The old prediction also refused a sum item replaced by an item (or the reverse) under a same-batch reference, a write that applies on its own. From
GROVE_V4it is accepted.2. Cost mismatch on an "unchanged" flags answer
What was done?
GROVE_V1toGROVE_V3and 1 inGROVE_V4:apply_batch.same_batch_reference_target_prediction(GroveDBApplyBatchVersions);merk_versions.tree.just_in_time_value_update(MerkTreeVersions).grovedb/src/batch/just_in_time_reference_update/): amod.rsdispatcher,v0.rswith the previous body unchanged, andv1.rs. Version 1:TreeNode::predict_put_final_value), once per target;Merk::replacement_runs_value_update, which reads only the nodes in memory);JustInTimeElementFlagsClientError), as it already is for a backward-references item. The check reads the node the pass just wrote and is not charged, so it changes no cost.merk/src/tree/just_in_time_value_update/): amod.rsdispatcher,v0.rswith the previous body unchanged, andv1.rs. Version 1:&GroveVersionto select it.How Has This Been Tested?
New
grovedb/src/batch/same_batch_reference_prediction_tests.rs:a_reference_rewritten_in_the_same_batch_commits_to_the_stored_bytes: the batch Platform writes. It covers anItemWithSumItemgrown by 1, 3 and 25 bytes in a later epoch, a sum item gaining an owner, and plain items, under seven ways of writing or refreshing the reference, and checks thatverify_grovedbis clean.grove_v3_keeps_the_legacy_sum_item_prediction: pins the old stale references onGROVE_V3.a_sum_item_and_an_item_replace_each_other_under_a_same_batch_reference: accepted on V4, refused on V3.a_reference_to_a_target_written_earlier_in_the_batch_commits_to_its_stored_bytesan_add_on_reference_commits_to_the_bytes_the_initial_segment_storedan_add_on_rewriting_an_initial_segment_target_predicts_its_new_bytesa_drive_shaped_reference_commits_to_the_stored_bytes: a flaggedReferenceWithSumItemin a sum tree, refreshed (trusted or not) or rewritten, to anItemWithSumItemunder aSumTree,CountSumTree,ProvableSumTreeorProvableCountProvableSumTree, grown in a later epoch or holding two epochs and shrunk in a later one, plus aNonCountedtarget in aCountSumTree.a_proof_through_a_same_batch_reference_verifies: a proof read through the refreshed reference verifies against the root hash on V4 and does not on V3.an_add_on_rewriting_an_item_the_initial_segment_inserted_commits_to_its_bytes: a partial batch whose add-on rewrites an item its initial segment inserted, with a deeper reference to it.a_reference_predicted_under_merk_put_value_0_commits_to_the_stored_bytes: a hand-built version pairing prediction v1 with Merkput_value0.a_replacement_keeping_flags_of_another_length_is_charged_for_its_bytes: V4, exact charge.grove_v3_refuses_a_replacement_keeping_flags_of_another_length: V3 cost mismatch, pinned.Merk (
merk/src/tree/just_in_time_value_update/mod.rs):predicted_provided_value_hash_put_matches_the_apply(the existing test, renamed)a_changed_round_then_unchanged_measures_the_value_it_storesLocally:
cargo fmt --all -- --checkclean;cargo check -p grovedb-versionandcargo check -p grovedb-merk --testspass with no warnings.cargo test -p grovedb-merk --lib: 802 passed.cargo test -p grovedb --libfiltered tosame_batch_reference_prediction,backward_references,just_in_timeandreference: 432 passed; after the last review fix, the 13 tests insame_batch_reference_prediction_tests.rspass, and the reference, backward-references and partial-batch tests (525) pass.cargo clippy -p grovedb -p grovedb-merk --all-targetsis clean. CI covers the full suite.In-place changes to shipped generations
Two edits touch code that
GROVE_V1toGROVE_V3also run. Neither changes what those versions store, charge or return:merk/src/tree/put_value/v0.rs: the line that installs an ordinary value moves intoinstall_ordinary_value_v0(the samekv.put_value_no_update_of_hashes(value)call), and the function passes the newgrove_versionargument through to the just-in-time value update dispatcher, which selects the unchanged v0 body on those versions.merk/src/element/insert.rs: the inline "sum item takes its specialized cost, anything else anOp::Put" choice, repeated in two places, becomes the sharedspecialized_put_cost, with the same condition, cost and error.grovedb/src/batch/mod.rs, the arm offollow_reference_get_value_hashfor a reference to a pending flagged item write: it now reads the target's stored bytes first and serializes the new element after, in the version-0 arm ofprocess_old_element_flags, which also decodes the stored element there. Successful batches are unchanged, since serialization is not metered. Only a batch whose new element fails to serialize, which a valid element cannot, would now report the cost of the read with the error.Breaking Changes
GroveDBApplyBatchVersionsgainssame_batch_reference_target_predictionandMerkTreeVersionsgainsjust_in_time_value_update. Code that builds either struct by hand must set them.TreeNode::put_value_with_fixed_cost,TreeNode::put_value_with_reference_value_hash_and_value_cost,TreeNode::put_value_with_two_reference_value_hashes_and_value_costand theirWalkercounterparts take a trailing&GroveVersion.TreeNode::provided_value_hash_put_final_valueis renamedTreeNode::predict_put_final_value. It also takes the stored node's value-defined cost, the put it predicts (newgrovedb_merk::tree::PredictedPut) and a&GroveVersion.GROVE_V4behaviour: a same-batch reference to a sum item now commits to the stored bytes, and the replacements above are now accepted. Dash Platform uses none of the renamed or re-signed functions, so adopting this needs only a rev bump; the fixes then apply from its protocol version 14.Checklist:
For repository code-owners and collaborators only
🤖 Generated with Claude Code
PR Hygiene ·
160c61d/skip-botsproceeds without the ones not yet reported/self-reviewedWhen every merge requirement is met, the
PR Hygienecheck passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.Summary by CodeRabbit
TreeNodeandWalkervalue-update APIs now require version information, and the final-value prediction method has been renamed. Earlier Grove versions retain their previous behavior.