Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
collection kept as one aux entry per member could not be read back. (#968)

### Changed
- **BREAKING**: `TreeNode::put_value_with_fixed_cost`,
`TreeNode::put_value_with_reference_value_hash_and_value_cost`,
`TreeNode::put_value_with_two_reference_value_hashes_and_value_cost` and
their `Walker` counterparts take a `&GroveVersion`, which selects the
just-in-time value update (see the `merk_versions.tree.just_in_time_value_update`
entry under Fixed). `TreeNode::provided_value_hash_put_final_value` is
renamed `TreeNode::predict_put_final_value` and takes the value-defined
cost the stored node is loaded with, the put it predicts (the new
`grovedb_merk::tree::PredictedPut`: an ordinary put as
`TreeNode::put_value` installs it at the grove version, a
provided-value-hash put, or a put at a specialized cost) and a
`&GroveVersion`. `grovedb_merk::element::insert::specialized_put_cost`,
the value-defined cost an ordinary write puts an element with, and
`Merk::replacement_runs_value_update`, whether a write over a key runs
the just-in-time value update, are new.
- **BREAKING**: `delete_operation_for_delete_internal` and
`delete_operations_for_delete_up_tree_while_empty` take the operations
already pending in the batch as any `impl PendingOperations` in place of a
Expand Down Expand Up @@ -170,6 +185,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
replay, through the new
`operations.worst_case.worst_case_sum_item_cost_size` slot (**BREAKING**
for code building `GroveDBOperationsWorstCaseVersions` by hand).
- From `GROVE_V4`, a reference a batch writes or refreshes alongside an
update of its target item commits to the bytes the update stores. Before,
the batch predicted those bytes with a hand-written copy of Merk's
just-in-time value update that took a sum item (`SumItem`,
`ItemWithSumItem`) to keep its stored flags, while the apply stores the
flags the flags update leaves it. When the flags update merges them (an
owned item updated in a later epoch, an unowned one gaining an owner) the
reference committed to bytes that are never stored, which
`verify_grovedb` reports. Drive writes this batch for a document of a
summable type: a refresh of a deeper index reference next to the
primary-storage update. The prediction also refused a sum item replaced
by an item, or the reverse, that applies on its own. Now a reference to
a target the batch has already written hashes the bytes stored there, and
one to a target still to be written commits to the bytes Merk's own
just-in-time value update gives for the put the apply performs, once per
target. V1..V3 keep the old prediction for replay, through the new
`apply_batch.same_batch_reference_target_prediction` slot (**BREAKING**
for code building `GroveDBApplyBatchVersions` by hand).
- From `GROVE_V4`, a replacement whose flags update leaves the flags
unchanged is charged for the bytes it stores. Before, Merk measured the
replacement with the new value carrying the old value's flags and kept
that measurement, so when the new value's own flags had another length
(a plain `apply_batch` replacing a flagged item with longer, shorter or
no flags, or Drive's merging callback on an unowned item gaining an owner
as it grows) the commit failed with `storage_cost cost mismatch`. A round
that rewrites the flags and then restores the put's value now also
restores its value-defined cost, so a sum item or tree measured again is
measured with the cost of the bytes it holds. Every replacement V1..V3
accept is charged the same. V1..V3 keep the old measurement for replay,
through the new `merk_versions.tree.just_in_time_value_update` slot
(**BREAKING** for code building `MerkTreeVersions` by hand).
- From `GROVE_V4`, a reference a batch resolves through a pending
`InsertIfNotExists` reaches the element already stored at that key when
there is one. Before, it always reached the op's element, so a reference
Expand Down
18 changes: 18 additions & 0 deletions grovedb-version/src/version/grovedb_versions.rs
Original file line number Diff line number Diff line change
Expand Up @@ -268,6 +268,24 @@ pub struct GroveDBApplyBatchVersions {
/// the same existence read the insert's own level makes; the op's
/// element otherwise.
pub reference_through_conditional_insert: FeatureVersion,
/// How a reference written in a batch gets the value hash of a target
/// item the same batch updates.
///
/// - `0` (V1..V3): a hand-written copy of Merk's just-in-time value
/// update over the target's bytes as the Merk cache holds them. For a
/// sum item it assumes the new element keeps the stored flags and does
/// not consult the flags update, while the apply stores the flags the
/// flags update leaves, so when those differ (an owned item updated in
/// a later epoch, an unowned one gaining an owner) the reference
/// commits to bytes that are never stored. A sum item replaced by an
/// item, or the reverse, is refused.
/// - `1` (V4+): once the batch has written the target, the hash of the
/// bytes it stores; before, Merk's own just-in-time value update run on
/// the stored and new bytes with the put the apply performs, once per
/// target. The reference commits to exactly the bytes the apply stores,
/// for every item kind, every answer of the flags update and every
/// order the batch writes the reference and its target in.
pub same_batch_reference_target_prediction: FeatureVersion,
}

#[derive(Clone, Debug, Default)]
Expand Down
18 changes: 18 additions & 0 deletions grovedb-version/src/version/merk_versions.rs
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,24 @@ pub struct MerkTreeVersions {
/// accounting is unchanged: it is always computed from the stored
/// predecessor bytes.
pub put_value: FeatureVersion,
/// `TreeNode::just_in_time_tree_node_value_update`: the client-driven
/// update a replacement of a stored value runs (flags carry-over,
/// flags-update callback, removal sectioning).
///
/// Version 0 measures the replacement with the new value carrying the
/// OLD value's flags, and keeps that measurement when the flags-update
/// callback answers that it left the flags unchanged. The new value is
/// then stored with its own flags, so when those differ in length from
/// the old ones the recorded cost does not match the bytes written and
/// the commit fails with a storage cost mismatch. Consensus-locked for
/// grove v1..v3.
///
/// Version 1 re-measures the replacement from the bytes it stores when
/// the callback answers unchanged and their size differs from the last
/// measurement taken, and restores the put's value-defined cost
/// whenever it restores the put's value. Every update whose cost
/// version 0 records for the bytes it stores is charged the same.
pub just_in_time_value_update: FeatureVersion,
}

#[derive(Clone, Debug, Default)]
Expand Down
6 changes: 5 additions & 1 deletion grovedb-version/src/version/v1.rs
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ pub const GROVE_V1: GroveVersion = GroveVersion {
backward_references_maintenance: 0,
move_element: 0,
reference_through_conditional_insert: 0,
same_batch_reference_target_prediction: 0,
},
element: GroveDBElementMethodVersions {
delete: 0,
Expand Down Expand Up @@ -294,7 +295,10 @@ pub const GROVE_V1: GroveVersion = GroveVersion {
},
// Ordinary replacements keep the loaded `value_defined_cost`
// metadata (issue #908); consensus-locked, bumped in v4.
tree: MerkTreeVersions { put_value: 0 },
tree: MerkTreeVersions {
put_value: 0,
just_in_time_value_update: 0,
},
},
// MMR hash charges: the shipped accounting, which billed the
// storage reads each operation performed but not the blake3 merges
Expand Down
6 changes: 5 additions & 1 deletion grovedb-version/src/version/v2.rs
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ pub const GROVE_V2: GroveVersion = GroveVersion {
backward_references_maintenance: 0,
move_element: 0,
reference_through_conditional_insert: 0,
same_batch_reference_target_prediction: 0,
},
element: GroveDBElementMethodVersions {
delete: 0,
Expand Down Expand Up @@ -293,7 +294,10 @@ pub const GROVE_V2: GroveVersion = GroveVersion {
},
// Ordinary replacements keep the loaded `value_defined_cost`
// metadata (issue #908); consensus-locked, bumped in v4.
tree: MerkTreeVersions { put_value: 0 },
tree: MerkTreeVersions {
put_value: 0,
just_in_time_value_update: 0,
},
},
// MMR hash charges: the shipped accounting, which billed the
// storage reads each operation performed but not the blake3 merges
Expand Down
6 changes: 5 additions & 1 deletion grovedb-version/src/version/v3.rs
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ pub const GROVE_V3: GroveVersion = GroveVersion {
backward_references_maintenance: 0,
move_element: 0,
reference_through_conditional_insert: 0,
same_batch_reference_target_prediction: 0,
},
element: GroveDBElementMethodVersions {
delete: 0,
Expand Down Expand Up @@ -311,7 +312,10 @@ pub const GROVE_V3: GroveVersion = GroveVersion {
},
// Ordinary replacements keep the loaded `value_defined_cost`
// metadata (issue #908); consensus-locked, bumped in v4.
tree: MerkTreeVersions { put_value: 0 },
tree: MerkTreeVersions {
put_value: 0,
just_in_time_value_update: 0,
},
},
// MMR hash charges: the shipped accounting, which billed the
// storage reads each operation performed but not the blake3 merges
Expand Down
40 changes: 37 additions & 3 deletions grovedb-version/src/version/v4.rs
Original file line number Diff line number Diff line change
Expand Up @@ -346,6 +346,29 @@
//! Identity-owned sections are unaffected; Drive accounts for the default
//! section as system removals, separately from identity fee refunds.
//!
//! - `apply_batch.same_batch_reference_target_prediction: 1`: a reference
//! written in the same batch as an update of its target commits to the
//! bytes the target stores. A target the batch has already written is
//! hashed from those bytes; one still to be written is predicted once,
//! through Merk's own just-in-time value update for the put the apply
//! performs. V1..V3 assume a sum item keeps its stored flags, so when the
//! flags update merges them (an owned item updated in a later epoch, an
//! unowned one gaining an owner) the reference commits to bytes never
//! stored and the grove stops verifying; they also refuse a sum item
//! replaced by an item, or the reverse, that applies on its own. Gated
//! because it moves a committed root and flips a rejected/accepted
//! outcome.
//!
//! - `merk_versions.tree.just_in_time_value_update: 1`: a replacement
//! whose flags update leaves the flags unchanged is charged for the bytes
//! it stores. V1..V3 keep the measurement taken with the old value's
//! flags, so a replacement whose own flags differ in length fails its
//! commit with a storage cost mismatch. A round that rewrites the flags
//! and then restores the put's value also restores its value-defined
//! cost, so a sum item or tree measured again is measured with the cost
//! of the bytes it holds. Gated because it changes tracked costs and
//! flips a failed commit into a successful one.
//!
//! Note that `GroveVersion::latest()` resolves to this version, so anything
//! defaulting to "latest" — tests, benchmarks, tools — exercises every gate
//! listed above rather than V3 behaviour.
Expand Down Expand Up @@ -417,6 +440,10 @@ pub const GROVE_V4: GroveVersion = GroveVersion {
// v1: a reference resolved through a skipped InsertIfNotExists
// reaches the element already stored there.
reference_through_conditional_insert: 1,
// v1: a reference written in the same batch as its target's
// update predicts the target's stored bytes through Merk's own
// just-in-time value update.
same_batch_reference_target_prediction: 1,
},
element: GroveDBElementMethodVersions {
delete: 0,
Expand Down Expand Up @@ -720,9 +747,16 @@ pub const GROVE_V4: GroveVersion = GroveVersion {
// proof envelope.
prove_count_offset_on_range: 0,
},
// Bumped 0 -> 1: ordinary (Item / Reference) replacements of a
// specialized value are charged from their own bytes (issue #908).
tree: MerkTreeVersions { put_value: 1 },
tree: MerkTreeVersions {
// Bumped 0 -> 1: ordinary (Item / Reference) replacements of a
// specialized value are charged from their own bytes (issue #908).
put_value: 1,
// Bumped 0 -> 1: a replacement whose flags update leaves the
// flags unchanged is measured from the bytes it stores, and a
// sum item or tree measured again after its value was restored
// is measured with that value's own value-defined cost.
just_in_time_value_update: 1,
},
},
// MMR hash charges: one hash per blake3 merge actually computed —
// `push` per collapsed peak, `get_root` and `gen_proof` per peak
Expand Down
Loading
Loading