diff --git a/CHANGELOG.md b/CHANGELOG.md index c77a39770..55f6b38d5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,22 +52,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 deletion sections it, every byte of the new element, key included, counts as added, and the element keeps the flags the callback leaves it with. GroveDB does not look at owners: `StorageFlags::update_element_flags_replacing_on_owner_change` - answers it when the new flags name a different owner. A batch accepts the - answer only with the new `BatchApplyOptions::allow_replacement_accounting`, - and refuses it on a write whose flags are byte-equal to the stored ones - (which covers an untrusted `RefreshReference` and a tree's root update), with - `Error::InvalidBatchOperation`. With the option, average-case and worst-case - estimates charge every `InsertOrReplace`, `Replace`, `Patch` and trusted - `RefreshReference` of an element with flags, and the write of a flagged tree - the batch also writes under, at least the bytes its node adds when inserted. - The option needs `apply_batch.replacement_accounting` (0 in V1..V4, 1 in V5), - and Merk accepts the answer from `merk_versions.tree.just_in_time_value_update` - version 2 (V5); - earlier versions refuse a batch or estimate that sets it. Off, every cost, - estimate, stored element and hash is what it was. **BREAKING** for code - building `BatchApplyOptions` or `GroveDBApplyBatchVersions` field by field, - and for callers of Merk's just-in-time apply, whose update callback now - answers `(ElementFlagsUpdate, Option)`. + answers it when the new flags name a different owner. A batch refuses it on + a write whose flags are byte-equal to the stored ones (which covers an + untrusted `RefreshReference` and a tree's root update), with + `Error::InvalidBatchOperation`. Merk accepts the answer from + `merk_versions.tree.just_in_time_value_update` version 2 (V5); earlier + versions refuse it. Estimates do not model it, because an estimator cannot + ask the callback: estimate such a write as an insertion. A callback that never + answers it sees every cost, stored element and hash as before. **BREAKING** + for callers of Merk's just-in-time apply, whose update callback now answers + `(ElementFlagsUpdate, Option)`. ### Changed - **BREAKING**: `Query` adds the public `offset: Option` field for branch-local committed-count-unit pagination. Exhaustive struct literals diff --git a/docs/crates/auxiliary.md b/docs/crates/auxiliary.md index bcb0b0f3b..3430dbe2d 100644 --- a/docs/crates/auxiliary.md +++ b/docs/crates/auxiliary.md @@ -215,7 +215,7 @@ These work on raw `ElementFlags` and match the callback signatures GroveDB expec It always merges with `MergingOwnersStrategy::UseTheirs`. When `old_flags` is `Some`, both the old and the new bytes must decode to flags, or it returns `RemovingFlagsError`. - `StorageFlags::update_element_flags_with_written_owner(cost, old_flags, new_flags) -> Result` is `update_element_flags`, except that the owner `new_flags` names always owns the element afterwards, including where `update_element_flags` keeps the old owner: an update that keeps the size, or one that shrinks an element holding bytes of one epoch when the new flags' epoch is later than that one. To leave the element with its owner while someone else pays for the write, write flags naming the current owner. To hand it over, write flags naming the new owner. New flags that name no owner get what `update_element_flags` gives. Bytes the update frees are still refunded to the old owner, and bytes that stay stored move with the element. -- `StorageFlags::update_element_flags_replacing_on_owner_change(cost, old_flags, new_flags) -> Result` is `update_element_flags_with_written_owner`, except that when the new flags name a different owner than the old ones it leaves them as written and answers `ElementFlagsUpdate::AccountAsReplacement`: the old owner is refunded for the whole old element and the new element's every byte is added. The batch must set `BatchApplyOptions::allow_replacement_accounting` (GROVE_V5+). +- `StorageFlags::update_element_flags_replacing_on_owner_change(cost, old_flags, new_flags) -> Result` is `update_element_flags_with_written_owner`, except that when the new flags name a different owner than the old ones it leaves them as written and answers `ElementFlagsUpdate::AccountAsReplacement`: the old owner is refunded for the whole old element and the new element's every byte is added. GroveDB accepts the answer from GROVE_V5. - `StorageFlags::split_removal_bytes(flags: &mut ElementFlags, removed_key_bytes, removed_value_bytes)` decodes the flags and calls `split_storage_removed_bytes`. It returns `BasicStorageRemoval` for both parts if the element has no flags. Callers hand these helpers `new_flags` of one epoch. What the helpers return may hold several, and that is safe: the batch puts the written flags back before every retry, so each retry starts from the same input and the size settles. Written flags that already hold several epochs can make the size flip between retries until the batch stops with its cyclic error, whether or not the owner changes. So can stored multi-epoch flags with no owner under written flags that name one, on `update_element_flags`. Either way the batch is refused and nothing is written. This is known and left as it is: shipped protocol versions depend on this merge, and a write of one-epoch flags with an owner over stored flags with an owner does not reach it. diff --git a/grovedb-epoch-based-storage-flags/src/update_element_flags.rs b/grovedb-epoch-based-storage-flags/src/update_element_flags.rs index 0d8652dc9..aa2c2166c 100644 --- a/grovedb-epoch-based-storage-flags/src/update_element_flags.rs +++ b/grovedb-epoch-based-storage-flags/src/update_element_flags.rs @@ -96,8 +96,7 @@ impl StorageFlags { /// Every other update (same owner, or no owner on one side) is answered by /// [`Self::update_element_flags_with_written_owner`]. /// - /// The batch accepts the answer only when its options set - /// `allow_replacement_accounting`. + /// GroveDB accepts the answer from grove version 5. /// /// Only an element written with the new owner's flags is handed over, under /// this helper and [`Self::update_element_flags_with_written_owner`] alike. diff --git a/grovedb-version/src/tests.rs b/grovedb-version/src/tests.rs index edd10bd43..a6749ad7a 100644 --- a/grovedb-version/src/tests.rs +++ b/grovedb-version/src/tests.rs @@ -94,7 +94,6 @@ fn v5_matches_v4_apart_from_its_gates() { path_query_methods.merge = GROVE_V4.grovedb_versions.path_query_methods.merge; path_query_methods.per_instance_query_offsets = 0; v5_as_v4.grovedb_versions.apply_batch.keyless_append_repeats = 0; - v5_as_v4.grovedb_versions.apply_batch.replacement_accounting = 0; v5_as_v4.merk_versions.tree.just_in_time_value_update = GROVE_V4.merk_versions.tree.just_in_time_value_update; assert_eq!(format!("{v5_as_v4:?}"), format!("{GROVE_V4:?}")); diff --git a/grovedb-version/src/version/grovedb_versions.rs b/grovedb-version/src/version/grovedb_versions.rs index 472ea3095..e424f620d 100644 --- a/grovedb-version/src/version/grovedb_versions.rs +++ b/grovedb-version/src/version/grovedb_versions.rs @@ -307,13 +307,6 @@ pub struct GroveDBApplyBatchVersions { /// slot gates an accepted/rejected outcome: a batch V1..V4 refuse is /// accepted on V5+. pub keyless_append_repeats: FeatureVersion, - /// Whether a batch may set `BatchApplyOptions::allow_replacement_accounting`. - /// - /// - `0` (grove v1..v4): a batch or estimate that sets the option is - /// refused. - /// - `1`: the flags update may answer `AccountAsReplacement`, and an - /// estimate covers it. - pub replacement_accounting: FeatureVersion, } #[derive(Clone, Debug, Default)] diff --git a/grovedb-version/src/version/v1.rs b/grovedb-version/src/version/v1.rs index 145ef7525..acdb38097 100644 --- a/grovedb-version/src/version/v1.rs +++ b/grovedb-version/src/version/v1.rs @@ -48,7 +48,6 @@ pub const GROVE_V1: GroveVersion = GroveVersion { reference_through_conditional_insert: 0, same_batch_reference_target_prediction: 0, keyless_append_repeats: 0, - replacement_accounting: 0, }, element: GroveDBElementMethodVersions { delete: 0, diff --git a/grovedb-version/src/version/v2.rs b/grovedb-version/src/version/v2.rs index 65ceb4220..08a08eb33 100644 --- a/grovedb-version/src/version/v2.rs +++ b/grovedb-version/src/version/v2.rs @@ -48,7 +48,6 @@ pub const GROVE_V2: GroveVersion = GroveVersion { reference_through_conditional_insert: 0, same_batch_reference_target_prediction: 0, keyless_append_repeats: 0, - replacement_accounting: 0, }, element: GroveDBElementMethodVersions { delete: 0, diff --git a/grovedb-version/src/version/v3.rs b/grovedb-version/src/version/v3.rs index 70d4be2af..51e928bc3 100644 --- a/grovedb-version/src/version/v3.rs +++ b/grovedb-version/src/version/v3.rs @@ -48,7 +48,6 @@ pub const GROVE_V3: GroveVersion = GroveVersion { reference_through_conditional_insert: 0, same_batch_reference_target_prediction: 0, keyless_append_repeats: 0, - replacement_accounting: 0, }, element: GroveDBElementMethodVersions { delete: 0, diff --git a/grovedb-version/src/version/v4.rs b/grovedb-version/src/version/v4.rs index cc4e41ace..e51994847 100644 --- a/grovedb-version/src/version/v4.rs +++ b/grovedb-version/src/version/v4.rs @@ -442,7 +442,6 @@ pub const GROVE_V4: GroveVersion = GroveVersion { same_batch_reference_target_prediction: 1, // v0: every operation is counted for repeats, keyless appends included. keyless_append_repeats: 0, - replacement_accounting: 0, }, element: GroveDBElementMethodVersions { delete: 0, diff --git a/grovedb-version/src/version/v5.rs b/grovedb-version/src/version/v5.rs index fdcd9cf1e..d9296c657 100644 --- a/grovedb-version/src/version/v5.rs +++ b/grovedb-version/src/version/v5.rs @@ -42,16 +42,11 @@ //! batch, as 6.1.1 did. Gated because it flips an accepted/rejected outcome //! (issue #1031). //! -//! - `apply_batch.replacement_accounting: 1`: a batch may set -//! `allow_replacement_accounting`, so a flags update can have an update -//! accounted as the removal of the old element plus the insertion of the -//! new one. V4 refuses the option. Gated because the accounting and the -//! stored flags are consensus-relevant once a caller uses it. -//! //! - `merk_versions.tree.just_in_time_value_update: 2`: Merk accepts that //! answer and records the removal plus the insertion. Versions 0 and 1 //! refuse it, so grove v1..v4 cannot account an update as a replacement at -//! any layer. Gated with the option above, for the same reason. +//! any layer. Gated because the accounting and the stored flags are +//! consensus-relevant once a caller uses it. //! //! It exists so that changes made after GroveDB 6.1.1 have somewhere to //! land. GROVE_V4 as 6.1.1 shipped it is what Dash Platform 5.0 (protocol @@ -86,7 +81,6 @@ pub const GROVE_V5: GroveVersion = GroveVersion { grovedb_versions: GroveDBVersions { apply_batch: GroveDBApplyBatchVersions { keyless_append_repeats: 1, // changed: a batch may repeat a keyless append - replacement_accounting: 1, // changed: a batch may account an update as a replacement ..GROVE_V4.grovedb_versions.apply_batch }, element: GroveDBElementMethodVersions { diff --git a/grovedb/src/batch/estimated_costs/average_case_costs.rs b/grovedb/src/batch/estimated_costs/average_case_costs.rs index ff4c52ca1..bc6239e69 100644 --- a/grovedb/src/batch/estimated_costs/average_case_costs.rs +++ b/grovedb/src/batch/estimated_costs/average_case_costs.rs @@ -45,38 +45,10 @@ use crate::{BackwardsReferences, Element}; #[cfg(feature = "minimal")] impl GroveOp { - /// [`Self::average_case_cost_with_options`] with the default batch - /// options. - #[cfg(test)] - #[allow(clippy::too_many_arguments)] - fn average_case_cost( - &self, - path: &KeyInfoPath, - key: &KeyInfo, - layer_element_estimates: &EstimatedLayerInformation, - append_tree_chunk_power: Option, - backwards_references: BackwardsReferences, - propagate: bool, - grove_version: &GroveVersion, - ) -> CostResult<(), Error> { - self.average_case_cost_with_options( - path, - key, - layer_element_estimates, - append_tree_chunk_power, - backwards_references, - propagate, - None, - &BatchApplyOptions::default(), - grove_version, - ) - } - /// Get the estimated average case cost of the op. Calls a lower level /// function to calculate the estimate based on the type of op. Returns /// CostResult. - #[allow(clippy::too_many_arguments)] - fn average_case_cost_with_options( + fn average_case_cost( &self, // The op's own path: sizes the inverted-registration growth bound // (every `invert()` output is built from the origin's qualified @@ -97,33 +69,9 @@ impl GroveOp { // it declares `Check`. backwards_references: BackwardsReferences, propagate: bool, - // The declared tree type of the layer an `InsertTreeWithRootHash` - // writes, which sizes it when it is accounted as a replacement. - written_tree_type: Option, - // The batch's options: with `allow_replacement_accounting`, a write that may - // be accounted as a replacement is charged as the insertion it then - // records too. - batch_apply_options: &BatchApplyOptions, grove_version: &GroveVersion, ) -> CostResult<(), Error> { let in_tree_type = layer_element_estimates.tree_type; - // A backward-references element is charged the typical referrer - // shape, each referrer at the entry of one at this op's own position. - // Only an estimate with the option reads that entry size, so it is measured - // only then. - let replacement = super::ReplacementAccounting::new( - batch_apply_options, - key, - in_tree_type.inner_node_type(), - super::CarriedReferrers::Average { - entry_bytes: if batch_apply_options.allow_replacement_accounting { - super::backward_reference_entry_bound(path, key) - } else { - 0 - }, - }, - grove_version, - ); let propagate_if_input = || { if propagate { Some(layer_element_estimates) @@ -145,10 +93,20 @@ impl GroveOp { } match element { Some(Element::BidirectionalReference(..)) => { - // The registration entry appended to the target, whose - // referrer is this op itself. + // The registration entry appended to the target: an + // inverted path built from the referrer's qualified + // origin (this op's path segments plus its key — an + // absolute inversion serializes them all), the cascade + // flag, and framing. + let origin_bytes: u32 = path + .0 + .iter() + .map(|segment| 4 + segment.max_length() as u32) + .sum::() + .saturating_add(4 + key.max_length() as u32); + let entry_bound = origin_bytes.saturating_add(16); Some(super::BackwardReferencesFanOut::average_reference( - super::backward_reference_entry_bound(path, key), + entry_bound, )) } // A backward-references ITEM write carries its own referrer @@ -255,44 +213,23 @@ impl GroveOp { not_summed, not_counted_or_summed, .. - } => replacement.raise_tree( - GroveDb::average_case_merk_insert_tree( - key, - flags, - aggregate_data.parent_tree_type(), - in_tree_type, - // Account for the wrapper byte if the op rebuilds the - // tree as `NonCounted(...)`, `NotSummed(...)`, or - // `NotCountedOrSummed(...)`. They share the same +1 - // discriminant overhead and are mutually exclusive on - // the rebuilt element. - super::wrapper_overhead_for(*non_counted, *not_summed, *not_counted_or_summed), - propagate_if_input(), - grove_version, - ), - // The write of a tree the batch also writes under may be - // accounted as a replacement as a write of its element is. - written_tree_type.unwrap_or(super::WORST_CASE_REPLACEMENT_TREE_TYPE), + } => GroveDb::average_case_merk_insert_tree( + key, flags, - *non_counted, - *not_summed, - *not_counted_or_summed, + aggregate_data.parent_tree_type(), + in_tree_type, + // Account for the wrapper byte if the op rebuilds the + // tree as `NonCounted(...)`, `NotSummed(...)`, or + // `NotCountedOrSummed(...)`. They share the same +1 + // discriminant overhead and are mutually exclusive on + // the rebuilt element. + super::wrapper_overhead_for(*non_counted, *not_summed, *not_counted_or_summed), + propagate_if_input(), + grove_version, ), GroveOp::InsertOrReplace { element } - | GroveOp::InsertOrReplaceDontCheckForBackwardsReferences { element } => with_fan_out( - replacement.raise( - GroveDb::average_case_merk_insert_element( - key, - element, - in_tree_type, - propagate_if_input(), - grove_version, - ), - element, - ), - backward_references_fan_out(Some(element)), - ), - GroveOp::InsertWithKnownToNotAlreadyExist { element } => with_fan_out( + | GroveOp::InsertOrReplaceDontCheckForBackwardsReferences { element } + | GroveOp::InsertWithKnownToNotAlreadyExist { element } => with_fan_out( GroveDb::average_case_merk_insert_element( key, element, @@ -376,26 +313,22 @@ impl GroveOp { } else { inner }; - let replace_cost = GroveDb::average_case_merk_replace_element( + GroveDb::average_case_merk_replace_element( key, &element, in_tree_type, propagate_if_input(), grove_version, - ); - replacement.raise_refresh(replace_cost, &element, mode) + ) } GroveOp::Replace { element } | GroveOp::ReplaceDontCheckForBackwardsReferences { element } => with_fan_out( - replacement.raise( - GroveDb::average_case_merk_replace_element( - key, - element, - in_tree_type, - propagate_if_input(), - grove_version, - ), + GroveDb::average_case_merk_replace_element( + key, element, + in_tree_type, + propagate_if_input(), + grove_version, ), backward_references_fan_out(Some(element)), ), @@ -407,16 +340,13 @@ impl GroveOp { element, change_in_bytes, } => with_fan_out( - replacement.raise( - GroveDb::average_case_merk_patch_element( - key, - element, - *change_in_bytes, - in_tree_type, - propagate_if_input(), - grove_version, - ), + GroveDb::average_case_merk_patch_element( + key, element, + *change_in_bytes, + in_tree_type, + propagate_if_input(), + grove_version, ), backward_references_fan_out(Some(element)), ), @@ -1052,7 +982,7 @@ impl TreeCache for AverageCaseTreeCacheKnownPaths { path: &KeyInfoPath, ops_at_path_by_key: BTreeMap, _ops_by_qualified_paths: &BTreeMap>, GroveOp>, - batch_apply_options: &BatchApplyOptions, + _batch_apply_options: &BatchApplyOptions, _flags_update: &mut G, _split_removal_bytes: &mut SR, grove_version: &GroveVersion, @@ -1215,30 +1145,15 @@ impl TreeCache for AverageCaseTreeCacheKnownPaths { &mut cost, path, &key, tree_type, ); } - // The layer a flagged `InsertTreeWithRootHash` writes, whose - // declared tree type sizes the tree when it is accounted as a - // replacement. Only an estimate with the option reads it, so the lookup is - // skipped otherwise. - let written_tree_type = if batch_apply_options.allow_replacement_accounting - && let GroveOp::InsertTreeWithRootHash { flags: Some(_), .. } = &op - { - let mut written_layer = path.clone(); - written_layer.push(key.clone()); - self.paths.get(&written_layer).map(|layer| layer.tree_type) - } else { - None - }; cost_return_on_error!( &mut cost, - op.average_case_cost_with_options( + op.average_case_cost( path, &key, layer_element_estimates, append_tree_chunk_power, op.backwards_references(), false, - written_tree_type, - batch_apply_options, grove_version ) ); diff --git a/grovedb/src/batch/estimated_costs/mod.rs b/grovedb/src/batch/estimated_costs/mod.rs index 85b249517..bc1fa4c9a 100644 --- a/grovedb/src/batch/estimated_costs/mod.rs +++ b/grovedb/src/batch/estimated_costs/mod.rs @@ -5,9 +5,8 @@ use std::collections::HashMap; #[cfg(feature = "minimal")] use grovedb_costs::{ - cost_return_on_error, cost_return_on_error_no_add, storage_cost::{removal::StorageRemovedBytes, StorageCost}, - CostsExt, OperationCost, + OperationCost, }; #[cfg(feature = "minimal")] use grovedb_merk::estimated_costs::{ @@ -42,281 +41,6 @@ pub(in crate::batch) fn wrapper_overhead_for( } } -/// The node type a worst-case estimate sizes a replacement with: -/// the worst-case layers do not say which tree a write lands in, and no node -/// carries more feature bytes than a big sum tree's. -#[cfg(feature = "minimal")] -pub(in crate::batch) const WORST_CASE_REPLACEMENT_NODE_TYPE: grovedb_merk::merk::NodeType = - grovedb_merk::merk::NodeType::BigSumNode; - -/// The allowance a registered referrer entry takes per segment of the -/// referrer's inverted path (and for its key). -#[cfg(feature = "minimal")] -const REFERRER_ENTRY_SEGMENT_BYTES: u32 = 4; - -/// The allowance a registered referrer entry takes for its cascade flag and -/// framing. -#[cfg(feature = "minimal")] -const REFERRER_ENTRY_FRAMING_BYTES: u32 = 16; - -/// An upper bound on the registered referrer entry of a referrer at `path` -/// under `key`: an inverted path built from the referrer's qualified origin -/// (its path segments plus its key; an absolute inversion serializes them -/// all), the cascade flag, and framing. -#[cfg(feature = "minimal")] -pub(in crate::batch) fn backward_reference_entry_bound( - path: &crate::batch::KeyInfoPath, - key: &crate::batch::KeyInfo, -) -> u32 { - use grovedb_storage::worst_case_costs::WorstKeyLength; - - path.0 - .iter() - .map(|segment| REFERRER_ENTRY_SEGMENT_BYTES + segment.max_length() as u32) - .sum::() - .saturating_add(REFERRER_ENTRY_SEGMENT_BYTES + key.max_length() as u32) - .saturating_add(REFERRER_ENTRY_FRAMING_BYTES) -} - -/// An upper bound on one registered referrer entry of a backward-references -/// element whose referrers may sit anywhere registration admits them: -/// [`backward_reference_entry_bound`] of a referrer at the deepest such -/// position (`MAX_BACKWARD_REFERENCES_GROVE_DEPTH` segments plus its key, -/// each of the biggest key size). -#[cfg(feature = "minimal")] -pub(in crate::batch) const WORST_CASE_BACKWARD_REFERENCE_ENTRY_BOUND: u32 = - (crate::bidirectional_references::MAX_BACKWARD_REFERENCES_GROVE_DEPTH as u32 + 1) - * (REFERRER_ENTRY_SEGMENT_BYTES - + grovedb_merk::estimated_costs::worst_case_costs::MERK_BIGGEST_KEY_SIZE) - + REFERRER_ENTRY_FRAMING_BYTES; - -/// How a replacement estimate sizes the referrer entries a backward-references -/// element lands with: the referrers registered on the element it replaces, -/// in place of the ones it was written with. (Only backward-references -/// elements declare a capacity.) -#[cfg(feature = "minimal")] -#[derive(Clone, Copy, Debug)] -pub(in crate::batch) enum CarriedReferrers { - /// Every referrer the element's declared capacity admits, each at - /// [`WORST_CASE_BACKWARD_REFERENCE_ENTRY_BOUND`]: the referrers can sit - /// anywhere registration admits them, so no apply accounted as a replacement adds more. - Worst, - /// The typical shape: as many referrers as the average item fan-out - /// ([`BACKWARD_REFERENCES_AVERAGE_ITEM_FAN_OUT`], never more than the - /// declared capacity), each with an entry of `entry_bytes`. - Average { - /// The bytes of one referrer entry. - entry_bytes: u32, - }, -} - -#[cfg(feature = "minimal")] -impl CarriedReferrers { - /// The bytes of the referrer entries an element declaring `capacity` - /// referrers lands with. - fn bytes(self, capacity: u16) -> u32 { - match self { - CarriedReferrers::Worst => { - (capacity as u32).saturating_mul(WORST_CASE_BACKWARD_REFERENCE_ENTRY_BOUND) - } - CarriedReferrers::Average { entry_bytes } => BACKWARD_REFERENCES_AVERAGE_ITEM_FAN_OUT - .min(capacity as u32) - .saturating_mul(entry_bytes), - } - } -} - -/// The bytes a write of `element` at a key of `key_len` bytes adds when it -/// is accounted as a replacement: the whole node, key and value, sized exactly as -/// the apply sizes the node it stores in a tree of `node_type` nodes. -/// -/// A backward-references element is sized without the referrer entries it -/// was written with, plus the entries `carried_referrers` gives it. (None of -/// those elements has a value-defined cost.) -#[cfg(feature = "minimal")] -fn replacement_accounting_added_bytes( - key_len: u32, - element: &crate::Element, - node_type: grovedb_merk::merk::NodeType, - carried_referrers: CarriedReferrers, - grove_version: &grovedb_version::version::GroveVersion, -) -> Result { - use grovedb_merk::{ - element::costs::ElementCostExtensions, - tree::kv::{ - ValueDefinedCostType::{LayeredValueDefinedCost, SpecializedValueDefinedCost}, - KV, - }, - }; - - let raw_value_len = match element.value_defined_cost(grove_version) { - Some(LayeredValueDefinedCost(value_cost)) => { - return Ok(KV::layered_node_byte_cost_size_for_key_and_value_lengths( - key_len, value_cost, node_type, - )) - } - Some(SpecializedValueDefinedCost(value_cost)) => value_cost, - None => match element.max_incoming_references() { - Some(capacity) => { - let element_bytes = if element - .backward_references() - .is_some_and(|refs| refs.is_empty()) - { - element.serialized_size(grove_version)? - } else { - element - .stripped_of_backward_references() - .serialized_size(grove_version)? - }; - (element_bytes as u32).saturating_add(carried_referrers.bytes(capacity)) - } - None => element.serialized_size(grove_version)? as u32, - }, - }; - Ok(KV::node_byte_cost_size_for_key_and_raw_value_lengths( - key_len, - raw_value_len, - node_type, - )) -} - -/// The Merk tree a worst-case estimate sizes a replaced -/// `InsertTreeWithRootHash` with (a count-sum tree, the largest any such op -/// writes): the op carries no aggregate data from which the tree type could -/// be read. -#[cfg(feature = "minimal")] -pub(in crate::batch) const WORST_CASE_REPLACEMENT_TREE_TYPE: grovedb_merk::tree_type::TreeType = - grovedb_merk::tree_type::TreeType::CountSumTree; - -/// The raise an estimate applies to a write that may be accounted as a replacement -/// ([`BatchApplyOptions::allow_replacement_accounting`](crate::batch::BatchApplyOptions::allow_replacement_accounting)): -/// the write's added bytes are raised to every byte of the node it writes, -/// key included, sized for a tree of `node_type` nodes with the referrer -/// entries `carried_referrers` gives a backward-references element, so the -/// estimate covers both the in-place update it models and the insertion a -/// replacement records. Without the option, or for an element -/// without flags (the flags update is never asked about one), the estimate -/// is left as it is. -#[cfg(feature = "minimal")] -pub(in crate::batch) struct ReplacementAccounting<'a> { - allow_replacement_accounting: bool, - key: &'a crate::batch::KeyInfo, - node_type: grovedb_merk::merk::NodeType, - carried_referrers: CarriedReferrers, - grove_version: &'a grovedb_version::version::GroveVersion, -} - -#[cfg(feature = "minimal")] -impl<'a> ReplacementAccounting<'a> { - /// The raise for a write at `key` in a tree of `node_type` nodes. - pub(in crate::batch) fn new( - batch_apply_options: &crate::batch::BatchApplyOptions, - key: &'a crate::batch::KeyInfo, - node_type: grovedb_merk::merk::NodeType, - carried_referrers: CarriedReferrers, - grove_version: &'a grovedb_version::version::GroveVersion, - ) -> Self { - Self { - allow_replacement_accounting: batch_apply_options.allow_replacement_accounting, - key, - node_type, - carried_referrers, - grove_version, - } - } - - /// `base`, the estimate of a write of `element` over a stored element, - /// raised for the case that the write is accounted as a replacement. - pub(in crate::batch) fn raise( - &self, - base: grovedb_costs::CostResult<(), crate::Error>, - element: &crate::Element, - ) -> grovedb_costs::CostResult<(), crate::Error> { - use grovedb_storage::worst_case_costs::WorstKeyLength; - - if !self.allow_replacement_accounting || element.get_flags().is_none() { - return base; - } - Self::raised_to( - base, - replacement_accounting_added_bytes( - self.key.max_length() as u32, - element, - self.node_type, - self.carried_referrers, - self.grove_version, - ), - ) - } - - /// `base`, its added bytes raised to at least `replacement_bytes`. An error - /// in `base` comes first and keeps its cost. - fn raised_to( - base: grovedb_costs::CostResult<(), crate::Error>, - replacement_bytes: Result, - ) -> grovedb_costs::CostResult<(), crate::Error> { - let mut cost = OperationCost::default(); - cost_return_on_error!(&mut cost, base); - let replacement_bytes = cost_return_on_error_no_add!(cost, replacement_bytes); - cost.storage_cost.added_bytes = cost.storage_cost.added_bytes.max(replacement_bytes); - Ok(()).wrap_with_cost(cost) - } - - /// [`Self::raise`] for a `RefreshReference` in `mode` writing `element`. - /// An untrusted refresh writes the stored flags back, and the batch - /// refuses to account as a replacement a write that keeps the stored - /// flags, so only a trusted one can be. - pub(in crate::batch) fn raise_refresh( - &self, - base: grovedb_costs::CostResult<(), crate::Error>, - element: &crate::Element, - mode: &crate::batch::RefreshReferenceMode, - ) -> grovedb_costs::CostResult<(), crate::Error> { - if mode.is_trusted() { - self.raise(base, element) - } else { - base - } - } - - /// [`Self::raise`] for the `InsertTreeWithRootHash` that a write of a - /// tree becomes when the batch also writes under it: the tree element - /// that op writes, a tree of `tree_type` with `flags` in the wrapper the - /// op rebuilds it in. Sized as - /// [`ElementCostExtensions::value_defined_cost`](grovedb_merk::element::costs::ElementCostExtensions::value_defined_cost) - /// sizes that element. - pub(in crate::batch) fn raise_tree( - &self, - base: grovedb_costs::CostResult<(), crate::Error>, - tree_type: grovedb_merk::tree_type::TreeType, - flags: &Option, - non_counted: bool, - not_summed: bool, - not_counted_or_summed: bool, - ) -> grovedb_costs::CostResult<(), crate::Error> { - use grovedb_merk::{tree::kv::KV, tree_type::CostSize}; - use grovedb_storage::worst_case_costs::WorstKeyLength; - use integer_encoding::VarInt; - - let (true, Some(flags)) = (self.allow_replacement_accounting, flags) else { - return base; - }; - let flags_len = flags.len() as u32; - let value_cost = tree_type.cost_size() - + flags_len - + flags_len.required_space() as u32 - + wrapper_overhead_for(non_counted, not_summed, not_counted_or_summed); - Self::raised_to( - base, - Ok(KV::layered_node_byte_cost_size_for_key_and_value_lengths( - self.key.max_length() as u32, - value_cost, - self.node_type, - )), - ) - } -} - // ── Backward-references fan-out estimation model ──────────────────────── // // Under `BatchApplyOptions::backwards_references` (GROVE_V4+), a @@ -904,334 +628,3 @@ pub enum EstimatedCostsType { /// Worst case estimated costs type WorstCaseCostsType(HashMap), } - -#[cfg(all(test, feature = "minimal"))] -mod tests { - use grovedb_costs::{CostsExt, OperationCost}; - use grovedb_merk::{ - merk::NodeType, - tree::kv::KV, - tree_type::{CostSize, TreeType}, - }; - use grovedb_version::version::GroveVersion; - - use crate::{ - batch::{BatchApplyOptions, KeyInfo}, - bidirectional_references::BackwardReference, - reference_path::ReferencePathType, - BackwardReferences, Element, - }; - - fn replacement_options() -> BatchApplyOptions { - BatchApplyOptions { - allow_replacement_accounting: true, - ..Default::default() - } - } - - /// The added bytes the raise gives a write of `element` at `key` - /// whose own estimate adds nothing. - fn raised(replacement: &super::ReplacementAccounting, element: &Element) -> u32 { - replacement - .raise(Ok(()).wrap_with_cost(OperationCost::default()), element) - .cost - .storage_cost - .added_bytes - } - - /// The worst-case size of a replaced tree bounds the node of every Merk - /// tree an `InsertTreeWithRootHash` can write, in a tree of any node - /// type. - #[test] - fn the_worst_case_replaced_tree_bounds_every_merk_tree_in_every_node_type() { - let grove_version = GroveVersion::latest(); - let options = replacement_options(); - let key = KeyInfo::KnownKey(b"tree".to_vec()); - let flags = Some(vec![0; 35]); - let raised = super::ReplacementAccounting::new( - &options, - &key, - super::WORST_CASE_REPLACEMENT_NODE_TYPE, - super::CarriedReferrers::Worst, - grove_version, - ) - .raise_tree( - Ok(()).wrap_with_cost(OperationCost::default()), - super::WORST_CASE_REPLACEMENT_TREE_TYPE, - &flags, - false, - false, - false, - ) - .cost - .storage_cost - .added_bytes; - for tree_type in [ - TreeType::NormalTree, - TreeType::SumTree, - TreeType::BigSumTree, - TreeType::CountTree, - TreeType::CountSumTree, - TreeType::ProvableCountTree, - TreeType::ProvableCountSumTree, - TreeType::ProvableSumTree, - TreeType::ProvableCountProvableSumTree, - ] { - for node_type in [ - NodeType::NormalNode, - NodeType::SumNode, - NodeType::BigSumNode, - NodeType::CountNode, - NodeType::CountSumNode, - NodeType::ProvableCountNode, - NodeType::ProvableCountSumNode, - NodeType::ProvableSumNode, - NodeType::ProvableCountProvableSumNode, - ] { - let replaced = KV::layered_node_byte_cost_size_for_key_and_value_lengths( - 4, - tree_type.cost_size() + 36, - node_type, - ); - assert!( - raised >= replaced, - "a {tree_type:?} in a tree of {node_type:?} nodes adds {replaced}, above \ - the worst-case {raised}" - ); - } - } - } - - /// The raise only ever adds: an estimate above what a replacement adds - /// stays as it is, and an estimate that failed keeps its error and cost. - #[test] - fn the_raise_never_lowers_an_estimate_or_hides_its_error() { - let grove_version = GroveVersion::latest(); - let options = replacement_options(); - let key = KeyInfo::KnownKey(b"item".to_vec()); - let replacement = super::ReplacementAccounting::new( - &options, - &key, - NodeType::NormalNode, - super::CarriedReferrers::Worst, - grove_version, - ); - let element = Element::new_item_with_flags(vec![7; 20], Some(vec![0; 35])); - let added = |added_bytes| OperationCost { - storage_cost: grovedb_costs::storage_cost::StorageCost { - added_bytes, - ..Default::default() - }, - ..Default::default() - }; - let replacement_bytes = raised(&replacement, &element); - assert!(replacement_bytes > 0); - - let above = replacement_bytes + 1000; - let kept = replacement.raise(Ok(()).wrap_with_cost(added(above)), &element); - assert_eq!(kept.cost.storage_cost.added_bytes, above); - - let failed = replacement.raise( - Err(crate::Error::InternalError("estimate".to_string())).wrap_with_cost(added(1)), - &element, - ); - assert_eq!(failed.cost.storage_cost.added_bytes, 1); - assert!(failed.value.is_err()); - } - - /// Only a trusted refresh can be a replacement, so only it is raised. - #[test] - fn only_a_trusted_refresh_is_raised() { - use crate::batch::RefreshReferenceMode; - - let grove_version = GroveVersion::latest(); - let options = replacement_options(); - let key = KeyInfo::KnownKey(b"reference".to_vec()); - let replacement = super::ReplacementAccounting::new( - &options, - &key, - NodeType::NormalNode, - super::CarriedReferrers::Worst, - grove_version, - ); - let element = Element::new_reference_with_flags( - ReferencePathType::AbsolutePathReference(vec![b"target".to_vec()]), - Some(vec![0; 35]), - ); - let replacement_bytes = raised(&replacement, &element); - assert!(replacement_bytes > 0); - for (mode, expected) in [ - ( - RefreshReferenceMode::PlainReferenceTrusted, - replacement_bytes, - ), - ( - RefreshReferenceMode::SumItemReferenceTrusted(5), - replacement_bytes, - ), - (RefreshReferenceMode::PlainReferenceUntrusted, 0), - ( - RefreshReferenceMode::SumItemReferenceUntrustedValueUpdate(5), - 0, - ), - ( - RefreshReferenceMode::SumItemReferenceUntrustedNoValueUpdate, - 0, - ), - ] { - let added_bytes = replacement - .raise_refresh( - Ok(()).wrap_with_cost(OperationCost::default()), - &element, - &mode, - ) - .cost - .storage_cost - .added_bytes; - assert_eq!(added_bytes, expected, "{mode:?}"); - } - } - - /// A replaced tree is raised to exactly the bytes of the element its - /// `InsertTreeWithRootHash` writes, for every Merk tree type and wrapper. - #[test] - fn a_replaced_tree_is_raised_to_the_bytes_of_the_element_it_writes() { - let grove_version = GroveVersion::latest(); - let options = replacement_options(); - let key = KeyInfo::KnownKey(b"tree".to_vec()); - let trees: [(TreeType, fn(Option>) -> Element); 9] = [ - (TreeType::NormalTree, Element::empty_tree_with_flags), - (TreeType::SumTree, Element::empty_sum_tree_with_flags), - (TreeType::BigSumTree, Element::empty_big_sum_tree_with_flags), - (TreeType::CountTree, Element::empty_count_tree_with_flags), - ( - TreeType::CountSumTree, - Element::empty_count_sum_tree_with_flags, - ), - ( - TreeType::ProvableCountTree, - Element::empty_provable_count_tree_with_flags, - ), - ( - TreeType::ProvableCountSumTree, - Element::empty_provable_count_sum_tree_with_flags, - ), - ( - TreeType::ProvableSumTree, - Element::empty_provable_sum_tree_with_flags, - ), - ( - TreeType::ProvableCountProvableSumTree, - Element::empty_provable_count_provable_sum_tree_with_flags, - ), - ]; - for (tree_type, tree) in trees { - // 200 bytes of flags take a two-byte length prefix. - for flags_len in [1, 35, 200] { - let flags = Some(vec![0; flags_len]); - let plain = || tree(flags.clone()); - for node_type in [NodeType::NormalNode, NodeType::CountSumNode] { - let replacement = super::ReplacementAccounting::new( - &options, - &key, - node_type, - super::CarriedReferrers::Worst, - grove_version, - ); - for (non_counted, not_summed, not_counted_or_summed, element) in [ - (false, false, false, plain()), - (true, false, false, Element::NonCounted(Box::new(plain()))), - (false, true, false, Element::NotSummed(Box::new(plain()))), - ( - false, - false, - true, - Element::NotCountedOrSummed(Box::new(plain())), - ), - ] { - let by_tree_type = replacement - .raise_tree( - Ok(()).wrap_with_cost(OperationCost::default()), - tree_type, - &flags, - non_counted, - not_summed, - not_counted_or_summed, - ) - .cost - .storage_cost - .added_bytes; - assert_eq!( - by_tree_type, - raised(&replacement, &element), - "{tree_type:?}, {flags_len} flag bytes, {node_type:?}, {element:?}" - ); - } - } - } - } - } - - /// A backward-references element lands with the referrers of the - /// element it replaces, so the raise sizes it without the - /// entries it was written with: the worst case adds its declared - /// capacity of the largest entries, the average case the typical - /// number of entries at the size it is given. - #[test] - fn a_replaced_backward_references_item_is_sized_by_its_carried_referrers() { - let grove_version = GroveVersion::latest(); - let options = replacement_options(); - let key = KeyInfo::KnownKey(b"item".to_vec()); - let flags = Some(vec![0; 35]); - let item = |entries: Vec| { - Element::ItemWithBackwardsReferences( - vec![7; 20], - BackwardReferences::new(32, entries), - flags.clone(), - ) - }; - let entry = |key: &[u8]| BackwardReference { - inverted_reference: ReferencePathType::SiblingReference(key.to_vec()), - cascade_on_update: false, - }; - let without_entries = item(vec![]); - let with_entries = item(vec![entry(b"r1"), entry(b"r2"), entry(b"r3")]); - let node_type = NodeType::NormalNode; - let entry_bytes = 40; - let worst = super::ReplacementAccounting::new( - &options, - &key, - node_type, - super::CarriedReferrers::Worst, - grove_version, - ); - let average = super::ReplacementAccounting::new( - &options, - &key, - node_type, - super::CarriedReferrers::Average { entry_bytes }, - grove_version, - ); - for replacement in [&worst, &average] { - assert_eq!( - raised(replacement, &with_entries), - raised(replacement, &without_entries), - "the entries the item was written with are not the ones it lands with" - ); - } - let element_bytes = without_entries.serialized_size(grove_version).unwrap() as u32; - let node_bytes = |raw_value_len| { - KV::node_byte_cost_size_for_key_and_raw_value_lengths(4, raw_value_len, node_type) - }; - assert_eq!( - raised(&worst, &without_entries), - node_bytes(element_bytes + 32 * super::WORST_CASE_BACKWARD_REFERENCE_ENTRY_BOUND) - ); - assert_eq!( - raised(&average, &without_entries), - node_bytes( - element_bytes + super::BACKWARD_REFERENCES_AVERAGE_ITEM_FAN_OUT * entry_bytes - ) - ); - } -} diff --git a/grovedb/src/batch/estimated_costs/worst_case_costs.rs b/grovedb/src/batch/estimated_costs/worst_case_costs.rs index 8161b4861..ee5c2919f 100644 --- a/grovedb/src/batch/estimated_costs/worst_case_costs.rs +++ b/grovedb/src/batch/estimated_costs/worst_case_costs.rs @@ -43,34 +43,7 @@ use crate::{BackwardsReferences, Element}; #[cfg(feature = "minimal")] impl GroveOp { - /// [`Self::worst_case_cost_with_options`] with the default batch - /// options. - #[cfg(test)] - #[allow(clippy::too_many_arguments)] fn worst_case_cost( - &self, - path: &KeyInfoPath, - key: &KeyInfo, - in_parent_tree_type: TreeType, - worst_case_layer_element_estimates: &WorstCaseLayerInformation, - backwards_references: BackwardsReferences, - propagate: bool, - grove_version: &GroveVersion, - ) -> CostResult<(), Error> { - self.worst_case_cost_with_options( - path, - key, - in_parent_tree_type, - worst_case_layer_element_estimates, - backwards_references, - propagate, - &BatchApplyOptions::default(), - grove_version, - ) - } - - #[allow(clippy::too_many_arguments)] - fn worst_case_cost_with_options( &self, // The op's own path: sizes the inverted-registration growth bound // (every `invert()` output is built from the origin's qualified @@ -85,10 +58,6 @@ impl GroveOp { // it declares `Check`. backwards_references: BackwardsReferences, propagate: bool, - // The batch's options: with `allow_replacement_accounting`, a write that may - // be accounted as a replacement is charged as the insertion it then - // records too. - batch_apply_options: &BatchApplyOptions, grove_version: &GroveVersion, ) -> CostResult<(), Error> { let propagate_if_input = || { @@ -98,13 +67,6 @@ impl GroveOp { None } }; - let replacement = super::ReplacementAccounting::new( - batch_apply_options, - key, - super::WORST_CASE_REPLACEMENT_NODE_TYPE, - super::CarriedReferrers::Worst, - grove_version, - ); let fan_out_version = grove_version .grovedb_versions .operations @@ -118,10 +80,20 @@ impl GroveOp { } match element { Some(Element::BidirectionalReference(..)) => { - // The registration entry appended to the target, whose - // referrer is this op itself. + // The registration entry appended to the target: an + // inverted path built from the referrer's qualified + // origin (this op's path segments plus its key — an + // absolute inversion serializes them all), the cascade + // flag, and framing. + let origin_bytes: u32 = path + .0 + .iter() + .map(|segment| 4 + segment.max_length() as u32) + .sum::() + .saturating_add(4 + key.max_length() as u32); + let entry_bound = origin_bytes.saturating_add(16); Some(super::BackwardReferencesFanOut::worst_reference( - super::backward_reference_entry_bound(path, key), + entry_bound, )) } // A backward-references ITEM write carries its own referrer @@ -227,40 +199,19 @@ impl GroveOp { not_summed, not_counted_or_summed, .. - } => replacement.raise_tree( - GroveDb::worst_case_merk_insert_tree( - key, - flags, - aggregate_data.parent_tree_type(), - in_parent_tree_type, - // See the comment in the corresponding average-case arm. - super::wrapper_overhead_for(*non_counted, *not_summed, *not_counted_or_summed), - propagate_if_input(), - grove_version, - ), - // The write of a tree the batch also writes under may be - // accounted as a replacement as a write of its element is. - super::WORST_CASE_REPLACEMENT_TREE_TYPE, + } => GroveDb::worst_case_merk_insert_tree( + key, flags, - *non_counted, - *not_summed, - *not_counted_or_summed, + aggregate_data.parent_tree_type(), + in_parent_tree_type, + // See the comment in the corresponding average-case arm. + super::wrapper_overhead_for(*non_counted, *not_summed, *not_counted_or_summed), + propagate_if_input(), + grove_version, ), GroveOp::InsertOrReplace { element } - | GroveOp::InsertOrReplaceDontCheckForBackwardsReferences { element } => with_fan_out( - replacement.raise( - GroveDb::worst_case_merk_insert_element( - key, - element, - in_parent_tree_type, - propagate_if_input(), - grove_version, - ), - element, - ), - backward_references_fan_out(Some(element)), - ), - GroveOp::InsertWithKnownToNotAlreadyExist { element } => with_fan_out( + | GroveOp::InsertOrReplaceDontCheckForBackwardsReferences { element } + | GroveOp::InsertWithKnownToNotAlreadyExist { element } => with_fan_out( GroveDb::worst_case_merk_insert_element( key, element, @@ -335,26 +286,22 @@ impl GroveOp { } else { inner }; - let replace_cost = GroveDb::worst_case_merk_replace_element( + GroveDb::worst_case_merk_replace_element( key, &element, in_parent_tree_type, propagate_if_input(), grove_version, - ); - replacement.raise_refresh(replace_cost, &element, mode) + ) } GroveOp::Replace { element } | GroveOp::ReplaceDontCheckForBackwardsReferences { element } => with_fan_out( - replacement.raise( - GroveDb::worst_case_merk_replace_element( - key, - element, - in_parent_tree_type, - propagate_if_input(), - grove_version, - ), + GroveDb::worst_case_merk_replace_element( + key, element, + in_parent_tree_type, + propagate_if_input(), + grove_version, ), backward_references_fan_out(Some(element)), ), @@ -366,15 +313,12 @@ impl GroveOp { element, change_in_bytes: _, } => with_fan_out( - replacement.raise( - GroveDb::worst_case_merk_replace_element( - key, - element, - in_parent_tree_type, - propagate_if_input(), - grove_version, - ), + GroveDb::worst_case_merk_replace_element( + key, element, + in_parent_tree_type, + propagate_if_input(), + grove_version, ), backward_references_fan_out(Some(element)), ), @@ -911,7 +855,7 @@ impl TreeCache for WorstCaseTreeCacheKnownPaths { path: &KeyInfoPath, ops_at_path_by_key: BTreeMap, _ops_by_qualified_paths: &BTreeMap>, GroveOp>, - batch_apply_options: &BatchApplyOptions, + _batch_apply_options: &BatchApplyOptions, _flags_update: &mut G, _split_removal_bytes: &mut SR, grove_version: &GroveVersion, @@ -961,14 +905,13 @@ impl TreeCache for WorstCaseTreeCacheKnownPaths { } cost_return_on_error!( &mut cost, - op.worst_case_cost_with_options( + op.worst_case_cost( path, &key, TreeType::NormalTree, worst_case_layer_element_estimates, op.backwards_references(), false, - batch_apply_options, grove_version ) ); diff --git a/grovedb/src/batch/just_in_time_value_update.rs b/grovedb/src/batch/just_in_time_value_update.rs index 7f1166e0a..dcff21ff1 100644 --- a/grovedb/src/batch/just_in_time_value_update.rs +++ b/grovedb/src/batch/just_in_time_value_update.rs @@ -37,17 +37,10 @@ use integer_encoding::VarInt; use crate::{Element, ElementFlags, Error}; -/// Why a batch whose flags update answered `AccountAsReplacement` was -/// refused: its options do not set `allow_replacement_accounting`. -pub(crate) const REPLACEMENT_ACCOUNTING_NOT_ALLOWED: &str = - "the flags update answered AccountAsReplacement, but the batch options do not set \ - allow_replacement_accounting"; - /// Why a batch whose flags update answered `AccountAsReplacement` was /// refused: the write's flags are byte-equal to the stored ones. An untrusted /// reference refresh and a tree's root update write the stored flags back, so -/// they bring no flags of their own to account for and the estimators do not -/// charge them as replacements. The callback cannot tell those from a write +/// they bring no flags of their own to account for. The callback cannot tell those from a write /// that happens to bring identical flags, which hands nothing over anyway, so /// that one is refused too. pub(crate) const REPLACED_A_WRITE_KEEPING_ITS_FLAGS: &str = @@ -55,8 +48,7 @@ pub(crate) const REPLACED_A_WRITE_KEEPING_ITS_FLAGS: &str = /// The caller's flags-update callback as a batch runs it: its answer as an /// [`ElementFlagsUpdate`] (a `bool` answer is `Changed` or `Unchanged`). -/// `AccountAsReplacement` is refused unless the batch's options set -/// `allow_replacement_accounting`, and refused on a write whose flags are +/// `AccountAsReplacement` is refused on a write whose flags are /// byte-equal to the stored ones, whatever the callback does with them: only /// a write that brings different flags can be accounted so. A refusal records /// its reason in `refused`, so the batch reports it with @@ -67,7 +59,6 @@ pub(crate) fn batch_flags_update( Option, &mut ElementFlags, ) -> Result, - allow_replacement_accounting: bool, refused: &Cell>, ) -> impl FnMut(&StorageCost, Option, &mut ElementFlags) -> Result where @@ -76,23 +67,14 @@ where move |storage_cost: &StorageCost, old_flags: Option, new_flags: &mut ElementFlags| { - // Compared before the callback, which may rewrite the flags. Without - // the option any such answer is refused anyway. - let keeps_stored_flags = - allow_replacement_accounting && old_flags.as_deref() == Some(new_flags.as_slice()); + // Compared before the callback, which may rewrite the flags. + let keeps_stored_flags = old_flags.as_deref() == Some(new_flags.as_slice()); let update = flags_update(storage_cost, old_flags, new_flags)?.into(); - if update == ElementFlagsUpdate::AccountAsReplacement { - let refusal = if !allow_replacement_accounting { - Some(REPLACEMENT_ACCOUNTING_NOT_ALLOWED) - } else if keeps_stored_flags { - Some(REPLACED_A_WRITE_KEEPING_ITS_FLAGS) - } else { - None - }; - if let Some(reason) = refusal { - refused.set(Some(reason)); - return Err(Error::JustInTimeElementFlagsClientError(reason.to_owned())); - } + if update == ElementFlagsUpdate::AccountAsReplacement && keeps_stored_flags { + refused.set(Some(REPLACED_A_WRITE_KEEPING_ITS_FLAGS)); + return Err(Error::JustInTimeElementFlagsClientError( + REPLACED_A_WRITE_KEEPING_ITS_FLAGS.to_owned(), + )); } Ok(update) } diff --git a/grovedb/src/batch/mod.rs b/grovedb/src/batch/mod.rs index d1508c4db..02e926ada 100644 --- a/grovedb/src/batch/mod.rs +++ b/grovedb/src/batch/mod.rs @@ -66,8 +66,8 @@ use estimated_costs::{ worst_case_costs::WorstCaseTreeCacheKnownPaths, }; use grovedb_costs::{ - cost_return_on_error, cost_return_on_error_default, cost_return_on_error_into, - cost_return_on_error_into_no_add, cost_return_on_error_no_add, + cost_return_on_error, cost_return_on_error_into, cost_return_on_error_into_no_add, + cost_return_on_error_no_add, storage_cost::{ removal::{StorageRemovedBytes, StorageRemovedBytes::BasicStorageRemoval}, transition::ElementFlagsUpdate, @@ -6247,10 +6247,6 @@ impl GroveDb { .apply_batch .apply_operations_without_batching ); - cost_return_on_error_default!(BatchApplyOptions::check_replacement_accounting_version( - &options, - grove_version, - )); let mut cost = OperationCost::default(); for op in ops.into_iter() { let declared = op.op.backwards_references(); @@ -7281,10 +7277,8 @@ impl GroveDb { /// batch if the apply stores anything else. /// /// The flags callback answers a `bool` (whether it rewrote the flags) or - /// an [`ElementFlagsUpdate`]; `AccountAsReplacement` is accepted only when - /// the options set - /// [`allow_replacement_accounting`](BatchApplyOptions::allow_replacement_accounting); - /// without it the batch fails with [`Error::InvalidBatchOperation`]. + /// an [`ElementFlagsUpdate`]; `AccountAsReplacement` is accepted from grove + /// version 5, and refused before. pub fn apply_batch_with_element_flags_update>( &self, ops: Vec, @@ -7305,14 +7299,9 @@ impl GroveDb { transaction: TransactionArg, grove_version: &GroveVersion, ) -> CostResult<(), Error> { - cost_return_on_error_default!(BatchApplyOptions::check_replacement_accounting_version( - &batch_apply_options, - grove_version, - )); let replacement_refused = Cell::new(None); let update_element_flags_function = just_in_time_value_update::batch_flags_update( update_element_flags_function, - BatchApplyOptions::allow_replacement_accounting_in(&batch_apply_options), &replacement_refused, ); self.apply_batch_with_batch_flags_update( @@ -7870,14 +7859,9 @@ impl GroveDb { transaction: TransactionArg, grove_version: &GroveVersion, ) -> CostResult<(), Error> { - cost_return_on_error_default!(BatchApplyOptions::check_replacement_accounting_version( - &batch_apply_options, - grove_version, - )); let replacement_refused = Cell::new(None); let update_element_flags_function = just_in_time_value_update::batch_flags_update( update_element_flags_function, - BatchApplyOptions::allow_replacement_accounting_in(&batch_apply_options), &replacement_refused, ); self.apply_partial_batch_with_batch_flags_update( @@ -8680,10 +8664,6 @@ impl GroveDb { #[cfg(feature = "estimated_costs")] /// Returns the estimated average or worst case cost for an entire batch of /// ops - /// - /// With [`allow_replacement_accounting`](BatchApplyOptions::allow_replacement_accounting) - /// set, every write that may be accounted as a replacement is charged at - /// least the bytes its element adds when inserted. pub fn estimated_case_operations_for_batch>( estimated_costs_type: EstimatedCostsType, ops: Vec, @@ -8710,13 +8690,8 @@ impl GroveDb { .apply_batch .estimated_case_operations_for_batch ); - cost_return_on_error_default!(BatchApplyOptions::check_replacement_accounting_version( - &batch_apply_options, - grove_version, - )); - // The estimators never call the flags update (an estimate follows - // the batch options, not the answers the flags update would give), - // so it only needs its answer type. + // The estimators never call the flags update, so it only needs its + // answer type. let update_element_flags_function = just_in_time_value_update::flags_update_answers(update_element_flags_function); let mut cost = OperationCost::default(); @@ -8963,7 +8938,6 @@ mod tests { disable_operation_consistency_check: true, base_root_storage_is_free: true, batch_pause_height: None, - allow_replacement_accounting: false, }), None, grove_version @@ -9569,7 +9543,6 @@ mod tests { disable_operation_consistency_check: false, base_root_storage_is_free: true, batch_pause_height: None, - allow_replacement_accounting: false, }), None, grove_version @@ -9611,7 +9584,6 @@ mod tests { validate_insertion_does_not_override: true, base_root_storage_is_free: true, batch_pause_height: None, - allow_replacement_accounting: false, }), None, grove_version @@ -9645,7 +9617,6 @@ mod tests { disable_operation_consistency_check: false, base_root_storage_is_free: true, batch_pause_height: None, - allow_replacement_accounting: false, }), None, grove_version diff --git a/grovedb/src/batch/options.rs b/grovedb/src/batch/options.rs index abc4d3b0d..fb88e97f4 100644 --- a/grovedb/src/batch/options.rs +++ b/grovedb/src/batch/options.rs @@ -6,10 +6,6 @@ use grovedb_merk::MerkOptions; #[cfg(feature = "minimal")] use crate::operations::{delete::DeleteOptions, insert::InsertOptions}; -#[cfg(feature = "minimal")] -use crate::Error; -#[cfg(feature = "minimal")] -use grovedb_version::{error::GroveVersionError, version::GroveVersion}; /// Batch apply options #[cfg(feature = "minimal")] @@ -82,42 +78,6 @@ pub struct BatchApplyOptions { /// At what height do we want to pause applying batch operations /// Most of the time this should be not set pub batch_pause_height: Option, - /// Account an update in place whose flags update answers - /// [`ElementFlagsUpdate::AccountAsReplacement`](grovedb_costs::storage_cost::transition::ElementFlagsUpdate::AccountAsReplacement) - /// as the removal of the old element plus the insertion of the new one. - /// - /// The old element's bytes, key included, count as removed and are - /// sectioned by the removal-bytes callback as a deletion sections them; - /// the new element's bytes, key included, count as added, and it keeps - /// the flags the callback leaves it with. A batch that does not set this - /// refuses a flags update that answers `AccountAsReplacement` with - /// [`Error::InvalidBatchOperation`](crate::Error::InvalidBatchOperation). - /// - /// The estimators read it too: an estimate with it set charges every - /// write that may be accounted so (`InsertOrReplace`, `Replace`, - /// `Patch` and a trusted `RefreshReference` of an element with flags, and - /// the write of a flagged tree the batch also writes under) at least the - /// bytes its element adds when inserted. The referrer entries a - /// backward-references element carries over are counted, in the worst - /// case, at its declared capacity, each at the largest entry - /// registration admits, so a worst-case estimate of a write it charges - /// so is never below what the apply records; the average case counts the - /// typical shape (the average item fan-out, each entry the size of one - /// from the write's own position). A write the caller declares new - /// (`InsertIfNotExists`, `InsertWithKnownToNotAlreadyExist`) is not charged - /// so, and its estimate holds only while the key is in fact new. - /// - /// An estimate cannot see owners, so it charges those writes whether or - /// not the update will change an owner. Nothing ties an estimate to its - /// apply: pass the same value to both, because an apply that sets this - /// over an estimate that does not can record more than was estimated. - /// - /// Off by default, and every cost, estimate, stored element and hash is - /// then what it was before the option existed. Setting it needs - /// `apply_batch.replacement_accounting` (GROVE_V5+); on an earlier grove - /// version the batch and its estimate fail with - /// [`Error::InvalidBatchOperation`](crate::Error::InvalidBatchOperation). - pub allow_replacement_accounting: bool, } #[cfg(feature = "minimal")] @@ -129,48 +89,12 @@ impl Default for BatchApplyOptions { disable_operation_consistency_check: false, base_root_storage_is_free: true, batch_pause_height: None, - allow_replacement_accounting: false, } } } #[cfg(feature = "minimal")] impl BatchApplyOptions { - /// Whether the batch options `options`, when given, set - /// [`Self::allow_replacement_accounting`]. - pub(crate) fn allow_replacement_accounting_in(options: &Option) -> bool { - options - .as_ref() - .is_some_and(|options| options.allow_replacement_accounting) - } - - /// Refuses options that set [`Self::allow_replacement_accounting`] on a - /// grove version that does not have it. - pub(crate) fn check_replacement_accounting_version( - options: &Option, - grove_version: &GroveVersion, - ) -> Result<(), Error> { - if !Self::allow_replacement_accounting_in(options) { - return Ok(()); - } - match grove_version - .grovedb_versions - .apply_batch - .replacement_accounting - { - 0 => Err(Error::InvalidBatchOperation( - "replacement accounting needs grove version 5 or later", - )), - 1 => Ok(()), - version => Err(GroveVersionError::UnknownVersionMismatch { - method: "check_replacement_accounting_version".to_string(), - known_versions: vec![0, 1], - received: version, - } - .into()), - } - } - /// As insert options pub(crate) fn as_insert_options(&self) -> InsertOptions { InsertOptions { diff --git a/grovedb/src/batch/replacement_accounting_cost_tests.rs b/grovedb/src/batch/replacement_accounting_cost_tests.rs index d1de9c206..bfe69a9b8 100644 --- a/grovedb/src/batch/replacement_accounting_cost_tests.rs +++ b/grovedb/src/batch/replacement_accounting_cost_tests.rs @@ -1,49 +1,31 @@ //! Costs of an update in place accounted as a replacement -//! (`BatchApplyOptions::allow_replacement_accounting`): the old element is removed +//! (`ElementFlagsUpdate::AccountAsReplacement`): the old element is removed //! and its bytes sectioned to its owner as a deletion sections them, and the //! new element is inserted and charged to the writer. #[cfg(feature = "minimal")] mod tests { - use std::collections::{BTreeMap, HashMap}; + use std::collections::BTreeMap; use grovedb_costs::{ - storage_cost::removal::StorageRemovedBytes::{NoStorageRemoval, SectionedStorageRemoval}, - OperationCost, + storage_cost::removal::StorageRemovedBytes::SectionedStorageRemoval, OperationCost, }; use grovedb_epoch_based_storage_flags::StorageFlags; use grovedb_merk::{ - element::{get::ElementFetchFromStorageExtensions, tree_type::ElementTreeTypeExtensions}, - estimated_costs::{ - average_case_costs::{ - EstimatedLayerCount::ApproximateElements, - EstimatedLayerInformation, - EstimatedLayerSizes::{AllItems, AllSubtrees}, - EstimatedSumTrees::NoSumTrees, - }, - worst_case_costs::WorstCaseLayerInformation::MaxElementsNumber, - }, - tree::kv::KV, - tree_type::TreeType, - }; - use grovedb_version::{ - error::GroveVersionError, - version::{v3::GROVE_V3, v4::GROVE_V4, GroveVersion}, + element::tree_type::ElementTreeTypeExtensions, tree::kv::KV, tree_type::TreeType, }; + use grovedb_version::version::{v3::GROVE_V3, v4::GROVE_V4, GroveVersion}; use intmap::IntMap; use crate::batch::replacement_test_support::{ - apply, merging_flags_update, options, owned_flags, replacing_flags_update, - split_removal_bytes, unowned_flags, value_bytes, write_op, Mode, KEY, NEW_OWNER, OLD_OWNER, + apply, owned_flags, replacing_flags_update, split_removal_bytes, unowned_flags, + value_bytes, write_op, Mode, KEY, NEW_OWNER, OLD_OWNER, }; use crate::{ - batch::{ - estimated_costs::EstimatedCostsType::{AverageCaseCostsType, WorstCaseCostsType}, - BatchApplyOptions, KeyInfoPath, QualifiedGroveDbOp, SubelementsDeletionBehavior, - }, + batch::{QualifiedGroveDbOp, SubelementsDeletionBehavior}, reference_path::ReferencePathType, tests::{common::EMPTY_PATH, make_empty_grovedb, TempGroveDb}, - BackwardReferences, Element, Error, GroveDb, + Element, Error, }; fn target_path() -> ReferencePathType { @@ -137,24 +119,6 @@ mod tests { .expect("expected the element") } - /// The element at `key1` as its Merk stores it, a backward-references - /// element's referrer list included (public reads strip it). - fn stored_in_merk(db: &TempGroveDb, grove_version: &GroveVersion) -> Element { - let tx = db.start_transaction(); - let merk = db - .open_transactional_merk_at_path( - [b"tree".as_slice()].as_ref().into(), - &tx, - None, - grove_version, - ) - .unwrap() - .expect("expected to open the tree"); - Element::get(&merk, KEY, true, grove_version) - .unwrap() - .expect("expected the element") - } - fn storage_flags_of(element: &Element) -> StorageFlags { StorageFlags::from_element_flags_ref(element.get_flags().as_ref().expect("expected flags")) .expect("expected valid flags") @@ -303,29 +267,6 @@ mod tests { assert_replaces(&old, &new, TreeType::NormalTree, grove_version); } - /// The option needs grove v5: an earlier version refuses a batch or an - /// estimate that sets it, before anything is written. - #[test] - fn replacement_accounting_refuses_a_slot_it_does_not_know() { - let mut grove_version = GroveVersion::latest().clone(); - grove_version - .grovedb_versions - .apply_batch - .replacement_accounting = 2; - let error = BatchApplyOptions::check_replacement_accounting_version( - &Some(options(Mode::Replacing)), - &grove_version, - ) - .expect_err("expected an unknown slot to be refused"); - assert!( - matches!( - error, - Error::VersionError(GroveVersionError::UnknownVersionMismatch { received: 2, .. }) - ), - "unexpected error {error}" - ); - } - #[test] fn replacement_accounting_is_refused_below_grove_v5() { for grove_version in [&GROVE_V3, &GROVE_V4] { @@ -336,7 +277,7 @@ mod tests { let applied = apply(&db, vec![write_op(&new)], Mode::Replacing, grove_version) .expect_err("expected the batch to be refused"); assert!( - matches!(applied, Error::InvalidBatchOperation(_)), + applied.to_string().contains("version 2"), "unexpected error {applied}" ); assert_eq!(stored(&db, grove_version), old); @@ -344,7 +285,7 @@ mod tests { let partial = db .apply_partial_batch_with_element_flags_update( vec![write_op(&new)], - Some(options(Mode::Replacing)), + None, replacing_flags_update, split_removal_bytes, |_cost, _ops| Ok(vec![]), @@ -354,41 +295,10 @@ mod tests { .cost_as_result() .expect_err("expected the partial batch to be refused"); assert!( - matches!(partial, Error::InvalidBatchOperation(_)), + partial.to_string().contains("version 2"), "unexpected error {partial}" ); - - let unbatched = db - .apply_operations_without_batching( - vec![write_op(&new)], - Some(options(Mode::Replacing)), - None, - grove_version, - ) - .cost_as_result() - .expect_err("expected the unbatched operations to be refused"); - assert!( - matches!(unbatched, Error::InvalidBatchOperation(_)), - "unexpected error {unbatched}" - ); assert_eq!(stored(&db, grove_version), old); - - let estimated = GroveDb::estimated_case_operations_for_batch( - WorstCaseCostsType(worst_case_layers()), - vec![write_op(&new)], - Some(options(Mode::Replacing)), - |_cost, _old_flags, _new_flags| Ok(false), - |_flags, _removed_key_bytes, _removed_value_bytes| { - Ok((NoStorageRemoval, NoStorageRemoval)) - }, - grove_version, - ) - .cost_as_result() - .expect_err("expected the estimate to be refused"); - assert!( - matches!(estimated, Error::InvalidBatchOperation(_)), - "unexpected error {estimated}" - ); } } @@ -468,140 +378,6 @@ mod tests { } } - /// A backward-references item keeps the referrers registered on the item - /// it replaces, and a replacement write adds their entries too. The - /// worst-case estimate covers them through the item's declared capacity, - /// wherever the referrers sit; the average case charges the typical - /// shape, which covers referrers beside the item. - #[test] - fn estimates_cover_the_referrers_a_replaced_item_carries_over() { - let grove_version = GroveVersion::latest(); - let family_item = |value: Vec, flags| { - Element::ItemWithBackwardsReferences( - value, - BackwardReferences::with_max_incoming(4), - flags, - ) - }; - let old = family_item(vec![7; 20], owned_flags(0, OLD_OWNER)); - let new = family_item(vec![8; 20], owned_flags(2, NEW_OWNER)); - let siblings = |db: &TempGroveDb| { - for referrer in [b"r1".as_slice(), b"r2", b"r3"] { - db.insert( - [b"tree".as_slice()].as_ref(), - referrer, - Element::new_bidirectional_reference(ReferencePathType::SiblingReference( - KEY.to_vec(), - )), - None, - None, - grove_version, - ) - .unwrap() - .expect("expected to register a referrer"); - } - }; - // Referrers far from the item under long keys, whose entries hold - // absolute paths much longer than the item's own. - let distant = |db: &TempGroveDb| { - let (outer, inner) = (vec![b'x'; 200], vec![b'y'; 200]); - db.insert( - [b"targets".as_slice()].as_ref(), - &outer, - Element::empty_tree(), - None, - None, - grove_version, - ) - .unwrap() - .expect("expected to insert a tree"); - db.insert( - [b"targets".as_slice(), outer.as_slice()].as_ref(), - &inner, - Element::empty_tree(), - None, - None, - grove_version, - ) - .unwrap() - .expect("expected to insert a tree"); - for i in 0..3u8 { - let mut referrer = vec![b'z'; 199]; - referrer.push(i); - db.insert( - [b"targets".as_slice(), outer.as_slice(), inner.as_slice()].as_ref(), - &referrer, - Element::new_bidirectional_reference(ReferencePathType::AbsolutePathReference( - vec![b"tree".to_vec(), KEY.to_vec()], - )), - None, - None, - grove_version, - ) - .unwrap() - .expect("expected to register a referrer"); - } - }; - let placements: [(&str, &dyn Fn(&TempGroveDb)); 2] = - [("siblings", &siblings), ("distant", &distant)]; - for (placement, register_referrers) in placements { - let db = grove_with(&old, TreeType::NormalTree, grove_version); - register_referrers(&db); - let applied = apply(&db, vec![write_op(&new)], Mode::Replacing, grove_version) - .expect("expected the replacing update to apply"); - let landed = stored_in_merk(&db, grove_version); - assert_eq!( - landed.backward_references().map(|refs| refs.len()), - Some(3), - "{placement}: the item keeps its referrers" - ); - assert_eq!( - storage_flags_of(&landed), - StorageFlags::SingleEpochOwned(2, NEW_OWNER) - ); - // the referrer entries are part of what the replacement apply adds - assert_eq!( - applied.storage_cost.added_bytes, - key_bytes() + value_bytes(&landed, TreeType::NormalTree, grove_version) - ); - assert!( - applied.storage_cost.added_bytes - > key_bytes() + value_bytes(&new, TreeType::NormalTree, grove_version) - ); - let worst_case = estimate( - vec![write_op(&new)], - Some(options(Mode::Replacing)), - None, - grove_version, - ); - assert!( - worst_case.storage_cost.added_bytes >= applied.storage_cost.added_bytes, - "{placement}: worst-case estimate {worst_case:?} adds less than the settled \ - apply {applied:?}" - ); - // The average case charges the typical referrer shape: it covers - // referrers beside the item, and stays below the worst case. - let average_case = estimate( - vec![write_op(&new)], - Some(options(Mode::Replacing)), - Some(TreeType::NormalTree), - grove_version, - ); - assert!( - average_case.storage_cost.added_bytes < worst_case.storage_cost.added_bytes, - "{placement}: average-case estimate {average_case:?} is not below the worst \ - case {worst_case:?}" - ); - if placement == "siblings" { - assert!( - average_case.storage_cost.added_bytes >= applied.storage_cost.added_bytes, - "{placement}: average-case estimate {average_case:?} adds less than the \ - replacement apply {applied:?}" - ); - } - } - } - /// A partial batch accounts an owner change as a replacement as a full batch does, and /// hands its add-on callback the replacement cost. #[test] @@ -616,7 +392,7 @@ mod tests { let cost = db .apply_partial_batch_with_element_flags_update( vec![write_op(&new)], - Some(options(Mode::Replacing)), + None, replacing_flags_update, split_removal_bytes, |cost, _leftover_operations| { @@ -734,124 +510,6 @@ mod tests { } } - /// Without the option, an owner change is merged into the new owner's - /// flags as it always was: the new owner takes the old epochs and pays - /// only the growth, and the old owner is refunded nothing. - #[test] - fn owner_change_without_the_option_merges_as_before() { - let grove_version = GroveVersion::latest(); - let old = Element::new_item_with_flags(vec![7; 20], owned_flags(0, OLD_OWNER)); - let new = Element::new_item_with_flags(vec![8; 60], owned_flags(2, NEW_OWNER)); - let db = grove_with(&old, TreeType::NormalTree, grove_version); - let cost = apply(&db, vec![write_op(&new)], Mode::Merging, grove_version) - .expect("expected the merging update to apply"); - assert_eq!(cost.storage_cost.removed_bytes, NoStorageRemoval); - let StorageFlags::MultiEpochOwned(0, epochs, NEW_OWNER) = - storage_flags_of(&stored(&db, grove_version)) - else { - panic!("expected the new owner to take the old owner's epochs"); - }; - assert_eq!(epochs.keys().copied().collect::>(), vec![2]); - let stored_element = stored(&db, grove_version); - assert_eq!( - cost.storage_cost.added_bytes, - value_bytes(&stored_element, TreeType::NormalTree, grove_version) - - value_bytes(&old, TreeType::NormalTree, grove_version) - ); - } - - /// The option alone changes nothing: a flags update that never answers it - /// is applied exactly as without it. - #[test] - fn the_option_alone_changes_nothing() { - let grove_version = GroveVersion::latest(); - let old = Element::new_item_with_flags(vec![7; 20], owned_flags(0, OLD_OWNER)); - for len in [60, 5, 20] { - let new = Element::new_item_with_flags(vec![8; len], owned_flags(2, NEW_OWNER)); - let mut costs = vec![]; - let mut hashes = vec![]; - for allow_replacement_accounting in [false, true] { - let db = grove_with(&old, TreeType::NormalTree, grove_version); - costs.push( - db.apply_batch_with_element_flags_update( - vec![write_op(&new)], - Some(BatchApplyOptions { - allow_replacement_accounting, - ..Default::default() - }), - merging_flags_update, - split_removal_bytes, - None, - grove_version, - ) - .cost_as_result() - .expect("expected the update to apply"), - ); - hashes.push(db.root_hash(None, grove_version).unwrap().unwrap()); - } - assert_eq!(costs[0], costs[1]); - assert_eq!(hashes[0], hashes[1]); - } - } - - /// A AccountAsReplacement answer without the option refuses the batch as an invalid - /// batch, whether the apply meets it first, a same-batch reference's - /// prediction does, or a partial batch's initial segment does. - #[test] - fn a_replacing_flags_update_is_refused_without_the_option() { - let grove_version = GroveVersion::latest(); - let old = Element::new_item_with_flags(vec![7; 20], owned_flags(0, OLD_OWNER)); - let new = Element::new_item_with_flags(vec![8; 60], owned_flags(2, NEW_OWNER)); - let reference = QualifiedGroveDbOp::insert_or_replace_op( - vec![b"targets".to_vec()], - b"reference".to_vec(), - Element::new_reference(ReferencePathType::AbsolutePathReference(vec![ - b"tree".to_vec(), - KEY.to_vec(), - ])), - ); - let assert_refused = |error: Error| { - assert!( - matches!(error, Error::InvalidBatchOperation(reason) if reason.contains("allow_replacement_accounting")), - "unexpected error: {error}" - ); - }; - for ops in [vec![write_op(&new)], vec![write_op(&new), reference]] { - let db = grove_with(&old, TreeType::NormalTree, grove_version); - let root_hash = db.root_hash(None, grove_version).unwrap().unwrap(); - assert_refused( - db.apply_batch_with_element_flags_update( - ops, - None, - replacing_flags_update, - split_removal_bytes, - None, - grove_version, - ) - .unwrap() - .expect_err("expected the AccountAsReplacement answer to be refused"), - ); - assert_eq!( - db.root_hash(None, grove_version).unwrap().unwrap(), - root_hash - ); - } - let db = grove_with(&old, TreeType::NormalTree, grove_version); - assert_refused( - db.apply_partial_batch_with_element_flags_update( - vec![write_op(&new)], - None, - replacing_flags_update, - split_removal_bytes, - |_cost, _leftover_operations| Ok(vec![]), - None, - grove_version, - ) - .unwrap() - .expect_err("expected the AccountAsReplacement answer to be refused"), - ); - } - /// A write that keeps the stored flags brings no owner of its own, so a /// AccountAsReplacement answer for it is refused, whatever the callback does with the /// flags: an untrusted refresh, which writes the stored reference's flags @@ -901,7 +559,7 @@ mod tests { false, false, )], - Some(options(Mode::Replacing)), + None, rewriting_to_a_new_owner, split_removal_bytes, None, @@ -939,7 +597,7 @@ mod tests { false, false, )], - Some(options(Mode::Replacing)), + None, rewriting_to_a_new_owner, split_removal_bytes, None, @@ -973,7 +631,7 @@ mod tests { b"child".to_vec(), Element::new_item(vec![1]), )], - Some(options(Mode::Replacing)), + None, always_replacing, split_removal_bytes, None, @@ -993,7 +651,7 @@ mod tests { let db = grove_with(&old, TreeType::NormalTree, grove_version); db.apply_batch_with_element_flags_update( vec![write_op(&new)], - Some(options(Mode::Replacing)), + None, always_replacing, split_removal_bytes, None, @@ -1009,7 +667,7 @@ mod tests { assert_refused( db.apply_batch_with_element_flags_update( vec![write_op(&same_flags)], - Some(options(Mode::Replacing)), + None, always_replacing, split_removal_bytes, None, @@ -1081,970 +739,4 @@ mod tests { assert!(issues.is_empty(), "reference issues: {issues:?}"); } } - - fn average_case_layers(tree_type: TreeType) -> HashMap { - let mut paths = HashMap::new(); - paths.insert( - KeyInfoPath(vec![]), - EstimatedLayerInformation { - tree_type: TreeType::NormalTree, - estimated_layer_count: ApproximateElements(2), - estimated_layer_sizes: AllSubtrees(7, NoSumTrees, None), - }, - ); - paths.insert( - KeyInfoPath::from_known_path([b"tree".as_slice()]), - EstimatedLayerInformation { - tree_type, - estimated_layer_count: ApproximateElements(2), - estimated_layer_sizes: AllItems(4, 100, Some(35)), - }, - ); - paths - } - - fn worst_case_layers() -> HashMap< - KeyInfoPath, - grovedb_merk::estimated_costs::worst_case_costs::WorstCaseLayerInformation, - > { - let mut paths = HashMap::new(); - paths.insert(KeyInfoPath(vec![]), MaxElementsNumber(2)); - paths.insert( - KeyInfoPath::from_known_path([b"tree".as_slice()]), - MaxElementsNumber(2), - ); - paths - } - - fn estimate( - ops: Vec, - batch_apply_options: Option, - average_case_tree_type: Option, - grove_version: &GroveVersion, - ) -> OperationCost { - let estimated_costs_type = match average_case_tree_type { - Some(tree_type) => AverageCaseCostsType(average_case_layers(tree_type)), - None => WorstCaseCostsType(worst_case_layers()), - }; - GroveDb::estimated_case_operations_for_batch( - estimated_costs_type, - ops, - batch_apply_options, - |_cost, _old_flags, _new_flags| Ok(false), - |_flags, _removed_key_bytes, _removed_value_bytes| { - Ok((NoStorageRemoval, NoStorageRemoval)) - }, - grove_version, - ) - .cost_as_result() - .expect("expected an estimate") - } - - /// Updates of every element kind an estimate may see replaced: the old - /// element, the new one, and the tree they are in. - fn estimate_cases() -> [(Element, Element, TreeType); 7] { - [ - ( - Element::new_item_with_flags(vec![7; 20], owned_flags(0, OLD_OWNER)), - Element::new_item_with_flags(vec![8; 60], owned_flags(2, NEW_OWNER)), - TreeType::NormalTree, - ), - ( - Element::new_item_with_flags(vec![7; 60], owned_flags(0, OLD_OWNER)), - Element::new_item_with_flags(vec![8; 5], owned_flags(2, NEW_OWNER)), - TreeType::NormalTree, - ), - ( - Element::new_item_with_sum_item_with_flags( - vec![7; 20], - 5, - owned_flags(0, OLD_OWNER), - ), - Element::new_item_with_sum_item_with_flags( - vec![8; 45], - 1, - owned_flags(2, NEW_OWNER), - ), - TreeType::SumTree, - ), - ( - Element::new_sum_item_with_flags(5, owned_flags(0, OLD_OWNER)), - Element::new_sum_item_with_flags(1, owned_flags(2, NEW_OWNER)), - TreeType::SumTree, - ), - ( - Element::new_reference_with_flags(target_path(), owned_flags(0, OLD_OWNER)), - Element::new_reference_with_flags(long_target_path(), owned_flags(2, NEW_OWNER)), - TreeType::NormalTree, - ), - ( - Element::new_reference_with_sum_item_with_flags( - target_path(), - 5, - owned_flags(0, OLD_OWNER), - ), - Element::new_reference_with_sum_item_with_flags( - long_target_path(), - 1, - owned_flags(2, NEW_OWNER), - ), - TreeType::SumTree, - ), - ( - Element::empty_tree_with_flags(owned_flags(0, OLD_OWNER)), - Element::empty_tree_with_flags(owned_flags(2, NEW_OWNER)), - TreeType::NormalTree, - ), - ] - } - - /// Every write of `new` at `key1` that may be accounted as a replacement. - fn replaceable_writes(new: &Element) -> Vec<(&'static str, QualifiedGroveDbOp)> { - let path = vec![b"tree".to_vec()]; - let mut ops = vec![ - ("insert_or_replace", write_op(new)), - ( - "insert_or_replace_dont_check", - write_op(new).dont_check_for_backwards_references(), - ), - ( - "replace", - QualifiedGroveDbOp::replace_op(path.clone(), KEY.to_vec(), new.clone()), - ), - ( - "replace_dont_check", - QualifiedGroveDbOp::replace_op(path.clone(), KEY.to_vec(), new.clone()) - .dont_check_for_backwards_references(), - ), - ]; - if let Element::Item(..) = new { - ops.push(( - "patch", - QualifiedGroveDbOp::patch_op(path.clone(), KEY.to_vec(), new.clone(), 0), - )); - ops.push(( - "patch_dont_check", - QualifiedGroveDbOp::patch_op(path.clone(), KEY.to_vec(), new.clone(), 0) - .dont_check_for_backwards_references(), - )); - } - if let Element::Reference(reference_path, max_hop, flags) = new { - ops.push(( - "refresh_trusted", - QualifiedGroveDbOp::refresh_reference_op( - path, - KEY.to_vec(), - reference_path.clone(), - *max_hop, - flags.clone(), - false, - true, - ), - )); - } - ops - } - - /// With the option, the average-case and worst-case estimates of every - /// write that may be accounted as a replacement are never below the replacement - /// apply of that same write. - #[test] - fn estimates_with_the_option_cover_a_replacement() { - let grove_version = GroveVersion::latest(); - for (old, new, tree_type) in estimate_cases() { - for (label, op) in replaceable_writes(&new) { - let db = grove_with(&old, tree_type, grove_version); - let applied = apply(&db, vec![op.clone()], Mode::Replacing, grove_version) - .unwrap_or_else(|e| panic!("expected {label} of {new:?} to apply: {e}")); - for average_case_tree_type in [Some(tree_type), None] { - let replacing = estimate( - vec![op.clone()], - Some(options(Mode::Replacing)), - average_case_tree_type, - grove_version, - ); - assert!( - replacing - .storage_cost - .worse_or_eq_than(&applied.storage_cost), - "estimate {replacing:?} of {label} of {new:?} is below the replacement apply \ - {applied:?}" - ); - } - } - } - } - - /// The estimates of those writes without the option on the latest grove - /// version, as they were before it existed: (case, write, average case, - /// seeks, added bytes, replaced bytes, loaded bytes, hash calls), taken - /// from develop at 0ea327d7. The worst-case rows whose replaced bytes - /// saturate at `u32::MAX` still pin every other figure. - const ESTIMATES_BEFORE_THE_OPTION: &[(usize, &str, bool, u32, u32, u32, u64, u32)] = &[ - (0, "insert_or_replace", true, 34, 243, 4289, 6170, 79), - ( - 0, - "insert_or_replace", - false, - 506892, - 243, - 4294967295, - 27857252525, - 87703315, - ), - ( - 0, - "insert_or_replace_dont_check", - true, - 13, - 243, - 1047, - 1379, - 23, - ), - ( - 0, - "insert_or_replace_dont_check", - false, - 12, - 243, - 409674, - 394925, - 275, - ), - (0, "replace", true, 34, 0, 4568, 6170, 80), - ( - 0, - "replace", - false, - 506892, - 206, - 4294967295, - 27857252525, - 87703315, - ), - (0, "replace_dont_check", true, 13, 0, 1326, 1379, 24), - (0, "replace_dont_check", false, 12, 206, 409711, 394925, 275), - (0, "patch", true, 34, 0, 4568, 6170, 80), - ( - 0, - "patch", - false, - 506892, - 206, - 4294967295, - 27857252525, - 87703315, - ), - (0, "patch_dont_check", true, 13, 0, 1326, 1379, 24), - (0, "patch_dont_check", false, 12, 206, 409711, 394925, 275), - (1, "insert_or_replace", true, 34, 187, 4289, 6170, 78), - ( - 1, - "insert_or_replace", - false, - 506892, - 187, - 4294967295, - 27857252525, - 87703314, - ), - ( - 1, - "insert_or_replace_dont_check", - true, - 13, - 187, - 1047, - 1379, - 22, - ), - ( - 1, - "insert_or_replace_dont_check", - false, - 12, - 187, - 409674, - 394925, - 274, - ), - (1, "replace", true, 34, 0, 4513, 6170, 79), - ( - 1, - "replace", - false, - 506892, - 150, - 4294967295, - 27857252525, - 87703314, - ), - (1, "replace_dont_check", true, 13, 0, 1271, 1379, 23), - (1, "replace_dont_check", false, 12, 150, 409711, 394925, 274), - (1, "patch", true, 34, 0, 4513, 6170, 79), - ( - 1, - "patch", - false, - 506892, - 150, - 4294967295, - 27857252525, - 87703314, - ), - (1, "patch_dont_check", true, 13, 0, 1271, 1379, 23), - (1, "patch_dont_check", false, 12, 150, 409711, 394925, 274), - (2, "insert_or_replace", true, 34, 253, 4554, 6578, 79), - ( - 2, - "insert_or_replace", - false, - 506892, - 237, - 4294967295, - 27857252525, - 87703315, - ), - ( - 2, - "insert_or_replace_dont_check", - true, - 13, - 253, - 1104, - 1451, - 23, - ), - ( - 2, - "insert_or_replace_dont_check", - false, - 12, - 237, - 409674, - 394925, - 275, - ), - (2, "replace", true, 34, 0, 4807, 6578, 79), - ( - 2, - "replace", - false, - 506892, - 200, - 4294967295, - 27857252525, - 87703315, - ), - (2, "replace_dont_check", true, 13, 0, 1357, 1451, 23), - (2, "replace_dont_check", false, 12, 200, 409711, 394925, 275), - (3, "insert_or_replace", true, 34, 206, 4554, 6578, 78), - ( - 3, - "insert_or_replace", - false, - 506892, - 190, - 4294967295, - 27857252525, - 87703314, - ), - ( - 3, - "insert_or_replace_dont_check", - true, - 13, - 206, - 1104, - 1451, - 22, - ), - ( - 3, - "insert_or_replace_dont_check", - false, - 12, - 190, - 409674, - 394925, - 274, - ), - (3, "replace", true, 34, 0, 4760, 6578, 78), - ( - 3, - "replace", - false, - 506892, - 0, - 4294967295, - 27857252525, - 87703314, - ), - (3, "replace_dont_check", true, 13, 0, 1310, 1451, 22), - (3, "replace_dont_check", false, 12, 0, 409864, 394925, 274), - (4, "insert_or_replace", true, 34, 223, 4289, 6170, 79), - ( - 4, - "insert_or_replace", - false, - 506892, - 223, - 4294967295, - 27857252525, - 87703315, - ), - ( - 4, - "insert_or_replace_dont_check", - true, - 13, - 223, - 1047, - 1379, - 23, - ), - ( - 4, - "insert_or_replace_dont_check", - false, - 12, - 223, - 409674, - 394925, - 275, - ), - (4, "replace", true, 34, 0, 4512, 6170, 79), - ( - 4, - "replace", - false, - 506892, - 186, - 4294967295, - 27857252525, - 87703315, - ), - (4, "replace_dont_check", true, 13, 0, 1270, 1379, 23), - (4, "replace_dont_check", false, 12, 186, 409711, 394925, 275), - (4, "refresh_trusted", true, 13, 0, 1270, 1379, 23), - (4, "refresh_trusted", false, 12, 186, 409711, 394925, 275), - (5, "insert_or_replace", true, 34, 240, 4554, 6578, 79), - ( - 5, - "insert_or_replace", - false, - 506892, - 224, - 4294967295, - 27857252525, - 87703315, - ), - ( - 5, - "insert_or_replace_dont_check", - true, - 13, - 240, - 1104, - 1451, - 23, - ), - ( - 5, - "insert_or_replace_dont_check", - false, - 12, - 224, - 409674, - 394925, - 275, - ), - (5, "replace", true, 34, 0, 4794, 6578, 79), - ( - 5, - "replace", - false, - 506892, - 187, - 4294967295, - 27857252525, - 87703315, - ), - (5, "replace_dont_check", true, 13, 0, 1344, 1451, 23), - (5, "replace_dont_check", false, 12, 187, 409711, 394925, 275), - (6, "insert_or_replace", true, 34, 151, 4289, 6170, 80), - ( - 6, - "insert_or_replace", - false, - 506892, - 151, - 4294967295, - 27857252525, - 87703316, - ), - ( - 6, - "insert_or_replace_dont_check", - true, - 13, - 151, - 1047, - 1379, - 24, - ), - ( - 6, - "insert_or_replace_dont_check", - false, - 12, - 151, - 409674, - 394925, - 276, - ), - (6, "replace", true, 34, 0, 4440, 6170, 80), - ( - 6, - "replace", - false, - 506892, - 0, - 4294967295, - 27857252525, - 87703316, - ), - (6, "replace_dont_check", true, 13, 0, 1198, 1379, 24), - (6, "replace_dont_check", false, 12, 0, 409825, 394925, 276), - ]; - - /// The same estimates on grove v3 (no backward-references fan-out), - /// taken from develop at 0ea327d7. - const GROVE_V3_ESTIMATES_BEFORE_THE_OPTION: &[(usize, &str, bool, u32, u32, u32, u64, u32)] = &[ - (0, "insert_or_replace", true, 13, 243, 1047, 1379, 23), - (0, "insert_or_replace", false, 12, 243, 409674, 394925, 275), - ( - 0, - "insert_or_replace_dont_check", - true, - 13, - 243, - 1047, - 1379, - 23, - ), - ( - 0, - "insert_or_replace_dont_check", - false, - 12, - 243, - 409674, - 394925, - 275, - ), - (0, "replace", true, 13, 0, 1326, 1379, 24), - (0, "replace", false, 12, 206, 409711, 394925, 275), - (0, "replace_dont_check", true, 13, 0, 1326, 1379, 24), - (0, "replace_dont_check", false, 12, 206, 409711, 394925, 275), - (0, "patch", true, 13, 0, 1326, 1379, 24), - (0, "patch", false, 12, 206, 409711, 394925, 275), - (0, "patch_dont_check", true, 13, 0, 1326, 1379, 24), - (0, "patch_dont_check", false, 12, 206, 409711, 394925, 275), - (1, "insert_or_replace", true, 13, 187, 1047, 1379, 22), - (1, "insert_or_replace", false, 12, 187, 409674, 394925, 274), - ( - 1, - "insert_or_replace_dont_check", - true, - 13, - 187, - 1047, - 1379, - 22, - ), - ( - 1, - "insert_or_replace_dont_check", - false, - 12, - 187, - 409674, - 394925, - 274, - ), - (1, "replace", true, 13, 0, 1271, 1379, 23), - (1, "replace", false, 12, 150, 409711, 394925, 274), - (1, "replace_dont_check", true, 13, 0, 1271, 1379, 23), - (1, "replace_dont_check", false, 12, 150, 409711, 394925, 274), - (1, "patch", true, 13, 0, 1271, 1379, 23), - (1, "patch", false, 12, 150, 409711, 394925, 274), - (1, "patch_dont_check", true, 13, 0, 1271, 1379, 23), - (1, "patch_dont_check", false, 12, 150, 409711, 394925, 274), - (2, "insert_or_replace", true, 13, 245, 1095, 1451, 23), - (2, "insert_or_replace", false, 12, 229, 409674, 394925, 275), - ( - 2, - "insert_or_replace_dont_check", - true, - 13, - 245, - 1095, - 1451, - 23, - ), - ( - 2, - "insert_or_replace_dont_check", - false, - 12, - 229, - 409674, - 394925, - 275, - ), - (2, "replace", true, 13, 0, 1340, 1451, 23), - (2, "replace", false, 12, 192, 409711, 394925, 275), - (2, "replace_dont_check", true, 13, 0, 1340, 1451, 23), - (2, "replace_dont_check", false, 12, 192, 409711, 394925, 275), - (3, "insert_or_replace", true, 13, 198, 1095, 1451, 22), - (3, "insert_or_replace", false, 12, 182, 409674, 394925, 274), - ( - 3, - "insert_or_replace_dont_check", - true, - 13, - 198, - 1095, - 1451, - 22, - ), - ( - 3, - "insert_or_replace_dont_check", - false, - 12, - 182, - 409674, - 394925, - 274, - ), - (3, "replace", true, 13, 0, 1301, 1451, 22), - (3, "replace", false, 12, 0, 409864, 394925, 274), - (3, "replace_dont_check", true, 13, 0, 1301, 1451, 22), - (3, "replace_dont_check", false, 12, 0, 409864, 394925, 274), - (4, "insert_or_replace", true, 13, 223, 1047, 1379, 23), - (4, "insert_or_replace", false, 12, 223, 409674, 394925, 275), - ( - 4, - "insert_or_replace_dont_check", - true, - 13, - 223, - 1047, - 1379, - 23, - ), - ( - 4, - "insert_or_replace_dont_check", - false, - 12, - 223, - 409674, - 394925, - 275, - ), - (4, "replace", true, 13, 0, 1270, 1379, 23), - (4, "replace", false, 12, 186, 409711, 394925, 275), - (4, "replace_dont_check", true, 13, 0, 1270, 1379, 23), - (4, "replace_dont_check", false, 12, 186, 409711, 394925, 275), - (4, "refresh_trusted", true, 13, 0, 1270, 1379, 23), - (4, "refresh_trusted", false, 12, 186, 409711, 394925, 275), - (5, "insert_or_replace", true, 13, 240, 1095, 1451, 23), - (5, "insert_or_replace", false, 12, 224, 409674, 394925, 275), - ( - 5, - "insert_or_replace_dont_check", - true, - 13, - 240, - 1095, - 1451, - 23, - ), - ( - 5, - "insert_or_replace_dont_check", - false, - 12, - 224, - 409674, - 394925, - 275, - ), - (5, "replace", true, 13, 0, 1335, 1451, 23), - (5, "replace", false, 12, 187, 409711, 394925, 275), - (5, "replace_dont_check", true, 13, 0, 1335, 1451, 23), - (5, "replace_dont_check", false, 12, 187, 409711, 394925, 275), - (6, "insert_or_replace", true, 13, 151, 1047, 1379, 24), - (6, "insert_or_replace", false, 12, 151, 409674, 394925, 276), - ( - 6, - "insert_or_replace_dont_check", - true, - 13, - 151, - 1047, - 1379, - 24, - ), - ( - 6, - "insert_or_replace_dont_check", - false, - 12, - 151, - 409674, - 394925, - 276, - ), - (6, "replace", true, 13, 0, 1198, 1379, 24), - (6, "replace", false, 12, 0, 409825, 394925, 276), - (6, "replace_dont_check", true, 13, 0, 1198, 1379, 24), - (6, "replace_dont_check", false, 12, 0, 409825, 394925, 276), - ]; - - /// Without the option, every estimate is what it was before the option - /// existed, on the latest grove version and on grove v3. - #[test] - fn estimates_without_the_option_are_unchanged() { - for (grove_version, pins) in [ - (GroveVersion::latest(), ESTIMATES_BEFORE_THE_OPTION), - (&GROVE_V3, GROVE_V3_ESTIMATES_BEFORE_THE_OPTION), - ] { - let mut pinned = pins.iter(); - for (case, (_, new, tree_type)) in estimate_cases().into_iter().enumerate() { - for (label, op) in replaceable_writes(&new) { - for average_case in [true, false] { - let &( - pinned_case, - pinned_label, - pinned_average_case, - seek_count, - added_bytes, - replaced_bytes, - storage_loaded_bytes, - hash_node_calls, - ) = pinned.next().expect("expected a pinned estimate"); - assert_eq!( - (pinned_case, pinned_label, pinned_average_case), - (case, label, average_case) - ); - let before = OperationCost { - seek_count, - storage_cost: grovedb_costs::storage_cost::StorageCost { - added_bytes, - replaced_bytes, - removed_bytes: NoStorageRemoval, - }, - storage_loaded_bytes, - hash_node_calls, - sinsemilla_hash_calls: 0, - }; - for batch_apply_options in [None, Some(options(Mode::Merging))] { - assert_eq!( - estimate( - vec![op.clone()], - batch_apply_options, - average_case.then_some(tree_type), - grove_version, - ), - before, - "case {case}, {label}, average case {average_case}" - ); - } - } - } - } - assert!(pinned.next().is_none(), "expected every pinned estimate"); - } - } - - /// A replacement estimated without the option charges no added bytes, - /// so it would be below a replacement; the option raises it to - /// the insertion the replacement apply records. - #[test] - fn a_replace_estimate_without_the_option_is_below_a_replacement() { - let grove_version = GroveVersion::latest(); - let old = Element::new_item_with_flags(vec![7; 20], owned_flags(0, OLD_OWNER)); - let new = Element::new_item_with_flags(vec![8; 20], owned_flags(2, NEW_OWNER)); - let db = grove_with(&old, TreeType::NormalTree, grove_version); - let applied = apply(&db, vec![write_op(&new)], Mode::Replacing, grove_version) - .expect("expected the replacing update to apply"); - let replace = - QualifiedGroveDbOp::replace_op(vec![b"tree".to_vec()], KEY.to_vec(), new.clone()); - let merging = estimate( - vec![replace.clone()], - None, - Some(TreeType::NormalTree), - grove_version, - ); - assert_eq!(merging.storage_cost.added_bytes, 0); - assert!(merging.storage_cost.added_bytes < applied.storage_cost.added_bytes); - let replacing = estimate( - vec![replace], - Some(options(Mode::Replacing)), - Some(TreeType::NormalTree), - grove_version, - ); - assert_eq!( - replacing.storage_cost.added_bytes, - applied.storage_cost.added_bytes - ); - assert_eq!( - replacing.storage_cost.replaced_bytes, - merging.storage_cost.replaced_bytes - ); - } - - /// A flagged tree written together with a write under it reaches the - /// estimators as the `InsertTreeWithRootHash` its propagation makes of - /// it; with the option that op too is charged the insertion a replacement - /// owner change records, whatever tree it is and whatever tree it lands - /// in. - #[test] - fn estimates_with_the_option_cover_a_replaced_tree_written_with_its_children() { - let grove_version = GroveVersion::latest(); - let trees: [fn(Option>) -> Element; 7] = [ - Element::empty_tree_with_flags, - Element::empty_sum_tree_with_flags, - Element::empty_big_sum_tree_with_flags, - Element::empty_count_tree_with_flags, - Element::empty_count_sum_tree_with_flags, - Element::empty_provable_count_tree_with_flags, - Element::empty_provable_count_sum_tree_with_flags, - ]; - for parent_tree in [ - Element::empty_tree(), - Element::empty_sum_tree(), - Element::empty_big_sum_tree(), - Element::empty_count_tree(), - ] { - let parent_tree_type = parent_tree.tree_type().expect("expected a tree"); - for tree in trees { - let tree_type = tree(None).tree_type().expect("expected a tree"); - let db = make_empty_grovedb(); - db.insert( - EMPTY_PATH, - b"parent", - parent_tree.clone(), - None, - None, - grove_version, - ) - .unwrap() - .expect("expected to insert the parent tree"); - db.insert( - [b"parent".as_slice()].as_ref(), - b"tree", - tree(owned_flags(0, OLD_OWNER)), - None, - None, - grove_version, - ) - .unwrap() - .expect("expected to insert the tree"); - db.insert( - [b"parent".as_slice(), b"tree"].as_ref(), - b"child", - Element::new_item(b"child".to_vec()), - None, - None, - grove_version, - ) - .unwrap() - .expect("expected to insert the child"); - let ops = vec![ - QualifiedGroveDbOp::insert_or_replace_op( - vec![b"parent".to_vec()], - b"tree".to_vec(), - tree(owned_flags(2, NEW_OWNER)), - ), - QualifiedGroveDbOp::insert_or_replace_op( - vec![b"parent".to_vec(), b"tree".to_vec()], - b"another child".to_vec(), - Element::new_item(b"another child".to_vec()), - ), - ]; - let applied = apply(&db, ops.clone(), Mode::Replacing, grove_version) - .expect("expected the replacing update to apply"); - - let mut average_case_layers = HashMap::new(); - average_case_layers.insert( - KeyInfoPath(vec![]), - EstimatedLayerInformation { - tree_type: TreeType::NormalTree, - estimated_layer_count: ApproximateElements(2), - estimated_layer_sizes: AllSubtrees(7, NoSumTrees, None), - }, - ); - average_case_layers.insert( - KeyInfoPath::from_known_path([b"parent".as_slice()]), - EstimatedLayerInformation { - tree_type: parent_tree_type, - estimated_layer_count: ApproximateElements(2), - estimated_layer_sizes: AllSubtrees(4, NoSumTrees, Some(35)), - }, - ); - average_case_layers.insert( - KeyInfoPath::from_known_path([b"parent".as_slice(), b"tree"]), - EstimatedLayerInformation { - tree_type, - estimated_layer_count: ApproximateElements(2), - estimated_layer_sizes: AllItems(13, 13, None), - }, - ); - let mut worst_case_layers = HashMap::new(); - worst_case_layers.insert(KeyInfoPath(vec![]), MaxElementsNumber(2)); - worst_case_layers.insert( - KeyInfoPath::from_known_path([b"parent".as_slice()]), - MaxElementsNumber(2), - ); - worst_case_layers.insert( - KeyInfoPath::from_known_path([b"parent".as_slice(), b"tree"]), - MaxElementsNumber(2), - ); - // The worst case sizes the item written under the tree as - // under a normal tree whatever the tree is, so only a normal - // tree's batch is bounded as a whole there; the worst-case - // replaced tree itself is bounded for every tree in - // `estimated_costs`' tests. - let mut estimated_costs_types = vec![AverageCaseCostsType(average_case_layers)]; - if tree_type == TreeType::NormalTree { - estimated_costs_types.push(WorstCaseCostsType(worst_case_layers)); - } - for estimated_costs_type in estimated_costs_types { - let estimate = GroveDb::estimated_case_operations_for_batch( - estimated_costs_type, - ops.clone(), - Some(options(Mode::Replacing)), - |_cost, _old_flags, _new_flags| Ok(false), - |_flags, _removed_key_bytes, _removed_value_bytes| { - Ok((NoStorageRemoval, NoStorageRemoval)) - }, - grove_version, - ) - .cost_as_result() - .expect("expected an estimate"); - assert!( - estimate.storage_cost.added_bytes >= applied.storage_cost.added_bytes, - "a {tree_type:?} in a {parent_tree_type:?}: estimate {estimate:?} is \ - below the replacement apply {applied:?}" - ); - } - } - } - } } diff --git a/grovedb/src/batch/replacement_test_support.rs b/grovedb/src/batch/replacement_test_support.rs index ac5942036..706d14624 100644 --- a/grovedb/src/batch/replacement_test_support.rs +++ b/grovedb/src/batch/replacement_test_support.rs @@ -13,11 +13,7 @@ use grovedb_epoch_based_storage_flags::StorageFlags; use grovedb_merk::{element::costs::ElementCostExtensions, tree_type::TreeType}; use grovedb_version::version::GroveVersion; -use crate::{ - batch::{BatchApplyOptions, QualifiedGroveDbOp}, - tests::TempGroveDb, - Element, ElementFlags, Error, -}; +use crate::{batch::QualifiedGroveDbOp, tests::TempGroveDb, Element, ElementFlags, Error}; pub(crate) const OLD_OWNER: [u8; 32] = [1; 32]; pub(crate) const NEW_OWNER: [u8; 32] = [2; 32]; @@ -82,15 +78,7 @@ pub(crate) fn split_removal_bytes( .map_err(|e| Error::SplitRemovalBytesClientError(e.to_string())) } -/// The batch options of `mode`. -pub(crate) fn options(mode: Mode) -> BatchApplyOptions { - BatchApplyOptions { - allow_replacement_accounting: mode == Mode::Replacing, - ..Default::default() - } -} - -/// Applies `ops` with the flags update and options of `mode`. +/// Applies `ops` with the flags update of `mode`. pub(crate) fn apply( db: &TempGroveDb, ops: Vec, @@ -99,7 +87,7 @@ pub(crate) fn apply( ) -> Result { db.apply_batch_with_element_flags_update( ops, - Some(options(mode)), + None, flags_update(mode), split_removal_bytes, None, diff --git a/grovedb/src/batch/same_batch_reference_prediction_tests.rs b/grovedb/src/batch/same_batch_reference_prediction_tests.rs index 2f9166d9f..5262b6f60 100644 --- a/grovedb/src/batch/same_batch_reference_prediction_tests.rs +++ b/grovedb/src/batch/same_batch_reference_prediction_tests.rs @@ -22,8 +22,8 @@ mod tests { use crate::{ batch::{ replacement_test_support::{ - apply, flags_update, options, owned_flags, split_removal_bytes, unowned_flags, - value_bytes, write_op, Mode, KEY, NEW_OWNER, OLD_OWNER, + apply, flags_update, owned_flags, split_removal_bytes, unowned_flags, value_bytes, + write_op, Mode, KEY, NEW_OWNER, OLD_OWNER, }, BatchApplyOptions, QualifiedGroveDbOp, }, @@ -753,7 +753,7 @@ mod tests { let db = grove_with(&old, TreeType::SumTree, false, grove_version); db.apply_partial_batch_with_element_flags_update( vec![write_op(&new)], - Some(options(mode)), + None, flags_update(mode), split_removal_bytes, |_cost, _leftover_operations| Ok(vec![reference_op(vec![], b"root reference")]), @@ -806,7 +806,7 @@ mod tests { partial .apply_partial_batch_with_element_flags_update( vec![write_op(&first)], - Some(options(mode)), + None, flags_update(mode), split_removal_bytes, |_cost, _leftover_operations| { @@ -869,7 +869,7 @@ mod tests { ); db.apply_partial_batch_with_element_flags_update( vec![write(&inserted)], - Some(options(mode)), + None, flags_update(mode), split_removal_bytes, |_cost, _leftover_operations| Ok(vec![write(&rewritten), reference.clone()]), diff --git a/grovedb/src/tests/backward_references_flags_update_tests.rs b/grovedb/src/tests/backward_references_flags_update_tests.rs index 6bbcd6108..ab12b3d0f 100644 --- a/grovedb/src/tests/backward_references_flags_update_tests.rs +++ b/grovedb/src/tests/backward_references_flags_update_tests.rs @@ -317,8 +317,7 @@ fn same_batch_referrers_of_an_updated_item_commit_to_the_landed_bytes() { assert_referrers_bound(&db, &[TEST_LEAF], &[b"n1"], b"world", grove_version); } -/// An update that hands a flagged item to a new owner is a replacement under -/// `allow_replacement_accounting`: the item lands with the new owner's flags as +/// An update that hands a flagged item to a new owner as a replacement: the item lands with the new owner's flags as /// written, and its referrers, old and same-batch, commit to those bytes. #[test] fn a_replacement_keeps_referrers_bound() { @@ -371,10 +370,7 @@ fn a_replacement_keeps_referrers_bound() { sibling_bidi(b"value"), ), ], - Some(crate::batch::BatchApplyOptions { - allow_replacement_accounting: true, - ..Default::default() - }), + None, crate::batch::replacement_test_support::replacing_flags_update, crate::batch::replacement_test_support::split_removal_bytes, None, diff --git a/grovedb/tests/written_owner_flags_update.rs b/grovedb/tests/written_owner_flags_update.rs index 19ac53853..a466d632d 100644 --- a/grovedb/tests/written_owner_flags_update.rs +++ b/grovedb/tests/written_owner_flags_update.rs @@ -14,9 +14,8 @@ use std::collections::BTreeMap; use grovedb::{ - batch::{BatchApplyOptions, QualifiedGroveDbOp}, - reference_path::ReferencePathType, - Element, ElementFlags, Error, GroveDb, + batch::QualifiedGroveDbOp, reference_path::ReferencePathType, Element, ElementFlags, Error, + GroveDb, }; use grovedb_costs::{ storage_cost::{ @@ -255,10 +254,7 @@ fn should_refund_the_old_owner_in_full_and_charge_the_new_owner_for_everything() let cost = db .apply_batch_with_element_flags_update( vec![write_op(new_element.clone())], - Some(BatchApplyOptions { - allow_replacement_accounting: true, - ..Default::default() - }), + None, |cost, old_flags, new_flags| { StorageFlags::update_element_flags_replacing_on_owner_change( cost, old_flags, new_flags,