diff --git a/CHANGELOG.md b/CHANGELOG.md index 348c4c85324..e786f1c45ba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -128,6 +128,15 @@ gains `DataContractCompiledMaxSizeExceededError` and `ProtocolValidationOperation` gains `ContractCompiledSizeValidation`, so struct literals and exhaustive matches need them. See [#5579](https://github.com/dashpay/platform/pull/5579). +- **drive (breaking):** `Drive::credit_storage_refunds_to_owners_operations` credits each + recorded owner of a storage refund without consulting keys or permissions, clearing an + owner's debt first, and returns a `StorageRefundCreditOutcome` whose + `processing_pool_share()` is the refunds without an owner balance plus the repaid debt, + for the caller's processing pool write. It is active from protocol version 15 + (`DRIVE_IDENTITY_METHOD_VERSIONS_V3`); nothing calls it yet, and settling refunds on + block lifecycle paths comes in a follow-up. `DriveIdentityUpdateMethodVersions` gains + `credit_storage_refunds_to_owners`, so struct literals need it. See + [#4706](https://github.com/dashpay/platform/pull/4706). ### [5.0.3](https://github.com/dashpay/platform/compare/v5.0.2...v5.0.3) (2026-11-03) diff --git a/book/src/fees/overview.md b/book/src/fees/overview.md index 30afc2e6ef3..ce3e02592fb 100644 --- a/book/src/fees/overview.md +++ b/book/src/fees/overview.md @@ -49,8 +49,9 @@ in `FeeStorageVersion`: | `storage_seek_cost` | 2,000 | Cost of a single disk seek | Storage fees are **refundable**: when data is deleted, a portion of the original -storage fee is returned to the identity that paid it (see [Refunds](#refunds) -below). +storage fee becomes a refund for the owner recorded in the stored bytes' storage +flags, which is not always the identity that paid the fee (see +[Refunds](#refunds) below). The documents of a type that declares a `ttl` (protocol version 14) are the exception: they carry no storage flags and refund nothing, their bytes are priced for the time they live, and @@ -494,6 +495,26 @@ rate. Every schedule so far has the same storage rate, so the two agree. There is a **dust limit**: refunds below 32 bytes worth of storage credits are discarded to prevent micro-refund spam. +### Refund ownership (protocol version 15 onward) + +Refunds follow the recorded owner in the element's storage flags. +`Drive::credit_storage_refunds_to_owners_operations` credits each owner that +has a balance element without consulting any key or permission, so a frozen +but existing owner still receives its bookkeeping refund. Two shares of a +refund never reach a balance and are reported for the caller instead: the part +that clears an owner's negative credit (identity debt, which lives outside the +credit sum trees) and the part whose owner has no balance element (the native +stand-in for a wiped owner). The caller moves both into the current epoch's +processing pool with a single pool write and records every refund against its +storage epoch in the pending epoch refunds, so the credit conservation check +stays balanced. This primitive is the settlement step block lifecycle paths +that remove owner-attributed bytes are meant to use in the block that removes +them; at protocol version 15 the vote poll end cleanup does not yet settle its +refunds, and wiring it up is a separate change. The protocol 12 schema +migration, which shrank stored contracts without refunding the stripped bytes, +ran once at that activation and is the recorded historical exception; it +replays exactly as executed. + ## Epoch-Based Fee Distribution Fees do not go directly to the block proposer. Instead, they accumulate in diff --git a/packages/rs-drive/src/drive/identity/update/methods/credit_storage_refunds_to_owners_operations/mod.rs b/packages/rs-drive/src/drive/identity/update/methods/credit_storage_refunds_to_owners_operations/mod.rs new file mode 100644 index 00000000000..6fe378b6e46 --- /dev/null +++ b/packages/rs-drive/src/drive/identity/update/methods/credit_storage_refunds_to_owners_operations/mod.rs @@ -0,0 +1,84 @@ +mod v0; + +use crate::drive::identity::update::storage_refund_credit_outcome::StorageRefundCreditOutcome; +use crate::drive::Drive; +use crate::error::drive::DriveError; +use crate::error::Error; +use crate::fees::op::LowLevelDriveOperation; +use dpp::fee::fee_result::refunds::FeeRefunds; +use dpp::version::PlatformVersion; +use grovedb::TransactionArg; + +impl Drive { + /// Credits storage refunds to their recorded owners and reports what could + /// not be routed. + /// + /// For every owner in `fee_refunds` except `skip_owner`, the owner's + /// per-epoch credits are summed with checked arithmetic. An owner with a + /// balance element is credited the way a state transition payer's own + /// refund is (one balance read feeding `add_to_previous_balance`, then the + /// balance and negative credit updates); no key, signature or permission + /// is consulted, so a frozen but existing owner receives its bookkeeping + /// refund. When that owner's balance is + /// zero the helper first clears its negative credit (identity debt), and + /// only the remainder reaches the balance; the cleared debt is reported as + /// `repaid_debt` because debt lives outside the credit sum trees and is + /// processing fee the pools were short of when it was incurred. An owner + /// without a balance element (the native proxy for a wiped owner) is not + /// credited and its amount is reported as `routed_to_processing_pool`. + /// The caller settles `processing_pool_share()` (both amounts) into the + /// current epoch's processing pool with one pool write and records the + /// pending refunds; the primitive does neither, so the block keeps a + /// single pending-refund and pool write per batch. + /// + /// # Parameters + /// + /// * `fee_refunds` - The refunds to settle, per owner and storage epoch. + /// * `skip_owner` - An owner whose refund the caller settles itself (a + /// state transition payer, whose refund folds into its balance change); + /// `None` on lifecycle paths. + /// * `transaction` - The current transaction. + /// * `drive_operations` - The accumulator the balance operations are + /// appended to; the caller applies them. + /// * `platform_version` - The platform version. + /// + /// # Returns + /// + /// * `Ok(StorageRefundCreditOutcome)` - The owners credited, the debt the + /// refunds repaid and the amount routed to the processing pool. + /// * `Err(Error)` - On overflow, a corrupted balance element, or when the + /// method is not active for the platform version. + pub fn credit_storage_refunds_to_owners_operations( + &self, + fee_refunds: &FeeRefunds, + skip_owner: Option<[u8; 32]>, + transaction: TransactionArg, + drive_operations: &mut Vec, + platform_version: &PlatformVersion, + ) -> Result { + match platform_version + .drive + .methods + .identity + .update + .credit_storage_refunds_to_owners + { + Some(0) => self.credit_storage_refunds_to_owners_operations_v0( + fee_refunds, + skip_owner, + transaction, + drive_operations, + platform_version, + ), + Some(version) => Err(Error::Drive(DriveError::UnknownVersionMismatch { + method: "credit_storage_refunds_to_owners_operations".to_string(), + known_versions: vec![0], + received: version, + })), + None => Err(Error::Drive(DriveError::VersionNotActive { + method: "credit_storage_refunds_to_owners_operations".to_string(), + known_versions: vec![0], + })), + } + } +} diff --git a/packages/rs-drive/src/drive/identity/update/methods/credit_storage_refunds_to_owners_operations/v0/mod.rs b/packages/rs-drive/src/drive/identity/update/methods/credit_storage_refunds_to_owners_operations/v0/mod.rs new file mode 100644 index 00000000000..9986aedc006 --- /dev/null +++ b/packages/rs-drive/src/drive/identity/update/methods/credit_storage_refunds_to_owners_operations/v0/mod.rs @@ -0,0 +1,814 @@ +use crate::drive::identity::update::add_to_previous_balance_outcome::AddToPreviousBalanceOutcomeV0Methods; +use crate::drive::identity::update::storage_refund_credit_outcome::StorageRefundCreditOutcome; +use crate::drive::Drive; +use crate::error::Error; +use crate::fees::op::LowLevelDriveOperation; +use dpp::fee::fee_result::refunds::FeeRefunds; +use dpp::fee::Credits; +use dpp::prelude::Identifier; +use dpp::version::PlatformVersion; +use dpp::ProtocolError; +use grovedb::TransactionArg; +use std::collections::BTreeMap; + +impl Drive { + /// Credits each recorded refund owner that has a balance element and + /// reports the rest as routed to the processing pool. See the dispatcher. + #[inline(always)] + pub(super) fn credit_storage_refunds_to_owners_operations_v0( + &self, + fee_refunds: &FeeRefunds, + skip_owner: Option<[u8; 32]>, + transaction: TransactionArg, + drive_operations: &mut Vec, + platform_version: &PlatformVersion, + ) -> Result { + let mut credited: BTreeMap = BTreeMap::new(); + let mut repaid_debt: Credits = 0; + let mut routed_to_processing_pool: Credits = 0; + + for (owner_id, credits_per_epoch) in fee_refunds.iter() { + if skip_owner.as_ref() == Some(owner_id) { + continue; + } + + let credits = credits_per_epoch + .values() + .try_fold(0u64, |sum, epoch_credits| sum.checked_add(*epoch_credits)) + .ok_or(ProtocolError::Overflow( + "storage refund credits for one owner overflow", + ))?; + + if credits == 0 { + continue; + } + + // A stateful read: `None` means the balance element does not exist, which + // is the only signal Drive has today that the owner is gone. + let existing_balance = self.fetch_identity_balance_operations( + *owner_id, + true, + transaction, + drive_operations, + platform_version, + )?; + + if let Some(existing_balance) = existing_balance { + // The same shape as the payer's own credit in + // `apply_balance_change_from_fee_to_identity`: the balance read above + // feeds the shipped helper directly, which reads the negative credit + // itself only when the balance is zero, so nothing is read twice. + let outcome = self.add_to_previous_balance( + *owner_id, + existing_balance, + credits, + true, + transaction, + drive_operations, + platform_version, + )?; + + if let Some(new_balance) = outcome.balance_modified() { + drive_operations + .push(self.update_identity_balance_operation_v0(*owner_id, new_balance)?); + } + + if let Some(new_negative_balance) = outcome.negative_credit_balance_modified() { + drive_operations.push(self.update_identity_negative_credit_operation( + *owner_id, + new_negative_balance, + platform_version, + )?); + } + + // From a zero balance the helper clears negative credit first and only + // the remainder becomes the new balance; from a positive balance the + // whole refund is added. The helper reports the part that repaid debt, + // which lives outside the sum trees and is passed on for the caller's + // processing pool write; the rest reached the balance. + let owner_repaid_debt = outcome.repaid_debt(); + let reached_balance = + credits + .checked_sub(owner_repaid_debt) + .ok_or(ProtocolError::Overflow( + "storage refund debt repayment cannot exceed the refund", + ))?; + + repaid_debt = + repaid_debt + .checked_add(owner_repaid_debt) + .ok_or(ProtocolError::Overflow( + "storage refund credits repaying identity debt overflow", + ))?; + + if reached_balance > 0 { + credited.insert(Identifier::from(*owner_id), reached_balance); + } + } else { + routed_to_processing_pool = routed_to_processing_pool.checked_add(credits).ok_or( + ProtocolError::Overflow( + "storage refund credits routed to the processing pool overflow", + ), + )?; + } + } + + Ok(StorageRefundCreditOutcome { + credited, + repaid_debt, + routed_to_processing_pool, + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::error::drive::DriveError; + use crate::util::batch::DriveOperation; + use crate::util::test_helpers::setup::setup_drive_with_initial_state_structure; + use dpp::block::block_info::BlockInfo; + use dpp::block::epoch::Epoch; + use dpp::fee::epoch::CreditsPerEpoch; + use dpp::identity::accessors::{IdentityGettersV0, IdentitySettersV0}; + use dpp::identity::Identity; + use grovedb::Transaction; + + const IDENTITY_BALANCE: Credits = 10_000_000; + const PROCESSING_POOL_SEED: Credits = 1_000_000; + + /// Refund credits per storage epoch, per owner. + type RefundsByOwner<'a> = [([u8; 32], &'a [(u16, Credits)])]; + + fn insert_identity( + drive: &Drive, + seed: u64, + transaction: &Transaction, + platform_version: &PlatformVersion, + ) -> Identity { + let mut identity = Identity::random_identity(3, Some(seed), platform_version) + .expect("expected a random identity"); + identity.set_balance(IDENTITY_BALANCE); + drive + .add_new_identity( + identity.clone(), + false, + &BlockInfo::default(), + true, + Some(transaction), + platform_version, + ) + .expect("expected to insert the identity"); + identity + } + + /// An identity with a zero balance and `debt` of negative credit, the + /// state an identity is left in after paying a processing fee it could + /// not fully cover. + fn insert_identity_with_debt( + drive: &Drive, + seed: u64, + debt: Credits, + transaction: &Transaction, + platform_version: &PlatformVersion, + ) -> Identity { + let mut identity = Identity::random_identity(3, Some(seed), platform_version) + .expect("expected a random identity"); + identity.set_balance(0); + drive + .add_new_identity( + identity.clone(), + false, + &BlockInfo::default(), + true, + Some(transaction), + platform_version, + ) + .expect("expected to insert the identity"); + apply( + drive, + vec![ + drive.update_identity_negative_credit_operation_v0(identity.id().to_buffer(), debt) + ], + transaction, + platform_version, + ); + identity + } + + fn debt( + drive: &Drive, + identity_id: [u8; 32], + transaction: &Transaction, + platform_version: &PlatformVersion, + ) -> Option { + drive + .fetch_identity_negative_balance_operations( + identity_id, + true, + Some(transaction), + &mut vec![], + platform_version, + ) + .expect("expected to fetch the negative credit") + } + + /// Settles a refund for an identity with a zero balance and `debt` of + /// negative credit exactly as a lifecycle caller would (credits, one pool + /// write for the pool share, pending refunds recorded) and returns the + /// outcome, the identity's balance and debt afterwards, and whether the + /// credit sum is balanced. + fn settle_refund_against_debt( + debt_amount: Credits, + refund: Credits, + ) -> ( + StorageRefundCreditOutcome, + Option, + Option, + bool, + ) { + let drive = setup_drive_with_initial_state_structure(None); + let platform_version = PlatformVersion::latest(); + let transaction = drive.grove.start_transaction(); + let epoch = Epoch::new(0).expect("epoch 0"); + + let owner = + insert_identity_with_debt(&drive, 8, debt_amount, &transaction, platform_version); + let owner_id = owner.id().to_buffer(); + + let seed_operation = drive + .add_epoch_processing_credits_for_distribution_operation( + &epoch, + PROCESSING_POOL_SEED, + Some(&transaction), + platform_version, + ) + .expect("expected the pool seed operation"); + apply(&drive, vec![seed_operation], &transaction, platform_version); + drive + .add_to_system_credits(PROCESSING_POOL_SEED, Some(&transaction), platform_version) + .expect("expected to record the system credits"); + + let fee_refunds = refunds(&[(owner_id, &[(0, refund)])]); + let mut operations = vec![]; + let outcome = drive + .credit_storage_refunds_to_owners_operations( + &fee_refunds, + None, + Some(&transaction), + &mut operations, + platform_version, + ) + .expect("expected to settle the refund"); + operations.push( + drive + .add_epoch_processing_credits_for_distribution_operation( + &epoch, + outcome + .processing_pool_share() + .expect("expected the pool share"), + Some(&transaction), + platform_version, + ) + .expect("expected the pool write"), + ); + apply(&drive, operations, &transaction, platform_version); + + let mut pending_refund_operations: Vec = vec![]; + Drive::add_update_pending_epoch_refunds_operations( + &mut pending_refund_operations, + fee_refunds.sum_per_epoch(), + &platform_version.drive, + ) + .expect("expected the pending refund operations"); + drive + .apply_drive_operations( + pending_refund_operations, + true, + &BlockInfo::default(), + Some(&transaction), + platform_version, + ) + .expect("expected to record the pending refunds"); + + let balanced = drive + .calculate_total_credits_balance(Some(&transaction), &platform_version.drive) + .expect("expected the balance") + .ok() + .expect("expected a well-formed balance"); + + ( + outcome, + balance(&drive, owner_id, &transaction, platform_version), + debt(&drive, owner_id, &transaction, platform_version), + balanced, + ) + } + + fn refunds(entries: &RefundsByOwner) -> FeeRefunds { + let mut fee_refunds = FeeRefunds::default(); + for (owner, credits_per_epoch) in entries { + let mut epochs = CreditsPerEpoch::default(); + for (epoch_index, credits) in credits_per_epoch.iter() { + epochs.insert(*epoch_index, *credits); + } + fee_refunds.0.insert(*owner, epochs); + } + fee_refunds + } + + fn apply( + drive: &Drive, + operations: Vec, + transaction: &Transaction, + platform_version: &PlatformVersion, + ) { + drive + .apply_batch_low_level_drive_operations( + None, + Some(transaction), + operations, + &mut vec![], + &platform_version.drive, + ) + .expect("expected to apply the operations"); + } + + fn balance( + drive: &Drive, + identity_id: [u8; 32], + transaction: &Transaction, + platform_version: &PlatformVersion, + ) -> Option { + drive + .fetch_identity_balance(identity_id, Some(transaction), platform_version) + .expect("expected to fetch the balance") + } + + #[test] + fn should_credit_each_recorded_owner_by_the_sum_of_its_epochs() { + let drive = setup_drive_with_initial_state_structure(None); + let platform_version = PlatformVersion::latest(); + let transaction = drive.grove.start_transaction(); + + let first = insert_identity(&drive, 1, &transaction, platform_version); + let second = insert_identity(&drive, 2, &transaction, platform_version); + let first_id = first.id().to_buffer(); + let second_id = second.id().to_buffer(); + + let fee_refunds = refunds(&[ + (first_id, &[(0, 300), (4, 700), (9, 1)]), + (second_id, &[(2, 5_000)]), + ]); + + let mut operations = vec![]; + let outcome = drive + .credit_storage_refunds_to_owners_operations( + &fee_refunds, + None, + Some(&transaction), + &mut operations, + platform_version, + ) + .expect("expected to credit the owners"); + apply(&drive, operations, &transaction, platform_version); + + assert_eq!( + outcome, + StorageRefundCreditOutcome { + credited: BTreeMap::from([(first.id(), 1_001), (second.id(), 5_000)]), + repaid_debt: 0, + routed_to_processing_pool: 0, + } + ); + assert_eq!( + balance(&drive, first_id, &transaction, platform_version), + Some(IDENTITY_BALANCE + 1_001) + ); + assert_eq!( + balance(&drive, second_id, &transaction, platform_version), + Some(IDENTITY_BALANCE + 5_000) + ); + } + + #[test] + fn should_report_refunds_for_an_owner_without_a_balance_instead_of_failing() { + let drive = setup_drive_with_initial_state_structure(None); + let platform_version = PlatformVersion::latest(); + let transaction = drive.grove.start_transaction(); + + let existing = insert_identity(&drive, 3, &transaction, platform_version); + let existing_id = existing.id().to_buffer(); + let missing_id = [0xAB; 32]; + assert_eq!( + balance(&drive, missing_id, &transaction, platform_version), + None + ); + + let fee_refunds = refunds(&[ + (existing_id, &[(1, 400)]), + (missing_id, &[(1, 250), (3, 50)]), + ]); + + let mut operations = vec![]; + let outcome = drive + .credit_storage_refunds_to_owners_operations( + &fee_refunds, + None, + Some(&transaction), + &mut operations, + platform_version, + ) + .expect("an owner without a balance is reported, not an error"); + apply(&drive, operations, &transaction, platform_version); + + assert_eq!( + outcome, + StorageRefundCreditOutcome { + credited: BTreeMap::from([(existing.id(), 400)]), + repaid_debt: 0, + routed_to_processing_pool: 300, + } + ); + assert_eq!(outcome.total(), Some(700)); + assert_eq!( + balance(&drive, existing_id, &transaction, platform_version), + Some(IDENTITY_BALANCE + 400) + ); + assert_eq!( + balance(&drive, missing_id, &transaction, platform_version), + None, + "no balance element is created for the missing owner" + ); + } + + #[test] + fn should_credit_an_owner_whose_keys_are_all_disabled() { + let drive = setup_drive_with_initial_state_structure(None); + let platform_version = PlatformVersion::latest(); + let transaction = drive.grove.start_transaction(); + + let frozen = insert_identity(&drive, 4, &transaction, platform_version); + let frozen_id = frozen.id().to_buffer(); + let key_ids = frozen.public_keys().keys().copied().collect::>(); + assert_eq!(key_ids.len(), 3); + drive + .disable_identity_keys( + frozen_id, + key_ids, + 1_000, + &BlockInfo::default(), + true, + Some(&transaction), + platform_version, + ) + .expect("expected to disable every key"); + + let fee_refunds = refunds(&[(frozen_id, &[(0, 12_345)])]); + + let mut operations = vec![]; + let outcome = drive + .credit_storage_refunds_to_owners_operations( + &fee_refunds, + None, + Some(&transaction), + &mut operations, + platform_version, + ) + .expect("no key or permission is consulted"); + apply(&drive, operations, &transaction, platform_version); + + assert_eq!(outcome.credited, BTreeMap::from([(frozen.id(), 12_345)])); + assert_eq!(outcome.routed_to_processing_pool, 0); + assert_eq!( + balance(&drive, frozen_id, &transaction, platform_version), + Some(IDENTITY_BALANCE + 12_345) + ); + } + + #[test] + fn should_skip_the_owner_the_caller_settles_itself() { + let drive = setup_drive_with_initial_state_structure(None); + let platform_version = PlatformVersion::latest(); + let transaction = drive.grove.start_transaction(); + + let payer = insert_identity(&drive, 5, &transaction, platform_version); + let other = insert_identity(&drive, 6, &transaction, platform_version); + let payer_id = payer.id().to_buffer(); + let other_id = other.id().to_buffer(); + + let fee_refunds = refunds(&[(payer_id, &[(0, 900)]), (other_id, &[(0, 100)])]); + + let mut operations = vec![]; + let outcome = drive + .credit_storage_refunds_to_owners_operations( + &fee_refunds, + Some(payer_id), + Some(&transaction), + &mut operations, + platform_version, + ) + .expect("expected to credit the other owner"); + apply(&drive, operations, &transaction, platform_version); + + assert_eq!(outcome.credited, BTreeMap::from([(other.id(), 100)])); + assert_eq!(outcome.routed_to_processing_pool, 0); + assert_eq!( + balance(&drive, payer_id, &transaction, platform_version), + Some(IDENTITY_BALANCE), + "the payer's refund folds into its own balance change elsewhere" + ); + assert_eq!( + balance(&drive, other_id, &transaction, platform_version), + Some(IDENTITY_BALANCE + 100) + ); + } + + #[test] + fn should_leave_total_credits_balanced_after_the_caller_records_the_pending_refunds() { + let drive = setup_drive_with_initial_state_structure(None); + let platform_version = PlatformVersion::latest(); + let transaction = drive.grove.start_transaction(); + let epoch = Epoch::new(0).expect("epoch 0"); + + let credited_owner = insert_identity(&drive, 7, &transaction, platform_version); + let credited_owner_id = credited_owner.id().to_buffer(); + let missing_owner_id = [0xCD; 32]; + + // Every credit in the system is accounted for before the refunds: two + // identity balances and a seeded processing pool. + let seed_operation = drive + .add_epoch_processing_credits_for_distribution_operation( + &epoch, + PROCESSING_POOL_SEED, + Some(&transaction), + platform_version, + ) + .expect("expected the pool seed operation"); + apply(&drive, vec![seed_operation], &transaction, platform_version); + drive + .add_to_system_credits( + IDENTITY_BALANCE + PROCESSING_POOL_SEED, + Some(&transaction), + platform_version, + ) + .expect("expected to record the system credits"); + assert!(drive + .calculate_total_credits_balance(Some(&transaction), &platform_version.drive) + .expect("expected the balance") + .ok() + .expect("expected a well-formed balance")); + + let fee_refunds = refunds(&[ + (credited_owner_id, &[(0, 100), (1, 50)]), + (missing_owner_id, &[(0, 70)]), + ]); + + // The primitive credits the owner that exists and reports the rest. + let mut operations = vec![]; + let outcome = drive + .credit_storage_refunds_to_owners_operations( + &fee_refunds, + None, + Some(&transaction), + &mut operations, + platform_version, + ) + .expect("expected to credit the owners"); + assert_eq!(outcome.routed_to_processing_pool, 70); + + // The caller routes the unrouted amount to the epoch's processing pool + // with one pool write and records every refund against its storage + // epoch, exactly as a lifecycle settlement does. + operations.push( + drive + .add_epoch_processing_credits_for_distribution_operation( + &epoch, + outcome + .processing_pool_share() + .expect("expected the pool share"), + Some(&transaction), + platform_version, + ) + .expect("expected the pool write"), + ); + apply(&drive, operations, &transaction, platform_version); + + let mut pending_refund_operations: Vec = vec![]; + Drive::add_update_pending_epoch_refunds_operations( + &mut pending_refund_operations, + fee_refunds.sum_per_epoch(), + &platform_version.drive, + ) + .expect("expected the pending refund operations"); + drive + .apply_drive_operations( + pending_refund_operations, + true, + &BlockInfo::default(), + Some(&transaction), + platform_version, + ) + .expect("expected to record the pending refunds"); + + assert_eq!( + balance(&drive, credited_owner_id, &transaction, platform_version), + Some(IDENTITY_BALANCE + 150) + ); + assert_eq!( + drive + .get_epoch_processing_credits_for_distribution( + &epoch, + Some(&transaction), + platform_version + ) + .expect("expected the pool balance"), + PROCESSING_POOL_SEED + 70 + ); + assert_eq!( + drive + .fetch_pending_epoch_refunds(Some(&transaction), &platform_version.drive) + .expect("expected the pending refunds"), + CreditsPerEpoch::from_iter([(0, 170), (1, 50)]) + ); + let total = drive + .calculate_total_credits_balance(Some(&transaction), &platform_version.drive) + .expect("expected the balance"); + assert!( + total.ok().expect("expected a well-formed balance"), + "credits moved between the pools and an identity balance must stay conserved: {:?}", + total + ); + } + + #[test] + fn should_report_a_refund_below_the_owners_debt_as_repaid_debt() { + let (outcome, balance, debt, balanced) = settle_refund_against_debt(100, 60); + + assert_eq!( + outcome, + StorageRefundCreditOutcome { + credited: BTreeMap::new(), + repaid_debt: 60, + routed_to_processing_pool: 0, + }, + "nothing reaches the balance; the whole refund clears debt" + ); + assert_eq!(balance, Some(0)); + assert_eq!(debt, Some(40)); + assert!(balanced, "the repaid debt went to the processing pool"); + } + + #[test] + fn should_report_a_refund_equal_to_the_owners_debt_as_repaid_debt() { + let (outcome, balance, debt, balanced) = settle_refund_against_debt(100, 100); + + assert_eq!( + outcome, + StorageRefundCreditOutcome { + credited: BTreeMap::new(), + repaid_debt: 100, + routed_to_processing_pool: 0, + } + ); + assert_eq!(balance, Some(0)); + assert_eq!(debt, Some(0)); + assert!(balanced); + } + + #[test] + fn should_split_a_refund_above_the_owners_debt_between_debt_and_balance() { + let (outcome, balance, debt, balanced) = settle_refund_against_debt(100, 150); + + let owner = Identity::random_identity(3, Some(8), PlatformVersion::latest()) + .expect("expected the same random identity") + .id(); + assert_eq!( + outcome, + StorageRefundCreditOutcome { + credited: BTreeMap::from([(owner, 50)]), + repaid_debt: 100, + routed_to_processing_pool: 0, + }, + "only the remainder above the debt is reported as credited" + ); + assert_eq!(outcome.processing_pool_share(), Some(100)); + assert_eq!(outcome.total(), Some(150)); + assert_eq!(balance, Some(50)); + assert_eq!(debt, Some(0)); + assert!(balanced); + } + + #[test] + fn should_not_touch_debt_when_the_owner_has_a_positive_balance() { + // The shipped helper repays debt only from a zero balance; an owner + // that is in debt yet holds a balance cannot exist through the fee + // path, but the primitive must still report what the helper does. + let drive = setup_drive_with_initial_state_structure(None); + let platform_version = PlatformVersion::latest(); + let transaction = drive.grove.start_transaction(); + + let owner = insert_identity(&drive, 9, &transaction, platform_version); + let owner_id = owner.id().to_buffer(); + // Outstanding debt beside the positive balance, so repaying it by + // mistake would show in the balance, the outcome and the debt. + let outstanding_debt = 75; + apply( + &drive, + vec![drive.update_identity_negative_credit_operation_v0(owner_id, outstanding_debt)], + &transaction, + platform_version, + ); + + let fee_refunds = refunds(&[(owner_id, &[(0, 250)])]); + let mut operations = vec![]; + let outcome = drive + .credit_storage_refunds_to_owners_operations( + &fee_refunds, + None, + Some(&transaction), + &mut operations, + platform_version, + ) + .expect("expected to credit the owner"); + apply(&drive, operations, &transaction, platform_version); + + assert_eq!(outcome.repaid_debt, 0); + assert_eq!(outcome.credited, BTreeMap::from([(owner.id(), 250)])); + assert_eq!( + balance(&drive, owner_id, &transaction, platform_version), + Some(IDENTITY_BALANCE + 250) + ); + assert_eq!( + debt(&drive, owner_id, &transaction, platform_version), + Some(outstanding_debt), + "the debt beside a positive balance is left as it was" + ); + } + + #[test] + fn should_read_each_owner_once_before_writing_its_balance() { + // The balance read decides whether the owner exists and then feeds the + // shipped helper, which reads the negative credit only from a zero + // balance: two stateful reads for an owner in debt, one otherwise. + // A second read of either element would show up as an extra cost + // operation and be billed to every refund the caller settles. + let drive = setup_drive_with_initial_state_structure(None); + let platform_version = PlatformVersion::latest(); + let transaction = drive.grove.start_transaction(); + + let funded = insert_identity(&drive, 10, &transaction, platform_version); + let in_debt = insert_identity_with_debt(&drive, 11, 100, &transaction, platform_version); + + let count_reads = |owner_id: [u8; 32]| { + let mut operations = vec![]; + drive + .credit_storage_refunds_to_owners_operations( + &refunds(&[(owner_id, &[(0, 150)])]), + None, + Some(&transaction), + &mut operations, + platform_version, + ) + .expect("expected to credit the owner"); + operations + .iter() + .filter(|operation| { + matches!( + operation, + LowLevelDriveOperation::CalculatedCostOperation(_) + ) + }) + .count() + }; + + assert_eq!(count_reads(funded.id().to_buffer()), 1); + assert_eq!(count_reads(in_debt.id().to_buffer()), 2); + } + + #[test] + fn should_not_be_active_before_the_new_drive_table() { + let drive = setup_drive_with_initial_state_structure(None); + let frozen_platform_version = PlatformVersion::get(14).expect("protocol version 14"); + let transaction = drive.grove.start_transaction(); + + let fee_refunds = refunds(&[([1; 32], &[(0, 100)])]); + + let result = drive.credit_storage_refunds_to_owners_operations( + &fee_refunds, + None, + Some(&transaction), + &mut vec![], + frozen_platform_version, + ); + + assert!( + matches!( + result, + Err(Error::Drive(DriveError::VersionNotActive { .. })) + ), + "protocol version 14 has no lifecycle refund settlement, got {:?}", + result + ); + } +} diff --git a/packages/rs-drive/src/drive/identity/update/methods/mod.rs b/packages/rs-drive/src/drive/identity/update/methods/mod.rs index c4bccd40633..7761f67037f 100644 --- a/packages/rs-drive/src/drive/identity/update/methods/mod.rs +++ b/packages/rs-drive/src/drive/identity/update/methods/mod.rs @@ -4,6 +4,7 @@ mod add_new_unique_keys_to_identity; mod add_to_identity_balance; mod add_to_previous_balance; mod apply_balance_change_from_fee_to_identity; +mod credit_storage_refunds_to_owners_operations; #[cfg(test)] mod debt_test_helpers; mod disable_identity_keys; diff --git a/packages/rs-drive/src/drive/identity/update/structs/mod.rs b/packages/rs-drive/src/drive/identity/update/structs/mod.rs index dbdb9a86d2f..71b93a3715f 100644 --- a/packages/rs-drive/src/drive/identity/update/structs/mod.rs +++ b/packages/rs-drive/src/drive/identity/update/structs/mod.rs @@ -2,3 +2,5 @@ pub mod add_to_previous_balance_outcome; /// Applying a balance chance outcome pub mod apply_balance_change_outcome; +/// The outcome of crediting storage refunds to their recorded owners +pub mod storage_refund_credit_outcome; diff --git a/packages/rs-drive/src/drive/identity/update/structs/storage_refund_credit_outcome/mod.rs b/packages/rs-drive/src/drive/identity/update/structs/storage_refund_credit_outcome/mod.rs new file mode 100644 index 00000000000..b80f1d38cab --- /dev/null +++ b/packages/rs-drive/src/drive/identity/update/structs/storage_refund_credit_outcome/mod.rs @@ -0,0 +1,48 @@ +use dpp::fee::Credits; +use dpp::prelude::Identifier; +use std::collections::BTreeMap; + +/// What `Drive::credit_storage_refunds_to_owners_operations` did with a set of +/// storage refunds: which recorded owners were credited, how much of the +/// refunds cleared identity debt instead of reaching a balance, and how much +/// could not be routed to any owner. +/// +/// The caller that knows the block's epoch settles `processing_pool_share()` +/// into that epoch's processing pool with a single pool write and records the +/// refunds against their storage epochs; the primitive itself never touches +/// the pools. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct StorageRefundCreditOutcome { + /// Credits that reached each recorded owner's balance element, summed + /// over the epochs the owner's bytes were stored in. An owner whose whole + /// refund went into clearing debt has no entry. + pub credited: BTreeMap, + /// Credits that cleared an owner's negative credit (identity debt) instead + /// of raising its balance. Debt is processing fee the pools were short of + /// when it was incurred and it lives outside the credit sum trees, so the + /// caller moves this amount into the current epoch's processing pool to + /// keep the credit sum balanced. + pub repaid_debt: Credits, + /// Credits whose recorded owner has no balance element. Until a typed + /// owner encoding exists this is the native proxy for a wiped owner, so + /// the caller moves this amount into the current epoch's processing pool. + pub routed_to_processing_pool: Credits, +} + +impl StorageRefundCreditOutcome { + /// What the caller writes into the current epoch's processing pool: the + /// unrouted refunds plus the debt they repaid. + pub fn processing_pool_share(&self) -> Option { + self.routed_to_processing_pool.checked_add(self.repaid_debt) + } + + /// The total settled: credited to balances, repaid as debt, or routed to + /// the pool. + pub fn total(&self) -> Option { + self.credited + .values() + .try_fold(self.processing_pool_share()?, |total, credits| { + total.checked_add(*credits) + }) + } +} diff --git a/packages/rs-platform-version/src/version/drive_versions/drive_identity_method_versions/mod.rs b/packages/rs-platform-version/src/version/drive_versions/drive_identity_method_versions/mod.rs index c55c74dd04e..1f91b861322 100644 --- a/packages/rs-platform-version/src/version/drive_versions/drive_identity_method_versions/mod.rs +++ b/packages/rs-platform-version/src/version/drive_versions/drive_identity_method_versions/mod.rs @@ -2,6 +2,7 @@ use versioned_feature_core::{FeatureVersion, OptionalFeatureVersion}; pub mod v1; pub mod v2; +pub mod v3; #[derive(Clone, Debug, Default)] pub struct DriveIdentityMethodVersions { @@ -248,4 +249,9 @@ pub struct DriveIdentityUpdateMethodVersions { /// Rewrites a key with a raised total budget or a later expiry (protocol version 14). Keys /// cannot carry limits before v14, so earlier tables keep the slot `None`. pub update_identity_key_limits: OptionalFeatureVersion, + /// Read by `Drive::credit_storage_refunds_to_owners_operations`: credits each recorded + /// owner of a storage refund and reports the amount whose owner has no balance element + /// so the caller can route it to the current epoch's processing pool. `None` before + /// protocol version 15, where no path settles refunds outside a state transition. + pub credit_storage_refunds_to_owners: OptionalFeatureVersion, } diff --git a/packages/rs-platform-version/src/version/drive_versions/drive_identity_method_versions/v1.rs b/packages/rs-platform-version/src/version/drive_versions/drive_identity_method_versions/v1.rs index fd26e1af65b..8641c91e2dc 100644 --- a/packages/rs-platform-version/src/version/drive_versions/drive_identity_method_versions/v1.rs +++ b/packages/rs-platform-version/src/version/drive_versions/drive_identity_method_versions/v1.rs @@ -120,6 +120,7 @@ pub const DRIVE_IDENTITY_METHOD_VERSIONS_V1: DriveIdentityMethodVersions = remove_from_identity_balance: 0, refresh_identity_key_reference_operations: 0, update_identity_key_limits: None, + credit_storage_refunds_to_owners: None, // backfill: the method is new and inactive here (VersionNotActive), nothing at these versions calls it }, insert: DriveIdentityInsertMethodVersions { add_new_identity: 0, diff --git a/packages/rs-platform-version/src/version/drive_versions/drive_identity_method_versions/v2.rs b/packages/rs-platform-version/src/version/drive_versions/drive_identity_method_versions/v2.rs index 360c20b0661..3c165f13853 100644 --- a/packages/rs-platform-version/src/version/drive_versions/drive_identity_method_versions/v2.rs +++ b/packages/rs-platform-version/src/version/drive_versions/drive_identity_method_versions/v2.rs @@ -177,6 +177,7 @@ pub const DRIVE_IDENTITY_METHOD_VERSIONS_V2: DriveIdentityMethodVersions = remove_from_identity_balance: 0, refresh_identity_key_reference_operations: 0, update_identity_key_limits: Some(0), + credit_storage_refunds_to_owners: None, // backfill: the method is new and inactive here (VersionNotActive), nothing at these versions calls it }, insert: DriveIdentityInsertMethodVersions { add_new_identity: 0, diff --git a/packages/rs-platform-version/src/version/drive_versions/drive_identity_method_versions/v3.rs b/packages/rs-platform-version/src/version/drive_versions/drive_identity_method_versions/v3.rs new file mode 100644 index 00000000000..043173a4439 --- /dev/null +++ b/packages/rs-platform-version/src/version/drive_versions/drive_identity_method_versions/v3.rs @@ -0,0 +1,25 @@ +use crate::version::drive_versions::drive_identity_method_versions::v2::DRIVE_IDENTITY_METHOD_VERSIONS_V2; +use crate::version::drive_versions::drive_identity_method_versions::{ + DriveIdentityMethodVersions, DriveIdentityUpdateMethodVersions, +}; + +/// V3 is protocol version 15's identity-method table. It differs from V2 in +/// one slot: +/// +/// * `update.credit_storage_refunds_to_owners` `None -> Some(0)`: the +/// primitive that credits each recorded owner of a storage refund and +/// reports the amount whose owner has no balance element, so a block +/// lifecycle path (or, later, the state transition refund path) can route +/// that amount to the current epoch's processing pool instead of halting. +/// No key or permission is consulted on any route: a frozen but existing +/// owner is credited, an owner without a balance is settled into the pool. +/// Nothing before v15 settles refunds outside a state transition, so V1 and +/// V2 keep the slot `None`. +pub const DRIVE_IDENTITY_METHOD_VERSIONS_V3: DriveIdentityMethodVersions = + DriveIdentityMethodVersions { + update: DriveIdentityUpdateMethodVersions { + credit_storage_refunds_to_owners: Some(0), // new in v15: credits recorded refund owners, reports the unrouted amount for the processing pool + ..DRIVE_IDENTITY_METHOD_VERSIONS_V2.update + }, + ..DRIVE_IDENTITY_METHOD_VERSIONS_V2 + }; diff --git a/packages/rs-platform-version/src/version/drive_versions/v10.rs b/packages/rs-platform-version/src/version/drive_versions/v10.rs index 19cf3009495..8987df68b9a 100644 --- a/packages/rs-platform-version/src/version/drive_versions/v10.rs +++ b/packages/rs-platform-version/src/version/drive_versions/v10.rs @@ -3,7 +3,7 @@ use crate::version::drive_versions::drive_contract_method_versions::{ DriveContractFeePotMethodVersions, DriveContractMethodVersions, }; use crate::version::drive_versions::drive_document_method_versions::v5::DRIVE_DOCUMENT_METHOD_VERSIONS_V5; -use crate::version::drive_versions::drive_identity_method_versions::v2::DRIVE_IDENTITY_METHOD_VERSIONS_V2; +use crate::version::drive_versions::drive_identity_method_versions::v3::DRIVE_IDENTITY_METHOD_VERSIONS_V3; use crate::version::drive_versions::drive_identity_method_versions::{ DriveIdentityCostEstimationMethodVersions, DriveIdentityMethodVersions, }; @@ -90,6 +90,10 @@ use grovedb_version::version::v5::GROVE_V5; /// V5 (`private_document_store.get_range` and `range_proof`), prices a store's creation as it /// applies it (`merk_insert_element` 1 in both estimators), and lets a batch carry the same /// keyless append more than once (`apply_batch.keyless_append_repeats`). +/// * **Recorded-owner refund credits** (protocol v15 note 28): `DRIVE_IDENTITY_METHOD_VERSIONS_V3` +/// turns on `update.credit_storage_refunds_to_owners`, the primitive block lifecycle paths +/// use to credit each recorded owner of a refund and report the amount whose owner has no +/// balance element. /// /// Everything else matches `DRIVE_VERSION_V9`. pub const DRIVE_VERSION_V10: DriveVersion = DriveVersion { @@ -109,9 +113,9 @@ pub const DRIVE_VERSION_V10: DriveVersion = DriveVersion { cost_estimation: DriveIdentityCostEstimationMethodVersions { for_balances: 1, // changed in v10: keeps a deeper root description for_token_balances: 1, // changed in v10: keeps a deeper root and Tokens layer description - ..DRIVE_IDENTITY_METHOD_VERSIONS_V2.cost_estimation + ..DRIVE_IDENTITY_METHOD_VERSIONS_V3.cost_estimation }, - ..DRIVE_IDENTITY_METHOD_VERSIONS_V2 + ..DRIVE_IDENTITY_METHOD_VERSIONS_V3 // changed in v10: credit_storage_refunds_to_owners credits recorded refund owners (protocol v15 note 28) }, document: DRIVE_DOCUMENT_METHOD_VERSIONS_V5, // changed in v10: private document type stores, delete estimates batch_operations: DriveBatchOperationsMethodVersion { diff --git a/packages/rs-platform-version/src/version/v15.rs b/packages/rs-platform-version/src/version/v15.rs index aa794085509..c5f020a9b36 100644 --- a/packages/rs-platform-version/src/version/v15.rs +++ b/packages/rs-platform-version/src/version/v15.rs @@ -567,9 +567,16 @@ pub const PROTOCOL_VERSION_15: ProtocolVersion = 15; /// block's rate. Every schedule so far has the same storage rate, so no refund changes. A /// removal from an earlier epoch pays once per such epoch for reading its record, the only /// fee this changes; generation 0 reads nothing. +/// +/// 28. **Recorded-owner refund credits**: `DRIVE_VERSION_V10` turns on +/// `identity.update.credit_storage_refunds_to_owners` (`DRIVE_IDENTITY_METHOD_VERSIONS_V3`), +/// the primitive that credits each recorded owner of a refund without consulting any key +/// or permission and reports the amount whose owner has no balance element, so block +/// lifecycle paths can settle it into the current epoch's processing pool. Nothing calls +/// it yet at this version; protocol v14 and earlier have no such method. pub const PLATFORM_V15: PlatformVersion = PlatformVersion { protocol_version: PROTOCOL_VERSION_15, - drive: DRIVE_VERSION_V10, // changed: calculate_fee 1 prices a refund at the storing epoch's rate (note 27); apply_contract_with_serialization 1 writes nothing for a contract whose format version 1 bytes match the stored one's (note 8); vote method versions v4: register_identity_vote 1 registers a yes/no vote, remove_all_votes_given_by_identities 1 also removes the yes/no votes of a masternode that left the list, every yes_no method and remove_vote_poll_end_date_query_operations Some(0) make yes/no polls active, add_vote_poll_end_date_query_operations 1 adds only the entry when the batch already queues the end date's tree with other storage flags, and verify methods v4 make the yes/no proof verifiers active (note 10) and verify_private_document_entries Some(0) a page of a private type's store (note 13); state transition method versions v5: document_create_transition 1 deletes a consumed document from what its lookup read (note 11); document method versions v5: insert_empty_private_document_store_operation Some(0) creates a private document type's store, and fetch_private_document_entries and prove_private_document_entries Some(0) read and prove a page of it (note 13); apply_drive_operations 2 resolves a batch's estimate, remove_reference_for_index_level_for_contract_operations 2, stateless_delete_of_non_tree_for_costs 1 and for_total_system_credits_update 1 describe their layers as they are (note 14); create_initial_state_structure 5 adds the token marketplace escrow trees; calculate_total_tokens_balance 2 counts escrowed tokens; the escrow primitives; the identity balance, token balance, prefunded specialized balance and contract fee pot estimations (for_balances 1, for_token_balances 1, estimated_cost_for_prefunded_specialized_balance_update 2, add_estimation_costs_for_contract_fee_pot_update 1) keep a deeper layer description (note 17); fetch_core_fork_activation_height and store_core_fork_activation_height Some(0) keep the Core fork activations (note 18); document method versions v5: append_private_document_entries_operations Some(0) appends a private document create's entries (note 25); grove_version GROVE_V5 (note 26) + drive: DRIVE_VERSION_V10, // changed: calculate_fee 1 prices a refund at the storing epoch's rate (note 27); apply_contract_with_serialization 1 writes nothing for a contract whose format version 1 bytes match the stored one's (note 8); vote method versions v4: register_identity_vote 1 registers a yes/no vote, remove_all_votes_given_by_identities 1 also removes the yes/no votes of a masternode that left the list, every yes_no method and remove_vote_poll_end_date_query_operations Some(0) make yes/no polls active, add_vote_poll_end_date_query_operations 1 adds only the entry when the batch already queues the end date's tree with other storage flags, and verify methods v4 make the yes/no proof verifiers active (note 10) and verify_private_document_entries Some(0) a page of a private type's store (note 13); state transition method versions v5: document_create_transition 1 deletes a consumed document from what its lookup read (note 11); document method versions v5: insert_empty_private_document_store_operation Some(0) creates a private document type's store, and fetch_private_document_entries and prove_private_document_entries Some(0) read and prove a page of it (note 13); apply_drive_operations 2 resolves a batch's estimate, remove_reference_for_index_level_for_contract_operations 2, stateless_delete_of_non_tree_for_costs 1 and for_total_system_credits_update 1 describe their layers as they are (note 14); create_initial_state_structure 5 adds the token marketplace escrow trees; calculate_total_tokens_balance 2 counts escrowed tokens; the escrow primitives; the identity balance, token balance, prefunded specialized balance and contract fee pot estimations (for_balances 1, for_token_balances 1, estimated_cost_for_prefunded_specialized_balance_update 2, add_estimation_costs_for_contract_fee_pot_update 1) keep a deeper layer description (note 17); fetch_core_fork_activation_height and store_core_fork_activation_height Some(0) keep the Core fork activations (note 18); document method versions v5: append_private_document_entries_operations Some(0) appends a private document create's entries (note 25); grove_version GROVE_V5 (note 26); identity method versions v3: credit_storage_refunds_to_owners Some(0) credits recorded refund owners (note 28) drive_abci: DriveAbciVersion { structs: DRIVE_ABCI_STRUCTURE_VERSIONS_V2, methods: DRIVE_ABCI_METHOD_VERSIONS_V11, // changed: the yes/no poll's tally, record, clean-up and finish hook are Some(0) (note 10); verify_chain_lock_locally 1 rejects a chain lock locally only when Platform holds the quorums active at its signing height, and verify_chain_lock_through_core 1 has Core's verifychainlock check the others before submitchainlock; consensus_params_update 2 installs the full evidence policy (note 16); update_core_info 1 records the chain locked height at which Core's V24 is first seen active (note 18); build_untied_withdrawal_transactions_from_documents 1 emits asset unlock payload version 2 once V24 is recorded as active (note 20); update_voter_identity 1 re-enables the voter key of the address a masternode returns to (note 22)