Skip to content

feat(platform)!: delete and erase lifecycle for keep-history documents - #4657

Merged
shumkov merged 202 commits into
v5.1-devfrom
keep-history-lifecycle
Nov 13, 2026
Merged

shumkov merged 202 commits into
v5.1-devfrom
keep-history-lifecycle

Conversation

@shumkov

@shumkov shumkov commented Sep 10, 2026 •

Copy link
Copy Markdown
Collaborator

Basic explanation

What this does: At protocol 15, a permitted delete removes a document from ordinary reads while retaining its history. A separately permitted erase removes that retained history in bounded chunks.

Value: Applications can keep revision history without making every document permanent. Historical queries distinguish active, deleted, erasing and absent documents.

Risks: This changes consensus validation, storage, proofs and fee accounting at protocol activation. Focused runtime, compatibility and build checks pass locally; full CI and validator-hardware capacity checks are separate verification layers.

Issue being fixed or feature implemented

Keep-history documents need an explicit lifecycle: removal from current reads must preserve revisions, while permanent removal of those revisions requires separate authorization. The lifecycle activates at protocol 15; protocols through 14 retain their existing behavior.

What was done?

Lifecycle and authorization

  • A keep-history type may permit deletion. canBeErased defaults to false, requires both history and deletion, and cannot change on a contract update. New registration still refuses a keep-history type with a contested index; non-validating reads preserve legacy contracts. History types cannot be consumed through a document reference.
  • Delete removes the current pointer and index references, including derived-index values, and retains revisions under the composite history layout. A versioned DPP lifecycle record in the type's [1] tree records the deletion. Multiple deletes share one lazy lifecycle-container insertion within a PV15 Drive batch.
  • The owner of the newest retained revision must start erasure. Once erasure starts, any identity may submit and pay for continuations. Each transition removes at most 100 revisions, newest first. The terminal chunk atomically removes the remaining history and lifecycle record, leaving no receipt.
  • Erase pays processing/storage fees in credits, has no action fee, and does not accept token payment. Its builder supports document-base versions 0, 1 and 2, including the PV15 default, and forwards signing settings before reserving a nonce for a constructible transition.
  • Deleted document ids cannot be recreated: ids commit to the identity contract nonce. The storage writer also refuses insertion over retained revisions. Terminal erasure does not make an id available to another valid create.

Reads and proofs

For a single-document lookup at time T:

State Answer
Active Newest retained revision at or before T, with the existing history-time clamp; absent before the first revision.
Deleted Retained revision before deleted_at; absent at or after it. The comparison uses the original, unclamped T, so a same-block delete wins.
Erasing Absent for every T, from the first erase chunk onward.
Erased Absent for every T; no terminal receipt is kept.

The PV15 temporal query authenticates the lifecycle record or its absence together with history. Whole-proof and subset verification share structural checks; typed verification also checks the returned document's id and revision. The raw serialized verifier keeps its public signature. Generation 0 remains available for cross-version execution proofs. DriveDocumentQuery with a block time remains refused for keep-history types.

GetDocumentHistory v1 reports ACTIVE, ABSENT, DELETED and ERASING, with remaining_revisions, deleted_at_ms, erasing_started_at_ms, erasing_from_time_ms and erasing_from_revision. Pages continue to expose surviving revisions during erasure. Proved responses use one merged GroveDB proof; lifecycle fields are derived from authenticated state. Released history v0 remains unchanged.

The Rust, WASM and JavaScript SDK surfaces expose erase and a separate current-lifecycle read. Erase confirmation authenticates affected state, rather than proving that this particular transition executed.

Fees and storage ownership

  • Delete estimates include owned flags for the history-tree read and lifecycle record. Total estimate-versus-applied tests cover first-container creation, summable/countable and derived indexes, sponsorship, a history node with two children, and migrated multi-epoch contract flags. Erase estimates include the storing-epoch reads needed for refunds and cover storage as well as processing.
  • Erase refund-recipient pricing has its own PV15-only optional Drive selector; its fee formula and recipient bound are unchanged.
  • Revision refunds go to their stored flag owners. A sponsored delete flags its lifecycle record to the sponsor. An erase-start record rewrite is billed to the eraser, while released record bytes refund the record's flag owner.
  • The client create-cost mirror admits insert generations 1 and 2. Actual insert matrices cover ordinary, keep-history, TTL and derived-index types: client storage matches applied storage and client processing covers applied processing. The client delete-refund mirror excludes retained history until erase; non-history and released-version pins remain intact.
  • The PV15 layout, 42-byte current pointer, upstream summable-index formulas, owner-following update v2 and activation migration are preserved. Default-deletable legacy history types become soft-deletable at activation. Static deletable-document references treat deleted/erasing/erased targets as missing; upstream dynamic-reference kind rules remain intact.

Protocol 15 version slots

The lifecycle selectors are documented in v15.rs, note 41:

Area Selection
Contract parser / document-type update validation try_from_schema 4 / validate_update 2
Drive insert / delete add_document_for_contract_operations 2 / delete_document_for_contract_operations 1
Primary pointer removal / estimate Both generation 1
Delete action conversion / structure validation Generation 1 / generation 2; upstream delete state validation 1 is retained
Typed single-document verifier Generation 1; existing unreleased temporal query/raw verifier 1 and history v1 are extended in place
Lifecycle fetch / erase operation / lifecycle and erase estimates / erase refund-recipient pricing Some(0)
Raw Drive-to-GroveDB batch conversion Generation 2; generation 1 remains selected at PV14
Erase structure / state validation / action conversion Some(0)
Erase serialization bounds / chunk limit Some(0..=0) / Some(100)

The PV15 tables are CONTRACT_VERSIONS_V7, DPP_VALIDATION_VERSIONS_V6, STATE_TRANSITION_SERIALIZATION_VERSIONS_V4, document/state-transition method tables V5, verify table V4, ABCI validation V11 and SYSTEM_LIMITS_V5. Erase handlers, bounds and limits are absent in released tables. DocumentTypeV2 gains the erasability flag; schema version 3 and compiled writer generation 0 are retained. StoredDeletion variants 0/1 preserve the previous false/true bytes; variant 2 represents erasability and requires history.

How Has This Been Tested?

Local macOS verification at a9c0e1ae82:

  • 632 Rust test cases across overlapping focused filters, plus 43 JavaScript decision-contract tests, with no failures. One pre-existing upstream contested-poll capacity test is ignored (should_end_a_poll_of_the_most_contenders_a_contest_accepts); it was not executed. This includes Drive lifecycle 49, delete 28, client costs 23, temporal/single-document verification 33, history verification 14, history decoder 19, migration 10 and contested 83; ABCI erase 19, references 8, activation 3, genesis 5 and commit-reveal 59; DPP contested 105 and lifecycle parser 20; and all 3 committed-byte compatibility tests.
  • Upstream decision-proposal integration: all 11 multiple-choice, 24 yes/no, 12 decision-proposal and 73 ABCI contested cases pass with the lifecycle implementation. The system contract's undeletable history proposals and non-history contested choices remain valid.
  • cargo fmt --check --all.
  • cargo clippy --workspace --all-targets --all-features --locked -- --no-deps -D warnings.
  • cargo check -p wasm-sdk --target wasm32-unknown-unknown --locked; wasm-dpp was also built for wasm32 and used by the JavaScript tests.
  • Earlier local port checks cover SDK history/erase/signing, request inventory, proof-verifier, gas sponsorship and persisted rollback/continuation. Compatibility captures include all 15 version tables, 1,956 non-lifecycle compiled-type observations and 28 unchanged non-lifecycle history query/proof/root/cost captures. The final merge preserves the same 218-path lifecycle delta across all 14,170 tree entries.

Follow-up verification at d6347691f2: the compatibility-v1 suite passes all 18 cases; structure lint/conformance/export tests pass all 12 cases; export freshness, workspace formatting and DPP/Drive clippy pass. The generated structure adds only the lifecycle container and record nodes, preserving all 281 existing nodes and 34 layer shapes. The requested changed-crate non-shielded runs completed 17,968 selected cases using the CI filter and --no-fail-fast: 17,967 passed initially and one reference-registration case aborted with the local stack setting unset. That case passes on both the frozen base and this branch with CI’s RUST_MIN_STACK=4194304; no test or production stack-size code was changed. Exact group counts, skips and passing process-leak warnings are recorded in the follow-up report.

Actual total delete-fee checks include:

Fixture Estimated / applied storage Estimated / applied processing
Middle history node with two children 9,072,000 / 8,640,000 89,371,160 / 7,322,960
Migrated history, multi-epoch contract flags 9,072,000 / 8,640,000 89,373,720 / 6,526,300

These assert total estimates cover applied fees. Processing has wide slack; storage margins are tight.

Verification at 70dd05914a5c: 2,047 focused Rust cases passed, including all 45 new upstream Rust tests for schema patterns, contact requests, proposal descriptions, contest poll identity and wallet addresses. The compatibility keyword inventory, signed lifecycle reference matrix, lifecycle/delete/history and proposal/contest groups pass with CI’s RUST_MIN_STACK=4194304; 16,224 cases were skipped by the focused selection. One passing process-leak warning was recorded and the case passed cleanly in isolation. All 46 decision-proposal JavaScript tests, workspace formatting and exact workspace clippy pass. The JavaScript run uses a WASM dependency rebuilt from this source. Swift wrapper tests remain CI-only because the required SDK framework is absent locally.

Local verification at 6edc4c051dc8: dpp, drive, drive-abci, dash-sdk, drive-proof-verifier, platform-version and native wasm-dpp2 pass 16,521 non-shielded cases with the CI filter and --no-fail-fast (1,214 skipped). The complete CI shielded shared-process phase passes 1,029 cases, including all 418 ABCI cases; three existing fixtures are ignored. Both use RUST_MIN_STACK=4194304. All six converter/refund-dispatch regressions pass, and workspace fmt, exact workspace clippy and all 38 platform-version cases pass. The shielded characterization keeps all eighteen original hashes: exact base/head captures differ only by two inactive default-false erasure fields in contract-type Debug output, which the non-history fixture excludes. Actions, errors, charges, lowered operations and settled state retain their original pins.

Final comment-only documentation commit 84ac1e73775c also passes workspace fmt and all 38 platform-version tests.

CI-only verification: the full-workspace non-shielded and full-chain phases, cargo test -p wasm-sdk --lib --locked -j 2, and workspace doctests. Those remaining phases are not established by these local runs. The network functional lifecycle test, representative validator-hardware activation rehearsal and maximum-block erase benchmarking are not established by these local checks. Client erase-refund modelling remains a separate follow-up.

In-place changes to shipped generations

Erase is appended to existing document/batch enums with existing discriminants and batch wire formats preserved. Released protocols publish no Erase bounds and refuse it in basic structure before executable handlers. Existing-input branches remain unchanged. This follows the same per-kind admission pattern as index-only deletion and shielded-payment branches.

The shipped versioned code gaining exhaustive Erase handling is:

  • DPP batch methods/v0, resolvers/v0, v0/v0_methods, v1/v0_methods, and validation/validate_basic_structure/v0.
  • ABCI batch advanced_structure/v0, advanced_structure/v1, state/v0 (including accepted_token_pool_nullifiers) and transformer/v0 (including property_constraint_aggregates).
  • Drive verify_state_transition_was_executed_with_proof/v0: Erase is explicitly classified as affected state, not execution-proved.

Unversioned exhaustive enum/accessor/dispatch updates accompany those arms. The complete appended-arm inventory is below.

Files with appended-kind handling
  • packages/rs-dpp/src/data_contract/document_type/action_fees/mod.rs
  • packages/rs-dpp/src/state_transition/mod.rs
  • packages/rs-dpp/src/state_transition/state_transitions/document/batch_transition/batched_transition/document_transition.rs
  • packages/rs-dpp/src/state_transition/state_transitions/document/batch_transition/batched_transition/document_transition_action_type.rs
  • packages/rs-dpp/src/state_transition/state_transitions/document/batch_transition/batched_transition/resolvers.rs
  • packages/rs-dpp/src/state_transition/state_transitions/document/batch_transition/batched_transition/token_transition.rs
  • packages/rs-dpp/src/state_transition/state_transitions/document/batch_transition/methods/mod.rs
  • packages/rs-dpp/src/state_transition/state_transitions/document/batch_transition/methods/v0/mod.rs
  • packages/rs-dpp/src/state_transition/state_transitions/document/batch_transition/resolvers/v0/mod.rs
  • packages/rs-dpp/src/state_transition/state_transitions/document/batch_transition/v0/v0_methods.rs
  • packages/rs-dpp/src/state_transition/state_transitions/document/batch_transition/v1/v0_methods.rs
  • packages/rs-dpp/src/state_transition/state_transitions/document/batch_transition/validation/validate_basic_structure/v0/mod.rs
  • packages/rs-drive-abci/src/execution/validation/state_transition/common/validate_document_not_expired.rs
  • packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/advanced_structure/v0/mod.rs
  • packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/advanced_structure/v1/mod.rs
  • packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/state/v0/accepted_token_pool_nullifiers.rs
  • packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/state/v0/mod.rs
  • packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/transformer/v0/mod.rs
  • packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/transformer/v0/property_constraint_aggregates.rs
  • packages/rs-drive/src/query/filter.rs
  • packages/rs-drive/src/state_transition_action/action_convert_to_operations/batch/document/document_transition.rs
  • packages/rs-drive/src/state_transition_action/batch/batched_transition/document_transition/document_transition_action_type.rs
  • packages/rs-drive/src/state_transition_action/batch/batched_transition/document_transition/mod.rs
  • packages/rs-drive/src/verify/state_transition/verify_state_transition_was_executed_with_proof/v0/mod.rs
  • packages/wasm-dpp/src/document/state_transition/batch_transition/document_transition/mod.rs
  • packages/wasm-dpp2/src/enums/batch/batch_enum.rs
  • packages/wasm-dpp2/src/state_transitions/batch/document_transition.rs
  • packages/rs-platform-wallet/src/wallet/identity/network/state_transition_summary.rs

Other shared-generation edits preserve released behavior:

  • DPP schema compatibility v1 freezes changes to a declared top-level canBeErased flag. Released grammars refuse that flag; comparisons without it, including contract definitions, keep the original compatibility rules.
  • Parser v3's body is shared through generation-owned constants; parser v1 explicitly keeps lifecycle admission off. Only parser 4 enables it. DocumentTypeV2 and compiled format v0 gain the flag using byte-compatible stored deletion variants; existing fixture bytes are pinned.
  • The public delete family, five delete wrappers v0 and contract update_keywords/v0 carry full BlockInfo and optional deleter context into version dispatch. Delete conversion 0 supplies no deleter, and generation 0 continues using the same timestamp and existing operations. Sponsorship sets the new deleter field, which only delete generation 1 reads. apply_drive_operations 0/1 and convert_drive_operations_to_grove_operations 0/1 pass no accumulated sibling operations; their PV15 generation 2 passes them for lifecycle-container deduplication. Shared force-delete lowering forwards siblings only for history types, which released force-delete handlers refuse; non-history lowering remains unchanged.
  • Insert v1 exposes the existing TTL helper to its sibling generation without changing its body. The retained-id guard in primary-storage writer v1 runs only on the composite history layout selected by PV15; released protocols retain the legacy layout.
  • Released version tables gain dormant optional fields only. Their existing values, batch formats and per-kind behavior are retained. Existing unreleased PV15 query/proof/history handlers are extended in place; no extra generation is stacked on them.

Breaking Changes

  • Rust document/batch/action enums gain appended Erase variants, and DocumentTypeV2 gains an erasability field; exhaustive matches and direct struct construction may require updates. No DocumentTypeV3 or new batch wire format is introduced.
  • Drive delete entry points accept the block context and optional deleter; DocumentOperationType::DeleteDocument gains deleter_id, and EraseDocument is added.
  • History v1 clients must handle DELETED, ERASING and the lifecycle fields. Generated gRPC clients are updated. Erase confirmation remains an affected-state observation.
  • At protocol 15, deletion is permitted for eligible history types, including default-deletable legacy types, and temporal reads hide deleted/erasing/erased documents according to the table above. Protocols through 14 retain their existing behavior and cannot admit Erase.

Checklist:

  • I have performed a self-review of my own code
  • My Rust, FFI, WASM, Swift or Kotlin changes follow the coding conventions
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have added "!" to the title and described breaking changes in the corresponding section if my code contains any
  • I have made corresponding changes to the documentation if needed
  • If I added or changed GroveDB structure, I described it in the area's structure.rs, regenerated grovedb-structure.json, and checked the structure viewer link posted on this pull request
  • If I added or changed a system contract's document schemas, I checked the contract viewer link posted on this pull request

For repository code-owners and collaborators only

  • I have assigned this pull request to a milestone

🤖 Generated with Claude Code

PR Hygiene · 84ac1e7

  • Bots — thepastaclaw ✓
  • Self-review — post /self-reviewed
  • Build failed
  • Approvals
    • files with no dedicated owner — you own it
    • js-wasm-sdk — you own it
    • dpp — you own it
    • rs-drive-abci — you own it
    • rs-drive — you own it
    • rs-platform-wallet (packages/rs-platform-wallet/src/wallet/identity/network/state_transition_summary.rs) — HashEngineering or ZocoLini or llbartekll or romchornyi
    • rust-sdk — you own it

When every merge requirement is met, the PR Hygiene check passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority: low Low priority — nice to have waiting-self-review Waiting for the author to post /self-reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants