Skip to content

feat(platform-wallet-storage): persist DashPay backfill coverage and outbound-account markers - #5479

Open
HashEngineering wants to merge 11 commits into
v5.1-devfrom
feat/sqlite-dashpay-backfill-persistence
Open

HashEngineering wants to merge 11 commits into
v5.1-devfrom
feat/sqlite-dashpay-backfill-persistence

Conversation

@HashEngineering

@HashEngineering HashEngineering commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Basic explanation

What this does: Teaches the SQLite wallet store to save two facts that #5026 added to the wallet's persistence contract: how far the DashPay "backfill" rescan has already got (so a restart does not start it again), and which account version each DashPay contact's outbound payment account was built for (so a restart does not rebuild it).

Value: Wallets on the SQLite backend stop re-walking the same block range on every launch once a DashPay backfill has run — the same fix #5026 gives the Kotlin host. On the reference mainnet wallet that re-walk was ~377,000 filters per launch. Contacts' outbound accounts also stop being torn down and rebuilt on every cold start.

Risks: Low. No consensus impact — this is client-side wallet storage only. The migration only adds nullable columns, so an upgraded database behaves exactly as before until the first backfill is recorded (one more re-run on first launch). The one behaviour change to existing data is that a stored sync cursor can now go down when Rust marks it as a deliberate rewind; ordinary advances still keep the highest value.

Issue being fixed or feature implemented

Follows #5026 (fixes #4302 for FFI hosts that store the record). That PR made the DIP-15 §12.6 coreHeight backfill durable through a new DashPayBackfillRecord that hosts store and hand back on load, plus a synced_height_is_rewind flag the host must honour. Kotlin implements it; the Swift SDK and the SQLite backend did not. This PR covers the SQLite backend:

  • It loaded every wallet with an empty record and kept a monotonic-max synced_height, so on this backend the backfill re-ran on every launch — the Persist DashPay coreHeight backfill completion (rescan_triggered is in-memory only) #4302 behaviour.
  • In FlushMode::Immediate it already qualified for outbound-account markers (ACCOUNT_ADDRESS_POOLS is an alias of the ASSET_LOCK_FUNDING_INDICES it declares), so markers were handed to it — and silently dropped, since no column held them.

iOS is not covered here. The Swift SDK does not register on_persist_wallet_dashpay_backfill_fn, so FFIPersister::stores_dashpay_backfill() is false there: coverage lasts one session and the backfill re-runs on a later launch. That needs its own Swift SDK follow-up (register the slot, keep the record in SwiftData, hand it back on load); this PR neither helps nor blocks it. Thanks to llbartekll for catching this.

What was done?

Migration V021__dashpay_backfill.rs (purely additive, nullable columns only — no new table, no blob)

Backfill record — src/sqlite/schema/dashpay_backfill.rs

  • apply replaces the wallet's record inside the store's transaction, after the round's account registrations: it clears every dashpay_covered_from, sets it on each covered account's row (addressed through the same typed-column helpers the account writer binds), then sets or clears the extent. An empty record only clears (the upstream invalidation).
  • A covered entry whose receiving account has no row is dropped, and if none remain the extent is cleared too. A contact missing from the record is only backfilled again, which is the safe direction.
  • The account upsert only rewrites account_xpub_bytes, so coverage survives the receiving account being re-registered. A receiving account's xpub derives from the wallet's own seed at index'/owner/contact, the row's key, so it never changes under a covered height.
  • load_for_wallet joins the columns back through DashPayBackfillRecord::from_entries (canonical order). Coverage without an extent reads as "no record". Wired into load_one_wallet → ClientWalletStartState::dashpay_backfill.
  • SqlitePersister::load() now reads inside one deferred read transaction, so the cursor and the coverage that suppresses the rescan below it come from one snapshot. Before, another process could commit a rewind plus its coverage between the two reads, and the load handed back the old cursor beside the new coverage.
  • SqlitePersister::stores_dashpay_backfill() returns true. Durable coverage still requires FlushMode::Immediate through the existing store_commits_inline gate in persists_backfill_coverage; Manual mode writes and clears records but keeps coverage session-only.
  • New Domain::DashPayBackfill in schema/versions.rs (replaces the explicit dashpay_backfill: _ ignore). Its version label is "dashpay_backfill", like "core" a domain rather than a table name.

Rewind — schema/core_state.rs::upsert_sync_state

  • Restore (rehydrate.rs::apply_persisted_core_state) now assigns the stored synced_height as written instead of maxing it against the birth_height - 1 seed — a backfill rewind can sit below the wallet's birth, and the FFI restore path already assigns it directly.
  • A synced_height marked synced_height_is_rewind overwrites the stored cursor; advances, last_processed_height and chainlock_height keep the monotonic max. A rewind can only lower the tombstone finality boundary (min(chainlock, synced)), so it never collects more.

Outbound-account marker — schema/contacts.rs

  • Written on established rows, restored on load. WalletPersister::store already strips it on backends that cannot keep it, and the manager's load clears restored markers it cannot trust. A trusted marker relies on apply_registrations overwriting the account xpub on rotation, plus the load-side check that distrusts a marker when an external pool holds addresses its current xpub does not derive.

Read-only backends — new defaulted PlatformWalletPersistence::refuses_writes() (false); SqlitePersister answers true under LoadPolicy::Recovery, where it already refuses every write. settle_published_backfill skips its replacement round for such a backend and keeps the result in memory, so load_from_persistor() serves the degraded projection instead of failing on ReadOnlyRecoveryMode. The record on file is left for a writable load to replace.

Identity API — ExternalAccountRegistration / RegisterExternalError are re-exported from platform_wallet::wallet::identity, so callers can match register_external_contact_account's outcome by type.

Docs — SCHEMA.md (the new columns on core_sync_state, account_registrations and contacts, the V021 log entry, the rewind exception on core_sync_state); a stale comment in rs-platform-wallet/src/changeset/changeset.rs that said SQLite ignores the rewind flag.

How Has This Been Tested?

cargo test -p platform-wallet-storage --all-features (589 integration + 438 unit, all pass; cargo test -p platform-wallet --all-features --lib: 1778 pass) and cargo clippy -p platform-wallet-storage -p platform-wallet --all-features --all-targets -- -D warnings (clean).

  • tests/integration/sqlite_dashpay_backfill_relaunch.rs (new, end-to-end): the real PlatformWalletManager over a real SqlitePersister, with the file closed and reopened between sessions:

    • Persist DashPay coreHeight backfill completion (rescan_triggered is in-memory only) #4302 regression: the backfill rewinds once (cursor 1000 → 100, the record on disk), the scan climbs to 400, then the relaunch restores 400 and does not rewind again. Control: with the record's columns cleared, the relaunch rewinds again. Mutation-checked against the rewind arm.
    • The outbound-account marker comes back with its DashpayExternalAccount present and the same xpub, so registering it again reports AlreadyExisted.
    • A Recovery reopen publishes the wallet without writing in each case where the manager would replace the record: a drifted covered receiving account, a Manual-mode backend over stored coverage, and a remove-then-reload on the same manager (mutation-checked: ignoring refuses_writes fails all three with ReadOnlyRecoveryMode).
    • Adds test_support::encrypt_contact_xpub_for_test (test-utils only) so the storage tests can build a contact's encrypted xpub offline.
  • tests/integration/sqlite_dashpay_backfill.rs (new):

    • Immediate vs Manual capability split.
    • The record round-trips across a reopen, is empty when never written, is replaced wholesale, and is cleared by an empty write.
    • Coverage without an extent reads as no record.
    • A covered entry whose receiving account has no row is dropped; dropping all of them clears the record.
    • Coverage survives the receiving account being re-registered.
    • Rewind vs advance on the stored cursor, and the rewound cursor restored beside its record.
    • Manual-mode last-write-wins.
    • Rewind ordering through the Manual buffer, across a failed-flush restore, and across identity re-add segments.
    • V020→V021 upgrade (no marker, no record) and delete-wallet cleanup.
    • Mutation-checked: dropping the rewind arm fails all five cursor tests.
    • A rewound cursor below the wallet's birth restores as written (mutation-checked).
    • load() reads the cursor and its coverage from one snapshot: a second connection commits a rewind plus coverage between the two reads (mutation-checked).
  • sqlite_contacts_keys_rehydration.rs: the outbound-account marker round-trips.

  • sqlite_version_bump.rs, sqlite_schema_pinning.rs, the new domain and the migration fingerprints.

Breaking Changes

None. The schema change is an additive migration; an older binary cannot open a V021 database, which is the existing forward-version gate for every migration.

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have added "!" to the title and described breaking changes in the corresponding section if my code contains any
  • I have made corresponding changes to the documentation if needed
  • If I added or changed GroveDB structure, I described it in the area's structure.rs, regenerated grovedb-structure.json, and checked the structure viewer link posted on this pull request

For repository code-owners and collaborators only

  • I have assigned this pull request to a milestone

🤖 Generated with Claude Code

PR Hygiene · 8e1ee2e

  • Bots — thepastaclaw ✓
  • Self-review — post /self-reviewed
  • Build green
  • Approvals
    • wallet-storage (packages/rs-platform-wallet-storage/SCHEMA.md, packages/rs-platform-wallet-storage/migrations/V021__dashpay_backfill.rs, packages/rs-platform-wallet-storage/src/sqlite/persister.rs and 13 more) — lklimek
    • rs-platform-wallet (packages/rs-platform-wallet/src/changeset/changeset.rs, packages/rs-platform-wallet/src/changeset/traits.rs, packages/rs-platform-wallet/src/manager/mod.rs and 4 more) — approved by llbartekll

When every merge requirement is met, the PR Hygiene check passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.

Summary by CodeRabbit

  • New Features
    • Wallet data now preserves DashPay backfill progress, covered receiving accounts, and outbound-account markers across reloads and relaunches.
    • Rewound sync cursors are restored directly, so backfill can resume from the saved position.
    • Recovery-mode loading avoids persisting changes to read-only storage.
    • External account registration types are now available through the public identity interface.
  • Bug Fixes
    • Wallet data is loaded from a consistent database snapshot.

@coderabbitai

coderabbitai Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

SQLite V021 adds storage for DashPay backfill records, covered receiving accounts, and established-contact external account references. Wallet persistence now stores and restores backfill records and rewind cursors. Read-only recovery handling avoids durable backfill writes when the persister refuses them.

Changes: DashPay SQLite persistence

Layer / File(s) Summary
V021 schema and persistence domain
packages/rs-platform-wallet-storage/migrations/V021__dashpay_backfill.rs, packages/rs-platform-wallet-storage/src/sqlite/schema/versions.rs, packages/rs-platform-wallet-storage/src/sqlite/schema/mod.rs, packages/rs-platform-wallet-storage/SCHEMA.md, packages/rs-platform-wallet-storage/tests/integration/sqlite_schema_pinning.rs, packages/rs-platform-wallet-storage/tests/integration/sqlite_version_bump.rs, packages/rs-platform-wallet-storage/tests/integration/sqlite_dashpay_backfill.rs
V021 adds nullable backfill extent, receiving-account coverage, and contact-marker columns. The SQLite changeset domain tracks backfill records, and migration and domain tests cover the added state.
Persist and restore backfill state
packages/rs-platform-wallet-storage/src/sqlite/schema/dashpay_backfill.rs, packages/rs-platform-wallet-storage/src/sqlite/schema/core_state.rs, packages/rs-platform-wallet-storage/src/sqlite/persister.rs, packages/rs-platform-wallet-storage/src/sqlite/rehydrate.rs, packages/rs-platform-wallet/src/changeset/changeset.rs, packages/rs-platform-wallet-storage/tests/integration/sqlite_dashpay_backfill.rs, packages/rs-platform-wallet-storage/tests/integration/sqlite_dashpay_backfill_relaunch.rs, packages/rs-platform-wallet-storage/tests/integration/sqlite_load_reconstruction.rs, packages/rs-platform-wallet-storage/tests/integration/main.rs
SQLite persists backfill coverage and rewind cursors, and loads the wallet state from one snapshot. Rehydration restores the persisted cursor. Integration tests cover record replacement, clearing, reload, and relaunch.
Handle backfill with write-refusing persistence
packages/rs-platform-wallet/src/changeset/traits.rs, packages/rs-platform-wallet/src/manager/mod.rs, packages/rs-platform-wallet-storage/src/sqlite/persister.rs, packages/rs-platform-wallet-storage/tests/integration/sqlite_dashpay_backfill_relaunch.rs
The persistence trait reports whether writes are refused. Backfill settlement skips durable replacement in that case. Recovery tests check loaded wallet coverage and unchanged database state.
Persist established-contact account markers
packages/rs-platform-wallet-storage/src/sqlite/schema/contacts.rs, packages/rs-platform-wallet-storage/SCHEMA.md, packages/rs-platform-wallet-storage/tests/integration/sqlite_contacts_keys_rehydration.rs, packages/rs-platform-wallet-storage/tests/integration/sqlite_dashpay_backfill_relaunch.rs, packages/rs-platform-wallet/src/wallet/identity/..., packages/rs-platform-wallet/src/test_support.rs
Established-contact storage writes and restores the optional external account reference. Reopen and relaunch tests check the marker and related outbound-account state. The identity network re-exports external registration types.

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant WalletManager
  participant SqlitePersister
  participant SQLite
  participant RelaunchedWalletManager
  WalletManager->>SqlitePersister: Submit changeset with rewind and backfill record
  SqlitePersister->>SQLite: Write cursor and record in transaction
  RelaunchedWalletManager->>SqlitePersister: Load wallet start state
  SqlitePersister->>SQLite: Read cursor and record in one snapshot
  SQLite-->>SqlitePersister: Return persisted wallet state
  SqlitePersister-->>RelaunchedWalletManager: Restore cursor and record
Loading

Suggested labels: Client Only

Suggested reviewers: thepastaclaw

Merge Risk: Minimal · up to 8e1ee

This change adds SQLite persistence for DashPay backfill coverage and contact account markers, with recovery-mode write refusal. No actionable merge-blocking risk was identified in the supplied evidence.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title clearly summarizes the main changes: SQLite persistence for DashPay backfill coverage and outbound-account markers.
Linked Issues check Passed Issues #5026 and #4302 are closed and provide historical context only. No active directly linked issue sets coding requirements for this PR.
Out of Scope Changes check Passed The changes support the current PR intent: SQLite persistence for DashPay backfill coverage and outbound-account markers, rewind-aware cursor handling, and recovery-mode loading. The migration, schema…
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Out of Scope Changes check

Explanation

The changes support the current PR intent: SQLite persistence for DashPay backfill coverage and outbound-account markers, rewind-aware cursor handling, and recovery-mode loading. The migration, schema documentation, re-exports, and tests support that work. The test-only sweep helper explicitly supports storage tests. No unrelated change is demonstrated.

Finishing Touches

Generate docstrings

  • Commit to this branch
  • Create a new PR
Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 2708f7d and 8e1ee2e.

Files selected for processing: 7

  • packages/rs-platform-wallet-storage/SCHEMA.md
  • packages/rs-platform-wallet-storage/migrations/V021__dashpay_backfill.rs
  • packages/rs-platform-wallet-storage/src/sqlite/persister.rs
  • packages/rs-platform-wallet-storage/tests/integration/sqlite_dashpay_backfill.rs
  • packages/rs-platform-wallet-storage/tests/integration/sqlite_load_reconstruction.rs
  • packages/rs-platform-wallet-storage/tests/integration/sqlite_schema_pinning.rs
  • packages/rs-platform-wallet-storage/tests/integration/sqlite_version_bump.rs
Files skipped from review as they are similar to previous changes: 1

  • packages/rs-platform-wallet-storage/SCHEMA.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

@thepastaclaw

thepastaclaw commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Final review complete — no blockers (commit 8e1ee2e) · triage: critical

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final validation — Phase 1 + Phase 2

Verified the supplied findings against head b3a06b1. Two blocking persistence defects remain: restoration can advance an explicitly rewound cursor, and the backfill writer accepts records that its reader rejects; the new integration tests also expose a smaller public-API usability issue. This was a static review: current CI checks pass, including Kotlin build/tests, but do not independently confirm the reported Rust storage-suite results.

🔴 2 blocking | 🟡 1 suggestion(s)

Review provenance

Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: architecture-layering); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: platform-versioning); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: rust-quality); reviewer 5: gemini-3.8-flash-high (agent: phase1-reviewer, role: general); reviewer 6: gemini-3.8-flash-high (agent: phase1-reviewer, role: rust-quality); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: critical by gpt-6.1-sol (effort low) — The change intricately couples durable backfill coverage, cursor rewinds, buffered persistence and restart recovery while changing storage schema in migrations/V019__dashpay_backfill.rs and cursor semantics in core_state.rs::upsert_sync_state.
  • Phase 1 reviewers: gemini-3.8-flash-high — general (completed, effort high); agent phase1-reviewer, gemini-3.8-flash-high — rust-quality (completed, effort high); agent phase1-reviewer
  • Phase 1 model: gemini-3.8-flash-high — antigravity quota: weekly 71% left, 5h 100% left
  • Single stage: Phase 1 and Phase 2 reviewed this head side by side, with no blocker gate between them (triage tier)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — architecture-layering (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — platform-versioning (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — rust-quality (completed, effort xhigh); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.

In `packages/rs-platform-wallet-storage/src/sqlite/rehydrate.rs`:
- [BLOCKING] packages/rs-platform-wallet-storage/src/sqlite/rehydrate.rs:293-299: Preserve an explicit rewind below the wallet-birth seed
  The claimed clamp does not exist in the pinned dependency: WalletInfoInterface::update_synced_height directly assigns its argument, and reconcile_dashpay_rescan passes the contact checkpoint without clamping it to wallet birth. For example, a wallet with birth_height = 501 can persist a backfill cursor of 100, but reopening seeds synced_height at 500 and this max advances the restored cursor to 500. Because this PR now restores the coverage record too, reconciliation treats the contact as covered and does not rewind again. When the shared scanner has history below this wallet's birth, such as alongside an older wallet, the unscanned range between the saved cursor and 500 is skipped for this wallet. Restore the explicit stored synced_height instead of treating the initialization seed as observed progress, and add a relaunch regression with a checkpoint below a nonzero birth height.

In `packages/rs-platform-wallet-storage/src/sqlite/schema/dashpay_backfill.rs`:
- [BLOCKING] packages/rs-platform-wallet-storage/src/sqlite/schema/dashpay_backfill.rs:37-42: Reject coverage blobs that the load path cannot read
  The writer inserts record.covered_bytes() without enforcing the 16 MiB limit checked by load_for_wallet at line 75. DashPayBackfillRecord::from_entries does not cap the entry count, V019 only requires a nonempty multiple of 72 bytes, and the connection's SQLite limit is 32 MiB. A record containing 233,017 distinct account keys therefore produces a 16,777,224-byte blob that store can successfully commit, but the next default load fails with BlobTooLarge and aborts restoration; Recovery mode omits the affected wallet. Although this requires a large coverage set, it lets a successful persistence operation make the wallet unrestorable through the normal load path. Check the encoded length before allocating and writing the blob, and add a boundary regression proving rejection preserves both the previous record and its cursor.

In `packages/rs-platform-wallet-storage/tests/integration/sqlite_dashpay_backfill_relaunch.rs`:
- [SUGGESTION] packages/rs-platform-wallet-storage/tests/integration/sqlite_dashpay_backfill_relaunch.rs:469: Expose registration outcomes instead of asserting Debug strings
  The new test compares the Debug strings "Built" and "AlreadyExisted" because ExternalAccountRegistration is public inside the private network::contacts module but is not re-exported. The public DashPayView method therefore returns an outcome whose variants external callers cannot name, and the integration test works around that API gap by depending on presentation text rather than typed outcomes. Re-export ExternalAccountRegistration and RegisterExternalError from the wallet's public identity API, then use typed assertions here and at the relaunch assertion. This is a non-blocking API-usability improvement, not a layer-ownership violation.
Out-of-scope follow-up suggestions (1)

These are valid observations, but they are outside this PR's scope and should be handled in separate issues or author/maintainer-requested PRs rather than blocking this review.

  • Track SQLite outbound-account rotation conflicts separately — Both the base and reviewed head reject an outbound-account rebuild when a genuinely rotated xpub supplies different typed pool keys at existing indices: core_pool::apply_pools returns TypedPoolKeyConflict. The rotation teardown removes only in-memory accounts, leaving the conflicting SQLite pool rows in place for subsequent retries. This predates the PR, and the registration transaction rolls back rather than persisting a falsely trusted marker.
    • Follow-up: Track a separate SQLite rotation operation that atomically replaces the old outbound pool with its account registration, preserves immutable-key protections for other account types, and covers rotation plus reopen through the real manager.

Base automatically changed from fix/dashpay-backfill-persistence-4302 to v5.0-dev October 17, 2026 03:29
@github-actions github-actions Bot added this to the v5.0.0 milestone Oct 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Waiting for bot review — coderabbitai not yet · thepastaclaw requested changes — dismiss the review or push a fix, 2 threads unresolved — resolve them. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added the waiting-bots Waiting for the review bots to report on this head label Oct 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix; thepastaclaw left review threads unresolved; resolve them.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed bot-review-skipped A required review bot did not report; it was skipped by the window or by a person. and removed waiting-bots Waiting for the review bots to report on this head labels Oct 17, 2026
@HashEngineering
HashEngineering force-pushed the feat/sqlite-dashpay-backfill-persistence branch from b3a06b1 to 88283da Compare October 17, 2026 17:21
@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Waiting for bot review — coderabbitai ✓ · thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed waiting-self-review Waiting for the author to post /self-reviewed bot-review-skipped A required review bot did not report; it was skipped by the window or by a person. labels Oct 17, 2026

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

Reviewed the complete diff at 88283da and independently confirmed that all three prior findings are fixed, with no remaining in-scope defects identified. SQLite preserves atomic record/cursor writes, restores deliberate rewinds faithfully, and persists outbound-account markers while leaving wallet policy in rs-platform-wallet. Validation was static locally; current exact-head CI passes scoped Rust wallet tests and clippy, Swift schema/build/tests, and Kotlin build/tests, while the full Rust workspace suite was skipped.

🔴 0 blocking | 🟡 0 suggestion(s) | 💬 0 nitpick(s)

Review provenance

Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: architecture-layering); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: platform-versioning); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: rust-quality); reviewer 5: gpt-6-astra (agent: phase2-reviewer, role: general); reviewer 6: gpt-6-astra (agent: phase2-reviewer, role: architecture-layering); reviewer 7: gpt-6-astra (agent: phase2-reviewer, role: platform-versioning); reviewer 8: gpt-6-astra (agent: phase2-reviewer, role: rust-quality); reviewer 9: gemini-3.8-flash-high (agent: phase1-reviewer, role: general); reviewer 10: gemini-3.8-flash-high (agent: phase1-reviewer, role: rust-quality); reviewer 11: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 12: gpt-6.1-sol (agent: phase2-reviewer, role: architecture-layering); reviewer 13: gpt-6.1-sol (agent: phase2-reviewer, role: platform-versioning); reviewer 14: gpt-6.1-sol (agent: phase2-reviewer, role: rust-quality); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: critical by gpt-6.1-sol (effort low) — The change combines a storage migration in migrations/V019__dashpay_backfill.rs with intricate persistence and recovery semantics spanning atomic backfill records, deliberate cursor rewinds, and restart rehydration.
  • Phase 1 reviewers: gemini-3.8-flash-high — general (completed, effort high); agent phase1-reviewer, gemini-3.8-flash-high — rust-quality (completed, effort high); agent phase1-reviewer
  • Phase 1 model: gemini-3.8-flash-high — antigravity quota: weekly 71% left, 5h 100% left
  • Single stage: Phase 1 and Phase 2 reviewed this head side by side, with no blocker gate between them (triage tier)
  • Fresh final gate: an independent Phase-2 review ran after iterative findings were reconciled
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — architecture-layering (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — platform-versioning (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — rust-quality (completed, effort xhigh); agent phase2-reviewer, gpt-6-astra — general (completed, effort xhigh); agent phase2-reviewer, gpt-6-astra — architecture-layering (completed, effort xhigh); agent phase2-reviewer, gpt-6-astra — platform-versioning (completed, effort xhigh); agent phase2-reviewer, gpt-6-astra — rust-quality (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — general (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — architecture-layering (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — platform-versioning (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — rust-quality (completed, effort xhigh); agent phase2-reviewer
  • Model comparison: every Phase-2 reviewer also ran on gpt-6-astra; the verifier weighed both sets without knowing which model wrote which
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify the current code and confirm that no unresolved issues remain.

No unresolved findings remain from the prior review on this head.
Out-of-scope follow-up suggestions (1)

These are valid observations, but they are outside this PR's scope and should be handled in separate issues or author/maintainer-requested PRs rather than blocking this review.

  • Coordinate explicit below-birth backfills with the SPV scan floor — Both the pinned and current default rust-dashcore revisions derive earliest_required_height from the minimum loaded-wallet birth height, and dash-spv applies that floor during scan startup and restart. With only a wallet born at 501 loaded, correctly restoring its cursor to 100 therefore does not make the scanner traverse heights 101–500, although reconcile_dashpay_rescan can record coverage from that earlier checkpoint. This is a concrete pre-existing wallet/scanner contract limitation, not a defect introduced by SQLite persistence; restoring the stored cursor remains the correct adapter behavior.
    • Follow-up: Track a separate coordinated platform-wallet/rust-dashcore fix that communicates explicit backfill requirements to the scanner and ensures recorded coverage reflects the reachable scan range. Include a real-scanner regression with a contact checkpoint below the only loaded wallet's birth height.

@github-actions

Copy link
Copy Markdown
Contributor

Bots are done — your move: post /self-reviewed.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Oct 17, 2026
@HashEngineering

Copy link
Copy Markdown
Collaborator Author

/self-reviewed

@HashEngineering HashEngineering self-assigned this Oct 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Ready for review — wallet-storage: lklimek · rs-platform-wallet: ZocoLini or llbartekll or romchornyi.
Full checklist in the description.

@github-actions github-actions Bot added ready-for-human Bots have reported, the author has self-reviewed, and the build is green: this needs a human. and removed waiting-self-review Waiting for the author to post /self-reviewed labels Oct 17, 2026
@github-actions github-actions Bot removed the waiting-bots Waiting for the review bots to report on this head label Nov 9, 2026
@HashEngineering

Copy link
Copy Markdown
Collaborator Author

/self-reviewed

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Ready for review — wallet-storage: lklimek · rs-platform-wallet: ZocoLini or llbartekll or romchornyi.
Full checklist in the description.

@github-actions github-actions Bot added ready-for-human Bots have reported, the author has self-reviewed, and the build is green: this needs a human. and removed waiting-self-review Waiting for the author to post /self-reviewed labels Nov 9, 2026
@github-actions
github-actions Bot requested a review from lklimek November 9, 2026 23:27
HashEngineering and others added 11 commits November 10, 2026 08:05
…outbound-account markers

The SQLite backend loaded every wallet with an empty DashPay backfill
record and kept a monotonic-max cursor, so on this backend the coreHeight
backfill re-ran on every launch (#4302). Outbound-account
markers were handed to it in Immediate mode (ACCOUNT_ADDRESS_POOLS is an
alias of the ASSET_LOCK_FUNDING_INDICES it already declares) but silently
dropped, since no column held them, so each cold start rebuilt every
outbound account.

- V019 adds `dashpay_backfill` (one row per wallet: floor, rewound_from,
  and the covered receival accounts as one `covered_bytes` blob — the
  layout the FFI hosts store) and a nullable
  `contacts.external_account_reference`.
- The record is written in the store's transaction beside the cursor it
  vouches for; every write replaces it whole and an empty record clears it.
  Load hands it back through `ClientWalletStartState::dashpay_backfill`; a
  torn cover set reads as no record, the codec's safe direction.
- `stores_dashpay_backfill()` is true; durable coverage still needs
  `FlushMode::Immediate` via `store_commits_inline`.
- A cursor marked `synced_height_is_rewind` overwrites the stored
  `synced_height`; advances keep the monotonic max. Tests pin the order
  through the Manual-mode buffer, a failed-flush restore, and identity
  re-add segments.
- New `DashPayBackfill` version domain; migration fingerprints bumped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…irth and refuse unreadable cover sets

- Restore the stored `synced_height` as written instead of maxing it
  against the `birth_height - 1` seed. A DashPay backfill rewind can lower
  the cursor below the wallet's birth (`reconcile_dashpay_rescan` passes
  the contact checkpoint to `update_synced_height` unclamped), and the
  restored record vouches for that cursor, so the seed skipped the range
  between them. Matches the FFI restore path; replaces a comment that
  claimed a clamp the pinned key-wallet does not have.
- Refuse, before writing, a backfill record whose encoded cover set
  exceeds the blob limit the loader reads under, so a successful store can
  never leave the wallet unloadable. The round fails whole; the record and
  cursor already on file stand.
- Re-export `ExternalAccountRegistration` and `RegisterExternalError` from
  the identity API so callers (and the relaunch test) can match the
  outcome of `register_external_contact_account` by type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…as plain columns

Review asked for values rather than a blob, on the tables that already
hold the facts. The record's extent moves to
core_sync_state.dashpay_backfill_floor / dashpay_backfill_rewound_from,
beside the cursor it lowered, and each covered entry becomes
account_registrations.dashpay_covered_from on the receiving account it
names. The dashpay_backfill table and its 72-byte cover-set blob are
gone, and with them the blob size limit the writer had to enforce.

A covered entry whose receiving account has no row is dropped (re-armed,
the safe direction); when none remain the extent is cleared too. The
account upsert only rewrites the xpub, so coverage survives a rotation.

The migration is renumbered V022: V019 is taken on v5.1-dev and V020 /
V021 are claimed by open PRs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
load() read each table in its own autocommit statement, so another
process could commit between two of them. A rewound cursor plus its
backfill coverage landing after the cursor read and before the coverage
read handed back the old cursor beside coverage that suppresses the
rescan below it. The whole load now runs in one deferred read
transaction, which in WAL mode pins the snapshot every read sees.

The regression test injects that commit from a second connection while
load() is between the two reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on to V020

v5.1-dev ends at V019 and now requires migration versions to be unique
and contiguous from V001 (#5553), so the migration takes the next free
version instead of skipping ahead of open PRs. The two schema
fingerprint pins follow the rename.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… own xpub and count the load snapshot

A receiving account's xpub derives from the wallet's seed at
index'/owner/contact, the row's key, so re-registering it carries the
same xpub; the test no longer implies coverage survives a different
key. The load statement-count bound allows the BEGIN and COMMIT of the
read snapshot load() now runs in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… instead of zeroing it

The coverage loader turned each identity column into an Identifier
through a helper that fell back to the all-zero id on a wrong length.
The width check before it made that unreachable, but a future caller
skipping the check would have attributed coverage to the zero identity.
Convert through the shared checked id32 instead, so a bad id is an
error like every other identity column.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…backend refuses writes

A SqlitePersister opened under LoadPolicy::Recovery refuses every store,
but publishing a wallet could still replace the host's DashPay backfill
record: when the load dropped coverage for a receiving account it did
not restore, when the backend cannot hold coverage durably (Manual), or
when the same manager had published the wallet before. The refused
write failed load_from_persistor instead of serving the degraded,
read-only projection.

PlatformWalletPersistence gains a defaulted refuses_writes(), which
SqlitePersister answers under Recovery. settle_published_backfill skips
the replacement round for such a backend and keeps its result in
memory; the record on file waits for a writable load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed type and the sent-sweep gate

Two base changes broke the SQLite relaunch tests after the rebase:

- rust-dashcore past the FFI removal (#5729) returns the BIP-39 seed as
  `Zeroizing<[u8; 64]>`, so `seed_wallet` dereferences it for
  `Wallet::from_seed_bytes`.
- #4740 defers a receiving account's rescan until a sent-request sweep has
  completed in this process. These tests seed contacts without a DashPay
  sync, so every pass returned `None` and the session-one rewind failed.
  After a relaunch the gate also made "a covered contact is not rewound
  again" pass whether or not the persisted record covered it.

`ManagedIdentity::mark_sent_sweep_completed_for_test` (behind `test-utils`)
lets the tests mark the sweep as each launch's sync would: after
establishing a contact and in every relaunched process before its pass. The
control that clears the record again rewinds, so the record is still what
stops the second rewind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uses_writes

`ensure_writable` repeated the `LoadPolicy::Recovery` comparison that
`refuses_writes` reports to the manager. Delegating to it keeps the gate that
refuses a write and the capability the manager reads before its own
housekeeping writes on one predicate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on to V021

#5762 merged `V020__asset_lock_shielded_recipient.rs`, and
`embedded_migration_versions_are_unique_and_contiguous` (#5553) requires one
migration per version from V001 with no gaps, so the backfill migration moves
to the next free version. Its SQL is unchanged.

- SCHEMA.md cites V021 and drops two stale history rows left by the rebase.
- The upgrade test migrates to V020, the last pre-backfill schema.
- Both schema pins are recomputed: refinery checksums the name and version.
- rusqlite 0.41 on the base returns a `Result` from `trace_v2` and needs the
  callback type when clearing it; the snapshot test now matches the base's
  pattern.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@HashEngineering
HashEngineering force-pushed the feat/sqlite-dashpay-backfill-persistence branch from 2708f7d to 8e1ee2e Compare November 10, 2026 16:45
@HashEngineering

HashEngineering commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator Author

Rebased onto v5.1-dev and renumbered the migration to V021 (V021__dashpay_backfill.rs): #5762 merged V020__asset_lock_shielded_recipient.rs, and #5553's contiguity test needs one migration per version. The migration SQL is unchanged.

The rebase also needed two test-only adaptations to the new base:

  • rusqlite 0.41: trace_v2 now returns a Result and needs a typed None to clear the callback. The snapshot test uses the base's pattern.
  • Schema pins: both recomputed for V001–V021.

@lklimek @llbartekll: since you wrote and merged #5762, note that the upgrade test now starts from V020, so it includes the asset_locks.shielded_recipient column, then applies V021. Storage tests (589 integration + 438 unit), platform-wallet lib (1778) and clippy -D warnings pass.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed ready-for-human Bots have reported, the author has self-reviewed, and the build is green: this needs a human. labels Nov 10, 2026

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

At head 8e1ee2e, the additive V021 migration follows the base's V020 migration, and the rebase adaptations preserve the intended persistence behavior; no actionable in-scope defects remain. Current CI reports successful Rust wallet, Swift SDK, and Kotlin SDK checks, while the full Rust workspace suite was skipped and review/hygiene checks remain pending. This verification was static; no local builds or tests were run.

🔴 0 blocking | 🟡 0 suggestion(s) | 💬 0 nitpick(s)

Review provenance

Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: architecture-layering); reviewer 3: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: platform-versioning); reviewer 5: gpt-6.1-sol (agent: phase2-reviewer, role: rust-quality); reviewer 6: gpt-6-astra (agent: phase2-reviewer, role: general); reviewer 7: gpt-6-astra (agent: phase2-reviewer, role: architecture-layering); reviewer 8: gpt-6-astra (agent: phase2-reviewer, role: platform-versioning); reviewer 9: gpt-6-astra (agent: phase2-reviewer, role: rust-quality); reviewer 10: muse-spark-1.3-contributor (agent: phase1-reviewer, role: rust-quality); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: critical by gpt-6.1-sol (effort low) — The change combines intricate transactional persistence, rewind handling, and restart reconstruction with a storage migration in packages/rs-platform-wallet-storage/migrations/V021__dashpay_backfill.rs, meeting both the complexity and critical-surface criteria.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer, muse-spark-1.3-contributor — rust-quality (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Single stage: Phase 1 and Phase 2 reviewed this head side by side, with no blocker gate between them (triage tier)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — architecture-layering (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — platform-versioning (completed, effort xhigh); agent phase2-reviewer, gpt-6.1-sol — rust-quality (completed, effort xhigh); agent phase2-reviewer, gpt-6-astra — general (completed, effort xhigh); agent phase2-reviewer, gpt-6-astra — architecture-layering (completed, effort xhigh); agent phase2-reviewer, gpt-6-astra — platform-versioning (completed, effort xhigh); agent phase2-reviewer, gpt-6-astra — rust-quality (completed, effort xhigh); agent phase2-reviewer
  • Model comparison: every Phase-2 reviewer also ran on gpt-6-astra; the verifier weighed both sets without knowing which model wrote which
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify the current code and confirm that no unresolved issues remain.

No unresolved findings remain from the prior review on this head.

@github-actions

Copy link
Copy Markdown
Contributor

Bots are done — your move: post /self-reviewed.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Nov 10, 2026
@lklimek

lklimek commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

@HashEngineering note the #5150 based stack also has conflicting migrations, we need to handle that carefully.

@llbartekll llbartekll left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving the rs-platform-wallet part at 8e1ee2e2. That is the only part I read, because it is the only part that links into the iOS FFI.

Scope

  • Read: the 7 files under packages/rs-platform-wallet/src.
  • Not read: rs-platform-wallet-storage (the V021 migration, the SQLite persister, the schema code) and its tests. iOS stores through SwiftData and does not use that backend, so I leave it to the other reviewers.

The wallet-crate part

  • No behaviour change for a host on FFIPersister. refuses_writes() defaults to false, only SqlitePersister overrides it (LoadPolicy::Recovery), and FFIPersister does not. The new early return in settle_published_backfill is therefore never taken on iOS.
  • The rest is inert for a host. A doc comment on synced_height_is_rewind, two re-exports of types that are already public in contacts.rs, and two test helpers. test_support is behind cfg(any(test, feature = "test-utils")) and mark_sent_sweep_completed_for_test carries the same gate, so neither reaches a release build.

iOS and the backfill record

You asked whether the re-run is an issue on iOS. I checked v5.1-dev at c8a284b6c0:

  • FFIPersister::stores_dashpay_backfill() is true only when the host registers on_persist_wallet_dashpay_backfill_fn in PersistenceCallbacksExtension.
  • The Swift SDK's makePersistenceCallbacksExtension() does not set that slot, and nothing under packages/swift-sdk/Sources refers to the record.

So on iOS the coverage is session-only, and by the reasoning in this PR's description the backfill runs again on a later launch. I have not measured it on a device. Two consequences:

  • The description's line that #5026 gives this fix to "the Kotlin and Swift hosts" holds for Kotlin only.
  • iOS needs its own follow-up in the Swift SDK: register the slot, keep the record in SwiftData and hand it back on load. This PR neither helps nor blocks that.

@HashEngineering

Copy link
Copy Markdown
Collaborator Author

@lklimek thanks, agreed. Here's where the numbers stand on v5.1-dev (which ends at V020 since #5762):

PR Migration
#5479 V021__dashpay_backfill.rs
#5150 V021__core_transaction_accounting.rs
#5220 (on #5150) V022__core_history_rescan.rs

#5781 and #5384 in the stack add none. Whichever side merges second renumbers. Which order do you prefer?

#5479 can go after the #5150 stack, so if that is easier for you, land the stack first and I'll renumber this one to V023.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-self-review Waiting for the author to post /self-reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Persist DashPay coreHeight backfill completion (rescan_triggered is in-memory only)

4 participants