diff --git a/AGENTS.md b/AGENTS.md index 47199ae7b..3f5ce5989 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -114,6 +114,7 @@ Code lives by responsibility, not convenience: - **`ui/components/`** — reusable **Component-pattern widgets ONLY**: a `show()` plus a `ComponentResponse`, a display-only render widget, or component infrastructure. If it does not render egui, it is not a component. - **`ui/state/`** — non-widget UI state: per-screen view-models and async fetch-state caches (e.g. `TrackedAssetLockCache`). Owned by screens, may return `BackendTask`, render nothing. - **`src/mcp/tools/`** — MCP tool logic, one file per domain (e.g. `wallet.rs`, `shielded.rs`, `identity.rs`) with multiple tool structs per file; never in `src/bin/det_cli/`. +- **`src/utils/file_deletion.rs`** — the single file-deletion chokepoint. Every production on-disk deletion goes through `delete_file` (or `delete_tree`) with a `DeletionIntent` stating its purpose; the chokepoint hard-refuses live databases, their sidecars, the vault and `.env`. `clippy.toml` disallows raw `std::fs`/`tokio::fs` remove calls; any deliberate exception gets `#[expect(clippy::disallowed_methods, reason = "...")]` at its call site. - **`src/localization.rs`** — localization logic. **`src/ui/theme.rs`** — theme/alignment helpers. Discriminator for `ui/components/` vs `ui/state/`: *does it render egui (`show`/`ui`/a render fn)?* Yes → component. No → state. diff --git a/CHANGELOG.md b/CHANGELOG.md index b88d8ec10..6009343b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -48,6 +48,13 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added +- **Upgrade backup retention setting**: Settings → Upgrade Backups controls how + long the backup copies taken before a storage upgrade are kept. By default, + backups older than 90 days are now deleted automatically; choose another + period (1 to 3650 days) or keep them forever. The newest complete backup of + each database is always kept (or the newest backup, if none is complete), and + a shorter period asks for confirmation because it deletes backups at once. + - **Add wallet-created identity keys**: for identities loaded from a wallet on this device, the Add Key screen defaults to "Create from wallet", which creates an ECDSA_SECP256K1 or ECDSA_HASH160 key from that wallet so it can be @@ -147,6 +154,28 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **Safer guidance for wallet data this version cannot open**: the message no + longer tells you to remove your local wallet data, which could have deleted + the `secrets` folder along with keys no recovery phrase can restore. It asks + you to write down your recovery phrases and imported keys in the version you + used before, close every running instance, and set aside the `.sqlite` files + while keeping the `secrets` folder. A database held by another session asks + you to close that session and try again. + +- **Upgrade backups no longer pile up**: repeated failed upgrades no longer + leave an identical backup on every attempt, and a failed upgrade that can + never succeed shows its message instead of retrying silently. Clearing a + network's data removes that network's upgrade backups; backups of the shared + app data are left to the retention setting. Removing a wallet or an identity + keeps upgrade backups. + +- **Clearing a network's data works with moved chain data**: chain-sync data + that was moved to another disk and linked back no longer stops the clear. + +- **Clearer identity funding errors**: when the wallet cannot save the data a + payment needs, the message now distinguishes a busy or full store from other + failures and says how to retry. + - **Identity keys added by hand no longer disappear**: unlocking a wallet, starting the app or loading an identity from a wallet refreshed the identity with only the keys the wallet can recreate, so keys you had pasted or diff --git a/clippy.toml b/clippy.toml new file mode 100644 index 000000000..257b2af12 --- /dev/null +++ b/clippy.toml @@ -0,0 +1,9 @@ +# Every production file deletion goes through `src/utils/file_deletion.rs`. +disallowed-methods = [ + { path = "std::fs::remove_file", reason = "delete through utils::file_deletion::delete_file" }, + { path = "std::fs::remove_dir", reason = "delete through utils::file_deletion::delete_tree" }, + { path = "std::fs::remove_dir_all", reason = "delete through utils::file_deletion::delete_tree" }, + { path = "tokio::fs::remove_file", reason = "delete through utils::file_deletion::delete_file" }, + { path = "tokio::fs::remove_dir", reason = "delete through utils::file_deletion::delete_tree" }, + { path = "tokio::fs::remove_dir_all", reason = "delete through utils::file_deletion::delete_tree" }, +] diff --git a/docs/ai-design/2026-09-10-platform-pin/upgrade-notes.md b/docs/ai-design/2026-09-10-platform-pin/upgrade-notes.md index 106152079..e078e231a 100644 --- a/docs/ai-design/2026-09-10-platform-pin/upgrade-notes.md +++ b/docs/ai-design/2026-09-10-platform-pin/upgrade-notes.md @@ -60,10 +60,36 @@ typed identity rows remain in the backup, while their opaque local metadata is preserved. An ambiguous or malformed saved identity roster fails closed. The encrypted seed vault is not migrated or rewritten. -Keep the retained backups. Downgrading does not automatically reverse the -database conversion; recovery requires the corresponding backup and the prior -application version. Validation uses synthetic upstream fixtures, not a user's -real profile. +The bridge takes its backup only after the converted copy validates, just before +the rebuild. Opening a database never deletes a published backup except an exact +duplicate of a newer backup of the same migration (each failed attempt copies the +unchanged original again); it also removes unfinished `.pending` copies left by a +crash. Upstream `backups/auto/pre-migration--*.db` snapshots are covered +too. Housekeeping is best-effort: an error is logged and retried on the next open, +and never blocks opening. Backup candidates must have the exact database-specific +name format and be regular files; symbolic links and directories are skipped, and +the deletion chokepoint refuses hard links and live databases. + +Backups older than a user-configurable retention period (default 90 days, or +never) are deleted by one global policy covering every database in the data +directory: bridge, upstream pre-migration, and legacy `data.db` backups. The +newest usable backup of each database is always kept, whatever its age: an empty or +truncated file never counts, and the upstream snapshot of the latest migration is +kept as well. If no backup is usable, the newest file is kept anyway. A backup +dated in the future does not displace one dated in the past, but only until real +time reaches that date; the migration order is the only clock-proof signal. A copy +that cannot be read, or has a non-empty rollback journal beside it, is never +deleted automatically. This version never writes new legacy `data.db` copies: it +opens an existing `data.db` read-only, so only copies earlier versions left behind +are expired. +Clearing a network's data deletes that network's wallet database backups, because +they copy its wallet and identity history; backups of the shared `det-app.sqlite` +also hold other networks' data and stay under retention. Removing a single wallet +or identity deletes no backups. + +Downgrading does not automatically reverse the database conversion; recovery +requires the corresponding backup and the prior application version. Validation +uses synthetic upstream fixtures, not a user's real profile. ## Validation and limits diff --git a/docs/user-stories.md b/docs/user-stories.md index 279d60a04..5c4c65896 100644 --- a/docs/user-stories.md +++ b/docs/user-stories.md @@ -100,6 +100,7 @@ As a user, I want to remove a wallet I no longer need so that it does not clutte - Confirmation prompt before removal. - Current wallet data is deleted from local storage. If an older recovery database exists, it remains untouched. +- Upgrade backups are not deleted with the wallet, since they also hold other wallets' data; they expire through backup retention (WAL-034). ### WAL-008: View wallet balances [Implemented] **Persona:** Alex, Priya, Jordan @@ -336,6 +337,18 @@ As a user updating from any earlier version I still have installed, I want every - Starting the updated app a second time changes nothing further, and a user whose data was already current keeps an untouched database rather than a needlessly rewritten one. - Currently a gap: each migration mechanism is unit-tested on synthetic data, but no automated check upgrades a data directory a *released* build actually wrote and confirms the result. The design for closing it is `docs/ai-design/2026-09-10-migration-matrix/design.md`; related implemented behaviour is covered by WAL-032, IDN-016 and IDN-020. +### WAL-034: Control how long upgrade backups are kept [Implemented] +**Persona:** Alex, Priya, Jordan + +As a user whose wallet data was copied before a storage upgrade, I want old upgrade backups to be deleted after a while so that copies of my wallet data do not stay on this device forever. + +- Upgrade backups older than 90 days are deleted automatically by default, covering app, wallet, upstream pre-migration and legacy `data.db` backups. +- Settings → Upgrade Backups lets the user change the period (1 to 3650 days) or keep backups forever. Turning automatic deletion on lets the user choose the period before applying it, and any stricter setting asks for confirmation first, because it deletes backups immediately. A period outside the range is shown with an error and never saved. +- The setting is one policy for every backup in the data directory, whichever network is open; a saved change applies to every network's backups as soon as the wallet data has opened. +- The newest complete backup of each database is always kept, whatever its age, so the last recovery copy is never deleted automatically. A backup dated in the future because the system clock ran ahead is not treated as the newest until that date arrives. An empty or cut-off copy never counts as that backup; if no complete copy exists, the newest copy is kept anyway. A copy that cannot be read is never deleted automatically. +- With "keep forever", no published backup is deleted automatically. Opening the data removes only unfinished copies left by a crash and exact duplicates of a newer backup. +- Removing a wallet or an identity does not delete upgrade backups. Clearing a network's data deletes that network's wallet backups; backups of the shared app data stay under the retention setting. + --- ## Send and Receive (SND) diff --git a/src/app.rs b/src/app.rs index 0804833e7..491d16e74 100644 --- a/src/app.rs +++ b/src/app.rs @@ -1328,6 +1328,15 @@ pub enum AppAction { SwitchIdentityHubTab(crate::ui::identity::IdentityHubTab), } +/// Whether results of the task behind `context` belong to the network chooser (settings) +/// screen whichever screen is visible: it tracks the task and must hear how it ended. +fn network_chooser_owns_task(context: &BackendTaskContext) -> bool { + matches!( + context, + BackendTaskContext::ClearNetworkDatabase | BackendTaskContext::UpdateBackupRetention + ) +} + impl BitOrAssign for AppAction { fn bitor_assign(&mut self, rhs: Self) { if matches!(rhs, AppAction::None) { @@ -1446,7 +1455,8 @@ impl AppState { pub(crate) fn boot_inputs() -> Result<(PathBuf, Arc), Box> { let data_dir = crate::boot::prepare_environment()?; - let db_file_path = data_file_path(&data_dir, "data.db")?; + let db_file_path = + data_file_path(&data_dir, crate::database::legacy_backups::LEGACY_DATABASE)?; let db = if db_file_path.exists() { Arc::new(Database::open_legacy_read_only(&db_file_path)?) } else { @@ -2955,6 +2965,16 @@ impl App for AppState { ); } } + result @ BackendTaskSuccessResult::UpdatedBackupRetention { .. } => { + // The settings form waits for this answer even after the user + // navigates away, so it goes to the chooser, not the visible screen. + if let Some(screen) = self + .main_screens + .get_mut(&RootScreenType::RootScreenNetworkChooser) + { + screen.display_backend_task_result(&context, result); + } + } BackendTaskSuccessResult::DashPayIncomingDetected(outputs) => { // The EventBridge surfaced received outputs on a // freshly-seen wallet transaction. Run the owner- @@ -3184,13 +3204,13 @@ impl App for AppState { &err, ); self.route_contact_request_error_to_hidden_hub(&err); - let is_database_clear = context == BackendTaskContext::ClearNetworkDatabase; - let suppress_stale_error = !is_database_clear + let chooser_owned = network_chooser_owns_task(&context); + let suppress_stale_error = !chooser_owned && !recovery_delivered && self .visible_screen_mut() .should_suppress_backend_task_error(&context, &err); - if is_database_clear { + if chooser_owned { if let Some(screen) = self .main_screens .get_mut(&RootScreenType::RootScreenNetworkChooser) @@ -3204,7 +3224,7 @@ impl App for AppState { // Let the screen handle specific error types first. // If handled, skip the generic error banner. let handled = suppress_stale_error - || (!is_database_clear + || (!chooser_owned && !recovery_delivered && self.visible_screen_mut().display_task_error(&err)); @@ -3241,7 +3261,7 @@ impl App for AppState { } _ => {} } - if !is_database_clear && !recovery_delivered { + if !chooser_owned && !recovery_delivered { self.visible_screen_mut() .display_message(&msg, MessageType::Error); } @@ -4513,3 +4533,22 @@ mod shutdown_tests { ); } } + +#[cfg(test)] +mod network_chooser_routing_tests { + use super::*; + use crate::backend_task::system_task::SystemTask; + use crate::model::backup_retention::BackupRetention; + + /// A retention save outlives navigation: its result and error must reach the + /// chooser even when another screen is visible, or its form stays disabled. + #[test] + fn backup_retention_task_results_route_to_network_chooser() { + let task = BackendTask::SystemTask(SystemTask::UpdateBackupRetention( + BackupRetention::KeepForever, + )); + let context = BackendTaskContext::from(&task); + assert_eq!(context, BackendTaskContext::UpdateBackupRetention); + assert!(network_chooser_owns_task(&context)); + } +} diff --git a/src/app_dir.rs b/src/app_dir.rs index 414ef778d..c91cc6ce0 100644 --- a/src/app_dir.rs +++ b/src/app_dir.rs @@ -152,10 +152,13 @@ pub fn copy_env_file_if_not_exists() { ensure_env_file(&app_data_dir); } +/// File name of the environment config in the data directory. +pub const ENV_FILE: &str = ".env"; + /// If .env file does not exist in the given data directory, /// copy the bundled `.env.example` file there. pub fn ensure_env_file(data_dir: &Path) { - let env_file_in_app_dir = data_dir.join(".env"); + let env_file_in_app_dir = data_dir.join(ENV_FILE); BundledResource::DotEnvExample .write_to_file(&env_file_in_app_dir, false) .expect("Failed to write bundled .env.example file"); diff --git a/src/backend_task/error.rs b/src/backend_task/error.rs index fc0c3db91..2a3053720 100644 --- a/src/backend_task/error.rs +++ b/src/backend_task/error.rs @@ -4,6 +4,7 @@ //! `Debug` → variant name + fields (logged and shown in collapsible details). use crate::model::fee_estimation::format_credits_as_dash; +use crate::wallet_backend::platform_compatibility::StorageFailure; use dash_sdk::Error as SdkError; use dash_sdk::dapi_client::DapiClientError; use dash_sdk::dapi_client::transport::TransportError; @@ -108,6 +109,12 @@ impl std::error::Error for BackendTaskJoinError {} /// Dash Core RPC error code: wallet file not specified (multi-wallet node). const RPC_WALLET_NOT_SPECIFIED: i32 = -19; +/// Some expired upgrade backups remain after a retention setting save. +pub(crate) const BACKUP_RETENTION_SAVED_CLEANUP_INCOMPLETE: &str = "Your backup retention setting was saved, but some old upgrade backups could not be deleted. The app tries again the next time it opens your wallet data."; + +/// Shown whenever another window or session holds the wallet database. +pub(crate) const WALLET_DATA_IN_USE: &str = "Your wallet data is open in another Dash Evo Tool window or command-line session. Close it and try again."; + /// App-level error envelope for backend tasks. #[derive(Debug, Error)] pub enum TaskError { @@ -188,15 +195,27 @@ pub enum TaskError { /// An identity-funding account was derived but could not be saved. Saving /// it is what lets a restart find the account again, so the operation is /// stopped rather than left able to strand a funding lock the app could no - /// longer spend. The technical cause lives in `Debug` and the logs. + /// longer spend. Covers non-retryable store failures (constraint, fatal); + /// retryable ones use [`Self::IdentityFundingAccountPersistBusy`]. #[error( - "Your wallet could not save the account this payment needs. Check that your disk is not full, then try again." + "Your wallet could not save the information it needs for this payment. Restart the application and try again." )] IdentityFundingAccountPersistFailed { #[source] source: Box, }, + /// Like [`Self::IdentityFundingAccountPersistFailed`], but the store kept + /// reporting a retryable condition until the retry budget ran out. Upstream + /// classifies a busy database and a full disk alike, so the copy names both. + #[error( + "Your wallet could not save the information it needs for this payment because its data is busy or the disk is full. Close any other copy of Dash Evo Tool, make sure there is free disk space, and try again." + )] + IdentityFundingAccountPersistBusy { + #[source] + source: Box, + }, + /// Single-key wallets are not supported in this version. Their data is /// preserved; HD (recovery-phrase) wallets remain fully functional. #[error( @@ -423,6 +442,13 @@ pub enum TaskError { source: platform_wallet_storage::WalletStorageError, }, + /// Another process currently owns the wallet database write lock. + #[error("{}", WALLET_DATA_IN_USE)] + WalletStorageInUse { + #[source] + source: platform_wallet_storage::WalletStorageError, + }, + /// A pinned-PR wallet database could not be upgraded without losing data. #[error(transparent)] PlatformDatabaseUpgrade { @@ -473,11 +499,11 @@ pub enum TaskError { /// /// The migration diagnostic is preserved through the `#[source]` chain for /// logs and the `Debug` view; it is kept out of the user-facing `Display` - /// copy. On the active development branch the storage layout changed in an - /// incompatible way, so the practical action is to remove the local wallet - /// data and let the app recreate it (see `docs/kv-keys.md`). + /// copy. The copy names only the rebuildable `.sqlite` stores and tells the + /// user to keep the `secrets` folder: the vault holds imported and + /// masternode keys that no recovery phrase can re-derive. #[error( - "Your wallet data is not compatible with this version of the app and cannot be opened. Remove the local wallet data so the app can create it fresh, then restart." + "Your wallet data was saved by an app version that this version cannot open, so it was left unchanged. Your recovery phrases and keys are still stored on this device. First open the app version you used before and write down the recovery phrase of every wallet. Separately, write down each imported private key. Close all Dash Evo Tool windows and command-line sessions. Then move each .sqlite file together with its matching .sqlite-wal and .sqlite-shm files, if present, out of the app data folder, leave the secrets folder where it is, and start this app again." )] WalletDataIncompatible { #[source] @@ -1164,6 +1190,25 @@ pub enum TaskError { )] WalletSeedDecryptFailed, + /// The upgrade-backup retention setting could not be read, so no backup was + /// deleted. + #[error( + "Your backup retention setting could not be read, so no old upgrade backups were deleted. Open Settings, choose a backup retention setting and save it." + )] + BackupRetentionRead { + #[source] + source: crate::wallet_backend::KvAdapterError, + }, + + /// Expired upgrade backups could not all be deleted. + #[error( + "Some old upgrade backups could not be deleted. The app tries again the next time it opens your wallet data." + )] + UpgradeBackupCleanup { + #[source] + source: std::io::Error, + }, + /// A local filesystem operation failed (e.g. creating a data directory). #[error( "Could not access local files. Check available disk space and restart the application." @@ -2976,18 +3021,23 @@ impl TaskError { /// - A divergent migration history (e.g. a database written under an /// earlier, incompatible storage layout that this build's migrations /// cannot reconcile) is surfaced as [`Self::WalletDataIncompatible`] so - /// the banner tells the user to remove the local wallet data — freeing - /// disk space or restarting never resolves a structural mismatch. + /// the banner tells the user to back up keys and set the databases aside + /// while keeping the vault — freeing disk space or restarting never + /// resolves a structural mismatch. + /// - A migration blocked by another SQLite writer is surfaced as + /// [`Self::WalletStorageInUse`] so the user can close the competing + /// process and retry instead of following incompatible-data recovery. /// - A folder on the database path that other accounts can modify is /// surfaced as [`Self::WalletDataFolderInsecure`] so the banner tells the /// user to tighten folder permissions. - /// - Every other storage failure keeps the generic disk/IO copy via - /// [`Self::WalletStorage`]. + /// - A migration that failed on a recoverable resource (disk full, OS I/O + /// failure, out of memory) and every other storage failure keep the + /// generic, retryable disk/IO copy via [`Self::WalletStorage`]. /// /// Discrimination is on the typed upstream variant /// (`WalletStorageError::SchemaVersionUnsupported` / - /// `WalletStorageError::Migration` / `WalletStorageError::InsecureParentDir`), - /// never on its `Display` text. + /// `WalletStorageError::Migration` / `WalletStorageError::InsecureParentDir`) + /// and the typed migration source chain, never on `Display` text. pub fn from_wallet_storage_open_error( source: platform_wallet_storage::WalletStorageError, ) -> Self { @@ -2999,6 +3049,16 @@ impl TaskError { found, max_supported, }, + other @ platform_wallet_storage::WalletStorageError::Migration(_) + if Self::wallet_storage_error_is_in_use(&other) => + { + Self::WalletStorageInUse { source: other } + } + other @ platform_wallet_storage::WalletStorageError::Migration(_) + if Self::wallet_storage_error_is_resource_exhausted(&other) => + { + Self::WalletStorage { source: other } + } other @ platform_wallet_storage::WalletStorageError::Migration(_) => { Self::WalletDataIncompatible { source: other } } @@ -3009,6 +3069,22 @@ impl TaskError { } } + fn wallet_storage_error_is_in_use( + source: &platform_wallet_storage::WalletStorageError, + ) -> bool { + StorageFailure::in_chain(source) == Some(StorageFailure::InUse) + } + + /// A migration that ran out of disk space, hit an OS I/O failure or ran + /// out of memory is recoverable by freeing the resource and retrying, so + /// it must not get the terminal incompatible-data guidance. + fn wallet_storage_error_is_resource_exhausted( + source: &platform_wallet_storage::WalletStorageError, + ) -> bool { + StorageFailure::in_chain(source) + .is_some_and(|failure| failure.is_retryable() && failure != StorageFailure::InUse) + } + /// Returns `true` when this is a [`Self::SecretStore`] open failure caused /// specifically by a vault passphrase mismatch /// ([`SecretStoreError::WrongPassphrase`]). @@ -6024,10 +6100,81 @@ mod tests { .expect_err("divergent checksum must abort") } + fn busy_migration_error() -> refinery::Error { + use refinery::{Migration, Runner}; + + let dir = tempfile::tempdir().expect("temporary directory"); + let path = dir.path().join("wallet.sqlite"); + let writer = rusqlite::Connection::open(&path).expect("writer connection"); + writer + .execute_batch("BEGIN IMMEDIATE; CREATE TABLE held (id INTEGER);") + .expect("hold write lock"); + let mut contender = rusqlite::Connection::open(&path).expect("contender connection"); + contender + .busy_timeout(std::time::Duration::from_millis(1)) + .expect("short deterministic timeout"); + let migration = Migration::unapplied("V1__init", "CREATE TABLE data (id INTEGER);") + .expect("valid migration"); + Runner::new(&[migration]) + .run(&mut contender) + .expect_err("held writer must make the migration busy") + } + + #[test] + fn busy_migration_error_maps_to_retryable_wallet_storage_message() { + let upstream = + platform_wallet_storage::WalletStorageError::Migration(busy_migration_error()); + let err = TaskError::from_wallet_storage_open_error(upstream); + assert!( + matches!(err, TaskError::WalletStorageInUse { .. }), + "Expected WalletStorageInUse, got: {err:?}" + ); + let message = err.to_string(); + assert!(message.contains("Close") && message.contains("try again")); + assert!(!message.contains("incompatible")); + assert!(std::error::Error::source(&err).is_some()); + } + + /// Wraps a SQLite failure with `code` in a refinery migration error the + /// same way refinery's rusqlite driver does, so the source chain matches + /// what `SqlitePersister::open` returns for a migration that ran out of a + /// resource. + fn resource_migration_error(code: std::ffi::c_int) -> refinery::Error { + use refinery::error::WrapMigrationError; + + Err::<(), _>(rusqlite::Error::SqliteFailure( + rusqlite::ffi::Error::new(code), + None, + )) + .migration_err("error applying migration", None) + .expect_err("wrapped SQLite failure") + } + + /// A migration that ran out of a resource (disk, OS I/O, memory) keeps the generic, + /// retryable storage copy instead of the terminal incompatible-data guidance. + #[test] + fn resource_migration_errors_map_to_retryable_wallet_storage() { + use rusqlite::ffi; + for code in [ffi::SQLITE_FULL, ffi::SQLITE_IOERR, ffi::SQLITE_NOMEM] { + let upstream = platform_wallet_storage::WalletStorageError::Migration( + resource_migration_error(code), + ); + let err = TaskError::from_wallet_storage_open_error(upstream); + assert!( + matches!(err, TaskError::WalletStorage { .. }), + "SQLite code {code}: {err:?}" + ); + assert!( + !crate::backend_task::is_terminal_storage_open_error(&err), + "SQLite code {code} must stay retryable" + ); + } + } + /// A divergent migration history (database written under an /// incompatible storage layout) maps to the dedicated - /// `WalletDataIncompatible` variant. Its `Display` tells the user to - /// remove the local wallet data, NOT the misleading "free disk space" copy. + /// `WalletDataIncompatible` variant. Its `Display` must never lead the user + /// to destroy the secrets vault, and must not blame disk space. #[test] fn migration_error_maps_to_wallet_data_incompatible() { let upstream = @@ -6039,9 +6186,18 @@ mod tests { ); let msg = err.to_string(); + assert!(msg.contains(".sqlite-wal") && msg.contains(".sqlite-shm")); + assert!(msg.contains("Close all") && msg.contains("write down each imported private key")); + assert!( + msg.contains("write down the recovery phrase") + && msg.contains(".sqlite") + && msg.contains("leave the secrets folder where it is"), + "Expected scoped, vault-preserving guidance, got: {msg}" + ); + let lower = msg.to_lowercase(); assert!( - msg.contains("not compatible") && msg.contains("Remove"), - "Expected incompatibility guidance, got: {msg}" + !lower.contains("remove") && !lower.contains("delete"), + "Incompatible-data guidance must not tell the user to delete anything, got: {msg}" ); assert!( !msg.contains("disk space"), diff --git a/src/backend_task/identity/remove_identity.rs b/src/backend_task/identity/remove_identity.rs index 80ed6d423..d4ff15d7f 100644 --- a/src/backend_task/identity/remove_identity.rs +++ b/src/backend_task/identity/remove_identity.rs @@ -149,6 +149,30 @@ mod tests { .any(|key_id| view.get(&MAIN, *key_id).unwrap().is_some()) } + /// Upgrade backups are governed by time-based retention alone; removing an + /// identity must leave every recovery snapshot in place. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn removing_an_identity_keeps_upgrade_backups() { + let staged = stage_identity_with_vaulted_keys(HIGH, MEDIUM).await; + let database = crate::wallet_backend::wallet_database_path( + staged.ctx.data_dir(), + staged.ctx.network(), + ); + let backup = + crate::wallet_backend::platform_compatibility::bridge_backup_path(&database, "fixture"); + std::fs::write(&backup, b"old identity rows").unwrap(); + + staged + .ctx + .remove_identity(staged.id) + .expect("the identity must be removed"); + + assert!( + backup.exists(), + "identity removal must not delete upgrade backups" + ); + } + /// A removal that fails before the identity is delisted did not happen: /// the identity is still on every screen, its keys are still on the /// device, and the button that started this still works. Reporting it as a diff --git a/src/backend_task/migration/finish_unwire.rs b/src/backend_task/migration/finish_unwire.rs index 0fab71dfd..31612efef 100644 --- a/src/backend_task/migration/finish_unwire.rs +++ b/src/backend_task/migration/finish_unwire.rs @@ -3125,6 +3125,10 @@ mod tests { } #[tokio::test] + #[expect( + clippy::disallowed_methods, + reason = "test fixture setup/teardown outside any production deletion path" + )] async fn dapi_config_persistence_reports_save_failure() { let _env_guard = CONFIG_ENV_LOCK.lock().await; let tmp = tempfile::tempdir().expect("tempdir"); diff --git a/src/backend_task/mod.rs b/src/backend_task/mod.rs index 59658184a..d2670596a 100644 --- a/src/backend_task/mod.rs +++ b/src/backend_task/mod.rs @@ -230,17 +230,19 @@ fn identity_load_ticket(task: &BackendTask) -> Option<(Identifier, IdentityLoadT } /// Whether a wallet-backend build error is terminal (storage written by a -/// newer/incompatible app build, or a data folder other accounts can modify). -/// These must surface their actionable message instead of being -/// logged-and-discarded as a transient deferral (F50); retrying cannot fix -/// them. Every other init error is retried by the cold-boot bridge. +/// newer/incompatible app build, a data folder other accounts can modify, or +/// a compatibility upgrade that fails the same way on every attempt). These +/// must surface their actionable message instead of being logged-and-discarded +/// as a transient deferral (F50); retrying cannot fix them. Every other init +/// error is retried by the cold-boot bridge. pub(crate) fn is_terminal_storage_open_error(error: &TaskError) -> bool { - matches!( - error, + match error { TaskError::WalletDataTooNew { .. } - | TaskError::WalletDataIncompatible { .. } - | TaskError::WalletDataFolderInsecure { .. } - ) + | TaskError::WalletDataIncompatible { .. } + | TaskError::WalletDataFolderInsecure { .. } => true, + TaskError::PlatformDatabaseUpgrade { source } => !source.is_retryable(), + _ => false, + } } /// Information about fees for a platform state transition. @@ -330,6 +332,8 @@ pub enum BackendTaskContext { TokenRewardEstimate(IdentityTokenIdentifier), /// The destructive per-network database clear. ClearNetworkDatabase, + /// Saving the upgrade-backup retention policy. + UpdateBackupRetention, /// A scheduled-vote sweep for one network. ScheduledVoteSweep { network: Network }, /// Receive-address derivation for one wallet's deposit flow. @@ -574,6 +578,9 @@ impl From<&BackendTask> for BackendTaskContext { identity_index: *identity_index, }, BackendTask::SystemTask(SystemTask::ClearNetworkDatabase) => Self::ClearNetworkDatabase, + BackendTask::SystemTask(SystemTask::UpdateBackupRetention(_)) => { + Self::UpdateBackupRetention + } BackendTask::WalletTask(WalletTask::GenerateReceiveAddress { seed_hash }) => { Self::GenerateReceiveAddress { seed_hash: *seed_hash, @@ -673,6 +680,16 @@ pub enum BackendTaskSuccessResult { prices: Option, }, UpdatedThemePreference(crate::ui::theme::ThemeMode), + /// The upgrade-backup retention policy was saved and applied. + UpdatedBackupRetention { + /// The policy now in effect. + retention: crate::model::backup_retention::BackupRetention, + /// Expired backups deleted under the new policy. + deleted: usize, + /// Why some expired backups could not be deleted; they are retried the next + /// time wallet data opens. + cleanup_failure: Option>, + }, PlatformInfo(PlatformInfoTaskResult), // DashPay related results @@ -2417,6 +2434,19 @@ mod tests { }, } )); + use crate::wallet_backend::platform_compatibility::UpgradeError; + assert!( + is_terminal_storage_open_error(&TaskError::PlatformDatabaseUpgrade { + source: UpgradeError::Verification, + }), + "a deterministic upgrade failure must surface instead of re-running" + ); + // A user-clearable upgrade failure keeps offering a retry. + assert!(!is_terminal_storage_open_error( + &TaskError::PlatformDatabaseUpgrade { + source: UpgradeError::from(std::io::Error::from(std::io::ErrorKind::StorageFull)), + } + )); // A transient pre-wire state must NOT be treated as terminal. assert!(!is_terminal_storage_open_error( &TaskError::WalletBackendNotYetWired diff --git a/src/backend_task/system_task/mod.rs b/src/backend_task/system_task/mod.rs index f7e729ac1..208c6031f 100644 --- a/src/backend_task/system_task/mod.rs +++ b/src/backend_task/system_task/mod.rs @@ -1,7 +1,8 @@ use crate::app::TaskResult; use crate::backend_task::BackendTaskSuccessResult; use crate::backend_task::error::TaskError; -use crate::context::AppContext; +use crate::context::{AppContext, BackupPruneReport}; +use crate::model::backup_retention::BackupRetention; use crate::ui::theme::ThemeMode; use std::sync::Arc; @@ -10,6 +11,8 @@ pub enum SystemTask { ClearNetworkDatabase, WipePlatformData, UpdateThemePreference(ThemeMode), + /// Persist the upgrade-backup retention policy and apply it right away. + UpdateBackupRetention(BackupRetention), } impl AppContext { @@ -29,6 +32,9 @@ impl AppContext { SystemTask::UpdateThemePreference(theme_mode) => { self.handle_update_theme_preference(theme_mode) } + SystemTask::UpdateBackupRetention(retention) => { + self.handle_update_backup_retention(retention) + } } } @@ -57,4 +63,29 @@ impl AppContext { Ok(BackendTaskSuccessResult::UpdatedThemePreference(theme_mode)) } + + /// Backend-task handler for `SystemTask::UpdateBackupRetention`. + /// + /// Only the save can fail the task. Pruning under the new policy is + /// best-effort and is retried the next time wallet data opens. + pub fn handle_update_backup_retention( + self: &Arc, + retention: BackupRetention, + ) -> Result { + let retention = retention.sanitized(); + self.set_backup_retention(retention) + .map_err(|source| TaskError::AppSettingsWrite { source })?; + // Before wallet storage is up, its start-up applies the saved policy; the + // newest usable backup of every database is kept either way. + let report = if self.wallet_backend().is_ok() { + self.prune_expired_upgrade_backups_best_effort() + } else { + BackupPruneReport::default() + }; + Ok(BackendTaskSuccessResult::UpdatedBackupRetention { + retention, + deleted: report.deleted, + cleanup_failure: report.failure.map(Arc::new), + }) + } } diff --git a/src/config.rs b/src/config.rs index 1042f4d6f..9e4c01221 100644 --- a/src/config.rs +++ b/src/config.rs @@ -142,8 +142,8 @@ impl Config { /// Preserves lines not modeled by `Config` and replaces the modeled fields. /// Uses a temporary file and atomic rename to prevent partial writes. pub fn save(&self, data_dir: &Path) -> Result<(), ConfigError> { - let env_file_path = - data_file_path(data_dir, ".env").map_err(|e| ConfigError::SaveError { source: e })?; + let env_file_path = data_file_path(data_dir, crate::app_dir::ENV_FILE) + .map_err(|e| ConfigError::SaveError { source: e })?; let existing_contents = read_env_contents(&env_file_path)?; validate_env_contents(&existing_contents)?; let modeled_entries = self.modeled_entries(); @@ -185,8 +185,8 @@ impl Config { network: Network, addresses: &str, ) -> Result<(), ConfigError> { - let env_file_path = - data_file_path(data_dir, ".env").map_err(|source| ConfigError::SaveError { source })?; + let env_file_path = data_file_path(data_dir, crate::app_dir::ENV_FILE) + .map_err(|source| ConfigError::SaveError { source })?; let existing_contents = read_env_contents(&env_file_path)?; let key = format!("{}dapi_addresses", dotenv_network_prefix(network)); atomic_replace_env_file(&env_file_path, |env_file| { @@ -203,15 +203,16 @@ impl Config { /// Loads the configuration for all networks from environment variables and `.env` file /// located in the default app data directory. pub fn load() -> Result { - let env_file_path = app_user_data_file_path(".env").expect("should create .env file path"); + let env_file_path = app_user_data_file_path(crate::app_dir::ENV_FILE) + .expect("should create .env file path"); Self::load_from_env_path(env_file_path) } /// Loads the configuration for all networks from environment variables and `.env` file /// located in the given data directory. pub fn load_from(data_dir: &Path) -> Result { - let env_file_path = - data_file_path(data_dir, ".env").map_err(|source| ConfigError::LoadError { source })?; + let env_file_path = data_file_path(data_dir, crate::app_dir::ENV_FILE) + .map_err(|source| ConfigError::LoadError { source })?; Self::load_from_env_path(env_file_path) } @@ -311,6 +312,9 @@ fn atomic_replace_env_file( "config file path has no parent directory", ), })?; + // Not routed through `delete_file`: on failure the temp file removes only + // the uniquely named file it created itself, and success is an atomic + // rename over `.env`, not a deletion. let mut env_file = NamedTempFile::new_in(parent_dir).map_err(|source| ConfigError::SaveError { source })?; write_contents(&mut env_file)?; diff --git a/src/context/mod.rs b/src/context/mod.rs index 30d4a14ed..6e339e258 100644 --- a/src/context/mod.rs +++ b/src/context/mod.rs @@ -13,6 +13,7 @@ mod settings_db; #[cfg(test)] pub(crate) mod test_support; mod wallet_lifecycle; +pub use wallet_lifecycle::BackupPruneReport; pub use wallet_lifecycle::PrepareGateGuard; pub use wallet_lifecycle::WalletUnlockRetention; @@ -617,7 +618,7 @@ impl AppContext { use platform_wallet_storage::SqlitePersisterConfig; crate::app_dir::ensure_data_dir_exists(data_dir) .map_err(|source| TaskError::FileSystem { source })?; - let path = data_dir.join("det-app.sqlite"); + let path = crate::wallet_backend::app_database_path(data_dir); let config = SqlitePersisterConfig::new(path); let persister = Arc::new(crate::wallet_backend::platform_compatibility::open(config)?); Ok(Arc::new(DetKv::new(persister))) @@ -667,7 +668,7 @@ impl AppContext { } /// Absolute path of the shared seed vault: `/secrets/det-secrets.pwsvault`. - fn secret_store_path(data_dir: &std::path::Path) -> std::path::PathBuf { + pub(crate) fn secret_store_path(data_dir: &std::path::Path) -> std::path::PathBuf { let mut path = data_dir.to_path_buf(); path.push("secrets"); path.push("det-secrets.pwsvault"); @@ -1159,6 +1160,9 @@ impl AppContext { .await?; self.wallet_backend.store(Some(Arc::new(backend))); drop(_build_guard); + // Wallet storage is up: apply the retention policy to every database's + // backups (the newest usable backup of each is always kept). + self.prune_expired_upgrade_backups_best_effort(); if let Err(error) = self.refresh_pending_dpns_usernames() { tracing::warn!( ?error, diff --git a/src/context/settings_db.rs b/src/context/settings_db.rs index 96635350c..933c10409 100644 --- a/src/context/settings_db.rs +++ b/src/context/settings_db.rs @@ -7,6 +7,7 @@ //! stale value. use super::{AppContext, SettingsCacheGuard}; +use crate::model::backup_retention::BackupRetention; use crate::model::settings::{AppSettings, detect_dash_qt_path}; use crate::model::user_role::UserRole; use crate::ui::RootScreenType; @@ -225,6 +226,25 @@ impl AppContext { Ok(with_default_user_role(settings)) } + /// The persisted upgrade-backup retention policy. + /// + /// An absent or out-of-range value resolves to the default. An unreadable + /// value is an error, never the default: pruning with a guessed policy could + /// delete backups a user chose to keep forever. + pub fn backup_retention(&self) -> Result { + Ok(self + .app_kv + .get::(DetScope::Global, BackupRetention::KV_KEY)? + .unwrap_or_default() + .sanitized()) + } + + /// Persist the upgrade-backup retention policy. + pub fn set_backup_retention(&self, retention: BackupRetention) -> Result<(), KvAdapterError> { + self.app_kv + .put(DetScope::Global, BackupRetention::KV_KEY, &retention) + } + /// Write the [`AppSettings`] blob to the shared app k/v store. /// /// The blob is stored verbatim — a `user_role` of `None` stays `None` on disk @@ -273,6 +293,34 @@ mod tests { DetKv::from_store(Arc::new(InMemoryKv::default())) } + #[test] + fn backup_retention_round_trips_and_defaults_when_absent_or_invalid() { + let tmp = tempfile::tempdir().unwrap(); + let ctx = test_app_context(tmp.path()); + assert_eq!(ctx.backup_retention().unwrap(), BackupRetention::default()); + for retention in [ + BackupRetention::KeepForever, + BackupRetention::DeleteAfterDays(7), + ] { + ctx.set_backup_retention(retention).unwrap(); + assert_eq!(ctx.backup_retention().unwrap(), retention); + } + ctx.set_backup_retention(BackupRetention::DeleteAfterDays(0)) + .unwrap(); + assert_eq!(ctx.backup_retention().unwrap(), BackupRetention::default()); + } + + #[test] + fn unreadable_backup_retention_is_an_error_not_the_default() { + let tmp = tempfile::tempdir().unwrap(); + let store = Arc::new(FailingKv::default()); + let ctx = test_app_context_with_kv(tmp.path(), Arc::new(DetKv::from_store(store.clone()))); + ctx.set_backup_retention(BackupRetention::KeepForever) + .unwrap(); + store.fail_reads(true); + assert!(ctx.backup_retention().is_err()); + } + /// F69 — the `AppSettings` blob survives a put/get round-trip through the /// k/v store with every field intact, and an absent blob reads as `None` /// (the path `get_app_settings` maps to its `Default`). diff --git a/src/context/wallet_lifecycle/mod.rs b/src/context/wallet_lifecycle/mod.rs index 6380db287..420613934 100644 --- a/src/context/wallet_lifecycle/mod.rs +++ b/src/context/wallet_lifecycle/mod.rs @@ -21,12 +21,13 @@ use crate::model::wallet::meta::WalletMeta; use crate::model::wallet::seed_envelope::StoredSeedEnvelope; use crate::model::wallet::single_key::SingleKeyWallet; use crate::model::wallet::{Wallet, WalletSeedHash}; +use crate::utils::file_deletion::{DeletionIntent, delete_file, delete_tree}; use crate::wallet_backend::poison::RwLockRecover; use crate::wallet_backend::{ - ClearAllOutcome, DetScope, WalletBackend, WalletMetaView, WalletSeedView, network_prefix, + ClearAllOutcome, DetScope, WalletBackend, WalletMetaView, WalletSeedView, spv_storage_dir, }; use dash_sdk::dpp::dashcore::Network; -use std::path::{Path, PathBuf}; +use std::path::Path; use std::sync::atomic::Ordering; use std::sync::{Arc, RwLock}; @@ -66,42 +67,37 @@ pub enum WalletUnlockRetention { UntilAppClose, } -/// Per-network SPV storage directory: `/spv//`. Mirrors -/// `WalletBackend::resolve_spv_storage_dir` so the path resolves identically -/// whether or not the wallet backend is wired yet. -fn spv_storage_dir(data_dir: &Path, network: Network) -> PathBuf { - data_dir.join("spv").join(network_prefix(network)) -} - /// Remove the upstream chain-sync cache files under `spv_dir`, leaving the /// legacy shielded sidecars in that directory untouched (the durable wallet /// databases are not in it at all — see [`SPV_CHAIN_STORAGE_ENTRIES`]). The /// `DiskStorageManager` lock lives at `.lock` (a sibling of the /// directory); it is removed too so a stale lock cannot block the next sync. /// A missing entry is the expected fresh/never-synced state and is tolerated. -fn clear_spv_chain_storage(spv_dir: &Path) -> Result<(), TaskError> { +/// Every unlink goes through the [`delete_file`] chokepoint, confined to this +/// network's SPV data by [`DeletionIntent::NetworkClear`]. +fn clear_spv_chain_storage(data_dir: &Path, network: Network) -> Result<(), TaskError> { + let intent = DeletionIntent::NetworkClear { data_dir, network }; + let spv_dir = spv_storage_dir(data_dir, network); for entry in SPV_CHAIN_STORAGE_ENTRIES { let path = spv_dir.join(entry); - let result = if path.is_dir() { - std::fs::remove_dir_all(&path) - } else { - std::fs::remove_file(&path) + let result = match std::fs::symlink_metadata(&path) { + Ok(metadata) if metadata.is_dir() => delete_tree(&path, intent), + Ok(_) => delete_file(&path, intent), + Err(error) => Err(error), }; - if let Err(e) = result - && e.kind() != std::io::ErrorKind::NotFound - { - return Err(TaskError::FileSystem { source: e }); - } + ignore_not_found(result)?; } + ignore_not_found(delete_file(&spv_dir.with_extension("lock"), intent)) +} - let lock_path = spv_dir.with_extension("lock"); - if let Err(e) = std::fs::remove_file(&lock_path) - && e.kind() != std::io::ErrorKind::NotFound - { - return Err(TaskError::FileSystem { source: e }); +/// Treat an already-missing file as deleted; map any other failure to [`TaskError::FileSystem`]. +fn ignore_not_found(result: std::io::Result<()>) -> Result<(), TaskError> { + match result { + Err(error) if error.kind() != std::io::ErrorKind::NotFound => { + Err(TaskError::FileSystem { source: error }) + } + _ => Ok(()), } - - Ok(()) } mod bootstrap; @@ -109,6 +105,8 @@ mod prepare; pub use prepare::PrepareGateGuard; mod registration; mod removal; +mod retention; +pub use retention::BackupPruneReport; mod spv; mod unlock; @@ -128,8 +126,8 @@ impl AppContext { } } -/// Unlink DET's two retired legacy shielded files from `spv_dir` (the active -/// network's spv directory), tolerating a missing file. +/// Unlink DET's two retired legacy shielded files from `network`'s spv +/// directory, tolerating a missing file. /// /// These are the files DET's deleted shielded subsystem owned: /// `det-shielded.sqlite` (the plaintext note sidecar) and @@ -137,17 +135,16 @@ impl AppContext { /// upstream coordinator's store (`det--shielded.sqlite`, outside this /// directory) is a DIFFERENT file and is deliberately NOT touched here — it is /// reset via the coordinator's own `clear_shielded`. Scoped strictly to -/// `spv_dir` so a clear of one network can never reach another network's files. -fn cleanup_legacy_shielded_files(spv_dir: &Path) -> Result<(), TaskError> { +/// that directory — enforced by the [`delete_file`] chokepoint's +/// [`DeletionIntent::NetworkClear`] scope — so a clear of one network can never +/// reach another network's files. +fn cleanup_legacy_shielded_files(data_dir: &Path, network: Network) -> Result<(), TaskError> { const LEGACY_SHIELDED_FILES: [&str; 2] = ["det-shielded.sqlite", "shielded-commitment-tree.sqlite"]; + let intent = DeletionIntent::NetworkClear { data_dir, network }; + let spv_dir = spv_storage_dir(data_dir, network); for file in LEGACY_SHIELDED_FILES { - let path = spv_dir.join(file); - if let Err(e) = std::fs::remove_file(&path) - && e.kind() != std::io::ErrorKind::NotFound - { - return Err(TaskError::FileSystem { source: e }); - } + ignore_not_found(delete_file(&spv_dir.join(file), intent))?; } Ok(()) } diff --git a/src/context/wallet_lifecycle/removal.rs b/src/context/wallet_lifecycle/removal.rs index 60762f5aa..312fd9536 100644 --- a/src/context/wallet_lifecycle/removal.rs +++ b/src/context/wallet_lifecycle/removal.rs @@ -77,6 +77,20 @@ impl AppContext { Ok(()) } + + /// Delete the retained compatibility-upgrade backups of this network's wallet + /// database, which copy the wallet and identity history a network clear wipes. + /// + /// The shared app database's backups also hold other networks' data, so they are + /// left to time-based retention. Filesystem cleanup is available even when the + /// wallet backend could not be opened. Removing a single wallet or identity never + /// calls this: those backups also hold other wallets' data. + pub(crate) fn remove_upgrade_backups(&self) -> Result<(), TaskError> { + crate::wallet_backend::platform_compatibility::remove_backups( + &crate::wallet_backend::wallet_database_path(self.data_dir(), self.network), + ) + .map_err(|source| TaskError::FileSystem { source }) + } } #[cfg(test)] diff --git a/src/context/wallet_lifecycle/retention.rs b/src/context/wallet_lifecycle/retention.rs new file mode 100644 index 000000000..87f9e1c5e --- /dev/null +++ b/src/context/wallet_lifecycle/retention.rs @@ -0,0 +1,234 @@ +//! Time-based retention of storage-upgrade backups. + +use super::*; +use std::time::SystemTime; + +/// What one retention pass did. +#[derive(Debug, Default)] +pub struct BackupPruneReport { + /// Expired backups deleted. Undercounts when a location fails part-way. + pub deleted: usize, + /// The first failure; the remaining expired backups are retried the next time + /// wallet data opens. + pub failure: Option, +} + +/// Every database in `data_dir` whose upgrade backups the global retention covers: +/// the shared app database and each network's wallet database. +pub(crate) fn retained_backup_databases(data_dir: &std::path::Path) -> Vec { + std::iter::once(crate::wallet_backend::app_database_path(data_dir)) + .chain( + crate::wallet_backend::all_networks() + .into_iter() + .map(|network| crate::wallet_backend::wallet_database_path(data_dir, network)), + ) + .collect() +} + +impl AppContext { + /// Delete expired upgrade and legacy backups of every database in the data directory, + /// keeping each database's newest usable one; attempts all and reports the first failure. + pub(crate) fn prune_expired_upgrade_backups(&self) -> BackupPruneReport { + let now = SystemTime::now(); + let mut report = BackupPruneReport::default(); + let retention = match self.backup_retention() { + Ok(retention) => retention, + Err(source) => { + report.failure = Some(TaskError::BackupRetentionRead { source }); + return report; + } + }; + let Some(max_age) = retention.max_age() else { + return report; + }; + let outcomes = retained_backup_databases(self.data_dir()) + .into_iter() + .map(|database| { + crate::wallet_backend::platform_compatibility::prune_expired_backups( + &database, max_age, now, + ) + }) + .chain(std::iter::once( + crate::database::legacy_backups::prune_expired(self.data_dir(), max_age, now), + )); + for outcome in outcomes { + match outcome { + Ok(count) => report.deleted += count, + Err(source) => { + report + .failure + .get_or_insert(TaskError::UpgradeBackupCleanup { source }); + } + } + } + if report.deleted > 0 { + tracing::info!( + deleted = report.deleted, + ?retention, + "Deleted expired upgrade backups" + ); + } + report + } + + /// [`Self::prune_expired_upgrade_backups`] with any failure logged; it never fails + /// the surrounding operation. + pub(crate) fn prune_expired_upgrade_backups_best_effort(&self) -> BackupPruneReport { + let report = self.prune_expired_upgrade_backups(); + match &report.failure { + Some(error @ TaskError::BackupRetentionRead { .. }) => tracing::warn!( + ?error, + "Backup retention setting unreadable; no upgrade backups are deleted until the setting is saved again" + ), + Some(error) => tracing::warn!( + ?error, + "Expired upgrade backups remain; retrying when wallet data next opens" + ), + None => {} + } + report + } +} + +#[cfg(test)] +mod tests { + use crate::context::test_support::test_app_context; + use crate::model::backup_retention::BackupRetention; + use std::path::Path; + use std::time::{Duration, SystemTime}; + + const DAY: Duration = Duration::from_secs(24 * 60 * 60); + + fn pruned(ctx: &crate::context::AppContext) -> usize { + let report = ctx.prune_expired_upgrade_backups(); + assert!(report.failure.is_none(), "{:?}", report.failure); + report.deleted + } + + fn write_aged(path: &Path, age: Duration) { + std::fs::create_dir_all(path.parent().unwrap()).unwrap(); + std::fs::write(path, b"backup").unwrap(); + std::fs::File::options() + .write(true) + .open(path) + .unwrap() + .set_modified(SystemTime::now() - age) + .unwrap(); + } + + struct Fixture { + expired: Vec, + fresh: Vec, + } + + fn backups(ctx: &crate::context::AppContext) -> Fixture { + let dir = ctx.data_dir(); + let [app, wallet] = [ + crate::wallet_backend::app_database_path(dir), + crate::wallet_backend::wallet_database_path(dir, ctx.network), + ]; + let bridge = crate::wallet_backend::platform_compatibility::bridge_backup_path; + let expired = vec![ + bridge(&app, "old"), + bridge(&wallet, "old"), + dir.join("backups/data_backup_20000101_000000.db"), + ]; + let fresh = vec![ + bridge(&app, "new"), + bridge(&wallet, "new"), + dir.join("backups") + .join(crate::database::legacy_backups::backup_file_name( + chrono::Utc::now(), + )), + ]; + for path in &expired { + write_aged(path, DAY * 120); + } + for path in &fresh { + write_aged(path, DAY); + } + Fixture { expired, fresh } + } + + #[test] + fn startup_retention_deletes_only_expired_backups() { + let tmp = tempfile::tempdir().unwrap(); + let ctx = test_app_context(tmp.path()); + let fixture = backups(&ctx); + + assert_eq!(pruned(&ctx), 3); + + assert!(fixture.expired.iter().all(|path| !path.exists())); + assert!(fixture.fresh.iter().all(|path| path.exists())); + } + + #[test] + fn shorter_retention_deletes_more() { + let tmp = tempfile::tempdir().unwrap(); + let ctx = test_app_context(tmp.path()); + ctx.set_backup_retention(BackupRetention::DeleteAfterDays(200)) + .unwrap(); + let fixture = backups(&ctx); + assert_eq!(pruned(&ctx), 0); + + ctx.set_backup_retention(BackupRetention::DeleteAfterDays(30)) + .unwrap(); + assert_eq!(pruned(&ctx), 3); + assert!(fixture.fresh.iter().all(|path| path.exists())); + } + + #[test] + fn retention_is_global_across_networks() { + let tmp = tempfile::tempdir().unwrap(); + let ctx = test_app_context(tmp.path()); + let other = dash_sdk::dpp::dashcore::Network::Mainnet; + assert_ne!(ctx.network, other); + let database = crate::wallet_backend::wallet_database_path(ctx.data_dir(), other); + let auto = platform_wallet_storage::default_auto_backup_dir(&database); + let expired = auto.join("pre-migration-det-mainnet-1-to-2-20000101T000000Z.db"); + let newest = auto.join("pre-migration-det-mainnet-2-to-3-20000102T000000Z.db"); + write_aged(&expired, DAY * 400); + write_aged(&newest, DAY * 300); + + assert_eq!(pruned(&ctx), 1); + assert!( + !expired.exists(), + "another network's expired backup is pruned" + ); + assert!( + newest.exists(), + "the newest backup of each database is kept" + ); + assert!( + !crate::wallet_backend::platform_compatibility::backup_lock_path( + &crate::wallet_backend::wallet_database_path( + ctx.data_dir(), + dash_sdk::dpp::dashcore::Network::Devnet + ) + ) + .unwrap() + .exists(), + "a database without backups is not locked or touched" + ); + } + + #[test] + fn database_in_use_is_skipped_not_failed() { + let tmp = tempfile::tempdir().unwrap(); + let ctx = test_app_context(tmp.path()); + let fixture = backups(&ctx); + let wallet = crate::wallet_backend::wallet_database_path(ctx.data_dir(), ctx.network); + let lock = + crate::wallet_backend::platform_compatibility::backup_lock_path(&wallet).unwrap(); + let held = std::fs::File::create(&lock).unwrap(); + held.try_lock().unwrap(); + + let report = ctx.prune_expired_upgrade_backups(); + assert!(report.failure.is_none(), "{:?}", report.failure); + assert_eq!( + report.deleted, 2, + "app and legacy backups; the locked wallet is skipped" + ); + assert!(fixture.expired[1].exists()); + } +} diff --git a/src/context/wallet_lifecycle/spv.rs b/src/context/wallet_lifecycle/spv.rs index b02345bd2..eb7ddc7ac 100644 --- a/src/context/wallet_lifecycle/spv.rs +++ b/src/context/wallet_lifecycle/spv.rs @@ -16,8 +16,7 @@ impl AppContext { /// `DiskStorageManager` has released its file lock and the deletes do not /// race a live writer. A missing directory (never synced) is success. pub fn clear_spv_data(&self) -> Result<(), TaskError> { - let spv_dir = spv_storage_dir(&self.data_dir, self.network); - clear_spv_chain_storage(&spv_dir) + clear_spv_chain_storage(&self.data_dir, self.network) } pub async fn clear_network_database(self: &Arc) -> Result<(), TaskError> { @@ -107,13 +106,23 @@ impl AppContext { // empties the per-network store) and unlink DET's two retired legacy // shielded files. The legacy-file unlinks are synchronous and scoped // strictly to THIS network's spv directory. - cleanup_legacy_shielded_files(backend.spv_storage_dir())?; + // A failure here must not skip the steps below: upgrade backups hold the + // wallet data being cleared, so record it and keep going. + if let Err(error) = cleanup_legacy_shielded_files(&self.data_dir, self.network) { + tracing::warn!(?error, "Legacy shielded file removal failed during clear"); + failures.push(error); + } if let Err(error) = backend.clear_shielded().await { tracing::warn!(%error, "Shielded coordinator reset failed during clear"); failures.push(error); } + if let Err(error) = self.remove_upgrade_backups() { + tracing::warn!(?error, "Upgrade backup removal failed during clear"); + failures.push(error); + } + self.wallet_context().clear(); // The per-wallet deletes above ran while the wallets were still loaded, // so a warm in flight could write an entry back; none can from here on. diff --git a/src/context/wallet_lifecycle/tests.rs b/src/context/wallet_lifecycle/tests.rs index 813187726..5c63b5e11 100644 --- a/src/context/wallet_lifecycle/tests.rs +++ b/src/context/wallet_lifecycle/tests.rs @@ -2500,6 +2500,51 @@ async fn clear_network_database_reports_incomplete_when_shielded_clear_fails() { } } +/// A legacy shielded-file removal failure must not skip upgrade-backup removal: +/// the backups copy the wallet data being cleared, and the failure still makes +/// the clear incomplete. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn clear_network_database_removes_backups_when_legacy_shielded_cleanup_fails() { + let (ctx, sender, _tmp) = offline_testnet_context(); + ctx.ensure_wallet_backend(sender) + .await + .expect("ensure_wallet_backend should succeed offline"); + let backend = ctx.wallet_backend().expect("backend wired"); + // A directory at a legacy file path makes its unlink fail. + let blocker = backend.spv_storage_dir().join("det-shielded.sqlite"); + std::fs::create_dir_all(blocker.join("occupied")).unwrap(); + let backup = ctx + .data_dir() + .join("det-testnet.sqlite.platform-67d4ef3-backup-fixture.sqlite"); + std::fs::write(&backup, b"old wallet history").unwrap(); + let shared = ctx + .data_dir() + .join("det-app.sqlite.platform-67d4ef3-backup-fixture.sqlite"); + std::fs::write(&shared, b"all networks' app data").unwrap(); + + let result = ctx.clear_network_database().await; + + backend.shutdown().await; + assert!(!backup.exists(), "upgrade backups must still be removed"); + assert!( + shared.exists(), + "the shared app database's backups are left to retention" + ); + match result { + Err(TaskError::WalletDataClearIncomplete { + failed, + first_error, + }) => { + assert_eq!(failed, 1, "the legacy cleanup should be the only failure"); + assert!( + matches!(*first_error, TaskError::FileSystem { .. }), + "the aggregate must preserve the legacy cleanup error: {first_error:?}" + ); + } + other => panic!("legacy cleanup failure must make clear incomplete: {other:?}"), + } +} + /// Clear-all must fail before changing any state when the wallet backend is /// unavailable, because persisted secrets from an earlier run may still exist. #[tokio::test] @@ -2593,6 +2638,10 @@ async fn lock_wipes_session_cached_seed() { /// with a directory: the store's atomic `persist` rename onto a directory /// path fails deterministically (root cannot bypass this). #[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[expect( + clippy::disallowed_methods, + reason = "test fixture setup/teardown outside any production deletion path" +)] async fn register_wallet_fails_closed_when_seed_envelope_write_fails() { let temp_dir = tempfile::tempdir().expect("tempdir"); let (ctx, _sender) = offline_testnet_context_at(temp_dir.path()); @@ -3361,7 +3410,7 @@ fn clear_spv_chain_storage_removes_chain_cache_but_keeps_wallet_sidecars() { std::fs::write(&wallet_sqlite, b"wallet").expect("write wallet sqlite"); std::fs::write(&shielded_tree, b"tree").expect("write shielded tree"); - clear_spv_chain_storage(&spv_dir).expect("clear must succeed"); + clear_spv_chain_storage(tmp.path(), Network::Testnet).expect("clear must succeed"); for entry in SPV_CHAIN_STORAGE_ENTRIES { assert!( @@ -3393,7 +3442,34 @@ fn clear_spv_chain_storage_is_ok_when_directory_absent() { !spv_dir.exists(), "precondition: no spv dir on a fresh install" ); - clear_spv_chain_storage(&spv_dir).expect("clearing an absent cache must succeed"); + clear_spv_chain_storage(tmp.path(), Network::Testnet) + .expect("clearing an absent cache must succeed"); +} + +/// Chain storage moved to another disk and symlinked back is cleared by +/// unlinking the links; the data they point at is never followed or touched. +#[cfg(unix)] +#[test] +fn clear_spv_chain_storage_unlinks_symlinked_entries() { + let tmp = tempfile::tempdir().expect("tempdir"); + let other_disk = tempfile::tempdir().expect("tempdir"); + std::fs::write(other_disk.path().join("segment.dat"), b"elsewhere").unwrap(); + let spv_dir = spv_storage_dir(tmp.path(), Network::Testnet); + std::fs::create_dir_all(&spv_dir).unwrap(); + std::os::unix::fs::symlink(other_disk.path(), spv_dir.join("block_headers")).unwrap(); + std::os::unix::fs::symlink( + other_disk.path().join("segment.dat"), + spv_dir.join("peers.dat"), + ) + .unwrap(); + + clear_spv_chain_storage(tmp.path(), Network::Testnet) + .expect("symlinked chain storage must not block the clear"); + + for entry in ["block_headers", "peers.dat"] { + assert!(spv_dir.join(entry).symlink_metadata().is_err(), "{entry}"); + } + assert!(other_disk.path().join("segment.dat").exists()); } /// Seed a legacy password-protected `single_key_wallet` row into the @@ -6423,8 +6499,8 @@ async fn an_exhausted_transient_persist_budget_keeps_memory_and_the_next_attempt .await .expect_err("an exhausted retry budget must surface as a typed error"); assert!( - matches!(error, TaskError::IdentityFundingAccountPersistFailed { .. }), - "the persist failure must keep its dedicated variant, got {error:?}", + matches!(error, TaskError::IdentityFundingAccountPersistBusy { .. }), + "a still-retryable failure must use the busy variant, got {error:?}", ); let view = backend @@ -6696,6 +6772,11 @@ async fn a_constraint_registration_persist_failure_rolls_back_the_in_memory_acco matches!(error, TaskError::IdentityFundingAccountPersistFailed { .. }), "the persist failure must keep its dedicated variant, got {error:?}", ); + let text = error.to_string(); + assert!( + !text.contains("disk") && !text.contains("account"), + "a constraint violation has nothing to do with disk space: {text}", + ); assert!( backend .identity_funding_account(&seed_hash, fault_topup_account_type()) @@ -7187,3 +7268,159 @@ async fn reconcile_managed_identities_skips_identities_linked_to_another_wallet( backend.shutdown().await; } + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn remove_wallet_keeps_upgrade_backups_without_backend() { + let (ctx, _sender, _tmp) = offline_testnet_context(); + assert!(ctx.wallet_backend().is_err()); + let wallet = + crate::model::wallet::Wallet::new_from_seed([0xA7; 64], Network::Testnet, None, None) + .expect("build wallet"); + let seed_hash = wallet.seed_hash(); + ctx.wallet_context() + .insert_test_wallet(seed_hash, Arc::new(RwLock::new(wallet))); + let backups = [ + "det-app.sqlite.platform-67d4ef3-backup-fixture.sqlite", + "det-testnet.sqlite.platform-67d4ef3-backup-fixture.pending", + "backups/auto/pre-migration-det-testnet-1-to-2-20260915T120001Z.db", + ] + .map(|name| ctx.data_dir().join(name)); + std::fs::create_dir_all(backups[2].parent().unwrap()).unwrap(); + for backup in &backups { + std::fs::write(backup, b"old wallet history").unwrap(); + } + ctx.remove_wallet(&seed_hash).unwrap(); + for backup in &backups { + assert!(backup.exists(), "{}", backup.display()); + } +} + +/// An upgrade-backup fixture next to `database`, last modified `days` ago. +fn aged_upgrade_backup(database: &std::path::Path, days: u64) -> std::path::PathBuf { + named_aged_upgrade_backup(database, "aged", days) +} + +/// A newer backup of `database`: retention always keeps the newest one, so it lets +/// the aged backup expire. +fn recent_upgrade_backup(database: &std::path::Path) -> std::path::PathBuf { + named_aged_upgrade_backup(database, "recent", 1) +} + +fn named_aged_upgrade_backup( + database: &std::path::Path, + suffix: &str, + days: u64, +) -> std::path::PathBuf { + let backup = + crate::wallet_backend::platform_compatibility::bridge_backup_path(database, suffix); + std::fs::write(&backup, b"old wallet history").unwrap(); + std::fs::File::options() + .write(true) + .open(&backup) + .unwrap() + .set_modified( + std::time::SystemTime::now() - std::time::Duration::from_secs(days * 24 * 60 * 60), + ) + .unwrap(); + backup +} + +/// Wiring: once the wallet backend opens, expired upgrade backups of both databases go. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn ensure_wallet_backend_prunes_expired_upgrade_backups() { + let (ctx, sender, _tmp) = offline_testnet_context(); + let databases = [ + ctx.data_dir().join("det-app.sqlite"), + crate::wallet_backend::wallet_database_path(ctx.data_dir(), Network::Testnet), + ]; + let expired = databases + .each_ref() + .map(|database| aged_upgrade_backup(database, 120)); + let newest = databases + .each_ref() + .map(|database| recent_upgrade_backup(database)); + + ctx.ensure_wallet_backend(sender) + .await + .expect("ensure_wallet_backend should succeed offline"); + + for backup in &expired { + assert!(!backup.exists(), "{} must expire", backup.display()); + } + for backup in &newest { + assert!( + backup.exists(), + "{} is the newest and stays", + backup.display() + ); + } + ctx.wallet_backend().unwrap().shutdown().await; +} + +/// A wallet database that fails to open may still need its snapshot, so nothing is pruned. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn failed_wallet_open_keeps_expired_upgrade_backups() { + let (ctx, sender, _tmp) = offline_testnet_context(); + let wallet_db = crate::wallet_backend::wallet_database_path(ctx.data_dir(), Network::Testnet); + std::fs::create_dir_all(wallet_db.parent().unwrap()).unwrap(); + std::fs::write(&wallet_db, b"not a sqlite database").unwrap(); + let expired = [ + aged_upgrade_backup(&ctx.data_dir().join("det-app.sqlite"), 120), + aged_upgrade_backup(&wallet_db, 120), + ]; + + ctx.ensure_wallet_backend(sender) + .await + .expect_err("a corrupt wallet database must not open"); + + for backup in &expired { + assert!(backup.exists(), "{} must be kept", backup.display()); + } +} + +/// Saving a policy always persists it, but prunes only once the wallet backend has +/// opened this network's databases; the result reports what was deleted. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn update_backup_retention_prunes_only_after_wallet_backend_opens() { + use crate::backend_task::BackendTaskSuccessResult; + use crate::model::backup_retention::BackupRetention; + + let (ctx, sender, _tmp) = offline_testnet_context(); + let backup = aged_upgrade_backup(&ctx.data_dir().join("det-app.sqlite"), 60); + recent_upgrade_backup(&ctx.data_dir().join("det-app.sqlite")); + let saved = |result| match result { + Ok(BackendTaskSuccessResult::UpdatedBackupRetention { + retention, + deleted, + cleanup_failure, + }) => (retention, deleted, cleanup_failure.is_some()), + other => panic!("unexpected result: {other:?}"), + }; + + let strict = BackupRetention::DeleteAfterDays(30); + assert_eq!( + saved(ctx.handle_update_backup_retention(strict)), + (strict, 0, false) + ); + assert_eq!(ctx.backup_retention().unwrap(), strict); + assert!( + backup.exists(), + "no pruning before the wallet backend opens" + ); + + ctx.handle_update_backup_retention(BackupRetention::KeepForever) + .unwrap(); + ctx.ensure_wallet_backend(sender) + .await + .expect("ensure_wallet_backend should succeed offline"); + assert!(backup.exists(), "keep-forever deletes nothing at startup"); + + assert_eq!( + saved(ctx.handle_update_backup_retention(strict)), + (strict, 1, false) + ); + assert!(!backup.exists()); + ctx.wallet_backend().unwrap().shutdown().await; +} + +mod deletion_sentinel; diff --git a/src/context/wallet_lifecycle/tests/deletion_sentinel.rs b/src/context/wallet_lifecycle/tests/deletion_sentinel.rs new file mode 100644 index 000000000..32d2a39ab --- /dev/null +++ b/src/context/wallet_lifecycle/tests/deletion_sentinel.rs @@ -0,0 +1,262 @@ +//! Sentinel for the file-deletion chokepoint: every production deletion path +//! runs against a realistic data directory, and every file it does not target +//! must survive byte-for-byte. + +use super::*; +use dash_sdk::dpp::identity::accessors::IdentityGettersV0; +use std::collections::{BTreeMap, BTreeSet}; +use std::path::PathBuf; +use std::time::{Duration, SystemTime}; + +const DAY: Duration = Duration::from_secs(24 * 60 * 60); + +fn plant(path: &Path, age: Duration) { + std::fs::create_dir_all(path.parent().unwrap()).unwrap(); + std::fs::write(path, path.to_string_lossy().as_bytes()).unwrap(); + std::fs::File::options() + .write(true) + .open(path) + .unwrap() + .set_modified(SystemTime::now() - age) + .unwrap(); +} + +fn snapshot(root: &Path) -> BTreeMap> { + let mut files = BTreeMap::new(); + let mut pending = vec![root.to_owned()]; + while let Some(dir) = pending.pop() { + for entry in std::fs::read_dir(&dir).unwrap() { + let path = entry.unwrap().path(); + if path.is_dir() { + pending.push(path); + } else { + files.insert(path.clone(), std::fs::read(&path).unwrap()); + } + } + } + files +} + +/// Live files the testnet context itself writes while the operations run: +/// they must survive, but their bytes may legitimately change. +fn written_by_context(data_dir: &Path, path: &Path) -> bool { + if path.starts_with(data_dir.join("secrets")) { + return true; + } + let Some(name) = path.file_name().and_then(|name| name.to_str()) else { + return false; + }; + let live = [ + "det-app.sqlite", + "det-testnet.sqlite", + "det-testnet-shielded.sqlite", + "data.db", + ]; + path.parent() == Some(data_dir) + && live.iter().any(|base| { + name == *base + || [ + "-wal", + "-shm", + "-journal", + crate::wallet_backend::platform_compatibility::LOCK_SUFFIX, + ] + .iter() + .any(|suffix| name == format!("{base}{suffix}")) + }) +} + +/// Retention prune, wallet removal, identity removal, log rotation, SPV-cache +/// clear and a full testnet network clear run against a data directory holding +/// every network's databases. Only the files each operation targets disappear; +/// every other file keeps its exact bytes, and the testnet clear never reaches +/// another network's data. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn deletion_paths_leave_every_untargeted_file_intact() { + let (ctx, sender, tmp) = offline_testnet_context(); + ctx.ensure_wallet_backend(sender.clone()) + .await + .expect("ensure_wallet_backend should succeed offline"); + let dir = tmp.path(); + let old = DAY * 400; + let fresh = DAY; + + // Other networks' live databases and sidecars: never deletable. + let mut sentinels = Vec::new(); + for network in [Network::Mainnet, Network::Devnet, Network::Regtest] { + let wallet = wallet_database_path(dir, network); + let shielded = crate::wallet_backend::shielded_database_path(dir, network); + for base in [wallet, shielded] { + for suffix in ["", "-wal", "-shm", "-journal"] { + let mut name = base.as_os_str().to_owned(); + name.push(suffix); + sentinels.push(PathBuf::from(name)); + } + } + } + // Other networks' SPV data, config, a pre-migration copy, current logs and + // a foreign-stem log. + sentinels.extend( + [ + "spv/mainnet/blocks/segment.dat", + "spv/mainnet/peers.dat", + "spv/mainnet/det-shielded.sqlite", + "spv/mainnet.lock", + "data.db.premigration", + "det.log", + "det-stderr.0000000001.log", + "notes.txt", + ] + .map(|relative| dir.join(relative)), + ); + for path in &sentinels { + plant(path, old); + } + // The newest backup of each database is kept by retention whatever its age, and + // the network clear keeps the shared app database's backups for retention too. + let newest: Vec = [ + "backups/auto/pre-migration-det-mainnet-1-to-2-20000101T000000Z.db", + "det-app.sqlite.platform-67d4ef3-backup-new.sqlite", + "backups/data_backup_20000102_000000.db", + ] + .map(|relative| dir.join(relative)) + .into(); + for path in &newest { + plant(path, fresh); + } + sentinels.extend(newest); + // The current log is rotated to a name from its mtime; keep it recent so the + // rotated copy is retained rather than expiring in the same pass. + plant(&dir.join("det.log"), fresh); + + let targeted: BTreeSet = [ + // Retention prune (global): expired backups of every network's database. + "det-app.sqlite.platform-67d4ef3-backup-old.sqlite", + "det-mainnet.sqlite.platform-67d4ef3-backup-old.sqlite", + "det-testnet.sqlite.platform-67d4ef3-backup-old.sqlite", + "backups/auto/pre-migration-det-testnet-1-to-2-20000101T000000Z.db", + "backups/data_backup_20000101_000000.db", + // Network clear: every remaining testnet upgrade backup, including the + // expired one retention keeps as the latest migration's snapshot. + "det-testnet.sqlite.platform-67d4ef3-backup-new.sqlite", + "backups/auto/pre-migration-det-testnet-2-to-3-20000102T000000Z.db", + // Network clear (SPV cache + retired shielded files): testnet only. + "spv/testnet/blocks/segment.dat", + "spv/testnet/filters/nested/segment.dat", + "spv/testnet/peers.dat", + "spv/testnet.lock", + "spv/testnet/det-shielded.sqlite", + "spv/testnet/shielded-commitment-tree.sqlite", + // Log rotation: an expired rotated log of the rotated stem. + "det.0000000001.log", + ] + .map(|relative| dir.join(relative)) + .into_iter() + .collect(); + for path in &targeted { + let age = if path.to_string_lossy().contains("-new.") { + fresh + } else { + old + }; + plant(path, age); + } + + // Wallet and identity to remove. + let seed = [0xC4u8; 64]; + let wallet = crate::model::wallet::Wallet::new_from_seed(seed, Network::Testnet, None, None) + .expect("build wallet"); + let seed_hash = wallet.seed_hash(); + ctx.register_wallet(wallet, &seed, WalletOrigin::Fresh) + .expect("register wallet"); + let identity = wallet_owned_qualified_identity(None); + ctx.insert_local_qualified_identity(&identity, &None) + .expect("insert identity"); + + let before = snapshot(dir); + + let report = ctx.prune_expired_upgrade_backups(); + assert!(report.failure.is_none(), "{:?}", report.failure); + assert_eq!(report.deleted, 5, "retention prune"); + ctx.remove_wallet(&seed_hash).expect("remove wallet"); + ctx.delete_local_qualified_identity(&identity.identity.id()) + .expect("remove identity"); + crate::logging::rotate_log_in_dir(dir, "det"); + ctx.clear_spv_data().expect("clear SPV data"); + ctx.clear_network_database() + .await + .expect("clear network database"); + ctx.wallet_backend() + .expect("backend wired") + .shutdown() + .await; + + let after = snapshot(dir); + for path in &targeted { + assert!(!path.exists(), "{path:?} should have been deleted"); + } + for (path, bytes) in &before { + if targeted.contains(path) { + continue; + } + if written_by_context(dir, path) { + assert!(after.contains_key(path), "live file {path:?} was deleted"); + } else if path == &dir.join("det.log") { + // Rotation moves the current log aside rather than deleting it. + assert!( + after.values().any(|after| after == bytes), + "the current log's contents must survive rotation" + ); + } else { + assert_eq!(after.get(path), Some(bytes), "{path:?} must be unchanged"); + } + } + for path in &sentinels { + if path != &dir.join("det.log") { + assert!(path.exists(), "sentinel {path:?} missing"); + } + } +} + +/// "Keep forever" keeps every published snapshot through the databases' opens and +/// repeated retention passes; only crash-left partials go. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn keep_forever_keeps_every_published_snapshot() { + let (ctx, sender, tmp) = offline_testnet_context(); + let dir = tmp.path(); + ctx.set_backup_retention(crate::model::backup_retention::BackupRetention::KeepForever) + .unwrap(); + let published: Vec = [ + "det-app.sqlite.platform-67d4ef3-backup-a1.sqlite", + "det-testnet.sqlite.platform-67d4ef3-backup-a1.sqlite", + "backups/auto/pre-migration-det-testnet-1-to-2-20000101T000000Z.db", + "backups/auto/pre-migration-det-testnet-2-to-3-20010101T000000Z.db", + "backups/auto/pre-migration-det-mainnet-1-to-2-20000101T000000Z.db", + "backups/data_backup_20000101_000000.db", + ] + .map(|relative| dir.join(relative)) + .into(); + for path in &published { + plant(path, DAY * 4000); + } + let partial = dir.join("det-testnet.sqlite.platform-67d4ef3-backup-crash.pending"); + plant(&partial, DAY * 4000); + + ctx.ensure_wallet_backend(sender) + .await + .expect("ensure_wallet_backend should succeed offline"); + for _ in 0..3 { + let report = ctx.prune_expired_upgrade_backups(); + assert!(report.failure.is_none(), "{:?}", report.failure); + assert_eq!(report.deleted, 0); + } + ctx.wallet_backend() + .expect("backend wired") + .shutdown() + .await; + + for path in &published { + assert!(path.exists(), "{path:?} must be kept forever"); + } + assert!(!partial.exists(), "a crash-left partial is cleaned up"); +} diff --git a/src/database/initialization.rs b/src/database/initialization.rs index 29f4aa8eb..5d6f1a410 100644 --- a/src/database/initialization.rs +++ b/src/database/initialization.rs @@ -1,7 +1,5 @@ use crate::database::Database; -use chrono::Utc; use rusqlite::{Connection, params}; -use std::fs; use std::path::Path; /// Error during database migration with structured context. @@ -212,7 +210,9 @@ impl Database { let current_version = self.db_schema_version()?; if current_version != DEFAULT_DB_VERSION { - self.backup_db(db_file_path)?; + // No backup is taken: production boot paths open an existing + // `data.db` read-only and call `initialize` only for a fresh file, + // so this ladder runs on test fixtures, never on user data. // Migrations may need to read `.env` from the data directory // (see v34 for one such arm). The DB file typically lives at // `/`, so its parent is the data dir. @@ -706,34 +706,6 @@ impl Database { u16::try_from(version).map_err(|_| rusqlite::Error::IntegralValueOutOfRange(0, version)) } - /// Backs up the existing database with a unique timestamped filename in backups directory. - fn backup_db(&self, db_file_path: &Path) -> rusqlite::Result<()> { - if db_file_path.exists() { - // Create a "backups" folder in the same directory as `data.db` if not exists - let backups_dir = db_file_path - .parent() - .expect("Expected parent directory in creating db backup folder") - .join("backups"); - fs::create_dir_all(&backups_dir).map_err(|e| { - rusqlite::Error::ToSqlConversionFailure( - format!("Failed to create db backups directory: {}", e).into(), - ) - })?; - - // Generate a unique filename with a timestamp for the backup - let timestamp = Utc::now().format("%Y%m%d_%H%M%S"); - let backup_filename = format!("data_backup_{}.db", timestamp); - let backup_path = backups_dir.join(backup_filename); - - // Copy `data.db` to the unique backup file - fs::copy(db_file_path, &backup_path) - .map_err(|e| rusqlite::Error::ToSqlConversionFailure(e.into()))?; - tracing::info!("Old database backed up to {:?}", backup_path); - } - - Ok(()) - } - /// Creates all required tables with indexes if they don't already exist. /// /// `include_legacy` controls whether the wallet-family tables diff --git a/src/database/legacy_backups.rs b/src/database/legacy_backups.rs new file mode 100644 index 000000000..66700618b --- /dev/null +++ b/src/database/legacy_backups.rs @@ -0,0 +1,229 @@ +//! Legacy `data.db` migration backups (`backups/data_backup_.db`). +//! +//! Earlier app versions copied `data.db` here before upgrading it; this version opens an +//! existing `data.db` read-only and never writes new copies. The copies already on disk +//! are subject to the same time-based retention as the storage-upgrade backups. + +use std::path::{Path, PathBuf}; +use std::time::{Duration, SystemTime}; + +/// Directory, next to the database, that holds the legacy backups. +pub(crate) const BACKUP_DIR: &str = "backups"; +/// The legacy database these backups copy. +pub(crate) const LEGACY_DATABASE: &str = "data.db"; +const PREFIX: &str = "data_backup_"; +const SUFFIX: &str = ".db"; +/// UTC timestamp format embedded in the backup name. +const TIMESTAMP_FORMAT: &str = "%Y%m%d_%H%M%S"; + +/// File name of a legacy backup taken at `timestamp` (UTC). Tests only: this version +/// never writes legacy backups, it only recognises and expires existing ones. +#[cfg(test)] +pub(crate) fn backup_file_name(timestamp: chrono::DateTime) -> String { + format!("{PREFIX}{}{SUFFIX}", timestamp.format(TIMESTAMP_FORMAT)) +} + +/// The UTC creation time embedded in a legacy backup name, or `None` for any other +/// file. Also the deletion chokepoint's definition of a legacy backup name. +pub(crate) fn backup_timestamp(name: &str) -> Option { + let timestamp = name.strip_prefix(PREFIX)?.strip_suffix(SUFFIX)?; + let parsed = chrono::NaiveDateTime::parse_from_str(timestamp, TIMESTAMP_FORMAT).ok()?; + Some(parsed.and_utc().into()) +} + +/// `/backups` if it exists; an error if it is a symlink or not a directory. +fn backup_directory(data_dir: &Path) -> std::io::Result> { + let directory = data_dir.join(BACKUP_DIR); + match std::fs::symlink_metadata(&directory) { + Ok(metadata) if metadata.is_dir() => Ok(Some(directory)), + Ok(_) => Err(std::io::Error::other( + "Legacy backup directory is not a regular directory", + )), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(None), + Err(error) => Err(error), + } +} + +/// [`prune_expired`](crate::utils::backup_prune::prune_expired) over the regular files +/// with a legacy backup name in `/backups`; returns the number deleted. +pub(crate) fn prune_expired( + data_dir: &Path, + max_age: Duration, + now: SystemTime, +) -> std::io::Result { + let Some(directory) = backup_directory(data_dir)? else { + return Ok(0); + }; + let mut candidates = Vec::new(); + for entry in std::fs::read_dir(&directory)? { + let entry = entry?; + let Some(named) = entry.file_name().to_str().and_then(backup_timestamp) else { + continue; + }; + let path = entry.path(); + if std::fs::symlink_metadata(&path).is_ok_and(|metadata| !metadata.is_file()) { + continue; + } + candidates.push(crate::utils::backup_prune::Candidate { + created: crate::utils::backup_prune::created_at(&path, Some(named)), + path, + sequence: None, + }); + } + crate::utils::backup_prune::prune_expired( + candidates, + now, + max_age, + crate::utils::file_deletion::DeletionIntent::Backup { + database: &data_dir.join(LEGACY_DATABASE), + }, + ) +} + +#[cfg(test)] +mod tests { + use super::*; + + const DAY: Duration = Duration::from_secs(24 * 60 * 60); + + fn set_mtime(path: &Path, time: SystemTime) { + std::fs::File::options() + .write(true) + .open(path) + .unwrap() + .set_modified(time) + .unwrap(); + } + + #[test] + fn legacy_backup_names_round_trip() { + let taken = chrono::DateTime::parse_from_rfc3339("2024-03-05T06:07:08Z") + .unwrap() + .to_utc(); + let name = backup_file_name(taken); + assert_eq!(name, "data_backup_20240305_060708.db"); + assert_eq!(backup_timestamp(&name), Some(taken.into())); + for other in [ + "data_backup_20240305_060708.db-journal", + "data_backup_latest.db", + "data.db", + ] { + assert_eq!(backup_timestamp(other), None, "{other}"); + } + } + + #[test] + fn legacy_backups_expire_by_later_of_name_and_mtime() { + let dir = tempfile::tempdir().unwrap(); + let backups = dir.path().join(BACKUP_DIR); + std::fs::create_dir_all(backups.join("auto")).unwrap(); + let now = SystemTime::now(); + let long_ago = now - DAY * 200; + let old_name = backup_file_name(chrono::DateTime::from(long_ago)); + let expired = backups.join(&old_name); + std::fs::write(&expired, b"old").unwrap(); + set_mtime(&expired, long_ago); + let recently_written = + backups.join(backup_file_name(chrono::DateTime::from(long_ago - DAY))); + std::fs::write(&recently_written, b"restored copy").unwrap(); + let fresh = backups.join(backup_file_name(chrono::DateTime::from(now))); + std::fs::write(&fresh, b"new").unwrap(); + set_mtime(&fresh, long_ago); + let unrelated = backups.join("notes.db"); + std::fs::write(&unrelated, b"keep").unwrap(); + set_mtime(&unrelated, long_ago); + + assert_eq!(prune_expired(dir.path(), DAY * 90, now).unwrap(), 1); + + assert!(!expired.exists()); + assert!(recently_written.exists()); + assert!(fresh.exists()); + assert!(unrelated.exists()); + assert!(backups.join("auto").is_dir()); + } + + #[test] + fn missing_legacy_backup_directory_is_a_no_op() { + let dir = tempfile::tempdir().unwrap(); + assert_eq!( + prune_expired(dir.path(), DAY, SystemTime::now()).unwrap(), + 0 + ); + } + + #[cfg(unix)] + #[test] + fn legacy_backup_directory_symlink_is_refused() { + let dir = tempfile::tempdir().unwrap(); + let elsewhere = tempfile::tempdir().unwrap(); + let victim = elsewhere.path().join("data_backup_20000101_000000.db"); + std::fs::write(&victim, b"keep").unwrap(); + std::os::unix::fs::symlink(elsewhere.path(), dir.path().join(BACKUP_DIR)).unwrap(); + + prune_expired(dir.path(), DAY, SystemTime::now()).unwrap_err(); + + assert!( + victim.exists(), + "a symlinked backups directory is never entered" + ); + } + + #[test] + fn legacy_backup_path_that_is_a_file_is_refused() { + let dir = tempfile::tempdir().unwrap(); + std::fs::write(dir.path().join(BACKUP_DIR), b"not a directory").unwrap(); + prune_expired(dir.path(), DAY, SystemTime::now()).unwrap_err(); + } + + /// Two contexts pruning the shared data directory at once must not report a + /// backup the other one already deleted as a failure. + #[test] + fn concurrent_legacy_backup_prunes_do_not_fail() { + let dir = tempfile::tempdir().unwrap(); + let backups = dir.path().join(BACKUP_DIR); + std::fs::create_dir_all(&backups).unwrap(); + let long_ago = SystemTime::now() - DAY * 400; + for second in 0..10 { + let path = backups.join(format!("data_backup_20000101_0000{second:02}.db")); + std::fs::write(&path, b"old").unwrap(); + set_mtime(&path, long_ago); + } + let barrier = std::sync::Barrier::new(4); + let results: Vec<_> = std::thread::scope(|scope| { + let handles: Vec<_> = (0..4) + .map(|_| { + scope.spawn(|| { + barrier.wait(); + prune_expired(dir.path(), DAY, SystemTime::now()) + }) + }) + .collect(); + handles.into_iter().map(|h| h.join().unwrap()).collect() + }); + let deleted: usize = results + .into_iter() + .map(|result| result.expect("a concurrently deleted backup is not a failure")) + .sum(); + assert_eq!(deleted, 9, "each backup is counted once; the newest stays"); + assert_eq!(std::fs::read_dir(&backups).unwrap().count(), 1); + } + + #[cfg(unix)] + #[test] + fn legacy_backup_symlinks_are_never_followed() { + let dir = tempfile::tempdir().unwrap(); + let backups = dir.path().join(BACKUP_DIR); + std::fs::create_dir_all(&backups).unwrap(); + let target = dir.path().join("precious.db"); + std::fs::write(&target, b"keep").unwrap(); + let link = backups.join("data_backup_20000101_000000.db"); + std::os::unix::fs::symlink(&target, &link).unwrap(); + + assert_eq!( + prune_expired(dir.path(), DAY, SystemTime::now()).unwrap(), + 0 + ); + assert!(link.symlink_metadata().is_ok()); + assert!(target.exists()); + } +} diff --git a/src/database/mod.rs b/src/database/mod.rs index 221dfef38..4ad077af2 100644 --- a/src/database/mod.rs +++ b/src/database/mod.rs @@ -6,6 +6,7 @@ mod initialization; /// it, and the cross-version migration matrix (`tests/migration-matrix/`, a /// separate integration-test crate) asserts against it directly. pub use initialization::DEFAULT_DB_VERSION; +pub(crate) mod legacy_backups; pub(crate) mod legacy_import; mod settings; mod single_key_wallet; diff --git a/src/logging.rs b/src/logging.rs index e45773404..68ec0255b 100644 --- a/src/logging.rs +++ b/src/logging.rs @@ -507,7 +507,7 @@ fn rotate_log_file() { /// Rotates `{stem}.log` in `dir` to a timestamped name and removes rotated /// copies of the same stem older than [`LOG_RETENTION_DAYS`]. -fn rotate_log_in_dir(dir: &Path, stem: &str) { +pub(crate) fn rotate_log_in_dir(dir: &Path, stem: &str) { let log_path = dir.join(format!("{stem}.log")); if log_path.exists() { let ts = fs::metadata(&log_path) @@ -516,6 +516,8 @@ fn rotate_log_in_dir(dir: &Path, stem: &str) { .unwrap_or_else(|_| Local::now()) .timestamp(); let rotated = dir.join(rotated_name(stem, ts)); + // Not a deletion, so not routed through `delete_file`: it moves the + // current log aside under a fresh timestamped name. let _ = fs::rename(&log_path, rotated); } @@ -531,7 +533,10 @@ fn rotate_log_in_dir(dir: &Path, stem: &str) { if let Some(ts) = parse_rotated_ts(name, stem) && ts < cutoff { - let _ = fs::remove_file(path); + let _ = crate::utils::file_deletion::delete_file( + &path, + crate::utils::file_deletion::DeletionIntent::Log { dir, stem }, + ); } } } @@ -541,12 +546,18 @@ fn rotated_name(stem: &str, ts: i64) -> String { format!("{stem}.{ts:010}.log") } -/// Parses the timestamp out of a rotated log file name produced by -/// [`rotated_name`], returning `None` for names that don't match the stem. -fn parse_rotated_ts(name: &str, stem: &str) -> Option { - name.strip_prefix(&format!("{stem}.")) - .and_then(|s| s.strip_suffix(".log")) - .and_then(|s| s.parse::().ok()) +/// The timestamp in a rotated log name `..log`, or `None` for any +/// other name. Also the deletion chokepoint's definition of a rotated log. +pub(crate) fn parse_rotated_ts(name: &str, stem: &str) -> Option { + if stem.is_empty() { + return None; + } + name.strip_prefix(stem)? + .strip_prefix('.')? + .strip_suffix(".log") + .filter(|digits| !digits.is_empty() && digits.bytes().all(|b| b.is_ascii_digit()))? + .parse() + .ok() } #[cfg(test)] @@ -659,6 +670,9 @@ mod tests { assert_eq!(parse_rotated_ts("det.log", "det"), None); assert_eq!(parse_rotated_ts("det.notanumber.log", "det"), None); assert_eq!(parse_rotated_ts("unrelated.txt", "det"), None); + assert_eq!(parse_rotated_ts("det.-5.log", "det"), None); + assert_eq!(parse_rotated_ts("det.+5.log", "det"), None); + assert_eq!(parse_rotated_ts(".5.log", ""), None); } #[test] diff --git a/src/mcp/server.rs b/src/mcp/server.rs index b088483d2..760a3e1e6 100644 --- a/src/mcp/server.rs +++ b/src/mcp/server.rs @@ -371,7 +371,7 @@ pub async fn init_app_context() -> Result, McpError> { tracing::warn!("Failed to load .env file at {}: {e}", env_path.display()); } - let db_file_path = data_file_path(&data_dir, "data.db") + let db_file_path = data_file_path(&data_dir, crate::database::legacy_backups::LEGACY_DATABASE) .map_err(|e| McpError::internal_error(format!("db path: {e}"), None))?; let db = if db_file_path.exists() { Arc::new( diff --git a/src/model/backup_retention.rs b/src/model/backup_retention.rs new file mode 100644 index 000000000..6f9192b1d --- /dev/null +++ b/src/model/backup_retention.rs @@ -0,0 +1,176 @@ +//! Time-based retention policy for storage-upgrade backups. +//! +//! Upgrade backups are copies of the app and wallet databases taken before a +//! storage upgrade. They exist for recovery only, so they are deleted once +//! they are older than the retention period. Pure data and validation — the +//! filesystem work lives in `wallet_backend::platform_compatibility` and +//! `database::legacy_backups`. + +use serde::{Deserialize, Serialize}; +use std::time::{Duration, SystemTime}; + +/// Seconds in one retention day. +const SECONDS_PER_DAY: u64 = 24 * 60 * 60; + +/// How long storage-upgrade backups are kept before they are deleted automatically. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub enum BackupRetention { + /// Never delete upgrade backups automatically. + KeepForever, + /// Delete upgrade backups older than this many days. + DeleteAfterDays(u16), +} + +impl BackupRetention { + /// Global k/v key of the persisted policy. A dedicated key, not an + /// `AppSettings` field: that blob has a positional wire layout, so a new + /// field would break decoding for existing installs and older builds. + pub const KV_KEY: &'static str = "det:upgrade_backup_retention:v1"; + + /// Default retention period: roughly three months. + pub const DEFAULT_DAYS: u16 = 90; + + /// Shortest accepted retention period, in days. + pub const MIN_DAYS: u16 = 1; + + /// Longest accepted retention period, in days (about ten years). + pub const MAX_DAYS: u16 = 3650; + + /// Maximum backup age, or `None` when backups are kept forever. + pub fn max_age(self) -> Option { + match self { + Self::KeepForever => None, + Self::DeleteAfterDays(days) => Some(Duration::from_secs( + u64::from(days).saturating_mul(SECONDS_PER_DAY), + )), + } + } + + /// Whether switching from `current` to this policy can delete backups `current` keeps. + pub fn is_stricter_than(self, current: Self) -> bool { + match (self.max_age(), current.max_age()) { + (Some(new), Some(old)) => new < old, + (Some(_), None) => true, + (None, _) => false, + } + } + + /// The policy itself when valid, otherwise the default. + /// + /// Guards against a persisted value outside the accepted range, which + /// would otherwise delete backups sooner than any user could choose. + pub fn sanitized(self) -> Self { + match self { + Self::DeleteAfterDays(days) if validate_retention_days(u32::from(days)).is_err() => { + Self::default() + } + policy => policy, + } + } +} + +impl Default for BackupRetention { + fn default() -> Self { + Self::DeleteAfterDays(Self::DEFAULT_DAYS) + } +} + +/// A retention period outside the accepted range. +#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)] +#[error("Enter a number of days between {min} and {max}.")] +pub struct InvalidRetentionDays { + /// Shortest accepted period. + pub min: u16, + /// Longest accepted period. + pub max: u16, +} + +/// Validate a retention period in days. +pub fn validate_retention_days(days: u32) -> Result { + u16::try_from(days) + .ok() + .filter(|days| (BackupRetention::MIN_DAYS..=BackupRetention::MAX_DAYS).contains(days)) + .ok_or(InvalidRetentionDays { + min: BackupRetention::MIN_DAYS, + max: BackupRetention::MAX_DAYS, + }) +} + +/// Whether a backup created at `created` is older than `max_age` at `now`. +/// +/// `created` must be the latest of every creation signal available (file +/// modification time, a timestamp in the file name), so a reset or skewed +/// signal only postpones deletion. A creation time in the future is never +/// expired. +pub fn backup_expired(created: SystemTime, now: SystemTime, max_age: Duration) -> bool { + now.duration_since(created).is_ok_and(|age| age > max_age) +} + +#[cfg(test)] +mod tests { + use super::*; + + const DAY: Duration = Duration::from_secs(SECONDS_PER_DAY); + + #[test] + fn default_retention_is_ninety_days() { + assert_eq!( + BackupRetention::default(), + BackupRetention::DeleteAfterDays(90) + ); + assert_eq!(BackupRetention::default().max_age(), Some(DAY * 90)); + assert_eq!(BackupRetention::KeepForever.max_age(), None); + } + + #[test] + fn stricter_retention_is_any_change_that_deletes_sooner() { + use BackupRetention::{DeleteAfterDays, KeepForever}; + assert!(DeleteAfterDays(30).is_stricter_than(DeleteAfterDays(90))); + assert!(DeleteAfterDays(3650).is_stricter_than(KeepForever)); + assert!(!DeleteAfterDays(90).is_stricter_than(DeleteAfterDays(90))); + assert!(!DeleteAfterDays(120).is_stricter_than(DeleteAfterDays(90))); + assert!(!KeepForever.is_stricter_than(DeleteAfterDays(1))); + assert!(!KeepForever.is_stricter_than(KeepForever)); + } + + #[test] + fn retention_days_are_bounded() { + assert_eq!(validate_retention_days(1), Ok(1)); + assert_eq!(validate_retention_days(3650), Ok(3650)); + for invalid in [0, 3651, u32::from(u16::MAX) + 1] { + let error = validate_retention_days(invalid).unwrap_err(); + assert_eq!( + error.to_string(), + "Enter a number of days between 1 and 3650." + ); + } + } + + #[test] + fn out_of_range_persisted_policy_falls_back_to_default() { + assert_eq!( + BackupRetention::DeleteAfterDays(0).sanitized(), + BackupRetention::default() + ); + assert_eq!( + BackupRetention::DeleteAfterDays(30).sanitized(), + BackupRetention::DeleteAfterDays(30) + ); + assert_eq!( + BackupRetention::KeepForever.sanitized(), + BackupRetention::KeepForever + ); + } + + #[test] + fn backup_expires_only_after_max_age() { + let now = SystemTime::UNIX_EPOCH + DAY * 1000; + assert!(backup_expired(now - DAY * 91, now, DAY * 90)); + assert!(!backup_expired(now - DAY * 90, now, DAY * 90)); + assert!(!backup_expired(now - DAY, now, DAY * 90)); + assert!( + !backup_expired(now + DAY * 365, now, DAY * 90), + "a future creation time is never expired" + ); + } +} diff --git a/src/model/mod.rs b/src/model/mod.rs index 3694af15b..189f369b0 100644 --- a/src/model/mod.rs +++ b/src/model/mod.rs @@ -1,6 +1,7 @@ pub mod address; pub mod amount; pub mod asset_lock; +pub mod backup_retention; pub mod contested_name; pub mod dashpay; pub mod dashpay_derivation; diff --git a/src/ui/network_chooser_screen.rs b/src/ui/network_chooser_screen.rs index b407b884c..eaa08811d 100644 --- a/src/ui/network_chooser_screen.rs +++ b/src/ui/network_chooser_screen.rs @@ -1,11 +1,12 @@ use crate::app::AppAction; use crate::backend_task::core::CoreTask; use crate::backend_task::dapi_discovery::persist_dapi_addresses; -use crate::backend_task::error::TaskError; +use crate::backend_task::error::{BACKUP_RETENTION_SAVED_CLEANUP_INCOMPLETE, TaskError}; use crate::backend_task::system_task::SystemTask; use crate::backend_task::{BackendTask, BackendTaskContext, BackendTaskSuccessResult}; use crate::context::AppContext; use crate::context::connection_status::OverallConnectionState; +use crate::model::backup_retention::{BackupRetention, validate_retention_days}; use crate::model::spv_status::{SpvStatus, SpvStatusSnapshot}; use crate::model::user_role::UserRole; use crate::ui::components::MessageBanner; @@ -80,6 +81,7 @@ pub struct NetworkChooserScreen { db_clear_in_progress: bool, wipe_platform_data_dialog: Option, auto_start_spv: bool, + backup_retention: BackupRetentionForm, discovery_in_progress: bool, fetch_confirmation_dialog: Option, /// Set when DAPI discovery completes and an SDK reinit is needed. @@ -100,6 +102,14 @@ impl NetworkChooserScreen { let settings = current_context.get_app_settings(); let theme_preference = settings.theme_mode; let auto_start_spv = settings.auto_start_spv; + let backup_retention = BackupRetentionForm::new( + current_context + .backup_retention() + .inspect_err(|error| { + tracing::warn!(?error, "Backup retention setting unreadable"); + }) + .ok(), + ); Self { network_contexts: contexts.clone(), @@ -120,6 +130,7 @@ impl NetworkChooserScreen { db_clear_in_progress: false, wipe_platform_data_dialog: None, auto_start_spv, + backup_retention, discovery_in_progress: false, fetch_confirmation_dialog: None, pending_reinit_after_discovery: false, @@ -695,6 +706,12 @@ impl NetworkChooserScreen { ui.separator(); ui.add_space(12.0); + self.backup_retention.show(ui, dark_mode); + + ui.add_space(12.0); + ui.separator(); + ui.add_space(12.0); + ui.label( egui::RichText::new("Database Maintenance") .strong() @@ -1362,6 +1379,266 @@ impl NetworkChooserScreen { } } +/// Shown when the retention setting cannot be read; the backend then deletes nothing. +const BACKUP_RETENTION_UNREADABLE: &str = "Your backup retention setting could not be read, so no old upgrade backups are deleted. Choose a backup retention setting and save it."; + +/// Result banner after saving a retention policy. +fn backup_retention_saved_banner( + deleted: usize, + cleanup_incomplete: bool, +) -> (String, MessageType) { + if cleanup_incomplete { + return ( + BACKUP_RETENTION_SAVED_CLEANUP_INCOMPLETE.to_owned(), + MessageType::Warning, + ); + } + let message = match deleted { + 0 => "Your backup retention setting was saved.".to_owned(), + 1 => "One old upgrade backup was deleted.".to_owned(), + count => format!("{count} old upgrade backups were deleted."), + }; + (message, MessageType::Success) +} + +/// Confirmation text for switching to a stricter retention of `days` days. +fn stricter_retention_confirmation(days: u16) -> String { + if days == 1 { + "Upgrade backups older than one day will be deleted now, and from then on automatically. Deleted backups cannot be restored. Continue?".to_owned() + } else { + format!( + "Upgrade backups older than {days} days will be deleted now, and from then on automatically. Deleted backups cannot be restored. Continue?" + ) + } +} + +/// Settings-screen state for the upgrade-backup retention policy. +/// +/// Edits are saved only once finished, and only after the backend confirms is the +/// new policy treated as saved. A policy that deletes backups sooner than the saved +/// one needs explicit confirmation, since saving it deletes backups right away. +struct BackupRetentionForm { + /// Whether old backups are deleted automatically. + enabled: bool, + /// Retention period being edited, in days. + days: u32, + /// The persisted policy, or `None` when it could not be read. + saved: Option, + /// Inputs of the last settled state; only edits away from it are saved. + baseline: (bool, u32), + /// A save ready to send, emitted by [`Self::take_task`] on a frame whose screen + /// action is otherwise empty, so no other action can overwrite it. + outbox: Option, + /// A save sent to the backend and not yet answered. + pending: Option, + /// A stricter policy awaiting the user's confirmation. + confirmation: Option<(BackupRetention, ConfirmationDialog)>, +} + +impl BackupRetentionForm { + fn new(saved: Option) -> Self { + // An unreadable setting deletes nothing, which the unchecked box reflects. + let inputs = Self::inputs(saved.unwrap_or(BackupRetention::KeepForever)); + Self { + enabled: inputs.0, + days: inputs.1, + saved, + baseline: inputs, + outbox: None, + pending: None, + confirmation: None, + } + } + + fn inputs(policy: BackupRetention) -> (bool, u32) { + match policy { + BackupRetention::DeleteAfterDays(days) => (true, u32::from(days)), + BackupRetention::KeepForever => (false, u32::from(BackupRetention::DEFAULT_DAYS)), + } + } + + /// The policy the form currently describes, or `None` while the period is invalid. + fn desired(&self) -> Option { + if !self.enabled { + return Some(BackupRetention::KeepForever); + } + validate_retention_days(self.days) + .ok() + .map(BackupRetention::DeleteAfterDays) + } + + fn busy(&self) -> bool { + self.outbox.is_some() || self.pending.is_some() || self.confirmation.is_some() + } + + /// Whether the user is turning automatic deletion on. That waits for an explicit + /// apply, so the period can be chosen before any backup is deleted. + fn enabling(&self) -> bool { + self.enabled && !self.baseline.0 + } + + /// Handle a finished edit: queue it, ask to confirm a stricter policy, or do nothing. + /// `apply` is an explicit user request to save: it is required to turn deletion + /// on, and saves even an unchanged form to repair an unreadable setting. + fn settle(&mut self, apply: bool) { + let inputs = (self.enabled, self.days); + if self.busy() || (!apply && (inputs == self.baseline || self.enabling())) { + return; + } + let Some(desired) = self.desired() else { + return; + }; + if self.saved == Some(desired) { + self.baseline = inputs; + return; + } + let stricter = match self.saved { + Some(saved) => desired.is_stricter_than(saved), + None => desired.max_age().is_some(), + }; + match desired { + BackupRetention::DeleteAfterDays(days) if stricter => { + let dialog = ConfirmationDialog::new( + "Delete old upgrade backups?", + stricter_retention_confirmation(days), + ) + .confirm_text(Some("Delete Old Backups")) + .cancel_text(Some("Keep Backups")) + .danger_mode(true); + self.confirmation = Some((desired, dialog)); + } + _ => self.outbox = Some(desired), + } + } + + /// The queued save task, if any; the save counts as in flight from here on. + fn take_task(&mut self) -> AppAction { + let Some(policy) = self.outbox.take() else { + return AppAction::None; + }; + self.pending = Some(policy); + AppAction::BackendTask(BackendTask::SystemTask(SystemTask::UpdateBackupRetention( + policy, + ))) + } + + /// Return the inputs to the last settled state. + fn revert(&mut self) { + (self.enabled, self.days) = self.baseline; + } + + /// Resolve a user's answer to the confirmation dialog. + fn resolve_confirmation(&mut self, status: ConfirmationStatus) { + let Some((policy, _)) = self.confirmation.take() else { + return; + }; + match status { + ConfirmationStatus::Confirmed => self.outbox = Some(policy), + ConfirmationStatus::Canceled => self.revert(), + } + } + + /// The backend persisted `retention`. + fn save_succeeded(&mut self, retention: BackupRetention) { + self.pending = None; + self.saved = Some(retention); + if self.desired() != Some(retention) { + (self.enabled, self.days) = Self::inputs(retention); + } + self.baseline = (self.enabled, self.days); + } + + /// The backend could not persist the policy; the saved one still applies. + fn save_failed(&mut self) { + self.pending = None; + self.revert(); + } + + /// Re-sync with the persisted policy when the screen is shown again. + /// + /// A save whose answer never arrived no longer blocks the form: its result, if it + /// still comes, is applied as usual. A queued save or an open confirmation is kept. + fn reload(&mut self, saved: Option) { + self.pending = None; + if !self.busy() { + *self = Self::new(saved); + } + } + + /// Render the section. The save task it may queue is emitted by [`Self::take_task`]. + fn show(&mut self, ui: &mut Ui, dark_mode: bool) { + ui.label( + egui::RichText::new("Upgrade Backups") + .strong() + .color(DashColors::text_primary(dark_mode)), + ); + ui.add_space(6.0); + ui.label( + egui::RichText::new( + "Before a storage upgrade, the app saves a backup copy of your wallet data on this device. Old backups can be deleted automatically.", + ) + .color(DashColors::text_secondary(dark_mode)), + ); + ui.add_space(8.0); + if self.saved.is_none() { + ui.colored_label(DashColors::WARNING, BACKUP_RETENTION_UNREADABLE); + ui.add_space(6.0); + } + + let mut editing = false; + let mut apply = false; + ui.add_enabled_ui(!self.busy(), |ui| { + StyledCheckbox::new( + &mut self.enabled, + "Delete old upgrade backups automatically", + ) + .show(ui); + if self.enabled { + ui.add_space(6.0); + ui.horizontal(|ui| { + ui.label("Keep upgrade backups for this many days:"); + // Not clamped: an out-of-range entry stays visible with the error + // below and is never saved, instead of silently becoming the limit. + let response = ui.add(egui::DragValue::new(&mut self.days).speed(1)); + // Save only a finished edit, not every intermediate drag or keystroke. + editing = response.dragged() || response.has_focus(); + }); + if let Err(error) = validate_retention_days(self.days) { + ui.colored_label(DashColors::ERROR, error.to_string()); + } + } else { + ui.label( + egui::RichText::new("Upgrade backups are kept until you delete them.") + .color(DashColors::text_secondary(dark_mode)), + ); + } + if self.enabling() { + ui.add_space(6.0); + ui.label( + egui::RichText::new( + "Choose how many days to keep upgrade backups, then turn on automatic deletion.", + ) + .color(DashColors::text_secondary(dark_mode)), + ); + ui.add_space(6.0); + apply = ui.button("Turn On Automatic Deletion").clicked(); + } else if self.saved.is_none() { + ui.add_space(6.0); + apply = ui.button("Save Setting").clicked(); + } + }); + + if !editing { + self.settle(apply); + } + if let Some((_, dialog)) = self.confirmation.as_mut() + && let Some(status) = dialog.show(ui).inner.dialog_response + { + self.resolve_confirmation(status); + } + } +} + fn wipe_platform_data_available(role: UserRole, network: Network) -> bool { role.at_least(UserRole::Developer) && network == Network::Devnet } @@ -1387,6 +1664,8 @@ impl ScreenLike for NetworkChooserScreen { // changed on another surface (e.g. the onboarding row) since this cached // root screen was constructed. self.selected_role = self.current_app_context().user_role(); + self.backup_retention + .reload(self.current_app_context().backup_retention().ok()); } fn ui(&mut self, ui: &mut egui::Ui) -> AppAction { @@ -1416,6 +1695,12 @@ impl ScreenLike for NetworkChooserScreen { action |= AppAction::BackendTask(BackendTask::ReinitCoreClientAndSdk); } + // Only on an otherwise idle frame, so no other action can overwrite the save; + // a busy frame leaves it queued for the next one. + if action == AppAction::None { + action = self.backup_retention.take_task(); + } + // Recheck both network status every 3 seconds let recheck_time = Duration::from_secs(3); if action == AppAction::None { @@ -1447,7 +1732,24 @@ impl ScreenLike for NetworkChooserScreen { // refresh ("Updated to N node addresses."). // Handle DapiNodesDiscovered (from "Refresh DAPI endpoints" button) - if let BackendTaskSuccessResult::NetworkDatabaseCleared { .. } = + if let BackendTaskSuccessResult::UpdatedBackupRetention { + retention, + deleted, + cleanup_failure, + } = backend_task_success_result + { + self.backup_retention.save_succeeded(retention); + let (message, message_type) = + backup_retention_saved_banner(deleted, cleanup_failure.is_some()); + let banner = MessageBanner::set_global( + self.current_app_context().egui_ctx(), + message, + message_type, + ); + if let Some(failure) = cleanup_failure { + banner.with_details(failure); + } + } else if let BackendTaskSuccessResult::NetworkDatabaseCleared { .. } = &backend_task_success_result { self.db_clear_in_progress = false; @@ -1491,6 +1793,9 @@ impl ScreenLike for NetworkChooserScreen { if matches!(context, BackendTaskContext::ClearNetworkDatabase) { self.db_clear_in_progress = false; } + if matches!(context, BackendTaskContext::UpdateBackupRetention) { + self.backup_retention.save_failed(); + } } fn display_message(&mut self, _msg: &str, msg_type: MessageType) { @@ -1513,6 +1818,229 @@ mod tests { assert_eq!(chooser_network_label(Network::Regtest), "Local"); } + fn saved_retention(action: AppAction) -> Option { + match action { + AppAction::BackendTask(BackendTask::SystemTask(SystemTask::UpdateBackupRetention( + retention, + ))) => Some(retention), + _ => None, + } + } + + /// Settle a finished edit and return the save task it emits, if any. + fn step(form: &mut BackupRetentionForm, apply: bool) -> Option { + form.settle(apply); + saved_retention(form.take_task()) + } + + fn answer( + form: &mut BackupRetentionForm, + status: ConfirmationStatus, + ) -> Option { + form.resolve_confirmation(status); + saved_retention(form.take_task()) + } + + #[test] + fn out_of_range_retention_days_are_never_saved() { + let mut form = BackupRetentionForm::new(Some(BackupRetention::default())); + for days in [0, 3651, 4000] { + form.days = days; + assert_eq!(step(&mut form, false), None, "{days} must not be saved"); + assert_eq!(step(&mut form, true), None, "{days} must not be saved"); + assert_eq!(form.days, days, "the entry stays visible, not clamped"); + } + } + + #[test] + fn lost_backup_retention_result_does_not_wedge_the_form() { + let mut form = BackupRetentionForm::new(Some(BackupRetention::default())); + form.days = 200; + assert!(step(&mut form, false).is_some()); + assert!(form.busy()); + + form.reload(Some(BackupRetention::default())); + assert!(!form.busy(), "arriving again unblocks the form"); + assert_eq!((form.enabled, form.days), (true, 90)); + + // A late answer is still applied. + form.save_succeeded(BackupRetention::DeleteAfterDays(200)); + assert_eq!(form.saved, Some(BackupRetention::DeleteAfterDays(200))); + assert_eq!(form.days, 200); + + // A setting that became readable is picked up. + let mut form = BackupRetentionForm::new(None); + form.reload(Some(BackupRetention::KeepForever)); + assert_eq!(form.saved, Some(BackupRetention::KeepForever)); + assert!(!form.enabled); + } + + /// Looser policies save without confirmation. The saved policy changes only when + /// the backend confirms it; a failed save restores the previous inputs. + #[test] + fn backup_retention_form_saves_looser_policies_without_confirmation() { + let mut form = BackupRetentionForm::new(Some(BackupRetention::default())); + assert!(form.enabled); + assert_eq!(form.days, 90); + assert_eq!(step(&mut form, false), None, "unchanged"); + + form.days = 120; + assert_eq!( + step(&mut form, false), + Some(BackupRetention::DeleteAfterDays(120)) + ); + assert!(form.confirmation.is_none()); + assert_eq!(step(&mut form, false), None, "a save is in flight"); + assert_eq!(form.saved, Some(BackupRetention::default())); + form.save_failed(); + assert_eq!((form.enabled, form.days), (true, 90)); + assert_eq!(step(&mut form, false), None, "no resend loop"); + + form.days = 120; + assert_eq!( + step(&mut form, false), + Some(BackupRetention::DeleteAfterDays(120)), + "the user can retry" + ); + form.save_succeeded(BackupRetention::DeleteAfterDays(120)); + + form.enabled = false; + assert_eq!(step(&mut form, false), Some(BackupRetention::KeepForever)); + form.save_succeeded(BackupRetention::KeepForever); + assert_eq!(step(&mut form, false), None, "settled"); + + form.enabled = true; + form.days = 0; + assert_eq!(step(&mut form, true), None, "invalid period"); + } + + /// Shortening the period deletes backups at once, so it needs confirmation; + /// cancelling restores the saved inputs. + #[test] + fn stricter_backup_retention_requires_confirmation() { + let mut form = BackupRetentionForm::new(Some(BackupRetention::default())); + form.days = 1; + assert_eq!(step(&mut form, false), None); + assert!(form.confirmation.is_some()); + assert_eq!(step(&mut form, false), None, "awaiting the user"); + + assert_eq!(answer(&mut form, ConfirmationStatus::Canceled), None); + assert!(form.confirmation.is_none()); + assert_eq!((form.enabled, form.days), (true, 90)); + + form.days = 30; + assert_eq!(step(&mut form, false), None); + assert_eq!( + answer(&mut form, ConfirmationStatus::Confirmed), + Some(BackupRetention::DeleteAfterDays(30)) + ); + } + + /// Turning deletion on waits for an explicit apply, so the user can choose the + /// period first; the confirmation then names the period actually chosen. + #[test] + fn enabling_backup_retention_waits_for_the_chosen_period() { + let mut form = BackupRetentionForm::new(Some(BackupRetention::KeepForever)); + assert!(!form.enabled); + assert_eq!(form.days, u32::from(BackupRetention::DEFAULT_DAYS)); + + form.enabled = true; + assert_eq!(step(&mut form, false), None); + assert!( + form.confirmation.is_none(), + "ticking the box alone neither saves nor asks to delete" + ); + form.days = 365; + assert_eq!( + step(&mut form, false), + None, + "editing the period still waits" + ); + + assert_eq!(step(&mut form, true), None); + assert!(form.confirmation.is_some(), "enabling deletion is stricter"); + assert_eq!( + answer(&mut form, ConfirmationStatus::Confirmed), + Some(BackupRetention::DeleteAfterDays(365)) + ); + form.save_succeeded(BackupRetention::DeleteAfterDays(365)); + assert!(!form.enabling()); + assert_eq!((form.enabled, form.days), (true, 365)); + + // Unticking before applying simply returns to keeping backups forever. + let mut form = BackupRetentionForm::new(Some(BackupRetention::KeepForever)); + form.enabled = true; + form.days = 10; + form.settle(false); + form.enabled = false; + assert_eq!(step(&mut form, false), None); + assert!(form.confirmation.is_none()); + } + + /// The save task leaves the form only when the screen can emit it, so a busy + /// frame never drops it and the form never waits on a task that was not sent. + #[test] + fn queued_backup_retention_save_survives_a_busy_frame() { + let mut form = BackupRetentionForm::new(Some(BackupRetention::default())); + form.days = 200; + form.settle(false); + assert!(form.pending.is_none(), "not in flight until emitted"); + assert!(form.busy(), "further edits wait for the queued save"); + // A frame whose action is already taken does not call take_task; the + // save stays queued and the next idle frame emits it. + assert_eq!( + saved_retention(form.take_task()), + Some(BackupRetention::DeleteAfterDays(200)) + ); + assert_eq!(form.pending, Some(BackupRetention::DeleteAfterDays(200))); + assert_eq!(saved_retention(form.take_task()), None, "emitted once"); + } + + /// An unreadable setting shows as unchecked (nothing is deleted) and can be + /// repaired by saving explicitly. + #[test] + fn unreadable_backup_retention_can_be_repaired() { + let mut form = BackupRetentionForm::new(None); + assert!(!form.enabled); + assert_eq!(step(&mut form, false), None, "never saved unasked"); + assert_eq!(step(&mut form, true), Some(BackupRetention::KeepForever)); + + let mut form = BackupRetentionForm::new(None); + form.enabled = true; + assert_eq!(step(&mut form, false), None); + assert_eq!(step(&mut form, true), None); + assert!( + form.confirmation.is_some(), + "enabling deletion from an unknown policy needs confirmation" + ); + } + + #[test] + fn backup_retention_banner_reports_deleted_count() { + assert_eq!( + backup_retention_saved_banner(0, false), + ( + "Your backup retention setting was saved.".to_owned(), + MessageType::Success + ) + ); + assert_eq!( + backup_retention_saved_banner(1, false).0, + "One old upgrade backup was deleted." + ); + assert_eq!( + backup_retention_saved_banner(3, false).0, + "3 old upgrade backups were deleted." + ); + assert_eq!( + backup_retention_saved_banner(3, true), + ( + BACKUP_RETENTION_SAVED_CLEANUP_INCOMPLETE.to_owned(), + MessageType::Warning + ) + ); + } + #[test] fn wipe_platform_data_is_available_only_to_developers_on_devnet() { assert!(!wipe_platform_data_available( diff --git a/src/ui/tokens/tokens_screen/mod.rs b/src/ui/tokens/tokens_screen/mod.rs index ff20837c7..6bfea6ced 100644 --- a/src/ui/tokens/tokens_screen/mod.rs +++ b/src/ui/tokens/tokens_screen/mod.rs @@ -3245,6 +3245,10 @@ mod tests { } #[test] + #[expect( + clippy::disallowed_methods, + reason = "test fixture setup/teardown outside any production deletion path" + )] fn test_token_creator_ui_builds_correct_contract() { let db_file_path = "test_db_token_creator"; let _ = std::fs::remove_file(db_file_path); // Clean up from previous runs @@ -3547,6 +3551,10 @@ mod tests { } #[test] + #[expect( + clippy::disallowed_methods, + reason = "test fixture setup/teardown outside any production deletion path" + )] fn test_distribution_function_random() { let db_file_path = "test_db_distribution_random"; let _ = std::fs::remove_file(db_file_path); // Clean up from previous runs @@ -3663,6 +3671,10 @@ mod tests { } #[test] + #[expect( + clippy::disallowed_methods, + reason = "test fixture setup/teardown outside any production deletion path" + )] fn test_parse_token_build_args_fails_with_empty_token_name() { let db_file_path = "test_db_empty_token_name"; let _ = std::fs::remove_file(db_file_path); // Clean up from previous runs diff --git a/src/utils/backup_prune.rs b/src/utils/backup_prune.rs new file mode 100644 index 000000000..d79f41959 --- /dev/null +++ b/src/utils/backup_prune.rs @@ -0,0 +1,407 @@ +//! Time-based expiry shared by every upgrade-backup location (bridge, upstream +//! `pre-migration-*`, legacy `data.db`). +//! +//! Backups older than the retention period are deleted, except the newest usable one +//! of each database by creation time (preferring those not dated in the future) and by +//! upgrade-history position. Copies whose usability is undetermined (unreadable, or +//! with a non-empty rollback journal beside them) are never deleted. With no usable +//! backup, the newest files are kept anyway. + +use super::file_deletion::{DeletionIntent, delete_file}; +use std::collections::BTreeSet; +use std::path::{Path, PathBuf}; +use std::time::{Duration, SystemTime}; + +/// How far ahead of `now` a creation time may be and still count as not in the +/// future when choosing the newest backup. Absorbs ordinary clock drift. +const FUTURE_TOLERANCE: Duration = Duration::from_secs(24 * 60 * 60); + +/// A published backup of one database. +#[derive(Debug)] +pub(crate) struct Candidate { + /// The backup file. + pub(crate) path: PathBuf, + /// When the backup was created, or the error that prevented reading it. + pub(crate) created: std::io::Result, + /// Clock-independent position in the database's upgrade history; a larger value + /// is a later snapshot. `None` when the name carries no such signal. + pub(crate) sequence: Option<(u32, u32)>, +} + +/// A candidate whose creation time is known. +struct Dated { + created: SystemTime, + path: PathBuf, + sequence: Option<(u32, u32)>, + usability: Usability, +} + +/// Whether a backup can be restored, as far as a cheap structural check can tell. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Usability { + /// A complete SQLite database. + Usable, + /// Empty, truncated, header-only or not SQLite at all. + Unusable, + /// Not judged: the file could not be read, or it looks complete but has a + /// non-empty rollback journal beside it (a torn write or a harmless leftover). + Undetermined, +} + +/// Delete expired `candidates` (all backups of `intent`'s database) as the module docs +/// describe. Attempts every one and returns the first failure, or the number deleted. +pub(crate) fn prune_expired( + candidates: Vec, + now: SystemTime, + max_age: Duration, + intent: DeletionIntent<'_>, +) -> std::io::Result { + let mut first_error = None; + let mut dated = Vec::with_capacity(candidates.len()); + for candidate in candidates { + let checked = candidate + .created + .map(|created| (created, check_snapshot(&candidate.path))); + let (created, usability) = match checked { + Ok((created, Ok(usability))) => (created, usability), + Ok((created, Err(error))) if error.kind() != std::io::ErrorKind::NotFound => { + // Unreadable is not proof of unusable: keep the file and retry later. + first_error.get_or_insert(error); + (created, Usability::Undetermined) + } + // Already gone, e.g. pruned concurrently from another network's context. + Ok((_, Err(_))) => continue, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => continue, + Err(error) => { + first_error.get_or_insert(error); + continue; + } + }; + dated.push(Dated { + created, + path: candidate.path, + sequence: candidate.sequence, + usability, + }); + } + let floors = floors(&dated, now); + let mut removed = 0; + let mut touched = BTreeSet::new(); + for (index, backup) in dated.iter().enumerate() { + if floors.contains(&index) + || backup.usability == Usability::Undetermined + || !crate::model::backup_retention::backup_expired(backup.created, now, max_age) + { + continue; + } + match delete_file(&backup.path, intent) { + Ok(()) => { + removed += 1; + if let Some(directory) = backup.path.parent() { + touched.insert(directory.to_owned()); + } + } + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + Err(error) => { + first_error.get_or_insert(error); + } + } + } + for directory in touched { + if let Err(error) = sync_directory(&directory) { + first_error.get_or_insert(error); + } + } + first_error.map_or(Ok(removed), Err) +} + +/// Indices of the backups retention keeps whatever their age: the newest usable one +/// by creation time (preferring those not dated in the future) and the newest usable +/// one by upgrade-history position. Ties break on the path, so every pass and every +/// concurrent context picks the same files. With no usable backup at all, the same +/// choice is made among every backup instead. +fn floors(dated: &[Dated], now: SystemTime) -> BTreeSet { + let usable = |backup: &Dated| backup.usability == Usability::Usable; + let any_usable = dated.iter().any(usable); + let eligible = || { + dated + .iter() + .enumerate() + .filter(move |(_, backup)| usable(backup) || !any_usable) + }; + let latest_trusted = now.checked_add(FUTURE_TOLERANCE); + let by_time = eligible() + .max_by_key(|(_, backup)| { + let trusted = latest_trusted.is_none_or(|latest| backup.created <= latest); + (trusted, backup.created, &backup.path) + }) + .map(|(index, _)| index); + let by_sequence = eligible() + .filter_map(|(index, backup)| Some((index, backup.sequence?, backup))) + .max_by_key(|(_, sequence, backup)| (*sequence, backup.created, &backup.path)) + .map(|(index, _, _)| index); + by_time.into_iter().chain(by_sequence).collect() +} + +/// When `backup` was created: the later of its modification time and `named` (the +/// timestamp in its name), so a reset or skewed signal only delays deletion. +pub(crate) fn created_at(backup: &Path, named: Option) -> std::io::Result { + let modified = std::fs::symlink_metadata(backup)?.modified()?; + Ok(named.map_or(modified, |named| named.max(modified))) +} + +/// Length of the SQLite database header. +const SQLITE_HEADER_LEN: usize = 100; +/// Magic string every SQLite 3 database file starts with. +const SQLITE_MAGIC: &[u8] = b"SQLite format 3\0"; + +/// Whether `backup` is proven to be a complete SQLite database. +#[cfg(test)] +pub(crate) fn usable_snapshot(backup: &Path) -> bool { + matches!(check_snapshot(backup), Ok(Usability::Usable)) +} + +/// How restorable `backup` looks. +/// +/// A structural check that reads only the 100-byte header and never opens the file +/// through SQLite, which could create `-wal`/`-shm` sidecars or roll back a journal. +/// Usable requires the magic string, a valid page size and file-format version, and +/// a file size covering every page the header claims. A complete-looking file with a +/// non-empty `-journal` beside it is [`Usability::Undetermined`]. A file that cannot +/// be read is an error, never a verdict. +fn check_snapshot(backup: &Path) -> std::io::Result { + use std::io::Read; + let mut file = std::fs::File::open(backup)?; + let length = file.metadata()?.len(); + if length < SQLITE_HEADER_LEN as u64 { + return Ok(Usability::Unusable); + } + let mut header = [0u8; SQLITE_HEADER_LEN]; + file.read_exact(&mut header)?; + if !header.starts_with(SQLITE_MAGIC) { + return Ok(Usability::Unusable); + } + let be_u32 = |offset: usize| { + u32::from_be_bytes([ + header[offset], + header[offset + 1], + header[offset + 2], + header[offset + 3], + ]) + }; + let page_size = match u16::from_be_bytes([header[16], header[17]]) { + 1 => 65_536, + size if (512..=32_768).contains(&size) && size.is_power_of_two() => u64::from(size), + _ => return Ok(Usability::Unusable), + }; + // File-format write/read versions: 1 = rollback journal, 2 = WAL. + if !matches!(header[18], 1 | 2) || !matches!(header[19], 1 | 2) { + return Ok(Usability::Unusable); + } + // The in-header page count is valid only when "version-valid-for" matches the + // change counter, as every SQLite since 3.7.0 writes it; otherwise require + // whole pages. + let (change_counter, pages, valid_for) = (be_u32(24), be_u32(28), be_u32(92)); + let complete = if pages > 0 && change_counter == valid_for { + length >= u64::from(pages) * page_size + } else { + length >= page_size && length % page_size == 0 + }; + if !complete { + return Ok(Usability::Unusable); + } + let mut journal = backup.as_os_str().to_owned(); + journal.push("-journal"); + match std::fs::symlink_metadata(journal) { + Ok(metadata) if metadata.len() > 0 => { + tracing::debug!(backup = %backup.display(), "Upgrade backup has a rollback journal beside it; kept"); + Ok(Usability::Undetermined) + } + Ok(_) => Ok(Usability::Usable), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(Usability::Usable), + Err(error) => Err(error), + } +} + +/// Persist directory-entry changes (unlinks) in `directory`. +/// +/// Windows cannot open a directory as a `File` without extra flags and NTFS +/// journals metadata changes, so this is a no-op there. +pub(crate) fn sync_directory(directory: &Path) -> std::io::Result<()> { + #[cfg(unix)] + std::fs::File::open(directory)?.sync_all()?; + #[cfg(not(unix))] + let _ = directory; + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + const DAY: Duration = Duration::from_secs(24 * 60 * 60); + + /// A small, complete SQLite database at `path`. + fn write_database(path: &Path) { + rusqlite::Connection::open(path) + .unwrap() + .execute_batch("CREATE TABLE t (v INTEGER); INSERT INTO t VALUES (1);") + .unwrap(); + } + + fn dated(path: &Path, created: SystemTime, sequence: Option<(u32, u32)>) -> Dated { + Dated { + created, + path: path.to_owned(), + sequence, + usability: check_snapshot(path).unwrap_or(Usability::Undetermined), + } + } + + #[test] + fn complete_databases_are_usable() { + let dir = tempfile::tempdir().unwrap(); + let rollback = dir.path().join("rollback.db"); + write_database(&rollback); + assert!(usable_snapshot(&rollback), "rollback-journal database"); + + let wal = dir.path().join("wal.db"); + let conn = rusqlite::Connection::open(&wal).unwrap(); + conn.pragma_update(None, "journal_mode", "WAL").unwrap(); + conn.execute_batch("CREATE TABLE t (v INTEGER); INSERT INTO t VALUES (1);") + .unwrap(); + drop(conn); + assert!(usable_snapshot(&wal), "WAL database"); + } + + #[test] + fn interrupted_copies_are_not_usable() { + let dir = tempfile::tempdir().unwrap(); + let complete = dir.path().join("complete.db"); + write_database(&complete); + let bytes = std::fs::read(&complete).unwrap(); + let zeroed = vec![0u8; bytes.len()]; + let cases: [(&str, &[u8]); 6] = [ + ("empty", b""), + ("header only", &bytes[..SQLITE_HEADER_LEN]), + ("one byte short", &bytes[..bytes.len() - 1]), + ("half", &bytes[..bytes.len() / 2]), + ("zero-filled", &zeroed), + ("not sqlite", b"candy wrappers, not a database"), + ]; + for (case, content) in cases { + let path = dir.path().join(format!("{case}.db")); + std::fs::write(&path, content).unwrap(); + assert_eq!( + check_snapshot(&path).unwrap(), + Usability::Unusable, + "{case}" + ); + } + assert!(!usable_snapshot(&dir.path().join("missing.db")), "missing"); + } + + #[test] + fn hot_journal_leaves_a_copy_undetermined() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("torn.db"); + write_database(&path); + let journal = dir.path().join("torn.db-journal"); + std::fs::write(&journal, b"").unwrap(); + assert_eq!( + check_snapshot(&path).unwrap(), + Usability::Usable, + "an empty journal is not hot" + ); + std::fs::write(&journal, b"uncommitted pages").unwrap(); + assert_eq!(check_snapshot(&path).unwrap(), Usability::Undetermined); + } + + /// A newer unusable file never becomes the floor; the newest usable one does. + #[test] + fn floor_skips_unusable_newer_files() { + let dir = tempfile::tempdir().unwrap(); + let now = SystemTime::now(); + let valid = dir.path().join("valid.db"); + write_database(&valid); + let killed = dir.path().join("killed.db"); + std::fs::write(&killed, b"").unwrap(); + let backups = [ + dated(&valid, now - DAY * 200, None), + dated(&killed, now, Some((9, 8))), + ]; + assert_eq!(floors(&backups, now), BTreeSet::from([0])); + } + + /// A newer copy of undetermined usability does not displace the newest usable one. + #[test] + fn floor_ignores_undetermined_newer_files() { + let dir = tempfile::tempdir().unwrap(); + let now = SystemTime::now(); + let [valid, journaled] = ["valid.db", "journaled.db"].map(|name| dir.path().join(name)); + write_database(&valid); + write_database(&journaled); + std::fs::write(dir.path().join("journaled.db-journal"), b"pages").unwrap(); + let backups = [ + dated(&valid, now - DAY * 200, None), + dated(&journaled, now - DAY * 100, Some((2, 1))), + ]; + assert_eq!(backups[1].usability, Usability::Undetermined); + assert_eq!(floors(&backups, now), BTreeSet::from([0])); + } + + /// Without any usable backup the newest file is still kept. + #[test] + fn floor_falls_back_to_newest_file_when_none_is_usable() { + let dir = tempfile::tempdir().unwrap(); + let now = SystemTime::now(); + let [older, newer] = ["older.db", "newer.db"].map(|name| dir.path().join(name)); + for path in [&older, &newer] { + std::fs::write(path, b"junk").unwrap(); + } + let backups = [ + dated(&older, now - DAY * 20, None), + dated(&newer, now - DAY * 10, None), + ]; + assert_eq!(floors(&backups, now), BTreeSet::from([1])); + } + + /// A backup dated in the future does not displace the newest backup dated in the + /// past, and the latest migration's snapshot is kept whatever its date. + #[test] + fn floor_resists_clock_skew() { + let dir = tempfile::tempdir().unwrap(); + let now = SystemTime::now(); + let paths = ["a.db", "b.db", "c.db"].map(|name| dir.path().join(name)); + for path in &paths { + write_database(path); + } + let backups = [ + dated(&paths[0], now + DAY * 30, None), + dated(&paths[1], now - DAY * 100, Some((2, 1))), + dated(&paths[2], now - DAY * 400, Some((3, 2))), + ]; + assert_eq!(floors(&backups, now), BTreeSet::from([1, 2])); + // Ordinary drift is not the future. + let backups = [ + dated(&paths[0], now + Duration::from_secs(60), None), + dated(&paths[1], now - DAY, None), + ]; + assert_eq!(floors(&backups, now), BTreeSet::from([0])); + // Once real time catches up with a skewed date, that backup is trusted + // again and, being dated later, is the time floor once more. A limit of + // any clock-based order; only the migration order is immune. + let skewed = now + DAY * 30; + let backups = [ + dated(&paths[0], skewed, None), + dated(&paths[1], now - DAY * 100, None), + ]; + assert_eq!(floors(&backups, skewed), BTreeSet::from([0])); + // Only future-dated backups: the newest of them. + let backups = [ + dated(&paths[0], now + DAY * 30, None), + dated(&paths[1], now + DAY * 20, None), + ]; + assert_eq!(floors(&backups, now), BTreeSet::from([0])); + } +} diff --git a/src/utils/file_deletion.rs b/src/utils/file_deletion.rs new file mode 100644 index 000000000..b55a10856 --- /dev/null +++ b/src/utils/file_deletion.rs @@ -0,0 +1,380 @@ +//! The single chokepoint for deleting files from disk: every production deletion goes +//! through [`delete_file`] (or [`delete_tree`]) with a [`DeletionIntent`]. Before any +//! unlink it refuses, in order: +//! +//! 1. anything but a regular file (a symlink is unlinked, never followed, only under +//! [`DeletionIntent::NetworkClear`]); +//! 2. live application files by name (app/legacy/wallet/shielded databases, vaults, +//! `.env`, `*.premigration`, with SQLite sidecars), independent of caller paths; +//! 3. the same files by canonical location and, on Unix, device/inode; +//! 4. on Unix, files with more than one hard link (on Windows, alias-prone names); +//! 5. anything outside the intent's directories and file-name grammar. +//! +//! Check and unlink are not atomic; a concurrent rename in the window is out of scope. + +use dash_sdk::dpp::dashcore::Network; +use std::path::{Path, PathBuf}; + +/// SQLite sidecars that belong to a database file and share its protection. +const SQLITE_SIDECAR_SUFFIXES: [&str; 3] = ["-wal", "-shm", "-journal"]; + +/// Why a deletion needs to happen, and therefore where it may reach. +#[derive(Debug, Clone, Copy)] +pub(crate) enum DeletionIntent<'a> { + /// A retained backup of `database`: a bridge snapshot of it next to it, an + /// upstream `pre-migration-*` snapshot of it in the auto-backup directory, or, + /// for the legacy `data.db`, a `backups/data_backup_*.db` copy. + Backup { + /// The live database the backup was taken from. + database: &'a Path, + }, + /// Resyncable per-network data removed by a network clear: anything under + /// `/spv//` (including emptied directories and symlinks, + /// which are unlinked, never followed), plus the `/spv/.lock` + /// storage lock. + NetworkClear { + /// The application data directory. + data_dir: &'a Path, + /// The network whose data is being cleared. + network: Network, + }, + /// A rotated log file named `..log` directly in `dir`. + Log { + /// The directory holding the logs; also the application data directory. + dir: &'a Path, + /// The log's base name, e.g. `det`. + stem: &'a str, + }, +} + +/// A deletion the chokepoint refused. Wrapped in an [`std::io::Error`], so +/// callers can downcast it from the error's inner value. +#[derive(Debug, thiserror::Error)] +pub(crate) enum DeletionRefused { + /// The target is a live application file (database, sidecar, vault, config). + #[error("A protected application file was not deleted.")] + ProtectedFile { + /// The refused target. + path: PathBuf, + }, + /// The target is a directory, a symlink outside a network clear, or another + /// non-regular file. + #[error("Only regular files can be deleted.")] + NotRegularFile { + /// The refused target. + path: PathBuf, + }, + /// The target has more than one hard link. + #[error("A file with more than one hard link was not deleted.")] + HardLinked { + /// The refused target. + path: PathBuf, + }, + /// The target lies outside what the deletion's intent may reach. + #[error("A file outside the permitted location was not deleted.")] + OutsideScope { + /// The refused target. + path: PathBuf, + }, +} + +impl From for std::io::Error { + fn from(refused: DeletionRefused) -> Self { + std::io::Error::other(refused) + } +} + +/// Delete the regular file at `path` if every chokepoint check passes. +/// +/// # Errors +/// +/// A refused deletion is an [`std::io::Error`] wrapping [`DeletionRefused`]. A +/// missing target is [`std::io::ErrorKind::NotFound`], so callers that tolerate +/// an already-gone file keep doing so. Other I/O failures pass through. +pub(crate) fn delete_file(path: &Path, intent: DeletionIntent<'_>) -> std::io::Result<()> { + let metadata = check(path, intent)?; + unlink(path, &metadata) +} + +/// Delete the directory tree at `dir`, unlinking every file through the same +/// checks as [`delete_file`] and then removing the emptied directories. +/// +/// Only [`DeletionIntent::NetworkClear`] may remove directories, and only strictly +/// inside its network's SPV directory; each directory is scope-checked before it is +/// read and again before it is removed. Symlinks are unlinked, never followed. The +/// walk uses an explicit stack, so depth cannot overflow the call stack. Stops at +/// the first failure. +/// +/// # Errors +/// +/// As [`delete_file`]; a missing `dir` is [`std::io::ErrorKind::NotFound`], a `dir` +/// that is not a real directory is refused as [`DeletionRefused::NotRegularFile`], +/// and one outside the intent's scope as [`DeletionRefused::OutsideScope`]. +pub(crate) fn delete_tree(dir: &Path, intent: DeletionIntent<'_>) -> std::io::Result<()> { + if !std::fs::symlink_metadata(dir)?.is_dir() { + return Err(DeletionRefused::NotRegularFile { + path: dir.to_owned(), + } + .into()); + } + check_dir_scope(dir, intent)?; + // Post-order walk: a directory is removed once all its entries are gone. + let mut stack = vec![(dir.to_owned(), false)]; + while let Some((directory, emptied)) = stack.pop() { + if emptied { + check_dir_scope(&directory, intent)?; + remove_empty_dir(&directory)?; + continue; + } + stack.push((directory.clone(), true)); + for entry in std::fs::read_dir(&directory)? { + let path = entry?.path(); + if std::fs::symlink_metadata(&path)?.is_dir() { + check_dir_scope(&path, intent)?; + stack.push((path, false)); + } else { + delete_file(&path, intent)?; + } + } + } + Ok(()) +} + +/// Refuse a directory outside what `intent` may remove. +fn check_dir_scope(dir: &Path, intent: DeletionIntent<'_>) -> std::io::Result<()> { + let located = canonical_location(dir)?; + let allowed = match intent { + DeletionIntent::NetworkClear { data_dir, network } => { + let root = canonical_dir(&crate::wallet_backend::spv_storage_dir(data_dir, network)); + located.starts_with(&root) && located != root + } + DeletionIntent::Backup { .. } | DeletionIntent::Log { .. } => false, + }; + if allowed { + Ok(()) + } else { + Err(DeletionRefused::OutsideScope { + path: dir.to_owned(), + } + .into()) + } +} + +/// The only raw directory removal; its caller has emptied and scope-checked `dir`. +#[expect( + clippy::disallowed_methods, + reason = "the deletion chokepoint itself; `dir` is empty and scope-checked" +)] +fn remove_empty_dir(dir: &Path) -> std::io::Result<()> { + std::fs::remove_dir(dir) +} + +/// The only raw file unlink; its caller has passed every chokepoint check. +#[expect( + clippy::disallowed_methods, + reason = "the deletion chokepoint itself; every check has passed" +)] +fn unlink(path: &Path, metadata: &std::fs::Metadata) -> std::io::Result<()> { + // Windows removes a symlink to a directory as a directory; the link is still + // unlinked without following it. + #[cfg(windows)] + { + use std::os::windows::fs::FileTypeExt; + if metadata.file_type().is_symlink_dir() { + return std::fs::remove_dir(path); + } + } + #[cfg(not(windows))] + let _ = metadata; + std::fs::remove_file(path) +} + +/// Every check before an unlink; returns the target's own (unfollowed) metadata. +fn check(path: &Path, intent: DeletionIntent<'_>) -> std::io::Result { + let refused = + |make: fn(PathBuf) -> DeletionRefused| -> std::io::Error { make(path.to_owned()).into() }; + let metadata = std::fs::symlink_metadata(path)?; + let unlinkable_link = + metadata.is_symlink() && matches!(intent, DeletionIntent::NetworkClear { .. }); + if !metadata.is_file() && !unlinkable_link { + return Err(refused(|path| DeletionRefused::NotRegularFile { path })); + } + let name = path + .file_name() + .and_then(|name| name.to_str()) + .ok_or_else(|| refused(|path| DeletionRefused::OutsideScope { path }))?; + if is_protected_name(name) || (cfg!(windows) && is_windows_alias_name(name)) { + return Err(refused(|path| DeletionRefused::ProtectedFile { path })); + } + let located = canonical_location(path)?; + let protected = protected_paths(intent); + if protected + .iter() + .any(|file| same_location(&located, file) || same_inode(&metadata, file)) + { + return Err(refused(|path| DeletionRefused::ProtectedFile { path })); + } + #[cfg(unix)] + { + use std::os::unix::fs::MetadataExt; + match metadata.nlink() { + 1 => {} + // Unlinked by a concurrent deletion between lookup and stat. + 0 => return Err(std::io::ErrorKind::NotFound.into()), + _ => return Err(refused(|path| DeletionRefused::HardLinked { path })), + } + } + if !in_scope(&located, name, intent) { + return Err(refused(|path| DeletionRefused::OutsideScope { path })); + } + Ok(metadata) +} + +/// Whether `name` is a live application file, regardless of its directory. +fn is_protected_name(name: &str) -> bool { + let name = name.to_ascii_lowercase(); + let base = SQLITE_SIDECAR_SUFFIXES + .iter() + .find_map(|suffix| name.strip_suffix(suffix)) + .unwrap_or(&name); + if [ + crate::wallet_backend::APP_DATABASE_FILE, + crate::database::legacy_backups::LEGACY_DATABASE, + crate::app_dir::ENV_FILE, + ] + .contains(&base) + || base.ends_with(".pwsvault") + || base.ends_with(".premigration") + { + return true; + } + crate::wallet_backend::all_networks() + .into_iter() + .any(|network| { + let prefix = crate::wallet_backend::network_prefix(network); + base == format!("det-{prefix}.sqlite") + || base == format!("det-{prefix}-shielded.sqlite") + }) +} + +/// The live files in the intent's data directory, with their SQLite sidecars. +fn protected_paths(intent: DeletionIntent<'_>) -> Vec { + let (data_dir, database) = match intent { + DeletionIntent::Backup { database } => (parent_of(database), Some(database)), + DeletionIntent::NetworkClear { data_dir, .. } => (data_dir, None), + DeletionIntent::Log { dir, .. } => (dir, None), + }; + let mut roots = vec![ + crate::wallet_backend::app_database_path(data_dir), + data_dir.join(crate::database::legacy_backups::LEGACY_DATABASE), + data_dir.join(crate::app_dir::ENV_FILE), + crate::context::AppContext::secret_store_path(data_dir), + ]; + for network in crate::wallet_backend::all_networks() { + roots.push(crate::wallet_backend::wallet_database_path( + data_dir, network, + )); + roots.push(crate::wallet_backend::shielded_database_path( + data_dir, network, + )); + } + roots.extend(database.map(Path::to_owned)); + let mut protected = Vec::with_capacity(roots.len() * (SQLITE_SIDECAR_SUFFIXES.len() + 1)); + for root in roots { + for suffix in SQLITE_SIDECAR_SUFFIXES { + let mut name = root.as_os_str().to_owned(); + name.push(suffix); + protected.push(PathBuf::from(name)); + } + protected.push(root); + } + protected +} + +/// `path` with its parent directory resolved, keeping the final component as is +/// so a symlink target is never followed. +fn canonical_location(path: &Path) -> std::io::Result { + let absolute = std::path::absolute(path)?; + let name = absolute + .file_name() + .ok_or_else(|| std::io::Error::from(std::io::ErrorKind::InvalidInput))?; + Ok(parent_of(&absolute).canonicalize()?.join(name)) +} + +fn parent_of(path: &Path) -> &Path { + match path.parent() { + Some(parent) if !parent.as_os_str().is_empty() => parent, + _ => Path::new("."), + } +} + +/// `dir` resolved when it exists; otherwise as given (so it matches nothing real). +fn canonical_dir(dir: &Path) -> PathBuf { + dir.canonicalize().unwrap_or_else(|_| dir.to_owned()) +} + +fn same_location(located: &Path, protected: &Path) -> bool { + canonical_location(protected).is_ok_and(|protected| { + protected + .as_os_str() + .eq_ignore_ascii_case(located.as_os_str()) + }) +} + +#[cfg(unix)] +fn same_inode(target: &std::fs::Metadata, protected: &Path) -> bool { + use std::os::unix::fs::MetadataExt; + std::fs::metadata(protected) + .is_ok_and(|protected| protected.dev() == target.dev() && protected.ino() == target.ino()) +} + +#[cfg(not(unix))] +fn same_inode(_target: &std::fs::Metadata, _protected: &Path) -> bool { + false +} + +/// Whether `name` could alias another file on Windows: an 8.3 short name (`~`), an +/// alternate data stream (`:`), or a trailing `.` or space that Win32 strips. +fn is_windows_alias_name(name: &str) -> bool { + name.contains(['~', ':']) || name.ends_with(['.', ' ']) +} + +fn in_scope(located: &Path, name: &str, intent: DeletionIntent<'_>) -> bool { + let Some(parent) = located.parent() else { + return false; + }; + let is = |dir: &Path| canonical_dir(dir) == parent; + match intent { + DeletionIntent::Backup { database } => { + let directory = parent_of(database); + let legacy = database.file_name().is_some_and(|file| { + file.eq_ignore_ascii_case(crate::database::legacy_backups::LEGACY_DATABASE) + }); + (is(directory) + && crate::wallet_backend::platform_compatibility::is_bridge_backup_name( + database, name, + )) + || (is(&platform_wallet_storage::default_auto_backup_dir(database)) + && crate::wallet_backend::platform_compatibility::upstream_backup_timestamp( + database, name, + ) + .is_some()) + || (legacy + && is(&directory.join(crate::database::legacy_backups::BACKUP_DIR)) + && crate::database::legacy_backups::backup_timestamp(name).is_some()) + } + DeletionIntent::NetworkClear { data_dir, network } => { + let spv = crate::wallet_backend::spv_root_dir(data_dir); + let lock = format!("{}.lock", crate::wallet_backend::network_prefix(network)); + parent.starts_with(canonical_dir(&crate::wallet_backend::spv_storage_dir( + data_dir, network, + ))) || (is(&spv) && name == lock) + } + DeletionIntent::Log { dir, stem } => { + is(dir) && crate::logging::parse_rotated_ts(name, stem).is_some() + } + } +} + +#[cfg(test)] +mod tests; diff --git a/src/utils/file_deletion/tests.rs b/src/utils/file_deletion/tests.rs new file mode 100644 index 000000000..b4c171616 --- /dev/null +++ b/src/utils/file_deletion/tests.rs @@ -0,0 +1,592 @@ +#![expect( + clippy::disallowed_methods, + reason = "test fixture setup/teardown outside any production deletion path" +)] + +use super::*; + +/// Every protected base name in a data directory, as the app lays them out. +fn protected_files(data_dir: &Path) -> Vec { + let mut files = vec![ + data_dir.join("det-app.sqlite"), + data_dir.join("data.db"), + data_dir.join(".env"), + data_dir.join("secrets/det-secrets.pwsvault"), + data_dir.join("data.db.premigration"), + ]; + for network in crate::wallet_backend::all_networks() { + files.push(crate::wallet_backend::wallet_database_path( + data_dir, network, + )); + files.push(crate::wallet_backend::shielded_database_path( + data_dir, network, + )); + } + let with_sidecars: Vec = files + .iter() + .flat_map(|file| { + SQLITE_SIDECAR_SUFFIXES.iter().map(move |suffix| { + let mut name = file.as_os_str().to_owned(); + name.push(suffix); + PathBuf::from(name) + }) + }) + .collect(); + files.extend(with_sidecars); + for file in &files { + std::fs::create_dir_all(file.parent().unwrap()).unwrap(); + std::fs::write(file, b"live").unwrap(); + } + files +} + +/// Every intent, each scoped to reach `data_dir` as widely as it can. +fn intents<'a>(data_dir: &'a Path, app_db: &'a Path) -> Vec> { + vec![ + DeletionIntent::Backup { database: app_db }, + DeletionIntent::NetworkClear { + data_dir, + network: Network::Testnet, + }, + DeletionIntent::Log { + dir: data_dir, + stem: "det", + }, + ] +} + +fn refusal(error: &std::io::Error) -> &DeletionRefused { + error + .get_ref() + .and_then(|inner| inner.downcast_ref::()) + .unwrap_or_else(|| panic!("expected a refusal, got {error:?}")) +} + +fn assert_protected(result: std::io::Result<()>, what: &str) { + let error = result.expect_err(what); + assert!( + matches!(refusal(&error), DeletionRefused::ProtectedFile { .. }), + "{what}: {error:?}" + ); +} + +#[test] +fn delete_file_refuses_every_protected_file_under_every_intent() { + let dir = tempfile::tempdir().unwrap(); + let files = protected_files(dir.path()); + let app_db = dir.path().join("det-app.sqlite"); + for intent in intents(dir.path(), &app_db) { + for file in &files { + assert_protected(delete_file(file, intent), &format!("{intent:?} {file:?}")); + assert!(file.exists(), "{file:?} must survive"); + } + } +} + +#[test] +fn delete_file_refuses_protected_names_in_any_directory() { + let dir = tempfile::tempdir().unwrap(); + // A protected name inside the network-clear scope is still refused. + let spv = dir.path().join("spv/testnet"); + std::fs::create_dir_all(&spv).unwrap(); + for name in [ + "det-testnet.sqlite", + "DET-MAINNET.SQLITE-wal", + "det-app.sqlite-journal", + "wallet.db.premigration", + "other.pwsvault", + ] { + let file = spv.join(name); + std::fs::write(&file, b"live").unwrap(); + assert_protected( + delete_file( + &file, + DeletionIntent::NetworkClear { + data_dir: dir.path(), + network: Network::Testnet, + }, + ), + name, + ); + assert!(file.exists(), "{name} must survive"); + } +} + +#[test] +fn delete_file_refuses_protected_files_via_dot_dot_paths() { + let dir = tempfile::tempdir().unwrap(); + let files = protected_files(dir.path()); + std::fs::create_dir_all(dir.path().join("backups/auto")).unwrap(); + let app_db = dir.path().join("det-app.sqlite"); + for intent in intents(dir.path(), &app_db) { + for file in &files { + let relative = file.strip_prefix(dir.path()).unwrap(); + let alias = dir.path().join("backups/auto/../..").join(relative); + assert_protected(delete_file(&alias, intent), &format!("{alias:?}")); + assert!(file.exists()); + } + } +} + +#[cfg(unix)] +#[test] +fn delete_file_refuses_protected_files_via_symlinks() { + let dir = tempfile::tempdir().unwrap(); + let files = protected_files(dir.path()); + let backups = dir.path().join("backups"); + std::fs::create_dir_all(&backups).unwrap(); + // A symlink to the live file, named like an ordinary backup. + let link = backups.join("data_backup_20000101_000000.db"); + // A symlinked directory aliasing the data directory. + let dir_link = dir.path().join("spv"); + std::os::unix::fs::symlink(dir.path(), &dir_link).unwrap(); + for file in &files { + let _ = std::fs::remove_file(&link); + std::os::unix::fs::symlink(file, &link).unwrap(); + let database = dir.path().join("data.db"); + let error = delete_file( + &link, + DeletionIntent::Backup { + database: &database, + }, + ) + .expect_err("a symlink must be refused"); + assert!(matches!( + refusal(&error), + DeletionRefused::NotRegularFile { .. } + )); + + let through_dir = dir_link.join(file.strip_prefix(dir.path()).unwrap()); + for intent in intents(dir.path(), &database) { + assert_protected( + delete_file(&through_dir, intent), + &format!("{through_dir:?}"), + ); + } + assert!(file.exists()); + } +} + +#[cfg(unix)] +#[test] +fn delete_file_refuses_hard_links_to_protected_files() { + let dir = tempfile::tempdir().unwrap(); + let files = protected_files(dir.path()); + let auto = dir.path().join("backups/auto"); + std::fs::create_dir_all(&auto).unwrap(); + let alias = auto.join("pre-migration-det-app-1-to-2-20000101T000000Z.db"); + let database = dir.path().join("det-app.sqlite"); + for file in &files { + let _ = std::fs::remove_file(&alias); + std::fs::hard_link(file, &alias).unwrap(); + let error = delete_file( + &alias, + DeletionIntent::Backup { + database: &database, + }, + ) + .expect_err("a hard link to a protected file must be refused"); + // Files protected by exact location are matched by inode; the rest + // (e.g. `*.premigration`, protected by name only) by the link count. + assert!( + matches!( + refusal(&error), + DeletionRefused::ProtectedFile { .. } | DeletionRefused::HardLinked { .. } + ), + "hard link to {file:?}: {error:?}" + ); + assert!(alias.exists() && file.exists()); + } +} + +#[cfg(unix)] +#[test] +fn delete_file_refuses_any_hard_linked_file() { + let dir = tempfile::tempdir().unwrap(); + let database = dir.path().join("det-app.sqlite"); + let backup = dir + .path() + .join("det-app.sqlite.platform-67d4ef3-backup-a.sqlite"); + std::fs::write(&backup, b"copy").unwrap(); + std::fs::hard_link(&backup, dir.path().join("elsewhere")).unwrap(); + let error = delete_file( + &backup, + DeletionIntent::Backup { + database: &database, + }, + ) + .unwrap_err(); + assert!(matches!( + refusal(&error), + DeletionRefused::HardLinked { .. } + )); + assert!(backup.exists()); +} + +#[test] +fn delete_file_refuses_targets_outside_the_intent_scope() { + let dir = tempfile::tempdir().unwrap(); + let database = dir.path().join("det-app.sqlite"); + let legacy = dir.path().join("data.db"); + let cases = [ + ( + dir.path() + .join("elsewhere/det-app.sqlite.platform-67d4ef3-backup-a.sqlite"), + DeletionIntent::Backup { + database: &database, + }, + ), + ( + dir.path().join("spv/mainnet/peers.dat"), + DeletionIntent::NetworkClear { + data_dir: dir.path(), + network: Network::Testnet, + }, + ), + ( + dir.path().join("spv/mainnet.lock"), + DeletionIntent::NetworkClear { + data_dir: dir.path(), + network: Network::Testnet, + }, + ), + ( + dir.path().join("notes.txt"), + DeletionIntent::NetworkClear { + data_dir: dir.path(), + network: Network::Testnet, + }, + ), + ( + dir.path().join("app.ron"), + DeletionIntent::Backup { + database: &database, + }, + ), + ( + crate::wallet_backend::platform_compatibility::backup_lock_path(&database).unwrap(), + DeletionIntent::Backup { + database: &database, + }, + ), + ( + dir.path() + .join("det-mainnet.sqlite.platform-67d4ef3-backup-a.sqlite"), + DeletionIntent::Backup { + database: &database, + }, + ), + ( + dir.path().join("backups/data_backup_20000101_000000.db"), + DeletionIntent::Backup { + database: &database, + }, + ), + ( + dir.path() + .join("backups/data_backup_20000101_000000.db.a1B2c3.pending"), + DeletionIntent::Backup { database: &legacy }, + ), + ( + dir.path() + .join("backups/auto/pre-migration-det-mainnet-1-to-2-20000101T000000Z.db"), + DeletionIntent::Backup { + database: &database, + }, + ), + ( + dir.path().join("det.log"), + DeletionIntent::Log { + dir: dir.path(), + stem: "det", + }, + ), + ( + dir.path().join("other.0000000001.log"), + DeletionIntent::Log { + dir: dir.path(), + stem: "det", + }, + ), + ( + dir.path().join("logs/det.0000000001.log"), + DeletionIntent::Log { + dir: dir.path(), + stem: "det", + }, + ), + ]; + for (file, intent) in cases { + std::fs::create_dir_all(file.parent().unwrap()).unwrap(); + std::fs::write(&file, b"keep").unwrap(); + let error = delete_file(&file, intent).unwrap_err(); + assert!( + matches!(refusal(&error), DeletionRefused::OutsideScope { .. }), + "{file:?}: {error:?}" + ); + assert!(file.exists(), "{file:?}"); + } +} + +#[test] +fn delete_file_deletes_targets_in_scope() { + let dir = tempfile::tempdir().unwrap(); + let database = dir.path().join("det-testnet.sqlite"); + let legacy = dir.path().join("data.db"); + let cases = [ + ( + dir.path() + .join("det-testnet.sqlite.platform-67d4ef3-backup-a.sqlite"), + DeletionIntent::Backup { + database: &database, + }, + ), + ( + dir.path() + .join("det-testnet.sqlite.platform-67d4ef3-backup-b.pending"), + DeletionIntent::Backup { + database: &database, + }, + ), + ( + dir.path().join("backups/data_backup_20000101_000000.db"), + DeletionIntent::Backup { database: &legacy }, + ), + ( + dir.path() + .join("backups/auto/pre-migration-det-testnet-1-to-2-20000101T000000Z.db"), + DeletionIntent::Backup { + database: &database, + }, + ), + ( + dir.path().join("spv/testnet/filters/segment.dat"), + DeletionIntent::NetworkClear { + data_dir: dir.path(), + network: Network::Testnet, + }, + ), + ( + dir.path().join("spv/testnet.lock"), + DeletionIntent::NetworkClear { + data_dir: dir.path(), + network: Network::Testnet, + }, + ), + ( + dir.path().join("det.0000000001.log"), + DeletionIntent::Log { + dir: dir.path(), + stem: "det", + }, + ), + ]; + for (file, intent) in cases { + std::fs::create_dir_all(file.parent().unwrap()).unwrap(); + std::fs::write(&file, b"expendable").unwrap(); + delete_file(&file, intent).unwrap_or_else(|e| panic!("{file:?}: {e:?}")); + assert!(!file.exists(), "{file:?}"); + } +} + +#[test] +fn delete_file_reports_a_missing_target_as_not_found() { + let dir = tempfile::tempdir().unwrap(); + let error = delete_file( + &dir.path().join("spv/testnet/peers.dat"), + DeletionIntent::NetworkClear { + data_dir: dir.path(), + network: Network::Testnet, + }, + ) + .unwrap_err(); + assert_eq!(error.kind(), std::io::ErrorKind::NotFound); +} + +#[test] +fn delete_file_refuses_directories() { + let dir = tempfile::tempdir().unwrap(); + let target = dir.path().join("spv/testnet/blocks"); + std::fs::create_dir_all(&target).unwrap(); + let error = delete_file( + &target, + DeletionIntent::NetworkClear { + data_dir: dir.path(), + network: Network::Testnet, + }, + ) + .unwrap_err(); + assert!(matches!( + refusal(&error), + DeletionRefused::NotRegularFile { .. } + )); + assert!(target.is_dir()); +} + +#[test] +fn delete_tree_removes_a_cache_tree_but_stops_at_a_protected_file() { + let dir = tempfile::tempdir().unwrap(); + let intent = DeletionIntent::NetworkClear { + data_dir: dir.path(), + network: Network::Testnet, + }; + let cache = dir.path().join("spv/testnet/blocks"); + std::fs::create_dir_all(cache.join("nested")).unwrap(); + std::fs::write(cache.join("a.dat"), b"x").unwrap(); + std::fs::write(cache.join("nested/b.dat"), b"x").unwrap(); + delete_tree(&cache, intent).unwrap(); + assert!(!cache.exists()); + + std::fs::create_dir_all(&cache).unwrap(); + let planted = cache.join("det-app.sqlite"); + std::fs::write(&planted, b"live").unwrap(); + assert_protected(delete_tree(&cache, intent), "planted live database"); + assert!(planted.exists()); +} + +/// A network clear unlinks a symlinked cache entry itself (e.g. chain data moved to +/// another disk) without following it; other intents still refuse symlinks. +#[cfg(unix)] +#[test] +fn network_clear_unlinks_symlinks_without_following_them() { + let dir = tempfile::tempdir().unwrap(); + let outside = tempfile::tempdir().unwrap(); + std::fs::write(outside.path().join("keep.dat"), b"keep").unwrap(); + let spv = dir.path().join("spv/testnet"); + std::fs::create_dir_all(&spv).unwrap(); + let intent = DeletionIntent::NetworkClear { + data_dir: dir.path(), + network: Network::Testnet, + }; + let dir_link = spv.join("block_headers"); + std::os::unix::fs::symlink(outside.path(), &dir_link).unwrap(); + let file_link = spv.join("peers.dat"); + std::os::unix::fs::symlink(outside.path().join("keep.dat"), &file_link).unwrap(); + for link in [&dir_link, &file_link] { + delete_file(link, intent).unwrap_or_else(|e| panic!("{link:?}: {e:?}")); + assert!( + link.symlink_metadata().is_err(), + "{link:?} must be unlinked" + ); + } + assert!(outside.path().join("keep.dat").exists()); + + let backup_link = dir + .path() + .join("det-app.sqlite.platform-67d4ef3-backup-a.sqlite"); + std::os::unix::fs::symlink(outside.path().join("keep.dat"), &backup_link).unwrap(); + let database = dir.path().join("det-app.sqlite"); + let error = delete_file( + &backup_link, + DeletionIntent::Backup { + database: &database, + }, + ) + .unwrap_err(); + assert!(matches!( + refusal(&error), + DeletionRefused::NotRegularFile { .. } + )); +} + +#[test] +fn delete_tree_refuses_directories_outside_its_scope() { + let dir = tempfile::tempdir().unwrap(); + let database = dir.path().join("det-app.sqlite"); + let clear = DeletionIntent::NetworkClear { + data_dir: dir.path(), + network: Network::Testnet, + }; + let cases = [ + (dir.path().join("spv/testnet"), clear), + (dir.path().join("spv/mainnet/blocks"), clear), + ( + dir.path().join("backups/auto"), + DeletionIntent::Backup { + database: &database, + }, + ), + ( + dir.path().join("logs"), + DeletionIntent::Log { + dir: dir.path(), + stem: "det", + }, + ), + ]; + for (target, intent) in cases { + std::fs::create_dir_all(&target).unwrap(); + let error = delete_tree(&target, intent).unwrap_err(); + assert!( + matches!(refusal(&error), DeletionRefused::OutsideScope { .. }), + "{target:?}: {error:?}" + ); + assert!(target.is_dir(), "{target:?}"); + } +} + +#[test] +fn delete_tree_handles_deep_trees_without_recursion() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().join("spv/testnet/filters"); + // Short components keep the full path within OS limits. + let mut deepest = root.clone(); + for _ in 0..1000 { + deepest.push("d"); + } + std::fs::create_dir_all(&deepest).unwrap(); + std::fs::write(deepest.join("x.dat"), b"x").unwrap(); + delete_tree( + &root, + DeletionIntent::NetworkClear { + data_dir: dir.path(), + network: Network::Testnet, + }, + ) + .unwrap(); + assert!(!root.exists()); +} + +#[test] +fn windows_alias_names_are_recognised() { + for name in [ + "DET-AP~1.SQL", + "det-app.sqlite::$DATA", + "det-app.sqlite.", + "det-app.sqlite ", + ] { + assert!(is_windows_alias_name(name), "{name}"); + } + for name in [ + "det-app.sqlite.platform-67d4ef3-backup-a1.sqlite", + "pre-migration-det-app-1-to-2-20000101T000000Z.db", + "det.0000000001.log", + ] { + assert!(!is_windows_alias_name(name), "{name}"); + } +} + +#[cfg(unix)] +#[test] +fn delete_tree_never_follows_a_symlinked_directory() { + let dir = tempfile::tempdir().unwrap(); + let outside = dir.path().join("outside"); + std::fs::create_dir_all(&outside).unwrap(); + std::fs::write(outside.join("keep.dat"), b"keep").unwrap(); + let cache = dir.path().join("spv/testnet/blocks"); + std::fs::create_dir_all(&cache).unwrap(); + std::os::unix::fs::symlink(&outside, cache.join("link")).unwrap(); + let intent = DeletionIntent::NetworkClear { + data_dir: dir.path(), + network: Network::Testnet, + }; + let error = delete_tree(&cache.join("link"), intent).unwrap_err(); + assert!(matches!( + refusal(&error), + DeletionRefused::NotRegularFile { .. } + )); + // Inside the tree the link itself is unlinked; its target is never entered. + delete_tree(&cache, intent).unwrap(); + assert!(!cache.exists()); + assert!(outside.join("keep.dat").exists()); +} diff --git a/src/utils/mod.rs b/src/utils/mod.rs index d81e1a8bf..574b3edf9 100644 --- a/src/utils/mod.rs +++ b/src/utils/mod.rs @@ -1,3 +1,5 @@ +pub(crate) mod backup_prune; pub mod egui_mpsc; +pub(crate) mod file_deletion; pub mod parsers; pub mod tasks; diff --git a/src/wallet_backend/identity_ops.rs b/src/wallet_backend/identity_ops.rs index 3f0e06132..df2ae2903 100644 --- a/src/wallet_backend/identity_ops.rs +++ b/src/wallet_backend/identity_ops.rs @@ -189,8 +189,11 @@ enum RegistrationPersistFailure { impl RegistrationPersistFailure { fn into_task_error(self) -> TaskError { let (Self::Staged(source) | Self::Discarded(source)) = self; - TaskError::IdentityFundingAccountPersistFailed { - source: Box::new(source), + let source = Box::new(source); + if source.is_transient() { + TaskError::IdentityFundingAccountPersistBusy { source } + } else { + TaskError::IdentityFundingAccountPersistFailed { source } } } } diff --git a/src/wallet_backend/mod.rs b/src/wallet_backend/mod.rs index 0f8815d36..cf6978c68 100644 --- a/src/wallet_backend/mod.rs +++ b/src/wallet_backend/mod.rs @@ -529,6 +529,28 @@ impl std::fmt::Debug for WalletBackend { } } +/// File name of the cross-network app database in the data directory. +pub(crate) const APP_DATABASE_FILE: &str = "det-app.sqlite"; + +/// The cross-network app k/v database: `/det-app.sqlite`. +pub(crate) fn app_database_path(data_dir: &Path) -> std::path::PathBuf { + data_dir.join(APP_DATABASE_FILE) +} + +/// Every network, via an exhaustive match: a new [`Network`] variant fails to +/// compile here until each caller (e.g. the deletion deny-list) is reviewed. +pub(crate) fn all_networks() -> [Network; 4] { + let _exhaustive = |network: Network| match network { + Network::Mainnet | Network::Testnet | Network::Devnet | Network::Regtest => (), + }; + [ + Network::Mainnet, + Network::Testnet, + Network::Devnet, + Network::Regtest, + ] +} + /// The durable per-network wallet database: `/det-.sqlite`. /// /// A sibling of `det-app.sqlite`, deliberately outside `/spv//` @@ -539,6 +561,20 @@ pub(crate) fn wallet_database_path(data_dir: &Path, network: Network) -> std::pa data_dir.join(format!("det-{}.sqlite", network_prefix(network))) } +/// Parent of every network's [`spv_storage_dir`], `/spv/`. +pub(crate) fn spv_root_dir(data_dir: &Path) -> std::path::PathBuf { + data_dir.join("spv") +} + +/// The per-network chain-sync (SPV) cache directory, `/spv//`. +/// +/// The single definition shared by the backend (which creates it), the network +/// clear (which empties it) and the deletion chokepoint (which confines that clear +/// to it). +pub(crate) fn spv_storage_dir(data_dir: &Path, network: Network) -> std::path::PathBuf { + spv_root_dir(data_dir).join(network_prefix(network)) +} + /// The upstream shielded coordinator's store, /// `/det--shielded.sqlite` — the sibling of /// [`wallet_database_path`] holding all Orchard state. @@ -3045,12 +3081,12 @@ impl WalletBackend { app_data_dir: &Path, network: Network, ) -> Result { - let mut dir = app_data_dir.join("spv"); - crate::app_dir::ensure_data_dir_exists(&dir) - .map_err(|source| TaskError::FileSystem { source })?; - dir.push(kv::network_prefix(network)); - crate::app_dir::ensure_data_dir_exists(&dir) - .map_err(|source| TaskError::FileSystem { source })?; + let dir = spv_storage_dir(app_data_dir, network); + // Create (and lock down) the shared `spv` root before the network directory. + for directory in [spv_root_dir(app_data_dir), dir.clone()] { + crate::app_dir::ensure_data_dir_exists(&directory) + .map_err(|source| TaskError::FileSystem { source })?; + } Ok(dir) } } diff --git a/src/wallet_backend/platform_compatibility/engine.rs b/src/wallet_backend/platform_compatibility/engine.rs index 6c8b3f9e1..af1fd4f7e 100644 --- a/src/wallet_backend/platform_compatibility/engine.rs +++ b/src/wallet_backend/platform_compatibility/engine.rs @@ -3,6 +3,8 @@ use std::path::{Path, PathBuf}; use rusqlite::{Connection, OpenFlags, OptionalExtension, TransactionBehavior}; +use super::storage_failure::StorageFailure; + const OLD_SCHEMA: &str = include_str!("fixtures/67d4ef3.sql"); const TARGET_SCHEMA: &str = include_str!("fixtures/e3cd7cf.sql"); const HISTORY: &str = "refinery_schema_history"; @@ -11,26 +13,45 @@ const IDENTITY_INDEX_KEY: &str = "det:identity_index:v1"; /// A compatibility upgrade stopped before committing changes to the original database. #[derive(Debug, thiserror::Error)] pub enum UpgradeError { + /// Another process holds the database (`SQLITE_BUSY` / `SQLITE_LOCKED`). + #[error("{}", crate::backend_task::error::WALLET_DATA_IN_USE)] + InUse(#[source] Box), #[error( - "Could not upgrade wallet data. Check available disk space and restart the application." + "Could not upgrade wallet data because the disk is full. Free up disk space and try again." )] - Sqlite(#[from] rusqlite::Error), + StorageFull(#[source] Box), #[error( - "Could not back up wallet data. Check available disk space and restart the application." + "Could not access wallet storage. Check that the drive is connected and available, then try again." )] - Io(#[from] std::io::Error), + StorageUnavailable(#[source] Box), #[error( - "Wallet data does not match a supported upgrade. Keep your data folder and reopen the previous application version." + "Could not upgrade wallet data because memory is exhausted. Close other applications and try again." )] - Unrecognized, + OutOfMemory(#[source] Box), #[error( - "The saved identity list could not be read. Keep your data folder and reopen the previous application version." + "Could not write wallet data. Allow write access to the app data folder, then restart the application." + )] + AccessDenied(#[source] Box), + #[error( + "Could not upgrade wallet data, and your data was not changed. Restart the application to try again, or keep your data folder and reopen the previous application version." + )] + Sqlite(#[source] rusqlite::Error), + #[error( + "Could not back up wallet data. Check that the app data folder can be written to and try again." )] - IdentityRoster, + Io(#[source] std::io::Error), + #[error( + "Wallet data does not match a supported upgrade. Keep your data folder and reopen the previous application version." + )] + Unrecognized, + /// The saved roster is missing, ambiguous, or does not decode. #[error( "The saved identity list could not be read. Keep your data folder and reopen the previous application version." )] - IdentityRosterDecode(#[from] bincode::error::DecodeError), + IdentityRoster { + #[source] + source: Option, + }, #[error( "Wallet data verification failed. Keep your data folder and reopen the previous application version." )] @@ -41,6 +62,61 @@ pub enum UpgradeError { TypedValidation(#[source] Box), } +impl UpgradeError { + /// Whether retrying can recover from contention or temporary resource exhaustion. + pub fn is_retryable(&self) -> bool { + matches!( + self, + Self::InUse(_) + | Self::StorageFull(_) + | Self::StorageUnavailable(_) + | Self::OutOfMemory(_) + ) + } +} + +impl UpgradeError { + /// The variant for an environmental storage failure, keeping `error` as the source. + pub(super) fn from_failure( + failure: StorageFailure, + error: Box, + ) -> Self { + match failure { + StorageFailure::InUse => Self::InUse(error), + StorageFailure::Full => Self::StorageFull(error), + StorageFailure::Unavailable => Self::StorageUnavailable(error), + StorageFailure::OutOfMemory => Self::OutOfMemory(error), + StorageFailure::AccessDenied => Self::AccessDenied(error), + } + } +} + +impl From for UpgradeError { + fn from(error: rusqlite::Error) -> Self { + match StorageFailure::of_sqlite(&error) { + Some(failure) => Self::from_failure(failure, Box::new(error)), + None => Self::Sqlite(error), + } + } +} + +impl From for UpgradeError { + fn from(error: std::io::Error) -> Self { + match StorageFailure::of_io(&error) { + Some(failure) => Self::from_failure(failure, Box::new(error)), + None => Self::Io(error), + } + } +} + +impl From for UpgradeError { + fn from(source: bincode::error::DecodeError) -> Self { + Self::IdentityRoster { + source: Some(source), + } + } +} + #[derive(Debug, PartialEq, Eq)] struct Object { kind: String, @@ -111,19 +187,19 @@ fn active_identities(conn: &Connection) -> Result>, UpgradeErro .optional()?; let Some(value) = value else { if conn.prepare("SELECT 1 FROM identities i JOIN meta_identity m ON i.identity_id = m.identity_id WHERE i.tombstoned = 1 AND m.key = 'det:identity:v1'")?.exists([])? { - return Err(UpgradeError::IdentityRoster); + return Err(UpgradeError::IdentityRoster { source: None }); } return Ok(BTreeSet::new()); }; let Some((&1, body)) = value.split_first() else { - return Err(UpgradeError::IdentityRoster); + return Err(UpgradeError::IdentityRoster { source: None }); }; let (ids, consumed): (Vec<[u8; 32]>, usize) = bincode::serde::decode_from_slice( body, bincode::config::standard().with_limit::<16777216>(), )?; if consumed != body.len() { - return Err(UpgradeError::IdentityRoster); + return Err(UpgradeError::IdentityRoster { source: None }); } Ok(ids.into_iter().map(Vec::from).collect()) } @@ -163,14 +239,417 @@ fn selected_rows(table: &str) -> String { .unwrap_or_default() } +fn backup_prefix(path: &Path) -> Option { + Some(format!( + "{}.platform-67d4ef3-backup-", + path.file_name()?.to_string_lossy() + )) +} + +/// The published bridge snapshot of `database` named with `suffix`; a test fixture. +#[cfg(test)] +pub(crate) fn bridge_backup_path(database: &Path, suffix: &str) -> PathBuf { + let prefix = backup_prefix(database).expect("database path has a file name"); + database.with_file_name(format!("{prefix}{suffix}.sqlite")) +} + +/// Upstream snapshot names: `pre-migration---to--.db`. +const UPSTREAM_PREFIX: &str = "pre-migration-"; +const UPSTREAM_SUFFIX: &str = ".db"; +/// UTC timestamp format in an upstream snapshot name. +const UPSTREAM_TIMESTAMP_FORMAT: &str = "%Y%m%dT%H%M%SZ"; + +/// Whether `name` is a bridge snapshot of `database` (published `.sqlite` or +/// unpublished `.pending`), which lives next to the database. +pub(crate) fn is_bridge_backup_name(database: &Path, name: &str) -> bool { + let Some(prefix) = backup_prefix(database) else { + return false; + }; + name.strip_prefix(&prefix) + .and_then(|suffix| { + suffix + .strip_suffix(".sqlite") + .or_else(|| suffix.strip_suffix(".pending")) + }) + .is_some_and(|suffix| { + !suffix.is_empty() && suffix.bytes().all(|b| b.is_ascii_alphanumeric()) + }) +} + +/// The migration an upstream snapshot of `database` precedes, and when it was taken. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct UpstreamSnapshot { + from: u32, + to: u32, + taken: std::time::SystemTime, +} + +fn parse_upstream_name(database: &Path, name: &str) -> Option { + let stem = database.file_stem()?.to_str()?; + // DET database names are already valid upstream stems; reject lossy/ambiguous names. + if stem.is_empty() + || stem.len() > 32 + || !stem + .bytes() + .all(|b| b.is_ascii_alphanumeric() || b == b'-' || b == b'_') + { + return None; + } + let suffix = name + .strip_prefix(UPSTREAM_PREFIX)? + .strip_prefix(stem)? + .strip_prefix('-')? + .strip_suffix(UPSTREAM_SUFFIX)?; + let (versions, timestamp) = suffix.rsplit_once('-')?; + let (from, to) = versions.split_once("-to-")?; + let taken = chrono::NaiveDateTime::parse_from_str(timestamp, UPSTREAM_TIMESTAMP_FORMAT).ok()?; + Some(UpstreamSnapshot { + from: from.parse().ok()?, + to: to.parse().ok()?, + taken: taken.and_utc().into(), + }) +} + +/// When the upstream snapshot `name` of `database` was taken, or `None` when +/// `name` is not one. +pub(crate) fn upstream_backup_timestamp( + database: &Path, + name: &str, +) -> Option { + parse_upstream_name(database, name).map(|snapshot| snapshot.taken) +} + +fn default_auto_dir(path: &Path) -> PathBuf { + platform_wallet_storage::default_auto_backup_dir(path) +} + +/// Single-link regular files named like a backup of the database at `path`, next to it +/// or in `auto_dir`. Anything else is skipped; [`remove_backup`] refuses what must never go. +fn backups_in(path: &Path, auto_dir: Option<&Path>) -> std::io::Result> { + let mut found = Vec::new(); + let Some(parent) = path.parent() else { + return Ok(found); + }; + for directory in [Some(parent), auto_dir].into_iter().flatten() { + let metadata = match std::fs::symlink_metadata(directory) { + Ok(metadata) => metadata, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => continue, + Err(error) => return Err(error), + }; + if !metadata.is_dir() { + return Err(std::io::Error::other( + "Backup directory is not a regular directory", + )); + } + for entry in std::fs::read_dir(directory)? { + let entry = entry?; + let name = entry.file_name(); + let Some(name) = name.to_str() else { continue }; + let bridge = directory == parent && is_bridge_backup_name(path, name); + let upstream = Some(directory) == auto_dir && parse_upstream_name(path, name).is_some(); + // Symlinks, directories and hard links (which may alias a live database) + // are skipped; an entry whose metadata cannot be read surfaces on use. + let other_type = std::fs::symlink_metadata(entry.path()) + .is_ok_and(|metadata| !is_single_link_file(&metadata)); + if (bridge || upstream) && !other_type { + found.push(entry.path()); + } + } + } + found.sort(); + found.dedup(); + Ok(found) +} + +fn remove_backup(backup: &Path, database: &Path) -> std::io::Result<()> { + crate::utils::file_deletion::delete_file( + backup, + crate::utils::file_deletion::DeletionIntent::Backup { database }, + ) +} + +pub(super) struct BackupGuard { + path: PathBuf, + _file: Option, +} + +/// How long a contended lifecycle lock is retried before reporting `WouldBlock`. +/// +/// `flock` belongs to the open file description, and a child that any thread of this +/// process forks shares it until the child execs. A lock just released by its guard +/// can therefore still look held for that brief window. +const LOCK_GRACE: std::time::Duration = std::time::Duration::from_millis(500); +const LOCK_RETRY_INTERVAL: std::time::Duration = std::time::Duration::from_millis(5); + +fn lock_with_grace(file: &std::fs::File) -> std::io::Result<()> { + let deadline = std::time::Instant::now() + LOCK_GRACE; + loop { + match file.try_lock() { + Ok(()) => return Ok(()), + Err(std::fs::TryLockError::WouldBlock) if std::time::Instant::now() < deadline => { + std::thread::sleep(LOCK_RETRY_INTERVAL); + } + Err(error) => return Err(error.into()), + } + } +} + +/// Suffix appended to a database's file name to form its lifecycle lock file. +pub(crate) const LOCK_SUFFIX: &str = ".platform-upgrade.lock"; + +/// The lifecycle lock file of the database at `database`; `None` when the path has +/// no file name. +pub(crate) fn backup_lock_path(database: &Path) -> Option { + let mut name = database.file_name()?.to_os_string(); + name.push(LOCK_SUFFIX); + Some(database.with_file_name(name)) +} + +/// Whether `metadata` (not followed) is a regular file with no other hard link. +fn is_single_link_file(metadata: &std::fs::Metadata) -> bool { + #[cfg(unix)] + let single_link = std::os::unix::fs::MetadataExt::nlink(metadata) == 1; + #[cfg(not(unix))] + let single_link = true; + metadata.is_file() && single_link +} + +/// Refuse an existing lock path that is a symlink, not a regular file, or hard-linked. +fn check_lock_file(lock: &Path) -> std::io::Result<()> { + if is_single_link_file(&std::fs::symlink_metadata(lock)?) { + Ok(()) + } else { + Err(std::io::Error::other( + "Upgrade lock path is not a single-link regular file", + )) + } +} + +pub(super) fn backup_lock(path: &Path) -> std::io::Result { + let Some(lock_path) = backup_lock_path(path) else { + return Err(std::io::ErrorKind::InvalidInput.into()); + }; + let mut options = std::fs::File::options(); + options.read(true).write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + options.mode(0o600); + } + let file = match options.open(&lock_path) { + Ok(file) => file, + Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => { + check_lock_file(&lock_path)?; + std::fs::File::options() + .read(true) + .write(true) + .open(&lock_path)? + } + // A missing parent has no snapshots to clean up. + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + return Ok(BackupGuard { + path: path.to_owned(), + _file: None, + }); + } + Err(error) => return Err(error), + }; + lock_with_grace(&file)?; + // Keep the pathname stable: unlinking it could let contenders lock different files. + Ok(BackupGuard { + path: path.to_owned(), + _file: Some(file), + }) +} + +/// [`tidy_backups_locked`] under a freshly taken lifecycle lock. +#[cfg(test)] +pub(super) fn tidy_backups(path: &Path, auto_dir: Option<&Path>) -> std::io::Result<()> { + let guard = backup_lock(path)?; + tidy_backups_locked(&guard, auto_dir) +} + +/// Delete crash-left `.pending` copies and published snapshots byte-identical to a newer +/// one of the same migration (each failed open writes another); the rest is left to retention. +pub(super) fn tidy_backups_locked( + guard: &BackupGuard, + auto_dir: Option<&Path>, +) -> std::io::Result<()> { + let path = &guard.path; + // Group published snapshots by the migration they precede; newest first. + let mut groups: std::collections::BTreeMap, Vec<_>> = + std::collections::BTreeMap::new(); + for backup in backups_in(path, auto_dir)? { + let name = backup + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or_default(); + if backup + .extension() + .is_some_and(|extension| extension == "pending") + { + remove_backup(&backup, path)?; + continue; + } + // Bridge snapshots all precede the same pinned-profile upgrade. + let migration = + parse_upstream_name(path, name).map(|snapshot| (snapshot.from, snapshot.to)); + let created = backup_created(path, &backup)?; + groups.entry(migration).or_default().push((created, backup)); + } + for mut group in groups.into_values() { + group.sort_by(|a, b| b.cmp(a)); + let mut kept: Vec = Vec::with_capacity(group.len()); + for (_, backup) in group { + let mut duplicate = false; + for newer in &kept { + if same_contents(newer, &backup)? { + duplicate = true; + break; + } + } + if duplicate { + remove_backup(&backup, path)?; + } else { + kept.push(backup); + } + } + } + Ok(()) +} + +/// Whether two files hold the same bytes. Sizes are compared first, then the +/// contents are streamed in fixed-size chunks, stopping at the first difference. +fn same_contents(a: &Path, b: &Path) -> std::io::Result { + use std::io::Read; + const CHUNK: usize = 64 * 1024; + if std::fs::symlink_metadata(a)?.len() != std::fs::symlink_metadata(b)?.len() { + return Ok(false); + } + let (mut a, mut b) = (std::fs::File::open(a)?, std::fs::File::open(b)?); + let (mut left, mut right) = (vec![0u8; CHUNK], vec![0u8; CHUNK]); + loop { + let read = a.read(&mut left)?; + if read == 0 { + // Equal lengths: `b` must be exhausted too, unless it grew meanwhile. + return Ok(b.read(&mut right[..1])? == 0); + } + b.read_exact(&mut right[..read])?; + if left[..read] != right[..read] { + return Ok(false); + } + } +} + +/// Delete every upgrade backup of the database at `path` and sync the directories +/// deleted from; attempts all and returns the first failure. +pub(crate) fn remove_backups(path: &Path) -> std::io::Result<()> { + let _guard = backup_lock(path)?; + let mut first_error = None; + let mut touched = BTreeSet::new(); + for backup in backups_in(path, Some(&default_auto_dir(path)))? { + match remove_backup(&backup, path) { + Ok(()) => touched.extend(backup.parent().map(Path::to_owned)), + Err(error) => { + first_error.get_or_insert(error); + } + } + } + for directory in touched { + if let Err(error) = crate::utils::backup_prune::sync_directory(&directory) { + first_error.get_or_insert(error); + } + } + first_error.map_or(Ok(()), Err) +} + +/// [`prune_expired`](crate::utils::backup_prune::prune_expired) over the bridge and +/// upstream backups of the database at `path`. Skips a database without backups or +/// whose lifecycle lock is held (the next pass covers it); returns the number deleted. +pub(crate) fn prune_expired_backups( + path: &Path, + max_age: std::time::Duration, + now: std::time::SystemTime, +) -> std::io::Result { + let auto_dir = default_auto_dir(path); + if backups_in(path, Some(&auto_dir))?.is_empty() { + return Ok(0); + } + let _guard = match backup_lock(path) { + Ok(guard) => guard, + Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => { + tracing::debug!(database = %path.display(), "Upgrade backup retention skipped a database in use"); + return Ok(0); + } + Err(error) => return Err(error), + }; + let candidates = backups_in(path, Some(&auto_dir))? + .into_iter() + .filter(|backup| { + backup + .extension() + .is_none_or(|extension| extension != "pending") + }) + .map(|backup| { + let created = backup_created(path, &backup); + // A later migration is a later snapshot, whatever the clock said. + let sequence = backup + .file_name() + .and_then(|name| name.to_str()) + .and_then(|name| parse_upstream_name(path, name)) + .map(|snapshot| (snapshot.to, snapshot.from)); + crate::utils::backup_prune::Candidate { + path: backup, + created, + sequence, + } + }) + .collect(); + crate::utils::backup_prune::prune_expired( + candidates, + now, + max_age, + crate::utils::file_deletion::DeletionIntent::Backup { database: path }, + ) +} + +fn backup_created(database: &Path, backup: &Path) -> std::io::Result { + let named = backup + .file_name() + .and_then(|name| name.to_str()) + .and_then(|name| upstream_backup_timestamp(database, name)); + crate::utils::backup_prune::created_at(backup, named) +} + +#[cfg(test)] fn backup(path: &Path) -> Result { + backup_with_hook(path, |_| Ok(())) +} + +#[cfg(test)] +fn backup_with_hook( + path: &Path, + pending_created: impl FnOnce(&Path) -> Result<(), UpgradeError>, +) -> Result { + let guard = backup_lock(path)?; + backup_with_hook_locked(&guard, pending_created) +} + +fn backup_with_hook_locked( + guard: &BackupGuard, + pending_created: impl FnOnce(&Path) -> Result<(), UpgradeError>, +) -> Result { + let path = &guard.path; let parent = path.parent().ok_or(UpgradeError::Unrecognized)?; - let filename = path.file_name().ok_or(UpgradeError::Unrecognized)?; - let prefix = format!("{}.platform-67d4ef3-backup-", filename.to_string_lossy()); + let prefix = backup_prefix(path).ok_or(UpgradeError::Unrecognized)?; + // Not routed through `delete_file`: on failure the temp file removes only + // the uniquely named `.pending` file it created itself, which keeps the + // cleanup tied to drop on every error path. let file = tempfile::Builder::new() .prefix(&prefix) - .suffix(".sqlite") + .suffix(".pending") .tempfile_in(parent)?; + pending_created(file.path())?; let source = Connection::open_with_flags( path, OpenFlags::SQLITE_OPEN_READ_ONLY | OpenFlags::SQLITE_OPEN_NOFOLLOW, @@ -185,10 +664,11 @@ fn backup(path: &Path) -> Result { verify(&dest)?; drop(dest); file.as_file().sync_all()?; - let (_, path) = file.keep().map_err(|e| e.error)?; - #[cfg(unix)] - std::fs::File::open(parent)?.sync_all()?; - Ok(path) + // Earlier snapshots stay: only time-based retention and open-time tidying remove them. + let kept = file.path().with_extension("sqlite"); + file.persist_noclobber(&kept).map_err(|e| e.error)?; + crate::utils::backup_prune::sync_directory(parent)?; + Ok(kept) } fn copy_rows( @@ -278,20 +758,42 @@ fn allowed_columns( } /// Translate only the exact pinned PR schema, preserving the original in a durable backup. +#[cfg(test)] pub(super) fn upgrade( path: &Path, target_path: &Path, validate: impl FnOnce(&Path) -> Result<(), UpgradeError>, ) -> Result, UpgradeError> { - upgrade_with_hook(path, target_path, validate, || Ok(())) + let guard = backup_lock(path)?; + upgrade_locked(&guard, target_path, validate) +} + +pub(super) fn upgrade_locked( + guard: &BackupGuard, + target_path: &Path, + validate: impl FnOnce(&Path) -> Result<(), UpgradeError>, +) -> Result, UpgradeError> { + upgrade_with_hook_locked(guard, target_path, validate, || Ok(())) } +#[cfg(test)] fn upgrade_with_hook( path: &Path, target_path: &Path, validate: impl FnOnce(&Path) -> Result<(), UpgradeError>, before_commit: impl FnOnce() -> Result<(), UpgradeError>, ) -> Result, UpgradeError> { + let guard = backup_lock(path)?; + upgrade_with_hook_locked(&guard, target_path, validate, before_commit) +} + +fn upgrade_with_hook_locked( + guard: &BackupGuard, + target_path: &Path, + validate: impl FnOnce(&Path) -> Result<(), UpgradeError>, + before_commit: impl FnOnce() -> Result<(), UpgradeError>, +) -> Result, UpgradeError> { + let path = &guard.path; let old = old_reference()?; let reference = target_reference()?; let mut source = Connection::open_with_flags( @@ -316,7 +818,6 @@ fn upgrade_with_hook( } verify(&source)?; let retired = retired_identities(&source)?; - let backup = backup(path)?; let mut target = Connection::open_with_flags( target_path, OpenFlags::SQLITE_OPEN_READ_WRITE | OpenFlags::SQLITE_OPEN_NOFOLLOW, @@ -396,6 +897,9 @@ fn upgrade_with_hook( target_tx.commit()?; drop(target); validate(target_path)?; + // The source transaction has written only temp tables, so a separate reader still + // sees the committed original; earlier failures roll back and need no backup. + let backup = backup_with_hook_locked(guard, |_| Ok(()))?; let target = Connection::open_with_flags( target_path, OpenFlags::SQLITE_OPEN_READ_ONLY | OpenFlags::SQLITE_OPEN_NOFOLLOW, diff --git a/src/wallet_backend/platform_compatibility/engine/tests.rs b/src/wallet_backend/platform_compatibility/engine/tests.rs index fb99701a6..a6132f4b1 100644 --- a/src/wallet_backend/platform_compatibility/engine/tests.rs +++ b/src/wallet_backend/platform_compatibility/engine/tests.rs @@ -216,7 +216,298 @@ fn platform_compatibility_typed_validation_failure_never_rewrites_original() { .unwrap_err(); assert!(matches!(error, UpgradeError::TypedValidation(_))); assert_eq!(snapshot(&path), before); - assert_eq!(snapshot(&backup_files(dir.path())[0]), before); + assert!( + backup_files(dir.path()).is_empty(), + "A failure before the rebuild leaves the original untouched, so no backup may remain." + ); +} + +#[test] +fn platform_compatibility_repeated_failed_rebuilds_are_deduplicated() { + let (dir, path, _target) = fixture(); + let before = snapshot(&path); + for attempt in 0..3 { + let stage = dir.path().join(format!("stage-{attempt}.sqlite")); + Connection::open(&stage) + .unwrap() + .execute_batch(TARGET_SCHEMA) + .unwrap(); + let error = upgrade_with_hook( + &path, + &stage, + |_| Ok(()), + || Err(UpgradeError::Verification), + ) + .unwrap_err(); + assert!(matches!(error, UpgradeError::Verification)); + } + assert_eq!( + backup_files(dir.path()).len(), + 3, + "publication never deletes an earlier snapshot" + ); + tidy_backups(&path, None).unwrap(); + let backups = backup_files(dir.path()); + assert_eq!( + backups.len(), + 1, + "identical copies of the unchanged original collapse to one" + ); + assert_eq!(snapshot(&backups[0]), before); +} + +#[test] +fn platform_compatibility_backup_removal_only_touches_the_named_database() { + let dir = tempfile::tempdir().unwrap(); + let names = [ + "det-app.sqlite.platform-67d4ef3-backup-a1.sqlite", + "det-testnet.sqlite.platform-67d4ef3-backup-b2.sqlite", + "det-mainnet.sqlite.platform-67d4ef3-backup-c3.sqlite", + "det-testnet.sqlite", + "det-testnet-shielded.sqlite", + ]; + for name in names { + std::fs::write(dir.path().join(name), b"x").unwrap(); + } + remove_backups(&dir.path().join("det-testnet.sqlite")).unwrap(); + let mut left = std::fs::read_dir(dir.path()) + .unwrap() + .map(|e| e.unwrap().file_name().to_string_lossy().into_owned()) + .collect::>(); + left.sort(); + assert_eq!( + left, + [ + "det-app.sqlite.platform-67d4ef3-backup-a1.sqlite", + "det-mainnet.sqlite.platform-67d4ef3-backup-c3.sqlite", + "det-testnet-shielded.sqlite", + "det-testnet.sqlite", + format!("det-testnet.sqlite{LOCK_SUFFIX}").as_str(), + ] + ); + remove_backups(&dir.path().join("absent.sqlite")).unwrap(); +} + +#[test] +fn platform_compatibility_removes_only_matching_upstream_backups() { + let dir = tempfile::tempdir().unwrap(); + let auto = dir.path().join("backups/auto"); + std::fs::create_dir_all(&auto).unwrap(); + let removed = auto.join("pre-migration-det-testnet-1-to-2-20260915T120000Z.db"); + let kept = auto.join("pre-migration-det-testnet-other-1-to-2-20260915T120000Z.db"); + for path in [&removed, &kept] { + std::fs::write(path, b"backup").unwrap(); + } + remove_backups(&dir.path().join("det-testnet.sqlite")).unwrap(); + assert!(!removed.exists()); + assert!(kept.exists()); +} + +/// The previous recovery snapshot must survive until its replacement is +/// published: a failed publication must not leave zero snapshots behind. +#[test] +fn platform_compatibility_failed_publication_keeps_previous_snapshot() { + let (_dir, path, _target) = fixture(); + let previous = backup(&path).unwrap(); + let error = backup_with_hook(&path, |pending| { + // Occupy the name the replacement would be published under. + std::fs::write(pending.with_extension("sqlite"), b"collision")?; + Ok(()) + }) + .unwrap_err(); + assert!( + matches!(&error, UpgradeError::Io(e) if e.kind() == std::io::ErrorKind::AlreadyExists), + "{error:?}" + ); + assert!( + previous.exists(), + "the published snapshot must survive a failed replacement" + ); + assert_eq!(snapshot(&previous), snapshot(&path)); +} + +/// Publication keeps earlier snapshots: only retention and tidying remove them. +#[test] +fn platform_compatibility_publication_keeps_earlier_snapshots() { + let (dir, path, _target) = fixture(); + let previous = backup(&path).unwrap(); + let replacement = backup(&path).unwrap(); + assert_ne!(previous, replacement); + assert!(previous.exists()); + assert!(replacement.exists()); + assert_eq!(backup_files(dir.path()).len(), 2); +} + +/// Tidying removes a snapshot only when a newer one of the same migration has the +/// same bytes; a snapshot that differs by even one byte, or precedes another +/// migration, is never removed. +#[test] +fn platform_compatibility_tidy_removes_only_identical_copies() { + let (dir, path, _target) = fixture(); + let auto = dir.path().join("backups/auto"); + std::fs::create_dir_all(&auto).unwrap(); + let original = backup(&path).unwrap(); + let copy = backup(&path).unwrap(); + set_mtime(&original, -3600); + let bytes = std::fs::read(©).unwrap(); + let mut differing = bytes.clone(); + *differing.last_mut().unwrap() ^= 0xFF; + let upstream = |versions: &str, second: u32| { + auto.join(format!( + "pre-migration-wallet-{versions}-20260915T12000{second}Z.db" + )) + }; + let (same_a, same_b, other_migration, changed) = ( + upstream("1-to-2", 1), + upstream("1-to-2", 2), + upstream("2-to-3", 3), + upstream("1-to-2", 4), + ); + for (file, contents) in [ + (&same_a, &bytes), + (&same_b, &bytes), + (&other_migration, &bytes), + (&changed, &differing), + ] { + std::fs::write(file, contents).unwrap(); + } + set_mtime(&same_a, -3600); + + tidy_backups(&path, Some(&auto)).unwrap(); + + assert!(!original.exists(), "the older identical bridge copy goes"); + assert!(copy.exists(), "the newest identical copy stays"); + assert!(!same_a.exists(), "the older identical upstream copy goes"); + assert!(same_b.exists()); + assert!( + other_migration.exists(), + "another migration is never merged" + ); + assert!(changed.exists(), "a differing snapshot is never removed"); + tidy_backups(&path, Some(&auto)).unwrap(); + assert_eq!(backup_files(dir.path()), vec![copy]); +} + +/// Removal deletes every backup and skips entries named like one that are directories, +/// symlinks or hard links (here to the live database), which are never followed. +#[cfg(unix)] +#[test] +fn platform_compatibility_removal_skips_links_and_directories() { + let (dir, path, _target) = fixture(); + let bridge = |suffix: &str| { + dir.path() + .join(format!("wallet.sqlite.platform-67d4ef3-backup-{suffix}")) + }; + let skipped = [ + bridge("link.sqlite"), + bridge("dir.pending"), + bridge("symlink.sqlite"), + ]; + std::fs::hard_link(&path, &skipped[0]).unwrap(); + std::fs::create_dir(&skipped[1]).unwrap(); + std::os::unix::fs::symlink(&path, &skipped[2]).unwrap(); + let valid = [bridge("b1.sqlite"), bridge("c2.pending")]; + for backup in &valid { + std::fs::write(backup, b"backup").unwrap(); + } + let auto = dir.path().join("backups/auto"); + std::fs::create_dir_all(&auto).unwrap(); + let upstream = auto.join("pre-migration-wallet-1-to-2-20260915T120000Z.db"); + std::fs::write(&upstream, b"backup").unwrap(); + + remove_backups(&path).unwrap(); + for backup in valid.iter().chain([&upstream]) { + assert!(!backup.exists(), "{} must be removed", backup.display()); + } + assert!(path.exists()); + for entry in &skipped { + assert!(entry.symlink_metadata().is_ok(), "{}", entry.display()); + } +} + +/// An existing lifecycle lock path that is a symlink is refused, never followed. +#[cfg(unix)] +#[test] +fn platform_compatibility_lock_refuses_a_symlinked_lock_file() { + let (dir, path, _target) = fixture(); + let elsewhere = dir.path().join("elsewhere"); + std::fs::write(&elsewhere, b"keep").unwrap(); + std::os::unix::fs::symlink(&elsewhere, backup_lock_path(&path).unwrap()).unwrap(); + assert!(backup_lock(&path).is_err()); + assert_eq!(std::fs::read(&elsewhere).unwrap(), b"keep"); +} + +fn sqlite_failure(code: std::ffi::c_int) -> rusqlite::Error { + rusqlite::Error::SqliteFailure(rusqlite::ffi::Error::new(code), None) +} + +#[test] +fn platform_compatibility_errors_name_their_actual_cause() { + let in_use = [ + UpgradeError::from(sqlite_failure(rusqlite::ffi::SQLITE_BUSY)), + UpgradeError::from(sqlite_failure(rusqlite::ffi::SQLITE_LOCKED)), + ]; + for error in &in_use { + assert!(matches!(error, UpgradeError::InUse(_)), "{error:?}"); + assert!(error.is_retryable()); + let text = error.to_string(); + assert!(text.contains("another Dash Evo Tool"), "{text}"); + assert!(!text.contains("disk"), "{text}"); + } + let full = [ + UpgradeError::from(sqlite_failure(rusqlite::ffi::SQLITE_FULL)), + UpgradeError::from(std::io::Error::from(std::io::ErrorKind::StorageFull)), + ]; + for error in &full { + assert!(matches!(error, UpgradeError::StorageFull(_)), "{error:?}"); + assert!(error.is_retryable()); + assert!(error.to_string().contains("disk")); + } + let io = UpgradeError::from(std::io::Error::from(std::io::ErrorKind::PermissionDenied)); + assert!(matches!(io, UpgradeError::AccessDenied(_)) && !io.is_retryable()); + assert!(!io.to_string().contains("disk"), "{io}"); + + let generic = UpgradeError::from(sqlite_failure(rusqlite::ffi::SQLITE_CORRUPT)); + assert!(matches!(generic, UpgradeError::Sqlite(_))); + assert!(!generic.is_retryable()); + assert!(!generic.to_string().contains("disk"), "{generic}"); + for error in [ + std::error::Error::source(&in_use[0]), + std::error::Error::source(&full[1]), + std::error::Error::source(&generic), + ] { + assert!( + error.is_some(), + "The technical cause must stay in the source chain." + ); + } + + for error in [ + UpgradeError::Unrecognized, + UpgradeError::IdentityRoster { source: None }, + UpgradeError::Verification, + UpgradeError::TypedValidation(Box::new(std::io::Error::other("fixture"))), + ] { + assert!(!error.is_retryable(), "{error:?}"); + } +} + +#[test] +fn platform_compatibility_malformed_roster_reports_one_message() { + let (_dir, path, target) = fixture(); + Connection::open(&path) + .unwrap() + .execute_batch("INSERT INTO meta_global VALUES ('det:identity_index:v1', X'01ff', 0)") + .unwrap(); + let decode = upgrade(&path, &target, |_| Ok(())).unwrap_err(); + assert!( + matches!(decode, UpgradeError::IdentityRoster { source: Some(_) }), + "{decode:?}" + ); + assert_eq!( + decode.to_string(), + UpgradeError::IdentityRoster { source: None }.to_string() + ); } #[test] @@ -266,6 +557,10 @@ fn platform_compatibility_wal_snapshot_includes_committed_uncheckpointed_data() live.execute_batch("PRAGMA journal_mode=WAL; PRAGMA wal_autocheckpoint=0; INSERT INTO meta_global VALUES ('wal-fixture', X'00ff', 42)").unwrap(); let before = snapshot(&path); let backup = upgrade(&path, &target, |_| Ok(())).unwrap().unwrap(); + assert!( + crate::utils::backup_prune::usable_snapshot(&backup), + "a snapshot of a WAL database must count as usable" + ); assert_eq!(snapshot(&backup), before); let conn = Connection::open(&path).unwrap(); assert_eq!( @@ -286,7 +581,7 @@ fn platform_compatibility_missing_roster_with_identity_sidecar_is_ambiguous() { let before = snapshot(&path); assert!(matches!( upgrade(&path, &target, |_| Ok(())), - Err(UpgradeError::IdentityRoster) + Err(UpgradeError::IdentityRoster { source: None }) )); assert_eq!(snapshot(&path), before); assert!(backup_files(dir.path()).is_empty()); @@ -330,3 +625,433 @@ fn platform_compatibility_writer_exclusion_covers_staged_validation() { Ok(()) }).unwrap(); } + +#[test] +fn platform_compatibility_pending_cleanup_preserves_published_snapshot() { + let (dir, path, _target) = fixture(); + let published = backup(&path).unwrap(); + let pending = dir + .path() + .join("wallet.sqlite.platform-67d4ef3-backup-crash.pending"); + std::fs::copy(&path, &pending).unwrap(); + tidy_backups(&path, None).unwrap(); + assert!(!pending.exists()); + assert!(published.exists()); + assert!(path.exists()); +} + +#[test] +fn platform_compatibility_deletion_removes_pending_snapshots() { + let (dir, path, _target) = fixture(); + let pending = dir + .path() + .join("wallet.sqlite.platform-67d4ef3-backup-crash.pending"); + let unrelated = dir + .path() + .join("other.sqlite.platform-67d4ef3-backup-crash.pending"); + std::fs::copy(&path, &pending).unwrap(); + std::fs::copy(&path, &unrelated).unwrap(); + remove_backups(&path).unwrap(); + assert!(!pending.exists()); + assert!(unrelated.exists()); + assert!(path.exists()); +} + +#[test] +fn platform_compatibility_active_snapshot_survives_concurrent_cleanup() { + use std::io::{BufRead, Read, Write}; + const CHILD_DIR: &str = "DET_PLATFORM_COMPAT_ACTIVE_FIXTURE_DIR"; + if let Some(dir) = std::env::var_os(CHILD_DIR) { + let path = PathBuf::from(dir).join("wallet.sqlite"); + backup_with_hook(&path, |pending| { + // Pause the real writer after creation, before it copies the database. + println!("PENDING:{}", pending.display()); + std::io::stdout().flush()?; + std::io::stdin().read_exact(&mut [0])?; + Ok(()) + }) + .unwrap(); + return; + } + for crash in [false, true] { + let (dir, path, _target) = fixture(); + let test_name = std::thread::current().name().unwrap().to_owned(); + let mut child = std::process::Command::new(std::env::current_exe().unwrap()) + .args(["--exact", &test_name, "--nocapture"]) + .env(CHILD_DIR, dir.path()) + .stdin(std::process::Stdio::piped()) + .stdout(std::process::Stdio::piped()) + .spawn() + .unwrap(); + let mut reader = std::io::BufReader::new(child.stdout.take().unwrap()); + let pending = loop { + let mut line = String::new(); + assert_ne!( + reader.read_line(&mut line).unwrap(), + 0, + "Writer exited before creating a snapshot" + ); + if let Some((_, path)) = line.trim().split_once("PENDING:") { + break PathBuf::from(path); + } + }; + let retained = tidy_backups(&path, None); + let removed = remove_backups(&path); + let exists = pending.exists(); + if crash { + child.kill().unwrap(); + } else { + child.stdin.take().unwrap().write_all(&[1]).unwrap(); + } + let status = child.wait().unwrap(); + assert!(exists, "Cleanup deleted another process's active snapshot"); + assert_eq!(retained.unwrap_err().kind(), std::io::ErrorKind::WouldBlock); + assert_eq!(removed.unwrap_err().kind(), std::io::ErrorKind::WouldBlock); + if crash { + assert!(!status.success()); + assert!(pending.exists()); + tidy_backups(&path, None).unwrap(); + assert!(!pending.exists()); + continue; + } + assert!(status.success()); + let published = pending.with_extension("sqlite"); + assert_eq!(snapshot(&published), snapshot(&path)); + tidy_backups(&path, None).unwrap(); + assert!(published.exists()); + remove_backups(&path).unwrap(); + assert!(!published.exists()); + } +} + +#[test] +fn platform_compatibility_one_guard_covers_retention_and_upgrade() { + let (_dir, path, target) = fixture(); + let before = snapshot(&path); + let guard = backup_lock(&path).unwrap(); + tidy_backups_locked(&guard, None).unwrap(); + let backup = upgrade_locked(&guard, &target, |_| Ok(())) + .unwrap() + .unwrap(); + tidy_backups_locked(&guard, None).unwrap(); + assert_eq!(snapshot(&backup), before); +} + +fn set_mtime(path: &Path, seconds_from_now: i64) { + let now = std::time::SystemTime::now(); + let offset = std::time::Duration::from_secs(seconds_from_now.unsigned_abs()); + let time = if seconds_from_now < 0 { + now - offset + } else { + now + offset + }; + std::fs::File::options() + .write(true) + .open(path) + .unwrap() + .set_modified(time) + .unwrap(); +} + +/// An interrupted copy left under a final snapshot name (empty or truncated) differs +/// from the complete snapshot, so tidying keeps both, in either format. +#[test] +fn platform_compatibility_tidy_keeps_differing_snapshots() { + let (dir, path, _target) = fixture(); + let before = snapshot(&path); + let valid = backup(&path).unwrap(); + let full = std::fs::read(&valid).unwrap(); + let auto = dir.path().join("backups/auto"); + std::fs::create_dir_all(&auto).unwrap(); + let bridge = dir + .path() + .join("wallet.sqlite.platform-67d4ef3-backup-killed.sqlite"); + let upstream = auto.join("pre-migration-wallet-1-to-2-20260915T120000Z.db"); + let cases: [(&str, &[u8], &PathBuf); 4] = [ + ("empty bridge", b"", &bridge), + ("truncated bridge", &full[..full.len() / 2], &bridge), + ("header-only bridge", &full[..100], &bridge), + ("empty upstream", b"", &upstream), + ]; + set_mtime(&valid, -3600); + for (case, bytes, incomplete) in cases { + std::fs::write(incomplete, bytes).unwrap(); + tidy_backups(&path, Some(&auto)).unwrap(); + assert_eq!(snapshot(&valid), before, "{case}"); + assert!(incomplete.exists(), "{case}: a differing copy is kept"); + } +} + +const RETENTION_DAY: std::time::Duration = std::time::Duration::from_secs(24 * 60 * 60); + +/// Expiry covers both backup formats, measures age from the later of the file's +/// modification time and its name timestamp, and never touches the database itself. +#[test] +fn platform_compatibility_prune_expired_backups_by_age() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("det-testnet.sqlite"); + std::fs::write(&path, b"live database").unwrap(); + let auto = dir.path().join("backups/auto"); + std::fs::create_dir_all(&auto).unwrap(); + let old_bridge = dir + .path() + .join("det-testnet.sqlite.platform-67d4ef3-backup-old.sqlite"); + let fresh_bridge = dir + .path() + .join("det-testnet.sqlite.platform-67d4ef3-backup-fresh.sqlite"); + let old_upstream = auto.join("pre-migration-det-testnet-1-to-2-20200101T000000Z.db"); + // An old name whose file was rewritten recently (e.g. restored from a copy). + let old_name_fresh_file = auto.join("pre-migration-det-testnet-2-to-3-20200102T000000Z.db"); + // A recent name whose modification time was reset to long ago. + let fresh_name_old_file = auto.join("pre-migration-det-testnet-3-to-4-20991231T000000Z.db"); + let unrelated = auto.join("pre-migration-det-mainnet-1-to-2-20200101T000000Z.db"); + for file in [ + &old_bridge, + &fresh_bridge, + &old_upstream, + &old_name_fresh_file, + &fresh_name_old_file, + &unrelated, + ] { + std::fs::write(file, b"backup").unwrap(); + } + let hundred_days = -100 * 24 * 60 * 60; + for file in [&old_bridge, &old_upstream, &fresh_name_old_file, &unrelated] { + set_mtime(file, hundred_days); + } + set_mtime(&path, hundred_days); + + let removed = + prune_expired_backups(&path, RETENTION_DAY * 90, std::time::SystemTime::now()).unwrap(); + + assert_eq!(removed, 2); + assert!(!old_bridge.exists()); + assert!(!old_upstream.exists()); + assert!(fresh_bridge.exists()); + assert!( + old_name_fresh_file.exists(), + "a recent modification time keeps an old-named snapshot" + ); + assert!( + fresh_name_old_file.exists(), + "a recent name timestamp keeps a snapshot with an old modification time" + ); + assert!(unrelated.exists(), "another database's snapshot is kept"); + assert!(path.exists(), "the live database is never a candidate"); +} + +/// An unreadable newest snapshot is kept and reported, and never hands the floor to an +/// older copy that would then be the only one left. +#[cfg(unix)] +#[test] +fn platform_compatibility_prune_keeps_unreadable_snapshots() { + use std::os::unix::fs::PermissionsExt; + let (_dir, path, _target) = fixture(); + let older = backup(&path).unwrap(); + let newest = backup(&path).unwrap(); + set_mtime(&older, -200 * 24 * 60 * 60); + set_mtime(&newest, -100 * 24 * 60 * 60); + std::fs::set_permissions(&newest, std::fs::Permissions::from_mode(0o000)).unwrap(); + if std::fs::File::open(&newest).is_ok() { + // Running as root: permissions cannot make it unreadable. + return; + } + + let result = prune_expired_backups(&path, RETENTION_DAY * 90, std::time::SystemTime::now()); + + std::fs::set_permissions(&newest, std::fs::Permissions::from_mode(0o600)).unwrap(); + result.unwrap_err(); + assert!(newest.exists(), "an unjudged snapshot is kept"); + assert!(older.exists(), "the newest proven snapshot is kept"); +} + +/// A snapshot written while the clock ran ahead is dated in the future; it must not +/// displace the genuinely newer snapshot taken after the clock was corrected. +#[test] +fn platform_compatibility_prune_floor_resists_forward_clock_skew() { + let (_dir, path, _target) = fixture(); + let skewed = backup(&path).unwrap(); + let current = backup(&path).unwrap(); + set_mtime(&skewed, 30 * 24 * 60 * 60); + set_mtime(¤t, -100 * 24 * 60 * 60); + + assert_eq!( + prune_expired_backups(&path, RETENTION_DAY * 90, std::time::SystemTime::now()).unwrap(), + 0 + ); + assert!( + current.exists(), + "the newest snapshot dated in the past is kept" + ); + assert!(skewed.exists(), "a future-dated snapshot is never expired"); +} + +/// Upstream snapshots name the migration they precede, a clock-independent order: +/// the snapshot of the latest migration survives even when an older migration's +/// snapshot carries later dates. +#[test] +fn platform_compatibility_prune_floor_keeps_latest_migration_snapshot() { + let (dir, path, _target) = fixture(); + let valid = backup(&path).unwrap(); + let auto = dir.path().join("backups/auto"); + std::fs::create_dir_all(&auto).unwrap(); + let latest_migration = auto.join("pre-migration-wallet-2-to-3-20200101T000000Z.db"); + let earlier_migration = auto.join("pre-migration-wallet-1-to-2-20200601T000000Z.db"); + std::fs::copy(&valid, &latest_migration).unwrap(); + std::fs::copy(&valid, &earlier_migration).unwrap(); + set_mtime(&latest_migration, -400 * 24 * 60 * 60); + set_mtime(&earlier_migration, -300 * 24 * 60 * 60); + set_mtime(&valid, -500 * 24 * 60 * 60); + + assert_eq!( + prune_expired_backups(&path, RETENTION_DAY * 90, std::time::SystemTime::now()).unwrap(), + 1 + ); + assert!(!valid.exists()); + assert!(latest_migration.exists(), "newest by migration order"); + assert!(earlier_migration.exists(), "newest by creation time"); +} + +/// A backup-named hard link (here to the live database) is never a candidate, so it +/// cannot become the floor and let the real newest backup expire. +#[cfg(unix)] +#[test] +fn platform_compatibility_prune_skips_hard_linked_backup_names() { + let (_dir, path, _target) = fixture(); + let real = backup(&path).unwrap(); + set_mtime(&real, -200 * 24 * 60 * 60); + let linked = bridge_backup_path(&path, "linked"); + std::fs::hard_link(&path, &linked).unwrap(); + + assert_eq!( + prune_expired_backups(&path, RETENTION_DAY * 90, std::time::SystemTime::now()).unwrap(), + 0 + ); + assert!(real.exists(), "the only real backup is the floor"); + assert!(linked.exists() && path.exists()); +} + +/// A held lifecycle lock means an open or upgrade is running: pruning skips the +/// database without failing, and a later pass covers it. +#[test] +fn platform_compatibility_prune_backs_off_while_lock_is_held() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("det-testnet.sqlite"); + let expired = dir + .path() + .join("det-testnet.sqlite.platform-67d4ef3-backup-old.sqlite"); + std::fs::write(&expired, b"backup").unwrap(); + set_mtime(&expired, -10 * 24 * 60 * 60); + std::fs::write( + dir.path() + .join("det-testnet.sqlite.platform-67d4ef3-backup-new.sqlite"), + b"newer backup", + ) + .unwrap(); + let guard = backup_lock(&path).unwrap(); + assert_eq!( + prune_expired_backups(&path, RETENTION_DAY, std::time::SystemTime::now()).unwrap(), + 0 + ); + assert!(expired.exists()); + drop(guard); + assert_eq!( + prune_expired_backups(&path, RETENTION_DAY, std::time::SystemTime::now()).unwrap(), + 1 + ); +} + +/// A database with no backups is skipped before its lock file would be created. +#[test] +fn platform_compatibility_prune_skips_database_without_backups() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("det-devnet.sqlite"); + assert_eq!( + prune_expired_backups(&path, RETENTION_DAY, std::time::SystemTime::now()).unwrap(), + 0 + ); + assert_eq!(std::fs::read_dir(dir.path()).unwrap().count(), 0); +} + +/// Only a well-formed UTC timestamp counts as a creation time. A malformed name is +/// not an upstream snapshot at all, so pruning never touches it however old it is. +#[test] +fn platform_compatibility_prune_rejects_malformed_upstream_timestamps() { + assert_eq!( + upstream_backup_timestamp( + Path::new("det-testnet.sqlite"), + "pre-migration-det-testnet-1-to-2-20200101T000000Z.db" + ), + Some( + chrono::NaiveDate::from_ymd_opt(2020, 1, 1) + .unwrap() + .and_hms_opt(0, 0, 0) + .unwrap() + .and_utc() + .into() + ) + ); + let malformed = [ + "pre-migration-det-testnet-1-to-2-20201301T000000Z.db", + "pre-migration-det-testnet-1-to-2-20200101T000000.db", + "pre-migration-det-testnet-1-to-2-2020-01-01.db", + "pre-migration-det-testnet-1-to-2-.db", + ]; + for name in malformed { + assert_eq!( + upstream_backup_timestamp(Path::new("det-testnet.sqlite"), name), + None, + "{name}" + ); + } + + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("det-testnet.sqlite"); + let auto = dir.path().join("backups/auto"); + std::fs::create_dir_all(&auto).unwrap(); + for name in malformed { + let file = auto.join(name); + std::fs::write(&file, b"backup").unwrap(); + set_mtime(&file, -400 * 24 * 60 * 60); + } + assert_eq!( + prune_expired_backups(&path, RETENTION_DAY, std::time::SystemTime::now()).unwrap(), + 0 + ); + for name in malformed { + assert!(auto.join(name).exists(), "{name}"); + } +} + +/// A child forked by another thread briefly shares the lock file's open description, +/// so a lock this thread just released can still look held; the grace period must +/// absorb that instead of reporting the database as busy. +#[test] +fn platform_compatibility_lock_survives_concurrent_process_spawns() { + use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("wallet.sqlite"); + let stop = AtomicBool::new(false); + let spawned = AtomicUsize::new(0); + std::thread::scope(|scope| { + for _ in 0..2 { + scope.spawn(|| { + while !stop.load(Ordering::Relaxed) { + let _ = std::process::Command::new(std::env::current_exe().unwrap()) + .arg("--help") + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .status(); + spawned.fetch_add(1, Ordering::Relaxed); + } + }); + } + // Keep re-acquiring while at least 40 processes are spawned alongside. + let mut outcome = Ok(()); + while outcome.is_ok() && spawned.load(Ordering::Relaxed) < 40 { + outcome = backup_lock(&path).map(drop); + } + stop.store(true, Ordering::Relaxed); + outcome.expect("a released lock must be re-acquirable while processes spawn"); + }); +} diff --git a/src/wallet_backend/platform_compatibility/mod.rs b/src/wallet_backend/platform_compatibility/mod.rs index 14cd2efee..0f3c3c3ab 100644 --- a/src/wallet_backend/platform_compatibility/mod.rs +++ b/src/wallet_backend/platform_compatibility/mod.rs @@ -1,7 +1,14 @@ //! Preserve databases created by the pinned Platform PR when opening the development release. mod engine; +mod storage_failure; pub use engine::UpgradeError; +#[cfg(test)] +pub(crate) use engine::{LOCK_SUFFIX, backup_lock_path, bridge_backup_path}; +pub(crate) use engine::{ + is_bridge_backup_name, prune_expired_backups, remove_backups, upstream_backup_timestamp, +}; +pub(crate) use storage_failure::StorageFailure; use platform_wallet::changeset::PlatformWalletPersistence; use platform_wallet_storage::{SqlitePersister, SqlitePersisterConfig, WalletStorageError}; @@ -9,20 +16,52 @@ use platform_wallet_storage::{SqlitePersister, SqlitePersisterConfig, WalletStor use crate::backend_task::error::TaskError; pub(crate) fn open(config: SqlitePersisterConfig) -> Result { + open_with_housekeeping(config, engine::tidy_backups_locked) +} + +fn open_with_housekeeping( + config: SqlitePersisterConfig, + mut tidy: impl FnMut(&engine::BackupGuard, Option<&std::path::Path>) -> std::io::Result<()>, +) -> Result { + let guard = engine::backup_lock(&config.path).map_err(lock_error)?; + let auto_dir = config.auto_backup_dir.as_deref(); + // Best-effort both times: tidy before a snapshot can be taken, and again to drop a + // copy a failed open just duplicated. A failure never decides the open's result. + let mut tidy_best_effort = || { + if let Err(error) = tidy(&guard, auto_dir) { + tracing::warn!( + ?error, + "Upgrade backup housekeeping failed; retrying on the next open" + ); + } + }; + tidy_best_effort(); + let result = open_inner(&config, &guard); + tidy_best_effort(); + result +} + +fn open_inner( + config: &SqlitePersisterConfig, + guard: &engine::BackupGuard, +) -> Result { let original = match SqlitePersister::open(config.clone()) { Ok(persister) => return Ok(persister), Err(error @ WalletStorageError::Migration(_)) => error, Err(error) => return Err(TaskError::from_wallet_storage_open_error(error)), }; let result = - upgrade(&config).map_err(|source| TaskError::PlatformDatabaseUpgrade { source })?; + upgrade(config, guard).map_err(|source| TaskError::PlatformDatabaseUpgrade { source })?; if !result { return Err(TaskError::from_wallet_storage_open_error(original)); } - SqlitePersister::open(config).map_err(TaskError::from_wallet_storage_open_error) + SqlitePersister::open(config.clone()).map_err(TaskError::from_wallet_storage_open_error) } -fn upgrade(config: &SqlitePersisterConfig) -> Result { +fn upgrade( + config: &SqlitePersisterConfig, + guard: &engine::BackupGuard, +) -> Result { let parent = config.path.parent().ok_or(UpgradeError::Unrecognized)?; let mut builder = tempfile::Builder::new(); builder.prefix("platform-upgrade-"); @@ -33,19 +72,12 @@ fn upgrade(config: &SqlitePersisterConfig) -> Result { } let stage_dir = builder.tempdir_in(parent)?; let stage_path = stage_dir.path().join("wallet.sqlite"); - drop( - SqlitePersister::open(SqlitePersisterConfig::new(&stage_path)) - .map_err(|e| UpgradeError::TypedValidation(Box::new(e)))?, - ); - let backup = engine::upgrade(&config.path, &stage_path, |path| { - let staged = SqlitePersister::open(SqlitePersisterConfig::new(path)) - .map_err(|e| UpgradeError::TypedValidation(Box::new(e)))?; - staged - .load() - .map_err(|e| UpgradeError::TypedValidation(Box::new(e)))?; - staged - .load_unowned_identities() - .map_err(|e| UpgradeError::TypedValidation(Box::new(e)))?; + drop(SqlitePersister::open(SqlitePersisterConfig::new(&stage_path)).map_err(validation_error)?); + let backup = engine::upgrade_locked(guard, &stage_path, |path| { + let staged = + SqlitePersister::open(SqlitePersisterConfig::new(path)).map_err(validation_error)?; + staged.load().map_err(validation_error)?; + staged.load_unowned_identities().map_err(validation_error)?; Ok(()) })?; if let Some(backup) = backup { @@ -56,6 +88,35 @@ fn upgrade(config: &SqlitePersisterConfig) -> Result { } } +fn lock_error(source: std::io::Error) -> TaskError { + if source.kind() == std::io::ErrorKind::WouldBlock { + TaskError::WalletStorageNotReady + } else { + TaskError::FileSystem { source } + } +} + +fn validation_error(error: impl std::error::Error + Send + Sync + 'static) -> UpgradeError { + use platform_wallet::changeset::PersistenceError; + + if let Some(failure) = StorageFailure::in_chain(&error) { + return UpgradeError::from_failure(failure, Box::new(error)); + } + let mut transient = false; + let mut cause: Option<&(dyn std::error::Error + 'static)> = Some(&error); + while let Some(current) = cause { + transient |= current + .downcast_ref::() + .is_some_and(PersistenceError::is_transient); + cause = current.source(); + } + if transient { + UpgradeError::StorageUnavailable(Box::new(error)) + } else { + UpgradeError::TypedValidation(Box::new(error)) + } +} + #[cfg(test)] mod real_fixture_tests; @@ -64,6 +125,150 @@ mod tests { use super::*; use dash_sdk::dpp::identity::accessors::IdentityGettersV0; + /// Every failed open of an unsupported database makes upstream write another + /// `pre-migration-*` copy of the unchanged database; the open's own tidying + /// collapses them to the newest, even when the open fails. + /// + /// Upstream refuses a group- or world-writable data or backup directory before + /// taking any snapshot, so the data directory is made owner-only and the backup + /// directory is left for upstream to create; neither may depend on the umask. + #[test] + fn platform_compatibility_bounds_upstream_backups_after_failed_opens() { + let dir = tempfile::tempdir().unwrap(); + crate::app_dir::ensure_data_dir_exists(dir.path()).unwrap(); + let path = dir.path().join("wallet.sqlite"); + rusqlite::Connection::open(&path) + .unwrap() + .execute_batch(include_str!("fixtures/67d4ef3.sql")) + .unwrap(); + rusqlite::Connection::open(&path) + .unwrap() + .execute_batch("ALTER TABLE wallets ADD COLUMN unsupported INTEGER;") + .unwrap(); + let auto = platform_wallet_storage::default_auto_backup_dir(&path); + let mut previous: Option = None; + for attempt in 1..=3 { + assert!(open(SqlitePersisterConfig::new(&path)).is_err()); + let retained = dir_entries(&auto); + assert_eq!( + retained.len(), + 1, + "attempt {attempt}: one snapshot must remain, got {retained:?}" + ); + let snapshot = retained[0].clone(); + assert!( + snapshot.starts_with("pre-migration-wallet-") && snapshot.ends_with(".db"), + "attempt {attempt}: unexpected entry {snapshot}" + ); + assert!( + crate::utils::backup_prune::usable_snapshot(&auto.join(&snapshot)), + "attempt {attempt}: a real upstream snapshot must count as usable" + ); + if let Some(previous) = &previous { + assert_ne!( + &snapshot, previous, + "attempt {attempt}: the newest copy must be the one kept" + ); + } + // Age the kept copy's name so the next open's copy gets a distinct, + // newer name even within the same second. + let (head, _) = snapshot.rsplit_once('-').unwrap(); + let aged = format!("{head}-2020010{attempt}T000000Z.db"); + std::fs::rename(auto.join(&snapshot), auto.join(&aged)).unwrap(); + previous = Some(aged); + } + } + + /// Opening never deletes a published snapshot that differs from the others: only + /// the retention setting removes those, so "Keep forever" keeps every one. + #[test] + fn platform_compatibility_repeated_opens_keep_every_distinct_snapshot() { + let dir = tempfile::tempdir().unwrap(); + crate::app_dir::ensure_data_dir_exists(dir.path()).unwrap(); + let path = dir.path().join("wallet.sqlite"); + drop(open(SqlitePersisterConfig::new(&path)).unwrap()); + let auto = platform_wallet_storage::default_auto_backup_dir(&path); + std::fs::create_dir_all(&auto).unwrap(); + let snapshots = [ + dir.path() + .join("wallet.sqlite.platform-67d4ef3-backup-first.sqlite"), + auto.join("pre-migration-wallet-1-to-2-20200101T000000Z.db"), + auto.join("pre-migration-wallet-2-to-3-20210101T000000Z.db"), + auto.join("pre-migration-wallet-2-to-3-20220101T000000Z.db"), + ]; + for (index, snapshot) in snapshots.iter().enumerate() { + std::fs::write(snapshot, format!("snapshot {index}")).unwrap(); + } + for _ in 0..3 { + drop(open(SqlitePersisterConfig::new(&path)).unwrap()); + } + for snapshot in &snapshots { + assert!(snapshot.exists(), "{} must be kept", snapshot.display()); + } + } + + /// Housekeeping is best-effort before and after the open: its failure neither + /// blocks nor discards a successful open, nor masks the open's own error. + #[test] + fn platform_compatibility_housekeeping_failure_never_decides_the_open() { + fn failing( + calls: &mut usize, + ) -> impl FnMut(&engine::BackupGuard, Option<&std::path::Path>) -> std::io::Result<()> + '_ + { + move |_, _| { + *calls += 1; + Err(std::io::Error::other("housekeeping failure")) + } + } + let dir = tempfile::tempdir().unwrap(); + crate::app_dir::ensure_data_dir_exists(dir.path()).unwrap(); + let mut calls = 0; + let persister = open_with_housekeeping( + SqlitePersisterConfig::new(dir.path().join("wallet.sqlite")), + failing(&mut calls), + ) + .expect("failed housekeeping must not block or discard an open"); + assert_eq!(calls, 2); + persister.load().unwrap(); + + let bad = dir.path().join("corrupt.sqlite"); + std::fs::write(&bad, b"not a sqlite database, just candy wrappers").unwrap(); + let mut calls = 0; + let error = + match open_with_housekeeping(SqlitePersisterConfig::new(&bad), failing(&mut calls)) { + Ok(_) => panic!("open unexpectedly succeeded on a corrupt database"), + Err(error) => error, + }; + assert_eq!(calls, 2); + assert!( + !matches!(&error, TaskError::FileSystem { source } if source.to_string() == "housekeeping failure"), + "the open error must not be masked by housekeeping: {error:?}" + ); + } + + fn dir_entries(dir: &std::path::Path) -> Vec { + let mut names: Vec = std::fs::read_dir(dir) + .unwrap() + .map(|entry| entry.unwrap().file_name().to_string_lossy().into_owned()) + .collect(); + names.sort(); + names + } + + #[test] + fn platform_compatibility_open_reports_lifecycle_lock_as_retryable() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("wallet.sqlite"); + rusqlite::Connection::open(&path).unwrap(); + let _guard = engine::backup_lock(&path).unwrap(); + let error = match open(SqlitePersisterConfig::new(&path)) { + Ok(_) => panic!("open unexpectedly acquired the held lifecycle lock"), + Err(error) => error, + }; + assert!(matches!(error, TaskError::WalletStorageNotReady)); + assert!(error.to_string().contains("try again")); + } + #[test] fn platform_compatibility_normal_open_upgrades_old_profile() { let dir = tempfile::tempdir().unwrap(); @@ -114,6 +319,55 @@ mod tests { 1 ); drop(persister); - open(SqlitePersisterConfig::new(&path)).unwrap(); + let upstream = platform_wallet_storage::default_auto_backup_dir(&path); + let upstream_snapshots = || std::fs::read_dir(&upstream).unwrap().count(); + let before_reopen = upstream_snapshots(); + drop(open(SqlitePersisterConfig::new(&path)).unwrap()); + // Both recovery points stay: the bridge copy of the pinned profile and any + // upstream copy taken before its own migration. A current database adds none. + assert_eq!(upstream_snapshots(), before_reopen); + assert!( + std::fs::read_dir(dir.path()).unwrap().any(|entry| entry + .unwrap() + .file_name() + .to_string_lossy() + .contains(".platform-67d4ef3-backup-")), + "the bridge snapshot is kept" + ); + } +} + +#[cfg(test)] +mod validation_tests { + use super::*; + use platform_wallet::changeset::PersistenceError; + + /// Staged-validation failures keep their storage class through any wrapper: + /// resource failures stay retryable, invalid data stays terminal. + #[test] + fn platform_compatibility_staged_validation_errors_keep_their_class() { + let full = || { + WalletStorageError::Sqlite(rusqlite::Error::SqliteFailure( + rusqlite::ffi::Error::new(rusqlite::ffi::SQLITE_FULL), + None, + )) + }; + for source in [ + validation_error(full()), + validation_error(PersistenceError::from(full())), + ] { + assert!(matches!(source, UpgradeError::StorageFull(_)), "{source:?}"); + assert!(std::error::Error::source(&source).is_some()); + assert!(!crate::backend_task::is_terminal_storage_open_error( + &TaskError::PlatformDatabaseUpgrade { source } + )); + } + let source = validation_error(PersistenceError::from( + WalletStorageError::SchemaHistoryMissing, + )); + assert!(matches!(source, UpgradeError::TypedValidation(_))); + assert!(crate::backend_task::is_terminal_storage_open_error( + &TaskError::PlatformDatabaseUpgrade { source } + )); } } diff --git a/src/wallet_backend/platform_compatibility/storage_failure.rs b/src/wallet_backend/platform_compatibility/storage_failure.rs new file mode 100644 index 000000000..f8cd5da64 --- /dev/null +++ b/src/wallet_backend/platform_compatibility/storage_failure.rs @@ -0,0 +1,127 @@ +//! The one policy for which storage failures are environmental (and which of +//! those retrying can fix), shared by the upgrade engine and `TaskError`. + +use std::error::Error; + +/// An environmental storage failure, as opposed to a defect in the data itself. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum StorageFailure { + /// Another connection or process holds the database. + InUse, + /// The disk is full. + Full, + /// The OS reported an I/O failure or a transient interruption. + Unavailable, + /// Memory ran out. + OutOfMemory, + /// Permissions or a read-only medium prevent writing. + AccessDenied, +} + +impl StorageFailure { + /// Classify one error, ignoring its sources. + pub(crate) fn of(error: &(dyn Error + 'static)) -> Option { + if let Some(sqlite) = error.downcast_ref::() { + return Self::of_sqlite(sqlite); + } + error.downcast_ref::().and_then(Self::of_io) + } + + /// Classify the first error in `error`'s source chain that has a category. + pub(crate) fn in_chain(error: &(dyn Error + 'static)) -> Option { + let mut cause = Some(error); + while let Some(current) = cause { + if let Some(failure) = Self::of(current) { + return Some(failure); + } + cause = current.source(); + } + None + } + + pub(crate) fn of_sqlite(error: &rusqlite::Error) -> Option { + use rusqlite::ErrorCode; + match error.sqlite_error_code()? { + ErrorCode::DatabaseBusy | ErrorCode::DatabaseLocked => Some(Self::InUse), + ErrorCode::DiskFull => Some(Self::Full), + ErrorCode::SystemIoFailure => Some(Self::Unavailable), + ErrorCode::OutOfMemory => Some(Self::OutOfMemory), + ErrorCode::PermissionDenied | ErrorCode::ReadOnly => Some(Self::AccessDenied), + _ => None, + } + } + + pub(crate) fn of_io(error: &std::io::Error) -> Option { + use std::io::ErrorKind; + match error.kind() { + ErrorKind::StorageFull => Some(Self::Full), + ErrorKind::OutOfMemory => Some(Self::OutOfMemory), + ErrorKind::Interrupted + | ErrorKind::WouldBlock + | ErrorKind::TimedOut + | ErrorKind::ResourceBusy => Some(Self::Unavailable), + ErrorKind::PermissionDenied | ErrorKind::ReadOnlyFilesystem => Some(Self::AccessDenied), + _ => None, + } + } + + /// Whether freeing the resource and retrying can recover. Access failures + /// never fix themselves. + pub(crate) fn is_retryable(self) -> bool { + !matches!(self, Self::AccessDenied) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn storage_failure_classifies_every_error_source() { + use rusqlite::ffi; + use std::io::ErrorKind; + let sqlite = |code| rusqlite::Error::SqliteFailure(ffi::Error::new(code), None); + let sqlite_cases = [ + (ffi::SQLITE_BUSY, Some(StorageFailure::InUse)), + (ffi::SQLITE_LOCKED, Some(StorageFailure::InUse)), + (ffi::SQLITE_FULL, Some(StorageFailure::Full)), + (ffi::SQLITE_IOERR, Some(StorageFailure::Unavailable)), + (ffi::SQLITE_NOMEM, Some(StorageFailure::OutOfMemory)), + (ffi::SQLITE_PERM, Some(StorageFailure::AccessDenied)), + (ffi::SQLITE_READONLY, Some(StorageFailure::AccessDenied)), + (ffi::SQLITE_CORRUPT, None), + ]; + for (code, expected) in sqlite_cases { + // Wrapped, so the source chain is walked too. + let wrapped = platform_wallet_storage::WalletStorageError::Sqlite(sqlite(code)); + assert_eq!(StorageFailure::in_chain(&wrapped), expected, "{code}"); + } + let io_cases = [ + (ErrorKind::StorageFull, Some(StorageFailure::Full)), + (ErrorKind::OutOfMemory, Some(StorageFailure::OutOfMemory)), + (ErrorKind::Interrupted, Some(StorageFailure::Unavailable)), + (ErrorKind::WouldBlock, Some(StorageFailure::Unavailable)), + (ErrorKind::TimedOut, Some(StorageFailure::Unavailable)), + (ErrorKind::ResourceBusy, Some(StorageFailure::Unavailable)), + ( + ErrorKind::PermissionDenied, + Some(StorageFailure::AccessDenied), + ), + ( + ErrorKind::ReadOnlyFilesystem, + Some(StorageFailure::AccessDenied), + ), + (ErrorKind::InvalidInput, None), + (ErrorKind::Other, None), + ]; + for (kind, expected) in io_cases { + assert_eq!( + StorageFailure::in_chain(&std::io::Error::from(kind)), + expected, + "{kind:?}" + ); + } + assert!(!StorageFailure::AccessDenied.is_retryable()); + assert!(StorageFailure::Full.is_retryable()); + } +} diff --git a/tests/backend-e2e/spv_reconnect.rs b/tests/backend-e2e/spv_reconnect.rs index f8c76b492..a134c5028 100644 --- a/tests/backend-e2e/spv_reconnect.rs +++ b/tests/backend-e2e/spv_reconnect.rs @@ -49,6 +49,10 @@ use std::time::Duration; /// It only needs live testnet peers (outbound TCP on port 19999). #[tokio_shared_rt::test(shared, flavor = "multi_thread", worker_threads = 12)] #[ignore = "network-dependent; requires testnet egress — no funded wallet needed"] +#[expect( + clippy::disallowed_methods, + reason = "test fixture setup/teardown outside any production deletion path" +)] async fn spv_reconnect_succeeds_without_already_open() { // ── Isolated context setup ────────────────────────────────────────────── let workdir =