Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 55 additions & 13 deletions packages/swift-sdk/SCHEMA_RELEASES.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,40 @@

SwiftData schemas become supported history when a build reaches App Store
distribution. TestFlight uploads capture provenance and a synthetic SQLite
fixture, but do not by themselves register a released schema. V1 is the agreed
existing baseline. Intermediate pre-release V2–V5 schemas have been collapsed
into the working V2, including the public-key usage-limit columns. A separate
compatibility bridge handles older, unregistered `1.0.0` database layouts when
they can migrate without losing existing data. Other intermediate development
databases are unsupported.
fixture, but do not by themselves register a released schema. The accepted frozen V1 remains unchanged. Historical V2 is now reconstructed
from `52e8d4ec68f0c772313fa1bbef223fb1eabbf1cc`; all 35 entity hashes and the
model checksum match the observed App Store 9.0.2 database. Active models are
V3. Other intermediate development shapes remain unsupported.

The old V2 fixture was generated from September 8 sources containing 13
properties added on August 28, after the August 27 App Store release. The
reconstructed V2 instead predates those additions. Its synthetic fixture and
source provenance live in `schema-releases.json` under `historical_schemas`,
separate from archive-captured releases. This identifies a matching model
source, not the confirmed build commit of Apple's binary. V2 is reserved:
automated release capture must not register another shape under that number.

The main migration plan is historical V2 → V3. Accepted V1 has a separate
V1 → V3 plan: V1 already contains the 13 properties missing from historical
V2, so a V1 → V2 → V3 chain could discard values. Routing uses model metadata
and runs after recovery; a version label alone never selects an unknown beta
schema. The former live V2 is accepted only when its complete graph matches
current V3 exactly, entity hashes and checksum alike. That alias therefore
lasts only until the next live-graph change: after it, every store still
labelled `2.0.0` with the former live shape becomes `unsupported-v2` and fails
closed. Check internal devices still carrying that label before the next
shape change and migrate or deliberately reset them then, rather than
discovering them afterwards as failed opens.

The route decision is logged as `store_migration_route` with the validated
`source_version`, `source_checksum` and one of `new-store`,
`accepted-v1-to-v3`, `historical-v2-to-v3`, `previous-live-v2-current-shape`,
`unsupported-v2`, `labelled-current-v3`, `ordinary-current-plan` or, from the
bridge, `legacy-v1-bridge-to-v3`. Resolving a frozen schema's identity builds
a temporary store, so it is memoized per process and consulted only where a
label is undecidable without it (`1.0.0`, `2.0.0`); a `3.0.0` label takes the
default plan without any probe, and a probe failure on the undecidable labels
refuses the open with the probe's own error while leaving the store untouched.

## Legacy stores before the release registry

Expand All @@ -19,7 +47,7 @@ the explicit migration plan; those databases still report version `1.0.0`.
The shared `DashModelContainer` factory recognizes registered schema
fingerprints before opening an existing store. Unknown legacy `1.0.0` stores
may use the compatibility bridge: take a consistent backup including committed
WAL data, migrate an isolated copy automatically to `DashSchemaV2`, verify that
WAL data, migrate an isolated copy automatically to `DashSchemaV3`, verify that
existing stored values and relationship rows survived, and reopen it through
the ordinary migration plan before installing it. The backup is retained for
recovery. Corruption, removed fields/entities, changed stored data, and newer
Expand Down Expand Up @@ -109,9 +137,9 @@ The run failed before upload, so this provenance establishes a tested source
layout rather than proof of publication. Regression tests exercise the public
factory, data/default preservation, writes, reopen, and failure recovery.

Keep the bridge for installations that skip the V2 app release. When advancing
to V3, bind `DashSchemaV2` to its released snapshot and retain the legacy-to-V2
step before the normal V2-to-current plan. The bridge must never automatically
Keep the bridge for installations that skip the V3 app release. When advancing
to V4, bind `DashSchemaV3` to its released snapshot and retain the legacy-to-V3
step before the normal V3-to-current plan. The bridge must never automatically
follow the latest live model graph. The release observer's one-time `bootstrap`
only records its observation baseline; it neither runs this migration nor
proves V1's App Store provenance.
Expand Down Expand Up @@ -181,8 +209,9 @@ all required releases, not merely a commit from before the snapshot merge.
access, and uses the PAT so its PR events trigger CI. Apple credentials stay
in the iOS repository. Do not put tokens in command arguments or manifests.
3. Initialize the iOS release baseline before the first newly tracked release,
then run its monitor manually in dry-run mode. The baseline accepts V1 as
agreed and does not claim to reconstruct an earlier binary.
then run its monitor manually in dry-run mode. The baseline must reference
the verified historical V2 binding for 9.0.2. Correct the old V1 association
without moving the publication cutoff; see the iOS runbook.
4. After publication, use the iOS manual monitor for the normal operator flow.
For a worker retry, select the recorded Apple version ID and a full commit
on `schema-release-data`. Select `dry_run` to validate and generate the patch
Expand Down Expand Up @@ -213,7 +242,7 @@ standard Swift/Foundation names are assumed not to be shadowed by application
types. It does not prove custom encoding or helper-method behavior, so native
captured-store hash/index checks and code review remain required.

A released snapshot has its own namespace, for example `DashSchemaSnapshotV2`.
A released snapshot has its own namespace, for example `DashSchemaSnapshotV3`.
It is not automatically registered alongside identical current models. When
changing the structure after a release, explicitly register the historical
snapshot as the old schema, introduce the next active version and its migration,
Expand Down Expand Up @@ -266,3 +295,16 @@ python3 packages/swift-sdk/scripts/historical_schema_fixture.py --check

Swift SDK CI additionally checks the generated schemas against the released
fixtures, including entity hashes, indexes and migration behavior.

## Local verification with a private database copy

`DashModelMigrationTests.testPrivateHistoricalStoreMigrationWhenExplicitlyProvided`
accepts a local standalone SQLite input through `DASH_PRIVATE_MIGRATION_STORE`.
For an Xcode simulator test run, forward it using the
`TEST_RUNNER_DASH_PRIVATE_MIGRATION_STORE` environment variable. The test copies
that input to a disposable directory, verifies preservation of every supported
SQLite table/relationship, writes through current models and reopens the copy.
Without the explicit input it skips; normal CI uses only synthetic fixtures.
Do not add a real wallet database to resources, XCTest attachments, commits or
CI artifacts. If capturing another input, preserve committed WAL contents with
a consistent SQLite backup rather than copying an active main file alone.
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,14 @@ private final class SDKLoggerState: @unchecked Sendable {
}
}

func removeSink() {
let removed: SDKLogFileSink? = lock.withLock {
defer { sink = nil }
return sink
}
removed?.flush()
}

func destination(for severity: SDKLogSeverity) -> SDKLogFileSink? {
lock.withLock {
guard severity != .debug || includeDebug else { return nil }
Expand Down Expand Up @@ -503,6 +511,13 @@ public enum SDKLogger {
state.updateDebugSetting(includeDebug)
}

/// Detach the process-wide file sink after flushing it. Tests that install
/// a sink in a disposable directory call this before removing the
/// directory, so later events are not written through a stale handle.
static func removeFileSink() {
state.removeSink()
}

public static func log(_ message: String, minimumLevel level: LoggingPreset = .medium) {
guard LoggingPreferences.allows(level) else { return }
// Mirror to NSLog (unified logging) in addition to stdout so
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@ import Darwin
import Foundation
import SwiftData

/// One-time, local-store bridge into the fixed V2 schema. The ordinary migration
/// One-time, local-store bridge into the fixed V3 schema. The ordinary migration
/// plan owns recognized versions. This path accepts only an older 1.0.0 graph
/// whose complete existing SQLite data survives an inferred migration unchanged.
/// Call before publishing any container/context for the same URL.
enum DashLegacySchemaBridge {
typealias SQLite = DashLegacyStoreSQLite
enum Phase { case afterSnapshot, afterMigration, beforeInstall, writeLocked, afterCommit }
enum Phase: String { case afterSnapshot, afterMigration, beforeInstall, writeLocked, afterCommit }
struct Hooks {
var visit: (Phase, URL) throws -> Void = { _, _ in }
var availableCapacity: (URL) throws -> Int64 = DashLegacySchemaBridge.availableCapacity(at:)
Expand All @@ -30,8 +30,16 @@ enum DashLegacySchemaBridge {
static func open(configuration: ModelConfiguration, schema: Schema,
plan: any SchemaMigrationPlan.Type, hooks: Hooks = Hooks()) throws -> ModelContainer {
let url = configuration.url
func visit(_ phase: Phase, _ candidate: URL) throws {
SDKLogger.event("legacy_migration_phase", category: .persistence,
fields: ["phase": .publicText(phase.rawValue)])
try hooks.visit(phase, candidate)
}
func ordinary() throws -> ModelContainer {
try ModelContainer(for: schema, migrationPlan: plan, configurations: [configuration])
let selectedPlan = configuration.isStoredInMemoryOnly ? plan :
try DashModelContainer.migrationPlan(at: url, defaultPlan: plan)
return try ModelContainer(for: schema, migrationPlan: selectedPlan,
configurations: [configuration])
}
guard !configuration.isStoredInMemoryOnly else { return try ordinary() }
let root = backupDirectory(for: url)
Expand All @@ -41,7 +49,7 @@ enum DashLegacySchemaBridge {
if !FileManager.default.fileExists(atPath: marker.path) {
// Detection must not impose bridge-specific metadata or locking
// requirements on stores handled by SwiftData's ordinary path.
let needsMigration = (try? identity(at: url)).flatMap { try? needsBridge($0, plan: plan) } ?? false
let needsMigration = (try? identity(at: url)).flatMap { try? needsBridge($0) } ?? false
// Recheck after reading identity: a concurrent bridge publishes its
// journal before the store can change to the destination schema.
if !needsMigration && !FileManager.default.fileExists(atPath: marker.path) {
Expand All @@ -59,12 +67,19 @@ enum DashLegacySchemaBridge {
throw SQLite.Failure.unsupported(
"The original database is missing while migration recovery is pending. Recovery files remain at \(root.path). Restore the original database from a verified backup with the app closed, or contact support for deliberate recovery. Do not delete the journal or create an empty database.")
}
SDKLogger.event("legacy_migration_phase", category: .persistence,
fields: ["phase": .publicText("recovery")])
try SQLite.recoverRollbackJournal(at: url)
try recoverIfNeeded(at: url, root: root)
// Another opener may have finished migration before this lock was
// acquired. Recovery errors above remain fatal; ordinary detection does not.
guard let source = try? identity(at: url),
(try? needsBridge(source, plan: plan)) == true else { return try ordinary() }
(try? needsBridge(source)) == true else { return try ordinary() }
SDKLogger.event("store_migration_route", category: .persistence, fields: [
"source_version": .publicText("1.0.0"),
"source_checksum": .publicText(Data(base64Encoded: source.checksum)?.count == 32 ? source.checksum : "unavailable"),
"route": .publicText("legacy-v1-bridge-to-v3"),
"target_version": .publicText("3.0.0")])
guard configuration.allowsSave else { throw SQLite.Failure.unsupported("The store is read-only") }
let permitted = Set(Schema(versionedSchema: DashSchemaV1.self).entities.map(\.name))
let required: Set<String> = ["PersistentWallet", "PersistentAccount", "PersistentTransaction", "PersistentTxo"]
Expand Down Expand Up @@ -108,23 +123,23 @@ enum DashLegacySchemaBridge {
guard try SQLite.rawDigest(url) == rawSource else { throw SQLite.Failure.sourceChanged }
try SQLite.integrityCheck(backup)
guard try identity(at: backup) == source else { throw SQLite.Failure.sourceChanged }
try hooks.visit(.afterSnapshot, url)
try visit(.afterSnapshot, url)
try createProtectedFile(candidate, like: url)
try SQLite.copy(from: backup, to: candidate)
// Never replace this with `schema` or the latest version. Users can skip
// the V2 app; later releases must keep V2's frozen graph as this target.
// the V3 app; later releases must keep V3's frozen graph as this target.
try autoreleasepool {
let bridgeSchema = Schema(versionedSchema: DashSchemaV2.self)
let bridgeSchema = Schema(versionedSchema: DashSchemaV3.self)
_ = try ModelContainer(for: bridgeSchema, configurations: [
ModelConfiguration(schema: bridgeSchema, url: candidate, cloudKitDatabase: .none)
])
}
try SQLite.checkpoint(candidate)
try hooks.visit(.afterMigration, candidate)
try visit(.afterMigration, candidate)
try rejectExternalStorage(at: candidate)
try SQLite.integrityCheck(candidate)
try SQLite.validatePreservation(from: backup, to: candidate)
// The normal plan must accept the fixed V2 result. In a future release
// The normal plan must accept the fixed V3 result. In a future release
// it may continue through additional explicitly registered stages here.
try autoreleasepool {
_ = try ModelContainer(for: schema, migrationPlan: plan, configurations: [
Expand All @@ -146,16 +161,16 @@ enum DashLegacySchemaBridge {
}
try synchronizeDirectory(directory)
try writeJournal(journal, at: root)
try hooks.visit(.beforeInstall, url)
try visit(.beforeInstall, url)
try SQLite.copy(from: candidate, to: url) {
// backup_step(0) owns SQLite's write lock, closing the race between
// comparison and commit. Main/WAL bytes must still be the snapshot's
// bytes; unrelated or concurrent writes cause a safe retry.
guard try SQLite.rawDigest(url) == rawSource else { throw SQLite.Failure.sourceChanged }
try hooks.visit(.writeLocked, url)
try visit(.writeLocked, url)
}
committed = true
try hooks.visit(.afterCommit, url)
try visit(.afterCommit, url)
let container = try ordinary()
// This container has not escaped to the app, so no application writes
// are permitted yet. Clearing must succeed before returning it: a stale
Expand Down Expand Up @@ -215,11 +230,11 @@ enum DashLegacySchemaBridge {
return max(0, try fileSystem())
}

private static func needsBridge(_ source: Identity, plan: any SchemaMigrationPlan.Type) throws -> Bool {
private static func needsBridge(_ source: Identity) throws -> Bool {
guard source.versions == ["1.0.0"] else { return false }
for registered in plan.schemas where version(registered.versionIdentifier) == "1.0.0" {
if source == (try identity(for: registered)) { return false }
}
// Accepted V1 has its own direct route to V3, independent of the
// primary historical V2 migration plan.
if source == (try identity(for: DashSchemaV1.self)) { return false }
return true
}

Expand Down Expand Up @@ -325,7 +340,7 @@ enum DashLegacySchemaBridge {
}
}

private static func identity(at url: URL) throws -> Identity {
static func identity(at url: URL) throws -> Identity {
let metadata = try NSPersistentStoreCoordinator.metadataForPersistentStore(type: .sqlite, at: url)
guard let versions = metadata[NSStoreModelVersionIdentifiersKey] as? [String],
let checksum = metadata["NSStoreModelVersionChecksumKey"] as? String,
Expand All @@ -336,7 +351,16 @@ enum DashLegacySchemaBridge {
return Identity(versions: versions, checksum: checksum, hashes: hashes)
}

private static func identity(for type: any VersionedSchema.Type) throws -> Identity {
/// The identity Core Data records for a versioned schema. It is a pure
/// function of the frozen model graph, but computing it writes a complete
/// temporary store, so each successful result is kept for the process.
static func identity(for type: any VersionedSchema.Type) throws -> Identity {
try identityCache.identity(for: type) { try computeIdentity(for: type) }
}

private static let identityCache = SchemaIdentityCache()

private static func computeIdentity(for type: any VersionedSchema.Type) throws -> Identity {
let directory = FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString)
try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
defer { try? FileManager.default.removeItem(at: directory) }
Expand All @@ -350,6 +374,23 @@ enum DashLegacySchemaBridge {
return try identity(at: url)
}

/// Process-lifetime memo of schema identities. Only successful
/// computations are stored: a failed probe (for example a full disk) is
/// retried on the next request rather than remembered.
final class SchemaIdentityCache: @unchecked Sendable {
private let lock = NSLock()
private var identities: [ObjectIdentifier: Identity] = [:]

func identity(for type: any VersionedSchema.Type,
compute: () throws -> Identity) throws -> Identity {
let key = ObjectIdentifier(type)
if let cached = lock.withLock({ identities[key] }) { return cached }
let computed = try compute()
lock.withLock { identities[key] = computed }
return computed
}
}

private static func rejectExternalStorage(at url: URL) throws {
let parent = url.deletingLastPathComponent()
let names = Set([url.lastPathComponent, url.deletingPathExtension().lastPathComponent])
Expand Down Expand Up @@ -393,10 +434,6 @@ enum DashLegacySchemaBridge {
defer { Darwin.close(descriptor) }
guard fsync(descriptor) == 0 else { throw SQLite.Failure.database("Cannot persist migration journal") }
}
private static func version(_ value: Schema.Version) -> String {
"\(value.major).\(value.minor).\(value.patch)"
}

private final class StoreLock {
private var descriptor: Int32
init(url: URL) throws {
Expand Down
Loading
Loading