diff --git a/website/docs/deployment/read-write-separation.md b/website/docs/deployment/read-write-separation.md index 4319609c1..285ad5d7d 100644 --- a/website/docs/deployment/read-write-separation.md +++ b/website/docs/deployment/read-write-separation.md @@ -138,7 +138,7 @@ datasets: duckdb_file: /data/customers.db ``` -Snapshots are partitioned by date and dataset (`month=YYYY-MM/day=YYYY-MM-DD/dataset=/...`), so retention is a normal object-store lifecycle rule. See [Snapshots](../features/data-acceleration/snapshots) for the full configuration reference. +Snapshots are partitioned by date and dataset (`month=YYYY-MM/day=YYYY-MM-DD/dataset=/...`), and the publishing instance keeps each dataset's snapshots of the last 14 days. See [Snapshot retention](../features/data-acceleration/snapshots#snapshot-retention), and [Snapshots](../features/data-acceleration/snapshots) for the full configuration reference. ### The read instances (read tier) @@ -245,13 +245,13 @@ s3://spiceai-snapshots/prod/ The extension names the engine that wrote the snapshot (`.duckdb`, `.sqlite`, `.cayenne`, or `.turso`), so the cluster and read tiers must accelerate a dataset with the same engine. -Apply an object-store lifecycle rule (S3 lifecycle, GCS Object Lifecycle Management, ADLS Lifecycle) to expire old partitions. Most deployments keep 24–72 hours of refresh-triggered snapshots and a daily archive beyond that. +The publishing instance deletes snapshots older than 14 days, keeping each dataset's current and newest snapshot at any age, so the bucket needs no expiration rule for current objects. On a bucket with versioning enabled, each delete leaves a noncurrent version, so add a noncurrent-version expiration rule. A replicated destination keeps deleted files unless replication carries deletes or the destination has its own lifecycle rule. See [Snapshot retention](../features/data-acceleration/snapshots#snapshot-retention). To keep a longer archive, copy snapshots to another prefix or bucket. The snapshot bucket is the only shared dependency between the tiers, so keep it in the same region as the read instances and apply VPC endpoints / Private Google Access to keep traffic on the private network. When read instances run in another region, replicate this standard bucket. Keep `snapshots.location` on standard S3, GCS, or ADLS for that replication. Cayenne's S3 Express One Zone settings (`cayenne_file_path`, `cayenne_s3_*`) store the Cayenne data tier in one zone and do not substitute for the snapshot bucket. See [Snapshot location and the Cayenne data tier](../features/data-acceleration/snapshots#snapshot-location-and-the-cayenne-data-tier). ### Full copies and incremental replication -Each snapshot object is the full acceleration file. Replicating the snapshot bucket between regions copies that entire file on every snapshot. Size the bucket and the lifecycle rule for dataset size times snapshot frequency. DuckDB `snapshots_compaction` shrinks each copy; the result is still a full file. See [What each snapshot contains](../features/data-acceleration/snapshots#what-each-snapshot-contains). +Each snapshot object is the full acceleration file. Replicating the snapshot bucket between regions copies that entire file on every snapshot. Size the bucket for dataset size times the number of snapshots written in about 15 days: 14 days of retention plus at least 24 hours before a dropped file is deleted. DuckDB `snapshots_compaction` shrinks each copy; the result is still a full file. See [What each snapshot contains](../features/data-acceleration/snapshots#what-each-snapshot-contains). When changed data should replicate as objects between regions or tiers, use a path that publishes the change: diff --git a/website/docs/features/data-acceleration/snapshots.md b/website/docs/features/data-acceleration/snapshots.md index 3bcb090b2..7ff31cf86 100644 --- a/website/docs/features/data-acceleration/snapshots.md +++ b/website/docs/features/data-acceleration/snapshots.md @@ -57,7 +57,7 @@ Every accelerated dataset must write to its own file (for example, `/nvme/my_dat Each snapshot is a complete copy of that dataset's acceleration file at the time Spice writes it. The object under the snapshot location is the whole accelerated dataset — DuckDB, SQLite, Cayenne, or Turso — ready for a reader to download and open. Spice writes that full file on every snapshot. It does not publish an incremental delta of the rows or objects that changed since the previous snapshot. -A bootstrap, and a [`refresh_mode: snapshot`](./data-refresh#snapshot) reload, replaces the local acceleration file with that copy. Every upload moves the full file, and bucket or object replication of the snapshot prefix moves the full file again. Storage grows with the size of the accelerated dataset times how often snapshots are written, until a lifecycle rule expires older objects. DuckDB [`snapshots_compaction`](#snapshot-compaction) can shrink each copy. The uploaded object is still a full file. +A bootstrap, and a [`refresh_mode: snapshot`](./data-refresh#snapshot) reload, replaces the local acceleration file with that copy. Every upload moves the full file, and bucket or object replication of the snapshot prefix moves the full file again. Storage grows with the size of the accelerated dataset times the number of snapshots kept. The [14-day retention period](#snapshot-retention) applies to the entries in `metadata.json`; dropped files stay at least 24 more hours, until a later publish deletes them, so budget for about 15 days of snapshot files, plus the current and newest snapshot when they are older. DuckDB [`snapshots_compaction`](#snapshot-compaction) can shrink each copy. The uploaded object is still a full file. When the workload needs incremental replication of changed data between regions or storage tiers, use a path that publishes the changed objects: @@ -66,6 +66,22 @@ When the workload needs incremental replication of changed data between regions Neither path bootstraps a file-mode accelerator. Keep snapshots for that. +## Snapshot retention + +Each dataset keeps the snapshots it published in the last 14 days. Its current snapshot and its newest snapshot are kept at any age, so the location always holds a snapshot to bootstrap from, however long ago it was published. The retention period is not configurable. + +The instance that creates a dataset's snapshots applies retention each time it publishes one. In the same [conditional write](#conditional-writes-on-the-snapshot-location) of `metadata.json` that records the new snapshot, it drops the entries that are older than 14 days. It deletes a dropped snapshot's file 24 hours after the drop, so a reader that is still downloading the file can finish. About once an hour, the publishing instance also lists the dataset's snapshot files and deletes the files that no entry names and that are at least 24 hours old, such as the upload of a publish that failed. A publish that drops entries or deletes files logs a message such as: + +```text +Snapshot cleanup of dataset 'orders' dropped 2 snapshots older than 14d and deleted 2 snapshot files. +``` + +Retention runs only when a snapshot is published, so the snapshots of a dataset that no longer publishes stay at the location. Spice v2.4.0-rc.2 and earlier keep every snapshot. After an upgrade, the first publish of each dataset drops its entries older than 14 days, and their files are deleted at a publish 24 hours or more later. + +A dropped snapshot is no longer listed by `GET /v1/datasets/{name}/acceleration/snapshots`, and `POST /v1/datasets/{name}/acceleration/snapshots/current` cannot make it current: the request returns `404 Not Found`. To roll back further than 14 days, keep a copy of the snapshot outside the location. + +A bucket lifecycle rule on the snapshot location can still delete files that `metadata.json` lists. Spice treats a snapshot whose file is gone as a failed download, and [`bootstrap_on_failure_behavior`](#failure-behavior) applies: `fallback` tries the next older snapshot, `warn` continues with an empty acceleration, and `retry` keeps retrying. When the file of an entry at least 24 hours old is gone, the publishing instance drops the entry and logs a warning that names the snapshot. The current and newest entries are never dropped this way, so a current or newest snapshot whose file is gone stays listed until a newer snapshot is published. + ## Configure snapshot storage Snapshots are controlled with a top-level `snapshots` block in the Spicepod. The location can point to S3, Azure ADLS Gen2, Google Cloud Storage, or the local filesystem. @@ -438,7 +454,7 @@ The dataset is read-only. Spice rejects a configuration that contradicts reading - **Pair with ephemeral storage:** Deployments commonly place the acceleration file on fast ephemeral disks (such as NVMe instance storage) while relying on snapshots for persistence across restarts. Local NVMe is the recommended medium for accelerations — see [Storage](../../reference/performance-tuning#storage) for the tiers, the instance-store lifetime, and the capacity figures. - **Enable compaction for large datasets:** Use `snapshots_compaction: enabled` for DuckDB accelerations to reduce snapshot size and improve bootstrap performance. - **Tune trigger thresholds for stream datasets:** For high-throughput streaming datasets, balance snapshot frequency against I/O overhead by adjusting `snapshots_trigger_threshold`. -- **Align retention policies:** Apply an object storage lifecycle rule that mirrors the desired snapshot retention policy. +- **Let Spice expire snapshots:** Spice keeps [14 days of snapshots](#snapshot-retention) and deletes older files itself. An expiration rule for current objects on the snapshot location is not needed. When one is required, set its expiration longer than 14 days. Such a rule can still delete the file of a current or newest snapshot older than its expiration. On a bucket with versioning enabled, a Spice delete leaves a noncurrent version, so add a noncurrent-version expiration rule to reclaim that storage. A replication destination keeps deleted files unless the replication carries deletes or the destination has its own lifecycle rule. - **Monitor bootstraps:** Track warning logs emitted when Spice falls back to an empty acceleration so operators can respond quickly if snapshot loading fails. - **Search indexes are not all inside the accelerator file.** A DuckDB HNSW index lives in the DuckDB file, so it is part of that dataset's snapshot. A built-in full-text index does not: the default in-memory Tantivy index is rebuilt on every start, and `index_store: file` writes a separate directory (`.spice/data/fts/...` unless `index_directory` is set) that acceleration snapshots do not upload. For restart parity of full-text search, keep that directory on the same durable volume as the acceleration file, or build the index once on a central tier and serve sidecars from the [search results cache](../caching). See [Where indexes are built](../search#where-indexes-are-built). diff --git a/website/docs/features/distributed-query/index.md b/website/docs/features/distributed-query/index.md index a0b5cf4f3..32eda680b 100644 --- a/website/docs/features/distributed-query/index.md +++ b/website/docs/features/distributed-query/index.md @@ -653,7 +653,7 @@ Job state and result chunks are stored in the shared object store configured via │ └── ... ``` -Each scheduler deletes expired jobs, with their result chunks, every 10 minutes. Only jobs whose results have already expired are deleted. An expired job answers its owner with `410 Gone` until the next cleanup deletes it, and with `404 Not Found` after that. +Each scheduler deletes, with their result chunks, the jobs whose results have expired and the jobs that failed or were cancelled at least 12 hours ago. It runs this cleanup about once an hour, at a random time between 30 and 90 minutes after the previous run. An expired job answers its owner with `410 Gone` until the next cleanup deletes it, and with `404 Not Found` after that. A cleanup that deletes jobs logs how many it deleted. ### Defaults and Limitations