docs(snapshots): keep 14 days of snapshots, and async job cleanup timing

[spiceemma]

spiceai/spiceai#15199 makes the instance that publishes a dataset's snapshots keep only the snapshots of the last 14 days, plus the current and newest snapshot at any age, and delete the dropped files 24 hours later. It also changes async query job cleanup: failed and cancelled jobs are now deleted 12 hours after they finish, and each scheduler runs cleanup about once an hour instead of every 10 minutes. Before this change, the docs said snapshot storage grows until a bucket lifecycle rule expires it, recommended such a rule (24–72 hours in the read/write separation guide), and said the scheduler deletes only expired jobs, every 10 minutes.

Pages changed

• website/docs/features/data-acceleration/snapshots.md: new "Snapshot retention" section (14-day retention, current and newest kept, 24-hour file grace, hourly sweep of unnamed files, log line, rollback limited to listed snapshots, lifecycle-deleted files treated as failed downloads, current and newest entries kept when their file is gone); storage sentence (about 15 days of files) and the lifecycle best practice updated, including versioned and replicated buckets.
• website/docs/deployment/read-write-separation.md: retention guidance now points to Spice's own retention instead of a 24–72 hour lifecycle rule, with noncurrent-version and replication notes and a 15-day sizing estimate.
• website/docs/features/distributed-query/index.md: async job cleanup covers failed and cancelled jobs and runs about once an hour.

Edits on the snapshots page avoid the sections changed by the open #2379 (pin and unpin, which already documents the 409 Conflict for a snapshot whose file is gone) and the per-dataset snapshot settings sections.

Versioned docs

vNext only. The merge commit 9fcf078e is in no release tag, including v2.4.0-rc.1 and v2.4.0-rc.2; the page notes that v2.4.0-rc.2 and earlier keep every snapshot, so rc testers see older entries dropped at the first publish after upgrading.

Verification

spiceai/spiceai trunk at 9a25991b0384c7fc6b25fc838d6d71ce6f65ea57.

• Reproduced: 14-day retention — pub(super) const SNAPSHOT_RETENTION: Duration = Duration::from_hours(14 * 24); (crates/runtime-acceleration/src/snapshot/retention.rs:35)
• Reproduced: 24-hour file grace — pub(super) const FILE_GRACE: Duration = Duration::from_hours(24); (crates/runtime-acceleration/src/snapshot/retention.rs:41)
• Reproduced: log line — "Snapshot cleanup of dataset '{}' dropped {aged_out} snapshots older than {} and deleted {deleted} snapshot files." (crates/runtime-acceleration/src/snapshot/mod.rs:2229), with 14 days rendered as 14d by format!("{}d", secs / DAY) (crates/object_store_occ/src/cleanup.rs:64)
• Reproduced: dropped snapshot returns 404 — SnapshotApiError::SnapshotNotFound { .. } => ( mapped to StatusCode::NOT_FOUND (crates/runtime/src/http/v1/snapshots.rs:452)
• Reproduced: async job result TTL — pub const DEFAULT_RESULT_TTL: Duration = Duration::from_hours(12); (crates/runtime/src/jobs/state.rs:25)
• Unverified, code inspection only: failed and cancelled jobs reclaimable — matches!(self.status, JobStatus::Failed | JobStatus::Cancelled) (crates/runtime/src/jobs/state.rs:312)
• Unverified, code inspection only: cleanup pass 30–90 minutes — pub const PASS_INTERVAL: Duration = Duration::from_hours(1); (crates/object_store_occ/src/cleanup.rs:29) and Duration::from_millis(half.saturating_add(rand::random_range(0..=interval_ms))) (crates/object_store_occ/src/cleanup.rs:39)
• Unverified, code inspection only: current and newest snapshots kept at any age, entries dropped in the publishing write of metadata.json, files deleted at a publish at least 24 hours after the drop, hourly sweep of unnamed files at least 24 hours old, retention runs only on publish, the first publish after upgrade drops older entries.
• Reproduced: current and newest entries are checked before missing-file removal — if Some(entry.snapshot_id) == current || Some(entry.snapshot_id) == newest { (crates/runtime-acceleration/src/snapshot/retention.rs:111) precedes if gone.contains(&entry.snapshot) { (crates/runtime-acceleration/src/snapshot/retention.rs:114)
• Reproduced: Spice deletes without a version ID — match store.delete(path).await { (crates/object_store_occ/src/cleanup.rs:88); the noncurrent-version and replication guidance comes from the S3 documentation and was not run here
• Unverified, code inspection only: storage budget of about 15 days of files (14-day entry retention plus at least 24 hours before a later publish deletes a dropped file).
• Unverified, code inspection only: a missing snapshot file is a failed download handled by bootstrap_on_failure_behavior, and the writer drops an entry whose file is gone once the entry is at least 24 hours old.
• Observed, not isolated: retention and cleanup unit tests added in the source PR, feat(state): clean up unused shared state and keep 14 days of snapshots spiceai#15199
• build: npm run build in website/ passed locally ([SUCCESS] Generated static files in "build"., exit 0; no broken links or anchors reported)

[github-actions]

✅ Pull with Spice Passed

Passing checks:

• ✅ Title meets minimum length requirement (10 characters)
• ✅ Has at least one of the required labels: area/blog, area/docs, area/cookbook, dependencies
• ✅ No banned labels detected
• ✅ Has at least one assignee: spiceemma

[github-actions]

🔍 Pull with Spice Failed

Passing checks:

• ✅ Title meets minimum length requirement (10 characters)
• ✅ Has at least one of the required labels: area/blog, area/docs, area/cookbook, dependencies
• ✅ No banned labels detected

Failed checks:

• ❌ At least one assignee is required for this pull request.

Please address these issues and update your pull request.

[github-actions]

🔍 Pull with Spice Failed

Passing checks:

• ✅ Title meets minimum length requirement (10 characters)
• ✅ Has at least one of the required labels: area/blog, area/docs, area/cookbook, dependencies
• ✅ No banned labels detected

Failed checks:

• ❌ At least one assignee is required for this pull request.

Please address these issues and update your pull request.

[github-actions]

🚀 deployed to https://c9fc8a50.spiceai-org-website.pages.dev

[Copilot]

🟡 Changes recommended

Lifecycle guidance can leave versioned storage growing indefinitely, and storage estimates omit retained files.

3 open findings

• Account for snapshot files retained during the deletion grace period · New
• Document retention exceptions for current and newest snapshots · New
• Clarify lifecycle rules for versioned and replicated S3 buckets · New

What changed in this PR

Updates vNext documentation for automatic snapshot retention and revised asynchronous query job cleanup.

Changes:

• Documents 14-day snapshot retention, protected snapshots, and delayed file deletion.
• Replaces lifecycle-based retention recommendations with publisher-managed cleanup.
• Documents hourly job cleanup, including failed and cancelled jobs.

File	Description
website/​docs/​features/​distributed-query/​index.md	Updates job cleanup eligibility and timing.
website/​docs/​features/​data-acceleration/​snapshots.md	Explains retention, deletion grace, and upgrade behavior.
website/​docs/​deployment/​read-write-separation.md	Revises snapshot retention and storage guidance.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[spiceemma]

Copilot's review body lists only the 3 inline findings. All three are addressed in 42a5816 and their threads are resolved. There were no separate previously-missed items.

[Copilot]

🟢 Approved

The documentation addresses prior feedback and has no unresolved blocking findings.

0 open findings

3 resolved since last review

• Clarify lifecycle rules for versioned and replicated S3 buckets
• Document retention exceptions for current and newest snapshots
• Account for snapshot files retained during the deletion grace period

🧠 Review effort: Balanced

[github-actions]

🚀 deployed to https://cbba191d.spiceai-org-website.pages.dev