From 1a70a88e50c41751d5e94e48a818083dbd5d3919 Mon Sep 17 00:00:00 2001 From: Emma <98318442+spiceemma@users.noreply.github.com> Date: Fri, 16 Oct 2026 10:34:49 -0700 Subject: [PATCH 1/2] docs(snapshots): per-dataset snapshot settings; deprecate top-level snapshots and runtime.scheduler state settings --- website/docs/cli/reference/query.md | 2 +- .../data-acceleration/data-refresh.md | 8 +- .../refresh-modes/snapshot.md | 27 ++--- .../features/data-acceleration/snapshots.md | 108 ++++++++++++------ .../docs/features/distributed-query/index.md | 36 +++--- website/docs/reference/spicepod/datasets.md | 14 ++- website/docs/reference/spicepod/index.md | 10 +- website/docs/reference/spicepod/runtime.md | 29 +++-- 8 files changed, 150 insertions(+), 84 deletions(-) diff --git a/website/docs/cli/reference/query.md b/website/docs/cli/reference/query.md index 986cbd964..651c47843 100644 --- a/website/docs/cli/reference/query.md +++ b/website/docs/cli/reference/query.md @@ -84,5 +84,5 @@ Time: 5.10000000 seconds. 1 rows. Query IDs can be abbreviated if they uniquely identify a query within the tracked session. :::note -The `spice query` command requires the runtime to be running in [cluster mode](/docs/features/distributed-query) with `--role scheduler` and a scheduler state location configured (`runtime.scheduler.state_location`, or [`runtime.state.location`](../../reference/spicepod/runtime#runtimestate)). +The `spice query` command requires the runtime to be running in [cluster mode](/docs/features/distributed-query) with `--role scheduler` and a scheduler state location configured ([`runtime.state.location`](../../reference/spicepod/runtime#runtimestate), or the deprecated `runtime.scheduler.state_location`). ::: diff --git a/website/docs/features/data-acceleration/data-refresh.md b/website/docs/features/data-acceleration/data-refresh.md index 2513ec5ee..7c5a9c566 100644 --- a/website/docs/features/data-acceleration/data-refresh.md +++ b/website/docs/features/data-acceleration/data-refresh.md @@ -146,11 +146,9 @@ See [Caching Mode](./refresh-modes/caching) for detailed documentation and examp The `snapshot` refresh mode creates a read-only acceleration that reloads exclusively from the [snapshot store](./snapshots). The federated data source is never queried for refreshes — instead, the runtime polls the snapshot store on a configurable interval and atomically swaps in newer snapshots when available. ```yaml -snapshots: - enabled: true - location: s3://my-bucket/snapshots/ - params: - s3_auth: iam_role +runtime: + state: + location: s3://my-bucket/spice/ # snapshots are read from s3://my-bucket/spice/snapshots/ datasets: - from: postgres:public.my_table diff --git a/website/docs/features/data-acceleration/refresh-modes/snapshot.md b/website/docs/features/data-acceleration/refresh-modes/snapshot.md index 2fed29a3e..33041faf4 100644 --- a/website/docs/features/data-acceleration/refresh-modes/snapshot.md +++ b/website/docs/features/data-acceleration/refresh-modes/snapshot.md @@ -20,11 +20,9 @@ To read snapshots published to S3 without naming the original federated source, ## Configuration ```yaml -snapshots: - enabled: true - location: s3://my-bucket/snapshots/ - params: - s3_auth: iam_role +runtime: + state: + location: s3://my-bucket/spice/ # snapshots are read from s3://my-bucket/spice/snapshots/ datasets: - from: postgres:public.my_table @@ -64,15 +62,14 @@ Use `refresh_mode: snapshot` for read-only replicas that should not access the f ## Reload on S3 event notifications -A reader that only polls loads a new snapshot up to one `refresh_check_interval` after a writer publishes it. When the snapshot location is on Amazon S3, set `snapshots.params.s3_queue_url` to an Amazon SQS queue that receives the location's [S3 event notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html), and `refresh_mode: snapshot` datasets reload as soon as a new snapshot is published: +A reader that only polls loads a new snapshot up to one `refresh_check_interval` after a writer publishes it. When the snapshot location is on Amazon S3, set `s3_queue_url` in the location's params (`runtime.state.params`, or a dataset's [`snapshots_params`](../snapshots#override-the-snapshot-location-per-dataset)) to an Amazon SQS queue that receives the location's [S3 event notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html), and `refresh_mode: snapshot` datasets reload as soon as a new snapshot is published: ```yaml -snapshots: - enabled: true - location: s3://my-bucket/snapshots/ - params: - s3_auth: iam_role - s3_queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/spice-reader-1 +runtime: + state: + location: s3://my-bucket/spice/ + params: + s3_queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/spice-reader-1 datasets: - from: postgres:public.my_table @@ -90,13 +87,13 @@ A writer makes a snapshot current by rewriting the location's `metadata.json`, s Set up the queue as follows: -- **One queue per reader process.** SQS delivers each message to one receiver, so readers that share a queue each see only some notifications. To serve several readers, publish the bucket's notifications to an Amazon SNS topic and subscribe one queue per reader process. All `refresh_mode: snapshot` datasets in one process share that process's queue. -- **Only this location's notifications.** Filter the bucket notification by the location's prefix. A notification for an object outside the location is logged as an error and left on the queue, which delivers it again after each visibility timeout until it expires. +- **One queue per reader process.** SQS delivers each message to one receiver, so readers that share a queue each see only some notifications. To serve several readers, publish the bucket's notifications to an Amazon SNS topic and subscribe one queue per reader process. All `refresh_mode: snapshot` datasets in one process share that process's queue. Snapshot locations in one process that name the same queue share one consumer, which reads the queue with the credentials of the first location. A location that names the queue with other credentials logs a warning that names both locations. +- **Only this location's notifications.** Filter the bucket notification by the location's prefix. A notification for an object outside the location is logged as an error and left on the queue, which delivers it again after each visibility timeout until it expires. When the location is the `snapshots/` folder of `runtime.state.location`, the notification can cover the whole `runtime.state` prefix: notifications for the other [runtime state folders](../../../reference/spicepod/runtime#runtime-state-layout), such as the scheduler's `cluster/` folder, are deleted from the queue without a log. - **Permissions.** Spice reads the queue with the snapshot location's credentials: `s3_key` and `s3_secret` when both are set, otherwise the default AWS credential chain. The region comes from the queue URL. Grant `sqs:ReceiveMessage` and `sqs:DeleteMessage` on the queue. Datasets keep checking the location every `refresh_check_interval`, which covers notifications that never arrive. When Spice cannot reach SQS, it logs a warning when the outage starts, an error if it lasts five minutes, and an info message when it recovers. Datasets reload on `refresh_check_interval` in the meantime. -A queue URL that is empty, is an ARN, or is not an SQS queue URL, and a snapshot location that is not `s3://`, fail the dataset at load with an error that names `s3_queue_url`. When `s3_queue_url` is set but no dataset uses `refresh_mode: snapshot`, Spice logs a warning and does not read the queue. A dataset that reads snapshots with [`file_format: snapshot`](../snapshots#serve-a-dataset-from-published-snapshots) does not accept `s3_queue_url` and checks for new snapshots every `refresh_check_interval`. +A queue URL that is empty, is an ARN, or is not an SQS queue URL, and a snapshot location that is not `s3://`, fail the dataset at load with an error that names `s3_queue_url`. When `s3_queue_url` is set but no dataset uses `refresh_mode: snapshot`, Spice logs a warning that names the setting that holds the queue, such as `runtime.state.params.s3_queue_url`, and does not read the queue. A cluster scheduler, which loads no accelerations, does not log this warning. A dataset that reads snapshots with [`file_format: snapshot`](../snapshots#serve-a-dataset-from-published-snapshots) does not accept `s3_queue_url` and checks for new snapshots every `refresh_check_interval`. ## Related Topics diff --git a/website/docs/features/data-acceleration/snapshots.md b/website/docs/features/data-acceleration/snapshots.md index 3bcb090b2..1b79aa385 100644 --- a/website/docs/features/data-acceleration/snapshots.md +++ b/website/docs/features/data-acceleration/snapshots.md @@ -12,12 +12,11 @@ Acceleration Snapshots are available in the Spice [Enterprise edition](https://d ## Spicepod Example ```yaml -snapshots: - enabled: true - location: s3://some_bucket/some_folder/ - bootstrap_on_failure_behavior: warn - params: - s3_auth: iam_role +runtime: + state: + location: s3://some_bucket/some_folder/ + params: + s3_region: us-east-1 datasets: - name: some_table @@ -44,7 +43,7 @@ Acceleration snapshots let Spice reuse a pre-built acceleration file on startup Snapshots are organized with Hive-style partitioning so they are easy to retain and prune. For a dataset named `my_dataset` accelerated with DuckDB, Spice writes files such as: ```bash -s3://some_bucket/some_folder/month=2025-09/day=2025-09-19/dataset=my_dataset/my_dataset_20250919T134522Z_01996238f1157a59ae4d69b4f18b22a4.duckdb +s3://some_bucket/some_folder/snapshots/month=2025-09/day=2025-09-19/dataset=my_dataset/my_dataset_20250919T134522Z_01996238f1157a59ae4d69b4f18b22a4.duckdb ``` The timestamp is recorded in UTC using ISO 8601 without punctuation. The UUIDv7 after the timestamp gives each snapshot its own object name, so two snapshots written in the same second do not overwrite each other. The file extension names the acceleration engine that wrote the snapshot: `.duckdb`, `.sqlite`, `.cayenne`, or `.turso`. The engine is also recorded in the snapshot metadata, and a snapshot whose engine differs from the dataset's current engine is rejected at bootstrap rather than restored. @@ -61,23 +60,14 @@ A bootstrap, and a [`refresh_mode: snapshot`](./data-refresh#snapshot) reload, r When the workload needs incremental replication of changed data between regions or storage tiers, use a path that publishes the changed objects: -- **Cayenne cold / datalake tier.** [`cayenne_datalake_location`](../../components/data-accelerators/cayenne#cold-object-store-tier) graduates data onto standard object storage (a general-purpose S3 or S3-compatible bucket). Promotion carries unchanged cold files forward and rewrites the cold files a change can touch. The tier requires `refresh_mode: changes` or `append` and a primary key. It is a data tier, separate from `snapshots.location`. +- **Cayenne cold / datalake tier.** [`cayenne_datalake_location`](../../components/data-accelerators/cayenne#cold-object-store-tier) graduates data onto standard object storage (a general-purpose S3 or S3-compatible bucket). Promotion carries unchanged cold files forward and rewrites the cold files a change can touch. The tier requires `refresh_mode: changes` or `append` and a primary key. It is a data tier, separate from the snapshot location. - **Iceberg batched writes with merge-on-read.** [Iceberg writes](../../components/data-connectors/iceberg#write-support) append new data files, and [deletes](../../components/data-connectors/iceberg#deleting-rows) are equality delete files that readers merge on scan. A catalog commit publishes those objects. Use this path when the dataset lives in Iceberg and readers apply delete files during scan. Neither path bootstraps a file-mode accelerator. Keep snapshots for that. ## Configure snapshot storage -Snapshots are controlled with a top-level `snapshots` block in the Spicepod. The location can point to S3, Azure ADLS Gen2, Google Cloud Storage, or the local filesystem. - -```yaml -snapshots: - enabled: true - location: s3://some_bucket/some_folder/ # Folder where snapshots are written - bootstrap_on_failure_behavior: warn # retry | fallback | warn - params: - s3_auth: iam_role # Defaults to iam_role for snapshots -``` +Snapshots are stored in the `snapshots/` folder of [`runtime.state.location`](../../reference/spicepod/runtime#runtimestate), the object store that holds the Spicepod's shared runtime state, and use `runtime.state.params` (see [Store snapshots under `runtime.state`](#store-snapshots-under-runtimestate)). A dataset can store its snapshots elsewhere, with other params, by [setting its own snapshot location](#override-the-snapshot-location-per-dataset). The location can point to S3, Azure ADLS Gen2, Google Cloud Storage, or the local filesystem. ### Supported storage backends @@ -88,15 +78,15 @@ snapshots: | Google Cloud Storage | `gs://` | `GOOGLE_APPLICATION_CREDENTIALS`, Workload Identity | | Local filesystem | `file://` | N/A | -`location` must be a URI with a scheme — a bare filesystem path such as `/nvme/snapshots` is not a valid URI, so it fails to parse and snapshots are disabled with an error logged. Use `file:///nvme/snapshots/` for a local folder. +A snapshot location must be a URI with a scheme — a bare filesystem path such as `/nvme/snapshots` is not a valid URI, so it fails to parse and snapshots are disabled with an error logged. Use `file:///nvme/snapshots/` for a local folder. -When the location is an S3 bucket, `params` accepts these [S3 parameters](../../components/data-connectors/s3): `s3_region`, `s3_endpoint`, `s3_auth` (`iam_role` or `key`; default `iam_role`), `s3_key`, `s3_secret`, `s3_session_token`, `s3_queue_url` (an SQS queue that receives the location's S3 event notifications, so [`refresh_mode: snapshot` readers reload on publication](./refresh-modes/snapshot#reload-on-s3-event-notifications)), `client_timeout`, and `allow_http`. Spice ignores other keys and logs a warning for each. With `s3_auth: key`, both `s3_key` and `s3_secret` must be set; without them, Spice logs an error and does not use the snapshot location, rather than connecting with credentials from the environment. Azure locations also accept the Azure connector parameters under `params` for explicit credential overrides. GCS locations accept the [GCS data connector](../../components/data-connectors/gcs#authentication-parameters) parameter names, such as `gcs_service_account_key` and `gcs_skip_signature`, and validate them as a [`runtime.state` location](../../reference/spicepod/runtime#state-location-parameters) does. Spice v2.4.0-rc.1 and earlier ignored these names on a snapshot location. When no explicit credentials are supplied, Spice reads standard environment variables for each cloud provider. Values in `params` can reference [secrets](../../components/secret-stores) with `${secrets:}` for S3, Azure, and GCS locations. +When the location is an S3 bucket, its params accept these [S3 parameters](../../components/data-connectors/s3): `s3_region`, `s3_endpoint`, `s3_auth` (`iam_role` or `key`; default `iam_role`), `s3_key`, `s3_secret`, `s3_session_token`, `s3_queue_url` (an SQS queue that receives the location's S3 event notifications, so [`refresh_mode: snapshot` readers reload on publication](./refresh-modes/snapshot#reload-on-s3-event-notifications)), `client_timeout`, and `allow_http`. Spice ignores other keys and logs a warning for each. With `s3_auth: key`, both `s3_key` and `s3_secret` must be set; without them, Spice logs an error and does not use the snapshot location, rather than connecting with credentials from the environment. Azure locations also accept the Azure connector parameters under `params` for explicit credential overrides. GCS locations accept the [GCS data connector](../../components/data-connectors/gcs#authentication-parameters) parameter names, such as `gcs_service_account_key` and `gcs_skip_signature`, and validate them as a [`runtime.state` location](../../reference/spicepod/runtime#state-location-parameters) does. Spice v2.4.0-rc.1 and earlier ignored these names on a snapshot location. When no explicit credentials are supplied, Spice reads standard environment variables for each cloud provider. Values in `params` can reference [secrets](../../components/secret-stores) with `${secrets:}` for S3, Azure, and GCS locations. An invalid S3 parameter value, such as `s3_auth: public`, disables snapshots for the dataset. Spice logs an error that names the dataset, the location, and the parameter, and does not fall back to default parameters. Those defaults would drop every other configured parameter, including `s3_endpoint`, `s3_region`, and the credentials, so snapshots could be written to a different store or with a different identity than the one configured. When `s3_queue_url` is set, the dataset fails to register with the parameter error instead. Fix the parameter and restart Spice. ### Store snapshots under `runtime.state` -When the Spicepod sets [`runtime.state`](../../reference/spicepod/runtime#runtimestate) and `snapshots.location` is not set, snapshots go to the `snapshots/` folder of `runtime.state.location`. The top-level `snapshots` section is then optional: +When the Spicepod sets [`runtime.state`](../../reference/spicepod/runtime#runtimestate), datasets that enable snapshots store them in the `snapshots/` folder of `runtime.state.location`: ```yaml runtime: @@ -115,17 +105,65 @@ datasets: This fragment omits the PostgreSQL connection settings that the `orders` dataset needs; see the [PostgreSQL connector `params`](../../components/data-connectors/postgres/index.md#params). With them, this Spicepod publishes the `orders` snapshots under `s3://my-bucket/spice/snapshots/`, next to the other [runtime state folders](../../reference/spicepod/runtime#runtime-state-layout), in the same layout as any other snapshot location. -`snapshots.location`, when set, takes precedence over `runtime.state`. The snapshot location then uses `snapshots.params` alone. For a location taken from `runtime.state`, the snapshots use `runtime.state.params`, unless `snapshots.params` is set, which replaces them as a whole. A `snapshots` section without a `location` still sets `enabled`, `bootstrap_on_failure_behavior`, and `params`, so `snapshots.enabled: false` turns snapshots off for every dataset. - In v2.4.0-rc.1 and earlier, a dataset with `snapshots: enabled` and no `snapshots.location` created and restored no snapshots, and Spice logged `Snapshots enabled but no location configured` when the Spicepod had a `snapshots` section. With `runtime.state` set, such a dataset now publishes snapshots to `/snapshots/` and bootstraps from them. Set `acceleration.snapshots: disabled` on a dataset that should not. +### Override the snapshot location per dataset + +A dataset that needs another snapshot location, other params, or another failure behavior sets them in its own `acceleration` block: + +```yaml +runtime: + state: + location: s3://my-bucket/spice/ + params: + s3_region: us-east-1 + +datasets: + - from: postgres:events + name: events + acceleration: + enabled: true + engine: cayenne + mode: file + snapshots: enabled + snapshots_location: s3://archive-bucket/snapshots/ + snapshots_params: + s3_region: eu-west-1 + snapshots_bootstrap_on_failure_behavior: fallback +``` + +| Field | Default | +| ----- | ------- | +| `snapshots_location` | The `snapshots/` folder of `runtime.state.location`. | +| `snapshots_params` | The params of the location the dataset uses: none for a `snapshots_location`, and `runtime.state.params` for the `runtime.state` folder. | +| `snapshots_bootstrap_on_failure_behavior` | `warn`. See [Failure behavior](#failure-behavior). | + +`snapshots_params`, when set, are the only params the dataset's snapshots use. A dataset that sets `snapshots_location` without `snapshots_params` uses no params, so `runtime.state.params`, such as an `s3_endpoint` or credentials, never apply to a location they were not written for. When the dataset's location needs credentials or a region that the environment does not supply, set them in `snapshots_params`. + +These fields require a Spice version newer than v2.4.0-rc.2. Spice v2.4.0-rc.2 and earlier reject a Spicepod that sets them. + +### Migrate from the top-level `snapshots` section + +The top-level `snapshots` section is deprecated and will be removed in a future release. Spicepods that set it still load, and each of its settings applies to every dataset that does not set the matching `acceleration` field: + +| Deprecated setting | Replacement | +| ------------------ | ----------- | +| `snapshots.location` | `runtime.state.location`, whose `snapshots/` folder holds the snapshots, or `acceleration.snapshots_location` on each dataset. | +| `snapshots.params` | `runtime.state.params`, or `acceleration.snapshots_params` on each dataset. | +| `snapshots.bootstrap_on_failure_behavior` | `acceleration.snapshots_bootstrap_on_failure_behavior` on each dataset. | +| `snapshots.enabled` | No replacement. `acceleration.snapshots` defaults to `disabled`, so leave it unset, or set it to `disabled`, on each dataset that should not use snapshots. | + +When the section is set, `snapshots.params` apply to `snapshots.location`, and they replace `runtime.state.params` for snapshots stored under `runtime.state`. `snapshots.enabled: false` still turns snapshots off for every dataset. + +At startup, a Spicepod with the section logs one warning that shows the configuration that keeps the existing snapshots where they are. For example, when `snapshots.location` is `s3://my-bucket/spice/snapshots/` and `runtime.state` is not set, the warning asks to delete the section and set `runtime.state.location: s3://my-bucket/spice/`. When the section's location is already the `snapshots/` folder of `runtime.state.location`, the warning asks to delete the section. In other cases, it asks to set `acceleration.snapshots_location` to the section's location on each dataset that enables snapshots. The warning names the keys of `snapshots.params` to move, never their values. A cluster executor does not log it, because the scheduler that shares its Spicepod does. + ### Conditional writes on the snapshot location Each location holds one `metadata.json` document that records the snapshots of every dataset publishing there, from every Spice instance. Spice changes this document only with conditional writes. It creates the document only if it does not exist yet (`If-None-Match: *`), and updates it only if it still has the version Spice read (`If-Match` on its `ETag`). Google Cloud Storage expresses both conditions with the object generation (`x-goog-if-generation-match`). When another writer changes the document first, the store refuses the write, and Spice reads the document again and reapplies its change. Concurrent publishers therefore do not overwrite each other's entries. The location's store must enforce both conditions. Amazon S3, Azure ADLS Gen2, Google Cloud Storage, and a local `file://` directory do. An S3-compatible store must honor `If-None-Match` and `If-Match` on `PUT`; Spice checks this before it publishes, as described below. -Before each Spice instance creates a dataset's first snapshot, it probes the store by creating, updating, and deleting one `.spice-conditional-write-probe-` object under the location, in at most five requests. When the store rejects conditional writes, or accepts a write whose condition does not hold, Spice uploads no snapshot and does not write `metadata.json`. The snapshot fails with an error that names the dataset, the location, and the reason. Spice keeps this result and does not probe the store again, so restart Spice after moving `location` to a store that enforces conditional writes. +Before each Spice instance creates a dataset's first snapshot, it probes the store by creating, updating, and deleting one `.spice-conditional-write-probe-` object under the location, in at most five requests. When the store rejects conditional writes, or accepts a write whose condition does not hold, Spice uploads no snapshot and does not write `metadata.json`. The snapshot fails with an error that names the dataset, the location, and the reason. Spice keeps this result and does not probe the store again, so restart Spice after moving the snapshot location to a store that enforces conditional writes. When the probe cannot reach a result, for example after a transient error or because the credentials cannot write the probe object, the snapshot is not published and the next attempt probes again. The error asks for permission to create, update, and delete `.spice-conditional-write-probe-*` objects under the location. Every instance that publishes snapshots probes with its own credentials, and each needs permission to create and update these objects. Permission to delete them is needed only for cleanup: a failed delete does not change the probe result and leaves one small probe object behind. @@ -153,15 +191,15 @@ Each instance is identified by the `SPICE_INSTANCE_ID` environment variable, or ### Snapshot location and the Cayenne data tier -Keep `location` on standard object storage: Amazon S3 (`s3://`), Google Cloud Storage (`gs://`), or Azure ADLS Gen2 (`abfss://`, `abfs://`). A local `file://` folder is valid on a single machine. Standard cloud storage is what bucket and object replication, and readers in another region, use. +Keep the snapshot location on standard object storage: Amazon S3 (`s3://`), Google Cloud Storage (`gs://`), or Azure ADLS Gen2 (`abfss://`, `abfs://`). A local `file://` folder is valid on a single machine. Standard cloud storage is what bucket and object replication, and readers in another region, use. -`location` is independent of Cayenne's S3 Express One Zone data tier. `cayenne_file_path` and the `cayenne_s3_*` parameters (`cayenne_s3_region`, `cayenne_s3_zone_ids`, `cayenne_s3_auth`, and the related keys) store Cayenne Vortex files on an Express One Zone directory bucket. That bucket is single-zone storage for the accelerator's data files. It does not substitute for the snapshot bucket, and it does not provide the replication or multi-region reads a standard snapshot location does. See [S3 Express One Zone storage](../../components/data-accelerators/cayenne#aws-s3-express-one-zone-storage). +The snapshot location is independent of Cayenne's S3 Express One Zone data tier. `cayenne_file_path` and the `cayenne_s3_*` parameters (`cayenne_s3_region`, `cayenne_s3_zone_ids`, `cayenne_s3_auth`, and the related keys) store Cayenne Vortex files on an Express One Zone directory bucket. That bucket is single-zone storage for the accelerator's data files. It does not substitute for the snapshot bucket, and it does not provide the replication or multi-region reads a standard snapshot location does. See [S3 Express One Zone storage](../../components/data-accelerators/cayenne#aws-s3-express-one-zone-storage). -The Cayenne [cold tier](../../components/data-accelerators/cayenne#cold-object-store-tier) uses a general-purpose S3 or S3-compatible bucket, set by `cayenne_datalake_location`, not GCS or ADLS. That prefix is the cold data tier, separate from `snapshots.location`. +The Cayenne [cold tier](../../components/data-accelerators/cayenne#cold-object-store-tier) uses a general-purpose S3 or S3-compatible bucket, set by `cayenne_datalake_location`, not GCS or ADLS. That prefix is the cold data tier, separate from the snapshot location. ### Failure behavior -`bootstrap_on_failure_behavior` controls what Spice does when it cannot load the most recent snapshot. +`acceleration.snapshots_bootstrap_on_failure_behavior` controls what Spice does when it cannot load a dataset's most recent snapshot. - `retry` – keep retrying the newest snapshot until it succeeds. - `fallback` – try older snapshot files until one loads successfully. @@ -185,6 +223,9 @@ acceleration: snapshots_trigger_threshold: # threshold for time_interval or stream_batches snapshots_compaction: enabled | disabled # default: disabled (DuckDB only) snapshots_reset_expiry_on_load: enabled | disabled # default: disabled (DuckDB only with Caching refresh mode) + snapshots_location: # default: the snapshots/ folder of runtime.state.location + snapshots_params: # default: the params of that location + snapshots_bootstrap_on_failure_behavior: warn | retry | fallback # default: warn ``` ### Snapshot triggers @@ -327,12 +368,11 @@ acceleration: ## Complete example ```yaml -snapshots: - enabled: true - location: s3://some_bucket/some_folder/ - bootstrap_on_failure_behavior: warn - params: - s3_auth: iam_role +runtime: + state: + location: s3://some_bucket/some_folder/ + params: + s3_region: us-east-1 datasets: # Batch dataset with refresh-triggered snapshots diff --git a/website/docs/features/distributed-query/index.md b/website/docs/features/distributed-query/index.md index a0b5cf4f3..cb1e71d9b 100644 --- a/website/docs/features/distributed-query/index.md +++ b/website/docs/features/distributed-query/index.md @@ -160,23 +160,23 @@ EXPLAIN SELECT count(id) FROM my_dataset; For long-running queries, the async queries API enables submitting queries for background execution, polling for status, and retrieving paginated results when ready. :::warning -The async queries API is experimental and requires a scheduler state location: `runtime.scheduler.state_location`, or the shared [`runtime.state.location`](../reference/spicepod/runtime#runtimestate) when `state_location` is omitted. +The async queries API is experimental and requires a scheduler state location: the shared [`runtime.state.location`](../../reference/spicepod/runtime.md#runtimestate), or the deprecated `runtime.scheduler.state_location`. ::: ### Prerequisites - Spice runtime running in cluster mode with `--role scheduler` -- `runtime.scheduler.state_location` or `runtime.state.location` configured in the Spicepod (see [High Availability > Configuration](#configuration)) +- `runtime.state.location` configured in the Spicepod (see [High Availability > Configuration](#configuration)) - At least one executor node connected to the scheduler ### Enabling Async Queries -Configure `runtime.scheduler.state_location` in your `spicepod.yaml` to enable the async queries API. A [`runtime.state`](../reference/spicepod/runtime#runtimestate) location also works when `state_location` is omitted: +Configure [`runtime.state.location`](../../reference/spicepod/runtime.md#runtimestate) in your `spicepod.yaml` to enable the async queries API. The scheduler keeps its state in the location's `cluster/` folder: ```yaml runtime: - scheduler: - state_location: s3://my-bucket/spice-state + state: + location: s3://my-bucket/spice-state params: s3_region: us-east-1 ``` @@ -187,10 +187,12 @@ For local development: ```yaml runtime: - scheduler: - state_location: "file://.data/scheduler-state" + state: + location: file:///tmp/spice-state/ ``` +`runtime.scheduler.state_location` and `runtime.scheduler.params` are deprecated. A scheduler that sets them logs a startup warning that shows how to move to `runtime.state`. See [`runtime.scheduler`](../../reference/spicepod/runtime.md#runtimescheduler). + ### Query Ownership A query is owned by the principal that submitted it, and **only that principal can list, poll, fetch results from, or cancel it**. This applies to the HTTP, Arrow Flight, and CLI surfaces alike. @@ -641,7 +643,7 @@ When a query fails, the `error` object contains an `error_code` field: ### Storage Layout -Job state and result chunks are stored in the shared object store configured via `runtime.scheduler.state_location`, or `runtime.state.location` when that is omitted. `{base_prefix}` is the root of `state_location`, or the `cluster/` folder below the path of `runtime.state.location` (see [Runtime state layout](../../reference/spicepod/runtime.md#runtime-state-layout)): +Job state and result chunks are stored in the shared object store configured via `runtime.state.location`, or the deprecated `runtime.scheduler.state_location`. `{base_prefix}` is the `cluster/` folder below the path of `runtime.state.location`, or the root of `state_location` (see [Runtime state layout](../../reference/spicepod/runtime.md#runtime-state-layout)): ``` {base_prefix}/ @@ -664,7 +666,7 @@ Each scheduler deletes expired jobs, with their result chunks, every 10 minutes. | List limit | 100 queries | - Only available in cluster mode with `--role scheduler` -- Requires `runtime.scheduler.state_location` or `runtime.state.location` to be configured +- Requires `runtime.state.location`, or the deprecated `runtime.scheduler.state_location`, to be configured - The `format` query parameter on the results endpoint is declared but not yet implemented (results are always JSON over HTTP, Arrow IPC over Flight) - Result TTL is not yet configurable per-query (fixed at 12 hours) - Chunk size is not yet configurable per-query (fixed at 10,000 rows) @@ -702,12 +704,12 @@ In an HA cluster: ### Configuration -Enable HA by configuring `runtime.scheduler.state_location` (or the shared [`runtime.state.location`](../reference/spicepod/runtime#runtimestate)) in the Spicepod to point to an S3-compatible object store: +Enable HA by configuring the shared [`runtime.state.location`](../../reference/spicepod/runtime.md#runtimestate) in the Spicepod to point to an S3-compatible object store. The schedulers keep their shared state in its `cluster/` folder: ```yaml runtime: - scheduler: - state_location: s3://my-bucket/spice-cluster + state: + location: s3://my-bucket/spice params: s3_region: us-east-1 ``` @@ -726,7 +728,7 @@ This failover applies to async queries, which require a scheduler state location ### S3 Configuration -The `runtime.scheduler.params` section supports the following S3 parameters: +`runtime.state.params` supports the following S3 parameters for the scheduler's state: | Parameter | Description | Default | |--------------------| ----------------------------------------------------- | ---------- | @@ -743,8 +745,8 @@ Example with explicit credentials: ```yaml runtime: - scheduler: - state_location: s3://my-bucket/spice-cluster + state: + location: s3://my-bucket/spice params: s3_region: us-east-1 s3_auth: key @@ -762,8 +764,8 @@ A `gs://` state location takes the [GCS data connector](../../components/data-co ```yaml runtime: - scheduler: - state_location: s3://my-bucket/spice-cluster + state: + location: s3://my-bucket/spice params: s3_region: us-east-1 ``` diff --git a/website/docs/reference/spicepod/datasets.md b/website/docs/reference/spicepod/datasets.md index abc385225..bf7f375be 100644 --- a/website/docs/reference/spicepod/datasets.md +++ b/website/docs/reference/spicepod/datasets.md @@ -519,7 +519,7 @@ For the per-engine adjustments each profile applies, and for guidance on choosin ## `acceleration.snapshots` -Optional. Controls how this dataset participates in managed acceleration snapshots. Requires the Spicepod to configure the top-level [`snapshots` block](.#snapshots), the acceleration engine to be `duckdb`, `sqlite`, `cayenne`, or `turso`, and `mode: file` with a dataset-specific file path (for example `acceleration.params.duckdb_file: /nvme/my_dataset.db`). +Optional. Controls how this dataset participates in managed acceleration snapshots. Requires a snapshot location, from [`runtime.state.location`](./runtime.md#runtimestate) or [`acceleration.snapshots_location`](#accelerationsnapshots_location), the acceleration engine to be `duckdb`, `sqlite`, `cayenne`, or `turso`, and `mode: file` with a dataset-specific file path (for example `acceleration.params.duckdb_file: /nvme/my_dataset.db`). Supported values: @@ -588,6 +588,18 @@ Supported values: - `disabled` (default) – Snapshot expiry is not reset on load; the original expiry time is preserved. - `enabled` – Reset the snapshot expiry when it is loaded during bootstrap. +## `acceleration.snapshots_location` + +Optional. The object store location of this dataset's snapshots, for example `s3://my-bucket/spice/snapshots/`. Defaults to the `snapshots/` folder of [`runtime.state.location`](./runtime.md#runtimestate). The deprecated top-level [`snapshots.location`](./index.md#snapshotslocation), when set, applies to datasets that do not set this field. See [Override the snapshot location per dataset](../../features/data-acceleration/snapshots#override-the-snapshot-location-per-dataset). + +## `acceleration.snapshots_params` + +Optional. Object store params for this dataset's snapshots, such as `s3_region` or `s3_queue_url`. When set, they are the only params the dataset's snapshots use. When not set, a dataset with `snapshots_location` uses no params, and a dataset whose snapshots are stored under `runtime.state` uses `runtime.state.params`. The deprecated top-level [`snapshots.params`](./index.md#snapshotsparams), when set, apply to `snapshots.location` and replace `runtime.state.params` for snapshots stored under `runtime.state`. The accepted keys are listed in [Configure snapshot storage](../../features/data-acceleration/snapshots#configure-snapshot-storage). + +## `acceleration.snapshots_bootstrap_on_failure_behavior` + +Optional. What Spice does when it cannot load this dataset's most recent snapshot: `warn` (default), `retry`, or `fallback`. The deprecated top-level [`snapshots.bootstrap_on_failure_behavior`](./index.md#snapshotsbootstrap_on_failure_behavior), when set, applies to datasets that do not set this field. See [Failure behavior](../../features/data-acceleration/snapshots#failure-behavior). + ## `acceleration.refresh_mode` Optional. How to refresh the dataset. The following values are supported: diff --git a/website/docs/reference/spicepod/index.md b/website/docs/reference/spicepod/index.md index fee0df1d9..53cf4120b 100644 --- a/website/docs/reference/spicepod/index.md +++ b/website/docs/reference/spicepod/index.md @@ -118,6 +118,10 @@ datasets: ## `snapshots` {#snapshots} +:::warning[Deprecated] +The top-level `snapshots` section is deprecated and will be removed in a future release. Snapshots are stored in the `snapshots/` folder of [`runtime.state.location`](./runtime.md#runtimestate). A dataset overrides the location, params, and failure behavior with [`acceleration.snapshots_location`](./datasets.md#accelerationsnapshots_location), [`acceleration.snapshots_params`](./datasets.md#accelerationsnapshots_params), and [`acceleration.snapshots_bootstrap_on_failure_behavior`](./datasets.md#accelerationsnapshots_bootstrap_on_failure_behavior). A Spicepod that sets the section logs a startup warning that shows the replacement configuration. See [Migrate from the top-level `snapshots` section](../../features/data-acceleration/snapshots.md#migrate-from-the-top-level-snapshots-section). +::: + Optional. Configure managed acceleration snapshots that Spice can use to bootstrap file-based accelerations. When enabled, datasets that opt in with [`acceleration.snapshots`](spicepod/datasets#accelerationsnapshots) will download database files from the snapshot location if the local file is missing, and will optionally write new snapshots after each refresh. DuckDB, SQLite, Cayenne, and Turso accelerations running in `mode: file` are supported, and each dataset must write to its own file path. ```yaml @@ -137,7 +141,7 @@ Enable or disable snapshot management globally. Defaults to `true`. The folder where snapshots are stored. Supports S3 bucket URIs (`s3://bucket/prefix/`), Azure ADLS Gen2 URIs (`abfss://container@account.dfs.core.windows.net/path/`), Google Cloud Storage URIs (`gs://bucket/prefix/`), and local filesystem URIs (`file:///path/to/folder/`). The value must be a URI with a scheme: a bare filesystem path such as `/var/spice/snapshots` is not a valid URI, so it fails to parse and snapshots are disabled with an error logged. The location must resolve to a single folder; Spice creates per-dataset folders underneath using Hive-style partitions (`month=YYYY-MM/day=YYYY-MM-DD/dataset=`). -When `location` is not set and the Spicepod sets [`runtime.state`](spicepod/runtime#runtimestate), snapshots go to `/snapshots/`. A `location` that is set takes precedence over `runtime.state`. See [Store snapshots under `runtime.state`](../features/data-acceleration/snapshots#store-snapshots-under-runtimestate). +When `location` is not set and the Spicepod sets [`runtime.state`](spicepod/runtime#runtimestate), snapshots go to `/snapshots/`. A `location` that is set takes precedence over `runtime.state`, and a dataset's `acceleration.snapshots_location` takes precedence over `location`. See [Store snapshots under `runtime.state`](../features/data-acceleration/snapshots#store-snapshots-under-runtimestate). The location's store must enforce conditional writes (create only if absent, update only at the version read), because every publisher updates the location's shared `metadata.json` with them. Amazon S3, Azure ADLS Gen2, Google Cloud Storage, and local `file://` directories do. Each Spice instance probes the store before it creates a dataset's first snapshot, and refuses a store that does not enforce them. The probe creates, updates, and then deletes a `.spice-conditional-write-probe-` object under the location. The credentials of every instance that publishes snapshots need permission to create and update that object. Permission to delete it is needed only for cleanup: a failed delete does not change the probe result and leaves the probe object behind. See [Conditional writes on the snapshot location](../features/data-acceleration/snapshots#conditional-writes-on-the-snapshot-location). @@ -145,7 +149,7 @@ Keep a cloud `location` on standard S3, GCS, or ADLS so bucket or object replica ### `snapshots.bootstrap_on_failure_behavior` -Controls what happens when Spice cannot load the most recent snapshot on startup. Valid values: +Controls what happens when Spice cannot load the most recent snapshot on startup, for datasets that do not set `acceleration.snapshots_bootstrap_on_failure_behavior`. Valid values: - `warn` (default) – Log a warning and continue with an empty acceleration. - `retry` – Retry the newest snapshot until it loads successfully. @@ -155,7 +159,7 @@ Controls what happens when Spice cannot load the most recent snapshot on startup Optional key-value map passed to the snapshot storage layer. When `location` points to S3, `params` accepts these [S3 parameters](../components/data-connectors/s3): `s3_region`, `s3_endpoint`, `s3_auth` (`iam_role` or `key`; default `iam_role`), `s3_key`, `s3_secret`, `s3_session_token`, `s3_queue_url`, `client_timeout`, and `allow_http`. Spice ignores other keys and logs a warning for each. Snapshots default to `s3_auth: iam_role`, which differs from the S3 dataset default of `public`. With `s3_auth: key`, both `s3_key` and `s3_secret` must be set, or the snapshot location is not used and Spice logs an error. Azure ADLS locations also accept the Azure connector parameters, and GCS locations the GCS connector parameter names (such as `gcs_service_account_key`), for explicit credential overrides; when no overrides are supplied, Spice reads standard environment variables for each cloud provider. Values can reference secrets with `${secrets:}` for all three. An invalid S3 parameter value, such as `s3_auth: public`, disables snapshots for the dataset with an error; Spice does not fall back to default parameters. When `s3_queue_url` is set, the dataset fails to register with the parameter error instead. -For snapshots stored under `runtime.state`, `params` replaces `runtime.state.params` as a whole. When `params` is not set, those snapshots use `runtime.state.params`. +For snapshots stored under `runtime.state`, `params` replaces `runtime.state.params` as a whole. When `params` is not set, those snapshots use `runtime.state.params`. A dataset that sets `acceleration.snapshots_params` uses those instead, and a dataset that sets `acceleration.snapshots_location` without `snapshots_params` uses no params. ## `models` diff --git a/website/docs/reference/spicepod/runtime.md b/website/docs/reference/spicepod/runtime.md index 27f51266c..935bd5761 100644 --- a/website/docs/reference/spicepod/runtime.md +++ b/website/docs/reference/spicepod/runtime.md @@ -1070,10 +1070,11 @@ Configures the cluster scheduler when running Spice in [cluster mode](../../depl ```yaml runtime: - scheduler: - state_location: s3://my-bucket/spice-cluster-state/ + state: + location: s3://my-bucket/spice-state/ params: s3_region: us-east-1 + scheduler: partition_assignment_interval: 30s max_partition_assignments_per_interval: 100 max_partitions_per_executor: 1000 @@ -1082,13 +1083,23 @@ runtime: | Parameter name | Optional | Default | Description | | -------------------------------------------------- | -------- | ------- | ---------------------------------------------------------------------- | -| `state_location` | Yes | - | Root URI for shared cluster state storage (e.g. `s3://bucket/path/`). Required unless [`runtime.state`](#runtimestate) is set, in which case `runtime.state.location` is used. | -| `params` | Yes | - | Object store parameters (e.g. `s3_region`). See [State location parameters](#state-location-parameters). When `state_location` is omitted, defaults to `runtime.state.params`. | +| `state_location` | Yes | - | Deprecated. Root URI for shared cluster state storage (e.g. `s3://bucket/path/`). Set [`runtime.state.location`](#runtimestate) instead. | +| `params` | Yes | - | Deprecated. Object store parameters (e.g. `s3_region`). See [State location parameters](#state-location-parameters). Set `runtime.state.params` instead. When `state_location` is omitted, defaults to `runtime.state.params`. | | `partition_assignment_interval` | Yes | `30s` | How often the scheduler runs partition assignment cycles. | | `max_partition_assignments_per_interval` | Yes | `100` | Maximum number of partition assignments per interval. | | `max_partitions_per_executor` | Yes | `1000` | Maximum number of partitions assigned to a single executor. | | `partition_discovery_timeout` | Yes | `60s` | How long the scheduler waits for executor discovery before timing out. | +The scheduler keeps its shared cluster state in the `cluster/` folder of [`runtime.state.location`](#runtimestate) and reads it with `runtime.state.params`. `runtime.scheduler.state_location` and `runtime.scheduler.params` are deprecated and will be removed in a future release. They still take precedence over `runtime.state`, and a scheduler or standalone instance that sets either one logs a startup warning that shows how to move to `runtime.state`: + +| Current setting | What the warning says | +| --------------- | --------------------- | +| `state_location` is the `cluster/` folder of `runtime.state.location` | Remove `state_location`: the state is already there. | +| `state_location` ends in `cluster/` and `runtime.state` is not set | Remove `state_location` and set `runtime.state.location` to its parent, for example `s3://my-bucket/spice/` for `s3://my-bucket/spice/cluster/`, which keeps the state where it is. | +| Any other `state_location`, and every `abfs://` or `abfss://` `state_location` | No `runtime.state` location holds the current state. Removing `state_location` (and setting `runtime.state.location`, when it is not set) makes the scheduler start empty in the new folder, so partition assignments are rebuilt and async query jobs are lost. An `abfs://` or `abfss://` `state_location` keeps its state at the container root, which is never a `runtime.state` folder. | + +The warning also asks to move the keys of `runtime.scheduler.params` into `runtime.state.params`. When `runtime.state.params` are already set, it asks to remove `runtime.scheduler.params` instead, because `runtime.state.params` serve every feature that keeps state under `runtime.state`, and to check that they give the scheduler access to its state. The warning names parameter keys, never their values. + ## `runtime.state` Optional. Sets one shared object store for runtime state, so each feature that persists state does not need its own location. Supported URI schemes are `file://`, `s3://`, `gs://` (or its alias `gcs://`), `abfs://`, and `abfss://`. @@ -1112,9 +1123,9 @@ The following features store their state under `runtime.state` when their own se | Feature | Uses `runtime.state` when | | ------- | ------------------------- | | [SQL results cache warmup](../../features/caching#warming-the-cache-after-a-restart) | `runtime.caching.sql_results.warmup` is `on_first_refresh`. Without `runtime.state`, recorded query plans are written to `.spice/data/results_cache_warmup.json`. | -| [Cluster scheduler](#runtimescheduler) | `runtime.scheduler.state_location` is not set. The scheduler's other settings still apply. | +| [Cluster scheduler](#runtimescheduler) | The deprecated `runtime.scheduler.state_location` is not set. The scheduler's other settings still apply. | | [HTTP rate control](#limits-across-instances) | Always, in a Spice.ai Enterprise build: instances that share the location coordinate per-second and per-minute limits for every origin that sets one. Open-source builds keep rate-control state in memory. Rate control has no location of its own. | -| [Acceleration snapshots](../../features/data-acceleration/snapshots#store-snapshots-under-runtimestate) | A dataset enables snapshots and `snapshots.location` is not set. Snapshots are available in Spice.ai Enterprise. | +| [Acceleration snapshots](../../features/data-acceleration/snapshots#store-snapshots-under-runtimestate) | A dataset enables snapshots and sets no `acceleration.snapshots_location`, and the deprecated top-level `snapshots.location` is not set. Snapshots are available in Spice.ai Enterprise. | A location set in a feature's own section takes precedence over `runtime.state`. Changing `runtime.state` requires a restart. @@ -1138,7 +1149,7 @@ For example, `spice-state/cluster.json` moves to `spice-state/cluster/cluster.js Move the rate-control objects first, one spicepod folder at a time, and move only the objects named `-<16 hex digits>.json` in that folder. A spicepod name can match another folder in the table, such as `heartbeats`, `jobs`, `scheduler`, `cluster`, `rate-control`, or `results-cache`, and moving whole folders would then mix rate-control objects with scheduler objects. Move the scheduler and warmup objects after that. -To keep the scheduler's previous layout instead of moving its objects, set `runtime.scheduler.state_location` to the old location. A scheduler with its own `state_location` does not use `runtime.state.params`, so also copy any parameters the location needs, such as credentials or an endpoint, into `runtime.scheduler.params`. Rate control and warmup have no location of their own, so their objects must be moved. Otherwise, the scheduler starts with fresh cluster state, rate control starts with no shared usage, and warmup records new query shapes. +To keep the scheduler's previous layout instead of moving its objects, set the deprecated `runtime.scheduler.state_location` to the old location. A scheduler with its own `state_location` does not use `runtime.state.params`, so also copy any parameters the location needs, such as credentials or an endpoint, into `runtime.scheduler.params`. Rate control and warmup have no location of their own, so their objects must be moved. Otherwise, the scheduler starts with fresh cluster state, rate control starts with no shared usage, and warmup records new query shapes. ::: @@ -1160,7 +1171,9 @@ v2.4.0-rc.1 did not check the location. After an upgrade from rc.1, an instance ### State location parameters -These rules apply to `runtime.state.params`, and to the `params` of a feature that sets its own state location, such as [`runtime.scheduler`](#runtimescheduler). A [snapshot location](../../features/data-acceleration/snapshots#configure-snapshot-storage) set with `snapshots.location` takes its parameters from `snapshots.params`. Snapshots stored under `runtime.state` use `runtime.state.params`, unless `snapshots.params` is set, which replaces them. +These rules apply to `runtime.state.params`, and to the `params` of a feature that sets its own state location, such as [`runtime.scheduler`](#runtimescheduler). Snapshots stored under `runtime.state` use `runtime.state.params`. A dataset that sets [`acceleration.snapshots_params`](./datasets.md#accelerationsnapshots_params) uses those instead, and a dataset that sets `acceleration.snapshots_location` without `snapshots_params` uses no params. A location set with the deprecated top-level `snapshots.location` takes its parameters from `snapshots.params`, and `snapshots.params`, when set, also replace `runtime.state.params` for snapshots stored under `runtime.state`. See [Configure snapshot storage](../../features/data-acceleration/snapshots#configure-snapshot-storage). + +`runtime.state.params` can hold `s3_queue_url`, the SQS queue for [snapshot notifications](../../features/data-acceleration/refresh-modes/snapshot#reload-on-s3-event-notifications). The scheduler, HTTP rate control, and SQL results cache warmup accept it there without a warning and do not use it. A feature's own `params`, such as `runtime.scheduler.params`, still log a warning for it. An `s3://` location takes `s3_region`, `s3_endpoint`, `s3_auth`, `s3_key`, `s3_secret`, `s3_session_token`, `client_timeout`, and `allow_http`. `s3_auth` is `iam_role` (the default) or `key`. With `s3_auth: key`, both `s3_key` and `s3_secret` must be set. A location that sets `s3_auth: key` without both keys fails instead of connecting with the credentials from the environment, and the error reads: From a71b7aeafb85bd47e1adc7f1584741fb60f2efe9 Mon Sep 17 00:00:00 2001 From: Emma <98318442+spiceemma@users.noreply.github.com> Date: Fri, 16 Oct 2026 11:18:44 -0700 Subject: [PATCH 2/2] docs(snapshots): reader examples use snapshots_location; queue filters cover every location; list deprecated snapshots.location --- .../features/data-acceleration/data-refresh.md | 5 +---- .../refresh-modes/snapshot.md | 18 +++++++----------- website/docs/reference/spicepod/datasets.md | 2 +- 3 files changed, 9 insertions(+), 16 deletions(-) diff --git a/website/docs/features/data-acceleration/data-refresh.md b/website/docs/features/data-acceleration/data-refresh.md index 7c5a9c566..6b81959ae 100644 --- a/website/docs/features/data-acceleration/data-refresh.md +++ b/website/docs/features/data-acceleration/data-refresh.md @@ -146,10 +146,6 @@ See [Caching Mode](./refresh-modes/caching) for detailed documentation and examp The `snapshot` refresh mode creates a read-only acceleration that reloads exclusively from the [snapshot store](./snapshots). The federated data source is never queried for refreshes — instead, the runtime polls the snapshot store on a configurable interval and atomically swaps in newer snapshots when available. ```yaml -runtime: - state: - location: s3://my-bucket/spice/ # snapshots are read from s3://my-bucket/spice/snapshots/ - datasets: - from: postgres:public.my_table name: my_table @@ -160,6 +156,7 @@ datasets: refresh_mode: snapshot refresh_check_interval: 30s # Poll interval; defaults to 1m snapshots: enabled + snapshots_location: s3://my-bucket/spice/snapshots/ # The writer's snapshot folder params: duckdb_file: /nvme/my_table.db ``` diff --git a/website/docs/features/data-acceleration/refresh-modes/snapshot.md b/website/docs/features/data-acceleration/refresh-modes/snapshot.md index 33041faf4..a083567db 100644 --- a/website/docs/features/data-acceleration/refresh-modes/snapshot.md +++ b/website/docs/features/data-acceleration/refresh-modes/snapshot.md @@ -20,10 +20,6 @@ To read snapshots published to S3 without naming the original federated source, ## Configuration ```yaml -runtime: - state: - location: s3://my-bucket/spice/ # snapshots are read from s3://my-bucket/spice/snapshots/ - datasets: - from: postgres:public.my_table name: my_table @@ -34,10 +30,13 @@ datasets: refresh_mode: snapshot refresh_check_interval: 30s # Poll interval; defaults to 1m snapshots: enabled + snapshots_location: s3://my-bucket/spice/snapshots/ # The writer's snapshot folder params: duckdb_file: /nvme/my_table.db ``` +This reader sets [`acceleration.snapshots_location`](../snapshots#override-the-snapshot-location-per-dataset) to the folder the writer publishes to, so it needs only read access to the snapshots. It could instead set [`runtime.state`](../../../reference/spicepod/runtime#runtimestate) to the writer's state location, whose `snapshots/` folder holds the snapshots. An instance with `runtime.state` checks the location at startup, which needs write access under `_spice/`, and does not report ready until the [check](../../../reference/spicepod/runtime#startup-check-of-the-state-location) passes, even though snapshot reads do not wait for it. + ## Requirements - `acceleration.snapshots` must be `enabled` or `bootstrap_only`. Snapshot mode is a snapshot *consumer* only, so the two behave identically here: creation is skipped for the mode entirely and the dataset never publishes new snapshots — a separate writer must produce them. @@ -65,12 +64,6 @@ Use `refresh_mode: snapshot` for read-only replicas that should not access the f A reader that only polls loads a new snapshot up to one `refresh_check_interval` after a writer publishes it. When the snapshot location is on Amazon S3, set `s3_queue_url` in the location's params (`runtime.state.params`, or a dataset's [`snapshots_params`](../snapshots#override-the-snapshot-location-per-dataset)) to an Amazon SQS queue that receives the location's [S3 event notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html), and `refresh_mode: snapshot` datasets reload as soon as a new snapshot is published: ```yaml -runtime: - state: - location: s3://my-bucket/spice/ - params: - s3_queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/spice-reader-1 - datasets: - from: postgres:public.my_table name: my_table @@ -81,6 +74,9 @@ datasets: refresh_mode: snapshot refresh_check_interval: 10m # Fallback check for missed notifications snapshots: bootstrap_only + snapshots_location: s3://my-bucket/spice/snapshots/ + snapshots_params: + s3_queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/spice-reader-1 ``` A writer makes a snapshot current by rewriting the location's `metadata.json`, so an `ObjectCreated` notification for that object triggers the reload. Spice reads `metadata.json` once for each batch of such notifications it receives and reloads only the datasets whose current snapshot changed, with the same snapshot id, schema, and checksum checks as a poll. Notifications for snapshot files and for removed objects are deleted from the queue without a reload. A reader that is still waiting for its first snapshot restores it when the notification arrives. @@ -88,7 +84,7 @@ A writer makes a snapshot current by rewriting the location's `metadata.json`, s Set up the queue as follows: - **One queue per reader process.** SQS delivers each message to one receiver, so readers that share a queue each see only some notifications. To serve several readers, publish the bucket's notifications to an Amazon SNS topic and subscribe one queue per reader process. All `refresh_mode: snapshot` datasets in one process share that process's queue. Snapshot locations in one process that name the same queue share one consumer, which reads the queue with the credentials of the first location. A location that names the queue with other credentials logs a warning that names both locations. -- **Only this location's notifications.** Filter the bucket notification by the location's prefix. A notification for an object outside the location is logged as an error and left on the queue, which delivers it again after each visibility timeout until it expires. When the location is the `snapshots/` folder of `runtime.state.location`, the notification can cover the whole `runtime.state` prefix: notifications for the other [runtime state folders](../../../reference/spicepod/runtime#runtime-state-layout), such as the scheduler's `cluster/` folder, are deleted from the queue without a log. +- **Only the notifications of the locations on the queue.** Filter the bucket notification by the prefix of each snapshot location that names the queue. A notification for an object outside every one of those locations is logged as an error and left on the queue, which delivers it again after each visibility timeout until it expires. When the location is the `snapshots/` folder of `runtime.state.location`, the notification can cover the whole `runtime.state` prefix: notifications for the other [runtime state folders](../../../reference/spicepod/runtime#runtime-state-layout), such as the scheduler's `cluster/` folder, are deleted from the queue without a log. - **Permissions.** Spice reads the queue with the snapshot location's credentials: `s3_key` and `s3_secret` when both are set, otherwise the default AWS credential chain. The region comes from the queue URL. Grant `sqs:ReceiveMessage` and `sqs:DeleteMessage` on the queue. Datasets keep checking the location every `refresh_check_interval`, which covers notifications that never arrive. When Spice cannot reach SQS, it logs a warning when the outage starts, an error if it lasts five minutes, and an info message when it recovers. Datasets reload on `refresh_check_interval` in the meantime. diff --git a/website/docs/reference/spicepod/datasets.md b/website/docs/reference/spicepod/datasets.md index bf7f375be..0888a2e9d 100644 --- a/website/docs/reference/spicepod/datasets.md +++ b/website/docs/reference/spicepod/datasets.md @@ -519,7 +519,7 @@ For the per-engine adjustments each profile applies, and for guidance on choosin ## `acceleration.snapshots` -Optional. Controls how this dataset participates in managed acceleration snapshots. Requires a snapshot location, from [`runtime.state.location`](./runtime.md#runtimestate) or [`acceleration.snapshots_location`](#accelerationsnapshots_location), the acceleration engine to be `duckdb`, `sqlite`, `cayenne`, or `turso`, and `mode: file` with a dataset-specific file path (for example `acceleration.params.duckdb_file: /nvme/my_dataset.db`). +Optional. Controls how this dataset participates in managed acceleration snapshots. Requires a snapshot location, from [`runtime.state.location`](./runtime.md#runtimestate), [`acceleration.snapshots_location`](#accelerationsnapshots_location), or the deprecated top-level [`snapshots.location`](./index.md#snapshotslocation), the acceleration engine to be `duckdb`, `sqlite`, `cayenne`, or `turso`, and `mode: file` with a dataset-specific file path (for example `acceleration.params.duckdb_file: /nvme/my_dataset.db`). Supported values: