diff --git a/website/versioned_docs/version-2.4.x/api/HTTP/post-search.RequestSchema.json b/website/versioned_docs/version-2.4.x/api/HTTP/post-search.RequestSchema.json index d92d5dbda..b615b7fda 100644 --- a/website/versioned_docs/version-2.4.x/api/HTTP/post-search.RequestSchema.json +++ b/website/versioned_docs/version-2.4.x/api/HTTP/post-search.RequestSchema.json @@ -1 +1 @@ -{"title":"Body","body":{"description":"Search request parameters","content":{"application/json":{"schema":{"allOf":[{"type":"object","required":["text"],"properties":{"additional_columns":{"type":"array","items":{"type":"string"},"description":"Additional columns to return from the dataset. If the column is a primary key, it will be\n returned within the response under `.primary_key`, not `.data`."},"datasets":{"type":["array","null"],"items":{"type":"string"},"description":"The datasets to search for similarity. If None, search across all datasets. For available datasets, use the `list_datasets` tool and ensure `can_search_documents==true`."},"limit":{"type":["integer","null"],"description":"Number of documents to return for each dataset","minimum":0},"text":{"type":"string","description":"The text to search documents for similarity"},"where":{"type":["string","null"],"description":"An SQL filter predicate to apply. Format: 'WHERE `where_cond`'."}},"title":"SearchRequestBaseJson"},{"type":"object","properties":{"keywords":{"type":["array","null"],"items":{"type":"string"}}}}],"description":"HTTP request schema is separate from AI requests, so that keywords can be supplied as an optional field for HTTP calls.\n`schemars` doesn't allow setting `#[serde(default)]` as well as `#[schemars(required)]` - the field does not become required.\nWhen the field is not required, the model ignores it.","title":"SearchRequestHTTPJson"},"example":{"datasets":["app_messages"],"text":"Tokyo plane tickets","where":"user=1234321","additional_columns":["timestamp"],"limit":3,"keywords":["plane","tickets"]}}},"required":true}} \ No newline at end of file +{"title":"Body","body":{"description":"Search request parameters","content":{"application/json":{"schema":{"allOf":[{"type":"object","required":["text"],"properties":{"additional_columns":{"type":"array","items":{"type":"string"},"description":"Additional columns to return from the dataset. If the column is a primary key, it will be\n returned within the response under `.primary_key`, not `.data`."},"datasets":{"type":["array","null"],"items":{"type":"string"},"description":"The datasets to search for similarity. If None, search across all datasets. For available datasets, use the `list_datasets` tool and ensure `can_search_documents==true`."},"limit":{"type":["integer","null"],"description":"Number of documents to return for each dataset","minimum":0},"text":{"type":"string","description":"The text to search documents for similarity. Must contain at least one non-whitespace character."},"where":{"type":["string","null"],"description":"An SQL filter predicate to apply. Format: 'WHERE `where_cond`'."}},"title":"SearchRequestBaseJson"},{"type":"object","properties":{"keywords":{"type":["array","null"],"items":{"type":"string"}}}}],"description":"HTTP request schema is separate from AI requests, so that keywords can be supplied as an optional field for HTTP calls.\n`schemars` doesn't allow setting `#[serde(default)]` as well as `#[schemars(required)]` - the field does not become required.\nWhen the field is not required, the model ignores it.","title":"SearchRequestHTTPJson"},"example":{"datasets":["app_messages"],"text":"Tokyo plane tickets","where":"user=1234321","additional_columns":["timestamp"],"limit":3,"keywords":["plane","tickets"]}}},"required":true}} \ No newline at end of file diff --git a/website/versioned_docs/version-2.4.x/api/HTTP/post-search.api.mdx b/website/versioned_docs/version-2.4.x/api/HTTP/post-search.api.mdx index 75bb04ab7..c4cbe3eda 100644 --- a/website/versioned_docs/version-2.4.x/api/HTTP/post-search.api.mdx +++ b/website/versioned_docs/version-2.4.x/api/HTTP/post-search.api.mdx @@ -5,7 +5,7 @@ description: "Perform a vector similarity search (VSS) operation on a dataset." sidebar_label: "Search" hide_title: true hide_table_of_contents: true -api: eJztWFtv2zgW/isH3Ie2gOzYjp04AvrQHXQxWXQ62SbdwcIJYko8jjmhSA1J2TUM//fFISVZTpxetq+bl8C8nOt3vnOoLfP8wbF0xq6R23zJ7hIm0OVWll4azVJ2hXZhbAEcVph7Y8HJQipupd+AC3fg9b+vr9+AKdFyugRGAwfBPXfo+7f6Vt8ssTm7P7WWSoFFX1kNfolQGOfBosIV1x4K7vMlOsi4Q0ESc+Okxq72tfTLcFPqsvIw9/jFz/tRW63dwV8VWokO3NJUSsCSrxC4BiwyFELqB8iNqgqdANciCONlaU1pJffYOVUYgQqU4QJFnyWsdeNSsJSVxvn76CBLmMW/KnT+70ZsWLp9Es4YZqjPQMktL9CjdSxhudEetadLvCyVzIOGkz8d3dwyly+x4GFXqd8XLJ1tmd+UyFJmsj8x97VuaVFQRikelE/yB62X6MJdISSJ5eo+uh5WazncWr5hCZMei+6681bqB7Z7Co53rbA6jg68aZK6sKYIIW2gAJeL8DseBemAQ2llwe0GHnGTgPQRFRne6loKipBnGTFi0ZVGO4RKC7Qw79fX7x9xM09AGw/zPqmb94OxNQr2nsxaF3WlFEXne109QJU3DZ4XBxURXPxoNCbNPs+tcQ64Uu3tPvzDWOArLhXP1F5qApXD4OZcSefvm/U5eGNUAChqV1mEec51jbd7YfKqQO3d27feVhgdV7KQvuu11B4f0Hb8PvTuY1VkaMEsoJXXzaSxgDxfNqayhBVSy6IqWDrYJRFpz0J4LIJ0shO9vbLDOJIP6yVa7PrQij3uwjsN1//6AAupPFooLQoqICRtVE2bEPaC+xRe/fHr+0/vYR403OdGi/mrPtuRI9IrbKv0U13I3OE/qQZ3yfOCOyyuR9ysjRX/E+B2u90zn369ublquSLWP5WNQ+INj7HE3l02R1wCzoBfcg+NJZBzDRmCq4hRUAB3xH+mrMt2IVGJEP2gK+dKuf6tnkdl1s1BGHT6lScImzU49J4Icf63mUMr8LXABa+Uf3M3J9FrVIr+03Yt4XXDSXSkF/AdlZLgULIZ5qZAaM71b/UfS9SdkzKeaw4kdb8gSpYP2lh0ID3R8tH8kWN1/hh+4UWpAqz25DAjur0v0Dn+gI5yFAHNbszjxkCpuEbwMn+k0y0wWeXQvh2OTsenoyFLjhLrjHlZoPO8KElsXZanSQcoMxbkB+OjhrsdQXHP5FTWYSFyX8DOaDB4sbfkhlz0KMBVeY7OLSqlNj/SX77WVSy6SoU4iCq2wPvCPW8z3c29xD0NPaEG47kCihV4/ki5N4BfMK98JMRIFwlIDYVUSjqkqnVPeaix7dsN7ahr9czBEnbvcmMpJw3hPfOvPrGXqAODsoQtAsuwlAlTZQqP0mBnijGxJwbd5Db9oKFl0zSwY1Y/68JtG6Yb+847r+2fNwyBArJN2CL09g9ge9V1cNc6vPnICzxKWPvwfBf3a15g4259EUItdQKw5tQKKi3I+yYf3wzATXMfYyPxS+kgguFlD1+CyO77XC/Q85fS81Mx7Yw03/b8aj8+vXZvQArUXi42xNBtUFEA+Qb1DLXv4T9hZqdX/kY6ns9LH6TzlO0mL6ErrSnbIb9PrHFH228kvCe83Rb5bNtFSE3fLGWXkBnziCKMU446S8OsXcQecv6zsEua688WYsJHw0lvMJ0Oe+PRUPR4joNeNro4zaan48komxyUaUv26fB8ND4fnk3Go11LJ+mgfzGkbkGDxFHThbSYhwHpIzV4/sMWT/loMrmYLHoXF4usNx4Pxr1pPh31cDgYn4npYpyPs69aPJlOhwcWT09HXzH4P+gSWOMrS5wWJwNvHtAv0f647ePREMV03MumZ6I3zoaDXjYWvHfOx6cjnF4MR/ng69Eejk4PbD+fnp9NQsDvDttVOh7tAojHx/ropV5xJcXPPdIOQYvWGktztokM7Uxlc3RQWrOSAgWL5kyOm+PRErc7tCu0EGX9rCWfNX4pMaeOII8pAJPnlbW1aZHwlqZ57VIKuV+ylJ2shift0zcKiLVZWcVStvS+TE9OlMm5Whrn0+ngYvCMxj7QNghcoTIlPQhqU/rwi9EL+UCPnvDcn/d6JJFeOXekLq/Cc4H08VJGOM3uaI/C8Gn/En+/j8DxQa2mtrvuq5F91o/arDXcbEpMIXSKJMzx7Sw3aIfFtvM182G7sB/2Xha4S5jUCxNxHVnQVpogfnyAKGWOAvZHlMyRuDLdMuqzNBWUPF9ib9SncNeNQaI93NkljBIWBY/6g/6gV2nn6V1Ke5TqguuO1Osm1Qc2bfdg/P8Ho6MfjOoGSipPSsVleI2EEtnWlTRjq2EooeY7HJULLW+35NNnq3Y7Wo7DYTq7S9iKWxlSRZhP2BK5oATPwkOUpeyXmJQeoY3RcVVh5OJDtiB6jzfe5TmW/qtn7zpUcPX79Q1LWFZ/6iKnCbh8TZM1X7OUhW9ldDs0jrC2ZYrrhyo2kCiT/v4L9z8rVg== +api: eJztWNtuGzkS/ZUC9yEJ0JIlWbLlBvKQHWQxWWQy3tjZwUI2LIosuTlmkz0kW4og6N8Xxb6oZctOsnldvxjipe51TrG3LPB7z9IZu0LuRMZuEybRC6eKoKxhKbtEt7QuBw4rFME68CpXmjsVNuDjHXj976urN2ALdJwugTXAQfLAPYb+jbkx1xk2Z/en1kprcBhKZyBkCLn1ARxqXHETIOdBZOhhwT1KkiisVwa72tcqZPGmMkUZYB7wa5j3K221dg9/legUevCZLbWEjK8QuAHMFyilMvcgrC5zkwA3MgrjReFs4RQP2DmVW4katOUSZZ8lrHXjg2QpK6wPd5WDLGEO/yrRh79buWHp9lE4qzBDfQYK7niOAZ1nCRPWBDSBLvGi0EpEDSd/erq5ZV5kmPO4q/XvS5bOtixsCmQps4s/UYRat3IoKaMUD8on+YMuKPTxrpSKxHJ9V7keV2s53Dm+YQlTAfPuug9OmXu2e1wc71phdRw9BNskdelsHkPalAJ8WMbf1VFQHjgUTuXcbeABNwmoUFXFAm9MLQVlzLOqasShL6zxCKWR6GDer6/fPeBmnoCxAeZ9UjfvR2PrKth7MmtdNKXWFJ3vdfWgqoJt6nl50BHRxU/WYNLsc+Gs98C1bm/34R/WAV9xpflC76UmUHqMbs618uGuWZ9DsFbHAkXjS4cwF9zU9XYnrShzNMG/fRtciZXjWuUqdL1WJuA9uo7fh959KvMFOrBLaOV1M2kdIBdZYypLWK6MysucpYNdUlXakxAeiyCd7ERvr+xxHH8rfQBqCK4M8AAauQ9gDYKxprfOVEBfcIEgMu64COii4+sMHXYdb2057vc7A1f/+ghLpQM6KBxK6jokE6kFNzFXOQ8pvPrj1/ef38M8argT1sj5qz7bkfcqaGxb+3Pd/dzjP6lxd8nTLj3syAfcrK2T/1OV7na7Jz79en192QJMBRrUax4JbAJWffnuQ3PEJ+AthIwHaCwBwQ0sEHxJMIQSuCfQtEXd60uFWsaURV2Ca+37N2ZeKXN+DtKiN68C1b1dg8cQCEXnf5t5dBJfS1zyUoc3t3MSvUat6T9t1xJeN0BGR3qxKSqlJDj2+QKFzRGac/0b80eGpnNSVeeaA0lNMoTj6t5Yhx5UICw/mj9yrM4fw688L3Qsqz2izAij73L0nt+jpxxVXcCu7cPGQqG5QQhKPNDptjBZ6dG9HY5Ox6ejIUuOovGMBZWjDzwvSGzdy6dJp1BmLMqPxlcabndUinv4JyyICxVgxtoZDQbPEpKw5GJACb4UAr1fllpvfoSUXqIih77UMQ6yrHjzLvdPuam7uZe4x65HeGID10CxgsAfKPcW8CuKMlQoWmFMAspArrRWHqlr/WPwamz7Ngseda0eVFjC7rywjnLSoOQT/+oTe4kmwi5L2DKiDEuZtOVC41Hs7Iw+tiLSqJvcph806Wwa1jtm9RPqbrmbbuzpel7bP28QAiUsNnGLqrd/ULaXXQd3rcObTzzHo4C1D893EYbhOTbu1hch9lInAGtO/FEaSd43+fhmAK6b+1ixT8iUh6oYnvfwuRLZfZ/rOQb+XHp+KqadOejbnl/uZ67X/g0oiSao5YYQug0qSiDfoB689sT/E2Z2uPI30vF0yPqoiOSXbV4iK60p2zG/j6zxR+m3ArxHuN02+WzbrZAavlnKPsDC2geUcQbzxCwNsnYr9hDzn4Rd0WPgbCknfDSc9AbT6bA3Hg1ljwsc9Baji9PF9HQ8GS0mB23agn06PB+Nz4dnk/Fo18JJOuhfDIktaJA4arpUDkWcqj4RwfMftnjKR5PJxWTZu7hYLnrj8WDcm4rpqIfDwfhMTpdjMV68aPFkOh0eWDw9Hb1g8H/QJ7DGV44wrZoMgr3HkKH7cdvHoyHK6bi3mJ7J3ngxHPQWY8l753x8OsLpxXAkBi9Hezg6PbD9fHp+NokBvz2kq3Q82sUiHh/j0Q9mxbWSP/eyOyxadM46Gs5thdDelk6gh8LZlZIoWWXO5Lg5AR1hu0e3QgeVrJ+15IvBrwUKYgR1TAFYIUrnatMqwMts80SmFPKQsZSdrIYn7Xu5ElD1Zuk0S1kWQpGenGgruM6sD+l0cDF4AmMfaRskrlDbgl4RtSl9+MWapbqnl1L8RjDv9UgiPY1uSZ0oiUSjPl6oqpxmt7RHYfi8f76/30fg+KBWQ9tt96nJvpgHY9cGrjcFphCZIolzfDvLDdphsWW+Zj5sF/bD3vMCdwlTZmmruq5Q0JWGSvz4AFEogRL2R7QSSFiZbhnxLE0FBRcZ9kZ9CndNDArd4c4uYZSwSvCoP+gPeqXxgR6ztEepzrnpSL1qUn1g03ZfjP//ynT0K1NNoKTypNBcxddIbJFt3UkzthrGFmo+3lG70PJ2Sz59cXq3o+VqOExntwlbcadiqqjmE5Yhl5TgWXyIspT9UiWlR9XG6LguscLiQ7QgeK9uvBMCi/Di2dsOFFz+fnXNEraov4+R01S4fE2TNV+zlMUPbHQ7Ekdc2zLNzX1ZEUglk/7+C7efPqk= sidebar_class_name: "post api-method" info_path: docs/api/HTTP/runtime custom_edit_url: null diff --git a/website/versioned_docs/version-2.4.x/cli/reference/query.md b/website/versioned_docs/version-2.4.x/cli/reference/query.md index 986cbd964..651c47843 100644 --- a/website/versioned_docs/version-2.4.x/cli/reference/query.md +++ b/website/versioned_docs/version-2.4.x/cli/reference/query.md @@ -84,5 +84,5 @@ Time: 5.10000000 seconds. 1 rows. Query IDs can be abbreviated if they uniquely identify a query within the tracked session. :::note -The `spice query` command requires the runtime to be running in [cluster mode](/docs/features/distributed-query) with `--role scheduler` and a scheduler state location configured (`runtime.scheduler.state_location`, or [`runtime.state.location`](../../reference/spicepod/runtime#runtimestate)). +The `spice query` command requires the runtime to be running in [cluster mode](/docs/features/distributed-query) with `--role scheduler` and a scheduler state location configured ([`runtime.state.location`](../../reference/spicepod/runtime#runtimestate), or the deprecated `runtime.scheduler.state_location`). ::: diff --git a/website/versioned_docs/version-2.4.x/components/data-accelerators/arrow/index.md b/website/versioned_docs/version-2.4.x/components/data-accelerators/arrow/index.md index 4cbc5fd49..95aaf6949 100644 --- a/website/versioned_docs/version-2.4.x/components/data-accelerators/arrow/index.md +++ b/website/versioned_docs/version-2.4.x/components/data-accelerators/arrow/index.md @@ -5,6 +5,12 @@ description: 'In-Memory Arrow Data Accelerator Documentation' sidebar_position: 2 --- +:::warning[Deprecated] + +The `arrow` acceleration engine is deprecated in v2.4 and will be removed in v2.5. A Spicepod with a dataset or view that uses it logs one warning at startup, and one on each hot reload that adds or changes such a component, naming those datasets and views. A cluster executor does not log it. Runtimes up to and including v2.4.0-rc.3 do not log this warning. Move these accelerations to [Spice Cayenne](../cayenne/index.md) before upgrading to v2.5. On Windows, where `arrow` is the default engine, an acceleration that sets no `engine` uses `arrow` and is included in the warning; Cayenne is not available on native Windows, so run Spice in Docker or WSL to use it; see [Migrating to Spice Cayenne](../cayenne/migration.md). + +::: + The In-Memory Arrow Data Accelerator uses Apache Arrow to store data in-memory for fast access and query performance. It is the default data accelerator on Windows. On Linux and macOS, the default is [Spice Cayenne](../cayenne/index.md); see [Default Engine](../index.md#default-engine). ## Configuration diff --git a/website/versioned_docs/version-2.4.x/components/data-accelerators/cayenne/index.md b/website/versioned_docs/version-2.4.x/components/data-accelerators/cayenne/index.md index 50227148f..7506936c3 100644 --- a/website/versioned_docs/version-2.4.x/components/data-accelerators/cayenne/index.md +++ b/website/versioned_docs/version-2.4.x/components/data-accelerators/cayenne/index.md @@ -96,7 +96,7 @@ Set under a dataset's `acceleration.params`: | `cayenne_compaction_trigger_protected_snapshots` | Number of protected snapshots before snapshot-maintenance compaction runs. Separate from `cayenne_compaction_trigger_files` so small-file tuning does not silently change scan amplification behavior. Defaults to `4` for `refresh_mode: caching` / `changes`, or `append` with `refresh_check_interval` ≤ 5m; `8` otherwise. A value of `1` is clamped to a minimum of `2`. | | `cayenne_compaction_trigger_snapshot_age_ms` | Maximum age in milliseconds of the oldest protected snapshot before snapshot-maintenance compaction runs. Set to `0` to disable the age trigger. Defaults to `60000` for `refresh_mode: caching` / `changes`, or `append` with `refresh_check_interval` ≤ 5m; `300000` otherwise. | | `cayenne_compaction_max_levels` | Maximum number of consecutive compaction passes per trigger. Bounds write amplification when promotion keeps producing new candidates. Defaults to `3`. | -| `cayenne_compaction_max_files_per_pick` | Maximum number of eligible file paths retained in one compaction candidate. For a table eligible for **subset** compaction — key-based deletes (`cayenne_deletion_mode: key`, including append-only tables), no configured `sort_columns`, and no protected snapshots — the runner rewrites only the picked candidate and carries the unpicked settled files into the new snapshot by hardlink (local filesystem) or copy (S3 or cross-device), so this value does bound rewrite IO and memory. A table using position deletes, a configured `sort_columns` (whose subset hardlinks would break the global sort attestation), or carrying protected snapshots (which the rewrite has to fold) still rewrites the whole current snapshot, and this value then only affects trigger selection and observability. Defaults to `32`. | +| `cayenne_compaction_max_files_per_pick` | Maximum number of eligible file paths retained in one compaction candidate. For a table eligible for **subset** compaction — key-based deletes (`cayenne_deletion_mode: key`, including append-only tables) or a table without a primary key that holds no position deletes, no configured `sort_columns` or `cayenne_cluster_by`, and no protected snapshots — the runner rewrites only the picked candidate and carries the unpicked settled files into the new snapshot by hardlink (local filesystem) or copy (S3 or cross-device), so this value does bound rewrite IO and memory on a key-delete table. A table without a primary key extends the candidate with its tier's next-smallest files up to one sixteenth of `cayenne_target_file_size_mb` (16 MiB at a 256 MB target), so its merge is bounded by bytes rather than by this value, and its small-file tier also qualifies once it holds this many small files (or `cayenne_compaction_trigger_files`, if larger), whatever their total size. Such a table holds writers off only to commit the pass, not while it encodes. A table that holds position deletes, has a configured `sort_columns` or `cayenne_cluster_by` (whose subset hardlinks would break the global sort attestation), or carries protected snapshots (which the rewrite has to fold) still rewrites the whole current snapshot, and this value then only affects trigger selection and observability. Defaults to `32`. | | `cayenne_compaction_background_interval_ms` | Background compaction interval in milliseconds. The accelerator runs a per-table background task at this interval. Set to `0` to disable the background task — inline compaction on writes still runs. Defaults to `10000` for `refresh_mode: caching` / `changes`, or `append` with `refresh_check_interval` ≤ 5m; `0` for `refresh_mode: full`, where each refresh replaces the whole table and leaves nothing for the background task to consolidate (set the parameter explicitly to turn it back on for a table that also takes `INSERT`s); `30000` otherwise. When `refresh_mode` is left unset the connector fills it in — `debezium` and `cdc` resolve to `changes`, `sink` to `disabled`, every other connector to `full` — so an unannotated dataset takes the `full` default here. | | `cayenne_cdc_durability` | Durability mode for the inline CDC write path. Accepts `memory` (default) or `file`. `memory` appends batches to an in-RAM tier and defers the source-slot acknowledgement to a periodic or cap-triggered checkpoint, collapsing per-batch durability cost; on crash the un-checkpointed tail is replayed from the source slot, and because the apply is primary-key-idempotent this remains exactly-once. The in-memory path is eligibility-gated — it applies only to the small-write / CDC profile on non-partitioned tables, and other profiles always use `file`. `file` persists each CDC batch durably before advancing the source slot and is the conservative opt-out. | | `cayenne_cdc_mem_tier_max_bytes` | Per-table RAM-tier byte cap that forces a spill (checkpoint) and source-slot advance, in `cayenne_cdc_durability: memory` mode only. Auto-derived from host memory (~1/64 of RAM, clamped to 256 MiB–1 GiB; 256 MiB on hosts at or under 16 GiB). A process-global byte budget also bounds aggregate resident memory across all tables; whichever cap is breached first triggers the spill. Set to `0` to disable the per-table cap (the global budget still applies). | diff --git a/website/versioned_docs/version-2.4.x/components/data-accelerators/cayenne/performance.md b/website/versioned_docs/version-2.4.x/components/data-accelerators/cayenne/performance.md index beeb34e26..c8f1aade5 100644 --- a/website/versioned_docs/version-2.4.x/components/data-accelerators/cayenne/performance.md +++ b/website/versioned_docs/version-2.4.x/components/data-accelerators/cayenne/performance.md @@ -333,7 +333,7 @@ Compaction consolidates the small files that appends and CDC produce into files | `cayenne_compaction_trigger_protected_snapshots` | `4` / `8` | Protected snapshots before snapshot-maintenance compaction runs | | `cayenne_compaction_trigger_snapshot_age_ms` | `60000` / `300000` | Age of the oldest protected snapshot before maintenance runs; `0` disables the age trigger | | `cayenne_compaction_max_levels` | `3` | Consecutive passes per trigger, bounding write amplification | -| `cayenne_compaction_max_files_per_pick` | `32` | Files rewritten per pass on tables eligible for subset compaction; the rest are carried forward by hardlink | +| `cayenne_compaction_max_files_per_pick` | `32` | Files rewritten per pass on key-delete tables eligible for subset compaction; the rest are carried forward by hardlink. A table without a primary key adds more small files to each pass, up to one sixteenth of the target file size | | `runtime.params.cayenne_compaction_memory_fraction` | `0.2` | Share of the query memory pool carved out for the dedicated compaction pool | Compaction is where the storage tier shows most clearly. A pass reads a file set and writes a new one, so on a network volume it consumes the same bandwidth ingest and queries need; on the `ebs` tier Cayenne writes the output with `O_DIRECT` so the rewritten bytes do not evict hot query data from the page cache, and the tuner lengthens the compaction interval when the table is I/O-bound. Guidance: diff --git a/website/versioned_docs/version-2.4.x/components/data-accelerators/duckdb/index.md b/website/versioned_docs/version-2.4.x/components/data-accelerators/duckdb/index.md index 7681d8075..27d0d5539 100644 --- a/website/versioned_docs/version-2.4.x/components/data-accelerators/duckdb/index.md +++ b/website/versioned_docs/version-2.4.x/components/data-accelerators/duckdb/index.md @@ -5,6 +5,12 @@ description: 'DuckDB Data Accelerator Documentation' sidebar_position: 3 --- +:::warning[Deprecated] + +The `duckdb` acceleration engine is deprecated in v2.4 and will be removed in v2.5. A Spicepod with a dataset or view that uses it logs one warning at startup, and one on each hot reload that adds or changes such a component, naming those datasets and views. A cluster executor does not log it. Runtimes up to and including v2.4.0-rc.3 do not log this warning. Move these accelerations to [Spice Cayenne](../cayenne/index.md) before upgrading to v2.5; see [Migrating to Spice Cayenne](../cayenne/migration.md). + +::: + The DuckDB Data Accelerator helps improve query performance by using [DuckDB](https://duckdb.org/), an embedded analytical database engine optimized for efficient data processing. It supports in-memory and file-based operation modes, enabling workloads that exceed available memory and optionally providing persistent storage for datasets. diff --git a/website/versioned_docs/version-2.4.x/components/data-accelerators/index.md b/website/versioned_docs/version-2.4.x/components/data-accelerators/index.md index 2bb828cfe..035e0a5cb 100644 --- a/website/versioned_docs/version-2.4.x/components/data-accelerators/index.md +++ b/website/versioned_docs/version-2.4.x/components/data-accelerators/index.md @@ -40,11 +40,13 @@ Some configurations load on Arrow but not on Cayenne. On Linux and macOS, with n | Name | Description | Status | Engine Modes | | ---------- | ------------------------------- | ----------------- | ---------------- | | `cayenne` | [Spice Cayenne][cayenne] | Stable | `memory`, `file`, `file_create`, `file_update` | -| `arrow` | In-Memory Arrow Records | Stable | `memory` | -| `duckdb` | Embedded [DuckDB][duckdb] | Stable | `memory`, `file`, `file_create`, `file_update` | -| `postgres` | Attached [PostgreSQL][postgres] (Spice.ai Enterprise) | Release Candidate | N/A | -| `sqlite` | Embedded [SQLite][sqlite] | Release Candidate | `memory`, `file`, `file_create`, `file_update` | -| `turso` | Embedded [Turso][turso] | Beta | `memory`, `file`, `file_create`, `file_update` | +| `arrow` | In-Memory Arrow Records | Deprecated | `memory` | +| `duckdb` | Embedded [DuckDB][duckdb] | Deprecated | `memory`, `file`, `file_create`, `file_update` | +| `postgres` | Attached [PostgreSQL][postgres] (Spice.ai Enterprise) | Deprecated | N/A | +| `sqlite` | Embedded [SQLite][sqlite] | Deprecated | `memory`, `file`, `file_create`, `file_update` | +| `turso` | Embedded [Turso][turso] | Deprecated | `memory`, `file`, `file_create`, `file_update` | + +Every engine other than `cayenne` is deprecated in v2.4 and will be removed in v2.5. See [Migrating to Spice Cayenne](./cayenne/migration.md). [cayenne]: ./cayenne/index.md [duckdb]: ./duckdb/index.md diff --git a/website/versioned_docs/version-2.4.x/components/data-accelerators/postgres/index.md b/website/versioned_docs/version-2.4.x/components/data-accelerators/postgres/index.md index f0b43ecdc..ab07a8695 100644 --- a/website/versioned_docs/version-2.4.x/components/data-accelerators/postgres/index.md +++ b/website/versioned_docs/version-2.4.x/components/data-accelerators/postgres/index.md @@ -10,6 +10,12 @@ sidebar_position: 5 The PostgreSQL Data Accelerator is available only in [Spice.ai Enterprise](https://docs.spice.ai/docs/enterprise). ::: +:::warning[Deprecated] + +The `postgres` acceleration engine is deprecated in v2.4 and will be removed in v2.5. A Spicepod with a dataset or view that uses it logs one warning at startup, and one on each hot reload that adds or changes such a component, naming those datasets and views. A cluster executor does not log it. Runtimes up to and including v2.4.0-rc.3 do not log this warning. Move these accelerations to [Spice Cayenne](../cayenne/index.md) before upgrading to v2.5; see [Migrating to Spice Cayenne](../cayenne/migration.md). + +::: + To use PostgreSQL as Data Accelerator, specify `postgres` as the `engine` for acceleration. ```yaml diff --git a/website/versioned_docs/version-2.4.x/components/data-accelerators/sqlite/index.md b/website/versioned_docs/version-2.4.x/components/data-accelerators/sqlite/index.md index 774e844c9..c90d65dce 100644 --- a/website/versioned_docs/version-2.4.x/components/data-accelerators/sqlite/index.md +++ b/website/versioned_docs/version-2.4.x/components/data-accelerators/sqlite/index.md @@ -6,6 +6,12 @@ sidebar_position: 4 pagination_next: null --- +:::warning[Deprecated] + +The `sqlite` acceleration engine is deprecated in v2.4 and will be removed in v2.5. A Spicepod with a dataset or view that uses it logs one warning at startup, and one on each hot reload that adds or changes such a component, naming those datasets and views. A cluster executor does not log it. Runtimes up to and including v2.4.0-rc.3 do not log this warning. Move these accelerations to [Spice Cayenne](../cayenne/index.md) before upgrading to v2.5; see [Migrating to Spice Cayenne](../cayenne/migration.md). + +::: + To use SQLite as Data Accelerator, specify `sqlite` as the `engine` for acceleration. ```yaml @@ -41,7 +47,7 @@ datasets: - The SQLite accelerator doesn't support arrow `Interval` types, as [SQLite](https://www.sqlite.org/lang_datefunc.html) doesn't have a native interval type. - The SQLite accelerator only supports arrow `List` types of primitive data types; lists with structs are not supported. - The SQLite accelerator doesn't support `Dictionary` or `Map` types. -- SQLite may not be suitable for high row count use cases with complex join queries. Use [DuckDB](duckdb) instead. +- SQLite may not be suitable for high row count use cases with complex join queries. Use [Spice Cayenne](../cayenne/index.md) instead. - `ROLLUP`, `CUBE`, and `GROUPING SETS` are evaluated in Spice, not in SQLite, because SQLite has no grouping sets. - `TRY_CAST` is never sent to SQLite, and a `CAST` is sent only when SQLite evaluates it the same way Spice does. See [Casts and Federation](#casts-and-federation). - `AVG` and `SUM` over a decimal column are never sent to SQLite. SQLite stores a decimal value as a double, so a value with more than about 15 significant digits reads back changed. See [Decimal Aggregates and Federation](#decimal-aggregates-and-federation). diff --git a/website/versioned_docs/version-2.4.x/components/data-accelerators/turso.md b/website/versioned_docs/version-2.4.x/components/data-accelerators/turso.md index e2fcb3390..2a11db76a 100644 --- a/website/versioned_docs/version-2.4.x/components/data-accelerators/turso.md +++ b/website/versioned_docs/version-2.4.x/components/data-accelerators/turso.md @@ -13,6 +13,12 @@ tags: The Turso Data Accelerator is in Beta. Features and configuration may change. ::: +:::warning[Deprecated] + +The `turso` acceleration engine is deprecated in v2.4 and will be removed in v2.5. A Spicepod with a dataset or view that uses it logs one warning at startup, and one on each hot reload that adds or changes such a component, naming those datasets and views. A cluster executor does not log it. Runtimes up to and including v2.4.0-rc.3 do not log this warning. Move these accelerations to [Spice Cayenne](./cayenne/index.md) before upgrading to v2.5; see [Migrating to Spice Cayenne](./cayenne/migration.md). + +::: + The Turso Data Accelerator uses [Turso](https://turso.tech/) (built on [libSQL](https://github.com/tursodatabase/libsql), a fork of SQLite) as the acceleration engine for caching and accelerating query performance. Turso provides native async support and modern improvements over traditional SQLite while maintaining SQLite compatibility. To enable Turso acceleration, set the dataset's `acceleration.engine` to `turso`: diff --git a/website/versioned_docs/version-2.4.x/components/vectors/duckdb.md b/website/versioned_docs/version-2.4.x/components/vectors/duckdb.md index a2356e8e2..e8bbf056f 100644 --- a/website/versioned_docs/version-2.4.x/components/vectors/duckdb.md +++ b/website/versioned_docs/version-2.4.x/components/vectors/duckdb.md @@ -6,6 +6,12 @@ sidebar_position: 3 pagination_next: null --- +:::warning[Deprecated] + +The DuckDB vector engine (`vectors.engine: duckdb`) is deprecated in v2.4 and will be removed in v2.5, together with the [DuckDB accelerator](../data-accelerators/duckdb) it requires, whose deprecation warning names the datasets and views that use it. Runtimes up to and including v2.4.0-rc.3 do not log this warning. To migrate, remove the `vectors` block and set `acceleration.engine: cayenne`, which indexes embedding columns for `vector_search` itself; see [Migrating the DuckDB vector engine](../data-accelerators/cayenne/migration.md#migrating-the-duckdb-vector-engine). + +::: + DuckDB can be used as a vector engine in Spice to store embeddings and execute vector similarity search using HNSW indexes via the [DuckDB VSS](https://duckdb.org/docs/extensions/vss) extension. This is useful when a dataset or view is already accelerated with DuckDB and a fully embedded, single-process vector store is preferred over an external service. The DuckDB vector engine requires the dataset or view to be accelerated with the [DuckDB accelerator](../data-accelerators/duckdb). Spice computes embeddings on the configured columns during refresh and write, stores them in the DuckDB accelerator alongside the source data, and creates an HNSW index that is used to answer `vector_search` and `/v1/search` queries. diff --git a/website/versioned_docs/version-2.4.x/components/vectors/index.md b/website/versioned_docs/version-2.4.x/components/vectors/index.md index 640f44946..00afdecf9 100644 --- a/website/versioned_docs/version-2.4.x/components/vectors/index.md +++ b/website/versioned_docs/version-2.4.x/components/vectors/index.md @@ -30,14 +30,14 @@ Supported Vector engines: | -------------------------------- | ----------------------------------- | | [`s3_vectors`][s3vectors] | AWS S3 vectors | | [`elasticsearch`][elasticsearch] | Elasticsearch (Spice.ai Enterprise) | -| [`duckdb`][duckdb] | DuckDB VSS (HNSW) extension | +| [`duckdb`][duckdb] | DuckDB VSS (HNSW) extension (deprecated) | ## Choosing an engine The engines differ in where the index lives: - **External engines**, for high-QPS serving where the index lives outside the Spice process and survives a restart of it. [S3 Vectors](./vectors/s3_vectors#handling-filters) stores vectors in S3 and pushes predicates on columns marked `metadata.vectors: filterable` into the index query. [Elasticsearch](./vectors/elasticsearch) keeps the kNN index in the cluster (and can combine it with full-text in that cluster). -- **In-process HNSW** ([DuckDB VSS](./vectors/duckdb)) when the dataset is already DuckDB-accelerated and modest in size. The index lives in the DuckDB file. A full refresh rebuilds it. Plan memory for the graph and a cold start that reopens or rebuilds it. +- **In-process HNSW** ([DuckDB VSS](./vectors/duckdb)) when the dataset is already DuckDB-accelerated and modest in size. The DuckDB vector engine is deprecated in v2.4 and will be removed in v2.5; a Cayenne acceleration indexes embedding columns for `vector_search` without a `vectors` block. The index lives in the DuckDB file. A full refresh rebuilds it. Plan memory for the graph and a cold start that reopens or rebuilds it. Add vector search only when semantic retrieval is required. Hybrid retrieval ([`rrf`](../reference/sql/search#reciprocal-rank-fusion-rrf) over `vector_search` and `text_search`) is for queries that need both semantic and literal matches. Otherwise stay on SQL, regex, or [full-text search](../features/search/full-text). See [Where indexes are built](../features/search#where-indexes-are-built). diff --git a/website/versioned_docs/version-2.4.x/deployment/read-write-separation.md b/website/versioned_docs/version-2.4.x/deployment/read-write-separation.md index 4319609c1..c23777a8a 100644 --- a/website/versioned_docs/version-2.4.x/deployment/read-write-separation.md +++ b/website/versioned_docs/version-2.4.x/deployment/read-write-separation.md @@ -138,7 +138,7 @@ datasets: duckdb_file: /data/customers.db ``` -Snapshots are partitioned by date and dataset (`month=YYYY-MM/day=YYYY-MM-DD/dataset=/...`), so retention is a normal object-store lifecycle rule. See [Snapshots](../features/data-acceleration/snapshots) for the full configuration reference. +Snapshots are partitioned by date and dataset (`month=YYYY-MM/day=YYYY-MM-DD/dataset=/...`), and the publishing instance keeps each dataset's snapshots of the last 28 days. See [Snapshot retention](../features/data-acceleration/snapshots#snapshot-retention), and [Snapshots](../features/data-acceleration/snapshots) for the full configuration reference. ### The read instances (read tier) @@ -245,13 +245,13 @@ s3://spiceai-snapshots/prod/ The extension names the engine that wrote the snapshot (`.duckdb`, `.sqlite`, `.cayenne`, or `.turso`), so the cluster and read tiers must accelerate a dataset with the same engine. -Apply an object-store lifecycle rule (S3 lifecycle, GCS Object Lifecycle Management, ADLS Lifecycle) to expire old partitions. Most deployments keep 24–72 hours of refresh-triggered snapshots and a daily archive beyond that. +The publishing instance deletes snapshots older than 28 days, keeping each dataset's current and newest snapshot at any age, so the bucket needs no expiration rule for current objects. On a bucket with versioning enabled, each delete leaves a noncurrent version, so add a noncurrent-version expiration rule. A replicated destination keeps deleted files unless replication carries deletes or the destination has its own lifecycle rule. See [Snapshot retention](../features/data-acceleration/snapshots#snapshot-retention). To keep a longer archive, copy snapshots to another prefix or bucket. The snapshot bucket is the only shared dependency between the tiers, so keep it in the same region as the read instances and apply VPC endpoints / Private Google Access to keep traffic on the private network. When read instances run in another region, replicate this standard bucket. Keep `snapshots.location` on standard S3, GCS, or ADLS for that replication. Cayenne's S3 Express One Zone settings (`cayenne_file_path`, `cayenne_s3_*`) store the Cayenne data tier in one zone and do not substitute for the snapshot bucket. See [Snapshot location and the Cayenne data tier](../features/data-acceleration/snapshots#snapshot-location-and-the-cayenne-data-tier). ### Full copies and incremental replication -Each snapshot object is the full acceleration file. Replicating the snapshot bucket between regions copies that entire file on every snapshot. Size the bucket and the lifecycle rule for dataset size times snapshot frequency. DuckDB `snapshots_compaction` shrinks each copy; the result is still a full file. See [What each snapshot contains](../features/data-acceleration/snapshots#what-each-snapshot-contains). +Each snapshot object is the full acceleration file. Replicating the snapshot bucket between regions copies that entire file on every snapshot. Size the bucket for dataset size times the number of snapshots written in about 29 days: 28 days of retention plus at least 24 hours before a dropped file is deleted. DuckDB `snapshots_compaction` shrinks each copy; the result is still a full file. See [What each snapshot contains](../features/data-acceleration/snapshots#what-each-snapshot-contains). When changed data should replicate as objects between regions or tiers, use a path that publishes the change: diff --git a/website/versioned_docs/version-2.4.x/features/cdc/mysql-replication.md b/website/versioned_docs/version-2.4.x/features/cdc/mysql-replication.md index 121f89029..ff7946aa0 100644 --- a/website/versioned_docs/version-2.4.x/features/cdc/mysql-replication.md +++ b/website/versioned_docs/version-2.4.x/features/cdc/mysql-replication.md @@ -163,6 +163,8 @@ SET SESSION net_write_timeout = GREATEST(@@SESSION.net_write_timeout, 180) It is a **floor**, not an assignment — raising `net_write_timeout` on the source is the manual workaround for this symptom, and a value already higher than 180 seconds is left alone. If the source rejects the statement (for example the replication user may not set session variables), Spice logs a warning and continues on the server's value; grant that permission or raise `net_write_timeout` on the source itself. Time the pump spends blocked on a member accrues per dataset in `replication_member_send_stalled_seconds_total`. +The shared connection reads the binlog on a dedicated thread, `mysql-binlog-pump`, separate from the CDC apply runtime. If that thread cannot start, Spice reads the binlog on the CDC apply runtime and logs a warning that begins `Failed to start a dedicated thread to read the MySQL binlog for connection`; replication can then fall behind under a heavy write load. + :::warning[A stalled member can force the group to re-bootstrap] Unlike a PostgreSQL replication slot, a held binlog position pins nothing server-side. If a member detaches — a schema change classified as DDL is member-fatal, detaching that one dataset while the group keeps running — its held floor holds back only the shared resume position. Should the source then purge binlogs past that point (`binlog_expire_logs_seconds`), the whole group must re-bootstrap. diff --git a/website/versioned_docs/version-2.4.x/features/cdc/postgres-replication.md b/website/versioned_docs/version-2.4.x/features/cdc/postgres-replication.md index 91445408b..cd50da137 100644 --- a/website/versioned_docs/version-2.4.x/features/cdc/postgres-replication.md +++ b/website/versioned_docs/version-2.4.x/features/cdc/postgres-replication.md @@ -304,6 +304,20 @@ Every Spice replica must have its own replication slot. Spice hashes the replica |---------------------------------|----------------------------------------------------------------| | `SPICE_INSTANCE_ID` env | Preferred — set it explicitly per replica. | | `HOSTNAME` / `COMPUTERNAME` env | Fallback — works on Kubernetes where each pod has a distinct hostname. | +| The host name | Last fallback, when none of the variables above is set. | + +This is the [instance identity](../../reference/spicepod/runtime#instance-identity) that every Spice feature coordinating through shared state uses. An empty variable counts as unset. When an instance's identity changes, for example when `SPICE_INSTANCE_ID` is set on an instance that had none, its default slot name changes too: Spice creates a new slot, and the old slot keeps retaining WAL on the source until it is dropped. Find slots that no replica consumes with `SELECT slot_name, active FROM pg_replication_slots WHERE slot_name LIKE 'spice_%'`, then drop a confirmed-abandoned slot with `SELECT pg_drop_replication_slot('')`. + +:::warning[Upgrading from v2.4.0-rc.2 and earlier] + +Earlier versions used the first of `SPICE_INSTANCE_ID`, `HOSTNAME`, and `COMPUTERNAME` that was set, even when its value was empty, and `unknown` when none was set. Later versions skip empty variables and fall back to the host name. Two kinds of instances therefore get a new default slot name, and create a new slot, on upgrade: + +- An instance with none of the variables set, such as a `systemd` service. Its old slot's name ends in `_3f484bfe`. +- An instance whose earlier version selected an empty variable, for example an empty `SPICE_INSTANCE_ID`, or an unset `SPICE_INSTANCE_ID` with an empty `HOSTNAME`. It now uses the next non-empty variable, or the host name. Its old slot's name ends in `_38d394c2`. + +The old slot keeps retaining WAL until it is dropped. An instance whose earlier version selected a non-empty variable keeps its slot name. + +::: ### Example: Kubernetes StatefulSet diff --git a/website/versioned_docs/version-2.4.x/features/data-acceleration/data-refresh.md b/website/versioned_docs/version-2.4.x/features/data-acceleration/data-refresh.md index 2513ec5ee..a1651e019 100644 --- a/website/versioned_docs/version-2.4.x/features/data-acceleration/data-refresh.md +++ b/website/versioned_docs/version-2.4.x/features/data-acceleration/data-refresh.md @@ -67,7 +67,7 @@ datasets: ``` :::info Readiness with snapshots -Append-mode accelerations that define a `time_column` wait to report ready until the first append refresh completes after [snapshot bootstrap](./snapshots). This keeps the dataset out of rotation until the freshest data is available while still benefiting from the snapshot-assisted startup. +An append-mode acceleration that defines a `time_column` and meets the conditions in [Serving an Existing Acceleration While the Source Is Unavailable](#serving-an-existing-acceleration-while-the-source-is-unavailable) is served as soon as its existing acceleration is available, whether it was reopened from disk or restored by [snapshot bootstrap](./snapshots), including while the source is unavailable. The first append refresh then catches up from the newest `time_column` value in the acceleration. [`last_refresh`](#monitoring-freshness) does not show how current the data is until a refresh completes: a dataset that creates acceleration snapshots or was restored from one does not restore `last_refresh` from its checkpoint at startup, so while the source is unavailable it serves the persisted data without reporting it. In v2.4.0-rc.3 and earlier, such a dataset with snapshots enabled waited to report ready until its first append refresh completed. ::: If late arriving data or clock-skew needs to be accounted for, an optional overlap can also be specified. See [`acceleration.refresh_append_overlap`](../../reference/spicepod/datasets#accelerationrefresh_append_overlap). @@ -146,12 +146,6 @@ See [Caching Mode](./refresh-modes/caching) for detailed documentation and examp The `snapshot` refresh mode creates a read-only acceleration that reloads exclusively from the [snapshot store](./snapshots). The federated data source is never queried for refreshes — instead, the runtime polls the snapshot store on a configurable interval and atomically swaps in newer snapshots when available. ```yaml -snapshots: - enabled: true - location: s3://my-bucket/snapshots/ - params: - s3_auth: iam_role - datasets: - from: postgres:public.my_table name: my_table @@ -162,6 +156,7 @@ datasets: refresh_mode: snapshot refresh_check_interval: 30s # Poll interval; defaults to 1m snapshots: enabled + snapshots_location: s3://my-bucket/spice/snapshots/ # The writer's snapshot folder params: duckdb_file: /nvme/my_table.db ``` @@ -178,8 +173,10 @@ datasets: - Each poll reads the snapshot store's metadata conditionally, sending the `ETag` recorded by the previous poll in `If-None-Match`. When the store reports that the metadata is unchanged, the poll ends without downloading it - When a newer snapshot is found, its schema is validated against the current acceleration schema before downloading - A poll reads the metadata once and uses that read for the snapshot id comparison, the schema validation, and the download, so the snapshot that is downloaded is the one whose schema was validated, even if a writer publishes another snapshot during the poll -- With [`bootstrap_on_failure_behavior: retry`](./snapshots#failure-behavior), a failed download retries the whole poll. Each attempt reads the metadata again and validates the snapshot before downloading it, so a snapshot published to replace a broken one is picked up -- A poll that does not load the current snapshot records no `ETag`, so the next poll reads the metadata in full. This applies when `bootstrap_on_failure_behavior: warn` skipped a failed download or `fallback` loaded an older snapshot, and when the store's current snapshot id is older than the loaded one, in which case every poll logs the `snapshot metadata current id is older than the locally loaded snapshot` warning +- Until a snapshot is loaded, with [`bootstrap_on_failure_behavior: retry`](./snapshots#failure-behavior), a failed download retries the whole poll. Each attempt reads the metadata again and validates the snapshot before downloading it, so a snapshot published to replace a broken one is picked up +- A poll that does not load the current snapshot records no `ETag`, so the next poll reads the metadata in full. This applies when `bootstrap_on_failure_behavior: warn` skipped a failed download or `fallback` loaded an older snapshot, and when the store's current snapshot id is older than the loaded one +- When a poll finds no published snapshot, or finds one older than the loaded snapshot, the dataset stays `Ready` and keeps serving the loaded snapshot. The poll increments `dataset_acceleration_refresh_errors` with `reason="snapshot_not_found"` or `reason="snapshot_older_than_loaded"` and logs a warning that begins `Dataset '' cannot refresh from its snapshot location, so it keeps serving snapshot ` +- Once a snapshot is loaded, a poll that cannot load a newer published snapshot does the same, whatever `bootstrap_on_failure_behavior` is set to: the dataset stays `Ready` and keeps serving the loaded snapshot, the poll increments `dataset_acceleration_refresh_errors` with `reason="snapshot_download_failed"` and logs the same warning, and the next poll tries again. A snapshot file that has not yet reached the location, because `metadata.json` arrived first, is checked every 10 seconds for up to 90 seconds before the poll reports it. A file that fails verification is reported at once - The accelerator file is swapped atomically — queries continue to be served from the previous snapshot until the swap completes - `INSERT`, `UPDATE`, `DELETE`, and `TRUNCATE` statements are all rejected with an error since the acceleration is driven exclusively from snapshots @@ -269,7 +266,7 @@ Query availability and source health are reported separately: a dataset can serv | `on_registration` | Served from the existing acceleration | `200` once registered | `Ready` at registration, `Error` after a failed connection attempt, `Ready` once reconnected | | `on_schema_resolved` | Served from the existing acceleration | `503` until the source is reached | `Initializing` at registration, `Error` after a failed connection attempt, `Ready` once reached | -`/v1/ready` stays `200` for `on_load` and `on_registration` after the status changes to `Error`, because runtime `on_load` readiness counts a dataset that has been `Ready` once. Two kinds of dataset do not follow this table, because the existing acceleration is not treated as loaded until a refresh completes in this process: a dataset with `refresh_mode: append`, a `time_column`, and snapshot bootstrapping enabled, and a dataset that reads acceleration snapshots (`file_format: snapshot`) before it restores one. Until that refresh completes, `on_load` is not ready and queries return an error. `on_registration` is ready at registration and `on_schema_resolved` is ready once the source is reached, but in both cases queries go to the federated source rather than the persisted data, so they fail while the source is unavailable. +`/v1/ready` stays `200` for `on_load` and `on_registration` after the status changes to `Error`, because runtime `on_load` readiness counts a dataset that has been `Ready` once. A dataset that reads acceleration snapshots (`file_format: snapshot`) does not follow this table before it restores a snapshot in this process, because its existing acceleration is not treated as loaded until then. Until it restores one, `on_load` is not ready and queries return an error. `on_registration` is ready at registration and `on_schema_resolved` is ready once the source is reached, but in both cases queries go to the federated source rather than the persisted data, so they fail while the source is unavailable. [`refresh_on_startup`](#refresh-on-startup) keeps its meaning for `refresh_mode: full`. With `auto`, the persisted data is served and the next refresh follows the refresh schedule. With `always`, the persisted data is served until the source is reached and the startup refresh replaces it. The other eligible refresh modes ignore `refresh_on_startup`: `append` refreshes at startup once the source is reached, adding new data to the persisted data instead of replacing it, `snapshot` checks for a newer snapshot at startup, and `disabled` does not refresh. @@ -593,7 +590,7 @@ Keep the interval aligned with the freshness SLA and with what the origin can se ## Isolating refresh from queries -By default the runtime runs acceleration refresh on a dedicated low-priority thread pool (`refresh-worker`), separate from the pool that executes queries. CDC apply (`refresh_mode: changes`) runs on its own default-priority pool (`cdc-apply-worker`) when any dataset streams changes, so a bulk refresh does not deprioritize the apply loop. Cayenne compaction runs on `compaction-worker` when a dataset can produce files to compact. +By default the runtime runs acceleration refresh on a dedicated low-priority thread pool (`refresh-worker`), separate from the pool that executes queries. CDC apply (`refresh_mode: changes`) runs on its own default-priority pool (`cdc-apply-worker`) when any dataset streams changes, so a bulk refresh does not deprioritize the apply loop. When any dataset streams changes, the query pool runs at nice 5, so CDC apply and primary-key lookups, which run at nice 0, stay responsive on a saturated host. Cayenne compaction runs on `compaction-worker` when a dataset can produce files to compact. The pools still share the machine's CPU and memory. A full refresh can raise latency if it saturates the node or the query memory pool. Leave the default in place, and set `runtime.params.dedicated_thread_pool: disabled` only when a single shared pool is intentional — that puts refresh back on the query runtime. See [`dedicated_thread_pool`](../../reference/spicepod/runtime#dedicated-thread-pools). @@ -699,6 +696,16 @@ datasets: refresh_check_interval: 30s ``` +### Source That Stops Sending Mid-Refresh + +Once the source has sent the first batch of a `full` or `append` refresh, Spice waits at most 10 minutes for each later batch. A connection that the network stops carrying can otherwise leave a refresh waiting indefinitely, with the dataset serving stale data and other refreshes queued behind it. A refresh that waits longer fails with a transient error, is counted in [`dataset_acceleration_refresh_errors`](../observability#available-metrics), and, unless `refresh_retry_enabled` is `false`, is retried on a new connection with the backoff described above: + +```text +The source of dataset 'orders' stopped sending data mid-refresh: no data for 600s after it started sending, so the connection is treated as lost and the refresh is retried +``` + +The wait for the first batch is not limited, so a source that takes a long time to plan and start a query is unaffected. A refresh that never receives its first batch, such as one that draws a pooled connection that died while idle, is not covered by this limit. The 10-minute limit cannot be configured. + ## Retention Policy | | | diff --git a/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md b/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md index cf2f62d3c..7a909fdf6 100644 --- a/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md +++ b/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md @@ -117,7 +117,7 @@ This is supported for connectors that accept the [file format parameter](../../. ## Readiness with Snapshots -Append-mode accelerations that define a `time_column` wait to report ready until the first append refresh completes after [snapshot bootstrap](../snapshots). This keeps the dataset out of rotation until the freshest data is available while still benefiting from snapshot-assisted startup. +An append-mode acceleration that defines a `time_column` and meets the conditions in [Serving an Existing Acceleration While the Source Is Unavailable](../data-refresh#serving-an-existing-acceleration-while-the-source-is-unavailable) is served as soon as its existing acceleration is available, whether it was reopened from disk or restored by [snapshot bootstrap](../snapshots), including while the source is unavailable. The first append refresh then catches up from the newest `time_column` value in the acceleration. In v2.4.0-rc.3 and earlier, such a dataset with snapshots enabled waited to report ready until its first append refresh completed. ## Combining with Upserts diff --git a/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/snapshot.md b/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/snapshot.md index 2fed29a3e..a852f3470 100644 --- a/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/snapshot.md +++ b/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/snapshot.md @@ -20,12 +20,6 @@ To read snapshots published to S3 without naming the original federated source, ## Configuration ```yaml -snapshots: - enabled: true - location: s3://my-bucket/snapshots/ - params: - s3_auth: iam_role - datasets: - from: postgres:public.my_table name: my_table @@ -36,10 +30,13 @@ datasets: refresh_mode: snapshot refresh_check_interval: 30s # Poll interval; defaults to 1m snapshots: enabled + snapshots_location: s3://my-bucket/spice/snapshots/ # The writer's snapshot folder params: duckdb_file: /nvme/my_table.db ``` +This reader sets [`acceleration.snapshots_location`](../snapshots#override-the-snapshot-location-per-dataset) to the folder the writer publishes to, so it needs only read access to the snapshots. It could instead set [`runtime.state`](../../../reference/spicepod/runtime#runtimestate) to the writer's state location, whose `snapshots/` folder holds the snapshots. An instance with `runtime.state` checks the location at startup, which needs write access under `_spice/`, and does not report ready until the [check](../../../reference/spicepod/runtime#startup-check-of-the-state-location) passes, even though snapshot reads do not wait for it. + ## Requirements - `acceleration.snapshots` must be `enabled` or `bootstrap_only`. Snapshot mode is a snapshot *consumer* only, so the two behave identically here: creation is skipped for the mode entirely and the dataset never publishes new snapshots — a separate writer must produce them. @@ -53,8 +50,10 @@ datasets: - Each poll reads the snapshot store's metadata conditionally, sending the `ETag` recorded by the previous poll in `If-None-Match`. When the store reports that the metadata is unchanged, the poll ends without downloading it. - When a newer snapshot is found, its schema is validated against the current acceleration schema before downloading. - A poll reads the metadata once and uses that read for the snapshot id comparison, the schema validation, and the download, so the snapshot that is downloaded is the one whose schema was validated, even if a writer publishes another snapshot during the poll. -- With [`bootstrap_on_failure_behavior: retry`](../snapshots#failure-behavior), a failed download retries the whole poll. Each attempt reads the metadata again and validates the snapshot before downloading it, so a snapshot published to replace a broken one is picked up. -- A poll that does not load the current snapshot records no `ETag`, so the next poll reads the metadata in full. This applies when `bootstrap_on_failure_behavior: warn` skipped a failed download or `fallback` loaded an older snapshot, and when the store's current snapshot id is older than the loaded one, in which case every poll logs the `snapshot metadata current id is older than the locally loaded snapshot` warning. +- Until a snapshot is loaded, with [`bootstrap_on_failure_behavior: retry`](../snapshots#failure-behavior), a failed download retries the whole poll. Each attempt reads the metadata again and validates the snapshot before downloading it, so a snapshot published to replace a broken one is picked up. +- A poll that does not load the current snapshot records no `ETag`, so the next poll reads the metadata in full. This applies when `bootstrap_on_failure_behavior: warn` skipped a failed download or `fallback` loaded an older snapshot, and when the store's current snapshot id is older than the loaded one. +- When a poll finds no published snapshot, or finds one older than the loaded snapshot, the dataset stays `Ready` and keeps serving the loaded snapshot. The poll increments [`dataset_acceleration_refresh_errors`](../../observability/index.md) with `reason="snapshot_not_found"` or `reason="snapshot_older_than_loaded"` and logs a warning that begins `Dataset '' cannot refresh from its snapshot location, so it keeps serving snapshot `. +- Once a snapshot is loaded, a poll that cannot load a newer published snapshot does the same, whatever `bootstrap_on_failure_behavior` is set to: the dataset stays `Ready` and keeps serving the loaded snapshot, the poll increments `dataset_acceleration_refresh_errors` with `reason="snapshot_download_failed"` and logs the same warning, and the next poll tries again. A snapshot file that has not yet reached the location, because `metadata.json` arrived first, is checked every 10 seconds for up to 90 seconds before the poll reports it. A file that fails verification is reported at once. - The accelerator file is swapped atomically — queries continue to be served from the previous snapshot until the swap completes. - `INSERT`, `UPDATE`, `DELETE`, and `TRUNCATE` statements are all rejected with an error since the acceleration is driven exclusively from snapshots. @@ -64,16 +63,9 @@ Use `refresh_mode: snapshot` for read-only replicas that should not access the f ## Reload on S3 event notifications -A reader that only polls loads a new snapshot up to one `refresh_check_interval` after a writer publishes it. When the snapshot location is on Amazon S3, set `snapshots.params.s3_queue_url` to an Amazon SQS queue that receives the location's [S3 event notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html), and `refresh_mode: snapshot` datasets reload as soon as a new snapshot is published: +A reader that only polls loads a new snapshot up to one `refresh_check_interval` after a writer publishes it. When the snapshot location is on Amazon S3, set `s3_queue_url` in the location's params (`runtime.state.params`, or a dataset's [`snapshots_params`](../snapshots#override-the-snapshot-location-per-dataset)) to an Amazon SQS queue that receives the location's [S3 event notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html), and `refresh_mode: snapshot` datasets reload as soon as a new snapshot is published: ```yaml -snapshots: - enabled: true - location: s3://my-bucket/snapshots/ - params: - s3_auth: iam_role - s3_queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/spice-reader-1 - datasets: - from: postgres:public.my_table name: my_table @@ -84,19 +76,22 @@ datasets: refresh_mode: snapshot refresh_check_interval: 10m # Fallback check for missed notifications snapshots: bootstrap_only + snapshots_location: s3://my-bucket/spice/snapshots/ + snapshots_params: + s3_queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/spice-reader-1 ``` A writer makes a snapshot current by rewriting the location's `metadata.json`, so an `ObjectCreated` notification for that object triggers the reload. Spice reads `metadata.json` once for each batch of such notifications it receives and reloads only the datasets whose current snapshot changed, with the same snapshot id, schema, and checksum checks as a poll. Notifications for snapshot files and for removed objects are deleted from the queue without a reload. A reader that is still waiting for its first snapshot restores it when the notification arrives. Set up the queue as follows: -- **One queue per reader process.** SQS delivers each message to one receiver, so readers that share a queue each see only some notifications. To serve several readers, publish the bucket's notifications to an Amazon SNS topic and subscribe one queue per reader process. All `refresh_mode: snapshot` datasets in one process share that process's queue. -- **Only this location's notifications.** Filter the bucket notification by the location's prefix. A notification for an object outside the location is logged as an error and left on the queue, which delivers it again after each visibility timeout until it expires. +- **One queue per reader process.** SQS delivers each message to one receiver, so readers that share a queue each see only some notifications. To serve several readers, publish the bucket's notifications to an Amazon SNS topic and subscribe one queue per reader process. All `refresh_mode: snapshot` datasets in one process share that process's queue. Snapshot locations in one process that name the same queue share one consumer, which reads the queue with the credentials of the first location. A location that names the queue with other credentials logs a warning that names both locations. +- **Only the notifications of the locations on the queue.** Filter the bucket notification by the prefix of each snapshot location that names the queue. A notification for an object outside every one of those locations is logged as an error and left on the queue, which delivers it again after each visibility timeout until it expires. When the location is the `snapshots/` folder of `runtime.state.location`, the notification can cover the whole `runtime.state` prefix: notifications for the other [runtime state folders](../../../reference/spicepod/runtime#runtime-state-layout), such as the scheduler's `cluster/` folder, are deleted from the queue without a log. - **Permissions.** Spice reads the queue with the snapshot location's credentials: `s3_key` and `s3_secret` when both are set, otherwise the default AWS credential chain. The region comes from the queue URL. Grant `sqs:ReceiveMessage` and `sqs:DeleteMessage` on the queue. Datasets keep checking the location every `refresh_check_interval`, which covers notifications that never arrive. When Spice cannot reach SQS, it logs a warning when the outage starts, an error if it lasts five minutes, and an info message when it recovers. Datasets reload on `refresh_check_interval` in the meantime. -A queue URL that is empty, is an ARN, or is not an SQS queue URL, and a snapshot location that is not `s3://`, fail the dataset at load with an error that names `s3_queue_url`. When `s3_queue_url` is set but no dataset uses `refresh_mode: snapshot`, Spice logs a warning and does not read the queue. A dataset that reads snapshots with [`file_format: snapshot`](../snapshots#serve-a-dataset-from-published-snapshots) does not accept `s3_queue_url` and checks for new snapshots every `refresh_check_interval`. +A queue URL that is empty, is an ARN, or is not an SQS queue URL, and a snapshot location that is not `s3://`, fail the dataset at load with an error that names `s3_queue_url`. When `s3_queue_url` is set but no dataset uses `refresh_mode: snapshot`, Spice logs a warning that names the setting that holds the queue, such as `runtime.state.params.s3_queue_url`, and does not read the queue. A cluster scheduler, which loads no accelerations, does not log this warning. A dataset that reads snapshots with [`file_format: snapshot`](../snapshots#serve-a-dataset-from-published-snapshots) does not accept `s3_queue_url` and checks for new snapshots every `refresh_check_interval`. ## Related Topics diff --git a/website/versioned_docs/version-2.4.x/features/data-acceleration/snapshots.md b/website/versioned_docs/version-2.4.x/features/data-acceleration/snapshots.md index 3bda07d4c..7ad82ed98 100644 --- a/website/versioned_docs/version-2.4.x/features/data-acceleration/snapshots.md +++ b/website/versioned_docs/version-2.4.x/features/data-acceleration/snapshots.md @@ -12,12 +12,11 @@ Acceleration Snapshots are available in the Spice [Enterprise edition](https://d ## Spicepod Example ```yaml -snapshots: - enabled: true - location: s3://some_bucket/some_folder/ - bootstrap_on_failure_behavior: warn - params: - s3_auth: iam_role +runtime: + state: + location: s3://some_bucket/some_folder/ + params: + s3_region: us-east-1 datasets: - name: some_table @@ -44,7 +43,7 @@ Acceleration snapshots let Spice reuse a pre-built acceleration file on startup Snapshots are organized with Hive-style partitioning so they are easy to retain and prune. For a dataset named `my_dataset` accelerated with DuckDB, Spice writes files such as: ```bash -s3://some_bucket/some_folder/month=2025-09/day=2025-09-19/dataset=my_dataset/my_dataset_20250919T134522Z_01996238f1157a59ae4d69b4f18b22a4.duckdb +s3://some_bucket/some_folder/snapshots/month=2025-09/day=2025-09-19/dataset=my_dataset/my_dataset_20250919T134522Z_01996238f1157a59ae4d69b4f18b22a4.duckdb ``` The timestamp is recorded in UTC using ISO 8601 without punctuation. The UUIDv7 after the timestamp gives each snapshot its own object name, so two snapshots written in the same second do not overwrite each other. The file extension names the acceleration engine that wrote the snapshot: `.duckdb`, `.sqlite`, `.cayenne`, or `.turso`. The engine is also recorded in the snapshot metadata, and a snapshot whose engine differs from the dataset's current engine is rejected at bootstrap rather than restored. @@ -57,28 +56,35 @@ Every accelerated dataset must write to its own file (for example, `/nvme/my_dat Each snapshot is a complete copy of that dataset's acceleration file at the time Spice writes it. The object under the snapshot location is the whole accelerated dataset — DuckDB, SQLite, Cayenne, or Turso — ready for a reader to download and open. Spice writes that full file on every snapshot. It does not publish an incremental delta of the rows or objects that changed since the previous snapshot. -A bootstrap, and a [`refresh_mode: snapshot`](./data-refresh#snapshot) reload, replaces the local acceleration file with that copy. Every upload moves the full file, and bucket or object replication of the snapshot prefix moves the full file again. Storage grows with the size of the accelerated dataset times how often snapshots are written, until a lifecycle rule expires older objects. DuckDB [`snapshots_compaction`](#snapshot-compaction) can shrink each copy. The uploaded object is still a full file. +A bootstrap, and a [`refresh_mode: snapshot`](./data-refresh#snapshot) reload, replaces the local acceleration file with that copy. Every upload moves the full file, and bucket or object replication of the snapshot prefix moves the full file again. Storage grows with the size of the accelerated dataset times the number of snapshots kept. The [28-day retention period](#snapshot-retention) applies to the entries in `metadata.json`; dropped files stay at least 24 more hours, until a later publish deletes them, so budget for about 29 days of snapshot files, plus the current and newest snapshot when they are older. DuckDB [`snapshots_compaction`](#snapshot-compaction) can shrink each copy. The uploaded object is still a full file. When the workload needs incremental replication of changed data between regions or storage tiers, use a path that publishes the changed objects: -- **Cayenne cold / datalake tier.** [`cayenne_datalake_location`](../../components/data-accelerators/cayenne#cold-object-store-tier) graduates data onto standard object storage (a general-purpose S3 or S3-compatible bucket). Promotion carries unchanged cold files forward and rewrites the cold files a change can touch. The tier requires `refresh_mode: changes` or `append` and a primary key. It is a data tier, separate from `snapshots.location`. +- **Cayenne cold / datalake tier.** [`cayenne_datalake_location`](../../components/data-accelerators/cayenne#cold-object-store-tier) graduates data onto standard object storage (a general-purpose S3 or S3-compatible bucket). Promotion carries unchanged cold files forward and rewrites the cold files a change can touch. The tier requires `refresh_mode: changes` or `append` and a primary key. It is a data tier, separate from the snapshot location. - **Iceberg batched writes with merge-on-read.** [Iceberg writes](../../components/data-connectors/iceberg#write-support) append new data files, and [deletes](../../components/data-connectors/iceberg#deleting-rows) are equality delete files that readers merge on scan. A catalog commit publishes those objects. Use this path when the dataset lives in Iceberg and readers apply delete files during scan. Neither path bootstraps a file-mode accelerator. Keep snapshots for that. -## Configure snapshot storage +## Snapshot retention -Snapshots are controlled with a top-level `snapshots` block in the Spicepod. The location can point to S3, Azure ADLS Gen2, Google Cloud Storage, or the local filesystem. +Each dataset keeps the snapshots it published in the last 28 days. Its current snapshot and its newest snapshot are kept at any age, so the location always holds a snapshot to bootstrap from, however long ago it was published. The retention period is not configurable. -```yaml -snapshots: - enabled: true - location: s3://some_bucket/some_folder/ # Folder where snapshots are written - bootstrap_on_failure_behavior: warn # retry | fallback | warn - params: - s3_auth: iam_role # Defaults to iam_role for snapshots +The instance that creates a dataset's snapshots applies retention each time it publishes one. In the same [conditional write](#conditional-writes-on-the-snapshot-location) of `metadata.json` that records the new snapshot, it drops the entries that are older than 28 days. It deletes a dropped snapshot's file 24 hours after the drop, so a reader that is still downloading the file can finish. About once an hour, the publishing instance also lists the dataset's snapshot files and deletes the files that no entry names and that are at least 24 hours old, such as the upload of a publish that failed. A publish that drops entries or deletes files logs a message such as: + +```text +Snapshot cleanup of dataset 'orders' dropped 2 snapshots older than 28d and deleted 2 snapshot files. ``` +Retention runs only when a snapshot is published, so the snapshots of a dataset that no longer publishes stay at the location. Spice v2.4.0-rc.2 and earlier keep every snapshot. After an upgrade, the first publish of each dataset drops its entries older than 28 days, and their files are deleted at a publish 24 hours or more later. + +A dropped snapshot is no longer listed by `GET /v1/datasets/{name}/acceleration/snapshots`, and `POST /v1/datasets/{name}/acceleration/snapshots/current` cannot make it current: the request returns `404 Not Found`. To roll back further than 28 days, keep a copy of the snapshot outside the location. + +A bucket lifecycle rule on the snapshot location can still delete files that `metadata.json` lists. Spice treats a snapshot whose file is gone as a failed download, and [`bootstrap_on_failure_behavior`](#failure-behavior) applies: `fallback` tries the next older snapshot, `warn` continues with an empty acceleration, and `retry` keeps retrying. A [`refresh_mode: snapshot`](./refresh-modes/snapshot.md) reader that has already loaded a snapshot never continues with an empty acceleration: when a poll loads no newer snapshot, the reader keeps serving the one it loaded and reports `reason="snapshot_download_failed"`. When the file of an entry at least 24 hours old is gone, the publishing instance drops the entry and logs a warning that names the snapshot. The current and newest entries are never dropped this way, so a current or newest snapshot whose file is gone stays listed until a newer snapshot is published. + +## Configure snapshot storage + +Snapshots are stored in the `snapshots/` folder of [`runtime.state.location`](../../reference/spicepod/runtime#runtimestate), the object store that holds the Spicepod's shared runtime state, and use `runtime.state.params` (see [Store snapshots under `runtime.state`](#store-snapshots-under-runtimestate)). A dataset can store its snapshots elsewhere, with other params, by [setting its own snapshot location](#override-the-snapshot-location-per-dataset). The location can point to S3, Azure ADLS Gen2, Google Cloud Storage, or the local filesystem. + ### Supported storage backends | Backend | URL scheme | Environment variables | @@ -88,15 +94,15 @@ snapshots: | Google Cloud Storage | `gs://` | `GOOGLE_APPLICATION_CREDENTIALS`, Workload Identity | | Local filesystem | `file://` | N/A | -`location` must be a URI with a scheme — a bare filesystem path such as `/nvme/snapshots` is not a valid URI, so it fails to parse and snapshots are disabled with an error logged. Use `file:///nvme/snapshots/` for a local folder. +A snapshot location must be a URI with a scheme — a bare filesystem path such as `/nvme/snapshots` is not a valid URI, so it fails to parse and snapshots are disabled with an error logged. Use `file:///nvme/snapshots/` for a local folder. -When the location is an S3 bucket, `params` accepts these [S3 parameters](../../components/data-connectors/s3): `s3_region`, `s3_endpoint`, `s3_auth` (`iam_role` or `key`; default `iam_role`), `s3_key`, `s3_secret`, `s3_session_token`, `s3_queue_url` (an SQS queue that receives the location's S3 event notifications, so [`refresh_mode: snapshot` readers reload on publication](./refresh-modes/snapshot#reload-on-s3-event-notifications)), `client_timeout`, and `allow_http`. Spice ignores other keys and logs a warning for each. With `s3_auth: key`, both `s3_key` and `s3_secret` must be set; without them, Spice logs an error and does not use the snapshot location, rather than connecting with credentials from the environment. Azure locations also accept the Azure connector parameters under `params` for explicit credential overrides. GCS locations accept the [GCS data connector](../../components/data-connectors/gcs#authentication-parameters) parameter names, such as `gcs_service_account_key` and `gcs_skip_signature`, and validate them as a [`runtime.state` location](../../reference/spicepod/runtime#state-location-parameters) does. Spice v2.4.0-rc.1 and earlier ignored these names on a snapshot location. When no explicit credentials are supplied, Spice reads standard environment variables for each cloud provider. Values in `params` can reference [secrets](../../components/secret-stores) with `${secrets:}` for S3, Azure, and GCS locations. +When the location is an S3 bucket, its params accept these [S3 parameters](../../components/data-connectors/s3): `s3_region`, `s3_endpoint`, `s3_auth` (`iam_role` or `key`; default `iam_role`), `s3_key`, `s3_secret`, `s3_session_token`, `s3_queue_url` (an SQS queue that receives the location's S3 event notifications, so [`refresh_mode: snapshot` readers reload on publication](./refresh-modes/snapshot#reload-on-s3-event-notifications)), `client_timeout`, and `allow_http`. Spice ignores other keys and logs a warning for each. With `s3_auth: key`, both `s3_key` and `s3_secret` must be set; without them, Spice logs an error and does not use the snapshot location, rather than connecting with credentials from the environment. Azure locations also accept the Azure connector parameters under `params` for explicit credential overrides. GCS locations accept the [GCS data connector](../../components/data-connectors/gcs#authentication-parameters) parameter names, such as `gcs_service_account_key` and `gcs_skip_signature`, and validate them as a [`runtime.state` location](../../reference/spicepod/runtime#state-location-parameters) does. Spice v2.4.0-rc.1 and earlier ignored these names on a snapshot location. When no explicit credentials are supplied, Spice reads standard environment variables for each cloud provider. Values in `params` can reference [secrets](../../components/secret-stores) with `${secrets:}` for S3, Azure, and GCS locations. An invalid S3 parameter value, such as `s3_auth: public`, disables snapshots for the dataset. Spice logs an error that names the dataset, the location, and the parameter, and does not fall back to default parameters. Those defaults would drop every other configured parameter, including `s3_endpoint`, `s3_region`, and the credentials, so snapshots could be written to a different store or with a different identity than the one configured. When `s3_queue_url` is set, the dataset fails to register with the parameter error instead. Fix the parameter and restart Spice. ### Store snapshots under `runtime.state` -When the Spicepod sets [`runtime.state`](../../reference/spicepod/runtime#runtimestate) and `snapshots.location` is not set, snapshots go to the `snapshots/` folder of `runtime.state.location`. The top-level `snapshots` section is then optional: +When the Spicepod sets [`runtime.state`](../../reference/spicepod/runtime#runtimestate), datasets that enable snapshots store them in the `snapshots/` folder of `runtime.state.location`: ```yaml runtime: @@ -115,17 +121,65 @@ datasets: This fragment omits the PostgreSQL connection settings that the `orders` dataset needs; see the [PostgreSQL connector `params`](../../components/data-connectors/postgres/index.md#params). With them, this Spicepod publishes the `orders` snapshots under `s3://my-bucket/spice/snapshots/`, next to the other [runtime state folders](../../reference/spicepod/runtime#runtime-state-layout), in the same layout as any other snapshot location. -`snapshots.location`, when set, takes precedence over `runtime.state`. The snapshot location then uses `snapshots.params` alone. For a location taken from `runtime.state`, the snapshots use `runtime.state.params`, unless `snapshots.params` is set, which replaces them as a whole. A `snapshots` section without a `location` still sets `enabled`, `bootstrap_on_failure_behavior`, and `params`, so `snapshots.enabled: false` turns snapshots off for every dataset. - In v2.4.0-rc.1 and earlier, a dataset with `snapshots: enabled` and no `snapshots.location` created and restored no snapshots, and Spice logged `Snapshots enabled but no location configured` when the Spicepod had a `snapshots` section. With `runtime.state` set, such a dataset now publishes snapshots to `/snapshots/` and bootstraps from them. Set `acceleration.snapshots: disabled` on a dataset that should not. +### Override the snapshot location per dataset + +A dataset that needs another snapshot location, other params, or another failure behavior sets them in its own `acceleration` block: + +```yaml +runtime: + state: + location: s3://my-bucket/spice/ + params: + s3_region: us-east-1 + +datasets: + - from: postgres:events + name: events + acceleration: + enabled: true + engine: cayenne + mode: file + snapshots: enabled + snapshots_location: s3://archive-bucket/snapshots/ + snapshots_params: + s3_region: eu-west-1 + snapshots_bootstrap_on_failure_behavior: fallback +``` + +| Field | Default | +| ----- | ------- | +| `snapshots_location` | The `snapshots/` folder of `runtime.state.location`. | +| `snapshots_params` | The params of the location the dataset uses: none for a `snapshots_location`, and `runtime.state.params` for the `runtime.state` folder. | +| `snapshots_bootstrap_on_failure_behavior` | `warn`. See [Failure behavior](#failure-behavior). | + +`snapshots_params`, when set, are the only params the dataset's snapshots use. A dataset that sets `snapshots_location` without `snapshots_params` uses no params, so `runtime.state.params`, such as an `s3_endpoint` or credentials, never apply to a location they were not written for. When the dataset's location needs credentials or a region that the environment does not supply, set them in `snapshots_params`. + +These fields require a Spice version newer than v2.4.0-rc.2. Spice v2.4.0-rc.2 and earlier reject a Spicepod that sets them. + +### Migrate from the top-level `snapshots` section + +The top-level `snapshots` section is deprecated and will be removed in a future release. Spicepods that set it still load, and each of its settings applies to every dataset that does not set the matching `acceleration` field: + +| Deprecated setting | Replacement | +| ------------------ | ----------- | +| `snapshots.location` | `runtime.state.location`, whose `snapshots/` folder holds the snapshots, or `acceleration.snapshots_location` on each dataset. | +| `snapshots.params` | `runtime.state.params`, or `acceleration.snapshots_params` on each dataset. | +| `snapshots.bootstrap_on_failure_behavior` | `acceleration.snapshots_bootstrap_on_failure_behavior` on each dataset. | +| `snapshots.enabled` | No replacement. `acceleration.snapshots` defaults to `disabled`, so leave it unset, or set it to `disabled`, on each dataset that should not use snapshots. | + +When the section is set, `snapshots.params` apply to `snapshots.location`, and they replace `runtime.state.params` for snapshots stored under `runtime.state`. `snapshots.enabled: false` still turns snapshots off for every dataset. + +At startup, a Spicepod with the section logs one warning that shows the configuration that keeps the existing snapshots where they are. For example, when `snapshots.location` is `s3://my-bucket/spice/snapshots/` and `runtime.state` is not set, the warning asks to delete the section and set `runtime.state.location: s3://my-bucket/spice/`. When the section's location is already the `snapshots/` folder of `runtime.state.location`, the warning asks to delete the section. In other cases, it asks to set `acceleration.snapshots_location` to the section's location on each dataset that enables snapshots. The warning names the keys of `snapshots.params` to move, never their values. A cluster executor does not log it, because the scheduler that shares its Spicepod does. + ### Conditional writes on the snapshot location Each location holds one `metadata.json` document that records the snapshots of every dataset publishing there, from every Spice instance. Spice changes this document only with conditional writes. It creates the document only if it does not exist yet (`If-None-Match: *`), and updates it only if it still has the version Spice read (`If-Match` on its `ETag`). Google Cloud Storage expresses both conditions with the object generation (`x-goog-if-generation-match`). When another writer changes the document first, the store refuses the write, and Spice reads the document again and reapplies its change. Concurrent publishers therefore do not overwrite each other's entries. The location's store must enforce both conditions. Amazon S3, Azure ADLS Gen2, Google Cloud Storage, and a local `file://` directory do. An S3-compatible store must honor `If-None-Match` and `If-Match` on `PUT`; Spice checks this before it publishes, as described below. -Before each Spice instance creates a dataset's first snapshot, it probes the store by creating, updating, and deleting one `.spice-conditional-write-probe-` object under the location, in at most five requests. When the store rejects conditional writes, or accepts a write whose condition does not hold, Spice uploads no snapshot and does not write `metadata.json`. The snapshot fails with an error that names the dataset, the location, and the reason. Spice keeps this result and does not probe the store again, so restart Spice after moving `location` to a store that enforces conditional writes. +Before each Spice instance creates a dataset's first snapshot, it probes the store by creating, updating, and deleting one `.spice-conditional-write-probe-` object under the location, in at most five requests. When the store rejects conditional writes, or accepts a write whose condition does not hold, Spice uploads no snapshot and does not write `metadata.json`. The snapshot fails with an error that names the dataset, the location, and the reason. Spice keeps this result and does not probe the store again, so restart Spice after moving the snapshot location to a store that enforces conditional writes. When the probe cannot reach a result, for example after a transient error or because the credentials cannot write the probe object, the snapshot is not published and the next attempt probes again. The error asks for permission to create, update, and delete `.spice-conditional-write-probe-*` objects under the location. Every instance that publishes snapshots probes with its own credentials, and each needs permission to create and update these objects. Permission to delete them is needed only for cleanup: a failed delete does not change the probe result and leaves one small probe object behind. @@ -149,24 +203,38 @@ Dataset 'orders' is not creating snapshots while instance 'spice-1' holds its sn A lease lasts twice the dataset's snapshot interval, kept between 30 seconds and 24 hours. The interval is `snapshots_trigger_threshold` with `snapshots_trigger: time_interval` and `refresh_check_interval` with `refresh_complete`, and otherwise 10 minutes. Another instance takes over a lease that has gone unrenewed that long, as measured on its own clock, so clock skew between instances does not expire a live lease. A holder whose snapshot fails releases the lease at once. A holder that loses the lease during an upload does not publish that snapshot over a newer one from the instance that took over. -Each instance is identified by the `SPICE_INSTANCE_ID` environment variable, or by its host name when the variable is unset, so a restarted instance resumes the lease it held. Give each replica a distinct `SPICE_INSTANCE_ID`: two processes with the same identity both upload, and each logs a warning naming the identity. +A holder that shuts down gracefully releases its leases when shutdown starts, so a replacement instance takes over each dataset's snapshots at its next snapshot instead of waiting for the lease to lapse. This keeps readers current during a rolling update that replaces instances with new identities. The release is bounded by [`runtime.shutdown_timeout`](../../reference/spicepod/runtime.md#runtimeshutdown_timeout), and a snapshot already in flight does not take a released lease back. The holder logs: + +```text +Released the snapshot writer lease of dataset 'orders', so another instance can take over its snapshots +``` + +A lease that cannot be released before the timeout lapses on its own. In v2.4.0-rc.2 and earlier, a holder keeps its leases on shutdown, and another instance takes over only after they lapse. + +Each instance is identified by its [instance identity](../../reference/spicepod/runtime#instance-identity): the `SPICE_INSTANCE_ID` environment variable, else `HOSTNAME`, else `COMPUTERNAME`, else the host name. The identity is stable across restarts, so an instance restarted after a crash resumes the lease it held. Give each replica a distinct identity: two processes with the same identity both upload, and a process that finds its lease renewed by the other logs an error naming the identity and the snapshot location: + +```text +Another running instance uses SPICE_INSTANCE_ID 'spice-1' at state location 's3://my-bucket/snapshots/'. Give each instance its own SPICE_INSTANCE_ID (for example the pod name). +``` ### Snapshot location and the Cayenne data tier -Keep `location` on standard object storage: Amazon S3 (`s3://`), Google Cloud Storage (`gs://`), or Azure ADLS Gen2 (`abfss://`, `abfs://`). A local `file://` folder is valid on a single machine. Standard cloud storage is what bucket and object replication, and readers in another region, use. +Keep the snapshot location on standard object storage: Amazon S3 (`s3://`), Google Cloud Storage (`gs://`), or Azure ADLS Gen2 (`abfss://`, `abfs://`). A local `file://` folder is valid on a single machine. Standard cloud storage is what bucket and object replication, and readers in another region, use. -`location` is independent of Cayenne's S3 Express One Zone data tier. `cayenne_file_path` and the `cayenne_s3_*` parameters (`cayenne_s3_region`, `cayenne_s3_zone_ids`, `cayenne_s3_auth`, and the related keys) store Cayenne Vortex files on an Express One Zone directory bucket. That bucket is single-zone storage for the accelerator's data files. It does not substitute for the snapshot bucket, and it does not provide the replication or multi-region reads a standard snapshot location does. See [S3 Express One Zone storage](../../components/data-accelerators/cayenne#aws-s3-express-one-zone-storage). +The snapshot location is independent of Cayenne's S3 Express One Zone data tier. `cayenne_file_path` and the `cayenne_s3_*` parameters (`cayenne_s3_region`, `cayenne_s3_zone_ids`, `cayenne_s3_auth`, and the related keys) store Cayenne Vortex files on an Express One Zone directory bucket. That bucket is single-zone storage for the accelerator's data files. It does not substitute for the snapshot bucket, and it does not provide the replication or multi-region reads a standard snapshot location does. See [S3 Express One Zone storage](../../components/data-accelerators/cayenne#aws-s3-express-one-zone-storage). -The Cayenne [cold tier](../../components/data-accelerators/cayenne#cold-object-store-tier) uses a general-purpose S3 or S3-compatible bucket, set by `cayenne_datalake_location`, not GCS or ADLS. That prefix is the cold data tier, separate from `snapshots.location`. +The Cayenne [cold tier](../../components/data-accelerators/cayenne#cold-object-store-tier) uses a general-purpose S3 or S3-compatible bucket, set by `cayenne_datalake_location`, not GCS or ADLS. That prefix is the cold data tier, separate from the snapshot location. ### Failure behavior -`bootstrap_on_failure_behavior` controls what Spice does when it cannot load the most recent snapshot. +`acceleration.snapshots_bootstrap_on_failure_behavior` controls what Spice does when it cannot load a dataset's most recent snapshot. - `retry` – keep retrying the newest snapshot until it succeeds. - `fallback` – try older snapshot files until one loads successfully. - `warn` – log a warning and continue with an empty acceleration. (Default.) +Once a [`refresh_mode: snapshot`](./refresh-modes/snapshot.md) reader has loaded a snapshot, a later poll that cannot load a newer one keeps serving the loaded snapshot under every setting, increments `dataset_acceleration_refresh_errors` with `reason="snapshot_download_failed"`, and tries again at the next poll. + ## Enable snapshots per dataset Each dataset opts into snapshotting through the `acceleration.snapshots` field. Four modes are available: @@ -185,6 +253,9 @@ acceleration: snapshots_trigger_threshold: # threshold for time_interval or stream_batches snapshots_compaction: enabled | disabled # default: disabled (DuckDB only) snapshots_reset_expiry_on_load: enabled | disabled # default: disabled (DuckDB only with Caching refresh mode) + snapshots_location: # default: the snapshots/ folder of runtime.state.location + snapshots_params: # default: the params of that location + snapshots_bootstrap_on_failure_behavior: warn | retry | fallback # default: warn ``` ### Snapshot triggers @@ -327,12 +398,11 @@ acceleration: ## Complete example ```yaml -snapshots: - enabled: true - location: s3://some_bucket/some_folder/ - bootstrap_on_failure_behavior: warn - params: - s3_auth: iam_role +runtime: + state: + location: s3://some_bucket/some_folder/ + params: + s3_region: us-east-1 datasets: # Batch dataset with refresh-triggered snapshots @@ -367,7 +437,7 @@ datasets: ``` :::info Readiness with append refreshes -Append-mode accelerations that define a `time_column` wait to report ready until the first append refresh completes after snapshot bootstrap. This keeps the dataset out of rotation until the freshest data is available while still benefiting from the snapshot-assisted startup. See [Fast Cold Starts](./data-refresh#fast-cold-starts-with-snapshots) for additional context. +An append-mode acceleration that defines a `time_column` and meets the conditions in [Serving an Existing Acceleration While the Source Is Unavailable](./data-refresh#serving-an-existing-acceleration-while-the-source-is-unavailable) is served as soon as it is restored from a snapshot, including while the source is unavailable, and the first append refresh then catches up from the newest `time_column` value in the acceleration. In v2.4.0-rc.3 and earlier, it waited to report ready until that first append refresh completed. See [Fast Cold Starts](./data-refresh#fast-cold-starts-with-snapshots) for additional context. ::: ## Serve a dataset from published snapshots @@ -438,7 +508,7 @@ The dataset is read-only. Spice rejects a configuration that contradicts reading - **Pair with ephemeral storage:** Deployments commonly place the acceleration file on fast ephemeral disks (such as NVMe instance storage) while relying on snapshots for persistence across restarts. Local NVMe is the recommended medium for accelerations — see [Storage](../../reference/performance-tuning#storage) for the tiers, the instance-store lifetime, and the capacity figures. - **Enable compaction for large datasets:** Use `snapshots_compaction: enabled` for DuckDB accelerations to reduce snapshot size and improve bootstrap performance. - **Tune trigger thresholds for stream datasets:** For high-throughput streaming datasets, balance snapshot frequency against I/O overhead by adjusting `snapshots_trigger_threshold`. -- **Align retention policies:** Apply an object storage lifecycle rule that mirrors the desired snapshot retention policy. +- **Let Spice expire snapshots:** Spice keeps [28 days of snapshots](#snapshot-retention) and deletes older files itself. An expiration rule for current objects on the snapshot location is not needed. When one is required, set its expiration longer than 28 days. Such a rule can still delete the file of a current or newest snapshot older than its expiration. On a bucket with versioning enabled, a Spice delete leaves a noncurrent version, so add a noncurrent-version expiration rule to reclaim that storage. A replication destination keeps deleted files unless the replication carries deletes or the destination has its own lifecycle rule. - **Monitor bootstraps:** Track warning logs emitted when Spice falls back to an empty acceleration so operators can respond quickly if snapshot loading fails. - **Search indexes are not all inside the accelerator file.** A DuckDB HNSW index lives in the DuckDB file, so it is part of that dataset's snapshot. A built-in full-text index does not: the default in-memory Tantivy index is rebuilt on every start, and `index_store: file` writes a separate directory (`.spice/data/fts/...` unless `index_directory` is set) that acceleration snapshots do not upload. For restart parity of full-text search, keep that directory on the same durable volume as the acceleration file, or build the index once on a central tier and serve sidecars from the [search results cache](../caching). See [Where indexes are built](../search#where-indexes-are-built). diff --git a/website/versioned_docs/version-2.4.x/features/distributed-query/index.md b/website/versioned_docs/version-2.4.x/features/distributed-query/index.md index 5e04d693f..d3d68d2fc 100644 --- a/website/versioned_docs/version-2.4.x/features/distributed-query/index.md +++ b/website/versioned_docs/version-2.4.x/features/distributed-query/index.md @@ -128,6 +128,16 @@ spiced --role executor \ Specifying `--scheduler-address` implies `--role executor`. +### Instance Identity + +Each scheduler and executor registers under its [instance identity](../../reference/spicepod/runtime.md#instance-identity): the `SPICE_INSTANCE_ID` environment variable, else `HOSTNAME`, else `COMPUTERNAME`, else the host name. Nodes on separate hosts or Kubernetes pods already have distinct identities. Processes on one host share its host name, so two schedulers, or two executors, on one host each need their own `SPICE_INSTANCE_ID`. This is in addition to what any two `spiced` processes on one host already need, such as their own instance directory, set with `SPICE_CONFIG_DIR`, because each process holds an exclusive lock on its instance directory. + +The identity is stable across restarts, so the scheduler recognizes an executor that restarts under the same identity as the same executor. It logs `Executor '' restarted, so the tasks and shuffle output of its earlier run are rescheduled` and reschedules that work instead of waiting for the earlier run to time out. + +A scheduler restarted after a crash under the same identity waits until its predecessor's heartbeat has stayed unchanged for 30 seconds, then registers. A scheduler whose identity is held by a running scheduler logs an error and is refused. See [Instance identity](../../reference/spicepod/runtime.md#instance-identity). + +In v2.4.0-rc.2 and earlier, a scheduler registered under its `host:port` address and an executor under its advertise address, so two nodes on one host did not need distinct identities. + ### Version Compatibility Run the same Spice version on every scheduler and executor in a cluster, and upgrade all of them together. A scheduler checks the cluster protocol version that each executor reports when it registers and polls for work, and rejects an executor built for a different Spice version. A rejected executor receives no work. The scheduler logs `Rejecting executor : protocol version mismatch`, and the executor logs a warning like the following on each retry: @@ -162,23 +172,23 @@ EXPLAIN SELECT count(id) FROM my_dataset; For long-running queries, the async queries API enables submitting queries for background execution, polling for status, and retrieving paginated results when ready. :::warning -The async queries API is experimental and requires a scheduler state location: `runtime.scheduler.state_location`, or the shared [`runtime.state.location`](../reference/spicepod/runtime#runtimestate) when `state_location` is omitted. +The async queries API is experimental and requires a scheduler state location: the shared [`runtime.state.location`](../../reference/spicepod/runtime.md#runtimestate), or the deprecated `runtime.scheduler.state_location`. ::: ### Prerequisites - Spice runtime running in cluster mode with `--role scheduler` -- `runtime.scheduler.state_location` or `runtime.state.location` configured in the Spicepod (see [High Availability > Configuration](#configuration)) +- `runtime.state.location` configured in the Spicepod (see [High Availability > Configuration](#configuration)) - At least one executor node connected to the scheduler ### Enabling Async Queries -Configure `runtime.scheduler.state_location` in your `spicepod.yaml` to enable the async queries API. A [`runtime.state`](../reference/spicepod/runtime#runtimestate) location also works when `state_location` is omitted: +Configure [`runtime.state.location`](../../reference/spicepod/runtime.md#runtimestate) in your `spicepod.yaml` to enable the async queries API. The scheduler keeps its state in the location's `cluster/` folder: ```yaml runtime: - scheduler: - state_location: s3://my-bucket/spice-state + state: + location: s3://my-bucket/spice-state params: s3_region: us-east-1 ``` @@ -189,10 +199,12 @@ For local development: ```yaml runtime: - scheduler: - state_location: "file://.data/scheduler-state" + state: + location: file:///tmp/spice-state/ ``` +`runtime.scheduler.state_location` and `runtime.scheduler.params` are deprecated. A scheduler that sets them logs a startup warning that shows how to move to `runtime.state`. See [`runtime.scheduler`](../../reference/spicepod/runtime.md#runtimescheduler). + ### Query Ownership A query is owned by the principal that submitted it, and **only that principal can list, poll, fetch results from, or cancel it**. This applies to the HTTP, Arrow Flight, and CLI surfaces alike. @@ -643,7 +655,7 @@ When a query fails, the `error` object contains an `error_code` field: ### Storage Layout -Job state and result chunks are stored in the shared object store configured via `runtime.scheduler.state_location`, or `runtime.state.location` when that is omitted. `{base_prefix}` is the root of `state_location`, or the `cluster/` folder below the path of `runtime.state.location` (see [Runtime state layout](../../reference/spicepod/runtime.md#runtime-state-layout)): +Job state and result chunks are stored in the shared object store configured via `runtime.state.location`, or the deprecated `runtime.scheduler.state_location`. `{base_prefix}` is the `cluster/` folder below the path of `runtime.state.location`, or the root of `state_location` (see [Runtime state layout](../../reference/spicepod/runtime.md#runtime-state-layout)): ``` {base_prefix}/ @@ -655,7 +667,7 @@ Job state and result chunks are stored in the shared object store configured via │ └── ... ``` -Each scheduler deletes expired jobs, with their result chunks, every 10 minutes. Only jobs whose results have already expired are deleted. An expired job answers its owner with `410 Gone` until the next cleanup deletes it, and with `404 Not Found` after that. +Each scheduler deletes, with their result chunks, the jobs whose results have expired and the jobs that failed or were cancelled at least 12 hours ago. It runs this cleanup about once an hour, at a random time between 30 and 90 minutes after the previous run. An expired job answers its owner with `410 Gone` until the next cleanup deletes it, and with `404 Not Found` after that. A cleanup that deletes jobs logs how many it deleted. ### Defaults and Limitations @@ -666,7 +678,7 @@ Each scheduler deletes expired jobs, with their result chunks, every 10 minutes. | List limit | 100 queries | - Only available in cluster mode with `--role scheduler` -- Requires `runtime.scheduler.state_location` or `runtime.state.location` to be configured +- Requires `runtime.state.location`, or the deprecated `runtime.scheduler.state_location`, to be configured - The `format` query parameter on the results endpoint is declared but not yet implemented (results are always JSON over HTTP, Arrow IPC over Flight) - Result TTL is not yet configurable per-query (fixed at 12 hours) - Chunk size is not yet configurable per-query (fixed at 10,000 rows) @@ -704,12 +716,12 @@ In an HA cluster: ### Configuration -Enable HA by configuring `runtime.scheduler.state_location` (or the shared [`runtime.state.location`](../reference/spicepod/runtime#runtimestate)) in the Spicepod to point to an S3-compatible object store: +Enable HA by configuring the shared [`runtime.state.location`](../../reference/spicepod/runtime.md#runtimestate) in the Spicepod to point to an S3-compatible object store. The schedulers keep their shared state in its `cluster/` folder: ```yaml runtime: - scheduler: - state_location: s3://my-bucket/spice-cluster + state: + location: s3://my-bucket/spice params: s3_region: us-east-1 ``` @@ -728,7 +740,7 @@ This failover applies to async queries, which require a scheduler state location ### S3 Configuration -The `runtime.scheduler.params` section supports the following S3 parameters: +`runtime.state.params` supports the following S3 parameters for the scheduler's state: | Parameter | Description | Default | |--------------------| ----------------------------------------------------- | ---------- | @@ -745,8 +757,8 @@ Example with explicit credentials: ```yaml runtime: - scheduler: - state_location: s3://my-bucket/spice-cluster + state: + location: s3://my-bucket/spice params: s3_region: us-east-1 s3_auth: key @@ -764,8 +776,8 @@ A `gs://` state location takes the [GCS data connector](../../components/data-co ```yaml runtime: - scheduler: - state_location: s3://my-bucket/spice-cluster + state: + location: s3://my-bucket/spice params: s3_region: us-east-1 ``` diff --git a/website/versioned_docs/version-2.4.x/features/observability/index.md b/website/versioned_docs/version-2.4.x/features/observability/index.md index ef96fc1a0..68410969a 100644 --- a/website/versioned_docs/version-2.4.x/features/observability/index.md +++ b/website/versioned_docs/version-2.4.x/features/observability/index.md @@ -292,7 +292,7 @@ Component-specific instruments are documented with their component. The `cayenne | `dataset_acceleration_refresh_data_fetches_skipped`

Number of refresh data fetches skipped due to unchanged file metadata. | _count_ | `dataset`, `mode` | | `dataset_acceleration_refresh_duration_ms`

Duration in milliseconds to load a full or appended refresh data. | _histogram_ | `dataset`, `mode` | | `dataset_acceleration_refresh_fallbacks`

Number of `append` refreshes of an Iceberg source that fall back to reading by time because the table's snapshot history cannot be followed, split by `reason`: `table_replaced`, `snapshot_not_ancestor`, `table_has_no_snapshot`, or `rows_rewritten_without_primary_key`. The first refresh, datasets with computed columns, and one-off refreshes with their own `refresh_sql` also read by time but are not counted. See [Iceberg sources follow table snapshots](../data-acceleration/refresh-modes/append.md#iceberg-sources-follow-table-snapshots). | _count_ | `dataset`, `mode`, `reason` | -| `dataset_acceleration_refresh_errors`

Number of errors refreshing the dataset. Always split by `reason`: `object_generation_changed` (a listed object was replaced while the scan still held the previous generation — a retriable precondition failure, not corruption), `parquet_decode` (the Parquet payload itself failed to decode), or `other`. | _count_ | `dataset`, `mode`, `reason` | +| `dataset_acceleration_refresh_errors`

Number of errors refreshing the dataset. Always split by `reason`: `object_generation_changed` (a listed object was replaced while the scan still held the previous generation — a retriable precondition failure, not corruption), `parquet_decode` (the Parquet payload itself failed to decode), `snapshot_not_found` (a `refresh_mode: snapshot` reader found no published snapshot and keeps serving the one it loaded), `snapshot_older_than_loaded` (the published snapshot is older than the loaded one, so it was not loaded), `snapshot_download_failed` (a newer published snapshot could not be loaded, so the reader keeps serving the one it loaded), or `other`. Snapshot readers report `mode="snapshot"`. | _count_ | `dataset`, `mode`, `reason` | | `dataset_acceleration_refresh_lag_ms`

Difference between the maximum time_column value after and before the refresh operation, in milliseconds. | _gauge_ | `dataset`, `mode` | | `dataset_acceleration_refresh_rows_written`

Cumulative number of rows read from the federated source and written into the accelerated table, counted as they are read. A row the table does not keep is also counted in `dataset_acceleration_rows_superseded` once its refresh succeeds. | _count_ | `dataset` | | `dataset_acceleration_refresh_bytes_written`

Cumulative number of bytes (Arrow in-memory size) read from the federated source and written into the accelerated table. | _count_ | `dataset` | @@ -302,7 +302,8 @@ Component-specific instruments are documented with their component. The `cayenne | `dataset_acceleration_snapshot_bootstrap_bytes`

Number of bytes downloaded when bootstrapping the acceleration from a snapshot. | _gauge_ | `dataset` | | `dataset_acceleration_snapshot_bootstrap_checksum`

Checksum of the snapshot downloaded during bootstrap (emitted with `checksum` attribute). | _gauge_ | `dataset`, `checksum` | | `dataset_acceleration_snapshot_bootstrap_duration_ms`

Time in milliseconds taken to download the snapshot used to bootstrap acceleration. | _count_ | `dataset` | -| `dataset_acceleration_snapshot_failure_count`

Number of failures encountered while writing snapshots. | _count_ | `dataset` | +| `dataset_acceleration_snapshot_failure_count`

Number of snapshots the dataset gave up creating after retries, whichever step failed. Each snapshot counts once, not once per retry. A dataset that creates snapshots reports the series from `0` when it loads. | _count_ | `dataset` | +| `dataset_acceleration_snapshot_skipped_count`

Number of snapshot creations skipped because the data did not change. | _count_ | `dataset` | | `dataset_acceleration_snapshot_write_bytes`

Number of bytes written for the most recent snapshot. | _gauge_ | `dataset` | | `dataset_acceleration_snapshot_write_checksum`

Checksum of the most recent snapshot write (emitted with `checksum` attribute). | _gauge_ | `dataset`, `checksum` | | `dataset_acceleration_snapshot_write_duration_ms`

Time in milliseconds taken to write the latest snapshot to object storage. | _histogram_ | `dataset` | @@ -416,6 +417,9 @@ Component-specific instruments are documented with their component. The `cayenne | `spiced_cpu_budget_millicores`

CPU millicores the runtime sizes itself for — the same entitlement, exact. | _gauge_ | `source` (`configured`, `cgroup_quota`, `request_burst`, `all_cores`, `affinity`, `fallback`) | | `spiced_cpu_limit_millicores`

CPU limit read from the cgroup CPU quota (Kubernetes `limits.cpu`). Not emitted when the cgroup expresses no limit, so it is never confused with a real zero. | _gauge_ | — | | `spiced_cpu_request_millicores`

The pod's own CPU request (Kubernetes `requests.cpu`), in millicores. Sizing derives from it only when nothing outranks it — no CPU limit, and no explicit `runtime.cpu.cores`; the `source` label above reports whether it did. Not emitted when no request was declared. The runtime cannot read the request itself: the pod spec must pass it in through `SPICE_CPU_REQUEST_MILLICORES` (see below). | _gauge_ | — | +| `state_store_cleanup_deleted_total`

Objects that cleanup deleted from a state location because no instance needs them: async query jobs and their result chunks (`scheduler`), rate-control files (`rate_control`), and snapshot files (`snapshots`). Published at zero. See [Metrics for the state location](../../reference/spicepod/runtime.md#metrics-for-the-state-location). | _count_ | `feature` | +| `state_store_conflicts_total`

Conditional writes to a state location that the store refused because another writer created or changed the object first. Published at zero. | _count_ | `feature` | +| `state_store_request_duration_ms`

Time in milliseconds each request to a state location took. `operation` is `read`, `conditional_write`, `write`, `list`, or `delete`. | _histogram_ | `feature`, `operation` | | `table_set_interner_rows`

Number of distinct input-table sets currently shared across cache entries. Populated by both the SQL results cache and the search results cache, so its bytes cannot be attributed to `results_cache_size_bytes` alone — read it against `search_results_cache_size_bytes` too. Shared table sets are not charged to any entry, so they appear in neither. | _gauge_ | — | | `table_set_interner_value_bytes`

Total size of the shared input-table sets, counted once per distinct set rather than once per cache entry holding it. | _gauge_ | — | | `table_set_interner_overhead_bytes`

The table-set pool's own bookkeeping: its hash-map slots and bucket vectors. | _gauge_ | — | diff --git a/website/versioned_docs/version-2.4.x/monitoring/datadog/index.md b/website/versioned_docs/version-2.4.x/monitoring/datadog/index.md index d5ade754d..619b6879a 100644 --- a/website/versioned_docs/version-2.4.x/monitoring/datadog/index.md +++ b/website/versioned_docs/version-2.4.x/monitoring/datadog/index.md @@ -98,7 +98,7 @@ The Spice dashboard identifies each running instance by a `service_instance_id` Datadog does not error on a missing tag. Without `service_instance_id`, each panel renders a single `N/A` series summing every instance — a 12-replica deployment reports 12 times its real dataset count rather than showing nothing. ::: -Panels from the Kubernetes integration (CPU, memory, and PVC utilization) group by `pod_name` instead, as those metrics never carry `service_instance_id`. +Panels from the Kubernetes integration (CPU, memory, and PVC utilization, and pod OOM kills, `CrashLoopBackOff` states, and restarts from `kubernetes_state`) group by `pod_name` instead, as those metrics never carry `service_instance_id`. ## Import the Spice Datadog Dashboard diff --git a/website/versioned_docs/version-2.4.x/reference/spicepod/datasets.md b/website/versioned_docs/version-2.4.x/reference/spicepod/datasets.md index 77ad540b1..efd7ba539 100644 --- a/website/versioned_docs/version-2.4.x/reference/spicepod/datasets.md +++ b/website/versioned_docs/version-2.4.x/reference/spicepod/datasets.md @@ -519,7 +519,7 @@ For the per-engine adjustments each profile applies, and for guidance on choosin ## `acceleration.snapshots` -Optional. Controls how this dataset participates in managed acceleration snapshots. Requires the Spicepod to configure the top-level [`snapshots` block](.#snapshots), the acceleration engine to be `duckdb`, `sqlite`, `cayenne`, or `turso`, and `mode: file` with a dataset-specific file path (for example `acceleration.params.duckdb_file: /nvme/my_dataset.db`). +Optional. Controls how this dataset participates in managed acceleration snapshots. Requires a snapshot location, from [`runtime.state.location`](./runtime.md#runtimestate), [`acceleration.snapshots_location`](#accelerationsnapshots_location), or the deprecated top-level [`snapshots.location`](./index.md#snapshotslocation), the acceleration engine to be `duckdb`, `sqlite`, `cayenne`, or `turso`, and `mode: file` with a dataset-specific file path (for example `acceleration.params.duckdb_file: /nvme/my_dataset.db`). Supported values: @@ -588,6 +588,18 @@ Supported values: - `disabled` (default) – Snapshot expiry is not reset on load; the original expiry time is preserved. - `enabled` – Reset the snapshot expiry when it is loaded during bootstrap. +## `acceleration.snapshots_location` + +Optional. The object store location of this dataset's snapshots, for example `s3://my-bucket/spice/snapshots/`. Defaults to the `snapshots/` folder of [`runtime.state.location`](./runtime.md#runtimestate). The deprecated top-level [`snapshots.location`](./index.md#snapshotslocation), when set, applies to datasets that do not set this field. See [Override the snapshot location per dataset](../../features/data-acceleration/snapshots#override-the-snapshot-location-per-dataset). + +## `acceleration.snapshots_params` + +Optional. Object store params for this dataset's snapshots, such as `s3_region` or `s3_queue_url`. When set, they are the only params the dataset's snapshots use. When not set, a dataset with `snapshots_location` uses no params, and a dataset whose snapshots are stored under `runtime.state` uses `runtime.state.params`. The deprecated top-level [`snapshots.params`](./index.md#snapshotsparams), when set, apply to `snapshots.location` and replace `runtime.state.params` for snapshots stored under `runtime.state`. The accepted keys are listed in [Configure snapshot storage](../../features/data-acceleration/snapshots#configure-snapshot-storage). + +## `acceleration.snapshots_bootstrap_on_failure_behavior` + +Optional. What Spice does when it cannot load this dataset's most recent snapshot: `warn` (default), `retry`, or `fallback`. The deprecated top-level [`snapshots.bootstrap_on_failure_behavior`](./index.md#snapshotsbootstrap_on_failure_behavior), when set, applies to datasets that do not set this field. See [Failure behavior](../../features/data-acceleration/snapshots#failure-behavior). + ## `acceleration.refresh_mode` Optional. How to refresh the dataset. The following values are supported: diff --git a/website/versioned_docs/version-2.4.x/reference/spicepod/index.md b/website/versioned_docs/version-2.4.x/reference/spicepod/index.md index fee0df1d9..53cf4120b 100644 --- a/website/versioned_docs/version-2.4.x/reference/spicepod/index.md +++ b/website/versioned_docs/version-2.4.x/reference/spicepod/index.md @@ -118,6 +118,10 @@ datasets: ## `snapshots` {#snapshots} +:::warning[Deprecated] +The top-level `snapshots` section is deprecated and will be removed in a future release. Snapshots are stored in the `snapshots/` folder of [`runtime.state.location`](./runtime.md#runtimestate). A dataset overrides the location, params, and failure behavior with [`acceleration.snapshots_location`](./datasets.md#accelerationsnapshots_location), [`acceleration.snapshots_params`](./datasets.md#accelerationsnapshots_params), and [`acceleration.snapshots_bootstrap_on_failure_behavior`](./datasets.md#accelerationsnapshots_bootstrap_on_failure_behavior). A Spicepod that sets the section logs a startup warning that shows the replacement configuration. See [Migrate from the top-level `snapshots` section](../../features/data-acceleration/snapshots.md#migrate-from-the-top-level-snapshots-section). +::: + Optional. Configure managed acceleration snapshots that Spice can use to bootstrap file-based accelerations. When enabled, datasets that opt in with [`acceleration.snapshots`](spicepod/datasets#accelerationsnapshots) will download database files from the snapshot location if the local file is missing, and will optionally write new snapshots after each refresh. DuckDB, SQLite, Cayenne, and Turso accelerations running in `mode: file` are supported, and each dataset must write to its own file path. ```yaml @@ -137,7 +141,7 @@ Enable or disable snapshot management globally. Defaults to `true`. The folder where snapshots are stored. Supports S3 bucket URIs (`s3://bucket/prefix/`), Azure ADLS Gen2 URIs (`abfss://container@account.dfs.core.windows.net/path/`), Google Cloud Storage URIs (`gs://bucket/prefix/`), and local filesystem URIs (`file:///path/to/folder/`). The value must be a URI with a scheme: a bare filesystem path such as `/var/spice/snapshots` is not a valid URI, so it fails to parse and snapshots are disabled with an error logged. The location must resolve to a single folder; Spice creates per-dataset folders underneath using Hive-style partitions (`month=YYYY-MM/day=YYYY-MM-DD/dataset=`). -When `location` is not set and the Spicepod sets [`runtime.state`](spicepod/runtime#runtimestate), snapshots go to `/snapshots/`. A `location` that is set takes precedence over `runtime.state`. See [Store snapshots under `runtime.state`](../features/data-acceleration/snapshots#store-snapshots-under-runtimestate). +When `location` is not set and the Spicepod sets [`runtime.state`](spicepod/runtime#runtimestate), snapshots go to `/snapshots/`. A `location` that is set takes precedence over `runtime.state`, and a dataset's `acceleration.snapshots_location` takes precedence over `location`. See [Store snapshots under `runtime.state`](../features/data-acceleration/snapshots#store-snapshots-under-runtimestate). The location's store must enforce conditional writes (create only if absent, update only at the version read), because every publisher updates the location's shared `metadata.json` with them. Amazon S3, Azure ADLS Gen2, Google Cloud Storage, and local `file://` directories do. Each Spice instance probes the store before it creates a dataset's first snapshot, and refuses a store that does not enforce them. The probe creates, updates, and then deletes a `.spice-conditional-write-probe-` object under the location. The credentials of every instance that publishes snapshots need permission to create and update that object. Permission to delete it is needed only for cleanup: a failed delete does not change the probe result and leaves the probe object behind. See [Conditional writes on the snapshot location](../features/data-acceleration/snapshots#conditional-writes-on-the-snapshot-location). @@ -145,7 +149,7 @@ Keep a cloud `location` on standard S3, GCS, or ADLS so bucket or object replica ### `snapshots.bootstrap_on_failure_behavior` -Controls what happens when Spice cannot load the most recent snapshot on startup. Valid values: +Controls what happens when Spice cannot load the most recent snapshot on startup, for datasets that do not set `acceleration.snapshots_bootstrap_on_failure_behavior`. Valid values: - `warn` (default) – Log a warning and continue with an empty acceleration. - `retry` – Retry the newest snapshot until it loads successfully. @@ -155,7 +159,7 @@ Controls what happens when Spice cannot load the most recent snapshot on startup Optional key-value map passed to the snapshot storage layer. When `location` points to S3, `params` accepts these [S3 parameters](../components/data-connectors/s3): `s3_region`, `s3_endpoint`, `s3_auth` (`iam_role` or `key`; default `iam_role`), `s3_key`, `s3_secret`, `s3_session_token`, `s3_queue_url`, `client_timeout`, and `allow_http`. Spice ignores other keys and logs a warning for each. Snapshots default to `s3_auth: iam_role`, which differs from the S3 dataset default of `public`. With `s3_auth: key`, both `s3_key` and `s3_secret` must be set, or the snapshot location is not used and Spice logs an error. Azure ADLS locations also accept the Azure connector parameters, and GCS locations the GCS connector parameter names (such as `gcs_service_account_key`), for explicit credential overrides; when no overrides are supplied, Spice reads standard environment variables for each cloud provider. Values can reference secrets with `${secrets:}` for all three. An invalid S3 parameter value, such as `s3_auth: public`, disables snapshots for the dataset with an error; Spice does not fall back to default parameters. When `s3_queue_url` is set, the dataset fails to register with the parameter error instead. -For snapshots stored under `runtime.state`, `params` replaces `runtime.state.params` as a whole. When `params` is not set, those snapshots use `runtime.state.params`. +For snapshots stored under `runtime.state`, `params` replaces `runtime.state.params` as a whole. When `params` is not set, those snapshots use `runtime.state.params`. A dataset that sets `acceleration.snapshots_params` uses those instead, and a dataset that sets `acceleration.snapshots_location` without `snapshots_params` uses no params. ## `models` diff --git a/website/versioned_docs/version-2.4.x/reference/spicepod/runtime.md b/website/versioned_docs/version-2.4.x/reference/spicepod/runtime.md index 9f32ad572..738bde7ff 100644 --- a/website/versioned_docs/version-2.4.x/reference/spicepod/runtime.md +++ b/website/versioned_docs/version-2.4.x/reference/spicepod/runtime.md @@ -1210,10 +1210,11 @@ Configures the cluster scheduler when running Spice in [cluster mode](../../depl ```yaml runtime: - scheduler: - state_location: s3://my-bucket/spice-cluster-state/ + state: + location: s3://my-bucket/spice-state/ params: s3_region: us-east-1 + scheduler: partition_assignment_interval: 30s max_partition_assignments_per_interval: 100 max_partitions_per_executor: 1000 @@ -1222,13 +1223,23 @@ runtime: | Parameter name | Optional | Default | Description | | -------------------------------------------------- | -------- | ------- | ---------------------------------------------------------------------- | -| `state_location` | Yes | - | Root URI for shared cluster state storage (e.g. `s3://bucket/path/`). Required unless [`runtime.state`](#runtimestate) is set, in which case `runtime.state.location` is used. | -| `params` | Yes | - | Object store parameters (e.g. `s3_region`). See [State location parameters](#state-location-parameters). When `state_location` is omitted, defaults to `runtime.state.params`. | +| `state_location` | Yes | - | Deprecated. Root URI for shared cluster state storage (e.g. `s3://bucket/path/`). Set [`runtime.state.location`](#runtimestate) instead. | +| `params` | Yes | - | Deprecated. Object store parameters (e.g. `s3_region`). See [State location parameters](#state-location-parameters). Set `runtime.state.params` instead. When `state_location` is omitted, defaults to `runtime.state.params`. | | `partition_assignment_interval` | Yes | `30s` | How often the scheduler runs partition assignment cycles. | | `max_partition_assignments_per_interval` | Yes | `100` | Maximum number of partition assignments per interval. | | `max_partitions_per_executor` | Yes | `1000` | Maximum number of partitions assigned to a single executor. | | `partition_discovery_timeout` | Yes | `60s` | How long the scheduler waits for executor discovery before timing out. | +The scheduler keeps its shared cluster state in the `cluster/` folder of [`runtime.state.location`](#runtimestate) and reads it with `runtime.state.params`. `runtime.scheduler.state_location` and `runtime.scheduler.params` are deprecated and will be removed in a future release. They still take precedence over `runtime.state`, and a scheduler or standalone instance that sets either one logs a startup warning that shows how to move to `runtime.state`: + +| Current setting | What the warning says | +| --------------- | --------------------- | +| `state_location` is the `cluster/` folder of `runtime.state.location` | Remove `state_location`: the state is already there. | +| `state_location` ends in `cluster/` and `runtime.state` is not set | Remove `state_location` and set `runtime.state.location` to its parent, for example `s3://my-bucket/spice/` for `s3://my-bucket/spice/cluster/`, which keeps the state where it is. | +| Any other `state_location`, and every `abfs://` or `abfss://` `state_location` | No `runtime.state` location holds the current state. Removing `state_location` (and setting `runtime.state.location`, when it is not set) makes the scheduler start empty in the new folder, so partition assignments are rebuilt and async query jobs are lost. An `abfs://` or `abfss://` `state_location` keeps its state at the container root, which is never a `runtime.state` folder. | + +The warning also asks to move the keys of `runtime.scheduler.params` into `runtime.state.params`. When `runtime.state.params` are already set, it asks to remove `runtime.scheduler.params` instead, because `runtime.state.params` serve every feature that keeps state under `runtime.state`, and to check that they give the scheduler access to its state. The warning names parameter keys, never their values. + ## `runtime.state` Optional. Sets one shared object store for runtime state, so each feature that persists state does not need its own location. Supported URI schemes are `file://`, `s3://`, `gs://` (or its alias `gcs://`), `abfs://`, and `abfss://`. @@ -1252,9 +1263,9 @@ The following features store their state under `runtime.state` when their own se | Feature | Uses `runtime.state` when | | ------- | ------------------------- | | [SQL results cache warmup](../../features/caching#warming-the-cache-after-a-restart) | `runtime.caching.sql_results.warmup` is `on_first_refresh`. Without `runtime.state`, recorded query plans are written to `.spice/data/results_cache_warmup.json`. | -| [Cluster scheduler](#runtimescheduler) | `runtime.scheduler.state_location` is not set. The scheduler's other settings still apply. | +| [Cluster scheduler](#runtimescheduler) | The deprecated `runtime.scheduler.state_location` is not set. The scheduler's other settings still apply. | | [HTTP rate control](#limits-across-instances) | Always, in a Spice.ai Enterprise build: instances that share the location coordinate per-second and per-minute limits for every origin that sets one. Open-source builds keep rate-control state in memory. Rate control has no location of its own. | -| [Acceleration snapshots](../../features/data-acceleration/snapshots#store-snapshots-under-runtimestate) | A dataset enables snapshots and `snapshots.location` is not set. Snapshots are available in Spice.ai Enterprise. | +| [Acceleration snapshots](../../features/data-acceleration/snapshots#store-snapshots-under-runtimestate) | A dataset enables snapshots and sets no `acceleration.snapshots_location`, and the deprecated top-level `snapshots.location` is not set. Snapshots are available in Spice.ai Enterprise. | A location set in a feature's own section takes precedence over `runtime.state`. Changing `runtime.state` requires a restart. @@ -1278,7 +1289,7 @@ For example, `spice-state/cluster.json` moves to `spice-state/cluster/cluster.js Move the rate-control objects first, one spicepod folder at a time, and move only the objects named `-<16 hex digits>.json` in that folder. A spicepod name can match another folder in the table, such as `heartbeats`, `jobs`, `scheduler`, `cluster`, `rate-control`, or `results-cache`, and moving whole folders would then mix rate-control objects with scheduler objects. Move the scheduler and warmup objects after that. -To keep the scheduler's previous layout instead of moving its objects, set `runtime.scheduler.state_location` to the old location. A scheduler with its own `state_location` does not use `runtime.state.params`, so also copy any parameters the location needs, such as credentials or an endpoint, into `runtime.scheduler.params`. Rate control and warmup have no location of their own, so their objects must be moved. Otherwise, the scheduler starts with fresh cluster state, rate control starts with no shared usage, and warmup records new query shapes. +To keep the scheduler's previous layout instead of moving its objects, set the deprecated `runtime.scheduler.state_location` to the old location. A scheduler with its own `state_location` does not use `runtime.state.params`, so also copy any parameters the location needs, such as credentials or an endpoint, into `runtime.scheduler.params`. Rate control and warmup have no location of their own, so their objects must be moved. Otherwise, the scheduler starts with fresh cluster state, rate control starts with no shared usage, and warmup records new query shapes. ::: @@ -1298,9 +1309,45 @@ Spice stops at startup and exits with an error in these cases: v2.4.0-rc.1 did not check the location. After an upgrade from rc.1, an instance whose `runtime.state` credentials are read-only does not become ready until they can write, and an instance whose store does not enforce conditional writes stops at startup. +### Metrics for the state location + +Spice reports each feature's requests to its state location on the [metrics endpoint](../../features/observability/index.md#available-metrics). The metrics cover the `runtime.state` location and a location set in a feature's own section, such as a dataset's `acceleration.snapshots_location`. + +| Metric | Type | Description | +| ------ | ---- | ----------- | +| `state_store_request_duration_ms` | Histogram | Time in milliseconds of each request, labeled by `feature` and `operation`. | +| `state_store_conflicts_total` | Counter | Conditional writes the store refused because another writer created or changed the object first, labeled by `feature`. | +| `state_store_cleanup_deleted_total` | Counter | Objects that cleanup deleted, labeled by `feature`. | + +`feature` is `scheduler`, `rate_control`, `results_cache_warmup`, or `snapshots`. `operation` is `read`, `conditional_write`, `write`, `list`, or `delete`. A `write` is an unconditional upload, including a multipart upload, a copy, or a rename; snapshot files are uploaded this way. A read is timed until the response starts, a list or delete until it ends, and a multipart upload until it completes or is aborted. The two counters are published at zero for every `feature` when the runtime starts, so a missing series points to a scrape problem rather than an idle feature. The [startup check](#startup-check-of-the-state-location) is not measured. + +A rising `state_store_conflicts_total` means instances that share the location are contending for the same objects. + +### Instance identity + +Every feature that coordinates with other instances through shared state names the instance with one identity. Spice reads the identity once per process from the first of these that is set and not empty: the `SPICE_INSTANCE_ID` environment variable, the `HOSTNAME` environment variable, the `COMPUTERNAME` environment variable, and otherwise the operating system's host name. The identity stays the same across restarts, so a restarted instance resumes the state it held. + +| Feature | Uses the identity for | +| ------- | --------------------- | +| [Cluster scheduler](../../features/distributed-query#instance-identity) | Its registration in the shared cluster state. | +| [Cluster executor](../../features/distributed-query#instance-identity) | Its executor ID when it registers with the schedulers. | +| [Acceleration snapshots](../../features/data-acceleration/snapshots#several-instances-creating-one-datasets-snapshots) | The holder of a dataset's snapshot writer lease. | +| [HTTP rate control](#limits-across-instances) | The name of each process's share of the per-window budget, paired with a per-process ID so that a restarted instance does not reuse its earlier run's share. | +| [PostgreSQL replication](../../features/cdc/postgres-replication#multi-replica-deployments) | The default replication slot name. | + +Give each instance that shares state its own identity. On Kubernetes, `HOSTNAME` defaults to the pod name, which is already distinct. Instances on one host share its host name, so two schedulers or two executors started on the same machine each need their own `SPICE_INSTANCE_ID`. When Spice finds another running process with the same identity, it logs an error such as: + +```text +Another running instance uses SPICE_INSTANCE_ID 'spice-0' at state location 's3://my-bucket/spice-state/'. Give each instance its own SPICE_INSTANCE_ID (for example the pod name). +``` + +A scheduler logs this error, and its registration is refused, when the scheduler already registered under its identity keeps sending heartbeats. A scheduler restarted after a crash is not reported as a duplicate, because the heartbeat its predecessor left stops advancing. It waits until it has seen that heartbeat unchanged for 30 seconds, then registers in its predecessor's place. In v2.4.0-rc.2 and earlier, a scheduler restarted within 30 seconds of its predecessor's last heartbeat failed to register and exited. A scheduler also logs the error when a second executor process connects under an ID that a running executor already uses, and a snapshot writer logs it when another process renews its lease. HTTP rate control does not detect a shared identity. + ### State location parameters -These rules apply to `runtime.state.params`, and to the `params` of a feature that sets its own state location, such as [`runtime.scheduler`](#runtimescheduler). A [snapshot location](../../features/data-acceleration/snapshots#configure-snapshot-storage) set with `snapshots.location` takes its parameters from `snapshots.params`. Snapshots stored under `runtime.state` use `runtime.state.params`, unless `snapshots.params` is set, which replaces them. +These rules apply to `runtime.state.params`, and to the `params` of a feature that sets its own state location, such as [`runtime.scheduler`](#runtimescheduler). Snapshots stored under `runtime.state` use `runtime.state.params`. A dataset that sets [`acceleration.snapshots_params`](./datasets.md#accelerationsnapshots_params) uses those instead, and a dataset that sets `acceleration.snapshots_location` without `snapshots_params` uses no params. A location set with the deprecated top-level `snapshots.location` takes its parameters from `snapshots.params`, and `snapshots.params`, when set, also replace `runtime.state.params` for snapshots stored under `runtime.state`. See [Configure snapshot storage](../../features/data-acceleration/snapshots#configure-snapshot-storage). + +`runtime.state.params` can hold `s3_queue_url`, the SQS queue for [snapshot notifications](../../features/data-acceleration/refresh-modes/snapshot#reload-on-s3-event-notifications). The scheduler, HTTP rate control, and SQL results cache warmup accept it there without a warning and do not use it. A feature's own `params`, such as `runtime.scheduler.params`, still log a warning for it. An `s3://` location takes `s3_region`, `s3_endpoint`, `s3_auth`, `s3_key`, `s3_secret`, `s3_session_token`, `client_timeout`, and `allow_http`. `s3_auth` is `iam_role` (the default) or `key`. With `s3_auth: key`, both `s3_key` and `s3_secret` must be set. A location that sets `s3_auth: key` without both keys fails instead of connecting with the credentials from the environment, and the error reads: diff --git a/website/versioned_docs/version-2.4.x/reference/sql/index.md b/website/versioned_docs/version-2.4.x/reference/sql/index.md index 8e7722716..0ebbb507d 100644 --- a/website/versioned_docs/version-2.4.x/reference/sql/index.md +++ b/website/versioned_docs/version-2.4.x/reference/sql/index.md @@ -143,4 +143,30 @@ Spice uses Apache Arrow data types internally. For data type compatibility with Use `CAST(expression AS type)` or `expression::type` to convert between types. +#### Comparing Decimals with Fractional Literals + +A fractional number literal such as `12345678.90` is planned as a `DOUBLE` (`Float64`). When the literal is compared with or cast to a `DECIMAL`, Spice converts it from its decimal digits rather than from its binary floating-point value, within the limits described below, so the comparison matches the value as written: + +```sql +SELECT count(*) AS n +FROM (SELECT CAST('12345678.90' AS DECIMAL(10, 2)) AS price) AS t +WHERE price = 12345678.90; +``` + +``` ++---+ +| n | ++---+ +| 1 | ++---+ +``` + +Bound `Float64` and `Float32` query parameters, such as `$1` in a [prepared statement](sql/prepared_statements), have no written digits, so Spice converts them from the shortest decimal spelling of the value. + +For both literals and parameters, the conversion applies when the digits fit: at most 15 significant digits and 15 digits before the decimal point (6 for a `Float32` parameter), no more fraction digits than the decimal's scale, and no more digits than its precision. Any other value keeps the standard floating-point cast, with its rounding and overflow behavior. For example, `CAST(1.234 AS DECIMAL(10, 2))` needs rounding, `CAST(0.30000000000000004 AS DECIMAL(38, 18))` has 17 significant digits, and a bound `1e30` cast to `DECIMAL(38, 0)` has 31 digits before the decimal point, so all three keep the floating-point cast. + +As an exception, a literal that a `Float64` stores as a different, shorter number, such as `4.0700000000000003` (stored as `4.07`), is converted from its written digits and rounded half away from zero to the decimal's scale. When one statement writes two different numbers that read as the same `Float64`, such as `price = 4.0700000000000003 OR price = 4.07`, Spice cannot tell those literals apart, and both keep the floating-point cast. + +To compare against a value outside these limits exactly, write it as a string cast, for example `CAST('0.30000000000000004' AS DECIMAL(38, 18))`, or pass a `Decimal128` parameter. + Refer to each section for detailed syntax, supported features, and examples. diff --git a/website/versioned_docs/version-2.4.x/reference/sql/json.md b/website/versioned_docs/version-2.4.x/reference/sql/json.md index ad9fa9b4e..545d09e84 100644 --- a/website/versioned_docs/version-2.4.x/reference/sql/json.md +++ b/website/versioned_docs/version-2.4.x/reference/sql/json.md @@ -651,10 +651,12 @@ The scalar form takes only the JSON argument and always runs with default caps; ## Federation and Pushdown These functions come from a Spice library, not from the source database, so a remote engine has no -equivalent to call. A connector that installs the Spice function deny-list will not federate a plan -containing one: a predicate such as `WHERE json_get_int(doc, 'id') = 1` disqualifies the node, the -column is streamed to Spice, and the filter runs there. That is correct, but it costs a full remote -scan. Whether — and which — JSON functions push down is therefore connector-specific. +equivalent to call. A connector that installs the Spice function deny-list keeps each node that calls +one of these functions in Spice and federates the largest parts of the plan beneath it. A predicate +such as `WHERE json_get_int(doc, 'id') = 1` runs in Spice, so the column is streamed from the source +at the cost of a full remote scan. Work beneath the function still runs at the source: +`json_get_str(MIN(doc), 'a')` sends `SELECT min(doc)` to the source and applies `json_get_str` to the +one row returned. Whether, and which, JSON functions push down is therefore connector-specific. **BigQuery** is the exception. A dataset read through the [ADBC data connector](../../components/data-connectors/adbc) with `adbc_driver: bigquery` uses a BigQuery diff --git a/website/versioned_docs/version-2.4.x/reference/sql/select.md b/website/versioned_docs/version-2.4.x/reference/sql/select.md index 2f6d83816..b6a86a4fb 100644 --- a/website/versioned_docs/version-2.4.x/reference/sql/select.md +++ b/website/versioned_docs/version-2.4.x/reference/sql/select.md @@ -72,7 +72,7 @@ SELECT a, b FROM x; ### SELECT clause -The `SELECT` clause is used to select data from a database by defining the colummns it returns. Each `select_expr` in the +The `SELECT` clause is used to select data from a database by defining the columns it returns. Each `select_expr` in the SELECT list can be an expression or wildcards. Example: @@ -100,7 +100,7 @@ SELECT t.a FROM table AS t; ### WHERE clause -The `WHERE` clause is used define the conditions to filter the query results. +The `WHERE` clause is used to define the conditions to filter the query results. Example: @@ -212,7 +212,7 @@ select * from x cross join x y; The `GROUP BY` clause groups together input rows that have the same value into summary rows. -`GROUP BY` is typically used with aggregrate functions (`COUNT()`, `MAX()`, `SUM()`), but if no aggregate functions are +`GROUP BY` is typically used with aggregate functions (`COUNT()`, `MAX()`, `SUM()`), but if no aggregate functions are included, the query with a `GROUP BY` clause is the same as `SELECT DISTINCT`. Example: