From 0a74b6c51605ca78cea59198798910808dcdfd37 Mon Sep 17 00:00:00 2001 From: Emma Spice Date: Tue, 27 Oct 2026 07:53:31 -0700 Subject: [PATCH 1/4] docs: keyed non-Cayenne Iceberg appends read by time, partitioned Cayenne snapshots, Cayenne on the cluster-sidecar example --- .../components/data-connectors/iceberg.md | 2 +- .../data-acceleration/partitioning.md | 1 - .../data-acceleration/refresh-modes/append.md | 12 +- .../features/data-acceleration/snapshots.md | 1 - website/docs/reference/spicepod/datasets.md | 2 +- .../architectures/_cluster-sidecar-v2.3.mdx | 287 ++++++++++++++++++ .../architectures/_cluster-sidecar.mdx | 12 +- .../architectures/cluster-sidecar.md | 2 +- .../architectures/cluster-sidecar.md | 2 +- .../architectures/cluster-sidecar.md | 2 +- .../architectures/cluster-sidecar.md | 2 +- .../components/data-connectors/iceberg.md | 2 +- .../data-acceleration/partitioning.md | 1 - .../data-acceleration/refresh-modes/append.md | 12 +- .../features/data-acceleration/snapshots.md | 1 - .../reference/spicepod/datasets.md | 2 +- 16 files changed, 322 insertions(+), 21 deletions(-) create mode 100644 website/src/partials/deployment/architectures/_cluster-sidecar-v2.3.mdx diff --git a/website/docs/components/data-connectors/iceberg.md b/website/docs/components/data-connectors/iceberg.md index f27dc8ab2..38e0c6487 100644 --- a/website/docs/components/data-connectors/iceberg.md +++ b/website/docs/components/data-connectors/iceberg.md @@ -414,7 +414,7 @@ Until that path exists, a workaround is **soft deletes** in the Iceberg table (a ## Append refresh and table layout -An accelerated Iceberg dataset with [`refresh_mode: append`](../../features/data-acceleration/refresh-modes/append.md) and a [`time_column`](../../reference/spicepod/datasets.md#time_column) follows the table's snapshots: each refresh reads the data files that the snapshots committed since the last refresh added, whatever their `time_column` values, so rows committed out of time order are loaded. See [Iceberg sources follow table snapshots](../../features/data-acceleration/refresh-modes/append.md#iceberg-sources-follow-table-snapshots) for how each snapshot operation is handled and when a refresh reads by time instead. +An accelerated Iceberg dataset with [`refresh_mode: append`](../../features/data-acceleration/refresh-modes/append.md) and a [`time_column`](../../reference/spicepod/datasets.md#time_column) follows the table's snapshots: each refresh reads the data files that the snapshots committed since the last refresh added, whatever their `time_column` values, so rows committed out of time order are loaded. A keyed acceleration on an engine other than Cayenne, whose `primary_key` does not include the `time_column`, reads by time instead. See [Iceberg sources follow table snapshots](../../features/data-acceleration/refresh-modes/append.md#iceberg-sources-follow-table-snapshots) for how each snapshot operation is handled and when a refresh reads by time instead. The first refresh, and a refresh that cannot follow the table's snapshots, polls with a filter on the `time_column`. [`refresh_append_overlap`](../../reference/spicepod/datasets.md#accelerationrefresh_append_overlap) widens the window. The scan skips data files whose partition values or column metrics cannot match the filter, and by default it skips Parquet row groups whose statistics cannot match. Page-level statistics inside a row group are not consulted. diff --git a/website/docs/features/data-acceleration/partitioning.md b/website/docs/features/data-acceleration/partitioning.md index c730e520c..ce2a25893 100644 --- a/website/docs/features/data-acceleration/partitioning.md +++ b/website/docs/features/data-acceleration/partitioning.md @@ -203,7 +203,6 @@ Partitioned Cayenne tables have these limitations: - **No in-place schema evolution.** Schema changes use the recreate fallback. Configuring [`on_schema_change`](../../reference/spicepod/datasets#on_schema_change) to evolve logs a warning; CDC batches with widened schemas are rejected. -- **No snapshot publication.** [Acceleration snapshots](snapshots) do not support partitioned tables. ## Full refreshes and partitions diff --git a/website/docs/features/data-acceleration/refresh-modes/append.md b/website/docs/features/data-acceleration/refresh-modes/append.md index 656781b03..8107842d3 100644 --- a/website/docs/features/data-acceleration/refresh-modes/append.md +++ b/website/docs/features/data-acceleration/refresh-modes/append.md @@ -34,12 +34,22 @@ datasets: To account for clock skew or late-arriving rows, configure an overlap window with [`acceleration.refresh_append_overlap`](../../../reference/spicepod/datasets#accelerationrefresh_append_overlap). Rows within the overlap are re-read on each refresh. A row that arrives later than the overlap, with a `time_column` value older than the window, is not loaded by any later `append` refresh; a `full` refresh loads it. -[Iceberg sources](#iceberg-sources-follow-table-snapshots) do not depend on the overlap for late rows: their append refresh reads what each new snapshot added. +[Iceberg sources](#iceberg-sources-follow-table-snapshots) do not depend on the overlap for late rows: their append refresh reads what each new snapshot added. The exception is a keyed acceleration on an engine other than Cayenne, which [reads by time](#keyed-accelerations-on-other-engines-read-by-time). ## Iceberg Sources Follow Table Snapshots An `append` refresh of an [Iceberg](../../../components/data-connectors/iceberg.md) dataset, or of an Iceberg table read through the [Glue](../../../components/data-connectors/glue.md) connector, follows the table's snapshots instead of a time window. Spice saves the snapshot it last loaded. The next refresh reads the data files that the snapshots committed since then added, whatever their `time_column` values, so rows that a writer commits out of time order are loaded. A refresh of a table with no new snapshot scans no data. +### Keyed accelerations on other engines read by time + +An acceleration on an engine other than Cayenne, with a `primary_key` that does not include the `time_column`, keeps the last row to arrive for each key. Its `append` refresh of an Iceberg source does not follow snapshots. It reads rows newer than the latest stored `time_column` value, less `refresh_append_overlap`, so a late row with an older `time_column` value does not replace the newer row already stored. A row that arrives later than the overlap is not loaded. Cayenne keeps the newest row for each key by `time_column`, so it follows snapshots and loads every new row. + +Such a dataset with no `refresh_append_overlap` logs this warning when it loads: + +```text +Dataset '' (`engine: `) has no `refresh_append_overlap`, so a row that arrives after a newer one is never loaded. Use `engine: cayenne` to load every new row, or set `refresh_append_overlap` to how late rows can arrive. See: https://spiceai.org/docs/features/data-acceleration/data-refresh +``` + The first refresh, which has no saved snapshot, reads by time as described in [Configuration](#configuration). Each kind of snapshot since the saved one is handled as follows: | Snapshot operation | What the refresh reads | diff --git a/website/docs/features/data-acceleration/snapshots.md b/website/docs/features/data-acceleration/snapshots.md index 3e67e7dd3..d08ea0725 100644 --- a/website/docs/features/data-acceleration/snapshots.md +++ b/website/docs/features/data-acceleration/snapshots.md @@ -519,7 +519,6 @@ For the production deployment pattern that uses snapshots to separate ingest fro :::warning[Limitations] - Only datasets are supported for snapshots. Views are not supported. -- **Partitioned Cayenne** accelerations (`engine: cayenne` with `partition_by`) skip periodic and pre-recreate snapshots, with a warning naming the dataset. - **Cayenne with a cold tier** (`cayenne_datalake_location` set) neither creates nor bootstraps from snapshots. The dataset loads from its source, with a warning naming the dataset. See [Cold Object-Store Tier](../../components/data-accelerators/cayenne/index.md#requirements-and-v1-constraints). ::: diff --git a/website/docs/reference/spicepod/datasets.md b/website/docs/reference/spicepod/datasets.md index bcc9c0e1f..14f296c83 100644 --- a/website/docs/reference/spicepod/datasets.md +++ b/website/docs/reference/spicepod/datasets.md @@ -852,7 +852,7 @@ Optional. A duration to specify how far back to include records based on the mos This setting can help mitigate missing data issues caused by late arriving data. -An [Iceberg](../../components/data-connectors/iceberg.md) source applies the overlap only to the refreshes that read by time: the first refresh, a refresh that cannot follow the table's snapshots, every refresh of a dataset that computes columns over the table (such as embedding columns), and a one-off refresh with its own `refresh_sql`. Other refreshes read the data files new snapshots added, whatever their time values. See [Iceberg sources follow table snapshots](../../features/data-acceleration/refresh-modes/append.md#iceberg-sources-follow-table-snapshots). +An [Iceberg](../../components/data-connectors/iceberg.md) source applies the overlap only to the refreshes that read by time: the first refresh, a refresh that cannot follow the table's snapshots, every refresh of a dataset that computes columns over the table (such as embedding columns), a one-off refresh with its own `refresh_sql`, and every refresh of a keyed acceleration on an engine other than Cayenne whose `primary_key` does not include the `time_column`. Other refreshes read the data files new snapshots added, whatever their time values. See [Iceberg sources follow table snapshots](../../features/data-acceleration/refresh-modes/append.md#iceberg-sources-follow-table-snapshots). Example: If the latest timestamp in the accelerated data table is `2020-01-01T02:00:00Z`, setting `refresh_append_overlap: 1h` will include records starting from `2020-01-01T01:00:00Z`. diff --git a/website/src/partials/deployment/architectures/_cluster-sidecar-v2.3.mdx b/website/src/partials/deployment/architectures/_cluster-sidecar-v2.3.mdx new file mode 100644 index 000000000..a818af2d9 --- /dev/null +++ b/website/src/partials/deployment/architectures/_cluster-sidecar-v2.3.mdx @@ -0,0 +1,287 @@ +Modern applications have two fundamentally different data access patterns, and no single deployment model serves both well. Large analytical queries — scanning terabytes of Iceberg data, joining Delta Lake tables, running cross-dataset aggregations — need distributed execution across many nodes. Hot operational queries — serving the working set a microservice actually uses, answering user-facing requests in under 5 milliseconds, feeding fresh context to an AI agent — need data materialized right next to the application, with no network hop. + +Those workloads share a third requirement: **isolation**. Giving an application — especially an AI agent that writes its own queries — direct credentials to production Postgres, a data lake, or a warehouse means a bad plan, a runaway loop, or a prompt injection can exhaust connection pools, scan petabytes, or touch rows it shouldn't. The retrieval layer needs to be a sandbox, not a passthrough. + +cluster-sidecar + +The [cluster-sidecar architecture](https://spice.ai/blog/cluster-sidecar-architecture) addresses all three. Application sidecars handle the hot path with a scoped, locally accelerated working set, while a centralized Spice [cluster](./cluster) (or the [Spice Cloud Platform](./hosted)) provides distributed compute for heavy queries, data ingestion, acceleration, and refresh. When a sidecar needs to reach beyond its materialized working set — a historical query, a cross-dataset join, a broad search — it transparently delegates to the cluster over Arrow Flight, which executes the query and returns results. The sidecar can then cache those results for future use. + +From the application's perspective, everything is `localhost`. From an infrastructure perspective, the system delivers the throughput of a distributed query engine, the latency of an embedded database, and a hard isolation boundary between the application and origin data systems — without ETL between them, sync jobs, or consistency gaps. + +Think of it as a [CDN for your data](../../use-cases/data/database-cdn): the cluster is the origin server, the sidecars are the edge nodes, and Spice handles the caching, invalidation, and routing. Origin databases, data lakes, and CDC streams never see the application fleet. + +```mermaid +flowchart LR + subgraph Node1["Node / Pod"] + direction LR + A1["App"] <-->|"loopback"| SC1["Spice Sidecar (cache)"] + end + + subgraph Node2["Node / Pod"] + direction LR + A2["App"] <-->|"loopback"| SC2["Spice Sidecar (cache)"] + end + + SC1 & SC2 -->|"Arrow Flight (gRPC)"| Cluster + + subgraph Cluster["Spice Cluster or Spice Cloud"] + direction LR + S1["Spice Node 1"] + S2["Spice Node 2"] + S3["Spice Node 3"] + end + + Cluster --> Sources["Data Sources (S3, PostgreSQL, Databricks, ...)"] +``` + +Each sidecar is configured declaratively via a `spicepod.yaml` — the datasets, views, acceleration engines, search indices, and AI models it manages. Sidecars start in seconds, typically run on a few hundred megabytes of memory, and scale horizontally with application pods: scale a deployment from 5 to 50 replicas and 50 sidecars come up automatically, each materializing only the working set its spicepod declares. Sidecars never talk to each other; they talk only to the cluster and, when they bootstrap from [acceleration snapshots](../../features/data-acceleration/snapshots), to the snapshot object store. + +For the engineering walkthrough of this pattern, including a request-path example and FAQ, see [Localhost Latency at Scale: The Spice Cluster-Sidecar Architecture](https://spice.ai/blog/cluster-sidecar-architecture). + +## How it works + +### The sidecar as a sandbox + +The most important property of the sidecar is isolation, not just latency. The sidecar is the only data-plane surface the application touches: + +- **Scoped working set.** A sidecar's `spicepod.yaml` declares exactly which datasets, views, and search indices the application may query. Anything not declared is physically absent from the catalog — not filtered by a policy, not hidden by a row-level rule. Compare this with row-level security, where the underlying data is still present and a single policy misconfiguration can expose it. Even a perfectly crafted prompt injection cannot query a table that is not in the catalog. +- **No origin credentials in the application.** The application connects to its sidecar with a local token. The sidecar connects to the cluster over Arrow Flight. Only the cluster holds credentials for Postgres, Snowflake, Databricks, S3, Kafka, and the rest. Compromising an application pod cannot leak origin credentials, because the pod never had them. +- **Narrow network surface.** The application's only outbound data dependency is the loopback interface. Network policy can pin the sidecar's egress to the cluster endpoint, plus the snapshot bucket when the sidecar bootstraps from snapshots. +- **Per-application data views.** Sidecars can be specialized per application class or per tenant. A customer-service agent, a fraud-review agent, and an internal dashboard can each run with different spicepods pointing at different slices of the same cluster. This is physical isolation, not policy-based filtering on a shared database. See [Multi-Tenant AI Agents](/docs/use-cases/ai/multi-tenant-agents). +- **Bounded resource use.** A rogue query plan or runaway loop exhausts the sidecar's local memory and CPU budget, not the cluster's and not the origin database's. +- **Local inference.** Sidecars can serve [LLM inference and tool calls](../../features/large-language-models) on loopback so sensitive prompts and retrieved context stay in the pod. Heavier models can still be routed to the cluster. +- **One audit point.** Every query, search, and inference call flows through the sidecar, so there is one place to log, rate-limit, and enforce policy. + +Data only moves along one path: **application → sidecar → (optionally) cluster → origin**. It never skips a tier. + +### Three latency tiers + +A request is served from the first tier that can answer it: + +1. **Sidecar results cache** — repeat queries return from the in-memory [results cache](../../features/caching) in microseconds (`Results-Cache-Status: HIT`). +2. **Sidecar working set** — novel queries that fit the locally materialized dataset execute on-node in single-digit milliseconds (Arrow, DuckDB, or SQLite). +3. **Cluster delegation** — queries that exceed the working set are forwarded to the cluster. [Apache Ballista](../../features/distributed-query) distributes execution; [Spice Cayenne](../../components/data-accelerators/cayenne) accelerates large scans. The sidecar caches the result so subsequent reads drop back to tier 1. + +The cluster has its own results cache, so one sidecar's miss can become a cluster cache hit for every subsequent sidecar. Typical application-visible latency: p50 at tier 1, p95 at tier 2, tail at tier 3. + +Useful cache knobs in this topology: + +- `cache_key_type: plan` (default) shares a cache entry across semantically equivalent SQL, which matters for ORM-generated queries. `sql` is a faster, string-exact lookup. +- `stale_while_revalidate_ttl` lets the sidecar serve a stale cached result immediately while a background refresh runs. +- Clients can send `Cache-Control` directives per request (`no-cache`, `min-fresh`, `max-stale`, `only-if-cached`). +- `encoding: zstd` typically cuts results-cache memory by 50–90%. + +### The cluster ingests once + +Refreshing from upstream sources, running Cayenne acceleration on large Iceberg tables, and keeping CDC streams connected are resource-intensive. Doing any of that *N* times for *N* sidecars is wasteful and often infeasible — source systems have connection limits, and per-pod CDC multiplies cloud cost with fleet size. + +The cluster ingests each dataset once and produces one authoritative materialization. Source load is bounded by cluster size, not fleet size. A new pod's sidecar pulls its working set from the cluster (or from an [acceleration snapshot](../../features/data-acceleration/snapshots)) rather than re-scanning the source, so new nodes are operational in seconds. + +Sidecars stay lightweight because they do **not** own ingest: + +- They start in seconds — important when application pods autoscale with traffic. +- They typically run on a few hundred megabytes of memory. +- Scaling from 5 to 50 replicas does **not** add 50 new connections to the source database. +- Each sidecar materializes only the working set its spicepod declares, not a full copy of the warehouse. Full accelerated datasets stay on the cluster. + +### Query delegation over Arrow Flight + +Sidecar-to-cluster communication is [Arrow Flight](https://arrow.apache.org/docs/format/Flight.html) over gRPC. Results flow as Arrow record batches directly into the sidecar's query engine with no JSON or row-based serialization detour. + +The sidecar decides locally whether a query can be served from its working set. If not, it forwards and streams results back. The application sees one endpoint and one query. It never knows whether execution happened locally or whether Ballista fanned the query out across cluster executors. + +Configure delegation with the [Spice.ai Data Connector](../../components/data-connectors/spiceai): point a sidecar dataset at a Spice Cloud app (`spice.ai///datasets/`) or at a self-hosted cluster (`spice.ai:https://cluster.example:50051`). Combine with local acceleration to materialize the hot working set; leave acceleration off to always delegate. + +Kubernetes is the most common place to run this topology, but it is not required. The architecture needs only a network path from each sidecar to the cluster. Sidecars run wherever the application runs — Kubernetes, a VPC, on-prem, or at the edge. + +### Acceleration snapshots + +The cluster-sidecar split maps onto [acceleration snapshots](../../features/data-acceleration/snapshots) as a single-writer / many-reader topology: + +- The **cluster** is the single writer (`snapshots: create_only`). After each refresh it uploads a snapshot to object storage. It never downloads snapshots on startup; it always refreshes from the source. +- Each **sidecar** is a reader (`snapshots: bootstrap_only`). On startup — or when ephemeral NVMe is recycled — the sidecar downloads the most recent snapshot and is immediately ready. It never writes snapshots back. + +This avoids snapshot conflicts, keeps the cluster as the authoritative refresh point, and gives every sidecar a warm start from the same materialization. For CDC-backed datasets with large initial state, the difference between a snapshot bootstrap and a full re-sync can be seconds versus minutes. + +Use DuckDB or SQLite in `mode: file` on the sidecar when snapshots are in play — snapshots persist and restore the acceleration file itself. Heavyweight [Cayenne](../../components/data-accelerators/cayenne) acceleration stays on the cluster. + +For production Spicepod and Helm reference configurations, see [Read/Write Separation](/docs/deployment/read-write-separation). + +### Cache coherency is a refresh policy, not a protocol + +Sidecars pull from the cluster on a configurable interval using append or full refresh. They do **not** participate in a distributed invalidation protocol. A pull-based model with explicit refresh intervals makes staleness bounded, predictable, and debuggable. + +For workloads that need sub-second freshness, the cluster consumes [CDC streams](../../features/cdc) (Postgres logical replication, DynamoDB Streams, Debezium, Kafka) once, and the sidecars pull the resulting accelerated dataset on a short interval. That gives near-real-time propagation without a fleet-wide invalidation bus. + +If the cluster is temporarily unavailable — a rolling upgrade, a network blip, a zone event — sidecars keep serving cached data and their accelerated working sets. Refreshes pause and resume when connectivity returns. The blast radius of a cluster incident is slightly staler data, not application downtime. + +## Example Spicepods + +The cluster materializes and accelerates datasets, consumes CDC, and exposes a results cache. It is the only tier with origin credentials. + +```yaml +version: v1 +kind: Spicepod +name: platform-cluster + +runtime: + caching: + sql_results: + enabled: true + max_size: 4GiB + item_ttl: 1m + stale_while_revalidate_ttl: 30s + encoding: zstd + +snapshots: + enabled: true + location: s3://my-bucket/spice-snapshots/ + params: + s3_auth: iam_role + +datasets: + - from: postgres:public.orders + name: orders + acceleration: + enabled: true + engine: cayenne + mode: file + refresh_mode: changes + snapshots: create_only + primary_key: id + on_conflict: + id: upsert + + - from: s3://lakehouse/events/ + name: events + time_column: event_time + params: + file_format: parquet + acceleration: + enabled: true + engine: cayenne + mode: file + refresh_mode: append + refresh_check_interval: 15m + snapshots: create_only +``` + +The sidecar is much smaller. It pulls from the cluster, keeps a working-set engine local, and caches results. It holds no origin credentials. + +```yaml +version: v1 +kind: Spicepod +name: app-sidecar + +runtime: + caching: + sql_results: + enabled: true + max_size: 256MiB + item_ttl: 30s + stale_while_revalidate_ttl: 30s + +snapshots: + enabled: true + location: s3://my-bucket/spice-snapshots/ + bootstrap_on_failure_behavior: warn + params: + s3_auth: iam_role + +datasets: + - from: spice.ai///datasets/orders + name: orders + params: + spiceai_api_key: ${secrets:SPICEAI_API_KEY} + spiceai_region: us-east-1 + acceleration: + enabled: true + engine: duckdb + mode: file + refresh_mode: append + refresh_check_interval: 10s + snapshots: bootstrap_only + + - from: spice.ai///datasets/events + name: events + params: + spiceai_api_key: ${secrets:SPICEAI_API_KEY} + spiceai_region: us-east-1 + # No local acceleration — delegate to the cluster on demand. + # Queries that match recent hot events still hit the results cache. +``` + +Spice Cloud datasets require `spiceai_api_key` and `spiceai_region`. For a self-hosted cluster, replace the `from:` URIs with `spice.ai:https://cluster.example:50051`, set `name:` to the upstream table name, and drop `spiceai_region`, which applies only to Spice Cloud endpoints. Keep `spiceai_api_key` only when the cluster requires API key authentication. See the [Spice.ai Data Connector](../../components/data-connectors/spiceai) for Cloud and self-hosted URI formats. + +## A request path + +1. The application (or agent) queries its sidecar on `localhost:8090`: `SELECT ... FROM orders WHERE tenant_id = $1 ORDER BY created_at DESC LIMIT 20`. +2. The sidecar checks its results cache. Hit → return in microseconds. +3. On a miss, the sidecar plans against its local catalog. `orders` is materialized locally (refreshed from the cluster every 10 seconds). It executes against DuckDB, returns in single-digit milliseconds, and populates the results cache. Origin Postgres sees no traffic. +4. A follow-up hybrid search over a multi-gigabyte `tickets` index is not in the sidecar catalog as a local acceleration. The sidecar opens an Arrow Flight stream to the cluster. Ballista distributes the search; Cayenne segment statistics prune most files. +5. Results stream back as Arrow record batches. The sidecar returns them to the application and caches them. A later LLM call can be served by a small local model on loopback, with the large-model call delegated to the cluster. +6. The cluster independently caches its own results. The next replica that asks the same question gets it from the cluster cache. + +The application sees one endpoint, one wire format, and one latency distribution. + +## Benefits + +- **Kubernetes-native** — designed to run on Kubernetes, leveraging pod-level sidecars with cluster-level orchestration. The same topology also works outside Kubernetes wherever the application has a network path to the cluster. +- Sub-millisecond reads via sidecar caching on loopback, with centralized data management in the cluster. +- Structural isolation — sidecars expose a scoped catalog, hold no origin credentials, and bound resource use per pod. +- Transparent query delegation — sidecars automatically route queries beyond their cached working set to the cluster. +- Sidecars remain lightweight — only a working set and a results cache, no ingest or heavy acceleration overhead. +- Cluster (or Spice Cloud) handles complex operations: data ingestion, [Spice Cayenne](../../components/data-accelerators/cayenne) acceleration, [distributed query](../../features/distributed-query), hybrid search, and refresh from sources. +- Works with both self-managed Spice clusters and the managed [Spice Cloud Platform](./hosted) as the centralized backend. The Spice Cloud cluster-sidecar model is the most common production topology. +- Sidecars can run anywhere — in your VPC, on-premises, at the edge, or in any Kubernetes cluster — while connecting securely to the managed cluster. +- Horizontal scalability — add sidecars without increasing load on data sources. +- Resilience — sidecars serve cached data even if the cluster is temporarily unavailable. +- Encrypted transport — the Spice Cloud endpoint built from `spiceai_region` uses TLS (`https://`), and a self-hosted cluster uses TLS when its `from:` endpoint is `https://` or `grpc+tls://`. Mutual TLS is opt-in: the sidecar presents a client certificate only when the [`spiceai_tls_client_*` parameters](../../components/data-connectors/spiceai#mutual-tls-mtls) are set. An `http://` endpoint connects without TLS and suits local development. Data is encrypted at rest. + +## Considerations + +- More complex deployment structure requiring both sidecar and cluster infrastructure. [Spice Cloud](./hosted) reduces this burden by managing the cluster. +- Cache coherency — sidecars must be configured with appropriate refresh intervals or TTLs to balance freshness with performance. There is no distributed invalidation bus. +- Requires a Spice cluster deployment or [Spice Cloud Platform](./hosted) subscription ([Spice.ai Enterprise](https://spice.ai/enterprise) for self-managed clustering with SSO, RBAC, and audit logs). +- Network connectivity between sidecars and the cluster must be reliable for cache refreshes and query delegation. + +## Use This Approach When + +- Applications or AI agents require sub-millisecond reads, unified retrieval (SQL, full-text, vector, hybrid), and inference on `localhost`, without giving the application direct database credentials. +- You need a clear isolation boundary between application code and origin data systems — scoped catalogs, no origin credentials in the pod, and one audit point per replica. +- Multiple application instances need fast access to the same datasets without each independently querying data sources. +- Reducing load on upstream data sources is a priority — the cluster ingests once, sidecars cache locally. +- The system benefits from separating the caching tier (sidecars) from the data processing tier (cluster). +- Workloads span both real-time operational queries and large-scale analytical queries on the same data (for example, an operational data lakehouse on S3/Iceberg). +- You already run Kubernetes sidecars for other concerns (service mesh, logging, config). + +## Not Ideal When + +- The application is simple with a single instance and no isolation requirement — the overhead of both sidecar and cluster infrastructure isn't justified. Consider [Sidecar](./sidecar) or [Microservice](./microservice). +- All queries are batch or analytical with relaxed latency requirements — a [Microservice](./microservice) deployment is simpler and sufficient. +- Network connectivity between sidecars and the cluster is unreliable — query delegation and cache refreshes will fail, leading to stale data. Consider standalone [Sidecar](./sidecar) deployments with direct source access. + +## Example Use Case + +A multi-tenant SaaS platform runs an AI support agent. Each tenant's agent pods include a Spice sidecar that materializes that tenant's working set (recent tickets, active customer records, the last 7 days of events, the tenant's private knowledge-base embeddings) into a local DuckDB + vector index. The spicepod exposes exactly those datasets — and no others — to the agent. The sidecar holds no credentials for Postgres, S3, or Snowflake. + +Agent turns hit the sidecar on `localhost` and return in single-digit milliseconds; repeat retrievals return from the results cache in microseconds. Behind the sidecars, a Spice cluster (often Spice Cloud) ingests from PostgreSQL via CDC, S3 Iceberg tables, and Databricks. Cayenne acceleration and refresh schedules run on the cluster. When an agent asks a broader question — "summarize this tenant's churn signal across the last 12 months" — the sidecar recognizes the query exceeds its working set and delegates over Arrow Flight. The cluster executes it, the sidecar caches the result, and subsequent turns for that tenant return in microseconds. + +The origin Postgres sees exactly one consumer (the cluster). If a replica is compromised, the attacker gets a loopback endpoint scoped to that tenant's working set, not database credentials and not a query interface to the whole warehouse. + +The same pattern — single ingestion path, per-pod sandboxing, tiered latency, one isolation boundary — works for any application fleet, not just agents: microservices serving real-time dashboards, services powering search, or internal tools querying operational data. + +## See also + +- [Localhost Latency at Scale: The Spice Cluster-Sidecar Architecture](https://spice.ai/blog/cluster-sidecar-architecture) — engineering walkthrough, request-path example, and FAQ. +- [Read/Write Separation](/docs/deployment/read-write-separation) — production guide for splitting ingest from reads using shared acceleration snapshots, including Spicepod and Helm reference configurations. +- [Spice.ai Data Connector](../../components/data-connectors/spiceai) — how sidecars federate to a cluster or to Spice Cloud over Arrow Flight. +- [Caching](../../features/caching) — results cache, stale-while-revalidate, and `Cache-Control`. +- [Acceleration Snapshots](../../features/data-acceleration/snapshots) — warm-start file-mode accelerations from object storage. +- [Distributed Query](../../features/distributed-query) — Apache Ballista schedulers and executors on the cluster. +- [Spice Cayenne](../../components/data-accelerators/cayenne) — cluster-side acceleration for datasets beyond 1 TB. +- [Multi-Tenant AI Agents](/docs/use-cases/ai/multi-tenant-agents) — per-tenant isolation patterns that pair with this architecture. diff --git a/website/src/partials/deployment/architectures/_cluster-sidecar.mdx b/website/src/partials/deployment/architectures/_cluster-sidecar.mdx index a818af2d9..f36f16003 100644 --- a/website/src/partials/deployment/architectures/_cluster-sidecar.mdx +++ b/website/src/partials/deployment/architectures/_cluster-sidecar.mdx @@ -63,7 +63,7 @@ Data only moves along one path: **application → sidecar → (optionally) clust A request is served from the first tier that can answer it: 1. **Sidecar results cache** — repeat queries return from the in-memory [results cache](../../features/caching) in microseconds (`Results-Cache-Status: HIT`). -2. **Sidecar working set** — novel queries that fit the locally materialized dataset execute on-node in single-digit milliseconds (Arrow, DuckDB, or SQLite). +2. **Sidecar working set** — novel queries that fit the locally materialized dataset execute on-node in single-digit milliseconds against a local [Cayenne](../../components/data-accelerators/cayenne) acceleration. 3. **Cluster delegation** — queries that exceed the working set are forwarded to the cluster. [Apache Ballista](../../features/distributed-query) distributes execution; [Spice Cayenne](../../components/data-accelerators/cayenne) accelerates large scans. The sidecar caches the result so subsequent reads drop back to tier 1. The cluster has its own results cache, so one sidecar's miss can become a cluster cache hit for every subsequent sidecar. Typical application-visible latency: p50 at tier 1, p95 at tier 2, tail at tier 3. @@ -107,7 +107,7 @@ The cluster-sidecar split maps onto [acceleration snapshots](../../features/data This avoids snapshot conflicts, keeps the cluster as the authoritative refresh point, and gives every sidecar a warm start from the same materialization. For CDC-backed datasets with large initial state, the difference between a snapshot bootstrap and a full re-sync can be seconds versus minutes. -Use DuckDB or SQLite in `mode: file` on the sidecar when snapshots are in play — snapshots persist and restore the acceleration file itself. Heavyweight [Cayenne](../../components/data-accelerators/cayenne) acceleration stays on the cluster. +Use the same engine on the sidecar as on the cluster when snapshots are in play. A snapshot restores only into an acceleration of the engine that created it: a sidecar on another engine skips the cluster's snapshots and loads its working set from the cluster instead. Use [Cayenne](../../components/data-accelerators/cayenne) in `mode: file` on both; acceleration engines other than Cayenne are deprecated and will be removed in v2.5 (see the [Cayenne migration guide](../../components/data-accelerators/cayenne/migration)). For production Spicepod and Helm reference configurations, see [Read/Write Separation](/docs/deployment/read-write-separation). @@ -153,8 +153,6 @@ datasets: refresh_mode: changes snapshots: create_only primary_key: id - on_conflict: - id: upsert - from: s3://lakehouse/events/ name: events @@ -200,7 +198,7 @@ datasets: spiceai_region: us-east-1 acceleration: enabled: true - engine: duckdb + engine: cayenne mode: file refresh_mode: append refresh_check_interval: 10s @@ -221,7 +219,7 @@ Spice Cloud datasets require `spiceai_api_key` and `spiceai_region`. For a self- 1. The application (or agent) queries its sidecar on `localhost:8090`: `SELECT ... FROM orders WHERE tenant_id = $1 ORDER BY created_at DESC LIMIT 20`. 2. The sidecar checks its results cache. Hit → return in microseconds. -3. On a miss, the sidecar plans against its local catalog. `orders` is materialized locally (refreshed from the cluster every 10 seconds). It executes against DuckDB, returns in single-digit milliseconds, and populates the results cache. Origin Postgres sees no traffic. +3. On a miss, the sidecar plans against its local catalog. `orders` is materialized locally (refreshed from the cluster every 10 seconds). It executes against the local Cayenne acceleration, returns in single-digit milliseconds, and populates the results cache. Origin Postgres sees no traffic. 4. A follow-up hybrid search over a multi-gigabyte `tickets` index is not in the sidecar catalog as a local acceleration. The sidecar opens an Arrow Flight stream to the cluster. Ballista distributes the search; Cayenne segment statistics prune most files. 5. Results stream back as Arrow record batches. The sidecar returns them to the application and caches them. A later LLM call can be served by a small local model on loopback, with the large-model call delegated to the cluster. 6. The cluster independently caches its own results. The next replica that asks the same question gets it from the cluster cache. @@ -267,7 +265,7 @@ The application sees one endpoint, one wire format, and one latency distribution ## Example Use Case -A multi-tenant SaaS platform runs an AI support agent. Each tenant's agent pods include a Spice sidecar that materializes that tenant's working set (recent tickets, active customer records, the last 7 days of events, the tenant's private knowledge-base embeddings) into a local DuckDB + vector index. The spicepod exposes exactly those datasets — and no others — to the agent. The sidecar holds no credentials for Postgres, S3, or Snowflake. +A multi-tenant SaaS platform runs an AI support agent. Each tenant's agent pods include a Spice sidecar that materializes that tenant's working set (recent tickets, active customer records, the last 7 days of events, the tenant's private knowledge-base embeddings) into a local Cayenne acceleration and vector index. The spicepod exposes exactly those datasets — and no others — to the agent. The sidecar holds no credentials for Postgres, S3, or Snowflake. Agent turns hit the sidecar on `localhost` and return in single-digit milliseconds; repeat retrievals return from the results cache in microseconds. Behind the sidecars, a Spice cluster (often Spice Cloud) ingests from PostgreSQL via CDC, S3 Iceberg tables, and Databricks. Cayenne acceleration and refresh schedules run on the cluster. When an agent asks a broader question — "summarize this tenant's churn signal across the last 12 months" — the sidecar recognizes the query exceeds its working set and delegates over Arrow Flight. The cluster executes it, the sidecar caches the result, and subsequent turns for that tenant return in microseconds. diff --git a/website/versioned_docs/version-2.0.x/deployment/architectures/cluster-sidecar.md b/website/versioned_docs/version-2.0.x/deployment/architectures/cluster-sidecar.md index ae9875d5c..55e600edb 100644 --- a/website/versioned_docs/version-2.0.x/deployment/architectures/cluster-sidecar.md +++ b/website/versioned_docs/version-2.0.x/deployment/architectures/cluster-sidecar.md @@ -7,6 +7,6 @@ pagination_prev: null pagination_next: null --- -import Content from '@site/src/partials/deployment/architectures/_cluster-sidecar.mdx'; +import Content from '@site/src/partials/deployment/architectures/_cluster-sidecar-v2.3.mdx'; diff --git a/website/versioned_docs/version-2.1.x/deployment/architectures/cluster-sidecar.md b/website/versioned_docs/version-2.1.x/deployment/architectures/cluster-sidecar.md index ae9875d5c..55e600edb 100644 --- a/website/versioned_docs/version-2.1.x/deployment/architectures/cluster-sidecar.md +++ b/website/versioned_docs/version-2.1.x/deployment/architectures/cluster-sidecar.md @@ -7,6 +7,6 @@ pagination_prev: null pagination_next: null --- -import Content from '@site/src/partials/deployment/architectures/_cluster-sidecar.mdx'; +import Content from '@site/src/partials/deployment/architectures/_cluster-sidecar-v2.3.mdx'; diff --git a/website/versioned_docs/version-2.2.x/deployment/architectures/cluster-sidecar.md b/website/versioned_docs/version-2.2.x/deployment/architectures/cluster-sidecar.md index ae9875d5c..55e600edb 100644 --- a/website/versioned_docs/version-2.2.x/deployment/architectures/cluster-sidecar.md +++ b/website/versioned_docs/version-2.2.x/deployment/architectures/cluster-sidecar.md @@ -7,6 +7,6 @@ pagination_prev: null pagination_next: null --- -import Content from '@site/src/partials/deployment/architectures/_cluster-sidecar.mdx'; +import Content from '@site/src/partials/deployment/architectures/_cluster-sidecar-v2.3.mdx'; diff --git a/website/versioned_docs/version-2.3.x/deployment/architectures/cluster-sidecar.md b/website/versioned_docs/version-2.3.x/deployment/architectures/cluster-sidecar.md index ae9875d5c..55e600edb 100644 --- a/website/versioned_docs/version-2.3.x/deployment/architectures/cluster-sidecar.md +++ b/website/versioned_docs/version-2.3.x/deployment/architectures/cluster-sidecar.md @@ -7,6 +7,6 @@ pagination_prev: null pagination_next: null --- -import Content from '@site/src/partials/deployment/architectures/_cluster-sidecar.mdx'; +import Content from '@site/src/partials/deployment/architectures/_cluster-sidecar-v2.3.mdx'; diff --git a/website/versioned_docs/version-2.4.x/components/data-connectors/iceberg.md b/website/versioned_docs/version-2.4.x/components/data-connectors/iceberg.md index c1533b831..0fd9b026b 100644 --- a/website/versioned_docs/version-2.4.x/components/data-connectors/iceberg.md +++ b/website/versioned_docs/version-2.4.x/components/data-connectors/iceberg.md @@ -391,7 +391,7 @@ Until that path exists, a workaround is **soft deletes** in the Iceberg table (a ## Append refresh and table layout -An accelerated Iceberg dataset with [`refresh_mode: append`](../../features/data-acceleration/refresh-modes/append.md) and a [`time_column`](../../reference/spicepod/datasets.md#time_column) follows the table's snapshots: each refresh reads the data files that the snapshots committed since the last refresh added, whatever their `time_column` values, so rows committed out of time order are loaded. See [Iceberg sources follow table snapshots](../../features/data-acceleration/refresh-modes/append.md#iceberg-sources-follow-table-snapshots) for how each snapshot operation is handled and when a refresh reads by time instead. +An accelerated Iceberg dataset with [`refresh_mode: append`](../../features/data-acceleration/refresh-modes/append.md) and a [`time_column`](../../reference/spicepod/datasets.md#time_column) follows the table's snapshots: each refresh reads the data files that the snapshots committed since the last refresh added, whatever their `time_column` values, so rows committed out of time order are loaded. A keyed acceleration on an engine other than Cayenne, whose `primary_key` does not include the `time_column`, reads by time instead. See [Iceberg sources follow table snapshots](../../features/data-acceleration/refresh-modes/append.md#iceberg-sources-follow-table-snapshots) for how each snapshot operation is handled and when a refresh reads by time instead. The first refresh, and a refresh that cannot follow the table's snapshots, polls with a filter on the `time_column`. [`refresh_append_overlap`](../../reference/spicepod/datasets.md#accelerationrefresh_append_overlap) widens the window. The scan skips data files whose partition values or column metrics cannot match the filter, and by default it skips Parquet row groups whose statistics cannot match. Page-level statistics inside a row group are not consulted. diff --git a/website/versioned_docs/version-2.4.x/features/data-acceleration/partitioning.md b/website/versioned_docs/version-2.4.x/features/data-acceleration/partitioning.md index c730e520c..ce2a25893 100644 --- a/website/versioned_docs/version-2.4.x/features/data-acceleration/partitioning.md +++ b/website/versioned_docs/version-2.4.x/features/data-acceleration/partitioning.md @@ -203,7 +203,6 @@ Partitioned Cayenne tables have these limitations: - **No in-place schema evolution.** Schema changes use the recreate fallback. Configuring [`on_schema_change`](../../reference/spicepod/datasets#on_schema_change) to evolve logs a warning; CDC batches with widened schemas are rejected. -- **No snapshot publication.** [Acceleration snapshots](snapshots) do not support partitioned tables. ## Full refreshes and partitions diff --git a/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md b/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md index 116bdf11d..09ea1984e 100644 --- a/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md +++ b/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md @@ -34,12 +34,22 @@ datasets: To account for clock skew or late-arriving rows, configure an overlap window with [`acceleration.refresh_append_overlap`](../../../reference/spicepod/datasets#accelerationrefresh_append_overlap). Rows within the overlap are re-read on each refresh. A row that arrives later than the overlap, with a `time_column` value older than the window, is not loaded by any later `append` refresh; a `full` refresh loads it. -[Iceberg sources](#iceberg-sources-follow-table-snapshots) do not depend on the overlap for late rows: their append refresh reads what each new snapshot added. +[Iceberg sources](#iceberg-sources-follow-table-snapshots) do not depend on the overlap for late rows: their append refresh reads what each new snapshot added. The exception is a keyed acceleration on an engine other than Cayenne, which [reads by time](#keyed-accelerations-on-other-engines-read-by-time). ## Iceberg Sources Follow Table Snapshots An `append` refresh of an [Iceberg](../../../components/data-connectors/iceberg.md) dataset, or of an Iceberg table read through the [Glue](../../../components/data-connectors/glue.md) connector, follows the table's snapshots instead of a time window. Spice saves the snapshot it last loaded. The next refresh reads the data files that the snapshots committed since then added, whatever their `time_column` values, so rows that a writer commits out of time order are loaded. A refresh of a table with no new snapshot scans no data. +### Keyed accelerations on other engines read by time + +An acceleration on an engine other than Cayenne, with a `primary_key` that does not include the `time_column`, keeps the last row to arrive for each key. Its `append` refresh of an Iceberg source does not follow snapshots. It reads rows newer than the latest stored `time_column` value, less `refresh_append_overlap`, so a late row with an older `time_column` value does not replace the newer row already stored. A row that arrives later than the overlap is not loaded. Cayenne keeps the newest row for each key by `time_column`, so it follows snapshots and loads every new row. + +Such a dataset with no `refresh_append_overlap` logs this warning when it loads: + +```text +Dataset '' (`engine: `) has no `refresh_append_overlap`, so a row that arrives after a newer one is never loaded. Use `engine: cayenne` to load every new row, or set `refresh_append_overlap` to how late rows can arrive. See: https://spiceai.org/docs/features/data-acceleration/data-refresh +``` + The first refresh, which has no saved snapshot, reads by time as described in [Configuration](#configuration). Each kind of snapshot since the saved one is handled as follows: | Snapshot operation | What the refresh reads | diff --git a/website/versioned_docs/version-2.4.x/features/data-acceleration/snapshots.md b/website/versioned_docs/version-2.4.x/features/data-acceleration/snapshots.md index 3bcb090b2..3bda07d4c 100644 --- a/website/versioned_docs/version-2.4.x/features/data-acceleration/snapshots.md +++ b/website/versioned_docs/version-2.4.x/features/data-acceleration/snapshots.md @@ -449,7 +449,6 @@ For the production deployment pattern that uses snapshots to separate ingest fro :::warning[Limitations] - Only datasets are supported for snapshots. Views are not supported. -- **Partitioned Cayenne** accelerations (`engine: cayenne` with `partition_by`) skip periodic and pre-recreate snapshots, with a warning naming the dataset. - **Cayenne with a cold tier** (`cayenne_datalake_location` set) neither creates nor bootstraps from snapshots. The dataset loads from its source, with a warning naming the dataset. See [Cold Object-Store Tier](../../components/data-accelerators/cayenne/index.md#requirements-and-v1-constraints). ::: diff --git a/website/versioned_docs/version-2.4.x/reference/spicepod/datasets.md b/website/versioned_docs/version-2.4.x/reference/spicepod/datasets.md index 69570e717..77ad540b1 100644 --- a/website/versioned_docs/version-2.4.x/reference/spicepod/datasets.md +++ b/website/versioned_docs/version-2.4.x/reference/spicepod/datasets.md @@ -832,7 +832,7 @@ Optional. A duration to specify how far back to include records based on the mos This setting can help mitigate missing data issues caused by late arriving data. -An [Iceberg](../../components/data-connectors/iceberg.md) source applies the overlap only to the refreshes that read by time: the first refresh, a refresh that cannot follow the table's snapshots, every refresh of a dataset that computes columns over the table (such as embedding columns), and a one-off refresh with its own `refresh_sql`. Other refreshes read the data files new snapshots added, whatever their time values. See [Iceberg sources follow table snapshots](../../features/data-acceleration/refresh-modes/append.md#iceberg-sources-follow-table-snapshots). +An [Iceberg](../../components/data-connectors/iceberg.md) source applies the overlap only to the refreshes that read by time: the first refresh, a refresh that cannot follow the table's snapshots, every refresh of a dataset that computes columns over the table (such as embedding columns), a one-off refresh with its own `refresh_sql`, and every refresh of a keyed acceleration on an engine other than Cayenne whose `primary_key` does not include the `time_column`. Other refreshes read the data files new snapshots added, whatever their time values. See [Iceberg sources follow table snapshots](../../features/data-acceleration/refresh-modes/append.md#iceberg-sources-follow-table-snapshots). Example: If the latest timestamp in the accelerated data table is `2020-01-01T02:00:00Z`, setting `refresh_append_overlap: 1h` will include records starting from `2020-01-01T01:00:00Z`. From 3c3afca1d162f686410892783a3f87b9203f25a5 Mon Sep 17 00:00:00 2001 From: Emma Spice Date: Tue, 27 Oct 2026 08:06:57 -0700 Subject: [PATCH 2/4] docs(append): a late row inside the overlap window can replace a newer row on last-arrival engines --- website/docs/features/data-acceleration/refresh-modes/append.md | 2 +- .../features/data-acceleration/refresh-modes/append.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/features/data-acceleration/refresh-modes/append.md b/website/docs/features/data-acceleration/refresh-modes/append.md index 8107842d3..ef2e4fc91 100644 --- a/website/docs/features/data-acceleration/refresh-modes/append.md +++ b/website/docs/features/data-acceleration/refresh-modes/append.md @@ -42,7 +42,7 @@ An `append` refresh of an [Iceberg](../../../components/data-connectors/iceberg. ### Keyed accelerations on other engines read by time -An acceleration on an engine other than Cayenne, with a `primary_key` that does not include the `time_column`, keeps the last row to arrive for each key. Its `append` refresh of an Iceberg source does not follow snapshots. It reads rows newer than the latest stored `time_column` value, less `refresh_append_overlap`, so a late row with an older `time_column` value does not replace the newer row already stored. A row that arrives later than the overlap is not loaded. Cayenne keeps the newest row for each key by `time_column`, so it follows snapshots and loads every new row. +An acceleration on an engine other than Cayenne, with a `primary_key` that does not include the `time_column`, keeps the last row to arrive for each key. Its `append` refresh of an Iceberg source does not follow snapshots. It reads rows newer than the latest stored `time_column` value, less `refresh_append_overlap`. A late row with a `time_column` value older than that window is not loaded, so it does not replace the newer row already stored. A late row inside the window is loaded and, because the last row to arrive wins, can replace a newer stored row with the same key. Cayenne keeps the newest row for each key by `time_column`, so it follows snapshots and loads every new row. Such a dataset with no `refresh_append_overlap` logs this warning when it loads: diff --git a/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md b/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md index 09ea1984e..7d83053da 100644 --- a/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md +++ b/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md @@ -42,7 +42,7 @@ An `append` refresh of an [Iceberg](../../../components/data-connectors/iceberg. ### Keyed accelerations on other engines read by time -An acceleration on an engine other than Cayenne, with a `primary_key` that does not include the `time_column`, keeps the last row to arrive for each key. Its `append` refresh of an Iceberg source does not follow snapshots. It reads rows newer than the latest stored `time_column` value, less `refresh_append_overlap`, so a late row with an older `time_column` value does not replace the newer row already stored. A row that arrives later than the overlap is not loaded. Cayenne keeps the newest row for each key by `time_column`, so it follows snapshots and loads every new row. +An acceleration on an engine other than Cayenne, with a `primary_key` that does not include the `time_column`, keeps the last row to arrive for each key. Its `append` refresh of an Iceberg source does not follow snapshots. It reads rows newer than the latest stored `time_column` value, less `refresh_append_overlap`. A late row with a `time_column` value older than that window is not loaded, so it does not replace the newer row already stored. A late row inside the window is loaded and, because the last row to arrive wins, can replace a newer stored row with the same key. Cayenne keeps the newest row for each key by `time_column`, so it follows snapshots and loads every new row. Such a dataset with no `refresh_append_overlap` logs this warning when it loads: From 13aad9475ea6cf6047d8dce2c510fd3079a656ca Mon Sep 17 00:00:00 2001 From: Emma Spice Date: Tue, 27 Oct 2026 08:13:38 -0700 Subject: [PATCH 3/4] docs(append): replacement inside the overlap window depends on on_conflict --- website/docs/features/data-acceleration/refresh-modes/append.md | 2 +- .../features/data-acceleration/refresh-modes/append.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/features/data-acceleration/refresh-modes/append.md b/website/docs/features/data-acceleration/refresh-modes/append.md index ef2e4fc91..643b246a7 100644 --- a/website/docs/features/data-acceleration/refresh-modes/append.md +++ b/website/docs/features/data-acceleration/refresh-modes/append.md @@ -42,7 +42,7 @@ An `append` refresh of an [Iceberg](../../../components/data-connectors/iceberg. ### Keyed accelerations on other engines read by time -An acceleration on an engine other than Cayenne, with a `primary_key` that does not include the `time_column`, keeps the last row to arrive for each key. Its `append` refresh of an Iceberg source does not follow snapshots. It reads rows newer than the latest stored `time_column` value, less `refresh_append_overlap`. A late row with a `time_column` value older than that window is not loaded, so it does not replace the newer row already stored. A late row inside the window is loaded and, because the last row to arrive wins, can replace a newer stored row with the same key. Cayenne keeps the newest row for each key by `time_column`, so it follows snapshots and loads every new row. +An acceleration on an engine other than Cayenne, with a `primary_key` that does not include the `time_column`, resolves a repeated key by arrival order under its [`on_conflict`](../constraints.md#handling-conflicts) setting, not by `time_column`. Its `append` refresh of an Iceberg source does not follow snapshots. It reads rows newer than the latest stored `time_column` value, less `refresh_append_overlap`. A late row with a `time_column` value older than that window is not loaded, so it cannot replace the newer row already stored. A late row inside the window is loaded, and with `on_conflict: upsert` it can replace a newer stored row with the same key. Cayenne keeps the newest row for each key by `time_column`, so it follows snapshots and loads every new row. Such a dataset with no `refresh_append_overlap` logs this warning when it loads: diff --git a/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md b/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md index 7d83053da..96559a551 100644 --- a/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md +++ b/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md @@ -42,7 +42,7 @@ An `append` refresh of an [Iceberg](../../../components/data-connectors/iceberg. ### Keyed accelerations on other engines read by time -An acceleration on an engine other than Cayenne, with a `primary_key` that does not include the `time_column`, keeps the last row to arrive for each key. Its `append` refresh of an Iceberg source does not follow snapshots. It reads rows newer than the latest stored `time_column` value, less `refresh_append_overlap`. A late row with a `time_column` value older than that window is not loaded, so it does not replace the newer row already stored. A late row inside the window is loaded and, because the last row to arrive wins, can replace a newer stored row with the same key. Cayenne keeps the newest row for each key by `time_column`, so it follows snapshots and loads every new row. +An acceleration on an engine other than Cayenne, with a `primary_key` that does not include the `time_column`, resolves a repeated key by arrival order under its [`on_conflict`](../constraints.md#handling-conflicts) setting, not by `time_column`. Its `append` refresh of an Iceberg source does not follow snapshots. It reads rows newer than the latest stored `time_column` value, less `refresh_append_overlap`. A late row with a `time_column` value older than that window is not loaded, so it cannot replace the newer row already stored. A late row inside the window is loaded, and with `on_conflict: upsert` it can replace a newer stored row with the same key. Cayenne keeps the newest row for each key by `time_column`, so it follows snapshots and loads every new row. Such a dataset with no `refresh_append_overlap` logs this warning when it loads: From 6317e204a9d556370dba1784675767696add20a9 Mon Sep 17 00:00:00 2001 From: Emma Spice Date: Tue, 27 Oct 2026 08:19:27 -0700 Subject: [PATCH 4/4] docs(append): the Late-Arriving Data exception names the primary_key condition --- website/docs/features/data-acceleration/refresh-modes/append.md | 2 +- .../features/data-acceleration/refresh-modes/append.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/features/data-acceleration/refresh-modes/append.md b/website/docs/features/data-acceleration/refresh-modes/append.md index 643b246a7..7a909fdf6 100644 --- a/website/docs/features/data-acceleration/refresh-modes/append.md +++ b/website/docs/features/data-acceleration/refresh-modes/append.md @@ -34,7 +34,7 @@ datasets: To account for clock skew or late-arriving rows, configure an overlap window with [`acceleration.refresh_append_overlap`](../../../reference/spicepod/datasets#accelerationrefresh_append_overlap). Rows within the overlap are re-read on each refresh. A row that arrives later than the overlap, with a `time_column` value older than the window, is not loaded by any later `append` refresh; a `full` refresh loads it. -[Iceberg sources](#iceberg-sources-follow-table-snapshots) do not depend on the overlap for late rows: their append refresh reads what each new snapshot added. The exception is a keyed acceleration on an engine other than Cayenne, which [reads by time](#keyed-accelerations-on-other-engines-read-by-time). +[Iceberg sources](#iceberg-sources-follow-table-snapshots) do not depend on the overlap for late rows: their append refresh reads what each new snapshot added. The exception is a keyed acceleration on an engine other than Cayenne whose `primary_key` does not include the `time_column`, which [reads by time](#keyed-accelerations-on-other-engines-read-by-time). ## Iceberg Sources Follow Table Snapshots diff --git a/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md b/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md index 96559a551..cf2f62d3c 100644 --- a/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md +++ b/website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md @@ -34,7 +34,7 @@ datasets: To account for clock skew or late-arriving rows, configure an overlap window with [`acceleration.refresh_append_overlap`](../../../reference/spicepod/datasets#accelerationrefresh_append_overlap). Rows within the overlap are re-read on each refresh. A row that arrives later than the overlap, with a `time_column` value older than the window, is not loaded by any later `append` refresh; a `full` refresh loads it. -[Iceberg sources](#iceberg-sources-follow-table-snapshots) do not depend on the overlap for late rows: their append refresh reads what each new snapshot added. The exception is a keyed acceleration on an engine other than Cayenne, which [reads by time](#keyed-accelerations-on-other-engines-read-by-time). +[Iceberg sources](#iceberg-sources-follow-table-snapshots) do not depend on the overlap for late rows: their append refresh reads what each new snapshot added. The exception is a keyed acceleration on an engine other than Cayenne whose `primary_key` does not include the `time_column`, which [reads by time](#keyed-accelerations-on-other-engines-read-by-time). ## Iceberg Sources Follow Table Snapshots