Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion website/docs/components/data-connectors/iceberg.md
Original file line number Diff line number Diff line change
Expand Up @@ -414,7 +414,7 @@ Until that path exists, a workaround is **soft deletes** in the Iceberg table (a

## Append refresh and table layout

An accelerated Iceberg dataset with [`refresh_mode: append`](../../features/data-acceleration/refresh-modes/append.md) and a [`time_column`](../../reference/spicepod/datasets.md#time_column) follows the table's snapshots: each refresh reads the data files that the snapshots committed since the last refresh added, whatever their `time_column` values, so rows committed out of time order are loaded. See [Iceberg sources follow table snapshots](../../features/data-acceleration/refresh-modes/append.md#iceberg-sources-follow-table-snapshots) for how each snapshot operation is handled and when a refresh reads by time instead.
An accelerated Iceberg dataset with [`refresh_mode: append`](../../features/data-acceleration/refresh-modes/append.md) and a [`time_column`](../../reference/spicepod/datasets.md#time_column) follows the table's snapshots: each refresh reads the data files that the snapshots committed since the last refresh added, whatever their `time_column` values, so rows committed out of time order are loaded. A keyed acceleration on an engine other than Cayenne, whose `primary_key` does not include the `time_column`, reads by time instead. See [Iceberg sources follow table snapshots](../../features/data-acceleration/refresh-modes/append.md#iceberg-sources-follow-table-snapshots) for how each snapshot operation is handled and when a refresh reads by time instead.

The first refresh, and a refresh that cannot follow the table's snapshots, polls with a filter on the `time_column`. [`refresh_append_overlap`](../../reference/spicepod/datasets.md#accelerationrefresh_append_overlap) widens the window. The scan skips data files whose partition values or column metrics cannot match the filter, and by default it skips Parquet row groups whose statistics cannot match. Page-level statistics inside a row group are not consulted.

Expand Down
1 change: 0 additions & 1 deletion website/docs/features/data-acceleration/partitioning.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,6 @@ Partitioned Cayenne tables have these limitations:
- **No in-place schema evolution.** Schema changes use the recreate fallback. Configuring
[`on_schema_change`](../../reference/spicepod/datasets#on_schema_change) to evolve logs a warning;
CDC batches with widened schemas are rejected.
- **No snapshot publication.** [Acceleration snapshots](snapshots) do not support partitioned tables.

## Full refreshes and partitions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,22 @@ datasets:

To account for clock skew or late-arriving rows, configure an overlap window with [`acceleration.refresh_append_overlap`](../../../reference/spicepod/datasets#accelerationrefresh_append_overlap). Rows within the overlap are re-read on each refresh. A row that arrives later than the overlap, with a `time_column` value older than the window, is not loaded by any later `append` refresh; a `full` refresh loads it.

[Iceberg sources](#iceberg-sources-follow-table-snapshots) do not depend on the overlap for late rows: their append refresh reads what each new snapshot added.
[Iceberg sources](#iceberg-sources-follow-table-snapshots) do not depend on the overlap for late rows: their append refresh reads what each new snapshot added. The exception is a keyed acceleration on an engine other than Cayenne whose `primary_key` does not include the `time_column`, which [reads by time](#keyed-accelerations-on-other-engines-read-by-time).

## Iceberg Sources Follow Table Snapshots

An `append` refresh of an [Iceberg](../../../components/data-connectors/iceberg.md) dataset, or of an Iceberg table read through the [Glue](../../../components/data-connectors/glue.md) connector, follows the table's snapshots instead of a time window. Spice saves the snapshot it last loaded. The next refresh reads the data files that the snapshots committed since then added, whatever their `time_column` values, so rows that a writer commits out of time order are loaded. A refresh of a table with no new snapshot scans no data.

### Keyed accelerations on other engines read by time

An acceleration on an engine other than Cayenne, with a `primary_key` that does not include the `time_column`, resolves a repeated key by arrival order under its [`on_conflict`](../constraints.md#handling-conflicts) setting, not by `time_column`. Its `append` refresh of an Iceberg source does not follow snapshots. It reads rows newer than the latest stored `time_column` value, less `refresh_append_overlap`. A late row with a `time_column` value older than that window is not loaded, so it cannot replace the newer row already stored. A late row inside the window is loaded, and with `on_conflict: upsert` it can replace a newer stored row with the same key. Cayenne keeps the newest row for each key by `time_column`, so it follows snapshots and loads every new row.

Such a dataset with no `refresh_append_overlap` logs this warning when it loads:

```text
Dataset '<name>' (`engine: <engine>`) has no `refresh_append_overlap`, so a row that arrives after a newer one is never loaded. Use `engine: cayenne` to load every new row, or set `refresh_append_overlap` to how late rows can arrive. See: https://spiceai.org/docs/features/data-acceleration/data-refresh
```

The first refresh, which has no saved snapshot, reads by time as described in [Configuration](#configuration). Each kind of snapshot since the saved one is handled as follows:

| Snapshot operation | What the refresh reads |
Expand Down
1 change: 0 additions & 1 deletion website/docs/features/data-acceleration/snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -519,7 +519,6 @@ For the production deployment pattern that uses snapshots to separate ingest fro
:::warning[Limitations]

- Only datasets are supported for snapshots. Views are not supported.
- **Partitioned Cayenne** accelerations (`engine: cayenne` with `partition_by`) skip periodic and pre-recreate snapshots, with a warning naming the dataset.
- **Cayenne with a cold tier** (`cayenne_datalake_location` set) neither creates nor bootstraps from snapshots. The dataset loads from its source, with a warning naming the dataset. See [Cold Object-Store Tier](../../components/data-accelerators/cayenne/index.md#requirements-and-v1-constraints).

:::
Expand Down
2 changes: 1 addition & 1 deletion website/docs/reference/spicepod/datasets.md
Original file line number Diff line number Diff line change
Expand Up @@ -852,7 +852,7 @@ Optional. A duration to specify how far back to include records based on the mos

This setting can help mitigate missing data issues caused by late arriving data.

An [Iceberg](../../components/data-connectors/iceberg.md) source applies the overlap only to the refreshes that read by time: the first refresh, a refresh that cannot follow the table's snapshots, every refresh of a dataset that computes columns over the table (such as embedding columns), and a one-off refresh with its own `refresh_sql`. Other refreshes read the data files new snapshots added, whatever their time values. See [Iceberg sources follow table snapshots](../../features/data-acceleration/refresh-modes/append.md#iceberg-sources-follow-table-snapshots).
An [Iceberg](../../components/data-connectors/iceberg.md) source applies the overlap only to the refreshes that read by time: the first refresh, a refresh that cannot follow the table's snapshots, every refresh of a dataset that computes columns over the table (such as embedding columns), a one-off refresh with its own `refresh_sql`, and every refresh of a keyed acceleration on an engine other than Cayenne whose `primary_key` does not include the `time_column`. Other refreshes read the data files new snapshots added, whatever their time values. See [Iceberg sources follow table snapshots](../../features/data-acceleration/refresh-modes/append.md#iceberg-sources-follow-table-snapshots).

Example: If the latest timestamp in the accelerated data table is `2020-01-01T02:00:00Z`, setting `refresh_append_overlap: 1h` will include records starting from `2020-01-01T01:00:00Z`.

Expand Down
Loading
Loading