You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
docs: keyed Iceberg appends on other engines read by time, partitioned Cayenne snapshots, Cayenne in the Cluster-Sidecar example - #2442
This PR combines two Forward updates and one Audit correction.
Forward (vNext and 2.4.x): both source PRs were cherry-picked into v2.4.0 (97d9b38549 and c87fa6236b are ancestors of v2.4.0). They change the current release's behavior, so both website/docs/ and version-2.4.x change.
Keyed Iceberg append on engines other than Cayenne reads by time (refresh-modes/append, components/data-connectors/iceberg, reference/spicepod/datasets). This applies to an acceleration on an engine other than Cayenne whose primary_key does not include the time_column. Its append refresh of an Iceberg source no longer follows snapshots. It reads by time, applies refresh_append_overlap, and does not load a row that arrives later than the overlap. Without an overlap, it logs a warning at load. A new subsection documents this, and the three pages that said Iceberg appends always follow snapshots now name the exception.
Partitioned Cayenne accelerations publish snapshots (snapshots, partitioning): the limitation bullets saying they skip snapshots are removed.
Audit (Cluster-Sidecar architecture, vNext and 2.4.x): the page is a shared partial that every version from 2.0.x to vNext imports. Its 2.0.x–2.3.x content is copied unchanged to _cluster-sidecar-v2.3.mdx, and those four version pages now import the copy. The original partial, still used by vNext and 2.4.x, is corrected:
The sidecar example used engine: duckdb with snapshots: bootstrap_only against snapshots that the cluster creates with engine: cayenne. Snapshots from another engine are skipped, so the sidecar would never bootstrap from them. The sidecar now uses engine: cayenne, and the guidance says to use the same engine on both sides.
The prose recommended DuckDB, SQLite, and Arrow for the sidecar working set. Engines other than Cayenne are deprecated in v2.4.0, so the prose now recommends Cayenne and links the migration guide.
The cluster example set on_conflict: {id: upsert} on a Cayenne dataset. v2.4.0 logs that Cayenne no longer uses on_conflict, so the example drops it.
Doc said (website/src/partials/deployment/architectures/_cluster-sidecar.mdx:203 at docs 31f71082): the sidecar orders uses engine: duckdb / snapshots: bootstrap_only, while the cluster's orders (line 151) uses engine: cayenne / snapshots: create_only, at the same location.
Code says (crates/runtime-acceleration/src/snapshot/mod.rs at v2.4.0, lines 2905–2917):
// Early engine filtering: skip snapshots created by a different engine before
// attempting any download. ...
if let Some(ref snap_engine) = snapshot.snapshot_engine
&& !snap_engine.eq_ignore_ascii_case(¤t_engine)
{ ... first_failure.get_or_insert_with(|| SnapshotDownloadError::EngineMismatch { ... })
Behavior claim: not run. This needs object storage plus a PostgreSQL CDC source. The correction is based on reading the code above, which skips every snapshot made by another engine.
Item 4: non-Cayenne engines deprecated (Reproduced, name and existence claim)
Doc said (_cluster-sidecar.mdx:66,110,224,270): "(Arrow, DuckDB, or SQLite)", "Use DuckDB or SQLite in mode: file on the sidecar", "It executes against DuckDB", "a local DuckDB + vector index".
Code says (crates/runtime/src/component/mod.rs:207 at v2.4.0):
"Acceleration engines other than `cayenne` are deprecated and will be removed in v2.5, so these accelerations must move to `cayenne` before upgrading: {engines}. ..."
Item 5: on_conflict on Cayenne (Reproduced, name and existence claim)
Doc said (_cluster-sidecar.mdx:155-157): primary_key: id / on_conflict: / id: upsert under engine: cayenne.
Code says (crates/runtime/src/init/dataset.rs:3378 at v2.4.0):
"Dataset '{dataset_name}' sets `acceleration.on_conflict`, which Cayenne no longer uses{change} Remove `on_conflict`. See: https://spiceai.org/docs/features/data-acceleration/constraints"
At v2.3.3, neither deprecation string exists. That is why the 2.0.x–2.3.x pages keep the old text through the copied partial:
$ git grep -n 'which Cayenne no longer uses' v2.3.3 -- crates -> (no output)
$ git grep -n 'other than `cayenne` are deprecated' v2.3.3 -- crates -> (no output)
$ git grep -n 'Snapshot engine mismatch' v2.3.3 -- crates
v2.3.3:crates/runtime-acceleration/src/snapshot/mod.rs:414: "Snapshot engine mismatch for dataset {dataset}. ..."
The engine mismatch also exists at v2.3.3, so item 3 applies to the copied 2.3.x-and-older partial as well. That copy is left unchanged here, because whether Cayenne snapshots work on those releases has not been checked. It is listed as a follow-up.
Verification
Forward items, read at v2.4.0 and origin/trunk814e639e:
crates/runtime/src/init/dataset.rs:3434 (trunk) / :3411 (v2.4.0)
fn reads_by_time_despite_commits(acceleration, time_column, refresh_mode) -> bool {
let (Some(time_column), Some(key)) = (time_column, acceleration.primary_key.as_ref()) else { return false; };
refresh_mode == RefreshMode::Append
&& !acceleration.orders_versions_by_time(Some(time_column), refresh_mode)
&& !key.iter().any(|column| column == time_column)
}
crates/runtime-acceleration/src/acceleration.rs:992-999
pub fn orders_versions_by_time(...) -> bool { self.engine == Engine::Cayenne && time_column.is_some() && matches!(refresh_mode, RefreshMode::Full | RefreshMode::Append) }
crates/runtime/src/init/dataset.rs:1725-1739 warning only when the source records commits (Iceberg) and refresh_append_overlap is none
v2.4.0:crates/runtime/src/init/dataset.rs:3428 "Dataset '{dataset_name}' (`engine: {engine}`) has no `refresh_append_overlap`, so a row that arrives after a newer one is never loaded. ..."
v2.4.0:crates/data-accelerator-api/src/snapshots.rs:257-261
pub fn archive_would_be_unrestorable(engine, engine_override) -> bool { *engine == AccelerationEngine::Cayenne && engine_override.is_none() } (no partition_by condition)
$ git grep -n 'snapshots of a partitioned Cayenne' origin/trunk v2.4.0 -- crates -> (no output)
Known limitations: spiceai/spiceai#15346 says a row later than the overlap window is not loaded on these engines, and the page states this.
Propagation:
$ grep -rln 'skip periodic and pre-recreate snapshots\|do not support partitioned tables' website/docs website/versioned_docs/version-2.4.x | wc -l
0
version-2.3.x keeps its partitioned-snapshot limitation and its Iceberg append text, because neither source commit is in v2.3.3.
Files changed: 16 (5 vNext pages, 5 2.4.x pages, the corrected partial, the new _cluster-sidecar-v2.3.mdx, and 4 version pages whose import now points at the copy).
Build: NODE_OPTIONS=--max-old-space-size=12288 npm run build:
[SUCCESS] Generated static files in "build".
exit=0
Style grep over the added lines: no hits (style exit=1, meaning grep found nothing).
Updates vNext and 2.4.x documentation for Iceberg append refreshes and partitioned Cayenne snapshots, and corrects the current Cluster-Sidecar example while preserving older-version content.
Changes:
Documents time-based reads and overlap limits for keyed Iceberg appends on non-Cayenne engines.
Non-Cayenne accelerations whose primary_key includes time_column still follow Iceberg snapshots, so this summary incorrectly implies that their late rows depend on an overlap window. Qualify the exception with primary_key not including time_column, matching the subsection below and the source change. Same issue in website/versioned_docs/version-2.4.x/features/data-acceleration/refresh-modes/append.md:37.
docs: keyed Iceberg appends on other engines read by time, partitioned Cayenne snapshots, Cayenne in the Cluster-Sidecar example #2442 (review) (Copilot, Needs a closer look; previously missed finding on append.md:37): fixed in 6317e20 for vNext and 2.4.x. The Late-Arriving Data summary now names the condition reads_by_time_despite_commits checks in crates/runtime/src/init/dataset.rs (origin/trunk and v2.4.0): && !key.iter().any(|column| column == time_column), so only a keyed non-Cayenne acceleration whose primary_key does not include the time_column reads by time. The iceberg.md and datasets.md sentences already carried this condition.
Local gate: cd website && npm run build succeeded on 6317e20.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR combines two Forward updates and one Audit correction.
Forward (vNext and 2.4.x): both source PRs were cherry-picked into
v2.4.0(97d9b38549 and c87fa6236b are ancestors ofv2.4.0). They change the current release's behavior, so bothwebsite/docs/andversion-2.4.xchange.appendon engines other than Cayenne reads by time (refresh-modes/append,components/data-connectors/iceberg,reference/spicepod/datasets). This applies to an acceleration on an engine other than Cayenne whoseprimary_keydoes not include thetime_column. Its append refresh of an Iceberg source no longer follows snapshots. It reads by time, appliesrefresh_append_overlap, and does not load a row that arrives later than the overlap. Without an overlap, it logs a warning at load. A new subsection documents this, and the three pages that said Iceberg appends always follow snapshots now name the exception.snapshots,partitioning): the limitation bullets saying they skip snapshots are removed.Audit (Cluster-Sidecar architecture, vNext and 2.4.x): the page is a shared partial that every version from 2.0.x to vNext imports. Its 2.0.x–2.3.x content is copied unchanged to
_cluster-sidecar-v2.3.mdx, and those four version pages now import the copy. The original partial, still used by vNext and 2.4.x, is corrected:engine: duckdbwithsnapshots: bootstrap_onlyagainst snapshots that the cluster creates withengine: cayenne. Snapshots from another engine are skipped, so the sidecar would never bootstrap from them. The sidecar now usesengine: cayenne, and the guidance says to use the same engine on both sides.v2.4.0, so the prose now recommends Cayenne and links the migration guide.on_conflict: {id: upsert}on a Cayenne dataset.v2.4.0logs that Cayenne no longer useson_conflict, so the example drops it.Source PRs
Reproduction
The audit items are checked against
spiceai/spiceaiv2.4.0. vNext is the same atorigin/trunk814e639e8eb55eacbe3f3fadaad2ec0aaf712c05.Item 3: snapshot engine mismatch (Unverified, code inspection only)
Doc said (
website/src/partials/deployment/architectures/_cluster-sidecar.mdx:203at docs31f71082): the sidecarordersusesengine: duckdb/snapshots: bootstrap_only, while the cluster'sorders(line 151) usesengine: cayenne/snapshots: create_only, at the same location.Code says (
crates/runtime-acceleration/src/snapshot/mod.rsatv2.4.0, lines 2905–2917):Behavior claim: not run. This needs object storage plus a PostgreSQL CDC source. The correction is based on reading the code above, which skips every snapshot made by another engine.
Item 4: non-Cayenne engines deprecated (Reproduced, name and existence claim)
Doc said (
_cluster-sidecar.mdx:66,110,224,270): "(Arrow, DuckDB, or SQLite)", "Use DuckDB or SQLite inmode: fileon the sidecar", "It executes against DuckDB", "a local DuckDB + vector index".Code says (
crates/runtime/src/component/mod.rs:207atv2.4.0):Item 5:
on_conflicton Cayenne (Reproduced, name and existence claim)Doc said (
_cluster-sidecar.mdx:155-157):primary_key: id/on_conflict:/id: upsertunderengine: cayenne.Code says (
crates/runtime/src/init/dataset.rs:3378atv2.4.0):At
v2.3.3, neither deprecation string exists. That is why the 2.0.x–2.3.x pages keep the old text through the copied partial:The engine mismatch also exists at
v2.3.3, so item 3 applies to the copied 2.3.x-and-older partial as well. That copy is left unchanged here, because whether Cayenne snapshots work on those releases has not been checked. It is listed as a follow-up.Verification
Forward items, read at
v2.4.0andorigin/trunk814e639e:Known limitations: spiceai/spiceai#15346 says a row later than the overlap window is not loaded on these engines, and the page states this.
Propagation:
version-2.3.xkeeps its partitioned-snapshot limitation and its Iceberg append text, because neither source commit is inv2.3.3.Files changed: 16 (5 vNext pages, 5 2.4.x pages, the corrected partial, the new
_cluster-sidecar-v2.3.mdx, and 4 version pages whose import now points at the copy).Build:
NODE_OPTIONS=--max-old-space-size=12288 npm run build:Style grep over the added lines: no hits (
style exit=1, meaning grep found nothing).Test plan
cd website && npm run buildpasses (output above)git diff --name-only origin/trunk..HEAD | wc -l= 16)Review gate
codexnot installed (receipt:attempt=codex result=not-installed,engine=none)/security-review: not run — docs-only Markdown diff; the same review on the sibling docs-only PR docs: request-context metric labels, HTTPS redirect guard, ORC metadata limit, scalar function memory, OTLP log record size #2440 found nothing/simplify: not run — docs-only diff, no code