diff --git a/website/docs/cli/reference/cloud.md b/website/docs/cli/reference/cloud.md index 022da13d6..ac3b6b2b4 100644 --- a/website/docs/cli/reference/cloud.md +++ b/website/docs/cli/reference/cloud.md @@ -300,10 +300,15 @@ The project half of the report comes from Spice Cloud, so the command needs an a "datasets_total": 0, "datasets_unhealthy": [], "runtime_error": null, + "health_not_reported": null, "link": { "connection": {}, "service": {}, "deployment": {} } } ``` +For Cloud Connect and BYOC runtimes, `instances` is empty and `runtime_error` is `null`. +`health_not_reported` explains why health is unavailable; the table output shows the same reason. +This does not affect the exit code. + ### `logs` ```shell diff --git a/website/docs/cli/reference/run.md b/website/docs/cli/reference/run.md index e0ccb5481..ac05d254e 100644 --- a/website/docs/cli/reference/run.md +++ b/website/docs/cli/reference/run.md @@ -7,7 +7,20 @@ pagination_next: null Run Spice - starts the Spice runtime, installing if necessary. -`spice run` is a wrapper around the [`spiced`](./spiced) runtime binary. It installs `spiced` on first use, applies developer-friendly defaults, and forwards arguments after `--` to the runtime. To invoke the runtime directly (for containers, systemd units, or CI), see the [`spiced` reference](./spiced). +`spice run` is a wrapper around the [`spiced`](./spiced) runtime binary. It applies developer-friendly defaults and forwards arguments after `--` to the runtime. To invoke the runtime directly (for containers, systemd units, or CI), see the [`spiced` reference](./spiced). + +### Runtime selection + +`spice run` and [`spice version`](./version) look for `spiced` in this order: + +1. `$SPICED_PATH`. +2. Beside the running `spice` binary. +3. `$HOME/.spice/bin/spiced`, managed by [`spice install`](./install). +4. Under `sudo`, the invoking user's `~/.spice/bin/spiced`. + +`PATH` is not searched. An invalid `$SPICED_PATH` causes an error; otherwise, `spice run` installs +the runtime if none is found. For a runtime outside the managed install, the CLI reports +its path and source. [`spice version`](./version) also reports these values. ### Usage diff --git a/website/docs/cli/reference/version.md b/website/docs/cli/reference/version.md index e59b06485..80f89c760 100644 --- a/website/docs/cli/reference/version.md +++ b/website/docs/cli/reference/version.md @@ -41,4 +41,16 @@ Learn more about upgrading the Spice CLI and runtime using `spice upgrade` [here CLI version: v1.1.0 Runtime version: v1.1.0+models - ``` \ No newline at end of file + ``` + +For a runtime outside the managed install, the output also includes its path and source: + +```shell +> spice version + +CLI version: v1.1.0 +Runtime version: v1.1.0+models +Runtime path: /usr/local/bin/spiced (beside the spice CLI) +``` + +JSON output includes `runtime_path` and `runtime_source`, following the same [runtime selection rules](./run#runtime-selection). diff --git a/website/docs/components/catalogs/adbc.md b/website/docs/components/catalogs/adbc.md index ec8266d52..12eb7392c 100644 --- a/website/docs/components/catalogs/adbc.md +++ b/website/docs/components/catalogs/adbc.md @@ -60,6 +60,7 @@ The following parameters are supported: | `adbc_driver_options` | Optional. Semicolon-delimited driver-specific database options (e.g., `key1=value1;key2=value2`). | | `connection_pool_size` | Optional. Maximum number of connections in the connection pool. Default: `5`. | | `connection_pool_min_idle` | Optional. Minimum number of idle connections to keep open in the pool. Default: `1`. | +| `query_federation` | Optional. Controls whether queries are federated to the ADBC source. Values: `enabled`, `disabled`. Default: `enabled`. | ## Examples diff --git a/website/docs/components/catalogs/glue.md b/website/docs/components/catalogs/glue.md index 61977e207..0a8e831cb 100644 --- a/website/docs/components/catalogs/glue.md +++ b/website/docs/components/catalogs/glue.md @@ -223,6 +223,9 @@ Write operations require `s3:PutObject` permission on the target S3 bucket in ad ::: +Tables with unsupported formats are skipped. A warning summarizes skipped tables per database; +debug logs provide each reason. Warnings are rate-limited unless the set of skipped tables changes. + ## Cookbook There is a [cookbook recipe](https://github.com/spiceai/cookbook/tree/trunk/catalogs/glue) to configure an AWS Glue Data Connector in Spice. diff --git a/website/docs/components/data-accelerators/cayenne/deployment.md b/website/docs/components/data-accelerators/cayenne/deployment.md index c0b979784..175cd666e 100644 --- a/website/docs/components/data-accelerators/cayenne/deployment.md +++ b/website/docs/components/data-accelerators/cayenne/deployment.md @@ -117,6 +117,7 @@ Generic acceleration metrics are available with the `dataset_acceleration_` pref | `cayenne_compaction_duration_ms` | Histogram | ms | Wall-clock time of Cayenne background compaction passes. The histogram's count doubles as the compaction-pass counter. | | `cayenne_compaction_memory_pool_bytes` | Gauge | By | Size of the dedicated compaction memory pool carved from the query memory limit (see `cayenne_compaction_memory_fraction`). | | `cayenne_compaction_memory_exhausted_total` | Counter | passes | Compaction passes that hit `ResourcesExhausted` on the dedicated compaction memory pool. | +| `cayenne_delete_main_visibility_downgrade_total` | Counter | deletes | Delete passes that deferred row cleanup to protect concurrent replacement rows. Label: `table`. A sustained increase indicates deferred cleanup under ingest load. | ### Memory Reconciliation Metrics diff --git a/website/docs/components/data-accelerators/cayenne/index.md b/website/docs/components/data-accelerators/cayenne/index.md index 79bbe7ace..07d9f834b 100644 --- a/website/docs/components/data-accelerators/cayenne/index.md +++ b/website/docs/components/data-accelerators/cayenne/index.md @@ -638,7 +638,7 @@ datasets: ## Data Type Support -Cayenne (via Vortex) supports most Arrow data types with the following considerations: +Cayenne supports most Arrow data types. The [accelerator data type table](../../reference/datatypes/accelerators) lists storage representations and engine comparisons. ### Fully Supported Types @@ -647,7 +647,7 @@ Cayenne (via Vortex) supports most Arrow data types with the following considera - Boolean - Utf8 and LargeUtf8 strings - Binary and LargeBinary -- Timestamps (normalized to Microsecond precision) +- Timestamps, preserving the source unit and timezone - Date32 and Date64 - Lists and FixedSizeLists - Maps @@ -655,10 +655,17 @@ Cayenne (via Vortex) supports most Arrow data types with the following considera ### Automatically Converted Types -| Original Type | Converted To | Notes | -| --------------------------- | ------------------------ | --------------------------------------------- | -| `Float16` | `Float32` | Automatic conversion for Vortex compatibility | -| `Timestamp(Nanosecond/...)` | `Timestamp(Microsecond)` | Precision normalized | +| Original Type | Converted To | Notes | +| ------------- | ------------ | --------------------------------------------- | +| `Float16` | `Float32` | Automatic conversion for Vortex compatibility | + +:::note Tables created before v2.2.0 + +These tables continue to normalize timestamps to microseconds. Preserving the source unit requires +recreating the table with `mode: file_create` in an empty directory. The +[`on_schema_change`](../../reference/spicepod/datasets#on_schema_change) setting does not migrate existing timestamps. + +::: ### Unsupported Types @@ -737,6 +744,8 @@ A file-mode Cayenne dataset whose resolved metastore directory falls inside its Paths are compared after `.`/`..` are collapsed and symlinks are resolved, so neither hides an overlap, and a sibling that merely shares a name prefix (`…/meta` next to `…/metadata`) is not affected. Datasets whose data lives on object storage (for example an S3 Express `cayenne_file_path`) are exempt — the metastore is always local, so it cannot sit inside an object-store data path. +Before recreating or deleting a data directory, Spice checks for `cayenne.db` and its SQLite sidecars, including those belonging to other datasets. Deletion is refused if the directory contains a metastore, links directly to one, or has unreadable entries. Metastore files must reside outside data directories. + ### CPU Query performance scales with available CPU cores. Vortex's columnar format supports parallel decompression and scanning across multiple threads. Allocate sufficient CPU for: @@ -779,14 +788,23 @@ Remove `on_conflict` to keep writes on the accelerator, or choose a different [` **Durable write-back requirements:** - The federated source must be **PostgreSQL** (see the warning above). -- The dataset must declare a **single-column [`acceleration.primary_key`](../../reference/spicepod/datasets#accelerationprimary_key)**. Delivery keys each committed row on that column, and a composite key cannot be expressed as the key filter it delivers with, so a multi-column key is rejected at registration with an error naming the columns rather than registering and then never delivering. -- The accelerator must be the **sole writer** of the rows it delivers. The upsert is an unconditional `ON CONFLICT ... DO UPDATE` with no compare-and-set against the source, so a second writer mutating the same source row directly can be overwritten by a later delivery. +- A **single-column [`acceleration.primary_key`](../../reference/spicepod/datasets#accelerationprimary_key)** is required. +- [`acceleration.mode`](../../reference/spicepod/datasets#accelerationmode) must be **`file`**. +- No acceleration retention may be configured, to preserve undelivered rows. +- The accelerator must be the **sole writer** of these source rows; delivery can overwrite changes made directly at the source. + +**Supported writes:** + +- `INSERT` and `UPDATE` must run inside a [transaction](#transactions). +- `DELETE` and `MERGE` are not supported. + +Delivery is asynchronous. Failed deliveries and temporarily unreadable rows remain pending for retry; an unreadable row does not trigger a source deletion. A persistent [`dataset_acceleration_write_back_pending_keys`](../../features/observability) backlog indicates a delivery problem. -Delivery is asynchronous and does not block accelerator commits: a delivery failure leaves the pending set to grow and retries on the next pass, and only a delivery that has been accepted by the source clears its marker. +Before a source deletion, writes must stop and pending keys must reach zero **while write-back is enabled**. Write-back can then be disabled and the source rows deleted, with CDC refreshing the accelerator. Disabling write-back resets the gauge without delivering pending rows. **Requirements and v1 limitations:** -- Write targets must be **accelerator-only, non-partitioned Cayenne datasets**. Other dataset modes route writes to the federated source — where the gate cannot govern them — and are rejected. +- Write targets must be **non-partitioned Cayenne datasets** configured as accelerator-only or for durable write-back. - Only **`INSERT` and `UPDATE`** writes are supported inside a transaction. `DELETE` and `MERGE` are rejected. - At most **one write per table** per transaction. Multiple tables may be written in the same transaction and are committed atomically together. - Reading a Cayenne table that is not a registered participant (for example, a partitioned table) fails the transaction closed. @@ -800,10 +818,11 @@ Consider the following limitations when using Spice Cayenne acceleration: - **Memory Mode Constraints**: `mode: memory` (fully in-RAM, ephemeral) is supported alongside `mode: file`, but it does not persist any data (the dataset reloads from its source on restart), does not support partitioned tables (`partition_by`), and enforces a hard per-table RAM bound instead of spilling to disk — a breach returns an error rather than growing without limit. Use `mode: file` when persistence across restarts is required. - **S3 Express Only**: Standard S3 buckets are not supported for remote storage. Only S3 Express One Zone directory buckets are supported. - **Unsupported Data Types**: `Interval`, `Duration`, `FixedSizeBinary`, `Union`, and `RunEndEncoded` types require `unsupported_type_action` configuration. -- **No Traditional Indexes**: Spice Cayenne does not support explicit index creation via the `indexes` configuration. Vortex's segment statistics and fast random access encodings provide equivalent or better performance for most point lookup workloads. +- **Indexes**: `indexes` is ignored with a warning, including `unique` indexes. Deduplication requires `primary_key` with [`on_conflict`](../../reference/spicepod/datasets#accelerationon_conflict). +- **SQL Retention**: [`retention_sql`](../../reference/spicepod/datasets#accelerationretention_sql) runs during maintenance after writes, full refreshes, and CDC checkpoints, independently of periodic retention settings. It is ignored in `mode: memory`, with a warning; matching rows remain queryable. +- **Time-Based Retention**: [`retention_period`](../../reference/spicepod/datasets#accelerationretention_period) hides expired rows in either mode. Scheduled deletion requires both [`retention_check_enabled: true`](../../reference/spicepod/datasets#accelerationretention_check_enabled) and [`retention_check_interval`](../../reference/spicepod/datasets#accelerationretention_check_interval), which has no default. Without both, a warning is logged and storage is reclaimed only when compaction rewrites affected files. - **No MVCC**: Multi-version concurrency control is not yet implemented. Snapshots and time-travel queries are planned for future releases. -- **Transaction Constraints**: [Transactions](#transactions) support gated `INSERT`/`UPDATE` writes on accelerator-only, non-partitioned Cayenne tables only (no `DELETE`/`MERGE`, one write per table). See [Transactions](#transactions) for the full list. -- **No `refresh_append_overlap`**: A dataset accelerated by Spice Cayenne that sets [`acceleration.refresh_append_overlap`](../../reference/spicepod/datasets#accelerationrefresh_append_overlap) fails to load, with `Cayenne data accelerator does not yet support refresh_append_overlap. Please remove this configuration`. [`refresh_mode: append`](../../features/data-acceleration/data-refresh) itself is supported — only the overlap window is not, so late-arriving rows behind the high-water mark are missed rather than re-read. The check runs during file-mode initialization, so a `mode: memory` dataset is not rejected. +- **Transaction Constraints**: [Transactions](#transactions) support `INSERT`/`UPDATE` on non-partitioned, accelerator-only or durable write-back datasets, with one write per table. `DELETE` and `MERGE` are not supported. ## Example Spicepod diff --git a/website/docs/components/data-accelerators/turso.md b/website/docs/components/data-accelerators/turso.md index 722c34581..9b9dcafb5 100644 --- a/website/docs/components/data-accelerators/turso.md +++ b/website/docs/components/data-accelerators/turso.md @@ -152,6 +152,7 @@ Turso supports query federation, where queries can span multiple data sources. T - **Remote databases not supported**: Only local Turso databases (file-based or in-memory) are supported as accelerators. Remote Turso databases using `turso_url` and `turso_auth_token` are not supported in this accelerator context. Remote Turso support will be available when Turso is implemented as a data connector. - **Arrow Interval types**: Not supported, as SQLite/libSQL doesn't have a native interval type. - **Complex List types**: Only Arrow `List` types of primitive data types are supported; lists with structs are not supported. +- **Older list encodings**: List values stored without an encoding-version marker read as `NULL` with a warning. Refreshing the acceleration or using a fresh file rewrites them in the supported encoding. - **Dictionary and Map types**: Not supported. - **Hot-reload federation**: Updating a dataset with Turso acceleration while the Spice Runtime is running (hot-reload) may disable query federation until the runtime is restarted. - **ROLLUP and GROUPING**: Advanced grouping features are not supported. diff --git a/website/docs/components/data-connectors/adbc.md b/website/docs/components/data-connectors/adbc.md index a353be072..e413cbb06 100644 --- a/website/docs/components/data-connectors/adbc.md +++ b/website/docs/components/data-connectors/adbc.md @@ -275,6 +275,8 @@ The ADBC connector maintains a pool of database connections for concurrent query Both values must be positive integers, and `connection_pool_min_idle` must not exceed `connection_pool_size` — a larger value causes connection pool initialization to fail. +Query cancellation or deadline expiry cancels the driver statement and releases the pooled connection. For BigQuery, this also stops the remote job. + ### Query Pushdown The ADBC connector pushes SQL operations down to the source database when possible, reducing the amount of data transferred: @@ -284,7 +286,9 @@ The ADBC connector pushes SQL operations down to the source database when possib - **Limit pushdown**: `LIMIT` clauses are applied at the source. - **Aggregation pushdown**: `GROUP BY`, `SUM`, `COUNT`, `AVG`, and other aggregations are executed on the source. - **Sort pushdown**: `ORDER BY` clauses are applied at the source. -- **Join pushdown**: Joins between datasets from the same ADBC URI are executed on the remote database. +- **Join pushdown**: Joins between datasets that reach the same remote engine are executed on the remote database. + +Join pushdown requires matching driver, URI, credentials, driver options, and explicit `adbc_catalog`/`adbc_schema` settings. BigQuery tables in different datasets within one project can share a job: dataset qualifiers in `from:` do not need to match, but explicit `adbc_schema` settings do. No special configuration is required. Pushdown happens automatically when the source database supports the operation. diff --git a/website/docs/components/data-connectors/ducklake.md b/website/docs/components/data-connectors/ducklake.md index 4598d7796..930ed844c 100644 --- a/website/docs/components/data-connectors/ducklake.md +++ b/website/docs/components/data-connectors/ducklake.md @@ -207,7 +207,7 @@ datasets: :::warning[Limitations] -- Spice uses DuckDB 1.5.3, which supports DuckLake 1.0. Older DuckLake catalogs require a metadata migration before use — set `ducklake_automatic_migration: true` to perform it on attach (this rewrites catalog metadata and cannot be undone). See [DuckLake migration guide](https://ducklake.select/docs/stable/duckdb/guides/troubleshooting#connecting-to-an-older-ducklake). +- Spice embeds DuckDB **1.4.4** and supports the catalog schema required by its `ducklake` extension. For older catalogs, `ducklake_automatic_migration: true` performs an irreversible [metadata migration](https://ducklake.select/docs/stable/duckdb/guides/troubleshooting#connecting-to-an-older-ducklake) on attach. - The DuckLake DuckDB extension is downloaded at runtime on first use, requiring network connectivity. - The `ducklake_connection_string` parameter is required — unlike the catalog connector, it cannot be omitted. - Each dataset creates its own DuckDB connection pool. For querying many tables from the same catalog, consider using the [DuckLake Catalog Connector](../catalogs/ducklake) instead, which shares a single connection pool. diff --git a/website/docs/components/data-connectors/postgres/index.md b/website/docs/components/data-connectors/postgres/index.md index 8d7b9b21c..5c58a4ff7 100644 --- a/website/docs/components/data-connectors/postgres/index.md +++ b/website/docs/components/data-connectors/postgres/index.md @@ -154,7 +154,7 @@ The table below shows the PostgreSQL data types supported, along with the type m | `money` | `Int64` | | `float4` | `Float32` | | `float8` | `Float64` | -| `numeric` | `Decimal128` | +| `numeric` | `Decimal128` [1](#numeric-precision-and-scale) | | `text` | `Utf8` | | `varchar` | `Utf8` | | `bpchar` | `Utf8` | @@ -187,6 +187,14 @@ The Postgres federated queries may result in unexpected result types due to the ::: +### `numeric` precision and scale + +`numeric(p,s)` maps to `Decimal128(p,s)`, including negative scales. An omitted scale defaults to +zero: `numeric(12)` becomes `Decimal128(12,0)`. + +Unconstrained `numeric` maps to `Decimal128(38,20)`. Reads fail for values requiring more than +18 integral or 20 fractional digits. An explicit column precision and scale controls this mapping. + ## Write Support The PostgreSQL connector supports writing data to PostgreSQL tables using SQL [`INSERT INTO`](../../reference/sql/dml#insert), `UPDATE`, and `DELETE FROM` statements. diff --git a/website/docs/components/embeddings/google.md b/website/docs/components/embeddings/google.md index 5154309c8..bda5e4782 100644 --- a/website/docs/components/embeddings/google.md +++ b/website/docs/components/embeddings/google.md @@ -1,36 +1,37 @@ --- -title: 'Google AI Embedding Models' -sidebar_label: 'Google AI' +title: 'Google Vertex AI Embedding Models' +sidebar_label: 'Google Vertex AI' sidebar_position: 4 --- -To use a hosted Google AI embedding model, specify the `google` path in the `from` field of your configuration. +`from: google:` selects a Vertex AI embedding model. A model ID, GCP project, location, +and Google Cloud credentials are required. Google AI Studio API keys are not supported. -Include the model ID in the `from` field; a model ID is required. For example, `google:gemini-embedding-2` selects Google's latest generally available embedding model. +| Parameter | Description | Default | +| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | +| `google_project` | Required. GCP project ID containing lowercase letters, digits, and hyphens. | - | +| `google_location` | Required. GCP region (for example `us-central1`) or `global`. | - | +| `google_service_account_path` | Path to a GCP service account JSON key file. | - | +| `google_service_account_key` | Service account JSON. Supports [secret replacement](../secret-stores). | - | +| `google_application_default_credentials` | Application Default Credentials from the file specified by `GOOGLE_APPLICATION_CREDENTIALS`. | `false` | +| `dimensions` | The output dimensionality of the embeddings. Some embedding models support dynamic output sizes. | - | -The following parameters are specific to Google AI embedding models: - -| Parameter | Description | Default | -| ---------------- | ------------------------------------------------------------------------------------------------ | ------- | -| `google_api_key` | The API key for accessing Google AI. | - | -| `dimensions` | The output dimensionality of the embeddings. Some embedding models support dynamic output sizes. | - | +Authentication requires exactly one of `google_service_account_path`, `google_service_account_key`, +or `google_application_default_credentials: true`. Below is an example configuration in `spicepod.yaml`: ```yaml embeddings: - - from: google:gemini-embedding-2 + - from: google:gemini-embedding-001 name: gemini_embeddings params: - google_api_key: ${ secrets:GEMINI_API_KEY } + google_project: my-gcp-project + google_location: us-central1 + google_service_account_path: /etc/spice/gcp-service-account.json dimensions: 768 # optional parameter - - - from: google:gemini-embedding-001 - name: legacy_embeddings - params: - google_api_key: ${ secrets:GEMINI_API_KEY } ``` -See [Google AI Embedding Models](https://ai.google.dev/gemini-api/docs/models/gemini#text-embedding) for a list of supported embedding models. +[Vertex AI text embeddings](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-text-embeddings) lists model availability by project and region. For detailed instructions and examples on running vector searches, refer to the [Vector-Based Search documentation](../../features/search/vector-search). diff --git a/website/docs/components/embeddings/index.md b/website/docs/components/embeddings/index.md index ae7595548..326c59b23 100644 --- a/website/docs/components/embeddings/index.md +++ b/website/docs/components/embeddings/index.md @@ -25,7 +25,7 @@ Spice supports a variety of embedding model sources and formats: | [`huggingface`][hf] | Models hosted on HuggingFace | Release Candidate | GGUF, GGML, SafeTensor | | [`openai`][openai] | OpenAI (or compatible) LLM endpoint | Release Candidate | OpenAI-compatible HTTP endpoint | | [`azure`][azure] | Azure OpenAI | Alpha | OpenAI-compatible HTTP endpoint | -| [`google`][google] | Google AI embedding models | Alpha | OpenAI-compatible HTTP endpoint | +| [`google`][google] | Google Vertex AI embedding models | Alpha | Vertex AI HTTP endpoint | | [`databricks`][databricks] | Models deployed to Databricks Mosaic AI | Alpha | OpenAI-compatible HTTP endpoint | | [`bedrock`][bedrock] | Models deployed on Amazon Bedrock | Alpha | OpenAI-compatible HTTP endpoint | | [`model2vec`][model2vec] | Model2Vec static word embeddings | Alpha | Model2Vec format | diff --git a/website/docs/components/models/google.md b/website/docs/components/models/google.md index ddb9cfdf3..065b7ab90 100644 --- a/website/docs/components/models/google.md +++ b/website/docs/components/models/google.md @@ -1,31 +1,39 @@ --- -title: 'Google AI Models' -description: 'Instructions for using language models hosted on Google AI with Spice.' -sidebar_label: 'Google AI' +title: 'Google Vertex AI Models' +description: 'Instructions for using language models hosted on Google Vertex AI with Spice.' +sidebar_label: 'Google Vertex AI' sidebar_position: 5 --- -To use a language model hosted on Google AI, specify `google` in the `from` field. +`from: google:` selects a Vertex AI language model. A model ID, GCP project, location, +and Google Cloud credentials are required. Google AI Studio API keys are not supported. -Include a model ID in the `from` field (see example below); a model ID is required. Spice does not apply a default model — the model fails to load if the ID is omitted (`from: google`). +| Parameter | Description | Default | +| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | +| `google_project` | Required. GCP project ID containing lowercase letters, digits, and hyphens. | - | +| `google_location` | Required. GCP region (for example `us-central1`) or `global`. | - | +| `google_service_account_path` | Path to a GCP service account JSON key file. | - | +| `google_service_account_key` | Service account JSON. Supports [secret replacement](../secret-stores). | - | +| `google_application_default_credentials` | Application Default Credentials from the file specified by `GOOGLE_APPLICATION_CREDENTIALS`. | `false` | -The following parameters are specific to Google AI models: - -| Parameter | Description | Default | -| ---------------- | ----------------------- | ------- | -| `google_api_key` | The Google AI API key. | - | +Authentication requires exactly one of `google_service_account_path`, `google_service_account_key`, +or `google_application_default_credentials: true`. Example `spicepod.yml` configuration: ```yaml models: - - from: google:gemini-3.5-flash - name: flash + - from: google:gemini-2.5-pro + name: gemini params: - google_api_key: ${ secrets:GEMINI_API_KEY } + google_project: my-gcp-project + google_location: us-central1 + google_service_account_path: /etc/spice/gcp-service-account.json ``` -See [Google AI Models](https://ai.google.dev/gemini-api/docs/models/gemini) for a list of supported model names. +`google_service_account_key: ${ secrets:GCP_SERVICE_ACCOUNT_KEY }` selects a secret instead of a file. + +[Vertex AI Generative AI models](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models) lists model availability by project and region. See [Large Language Models](../../features/large-language-models) for additional configuration options: diff --git a/website/docs/components/models/index.md b/website/docs/components/models/index.md index 5d27dd586..a53ab6e3d 100644 --- a/website/docs/components/models/index.md +++ b/website/docs/components/models/index.md @@ -20,7 +20,7 @@ Spice supports various model providers for large language models (LLMs). | [`spice.ai`][spice] | Models hosted on the Spice.ai Cloud Platform | Release Candidate | OpenAI-compatible HTTP endpoint | | [`azure`][azure] | Azure OpenAI | Alpha | OpenAI-compatible HTTP endpoint | | [`anthropic`][ant] | Models hosted on Anthropic | Alpha | OpenAI-compatible HTTP endpoint | -| [`google`][google] | Google AI language models | Alpha | OpenAI-compatible HTTP endpoint | +| [`google`][google] | Google Vertex AI language models | Alpha | Vertex AI HTTP endpoint | | [`databricks`][databricks] | Models deployed to Databricks Mosaic AI | Alpha | OpenAI-compatible HTTP endpoint | | ~~`perplexity`~~ | ~~Perplexity~~ ([Deprecated][perplexity]) | Deprecated | - | @@ -66,7 +66,7 @@ The following provider prefixes are supported: | `azure` | Azure OpenAI | | `xai` | xAI | | `anthropic` | Anthropic | -| `google` | Google AI | +| `google` | Google Vertex AI | | `hf` | Hugging Face | | `file` | Local filesystem | | `spiceai` | Spice.ai Cloud Platform | diff --git a/website/docs/components/vectors/elasticsearch.md b/website/docs/components/vectors/elasticsearch.md index ecee64976..1dadfd617 100644 --- a/website/docs/components/vectors/elasticsearch.md +++ b/website/docs/components/vectors/elasticsearch.md @@ -118,6 +118,13 @@ columns: row_id: product_id ``` +### Index write and delete errors + +Spice checks per-document write and delete results. Partial failures, including version conflicts, +fail the operation and are logged to [`runtime.task_history`](../../reference/task_history) with the document and cause. +Deletes require a synchronous `_delete_by_query` response; async task handles are rejected. +Deletes are not retried automatically because a retry could remove a concurrently updated document. + ### Custom Index and Vector Field Names By default the index name is a sanitized `{dataset}-{column}-{model}` and the vector field is `{column}_embedding`. Override either with `elasticsearch_index` and `elasticsearch_vector_field`: diff --git a/website/docs/deployment/gcp/integrations.md b/website/docs/deployment/gcp/integrations.md index bba228fcf..a0b12c369 100644 --- a/website/docs/deployment/gcp/integrations.md +++ b/website/docs/deployment/gcp/integrations.md @@ -91,42 +91,40 @@ datasets: For [Postgres replication-based CDC](../../features/cdc/postgres-replication), Cloud SQL requires the `cloudsql.logical_decoding = on` flag. -## AI Models (Google AI) +## AI Models (Vertex AI) -Spice integrates with [Google AI Studio](https://aistudio.google.com) for chat completion and reasoning models, including the Gemini family. - -| Provider | Supported Models | Documentation | -| ------------- | ----------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | -| **Google AI** | Gemini 2.0/2.5/Pro, Gemini Flash, and other models from the [Gemini API](https://ai.google.dev/gemini-api). | [Google AI Models](../../components/models/google) | +[Vertex AI](https://cloud.google.com/vertex-ai) Gemini models require a GCP project, location, and +service account or Application Default Credentials. [Google Vertex AI Models](../../components/models/google) lists parameters and supported models. ### Example: Gemini Chat Model ```yaml models: - - from: google:gemini-2.0-flash-exp + - from: google:gemini-2.5-flash name: gemini params: - google_api_key: ${ secrets:GEMINI_API_KEY } + google_project: my-gcp-project + google_location: us-central1 + google_service_account_path: /etc/spice/gcp-service-account.json ``` -See [Google AI Models](https://ai.google.dev/gemini-api/docs/models/gemini) for the full list of supported model names. - -## Embeddings (Google AI) +Application Default Credentials require `google_application_default_credentials: true` instead of +`google_service_account_path`, with `GOOGLE_APPLICATION_CREDENTIALS` pointing to the credentials file. -Generate vector embeddings using Gemini embedding models for semantic search and retrieval-augmented generation (RAG). +## Embeddings (Vertex AI) -| Provider | Supported Models | Documentation | -| ------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | -| **Google AI** | `gemini-embedding-2` and other models from [Gemini API embeddings](https://ai.google.dev/gemini-api/docs/models/gemini#text-embedding). | [Google AI Embeddings](../../components/embeddings/google) | +[Vertex AI embedding models](../../components/embeddings/google) support semantic search and retrieval-augmented generation (RAG). -### Example: Google AI Embeddings +### Example: Vertex AI Embeddings ```yaml embeddings: - - from: google:gemini-embedding-2 + - from: google:gemini-embedding-001 name: gemini_embeddings params: - google_api_key: ${ secrets:GEMINI_API_KEY } + google_project: my-gcp-project + google_location: us-central1 + google_service_account_path: /etc/spice/gcp-service-account.json ``` ## Snapshots and shared state diff --git a/website/docs/features/caching/index.md b/website/docs/features/caching/index.md index 90eb7eeb1..ddc3f4a07 100644 --- a/website/docs/features/caching/index.md +++ b/website/docs/features/caching/index.md @@ -51,6 +51,17 @@ Every cache type (`sql_results`, `search_results`, `embeddings`) supports the fo | `eviction_policy` | Yes | `lru` | Cache replacement policy when the cache reaches `max_size`. Defaults to `lru`. Supports `lru` (Least Recently Used) and `tiny_lfu` (Tiny Least Frequently Used, higher hit rate for skewed access patterns). | | `item_ttl` | Yes | `1s` | Cache entry expiration duration (Time to Live). Defaults to 1 second. | | `hashing_algorithm` | Yes | `xxh3` | Selects which hashing algorithm is used to hash the cache keys when storing the results. Defaults to `xxh3`. Supports `xxh3`, `ahash`, `siphash`, `blake3`, `xxh32`, `xxh64`, or `xxh128`. | +| `engine` | Yes | `moka` | Cache backend: `moka` or `pingora`. See [Choosing an `engine`](#choosing-an-engine). | + +### Choosing an `engine` + +- **`moka` (default):** Supports TTL expiration and lazy table invalidation. +- **`pingora`:** Uses a sharded LRU cache. Reads lock the key's shard; table invalidations that + evict entries scan the cache, with cost proportional to its size. + +With a non-zero [`stale_while_revalidate_ttl`](#serving-stale-after-an-acceleration-refresh), SQL +result invalidations mark entries stale without eviction, so neither engine scans. +Search result invalidations always evict entries. ## `caching.sql_results` Parameters diff --git a/website/docs/features/cdc/mysql-replication.md b/website/docs/features/cdc/mysql-replication.md index 537340ccb..98d54d89e 100644 --- a/website/docs/features/cdc/mysql-replication.md +++ b/website/docs/features/cdc/mysql-replication.md @@ -105,7 +105,7 @@ Configure replication behavior with the following `params` on the MySQL dataset: | Parameter | Default | Description | | -------------------------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mysql_replication_server_id` | derived | The `server_id` this replica registers with. Must be unique among all replicas attached to the source. The default is derived from the **connection identity** (host, port, user, credentials, TLS) mixed with a per-process nonce, so datasets sharing a connection coalesce onto one binlog connection while two spiced instances don't collide, and derived values stay at or above `100000` to avoid hand-assigned replica ids. Setting distinct explicit values is how to opt a dataset out of [connection sharing](#sharing-one-binlog-connection). | -| `mysql_replication_initial_snapshot` | `auto` | When existing rows load: `auto` snapshots when no resumable position exists and resumes without a snapshot when one does; `disabled` streams changes only; `always` re-snapshots on every start, discarding any persisted position. | +| `mysql_replication_initial_snapshot` | `auto` | `auto` resumes a saved position or snapshots if none exists; `disabled` initially streams changes only; `always` discards the saved position on every start. Existing accelerations may still require a [rebuild](#when-the-position-is-purged). | | `mysql_replication_checkpoint_interval` | `10s` | How often the committed position persists to the sidecar. Bounds crash-replay volume. | | `mysql_replication_bootstrap_batch_size` | `8192` | Rows per emitted snapshot batch. Maximum: `1048576`. | | `mysql_replication_invalid_checkpoint_behavior` | `error` | What to do when the persisted position cannot be resumed losslessly — it was purged from the source, the source's GTID history diverged from the checkpoint, or the source table's column layout drifted incompatibly with the recorded position: `error`, or `restart` (drop the position and re-snapshot). | @@ -140,7 +140,12 @@ params: mysql_replication_invalid_checkpoint_behavior: restart ``` -to instead drop the stale position, truncate the accelerator, and re-snapshot the table automatically. +to instead drop the stale position and rebuild the accelerated table from the source automatically. + +With `restart`, an existing acceleration is rebuilt atomically even when +`mysql_replication_initial_snapshot: disabled`. That setting controls the initial load and does +not preserve stale rows after a lost position. Setting `mysql_replication_initial_snapshot: always` +also rebuilds an existing acceleration by discarding its saved position. ## Sharing one binlog connection diff --git a/website/docs/features/cdc/postgres-replication.md b/website/docs/features/cdc/postgres-replication.md index 9d43b0e42..b2f5d3fc9 100644 --- a/website/docs/features/cdc/postgres-replication.md +++ b/website/docs/features/cdc/postgres-replication.md @@ -389,6 +389,7 @@ On each start the decision is arithmetic rather than an inference: | None, on a durable acceleration that can record one | any | **Rebuild** — a table that outlives the process may already hold rows this start did not load | | Present | Slot's `restart_lsn` is at or before the watermark | **Resume** — the WAL in between is still retained and is replayed | | Present | Slot's `restart_lsn` is past the watermark, or the slot is gone | **Rebuild** — the missing changes no longer exist on the source | +| Present, but the accelerated table is empty or unreadable | Slot can still stream from it | **Rebuild** — changes before the watermark will not be replayed | | Recorded against a different source | any | **Rebuild** — LSNs are only comparable within one source's history | A rebuild replaces the accelerated table's contents through the ordinary full-refresh write path, so it is atomic: on Cayenne, readers keep seeing the pre-rebuild table until the new snapshot swaps in. @@ -401,7 +402,14 @@ A snapshot bootstrap emits only insert events, and nothing clears a durable acce The first start after upgrading a durable `refresh_mode: changes` dataset to a version that records watermarks has no recorded position, so it rebuilds once and records one from then on. -A slot lost **while Spice is streaming** — dropped by an operator, or invalidated by PostgreSQL for exceeding `max_slot_wal_keep_size` or `idle_replication_slot_timeout` — is recovered on the same reconnect path, without a restart: the unusable slot is dropped and replaced, every acceleration on it is rebuilt from the source, and streaming continues on the replacement. +An empty table is rebuilt even with a valid watermark, for example after +[`mode: file_update`](../../reference/spicepod/datasets#accelerationmode) recreates it or its rows +are deleted. If the source is also empty, the rebuild reads no rows. + +If a slot is lost while streaming, Spice replaces it, rebuilds every acceleration sharing it, and +resumes streaming without a runtime restart. This also applies when a newly joining dataset +triggers replacement. Slots can be dropped manually or invalidated by PostgreSQL's +`max_slot_wal_keep_size` or `idle_replication_slot_timeout` limits. Replacement is rate-limited to **3 slots per hour per replication connection**. A process running for months may legitimately be invalidated a few times, each one a genuine recovery, but three inside an hour means the source is not retaining enough WAL to cover the dataset — and every replacement costs a full re-read of every table on the slot, so retrying indefinitely would turn a retention limit into sustained load on the source. Past the budget the dataset surfaces a terminal error instead: raise `max_slot_wal_keep_size` on the source, or reduce replication lag, then reload the dataset. @@ -441,6 +449,10 @@ Shared-slot delivery and coalescing (auto-registered; reported only for datasets | `dataset_postgres_replication_member_envelope_mailbox_merges_total` | Counter | Committed transactions folded into an envelope already sitting unclaimed in this dataset's buffer (stage 2). Rising alongside a flat `dataset_postgres_replication_member_send_stalled_seconds_total` means back-pressure is being absorbed rather than stalling the slot. | | `dataset_postgres_replication_member_mailbox_coalesce_limited_total` | Counter | Times a committed transaction could not be folded into the unclaimed buffer tail because a configured bound refused it, rather than because the changes were not foldable. `0` means the bounds never bind. | +[Rebuilds](#recovering-from-a-lost-replication-slot) appear in +`dataset_acceleration_refresh_duration_ms{mode="full"}`. For `refresh_mode: changes` datasets, +each full-refresh event is a rebuild; the preceding log gives its reason. + ## Troubleshooting | Symptom | Cause and fix | diff --git a/website/docs/features/data-acceleration/partitioning.md b/website/docs/features/data-acceleration/partitioning.md index 899490610..b8293c30c 100644 --- a/website/docs/features/data-acceleration/partitioning.md +++ b/website/docs/features/data-acceleration/partitioning.md @@ -196,6 +196,13 @@ Each partition is a separate Cayenne (Vortex) table; the partition catalog is tr Requires `mode: file`. +Partitioned Cayenne tables have these limitations: + +- **No in-place schema evolution.** Schema changes use the recreate fallback. Configuring + [`on_schema_change`](../../reference/spicepod/datasets#on_schema_change) to evolve logs a warning; + CDC batches with widened schemas are rejected. +- **No snapshot publication.** [Acceleration snapshots](snapshots) do not support partitioned tables. + ## Changing `partition_by` after refresh Once partitions exist on disk, changing `partition_by` is rejected: diff --git a/website/docs/features/data-acceleration/snapshots.md b/website/docs/features/data-acceleration/snapshots.md index cb4d568cf..688544ac3 100644 --- a/website/docs/features/data-acceleration/snapshots.md +++ b/website/docs/features/data-acceleration/snapshots.md @@ -305,5 +305,6 @@ For the production deployment pattern that uses snapshots to separate ingest fro :::warning[Limitations] - Only datasets are supported for snapshots. Views are not supported. +- **Partitioned Cayenne** accelerations (`engine: cayenne` with `partition_by`) skip periodic and pre-recreate snapshots, with a warning naming the dataset. ::: diff --git a/website/docs/features/data-ingestion/index.md b/website/docs/features/data-ingestion/index.md index b60f21191..0765f2132 100644 --- a/website/docs/features/data-ingestion/index.md +++ b/website/docs/features/data-ingestion/index.md @@ -143,7 +143,11 @@ A metric whose existing table has more than one column of the same name (Arrow p Histograms carry the same `time_unix_nano` and `start_time_unix_nano` columns as number data points. -Data point attributes become additional columns named after the attribute key, typed by the attribute value (`Utf8`, `Boolean`, `Int64`, `Float64`, or `Binary`). When a metric starts reporting a new attribute, Spice evolves an accelerated table's schema in place before writing, subject to the dataset's [`on_schema_change`](../../reference/spicepod/datasets.md#on_schema_change) policy. +Data point and resource attributes, such as `service.name` and `service.instance.id`, become columns +named after the attribute key. Values determine the type: `Utf8`, `Boolean`, `Int64`, `Float64`, or +`Binary`. Data point values take precedence when keys overlap. New attributes trigger in-place +schema evolution before writing, subject to the dataset's +[`on_schema_change`](../../reference/spicepod/datasets.md#on_schema_change) policy. Because OTLP timestamps are nanoseconds, set [`time_format: unix_nanos`](../../reference/spicepod/datasets.md#time_format) when using `time_unix_nano` as a dataset's `time_column`: diff --git a/website/docs/features/functions/index.md b/website/docs/features/functions/index.md index 1b372dea6..5469de1f6 100644 --- a/website/docs/features/functions/index.md +++ b/website/docs/features/functions/index.md @@ -305,6 +305,16 @@ functions: body: 'upper(s)' ``` +## Functions and query federation + +User-defined functions execute in Spice. Filters using them, such as `WHERE shout(name) = 'ACME'`, +can require more data from the source than equivalent filters written in the source's SQL. +Source-side filtering in [`refresh_sql`](../data-acceleration/data-refresh.md) or a source view can +reduce that transfer. + +Functions added after startup or through hot reload follow the same rule. Changes to the function +registry invalidate cached plans. + ## Types Argument and return types use Arrow logical types. Both Spicepod aliases and Arrow display forms are accepted. diff --git a/website/docs/features/observability/index.md b/website/docs/features/observability/index.md index 63fc1d78d..16f9f3bae 100644 --- a/website/docs/features/observability/index.md +++ b/website/docs/features/observability/index.md @@ -284,6 +284,7 @@ Component-specific instruments are documented with their component. The `cayenne | `dataset_acceleration_snapshot_write_checksum`

Checksum of the most recent snapshot write (emitted with `checksum` attribute). | _gauge_ | `dataset`, `checksum` | | `dataset_acceleration_snapshot_write_duration_ms`

Time in milliseconds taken to write the latest snapshot to object storage. | _histogram_ | `dataset` | | `dataset_acceleration_snapshot_write_timestamp`

Unix timestamp (seconds) when the most recent snapshot write completed. | _gauge_ | `dataset` | +| `dataset_acceleration_write_back_pending_keys`

Committed keys awaiting [durable write-back](../components/data-accelerators/cayenne/#transactions). Updated on every delivery pass, including failures; resets to `0` when the dataset unloads. A persistent backlog indicates a delivery problem. | _gauge_ | `dataset` | | `dataset_active_count`

Number of currently loaded datasets. | _gauge_ | `engine` | | `dataset_load_errors`

Number of errors loading the dataset. | _count_ | — | | `dataset_load_state`

Status of the dataset. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown. | _gauge_ | `dataset` | diff --git a/website/docs/reference/datatypes/accelerators.md b/website/docs/reference/datatypes/accelerators.md index 71854e98e..eec73cf0b 100644 --- a/website/docs/reference/datatypes/accelerators.md +++ b/website/docs/reference/datatypes/accelerators.md @@ -38,7 +38,7 @@ Spice adheres to Apache Arrow data [types](https://docs.rs/arrow/latest/arrow/da | sparse_union | Sparse unions of logical types. | | | | | | dense_union | Dense unions of logical types. | | | | | | dictionary | Dictionary-encoded type, | | | | | -| map | Map, a repeated struct logical type. | | | | | +| map | Map, a repeated struct logical type. | `List(Struct)` | | | | | extension | Custom data type, implemented by user. | `Extension` | | | | | fixed_size_list | Fixed size list of some logical type. | `FixedSizeList` | `TYPE[N]` | | | | duration | Elapsed time in seconds, milliseconds, microseconds or nanoseconds. | | | | | @@ -52,4 +52,11 @@ Spice adheres to Apache Arrow data [types](https://docs.rs/arrow/latest/arrow/da | list_view | A list of some logical data type represented by offset and size. | | | | | | large_list_view | Like LIST_VIEW, but with 64-bit offsets and sizes. | | | | | -Note: Where `TYPE` is used (e.g. `TYPE[]`), it refers an established supported type for the specific data accelerator (e.g. `INTEGER[]`). Spice Cayenne (Vortex) provides zero-copy compatibility with Apache Arrow and supports most Arrow types through its logical type system. A `half_float` (Arrow `Float16`) column is widened to `Float32` before it is written, so it is stored — and read back — as `Primitive(F32)`; Vortex has no 2-byte float type. For DuckDB, the accelerator hands DuckDB the Arrow schema and uses the `CREATE TABLE` statement DuckDB derives from it, so unsigned Arrow integers keep their unsignedness (`uint32` becomes `UINTEGER`, not `INTEGER`) and `decimal128` keeps its precision and scale. `TYPE[N]` denotes a fixed-length list (e.g. `INTEGER[4]`), and a timezone-aware `timestamp` becomes `TIMESTAMP WITH TIME ZONE`. +`TYPE` denotes a supported accelerator type, such as `INTEGER` in `INTEGER[]`. `TYPE[N]` denotes +a fixed-length list, such as `INTEGER[4]`. + +- **Cayenne:** `Float16` is stored and read as `Primitive(F32)`. A `map` is stored as + `List>` and read back as a `map`. +- **DuckDB:** Unsigned integers retain their unsignedness (`uint32` maps to `UINTEGER`), + `decimal128` retains its precision and scale, and timezone-aware timestamps map to + `TIMESTAMP WITH TIME ZONE`. diff --git a/website/docs/reference/spicepod/datasets.md b/website/docs/reference/spicepod/datasets.md index f666004d0..41630b86e 100644 --- a/website/docs/reference/spicepod/datasets.md +++ b/website/docs/reference/spicepod/datasets.md @@ -259,7 +259,15 @@ datasets: :::note -In-place evolution (no restart) is supported for the `duckdb`, `sqlite`, `turso`, and Spice Cayenne (`cayenne`) acceleration engines, including for PostgreSQL CDC (`refresh_mode: changes`). Other engines (for example `arrow` and the PostgreSQL accelerator) log a clear unsupported message and degrade safely, applying additive changes on restart. Constraint and primary-key columns cannot be widened in place. For destructive schema changes (column removals or narrowing), set `on_schema_change: drop_and_recreate` with `refresh_mode: full` to drop and recreate the accelerated table from the source, or use [`mode: file_update`](#accelerationmode), which recreates the acceleration file on any change. +In-place evolution is supported by `duckdb`, `sqlite`, `turso`, and non-partitioned Spice Cayenne +(`cayenne`), including PostgreSQL CDC (`refresh_mode: changes`). Partitioned Cayenne tables log a +warning and use the recreate fallback. Other engines, including `arrow` and PostgreSQL, log an +unsupported message and apply additive changes on restart. Constraint and primary-key columns +cannot be widened in place. + +For column removals or narrowing, `on_schema_change: drop_and_recreate` with `refresh_mode: full` +recreates the table from the source. [`mode: file_update`](#accelerationmode) recreates the +acceleration file on any schema change. ::: @@ -571,7 +579,7 @@ Optional. Controls how writes to a `read_write` accelerated dataset propagate be Supported values: - `write_through` (default) – Writes are sent to the federated source synchronously. The client receives an ACK only after the source commits the change, providing ACID guarantees. The local accelerator is updated through the configured refresh path (for example, the WAL stream when `refresh_mode: changes`). -- `write_back` – Writes are applied to the local accelerator first (fast ACK), then forwarded asynchronously to the federated source. Choose this for write throughput when eventual consistency at the source is acceptable. +- `write_back` – Writes commit to the local accelerator before asynchronous delivery to the source. This provides eventual consistency at the source. [Durable write-back](../../components/data-accelerators/cayenne/#transactions) requires Cayenne over PostgreSQL, a single-column `primary_key`, `mode: file`, and no acceleration retention. Writes must be transactional; `DELETE` is unsupported. ## `acceleration.refresh_check_interval` diff --git a/website/docs/reference/spicepod/embeddings.md b/website/docs/reference/spicepod/embeddings.md index e8d8ff27a..d1dda726a 100644 --- a/website/docs/reference/spicepod/embeddings.md +++ b/website/docs/reference/spicepod/embeddings.md @@ -32,7 +32,7 @@ The `from` field specifies the source of the embedding model. It supports the fo - `azure` - Azure OpenAI models - `databricks` - Databricks-hosted models - `bedrock` - Amazon Bedrock models -- `google` - Google AI models +- `google` - Google Vertex AI models - `model2vec` - Model2Vec static embedding models Follows the same convention as [`models.from`](./models#from). diff --git a/website/docs/reference/spicepod/runtime.md b/website/docs/reference/spicepod/runtime.md index f1998499e..b736ed97a 100644 --- a/website/docs/reference/spicepod/runtime.md +++ b/website/docs/reference/spicepod/runtime.md @@ -559,12 +559,13 @@ These two lines are the primary diagnostic: between them they name what the runt The derived line reports **defaults**, not necessarily the values in force: several are overridable by their own setting (`runtime.query.target_partitions`, `runtime.query.max_concurrent_queries`, DuckDB's `threads`, a model's parallelism), and the line is logged before that configuration is resolved. Each overridable consumer separately logs the value it used and where that value came from. -Three warnings cover the cases the summary cannot state on its own. Each names a cause and an action; none of them fires for a deployment that is merely sized small on purpose, which the summary already records. +The following warnings identify CPU configuration issues and remedies: | Warning | Fires when | | ---------------------------------------------- | ------------------------------------------------------------------------------------------------- | | CPU request present but not passed through | Running under Kubernetes with a cgroup share but no `SPICE_CPU_REQUEST_MILLICORES` — the deployment surface is not emitting the block above, so sizing fell through to the machine. | | Declared request implausibly small | A declared request below 10 millicores, which is what a `resourceFieldRef` missing its `divisor: 1m` produces for a request of one to nine cores. | +| Configured cores above the real ceiling | The explicit CPU entitlement exceeds the cgroup or affinity limit. The configured value remains unchanged; the warning identifies the limit so the entitlement or limit can be adjusted. | | CPU share changed after startup | The cgroup share moved from its value at startup — the pod was resized in place. The entitlement cannot change without a restart, so this reports the drift rather than acting on it. | The `spiced_cpu_budget_cores`, `spiced_cpu_budget_millicores`, `spiced_cpu_limit_millicores`, and `spiced_cpu_request_millicores` gauges report the same figures — see [Observability](../../features/observability). The `source` label on `spiced_cpu_budget_cores` is the authority on which rung won, which is what makes a fleet greppable for pods that resolved somewhere unexpected. `tokio_runtime_workers` is the cross-check on the thread pools the entitlement sized. diff --git a/website/docs/reference/sql/json.md b/website/docs/reference/sql/json.md index 206d2c65b..ad9fa9b4e 100644 --- a/website/docs/reference/sql/json.md +++ b/website/docs/reference/sql/json.md @@ -696,7 +696,8 @@ arrive in Spice as a string. The three rows above that are conditional are the o translation depends on which it is, so they read the remote type the driver reports for the column (the Arrow `arrow.json` extension name, or the driver's own `BIGQUERY:type`). An expression the source did not type — a computed string, or a literal — establishes neither, and the call stays -local. +local. `json_as_text` and `json_get(...) IS NULL` also support `COALESCE` of columns with the same +declared type. `json_contains` requires a declared column. - **`json_contains`** needs the declared type in order to pick between two BigQuery expressions that are *not* interchangeable: `JSON_QUERY(doc, '') IS NOT NULL` on a native `JSON` @@ -707,9 +708,9 @@ local. native `JSON` column, BigQuery need not retain a numeric token's input spelling — `1.50` and `1e+00` do not survive — so the local answer and the remote one would differ. -Two values cannot be extracted from a `STRING` document without changing the result, and rather -than return a different answer the remote query fails with an error telling you to set -[`query_federation: disabled`](../../components/data-connectors/adbc) on the dataset: +The following extractions from `STRING` documents fail because BigQuery changes the result. +These queries require [`query_federation: disabled`](../../components/data-connectors/adbc) on the dataset or its +[ADBC catalog](../../components/catalogs/adbc): - **A container.** `json_as_text` returns the matched array or object's original serialization, and BigQuery's `JSON_QUERY` re-spaces it. diff --git a/website/docs/reference/sql/subqueries.md b/website/docs/reference/sql/subqueries.md index 274da72a2..5af6e272a 100644 --- a/website/docs/reference/sql/subqueries.md +++ b/website/docs/reference/sql/subqueries.md @@ -338,6 +338,16 @@ HAVING +-------+--------+ ``` +## Federation of a correlated `EXISTS` + +A correlated `[NOT] EXISTS` can be pushed down to a federated source. Spice rejects queries when: + +- The subquery reuses the outer reference's only qualifier, making the reference ambiguous. +- The correlation spans multiple build-side inputs that cannot share one scope name. + +The error identifies the correlation. Distinct relation aliases allow the outer reference to +remain unambiguous inside the subquery. + ## Subquery categories Subqueries can be categorized as one or more of the following based on the