diff --git a/docs/deploy/configuration/backups.md b/docs/deploy/configuration/backups.md index de8ac6b1..bbb001fc 100644 --- a/docs/deploy/configuration/backups.md +++ b/docs/deploy/configuration/backups.md @@ -316,7 +316,7 @@ The `*` character matches any sequence of characters. For example, `Article*` ma | name | type | required | default | description | | ---- | ---- | ---- | ---- |---- | | `CPUPercentage` | number | no | `50%` | An optional integer to set the desired CPU core utilization ranging from 1%-80%. | -| `ChunkSize` | number | no | `128MB` | An optional integer represents the desired size for chunks. Weaviate will attempt to come close the specified size, with a minimum of 2MB, default of 128MB, and a maximum of 512MB.| +| `ChunkSize` | number | no | - | **Deprecated. This option has no effect.** Weaviate ignores any value sent here, so it neither sets nor caps the chunk size. Chunk sizing is now controlled by the [`BACKUP_CHUNK_TARGET_SIZE`](#chunking-and-file-splitting) environment variable, which replaced it. | | `CompressionLevel`| string | no | `DefaultCompression` | An optional [compression level](#compression-levels) to be used. | | `Path` | string | no | `""` | An optional string to manually set the backup location. If not provided, the backup will be stored in the default location. Introduced in Weaviate `v1.27.2`. | | `incremental_base_backup_id` | string | no | `None` | The ID of a previous backup to use as the base for an [incremental backup](#incremental-backups). Files unchanged since the base backup are stored as references rather than copied. Introduced in Weaviate `v1.37`. | @@ -466,7 +466,9 @@ This can result in dramatically smaller backups and much faster backup times. #### How it works -When creating a backup, Weaviate splits large files into individual chunks. During an incremental backup, Weaviate compares each file against the base backup. Files that haven't changed are stored as pointers to the base backup rather than being copied again. On restore, Weaviate automatically fetches the referenced files from the base backup. +When creating a backup, Weaviate packs a shard's files into chunks. During an incremental backup, Weaviate compares each file against the base backup. Files that haven't changed are stored as pointers to the base backup rather than being copied again. On restore, Weaviate automatically fetches the referenced files from the base backup. + +Only a file large enough to get a chunk of its own can be referenced individually, so the way Weaviate groups files into chunks determines how much an incremental backup can reuse. For how Weaviate decides which files get their own chunk, and the environment variables that control chunking, see [Chunking and file splitting](#chunking-and-file-splitting). #### Create a full (base) backup @@ -778,6 +780,39 @@ The backup API is built in a way that no long-running network requests are requi If you would like your application to wait for the background backup process to complete, you can use the "wait for completion" feature that is present in all language clients. The clients will poll the status endpoint in the background and block until the status is either `SUCCESS` or `FAILED`. This makes it easy to write simple synchronous backup scripts, even with the async nature of the API. +### Chunking and file splitting + +Weaviate does not upload a [shard's](/weaviate/concepts/storage.md#logical-storage-units-indexes-shards-stores) files one by one. It packs them into chunks, and the way files are grouped into chunks determines how much of that backup a later [incremental backup](#incremental-backups) can reuse. Chunking runs on every backup, not only on incremental ones. + +Weaviate gives each of a shard's biggest files a chunk of its own, and only a file that gets its own chunk can be referenced individually by a later incremental backup. A chunk of its own is a requirement rather than a guarantee, because Weaviate only reuses files that it treats as immutable, so a large file that it keeps rewriting is uploaded again with every backup. All remaining files are packed together into shared chunks, and a shared chunk is uploaded again on every incremental backup whether or not anything in it changed, because only a chunk that holds a single big file can be skipped. A file larger than `BACKUP_SPLIT_FILE_SIZE` is spread across several chunks instead of being written into one. + +A file gets its own chunk once it reaches the qualifying size: the larger of `BACKUP_MIN_CHUNK_SIZE` and the size of the Nth largest file in the shard, where N is `BACKUP_MAX_INDIVIDUAL_FILES`. + +Because the larger of the two wins, `BACKUP_MIN_CHUNK_SIZE` is a floor on the qualifying size rather than a dial for it. Lowering it never reduces how many files get their own chunk, and it only increases that number while the shard holds fewer than `BACKUP_MAX_INDIVIDUAL_FILES` files above its current value. At the defaults, a shard with 100 or more files of at least `1MiB` is unaffected by a lower `BACKUP_MIN_CHUNK_SIZE`, because the size of the 100th largest file wins the comparison. To have more files referenced individually in that case, raise `BACKUP_MAX_INDIVIDUAL_FILES` instead, at the cost of more chunks per backup. + +On an incremental backup, the `BACKUP_MAX_INDIVIDUAL_FILES` budget is reduced by the number of files that are reused from the base backup, so it is shared across a whole backup chain rather than being renewed for each backup in it. + +A chunk that carries part of a split file holds nothing else, because Weaviate deliberately leaves the rest of that chunk empty rather than mixing split parts with regular files. The size of such a chunk is governed by `BACKUP_SPLIT_FILE_SIZE` rather than by `BACKUP_CHUNK_TARGET_SIZE`: Weaviate divides the file into roughly equal parts that are each at least half of the split size and no larger than the split size, so at the defaults these chunks are far larger than the target, not smaller. + +The three size variables below accept a plain number of bytes or a number with a unit suffix (`B`, `KB`, `MB`, `GB`, `TB`, `KiB`, `MiB`, `GiB`, `TiB`), for example `4MiB`. The decimal and binary suffixes are distinct: `MB` is 1,000,000 bytes while `MiB` is 1,048,576 bytes. They also accept `unlimited` or `nolimit`, which is how you disable file splitting through `BACKUP_SPLIT_FILE_SIZE`. All three are read at startup, so a restart is required to change them. `BACKUP_MAX_INDIVIDUAL_FILES` is a plain count instead of a size, and Weaviate reads it for each backup, so you can change it without a restart through the `backup_max_individual_files` [runtime configuration](./env-vars/runtime-config.md) key. + +| Environment variable | Required | Description | +| --- | --- | --- | +| `BACKUP_MIN_CHUNK_SIZE` | no | The floor on the qualifying size a file must reach before it is stored in its own chunk and becomes eligible to be referenced individually by a later incremental backup. Defaults to `1MiB`. | +| `BACKUP_CHUNK_TARGET_SIZE` | no | The size Weaviate aims for when packing several smaller files into a single chunk. Defaults to `10MiB`. | +| `BACKUP_SPLIT_FILE_SIZE` | no | The size above which a single file is split across multiple chunks instead of being written to one. Set it to `unlimited` to disable splitting. Defaults to `50GiB`. | +| `BACKUP_MAX_INDIVIDUAL_FILES` | no | How many of a shard's biggest files Weaviate aims to give a chunk of their own, and therefore how many become eligible to be referenced individually by a later incremental backup. This is a count rather than a size, and it must be greater than `0`. Defaults to `100`. Also settable without a restart through the `backup_max_individual_files` [runtime configuration](./env-vars/runtime-config.md) key.

Added in `v1.37.14` and `v1.38.7`. | + +:::note How Weaviate adjusts these values + +These settings are lower bounds, not exact values, and Weaviate raises them when a smaller value would have no useful effect: + +- `BACKUP_CHUNK_TARGET_SIZE` is raised to the qualifying size if you set it lower, since a chunk that packs smaller files must still be able to hold one big file. +- `BACKUP_SPLIT_FILE_SIZE` is likewise raised to that size if you set it lower; otherwise no file would ever be big enough to split. +- If a shard holds fewer files than `BACKUP_MAX_INDIVIDUAL_FILES`, or than the reduced budget on an incremental backup, the qualifying size falls back to the size of the shard's smallest file, still raised to `BACKUP_MIN_CHUNK_SIZE` if that is larger. + +::: + ### Skip the storage access check When a cloud backup backend (`backup-s3`, `backup-gcs`, or `backup-azure`) initializes, Weaviate verifies that the configured credentials can write to and delete from the target bucket. It does this by writing a temporary `access-check` object and then removing it. This probe fails on immutable (write-once / WORM) buckets, or with least-privilege credentials that are not permitted to delete objects. diff --git a/docs/deploy/configuration/env-vars/runtime-config.md b/docs/deploy/configuration/env-vars/runtime-config.md index 137ead48..a44ea2c8 100644 --- a/docs/deploy/configuration/env-vars/runtime-config.md +++ b/docs/deploy/configuration/env-vars/runtime-config.md @@ -61,6 +61,7 @@ The following overrides are currently supported: | `async_replication_hashtree_init_concurrency` | `ASYNC_REPLICATION_HASHTREE_INIT_CONCURRENCY`| | `async_replication_cluster_max_workers` _(removed in `v1.38`)_ | `ASYNC_REPLICATION_CLUSTER_MAX_WORKERS` _(removed in `v1.38`)_ | | `autoschema_enabled` | `AUTOSCHEMA_ENABLED` | +| `backup_max_individual_files` | `BACKUP_MAX_INDIVIDUAL_FILES` | | `debug_endpoints_enabled` | `DEBUG_ENDPOINTS_ENABLED` | | `default_quantization` | `DEFAULT_QUANTIZATION` | | `default_sharding_count` | `DEFAULT_SHARDING_COUNT` | diff --git a/docs/weaviate/model-providers/google/embeddings.md b/docs/weaviate/model-providers/google/embeddings.md index 802b5d0d..be5cb647 100644 --- a/docs/weaviate/model-providers/google/embeddings.md +++ b/docs/weaviate/model-providers/google/embeddings.md @@ -234,9 +234,12 @@ The following examples show how to configure Google-specific options. **Vertex AI parameters:** - `projectId` (Required): Your Google Cloud project ID, e.g. `cloud-large-language-models` +- `location` (Optional): The Google Cloud region to send requests to, e.g. `europe-west1`. - `apiEndpoint` (Optional): Regional endpoint, e.g. `us-central1-aiplatform.googleapis.com` - `modelId` (Optional): e.g. `gemini-embedding-001`, `text-embedding-005` +Set `location` together with a matching `apiEndpoint` to keep data in a specific region. +