diff --git a/kits/README.md b/kits/README.md index e9fa85ffec..48639d426e 100644 --- a/kits/README.md +++ b/kits/README.md @@ -36,7 +36,10 @@ functions it exports. ## Using a kit -Kits need Firebase CLI 15.25.1 or later with the `kits` experiment enabled: +Kits need Firebase CLI 15.27.0 or later with the `kits` experiment enabled. +The CLI provides each instance its id as the `FIREBASE_KIT_INSTANCE_ID` +environment variable; earlier versions do not set it, and kits that use the +instance id will not deploy or run without it. ```sh firebase experiments:enable kits diff --git a/kits/bigquery-firestore-export/CHANGELOG.md b/kits/bigquery-firestore-export/CHANGELOG.md index cab8581a02..d026950fa7 100644 --- a/kits/bigquery-firestore-export/CHANGELOG.md +++ b/kits/bigquery-firestore-export/CHANGELOG.md @@ -1,2 +1,3 @@ - Initial release of kit, see README for differences between the legacy extension and this kit +- The instance id now comes from `FIREBASE_KIT_INSTANCE_ID`, which the Firebase CLI (15.27.0 or later) provides to each kit instance; `INSTANCE_ID` is no longer a configuration parameter - A BigQuery `TIME` column no longer crashes a run: its value is written to Firestore as the string BigQuery returned (`"10:30:00"`) rather than passed to `Timestamp.fromDate`, which threw and lost the whole run. This is a deliberate divergence from the legacy extension, which throws on the same line diff --git a/kits/bigquery-firestore-export/README.md b/kits/bigquery-firestore-export/README.md index 6a0bfd9e80..0d3b655880 100644 --- a/kits/bigquery-firestore-export/README.md +++ b/kits/bigquery-firestore-export/README.md @@ -49,7 +49,6 @@ export { and configure them with a `.env` (or `.env.`): ```sh -INSTANCE_ID=analytics-export BIGQUERY_DATASET_LOCATION=US DATASET_ID=analytics TABLE_NAME=users @@ -67,8 +66,9 @@ only deploys what your entry file exports. ## Deploy -The package's `firebase.json` declares a `kit` stanza (Firebase CLI 15.25.1 or -later, behind the `kits` experiment): +The package's `firebase.json` declares a `kit` stanza (Firebase CLI 15.27.0 or +later, behind the `kits` experiment - earlier CLIs do not provide the +`FIREBASE_KIT_INSTANCE_ID` variable this kit reads its instance id from): ```json { @@ -103,9 +103,13 @@ Set these values in a `.env` (or `.env.`) file. The Firebase CLI loads them at deploy time and prompts for required values that are missing. `PROJECT_ID` is supplied by the Firebase CLI. +The instance id is not a setting: the CLI provides it to each instance as +`FIREBASE_KIT_INSTANCE_ID`, set to that instance's key in the `instances` map. +`FIREBASE_` is a reserved prefix in `.env` files, so it cannot be set or +overridden there. + | Field | Env var | Required | Default | Description | | ------------------------- | --------------------------- | -------- | ----------------- | ------------------------------------------------------------ | -| `instanceId` | `INSTANCE_ID` | yes | — | Must match this instance's key in the `instances` map | | `bigqueryDatasetLocation` | `BIGQUERY_DATASET_LOCATION` | no | `US` | BigQuery destination dataset location | | `transferConfigName` | `TRANSFER_CONFIG_NAME` | no | (empty) | Existing DTS config resource to link instead of creating one | | `datasetId` | `DATASET_ID` | yes | — | BigQuery destination dataset id | @@ -114,7 +118,7 @@ loads them at deploy time and prompts for required values that are missing. | `displayName` | `DISPLAY_NAME` | yes | — | Human-readable scheduled-query name | | `partitioningField` | `PARTITIONING_FIELD` | no | (empty) | Destination-table partitioning field | | `schedule` | `SCHEDULE` | yes | — | DTS schedule, such as `every 24 hours` | -| `pubSubTopic` | `PUB_SUB_TOPIC` | no | `kit--processMessages` | Pub/Sub topic ID, not a full resource name, receiving DTS completion notifications | +| `pubSubTopic` | `PUB_SUB_TOPIC` | no | `kit--processMessages` | Pub/Sub topic ID, not a full resource name, receiving DTS completion notifications | | `firestoreCollection` | `COLLECTION_PATH` | no | `transferConfigs` | Root Firestore collection for configs and output | | `logLevel` | `LOG_LEVEL` | no | `info` | `debug`, `info`, `warn`, `error`, or `silent` | @@ -140,10 +144,10 @@ To run several reverse-sync instances, add one entry per instance to the Instance ids must be unique across all kit stanzas in the project, and every instance's function names are namespaced by its `kit--` prefix, so -the instances cannot collide. Set `INSTANCE_ID` in each config directory to the -same value as that directory's key in the `instances` map; it also namespaces -the Pub/Sub notification topic and associates the deployment with its transfer -config. +the instances cannot collide. Each instance learns its own id from the +`FIREBASE_KIT_INSTANCE_ID` variable the CLI provides; there is nothing to keep +in sync by hand. The id also namespaces the Pub/Sub notification topic and +associates the deployment with its transfer config. ## Provisioning @@ -185,18 +189,19 @@ variable name and default, so a `.env` copied from your installed instance needs no value changes. What changes is the instance id, the Pub/Sub topic, the identity the scheduled query runs as, and how repeated BigQuery columns land in Firestore. -### You set `INSTANCE_ID` yourself, and the Pub/Sub topic is renamed +### The instance id comes from `firebase.json`, and the Pub/Sub topic is renamed The extension derived an instance id at install and used it to name its notification topic (`ext--processMessages`) and to tag its transfer -config document with `extInstanceId`. Here `INSTANCE_ID` is a setting you -provide, and it must match this instance's key in the `instances` map in -`firebase.json`. - -The topic defaults to `kit--processMessages`, and the kit creates it -on first run if it does not already exist. Set `INSTANCE_ID` to your installed -instance's id if you want the kit to adopt the scheduled query that instance -created, because the lookup is by `extInstanceId` on the documents in +config document with `extInstanceId`. Here the CLI derives it from this +instance's key in the `instances` map in `firebase.json` and provides it to the +functions as `FIREBASE_KIT_INSTANCE_ID`. There is no `INSTANCE_ID` setting to +configure. + +The topic defaults to `kit--processMessages`, and the kit creates it +on first run if it does not already exist. Use your installed instance's id as +the `instances` key if you want the kit to adopt the scheduled query that +instance created, because the lookup is by `extInstanceId` on the documents in `COLLECTION_PATH`. With a different id the kit finds nothing, creates a second scheduled query, and you end up with two writing into the same collection. @@ -209,7 +214,7 @@ there are two ways to migrate: left untouched: anything else subscribed to that topic, including an extension instance still installed, keeps receiving run notifications. - Take the default. The first deploy repoints the transfer config at - `kit--processMessages`, which stops notifications reaching the + `kit--processMessages`, which stops notifications reaching the extension and any other subscriber on the old topic. Do this once the extension is uninstalled, and the old topic can then be deleted. @@ -299,7 +304,7 @@ result query where your dataset lives. extension's 1st gen trigger did not retry. A run whose results fail to copy, for example because BigQuery or Firestore is briefly unavailable, is now retried rather than dropped. A notification that keeps failing, such as one for a transfer -config not tagged with this `INSTANCE_ID`, is also retried until Pub/Sub gives up. +config not tagged with this instance id, is also retried until Pub/Sub gives up. Both functions' service accounts need `roles/eventarc.eventReceiver` and `roles/run.invoker` on top of the three roles the extension asked for, and the diff --git a/kits/bigquery-firestore-export/src/config.ts b/kits/bigquery-firestore-export/src/config.ts index 10c785faa8..15dc4f432f 100644 --- a/kits/bigquery-firestore-export/src/config.ts +++ b/kits/bigquery-firestore-export/src/config.ts @@ -14,12 +14,7 @@ * limitations under the License. */ -import { - defineString, - expr, - projectID, - select, -} from "firebase-functions/params"; +import { defineString, projectID, select } from "firebase-functions/params"; import type { BigqueryFirestoreExportConfig, DeployTimeOptions, @@ -27,10 +22,29 @@ import type { } from "./export-config"; const LOG_LEVEL_OPTIONS = ["debug", "info", "warn", "error", "silent"] as const; -const instanceId = defineString("INSTANCE_ID"); + +// firebase-tools injects this for kit instances (set to the instance's key in +// firebase.json) during discovery, in the emulator, and on deployed functions. +// The FIREBASE_ prefix is reserved in .env files and the params machinery never +// sees injected values, so it must be a plain env read, not a defineString. +function instanceIdFromEnv(): string { + const instanceId = process.env.FIREBASE_KIT_INSTANCE_ID; + if (!instanceId) { + throw new Error( + "FIREBASE_KIT_INSTANCE_ID is not set. It is provided automatically to " + + "kit instances by firebase-tools >= 15.27.0; deploy or emulate this " + + "kit with a supported CLI version." + ); + } + return instanceId; +} + +// Resolved at import so the topic default is a concrete name at discovery. An +// unsupported CLI fails the discovery pass here rather than freezing +// "kit-undefined-processMessages" into the manifest. +const instanceId = instanceIdFromEnv(); const params = { - instanceId, bigqueryDatasetLocation: defineString("BIGQUERY_DATASET_LOCATION", { label: "BigQuery Dataset Location", description: @@ -81,7 +95,7 @@ const params = { description: "Which Pub/Sub topic should receive BigQuery Data Transfer completion notifications? Leave the default unless you are migrating from the bigquery-firestore-export extension, whose topic is named ext--processMessages. Pointing this at the extension's topic keeps the existing scheduled query's notification settings untouched.", - default: expr`kit-${instanceId}-processMessages`, + default: `kit-${instanceId}-processMessages`, input: { text: { nonEmpty: true, @@ -201,12 +215,10 @@ function normalizeLogLevel(value: string): LogLevel { /** Reads runtime values from Firebase deploy-time parameters. */ export function configFromEnv(): BigqueryFirestoreExportConfig { - const resolvedInstanceId = params.instanceId.value(); - return { bigqueryDatasetLocation: params.bigqueryDatasetLocation.value(), projectId: projectID.value(), - instanceId: resolvedInstanceId, + instanceId: instanceIdFromEnv(), transferConfigName: optional(params.transferConfigName.value()), datasetId: params.datasetId.value(), tableName: params.tableName.value(), diff --git a/kits/bigquery-firestore-export/tests/config.test.ts b/kits/bigquery-firestore-export/tests/config.test.ts index 1c3ad1ee40..702a70a4d4 100644 --- a/kits/bigquery-firestore-export/tests/config.test.ts +++ b/kits/bigquery-firestore-export/tests/config.test.ts @@ -14,32 +14,59 @@ * limitations under the License. */ -import { Expression } from "firebase-functions/params"; +import { declaredParams, Expression } from "firebase-functions/params"; import { afterEach, describe, expect, test, vi } from "vitest"; -import { CONFIG_EXPRESSIONS, configFromEnv } from "../src/config"; + +const INSTANCE_ID = "users-export"; + +// The instance id is read when the module loads, so the environment has to be +// in place before each import. +async function importConfig(instanceId: string | undefined) { + vi.resetModules(); + vi.stubEnv("FIREBASE_KIT_INSTANCE_ID", instanceId); + return import("../src/config"); +} + +function stubRuntimeEnv() { + vi.stubEnv("FIREBASE_CONFIG", JSON.stringify({ projectId: "test-project" })); + vi.stubEnv("BIGQUERY_DATASET_LOCATION", "EU"); + vi.stubEnv("DATASET_ID", "analytics"); + vi.stubEnv("TABLE_NAME", "users"); + vi.stubEnv("QUERY_STRING", "SELECT * FROM source.users"); + vi.stubEnv("DISPLAY_NAME", "Users export"); + vi.stubEnv("SCHEDULE", "every 24 hours"); + vi.stubEnv("COLLECTION_PATH", "transferConfigs"); + vi.stubEnv("LOG_LEVEL", "info"); +} afterEach(() => { vi.unstubAllEnvs(); }); describe("CONFIG_EXPRESSIONS", () => { - test("binds the trigger to the Pub/Sub topic parameter", () => { + test("binds the trigger to the Pub/Sub topic parameter", async () => { + const { CONFIG_EXPRESSIONS } = await importConfig(INSTANCE_ID); + expect(CONFIG_EXPRESSIONS.pubSubTopic).toBeInstanceOf(Expression); expect((CONFIG_EXPRESSIONS.pubSubTopic as Expression).toCEL()).toBe( "{{ params.PUB_SUB_TOPIC }}" ); }); - test("defaults the topic parameter to the instance-namespaced kit topic", () => { + test("defaults the topic parameter to the instance-namespaced kit topic", async () => { + const { CONFIG_EXPRESSIONS } = await importConfig(INSTANCE_ID); + const spec = ( CONFIG_EXPRESSIONS.pubSubTopic as unknown as { toSpec: () => { default?: string }; } ).toSpec(); - expect(spec.default).toBe("kit-{{ params.INSTANCE_ID }}-processMessages"); + expect(spec.default).toBe("kit-users-export-processMessages"); }); - test("accepts a topic ID but rejects a full resource name", () => { + test("accepts a topic ID but rejects a full resource name", async () => { + const { CONFIG_EXPRESSIONS } = await importConfig(INSTANCE_ID); + const spec = ( CONFIG_EXPRESSIONS.pubSubTopic as unknown as { toSpec: () => { input?: { text?: { validationRegex?: string } } }; @@ -60,22 +87,40 @@ describe("CONFIG_EXPRESSIONS", () => { }); }); -describe("configFromEnv", () => { - test("reads runtime parameters and derives the same topic", () => { - vi.stubEnv( - "FIREBASE_CONFIG", - JSON.stringify({ projectId: "test-project" }) +describe("instance id", () => { + // The CLI injects FIREBASE_KIT_INSTANCE_ID as a reserved env var; declaring + // it (or INSTANCE_ID) as a param makes the CLI prompt for a value it cannot + // accept and abort loading the kit. + test("is not declared as a param", async () => { + await importConfig(INSTANCE_ID); + + const declared = declaredParams.map((param) => param.name); + expect(declared).toContain("PUB_SUB_TOPIC"); + expect(declared).not.toContain("INSTANCE_ID"); + expect(declared).not.toContain("FIREBASE_KIT_INSTANCE_ID"); + }); + + test("fails discovery when FIREBASE_KIT_INSTANCE_ID is missing", async () => { + await expect(importConfig(undefined)).rejects.toThrow( + /FIREBASE_KIT_INSTANCE_ID is not set/ ); - vi.stubEnv("INSTANCE_ID", "users-export"); - vi.stubEnv("BIGQUERY_DATASET_LOCATION", "EU"); - vi.stubEnv("DATASET_ID", "analytics"); - vi.stubEnv("TABLE_NAME", "users"); - vi.stubEnv("QUERY_STRING", "SELECT * FROM source.users"); - vi.stubEnv("DISPLAY_NAME", "Users export"); - vi.stubEnv("SCHEDULE", "every 24 hours"); - vi.stubEnv("COLLECTION_PATH", "transferConfigs"); - vi.stubEnv("LOG_LEVEL", "info"); + }); + + test("throws at runtime when FIREBASE_KIT_INSTANCE_ID is missing", async () => { + const { configFromEnv } = await importConfig(INSTANCE_ID); + stubRuntimeEnv(); + vi.stubEnv("FIREBASE_KIT_INSTANCE_ID", undefined); + expect(() => configFromEnv()).toThrow( + /FIREBASE_KIT_INSTANCE_ID is not set/ + ); + }); +}); + +describe("configFromEnv", () => { + test("reads runtime parameters and derives the same topic", async () => { + const { configFromEnv } = await importConfig(INSTANCE_ID); + stubRuntimeEnv(); vi.stubEnv("PUB_SUB_TOPIC", "kit-users-export-processMessages"); expect(configFromEnv()).toMatchObject({ @@ -90,20 +135,9 @@ describe("configFromEnv", () => { }); }); - test("passes through a topic pointing at the extension's own topic", () => { - vi.stubEnv( - "FIREBASE_CONFIG", - JSON.stringify({ projectId: "test-project" }) - ); - vi.stubEnv("INSTANCE_ID", "users-export"); - vi.stubEnv("BIGQUERY_DATASET_LOCATION", "EU"); - vi.stubEnv("DATASET_ID", "analytics"); - vi.stubEnv("TABLE_NAME", "users"); - vi.stubEnv("QUERY_STRING", "SELECT * FROM source.users"); - vi.stubEnv("DISPLAY_NAME", "Users export"); - vi.stubEnv("SCHEDULE", "every 24 hours"); - vi.stubEnv("COLLECTION_PATH", "transferConfigs"); - vi.stubEnv("LOG_LEVEL", "info"); + test("passes through a topic pointing at the extension's own topic", async () => { + const { configFromEnv } = await importConfig(INSTANCE_ID); + stubRuntimeEnv(); vi.stubEnv("PUB_SUB_TOPIC", "ext-users-export-processMessages"); expect(configFromEnv().pubSubTopic).toBe( diff --git a/kits/firestore-incremental-capture/CHANGELOG.md b/kits/firestore-incremental-capture/CHANGELOG.md index f83331ddbc..0e88ae2174 100644 --- a/kits/firestore-incremental-capture/CHANGELOG.md +++ b/kits/firestore-incremental-capture/CHANGELOG.md @@ -1,3 +1,4 @@ +- The instance id now comes from `FIREBASE_KIT_INSTANCE_ID`, which the Firebase CLI (15.27.0 or later) provides to each kit instance; `INSTANCE_ID` is no longer a configuration parameter - Fixed task dispatch failing with "Queue does not exist": the kit prefixed queue names with `kit--` itself, which the Admin SDK then prefixed again from `FIREBASE_KIT_INSTANCE_ID`. Changelog rows never reached BigQuery and restorations never started - Full implementation, replacing the skeleton package: Firestore capture to a BigQuery changelog, Dataflow-based point-in-time restoration, and first-deploy provisioning. See the README for differences between the legacy extension and this kit. - The restoration pipeline is consumed as a pinned, digest-verified release of GoogleCloudPlatform/firebase-extensions (`firestore-incremental-capture-pipeline-v0.1.0`) downloaded by `scripts/setup.sh`; Maven is no longer required. diff --git a/kits/firestore-incremental-capture/README.md b/kits/firestore-incremental-capture/README.md index 6176a155d1..9f10086763 100644 --- a/kits/firestore-incremental-capture/README.md +++ b/kits/firestore-incremental-capture/README.md @@ -105,8 +105,9 @@ the config values to use below. PITR only covers writes made after it is enabled, so a restoration can only target a point in time after setup ran. -The package's `firebase.json` declares a `kit` stanza (Firebase CLI 15.25.1 or -later, behind the `kits` experiment): +The package's `firebase.json` declares a `kit` stanza (Firebase CLI 15.27.0 or +later, behind the `kits` experiment - earlier CLIs do not provide the +`FIREBASE_KIT_INSTANCE_ID` variable this kit reads its instance id from): ```json { @@ -149,9 +150,13 @@ Set these values in a `.env` (or `.env.`) file. The Firebase CLI loads them at deploy time and prompts for any required values that are missing. `PROJECT_ID` is supplied by the Firebase CLI. +The instance id is not a setting: the CLI provides it to each instance as +`FIREBASE_KIT_INSTANCE_ID`, set to that instance's key in the `instances` map. +`FIREBASE_` is a reserved prefix in `.env` files, so it cannot be set or +overridden there. + | Field | Env var | Required | Default | Description | | -------------------- | ---------------------- | -------- | ---------------- | ----------------------------------------------------------- | -| `instanceId` | `INSTANCE_ID` | yes | - | Must match this instance's key in the `instances` map | | `backupInstanceId` | `BACKUP_INSTANCE_ID` | yes | - | Firestore database to restore into; must not be `(default)` | | `syncCollectionPath` | `SYNC_COLLECTION_PATH` | no | `posts` | Collection to capture | | `datasetId` | `SYNC_DATASET` | no | `backup_dataset` | BigQuery dataset for the changelog | @@ -204,9 +209,10 @@ map, each pointing at its own config directory with its own `.env`: Instance ids must be unique across all kit stanzas in the project, and every instance's function names are namespaced by its `kit--` prefix, so -the instances cannot collide. Set `INSTANCE_ID` in each config directory to that -instance's key - it namespaces the Dataflow jobs and run-status documents, and a -mismatch makes two instances share them. +the instances cannot collide. Each instance learns its own id from the +`FIREBASE_KIT_INSTANCE_ID` variable the CLI provides; it namespaces the task +queues, the Dataflow jobs and the run-status documents, and there is nothing to +keep in sync by hand. Give each instance its own `SYNC_DATASET`/`SYNC_TABLE` or its own `BACKUP_INSTANCE_ID`. Two instances sharing a changelog table would replay each @@ -315,8 +321,10 @@ correctly. Everything around the format moved: `gs:///-dataflow-restore`, not the extension's `gs:////templates/myTemplate`, so an extension-staged template is not reused - run `scripts/setup.sh` before the first restoration. -- **Configuration.** `INSTANCE_ID` is new and required (the extension injected - `EXT_INSTANCE_ID` itself). `LOCATION` is free-form and mutable instead of an +- **Configuration.** The instance id comes from this instance's key in the + `instances` map in `firebase.json`, which the CLI provides to the functions + as `FIREBASE_KIT_INSTANCE_ID` (the extension injected `EXT_INSTANCE_ID` + itself). `LOCATION` is free-form and mutable instead of an immutable install-time select. `SYNC_COLLECTION_PATH` is optional with a default, and no longer advertises `{document=**}` whole-database capture - that pattern never produced a deployable trigger (see Configuration). diff --git a/kits/firestore-incremental-capture/scripts/setup.sh b/kits/firestore-incremental-capture/scripts/setup.sh index 55b8f1aaba..1e1dc9dcd5 100755 --- a/kits/firestore-incremental-capture/scripts/setup.sh +++ b/kits/firestore-incremental-capture/scripts/setup.sh @@ -35,9 +35,10 @@ # BUCKET_NAME Bucket holding the flex template. Defaults to the # project's default bucket. # INSTANCE_ID This instance's key in the `instances` map of the kit -# stanza, and the kit's INSTANCE_ID param. Must match -# both: it names the flex template object the deployed -# function launches. Default "default". +# stanza in firebase.json, which the Firebase CLI +# provides to the functions as FIREBASE_KIT_INSTANCE_ID. +# Must match: it names the flex template object the +# deployed function launches. Default "default". # WORKER_SERVICE_ACCOUNT Service account the Dataflow workers run as. # Defaults to the Compute Engine default service account. # @@ -288,10 +289,12 @@ main() { echo echo "Set these in .env before deploying:" echo " BACKUP_INSTANCE_ID=${BACKUP_INSTANCE_ID}" - echo " INSTANCE_ID=${INSTANCE_ID}" echo " LOCATION=${LOCATION}" echo " BUCKET_NAME=${bucket}" echo + echo "Deploy this instance under the key \"${INSTANCE_ID}\" in the kit stanza's" + echo "\"instances\" map in firebase.json; the CLI passes that key to the functions." + echo echo "The functions' own roles are granted by the Firebase CLI on first deploy." } diff --git a/kits/firestore-incremental-capture/src/capture-config.ts b/kits/firestore-incremental-capture/src/capture-config.ts index 86693e8d5f..bbfe76659b 100644 --- a/kits/firestore-incremental-capture/src/capture-config.ts +++ b/kits/firestore-incremental-capture/src/capture-config.ts @@ -66,8 +66,8 @@ export interface CaptureConfig { * This instance's key in the `instances` map of the kit stanza. Required: it * namespaces the flex template object, the Dataflow job names and the * run-status documents, which is what keeps two instances in one project from - * colliding. Task queue names come from `FIREBASE_KIT_INSTANCE_ID` instead, - * which the Admin SDK reads directly. + * colliding. The Admin SDK reads the same id from `FIREBASE_KIT_INSTANCE_ID` + * to prefix the task queues. */ instanceId: string; /** Defaults to `info`. */ @@ -131,8 +131,8 @@ export function resolveCaptureConfig( if (!config.instanceId) { invalid( - "INSTANCE_ID is required. It must match this instance's key in the " + - "`instances` map of the kit stanza in firebase.json." + "instanceId is required. The Firebase CLI (15.27.0 or later) provides " + + "it to each kit instance as FIREBASE_KIT_INSTANCE_ID." ); } diff --git a/kits/firestore-incremental-capture/src/config.ts b/kits/firestore-incremental-capture/src/config.ts index b3a4bcf5be..6c4daf2660 100644 --- a/kits/firestore-incremental-capture/src/config.ts +++ b/kits/firestore-incremental-capture/src/config.ts @@ -108,9 +108,6 @@ const params = { }), dataflowRegion: defineString("DATAFLOW_REGION", { default: "" }), bucketName: defineString("BUCKET_NAME", { default: "" }), - // No default: it has to match this instance's key in the `instances` map, and - // a wrong value silently misnames the task queues. - instanceId: defineString("INSTANCE_ID"), logLevel: defineString("LOG_LEVEL", { default: "info", input: select([...LOG_LEVEL_OPTIONS]), @@ -122,6 +119,22 @@ export const CONFIG_EXPRESSIONS: ConfigExpressions = { location: params.location, }; +// firebase-tools injects this for kit instances (set to the instance's key in +// firebase.json) during discovery, in the emulator, and on deployed functions. +// The FIREBASE_ prefix is reserved in .env files and the params machinery never +// sees injected values, so it must be a plain env read, not a defineString. +function instanceIdFromEnv(): string { + const instanceId = process.env.FIREBASE_KIT_INSTANCE_ID; + if (!instanceId) { + throw new Error( + "FIREBASE_KIT_INSTANCE_ID is not set. It is provided automatically to " + + "kit instances by firebase-tools >= 15.27.0; deploy or emulate this " + + "kit with a supported CLI version." + ); + } + return instanceId; +} + /** Coerce an empty-string param value to `undefined`. */ function optional(value: string): string | undefined { return value.length > 0 ? value : undefined; @@ -163,7 +176,7 @@ export function configFromEnv(defaultBucketName?: string): CaptureConfig { location: optional(params.location.value()), dataflowRegion: optional(params.dataflowRegion.value()), bucketName: optional(params.bucketName.value()) || defaultBucketName || "", - instanceId: params.instanceId.value(), + instanceId: instanceIdFromEnv(), logLevel: normalizeLogLevel(params.logLevel.value()), }; } diff --git a/kits/firestore-incremental-capture/tests/capture-config.test.ts b/kits/firestore-incremental-capture/tests/capture-config.test.ts index 7d0f357d2e..4096a674c9 100644 --- a/kits/firestore-incremental-capture/tests/capture-config.test.ts +++ b/kits/firestore-incremental-capture/tests/capture-config.test.ts @@ -44,9 +44,9 @@ describe("resolveCaptureConfig", () => { expect(resolved.logLevel).toBe("info"); }); - test("requires an instance id, which must match the instances map key", () => { + test("requires an instance id, which namespaces the Dataflow jobs, flex template and run-status documents", () => { expect(() => resolveCaptureConfig(config({ instanceId: "" }))).toThrow( - /INSTANCE_ID is required/ + /instanceId is required/ ); }); diff --git a/kits/firestore-incremental-capture/tests/config.test.ts b/kits/firestore-incremental-capture/tests/config.test.ts new file mode 100644 index 0000000000..a1e576380f --- /dev/null +++ b/kits/firestore-incremental-capture/tests/config.test.ts @@ -0,0 +1,58 @@ +/* + * Copyright 2026 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { declaredParams } from "firebase-functions/params"; +import { afterEach, beforeEach, describe, expect, test, vi } from "vitest"; +import { configFromEnv } from "../src/config"; + +beforeEach(() => { + vi.stubEnv("FIREBASE_CONFIG", JSON.stringify({ projectId: "test-project" })); + vi.stubEnv("BACKUP_INSTANCE_ID", "backup-db"); + vi.stubEnv("FIREBASE_KIT_INSTANCE_ID", "test-instance"); +}); + +afterEach(() => { + vi.unstubAllEnvs(); +}); + +describe("configFromEnv", () => { + // The CLI injects FIREBASE_KIT_INSTANCE_ID as a reserved env var; declaring + // it (or INSTANCE_ID) as a param makes the CLI prompt for a value it cannot + // accept and abort loading the kit. + test("does not declare an instance-id param", () => { + const declared = declaredParams.map((param) => param.name); + + expect(declared).toContain("BACKUP_INSTANCE_ID"); + expect(declared).not.toContain("INSTANCE_ID"); + expect(declared).not.toContain("FIREBASE_KIT_INSTANCE_ID"); + }); + + test("reads the instance id from the injected environment", () => { + expect(configFromEnv()).toMatchObject({ + projectId: "test-project", + instanceId: "test-instance", + backupInstanceId: "backup-db", + }); + }); + + test("throws when FIREBASE_KIT_INSTANCE_ID is missing", () => { + vi.stubEnv("FIREBASE_KIT_INSTANCE_ID", undefined); + + expect(() => configFromEnv()).toThrow( + /FIREBASE_KIT_INSTANCE_ID is not set/ + ); + }); +}); diff --git a/kits/firestore-incremental-capture/tests/tasks.enqueue.test.ts b/kits/firestore-incremental-capture/tests/tasks.enqueue.test.ts index 6aa878d8a7..a98aca39f5 100644 --- a/kits/firestore-incremental-capture/tests/tasks.enqueue.test.ts +++ b/kits/firestore-incremental-capture/tests/tasks.enqueue.test.ts @@ -27,9 +27,8 @@ const INSTANCE_ID = "capture"; let server: Server; let paths: string[] = []; -// The Admin SDK only reads CLOUD_TASKS_EMULATOR_HOST and FIREBASE_KIT_INSTANCE_ID -// when the app and the functions client are constructed, so both are set before -// firebase-admin is imported. +// The Admin SDK reads CLOUD_TASKS_EMULATOR_HOST when the functions client is +// constructed, so it is set before firebase-admin is imported. beforeAll(async () => { server = createServer((request, response) => { paths.push(request.url ?? ""); @@ -89,26 +88,6 @@ describe("enqueue", () => { ]); }); - // resolveResourceId reads the env var per call, so this state is reachable - // from the same app: it is what an instance deployed by a CLI that does not - // set FIREBASE_KIT_INSTANCE_ID would enqueue onto. - test("has no prefix of its own when the kit instance id is absent", async () => { - paths = []; - const { CHANGELOG_TASK_FUNCTION, enqueue } = await import("../src/tasks"); - delete process.env.FIREBASE_KIT_INSTANCE_ID; - - try { - await enqueue(config(), CHANGELOG_TASK_FUNCTION, { path: "users/alice" }); - } finally { - process.env.FIREBASE_KIT_INSTANCE_ID = INSTANCE_ID; - } - - expect(paths).toEqual([ - "/projects/test-project/locations/us-central1/queues/" + - "syncChangelogTask/tasks", - ]); - }); - test("targets the restoration queue, prefixed exactly once", async () => { paths = []; const { enqueue, RESTORATION_TASK_FUNCTION } = await import("../src/tasks"); diff --git a/kits/firestore-vector-search/CHANGELOG.md b/kits/firestore-vector-search/CHANGELOG.md index bccb4a65f6..acb2222168 100644 --- a/kits/firestore-vector-search/CHANGELOG.md +++ b/kits/firestore-vector-search/CHANGELOG.md @@ -1,3 +1,4 @@ +- The instance id now comes from `FIREBASE_KIT_INSTANCE_ID`, which the Firebase CLI (15.27.0 or later) provides to each kit instance; `INSTANCE_ID` is no longer a configuration parameter - Fixed the backfill and update task dispatch failing with "Queue does not exist": the kit prefixed queue names with `kit--` itself, which the Admin SDK then prefixed again from `FIREBASE_KIT_INSTANCE_ID`. The four `*_QUEUE_NAME` settings now take the deployed function name without that prefix - OpenAI embeddings are back on the extension's model and size: `EMBEDDING_PROVIDER: openai` requests `text-embedding-ada-002` at its native 1536 dimensions with a batch size of 16, replacing `text-embedding-3-small` pinned at 512 with a batch size of 1. Vectors written by an earlier version of the kit are not comparable with the ones it writes now, so re-embed the collection after upgrading. The vector index the kit creates for OpenAI is still declared with 512 dimensions, exactly as the extension declared it, so it does not cover the 1536-dimension vectors and `findNearest` fails against it; create the 1536-dimension index yourself if you query an OpenAI-embedded collection. - Initial release of kit, see README for differences between the legacy extension and this kit diff --git a/kits/firestore-vector-search/README.md b/kits/firestore-vector-search/README.md index 59806d10d4..08c33e266e 100644 --- a/kits/firestore-vector-search/README.md +++ b/kits/firestore-vector-search/README.md @@ -59,8 +59,9 @@ only deploys what your entry file exports. ## Deploy -The package's `firebase.json` declares a `kit` stanza (Firebase CLI 15.25.1 or -later, behind the `kits` experiment): +The package's `firebase.json` declares a `kit` stanza (Firebase CLI 15.27.0 or +later, behind the `kits` experiment - earlier CLIs do not provide the +`FIREBASE_KIT_INSTANCE_ID` variable this kit reads its instance id from): ```json { @@ -96,9 +97,13 @@ loads them at deploy time and prompts for any required values that are missing. Rows marked `secret` live in Secret Manager. You can reuse existing secrets; the CLI connects them to the function at deploy time. +The instance id is not a setting: the CLI provides it to each instance as +`FIREBASE_KIT_INSTANCE_ID`, set to that instance's key in the `instances` map. +`FIREBASE_` is a reserved prefix in `.env` files, so it cannot be set or +overridden there. + | Field | Env var | Required | Default | Description | |---|---|---|---|---| -| `instanceId` | `INSTANCE_ID` | yes | — | Must match this instance's key in the `instances` map | | `embeddingProvider` | `EMBEDDING_PROVIDER` | no | `gemini` | Embedding provider | | `customEmbeddingsEndpoint` | `CUSTOM_EMBEDDINGS_ENDPOINT` | no | (empty) | Custom embeddings endpoint | | `customEmbeddingsBatchSize` | `CUSTOM_EMBEDDINGS_BATCH_SIZE` | no | (empty) | Custom batch size | @@ -140,9 +145,10 @@ To run several vector-search indexes, add one entry per instance to the Instance ids must be unique across all kit stanzas in the project, and every instance's function names are namespaced by its `kit--` prefix, so -the instances cannot collide. Set `INSTANCE_ID` in each config directory to the -same value as that directory's key in the `instances` map; it also namespaces -the internal Firestore metadata/query paths and task queue references. +the instances cannot collide. Each instance learns its own id from the +`FIREBASE_KIT_INSTANCE_ID` variable the CLI provides; there is nothing to keep +in sync by hand. The id also namespaces the internal Firestore metadata/query +paths and task queue references. ## Differences from the Vector Search with Firestore extension @@ -175,18 +181,20 @@ dimension both the extension and the kit declare their vector index with, so the default provider works. This is the one place the kit deliberately does not match the extension's behaviour, because matching it means writing nothing at all. -### You set `INSTANCE_ID` yourself, and it names the query collection +### The instance id comes from `firebase.json`, and it names the query collection The extension derived its instance id at install and used it for the query collection (`_/index/queries`), the index metadata document -(`_/index`) and its task queues. Here `INSTANCE_ID` is a setting you -provide, and it must match this instance's key in the `instances` map in -`firebase.json`. To keep serving the query documents your clients already write -to, set it to your installed instance's id. The four task queue names can also be -overridden individually with `UPDATE_TRIGGER_QUEUE_NAME`, `UPDATE_TASK_QUEUE_NAME`, -`BACKFILL_TRIGGER_QUEUE_NAME` and `BACKFILL_TASK_QUEUE_NAME`, which the extension -did not allow. Each names the deployed function, without the -`kit--` prefix: the Admin SDK adds that when it resolves the queue. +(`_/index`) and its task queues. Here the CLI derives it from this +instance's key in the `instances` map in `firebase.json` and provides it to the +functions as `FIREBASE_KIT_INSTANCE_ID`. There is no `INSTANCE_ID` setting to +configure. To keep serving the query documents your clients already write to, +use your installed instance's id as the `instances` key. The four task queue +names can also be overridden individually with `UPDATE_TRIGGER_QUEUE_NAME`, +`UPDATE_TASK_QUEUE_NAME`, `BACKFILL_TRIGGER_QUEUE_NAME` and +`BACKFILL_TASK_QUEUE_NAME`, which the extension did not allow. Each names the +deployed function, without the `kit--` prefix: the Admin SDK adds +that when it resolves the queue. ### Create the `GEMINI_API_KEY` and `OPENAI_API_KEY` secrets, both of them diff --git a/kits/firestore-vector-search/src/config.ts b/kits/firestore-vector-search/src/config.ts index 8b1075c863..673e26a700 100644 --- a/kits/firestore-vector-search/src/config.ts +++ b/kits/firestore-vector-search/src/config.ts @@ -41,10 +41,29 @@ type ConfigExpression = T | Expression; export interface ConfigExpressions { collectionDocument: ConfigExpression; - queryCollectionDocument: ConfigExpression; } -const instanceId = defineString("INSTANCE_ID"); +/** + * Reads the instance id firebase-tools injects for kit instances (set to the + * instance's key in firebase.json) during discovery, in the emulator, and on + * deployed functions. The FIREBASE_ prefix is reserved in .env files and the + * params machinery never sees injected values, so it is a plain env read, not + * a defineString. Not evaluated at import: the `./lib` entry re-exports from + * this module and must load without the variable. + * + * @throws If the variable is missing, naming the CLI version that provides it. + */ +export function instanceIdFromEnv(): string { + const instanceId = process.env.FIREBASE_KIT_INSTANCE_ID; + if (!instanceId) { + throw new Error( + "FIREBASE_KIT_INSTANCE_ID is not set. It is provided automatically to " + + "kit instances by firebase-tools >= 15.27.0; deploy or emulate this " + + "kit with a supported CLI version." + ); + } + return instanceId; +} const EMBEDDING_PROVIDER_OPTIONS = [ "gemini", @@ -59,7 +78,6 @@ const DISTANCE_MEASURE_OPTIONS = [ "DOT_PRODUCT", ] as const; const params = { - instanceId, embeddingProvider: defineString("EMBEDDING_PROVIDER", { label: "LLM", description: @@ -187,7 +205,6 @@ const params = { export const CONFIG_EXPRESSIONS = { collectionDocument: expr`${params.collectionPath}/{docId}`, - queryCollectionDocument: expr`_${instanceId}/index/queries/{queryId}`, } as const satisfies ConfigExpressions; function optionalString(value: string): string | undefined { @@ -223,7 +240,7 @@ export function configFromEnv(): VectorSearchConfig { updateOnConfigure: params.updateOnConfigure.value(), region: process.env.FUNCTION_REGION, projectId: projectID.value(), - instanceId: params.instanceId.value(), + instanceId: instanceIdFromEnv(), geminiApiKey: optionalString(geminiApiKey.value()), openAiApiKey: optionalString(openAiApiKey.value()), bucketName: optionalString(storageBucket.value()), diff --git a/kits/firestore-vector-search/src/index.ts b/kits/firestore-vector-search/src/index.ts index aa4286f42f..a90a84f6c4 100644 --- a/kits/firestore-vector-search/src/index.ts +++ b/kits/firestore-vector-search/src/index.ts @@ -29,6 +29,7 @@ import { CONFIG_EXPRESSIONS, configFromEnv, geminiApiKey, + instanceIdFromEnv, openAiApiKey, } from "./config"; import { @@ -107,6 +108,12 @@ const CALLABLE_FUNCTION_OPTIONS = { secrets: FUNCTION_SECRETS, } as const; +// Resolved at import so the query trigger path is a concrete document path at +// discovery. An unsupported CLI fails the discovery pass here, before anything +// is registered, rather than freezing "_undefined/index/queries/{queryId}" +// into the manifest. +const QUERY_COLLECTION_DOCUMENT = `_${instanceIdFromEnv()}/index/queries/{queryId}`; + for (const role of REQUIRED_ROLES) { requiresRole(role); } @@ -188,7 +195,7 @@ export const embedOnWrite = onDocumentWritten( export const queryOnWrite = onDocumentWritten( { ...FIRESTORE_FUNCTION_OPTIONS, - document: CONFIG_EXPRESSIONS.queryCollectionDocument, + document: QUERY_COLLECTION_DOCUMENT, }, (event) => handleQueryOnWrite(event, getContext()) ); diff --git a/kits/firestore-vector-search/tests/config.test.ts b/kits/firestore-vector-search/tests/config.test.ts new file mode 100644 index 0000000000..50061dcd1b --- /dev/null +++ b/kits/firestore-vector-search/tests/config.test.ts @@ -0,0 +1,64 @@ +/** + * Copyright 2026 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { declaredParams, type Expression } from "firebase-functions/params"; +import { afterEach, beforeEach, describe, expect, test, vi } from "vitest"; +import { CONFIG_EXPRESSIONS, configFromEnv } from "../src/config"; + +beforeEach(() => { + vi.stubEnv("FIREBASE_CONFIG", JSON.stringify({ projectId: "test-project" })); + vi.stubEnv("FIREBASE_KIT_INSTANCE_ID", "test-instance"); +}); + +afterEach(() => { + vi.unstubAllEnvs(); +}); + +describe("CONFIG_EXPRESSIONS", () => { + test("binds the embed trigger to the collection name parameter", () => { + expect( + (CONFIG_EXPRESSIONS.collectionDocument as Expression).toCEL() + ).toBe("{{ params.COLLECTION_NAME }}/{docId}"); + }); +}); + +describe("instance id", () => { + // The CLI injects FIREBASE_KIT_INSTANCE_ID as a reserved env var; declaring + // it (or INSTANCE_ID) as a param makes the CLI prompt for a value it cannot + // accept and abort loading the kit. + test("is not declared as a param", () => { + const declared = declaredParams.map((param) => param.name); + + expect(declared).toContain("COLLECTION_NAME"); + expect(declared).not.toContain("INSTANCE_ID"); + expect(declared).not.toContain("FIREBASE_KIT_INSTANCE_ID"); + }); + + test("is read from the injected environment", () => { + expect(configFromEnv()).toMatchObject({ + projectId: "test-project", + instanceId: "test-instance", + }); + }); + + test("throws at runtime when FIREBASE_KIT_INSTANCE_ID is missing", () => { + vi.stubEnv("FIREBASE_KIT_INSTANCE_ID", undefined); + + expect(() => configFromEnv()).toThrow( + /FIREBASE_KIT_INSTANCE_ID is not set/ + ); + }); +}); diff --git a/kits/firestore-vector-search/tests/index.test.ts b/kits/firestore-vector-search/tests/index.test.ts new file mode 100644 index 0000000000..f3d5391cc2 --- /dev/null +++ b/kits/firestore-vector-search/tests/index.test.ts @@ -0,0 +1,60 @@ +/** + * Copyright 2026 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { afterEach, describe, expect, test, vi } from "vitest"; + +// `queries/setup` builds a FirestoreAdminClient at module scope and +// `embeddings` pulls in the provider SDKs; neither is needed to register the +// triggers. +vi.mock("../src/queries/setup", () => ({ createIndex: vi.fn() })); +vi.mock("../src/embeddings", () => ({ createEmbedClient: vi.fn() })); + +// firebase-functions allows one lifecycle hook of each kind per process, and +// the entry is imported once per test. +vi.mock("firebase-functions/v2/lifecycle", () => ({ + afterFirstDeploy: vi.fn(), + afterRedeploy: vi.fn(), +})); + +// The deploy entry resolves the query trigger path when it loads, so the +// environment has to be in place before each import. +async function importIndex(instanceId: string | undefined) { + vi.resetModules(); + vi.stubEnv("FIREBASE_KIT_INSTANCE_ID", instanceId); + return import("../src/index"); +} + +afterEach(() => { + vi.unstubAllEnvs(); +}); + +describe("deploy entry", () => { + test("names the query trigger collection after the instance id", async () => { + const { queryOnWrite } = await importIndex("test-instance"); + + expect(queryOnWrite.__endpoint.eventTrigger).toMatchObject({ + eventFilterPathPatterns: { + document: "_test-instance/index/queries/{queryId}", + }, + }); + }); + + test("fails discovery when FIREBASE_KIT_INSTANCE_ID is missing", async () => { + await expect(importIndex(undefined)).rejects.toThrow( + /FIREBASE_KIT_INSTANCE_ID is not set/ + ); + }); +}); diff --git a/kits/firestore-vector-search/tests/lib.test.ts b/kits/firestore-vector-search/tests/lib.test.ts new file mode 100644 index 0000000000..0c4d8deb6d --- /dev/null +++ b/kits/firestore-vector-search/tests/lib.test.ts @@ -0,0 +1,33 @@ +/** + * Copyright 2026 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { afterEach, expect, test, vi } from "vitest"; + +afterEach(() => { + vi.unstubAllEnvs(); +}); + +// The README promises `./lib` reads no environment when it loads, so it can be +// imported from a user's own triggers and tests outside the Firebase CLI. +test("the library entry loads without FIREBASE_KIT_INSTANCE_ID", async () => { + vi.resetModules(); + vi.stubEnv("FIREBASE_KIT_INSTANCE_ID", undefined); + + const lib = await import("../src/lib"); + + expect(lib.configFromEnv).toBeTypeOf("function"); + expect(lib.resolveVectorSearchConfig).toBeTypeOf("function"); +});