refactor(kits): read the kit instance id from FIREBASE_KIT_INSTANCE_ID

kits/README.md

@@ -36,7 +36,10 @@ functions it exports.
 
 ## Using a kit
 
-Kits need Firebase CLI 15.25.1 or later with the `kits` experiment enabled:
+Kits need Firebase CLI 15.27.0 or later with the `kits` experiment enabled.
+The CLI provides each instance its id as the `FIREBASE_KIT_INSTANCE_ID`
+environment variable; earlier versions do not set it, and kits that use the
+instance id will not deploy or run without it.
 
 ```sh
 firebase experiments:enable kits

kits/bigquery-firestore-export/CHANGELOG.md

@@ -1 +1,2 @@
 - Initial release of kit, see README for differences between the legacy extension and this kit
+- The instance id now comes from `FIREBASE_KIT_INSTANCE_ID`, which the Firebase CLI (15.27.0 or later) provides to each kit instance; `INSTANCE_ID` is no longer a configuration parameter

kits/bigquery-firestore-export/README.md

@@ -49,7 +49,6 @@ export {
 and configure them with a `.env` (or `.env.<projectId>`):
 
 ```sh
-INSTANCE_ID=analytics-export
 BIGQUERY_DATASET_LOCATION=US
 DATASET_ID=analytics
 TABLE_NAME=users
@@ -67,8 +66,9 @@ only deploys what your entry file exports.
 
 ## Deploy
 
-The package's `firebase.json` declares a `kit` stanza (Firebase CLI 15.25.1 or
-later, behind the `kits` experiment):
+The package's `firebase.json` declares a `kit` stanza (Firebase CLI 15.27.0 or
+later, behind the `kits` experiment - earlier CLIs do not provide the
+`FIREBASE_KIT_INSTANCE_ID` variable this kit reads its instance id from):
 
 ```json
 {
@@ -103,9 +103,13 @@ Set these values in a `.env` (or `.env.<projectId>`) file. The Firebase CLI
 loads them at deploy time and prompts for required values that are missing.
 `PROJECT_ID` is supplied by the Firebase CLI.
 
+The instance id is not a setting: the CLI provides it to each instance as
+`FIREBASE_KIT_INSTANCE_ID`, set to that instance's key in the `instances` map.
+`FIREBASE_` is a reserved prefix in `.env` files, so it cannot be set or
+overridden there.
+
 | Field                     | Env var                     | Required | Default           | Description                                                  |
 | ------------------------- | --------------------------- | -------- | ----------------- | ------------------------------------------------------------ |
-| `instanceId`              | `INSTANCE_ID`               | yes      | —                 | Must match this instance's key in the `instances` map        |
 | `bigqueryDatasetLocation` | `BIGQUERY_DATASET_LOCATION` | no       | `US`              | BigQuery destination dataset location                        |
 | `transferConfigName`      | `TRANSFER_CONFIG_NAME`      | no       | (empty)           | Existing DTS config resource to link instead of creating one |
 | `datasetId`               | `DATASET_ID`                | yes      | —                 | BigQuery destination dataset id                              |
@@ -114,7 +118,7 @@ loads them at deploy time and prompts for required values that are missing.
 | `displayName`             | `DISPLAY_NAME`              | yes      | —                 | Human-readable scheduled-query name                          |
 | `partitioningField`       | `PARTITIONING_FIELD`        | no       | (empty)           | Destination-table partitioning field                         |
 | `schedule`                | `SCHEDULE`                  | yes      | —                 | DTS schedule, such as `every 24 hours`                       |
-| `pubSubTopic`             | `PUB_SUB_TOPIC`             | no       | `kit-<INSTANCE_ID>-processMessages` | Pub/Sub topic ID, not a full resource name, receiving DTS completion notifications |
+| `pubSubTopic`             | `PUB_SUB_TOPIC`             | no       | `kit-<instance id>-processMessages` | Pub/Sub topic ID, not a full resource name, receiving DTS completion notifications |
 | `firestoreCollection`     | `COLLECTION_PATH`           | no       | `transferConfigs` | Root Firestore collection for configs and output             |
 | `logLevel`                | `LOG_LEVEL`                 | no       | `info`            | `debug`, `info`, `warn`, `error`, or `silent`                |
 
@@ -140,10 +144,10 @@ To run several reverse-sync instances, add one entry per instance to the
 
 Instance ids must be unique across all kit stanzas in the project, and every
 instance's function names are namespaced by its `kit-<instance id>-` prefix, so
-the instances cannot collide. Set `INSTANCE_ID` in each config directory to the
-same value as that directory's key in the `instances` map; it also namespaces
-the Pub/Sub notification topic and associates the deployment with its transfer
-config.
+the instances cannot collide. Each instance learns its own id from the
+`FIREBASE_KIT_INSTANCE_ID` variable the CLI provides; there is nothing to keep
+in sync by hand. The id also namespaces the Pub/Sub notification topic and
+associates the deployment with its transfer config.
 
 ## Provisioning
 
@@ -185,18 +189,19 @@ variable name and default, so a `.env` copied from your installed instance needs
 no value changes. What changes is the instance id, the Pub/Sub topic, the identity
 the scheduled query runs as, and how repeated BigQuery columns land in Firestore.
 
-### You set `INSTANCE_ID` yourself, and the Pub/Sub topic is renamed
+### The instance id comes from `firebase.json`, and the Pub/Sub topic is renamed
 
 The extension derived an instance id at install and used it to name its
 notification topic (`ext-<instance id>-processMessages`) and to tag its transfer
-config document with `extInstanceId`. Here `INSTANCE_ID` is a setting you
-provide, and it must match this instance's key in the `instances` map in
-`firebase.json`.
-
-The topic defaults to `kit-<INSTANCE_ID>-processMessages`, and the kit creates it
-on first run if it does not already exist. Set `INSTANCE_ID` to your installed
-instance's id if you want the kit to adopt the scheduled query that instance
-created, because the lookup is by `extInstanceId` on the documents in
+config document with `extInstanceId`. Here the CLI derives it from this
+instance's key in the `instances` map in `firebase.json` and provides it to the
+functions as `FIREBASE_KIT_INSTANCE_ID`. There is no `INSTANCE_ID` setting to
+configure.
+
+The topic defaults to `kit-<instance id>-processMessages`, and the kit creates it
+on first run if it does not already exist. Use your installed instance's id as
+the `instances` key if you want the kit to adopt the scheduled query that
+instance created, because the lookup is by `extInstanceId` on the documents in
 `COLLECTION_PATH`. With a different id the kit finds nothing, creates a second
 scheduled query, and you end up with two writing into the same collection.
 
@@ -209,7 +214,7 @@ there are two ways to migrate:
   left untouched: anything else subscribed to that topic, including an extension
   instance still installed, keeps receiving run notifications.
 - Take the default. The first deploy repoints the transfer config at
-  `kit-<INSTANCE_ID>-processMessages`, which stops notifications reaching the
+  `kit-<instance id>-processMessages`, which stops notifications reaching the
   extension and any other subscriber on the old topic. Do this once the extension
   is uninstalled, and the old topic can then be deleted.
 
@@ -285,7 +290,7 @@ result query where your dataset lives.
 extension's 1st gen trigger did not retry. A run whose results fail to copy, for
 example because BigQuery or Firestore is briefly unavailable, is now retried
 rather than dropped. A notification that keeps failing, such as one for a transfer
-config not tagged with this `INSTANCE_ID`, is also retried until Pub/Sub gives up.
+config not tagged with this instance id, is also retried until Pub/Sub gives up.
 
 Both functions' service accounts need `roles/eventarc.eventReceiver` and
 `roles/run.invoker` on top of the three roles the extension asked for, and the

kits/bigquery-firestore-export/src/config.ts

@@ -14,23 +14,37 @@
  * limitations under the License.
  */
 
-import {
-  defineString,
-  expr,
-  projectID,
-  select,
-} from "firebase-functions/params";
+import { defineString, projectID, select } from "firebase-functions/params";
 import type {
   BigqueryFirestoreExportConfig,
   DeployTimeOptions,
   LogLevel,
 } from "./export-config";
 
 const LOG_LEVEL_OPTIONS = ["debug", "info", "warn", "error", "silent"] as const;
-const instanceId = defineString("INSTANCE_ID");
+
+// firebase-tools injects this for kit instances (set to the instance's key in
+// firebase.json) during discovery, in the emulator, and on deployed functions.
+// The FIREBASE_ prefix is reserved in .env files and the params machinery never
+// sees injected values, so it must be a plain env read, not a defineString.
+function instanceIdFromEnv(): string {
+  const instanceId = process.env.FIREBASE_KIT_INSTANCE_ID;
+  if (!instanceId) {
+    throw new Error(
+      "FIREBASE_KIT_INSTANCE_ID is not set. It is provided automatically to " +
+        "kit instances by firebase-tools >= 15.27.0; deploy or emulate this " +
+        "kit with a supported CLI version."
+    );
+  }
+  return instanceId;
+}
+
+// Resolved at import so the topic default is a concrete name at discovery. An
+// unsupported CLI fails the discovery pass here rather than freezing
+// "kit-undefined-processMessages" into the manifest.
+const instanceId = instanceIdFromEnv();
 
 const params = {
-  instanceId,
   bigqueryDatasetLocation: defineString("BIGQUERY_DATASET_LOCATION", {
     label: "BigQuery Dataset Location",
     description:
@@ -81,7 +95,7 @@ const params = {
     description:
       "Which Pub/Sub topic should receive BigQuery Data Transfer completion notifications? Leave the default unless you are migrating from the bigquery-firestore-export extension, whose topic is named ext-<instance id>-processMessages. Pointing this at the extension's topic keeps the existing scheduled query's notification settings untouched.",
 
-    default: expr`kit-${instanceId}-processMessages`,
+    default: `kit-${instanceId}-processMessages`,
     input: {
       text: {
         nonEmpty: true,
@@ -201,12 +215,10 @@ function normalizeLogLevel(value: string): LogLevel {
 
 /** Reads runtime values from Firebase deploy-time parameters. */
 export function configFromEnv(): BigqueryFirestoreExportConfig {
-  const resolvedInstanceId = params.instanceId.value();
-
   return {
     bigqueryDatasetLocation: params.bigqueryDatasetLocation.value(),
     projectId: projectID.value(),
-    instanceId: resolvedInstanceId,
+    instanceId: instanceIdFromEnv(),
     transferConfigName: optional(params.transferConfigName.value()),
     datasetId: params.datasetId.value(),
     tableName: params.tableName.value(),

kits/bigquery-firestore-export/tests/config.test.ts

@@ -14,32 +14,59 @@
  * limitations under the License.
  */
 
-import { Expression } from "firebase-functions/params";
+import { declaredParams, Expression } from "firebase-functions/params";
 import { afterEach, describe, expect, test, vi } from "vitest";
-import { CONFIG_EXPRESSIONS, configFromEnv } from "../src/config";
+
+const INSTANCE_ID = "users-export";
+
+// The instance id is read when the module loads, so the environment has to be
+// in place before each import.
+async function importConfig(instanceId: string | undefined) {
+  vi.resetModules();
+  vi.stubEnv("FIREBASE_KIT_INSTANCE_ID", instanceId);
+  return import("../src/config");
+}
+
+function stubRuntimeEnv() {
+  vi.stubEnv("FIREBASE_CONFIG", JSON.stringify({ projectId: "test-project" }));
+  vi.stubEnv("BIGQUERY_DATASET_LOCATION", "EU");
+  vi.stubEnv("DATASET_ID", "analytics");
+  vi.stubEnv("TABLE_NAME", "users");
+  vi.stubEnv("QUERY_STRING", "SELECT * FROM source.users");
+  vi.stubEnv("DISPLAY_NAME", "Users export");
+  vi.stubEnv("SCHEDULE", "every 24 hours");
+  vi.stubEnv("COLLECTION_PATH", "transferConfigs");
+  vi.stubEnv("LOG_LEVEL", "info");
+}
 
 afterEach(() => {
   vi.unstubAllEnvs();
 });
 
 describe("CONFIG_EXPRESSIONS", () => {
-  test("binds the trigger to the Pub/Sub topic parameter", () => {
+  test("binds the trigger to the Pub/Sub topic parameter", async () => {
+    const { CONFIG_EXPRESSIONS } = await importConfig(INSTANCE_ID);
+
     expect(CONFIG_EXPRESSIONS.pubSubTopic).toBeInstanceOf(Expression);
     expect((CONFIG_EXPRESSIONS.pubSubTopic as Expression<string>).toCEL()).toBe(
       "{{ params.PUB_SUB_TOPIC }}"
     );
   });
 
-  test("defaults the topic parameter to the instance-namespaced kit topic", () => {
+  test("defaults the topic parameter to the instance-namespaced kit topic", async () => {
+    const { CONFIG_EXPRESSIONS } = await importConfig(INSTANCE_ID);
+
     const spec = (
       CONFIG_EXPRESSIONS.pubSubTopic as unknown as {
         toSpec: () => { default?: string };
       }
     ).toSpec();
-    expect(spec.default).toBe("kit-{{ params.INSTANCE_ID }}-processMessages");
+    expect(spec.default).toBe("kit-users-export-processMessages");
   });
 
-  test("accepts a topic ID but rejects a full resource name", () => {
+  test("accepts a topic ID but rejects a full resource name", async () => {
+    const { CONFIG_EXPRESSIONS } = await importConfig(INSTANCE_ID);
+
     const spec = (
       CONFIG_EXPRESSIONS.pubSubTopic as unknown as {
         toSpec: () => { input?: { text?: { validationRegex?: string } } };
@@ -60,22 +87,40 @@ describe("CONFIG_EXPRESSIONS", () => {
   });
 });
 
-describe("configFromEnv", () => {
-  test("reads runtime parameters and derives the same topic", () => {
-    vi.stubEnv(
-      "FIREBASE_CONFIG",
-      JSON.stringify({ projectId: "test-project" })
+describe("instance id", () => {
+  // The CLI injects FIREBASE_KIT_INSTANCE_ID as a reserved env var; declaring
+  // it (or INSTANCE_ID) as a param makes the CLI prompt for a value it cannot
+  // accept and abort loading the kit.
+  test("is not declared as a param", async () => {
+    await importConfig(INSTANCE_ID);
+
+    const declared = declaredParams.map((param) => param.name);
+    expect(declared).toContain("PUB_SUB_TOPIC");
+    expect(declared).not.toContain("INSTANCE_ID");
+    expect(declared).not.toContain("FIREBASE_KIT_INSTANCE_ID");
+  });
+
+  test("fails discovery when FIREBASE_KIT_INSTANCE_ID is missing", async () => {
+    await expect(importConfig(undefined)).rejects.toThrow(
+      /FIREBASE_KIT_INSTANCE_ID is not set/
     );
-    vi.stubEnv("INSTANCE_ID", "users-export");
-    vi.stubEnv("BIGQUERY_DATASET_LOCATION", "EU");
-    vi.stubEnv("DATASET_ID", "analytics");
-    vi.stubEnv("TABLE_NAME", "users");
-    vi.stubEnv("QUERY_STRING", "SELECT * FROM source.users");
-    vi.stubEnv("DISPLAY_NAME", "Users export");
-    vi.stubEnv("SCHEDULE", "every 24 hours");
-    vi.stubEnv("COLLECTION_PATH", "transferConfigs");
-    vi.stubEnv("LOG_LEVEL", "info");
+  });
+
+  test("throws at runtime when FIREBASE_KIT_INSTANCE_ID is missing", async () => {
+    const { configFromEnv } = await importConfig(INSTANCE_ID);
+    stubRuntimeEnv();
+    vi.stubEnv("FIREBASE_KIT_INSTANCE_ID", undefined);
 
+    expect(() => configFromEnv()).toThrow(
+      /FIREBASE_KIT_INSTANCE_ID is not set/
+    );
+  });
+});
+
+describe("configFromEnv", () => {
+  test("reads runtime parameters and derives the same topic", async () => {
+    const { configFromEnv } = await importConfig(INSTANCE_ID);
+    stubRuntimeEnv();
     vi.stubEnv("PUB_SUB_TOPIC", "kit-users-export-processMessages");
 
     expect(configFromEnv()).toMatchObject({
@@ -90,20 +135,9 @@ describe("configFromEnv", () => {
     });
   });
 
-  test("passes through a topic pointing at the extension's own topic", () => {
-    vi.stubEnv(
-      "FIREBASE_CONFIG",
-      JSON.stringify({ projectId: "test-project" })
-    );
-    vi.stubEnv("INSTANCE_ID", "users-export");
-    vi.stubEnv("BIGQUERY_DATASET_LOCATION", "EU");
-    vi.stubEnv("DATASET_ID", "analytics");
-    vi.stubEnv("TABLE_NAME", "users");
-    vi.stubEnv("QUERY_STRING", "SELECT * FROM source.users");
-    vi.stubEnv("DISPLAY_NAME", "Users export");
-    vi.stubEnv("SCHEDULE", "every 24 hours");
-    vi.stubEnv("COLLECTION_PATH", "transferConfigs");
-    vi.stubEnv("LOG_LEVEL", "info");
+  test("passes through a topic pointing at the extension's own topic", async () => {
+    const { configFromEnv } = await importConfig(INSTANCE_ID);
+    stubRuntimeEnv();
     vi.stubEnv("PUB_SUB_TOPIC", "ext-users-export-processMessages");
 
     expect(configFromEnv().pubSubTopic).toBe(

kits/firestore-incremental-capture/CHANGELOG.md

@@ -1,3 +1,4 @@
+- The instance id now comes from `FIREBASE_KIT_INSTANCE_ID`, which the Firebase CLI (15.27.0 or later) provides to each kit instance; `INSTANCE_ID` is no longer a configuration parameter
 - Fixed task dispatch failing with "Queue does not exist": the kit prefixed queue names with `kit-<instance id>-` itself, which the Admin SDK then prefixed again from `FIREBASE_KIT_INSTANCE_ID`. Changelog rows never reached BigQuery and restorations never started
 - Full implementation, replacing the skeleton package: Firestore capture to a BigQuery changelog, Dataflow-based point-in-time restoration, and first-deploy provisioning. See the README for differences between the legacy extension and this kit.
 - The restoration pipeline is consumed as a pinned, digest-verified release of GoogleCloudPlatform/firebase-extensions (`firestore-incremental-capture-pipeline-v0.1.0`) downloaded by `scripts/setup.sh`; Maven is no longer required.