diff --git a/kits/firestore-bigquery-export/CHANGELOG.md b/kits/firestore-bigquery-export/CHANGELOG.md index a3a12579e..250fddd0e 100644 --- a/kits/firestore-bigquery-export/CHANGELOG.md +++ b/kits/firestore-bigquery-export/CHANGELOG.md @@ -1,3 +1,4 @@ +- fix: `USE_NEW_SNAPSHOT_QUERY_SYNTAX` and `EXCLUDE_OLD_DATA` take the extension's `yes` / `no` values again, so a config exported from the extension works unchanged. `WILDCARD_IDS` is unchanged: the extension already used `true` / `false` there. - feat: reinstate the extension's Cloud Tasks write buffer. A failed inline BigQuery write now enqueues onto a new `syncBigQuery` task queue (5 attempts, 60s minimum backoff, throttled by the restored `MAX_DISPATCHES_PER_SECOND` param, default 100) instead of replaying the Firestore event through Eventarc redelivery for up to 24 hours; `MAX_ENQUEUE_ATTEMPTS` (default 3) is also back. The `onSuccess` event returns with the queue handler. Two behavior changes against earlier release candidates: a row that exhausts the queue is dropped unless `BACKUP_COLLECTION` is set (extension parity - the tracker backs the row up on every terminal insert failure, so configure a backup collection), and deleting or moving the functions can leave the Cloud Tasks queue behind. A failed enqueue is logged at error level, published as an `onError` event, and dropped, as in the extension; the trigger no longer declares `retry: true`, so nothing is redelivered through Eventarc. Export the new `syncBigQuery` function from your codebase entry, and deploy with Firebase CLI 15.28.0+ so the trigger can address its own queue (`FIREBASE_KIT_INSTANCE_ID`); requires firebase-admin 14.2.0+. - fix: restore explicit function placement from `DATABASE_REGION`, now with the Firestore-location-to-Cloud-Run-region mapping. The `DATABASE_REGION` parameter is back and all three functions deploy to the region derived from it: regional locations pass through unchanged, and the multi-region locations map to a Cloud Run region (`nam5`/`nam7` to `us-central1`, `eur3` to `europe-west1`) instead of failing the deploy. With the parameter unset the functions still declare no region and the CLI falls back as before (`us-central1` by default, `FIREBASE_FUNCTIONS_DEFAULT_REGION` to override). Placement requires firebase-tools >= 15.28.0 (older CLIs do not load `.env` at discovery and keep the fallback). If your `.env` already carries `DATABASE_REGION` from an extension migration, upgrading to this version moves the functions to the mapped region on your next deploy, which deletes and recreates them. - fix: stop deploying functions to the `DATABASE_REGION` value. Firestore multi-region locations (`eur3`, `nam5`, `nam7`) are not Cloud Run regions, so any multi-region database made every deploy fail. The `DATABASE_REGION` parameter is removed; the functions now declare no region and deploy to `us-central1` by default (set `FIREBASE_FUNCTIONS_DEFAULT_REGION` when deploying to choose another region), while the Firestore trigger is always pinned to the database's own region. `ExportConfig.location` is removed from the library surface. diff --git a/kits/firestore-bigquery-export/README.md b/kits/firestore-bigquery-export/README.md index fdc8cf07a..c8205556a 100644 --- a/kits/firestore-bigquery-export/README.md +++ b/kits/firestore-bigquery-export/README.md @@ -133,8 +133,8 @@ loads them at deploy time and prompts for any required values that are missing. | `timePartitioningFirestoreField` | `TIME_PARTITIONING_FIRESTORE_FIELD` | no | (empty) | Firestore field for partitioning | | `clustering` | `CLUSTERING` | no | (empty) | Clustering columns (max 4) | | `wildcardIds` | `WILDCARD_IDS` | no | `false` | Store path-param values as columns | -| `useNewSnapshotQuerySyntax` | `USE_NEW_SNAPSHOT_QUERY_SYNTAX` | no | `false` | Use newer snapshot query syntax | -| `excludeOldData` | `EXCLUDE_OLD_DATA` | no | `false` | Skip previous document state on updates | +| `useNewSnapshotQuerySyntax` | `USE_NEW_SNAPSHOT_QUERY_SYNTAX` | no | `no` | Use newer snapshot query syntax (`yes` / `no`) | +| `excludeOldData` | `EXCLUDE_OLD_DATA` | no | `no` | Skip previous document state on updates (`yes` / `no`) | | `viewType` | `VIEW_TYPE` | no | `view` | `view`, `materialized_incremental`, `materialized_non_incremental` | | `maxStaleness` | `MAX_STALENESS` | no | (empty) | Materialized view max staleness | | `refreshIntervalMinutes` | `REFRESH_INTERVAL_MINUTES` | no | (empty) | Materialized view refresh interval | @@ -348,13 +348,6 @@ This kit is the extension repackaged as an npm package, but a few things behave differently. If you are moving from an installed extension instance, read this section before you deploy. -### Boolean settings use `true` / `false` - -`WILDCARD_IDS`, `USE_NEW_SNAPSHOT_QUERY_SYNTAX` and `EXCLUDE_OLD_DATA` are -boolean params, and only the literal string `true` enables them. The extension -used `yes` / `no` for the last two, so copying an old config across leaves them -silently disabled. Change any `yes` to `true` in your `.env`. - ### Failed writes: same buffer The kit keeps the extension's write-path architecture: a failed BigQuery write diff --git a/kits/firestore-bigquery-export/src/config.ts b/kits/firestore-bigquery-export/src/config.ts index cd6f958d3..28e9be215 100644 --- a/kits/firestore-bigquery-export/src/config.ts +++ b/kits/firestore-bigquery-export/src/config.ts @@ -398,19 +398,21 @@ const params = { default: false, }), - useNewSnapshotQuerySyntax: defineBoolean("USE_NEW_SNAPSHOT_QUERY_SYNTAX", { + useNewSnapshotQuerySyntax: defineString("USE_NEW_SNAPSHOT_QUERY_SYNTAX", { label: "Use new query syntax for snapshots", description: "If enabled, snapshots will be generated with the new query syntax, which should be more performant, and avoid potential resource limitations.", - default: false, + default: "no", + input: select({ Yes: "yes", No: "no" }), }), - excludeOldData: defineBoolean("EXCLUDE_OLD_DATA", { + excludeOldData: defineString("EXCLUDE_OLD_DATA", { label: "Exclude old data payloads", description: "If enabled, table rows will never contain old data (document snapshot before the Firestore onDocumentUpdate event: `change.before.data()`). The reduction in data should be more performant, and avoid potential resource limitations.", - default: false, + default: "no", + input: select({ Yes: "yes", No: "no" }), }), viewType: defineString("VIEW_TYPE", { label: "View Type", @@ -633,6 +635,12 @@ function normalizePositiveInt(value: string): number | undefined { return normalized > 0 ? normalized : undefined; } +// The extension's select emits `yes` / `no`; its label is `Yes`, and the CLI +// copies .env values verbatim, so case and whitespace are forgiven. +function yesNo(value: string): boolean { + return value.trim().toLowerCase() === "yes"; +} + /** Coerce an empty-string param value to `undefined`. */ function optional(value: string): string | undefined { return value.length > 0 ? value : undefined; @@ -674,8 +682,8 @@ export function configFromEnv(): ExportConfig { projectId: projectID.value(), databaseId: optional(params.database.value()) || "(default)", wildcardIds: params.wildcardIds.value(), - excludeOldData: params.excludeOldData.value(), - useNewSnapshotQuerySyntax: params.useNewSnapshotQuerySyntax.value(), + excludeOldData: yesNo(params.excludeOldData.value()), + useNewSnapshotQuerySyntax: yesNo(params.useNewSnapshotQuerySyntax.value()), viewType: (optional(params.viewType.value()) || "view") as ViewType, partitioning: buildPartitioningConfig({ timePartitioning: timePartitioning(tablePartitioning), diff --git a/kits/firestore-bigquery-export/tests/config.test.ts b/kits/firestore-bigquery-export/tests/config.test.ts index 191a85c5e..fa1df9db1 100644 --- a/kits/firestore-bigquery-export/tests/config.test.ts +++ b/kits/firestore-bigquery-export/tests/config.test.ts @@ -33,15 +33,10 @@ vi.mock("firebase-functions/params", () => ({ toJSON(): string { return this.toString(); } - }, - defineString: ( - _name: string, - opts?: { default?: string | { value(): string } } - ) => ({ - value: () => - typeof opts?.default === "string" - ? opts.default - : opts?.default?.value() ?? "", + }, // Mirrors the real StringParam: the env var or "", never the declared + // default, which only the CLI prompt consults. + defineString: (_name: string) => ({ + value: () => process.env[_name] || "", toString: () => `params.${_name}`, }), // Mirrors the real IntParam: a missing or blank env var resolves to 0, and @@ -166,12 +161,15 @@ describe("buildPartitioningConfig", () => { describe("configFromEnv", () => { test("maps params", () => { + // The CLI resolves the param's default into the env at deploy. + vi.stubEnv("BIGQUERY_PROJECT_ID", "test-project"); const config = configFromEnv(); expect(config.projectId).toBe("test-project"); expect(config.bqProjectId).toBe("test-project"); expect(config.databaseId).toBe("(default)"); expect(config.viewType).toBe("view"); expect(resolveExportConfig(config)).not.toHaveProperty("location"); + vi.unstubAllEnvs(); }); test("reports unset queue params as undefined so the documented defaults apply", () => { @@ -201,6 +199,37 @@ describe("configFromEnv", () => { vi.unstubAllEnvs(); }); + test("yes/no selects default to off", () => { + const config = configFromEnv(); + expect(config.useNewSnapshotQuerySyntax).toBe(false); + expect(config.excludeOldData).toBe(false); + }); + + test("yes/no selects enable on the extension's literal yes", () => { + vi.stubEnv("USE_NEW_SNAPSHOT_QUERY_SYNTAX", "yes"); + vi.stubEnv("EXCLUDE_OLD_DATA", "yes"); + const config = configFromEnv(); + expect(config.useNewSnapshotQuerySyntax).toBe(true); + expect(config.excludeOldData).toBe(true); + vi.unstubAllEnvs(); + }); + test("yes/no selects forgive the label's case and surrounding whitespace", () => { + vi.stubEnv("USE_NEW_SNAPSHOT_QUERY_SYNTAX", "Yes"); + vi.stubEnv("EXCLUDE_OLD_DATA", " yes "); + const config = configFromEnv(); + expect(config.useNewSnapshotQuerySyntax).toBe(true); + expect(config.excludeOldData).toBe(true); + vi.unstubAllEnvs(); + }); + + test("yes/no selects treat no and anything else as off", () => { + vi.stubEnv("USE_NEW_SNAPSHOT_QUERY_SYNTAX", "no"); + vi.stubEnv("EXCLUDE_OLD_DATA", "true"); + expect(configFromEnv().useNewSnapshotQuerySyntax).toBe(false); + expect(configFromEnv().excludeOldData).toBe(false); + vi.unstubAllEnvs(); + }); + test("exposes deploy-time expressions for trigger metadata", () => { expect(CONFIG_EXPRESSIONS.collectionPath.toString()).toBe( "params.COLLECTION_PATH"