Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions kits/firestore-bigquery-export/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
- fix: `USE_NEW_SNAPSHOT_QUERY_SYNTAX` and `EXCLUDE_OLD_DATA` take the extension's `yes` / `no` values again, so a config exported from the extension works unchanged. `WILDCARD_IDS` is unchanged: the extension already used `true` / `false` there.
- feat: reinstate the extension's Cloud Tasks write buffer. A failed inline BigQuery write now enqueues onto a new `syncBigQuery` task queue (5 attempts, 60s minimum backoff, throttled by the restored `MAX_DISPATCHES_PER_SECOND` param, default 100) instead of replaying the Firestore event through Eventarc redelivery for up to 24 hours; `MAX_ENQUEUE_ATTEMPTS` (default 3) is also back. The `onSuccess` event returns with the queue handler. Two behavior changes against earlier release candidates: a row that exhausts the queue is dropped unless `BACKUP_COLLECTION` is set (extension parity - the tracker backs the row up on every terminal insert failure, so configure a backup collection), and deleting or moving the functions can leave the Cloud Tasks queue behind. A failed enqueue is logged at error level, published as an `onError` event, and dropped, as in the extension; the trigger no longer declares `retry: true`, so nothing is redelivered through Eventarc. Export the new `syncBigQuery` function from your codebase entry, and deploy with Firebase CLI 15.28.0+ so the trigger can address its own queue (`FIREBASE_KIT_INSTANCE_ID`); requires firebase-admin 14.2.0+.
- fix: restore explicit function placement from `DATABASE_REGION`, now with the Firestore-location-to-Cloud-Run-region mapping. The `DATABASE_REGION` parameter is back and all three functions deploy to the region derived from it: regional locations pass through unchanged, and the multi-region locations map to a Cloud Run region (`nam5`/`nam7` to `us-central1`, `eur3` to `europe-west1`) instead of failing the deploy. With the parameter unset the functions still declare no region and the CLI falls back as before (`us-central1` by default, `FIREBASE_FUNCTIONS_DEFAULT_REGION` to override). Placement requires firebase-tools >= 15.28.0 (older CLIs do not load `.env` at discovery and keep the fallback). If your `.env` already carries `DATABASE_REGION` from an extension migration, upgrading to this version moves the functions to the mapped region on your next deploy, which deletes and recreates them.
- fix: stop deploying functions to the `DATABASE_REGION` value. Firestore multi-region locations (`eur3`, `nam5`, `nam7`) are not Cloud Run regions, so any multi-region database made every deploy fail. The `DATABASE_REGION` parameter is removed; the functions now declare no region and deploy to `us-central1` by default (set `FIREBASE_FUNCTIONS_DEFAULT_REGION` when deploying to choose another region), while the Firestore trigger is always pinned to the database's own region. `ExportConfig.location` is removed from the library surface.
Expand Down
11 changes: 2 additions & 9 deletions kits/firestore-bigquery-export/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,8 +133,8 @@ loads them at deploy time and prompts for any required values that are missing.
| `timePartitioningFirestoreField` | `TIME_PARTITIONING_FIRESTORE_FIELD` | no | (empty) | Firestore field for partitioning |
| `clustering` | `CLUSTERING` | no | (empty) | Clustering columns (max 4) |
| `wildcardIds` | `WILDCARD_IDS` | no | `false` | Store path-param values as columns |
| `useNewSnapshotQuerySyntax` | `USE_NEW_SNAPSHOT_QUERY_SYNTAX` | no | `false` | Use newer snapshot query syntax |
| `excludeOldData` | `EXCLUDE_OLD_DATA` | no | `false` | Skip previous document state on updates |
| `useNewSnapshotQuerySyntax` | `USE_NEW_SNAPSHOT_QUERY_SYNTAX` | no | `no` | Use newer snapshot query syntax (`yes` / `no`) |
| `excludeOldData` | `EXCLUDE_OLD_DATA` | no | `no` | Skip previous document state on updates (`yes` / `no`) |
| `viewType` | `VIEW_TYPE` | no | `view` | `view`, `materialized_incremental`, `materialized_non_incremental` |
| `maxStaleness` | `MAX_STALENESS` | no | (empty) | Materialized view max staleness |
| `refreshIntervalMinutes` | `REFRESH_INTERVAL_MINUTES` | no | (empty) | Materialized view refresh interval |
Expand Down Expand Up @@ -348,13 +348,6 @@ This kit is the extension repackaged as an npm package, but a few things behave
differently. If you are moving from an installed extension instance, read this
section before you deploy.

### Boolean settings use `true` / `false`

`WILDCARD_IDS`, `USE_NEW_SNAPSHOT_QUERY_SYNTAX` and `EXCLUDE_OLD_DATA` are
boolean params, and only the literal string `true` enables them. The extension
used `yes` / `no` for the last two, so copying an old config across leaves them
silently disabled. Change any `yes` to `true` in your `.env`.

### Failed writes: same buffer

The kit keeps the extension's write-path architecture: a failed BigQuery write
Expand Down
20 changes: 14 additions & 6 deletions kits/firestore-bigquery-export/src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -398,19 +398,21 @@ const params = {

default: false,
}),
useNewSnapshotQuerySyntax: defineBoolean("USE_NEW_SNAPSHOT_QUERY_SYNTAX", {
useNewSnapshotQuerySyntax: defineString("USE_NEW_SNAPSHOT_QUERY_SYNTAX", {
label: "Use new query syntax for snapshots",
description:
"If enabled, snapshots will be generated with the new query syntax, which should be more performant, and avoid potential resource limitations.",

default: false,
default: "no",
input: select({ Yes: "yes", No: "no" }),
}),
excludeOldData: defineBoolean("EXCLUDE_OLD_DATA", {
excludeOldData: defineString("EXCLUDE_OLD_DATA", {
label: "Exclude old data payloads",
description:
"If enabled, table rows will never contain old data (document snapshot before the Firestore onDocumentUpdate event: `change.before.data()`). The reduction in data should be more performant, and avoid potential resource limitations.",

default: false,
default: "no",
input: select({ Yes: "yes", No: "no" }),
}),
viewType: defineString("VIEW_TYPE", {
label: "View Type",
Expand Down Expand Up @@ -633,6 +635,12 @@ function normalizePositiveInt(value: string): number | undefined {
return normalized > 0 ? normalized : undefined;
}

// The extension's select emits `yes` / `no`; its label is `Yes`, and the CLI
// copies .env values verbatim, so case and whitespace are forgiven.
function yesNo(value: string): boolean {
Comment thread
cabljac marked this conversation as resolved.
return value.trim().toLowerCase() === "yes";
}
Comment thread
cabljac marked this conversation as resolved.

/** Coerce an empty-string param value to `undefined`. */
function optional(value: string): string | undefined {
return value.length > 0 ? value : undefined;
Expand Down Expand Up @@ -674,8 +682,8 @@ export function configFromEnv(): ExportConfig {
projectId: projectID.value(),
databaseId: optional(params.database.value()) || "(default)",
wildcardIds: params.wildcardIds.value(),
excludeOldData: params.excludeOldData.value(),
useNewSnapshotQuerySyntax: params.useNewSnapshotQuerySyntax.value(),
excludeOldData: yesNo(params.excludeOldData.value()),
useNewSnapshotQuerySyntax: yesNo(params.useNewSnapshotQuerySyntax.value()),
viewType: (optional(params.viewType.value()) || "view") as ViewType,
partitioning: buildPartitioningConfig({
timePartitioning: timePartitioning(tablePartitioning),
Expand Down
47 changes: 38 additions & 9 deletions kits/firestore-bigquery-export/tests/config.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,15 +33,10 @@ vi.mock("firebase-functions/params", () => ({
toJSON(): string {
return this.toString();
}
},
defineString: (
_name: string,
opts?: { default?: string | { value(): string } }
) => ({
value: () =>
typeof opts?.default === "string"
? opts.default
: opts?.default?.value() ?? "",
}, // Mirrors the real StringParam: the env var or "", never the declared
// default, which only the CLI prompt consults.
defineString: (_name: string) => ({
value: () => process.env[_name] || "",
toString: () => `params.${_name}`,
}),
// Mirrors the real IntParam: a missing or blank env var resolves to 0, and
Expand Down Expand Up @@ -166,12 +161,15 @@ describe("buildPartitioningConfig", () => {

describe("configFromEnv", () => {
test("maps params", () => {
// The CLI resolves the param's default into the env at deploy.
vi.stubEnv("BIGQUERY_PROJECT_ID", "test-project");
const config = configFromEnv();
expect(config.projectId).toBe("test-project");
expect(config.bqProjectId).toBe("test-project");
expect(config.databaseId).toBe("(default)");
expect(config.viewType).toBe("view");
expect(resolveExportConfig(config)).not.toHaveProperty("location");
vi.unstubAllEnvs();
});

test("reports unset queue params as undefined so the documented defaults apply", () => {
Expand Down Expand Up @@ -201,6 +199,37 @@ describe("configFromEnv", () => {
vi.unstubAllEnvs();
});

test("yes/no selects default to off", () => {
const config = configFromEnv();
expect(config.useNewSnapshotQuerySyntax).toBe(false);
expect(config.excludeOldData).toBe(false);
});

test("yes/no selects enable on the extension's literal yes", () => {
vi.stubEnv("USE_NEW_SNAPSHOT_QUERY_SYNTAX", "yes");
vi.stubEnv("EXCLUDE_OLD_DATA", "yes");
const config = configFromEnv();
expect(config.useNewSnapshotQuerySyntax).toBe(true);
expect(config.excludeOldData).toBe(true);
vi.unstubAllEnvs();
});
test("yes/no selects forgive the label's case and surrounding whitespace", () => {
vi.stubEnv("USE_NEW_SNAPSHOT_QUERY_SYNTAX", "Yes");
vi.stubEnv("EXCLUDE_OLD_DATA", " yes ");
const config = configFromEnv();
expect(config.useNewSnapshotQuerySyntax).toBe(true);
expect(config.excludeOldData).toBe(true);
vi.unstubAllEnvs();
});

test("yes/no selects treat no and anything else as off", () => {
vi.stubEnv("USE_NEW_SNAPSHOT_QUERY_SYNTAX", "no");
vi.stubEnv("EXCLUDE_OLD_DATA", "true");
expect(configFromEnv().useNewSnapshotQuerySyntax).toBe(false);
expect(configFromEnv().excludeOldData).toBe(false);
vi.unstubAllEnvs();
});

test("exposes deploy-time expressions for trigger metadata", () => {
expect(CONFIG_EXPRESSIONS.collectionPath.toString()).toBe(
"params.COLLECTION_PATH"
Expand Down
Loading