Skip to content

docs(firestore-bigquery-export): document failure handling and recovery #3131

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cabljac merged 1 commit into from
Sep 8, 2026
+162 −37
Merged

docs(firestore-bigquery-export): document failure handling and recovery #3131

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
199 changes: 162 additions & 37 deletions kits/firestore-bigquery-export/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ BigQuery Firebase Extension as an npm package you add to your own Firebase
Functions codebase and deploy.

It listens for document writes on a collection, serializes each change, and
writes it to a BigQuery changelog table. Failed writes are retried through a
Firebase Functions runtime retry policy. The functions run in your own Firebase
project; there is no hosted version, so you deploy them yourself.
writes it to a BigQuery changelog table. Failed writes buffer through a Cloud
Tasks queue (`syncBigQuery`), which retries them on its own throttled schedule.
The functions run in your own Firebase project; there is no hosted version, so
you deploy them yourself.

## Install

Expand All @@ -30,6 +31,7 @@ conflicts with that automatic setup.
| `roles/datastore.user` | write failed-row records back to Firestore (only if you configure a backup collection) |
| `roles/eventarc.eventReceiver` | receive Gen2 Firestore trigger events |
| `roles/run.invoker` | allow Eventarc to invoke the Gen2 Cloud Run service |
| `roles/cloudtasks.enqueuer` | enqueue failed writes onto the kit's own `syncBigQuery` task queue |
| `bigquery.googleapis.com` | mirror Firestore collection changes in BigQuery |

If the dataset lives in a different project (`BIGQUERY_PROJECT_ID`), grant the
Expand All @@ -38,12 +40,13 @@ CMEK dataset, also grant the BigQuery service account access to your KMS key.

## Usage

Export the three functions from your functions codebase entry:
Export the four functions from your functions codebase entry:

```ts
// functions/src/index.ts
export {
fsexportbigquery,
syncBigQuery,
initBigQuerySync,
setupBigQuerySync,
} from "@firebase-function-kits/firestore-bigquery-export";
Expand All @@ -59,6 +62,7 @@ DATABASE_REGION=europe-west2
```

- `fsexportbigquery` is the Firestore trigger.
- `syncBigQuery` is the write-buffer task queue that retries failed writes.
- `initBigQuerySync` is the first-deploy provisioning lifecycle task.
- `setupBigQuerySync` is the reconfigure provisioning lifecycle task.

Expand Down Expand Up @@ -87,8 +91,15 @@ later, behind the `kits` experiment):
`instances` maps each instance id to the directory (relative to
`firebase.json`) holding that instance's `.env`. The CLI prefixes every
function and task queue name with `kit-<instance id>-`, so the functions above
deploy as `kit-default-fsexportbigquery`, `kit-default-initBigQuerySync`, and
`kit-default-setupBigQuerySync`.
deploy as `kit-default-fsexportbigquery`, `kit-default-syncBigQuery`,
`kit-default-initBigQuerySync`, and `kit-default-setupBigQuerySync`.

Deploy with Firebase CLI 15.28.0 or later: it sets the
`FIREBASE_KIT_INSTANCE_ID` env var on the deployed functions, which the trigger
needs to address its own `syncBigQuery` queue. On functions deployed with an
older CLI, enqueues fail (logged at error level and published as an `onError`
event; the event is dropped, as in the extension) until you redeploy with a
newer CLI.

```sh
firebase experiments:enable kits
Expand All @@ -112,7 +123,9 @@ loads them at deploy time and prompts for any required values that are missing.
| `datasetLocation` | `DATASET_LOCATION` | no | `us` | BigQuery dataset location |
| `database` | `DATABASE` | no | `(default)` | Firestore database id |
| `bigqueryProjectId` | `BIGQUERY_PROJECT_ID` | no | project id | Dataset project, if different |
| `backupCollection` | `BACKUP_COLLECTION` | no | (empty) | Firestore collection for failed rows |
| `backupCollection` | `BACKUP_COLLECTION` | no | (empty) | Strongly recommended: collection for rows whose BigQuery insert failed |
| `maxDispatchesPerSecond` | `MAX_DISPATCHES_PER_SECOND` | no | `100` | `syncBigQuery` queue dispatch rate (1-500) |
| `maxEnqueueAttempts` | `MAX_ENQUEUE_ATTEMPTS` | no | `3` | In-process enqueue attempts before giving up (1-10) |
| `transformFunction` | `TRANSFORM_FUNCTION` | no | (empty) | Optional transform Cloud Function |
| `tablePartitioning` | `TABLE_PARTITIONING` | no | `NONE` | Table partitioning strategy |
| `timePartitioningField` | `TIME_PARTITIONING_FIELD` | no | (empty) | Time-partitioning column name |
Expand All @@ -127,6 +140,8 @@ loads them at deploy time and prompts for any required values that are missing.
| `refreshIntervalMinutes` | `REFRESH_INTERVAL_MINUTES` | no | (empty) | Materialized view refresh interval |
| `kmsKeyName` | `KMS_KEY_NAME` | no | (empty) | CMEK key for the dataset |
| `logLevel` | `LOG_LEVEL` | no | `info` | `debug`, `info`, `warn`, `error`, `silent` |
| (env only) | `EVENTARC_CHANNEL` | no | (empty) | Eventarc channel to publish lifecycle events on; unset disables events |
| (env only) | `EXT_SELECTED_EVENTS` | no | (empty) | Comma-separated allowlist of event types to publish (see Events) |

## Multiple instances

Expand Down Expand Up @@ -154,11 +169,19 @@ the instances cannot collide.

## Events

When `EVENTARC_CHANNEL` is configured, the function publishes `onStart` and
`onError` lifecycle events under
`firebase.extensions.firestore-bigquery-export.v1.*`. The extension's
`onSuccess` event is not published; see the events entry under
"Differences from the Stream Firestore to BigQuery extension" below.
When `EVENTARC_CHANNEL` is configured, the functions publish lifecycle events
under `firebase.extensions.firestore-bigquery-export.v1.*`: `onStart` and
`onError` from the write path, and `onSuccess` from the `syncBigQuery` task
Comment thread
cabljac marked this conversation as resolved.
when a buffered write lands (matching the extension, which only emitted
`onSuccess` from its queue handler).

Publishing is filtered by `EXT_SELECTED_EVENTS`: the value is split on commas
and only exactly matching event types are published, silently. An empty value
suppresses every event, and a value carrying only another product's types
(the extension offered more than one namespace to tick) publishes nothing. A
config exported from the extension brings its `EXT_SELECTED_EVENTS` along, so
check it lists the `firebase.extensions.firestore-bigquery-export.v1.*` types
you expect, `onSuccess` included.

## Provisioning

Expand Down Expand Up @@ -216,10 +239,108 @@ curl -fsS -X POST -H "Content-Type: application/json" -d '{"data":{}}' \
```

The Firestore write path never provisions on the hot path. If resources are
missing when a write arrives, the inline write fails, the handler calls
`ensureInitialized()` once as a self-heal and retries the write, and a remaining
failure is surfaced to the function runtime retry policy (`retry: true` on
`fsexportbigquery`).
missing when a write arrives, the inline write fails and the change buffers
through the `syncBigQuery` queue, which re-attempts the write on Cloud Tasks'
schedule. The queue handler does not provision, as in the extension: if the
resources are still missing the retries fail and the row lands in
`BACKUP_COLLECTION`; run the lifecycle task (redeploy) to recreate them.

## Failure handling

The write path mirrors the extension's Cloud Tasks buffer:

1. The trigger attempts the BigQuery insert inline. On success, done.
2. On failure, it enqueues the serialized change onto the `syncBigQuery` queue
(up to `MAX_ENQUEUE_ATTEMPTS` in-process attempts with backoff, keyed by
event id so a retried enqueue cannot buffer the same event twice) and the
execution succeeds. The trigger declares no retry policy, as in the
extension: a failure _before_ the write is attempted (serializing the
change, publishing the `onStart` event) fails the execution once and the
event is not redelivered.
3. `syncBigQuery` re-attempts the write on the queue's schedule: 5 attempts,
60 seconds minimum backoff, throttled to `MAX_DISPATCHES_PER_SECOND`
dispatches per second (500 concurrent max).
4. On every terminal insert failure the tracker writes the row to
`BACKUP_COLLECTION` (when configured), keyed by the event id, before the
task fails. After the fifth attempt the task is dropped. **Without a backup collection, the row is dropped with the task** -
configure `BACKUP_COLLECTION`.
5. If the enqueue itself fails (BigQuery AND Cloud Tasks both failing), the
trigger logs at error level, publishes an `onError` event, and the
execution succeeds: the event is dropped, exactly as the extension did in
this window.

### Recovering parked rows

Rows in `BACKUP_COLLECTION` are changelog-shaped documents, not plain document
snapshots, so `fs-bq-import-collection` cannot consume them. Treat them as
"possibly failed": a transient failure that later succeeded on retry also
leaves one behind, and nothing cleans them up. To recover after an outage,
load the backup docs into a temp table and `MERGE` them into the changelog
table with a `WHEN NOT MATCHED` condition on `event_id` (the anti-join is
mandatory because of those stale rows).

Each backup document is keyed by the event id and shaped like the streaming
insert row the tracker sent, plus the error:

```json
{
"insertId": "<event id>",
"json": {
"timestamp": "...",
"event_id": "<event id>",
"document_name": "...",
"document_id": "...",
"operation": "CREATE",
"data": "<JSON string>",
"old_data": "<JSON string or null>",
"path_params": "<JSON string, only with WILDCARD_IDS>"
},
"error_details": "..."
}
```

The changelog columns sit under `json`, not at the top level. Load the `json`
objects of the backup documents into a temp table with the changelog's schema
(for example by exporting the collection and running `bq load` on the `json`
field), then:

```sql
MERGE `<project>.<dataset>.<table>_raw_changelog` AS target
USING `<project>.<dataset>.<temp table>` AS backup
ON target.event_id = backup.event_id
WHEN NOT MATCHED THEN
INSERT (timestamp, event_id, document_name, document_id, operation, data, old_data)
VALUES (backup.timestamp, backup.event_id, backup.document_name,
backup.document_id, backup.operation, backup.data, backup.old_data)
```

Add `path_params` and any partition column to both lists if your table has
them.

### Known limits

- A task queue is a project-level resource created for each task function.
Deleting the functions (or moving them to another region) disables the old
queue rather than removing it; it shows as `DISABLED` in the Cloud Tasks
console until you delete it there.
- A row whose insert still fails on the last queue attempt with no
`BACKUP_COLLECTION` configured is gone. This matches the extension; it is
the reason the backup collection is strongly recommended.
- The changelog can carry a duplicate `event_id`. BigQuery's `insertId`
dedupe on streaming inserts is best effort for about a minute and the
queue's minimum backoff is 60 seconds, so an insert that landed but reported
an error can be written again by the retry. The `_raw_latest` view keys on
`document_name` and takes the newest change, so duplicates do not affect it;
the `MERGE` above assumes them.
- `BACKUP_COLLECTION` captures rows whose BigQuery insert fails. A failure
earlier in the tracker, such as a `TRANSFORM_FUNCTION` endpoint that is down
or returns malformed JSON, throws before the insert and is not backed up.
Same as the extension.
- A change whose serialized payload exceeds the Cloud Tasks task size limit
(1 MB) cannot be enqueued: the row is logged and dropped, and never reaches
`BACKUP_COLLECTION`. An update carries both `data` and `old_data`, so large
documents get there first; `EXCLUDE_OLD_DATA=yes` halves the payload. Same as
the extension.

## Differences from the Stream Firestore to BigQuery extension

Expand All @@ -234,23 +355,25 @@ boolean params, and only the literal string `true` enables them. The extension
used `yes` / `no` for the last two, so copying an old config across leaves them
silently disabled. Change any `yes` to `true` in your `.env`.

### Failed writes retry differently
### Failed writes: same buffer

The extension pushed a failed BigQuery write onto a Cloud Tasks queue
(`syncBigQuery`) and retried it from there. This kit has no task queue on the
write path. A failed write is retried once in place, and anything still failing
is handed to the Cloud Functions runtime retry policy, which redelivers the
Firestore event.
The kit keeps the extension's write-path architecture: a failed BigQuery write
buffers through the `syncBigQuery` Cloud Tasks queue, with the same shape (5
attempts, 60s minimum backoff, `MAX_DISPATCHES_PER_SECOND` throttling) and the
same knobs (`MAX_DISPATCHES_PER_SECOND`, `MAX_ENQUEUE_ATTEMPTS`) - your
migrated `.env` values carry over unchanged.

The practical effects: retries no longer show up as a separate function or
queue in the console, and the two knobs that tuned that queue,
`MAX_DISPATCHES_PER_SECOND` and `MAX_ENQUEUE_ATTEMPTS`, no longer exist.
When the enqueue itself fails, the kit does what the extension does: logs at
error level, publishes an `onError` event, and drops the event. The trigger
declares no retry policy, so nothing is redelivered through Eventarc.

### Events
Earlier release candidates of this kit had no queue: they retried every failed
write through Eventarc redelivery for up to 24 hours and never lost a row
inside that window. That property is gone by design - a row that exhausts the
queue without a configured `BACKUP_COLLECTION` is dropped, exactly as in the
extension. Set `BACKUP_COLLECTION`.

`onSuccess` is no longer published. The extension emitted it from the task
queue handler, which is gone, so the kit publishes `onStart` and `onError`
only.
### Events

Events are published under `firebase.extensions.firestore-bigquery-export.v1.*`
only. The extension also published a duplicate copy of every event under
Expand Down Expand Up @@ -321,14 +444,16 @@ BigQuery changelog table, so they still work against data this kit writes.
## API surface

- **Main entry** (`@firebase-function-kits/firestore-bigquery-export`): exports
`fsexportbigquery`, `initBigQuerySync`, and `setupBigQuerySync`, and
registers the first-deploy / redeploy provisioning hooks. Runtime config is
resolved lazily on first invocation. Use this entry from Firebase
deploy/emulator/runtime. For your own triggers, import from `./lib` instead.
- **Library entry** (`./lib`): `handleDocumentWrite`, the raw handler for owning
trigger registration yourself, plus the config types and helpers
(`ExportConfig`, `resolveExportConfig`, `toTrackerConfig`) for building its
injected `HandlerContext`. Safe to import anywhere.
`fsexportbigquery`, `syncBigQuery`, `initBigQuerySync`, and
`setupBigQuerySync`, and registers the first-deploy / redeploy provisioning
hooks. Runtime config is resolved lazily on first invocation. Use this entry
from Firebase deploy/emulator/runtime. For your own triggers, import from
`./lib` instead.
- **Library entry** (`./lib`): `handleDocumentWrite` and
`handleSyncBigQueryTask`, the raw handlers for owning trigger registration
yourself, plus the config types and helpers (`ExportConfig`,
`resolveExportConfig`, `toTrackerConfig`, `SerializedDocumentChange`) for
building their injected `HandlerContext`. Safe to import anywhere.

The change-tracker engine is an internal dependency and is not exported.

Expand Down
Loading