feat(firestore-bigquery-export): design write-path buffering

[cabljac]

The extension buffers failed BigQuery writes in a Cloud Tasks queue (syncBigQuery, maxAttempts 5, dispatch throttling); on terminal failure it swallows the error, writing the row to BACKUP_COLLECTION only when that optional param is set. The kit has no queue: one self-heal re-attempt, then rethrow with retry: true, so it never loses a row but a persistent BigQuery error replays every write through Eventarc backoff for up to 24h. Aggregate cost scales with write rate x outage duration (each retried invocation performs up to 4 insert attempts plus 2 backup commits), and the flood can saturate instances. Parity ledger: #2974, firestore-bigquery-export §1.

DECISION (2026-09-02): reinstate the Cloud Tasks buffer (option D)

Team decision: port the extension's Cloud Tasks buffer architecture into the kit rather than shipping the retry-shape design below. Rationale: migration safety on our delivery timeframe - migrating users keep exactly the failure behavior and cost profile they have today (about 5 attempts, dispatch-throttled), no billing-spike surprise, no novel classifier/gate code to debug in release candidates. A native platform solution is expected around late 2027, so the sophisticated design solves a problem the platform will eventually own; if needed we follow up in a later version.

Accepted trade-offs, recorded deliberately: the current kit's never-loses-within-24h property goes away (rows drop after the queue's attempts when no backup collection is set - which is extension parity, i.e. the bar), and the known kit queue-leak on function deletion now applies to this kit (document it).

Port-time cautions - fix these legacy bugs while porting, they are bug fixes not divergence:

1. Do NOT port attemptToEnqueue's silent swallow (legacy index.ts:290-322 dropped the event outright when the enqueue itself failed, with no visibility). Log loudly at error level and rethrow so the trigger retry covers the enqueue-failure window.
2. Do NOT reintroduce tracker 2.0.4 semantics: the always-truthy retry guard and the !retry-only backup are fixed in 2.1.0, which the kit consumes. The port must keep 2.1.0's back-up-on-every-terminal-failure behavior.
3. Keep the strongly-recommended BACKUP_COLLECTION framing (prompt description is the delivery vehicle: the CLI prompts every param absent from dotenv, defaulted or not, and hard-errors non-interactively - verified against firebase-tools 15.26.0 lib/deploy/functions/params.js resolveParams).

Considered alternative, revisit in a later version: retry shape + permanent-failure gate (option B - investigated, adversarially reviewed, and red-teamed; key claims verified against source)

Keep the kit's retry-based shape - it is the data-safe baseline - and cap its cost. Since tracker 2.1.0 the tracker writes failed rows to the backup collection on EVERY terminal insert failure, transient or permanent, before rethrowing (2.0.4 only backed up when !retry, masked by the always-truthy guard bug - do not backport against 2.0.4 semantics), keyed idempotently by insertId (= eventId, so redeliveries overwrite one doc). The kit change is therefore "stop rethrowing when the backup is durable and the failure is permanent", not "add a backup write". Family convention this extends (from #3086's terminal-vs-retryable split): terminal failures may be swallowed only when nothing durable is lost.

Dependency status: unblocked. #2937 merged 2026-09-01 and shipped as tracker 2.1.0, which the kit already consumes (^2.1.0). Remaining tracker work: the backed-up signal (a 2.1.1+), plus syncing the kits-branch tracker source copy (currently stale at 2.0.4, large diff vs next).

1. Tracker: expose a typed backed-up signal on the rethrown insert error (set only when the backup commit resolved, attached between backup resolution and the rethrow). Shape: a Symbol.for(...)-keyed property (or a plain string key) plus an exported type-guard predicate - robust to duplicate tracker copies (tests, transitive pins); a module-local Symbol() or an error subclass shares the dual-copy identity weakness. The signal legitimately appears on transient rethrows too; the kit gate is permanent AND signal. Additive; legacy extension behavior unchanged. Also export the tracker's extractInsertErrors so the kit classifier reuses it (e.errors drops location; the tracker reads e.response.insertErrors).

   Implementation notes for the 2.1.1 PR (from the catch block at bigquery/index.ts:357-401):

   • e is not guaranteed to be an object - describeError's own comment says insertData reports "whatever it caught", and the file compiles strict, so e[SIGNAL] = true on a string throws a TypeError that would replace the real insert error. Guard with typeof e === "object" && e !== null before attaching, and attach inside the existing try only after handleFailedTransactions resolves, so a failed backup leaves the property absent rather than half-set.
   • extractInsertErrors is module-private in bigquery/index.ts today; export it through the package root, and capture the kit classifier's test fixtures against the same call so the tracker and kit suites share them.
   • Land on next, then sync the kits-branch tracker copy in the same change window - the 2.0.4 staleness there is the one place the dual-copy risk the Symbol.for choice guards against is currently real.

2. Kit classification, v1 = schema errors only: new errors.ts (following fix(bigquery-firestore-export): stop retrying the two failures the extension treated as terminal #3086's conventions) classifying PartialFailureError rows with reason invalid as permanent; everything else stays transient and rethrows. Request-level errors (403/401 etc.) are deliberately excluded from v1: the self-heal path wraps init-time failures in a plain Error, so permission errors reach the classifier shapeless on cold instances, and accessDenied has an IAM-propagation false-positive window. Widen later with telemetry. Schema drift healing folds into the EXISTING self-heal rather than adding a third attempt: retryAfterSelfHeal invokes a new reset hook on createEnsureInitialized when the first error is schema-shaped (invalid / no-such-field), so its existing re-attempt runs against a really re-initialized table (today init success is memoized forever and the kit's skipInit: true makes the tracker's own _initialized reset unreachable). Classification happens on the self-heal's failure, as it does today - the "4 inserts + 2 backups" ceiling is unchanged. Complementary to, not a copy of, the tracker's schemaLagColumns strip-retry: that path covers columns that exist but are not yet streamable (this instance just added them); the re-init covers columns genuinely missing from the table (dropped, recreated, or initialized elsewhere). The classifier carries a single-row-assumption comment: the kit write path is one event = one row (record([event])), so mixed batches cannot occur here - do not reuse it for the import script's multi-row batches (and note a TRANSFORM_FUNCTION can change the row count).

3. Permanent + backup confirmed: log a constant-message structured error line (stable interface for log-based metrics: message plus backup_collection, event_id, reason fields), fire recordErrorEvent exactly once (retryAfterSelfHeal already emits it before rethrowing; the swallow path must not double-emit), suppress the duplicate complete() log, and swallow. Permanent without confirmed backup (including BACKUP_COLLECTION unset): keep rethrowing - the row has nowhere durable to go. README ships an alert recipe: log-based counter on the parked-row line, a second metric on the tracker's failedBackupWrite line (propose raising it warn -> error in 2.1.1: a failing backup during an outage is the worst path - full retry cost, zero durability), and the built-in execution-failure-rate metric for the storm itself. Optional, cheap: an "event age > ~20h, loss imminent" error computed from event.time (the CloudEvent carries no attempt counter), making the 24h cliff observable in backup-unset or backup-failing installs.

4. MAX_INSTANCES param (defineInt, default 20) on the export trigger. Do not pin concurrency - that would make the cap a hard organic-throughput ceiling users cannot lift by adding memory. Instead document the formula: the effective event cap is MAX_INSTANCES x concurrency, and concurrency is 1 at the kit's default resources but flips to the gen2 default of 80 once memory/cpu reach 1 vCPU (turning "20" into ~1600). The param description must state it also throttles sustained organic traffic (backlogged events share Eventarc's 24h clock).

5. Loop guard at config load: reject a BACKUP_COLLECTION whose document path could match the trigger pattern ${COLLECTION_PATH}/{documentId} - segment-aware comparison (a {wildcard} segment matches any one segment; be conservative with {x=**}: reject unless a literal segment provably differs), never a prefix check. Segment depth alone can never guarantee safety (collection paths are odd, doc paths even), so the comparison is against the actual configured pattern.

6. Recovery recipe is a deliverable, not a footnote: backup rows are changelog-shaped and fs-bq-import-collection cannot consume them. Ship a script (or documented commands): NDJSON export of each backup doc's json fields -> bq load into a temp table with the changelog schema -> MERGE into the raw changelog with WHEN NOT MATCHED on event_id, partition-pruned to the outage window -> verify counts, delete backup docs. The event_id anti-join is MANDATORY: stale backup rows are the norm (every transient blip that later succeeds leaves one - nothing cleans them up, and delete-on-success is rejected: the kit cannot know a backup exists without a read or a blind write per success), and BigQuery streaming insertId dedupe only spans ~1 minute, so it does not protect recovery-time re-inserts. Precondition: permanent-classified rows will fail again unless the schema cause is fixed first.

7. README + param description: rewrite the BACKUP_COLLECTION description as strongly recommended, with the failure model in one sentence ("without it, rows that can never insert are retried for 24 hours and then dropped"). The param keeps its default: "" - verified against firebase-tools 15.26.0 lib/deploy/functions/params.js (resolveParams): the CLI partitions params purely on dotenv presence, so a defaulted param absent from dotenv still prompts interactively (the default only pre-fills the prompt; Enter-through writes the explicit empty back to .env, which optional() maps to unset) and still hard-errors in non-interactive mode. The description text is therefore the delivery vehicle on both deploy paths - the recommended-param middle path needs no new mechanism. Backup rows must be described as "possibly failed" (see recovery recipe), the collection must not live under COLLECTION_PATH (enforced by the loop guard), and the recovery doc explains the reconcile.

Known limits (accepted, documented rather than fixed here)

• transformFunction failures occur before insertData and bypass backup entirely: a >24h transform-endpoint outage loses events even with BACKUP_COLLECTION set. Out of scope for v1; candidate for a later classifier/backup extension.
• A failing backup write (revoked Firestore perms, >1MiB doc) keeps full retry cost with zero durability; mitigated by the failedBackupWrite alert metric, not eliminated.
• Oversized rows can fail BigQuery (request-level 400, retried as transient) and simultaneously exceed Firestore's 1MiB backup-doc limit; after Eventarc's 24h window such a row is lost with no durable copy. Pre-existing in the legacy extension too.
• Default installs (BACKUP_COLLECTION unset) trade against unset-backup legacy directly: worse on cost (24h of redeliveries x write rate vs legacy's ~5 queue attempts in under an hour), better on loss window (24h of chances to recover vs silent loss within the hour). Step 2's re-init still heals schema drift there; the swallow path is inert by design.

Decided

• No dispatch-throttling knob: legacy's queue throttled only the failure path, not healthy traffic, so nothing is lost; revisit on demand.
• No auto-default backup collection in v1; the strongly-recommended prompt (step 7) is the mitigation. Revisit a derived default AFTER the loop guard lands - the guard removes the trigger-loop objection, leaving only surprise-writes. Note the obvious candidate (a top-level _${TABLE_ID}_bq_export_backup) fails its own guard under a top-level-wildcard trigger ({col}/{doc} matches every two-segment path), and those installs need a default most - a revisited default must derive its shape from the configured pattern or be scoped to literal-rooted triggers.

Tests

Classifier tests use captured real BigQuery error fixtures (reason invalid, both no-such-field shapes, backendError, a request-level 403), captured during live validation and shared with the tracker's suites - the PartialFailureError shape is too subtle for hand-built objects (e.errors vs e.response.insertErrors, dropped location). Live validation: induce a schema-permanent failure (retype a changelog column) and a transient one (revoke bigquery.dataEditor), verify the swallow gate in both backup-set and backup-unset installs, the stale-row reconcile, and the MAX_INSTANCES cost cap.

Sequencing: #2937 (merged, shipped as 2.1.0) -> tracker signal + extractInsertErrors export + failedBackupWrite severity as 2.1.1 (tracker source needs syncing across next and kits) -> one kit PR (dep bump, errors.ts, handler, params, loop guard, README, tests) -> recovery script. Watch for overlap with #2817 and #3039 (merged).