Skip to content

feat(firestore-bigquery-export): reinstate the Cloud Tasks write buffer - #3130

Open
cabljac wants to merge 1 commit into
feat/kits-fbe-sync-queue-paramsfrom
feat/kits-fbe-cloud-tasks-write-buffer
Open

feat(firestore-bigquery-export): reinstate the Cloud Tasks write buffer#3130
cabljac wants to merge 1 commit into
feat/kits-fbe-sync-queue-paramsfrom
feat/kits-fbe-cloud-tasks-write-buffer

Conversation

@cabljac

@cabljac cabljac commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Fourth layer of the Cloud Tasks write buffer stack and the behaviour change, per the option D decision on #3031: migrating users keep the failure behaviour and cost profile they have today. Supersedes #3103.

A failed inline write now enqueues onto a new syncBigQuery task queue (5 attempts, 60s minimum backoff, throttled by MAX_DISPATCHES_PER_SECOND) after up to MAX_ENQUEUE_ATTEMPTS in-process attempts, instead of self-healing once and rethrowing to the trigger retry policy. The task handler re-attempts the write and rethrows so Cloud Tasks retries; the tracker parks terminal failures in BACKUP_COLLECTION. The trigger drops its retry policy, and a failed enqueue is logged, published as onError, and dropped, both exactly as the extension does. The function needs roles/cloudtasks.enqueuer. The queue function sets maxInstances to 500 to match the gen1 handler, since the gen2 default of 100 would 429 dispatches and burn attempts.

One deliberate change from the extension: the success event is published after the insert and swallowed on failure, closing a duplicate-row route.

89 tests pass. Not verified live: that the deployed env carries FIREBASE_KIT_INSTANCE_ID and FUNCTION_REGION, that the gen2 task function accepts the SDK's ID token, queue creation with the resolved throttle, and the task size limit on large documents.

Fixes #3031.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request reinstates the Cloud Tasks write buffer for the Firestore BigQuery Export kit, routing failed inline writes to a new syncBigQuery task queue instead of relying on Eventarc redelivery. Feedback suggests introducing a separate terminalTracker in HandlerContext to be used exclusively on the final Cloud Tasks retry attempt. This prevents transient failures from writing premature or duplicate entries to the backup collection.

Comment thread kits/firestore-bigquery-export/src/handlers.ts
Comment thread kits/firestore-bigquery-export/src/handlers.ts
Comment thread kits/firestore-bigquery-export/tests/handlers.test.ts
@cabljac
cabljac force-pushed the feat/kits-fbe-cloud-tasks-write-buffer branch from d228429 to f1c2448 Compare September 7, 2026 16:45
@cabljac
cabljac marked this pull request as ready for review September 7, 2026 16:50
@cabljac
cabljac force-pushed the feat/kits-fbe-cloud-tasks-write-buffer branch from f1c2448 to fd7c0d1 Compare September 7, 2026 16:52
@cabljac cabljac mentioned this pull request Sep 7, 2026
62 tasks
A failed inline write now enqueues onto the syncBigQuery task queue
instead of self-healing once and rethrowing to the trigger retry policy.
The queue retries five times with a 60s minimum backoff, throttled by
MAX_DISPATCHES_PER_SECOND, and the tracker parks terminal failures in
BACKUP_COLLECTION. The trigger drops its retry policy and a failed enqueue
is logged, published as onError, and dropped, as the extension did.

The success event is published after the insert and swallowed on failure,
so a task retry cannot duplicate a row that already landed.
@cabljac
cabljac force-pushed the feat/kits-fbe-cloud-tasks-write-buffer branch from fd7c0d1 to 7b2c7a0 Compare September 7, 2026 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant