Skip to content

fix(bigquery-firestore-export): soft failures throw and retry-storm #3008

Description

@cabljac

Originally filed as "soft failures throw and retry-storm" from the #2974 parity analysis (bigquery-firestore-export §7a-7c, §8). The state has moved, so reframing.

Current state

Possible remaining problem

retry: false may trade the storm for silent loss: a transient failure (Firestore blip, timeout) acks and drops the completion notification, and that run's output never reaches Firestore. The extension appears to have the same flaw, so this is parity, but arguably parity with a flaw. With retry: true, v2 pub/sub retries go through Eventarc with exponential backoff (10s to 600s, up to 24h), so retries look bounded rather than stormy - the storm concern seems to only apply to permanent errors, which the #2983 pattern addresses.

Proposed plan (needs agreement, since it goes beyond parity)

  1. Land fix(bigquery-firestore-export): write run output to deterministic ids #2963 (deterministic run-output ids) so redelivery cannot duplicate docs.
  2. Extend the fix(bigquery-firestore-export): stop retrying deploy-time misconfigurations #2983 classification to handleMessagePublished (non-associated config, malformed payload → log and return).
  3. Then flip processMessages to retry: true. Order matters.
  4. Sweep the remaining handler throw sites and classify permanent vs transient.

The lost setProcessingState visibility is out of scope here, tracked in #3035.

Depends on: #2983, #2963, #2985. Parity ledger: #2974.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions