Repository navigation
fix(pump): retry leader and source bootstrap failures - #128
Merged
Merged
Conversation
When an instance won the leader lease and registering the virtual pathway failed (e.g. a 502 from the control plane for longer than the provisioning retries cover), bootstrapLeaderPump stopped the pump and threw. The error was only logged, while the cluster manager kept renewing the lease, so no instance ran a pump until the process restarted. Retry the bootstrap with backoff (1s, 2s, 5s, 10s, then every 30s) for as long as this instance is still the current leader. A leadership change bumps a generation counter so stale retry loops stop. Seen in production on klokkan: events were written but not delivered for about five hours. Co-Authored-By: Claude <noreply@anthropic.com>
olavurellefsen
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Transient failures during a leader change can leave a live lease with no working source pumps. A source that throws during initial bootstrap also leaves a partially started instance pulsing, and prevents later sources from starting.
The builder retries leader bootstrap while its leadership generation remains current. Each source now stops and discards a partially started pump, retries with capped exponential backoff, and lets healthy peer sources start. Losing leadership or stopping the runtime cancels pending retries. Existing checkpoint factories and cursor positions are preserved.
Verification: full PostgreSQL-backed
deno test -A: 102 passed (253 steps), zero failures. Focused initial-source and leadership retry tests: 4 passed (19 steps).deno lint,deno check src/mod.ts, and the npm ESM/CJS build pass. Added an explicit Node process import required by the current Deno linter in an existing Bun regression test.Production evidence: compute-api and compute-reconciler were independently recovered by rolling their existing versions on 2026-11-01. All seven subscribed sources became live with fresh pulses and empty buffers, without resetting saved checkpoints. This PR is permanent prevention; its release still needs adoption by those applications.