Skip to content

feat(pathways): resilient cluster start that never stalls the data pump - #1

Merged
jbiskur merged 1 commit into
mainfrom
fix/pathways-resilient-cluster-start
Sep 30, 2026
Merged

jbiskur merged 1 commit into
mainfrom
fix/pathways-resilient-cluster-start

Conversation

@jbiskur

@jbiskur jbiskur commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Summary

  • New skills/flowcore-pathways/references/resilient-startup.md: production start runtime for virtual pathways. It retries the whole start with stopPump() + stopCluster() teardown, exits after the last attempt, exits on Failed to bootstrap leader runtime after becoming leader (detected in the logger adapter), and runs a leader watchdog. It separates liveness from readiness and adds handler timeouts sized to deliveryTimeoutMs.
  • SKILL.md: new rule 11 (a start error must never leave a silent stall). The minimal example now uses a single-attempt startOnce() plus the runtime. Checklist items added.
  • cluster-and-pump.md: why the naive start stalls. Fixes the PostgreSQL-outage row: the leader does not step down, because renewLease throws and the error is logged as Lease loop error.
  • Troubleshooting (both pathway skills) and a new evaluation scenario.
  • Version 0.2.0.

Verification (source)

Every claim was checked against the @flowcore/pathways 2.10.4 source (builder.ts startCluster/startPump/handleLeadershipChange/bootstrapLeaderPump, cluster-manager.ts, postgres-coordinator.ts, pathway-pump.ts) and the @flowcore/data-pump 0.23.3 source (data-pump.ts redelivery and fetch loop). The patterns come from about 40 Flowcore workspace solution and outage fragments on pump stalls.

Test plan

  • node scripts/validate-package.mjs passes (13 checks), and the self-test passes 33/33
  • deno check of pathways-logger.ts, pathways-runtime.ts, the SKILL.md example and the handler-timeout block against the pathways 2.10.4 source
  • Behavior test of the runtime with a fake pathways module:
    • recovers after 2 failed attempts, with teardown between attempts
    • exits 1 after 6 attempts
    • exits 1 on the fatal bootstrap log line
    • the watchdog exits 1 when the leader has no running pump past the grace period
  • CI green

🤖 Generated with Claude Code

A failed startPump() or leader bootstrap left a leader that holds the
lease with no running pump, so no other instance could take over. The
skill now ships a production start runtime: retry the whole start with
stopPump() + stopCluster() teardown, exit after the last attempt, exit
on "Failed to bootstrap leader runtime after becoming leader", and a
leader watchdog. Verified against @flowcore/pathways 2.10.4 and
@flowcore/data-pump 0.23.3 source.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jbiskur
jbiskur merged commit 666fe54 into main Sep 30, 2026
3 checks passed
@jbiskur
jbiskur deleted the fix/pathways-resilient-cluster-start branch September 30, 2026 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant