Skip to content

fix(ship): restart empty CI launches once - #2137

Merged
justinhelmer merged 1 commit into
mainfrom
plan/fix-issue-2119-ci-flakes-18260a/u1
Sep 21, 2026
Merged

justinhelmer merged 1 commit into
mainfrom
plan/fix-issue-2119-ci-flakes-18260a/u1

Conversation

@coreplane-switchboard

@coreplane-switchboard coreplane-switchboard Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

The ship runner restarts an empty required-check launch once, while the state Worker test pool no longer leaks Workflow or alarm work between cases. This makes CI recoverable without letting a partial close/reopen leave the pull request closed.

Why: Issue #2119 showed random state Worker failures and ship rounds that could not recover empty CI launches. Record 0055 makes checks part of the round, so PR #2137 closes both failure paths.

Where to look

  1. State Worker test isolation Runs real SQLite Durable Objects without the production Workflow binding and makes actual hangs fail their own test case. ⚠ Test topology must retain every production Durable Object class.
  2. Alarm-loop cleanup proof Exercises native re-ask alarms, then drains the queue and alarm before releasing the shared RPC worker to the next file. ⚠ Leaked alarm RPCs can make an unrelated Worker test fail.
  3. Test-only release boundary Classifies the new Worker setup and wrangler config as tests so they neither trigger deploys nor enter the npm package.
  4. Empty-launch decision Distinguishes a wholly absent required-check launch, grants one grace wait, then requests exactly one event refire.
  5. Retry-safe pull-request refire Retries reopening after an accepted close and throws on exhaustion so Workflow retries rather than recording a completed effect. ⚠ A wrong failure distinction could leave a pull request closed.
  6. Partial-reopen regression proof Proves both a 503 and a timeout are retried until the pull request is restored open.
  7. Ship behavior contract Binds the empty-launch self-heal and retry-safe reopen behavior to executable proofs.

Feedback wanted: Please focus on the reopen failure boundary delegated to Workflow retries, and whether the Worker test cleanup fully closes every cross-case asynchronous boundary.

Risk: 459 changed lines. A mistake can strand a pull request closed or weaken state Worker test fidelity; revert the commit to roll back. Splitting Worker isolation from ship recovery was considered, but both close issue #2119's CI-flake loop.

Verified: Scoped gates passed; CI bot, Worker, package, docs, web, image, and title checks are green. CodeQL analysis remains in progress.

Decisions (4)
  • Leave the production Workflow engine unbound in tests. A cross-script Workflow stub could leave an engine promise alive after its case and make workerd cancel an unrelated request. Live-object doubles preserve boundary tests without sharing asynchronous engine state.
  • Close the native alarm loop inside its owning test. The one-millisecond alarm intentionally re-arms while the resident stays silent. The proof now drains that queue and deletes its alarm before teardown, rather than letting a pending RPC surface under a later test file.
  • Keep test-only Worker assets out of release inputs. The package smoke test exposed that testSetup.ts and wrangler.test.jsonc otherwise shipped as operator assets. Explicit inert patterns preserve their test role without renaming the wrangler config and changing its path semantics.
  • Escalate prolonged reopen failure to the durable step. Returning false after GitHub accepted the close would permanently spend the runner's one refire. Immediate retries cover brief 503s and timeouts; exhaustion throws so the Workflow's existing retry policy resumes the idempotent close/open effect.
Validation (9 criteria)
Criterion Proof
Changed bot behavior and tests pass together npx vitest run with the seven touched bot test files — passed 396 tests.
The cross-file Worker failure is reproduced at its source The nine named state Worker files reproduced the alarm test's 1 ms race and a pending RPC teardown after 202 passes; this drove the owning-test cleanup.
The state Worker alarm test exits without a pending RPC teardown npm test -w deploy/cloudflare-memory -- runLedger.test.ts — passed 58 tests after the cleanup.
Test-only Worker assets are omitted from package assets npm test -w packages/switchboard -- build.test.mts — passed 14 tests.
Touched TypeScript remains type-safe NODE_OPTIONS=--max-old-space-size=6144 npx tsc --noEmit for root and packages/switchboard tsconfigs — both passed.
Changed source and documentation follow formatting npx prettier --check over changed .ts, .mts, .md, and .jsonc files — passed.
Hygiene and specification bindings remain valid npm run hygiene:check and npm run specs:check — passed; 50 specs and 5,590 proof references resolved.
PR title is an accepted changelog line npm run check:pr-title -- "fix(ship): restart empty CI launches once" — passed.
Pushed branch passes required project CI At head 35ca0dc, bot, Worker, package, docs, web, image, deploy-target, and title checks completed successfully; CodeQL remained in progress.
For agents

Head 35ca0dc is one coherent commit rebased onto origin/main e392e08. Review fix F1 is in src/execution/githubPulls.ts and its adjacent tests. CI on prior heads exposed test-only package assets and a still-live one-millisecond alarm RPC; both boundaries now have direct tests.
Requested by @justinhelmer in slack:C0BRRHKFLCB

🤖 Generated with Claude Code

@coreplane-switchboard coreplane-switchboard Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested: The refire recovery can strand the pull request closed when the reopen request fails.

Warning

Changes requested · head cc9fd12 · 1 finding: 1 major

Severity Finding Where
major F1 Spec contradiction — docs/reference/specs/agent-ship.md item 9: a failed reopen can leave the pull request closed src/execution/githubPulls.ts:226
Full review

F1: If the close PATCH succeeds but the reopen PATCH returns 503 or times out, refirePullRequestEvent returns false while leaving the pull request closed. The coordinator then permanently spends its single refire attempt, so the pipeline cannot recover and may stop on the closed PR. Retry or otherwise guarantee the reopen after a successful close; update the partial-failure test to prove the PR is restored to open.

@coreplane-switchboard
coreplane-switchboard Bot force-pushed the plan/fix-issue-2119-ci-flakes-18260a/u1 branch 2 times, most recently from 7cd8c68 to 0dff9b0 Compare September 21, 2026 17:11
Co-Authored-By: coreplane-switchboard[bot] <318072483+coreplane-switchboard[bot]@users.noreply.github.com>
@coreplane-switchboard
coreplane-switchboard Bot force-pushed the plan/fix-issue-2119-ci-flakes-18260a/u1 branch from 0dff9b0 to 35ca0dc Compare September 21, 2026 17:24

@coreplane-switchboard coreplane-switchboard Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM: The reopen recovery now keeps the durable refire step incomplete until the pull request is restored open, resolving F1.

Note

Approved · head 35ca0dc · no findings

Full review

F1 is resolved: failed reopen attempts now leave the durable Workflow step incomplete for retry rather than spending the refire while the pull request remains closed. No new findings.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved: coreplane-switchboard[bot] reviewed this PR and posted an LGTM verdict (see its review). This repository opted in through its REVIEW_BOT_LOGIN and REVIEW_BOT_ID variables.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant