Skip to content

test: keep the lifecycle tests stable on a slow runner and cut fixed waits - #311

Merged
ThomasK33 merged 1 commit into
mainfrom
test/lifecycle-load-timing
Nov 4, 2026
Merged

ThomasK33 merged 1 commit into
mainfrom
test/lifecycle-load-timing

Conversation

@ThomasK33

Copy link
Copy Markdown
Member

Why

On a slow macOS runner, merge-group run 41718251342 failed test/integration/lifecycle.test.ts "destroy --force before the host names the program prints its result" after 2 retries. "stale host recovery" needed 2 retries (99 s), and the file took 527 s instead of the usual 230 s. That runner ran every test 2 to 4 times slower than usual: "full lifecycle" took 19 s instead of 5 s. In 50 macOS jobs from 2026-11-03, this was the only failure of that test.

Cause

  1. destroy --force before the host names the program. The test holds the host's start-up write for 4 s, so the host never answers create. create gives up after about 5 s and sends the host SIGTERM. The test passed only while the program's inner destroy --force killed the host before that SIGTERM. That destroy --force needs two Node starts, because it runs itself again in a new session. On a slow runner the SIGTERM came first, and the host shut the session down itself. Then the inner destroy read SESSION_ALREADY_DESTROYED (ok: false), or the host's hang-up killed it before it printed anything. This is not a product bug: a destroy whose session another actor ended reports that.
  2. stale host recovery. The test runs eight CLI commands. On macOS it takes about 10 s, and on a 3 times slower runner it takes about 33 s, which exceeds the file's 30 s per-test time-out.

I reproduced cause 1 on Linux with a preload that delays the start of the host and of each destroy CLI (Atomics.wait). Main fails 3 of 3 with a delay of 1.5 s (ok: false, SESSION_ALREADY_DESTROYED) and 3 of 3 with 2.5 s (no envelope). This branch passes with both delays.

Repro preload (not committed)
// NODE_OPTIONS=--import=file:///abs/slow.mjs SLOW_HOST_MS=1500 SLOW_DESTROY_MS=1500 npx vitest run --retry=0 test/integration/lifecycle.test.ts -t "before the host names the program"
const ms = process.argv.includes('_host')
  ? Number(process.env.SLOW_HOST_MS ?? '0')
  : process.argv.includes('destroy')
    ? Number(process.env.SLOW_DESTROY_MS ?? '0')
    : 0;
if (ms > 0) Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms);

What changes

Only tests and the replay-hold.mjs test helper change. The product code does not change.

  • Early destroy. The test starts create in the background and kills it once the program runs, so no give-up SIGTERM reaches the host. It holds the host's start-up write until the inner destroy --force kills the host. It also checks that childPid stays null, so the destroy really ran before the host named the program. Before, a destroy that ran after the 4 s hold also passed.
  • replay-hold.mjs. REPLAY_RELEASE=<file> ends a hold as soon as the file exists, so the hold time is only a cap. The two "a create that gives up" tests release the host's write once create has exited, instead of after a fixed 10 s. That is about 3 s faster per test, and the test no longer depends on create giving up within 10 s.
  • Fixed sleeps. Four tests waited 0.5 s or 1 s after killing the host. They now wait until the host process has ended (new helper processEnds, and crashSession returns the host's pid). The offline snapshot test waited 1 s for the program's output. It now waits until events.jsonl contains that output.
  • Per-test time-out. The file's per-test time-out is now 60 s instead of 30 s. Many tests spend 5 to 15 s in real waits and CLI starts, and a runner that is 3 times slower needs up to about 35 s for them. A passing test does not get slower.

What I did not cut

Most of the file's real time is product timing: the 5 s destroy wait before HOST_TIMEOUT, the 2 s shutdown grace, and about 0.7 s per CLI start through tsx. The tests that check HOST_TIMEOUT must wait out the destroy wait. Cutting it needs a test-only timing setting in the product code. I did not add one.

Validation

  • tsc --noEmit, oxlint and oxfmt --check pass.
  • On the changed tests after merging main (8064471d, fix: create stops its host and names the session on every error of its readiness wait #294), with no retries, 7 of 7 tests pass. The early-destroy test took 4.2 to 4.7 s on this branch and 6.7 to 7.1 s on main, run back to back on the same host.
  • With slow starts, the early-destroy test passes at 1.5 s and 2.5 s on this branch (5 of 5 runs). Main fails 6 of 6.
  • I ran the whole file with --maxWorkers=1 --retry=2 on this shared host at load averages of 30 to 80. This branch passed 63 of 63 in 491 s. Main failed 1 test in 730 s: "destroy of an exited session from a CLI that sees no /proc, in the pid namespace of the host" hit the 30 s time-out 3 times. On main, the early-destroy test also needed 2 retries.
  • Under that load, both trees needed 2 retries for "destroy ends a child of the program whose main thread exited while another thread runs". I could not reproduce that alone (3 of 3 runs passed), so this PR does not change it.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

…waits

- destroy --force before the host names the program: a held host never
  answers create, which gave up after about 5 s and sent the host SIGTERM.
  On a slow macOS runner that came before the program's destroy, which then
  read SESSION_ALREADY_DESTROYED or died with the program. The test now
  kills create once the program runs and holds the host's write until the
  destroy kills the host.
- replay-hold.mjs: REPLAY_RELEASE=<file> ends a hold early. The create
  give-up tests release the host's write once create has exited instead
  of after a fixed 10 s.
- Tests wait for the crashed host to end instead of a fixed sleep, and
  the offline snapshot test waits for the program's output in the log.
- The file's per-test time-out is 60 s: a runner three times slower than
  usual took 33 s for the stale host recovery test.
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T23:10:39.404736Z ea3e1c3 Manual request
🔒 Security Review ✅ Completed 2026-11-03T23:10:19.224103Z ea3e1c3 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: ea3e1c3b0c

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: ea3e1c3b0c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33
ThomasK33 added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit 4357e94 Nov 4, 2026
13 checks passed
@ThomasK33
ThomasK33 deleted the test/lifecycle-load-timing branch November 4, 2026 00:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant