Skip to content

kernel: a dead kernel's close reason waits for its last stderr lines - #291

Merged
ymote merged 1 commit into
mainfrom
fix/flaky-kernel-last-words
Oct 2, 2026
Merged

ymote merged 1 commit into
mainfrom
fix/flaky-kernel-last-words

Conversation

@ymote

@ymote ymote commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Fixes the flaky a_kernel_that_dies_closes_its_connections_with_its_last_words (crates/kernel/tests/core.rs:241), which failed apps.yml / services on PR #278 (run 36979961660, job 110752006291) with:

the kernel closed its output: fake kernel up

Cause

The supervisor watches the kernel's stdout and its exit status, but the kernel's stderr is copied into the reason's tail by a task of its own. When the kernel dies, the line it wrote last (fake kernel: asked to exit) and the end of its stdout arrive at almost the same moment. The supervisor saw the EOF first and read the tail at once, before the stderr task had pushed that line, so the close reason lost the kernel's last words. This is an ordering bug, not a slow machine: it needs only the stderr task to be scheduled a moment after the supervisor, which a busy CI runner does now and then.

Fix

  • Io keeps the stderr reader's JoinHandle.
  • After stop() has reaped the kernel, last_words() waits for that reader to reach the end of the pipe, and only then appends the tail. Exited and Failed reasons both go through it. The pipe closes with the kernel, so the wait normally ends at once. It is bounded at 2 s only for a process the kernel started that still holds the pipe. There are no sleeps or retries.
  • The test now asks for the last words themselves (asked to exit), not just any "exit".

Side effects: the "writing to / reading from the kernel failed" reasons now carry the tail too. The stopping kernel N: log line no longer repeats it, since every stderr line is already logged as octos: ….

Evidence

All runs were on the Linux build host, using the core test binary (all 13 tests at once, as CI runs them).

Setup main (b221f7b) this branch
300 runs each, before and after side by side, each pinned to 3 CPUs shared with 3 busy loops 58 of 300 failed, all this test, all the kernel closed its output: fake kernel up 0 of 300
Idle 72-core host, 200 runs 0 of 200 (the race needs contention) —
Pinned to 2, 3 or 4 CPUs beside a cargo build, 300 runs each 14, 15 and 8 failures (all this test) —

cargo test --locked -p octosense-kernel and cargo clippy --locked -p octosense-kernel --all-targets --no-deps -- -D warnings (the apps.yml commands) pass.

🤖 Generated with Claude Code

The supervisor saw the kernel end (its stdout closing, or its exit
status) and read the stderr tail at once, but stderr is copied into the
tail by a task of its own. The line the kernel wrote just before it
exited could still be in the pipe, so the reason lost it: CI's
`a_kernel_that_dies_closes_its_connections_with_its_last_words` failed
with "the kernel closed its output: fake kernel up", without the
"asked to exit" the fake kernel wrote last.

The tail is now read after the kernel is stopped and the stderr task has
reached the end of the pipe (bounded at 2 s, for a process the kernel
started that still holds the pipe). Failed reasons get the same complete
tail. The test asks for the last words themselves, not just "exit".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ymote
ymote force-pushed the fix/flaky-kernel-last-words branch from fd7e2ac to 9586193 Compare October 2, 2026 20:57
@ymote
ymote merged commit cd3905b into main Oct 2, 2026
3 of 5 checks passed
@ymote

ymote commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Merged in a merge train; local CI (Mac with rom toolchain + --linux-host) passed on the train head 72e5d55, which contains this PR.

  • tools/ci-local.sh --only all --linux-host, started 2026-10-02T14:00:45: passed. Desktop, phone and apps passed on the Mac; apps services, kernel-security and linux-host / sandbox passed on the Linux build host. Its rom.yml steps were skipped on the Linux host, and the train doesn't trigger rom.yml.
  • tools/ci-local.sh --only all on the Mac with the rom toolchain (JDK 17, Android SDK, node 24), started 2026-10-02T14:51:21: 42/42 steps passed, none skipped.

The train was #291, #293, #297, #294, #298, #295, #296, #299, #300 and #292, on main f6678b5. After the last merge, main 301be0c has the tested head's tree.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant