Repository navigation
kernel: a dead kernel's close reason waits for its last stderr lines - #291
Merged
Merged
Conversation
The supervisor saw the kernel end (its stdout closing, or its exit status) and read the stderr tail at once, but stderr is copied into the tail by a task of its own. The line the kernel wrote just before it exited could still be in the pipe, so the reason lost it: CI's `a_kernel_that_dies_closes_its_connections_with_its_last_words` failed with "the kernel closed its output: fake kernel up", without the "asked to exit" the fake kernel wrote last. The tail is now read after the kernel is stopped and the stderr task has reached the end of the pipe (bounded at 2 s, for a process the kernel started that still holds the pipe). Failed reasons get the same complete tail. The test asks for the last words themselves, not just "exit". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ymote
force-pushed
the
fix/flaky-kernel-last-words
branch
from
October 2, 2026 20:57
fd7e2ac to
9586193
Compare
Contributor
Author
|
Merged in a merge train; local CI (Mac with rom toolchain + --linux-host) passed on the train head 72e5d55, which contains this PR.
The train was #291, #293, #297, #294, #298, #295, #296, #299, #300 and #292, on main f6678b5. After the last merge, main 301be0c has the tested head's tree. |
This was referenced Oct 2, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the flaky
a_kernel_that_dies_closes_its_connections_with_its_last_words(crates/kernel/tests/core.rs:241), which failedapps.yml / serviceson PR #278 (run 36979961660, job 110752006291) with:Cause
The supervisor watches the kernel's stdout and its exit status, but the kernel's stderr is copied into the reason's tail by a task of its own. When the kernel dies, the line it wrote last (
fake kernel: asked to exit) and the end of its stdout arrive at almost the same moment. The supervisor saw the EOF first and read the tail at once, before the stderr task had pushed that line, so the close reason lost the kernel's last words. This is an ordering bug, not a slow machine: it needs only the stderr task to be scheduled a moment after the supervisor, which a busy CI runner does now and then.Fix
Iokeeps the stderr reader'sJoinHandle.stop()has reaped the kernel,last_words()waits for that reader to reach the end of the pipe, and only then appends the tail.ExitedandFailedreasons both go through it. The pipe closes with the kernel, so the wait normally ends at once. It is bounded at 2 s only for a process the kernel started that still holds the pipe. There are no sleeps or retries.asked to exit), not just any "exit".Side effects: the "writing to / reading from the kernel failed" reasons now carry the tail too. The
stopping kernel N:log line no longer repeats it, since every stderr line is already logged asoctos: ….Evidence
All runs were on the Linux build host, using the
coretest binary (all 13 tests at once, as CI runs them).the kernel closed its output: fake kernel upcargo test --locked -p octosense-kernelandcargo clippy --locked -p octosense-kernel --all-targets --no-deps -- -D warnings(theapps.ymlcommands) pass.🤖 Generated with Claude Code