Repository navigation
fix(manager): report a spawned run as finished, not failed_to_start - #48
Merged
kp2pml30 merged 1 commit intoOct 6, 2026
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review (3)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
GenVM PR actionsTick a box to run it (the box unticks itself when handled). Actions only run while the PR has the
Commands
|
The protocol separates `failed_to_start` from `finished` by whether a process was created, not by which code path noticed the error. The manager did not track that fact: `supervise_genvm` reported every error out of `supervise_genvm_inner` as `failed_to_start`. That misreported a real race. `run_genvm_process` selects between the stdin writer and `child.wait()`. If the executor rejects a flag it exits immediately, so the writer gets `EPIPE` and `child.wait()` completes at nearly the same moment. Whichever loses the race returned an error, and the host was told no process ever ran, even though one did. Which of the two events a caller saw depended on a race it cannot observe. Record the spawned state at the one point where it is known, immediately after `proc.spawn()` returns, and let that decide the terminal event. There is no exit status to report when the failure precedes `child.wait()`, so the `finished` event carries a null `exit_code`, which the protocol allows for a run killed before its exit code was known. The error still goes to the log, since `finished` has no field for it. Refs genlayerlabs#14 (cherry picked from commit 5550c18)
HusseinAdeiza
force-pushed
the
fix/failed-to-start-after-spawn
branch
from
September 30, 2026 05:36
5550c18 to
5d64e72
Compare
kp2pml30
merged commit Oct 6, 2026
1dff5e2
into
genlayerlabs:feat/manager-socket-train
6 checks passed
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #14
Problem
The protocol distinguishes
failed_to_startfromfinishedby whether a process was created, not by which code path noticed the error:The manager did not track that fact.
supervise_genvmreported every error returned bysupervise_genvm_innerasfail_to_start.That misreports a real race in
run_genvm_process. The stdinselect!has two arms that can fire at nearly the same instant once the child has exited:EPIPE(failed to write execution data to child stdin), andchild.wait()completing normally.Whichever loses the race returns an error, so the host is told no process ever ran even though one did. The caller cannot observe which arm won, and the reported terminal event depended on it.
The existing system test at
tests/system/manager-socket/test.pyonly passes because it happens to lose the race in the expected direction, which is why it reads as load-bearing on that race.Fix
Record the spawned state at the one point where it is known, immediately after
proc.spawn()returns, and let that decide the terminal event.There is no exit status to report when the failure precedes
child.wait(), so thefinishedevent carries a nullexit_code. The protocol allows this:exit_codeis documented as null when a run was killed before its exit code was known. The error still goes to the operator log, sincefinishedhas no field to carry it.Failures that happen before
proc.spawn()returns are unchanged: they are stillfailed_to_start.Verification
The test drives
report_run_failure, the function production calls for every error out of a run, so it covers the classification rather than a reimplementation of it. Neutralising the branch to the old behaviour fails it with exactly the reported symptom:The test also keeps a control case (the same error with no process is still
failed_to_start), so it would still fail if the branch were inverted and every failure became afinished.implementationlib suite: 80 passed, 0 failedcargo fmt --check: cleanBuilt against
v0.6-dev, which is where this ships.Not included
I did not touch the system test. Making the race deterministic at that level requires an execution-data body too large for the pipe buffer, which is a larger change to shared test infrastructure and reads as a separate concern from the classification fix. Flagged on the issue in case you would rather the protocol-level test also stopped depending on which arm wins.