Skip to content

fix(manager): report a spawned run as finished, not failed_to_start - #48

Merged
kp2pml30 merged 1 commit into
genlayerlabs:feat/manager-socket-trainfrom
HusseinAdeiza:fix/failed-to-start-after-spawn
Oct 6, 2026
Merged

kp2pml30 merged 1 commit into
genlayerlabs:feat/manager-socket-trainfrom
HusseinAdeiza:fix/failed-to-start-after-spawn

Conversation

@HusseinAdeiza

Copy link
Copy Markdown
Contributor

Fixes #14

Problem

The protocol distinguishes failed_to_start from finished by whether a process was created, not by which code path noticed the error:

failed_to_start Terminal. The run never reached a spawned executor: request validation, version resolution, or the spawn itself failed.

The manager did not track that fact. supervise_genvm reported every error returned by supervise_genvm_inner as fail_to_start.

That misreports a real race in run_genvm_process. The stdin select! has two arms that can fire at nearly the same instant once the child has exited:

  • the stdin writer returning EPIPE (failed to write execution data to child stdin), and
  • child.wait() completing normally.

Whichever loses the race returns an error, so the host is told no process ever ran even though one did. The caller cannot observe which arm won, and the reported terminal event depended on it.

The existing system test at tests/system/manager-socket/test.py only passes because it happens to lose the race in the expected direction, which is why it reads as load-bearing on that race.

Fix

Record the spawned state at the one point where it is known, immediately after proc.spawn() returns, and let that decide the terminal event.

There is no exit status to report when the failure precedes child.wait(), so the finished event carries a null exit_code. The protocol allows this: exit_code is documented as null when a run was killed before its exit code was known. The error still goes to the operator log, since finished has no field to carry it.

Failures that happen before proc.spawn() returns are unchanged: they are still failed_to_start.

Verification

The test drives report_run_failure, the function production calls for every error out of a run, so it covers the classification rather than a reimplementation of it. Neutralising the branch to the old behaviour fails it with exactly the reported symptom:

spawned run reported Event(FailedToStart { genvm_id: GenVMId(1), ...,
error: "failed to write execution data to child stdin: EPIPE" }),
expected a finished event

The test also keeps a control case (the same error with no process is still failed_to_start), so it would still fail if the branch were inverted and every failure became a finished.

  • implementation lib suite: 80 passed, 0 failed
  • cargo fmt --check: clean

Built against v0.6-dev, which is where this ships.

Not included

I did not touch the system test. Making the race deterministic at that level requires an execution-data body too large for the pipe buffer, which is a larger change to shared test infrastructure and reads as a separate concern from the classification fix. Flagged on the issue in case you would rather the protocol-level test also stopped depending on which arm wins.

@coderabbitai

coderabbitai Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (3)
  • main
  • v\d+.\d+
  • v\d+.\d+-dev

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository: genlayerlabs/genvm-manager/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f5af2ad8-86bf-4c4a-94c3-3b3c35534bf7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

GenVM PR actions

Tick a box to run it (the box unticks itself when handled). Actions only run while the PR has the ci-safe label.

  • Force run full tests
  • Provision executor PRs
Commands
  • /genvm-run-tests — run full tests once for the current manager snapshot
  • /merge — queue the exact manager snapshot through the App-owned E2E merge train

@github-actions github-actions Bot added the not rebased branch is behind its base; rebase before it can be merged label Sep 30, 2026
The protocol separates `failed_to_start` from `finished` by whether a
process was created, not by which code path noticed the error. The
manager did not track that fact: `supervise_genvm` reported every error
out of `supervise_genvm_inner` as `failed_to_start`.

That misreported a real race. `run_genvm_process` selects between the
stdin writer and `child.wait()`. If the executor rejects a flag it exits
immediately, so the writer gets `EPIPE` and `child.wait()` completes at
nearly the same moment. Whichever loses the race returned an error, and
the host was told no process ever ran, even though one did. Which of the
two events a caller saw depended on a race it cannot observe.

Record the spawned state at the one point where it is known, immediately
after `proc.spawn()` returns, and let that decide the terminal event.

There is no exit status to report when the failure precedes
`child.wait()`, so the `finished` event carries a null `exit_code`, which
the protocol allows for a run killed before its exit code was known. The
error still goes to the log, since `finished` has no field for it.

Refs genlayerlabs#14

(cherry picked from commit 5550c18)
@HusseinAdeiza
HusseinAdeiza force-pushed the fix/failed-to-start-after-spawn branch from 5550c18 to 5d64e72 Compare September 30, 2026 05:36
@github-actions github-actions Bot added not rebased branch is behind its base; rebase before it can be merged and removed not rebased branch is behind its base; rebase before it can be merged labels Sep 30, 2026
@kp2pml30
kp2pml30 changed the base branch from v0.6-dev to feat/manager-socket-train October 6, 2026 09:22
@github-actions github-actions Bot removed the not rebased branch is behind its base; rebase before it can be merged label Oct 6, 2026
@kp2pml30
kp2pml30 merged commit 1dff5e2 into genlayerlabs:feat/manager-socket-train Oct 6, 2026
6 checks passed
@kp2pml30

kp2pml30 commented Oct 6, 2026

Copy link
Copy Markdown
Member

Thanks! Landed via the feat/manager-socket-train train (#58), reworked after review with you credited as co-author; it reaches v0.6-dev when #58 merges

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants