Repository navigation
fix(manager): the QEMU guest boots with network and its fw_cfg arguments - #16
Conversation
The guest's e1000.ko and qemu_fw_cfg.ko were committed copies built for
6.8.0-1064-azure, while the workflow takes the newest build-lite kernel,
now 6.8.0-1070-azure. Both modules fail to load ("disagrees about version
of symbol"), so the guest has no eth0 and never reads manager_args: the
manager runs without the pinned payload, fails to download the newest
one, panics as PID 1, and the kernel panic keeps QEMU running until the
host gives up.
- build.sh takes the kernel and both modules from the same qemu-assets
artifact (KERNEL_ASSETS) and fails when a module's vermagic differs
from the kernel's version; the stale guest/modules copies are gone.
- init no longer execs the manager: when it exits the guest powers off,
so QEMU exits instead of hanging on "Attempted to kill init". Failed
insmods are logged, and the kernel cmdline fallback for the manager
arguments now runs when fw_cfg gave none (its old test never held).
- PrepareWrapper retries a failed payload download with backoff (5 s
doubling to 1 min) instead of panicking.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: LYJW131 <153256373+LYJW131@users.noreply.github.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| err := prepareWrapper(mirror) | ||
| if err == nil { | ||
| return | ||
| } | ||
| log.Errorf("wrapper-lite is not ready: %v; trying again in %s", err, delay) | ||
| time.Sleep(delay) |
There was a problem hiding this comment.
🔴 Invalid payload pin blocks startup indefinitely
When a pinned download fails its digest check, PrepareWrapper retries the same artifact forever. The manager never serves requests, and -prepare never reports failure.
Learn more
The payload digest check in downloadWrapperLite returns errPayloadSHA256 when the fetched bytes differ from the configured pin. prepareWrapper preserves that error even when an older payload is installed. The retry loop treats it as a network failure, so a fixed wrong pin prevents the main process from ever reaching its server or finishing -prepare.
Example: Start with -payload-url pointing at a valid zip and -payload-sha256 set to 64 zeroes. Every successful fetch fails verification; the process waits and fetches again instead of terminating with an error.
Recommended fix: Distinguish terminal validation failures such as errPayloadSHA256 from transient download failures in PrepareWrapper. Propagate or terminate on terminal errors while retaining backoff for transient failures; ensure the -prepare caller exits nonzero.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Fixed in a8eff91: a digest mismatch now returns from PrepareWrapper and main exits 1; other download failures still retry with backoff. Test: TestPrepareWrapperGivesUpOnADigestMismatch.
| /wrapper-manager $MANAGER_ARGS | ||
| echo "[init] wrapper-manager exited with status $?, powering off" | ||
| /bin/busybox poweroff -f |
There was a problem hiding this comment.
🟡 Failed manager exits look successful to host
When /wrapper-manager fails, init powers off the guest without passing its status to the host. runQemu treats that shutdown as success, so launchers report a clean exit.
Learn more
The guest init process now remains alive while wrapper-manager runs. Regardless of the manager's exit status, it calls poweroff -f; runQemu interprets the resulting zero QEMU exit status as a clean shutdown. A manager startup failure can therefore appear successful to host scripts and supervisors.
Example: If the manager exits with status 2 after startup, init logs status 2 and powers off. QEMU exits normally and wrapper-manager-qemu returns 0 instead of signaling that the manager failed.
Recommended fix: Preserve the manager's exit status and communicate failures across the QEMU boundary, such as through an explicit QEMU guest-to-host exit channel handled by the launcher; keep normal poweroff for successful exits.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Not changing this here. amdl, the host that pins this guest, treats any QEMU exit before /status answers as a failed boot regardless of exit status, so the clean poweroff is what it needs. Carrying the manager's status to wrapper-manager-qemu would need a guest-to-host channel (e.g. isa-debug-exit); worth it only if a script depends on that launcher's exit code.
| KERNEL_ASSETS="${KERNEL_ASSETS:?set KERNEL_ASSETS to the unzipped build-lite qemu-assets artifact}" | ||
| KERNEL_ASSETS="$(cd "$KERNEL_ASSETS" && pwd)" |
There was a problem hiding this comment.
🔍 Parent build instructions need updating
The documented guest build command omits mandatory KERNEL_ASSETS. Readers following it stop before building the initramfs; the guest-specific README has the revised steps.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Fixed in a8eff91: manager/README.md now fetches qemu-assets and passes KERNEL_ASSETS.
| magic="$(modinfo -F vermagic "$OVERLAY/$mod.ko" | cut -d' ' -f1)" | ||
| [ "$magic" = "$KERNEL_VER" ] || { echo "$mod.ko is built for $magic, the kernel is $KERNEL_VER"; exit 1; } |
There was a problem hiding this comment.
📝 Info: Vermagic does not verify symbol compatibility
The release-string check cannot detect symbol-version mismatches. Matching artifacts lower that risk, but a matching vermagic alone does not prove the modules will load.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Agreed, vermagic alone is a release check. The modules now come from the same qemu-assets initramfs that build-lite built against that kernel, so symbol versions match by construction; the vermagic check catches a mixed-up artifact, which is the failure that shipped.
Retrying cannot help when the pin itself does not match: PrepareWrapper returns that error and main exits non-zero, while a download that fails for any other reason is still retried with backoff. The parent README's build steps now pass KERNEL_ASSETS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: LYJW131 <153256373+LYJW131@users.noreply.github.com>
… a dead one The pinned guest (run 41583637553) had e1000.ko and qemu_fw_cfg.ko built for another kernel than its own, so it booted without network and without its fw_cfg arguments: the manager ignored the pinned payload, could not download one, panicked as PID 1, and the guest hung in a kernel panic until the supervisor gave up after three minutes, then booted it again. AMDL-Web/wrapper#16 takes the modules from the kernel's own artifact and powers the guest off when the manager exits. - Builtin pins run 41594421666 (3ff150fdbd27), which boots with eth0 up, reads manager_args and installs the pinned wrapper-lite. - The kernel command line has panic=1: with -no-reboot a guest kernel panic now ends QEMU, so a broken guest fails in seconds. - readyTimeout is 30 minutes: the manager downloads the ~50 MB payload before it answers, and at the 40 KB/s measured from China that takes about 21 minutes. Three minutes cut every first boot short, and each retry started the download over. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: LYJW131 <153256373+LYJW131@users.noreply.github.com>
The pinned manager guest (run 41583637553) does not boot usefully: its
e1000.koandqemu_fw_cfg.kowere committed copies built for6.8.0-1064-azure, whilemanager.ymltakes the newest build-lite kernel,6.8.0-1070-azure. Both modules are rejected ("disagrees about version of symbol"), so the guest has noeth0and never readsmanager_args; the manager starts without the pinned payload, cannot download the newest one, panics as PID 1, and the kernel panic keeps QEMU running until the host gives up.build.shtakes the kernel and both modules from oneqemu-assetsartifact (KERNEL_ASSETS) and fails on a vermagic mismatch;guest/modules/is removed.initpowers off when the manager exits instead ofexecing it, logs failedinsmods, and its kernel-cmdline fallback runs when fw_cfg gave no arguments.PrepareWrapperretries a failed payload download with backoff instead of panicking.Checked on an Apple Silicon Mac (TCG) with the 1070 modules and this init swapped into the pinned initramfs:
eth0 carrier: 1, DNS resolves, the args come from fw_cfg and the manager downloads the pinned payload URL.🤖 Generated with Claude Code