Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 11 additions & 9 deletions .github/workflows/manager.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,21 +50,23 @@ jobs:
sudo apt-get update
sudo apt-get install -y busybox-static cpio gzip e2fsprogs

- name: Fetch the build-lite kernel and its modules
run: |
# The kernel (vmlinuz-lite-qemu) is produced by the build-lite
# workflow. The e1000 and qemu_fw_cfg modules come from the same
# artifact's initramfs, so they are built for that kernel.
curl -fsSL -o qemu-assets.zip \
"https://nightly.link/AMDL-Web/wrapper/workflows/build-lite/lite/qemu-assets.zip"
unzip -o -q qemu-assets.zip vmlinuz-lite-qemu lite-initramfs.cpio.gz -d kernel-assets/

- name: Build guest assets
env:
KERNEL_ASSETS: kernel-assets
run: |
chmod +x guest/build.sh guest/init
./guest/build.sh dist-guest
CREATE_DATA_IMG=1 ./guest/build.sh dist-guest

- name: Fetch upstream kernel
run: |
# The kernel (vmlinuz-lite-qemu) is produced by the upstream wrapper
# build-lite workflow; fetch it via nightly.link so the guest image
# package is complete.
curl -sL -o qemu-assets.zip \
"https://nightly.link/AMDL-Web/wrapper/workflows/build-lite/lite/qemu-assets.zip"
unzip -o -q qemu-assets.zip vmlinuz-lite-qemu -d dist-guest/

- name: Upload guest assets
uses: actions/upload-artifact@v4
with:
Expand Down
14 changes: 7 additions & 7 deletions manager/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -276,13 +276,13 @@ The launcher is modeled after upstream
# 1. the launcher (any platform)
go build -o wrapper-manager-qemu ./cmd/wrapper-manager-qemu

# 2. the guest initramfs (on Linux; see guest/build.sh)
./guest/build.sh out # builds wrapper-manager-initramfs.cpio.gz
CREATE_DATA_IMG=1 ./guest/build.sh out # also create a fresh data.img

# 3. the kernel is reused from the upstream wrapper qemu-assets artifact:
# https://nightly.link/AMDL-Web/wrapper/workflows/build-lite/lite/qemu-assets.zip
# (vmlinuz-lite-qemu). Put it next to the initramfs in the assets dir.
# 2. the guest (on Linux; see guest/README.md). The kernel and its e1000 and
# qemu_fw_cfg modules come from the build-lite qemu-assets artifact.
curl -fsSL -o qemu-assets.zip \
"https://nightly.link/AMDL-Web/wrapper/workflows/build-lite/lite/qemu-assets.zip"
unzip -o qemu-assets.zip vmlinuz-lite-qemu lite-initramfs.cpio.gz -d kernel-assets/
KERNEL_ASSETS=kernel-assets ./guest/build.sh out # kernel + initramfs
KERNEL_ASSETS=kernel-assets CREATE_DATA_IMG=1 ./guest/build.sh out # also a fresh data.img
```

Prebuilt artifacts (launchers for linux/windows/macos + guest image) are
Expand Down
36 changes: 18 additions & 18 deletions manager/guest/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,7 @@ runs on Windows / macOS / any Linux host (see the parent README).
```text
guest/
├── build.sh # builds wrapper-manager-initramfs.cpio.gz (+ optional data.img)
├── init # guest init script (busybox sh)
└── modules/ # kernel modules reused from the upstream wrapper initramfs
├── e1000.ko
└── qemu_fw_cfg.ko
└── init # guest init script (busybox sh)
```

`overlay/` is a build output (gitignored): build.sh rebuilds it from scratch
Expand All @@ -22,34 +19,37 @@ certificates and the kernel modules.
## Build

```shell
# requires: go, cpio, gzip, e2fsprogs, glibc + ca-certificates (any Debian/Ubuntu)
./guest/build.sh out # initramfs only
CREATE_DATA_IMG=1 ./guest/build.sh out # also create a fresh 512MB data.img
# requires: go, cpio, gzip, e2fsprogs, file, kmod, glibc + ca-certificates (any Debian/Ubuntu)
curl -fsSL -o qemu-assets.zip \
"https://nightly.link/AMDL-Web/wrapper/workflows/build-lite/lite/qemu-assets.zip"
unzip -o qemu-assets.zip vmlinuz-lite-qemu lite-initramfs.cpio.gz -d kernel-assets/
KERNEL_ASSETS=kernel-assets ./guest/build.sh out # kernel + initramfs
KERNEL_ASSETS=kernel-assets CREATE_DATA_IMG=1 ./guest/build.sh out # also a fresh 512MB data.img
```

## Kernel

`vmlinuz-lite-qemu` is **not** built here. It is the kernel produced by the
[AMDL-Web/wrapper](https://github.com/AMDL-Web/wrapper) (a fork of
[WorldObservationLog/wrapper](https://github.com/WorldObservationLog/wrapper))
`build-lite` workflow and is distributed in its `qemu-assets` artifact:

```shell
curl -sL -o qemu-assets.zip \
"https://nightly.link/AMDL-Web/wrapper/workflows/build-lite/lite/qemu-assets.zip"
unzip -o qemu-assets.zip vmlinuz-lite-qemu -d out/
```

Copy it next to the initramfs (the launcher's `--assets-dir`, default
`<exe>/guest`).
`build-lite` workflow and is distributed in its `qemu-assets` artifact.
build.sh copies it next to the initramfs, and takes `e1000.ko` and
`qemu_fw_cfg.ko` from the same artifact's `lite-initramfs.cpio.gz`: modules
built for another kernel do not load, which leaves the guest without network
and without its fw_cfg arguments. build.sh fails when their vermagic differs
from the kernel's version.

## What the guest does on boot (init)

1. Mounts proc/sys/dev, loads `e1000.ko`, configures static NAT networking
(10.0.2.15, DNS 10.0.2.3) and writes `/etc/resolv.conf`.
2. Mounts the ext4 data disk (`/dev/vda` → `/data`); formats it with
`mke2fs` on first boot when it is a raw empty image.
3. Executes `/wrapper-manager -host 0.0.0.0 -port 8080` from `/data`, so all
3. Runs `/wrapper-manager -host 0.0.0.0 -port 8080` from `/data`, so all
manager state (wrapper-lite payload, accounts, tokens) persists in
`data.img`. Optional manager args can be supplied via qemu fw_cfg
(`manager_args`) or the kernel cmdline (`manager_args_b64=`).
4. Powers the guest off when the manager exits, so QEMU exits with it rather
than hanging on a kernel panic. The manager itself retries a failed
wrapper-lite download with backoff instead of exiting; only a download
that fails its pinned digest ends it.
32 changes: 23 additions & 9 deletions manager/guest/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,15 @@
# - cpio, gzip, e2fsprogs (mke2fs)
#
# The kernel (vmlinuz-lite-qemu) is NOT built here - it is reused from the
# upstream wrapper qemu-assets artifact (see guest/README.md).
#
# Kernel modules (e1000.ko / qemu_fw_cfg.ko) live in guest/modules/ and are
# copied verbatim from the upstream wrapper initramfs.
# build-lite qemu-assets artifact (see guest/README.md). KERNEL_ASSETS names
# the unzipped artifact: the kernel and the e1000 / qemu_fw_cfg modules both
# come from it, so they always match. A module built for another kernel fails
# to load, and the guest boots without network or fw_cfg arguments.
set -euo pipefail

KERNEL_ASSETS="${KERNEL_ASSETS:?set KERNEL_ASSETS to the unzipped build-lite qemu-assets artifact}"
KERNEL_ASSETS="$(cd "$KERNEL_ASSETS" && pwd)"
Comment on lines +19 to +20

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Parent build instructions need updating

The documented guest build command omits mandatory KERNEL_ASSETS. Readers following it stop before building the initramfs; the guest-specific README has the revised steps.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a8eff91: manager/README.md now fetches qemu-assets and passes KERNEL_ASSETS.


REPO_DIR="$(cd "$(dirname "$0")/.." && pwd)"
GUEST_DIR="$REPO_DIR/guest"
OVERLAY="$GUEST_DIR/overlay"
Expand Down Expand Up @@ -70,9 +73,21 @@ if [ ! -f "$OVERLAY/etc/ssl/certs/ca-certificates.crt" ]; then
fi
fi

# 5. Kernel modules.
echo "[build] staging kernel modules..."
cp "$GUEST_DIR"/modules/*.ko "$OVERLAY/" 2>/dev/null || true
# 5. Kernel and the modules built for it, from the same qemu-assets artifact.
echo "[build] staging the kernel and its modules..."
KERNEL_VER="$(file -bL "$KERNEL_ASSETS/vmlinuz-lite-qemu" | sed -n 's/.*version \([^ ]*\).*/\1/p')"
[ -n "$KERNEL_VER" ] || { echo "cannot read the version of $KERNEL_ASSETS/vmlinuz-lite-qemu"; exit 1; }
LITE_ROOT="$(mktemp -d)"
trap 'rm -rf "$LITE_ROOT"' EXIT
(cd "$LITE_ROOT" && gzip -dc "$KERNEL_ASSETS/lite-initramfs.cpio.gz" | cpio -id --quiet)
for mod in e1000 qemu_fw_cfg; do
[ -f "$LITE_ROOT/$mod.ko" ] || { echo "$mod.ko is missing from the qemu-assets initramfs"; exit 1; }
cp "$LITE_ROOT/$mod.ko" "$OVERLAY/$mod.ko"
magic="$(modinfo -F vermagic "$OVERLAY/$mod.ko" | cut -d' ' -f1)"
[ "$magic" = "$KERNEL_VER" ] || { echo "$mod.ko is built for $magic, the kernel is $KERNEL_VER"; exit 1; }
Comment on lines +86 to +87

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Vermagic does not verify symbol compatibility

The release-string check cannot detect symbol-version mismatches. Matching artifacts lower that risk, but a matching vermagic alone does not prove the modules will load.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, vermagic alone is a release check. The modules now come from the same qemu-assets initramfs that build-lite built against that kernel, so symbol versions match by construction; the vermagic check catches a mixed-up artifact, which is the failure that shipped.

done
cp "$KERNEL_ASSETS/vmlinuz-lite-qemu" "$OUT_DIR/vmlinuz-lite-qemu"
echo "[build] kernel and modules: $KERNEL_VER"

# 6. Init script at the initramfs root (required by the kernel).
cp "$GUEST_DIR/init" "$OVERLAY/init"
Expand All @@ -96,5 +111,4 @@ if [ "${CREATE_DATA_IMG:-0}" = "1" ]; then
echo "[build] data.img created: $DATA_IMG"
fi

echo "[build] done. Copy vmlinuz-lite-qemu next to the initramfs:"
echo " cp <wrapper qemu-assets>/vmlinuz-lite-qemu $OUT_DIR/"
echo "[build] done."
14 changes: 10 additions & 4 deletions manager/guest/init
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
/bin/busybox hostname wrapper-manager

# --- network (QEMU user-mode NAT) ---
/bin/busybox insmod /e1000.ko 2>/dev/null || true
/bin/busybox insmod /e1000.ko || echo "[init] e1000.ko did not load: no network"
/bin/busybox ip link set lo up
/bin/busybox ip link set eth0 up 2>/dev/null || true

Expand Down Expand Up @@ -45,17 +45,19 @@ fi
# read it if present. The content is split on whitespace into arguments, so
# no argument may contain a space.
MANAGER_ARGS="-host 0.0.0.0 -port 8080"
/bin/busybox insmod /qemu_fw_cfg.ko 2>/dev/null || true
FROM_FW_CFG=0
/bin/busybox insmod /qemu_fw_cfg.ko || echo "[init] qemu_fw_cfg.ko did not load"
if [ -f /sys/firmware/qemu_fw_cfg/by_name/manager_args/raw ]; then
echo "[init] reading manager args from fw_cfg"
ARGS_LINE=$(/bin/busybox cat /sys/firmware/qemu_fw_cfg/by_name/manager_args/raw 2>/dev/null)
if [ -n "$ARGS_LINE" ]; then
MANAGER_ARGS="$ARGS_LINE"
FROM_FW_CFG=1
fi
fi

# Fallback: kernel cmdline base64 (works without qemu_fw_cfg driver)
if [ -z "$(echo "$MANAGER_ARGS" | /bin/busybox grep -c 'fw_cfg')" ]; then
if [ "$FROM_FW_CFG" = 0 ]; then
for w in $(/bin/busybox cat /proc/cmdline); do
case "$w" in
manager_args_b64=*)
Expand All @@ -72,5 +74,9 @@ echo "[init] eth0 carrier: $(cat /sys/class/net/eth0/carrier 2>/dev/null)"
echo "[init] dns test: $(/bin/busybox nslookup nightly.link 10.0.2.3 2>&1 | head -3 | tr '\n' '|')"
/bin/busybox chmod +x /wrapper-manager
cd /data
# Not exec: the kernel panics when init exits, and a panicked guest keeps
# QEMU running. Powering off ends QEMU, so the host sees the manager stop.
# shellcheck disable=SC2086
exec /wrapper-manager $MANAGER_ARGS
/wrapper-manager $MANAGER_ARGS
echo "[init] wrapper-manager exited with status $?, powering off"
/bin/busybox poweroff -f
Comment on lines +80 to +82

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Failed manager exits look successful to host

When /wrapper-manager fails, init powers off the guest without passing its status to the host. runQemu treats that shutdown as success, so launchers report a clean exit.

Learn more

The guest init process now remains alive while wrapper-manager runs. Regardless of the manager's exit status, it calls poweroff -f; runQemu interprets the resulting zero QEMU exit status as a clean shutdown. A manager startup failure can therefore appear successful to host scripts and supervisors.

Example: If the manager exits with status 2 after startup, init logs status 2 and powers off. QEMU exits normally and wrapper-manager-qemu returns 0 instead of signaling that the manager failed.

Recommended fix: Preserve the manager's exit status and communicate failures across the QEMU boundary, such as through an explicit QEMU guest-to-host exit channel handled by the launcher; keep normal poweroff for successful exits.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing this here. amdl, the host that pins this guest, treats any QEMU exit before /status answers as a failed boot regardless of exit status, so the clean poweroff is what it needs. Carrying the manager's status to wrapper-manager-qemu would need a guest-to-host channel (e.g. isa-debug-exit); worth it only if a script depends on that launcher's exit code.

Binary file removed manager/guest/modules/e1000.ko
Binary file not shown.
Binary file removed manager/guest/modules/qemu_fw_cfg.ko
Binary file not shown.
5 changes: 4 additions & 1 deletion manager/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,10 @@ func main() {
if !wrapperPayloadInstalled() {
log.Warn("wrapper-lite does not exist, downloading...")
}
PrepareWrapper(*mirror)
if err := PrepareWrapper(*mirror); err != nil {
log.Errorf("wrapper-lite cannot be installed: %v", err)
os.Exit(1)
}
}

if *prepare {
Expand Down
69 changes: 62 additions & 7 deletions manager/payload_source_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,9 @@ import (
"path/filepath"
"runtime"
"strings"
"sync/atomic"
"testing"
"time"
)

// writePayloadZip builds a zip with the launcher and lite binary laid out like
Expand Down Expand Up @@ -88,14 +90,28 @@ func TestPayloadFromAnotherSourceIsNotReady(t *testing.T) {

// servePayload serves a payload zip and returns its URL and sha256.
func servePayload(t *testing.T) (url, sum string) {
t.Helper()
return servePayloadAfter(t, 0)
}

// servePayloadAfter serves the payload once failures requests have been
// answered 503.
func servePayloadAfter(t *testing.T, failures int32) (url, sum string) {
t.Helper()
zipPath := filepath.Join(t.TempDir(), "p.zip")
writePayloadZip(t, zipPath)
body, err := os.ReadFile(zipPath)
if err != nil {
t.Fatal(err)
}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write(body) }))
var seen atomic.Int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
if seen.Add(1) <= failures {
w.WriteHeader(http.StatusServiceUnavailable)
return
}
_, _ = w.Write(body)
}))
t.Cleanup(srv.Close)
h := sha256.Sum256(body)
return srv.URL + "/p.zip", hex.EncodeToString(h[:])
Expand Down Expand Up @@ -148,7 +164,7 @@ func TestPayloadSHA256RefusesAMismatchAndInstallsNothing(t *testing.T) {
func TestPayloadSHA256MismatchDoesNotKeepAnInstalledPayload(t *testing.T) {
t.Chdir(t.TempDir())
// An installed payload from another source is normally kept when the
// replacement fails; a digest mismatch must panic instead.
// replacement fails; a digest mismatch must fail instead.
for _, p := range []string{launcherPath(), filepath.Join(wrapperDir, "rootfs", "system", "bin", "lite")} {
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
Expand All @@ -159,10 +175,49 @@ func TestPayloadSHA256MismatchDoesNotKeepAnInstalledPayload(t *testing.T) {
}
url, _ := servePayload(t)
setPin(t, url, strings.Repeat("0", 64))
defer func() {
if r := recover(); r == nil {
t.Fatal("PrepareWrapper kept the unverified payload")
if err := prepareWrapper(false); !errors.Is(err, errPayloadSHA256) {
t.Fatalf("prepareWrapper kept the unverified payload: %v", err)
}
if wrapperPayloadReady() {
t.Fatal("the unverified payload reported ready")
}
}

func TestPrepareWrapperGivesUpOnADigestMismatch(t *testing.T) {
t.Chdir(t.TempDir())
url, _ := servePayload(t)
setPin(t, url, strings.Repeat("0", 64))
done := make(chan error, 1)
go func() { done <- PrepareWrapper(false) }()
select {
case err := <-done:
if !errors.Is(err, errPayloadSHA256) {
t.Fatalf("PrepareWrapper: %v", err)
}
}()
PrepareWrapper(false)
case <-time.After(10 * time.Second):
t.Fatal("PrepareWrapper retried a pin that cannot match")
}
}

func TestPrepareWrapperRetriesUntilThePayloadArrives(t *testing.T) {
t.Chdir(t.TempDir())
first, max := prepareRetryFirst, prepareRetryMax
prepareRetryFirst, prepareRetryMax = time.Millisecond, 2*time.Millisecond
t.Cleanup(func() { prepareRetryFirst, prepareRetryMax = first, max })
url, sum := servePayloadAfter(t, 3)
setPin(t, url, sum)
done := make(chan struct{})
var err error
go func() { err = PrepareWrapper(false); close(done) }()
select {
case <-done:
if err != nil {
t.Fatal(err)
}
case <-time.After(10 * time.Second):
t.Fatal("PrepareWrapper did not install the payload once it was served")
}
if !wrapperPayloadReady() {
t.Fatal("payload not ready after PrepareWrapper returned")
}
}
40 changes: 34 additions & 6 deletions manager/wrapper.go
Original file line number Diff line number Diff line change
Expand Up @@ -252,12 +252,39 @@ func describeInstall(res installResult) {
}
}

// PrepareWrapper downloads and extracts the wrapper-lite native package when
// prepareRetryFirst and prepareRetryMax bound the wait between attempts to
// install the payload.
var (
prepareRetryFirst = 5 * time.Second
prepareRetryMax = time.Minute
)

// PrepareWrapper installs the wrapper-lite payload, retrying a failed
// download with backoff until it succeeds. Nothing can be served without it,
// and exiting is no better: in the QEMU guest the manager is what init runs,
// so its exit stops the guest, and a host that sees a guest stop at once may
// blame its accelerator for a network that is merely down. A download that
// fails its pinned digest is returned instead: the same pin would fail again.
func PrepareWrapper(mirror bool) error {
delay := prepareRetryFirst
for {
err := prepareWrapper(mirror)
if err == nil || errors.Is(err, errPayloadSHA256) {
return err
}
log.Errorf("wrapper-lite is not ready: %v; trying again in %s", err, delay)
time.Sleep(delay)
Comment on lines +271 to +276

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Invalid payload pin blocks startup indefinitely

When a pinned download fails its digest check, PrepareWrapper retries the same artifact forever. The manager never serves requests, and -prepare never reports failure.

Learn more

The payload digest check in downloadWrapperLite returns errPayloadSHA256 when the fetched bytes differ from the configured pin. prepareWrapper preserves that error even when an older payload is installed. The retry loop treats it as a network failure, so a fixed wrong pin prevents the main process from ever reaching its server or finishing -prepare.

Example: Start with -payload-url pointing at a valid zip and -payload-sha256 set to 64 zeroes. Every successful fetch fails verification; the process waits and fetches again instead of terminating with an error.

Recommended fix: Distinguish terminal validation failures such as errPayloadSHA256 from transient download failures in PrepareWrapper. Propagate or terminate on terminal errors while retaining backoff for transient failures; ensure the -prepare caller exits nonzero.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a8eff91: a digest mismatch now returns from PrepareWrapper and main exits 1; other download failures still retry with backoff. Test: TestPrepareWrapperGivesUpOnADigestMismatch.

delay = min(2*delay, prepareRetryMax)
}
}

// prepareWrapper downloads and extracts the wrapper-lite native package when
// missing or installed from another source. mirror routes the download through
// gh-proxy.com. Failing to replace an installed payload keeps it.
func PrepareWrapper(mirror bool) {
// gh-proxy.com. Failing to replace an installed payload keeps it, unless the
// download failed its pinned digest.
func prepareWrapper(mirror bool) error {
if wrapperPayloadReady() {
return
return nil
}
installed := wrapperPayloadInstalled()
if installed {
Expand All @@ -271,12 +298,13 @@ func PrepareWrapper(mirror bool) {
if err != nil {
if installed && !errors.Is(err, errPayloadSHA256) {
log.Warnf("keeping the installed wrapper-lite: %v", err)
return
return nil
}
panic(err)
return err
}
log.Info("wrapper-lite ready")
describeInstall(res)
return nil
}

// UpdateWrapper force-reinstalls the latest wrapper-lite payload from
Expand Down
Loading