From 4c7d4a757428d4d648d2dda28408f1e528e584f7 Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Sat, 19 Sep 2026 14:47:31 -0700 Subject: [PATCH] fix(groom): refresh the apt index before installing bubblewrap in the sandbox preflight MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `preflight()` ran `sudo apt-get install -y bubblewrap` against the runner image's apt index, which is frozen at image-build time. noble-updates keeps superseding packages between image rebuilds, so the install resolves to a .deb version that has since been withdrawn from the pool and every mirror answers 404 — with a perfectly good package sitting there under a newer version. Observed 2026-09-19 on Comfy-Org/evals run 35443452088: the same image (ubuntu-24.04 20260907.300.1) that installed bubblewrap 0.9.0-1ubuntu0.1 fine the day before 404'd on it across the azure, archive and security mirrors. No bwrap binary means the AppArmor and sysctl fallbacks cannot help either, so the whole sweep died at `Preflight the sandbox` before the finder ever ran. Nothing self-heals: without an index refresh it stays broken until the runner image is rebuilt or the mirror restores the withdrawn file. Both halves now tolerate failure, so the verdict stays with the selftest rather than apt's exit code. That also converges the two call sites: preflight() runs bare (errexit live) on the agent path and as `preflight || exit $?` on the --preflight-only path, so an apt failure used to exit 100 with no `::error::` annotation on one and reach the fail-loud on the other. Both now end at the explicit fail-loud. Section 8c of sandbox-tests.sh pins the ordering: with bwrap absent from PATH and sudo stubbed to log its argv, `apt-get update` must be logged before `apt-get install`, and preflight must still exit non-zero because nothing was really installed. Co-Authored-By: Claude Opus 5 --- .github/groom/README.md | 5 ++-- .github/groom/agent-sandbox.sh | 21 +++++++++++++- .github/groom/tests/sandbox-tests.sh | 41 ++++++++++++++++++++++++++++ 3 files changed, 64 insertions(+), 3 deletions(-) diff --git a/.github/groom/README.md b/.github/groom/README.md index d1ebe61b..76b91738 100644 --- a/.github/groom/README.md +++ b/.github/groom/README.md @@ -836,8 +836,9 @@ agent step dies instead of becoming a green no-op that runs no agent. `preflight()` takes its idempotent fast path), and the sole remaining side effect is the `mkdir -p` on the out-dir that the real run performs anyway; the agent step's own copies of both are then no-ops. Run standalone on a host where the -sandbox is *not* yet usable, the same call will `sudo apt-get install bubblewrap`, -write `/etc/apparmor.d/bwrap`, and as a last resort `sudo sysctl -w +sandbox is *not* yet usable, the same call will `sudo apt-get update` and then +`sudo apt-get install bubblewrap`, write `/etc/apparmor.d/bwrap`, and as a last +resort `sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0` — the bring-up's host-wide mutations, from a mode named for validation. Pair it with `--preflight-only`, as groom.yml does, or expect the bring-up. diff --git a/.github/groom/agent-sandbox.sh b/.github/groom/agent-sandbox.sh index 047decfb..5b082215 100755 --- a/.github/groom/agent-sandbox.sh +++ b/.github/groom/agent-sandbox.sh @@ -143,7 +143,26 @@ preflight() { fi if ! command -v bwrap >/dev/null 2>&1; then - sudo apt-get install -y bubblewrap >&2 + # `apt-get update` FIRST, every time. The runner image ships an apt index + # frozen at image-build time, and noble-updates supersedes packages + # between image rebuilds: installing against the stale index asks the + # pool for a .deb version that has since been withdrawn, so EVERY mirror + # answers 404 and the install fails with a perfectly good package + # sitting there under a newer version. Observed 2026-09-19 — the same + # image (ubuntu-24.04 20260907.300.1) that installed + # bubblewrap 0.9.0-1ubuntu0.1 fine the day before 404'd on it across the + # azure, archive and security mirrors, failing the sweep at preflight. + # + # Both halves tolerate failure so the verdict stays with the selftest + # below rather than with apt's exit code: a transient index refresh + # failure must not skip an install the on-image cache can still satisfy, + # and a failed install must not abort the script before the AppArmor and + # sysctl fallbacks — preflight() is called bare (errexit live) on the + # agent path and as `preflight || exit $?` on the --preflight-only path, + # so without these an apt failure would exit 100 with no `::error::` on + # one path and reach the fail-loud on the other. + sudo apt-get update >&2 || true + sudo apt-get install -y bubblewrap >&2 || true fi local restrict=/proc/sys/kernel/apparmor_restrict_unprivileged_userns diff --git a/.github/groom/tests/sandbox-tests.sh b/.github/groom/tests/sandbox-tests.sh index 54a21898..987d0e4c 100755 --- a/.github/groom/tests/sandbox-tests.sh +++ b/.github/groom/tests/sandbox-tests.sh @@ -483,6 +483,47 @@ if PATH="$failbin:$PATH" "$SANDBOX" --preflight-only >/dev/null 2>&1; then fi pass "--preflight-only fails loud when the sandbox self-test cannot pass" +# 8c. With bwrap MISSING, the bring-up must refresh the apt index BEFORE it +# installs bubblewrap. The runner image carries an apt index frozen at image +# build time while noble-updates keeps superseding packages, so an install +# against that stale index asks the pool for a withdrawn .deb version, 404s on +# every mirror, and kills the whole sweep at preflight with a working package one +# `apt-get update` away (observed 2026-09-19 on ubuntu-24.04 20260907.300.1, +# bubblewrap 0.9.0-1ubuntu0.1 — installed fine the day before). +# +# PATH is the stub dir ALONE, holding only a `sudo` that logs its argv and +# succeeds plus symlinks to the two binaries the bring-up itself needs (`bash`, +# which the `#!/usr/bin/env bash` shebang resolves through PATH, and `cat`). That +# is what makes bwrap unfindable, so preflight takes the install branch; nothing +# is really installed, so it still fails loud at the end. +aptbin="$work/aptbin" +mkdir -p "$aptbin" +cat > "$aptbin/sudo" <<'STUB' +#!/bin/sh +echo "$@" >> "$SUDO_LOG" +exit 0 +STUB +chmod +x "$aptbin/sudo" +ln -s "$(command -v bash)" "$aptbin/bash" +ln -s "$(command -v cat)" "$aptbin/cat" +export SUDO_LOG="$work/sudo-argv.log" +: > "$SUDO_LOG" +if PATH="$aptbin" "$SANDBOX" --preflight-only >/dev/null 2>&1; then + fail "--preflight-only exited 0 with no bwrap and a no-op sudo (nothing was installed, so it must fail loud)" +fi +apt_update_line="$(grep -n -m1 '^apt-get update' "$SUDO_LOG" | cut -d: -f1 || true)" +apt_install_line="$(grep -n -m1 '^apt-get install' "$SUDO_LOG" | cut -d: -f1 || true)" +if [[ -z "$apt_install_line" ]]; then + fail "preflight never attempted the bubblewrap install with bwrap absent from PATH" +fi +if [[ -z "$apt_update_line" ]]; then + fail "preflight installed bubblewrap without refreshing the apt index first — a stale runner index 404s the .deb" +fi +if (( apt_update_line >= apt_install_line )); then + fail "preflight ran \`apt-get update\` AFTER the install — the install still resolves against the stale index" +fi +pass "preflight refreshes the apt index before installing bubblewrap (and still fails loud when it cannot)" + # --- 9. --validate-only: the pre-exec guards, off the billed step (BE-14771) -- # The groom jobs run this in the SAME separate "Preflight the sandbox" step as # --preflight-only. Everything the wrapper checks before `exec bwrap` is no-spend