Skip to content

fix(groom): refresh the apt index before installing bubblewrap in the sandbox preflight - #329

Merged
mattmillerai merged 1 commit into
mainfrom
fix/groom-sandbox-apt-update
Sep 19, 2026
Merged

mattmillerai merged 1 commit into
mainfrom
fix/groom-sandbox-apt-update

Conversation

@mattmillerai

Copy link
Copy Markdown
Member

ELI-5

The groom sandbox preflight installs bubblewrap when the runner doesn't have it, but it never refreshed the apt index first. The runner image's index is frozen at image-build time while noble-updates keeps superseding packages, so apt-get install asks the pool for a .deb version that has since been withdrawn — every mirror answers 404, the install fails, and there is no bwrap for the AppArmor or sysctl fallbacks to rescue. The preflight then correctly refuses to run the agent unsandboxed, and the whole sweep dies before the finder starts.

One apt-get update ahead of the install fixes it.

The failure this reproduces

Comfy-Org/evals run 35443452088 (scheduled, 2026-09-19), job Audit — finder, step Preflight the sandbox:

Ign:2 http://azure.archive.ubuntu.com/ubuntu noble-updates/main amd64 bubblewrap amd64 0.9.0-1ubuntu0.1
Ign:2 https://archive.ubuntu.com/ubuntu noble-updates/main amd64 bubblewrap amd64 0.9.0-1ubuntu0.1
Ign:2 https://security.ubuntu.com/ubuntu noble-updates/main amd64 bubblewrap amd64 0.9.0-1ubuntu0.1
Err:2 mirror+file:/etc/apt/apt-mirrors.txt noble-updates/main amd64 bubblewrap amd64 0.9.0-1ubuntu0.1
  404  Not Found [IP: 52.161.185.214 80]
E: Failed to fetch .../bubblewrap_0.9.0-1ubuntu0.1_amd64.deb  404  Not Found
kernel.apparmor_restrict_unprivileged_userns = 0
::error::bwrap sandbox unavailable on this runner image — refusing to run the agent unsandboxed

It is a stale index, not a broken mirror or a bad runner: the previous day's run (35346094396) installed that exact version from that exact mirror on the exact same image — ubuntu-24.04 20260907.300.1 both times. The package was superseded in between; the image's index still named the withdrawn one.

Nothing about this self-heals. Without an index refresh a re-run resolves to the same withdrawn version, so the sweep stays down until the runner image is rebuilt or the mirror restores the file. That is what makes it worth fixing rather than retrying.

What changed

  • .github/groom/agent-sandbox.sh (preflight()) — sudo apt-get update >&2 || true ahead of the install. Only reached when bwrap is missing; the idempotent fast path (bwrap present + selftest green) still returns before any of this, so the repeated --preflight-only / --validate-only / agent-step invocations stay quiet and cost nothing.

  • Both halves now tolerate failure (|| true), so the verdict stays with the selftest below rather than with apt's exit code. A transient index-refresh failure must not skip an install the on-image cache can still satisfy, and a failed install must not abort before the AppArmor and sysctl fallbacks.

    This also converges the two call sites, which behaved differently on an apt failure: preflight() is called bare on the agent path (line 292, errexit live → apt's exit 100 propagated, killing the script with no ::error:: annotation and no fallbacks) and as preflight || exit $? on the --preflight-only path (errexit suppressed inside a || list → it ran on to the fallbacks and the explicit fail-loud). Both now reach the fail-loud, which is the documented contract.

  • .github/groom/README.md — the --validate-only standalone-side-effects paragraph names the bring-up's host mutations one by one; apt-get update is now one of them.

Tests

New section 8c in sandbox-tests.sh, beside 8b's fail-loud proof.

PATH is the stub dir alone: a sudo that logs its argv and succeeds, plus symlinks to the only two binaries the bring-up itself needs — bash (the #!/usr/bin/env bash shebang resolves it through PATH) and cat (the apparmor_restrict_unprivileged_userns read). That is what makes bwrap unfindable, so preflight takes the install branch; nothing is really installed, so it still fails loud at the end. The assertions:

  1. --preflight-only exits non-zero (no silent green with no sandbox),
  2. an apt-get install was attempted at all (guards against a vacuous pass if the branch stops being reached),
  3. an apt-get update was logged,
  4. and its line number is strictly less than the install's — ordering is the whole point; an update after the install leaves the stale index in force.

Non-vacuity

Section 8c extracted into a standalone harness and run against both scripts:

=== against FIXED script ===
PASS: preflight refreshes the apt index before installing bubblewrap (and still fails loud when it cannot)
rc=0
=== against OLD (pre-fix) script ===
FAIL: preflight installed bubblewrap without refreshing the apt index first — a stale runner index 404s the .deb
rc=1

The old script's sudo log is apt-get install -y bubblewrap followed by sysctl -w … — install first, no update, fail loud. The new one logs apt-get update then the install then the sysctl.

Verification

  • shellcheck -x .github/groom/agent-sandbox.sh .github/groom/tests/sandbox-tests.sh — clean.
  • bash -n on both — clean.
  • Section 8c run standalone, both directions (above).
  • Not run locally: the full sandbox-tests.sh. Sections 1–7 need a Linux host with unprivileged user namespaces; this was authored on macOS. The sandbox-tests job on this PR is the real run — and it is also the live proof of the fix, since that job's own bring-up takes the very apt-get install path this changes.

Residual

  • Only the groom sandbox preflight is touched. Any other apt-get install in this repo has the same stale-index exposure and is out of scope here.
  • apt-get update adds a few seconds, once, on the runs that actually have to install bubblewrap. The fast path is untouched.
  • Not marked Skip-caller-bump — this is a behavioral change in a watched surface, so the groom fleet should bump.

Provenance

  • Authored by: Claude Opus 5 (Claude Code), from a triage of evals run 35443452088.
  • Verified: shellcheck clean; bash -n clean; section 8c passes against the fix and fails against the pre-fix script (output above). Full suite deferred to this PR's sandbox-tests job.
  • Deviations: none. The comment in preflight() is longer than the code it explains, deliberately — the next person to see this 404 should not have to re-derive why the index is stale.

🤖 Generated with Claude Code

… sandbox preflight

`preflight()` ran `sudo apt-get install -y bubblewrap` against the runner
image's apt index, which is frozen at image-build time. noble-updates keeps
superseding packages between image rebuilds, so the install resolves to a .deb
version that has since been withdrawn from the pool and every mirror answers
404 — with a perfectly good package sitting there under a newer version.

Observed 2026-09-19 on Comfy-Org/evals run 35443452088: the same image
(ubuntu-24.04 20260907.300.1) that installed bubblewrap 0.9.0-1ubuntu0.1 fine
the day before 404'd on it across the azure, archive and security mirrors. No
bwrap binary means the AppArmor and sysctl fallbacks cannot help either, so the
whole sweep died at `Preflight the sandbox` before the finder ever ran. Nothing
self-heals: without an index refresh it stays broken until the runner image is
rebuilt or the mirror restores the withdrawn file.

Both halves now tolerate failure, so the verdict stays with the selftest rather
than apt's exit code. That also converges the two call sites: preflight() runs
bare (errexit live) on the agent path and as `preflight || exit $?` on the
--preflight-only path, so an apt failure used to exit 100 with no `::error::`
annotation on one and reach the fail-loud on the other. Both now end at the
explicit fail-loud.

Section 8c of sandbox-tests.sh pins the ordering: with bwrap absent from PATH
and sudo stubbed to log its argv, `apt-get update` must be logged before
`apt-get install`, and preflight must still exit non-zero because nothing was
really installed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Team

Run ID: c07d412e-18e8-45ab-88cb-7f5477e315b5

📥 Commits

Reviewing files that changed from the base of the PR and between 525a713 and 4c7d4a7.

📒 Files selected for processing (3)
  • .github/groom/README.md
  • .github/groom/agent-sandbox.sh
  • .github/groom/tests/sandbox-tests.sh
 _______________________________
< The only circle I like is CI. >
 -------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       /   づ
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
✨ Simplify code
  • Commit to this branch
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@mattmillerai
mattmillerai merged commit ef9b983 into main Sep 19, 2026
8 of 9 checks passed
@mattmillerai
mattmillerai deleted the fix/groom-sandbox-apt-update branch September 19, 2026 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants