Repository navigation
fix(groom): refresh the apt index before installing bubblewrap in the sandbox preflight - #329
Merged
Merged
Conversation
… sandbox preflight `preflight()` ran `sudo apt-get install -y bubblewrap` against the runner image's apt index, which is frozen at image-build time. noble-updates keeps superseding packages between image rebuilds, so the install resolves to a .deb version that has since been withdrawn from the pool and every mirror answers 404 — with a perfectly good package sitting there under a newer version. Observed 2026-09-19 on Comfy-Org/evals run 35443452088: the same image (ubuntu-24.04 20260907.300.1) that installed bubblewrap 0.9.0-1ubuntu0.1 fine the day before 404'd on it across the azure, archive and security mirrors. No bwrap binary means the AppArmor and sysctl fallbacks cannot help either, so the whole sweep died at `Preflight the sandbox` before the finder ever ran. Nothing self-heals: without an index refresh it stays broken until the runner image is rebuilt or the mirror restores the withdrawn file. Both halves now tolerate failure, so the verdict stays with the selftest rather than apt's exit code. That also converges the two call sites: preflight() runs bare (errexit live) on the agent path and as `preflight || exit $?` on the --preflight-only path, so an apt failure used to exit 100 with no `::error::` annotation on one and reach the fail-loud on the other. Both now end at the explicit fail-loud. Section 8c of sandbox-tests.sh pins the ordering: with bwrap absent from PATH and sudo stubbed to log its argv, `apt-get update` must be logged before `apt-get install`, and preflight must still exit non-zero because nothing was really installed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Team Run ID: 📒 Files selected for processing (3)
✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ELI-5
The groom sandbox preflight installs
bubblewrapwhen the runner doesn't have it, but it never refreshed the apt index first. The runner image's index is frozen at image-build time whilenoble-updateskeeps superseding packages, soapt-get installasks the pool for a.debversion that has since been withdrawn — every mirror answers 404, the install fails, and there is nobwrapfor the AppArmor or sysctl fallbacks to rescue. The preflight then correctly refuses to run the agent unsandboxed, and the whole sweep dies before the finder starts.One
apt-get updateahead of the install fixes it.The failure this reproduces
Comfy-Org/evals run 35443452088 (scheduled, 2026-09-19), job
Audit — finder, stepPreflight the sandbox:It is a stale index, not a broken mirror or a bad runner: the previous day's run (35346094396) installed that exact version from that exact mirror on the exact same image — ubuntu-24.04
20260907.300.1both times. The package was superseded in between; the image's index still named the withdrawn one.Nothing about this self-heals. Without an index refresh a re-run resolves to the same withdrawn version, so the sweep stays down until the runner image is rebuilt or the mirror restores the file. That is what makes it worth fixing rather than retrying.
What changed
.github/groom/agent-sandbox.sh(preflight()) —sudo apt-get update >&2 || trueahead of the install. Only reached whenbwrapis missing; the idempotent fast path (bwrap present + selftest green) still returns before any of this, so the repeated--preflight-only/--validate-only/ agent-step invocations stay quiet and cost nothing.Both halves now tolerate failure (
|| true), so the verdict stays with the selftest below rather than with apt's exit code. A transient index-refresh failure must not skip an install the on-image cache can still satisfy, and a failed install must not abort before the AppArmor and sysctl fallbacks.This also converges the two call sites, which behaved differently on an apt failure:
preflight()is called bare on the agent path (line 292, errexit live → apt's exit 100 propagated, killing the script with no::error::annotation and no fallbacks) and aspreflight || exit $?on the--preflight-onlypath (errexit suppressed inside a||list → it ran on to the fallbacks and the explicit fail-loud). Both now reach the fail-loud, which is the documented contract..github/groom/README.md— the--validate-onlystandalone-side-effects paragraph names the bring-up's host mutations one by one;apt-get updateis now one of them.Tests
New section 8c in
sandbox-tests.sh, beside 8b's fail-loud proof.PATHis the stub dir alone: asudothat logs its argv and succeeds, plus symlinks to the only two binaries the bring-up itself needs —bash(the#!/usr/bin/env bashshebang resolves it throughPATH) andcat(theapparmor_restrict_unprivileged_usernsread). That is what makesbwrapunfindable, so preflight takes the install branch; nothing is really installed, so it still fails loud at the end. The assertions:--preflight-onlyexits non-zero (no silent green with no sandbox),apt-get installwas attempted at all (guards against a vacuous pass if the branch stops being reached),apt-get updatewas logged,Non-vacuity
Section 8c extracted into a standalone harness and run against both scripts:
The old script's sudo log is
apt-get install -y bubblewrapfollowed bysysctl -w …— install first, no update, fail loud. The new one logsapt-get updatethen the install then the sysctl.Verification
shellcheck -x .github/groom/agent-sandbox.sh .github/groom/tests/sandbox-tests.sh— clean.bash -non both — clean.sandbox-tests.sh. Sections 1–7 need a Linux host with unprivileged user namespaces; this was authored on macOS. Thesandbox-testsjob on this PR is the real run — and it is also the live proof of the fix, since that job's own bring-up takes the veryapt-get installpath this changes.Residual
apt-get installin this repo has the same stale-index exposure and is out of scope here.apt-get updateadds a few seconds, once, on the runs that actually have to install bubblewrap. The fast path is untouched.Skip-caller-bump— this is a behavioral change in a watched surface, so the groom fleet should bump.Provenance
bash -nclean; section 8c passes against the fix and fails against the pre-fix script (output above). Full suite deferred to this PR'ssandbox-testsjob.preflight()is longer than the code it explains, deliberately — the next person to see this 404 should not have to re-derive why the index is stale.🤖 Generated with Claude Code