Repository navigation
Grade with the producing run's network and resource posture - #2080
Merged
Merged
Conversation
ppXD
changed the base branch from
fix/keep-model-authored-acceptance-from-carrying-setup
to
main
October 7, 2026 04:41
ppXD
force-pushed
the
fix/grade-with-the-producing-runs-posture
branch
from
October 7, 2026 04:41
44bb768 to
19467a2
Compare
An acceptance grade's setup step was hard-coded to AllowNetwork=true, so under bubblewrap it shared the host network whatever the producing run's tier, and neither the setup nor the check had a memory or CPU ceiling. A setup like `npm ci` or `pip install` runs manifests the agent wrote, after the agent's own sandbox is gone. So a network-off Standard run could have its planted code executed with full egress, and a runaway install had no cgroup bound. Benchmark and qualification grading ran the fixture's tests over the agent's workspace on the raw local runner, with no ceiling either. Every lane now carries an AcceptanceGradingPosture, derived once from the run whose bytes it grades. The posture is that run's network grant and egress allowlist plus its tier's resource ceilings, clamped by Sandbox:MaxAutonomy the way RunCommandService.BuildSpec clamps agent.run_command. The executor lanes (branch, patch, multi-repo, local) take it from the run's own task. The supervisor's per-unit, baseline, captured, resolve and branchless-stop lanes read the unit's stored task. The stop over the integrated head uses the run profile's tier, which is the tier every unit is clamped to. BenchmarkRunner uses its agent's task; a TaskLaunch cell uses the posture its attempts agree on, and fails closed when they do not. AcceptanceGradingPosturePolicy.Bind wraps the grading runner once, so the setup and every oracle's command run narrowed, narrow-only. The setup keeps the network only when the producer had it. An allowlist producer is filtered to its operator-configured hosts, and an allowlist with no host severs. Both steps run under the tier's ceilings. A request with no posture grades with network off under the Confined ceilings. What the grade reports follows what the sandbox did, not what the spec asked. The runner now says which egress it enforces for a spec (ISandboxEgressEnforcement). Only when it severed or filtered the setup does one notice head the grade's evidence, so an unconfined host never reads "off" for a setup that kept its network. A setup the sandbox severed fails as "setup-failed-network-severed:": still infra, but the posture comes from the same stored task on every attempt, so agent.code no longer re-buys an agent run to sever it again. Grades can now hit a memory ceiling. A check the runner kills there (ResourceExhausted) is "tests-resource-exhausted", an Environment fact like tests-timed-out, instead of a genuine failure that bought revise rounds and recorded a verdict on the code. This changes what operators see: an operator setup on a network-off run no longer downloads where the sandbox confines. The fix is an egress allowlist or a higher tier. Trusted and Unleashed runs keep full egress. Baselines are now memoized per posture, so two units of different tiers off one base are no longer compared across two different sandboxes. EvaluatorVersion moves to v9.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AllowNetwork=truehard-coded (host network under bubblewrap), and neither setup nor check had cgroup ceilings. Benchmark and qualification grading ran the fixture's tests over the agent's workspace on the raw local runner.AcceptanceGradingPosturetaken from the run whose bytes it executes: its network grant and egress allowlist, plus its tier's memory and CPU row, clamped bySandbox:MaxAutonomyand narrowed bySandbox:AgentMemoryCeilingMb.AcceptanceGradingPosturePolicy.Bindwraps the grading runner once, narrow-only, for the setup and every oracle.ISandboxEgressEnforcement;LocalProcessRunner.EnforcedEgress).setup network: …) heads the evidence only when the setup was actually severed or filtered.setup-failed-network-severed: …. That is infra (no revise round), andagent.codedoes not respawn it, because the posture comes from the same stored task every time.ResourceExhausted) istests-resource-exhausted, class Environment, liketests-timed-out. Before, it was a Genuinetests-failed-exit-137.BenchmarkRunner: its agent's task.Owner-visible behaviour (setup network applies where bubblewrap confines; ceilings apply where
Sandbox:CgroupRootis delegated):EgressAllowHosts, filtered where the host filters, severed where it cannot; no hosts → severedtests-timed-out.EvaluatorVersionissupervisor-acceptance/v9.Test plan
For()read through real configuration (Sandbox:MaxAutonomy,AgentMemoryCeilingMb)EnforcedEgresstable and notice tableTestsPassGradertests-resource-exhausted,setup-failed-network-severed:)agent.coderetry rowsBenchmarkTaskGradingposture and fail-closedGradedPostureLocalProcessRunnerBenchmarkRunnerhands the oracle a runner bound to its agent's posturePosture = nullon the resolve patch arm fails 2 rows;For()without the host budget fails the configuration testAcceptanceGradingPostureE2ETests, covering a real severed setup that reportssetup-failed-network-severed:and a host that reports the severance it enforces