Skip to content

Refuse a network-off run that cannot be sealed before it spends - #2036

Merged
ppXD merged 1 commit into
mainfrom
fix/refuse-a-network-off-run-that-cannot-be-sealed
Sep 27, 2026
Merged

ppXD merged 1 commit into
mainfrom
fix/refuse-a-network-off-run-that-cannot-be-sealed

Conversation

@ppXD

@ppXD ppXD commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • A confining host that cannot seal used to sever a network-off brokered run. That cut the run off from its broker as well, so it spent its whole timeout and the CLI's retries on failures that looked like a provider outage (AgentAutonomyPolicy.cs, the shape its network doc describes). Such a run is now refused before it spends anything, with sandbox_sealed_egress_unavailable.
  • New runner capability, ISandboxEgressAdmission (Rule 7: a sibling interface, not a wider ISandboxRunner). The executor calls it once the spec is final, before local acceptance is prepared, spend is admitted or a process starts.
  • LocalProcessRunner.EnsureEgressAdmissible refuses only a spec that carries a broker port on a host that confines. The refusal names one of three causes: ip/nft missing, no privilege to build a namespace, or a broker that could only bind loopback. Every other spec is admitted untouched, including unbrokered network-off runs, which have no model to reach.
  • BrokeredModelCredential.ReachableFromNamespace records whether the lease took the wide bind. Its default is false, so an unknown answer fails closed.
  • A sealed setup that still fails at launch (a name collision, a kernel refusal) is retyped to the same refusal, replacing a generic InvalidOperationException.
  • The code joins FailureCodes.InfraExitReasons and gets its own supervisor remedy in LlmSupervisorDecider.EndedByDeploymentSteer: retry once in case another worker can seal, then ask_human.
  • The headline lane (backend-e2e.yml) is already root and privileged; it now also installs iproute2 nftables, so any brokered agent there is sealed instead of refused. It has no allowlist tests, so nothing else there changes.
  • A setup step that fails on a host that can seal (a route the kernel will not send the run's replies down, a name left behind) gets its own remedy (SealedEgressUnavailableException.SetupFailed): fix what the step names, since every launch runs the setup afresh. It is not told to grant privileges or wait for a re-probe, neither of which would help.
  • A refusal for missing privilege carries the seal probe's own account of the failed step (FilteredEgressNetns.SealUnavailableReason). Because a probe failure can be transient, the remedy now says a retry on the host helps once it can build a namespace, which it re-checks at most once a minute.

Test plan

  • Unit:
    • SealedEgressAdmissionTests: each cause, including the admitted case.
    • A spec with no broker port is admitted on any host.
    • Host-honest refusal: only on a confining host that cannot seal.
    • Exception kind, code, client message and remedy.
    • FailureTaxonomyTests: literal code, and the infra set member by member.
    • SupervisorDeciderTests: the steer arm, byte for byte.
    • ModelCredentialBrokerTests: ReachableFromNamespace matches whether the wide bind is tried.
  • Integration:
    • A runner that cannot seal refuses a brokered network-off run of a workflow run. The run lands Failed under the code; the runner was asked with the lease port; no process is launched; no spend row exists (an admitted run there always records an unbudgeted one); and the lease is withdrawn.
    • An unbrokered network-off run is admitted.
    • Mutation-checked: moving the admission after spend admission turns the first test red.
  • Unit suite: full run green locally (11245).
  • Sandbox lane (real kernel): new A_sealed_setup_that_fails_on_this_host_refuses_the_launch_typed_and_leaks_nothing. It occupies the host veth name so the plan's own ip link add fails, then checks that the durable launch refuses typed, the refusal names the failed step, and no namespace is left behind.
  • Unit: SealedEgressAdmissionTests pins the setup-failure remedy: same code, the failed step's own error as the cause, and neither the Dockerfile nor the probe interval in the message.
  • Rebased on Seal a network-off run to its model broker #2035 after its sixth review round: unit suite 11315 green, and the touched integration classes green.
  • Rebased on Seal a network-off run to its model broker #2035 after its fifth review round. A setup that fails Seal a network-off run to its model broker #2035's new route check is refused under the same typed code, since it fails through the same setup path. Unit suite (11308) and the touched integration classes green. The sandbox lane floor is 66.

@ppXD
ppXD force-pushed the fix/seal-a-network-off-run-to-its-broker branch from eb9be4b to 66df2db Compare September 26, 2026 23:54
@ppXD
ppXD force-pushed the fix/refuse-a-network-off-run-that-cannot-be-sealed branch from c984ba6 to fc9ebfb Compare September 27, 2026 00:04
@ppXD
ppXD force-pushed the fix/seal-a-network-off-run-to-its-broker branch from 66df2db to f430284 Compare September 27, 2026 00:44
@ppXD
ppXD force-pushed the fix/refuse-a-network-off-run-that-cannot-be-sealed branch from fc9ebfb to 0230dd3 Compare September 27, 2026 00:53
@ppXD
ppXD force-pushed the fix/seal-a-network-off-run-to-its-broker branch from f430284 to 9f4394f Compare September 27, 2026 01:26
@ppXD
ppXD force-pushed the fix/refuse-a-network-off-run-that-cannot-be-sealed branch from 0230dd3 to 3a12635 Compare September 27, 2026 01:32
@ppXD
ppXD force-pushed the fix/seal-a-network-off-run-to-its-broker branch from 9f4394f to f71bbf6 Compare September 27, 2026 02:00
@ppXD
ppXD force-pushed the fix/refuse-a-network-off-run-that-cannot-be-sealed branch from 3a12635 to 2792bdc Compare September 27, 2026 02:04
@ppXD
ppXD force-pushed the fix/seal-a-network-off-run-to-its-broker branch from f71bbf6 to 3aaa4bb Compare September 27, 2026 02:54
@ppXD
ppXD force-pushed the fix/refuse-a-network-off-run-that-cannot-be-sealed branch from 2792bdc to 3bea2fb Compare September 27, 2026 03:01
@ppXD
ppXD force-pushed the fix/seal-a-network-off-run-to-its-broker branch from 3aaa4bb to 0baf9a0 Compare September 27, 2026 08:16
@ppXD
ppXD force-pushed the fix/refuse-a-network-off-run-that-cannot-be-sealed branch from 3bea2fb to 1b65519 Compare September 27, 2026 08:16
@ppXD
ppXD force-pushed the fix/seal-a-network-off-run-to-its-broker branch from 0baf9a0 to d3c6729 Compare September 27, 2026 09:10
@ppXD
ppXD force-pushed the fix/refuse-a-network-off-run-that-cannot-be-sealed branch from 1b65519 to 567b4bd Compare September 27, 2026 09:10
@ppXD
ppXD changed the base branch from fix/seal-a-network-off-run-to-its-broker to main September 27, 2026 10:03
On a host that confines but cannot seal, a network-off brokered run was
severed from its broker along with everything else. It then burned its
whole timeout and the CLI's retries on failures that read like a
provider outage.

The runner now gains an optional admission capability. The executor
asks it once the spec is final, before local acceptance is prepared,
spend is admitted or a process starts. The local runner refuses such a
spec under sandbox_sealed_egress_unavailable, naming the wall: missing
ip/nft, no privilege to build a namespace, or a broker that could only
bind loopback. The broker lease now says whether it bound where a
namespace can reach it. A sealed setup that still fails at launch is
retyped to the same refusal, with a remedy of its own: fix what the
failed step names, since granting privileges or waiting for a re-probe
cannot help a host that can already seal.

The code joins the infra exit reasons with its own supervisor remedy:
retry once in case another worker can seal, then ask a human. The
headline lane installs iproute2 and nftables, so its brokered agents
are sealed rather than refused.
@ppXD
ppXD force-pushed the fix/refuse-a-network-off-run-that-cannot-be-sealed branch from 567b4bd to 76f67cf Compare September 27, 2026 10:03
@ppXD
ppXD merged commit 02a88b4 into main Sep 27, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant