Skip to content

Reach a namespaced run's model broker through its socket - #2044

Merged
ppXD merged 1 commit into
mainfrom
feat/reach-a-namespaced-runs-broker-through-its-socket
Sep 28, 2026
Merged

ppXD merged 1 commit into
mainfrom
feat/reach-a-namespaced-runs-broker-through-its-socket

Conversation

@ppXD

@ppXD ppXD commented Sep 28, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Relay for network-off and allowlist runs. On Linux, a brokered run whose network is its own (off, or an allowlist) now gets a per-run Unix socket on its lease. Its CLI starts behind codespace-mcp relay, just inside bubblewrap. The socket's directory and the helper's files are bound read-only. This is ModelBrokerRelay.Wrap and RelaysModelBroker in LocalProcessRunner.Durable.cs.

    • The shipped worker (uid 1654, no capabilities) used to refuse Standard, the default tier; it now admits and relays these runs.
    • Removed: the veth seal (SetupSealedNetnsAsync, BuildSealed/BuildSealedNftRuleset, SandboxEgressMode.Sealed) and the seal-probe prepay in NativeLaunch.
    • Every child now resolves its broker to 127.0.0.1.
  • Refusal before spend. LocalProcessRunner.RelayRefusal refuses under sandbox_sealed_egress_unavailable in two cases:

    • the lease has no socket;
    • no helper can run the relay: it is missing, it is a self-contained publish, or it does not answer as the relay (ModelBrokerRelay.HelperRunsRelay). A build from before the relay, which CODESPACE_MCP_PROXY_PATH can name, is the last case.

    HelperRunsRelay asks the helper to listen on a loopback port the worker holds and expects exit 125, having started nothing. Only a yes is cached, per file identity.

  • Runs already in flight keep what they had. A handle with an EgressNetnsKey and no socket path keeps the legacy re-bind: wide first, the 10/8 source gate, and teardown by name, which still deletes a sealed run's inet table. A handle with neither re-binds on loopback (LoopbackModelCredentialBroker.CandidateHosts).

    • Unconfined hosts and macOS are byte-identical.
    • Trusted runs get no socket.
    • This must deploy after Serve a model-broker lease over a per-run Unix socket #2042, which serves a lease over a Unix socket. A rollback from this change then lands on a worker that still reopens the sockets of relayed runs in flight.

Test plan

  • Unit: full suite on macOS, 11470/11470 (1 skipped). On Linux, in a privileged container, the same 59 environment-only failures as the pre-fix tree and no new ones. SealedEgressAdmissionTests uses real stand-in helpers and the shipped apphost. ModelCredentialBrokerTests has a socket-less, no-namespace re-bind row that must bind loopback.
  • Integration (AgentRunExecutorTests, macOS): 144/144.
  • Integration in a confining Linux container, broker tests plus the heartbeat test: the runs that regressed on this branch pass. 5 tests fail there identically on the base (environment). Brokered runs point CODESPACE_MCP_PROXY_PATH at the built helper while they execute.
  • Sandbox, root, with the real claude/codex CLIs: 76/76. In both lanes the relayed arms assert EROFS on unlinking the lease's socket and on creating a file beside it. The legacy arm tears down a sealed survivor's inet table (marker legacy-sealed-teardown).
  • Sandbox, non-root (uid 1654, no capabilities, no_new_privs): 9/9. The workflow's assertion blocks pass on both trx files.
  • Mutations turn red and then pass once restored:
    • wide re-bind without a namespace;
    • admission back to File.Exists;
    • socket directory writable;
    • inet teardown removed;
    • no relay wrap;
    • no socket-directory bind;
    • SealedEgress not redefined;
    • relay outside bwrap;
    • admission keyed on CanSeal.
  • Staging: watch for the legacy model-broker re-bind log line to go quiet before retiring the gateway path.
  • The sandbox lane replayed step by step in the CI image (mcr.microsoft.com/dotnet/sdk:10.0, HOME=/github/home, workspace at /__w), with GITHUB_ENV carried between steps: root 76/76 with the real claude 2.1.263 and codex 0.142.2 CLIs, and uid 1654 9/9. Every marker assertion passed. The non-root step asks NuGet for its global-packages folder, because the container job's HOME is /github/home, not /root.

@ppXD
ppXD force-pushed the feat/reach-a-namespaced-runs-broker-through-its-socket branch from 59f6e26 to d57e91a Compare September 28, 2026 01:15
A network-off brokered run on a confining host reached its broker only
through a veth namespace sealed to it, which needs root with
CAP_NET_ADMIN and CAP_SYS_ADMIN. The shipped worker is uid 1654 with no
capabilities, so there every such run was refused before it spent, and
Standard, the default tier, reached no model. An allowlist run reached
its broker at its namespace's gateway on a lease bound to every
address.

The executor now asks the broker for a per-run socket when the run's
network is its own (off, or narrowed to an allowlist) on Linux, and
stamps the lease's port and socket onto the spec for exactly those
runs. The runner has one predicate, RelaysModelBroker, read by the
argv, the proxy strip, the confinement record and the admission. Where
it holds, the CLI starts behind `codespace-mcp relay`, just inside
bubblewrap (or directly in an allowlist namespace with no bubblewrap),
so the relay binds the sandbox's own loopback and Codex's stand-down
still lands on the CLI's argv. bubblewrap binds the socket's directory
and the helper's files read-only, and the CLI's directory when it is an
absolute path outside the read-only roots. Every child resolves its
broker to 127.0.0.1, and a network-off run needs no namespace of the
worker's at all, so the seal is gone: its branch, SetupSealedAsync, the
sealed plan and ruleset, the seal-probe prepay, SandboxEgressMode.Sealed
and BrokeredModelCredential.ReachableFromNamespace. A new lease binds
loopback only.

The admission no longer asks whether a namespace can be built. It
refuses a brokered child whose network is its own when the lease came
back without its socket, or when no helper on the worker can run the
relay: none is installed, it is a self-contained publish the sandbox
cannot start, or it does not answer as the relay does when asked to
listen on a port the worker holds. That last one is a build from before
the relay, which CODESPACE_MCP_PROXY_PATH can name; admitted, its CLI
would never start, after the run had spent. The refusal keeps the
sandbox_sealed_egress_unavailable code, which the supervisor and stored
results key on. A worker that cannot build a namespace now admits and
relays such a run; before, it refused it. OffQualifier drops the /30
caveat from the sealed sentence, since a relayed run holds none.

Runs already in flight keep what they had. A handle with a namespace
key and no socket path takes the legacy re-bind: the recorded port,
wide first where namespaces can exist, the 10/8 source gate, and
teardown by name, which still deletes the inet table a sealed run left.
A handle with neither, whose child shares the worker's network, is
re-bound on loopback, where that child calls. An unconfined host never
relays, and macOS mints no socket, so their command, argv and
environment are unchanged; a network-on run with no allowlist gets no
socket. This must deploy after the change that serves a lease over a
socket, so a rollback lands on a worker that re-opens these sockets. A
launch request written by the old code and retried by this one fails
binding-conflict, as the seal's own change accepted.

The sandbox lane gains a step that runs as uid 1654 with no
capabilities and no_new_privs: each of its arms asserts it is not root,
that bubblewrap confines and that no namespace can be built, then runs
the same arm the root lane does, the real claude and codex reviewers
included. In both lanes a relayed child is shown unable to unlink the
lease's socket or plant a file beside it. The broker's integration
tests point the runner at the helper this build produces while a
brokered run executes, as production has it beside the worker, since
that assembly's bin deliberately carries none.
@ppXD
ppXD merged commit 59fc9b1 into main Sep 28, 2026
8 of 9 checks passed
@ppXD ppXD mentioned this pull request Sep 30, 2026
7 of 8 tasks
ppXD added a commit that referenced this pull request Oct 4, 2026
Since #2044 every brokered child with a network of its own reaches its
broker through the codespace-mcp relay and a per-run Unix socket, so
every lease already binds loopback. What was left existed only for a
namespaced run launched before that: a re-bind without a socket path
bound every address, the source gate admitted 10/8, the setup result
carried the veth gateway, and a fixed log line named each such re-bind
so a deployment could tell when none was left.

A re-bind now binds loopback like every open, the source gate admits
loopback alone, and SetupResult.HostIp, the request's
ChildInNetworkNamespace flag and the legacy log line are gone. A handle
that still records a network namespace and no broker socket is not
re-bound at all: nothing listens at its gateway any more, so a re-bind
would clear the posture that says its model access is gone. Such a
straggler lands as ModelCredentialLeaseLost and its agent is stopped.

The legacy E2E arm is flipped rather than dropped. The same pre-guard
namespace on a 10.x lease, re-bound without a socket, now has its child
refused at the gateway while a listener bound at that address and port
is answered, so the kernel says the gateway path is closed, not only
the listener's prefix.

The inet teardown line stays: the veth guard is an inet table of the
same name, and a sealed survivor's table may still need deleting. The
teardown-by-name arm now asserts the kernel lists nothing of the run
afterwards, which the removed legacy arm used to check.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant