Skip to content

Retire the gateway path to the model broker - #2055

Draft
ppXD wants to merge 1 commit into
mainfrom
fix/retire-the-gateway-path-to-the-model-broker
Draft

ppXD wants to merge 1 commit into
mainfrom
fix/retire-the-gateway-path-to-the-model-broker

Conversation

@ppXD

@ppXD ppXD commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • A rebind now binds its recorded port on loopback, with or without a socket path. The wide http://+:<port>/ bind for socketless rebinds is removed: AnyHost, CandidateHosts and WarnIfUnreachableFromNetns are gone, BindPort is at backend/src/CodeSpace.Core/Services/Agents/Credentials/Broker/LoopbackModelCredentialBroker.cs:482 and its caller at :264.
  • The broker's source gate admits loopback only; the 10/8 branch is dropped (LoopbackModelCredentialBroker.cs:687).
  • The legacy model-broker re-bind log line and its request field ModelCredentialRebindRequest.ChildInNetworkNamespace are removed.
  • FilteredEgressNetns.SetupResult.HostIp is removed; setup now returns only the exec prefix (backend/src/CodeSpace.Core/Services/Agents/Sandbox/Isolation/FilteredEgressNetns.cs:201). E2E tests read the gateway off the veth (backend/tests/CodeSpace.SandboxTests/FilteredEgressNetnsE2ETests.cs:764). That includes the port-53 checks from Pin an allowlist run's DNS to the resolvers it uses #2054: the worker-shut arm's gateway_53 assertion, and the REDIRECT arm's proxy and unlisted-port checks.
  • A handle that records a network namespace and no broker socket is no longer re-bound, because its child calls a gateway where nothing listens (backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs:4593, :4611). Such a straggler lands as ModelCredentialLeaseLost and its agent is stopped, instead of being recorded as restored.
  • The inet teardown line in FilteredEgressPlan.TeardownCommandsFor stays (backend/src/CodeSpace.Core/Services/Agents/Sandbox/Isolation/FilteredEgressPlan.cs:162). The durable teardown arm now asserts that the kernel lists no guard table, no forward table and no namespace after teardown (FilteredEgressNetnsE2ETests.cs:69-101).
  • The legacy gateway E2E arm is replaced by a retirement arm, A_gateway_addressed_child_reaches_nothing_after_a_socketless_rebind (backend/tests/CodeSpace.SandboxTests/ModelCredentialBrokerNetnsE2ETests.cs:291). It stages the pre-guard namespace on a 10.x lease and does a socketless rebind. It then asserts three things: the child is refused at <gateway>:<port> (curl exit 7), the worker gets 200 on 127.0.0.1:<port>, and a control listener bound at the same gateway address and port is answered from inside.
  • The sandbox lane's case count does not change, so its floor stays where main has it (90). The two legacy markers are replaced by teardown-by-name and gateway-retired, and the durable teardown arm joins the allowlist-plan list, which now has 15 arms (.github/workflows/sandbox-isolation.yml:237, :290, :298).

Merge precondition

Keep this PR in draft until all four conditions hold. Check them immediately before merging, and again when this PR's deploy starts.

  1. Every running worker is on Reach a namespaced run's model broker through its socket #2044 or later. Any worker on a build older than 59fc9b1ac (Reach a namespaced run's model broker through its socket #2044) still launches gateway-addressed runs, and the query below counts only rows that already exist when it runs.

    • Each process prints its build first at boot (Running build {Build}, backend/src/CodeSpace.Api/Program.cs:50) and stamps it on every log event (Build, :121).
    • For every worker process running now, read its boot line (for example kubectl logs <pod> | grep -m1 'Running build'). Confirm that the sha after the + descends from Reach a namespaced run's model broker through its socket #2044: git merge-base --is-ancestor 59fc9b1ac <sha> must exit 0.
    • A build with no sha cannot be placed, so it counts as a failure.
    • Cross-check in Seq: select count(*) from stream where @Timestamp > Now() - 1d group by Application, Build. This does not replace the per-worker check, because a process that logged nothing in the window is missing from it.
    • Once this condition holds, no new gateway-addressed handle can be written, so the count below can only fall.
  2. The query returns in_flight = 0 and gate_holds = true against the production database:

SELECT count(*) FILTER (WHERE status IN ('Queued', 'Running'))              AS in_flight,
       count(*) FILTER (WHERE status NOT IN ('Queued', 'Running'))          AS terminal_unreaped,
       max(completed_at) FILTER (WHERE status NOT IN ('Queued', 'Running')) AS last_completed_at,
       count(*) FILTER (WHERE status IN ('Queued', 'Running')) = 0
         AND coalesce(max(completed_at) FILTER (WHERE status NOT IN ('Queued', 'Running')) < statement_timestamp() - interval '24 hours', true) AS gate_holds
FROM agent_run
WHERE runner_handle ->> 'modelBrokerPort' IS NOT NULL
  AND coalesce(runner_handle ->> 'egressNetnsKey', '') <> ''
  AND coalesce(runner_handle ->> 'modelBrokerSocketPath', '') = '';
  1. The 24 h spool-reaper window has passed for every such run that is terminal. gate_holds encodes this. If CODESPACE_AGENT_RUN_SPOOL_RETENTION is set longer than 24 h, use that interval in the query instead.

  2. The legacy log line has gone quiet. No log entry containing legacy model-broker re-bind (Information level, source LoopbackModelCredentialBroker) has appeared in the last 24 h on any worker, and at least one worker restart falls inside that window. The line is only logged when a legacy rebind succeeds, which happens on a restart. Builds older than 6dbc339f1 (Serve a model-broker lease over a per-run Unix socket #2042) never log it. So a quiet line means nothing unless condition 0 also holds.

A run that slips past the gate is not silently broken. When this build re-attaches it, RebindRequestFor returns null (AgentRunExecutor.cs:4593), the lost lease is recorded, its agent is stopped, and the run lands Failed with exit reason ModelCredentialLeaseLost.

Test plan

  • Unit, full suite: 11592 passed, 1 skipped. New or flipped cases:
    • A_rebind_binds_loopback_only_with_a_socket_or_without_even_where_a_per_run_namespace_can_exist
    • Only_loopback_is_a_plausible_sandbox_source
    • A_rebind_is_not_built_for_a_handle_whose_child_calls_its_namespace_gateway
  • Integration: AgentRunExecutorTests, 145 passed. This includes the new A_reattach_of_a_run_whose_child_calls_its_namespace_gateway_asks_for_no_re_bind_and_lands_it_typed and the loopback row of A_reattach_re_opens_the_broker_socket_its_handle_recorded_and_a_handle_without_one_is_served_on_loopback.
  • Integration: the re-attach, recovery, spool-reaper and cross-host-abandon classes, 46 passed.
  • Sandbox lane replayed on a privileged container with the real CLIs: executed=87 passed=87 (before the Pin every agent run to its own CLI settings #2068 rebase), non-root 10/10, unconfined 3/3. Every marker assertion passes, including "All 15 allowlist-plan arms ran and passed.", "All 4 broker arms ran.", [broker-socket-e2e] ran gateway-retired uid=0 lease=10.254.134.152/30 gatewayExit=7, and the Pin an allowlist run's DNS to the resolvers it uses #2054 DNS arms (dns-pinned, resolv-conf-view, dns-dnat, dns-redirect, and gateway_53 shut under the guard and open in the control).
  • Rebased onto Pin every agent run to its own CLI settings #2068 (floor 90, non-root 11): the only conflict was the floor line, which keeps both edits. Locally green after the rebase: broker, executor, egress, harness and sandbox unit tests (1079), and AgentRunExecutor* integration (150).
  • Mutations turn the suite red. They were run before this branch was rebased onto Pin an allowlist run's DNS to the resolvers it uses #2054, which changed none of the lines they target:
    • Restoring the 10/8 branch.
    • Restoring the wide bind. Unit fails; the retirement arm fails on the listener prefix; and with that assertion removed, it still fails on the kernel, where the gateway answered 401 instead of refusing.
    • Swapping the arm's pre-guard fixture for the guarded plan. The arm fails because curl times out (exit 28) instead of being refused, so the arm cannot pass on a namespace that cannot reach its gateway.
    • Dropping the executor gate, at both the unit and the integration tier.
    • Dropping the inet teardown line, on a real kernel.
  • Gate query checked against handles written by a real re-attach. It counts the staged pre-socket survivor and excludes runs that have a socket or no namespace.
  • Merge precondition above holds in production.

Since #2044 every brokered child with a network of its own reaches its
broker through the codespace-mcp relay and a per-run Unix socket, so
every lease already binds loopback. What was left existed only for a
namespaced run launched before that: a re-bind without a socket path
bound every address, the source gate admitted 10/8, the setup result
carried the veth gateway, and a fixed log line named each such re-bind
so a deployment could tell when none was left.

A re-bind now binds loopback like every open, the source gate admits
loopback alone, and SetupResult.HostIp, the request's
ChildInNetworkNamespace flag and the legacy log line are gone. A handle
that still records a network namespace and no broker socket is not
re-bound at all: nothing listens at its gateway any more, so a re-bind
would clear the posture that says its model access is gone. Such a
straggler lands as ModelCredentialLeaseLost and its agent is stopped.

The legacy E2E arm is flipped rather than dropped. The same pre-guard
namespace on a 10.x lease, re-bound without a socket, now has its child
refused at the gateway while a listener bound at that address and port
is answered, so the kernel says the gateway path is closed, not only
the listener's prefix.

The inet teardown line stays: the veth guard is an inet table of the
same name, and a sealed survivor's table may still need deleting. The
teardown-by-name arm now asserts the kernel lists nothing of the run
afterwards, which the removed legacy arm used to check.
@ppXD
ppXD force-pushed the fix/retire-the-gateway-path-to-the-model-broker branch from 7716d48 to f4e1251 Compare October 4, 2026 07:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant