Conversation
Since #2044 every brokered child with a network of its own reaches its broker through the codespace-mcp relay and a per-run Unix socket, so every lease already binds loopback. What was left existed only for a namespaced run launched before that: a re-bind without a socket path bound every address, the source gate admitted 10/8, the setup result carried the veth gateway, and a fixed log line named each such re-bind so a deployment could tell when none was left. A re-bind now binds loopback like every open, the source gate admits loopback alone, and SetupResult.HostIp, the request's ChildInNetworkNamespace flag and the legacy log line are gone. A handle that still records a network namespace and no broker socket is not re-bound at all: nothing listens at its gateway any more, so a re-bind would clear the posture that says its model access is gone. Such a straggler lands as ModelCredentialLeaseLost and its agent is stopped. The legacy E2E arm is flipped rather than dropped. The same pre-guard namespace on a 10.x lease, re-bound without a socket, now has its child refused at the gateway while a listener bound at that address and port is answered, so the kernel says the gateway path is closed, not only the listener's prefix. The inet teardown line stays: the veth guard is an inet table of the same name, and a sealed survivor's table may still need deleting. The teardown-by-name arm now asserts the kernel lists nothing of the run afterwards, which the removed legacy arm used to check.
ppXD
force-pushed
the
fix/retire-the-gateway-path-to-the-model-broker
branch
from
October 4, 2026 07:12
7716d48 to
f4e1251
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
http://+:<port>/bind for socketless rebinds is removed:AnyHost,CandidateHostsandWarnIfUnreachableFromNetnsare gone,BindPortis atbackend/src/CodeSpace.Core/Services/Agents/Credentials/Broker/LoopbackModelCredentialBroker.cs:482and its caller at:264.LoopbackModelCredentialBroker.cs:687).legacy model-broker re-bindlog line and its request fieldModelCredentialRebindRequest.ChildInNetworkNamespaceare removed.FilteredEgressNetns.SetupResult.HostIpis removed; setup now returns only the exec prefix (backend/src/CodeSpace.Core/Services/Agents/Sandbox/Isolation/FilteredEgressNetns.cs:201). E2E tests read the gateway off the veth (backend/tests/CodeSpace.SandboxTests/FilteredEgressNetnsE2ETests.cs:764). That includes the port-53 checks from Pin an allowlist run's DNS to the resolvers it uses #2054: the worker-shut arm'sgateway_53assertion, and the REDIRECT arm's proxy and unlisted-port checks.backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs:4593,:4611). Such a straggler lands asModelCredentialLeaseLostand its agent is stopped, instead of being recorded as restored.FilteredEgressPlan.TeardownCommandsForstays (backend/src/CodeSpace.Core/Services/Agents/Sandbox/Isolation/FilteredEgressPlan.cs:162). The durable teardown arm now asserts that the kernel lists no guard table, no forward table and no namespace after teardown (FilteredEgressNetnsE2ETests.cs:69-101).A_gateway_addressed_child_reaches_nothing_after_a_socketless_rebind(backend/tests/CodeSpace.SandboxTests/ModelCredentialBrokerNetnsE2ETests.cs:291). It stages the pre-guard namespace on a 10.x lease and does a socketless rebind. It then asserts three things: the child is refused at<gateway>:<port>(curl exit 7), the worker gets 200 on127.0.0.1:<port>, and a control listener bound at the same gateway address and port is answered from inside.teardown-by-nameandgateway-retired, and the durable teardown arm joins the allowlist-plan list, which now has 15 arms (.github/workflows/sandbox-isolation.yml:237,:290,:298).Merge precondition
Keep this PR in draft until all four conditions hold. Check them immediately before merging, and again when this PR's deploy starts.
Every running worker is on Reach a namespaced run's model broker through its socket #2044 or later. Any worker on a build older than
59fc9b1ac(Reach a namespaced run's model broker through its socket #2044) still launches gateway-addressed runs, and the query below counts only rows that already exist when it runs.Running build {Build},backend/src/CodeSpace.Api/Program.cs:50) and stamps it on every log event (Build,:121).kubectl logs <pod> | grep -m1 'Running build'). Confirm that the sha after the+descends from Reach a namespaced run's model broker through its socket #2044:git merge-base --is-ancestor 59fc9b1ac <sha>must exit 0.select count(*) from stream where @Timestamp > Now() - 1d group by Application, Build. This does not replace the per-worker check, because a process that logged nothing in the window is missing from it.The query returns
in_flight = 0andgate_holds = trueagainst the production database:The 24 h spool-reaper window has passed for every such run that is terminal.
gate_holdsencodes this. IfCODESPACE_AGENT_RUN_SPOOL_RETENTIONis set longer than 24 h, use that interval in the query instead.The legacy log line has gone quiet. No log entry containing
legacy model-broker re-bind(Information level, sourceLoopbackModelCredentialBroker) has appeared in the last 24 h on any worker, and at least one worker restart falls inside that window. The line is only logged when a legacy rebind succeeds, which happens on a restart. Builds older than6dbc339f1(Serve a model-broker lease over a per-run Unix socket #2042) never log it. So a quiet line means nothing unless condition 0 also holds.A run that slips past the gate is not silently broken. When this build re-attaches it,
RebindRequestForreturns null (AgentRunExecutor.cs:4593), the lost lease is recorded, its agent is stopped, and the run landsFailedwith exit reasonModelCredentialLeaseLost.Test plan
A_rebind_binds_loopback_only_with_a_socket_or_without_even_where_a_per_run_namespace_can_existOnly_loopback_is_a_plausible_sandbox_sourceA_rebind_is_not_built_for_a_handle_whose_child_calls_its_namespace_gatewayAgentRunExecutorTests, 145 passed. This includes the newA_reattach_of_a_run_whose_child_calls_its_namespace_gateway_asks_for_no_re_bind_and_lands_it_typedand the loopback row ofA_reattach_re_opens_the_broker_socket_its_handle_recorded_and_a_handle_without_one_is_served_on_loopback.[broker-socket-e2e] ran gateway-retired uid=0 lease=10.254.134.152/30 gatewayExit=7, and the Pin an allowlist run's DNS to the resolvers it uses #2054 DNS arms (dns-pinned,resolv-conf-view,dns-dnat,dns-redirect, andgateway_53shut under the guard and open in the control).AgentRunExecutor*integration (150).