Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 11 additions & 12 deletions .github/workflows/sandbox-isolation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ jobs:
passed=$(grep -oE 'passed="[0-9]+"' "$trx" | head -1 | grep -oE '[0-9]+')
echo "executed=${executed:-0} passed=${passed:-0}"
if [ "${executed:-0}" -lt 90 ]; then
echo "::error::Expected >=90 sandbox isolation tests to run (bwrap/prlimit confinement + cap-drop + cgroup-namespace re-root + egress-allowlist filter + cgroup resource cap + durable-launch cgroup wiring + argv/envp per-string kernel ceiling + a prompt past it riding stdin + a read-only workspace mount + a network-off run reaching its broker through the relay and nothing else + an allowlist run relayed to its broker + a read-only reviewer reading its diff with the real CLIs + a bwrap probe that runs the launch argv + the MCP helper bound file by file behind a read-only socket dir + a CLI reaching its broker socket through the relay + a severed child reaching its broker over the lease socket across a worker restart + a pre-relay namespaced run re-bound at its gateway and torn down with its seal + an allowlist run's veth guarded both ways, a flow the worker opened before the run included + forwarding a root worker may not write named before an allowlist is planned + an allowlist run's port 53 open only at the resolvers of the resolv.conf its namespace reads, and still to a resolver address the worker's own NAT rewrites before its forward and its input hooks + a target repository's own CLI config kept out of the run with the real CLIs, every repository's memory read in a multi-repo workspace included), but only ${executed:-0} did — the Category=Sandbox filter matched too few (trait regression?). If a case was deliberately removed, lower this number in the same PR."
echo "::error::Expected >=90 sandbox isolation tests to run (bwrap/prlimit confinement + cap-drop + cgroup-namespace re-root + egress-allowlist filter + cgroup resource cap + durable-launch cgroup wiring + argv/envp per-string kernel ceiling + a prompt past it riding stdin + a read-only workspace mount + a network-off run reaching its broker through the relay and nothing else + an allowlist run relayed to its broker + a read-only reviewer reading its diff with the real CLIs + a bwrap probe that runs the launch argv + the MCP helper bound file by file behind a read-only socket dir + a CLI reaching its broker socket through the relay + a severed child reaching its broker over the lease socket across a worker restart + a pre-relay namespaced run's gateway answering nothing after its re-bind + an allowlist run's veth guarded both ways, a flow the worker opened before the run included + forwarding a root worker may not write named before an allowlist is planned + an allowlist run's port 53 open only at the resolvers of the resolv.conf its namespace reads, and still to a resolver address the worker's own NAT rewrites before its forward and its input hooks + a target repository's own CLI config kept out of the run with the real CLIs, every repository's memory read in a multi-repo workspace included), but only ${executed:-0} did — the Category=Sandbox filter matched too few (trait regression?). If a case was deliberately removed, lower this number in the same PR."
exit 1
fi

Expand Down Expand Up @@ -283,22 +283,21 @@ jobs:
assert len(cases) == rows and all(r.get('outcome') == 'Passed' for r in cases), f'{method}: all {rows} case(s) must pass'
print('All 6 sealed-egress arms ran and passed.')

# The allowlist plan's host-routing arms, the guard on its veth, its DNS pinned to the resolvers of the resolv.conf
# its namespace reads and still answered where the worker's own NAT rewrites one, the relayed allowlist launch,
# and the forwarding probe's read-only arm return early without ip and nft, or off root; require their markers.
# The IPv6 arm prints its marker only where the veth has a link-local.
for marker in ('[filtered-egress-e2e] ran restart-reissue', '[filtered-egress-e2e] ran policy-route-discard-dst', '[filtered-egress-e2e] ran worker-shut', '[filtered-egress-e2e] ran ipv6-link-local', '[filtered-egress-e2e] ran shadowed-peer-refused', '[filtered-egress-e2e] ran established-flow-refused', '[filtered-egress-e2e] ran pmtu-upload', '[filtered-egress-e2e] ran stale-guard-replaced', '[filtered-egress-e2e] ran forwarding-read-only', '[filtered-egress-e2e] ran dns-pinned', '[filtered-egress-e2e] ran resolv-conf-view', '[filtered-egress-e2e] ran dns-dnat', '[filtered-egress-e2e] ran dns-redirect', '[durable-egress-e2e] ran allowlist-relay'):
# The allowlist plan's host-routing arms, the guard on its veth, the teardown by name, its DNS pinned to the
# resolvers of the resolv.conf its namespace reads and still answered where the worker's own NAT rewrites one, the
# relayed allowlist launch, and the forwarding probe's read-only arm return early without ip and nft, or off root;
# require their markers. The IPv6 arm prints its marker only where the veth has a link-local.
for marker in ('[filtered-egress-e2e] ran teardown-by-name', '[filtered-egress-e2e] ran restart-reissue', '[filtered-egress-e2e] ran policy-route-discard-dst', '[filtered-egress-e2e] ran worker-shut', '[filtered-egress-e2e] ran ipv6-link-local', '[filtered-egress-e2e] ran shadowed-peer-refused', '[filtered-egress-e2e] ran established-flow-refused', '[filtered-egress-e2e] ran pmtu-upload', '[filtered-egress-e2e] ran stale-guard-replaced', '[filtered-egress-e2e] ran forwarding-read-only', '[filtered-egress-e2e] ran dns-pinned', '[filtered-egress-e2e] ran resolv-conf-view', '[filtered-egress-e2e] ran dns-dnat', '[filtered-egress-e2e] ran dns-redirect', '[durable-egress-e2e] ran allowlist-relay'):
assert marker in text, f'"{marker}" is missing — this lane is root with ip and nft, so the allowlist plan must run'
for method in ('FilteredEgressNetnsE2ETests.A_30_still_held_by_a_run_that_outlived_its_worker_is_not_handed_to_the_next_run', 'FilteredEgressNetnsE2ETests.A_host_whose_policy_rule_discards_the_run_s_replies_fails_the_setup_and_leaks_nothing', 'FilteredEgressNetnsE2ETests.An_allowlist_run_reaches_neither_the_worker_s_gateway_nor_its_address_while_dns_on_the_worker_and_the_allowlist_still_answer', 'FilteredEgressNetnsE2ETests.An_allowlist_run_has_no_path_to_the_worker_over_the_veth_s_ipv6_link_local_either', 'FilteredEgressNetnsE2ETests.A_peer_the_worker_reaches_at_the_run_s_address_is_refused_and_the_sandbox_receives_nothing', 'FilteredEgressNetnsE2ETests.A_flow_the_worker_opened_to_the_shadowed_peer_before_the_run_is_neither_handed_to_the_sandbox_nor_answered_from_it', 'FilteredEgressNetnsE2ETests.An_upload_across_a_narrower_uplink_completes_because_the_worker_s_frag_needed_reaches_the_run', 'FilteredEgressNetnsE2ETests.A_guard_an_earlier_teardown_left_behind_is_replaced_not_added_to', 'FilteredEgressNetnsE2ETests.Forwarding_a_root_worker_may_not_write_is_named_before_an_allowlist_is_planned_on_it', 'FilteredEgressNetnsE2ETests.An_allowlist_run_reaches_port_53_only_at_the_resolver_its_resolv_conf_names', 'FilteredEgressNetnsE2ETests.A_namespace_the_setup_builds_reads_the_worker_s_resolv_conf_and_opens_port_53_to_its_resolvers_alone', 'FilteredEgressNetnsE2ETests.A_resolver_address_the_worker_dnats_before_its_forward_hook_still_answers_the_run', 'FilteredEgressNetnsE2ETests.A_resolver_address_the_worker_redirects_to_itself_still_answers_the_run', 'DurableLaunchEgressE2ETests.An_allowlist_run_reaches_its_broker_through_the_relay_and_its_allowlist_still_holds'):
for method in ('FilteredEgressNetnsE2ETests.The_durable_setup_teardown_split_enforces_the_filter_and_teardown_is_reconstructable_from_runId', 'FilteredEgressNetnsE2ETests.A_30_still_held_by_a_run_that_outlived_its_worker_is_not_handed_to_the_next_run', 'FilteredEgressNetnsE2ETests.A_host_whose_policy_rule_discards_the_run_s_replies_fails_the_setup_and_leaks_nothing', 'FilteredEgressNetnsE2ETests.An_allowlist_run_reaches_neither_the_worker_s_gateway_nor_its_address_while_dns_on_the_worker_and_the_allowlist_still_answer', 'FilteredEgressNetnsE2ETests.An_allowlist_run_has_no_path_to_the_worker_over_the_veth_s_ipv6_link_local_either', 'FilteredEgressNetnsE2ETests.A_peer_the_worker_reaches_at_the_run_s_address_is_refused_and_the_sandbox_receives_nothing', 'FilteredEgressNetnsE2ETests.A_flow_the_worker_opened_to_the_shadowed_peer_before_the_run_is_neither_handed_to_the_sandbox_nor_answered_from_it', 'FilteredEgressNetnsE2ETests.An_upload_across_a_narrower_uplink_completes_because_the_worker_s_frag_needed_reaches_the_run', 'FilteredEgressNetnsE2ETests.A_guard_an_earlier_teardown_left_behind_is_replaced_not_added_to', 'FilteredEgressNetnsE2ETests.Forwarding_a_root_worker_may_not_write_is_named_before_an_allowlist_is_planned_on_it', 'FilteredEgressNetnsE2ETests.An_allowlist_run_reaches_port_53_only_at_the_resolver_its_resolv_conf_names', 'FilteredEgressNetnsE2ETests.A_namespace_the_setup_builds_reads_the_worker_s_resolv_conf_and_opens_port_53_to_its_resolvers_alone', 'FilteredEgressNetnsE2ETests.A_resolver_address_the_worker_dnats_before_its_forward_hook_still_answers_the_run', 'FilteredEgressNetnsE2ETests.A_resolver_address_the_worker_redirects_to_itself_still_answers_the_run', 'DurableLaunchEgressE2ETests.An_allowlist_run_reaches_its_broker_through_the_relay_and_its_allowlist_still_holds'):
cases = [r for r in results if method in r.get('testName', '')]
assert len(cases) == 1 and cases[0].get('outcome') == 'Passed', f'{method}: must pass'
print('All 14 allowlist-plan arms ran and passed.')
print('All 15 allowlist-plan arms ran and passed.')

# The broker's socket, relay-revoke and legacy-gateway arms return early without bwrap or ip/nft; require each
# marker, the legacy survivor's teardown of the seal it carried included.
for arm in ('socket-channel', 'revoke-network-off', 'revoke-allowlist', 'legacy-gateway-rebind', 'legacy-sealed-teardown'):
# The broker's socket, relay-revoke and retired-gateway arms return early without bwrap or ip/nft; require each marker.
for arm in ('socket-channel', 'revoke-network-off', 'revoke-allowlist', 'gateway-retired'):
assert f'[broker-socket-e2e] ran {arm}' in text, f'broker E2E arm "{arm}" did not run — this lane is root with bwrap, ip and nft'
print('All 5 broker arms ran.')
print('All 4 broker arms ran.')

# The bwrap probe E2E returns early off root, which reads as Passed; require each arm's marker.
for arm in ('masked-proc', 'unmasked-proc'):
Expand Down
20 changes: 17 additions & 3 deletions backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs
Original file line number Diff line number Diff line change
Expand Up @@ -4568,12 +4568,16 @@
}

/// <summary>
/// The re-bind this run's handle makes possible, or null when it makes none. Three gates, each of which would
/// The re-bind this run's handle makes possible, or null when it makes none. Four gates, each of which would
/// otherwise produce a lease that answers the wrong thing:
///
/// <para><b>The address.</b> Port + route + bearer must all be recorded. A handle stamped before they were is a
/// run whose port nobody wrote down, and that is the mixed-version deploy case: it keeps the typed landing.</para>
///
/// <para><b>The door.</b> See <see cref="CallsItsBrokerAtAGateway"/> — a child that calls an address no broker
/// serves any more is not restored by a lease on loopback, and a re-bind that took would clear the posture that says
/// its model access is gone.</para>
///
/// <para><b>The host.</b> The agent calls a port on the machine it was launched on. Binding that number HERE, on a
/// worker that is not that machine, would answer nobody at all — while clearing the posture that says the run's
/// access is gone. Same predicate the runner uses before answering any other pid-derived question, and it admits
Expand All @@ -4584,22 +4588,32 @@
/// <para><b>The credential.</b> See <see cref="FrontsTheSameCredential"/> — a resolve that landed on a different
/// ROW is not a restoration.</para>
///
/// <para>Takes what it reads rather than the whole context, so the three gates are directly testable (Rule 1 —
/// <para>Takes what it reads rather than the whole context, so the four gates are directly testable (Rule 1 —
/// four parameters, under the cap).</para>
/// </summary>
internal static ModelCredentialRebindRequest? RebindRequestFor(AgentRunOwnerToken owner, Guid teamId, SandboxHandle handle, ResolvedModelCredential? upstream)
{
if (handle.ModelBrokerRunToken is not { Length: > 0 } token || handle.ModelBrokerRoute is not { Length: > 0 } route || handle.ModelBrokerPort is not { } port) return null;
if (CallsItsBrokerAtAGateway(handle)) return null;
if (!LocalProcessRunner.PidAnswerableHere(handle)) return null;
if (upstream is not { } resolved || !FrontsTheSameCredential(handle, resolved)) return null;

return new()
{
RunId = owner.RunId, TeamId = teamId, Epoch = owner.Epoch, Port = port, PathId = route, RunToken = token, Upstream = resolved, Ttl = Credentials.ModelCredentialLease.Ttl,
SocketPath = handle.ModelBrokerSocketPath, ChildInNetworkNamespace = handle.EgressNetnsKey is { Length: > 0 },
SocketPath = handle.ModelBrokerSocketPath,
};
}

/// <summary>
/// Whether this handle's child reaches its broker at its network namespace's gateway rather than through a socket:
/// a namespace recorded and no broker socket. Only a run launched before a namespaced child reached its broker
/// through a socket has that shape — every brokered launch into a network of its own since mints one or is refused
/// — and nothing listens at a gateway any more, since every lease binds loopback. So such a run keeps the typed
/// landing, and its agent is stopped rather than left calling nothing.
/// </summary>
private static bool CallsItsBrokerAtAGateway(SandboxHandle handle) => handle.EgressNetnsKey is { Length: > 0 } && handle.ModelBrokerSocketPath is null;

/// <summary>
/// Whether the credential this pass resolved is the SAME one the launch's lease fronted — the row id when the
/// launch named one (both null is the operator-global key, which has no row) and the provider tag either way.
Expand Down Expand Up @@ -5006,10 +5020,10 @@
/// <summary>The same reconstruction from a payload that came from somewhere other than the row — an offloaded one fetched back out of the artifact store.</summary>
private static AgentEvent ReplayedEvent(AgentEventKind kind, string? text, string? dataJson)
{
if (dataJson is not { Length: > 0 } json) return new AgentEvent { Kind = kind, Text = text };

Check warning on line 5023 in backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs

View workflow job for this annotation

GitHub Actions / recurring jobs fire (worker host · Postgres)

Possible null reference assignment.

Check warning on line 5023 in backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs

View workflow job for this annotation

GitHub Actions / dotnet test (E2ETests · HTTP · Postgres)

Possible null reference assignment.

Check warning on line 5023 in backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs

View workflow job for this annotation

GitHub Actions / dotnet test (UnitTests)

Possible null reference assignment.

Check warning on line 5023 in backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs

View workflow job for this annotation

GitHub Actions / dotnet test (UnitTests)

Possible null reference assignment.

Check warning on line 5023 in backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs

View workflow job for this annotation

GitHub Actions / dotnet test (IntegrationTests · Postgres)

Possible null reference assignment.

try { using var doc = JsonDocument.Parse(json); return new AgentEvent { Kind = kind, Text = text, Data = doc.RootElement.Clone() }; }

Check warning on line 5025 in backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs

View workflow job for this annotation

GitHub Actions / recurring jobs fire (worker host · Postgres)

Possible null reference assignment.

Check warning on line 5025 in backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs

View workflow job for this annotation

GitHub Actions / dotnet test (E2ETests · HTTP · Postgres)

Possible null reference assignment.

Check warning on line 5025 in backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs

View workflow job for this annotation

GitHub Actions / dotnet test (UnitTests)

Possible null reference assignment.

Check warning on line 5025 in backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs

View workflow job for this annotation

GitHub Actions / dotnet test (UnitTests)

Possible null reference assignment.

Check warning on line 5025 in backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs

View workflow job for this annotation

GitHub Actions / dotnet test (IntegrationTests · Postgres)

Possible null reference assignment.
catch (JsonException) { return new AgentEvent { Kind = kind, Text = text }; }

Check warning on line 5026 in backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs

View workflow job for this annotation

GitHub Actions / recurring jobs fire (worker host · Postgres)

Possible null reference assignment.

Check warning on line 5026 in backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs

View workflow job for this annotation

GitHub Actions / dotnet test (E2ETests · HTTP · Postgres)

Possible null reference assignment.

Check warning on line 5026 in backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs

View workflow job for this annotation

GitHub Actions / dotnet test (UnitTests)

Possible null reference assignment.

Check warning on line 5026 in backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs

View workflow job for this annotation

GitHub Actions / dotnet test (UnitTests)

Possible null reference assignment.

Check warning on line 5026 in backend/src/CodeSpace.Core/Services/Agents/AgentRunExecutor.cs

View workflow job for this annotation

GitHub Actions / dotnet test (IntegrationTests · Postgres)

Possible null reference assignment.
}

/// <summary>Ask the row, on a token of its own, whether the run actually reached a terminal state — the only honest answer to "did the landing take?" once an exception has been raised somewhere after the fenced write.</summary>
Expand Down
Loading
Loading