Skip to content

Name the broker reply in the sealed route check - #2038

Merged
ppXD merged 1 commit into
mainfrom
fix/name-the-broker-reply-in-the-route-check
Sep 27, 2026
Merged

ppXD merged 1 commit into
mainfrom
fix/name-the-broker-reply-in-the-route-check

Conversation

@ppXD

@ppXD ppXD commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • A sealed namespace's setup check (FilteredEgressPlan.RouteCheckArgv, added in Seal a network-off run to its model broker #2035) asked ip route get <ns> from <gateway>. The broker's replies are TCP from the broker port, so a policy rule keyed on the protocol or the port was invisible to it. Two examples: ip rule add ipproto tcp lookup <vpn-table>, or a null route behind an ipproto/sport rule. The check then admitted a sealed run, and every brokered call timed out. A sealed plan now asks the reply's own lookup, adding ipproto 6 sport <port>.
    • The protocol is given by number. The name tcp needs /etc/protocols, which minimal Debian and Ubuntu images lack, and there ip answers Invalid "ipproto" value.
    • It still cannot name what the broker cannot know ahead of time. Each reply goes to the agent's own ephemeral port and may carry a mark, so a rule keyed on the destination port or on a mark still gets past the check. The doc says so.
    • An allowlist plan has no single port to name, so its check is unchanged. Its doc now says the check does not see rules keyed on those selectors or on the uplink.
  • Two pins the last review found missing:
    • The launch-time setup failure carries the "fix what that step names" remedy. Reverting the raise site to the grant-remedy constructor stayed green in every tier before.
    • The proxy gate counts a task-set ALL_PROXY, which the child gets unfiltered. The worker's own NO_PROXY stays in the exemption list.

Test plan

  • Unit:
    • FilteredEgressPlanTests pins the sealed argv with ipproto 6 sport <port> and the allowlist argv without it.
    • LocalProcessRunnerEnvScrubTests adds a task ALL_PROXY row and a worker NO_PROXY union row.
    • Mutation-checked: filtering the task's own value through the env allowlist, or dropping the worker's NO_PROXY from the union, turns the new rows red.
  • Unit suite: full run green locally (11317).
  • Sandbox classes on a real kernel, locally (privileged Ubuntu 24.04 container, IPv6 on, iproute2 6.1, nft 1.0.9): SealedEgressE2ETests, FilteredEgressNetnsE2ETests, ModelCredentialBrokerNetnsE2ETests, DurableLaunchEgressE2ETests, 14/14, no host residue.
    • A_host_whose_policy_rule_discards_the_run_s_replies_fails_the_setup_and_leaks_nothing becomes a theory. The new row installs a rule that only TCP from the broker port meets.
    • Mutation-checked: removing the selectors from the sealed argv turns that row red, while the destination-keyed row stays green.
    • The setup-failure arm now requires the fix-that-step remedy in the message.
  • Adversarial review on a real kernel: the production ApplyAsync(BuildSealed) was compared against a real TCP exchange to the broker port.
    • Head agrees with ground truth on every configuration tested: clean, rp_filter=1, a real wg-quick tunnel, Tailscale with and without an exit node, a Cilium-style setup, AWS VPC CNI, a Calico blackhole, and rules keyed on ipproto, sport, dport, uidrange, fwmark, tos and iif. The one exception is a contrived mark case, and base behaves identically there.
    • Wherever base and head disagree, head is right. Base admitted a dead broker behind ipproto 6 and sport rules. It also refused a working one under an ipproto 17 rule, because a route get with no protocol is answered as UDP.
    • The qualified argv also works end to end on Debian 10's iproute2 4.20. Builds that reject ipproto (4.17 and earlier) already fail at the route listing.
  • Sandbox lane: floor 67. The arms policy-route-discard-dst and policy-route-discard-l4 each have their own marker, so neither can be satisfied by the other's.

@ppXD
ppXD force-pushed the fix/name-the-broker-reply-in-the-route-check branch from f6f4a34 to 7bc0b35 Compare September 27, 2026 10:46
The setup's route check asked `ip route get <ns> from <gateway>`, but
the broker's replies are TCP from the broker port. A policy rule keyed
on the protocol or the port (all TCP through a VPN table, a null route
behind an ipproto or sport rule) was invisible to the check, so a
sealed run was admitted and every brokered call then timed out. A
sealed plan now asks the reply's own lookup, adding `ipproto 6 sport
<port>`. The protocol is given by number because the name needs
/etc/protocols, which minimal images lack. A rule keyed on the
destination port or a mark still gets past it, since each reply goes to
the agent's ephemeral port; the doc says so. An allowlist plan has no
single port to name, so its check is unchanged and its doc says what it
does not see.

Two pins the last review found missing are added. The launch-time
setup failure must carry the fix-that-step remedy, not the grant
remedy. The proxy gate must count a task-set ALL_PROXY, and must keep
the worker's own NO_PROXY in the exemption list.
@ppXD
ppXD merged commit 6afe829 into main Sep 27, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant