Skip to content

Document the rootless worker posture and commit its seccomp profile - #2043

Merged
ppXD merged 1 commit into
mainfrom
docs/document-the-rootless-worker-posture
Sep 27, 2026
Merged

ppXD merged 1 commit into
mainfrom
docs/document-the-rootless-worker-posture

Conversation

@ppXD

@ppXD ppXD commented Sep 27, 2026

Copy link
Copy Markdown
Owner

Summary

  • Commit backend/deploy/seccomp/codespace-worker.json. It is moby v20.10.20's default profile resolved for a container with no capabilities, plus one unconditional allow of the seven calls bubblewrap needs (clone, clone3, mount, pivot_root, setns, umount2, unshare).
    • With it, plus apparmor=unconfined and systempaths=unconfined, the unmodified worker image confines as uid 1654. It needs no --privileged and no capability, and a network-off child sees only loopback.
    • The file is in OCI runtime-spec form, with no includes, excludes or archMap. containerd decodes a Localhost profile straight into specs.LinuxSeccomp (contrib/seccomp/seccomp.go) and silently drops those keys. A moby-format file would therefore have become an unconditional allow of 39 capability-gated calls on the Kubernetes path, among them the new mount API, open_by_handle_at, bpf and process_vm_readv/writev.
    • derive-codespace-worker.sh regenerates the file from the sha256-pinned moby source.
  • Rewrite the Dockerfile.worker confinement header per tier, with a Kubernetes securityContext for the same posture (hostUsers: false, procMount: Unmasked, Localhost seccomp, AppArmor Unconfined, non-root, drop ALL; privileged: true with runAsUser: 1654 on older clusters). Two old statements were wrong:
    • an allowlist run on an unprivileged pod aborts at launch rather than "just gets Denied";
    • an allowlist run also needs CAP_SYS_ADMIN, because ip netns add needs it.
  • Add a commented security_opt block to docker-compose.yml; the default stack is unchanged. Both the compose comment and the Dockerfile say that a masked /proc still passes the bubblewrap probe (the probe mounts no /proc) while every launch fails. An operator should therefore check that the first run's agent starts, not rely on the boot line alone.
  • Log one Sandbox posture: line at worker boot. It gives the bubblewrap probe and its reason, whether the codespace-mcp helper is present, and whether FilteredEgressNetns.CanSeal holds and why not.

Test plan

  • Unit: RootlessWorkerPostureTests (13):
    • the seven calls are allowed unconditionally;
    • none of the other 39 calls moby gates on a capability is allowed;
    • only OCI LinuxSeccomp/LinuxSyscall keys appear, so Docker, containerd and CRI-O read one filter;
    • the file's SHA-256 is pinned to the derivation;
    • the boot line carries each probe.
  • Unit: full suite (11330 passed, 1 skipped)
  • E2E: RecurringJobWorkerSmokeE2ETests.The_worker_host_logs_its_sandbox_posture_once_at_boot. It boots the real Worker-role host and fails if the registrar stops emitting the line.
  • Integration: HangfireHostingRoleTests, SandboxConfigurationStartupTests (25)
  • Sandbox lane, privileged: BubblewrapConfinementSandboxTests, DurableLaunchEgressE2ETests, SealedEgressE2ETests, CommandIsolationE2ETests (26), no host residue
  • Container proof: worker image, not privileged, arm64 kernel.
    • uid 1654 under the profile: confines, and the network-off child has only lo.
    • Same with drop ALL + no-new-privs under containerd's reading of the file.
    • Four capability-gated calls return EPERM under both the Docker and the containerd reading.
    • Profile with pivot_root removed: the probe reports unavailable and RequireConfinement refuses.
    • root + NET_ADMIN + SYS_ADMIN under the profile: confines, and CanSeal holds.
  • Mutations: widening the rule, removing pivot_root, adding a capability-gated rule or an archMap key, and deleting the registrar call each turn a test red.
  • amd64 node: confirm the worker confines under the profile (only arm64 has been measured).
  • containerd 2.x node: confirm it reads the Localhost profile as the emulated reading did.

@ppXD
ppXD force-pushed the feat/serve-a-broker-lease-over-a-unix-socket branch from 41e1daf to 3af4728 Compare September 27, 2026 18:56
@ppXD
ppXD changed the base branch from feat/serve-a-broker-lease-over-a-unix-socket to main September 27, 2026 18:56
@ppXD
ppXD force-pushed the docs/document-the-rootless-worker-posture branch from e2c0994 to e5866c5 Compare September 27, 2026 18:56
The worker image runs as uid 1654 with no capabilities, but nothing in
the repo said how to let that user confine. Docker's default seccomp
profile reserves clone/unshare with namespace flags, setns, mount and
umount2 for CAP_SYS_ADMIN and denies pivot_root outright, so bubblewrap
fails its probe and every run is recorded unconfined. The only remedies
people reached for were --privileged or seccomp=unconfined.

codespace-worker.json is moby v20.10.20's default resolved for a
container with no capabilities, plus one unconditional allow of those
seven calls. It is written in the OCI runtime-spec form, with no
includes, excludes or archMap. containerd decodes a Localhost profile
straight into the OCI LinuxSeccomp struct and silently drops those
keys, so a moby-format file would have turned every capability-gated
rule into an unconditional allow on the Kubernetes path the Dockerfile
recommends first: 39 calls, among them the new mount API,
open_by_handle_at, bpf and process_vm_readv/writev. Resolving the
conditions up front makes Docker, containerd and CRI-O apply one filter.
derive-codespace-worker.sh regenerates the file from the pinned moby
source, and the unit test pins its SHA-256, so any change to what the
worker may call is a reviewed diff.

Under the profile, plus apparmor=unconfined and systempaths=unconfined,
the unmodified worker image confines as uid 1654 without --privileged
or any capability, and a network-off child sees only loopback.

The Dockerfile header said an allowlist run on an unprivileged pod
"just gets Denied"; it actually aborts at launch, because the tool
probe passes on the installed binaries. It also left CAP_SYS_ADMIN out
of the allowlist tier, which `ip netns add` needs. The header now states
per tier what the deployment must grant, with a Kubernetes
securityContext for the same posture, and compose carries the
security_opt block commented out so the default stack is unchanged.
Both say that a masked /proc still passes the bubblewrap probe, which
mounts no /proc, while every launch fails.

A boot line on the worker reports the bubblewrap probe and its reason,
whether the codespace-mcp helper is present, and whether the namespace
probe (CanSeal) holds, so an operator sees the posture before the first
run instead of reconstructing it from a refused or unconfined one. The
worker-host E2E pins that the registrar emits it exactly once.
@ppXD
ppXD merged commit 3875fe1 into main Sep 27, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant