prod demands a kernel of its own, where one can exist - #88
Conversation
Task 3 §2 and §3. A container shares the host kernel, and no amount of capability-dropping changes what a kernel vulnerability means. prod may carry untrusted agents, so on a host that can give a run its own kernel it now has to: ValidateProfile refuses a prod configuration that has not named one, and `doctor --profile prod` fails before anything is scheduled on that machine. Two decisions shape it, and both are departures from the scope document. **prod demands a runtime, and refuses to name one.** §3 said prod should "set a runtime rather than inheriting the host default". It does not, because which of Kata or gVisor a machine has is a property of the machine: writing `runsc` into the profile would refuse every host that has Kata, and `kata-runtime` every host that has gVisor. A profile that guesses fails on the machine it was written to protect. So the demand is the profile's and the choice is the user's, in a config only they can write — and the refusal says where to look. **The demand applies where a stronger runtime can exist.** Docker Desktop runs every container inside its own managed Linux VM and cannot register a custom OCI runtime, so demanding one on macOS or Windows would not be a boundary control — it would be a refusal to run on the platform most developers use, in exchange for a boundary already present. hostCanRegisterStrongerRuntime is that question, a var for the reason hostTimezone and hostPrimaryGID are: it is the one input that differs per machine, and the tests pin it both ways rather than asserting whichever answer the machine running them happens to give. The rule reads the **daemon before the platform**. A host with a stronger runtime registered and nothing selecting it fails under prod wherever it is — that is the sharpest case, since the boundary prod promises was available and unused. Only when nothing is registered does the platform decide: install one (Linux), or accept the VM you already have (Desktop). doctor also now fails early on a runtime the daemon has never heard of, which the launch would have refused anyway. doctor asks config for the platform answer rather than deciding again, so the preflight and the profile cannot disagree about which hosts are held to which rule — the preflight exists to say what a run would do before it does it. Three existing tests encoded the old rule and were rewritten rather than patched: two asserted that the runtime gap is reported and never fatal, and one asserted prod passes with only an unbuilt image on a host with just runc — which would now pass on macOS and fail on Linux while the code was right on both. The platform branches are tested in internal/doctor, where the var lives.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Nine findings, and the first three said the feature did not hold. Both causes
were mine and both were structural, so this is a rework rather than a patch.
**The demand was made against the wrong value.** ValidateProfile checks a
resolved Config, and `--runtime` does not arrive that way: it comes through
sandbox.Options and wins over cfg in BuildSpec. So `--profile prod --runtime
runc` passed validation and launched on a shared kernel — the persist_auth class
of leak CLAUDE.md already records, walked into while quoting it. The check now
lives in sandbox.enforceKernelBoundary, beside enforceSeccomp, and reads
spec.Runtime: the value that reaches docker.
**It also strangled the command that explains it.** `doctor` resolves its target
through LoadProfile, which ran the new refusal — so on exactly the hosts the rule
targets, `doctor --profile prod` printed no checks at all and exited with a
remedy pointing at itself. Moving the demand out of ValidateProfile fixes that,
and the vacuous-test problem with it: TestValidateProfileCatchesEachWeakening was
passing on Linux for the runtime reason alone, so deleting the seccomp or network
assertion from ValidateProfile would have shipped green.
**And it asked the wrong machine.** hostCanRegisterStrongerRuntime read
runtime.GOOS of the *client*. A macOS developer with DOCKER_HOST pointed at a
Linux build box had the demand waived on a host that could satisfy it, while
doctor — which does read the daemon — failed on the same machine, in the exact
way the comment claimed could not happen. The question now goes to the engine:
DockerCLI.StrongerRuntimeSupport reads which stronger runtimes are registered
and, when none are, whether one could be — Docker Desktop reports itself and
cannot register one, anything else that answers can. A daemon that cannot be
asked refuses under prod, the bargain enforceSeccomp already makes.
runtime.ClassifyRuntimeGap is the verdict both callers share, so the preflight
and the launch cannot reach different conclusions from the same evidence. Only
the wording and the fatality are the profile's business, and those stay in
doctor.
The rest:
- doctor's "selected but not registered" branch tested len(strong) == 0 rather
than membership, so `runtime: runsc` on a kata host reported ok and every
prod run then died at launch. It is membership now, and the case has a test.
- ValidateProfile's doc comment had been orphaned onto the new var by where I
inserted it; the var is gone and the comment is back on the function.
- The two cli-level runtime tests duplicated internal/doctor's from a package
that cannot vary the daemon's answer. Removed; the doctor package now covers
all six verdicts under both profiles.
- A prod run's argv is covered without a Kata host: BuildSpec renders
--runtime alongside the confinement flags rather than instead of them.
The podman branch of the registrability probe is written from the documented
serviceIsRemote field and is not yet exercised against a live podman machine; an
absent field reads as "could not be asked", which prod refuses, rather than as a
permissive default.
The site is the surface most people meet first, so a stale claim there is worse
than a missing one — it tells a reader the previous world is still true. One
sentence in the deployment guide had become exactly that: "the profile
deliberately does not select a stronger runtime for you — which ones are
registered is a property of the machine, and doctor reports what it found rather
than pretending". Under prod, doctor no longer reports. It fails.
What changed, matched to the behaviour it describes:
- PROFILE_MATRIX gains the row it was missing. The site's own answer to "what
actually differs between dev and prod" had six rows and none of them was the
boundary this release is about. The note carries the two limits: prod never
names the runtime, and the engine is asked rather than the client, so a Mac
driving a Linux daemon is held to what that daemon can do.
- The untrusted-agent step is no longer advice. It is a requirement with a
remedy, and it says who chooses.
- doctor's capability entry ends where the behaviour now ends — with teeth
under prod rather than with "which OCI runtimes are registered".
- --runtime is opt-in under dev and required under prod, which is what its
entry, the tutorial's flag catalogue and the prod config sample now say.
- The platform table's Docker Desktop footnote says what prod does there,
since "you cannot select one" invites the question.
And two things that shipped earlier without reaching the site at all:
- The RUNTIME column and the audit log's runtime field (#87) — a capability
entry of their own, and a line in the `list` tab's copy.
- /shared being group-accessible on native Linux (#31/#79). That one is
user-visible in a way worth stating: if your primary group is shared, its
members can read the handoff directory.
Kept in step: the capability count in the section title and the nav hint, which
the new entry moves to thirty-three.
|
Pushed the website half, which this PR was missing: Corrected — Added — a Caught up while there — two things that shipped earlier and never reached the site: the
|
Ten findings, and eight of them were one mistake: the gate tried to decide
whether a host *could* have been given a stronger runtime, and every signal
available for that is wrong somewhere.
- "Docker Desktop" as a product-name match: colima and OrbStack report a Linux
distro, so their users were refused with "install gVisor or Kata, register it
with the engine" inside a VM image they do not compose.
- podman's serviceIsRemote as a proxy for "in a machine VM": it is true for any
remote client, so `podman --remote` at a bare-metal Linux daemon had the
demand waived silently — the exact fail-open the docker branch's own comment
forty lines above said must not happen.
- podman's registered-runtime list: `podman info` answers with the *active*
runtime, so a host with kata in containers.conf read as having none. Every
prod run on podman was refused with no way to comply, including one told to
install what was already installed.
- strongerRuntimes as the gate on refusal: gVisor's own installer produces
runsc-hostnet, so selecting it was refused on a host that had gVisor, and
isolation.go's promise that the list's incompleteness "only ever costs a
missing label" was broken by making it decide a refusal.
- a default runtime already stronger (podman with kata as its default) was
refused for not having "selected" anything.
So the inference is gone. Three outcomes now, and only one refuses: a run that
names any non-default runtime is accepted (the launch settles whether it exists,
and second-guessing a name the operator chose is what produced the wrong
refusals); an engine that *reports* a stronger runtime while nothing selected it
is refused, because that is the one provable gap; everything else warns on every
run and proceeds. An engine's silence is not evidence — podman's answer is not a
registered set, and nothing distinguishes a Linux host that could install Kata
from a VM image its user does not compose.
That is weaker than the rule this PR opened with, deliberately: it no longer
refuses a bare Linux host that installed nothing. What it buys is that every
refusal is one the operator can act on, and every acceptance is one the tool can
defend. §1 is what closes the gap properly — once a tested Kata path exists,
"install this" is advice a user can follow.
The rest:
- doctor and the run path disagreed on their input: doctor saw cfg.Runtime
while the run enforced spec.Runtime, which `--runtime` overrides — so a CI
gate could pass and the run refuse, or the reverse. `doctor --runtime NAME`
now asks about the run you are about to make.
- doctor's fallback for a backend without the interface upgrade fabricated
Registrable from the names it had, which produced the *permissive* verdict
for a plain runc host while claiming to be conservative. It reports Known:
false now.
- the GapMissing remedy listed every registered runtime including runc, so
following it produced a second refusal. It lists the stronger ones.
- the only end-to-end prod launch test wrapped Session.Runtime in an embedded
interface, which erased StrongerRuntimeSupport from the method set and made
the gate skip itself. It answers both questions now, the way a CI machine
really would, so the gate takes a real decision and the container still
starts. The comment that explained the skip named ValidateProfile, which has
nothing to do with it.
- CLAUDE.md still said the runtime check "reports rather than refuses, because
prod does not yet select a stronger runtime". It is the file every session
reads first; it now carries the rule the code follows, and why the inference
was removed.
docs/security, docs/platforms, the roadmap, the CHANGELOG and the website all
describe the rule as it now is — including the new doctor flag.
Ten findings, and the sharpest says the gate refused precisely the hosts that
had already done the right thing: a daemon.json with `default-runtime: runsc`
gives every container its own kernel, and prod rejected it for "nothing selected
one". The signal that settles it — docker's DefaultRuntime, and podman's answer
being its active runtime — was available and unread.
So the classification now reads evidence in order: what the run selected, then
what the engine runs by default, then whether the engine has that name at all —
on docker, whose list is complete, and never on podman, which reports only the
runtime it is using. Six outcomes where there were three, because the evidence
really does come in six shapes and collapsing them is what produced the last two
rounds of wrong answers:
- the run gets a recognised strong runtime, by name or by the engine's default
→ permitted;
- it gets a non-default runtime nobody here recognises → permitted and
explicitly *not* vouched for, because sysbox-runc is also a non-default name
and shares the host kernel. Permitting a deliberate choice is not the same as
asserting a boundary, and the message now says which one it is doing;
- it names a runtime a complete list does not have → refused before the launch
that would have failed on it, which is what `doctor --runtime kata-runtime`
on a laptop without kata should have said instead of exiting 0;
- a stronger runtime is registered and neither the run nor the default uses it
→ refused;
- nothing stronger reported → warned, and the warning now names what the engine
*did* report, because a host with gVisor installed as runsc-hostnet was being
told to install gVisor;
- the engine could not be asked → refused, by the run path *and* by doctor. The
two disagreed before, with the run being the permissive one, and the tests
pinned them as opposites.
The rest:
- `fleet` asked the question once, before the first branch and worktree, rather
than refusing each task after creating both. Session.CheckKernelBoundary is a
dry run of the same gate, so a fleet cannot be held to a different standard
than the runs it is made of, and the refusal names `runtime:` in config —
there is no per-task flag to point at.
- one `docker info` per doctor run instead of two, fetching DefaultRuntime and
Runtimes together. The duplicate could also disagree with itself between
calls.
- enforceSeccomp's doc comment had been spliced onto enforceKernelBoundary,
where it described the opposite of what that function does. Both are on their
own functions again.
- the roadmap's summary line said the demand is enforced by ValidateProfile,
contradicting the same file 17 lines later, CLAUDE.md and the code. A
maintainer following it would have reintroduced the --runtime bypass.
- RuntimeSupport.Known is load-bearing again rather than redundant: it is what
separates "reported none" from "could not be asked", which is now the
difference between a warning and a refusal.
CLAUDE.md, docs/security, the roadmap, the CHANGELOG and the website all describe
the order the evidence is read in, including that the engine's default counts.
|
Verification steps for a Rocky Linux host with hardware virtualization are now a ticket: #89. It is the reason this PR is open. Phase 2 alone (three A contradiction there is the intended outcome rather than a setback. |
Merges main and fixes the one comparison in ClassifyRuntimeGap that still read names as strings. `containsRuntime` decides GapMissing, and GapMissing is a prod refusal. A containerd-backed daemon keys .Runtimes by shim name — `io.containerd.runc.v2` on Rocky Linux 10.2, in the same `docker info` that calls its default `runc` — so `--runtime kata` against a host listing `io.containerd.kata.v2` read as missing and prod refused a machine that had the kernel it was demanding. It now asks SameRuntime, which #98 added for exactly this and which the rest of this file already uses. The permissive direction is unchanged: it can only find a runtime the engine really listed, and a name no entry means is still GapMissing, so prod still refuses before the launch fails. This matters before #89 phase 3 rather than after it. That checklist walks the gate with `--runtime` values against a real daemon, and on a containerd host it would have been measuring a comparison already known to be wrong here — the same trap that would have had its `--runtime runc` case pass for the wrong reason. The merge conflict in isolation.go was two rewrites of notHostDefault's doc comment over an identical body: main's account of why normalisation is deliberately asymmetric between the two lists, and this branch's account of why both callers need the unrecognised-name direction. Both are kept.
|
Blocked on a decision, now written up rather than left in an issue thread: Short version: prod demands a kernel of its own and an egress allowlist, and |
The options already here are about what prod does. The narrower question is what #88 ships as, and it needed stating separately because the answer changes depending on whether the demand is enforced today or later. Records what the earlier version of this note got wrong: #88 is not merged, so prod has no opinion about runtimes today. It runs on runc with the allowlist, verified on the gVisor host itself. Merging #88 changes a working run into some other outcome, which is the thing being decided rather than a hypothetical. Refusal, warning and hold, each with the gVisor limitation spelled out where it lands. For the refusal that is the sharp one: on a host whose only stronger runtime is gVisor, prod becomes unusable rather than degraded, because not selecting runsc is refused for not selecting it and selecting it is refused because the allowlist cannot be programmed inside it. There is no third choice short of installing Kata or removing gVisor. The warning has its own version of the same trap: "a stronger runtime is available" is true only in the sense that it is registered, so the limitation has to be in the warning text or the advice is misleading.
* Pick the iptables backend the kernel will serve
The egress allowlist cannot be programmed under gVisor, so `--profile prod
--runtime runsc` refuses to start:
ip6tables: Failed to initialize nft: Protocol not supported
sandbox-cli: egress firewall setup failed; refusing to run without the requested allowlist
Debian points `iptables` at the nft variant, which is right on an ordinary host.
gVisor implements only the older setsockopt interface, so the nft binaries fail
before touching a rule while the `-legacy` ones work. Measured inside this image
on Rocky Linux 10.2:
iptables: FAILS ip6tables: FAILS
iptables-legacy: ok ip6tables-legacy: ok
That made prod's two demands mutually exclusive on a gVisor host: it requires a
kernel of its own *and* an egress allowlist, and the allowlist could not be
programmed inside the kernel it was demanding.
pick_iptables tries the nft binaries first and falls back to legacy, and every
call site now goes through what it chose. Order matters: an ordinary runc host
keeps the backend its distribution selected and nothing about those runs
changes. Chosen by trying rather than by detecting the runtime, for the reason
the host-side probe already gives — a runtime name says what somebody installed,
and what matters here is what the kernel will answer.
Still fails closed. A container that can serve neither backend refuses, because
running without the requested allowlist is the one thing this script must never
do; the message names `runsc install -- --net-raw`, which is the other half of
this on a gVisor host. Without that flag gVisor serves neither backend, and no
change here can fix it — the container simply cannot filter.
Found by #89 phase 3/4 on a host with gVisor actually installed. The Dockerfile
is content-addressed into the image tag, so the first run after this rebuilds.
* Ask one question about the backend, in one place
Six review findings, and the first two say the fallback was added in the wrong
shape: only the container script learned it, while everything else that touches
iptables kept hardcoding the bare binary.
**doctor and the launch now disagreed on the hosts this fixes.**
FirewallProgrammable's probe ran bare `iptables`, which resolves to the nft
variant — the one gVisor cannot serve. On a runsc-default host the preflight
reported "a container here cannot program the firewall" and advised
`--network default`, while the run itself worked. Before this branch they at
least agreed; the fix made them disagree, which CLAUDE.md names as the failure
to avoid.
**The only end-to-end firewall assertion read the wrong store.** The integration
test execs `iptables -S INPUT`; on the legacy path the rules live elsewhere, so
it would report "the INPUT chain was never programmed" for a container that is
fully default-deny. The ~42 rewritten call sites had no test that could observe
them on the backend being added.
So the choice moves into the image as `sandbox-iptables`, and all three callers
ask it: the entrypoint programs with it, doctor's probe uses it, the suite reads
back through it. Three copies of "which binary works" would drift, and the drift
is a preflight that answers differently from the launch.
**It writes rather than lists.** `-L -n` only proves the table can be read —
docker_cli.go already argues this, which is why the host probe creates rules —
and today's measurements prove it: legacy passes `-L` under gVisor and then
fails on `-m conntrack`. Creating and deleting one empty chain is the cheapest
thing that proves the kernel accepts a write.
**The kernel's own error survives.** The first version swallowed stderr from
both candidates and offered only a gVisor remedy, so a rootless daemon — whose
real error is "Permission denied (you must be root)" — was told to install
runsc. The last candidate's stderr is now ours.
Also: the chosen backend is logged, because IPv4 and IPv6 are picked
independently and an operator reading back with the wrong binary sees an empty
chain; and a CHANGELOG entry, which says plainly that this removes one barrier
to gVisor and not the last one — the allowlist still cannot be enforced there,
because gVisor has no connection tracking at all.
* Probe the firewall under the runtime the run will select
doctor asked whether a container can program iptables by starting one on the
engine's *default* runtime. Whether it can is a property of the kernel the
container gets, and runtimes differ: gVisor serves only the legacy backend, and
only when installed with --net-raw.
Measured on a Rocky Linux host with gVisor registered — doctor reported
ok egress firewall a container here can program the nat, redirect, owner
and conntrack rules the firewall needs
from a runc probe, while `--runtime runsc` could not program a single rule. The
preflight passing and the launch failing is what ClassifyRuntimeGap was
centralised to prevent; this check was simply never centralised with it.
FirewallProgrammable now takes the runtime and renders --runtime on the probe
container, and doctor passes the same selectedRuntime it already resolves for
the kernel-boundary check — so `doctor --runtime runsc` asks about runsc, and a
config naming one asks about that.
Empty stays empty: with nothing selected the probe runs on the engine's default,
which is what it should answer for.
Found by #89 phase 4. It is on this branch rather than main because the runtime
is already plumbed into RunChecks here; doing it on main would have duplicated
that and conflicted.
* Resolve the probe's runtime, and say what it actually found
Five review findings, and the first two are the bug this commit set out to fix,
inverted.
**The probe sent the user's spelling, not the engine's.** The run path
translates a runtime through resolveRuntime — a containerd daemon lists
`io.containerd.runc.v2` while a config says `runtime: runc` — and the probe did
not. So on that host the probe died with "unknown or invalid runtime name" and
doctor reported that a machine whose runs program the firewall perfectly well
cannot. Scoping the probe to a runtime and then naming it differently from the
run reintroduced the disagreement it exists to prevent.
**A runtime the engine does not have was reported as a host defect.** That is
FirewallUnknown, not FirewallBlocked: the probe never ran, so it learned nothing
about whether this host can filter. As Blocked it produced StatusWeak, blamed
the daemon, and offered `--network default` — telling a prod operator to switch
the egress allowlist off to fix a misspelled --runtime, while checkRuntimes
reported the real GapMissing one line below.
**The remedy still named only daemon causes.** Once the probe runs under a
selected runtime, "rootless or userns-remapped" is not the only explanation and
on gVisor it is not the explanation at all — iptables is gated behind
`runsc install -- --net-raw`. The remedy now says so when a runtime was
selected, and is unchanged when none was, which is the case it was written for.
Also: the probe gets its own half of doctor's budget, because it is the only
check that starts a container and under a stronger runtime that means booting a
micro-VM — sharing one deadline turned a slow probe into two prod failures. And
the argv moves into a pure firewallProbeArgs so it can be asserted: it is built
outside BuildArgs, so the golden --dry-run test never saw it, and deleting the
--runtime append left everything green. The new test also pins the ordering,
since a flag after the image reference is handed to the container instead.
* Say which cause the firewall probe found, and advise on that one
The review of this PR found five things, all in the seams where the new
runtime dimension meets messaging that predates it.
"Unknown" now covers three causes — an unbuilt image, a host too busy to
answer, and a runtime the engine does not have — and one canned remedy
told every one of them to build the image. Under prod that is
unactionable advice attached to a failure. The cause is now a **value**
carried beside the verdict (`FirewallReport`), for the reason its
neighbour gives: the alternative is the caller matching on reason prose,
which is the substring-across-a-package-boundary the enum replaced.
The blocked remedy keyed on the runtime a *flag* named, so a daemon whose
default-runtime is runsc — nothing selected, gVisor nonetheless what runs
— was told "rootless or userns-remapped daemons often cannot; use
--network default". That is the one instruction the comment three lines
above says a mis-identified check must never give, and it does not fix
gVisor: its netstack has no conntrack, which the rules need, so the
`runsc install -- --net-raw` this used to lead with sends an operator
away and back to a byte-identical failure.
An engine refusing the runtime *name* is not a container failing to
program rules — the container never started. resolveRuntime is non-fatal
in two paths by design, so an unaccepted spelling still reaches the
launch, and reading the refusal as Blocked blamed the daemon while the
runtime check one line below reported that same runtime as fine.
The probe's half-budget is gone, and the ordering is what replaces it:
the runtime facts come from one `info` call made *before* the probe, so
the expensive check can take the time a micro-VM needs without starving
the cheap one. That also removes a second `info` call, and two dead
parameters from checkRuntimes, which no longer asks anything.
Not fixed here, deliberately: the `sandbox-iptables` block splits the
sandbox-egress-setup comment mid-sentence. This branch changes nothing
else in the Dockerfile, and image.Ref hashes it — so rejoining a sentence
would invalidate every cached base image and cost every user a
multi-minute rebuild. It should ride along with the next change that
alters the image for a reason.
Task 3 §2 and §3. A container shares the host kernel, and no amount of
capability-dropping changes what a kernel vulnerability means. prod may carry
untrusted agents, so on a host that can give a run its own kernel it now has
to:
ValidateProfilerefuses a prod configuration that has not named one, anddoctor --profile prodfails before anything is scheduled on that machine.Two departures from the scope document
prod demands a runtime and refuses to name one. §3 said prod should "set a
runtime rather than inheriting the host default". It does not. Which of Kata or
gVisor a machine has is a property of the machine: writing
runscinto theprofile would refuse every host that has Kata, and
kata-runtimeevery host thathas gVisor. A profile that guesses fails on the machine it was written to
protect. So the demand is the profile's, the choice is the user's, and the
refusal says where to look.
The demand applies where a stronger runtime can exist. Docker Desktop runs
every container inside its own managed Linux VM and cannot register a custom OCI
runtime. Demanding one there would not be a boundary control — it would be a
refusal to run on the platform most developers use, in exchange for a boundary
that is already present.
Both are recorded in the task document rather than left as surprises in the code.
The rule reads the daemon before the platform
devis untouched on every row: a laptop is allowed not to have Kata, andallowed not to use the Kata it has.
hostCanRegisterStrongerRuntimeis a var for the reasonhostTimezoneandhostPrimaryGIDare — it is the one input that differs per machine — anddoctorasks config for the answer rather than deciding again, so thepreflight and the profile cannot disagree about which hosts are held to which
rule.
Tests
Three existing tests encoded the old rule and were rewritten rather than patched:
two asserted the runtime gap is reported and never fatal, and one asserted prod
passes with only an unbuilt image on a host with just
runc— which would nowpass on macOS and fail on Linux while the code was right on both. Every new test
pins the platform explicitly instead of asserting whichever answer the machine
running it happens to give; the platform branches live in
internal/doctor,beside the var.
go test ./...clean on macOS and in agolang:1.25container as a non-root usergo vet ./...cleanTestProdActuallyLaunchesAContainerand the doctor tests passNot in this PR
§1 — a tested Kata path, and the per-distribution setup documentation — is what
remains of task 3, and it needs a Linux host with hardware virtualization to
verify. Everything here is the machinery that makes §1 demandable once it
exists.
Note for existing prod users on Linux
This is a deliberate behaviour change: a Linux host running
--profile prodwithonly
runcwill now refuse. That is the point of the task — prod's promise wasnever a shared kernel — but it is worth a release note, which the CHANGELOG entry
carries.
🤖 Generated with Claude Code