Skip to content

prod demands a kernel of its own, where one can exist - #88

Open
Amitgb14 wants to merge 10 commits into
mainfrom
feat/prod-demands-a-kernel
Open

prod demands a kernel of its own, where one can exist#88
Amitgb14 wants to merge 10 commits into
mainfrom
feat/prod-demands-a-kernel

Conversation

@Amitgb14

@Amitgb14 Amitgb14 commented Aug 7, 2026

Copy link
Copy Markdown
Owner

Task 3 §2 and §3. A container shares the host kernel, and no amount of
capability-dropping changes what a kernel vulnerability means. prod may carry
untrusted agents, so on a host that can give a run its own kernel it now has
to: ValidateProfile refuses a prod configuration that has not named one, and
doctor --profile prod fails before anything is scheduled on that machine.

$ sandbox-cli doctor --profile prod
  FAIL  isolation runtime  this host can give a run its own kernel and nothing selected one: kata-runtime
                           set `runtime: kata-runtime` in your own config — prod refuses a shared kernel where one can be avoided

Two departures from the scope document

prod demands a runtime and refuses to name one. §3 said prod should "set a
runtime rather than inheriting the host default". It does not. Which of Kata or
gVisor a machine has is a property of the machine: writing runsc into the
profile would refuse every host that has Kata, and kata-runtime every host that
has gVisor. A profile that guesses fails on the machine it was written to
protect. So the demand is the profile's, the choice is the user's, and the
refusal says where to look.

The demand applies where a stronger runtime can exist. Docker Desktop runs
every container inside its own managed Linux VM and cannot register a custom OCI
runtime. Demanding one there would not be a boundary control — it would be a
refusal to run on the platform most developers use, in exchange for a boundary
that is already present.

Both are recorded in the task document rather than left as surprises in the code.

The rule reads the daemon before the platform

Daemon says Platform prod
a stronger runtime is registered, none selected either FAIL — the boundary was available and unused
the selected runtime is registered either ok, and the check names what runs are getting
the selected runtime is not registered either FAIL early, rather than at the launch that would have refused
nothing registered Linux FAIL — install gVisor or Kata
nothing registered Docker Desktop ok — reports the VM every container is already inside

dev is untouched on every row: a laptop is allowed not to have Kata, and
allowed not to use the Kata it has.

hostCanRegisterStrongerRuntime is a var for the reason hostTimezone and
hostPrimaryGID are — it is the one input that differs per machine — and
doctor asks config for the answer rather than deciding again, so the
preflight and the profile cannot disagree about which hosts are held to which
rule.

Tests

Three existing tests encoded the old rule and were rewritten rather than patched:
two asserted the runtime gap is reported and never fatal, and one asserted prod
passes with only an unbuilt image on a host with just runc — which would now
pass on macOS and fail on Linux while the code was right on both. Every new test
pins the platform explicitly instead of asserting whichever answer the machine
running it happens to give; the platform branches live in internal/doctor,
beside the var.

  • go test ./... clean on macOS and in a golang:1.25 container as a non-root user
  • go vet ./... clean
  • Docker integration: TestProdActuallyLaunchesAContainer and the doctor tests pass
  • All relative doc links resolve

Not in this PR

§1 — a tested Kata path, and the per-distribution setup documentation — is what
remains of task 3, and it needs a Linux host with hardware virtualization to
verify. Everything here is the machinery that makes §1 demandable once it
exists.

Note for existing prod users on Linux

This is a deliberate behaviour change: a Linux host running --profile prod with
only runc will now refuse. That is the point of the task — prod's promise was
never a shared kernel — but it is worth a release note, which the CHANGELOG entry
carries.

🤖 Generated with Claude Code

Task 3 §2 and §3. A container shares the host kernel, and no amount of
capability-dropping changes what a kernel vulnerability means. prod may carry
untrusted agents, so on a host that can give a run its own kernel it now has to:
ValidateProfile refuses a prod configuration that has not named one, and
`doctor --profile prod` fails before anything is scheduled on that machine.

Two decisions shape it, and both are departures from the scope document.

**prod demands a runtime, and refuses to name one.** §3 said prod should "set a
runtime rather than inheriting the host default". It does not, because which of
Kata or gVisor a machine has is a property of the machine: writing `runsc` into
the profile would refuse every host that has Kata, and `kata-runtime` every host
that has gVisor. A profile that guesses fails on the machine it was written to
protect. So the demand is the profile's and the choice is the user's, in a config
only they can write — and the refusal says where to look.

**The demand applies where a stronger runtime can exist.** Docker Desktop runs
every container inside its own managed Linux VM and cannot register a custom OCI
runtime, so demanding one on macOS or Windows would not be a boundary control —
it would be a refusal to run on the platform most developers use, in exchange for
a boundary already present. hostCanRegisterStrongerRuntime is that question, a
var for the reason hostTimezone and hostPrimaryGID are: it is the one input that
differs per machine, and the tests pin it both ways rather than asserting
whichever answer the machine running them happens to give.

The rule reads the **daemon before the platform**. A host with a stronger runtime
registered and nothing selecting it fails under prod wherever it is — that is the
sharpest case, since the boundary prod promises was available and unused. Only
when nothing is registered does the platform decide: install one (Linux), or
accept the VM you already have (Desktop). doctor also now fails early on a
runtime the daemon has never heard of, which the launch would have refused
anyway.

doctor asks config for the platform answer rather than deciding again, so the
preflight and the profile cannot disagree about which hosts are held to which
rule — the preflight exists to say what a run would do before it does it.

Three existing tests encoded the old rule and were rewritten rather than patched:
two asserted that the runtime gap is reported and never fatal, and one asserted
prod passes with only an unbuilt image on a host with just runc — which would now
pass on macOS and fail on Linux while the code was right on both. The platform
branches are tested in internal/doctor, where the var lives.
@vercel

vercel Bot commented Aug 7, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
sandbox-cli Ready Ready Preview Aug 22, 2026 5:18pm

Nine findings, and the first three said the feature did not hold. Both causes
were mine and both were structural, so this is a rework rather than a patch.

**The demand was made against the wrong value.** ValidateProfile checks a
resolved Config, and `--runtime` does not arrive that way: it comes through
sandbox.Options and wins over cfg in BuildSpec. So `--profile prod --runtime
runc` passed validation and launched on a shared kernel — the persist_auth class
of leak CLAUDE.md already records, walked into while quoting it. The check now
lives in sandbox.enforceKernelBoundary, beside enforceSeccomp, and reads
spec.Runtime: the value that reaches docker.

**It also strangled the command that explains it.** `doctor` resolves its target
through LoadProfile, which ran the new refusal — so on exactly the hosts the rule
targets, `doctor --profile prod` printed no checks at all and exited with a
remedy pointing at itself. Moving the demand out of ValidateProfile fixes that,
and the vacuous-test problem with it: TestValidateProfileCatchesEachWeakening was
passing on Linux for the runtime reason alone, so deleting the seccomp or network
assertion from ValidateProfile would have shipped green.

**And it asked the wrong machine.** hostCanRegisterStrongerRuntime read
runtime.GOOS of the *client*. A macOS developer with DOCKER_HOST pointed at a
Linux build box had the demand waived on a host that could satisfy it, while
doctor — which does read the daemon — failed on the same machine, in the exact
way the comment claimed could not happen. The question now goes to the engine:
DockerCLI.StrongerRuntimeSupport reads which stronger runtimes are registered
and, when none are, whether one could be — Docker Desktop reports itself and
cannot register one, anything else that answers can. A daemon that cannot be
asked refuses under prod, the bargain enforceSeccomp already makes.

runtime.ClassifyRuntimeGap is the verdict both callers share, so the preflight
and the launch cannot reach different conclusions from the same evidence. Only
the wording and the fatality are the profile's business, and those stay in
doctor.

The rest:

  - doctor's "selected but not registered" branch tested len(strong) == 0 rather
    than membership, so `runtime: runsc` on a kata host reported ok and every
    prod run then died at launch. It is membership now, and the case has a test.
  - ValidateProfile's doc comment had been orphaned onto the new var by where I
    inserted it; the var is gone and the comment is back on the function.
  - The two cli-level runtime tests duplicated internal/doctor's from a package
    that cannot vary the daemon's answer. Removed; the doctor package now covers
    all six verdicts under both profiles.
  - A prod run's argv is covered without a Kata host: BuildSpec renders
    --runtime alongside the confinement flags rather than instead of them.

The podman branch of the registrability probe is written from the documented
serviceIsRemote field and is not yet exercised against a live podman machine; an
absent field reads as "could not be asked", which prod refuses, rather than as a
permissive default.
The site is the surface most people meet first, so a stale claim there is worse
than a missing one — it tells a reader the previous world is still true. One
sentence in the deployment guide had become exactly that: "the profile
deliberately does not select a stronger runtime for you — which ones are
registered is a property of the machine, and doctor reports what it found rather
than pretending". Under prod, doctor no longer reports. It fails.

What changed, matched to the behaviour it describes:

  - PROFILE_MATRIX gains the row it was missing. The site's own answer to "what
    actually differs between dev and prod" had six rows and none of them was the
    boundary this release is about. The note carries the two limits: prod never
    names the runtime, and the engine is asked rather than the client, so a Mac
    driving a Linux daemon is held to what that daemon can do.
  - The untrusted-agent step is no longer advice. It is a requirement with a
    remedy, and it says who chooses.
  - doctor's capability entry ends where the behaviour now ends — with teeth
    under prod rather than with "which OCI runtimes are registered".
  - --runtime is opt-in under dev and required under prod, which is what its
    entry, the tutorial's flag catalogue and the prod config sample now say.
  - The platform table's Docker Desktop footnote says what prod does there,
    since "you cannot select one" invites the question.

And two things that shipped earlier without reaching the site at all:

  - The RUNTIME column and the audit log's runtime field (#87) — a capability
    entry of their own, and a line in the `list` tab's copy.
  - /shared being group-accessible on native Linux (#31/#79). That one is
    user-visible in a way worth stating: if your primary group is shared, its
    members can read the handoff directory.

Kept in step: the capability count in the section title and the nav hint, which
the new entry moves to thirty-three.
@Amitgb14

Amitgb14 commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Pushed the website half, which this PR was missing: web/ still described the world before it.

Correctedlib/deploy.ts said "the profile deliberately does not select a stronger runtime for you … doctor reports what it found rather than pretending". Under prod doctor no longer reports; it fails.

Added — a PROFILE_MATRIX row for the kernel demand (the site's own dev-vs-prod table had six rows and not this one), with the two limits in its note: prod never names the runtime, and the engine is asked rather than the client. Plus doctor's capability entry, the --runtime entry, the tutorial flag catalogue, the prod config sample and the platform-table footnote.

Caught up while there — two things that shipped earlier and never reached the site: the RUNTIME column and the audit runtime field from #87, and /shared being group-accessible on native Linux from #31/#79.

tsc --noEmit, eslint src and next build all clean; the new copy is in the prerendered HTML.

Ten findings, and eight of them were one mistake: the gate tried to decide
whether a host *could* have been given a stronger runtime, and every signal
available for that is wrong somewhere.

  - "Docker Desktop" as a product-name match: colima and OrbStack report a Linux
    distro, so their users were refused with "install gVisor or Kata, register it
    with the engine" inside a VM image they do not compose.
  - podman's serviceIsRemote as a proxy for "in a machine VM": it is true for any
    remote client, so `podman --remote` at a bare-metal Linux daemon had the
    demand waived silently — the exact fail-open the docker branch's own comment
    forty lines above said must not happen.
  - podman's registered-runtime list: `podman info` answers with the *active*
    runtime, so a host with kata in containers.conf read as having none. Every
    prod run on podman was refused with no way to comply, including one told to
    install what was already installed.
  - strongerRuntimes as the gate on refusal: gVisor's own installer produces
    runsc-hostnet, so selecting it was refused on a host that had gVisor, and
    isolation.go's promise that the list's incompleteness "only ever costs a
    missing label" was broken by making it decide a refusal.
  - a default runtime already stronger (podman with kata as its default) was
    refused for not having "selected" anything.

So the inference is gone. Three outcomes now, and only one refuses: a run that
names any non-default runtime is accepted (the launch settles whether it exists,
and second-guessing a name the operator chose is what produced the wrong
refusals); an engine that *reports* a stronger runtime while nothing selected it
is refused, because that is the one provable gap; everything else warns on every
run and proceeds. An engine's silence is not evidence — podman's answer is not a
registered set, and nothing distinguishes a Linux host that could install Kata
from a VM image its user does not compose.

That is weaker than the rule this PR opened with, deliberately: it no longer
refuses a bare Linux host that installed nothing. What it buys is that every
refusal is one the operator can act on, and every acceptance is one the tool can
defend. §1 is what closes the gap properly — once a tested Kata path exists,
"install this" is advice a user can follow.

The rest:

  - doctor and the run path disagreed on their input: doctor saw cfg.Runtime
    while the run enforced spec.Runtime, which `--runtime` overrides — so a CI
    gate could pass and the run refuse, or the reverse. `doctor --runtime NAME`
    now asks about the run you are about to make.
  - doctor's fallback for a backend without the interface upgrade fabricated
    Registrable from the names it had, which produced the *permissive* verdict
    for a plain runc host while claiming to be conservative. It reports Known:
    false now.
  - the GapMissing remedy listed every registered runtime including runc, so
    following it produced a second refusal. It lists the stronger ones.
  - the only end-to-end prod launch test wrapped Session.Runtime in an embedded
    interface, which erased StrongerRuntimeSupport from the method set and made
    the gate skip itself. It answers both questions now, the way a CI machine
    really would, so the gate takes a real decision and the container still
    starts. The comment that explained the skip named ValidateProfile, which has
    nothing to do with it.
  - CLAUDE.md still said the runtime check "reports rather than refuses, because
    prod does not yet select a stronger runtime". It is the file every session
    reads first; it now carries the rule the code follows, and why the inference
    was removed.

docs/security, docs/platforms, the roadmap, the CHANGELOG and the website all
describe the rule as it now is — including the new doctor flag.
Ten findings, and the sharpest says the gate refused precisely the hosts that
had already done the right thing: a daemon.json with `default-runtime: runsc`
gives every container its own kernel, and prod rejected it for "nothing selected
one". The signal that settles it — docker's DefaultRuntime, and podman's answer
being its active runtime — was available and unread.

So the classification now reads evidence in order: what the run selected, then
what the engine runs by default, then whether the engine has that name at all —
on docker, whose list is complete, and never on podman, which reports only the
runtime it is using. Six outcomes where there were three, because the evidence
really does come in six shapes and collapsing them is what produced the last two
rounds of wrong answers:

  - the run gets a recognised strong runtime, by name or by the engine's default
    → permitted;
  - it gets a non-default runtime nobody here recognises → permitted and
    explicitly *not* vouched for, because sysbox-runc is also a non-default name
    and shares the host kernel. Permitting a deliberate choice is not the same as
    asserting a boundary, and the message now says which one it is doing;
  - it names a runtime a complete list does not have → refused before the launch
    that would have failed on it, which is what `doctor --runtime kata-runtime`
    on a laptop without kata should have said instead of exiting 0;
  - a stronger runtime is registered and neither the run nor the default uses it
    → refused;
  - nothing stronger reported → warned, and the warning now names what the engine
    *did* report, because a host with gVisor installed as runsc-hostnet was being
    told to install gVisor;
  - the engine could not be asked → refused, by the run path *and* by doctor. The
    two disagreed before, with the run being the permissive one, and the tests
    pinned them as opposites.

The rest:

  - `fleet` asked the question once, before the first branch and worktree, rather
    than refusing each task after creating both. Session.CheckKernelBoundary is a
    dry run of the same gate, so a fleet cannot be held to a different standard
    than the runs it is made of, and the refusal names `runtime:` in config —
    there is no per-task flag to point at.
  - one `docker info` per doctor run instead of two, fetching DefaultRuntime and
    Runtimes together. The duplicate could also disagree with itself between
    calls.
  - enforceSeccomp's doc comment had been spliced onto enforceKernelBoundary,
    where it described the opposite of what that function does. Both are on their
    own functions again.
  - the roadmap's summary line said the demand is enforced by ValidateProfile,
    contradicting the same file 17 lines later, CLAUDE.md and the code. A
    maintainer following it would have reintroduced the --runtime bypass.
  - RuntimeSupport.Known is load-bearing again rather than redundant: it is what
    separates "reported none" from "could not be asked", which is now the
    difference between a warning and a refusal.

CLAUDE.md, docs/security, the roadmap, the CHANGELOG and the website all describe
the order the evidence is read in, including that the engine's default counts.
@Amitgb14

Amitgb14 commented Aug 9, 2026

Copy link
Copy Markdown
Owner Author

Verification steps for a Rocky Linux host with hardware virtualization are now a ticket: #89.

It is the reason this PR is open. Phase 2 alone (three docker info strings, no Kata needed) settles the biggest unknown — DefaultRuntime is the signal the last review round found unread, and it has only ever been observed on Docker Desktop. Phase 3 walks the gate matrix, including the case the code got wrong twice: an engine whose default is already a stronger runtime must satisfy prod without naming one. Phase 4 asks whether the egress firewall still programs inside the guest, which task 3 §1 flags as the likeliest obstacle.

A contradiction there is the intended outcome rather than a setback.

Brings in #81's umask work, which landed while this waited on a Kata host (#89).
No conflicts: both touched CHANGELOG.md and CLAUDE.md in different sections.
Merges main and fixes the one comparison in ClassifyRuntimeGap that still read
names as strings.

`containsRuntime` decides GapMissing, and GapMissing is a prod refusal. A
containerd-backed daemon keys .Runtimes by shim name — `io.containerd.runc.v2`
on Rocky Linux 10.2, in the same `docker info` that calls its default `runc` —
so `--runtime kata` against a host listing `io.containerd.kata.v2` read as
missing and prod refused a machine that had the kernel it was demanding.

It now asks SameRuntime, which #98 added for exactly this and which the rest of
this file already uses. The permissive direction is unchanged: it can only find
a runtime the engine really listed, and a name no entry means is still
GapMissing, so prod still refuses before the launch fails.

This matters before #89 phase 3 rather than after it. That checklist walks the
gate with `--runtime` values against a real daemon, and on a containerd host it
would have been measuring a comparison already known to be wrong here — the same
trap that would have had its `--runtime runc` case pass for the wrong reason.

The merge conflict in isolation.go was two rewrites of notHostDefault's doc
comment over an identical body: main's account of why normalisation is
deliberately asymmetric between the two lists, and this branch's account of why
both callers need the unrecognised-name direction. Both are kept.
@Amitgb14

Amitgb14 commented Aug 9, 2026

Copy link
Copy Markdown
Owner Author

Blocked on a decision, now written up rather than left in an issue thread:
docs/roadmap/task-3-gvisor-egress.md.

Short version: prod demands a kernel of its own and an egress allowlist, and
gVisor provides no connection tracking, so it can serve only the first. The gate
in this PR is verified working (#89 phase 3) — what is unresolved is what prod
should do on a host where satisfying one demand costs the other.

Amitgb14 added a commit that referenced this pull request Aug 10, 2026
The options already here are about what prod does. The narrower question is what
#88 ships as, and it needed stating separately because the answer changes
depending on whether the demand is enforced today or later.

Records what the earlier version of this note got wrong: #88 is not merged, so
prod has no opinion about runtimes today. It runs on runc with the allowlist,
verified on the gVisor host itself. Merging #88 changes a working run into some
other outcome, which is the thing being decided rather than a hypothetical.

Refusal, warning and hold, each with the gVisor limitation spelled out where it
lands. For the refusal that is the sharp one: on a host whose only stronger
runtime is gVisor, prod becomes unusable rather than degraded, because not
selecting runsc is refused for not selecting it and selecting it is refused
because the allowlist cannot be programmed inside it. There is no third choice
short of installing Kata or removing gVisor.

The warning has its own version of the same trap: "a stronger runtime is
available" is true only in the sense that it is registered, so the limitation
has to be in the warning text or the advice is misleading.
* Pick the iptables backend the kernel will serve

The egress allowlist cannot be programmed under gVisor, so `--profile prod
--runtime runsc` refuses to start:

    ip6tables: Failed to initialize nft: Protocol not supported
    sandbox-cli: egress firewall setup failed; refusing to run without the requested allowlist

Debian points `iptables` at the nft variant, which is right on an ordinary host.
gVisor implements only the older setsockopt interface, so the nft binaries fail
before touching a rule while the `-legacy` ones work. Measured inside this image
on Rocky Linux 10.2:

    iptables: FAILS          ip6tables: FAILS
    iptables-legacy: ok      ip6tables-legacy: ok

That made prod's two demands mutually exclusive on a gVisor host: it requires a
kernel of its own *and* an egress allowlist, and the allowlist could not be
programmed inside the kernel it was demanding.

pick_iptables tries the nft binaries first and falls back to legacy, and every
call site now goes through what it chose. Order matters: an ordinary runc host
keeps the backend its distribution selected and nothing about those runs
changes. Chosen by trying rather than by detecting the runtime, for the reason
the host-side probe already gives — a runtime name says what somebody installed,
and what matters here is what the kernel will answer.

Still fails closed. A container that can serve neither backend refuses, because
running without the requested allowlist is the one thing this script must never
do; the message names `runsc install -- --net-raw`, which is the other half of
this on a gVisor host. Without that flag gVisor serves neither backend, and no
change here can fix it — the container simply cannot filter.

Found by #89 phase 3/4 on a host with gVisor actually installed. The Dockerfile
is content-addressed into the image tag, so the first run after this rebuilds.

* Ask one question about the backend, in one place

Six review findings, and the first two say the fallback was added in the wrong
shape: only the container script learned it, while everything else that touches
iptables kept hardcoding the bare binary.

**doctor and the launch now disagreed on the hosts this fixes.**
FirewallProgrammable's probe ran bare `iptables`, which resolves to the nft
variant — the one gVisor cannot serve. On a runsc-default host the preflight
reported "a container here cannot program the firewall" and advised
`--network default`, while the run itself worked. Before this branch they at
least agreed; the fix made them disagree, which CLAUDE.md names as the failure
to avoid.

**The only end-to-end firewall assertion read the wrong store.** The integration
test execs `iptables -S INPUT`; on the legacy path the rules live elsewhere, so
it would report "the INPUT chain was never programmed" for a container that is
fully default-deny. The ~42 rewritten call sites had no test that could observe
them on the backend being added.

So the choice moves into the image as `sandbox-iptables`, and all three callers
ask it: the entrypoint programs with it, doctor's probe uses it, the suite reads
back through it. Three copies of "which binary works" would drift, and the drift
is a preflight that answers differently from the launch.

**It writes rather than lists.** `-L -n` only proves the table can be read —
docker_cli.go already argues this, which is why the host probe creates rules —
and today's measurements prove it: legacy passes `-L` under gVisor and then
fails on `-m conntrack`. Creating and deleting one empty chain is the cheapest
thing that proves the kernel accepts a write.

**The kernel's own error survives.** The first version swallowed stderr from
both candidates and offered only a gVisor remedy, so a rootless daemon — whose
real error is "Permission denied (you must be root)" — was told to install
runsc. The last candidate's stderr is now ours.

Also: the chosen backend is logged, because IPv4 and IPv6 are picked
independently and an operator reading back with the wrong binary sees an empty
chain; and a CHANGELOG entry, which says plainly that this removes one barrier
to gVisor and not the last one — the allowlist still cannot be enforced there,
because gVisor has no connection tracking at all.

* Probe the firewall under the runtime the run will select

doctor asked whether a container can program iptables by starting one on the
engine's *default* runtime. Whether it can is a property of the kernel the
container gets, and runtimes differ: gVisor serves only the legacy backend, and
only when installed with --net-raw.

Measured on a Rocky Linux host with gVisor registered — doctor reported

    ok  egress firewall  a container here can program the nat, redirect, owner
                         and conntrack rules the firewall needs

from a runc probe, while `--runtime runsc` could not program a single rule. The
preflight passing and the launch failing is what ClassifyRuntimeGap was
centralised to prevent; this check was simply never centralised with it.

FirewallProgrammable now takes the runtime and renders --runtime on the probe
container, and doctor passes the same selectedRuntime it already resolves for
the kernel-boundary check — so `doctor --runtime runsc` asks about runsc, and a
config naming one asks about that.

Empty stays empty: with nothing selected the probe runs on the engine's default,
which is what it should answer for.

Found by #89 phase 4. It is on this branch rather than main because the runtime
is already plumbed into RunChecks here; doing it on main would have duplicated
that and conflicted.

* Resolve the probe's runtime, and say what it actually found

Five review findings, and the first two are the bug this commit set out to fix,
inverted.

**The probe sent the user's spelling, not the engine's.** The run path
translates a runtime through resolveRuntime — a containerd daemon lists
`io.containerd.runc.v2` while a config says `runtime: runc` — and the probe did
not. So on that host the probe died with "unknown or invalid runtime name" and
doctor reported that a machine whose runs program the firewall perfectly well
cannot. Scoping the probe to a runtime and then naming it differently from the
run reintroduced the disagreement it exists to prevent.

**A runtime the engine does not have was reported as a host defect.** That is
FirewallUnknown, not FirewallBlocked: the probe never ran, so it learned nothing
about whether this host can filter. As Blocked it produced StatusWeak, blamed
the daemon, and offered `--network default` — telling a prod operator to switch
the egress allowlist off to fix a misspelled --runtime, while checkRuntimes
reported the real GapMissing one line below.

**The remedy still named only daemon causes.** Once the probe runs under a
selected runtime, "rootless or userns-remapped" is not the only explanation and
on gVisor it is not the explanation at all — iptables is gated behind
`runsc install -- --net-raw`. The remedy now says so when a runtime was
selected, and is unchanged when none was, which is the case it was written for.

Also: the probe gets its own half of doctor's budget, because it is the only
check that starts a container and under a stronger runtime that means booting a
micro-VM — sharing one deadline turned a slow probe into two prod failures. And
the argv moves into a pure firewallProbeArgs so it can be asserted: it is built
outside BuildArgs, so the golden --dry-run test never saw it, and deleting the
--runtime append left everything green. The new test also pins the ordering,
since a flag after the image reference is handed to the container instead.

* Say which cause the firewall probe found, and advise on that one

The review of this PR found five things, all in the seams where the new
runtime dimension meets messaging that predates it.

"Unknown" now covers three causes — an unbuilt image, a host too busy to
answer, and a runtime the engine does not have — and one canned remedy
told every one of them to build the image. Under prod that is
unactionable advice attached to a failure. The cause is now a **value**
carried beside the verdict (`FirewallReport`), for the reason its
neighbour gives: the alternative is the caller matching on reason prose,
which is the substring-across-a-package-boundary the enum replaced.

The blocked remedy keyed on the runtime a *flag* named, so a daemon whose
default-runtime is runsc — nothing selected, gVisor nonetheless what runs
— was told "rootless or userns-remapped daemons often cannot; use
--network default". That is the one instruction the comment three lines
above says a mis-identified check must never give, and it does not fix
gVisor: its netstack has no conntrack, which the rules need, so the
`runsc install -- --net-raw` this used to lead with sends an operator
away and back to a byte-identical failure.

An engine refusing the runtime *name* is not a container failing to
program rules — the container never started. resolveRuntime is non-fatal
in two paths by design, so an unaccepted spelling still reaches the
launch, and reading the refusal as Blocked blamed the daemon while the
runtime check one line below reported that same runtime as fine.

The probe's half-budget is gone, and the ordering is what replaces it:
the runtime facts come from one `info` call made *before* the probe, so
the expensive check can take the time a micro-VM needs without starving
the cheap one. That also removes a second `info` call, and two dead
parameters from checkRuntimes, which no longer asks anything.

Not fixed here, deliberately: the `sandbox-iptables` block splits the
sandbox-egress-setup comment mid-sentence. This branch changes nothing
else in the Dockerfile, and image.Ref hashes it — so rejoining a sentence
would invalidate every cached base image and cost every user a
multi-minute rebuild. It should ride along with the next change that
alters the image for a reason.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant