Container base audit: what moving off Alpine does and does not buy

[HuggeK]

Audit of the core container's base image, prompted by the review on #714 arguing
that the official image cannot resolve mDNS .local names because "musl has no
NSS plugin mechanism, so the usual nss-mdns fix is not available the way it is
on glibc."

The conclusion in that review is correct — .local genuinely does not resolve.
The diagnosis is not, and it matters, because it points at the wrong fix.

1. The base image was never the variable

Go routes a .local lookup to libc only when cgo is available.
net/conf.go:395 opens if canUseCgo {, and the mDNS case sits inside it at
408-409:

case hostname != "" && stringslite.HasPrefix(src.source, "mdns"):
    if stringsHasSuffixFold(hostname, ".local") {
        return hostLookupCgo, dnsConf
    }

mustUseGoResolver begins if !cgoAvailable { return true }. Every FTW build
sets CGO_ENABLED=0, so canUseCgo is false and that branch is dead code. An
mdns4_minimal entry in nsswitch.conf falls through to hostLookupFilesDNS —
/etc/hosts, then a unicast DNS query to whatever resolv.conf points at.

This is true on every base image and every libc. Installing nss-mdns on a
Debian base would not have changed anything.

What NSS-based mDNS would actually require

All five of:

1. a glibc base;
2. CGO_ENABLED=1 — forfeits the static binary;
3. dynamic linking — glibc NSS resolves modules with dlopen, which a
   statically linked glibc binary cannot do. This also breaks the
   --platform=$BUILDPLATFORM cross-compile in Dockerfile and make verify-all;
4. libnss-mdns installed;
5. a reachable /run/avahi-daemon/socket — not present in the container, absent
   on a generic Docker host, and impossible under Docker Desktop.

Cost: CGO plus dynamic linking plus a new host dependency. Benefit: zero on any
host without avahi.

Our own Pi image confirms the host side: it runs NetworkManager (dns=default,
router nameservers straight into resolv.conf) plus avahi-daemon, with
dhcpcd masked and no systemd-resolved. Avahi serves glibc host programs
through NSS; it does not answer DNS queries. With network_mode: host the
container inherits that resolv.conf, so zap.local becomes a unicast query to
a router that will not answer it.

This is a live path, not a #714 hypothetical — the shipped
config.example.yaml binds devices by .local name (zap.local,
sid-os.local), and docs/sourceful-zap.md documents reaching the device at
http://zap.local/api/system.

Fix: #728 adds go/internal/mdnsresolve, forward .local resolution in Go,
wired into all six dial sites. Works on any base, any libc, no container change.

2. Measured comparison

Base layers via docker manifest inspect (arm64, compressed); full images built
locally (amd64, uncompressed).

	alpine:3.22	debian:trixie-slim	distroless/static	scratch
Base layer, compressed	4.12 MB	30.14 MB	0.70 MB	0
Full core image, uncompressed	52.5 MB	133 MB (measured, trixie)	~17 MB est.	~16 MB est.
Solves the mDNS motivation	n/a	no	no	no
arm64 runtime layer under QEMU	trivial (apk add ×2)	apt/dpkg emulated	none	none
Files to change	0	~6	~8 incl. the update path	~8 + own certs/zoneinfo/passwd
RUN in final stage	fine	fine	impossible (no shell)	impossible
On-site docker exec debugging	busybox shell	best	none	none
CVE surface	small	largest	smallest	smallest

3. Blockers that apply to any base change

1. scripts/test-container-boundaries.sh asserted ^FROM alpine: with no
   error message. It is a proxy for the real invariant — core must not drag in a
   Python interpreter — since Python originally reached core via the optimizer's
   base image. Worth stating directly instead.
2. wget is contractual, and absent from every candidate base. It is used by
   the HEALTHCHECK and by go/cmd/ftw-updater/main.go:1217, the readiness
   probe that decides whether an update commits. On failure the updater
   auto-rolls back.
3. gid 101 must be preserved. optimizer/ftw_optimizer/worker.py:101 chmods
   its socket 0o660 — group-only. Core reaches it purely because both
   containers share gid 101. Changing it degrades the planner to the Go DP
   fallback silently.

4. Why distroless/scratch is currently unreachable

ftw-updater can never update itself. componentSpec handles only core
and optimizer, and compose up -d is scoped to a single service. Old updaters
therefore persist in the field indefinitely — there is no release cadence that
drains them.

So a wget-less core would meet an old updater that docker execs wget, fail
its gate forever, and pin the host in an update → rollback loop. Both call sites
are stubbed in main_test.go, so CI would not catch it.

Making distroless reachable needs updater added as a third component first,
then a wait for convergence. Filing this as the standalone gap it is.

5. Latent issues found along the way

Independent of any base decision:

• time.Local silently degrades to UTC when zoneinfo is missing, mis-timing
  price and plan windows with no error. _ "time/tzdata" is not imported, and
  the tzdata tests t.Skipf rather than fail — so a base image without tzdata
  would keep CI green. Addressed in build: run one base across the whole stack (debian:trixie-slim) #731.
• Nothing watches the base images. No trivy/grype/scout/SBOM/provenance
  anywhere, and .github/dependabot.yml has no package-ecosystem: docker, so
  alpine:3.22, golang:1.26-alpine, docker:27-cli and
  python:3.12-slim-bookworm are unmonitored. The "Alpine has fewer CVEs"
  argument is currently unmeasured either way.
• Two comments describe a user that has never existed. scripts/install.sh
  and deploy/pi-gen/stage-ftw/01-ftw-setup/00-run.sh both attribute uid 100 to
  an alpine adduser -S. The Dockerfile creates no account on any base —
  USER 100:101 is bare numeric, which is why ENV HOME=/app/data is
  load-bearing. Verified: uid 100 and gid 101 have no passwd/group entry on
  debian:bookworm-slim either.
• No image-size assertion and no on-device digest pinning exist.
  selfupdate/registry.go checks tag existence only; rollback is local
  retagging. Beta and stable built from the same commit at different times can
  already contain different base rootfs.

6. Related

• feat(net): resolve device .local names over mDNS #728 — Go-side mDNS resolution. The actual fix for .local.
• build: run one base across the whole stack (debian:trixie-slim) #731 — moves core, updater and optimizer to debian:trixie-slim, one base across the stack.
• ci: notice when the container base falls behind Debian stable #733 — weekly check that files an issue when a newer Debian stable ships, so the pin moves deliberately.