Skip to content

perf(provider): cut the docker image from 1.84 GB to 376 MB (distroless) - #3322

Merged
forgetso merged 1 commit into
mainfrom
perf/slim-provider-image
Sep 22, 2026
Merged

forgetso merged 1 commit into
mainfrom
perf/slim-provider-image

Conversation

@forgetso

@forgetso forgetso commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

What changed

The provider image goes from 1.84 GB to 376 MB — a 80% cut — by making the build two-stage and moving the runtime to distroless.

size
before (node:24) 1.84 GB
node:24-slim 538 MB
distroless (this PR) 376 MB

Why it was so large

It was never our code. docker history on the old image:

layer size
node:24 base 1.63 GB (89%)
GeoLite2 databases (3 layers) 86.5 MB
npm i sharp 42.4 MB
CLI bundle 7.0 MB
locales 0.9 MB

node:24 carries a full build toolchain — gcc, python, git, the apt cache — for a container that runs a pre-bundled JS file and installs one dependency from a prebuilt binary.

A node:24-slim builder now does everything needing a shell, apt or the network (install curl, fetch the GeoLite databases, npm i sharp), and the runtime stage is gcr.io/distroless/nodejs24-debian12 with the result copied in. The three GeoLite RUNs became one layer, and the npm cache is cleaned in the layer that creates it.

Why not Alpine

It would save more and break everything. All four native addons are linux-x64-**gnu**:

/usr/src/app/node_modules/@img/sharp-linux-x64/lib/sharp-linux-x64-0.35.3.node
/usr/src/app/prosopo-native-puzzle.node
/usr/src/app/prosopo-native-merkle.node
/usr/src/app/prosopo-native-ja4.node

Alpine is musl; each needs separately compiled binaries. The distroless runtime is Debian 12, matching the builder, so the glibc the addons were resolved against is the one that loads them.

Runs as non-root

The runtime stage is gcr.io/distroless/nodejs24-debian12:nonroot with an explicit USER 65532:65532, and the copied application tree is chowned to that user. Closes captcha-private#4749.

This has a hard deployment prerequisite. The provider writes at runtime to a host bind mount (./data/detector-pool.N → /app/data/detector-pool) — the pool-replace endpoint does mkdir + writeFile, and assignSecret drops a 0600 secret alongside. Those directories are root-owned today. Tested against the built image:

pool dir owner result
root (today) every write fails EACCES
chowned to 65532 write succeeds, mode 600 preserved

That failure is silent — an unreadable pool is not an error, getDetectorBundlePool() returns empty and runEmptyDetectorPoolPowFallback serves proof of work, so scoring would degrade fleet-wide with nothing in the logs.

prosopo/captcha-private#4750 must be deployed and verified before this image reaches any pronode.

Self-hosted deployments are unaffected: docker-compose.self-hosted.yml mounts no pool directory, so the path is created inside the container by the runtime user. Privileged ports are a non-issue — Docker sets ip_unprivileged_port_start=0, and the API listens on 9229 with Caddy fronting 80/443.

⚠️ Breaking for operators

There is no shell, no npm and no npx in the runtime image. npx provider <command> was only ever human convenience, so every call site is now an explicit node invocation:

docker exec provider1 /nodejs/bin/node /usr/src/app/provider.cli.bundle.js <command>

docker exec -it provider1 bash no longer works. That is the real cost of this change.

Call sites updated in this PR:

  • provider_image CI smoke test — ran /bin/sh -c '... npx provider version', which would have failed on the first build
  • both provider healthchecks in docker/docker-compose.provider.yml — curl → node -e "fetch(...)"
  • the healthcheck in docker-compose.self-hosted.yml
  • SELF_HOSTING.md and packages/datasets-fs/README.md

Updated alongside, outside this repo:

  • captcha-private: providerEnsureIndexes, providerLoadDataset, providerMigrateIps, providerUpdateSpamEmailDomains playbooks (all used docker exec ... bash -c "npx provider ...") and RUNBOOK.md
  • Add a Self-Hosting guide, including Bring Your Own Detector docs#100: the self-hosting page, plus a note explaining why the CLI examples name node explicitly

I swept the repos for any remaining bash/npx/curl assumption against this container and found none. docker-compose.provider-mock.yml still uses a curl healthcheck, correctly — that is prosopo/provider-mock, a different image built from demos/provider-mock/Dockerfile.

Test coverage

No unit tests — it's a Dockerfile. Built all three variants from the same bundle and ran the distroless one as a full stack against Mongo and Redis:

  • boots; /healthz returns ok; the node-based container healthcheck reports healthy
  • MaxMind City and ASN readers initialise; IpInfoService initialized
  • CLI works via explicit node: version, ensure_indexes, provider_set_data_set (dataset loads), site_key_register
  • all three captcha types serve — pow, puzzle (29 KB WebP returned, so sharp renders), and image from a loaded dataset with signature present
  • zero error or fatal lines in the provider log across the whole run
  • re-verified as uid 65532 after the non-root switch: healthz ok, healthcheck green, CLI works, puzzle renders, pool directory writable at mode 600

Left for later

Moving the GeoLite2 databases to a mounted volume would take this to ~290 MB, but operators would then have to supply them — a separate operational change.

Below that means dropping the Node runtime: the node binary alone is 120.7 MB (125.3 MB on musl), so 50–100 MB is not reachable without a rewrite in a compiled language.

🤖 Generated with Claude Code

@forgetso
forgetso force-pushed the perf/slim-provider-image branch 2 times, most recently from 9e1b2d5 to 3cce1ba Compare September 22, 2026 10:06
@forgetso forgetso changed the title perf(provider): cut the docker image from 1.84 GB to 538 MB perf(provider): cut the docker image from 1.84 GB to 376 MB (distroless) Sep 22, 2026
@forgetso
forgetso force-pushed the perf/slim-provider-image branch 2 times, most recently from eb45bfa to d2d916b Compare September 22, 2026 10:46
node:24 is 1.63 GB -- 89% of the old image -- and ships a build toolchain this
container never uses. The build is now two stages: a node:24-slim builder does
everything needing a shell, apt or the network, and the runtime stage is
distroless with the result copied in.

Alpine was rejected rather than overlooked: all four native addons are
linux-x64-gnu and musl needs separately compiled binaries. The distroless
runtime is Debian 12, matching the builder.

BREAKING for operators: no shell, no npm, no npx in the runtime image. npx was
only human convenience; every call site now invokes node explicitly as
`/nodejs/bin/node /usr/src/app/provider.cli.bundle.js <command>`. Healthchecks
move from curl to `node -e "fetch(...)"`. `docker exec -it provider1 bash` no
longer works.

Updated here: the provider_image CI smoke test, both provider healthchecks in
docker-compose.provider.yml, SELF_HOSTING.md and datasets-fs/README.md. The
ansible playbooks and RUNBOOK are updated in captcha-private.

Verified as a full stack: healthz ok, healthcheck healthy, MaxMind readers up,
dataset loads, all three captcha types serve, no errors logged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@forgetso
forgetso force-pushed the perf/slim-provider-image branch from d2d916b to a006053 Compare September 22, 2026 10:50
@forgetso
forgetso enabled auto-merge (squash) September 22, 2026 10:51
@forgetso
forgetso merged commit 56ca621 into main Sep 22, 2026
19 checks passed
@forgetso
forgetso deleted the perf/slim-provider-image branch September 22, 2026 11:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant