perf(provider): cut the docker image from 1.84 GB to 376 MB (distroless)

.changeset/slim-provider-image.md

@@ -0,0 +1,80 @@
+---
+---
+
+Cut the provider image from 1.84 GB to 376 MB, and drop the shell from it.
+
+Almost all of the old size was the base. `node:24` is 1.63 GB — 89% of the image — and ships a build toolchain this
+container never uses: it runs a pre-bundled JS file and installs one dependency from a prebuilt binary. Our own layers
+only ever accounted for ~137 MB.
+
+The build is now two stages. A `node:24-slim` builder does everything that needs a shell, apt or the network — install
+curl, fetch the three GeoLite2 databases, `npm i sharp` — and the runtime stage is `gcr.io/distroless/nodejs24-debian12`
+with the result copied in. Nothing but the Node runtime, the bundle and sharp reaches the final image.
+
+Smaller things folded in: the three GeoLite `RUN`s are one layer instead of three, and the npm cache is cleaned in the
+layer that creates it.
+
+**Alpine was rejected, not overlooked.** All four native addons are `linux-x64-gnu` — `@img/sharp`,
+`@prosopo/native-ja4`, `@prosopo/native-merkle`, `@prosopo/native-puzzle` — and musl needs separately compiled
+binaries. The distroless runtime is Debian 12, matching the builder, so the glibc the addons were resolved against is
+the one that loads them.
+
+## Breaking for operators
+
+There is no shell, no `npm` and no `npx` in the runtime image. `npx provider <command>` was only ever human
+convenience; every call site has been moved to an explicit node invocation:
+
+```bash
+docker exec provider1 /nodejs/bin/node /usr/src/app/provider.cli.bundle.js <command>
+```
+
+Updated here: `SELF_HOSTING.md`, `packages/datasets-fs/README.md`, and the `provider_image` CI smoke test (which ran
+`/bin/sh -c '... npx provider version'` and would otherwise have failed on the first build). Healthchecks in
+`docker/docker-compose.provider.yml` for both provider containers no longer use `curl` — it is not in the image — and
+run `node -e "fetch(...)"` instead. The matching ansible playbooks (`providerEnsureIndexes`,
+`providerLoadDataset`, `providerMigrateIps`, `providerUpdateSpamEmailDomains`) and `RUNBOOK.md` are updated in
+captcha-private.
+
+Healthchecks in `docker-compose.self-hosted.yml` move to node for the same reason.
+
+`docker exec -it provider1 bash` no longer works. That is the real cost of this change.
+
+## Verified
+
+Built all three variants from the same bundle and ran the distroless one as a full stack against Mongo and Redis:
+
+- boots; `/healthz` ok; the node-based container healthcheck reports `healthy`
+- MaxMind City and ASN readers initialise
+- CLI works via explicit node: `version`, `ensure_indexes`, `provider_set_data_set` (dataset loads), `site_key_register`
+- all three captcha types serve — `pow`, `puzzle` (29 KB WebP, so sharp renders), `image` from a loaded dataset with
+  `signature` present
+- zero `error` or `fatal` lines in the provider log
+
+## Runs as non-root
+
+The runtime stage is `gcr.io/distroless/nodejs24-debian12:nonroot` — uid/gid 65532 — and the copied application tree
+is chowned to that user. The container no longer runs as root.
+
+This has a hard deployment prerequisite on the fleet. The provider writes at runtime to a host bind mount
+(`./data/detector-pool.N` → `/app/data/detector-pool`): the pool-replace admin endpoint does `mkdir` + `writeFile`,
+and `assignSecret` drops a `0600` secret alongside. Those directories are root-owned today, and docker auto-creates
+them as root on `up`. Confirmed by running the non-root image against both: a root-owned pool directory fails every
+write with `EACCES`, and the same directory chowned to 65532 writes fine at mode `600`.
+
+**That failure is silent.** An unwritable or unreadable pool is not an error — `getDetectorBundlePool()` returns empty
+and `runEmptyDetectorPoolPowFallback` serves proof of work instead, so scoring would degrade fleet-wide with nothing
+in the logs to explain it.
+
+The matching ansible task (captcha-private) creates and chowns both pool directories, and **must be deployed and
+verified before this image reaches any pronode.**
+
+Self-hosted deployments are unaffected: `docker-compose.self-hosted.yml` mounts no pool directory, so the path is
+created inside the container by the runtime user.
+
+Privileged ports are a non-issue — Docker sets `ip_unprivileged_port_start=0`, and the API listens on
+`PROSOPO_API_PORT` (9229) with Caddy fronting 80/443 regardless.
+
+## Remaining
+
+Moving the GeoLite2 databases to a mounted volume would take this to ~290 MB — deliberately separate because it
+changes how the image is operated.

.github/workflows/provider_image.yml

@@ -113,7 +113,10 @@ jobs:
           set -euxo pipefail # stop on errors, print commands, fail on pipe fails
 
           # Run container and capture exit code
-          CONTAINER="$(docker run -d -v "./.env.production:/usr/src/app/.env.production" prosopo/provider:dev /bin/sh -c 'NODE_ENV=production npx provider version')"
+          # The runtime image is distroless: no shell, no npx. ENTRYPOINT is
+          # node, so the bundle path and its args are passed as argv, and
+          # NODE_ENV comes in as a real env var rather than a shell prefix.
+          CONTAINER="$(docker run -d -e NODE_ENV=production -v "./.env.production:/usr/src/app/.env.production" prosopo/provider:dev /usr/src/app/provider.cli.bundle.js version)"
 
           # Wait for container to complete
           sleep 20s

SELF_HOSTING.md

@@ -72,7 +72,7 @@ Pin different versions with `PROSOPO_PROVIDER_IMAGE_VERSION` and `CADDY_IMAGE_VE
 ## 3. Register a site key
 
 ```bash
-docker compose exec provider npx provider site_key_register \
+docker compose exec provider /nodejs/bin/node /usr/src/app/provider.cli.bundle.js site_key_register \
   "$(grep '^PROSOPO_SITE_KEY=' .env | cut -d= -f2)" enterprise \
   --captcha_type puzzle \
   --domains example.com \
@@ -144,10 +144,10 @@ node dist/cli.js generate-v2 --out ./captchas.json \
 
 ```bash
 docker compose cp ./captchas.json provider:/usr/src/app/captchas.json
-docker compose exec provider npx provider provider_set_data_set --file /usr/src/app/captchas.json
+docker compose exec provider /nodejs/bin/node /usr/src/app/provider.cli.bundle.js provider_set_data_set --file /usr/src/app/captchas.json
 docker compose restart provider
 
-docker compose exec provider npx provider site_key_register "$(grep '^PROSOPO_SITE_KEY=' .env | cut -d= -f2)" enterprise \
+docker compose exec provider /nodejs/bin/node /usr/src/app/provider.cli.bundle.js site_key_register "$(grep '^PROSOPO_SITE_KEY=' .env | cut -d= -f2)" enterprise \
   --captcha_type image --domains example.com \
   --pow_difficulty 4 --frictionless_threshold 0.5 --image_threshold 0.8
 ```

docker-compose.self-hosted.yml

@@ -61,7 +61,13 @@ services:
         max-size: '100m'
         max-file: '1'
     healthcheck:
-      test: ['CMD', 'curl', '--fail', 'localhost:9229/healthz']
+      # The provider image is distroless: no curl, no shell. node is the only
+      # binary in it, so the check runs in node.
+      test:
+        - CMD
+        - /nodejs/bin/node
+        - -e
+        - fetch('http://localhost:9229/healthz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))
       interval: 5m
       retries: 3
       start_period: 30s

docker/docker-compose.provider.yml

@@ -33,7 +33,8 @@ services:
         max-size: '100m'
         max-file: '1'
     healthcheck:
-      test: [ "CMD", "curl", "--fail", "localhost:9229/healthz" ] # ping the healthz endpoint
+      # Distroless runtime: no curl. node is the only binary in the image.
+      test: [ "CMD", "/nodejs/bin/node", "-e", "fetch('http://localhost:9229/healthz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))" ]
       interval: 5m
       retries: 3
       start_period: 30s
@@ -72,7 +73,8 @@ services:
         max-size: '100m'
         max-file: '1'
     healthcheck:
-      test: [ "CMD", "curl", "--fail", "localhost:9339/healthz" ] # ping the healthz endpoint
+      # Distroless runtime: no curl. node is the only binary in the image.
+      test: [ "CMD", "/nodejs/bin/node", "-e", "fetch('http://localhost:9339/healthz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))" ]
       interval: 5m
       retries: 3
       start_period: 30s

packages/cli/Dockerfile

@@ -1,30 +1,82 @@
-FROM node:24
+# Two stages. Everything that needs a shell, a package manager or a network
+# fetch happens in the builder; the runtime image is distroless and contains
+# only the Node runtime, this bundle and its one dependency.
+#
+# The builder is `node:24-slim` rather than `node:24` for the same reason the
+# runtime is distroless: `node:24` is 1.63 GB of build toolchain that a
+# pre-bundled JS file and a single prebuilt-binary dependency never touch.
+FROM node:24-slim AS builder
 
-WORKDIR /usr/src/app
+WORKDIR /build
 
 # set shell to bash and error on any failure, unset variables, and pipefail
 SHELL ["/bin/bash", "-euo", "pipefail", "-c"]
 
+# slim ships no curl, and the GeoLite fetches below need it. Builder-only, so
+# none of this reaches the runtime image.
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends curl ca-certificates && \
+    rm -rf /var/lib/apt/lists/*
+
 COPY ./packages/cli/dist/bundle/ ./
 
 COPY ./packages/locale/dist/locales/ ./locales/
 
-# download the GeoLite2 databases into the same directory
-RUN curl -L -o GeoLite2-Country.mmdb https://git.io/GeoLite2-Country.mmdb
-RUN curl -L -o GeoLite2-City.mmdb https://git.io/GeoLite2-City.mmdb
-RUN curl -L -o GeoLite2-ASN.mmdb https://git.io/GeoLite2-ASN.mmdb
-
-# make the provider cli executable and set the binary for npx. The
-# generated package.json declares `sharp` as a runtime dependency so
-# `npm i` installs the linux-x64 native binary — vite bundles the JS
-# shim but not the platform-specific .node addon, and the puzzle
-# renderer imports sharp from @prosopo/puzzle-assets at runtime.
-# --os=linux --cpu=x64 --include=optional forces npm to fetch the
-# correct optional platform variant regardless of the build host.
-RUN echo '#!/usr/bin/env node' | cat - provider.cli.bundle.js > temp && mv temp provider.cli.bundle.js && \
-    echo '{ "name": "prosopo-provider", "bin": { "provider": "./provider.cli.bundle.js" }, "type": "module", "dependencies": { "sharp": "0.35.3" } }' > /usr/src/app/package.json && \
-    npm i --include=optional --os=linux --cpu=x64
+# Download the GeoLite2 databases into the same directory. One layer rather
+# than three so the intermediate states are not retained.
+RUN curl -fLsS -o GeoLite2-Country.mmdb https://git.io/GeoLite2-Country.mmdb && \
+    curl -fLsS -o GeoLite2-City.mmdb https://git.io/GeoLite2-City.mmdb && \
+    curl -fLsS -o GeoLite2-ASN.mmdb https://git.io/GeoLite2-ASN.mmdb
+
+# The generated package.json declares `sharp` as a runtime dependency so
+# `npm i` installs the linux-x64 native binary — vite bundles the JS shim but
+# not the platform-specific .node addon, and the puzzle renderer imports sharp
+# from @prosopo/puzzle-assets at runtime. --os=linux --cpu=x64
+# --include=optional forces npm to fetch the correct optional platform variant
+# regardless of the build host.
+#
+# The shebang points at the absolute node path rather than `/usr/bin/env node`:
+# there is no `env` on PATH in the runtime image to resolve a relative
+# interpreter, but the kernel honours an absolute one. It is a convenience
+# only — every caller in this repo invokes node explicitly, because `docker
+# exec` does not apply ENTRYPOINT:
+#   docker exec provider1 /nodejs/bin/node /usr/src/app/provider.cli.bundle.js <command>
+RUN echo '#!/nodejs/bin/node' | cat - provider.cli.bundle.js > temp && \
+    mv temp provider.cli.bundle.js && \
+    chmod +x provider.cli.bundle.js && \
+    echo '{ "name": "prosopo-provider", "type": "module", "dependencies": { "sharp": "0.35.3" } }' > package.json && \
+    npm i --include=optional --os=linux --cpu=x64 --no-fund --no-audit && \
+    npm cache clean --force
+
+# `nodejs24-debian12` matches the builder's Debian release, so the glibc the
+# native addons were resolved against is the same one that loads them. All
+# four (@img/sharp, native-ja4, native-merkle, native-puzzle) are
+# linux-x64-gnu; an Alpine/musl runtime would break every one.
+#
+# The `:nonroot` tag runs as uid/gid 65532 rather than root. The bind-mounted
+# detector-pool directory must be owned by that uid on the host or the pool
+# cannot be written or re-read — and an unreadable pool fails silently, with
+# the provider falling back to proof of work. The ansible provider role
+# chowns it; see captcha-private issue #4749.
+#
+# There is no shell, no package manager and no npx in here. That is the point,
+# and it is also the operational cost: `docker exec ... bash` no longer works,
+# CLI commands must name node explicitly (see above), and container
+# healthchecks cannot use curl — they run `node -e "fetch(...)"` instead. Every
+# call site in this repo and in the ansible playbooks has been moved over.
+FROM gcr.io/distroless/nodejs24-debian12:nonroot
+
+WORKDIR /usr/src/app
+
+# --chown so the runtime user owns what it needs to read and write.
+COPY --from=builder --chown=65532:65532 /build/ ./
 
 EXPOSE 9229 9339 80 443
 
-CMD ["npx" , "provider" ,"--api"]
+# Redundant with the `:nonroot` tag, which already runs as this uid, but
+# stated explicitly: it is the only form a Dockerfile reader — or a scanner —
+# can see, and it keeps the guarantee if the tag is ever changed.
+USER 65532:65532
+
+# ENTRYPOINT is /nodejs/bin/node, so this is argv to node.
+CMD ["/usr/src/app/provider.cli.bundle.js", "--api"]

packages/datasets-fs/README.md

@@ -126,7 +126,7 @@ counts instead of `--count`.
 
 ```bash
 docker compose cp ./captchas.json provider:/usr/src/app/captchas.json
-docker compose exec provider npx provider provider_set_data_set --file /usr/src/app/captchas.json
+docker compose exec provider /nodejs/bin/node /usr/src/app/provider.cli.bundle.js provider_set_data_set --file /usr/src/app/captchas.json
 docker compose restart provider
 ```
 
@@ -137,7 +137,7 @@ dataset uploaded into a running provider is stored but not served — image chal
 Then register your site key for image captchas:
 
 ```bash
-docker compose exec provider npx provider site_key_register <sitekey> enterprise \
+docker compose exec provider /nodejs/bin/node /usr/src/app/provider.cli.bundle.js site_key_register <sitekey> enterprise \
   --captcha_type image --domains example.com \
   --pow_difficulty 4 --frictionless_threshold 0.5 --image_threshold 0.8
 ```