diff --git a/.changeset/slim-provider-image.md b/.changeset/slim-provider-image.md new file mode 100644 index 0000000000..9be389576b --- /dev/null +++ b/.changeset/slim-provider-image.md @@ -0,0 +1,80 @@ +--- +--- + +Cut the provider image from 1.84 GB to 376 MB, and drop the shell from it. + +Almost all of the old size was the base. `node:24` is 1.63 GB — 89% of the image — and ships a build toolchain this +container never uses: it runs a pre-bundled JS file and installs one dependency from a prebuilt binary. Our own layers +only ever accounted for ~137 MB. + +The build is now two stages. A `node:24-slim` builder does everything that needs a shell, apt or the network — install +curl, fetch the three GeoLite2 databases, `npm i sharp` — and the runtime stage is `gcr.io/distroless/nodejs24-debian12` +with the result copied in. Nothing but the Node runtime, the bundle and sharp reaches the final image. + +Smaller things folded in: the three GeoLite `RUN`s are one layer instead of three, and the npm cache is cleaned in the +layer that creates it. + +**Alpine was rejected, not overlooked.** All four native addons are `linux-x64-gnu` — `@img/sharp`, +`@prosopo/native-ja4`, `@prosopo/native-merkle`, `@prosopo/native-puzzle` — and musl needs separately compiled +binaries. The distroless runtime is Debian 12, matching the builder, so the glibc the addons were resolved against is +the one that loads them. + +## Breaking for operators + +There is no shell, no `npm` and no `npx` in the runtime image. `npx provider ` was only ever human +convenience; every call site has been moved to an explicit node invocation: + +```bash +docker exec provider1 /nodejs/bin/node /usr/src/app/provider.cli.bundle.js +``` + +Updated here: `SELF_HOSTING.md`, `packages/datasets-fs/README.md`, and the `provider_image` CI smoke test (which ran +`/bin/sh -c '... npx provider version'` and would otherwise have failed on the first build). Healthchecks in +`docker/docker-compose.provider.yml` for both provider containers no longer use `curl` — it is not in the image — and +run `node -e "fetch(...)"` instead. The matching ansible playbooks (`providerEnsureIndexes`, +`providerLoadDataset`, `providerMigrateIps`, `providerUpdateSpamEmailDomains`) and `RUNBOOK.md` are updated in +captcha-private. + +Healthchecks in `docker-compose.self-hosted.yml` move to node for the same reason. + +`docker exec -it provider1 bash` no longer works. That is the real cost of this change. + +## Verified + +Built all three variants from the same bundle and ran the distroless one as a full stack against Mongo and Redis: + +- boots; `/healthz` ok; the node-based container healthcheck reports `healthy` +- MaxMind City and ASN readers initialise +- CLI works via explicit node: `version`, `ensure_indexes`, `provider_set_data_set` (dataset loads), `site_key_register` +- all three captcha types serve — `pow`, `puzzle` (29 KB WebP, so sharp renders), `image` from a loaded dataset with + `signature` present +- zero `error` or `fatal` lines in the provider log + +## Runs as non-root + +The runtime stage is `gcr.io/distroless/nodejs24-debian12:nonroot` — uid/gid 65532 — and the copied application tree +is chowned to that user. The container no longer runs as root. + +This has a hard deployment prerequisite on the fleet. The provider writes at runtime to a host bind mount +(`./data/detector-pool.N` → `/app/data/detector-pool`): the pool-replace admin endpoint does `mkdir` + `writeFile`, +and `assignSecret` drops a `0600` secret alongside. Those directories are root-owned today, and docker auto-creates +them as root on `up`. Confirmed by running the non-root image against both: a root-owned pool directory fails every +write with `EACCES`, and the same directory chowned to 65532 writes fine at mode `600`. + +**That failure is silent.** An unwritable or unreadable pool is not an error — `getDetectorBundlePool()` returns empty +and `runEmptyDetectorPoolPowFallback` serves proof of work instead, so scoring would degrade fleet-wide with nothing +in the logs to explain it. + +The matching ansible task (captcha-private) creates and chowns both pool directories, and **must be deployed and +verified before this image reaches any pronode.** + +Self-hosted deployments are unaffected: `docker-compose.self-hosted.yml` mounts no pool directory, so the path is +created inside the container by the runtime user. + +Privileged ports are a non-issue — Docker sets `ip_unprivileged_port_start=0`, and the API listens on +`PROSOPO_API_PORT` (9229) with Caddy fronting 80/443 regardless. + +## Remaining + +Moving the GeoLite2 databases to a mounted volume would take this to ~290 MB — deliberately separate because it +changes how the image is operated. diff --git a/.github/workflows/provider_image.yml b/.github/workflows/provider_image.yml index 0f924a389d..8833143e42 100644 --- a/.github/workflows/provider_image.yml +++ b/.github/workflows/provider_image.yml @@ -113,7 +113,10 @@ jobs: set -euxo pipefail # stop on errors, print commands, fail on pipe fails # Run container and capture exit code - CONTAINER="$(docker run -d -v "./.env.production:/usr/src/app/.env.production" prosopo/provider:dev /bin/sh -c 'NODE_ENV=production npx provider version')" + # The runtime image is distroless: no shell, no npx. ENTRYPOINT is + # node, so the bundle path and its args are passed as argv, and + # NODE_ENV comes in as a real env var rather than a shell prefix. + CONTAINER="$(docker run -d -e NODE_ENV=production -v "./.env.production:/usr/src/app/.env.production" prosopo/provider:dev /usr/src/app/provider.cli.bundle.js version)" # Wait for container to complete sleep 20s diff --git a/SELF_HOSTING.md b/SELF_HOSTING.md index bd9215f576..d8cc29f7cd 100644 --- a/SELF_HOSTING.md +++ b/SELF_HOSTING.md @@ -72,7 +72,7 @@ Pin different versions with `PROSOPO_PROVIDER_IMAGE_VERSION` and `CADDY_IMAGE_VE ## 3. Register a site key ```bash -docker compose exec provider npx provider site_key_register \ +docker compose exec provider /nodejs/bin/node /usr/src/app/provider.cli.bundle.js site_key_register \ "$(grep '^PROSOPO_SITE_KEY=' .env | cut -d= -f2)" enterprise \ --captcha_type puzzle \ --domains example.com \ @@ -144,10 +144,10 @@ node dist/cli.js generate-v2 --out ./captchas.json \ ```bash docker compose cp ./captchas.json provider:/usr/src/app/captchas.json -docker compose exec provider npx provider provider_set_data_set --file /usr/src/app/captchas.json +docker compose exec provider /nodejs/bin/node /usr/src/app/provider.cli.bundle.js provider_set_data_set --file /usr/src/app/captchas.json docker compose restart provider -docker compose exec provider npx provider site_key_register "$(grep '^PROSOPO_SITE_KEY=' .env | cut -d= -f2)" enterprise \ +docker compose exec provider /nodejs/bin/node /usr/src/app/provider.cli.bundle.js site_key_register "$(grep '^PROSOPO_SITE_KEY=' .env | cut -d= -f2)" enterprise \ --captcha_type image --domains example.com \ --pow_difficulty 4 --frictionless_threshold 0.5 --image_threshold 0.8 ``` diff --git a/docker-compose.self-hosted.yml b/docker-compose.self-hosted.yml index e1b4a63219..30a4067100 100644 --- a/docker-compose.self-hosted.yml +++ b/docker-compose.self-hosted.yml @@ -61,7 +61,13 @@ services: max-size: '100m' max-file: '1' healthcheck: - test: ['CMD', 'curl', '--fail', 'localhost:9229/healthz'] + # The provider image is distroless: no curl, no shell. node is the only + # binary in it, so the check runs in node. + test: + - CMD + - /nodejs/bin/node + - -e + - fetch('http://localhost:9229/healthz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1)) interval: 5m retries: 3 start_period: 30s diff --git a/docker/docker-compose.provider.yml b/docker/docker-compose.provider.yml index ffa08e2182..cfa9f9a029 100644 --- a/docker/docker-compose.provider.yml +++ b/docker/docker-compose.provider.yml @@ -33,7 +33,8 @@ services: max-size: '100m' max-file: '1' healthcheck: - test: [ "CMD", "curl", "--fail", "localhost:9229/healthz" ] # ping the healthz endpoint + # Distroless runtime: no curl. node is the only binary in the image. + test: [ "CMD", "/nodejs/bin/node", "-e", "fetch('http://localhost:9229/healthz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))" ] interval: 5m retries: 3 start_period: 30s @@ -72,7 +73,8 @@ services: max-size: '100m' max-file: '1' healthcheck: - test: [ "CMD", "curl", "--fail", "localhost:9339/healthz" ] # ping the healthz endpoint + # Distroless runtime: no curl. node is the only binary in the image. + test: [ "CMD", "/nodejs/bin/node", "-e", "fetch('http://localhost:9339/healthz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))" ] interval: 5m retries: 3 start_period: 30s diff --git a/packages/cli/Dockerfile b/packages/cli/Dockerfile index ba27ad798e..1967ecf557 100644 --- a/packages/cli/Dockerfile +++ b/packages/cli/Dockerfile @@ -1,30 +1,82 @@ -FROM node:24 +# Two stages. Everything that needs a shell, a package manager or a network +# fetch happens in the builder; the runtime image is distroless and contains +# only the Node runtime, this bundle and its one dependency. +# +# The builder is `node:24-slim` rather than `node:24` for the same reason the +# runtime is distroless: `node:24` is 1.63 GB of build toolchain that a +# pre-bundled JS file and a single prebuilt-binary dependency never touch. +FROM node:24-slim AS builder -WORKDIR /usr/src/app +WORKDIR /build # set shell to bash and error on any failure, unset variables, and pipefail SHELL ["/bin/bash", "-euo", "pipefail", "-c"] +# slim ships no curl, and the GeoLite fetches below need it. Builder-only, so +# none of this reaches the runtime image. +RUN apt-get update && \ + apt-get install -y --no-install-recommends curl ca-certificates && \ + rm -rf /var/lib/apt/lists/* + COPY ./packages/cli/dist/bundle/ ./ COPY ./packages/locale/dist/locales/ ./locales/ -# download the GeoLite2 databases into the same directory -RUN curl -L -o GeoLite2-Country.mmdb https://git.io/GeoLite2-Country.mmdb -RUN curl -L -o GeoLite2-City.mmdb https://git.io/GeoLite2-City.mmdb -RUN curl -L -o GeoLite2-ASN.mmdb https://git.io/GeoLite2-ASN.mmdb - -# make the provider cli executable and set the binary for npx. The -# generated package.json declares `sharp` as a runtime dependency so -# `npm i` installs the linux-x64 native binary — vite bundles the JS -# shim but not the platform-specific .node addon, and the puzzle -# renderer imports sharp from @prosopo/puzzle-assets at runtime. -# --os=linux --cpu=x64 --include=optional forces npm to fetch the -# correct optional platform variant regardless of the build host. -RUN echo '#!/usr/bin/env node' | cat - provider.cli.bundle.js > temp && mv temp provider.cli.bundle.js && \ - echo '{ "name": "prosopo-provider", "bin": { "provider": "./provider.cli.bundle.js" }, "type": "module", "dependencies": { "sharp": "0.35.3" } }' > /usr/src/app/package.json && \ - npm i --include=optional --os=linux --cpu=x64 +# Download the GeoLite2 databases into the same directory. One layer rather +# than three so the intermediate states are not retained. +RUN curl -fLsS -o GeoLite2-Country.mmdb https://git.io/GeoLite2-Country.mmdb && \ + curl -fLsS -o GeoLite2-City.mmdb https://git.io/GeoLite2-City.mmdb && \ + curl -fLsS -o GeoLite2-ASN.mmdb https://git.io/GeoLite2-ASN.mmdb + +# The generated package.json declares `sharp` as a runtime dependency so +# `npm i` installs the linux-x64 native binary — vite bundles the JS shim but +# not the platform-specific .node addon, and the puzzle renderer imports sharp +# from @prosopo/puzzle-assets at runtime. --os=linux --cpu=x64 +# --include=optional forces npm to fetch the correct optional platform variant +# regardless of the build host. +# +# The shebang points at the absolute node path rather than `/usr/bin/env node`: +# there is no `env` on PATH in the runtime image to resolve a relative +# interpreter, but the kernel honours an absolute one. It is a convenience +# only — every caller in this repo invokes node explicitly, because `docker +# exec` does not apply ENTRYPOINT: +# docker exec provider1 /nodejs/bin/node /usr/src/app/provider.cli.bundle.js +RUN echo '#!/nodejs/bin/node' | cat - provider.cli.bundle.js > temp && \ + mv temp provider.cli.bundle.js && \ + chmod +x provider.cli.bundle.js && \ + echo '{ "name": "prosopo-provider", "type": "module", "dependencies": { "sharp": "0.35.3" } }' > package.json && \ + npm i --include=optional --os=linux --cpu=x64 --no-fund --no-audit && \ + npm cache clean --force + +# `nodejs24-debian12` matches the builder's Debian release, so the glibc the +# native addons were resolved against is the same one that loads them. All +# four (@img/sharp, native-ja4, native-merkle, native-puzzle) are +# linux-x64-gnu; an Alpine/musl runtime would break every one. +# +# The `:nonroot` tag runs as uid/gid 65532 rather than root. The bind-mounted +# detector-pool directory must be owned by that uid on the host or the pool +# cannot be written or re-read — and an unreadable pool fails silently, with +# the provider falling back to proof of work. The ansible provider role +# chowns it; see captcha-private issue #4749. +# +# There is no shell, no package manager and no npx in here. That is the point, +# and it is also the operational cost: `docker exec ... bash` no longer works, +# CLI commands must name node explicitly (see above), and container +# healthchecks cannot use curl — they run `node -e "fetch(...)"` instead. Every +# call site in this repo and in the ansible playbooks has been moved over. +FROM gcr.io/distroless/nodejs24-debian12:nonroot + +WORKDIR /usr/src/app + +# --chown so the runtime user owns what it needs to read and write. +COPY --from=builder --chown=65532:65532 /build/ ./ EXPOSE 9229 9339 80 443 -CMD ["npx" , "provider" ,"--api"] +# Redundant with the `:nonroot` tag, which already runs as this uid, but +# stated explicitly: it is the only form a Dockerfile reader — or a scanner — +# can see, and it keeps the guarantee if the tag is ever changed. +USER 65532:65532 + +# ENTRYPOINT is /nodejs/bin/node, so this is argv to node. +CMD ["/usr/src/app/provider.cli.bundle.js", "--api"] diff --git a/packages/datasets-fs/README.md b/packages/datasets-fs/README.md index f10a95379d..acdf3a97be 100644 --- a/packages/datasets-fs/README.md +++ b/packages/datasets-fs/README.md @@ -126,7 +126,7 @@ counts instead of `--count`. ```bash docker compose cp ./captchas.json provider:/usr/src/app/captchas.json -docker compose exec provider npx provider provider_set_data_set --file /usr/src/app/captchas.json +docker compose exec provider /nodejs/bin/node /usr/src/app/provider.cli.bundle.js provider_set_data_set --file /usr/src/app/captchas.json docker compose restart provider ``` @@ -137,7 +137,7 @@ dataset uploaded into a running provider is stored but not served — image chal Then register your site key for image captchas: ```bash -docker compose exec provider npx provider site_key_register enterprise \ +docker compose exec provider /nodejs/bin/node /usr/src/app/provider.cli.bundle.js site_key_register enterprise \ --captcha_type image --domains example.com \ --pow_difficulty 4 --frictionless_threshold 0.5 --image_threshold 0.8 ```