https-app: an app can claim a handle ending, so its handles resolve through nginx

README.md

@@ -600,6 +600,7 @@ runs against it with no devnet code:
 | `prod.devnet.test`, `*.prod.devnet.test` | `pds-prod`, the build you run in prod, and its handles |
 | `plc.directory` | this stack's own PLC |
 | `<name>.devnet.internal` | an app on this machine, at the port `scripts/https-app.sh` recorded for `<name>` |
+| `*.<ending>.devnet.test` | the app that claimed `<ending>` with `scripts/https-app.sh`, for the handles it hands out |
 
 nginx answers all of them on `127.0.0.1:443` with one certificate from a local CA. Each PDS's
 public URL, OAuth issuer and DID document endpoint is `https://<its name>`, and the relay crawls
@@ -753,6 +754,19 @@ metadata and keys from there over TLS. A client_id under `.test` is refused by t
 (`invalid_client_id: The client_id's TLD must not be a local hostname`), which is why apps live under
 `.devnet.internal`.
 
+**A handle ending**, for an app that hands out handles of its own, as a PDS does. A third word
+claims one:
+
+```bash
+scripts/https-app.sh grouphost 7070 gh  # also https://<handle>.gh.devnet.test -> port 7070
+```
+
+nginx then sends every name under `gh.devnet.test` to the app, Host and all, so the app answers
+`/.well-known/atproto-did` for its handles and they resolve from anywhere on the devnet. The leaf
+gains `*.gh.devnet.test` from the same CA, so nothing has to be trusted again. `alpha`, `regular` and
+`prod` belong to the PDSes, and an ending belongs to one app. A rerun without the ending drops it.
+Under `https-run`, list each such handle in `HTTPS_RUN_EXTRA_NAMES`, as for any handle.
+
 `scripts/https-probe.mjs` checks the whole path for any app name. It resolves an account through
 `https://plc.directory` with stock atcute at its defaults, serves a confidential client at
 `https://<PROBE_APP>.devnet.internal/`, pushes a PAR with that client_id, and pushes a raw PAR with a
@@ -803,9 +817,11 @@ REFUSED .test client_id: HTTP 400 invalid_client_id: The client_id's TLD must no
 `scripts/https-ca.sh` makes the CA once. `https-up.sh` runs it when there is no leaf, or when the
 leaf does not name `*.devnet.internal`. Rerunning it keeps the CA and only reissues the leaf, so a
 browser or OS that trusts `ca.crt` keeps trusting the devnet. The leaf names the three PDSes,
-`*.devnet.test`, `*.regular.devnet.test`, `*.prod.devnet.test`, `plc.directory` and
-`*.devnet.internal`; a TLS wildcard covers one label, so each handle domain needs its own. Add
-deeper names with `HTTPS_EXTRA_NAMES="x.mine.devnet.test"`, then restart nginx. A name one label
+`*.devnet.test`, `*.regular.devnet.test`, `*.prod.devnet.test`, `plc.directory`,
+`*.devnet.internal`, and `*.<ending>.devnet.test` for each handle ending an app claimed; a TLS
+wildcard covers one label, so each handle domain needs its own. `https-app.sh` reissues the leaf
+when it claims an ending, and `https-up.sh` when the leaf lacks any of these names. Add deeper names
+with `HTTPS_EXTRA_NAMES="x.mine.devnet.test"` on every run, then restart nginx. A name one label
 under `devnet.test` is already covered. To start over, delete `data/https/ca.*` and trust the new
 `ca.crt` wherever the old one was trusted.
 

https/nginx.conf.template

@@ -82,7 +82,9 @@ server {
 # data/https/nginx/ (mounted at /etc/nginx/devnet/), one upstream per app, naming
 # host.docker.internal, and one map line per name. An upstream's name is resolved
 # once, when nginx loads it, through the container's hosts file (extra_hosts), which
-# Docker's DNS does not serve. Neither file has to exist.
+# Docker's DNS does not serve. Neither file has to exist. An app that claimed a handle
+# ending (scripts/https-app.sh <name> <port> <ending>) also gets a server there for
+# *.<ending>.devnet.test, which wins over the alpha's *.devnet.test as the longer wildcard.
 include /etc/nginx/devnet/*.conf;
 
 map $host $devnet_app {

scripts/https-app.sh

@@ -5,16 +5,23 @@ set -eu
 # devnet's nginx, for any app: its pages, its OAuth client metadata and its keys, at an
 # https origin the devnet's PDSes and https-run's processes both reach.
 #
-# Usage: ./scripts/https-app.sh <name> <port>
-#   <name>  one lowercase DNS label; the app is https://<name>.devnet.internal
-#   <port>  where the app listens on this machine. Listen on the address Docker's host
-#           gateway lands on (host.docker.internal; the docker0 address here), or on
-#           0.0.0.0, so nginx can reach it.
+# Usage: ./scripts/https-app.sh <name> <port> [<ending>]
+#   <name>    one lowercase DNS label; the app is https://<name>.devnet.internal
+#   <port>    where the app listens on this machine. Listen on the address Docker's host
+#             gateway lands on (host.docker.internal; the docker0 address here), or on
+#             0.0.0.0, so nginx can reach it.
+#   <ending>  for an app that hands out handles of its own, as a PDS does: one lowercase
+#             DNS label. Every name under <ending>.<domain> (DEVNET_DOMAIN, devnet.test by
+#             default) goes to the app too, so a handle there resolves at the app. The
+#             leaf gains *.<ending>.<domain>, from the same CA, so nothing has to be
+#             trusted again. alpha, regular and prod are the PDSes' own; an ending is one
+#             app's.
 #
-# The route is kept in data/https/apps (a rerun with the same name changes its port);
-# data/https/nginx/ is rendered from it, and nginx in DEVNET_PROJECT's stack is
-# reloaded (DEVNET_PROJECT defaults to the one data/devnet.env names). If the stack is
-# down, the route is kept and nginx loads it when https-up.sh starts it.
+# The route is kept in data/https/apps (a rerun with the same name changes its port, and
+# drops its ending unless given again); data/https/nginx/ is rendered from it, and nginx
+# in DEVNET_PROJECT's stack is reloaded (DEVNET_PROJECT defaults to the one
+# data/devnet.env names). If the stack is down, the route is kept and nginx loads it when
+# https-up.sh starts it.
 #
 # A name with no route gets a 404 from nginx and reaches nothing. Inside https-run the
 # name resolves to 127.0.0.1; restart a process already running under it to pick up a
@@ -25,18 +32,41 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 
 NAME="${1:-}"
 PORT="${2:-}"
-if ! echo "${NAME}" | grep -qxE '[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?' || ! echo "${PORT}" | grep -qxE '[1-9][0-9]{0,4}' || [ "${PORT}" -gt 65535 ]; then
-  echo "Usage: https-app.sh <name> <port>   (name: one lowercase DNS label)" >&2
+ENDING="${3:-}"
+label() { echo "$1" | grep -qxE '[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?'; }
+if ! label "${NAME}" || ! echo "${PORT}" | grep -qxE '[1-9][0-9]{0,4}' || [ "${PORT}" -gt 65535 ] \
+   || { [ -n "${ENDING}" ] && ! label "${ENDING}"; } || [ $# -gt 3 ]; then
+  echo "Usage: https-app.sh <name> <port> [<ending>]   (name, ending: one lowercase DNS label each)" >&2
   exit 2
 fi
 
 mkdir -p "${HTTPS_DIR}"
 APPS="${HTTPS_DIR}/apps"
 touch "${APPS}"
-{ awk -v n="${NAME}" '$1 != n' "${APPS}"; echo "${NAME} ${PORT}"; } > "${APPS}.tmp"
+if [ -n "${ENDING}" ]; then
+  case "${ENDING}" in
+    alpha | regular | prod)
+      echo "https-app: ${ENDING}.${DEVNET_DOMAIN} is a PDS's own" >&2
+      exit 1
+      ;;
+  esac
+  OWNER="$(awk -v n="${NAME}" -v e="${ENDING}" '$1 != n && $3 == e { print $1; exit }' "${APPS}")"
+  if [ -n "${OWNER}" ]; then
+    echo "https-app: handles under ${ENDING}.${DEVNET_DOMAIN} already go to ${OWNER} (data/https/apps)" >&2
+    exit 1
+  fi
+fi
+{ awk -v n="${NAME}" '$1 != n' "${APPS}"; echo "${NAME} ${PORT}${ENDING:+ ${ENDING}}"; } > "${APPS}.tmp"
 mv "${APPS}.tmp" "${APPS}"
+# The CA is kept, and a new leaf is written over the old file, which nginx's mount follows.
+NEW_CERTS=""
+if leaf_lacks; then
+  "${SCRIPT_DIR}/https-ca.sh"
+  NEW_CERTS=1
+fi
 render_apps
 echo "https://${NAME}.devnet.internal -> port ${PORT} on this machine (data/https/apps)" >&2
+[ -z "${ENDING}" ] || echo "https://<handle>.${ENDING}.${DEVNET_DOMAIN} -> the same port" >&2
 
 DEVNET_PROJECT="${DEVNET_PROJECT:-$(env_get DEVNET_PROJECT "${DATA_DIR}/devnet.env")}"
 if [ -z "${DEVNET_PROJECT}" ]; then
@@ -51,5 +81,6 @@ if ! dc exec -T nginx nginx -t -q; then
   echo "https-app: nginx refused the new routes; data/https/nginx/ holds what it read" >&2
   exit 1
 fi
+# A reload rereads the certificate too.
 dc exec -T nginx nginx -s reload
-echo "Reloaded nginx in ${DEVNET_PROJECT}" >&2
+echo "Reloaded nginx in ${DEVNET_PROJECT}${NEW_CERTS:+, with the new leaf}" >&2

scripts/https-ca.sh

@@ -2,9 +2,10 @@
 set -eu
 
 # The https devnet's certificates: one local CA, made once, and one leaf that nginx
-# serves for every devnet name: the three PDSes and their handles, plc.directory, and
-# every app at <name>.devnet.internal. The PDS names sit under DEVNET_DOMAIN
-# (devnet.test by default; scripts/https-lib.sh), and the names below use the default.
+# serves for every devnet name: the three PDSes and their handles, plc.directory, every
+# app at <name>.devnet.internal, and the handles under each ending an app claimed with
+# scripts/https-app.sh. The PDS names sit under DEVNET_DOMAIN (devnet.test by default;
+# scripts/https-lib.sh, whose leaf_names lists them), and the names below use the default.
 #
 # Usage: ./scripts/https-ca.sh
 #
@@ -20,20 +21,19 @@ set -eu
 # a new DEVNET_DOMAIN, means a new CA, trusted again.
 #
 # A TLS wildcard covers one label, so *.devnet.test does not cover a handle on the
-# regular PDS (<name>.regular.devnet.test); each PDS's handle domain has its own.
+# regular PDS (<name>.regular.devnet.test); each handle domain has its own.
 #
 # Nothing here prints a key. Pass extra leaf names with HTTPS_EXTRA_NAMES
-# (space-separated), for instance when a PDS is added.
+# (space-separated), for instance when a PDS is added, on every run.
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 . "${SCRIPT_DIR}/https-lib.sh"
 D="${DEVNET_DOMAIN}"
+NAMES="$(leaf_names | tr '\n' ' ')"
 OUT="${SCRIPT_DIR}/../data/https"
 mkdir -p "${OUT}"
 cd "${OUT}"
 
-NAMES="alpha.${D} regular.${D} prod.${D} *.${D} *.regular.${D} *.prod.${D} plc.directory *.devnet.internal ${HTTPS_EXTRA_NAMES:-}"
-
 umask 077
 
 if [ -f ca.crt ] && [ -f ca.key ]; then

scripts/https-lib.sh

@@ -109,23 +109,59 @@ add_name() {
   grep -qxF "$1" "${HTTPS_DIR}/names" || echo "$1" >> "${HTTPS_DIR}/names"
 }
 
-# render_apps: nginx's app routes, from data/https/apps (lines of "<name> <port>"), into
-# data/https/nginx/: one upstream per app at host.docker.internal:<port> (apps.conf) and
-# one map line per name (apps.map). See https/nginx.conf.template.
+# app_handles: the handle endings apps claimed with scripts/https-app.sh, one per line.
+app_handles() {
+  [ -f "${HTTPS_DIR}/apps" ] || return 0
+  awk 'NF >= 3 { print $3 }' "${HTTPS_DIR}/apps"
+}
+
+# leaf_names: every name the leaf carries, one or more per line. A TLS wildcard covers one
+# label, so each handle domain has its own: the three PDSes', and *.<ending>.<domain> for
+# each ending an app claimed. HTTPS_EXTRA_NAMES adds names (space-separated).
+leaf_names() {
+  _d="${DEVNET_DOMAIN}"
+  echo "alpha.${_d} regular.${_d} prod.${_d} *.${_d} *.regular.${_d} *.prod.${_d} plc.directory *.devnet.internal"
+  app_handles | sed "s/.*/*.&.${_d}/"
+  echo "${HTTPS_EXTRA_NAMES:-}"
+}
+
+# leaf_lacks: true when there is no leaf, or it lacks a name leaf_names lists.
+leaf_lacks() {
+  [ -f "${HTTPS_DIR}/leaf.crt" ] && [ -f "${HTTPS_DIR}/leaf.key" ] || return 0
+  _san="$(openssl x509 -in "${HTTPS_DIR}/leaf.crt" -noout -ext subjectAltName 2>/dev/null | tr -d ' ' | tr ',' '\n')"
+  set -f # a wildcard name is a name, not a glob
+  for _n in $(leaf_names); do
+    if ! printf '%s\n' "${_san}" | grep -qxF "DNS:${_n}"; then
+      set +f
+      return 0
+    fi
+  done
+  set +f
+  return 1
+}
+
+# render_apps: nginx's app routes, from data/https/apps (lines of "<name> <port>
+# [<ending>]"), into data/https/nginx/: one upstream per app at host.docker.internal:<port>
+# and, for an app that claimed a handle ending, a server for *.<ending>.<domain> (apps.conf),
+# and one map line per name (apps.map). See https/nginx.conf.template.
 render_apps() {
   mkdir -p "${HTTPS_DIR}/nginx"
   _apps="${HTTPS_DIR}/apps"
   _out="${HTTPS_DIR}/nginx"
   {
     echo "# Written by scripts/https-app.sh from data/https/apps; rerun it instead of editing."
     if [ -f "${_apps}" ]; then
-      awk 'NF == 2 { u = $1; gsub("-", "_", u); printf "upstream devnet_app_%s { server host.docker.internal:%s; }\n", u, $2 }' "${_apps}"
+      awk 'NF >= 2 { u = $1; gsub("-", "_", u); printf "upstream devnet_app_%s { server host.docker.internal:%s; }\n", u, $2 }' "${_apps}"
+      # Host is passed through (the http block sets it), so the app answers
+      # /.well-known/atproto-did for each handle. nginx takes the longer of two matching
+      # wildcards, so these win over the alpha's *.<domain>.
+      awk -v d="${DEVNET_DOMAIN}" 'NF >= 3 { u = $1; gsub("-", "_", u); printf "\n# Handles under %s.%s go to %s.\nserver {\n    listen 443 ssl;\n    server_name *.%s.%s;\n    location / {\n        proxy_pass http://devnet_app_%s;\n    }\n}\n", $3, d, $1, $3, d, u }' "${_apps}"
     fi
   } > "${_out}/apps.conf.tmp"
   {
     echo "# Written by scripts/https-app.sh from data/https/apps; rerun it instead of editing."
     if [ -f "${_apps}" ]; then
-      awk 'NF == 2 { u = $1; gsub("-", "_", u); printf "%s.devnet.internal devnet_app_%s;\n", $1, u }' "${_apps}"
+      awk 'NF >= 2 { u = $1; gsub("-", "_", u); printf "%s.devnet.internal devnet_app_%s;\n", $1, u }' "${_apps}"
     fi
   } > "${_out}/apps.map.tmp"
   mv "${_out}/apps.conf.tmp" "${_out}/apps.conf"

scripts/https-run

@@ -46,7 +46,7 @@ if [ -f "${DATA}/names" ]; then
   NAMES="${NAMES} $(grep -E '^[A-Za-z0-9.-]+$' "${DATA}/names" | tr '\n' ' ')"
 fi
 if [ -f "${DATA}/apps" ]; then
-  NAMES="${NAMES} $(awk 'NF == 2 { printf "%s.devnet.internal ", $1 }' "${DATA}/apps")"
+  NAMES="${NAMES} $(awk 'NF >= 2 { printf "%s.devnet.internal ", $1 }' "${DATA}/apps")"
 fi
 NAMES="${NAMES} ${HTTPS_RUN_EXTRA_NAMES:-}"
 

scripts/https-up.sh

@@ -39,10 +39,10 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 
 mkdir -p "${HTTPS_DIR}/nginx"
 
-# The leaf must name *.devnet.internal, which a leaf from before the app routes lacks.
+# The leaf must carry every name leaf_names lists: a leaf from before the app routes lacks
+# *.devnet.internal, and one from before an app claimed a handle ending lacks its wildcard.
 NEW_CERTS=""
-if [ ! -f "${HTTPS_DIR}/leaf.crt" ] || [ ! -f "${HTTPS_DIR}/leaf.key" ] \
-   || ! openssl x509 -in "${HTTPS_DIR}/leaf.crt" -noout -ext subjectAltName 2>/dev/null | grep -qF 'DNS:*.devnet.internal'; then
+if leaf_lacks; then
   "${SCRIPT_DIR}/https-ca.sh"
   NEW_CERTS=1
 fi