fix: healthcheck every container and make the host cron check catch real outages

.claude/memory/MEMORY.md

@@ -0,0 +1,2 @@
+- [Rabbit dead while container Up](rabbit-dead-while-container-up.md) — corrupt queue index after an unclean reboot; symptoms, the one-directory fix, the clean-up, and what monitors it now
+- [Aggregator /health is broken](aggregator-health-endpoint-broken.md) — HealthModule lacks HttpModule, so compose checks /metrics for now

.claude/memory/aggregator-health-endpoint-broken.md

@@ -0,0 +1,17 @@
+---
+name: aggregator-health-endpoint-broken
+description: The aggregator's /health returns 500 because its HealthModule never imports HttpModule; compose checks /metrics instead until that is fixed
+metadata:
+  type: project
+  volatility: normal
+  lastVerified: 2026-09-15
+---
+
+The aggregator exposes `/health` via `@nestjs/terminus` with an API ping, a Redis ping and a
+RabbitMQ ping, which would be the ideal Docker healthcheck. It returns 500 in production:
+`It seems like "HttpService" is not available in the current context` because the
+HealthModule does not import `HttpModule`. Until that is fixed in the aggregator repo and a
+new image is published, the compose healthcheck hits `/metrics`, which only proves the
+process answers. Switch it back to `/health` once the fix ships.
+
+Related: [[rabbit-dead-while-container-up]].

.claude/memory/rabbit-dead-while-container-up.md

@@ -0,0 +1,41 @@
+---
+name: rabbit-dead-while-container-up
+description: RabbitMQ can crash during queue-index recovery and leave the container "Up" with no broker; how it presents, the fix, and what now catches it
+metadata:
+  type: project
+  volatility: durable
+  lastVerified: 2026-09-15
+---
+
+After an unclean host reboot RabbitMQ's on-disk queue index can come back zero-filled. The
+broker app then crashes during recovery (`rabbit_queue_index:parse_pub_record_body` throws
+`badarg`) but the Erlang VM stays up, so `docker ps` shows `rabbit` as "Up" indefinitely.
+Happened 2026-09-13; alerts stopped for ~45 hours before anyone noticed.
+
+**How it presents:** collectors log `[RabbitMQ] Connection failed` every 5s while still
+reporting "Connected to Census"; the aggregators crash-loop on `MQ Publisher queue did not
+connect in time` and burn ~60% CPU each because every restart is a fresh `nest start`
+compile (thousands of restarts). `docker exec rabbit rabbitmqctl status` says the `rabbit`
+app is not running. The broker log names the corrupt queue directory.
+
+**Fix:** `docker stop rabbit`, delete just that queue's directory under
+`volumes/rabbitmq/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/<vhost-hash>/queues/`
+(each directory's `.queue_name` file says which queue it is), `docker start rabbit`. The
+apps re-assert their queues on connect. Only the pending messages in that one queue are lost;
+the README's "wipe the whole mq volume" remedy is the blunt version of the same thing.
+
+**Aftermath to clean:** alerts that started during the outage sit in `state: 0` ("failed to
+start recording"), and the queued MetagameEvent backlog replays when the broker returns,
+"ending" alerts with bogus 0/100/0 or 7/5/5 territory. Delete those
+`instance_metagame_territories` docs and the matching `instance` rows in
+`instance_facility_controls` and `aggregate_instance_*`. Per-alert queues for instances the
+aggregator never tracked are left orphaned in Rabbit and need `rabbitmqctl delete_queue`.
+
+**What catches it now:** every compose service has a healthcheck (rabbit's is
+`rabbitmq-diagnostics check_running`, which fails in exactly this state), and
+`docker/production/healthcheck.sh` runs the diagnostics itself every minute, requires a
+consumer on the API and per-world MetagameEvent queues, and fails if no alert has recorded
+for 12 hours. Verified by stopping the broker app in place: the cron check went red within a
+minute, Docker's health state ~90s later.
+
+Related: [[aggregator-health-endpoint-broken]].

.claude/settings.json

@@ -0,0 +1 @@
+{ "autoMemoryDirectory": "~/code/ps2alerts/stack/.claude/memory" }

.gitignore

@@ -8,3 +8,4 @@ archive/terraform/.auto.tfvars
 archive/terraform/kube-config
 archive/terraform/tfplan.out
 .DS_Store
+.claude/settings.local.json

README.md

@@ -159,4 +159,9 @@ We use grafana to monitor metrics from within the applications.
 
 #### Deployment
 
-Deployment is currently manually done by Maelstromeous. Upon merge into `master`, run `./sync-to-ceres.sh`
+Deployment is manual. After merging into `master`, run `./deploy.sh` with an ssh alias named
+`ps2alerts` pointing at the production host. It copies the compose file, the Grafana and
+Prometheus config and the host healthcheck to the box, splices the host's Erlang cookie into the
+compose file, validates it with `docker compose config`, keeps `.bak` copies of what it replaces,
+then runs `docker compose up -d`. Secrets (`*.env`, the cookie, `/root/healthcheck.env`) live only
+on the host and are never overwritten by the deploy.

deploy.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+# Deploys the production compose, monitoring config and host healthcheck to the ps2alerts box.
+# Secrets never live in this repo: env files and the Erlang cookie stay on the host.
+set -euo pipefail
+HOST=ps2alerts
+REMOTE=/root/docker
+
+echo "Syncing files to $HOST"
+scp -r monitoring/grafana/ $HOST:$REMOTE/files
+scp -r monitoring/prometheus/ $HOST:$REMOTE/files/prometheus
+scp docker/production/docker-compose.yml $HOST:$REMOTE/docker-compose.yml.new
+scp docker/production/healthcheck.sh $HOST:/root/healthcheck.sh.new
+
+ssh $HOST bash -s <<'REMOTE'
+set -euo pipefail
+cd /root/docker
+cookie=$(grep -oE 'RABBITMQ_ERL_COOKIE: "[^"]+"' docker-compose.yml | head -1)
+sed -i "s|RABBITMQ_ERL_COOKIE: \"REDACTED\"|$cookie|" docker-compose.yml.new
+docker compose -f docker-compose.yml.new config -q
+cp docker-compose.yml docker-compose.yml.bak
+mv docker-compose.yml.new docker-compose.yml
+bash -n /root/healthcheck.sh.new
+cp /root/healthcheck.sh /root/healthcheck.sh.bak
+mv /root/healthcheck.sh.new /root/healthcheck.sh
+chmod 700 /root/healthcheck.sh
+docker compose up -d --remove-orphans
+REMOTE
+echo "Done."

docker/aggregator.env

@@ -2,14 +2,14 @@ NODE_ENV=production
 VERSION=12345
 CENSUS_ENVIRONMENT=ps2
 CENSUS_SERVICE_ID=REDACTED
-RABBITMQ_HOST=ps2alerts-rabbitmq
+RABBITMQ_HOST=rabbit
 RABBITMQ_USER=REDACTED
 RABBITMQ_PASS="REDACTED"
 RABBITMQ_API_QUEUE="api-queue-production"
 RABBITMQ_API_QUEUE_DELAY="api-queue-production-delay"
-REDIS_HOST=ps2alerts-redis
+REDIS_HOST=redis
 REDIS_DB=0
-INTERNAL_API_HOST="http://ps2alerts-api-rest-production:3000"
+INTERNAL_API_HOST="http://api-rest:3000"
 INTERNAL_API_USER=REDACTED
 INTERNAL_API_PASS="REDACTED"
 LOGGER_TRANSPORTS=console

docker/api.env

@@ -1,10 +1,10 @@
 NODE_ENV=production
 DB_USER=REDACTED
 DB_PASS="REDACTED"
-DB_HOST="ps2alerts-db"
+DB_HOST="db"
 DB_POOL_SIZE=10
 DB_AUTH_SOURCE=REDACTED
-REDIS_HOST="ps2alerts-redis"
+REDIS_HOST="redis"
 REDIS_DB="10"
 AGGREGATOR_ENABLED=false
 CRON_ENABLED=false

docker/collector.env

@@ -1,5 +1,5 @@
 SERVICE_ID=REDACTED
 STREAM_ENDPOINT="wss://push.nanite-systems.net/streaming"
-RABBITMQ_URL="amqp://REDACTED:REDACTED@ps2alerts-mq:5672"
+RABBITMQ_URL="amqp://REDACTED:REDACTED@rabbit:5672"
 RABBITMQ_COLLECTOR_EXCHANGE_NAME="ps2alerts-census"
 SUBSCRIBE_EVENTS="Death,FacilityControl,GainExperience,MetagameEvent,VehicleDestroy"

docker/production/docker-compose.yml

@@ -3,6 +3,12 @@ services:
     container_name: website
     image: maelstromeous/ps2alerts:website-production-latest
     restart: unless-stopped
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS -o /dev/null http://127.0.0.1/"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
     ports:
       - "80:80"
   api-aggregator:
@@ -17,6 +23,12 @@ services:
       - rabbit.env
       - redis.env
     restart: unless-stopped
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS -u \"$$RABBITMQ_USER:$$RABBITMQ_PASS\" http://rabbit:15672/api/queues/%2F/api-queue-production | grep -qE '\"consumers\":[1-9]'"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 120s
     environment:
       AGGREGATOR_ENABLED: "true"
       RABBITMQ_QUEUE: api-queue-production
@@ -27,6 +39,12 @@ services:
     container_name: api-cron
     image: maelstromeous/ps2alerts:api-production-latest
     restart: unless-stopped
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS -o /dev/null http://127.0.0.1:3000/healthcheck"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
     depends_on:
       - db
       - redis
@@ -41,6 +59,12 @@ services:
     container_name: api-rest
     image: maelstromeous/ps2alerts:api-production-latest
     restart: unless-stopped
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS -o /dev/null http://127.0.0.1:3000/healthcheck"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
     depends_on:
       - db
       - redis
@@ -57,6 +81,12 @@ services:
     container_name: assets
     image: maelstromeous/ps2alerts:assets-latest
     restart: unless-stopped
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS -o /dev/null http://127.0.0.1/"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
     env_file:
       - collector.env
     ports:
@@ -65,6 +95,12 @@ services:
     container_name: collector-pc
     image: ghcr.io/nanite-systems/stream-collector:1.3.2
     restart: unless-stopped
+    healthcheck:
+      test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:3000/health >/dev/null"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
     depends_on:
       - rabbit
     env_file:
@@ -77,6 +113,12 @@ services:
     container_name: collector-ps4eu
     image: ghcr.io/nanite-systems/stream-collector:1.3.2
     restart: unless-stopped
+    healthcheck:
+      test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:3000/health >/dev/null"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
     depends_on:
       - rabbit
     env_file:
@@ -89,6 +131,12 @@ services:
     container_name: collector-ps4us
     image: ghcr.io/nanite-systems/stream-collector:1.3.2
     restart: unless-stopped
+    healthcheck:
+      test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:3000/health >/dev/null"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
     depends_on:
       - rabbit
     env_file:
@@ -101,6 +149,12 @@ services:
     container_name: aggregator-pc
     image: maelstromeous/ps2alerts:aggregator-production-latest
     restart: unless-stopped
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS -o /dev/null http://127.0.0.1:$${APP_PORT:-1337}/metrics"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 120s
     depends_on:
       - rabbit
       - redis
@@ -117,14 +171,20 @@ services:
     container_name: aggregator-ps4eu
     image: maelstromeous/ps2alerts:aggregator-production-latest
     restart: unless-stopped
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS -o /dev/null http://127.0.0.1:$${APP_PORT:-1337}/metrics"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 120s
     depends_on:
       - rabbit
       - redis
     env_file:
       - aggregator.env
       - redis.env
     ports:
-      - "1338:1337"
+      - "1338:1338"
     environment:
       APP_PORT: 1338
       CENSUS_ENVIRONMENT: ps2ps4eu
@@ -134,23 +194,35 @@ services:
     container_name: aggregator-ps4us
     image: maelstromeous/ps2alerts:aggregator-production-latest
     restart: unless-stopped
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS -o /dev/null http://127.0.0.1:$${APP_PORT:-1337}/metrics"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 120s
     depends_on:
       - rabbit
       - redis
     env_file:
       - aggregator.env
       - redis.env
     ports:
-      - "1339:1337"
+      - "1339:1339"
     environment:
       APP_PORT: 1339
       CENSUS_ENVIRONMENT: ps2ps4us
     networks:
       - ps2alerts
   db:
     container_name: db
-    image: bitnami/mongodb:5.0.14-debian-11-r13
+    image: bitnamilegacy/mongodb:5.0.14-debian-11-r31
     restart: unless-stopped
+    healthcheck:
+      test: ["CMD-SHELL", "mongosh --quiet --eval 'db.adminCommand({ping:1}).ok' | grep -q 1"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 120s
     env_file:
       - db.env
     ports:
@@ -161,8 +233,14 @@ services:
       - ps2alerts
   rabbit:
     container_name: rabbit
-    image: bitnami/rabbitmq:3.10.13
+    image: bitnamilegacy/rabbitmq:3.10.13
     restart: unless-stopped
+    healthcheck:
+      test: ["CMD-SHELL", "rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_local_alarms"]
+      interval: 30s
+      timeout: 20s
+      retries: 3
+      start_period: 120s
     env_file:
       - rabbit.env
     environment:
@@ -178,6 +256,12 @@ services:
     container_name: redis
     image: bitnamilegacy/redis:6.2
     restart: unless-stopped
+    healthcheck:
+      test: ["CMD-SHELL", "redis-cli -a \"$$REDIS_PASSWORD\" --no-auth-warning ping | grep -q PONG"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
     env_file:
       - redis.env
     ports:
@@ -186,23 +270,33 @@ services:
       - ./volumes/redis:/bitnami/redis/data
     networks:
       - ps2alerts
-  ps2alerts-prometheus:
-    container_name: ps2alerts-prom
+  prometheus:
+    container_name: prometheus
     image: prom/prometheus
     restart: unless-stopped
-    user: "1001"
+    healthcheck:
+      test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:9090/-/healthy >/dev/null"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
     ports:
       - "9090:9090"
     volumes:
       - ./files/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
       - ./volumes/prometheus:/prometheus
     networks:
       - ps2alerts
-  ps2alerts-grafana:
-    container_name: ps2alerts-grafana
+  grafana:
+    container_name: grafana
     image: grafana/grafana
     restart: unless-stopped
-    user: "1001"
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS -o /dev/null http://127.0.0.1:3000/api/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
     ports:
       - "9091:3000"
     volumes:

docker/production/healthcheck.env.example

@@ -0,0 +1,4 @@
+# Copied to /root/healthcheck.env on the host, mode 600. Never committed with real values.
+PING_URL=https://hc-ping.com/<uuid>
+PUSHOVER_API_TOKEN=
+PUSHOVER_USER_KEY=