diff --git a/.claude/memory/MEMORY.md b/.claude/memory/MEMORY.md new file mode 100644 index 0000000..f0ef791 --- /dev/null +++ b/.claude/memory/MEMORY.md @@ -0,0 +1,2 @@ +- [Rabbit dead while container Up](rabbit-dead-while-container-up.md) — corrupt queue index after an unclean reboot; symptoms, the one-directory fix, the clean-up, and what monitors it now +- [Aggregator /health is broken](aggregator-health-endpoint-broken.md) — HealthModule lacks HttpModule, so compose checks /metrics for now diff --git a/.claude/memory/aggregator-health-endpoint-broken.md b/.claude/memory/aggregator-health-endpoint-broken.md new file mode 100644 index 0000000..cc275d1 --- /dev/null +++ b/.claude/memory/aggregator-health-endpoint-broken.md @@ -0,0 +1,17 @@ +--- +name: aggregator-health-endpoint-broken +description: The aggregator's /health returns 500 because its HealthModule never imports HttpModule; compose checks /metrics instead until that is fixed +metadata: + type: project + volatility: normal + lastVerified: 2026-09-15 +--- + +The aggregator exposes `/health` via `@nestjs/terminus` with an API ping, a Redis ping and a +RabbitMQ ping, which would be the ideal Docker healthcheck. It returns 500 in production: +`It seems like "HttpService" is not available in the current context` because the +HealthModule does not import `HttpModule`. Until that is fixed in the aggregator repo and a +new image is published, the compose healthcheck hits `/metrics`, which only proves the +process answers. Switch it back to `/health` once the fix ships. + +Related: [[rabbit-dead-while-container-up]]. diff --git a/.claude/memory/rabbit-dead-while-container-up.md b/.claude/memory/rabbit-dead-while-container-up.md new file mode 100644 index 0000000..117fea6 --- /dev/null +++ b/.claude/memory/rabbit-dead-while-container-up.md @@ -0,0 +1,41 @@ +--- +name: rabbit-dead-while-container-up +description: RabbitMQ can crash during queue-index recovery and leave the container "Up" with no broker; how it presents, the fix, and what now catches it +metadata: + type: project + volatility: durable + lastVerified: 2026-09-15 +--- + +After an unclean host reboot RabbitMQ's on-disk queue index can come back zero-filled. The +broker app then crashes during recovery (`rabbit_queue_index:parse_pub_record_body` throws +`badarg`) but the Erlang VM stays up, so `docker ps` shows `rabbit` as "Up" indefinitely. +Happened 2026-09-13; alerts stopped for ~45 hours before anyone noticed. + +**How it presents:** collectors log `[RabbitMQ] Connection failed` every 5s while still +reporting "Connected to Census"; the aggregators crash-loop on `MQ Publisher queue did not +connect in time` and burn ~60% CPU each because every restart is a fresh `nest start` +compile (thousands of restarts). `docker exec rabbit rabbitmqctl status` says the `rabbit` +app is not running. The broker log names the corrupt queue directory. + +**Fix:** `docker stop rabbit`, delete just that queue's directory under +`volumes/rabbitmq/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts//queues/` +(each directory's `.queue_name` file says which queue it is), `docker start rabbit`. The +apps re-assert their queues on connect. Only the pending messages in that one queue are lost; +the README's "wipe the whole mq volume" remedy is the blunt version of the same thing. + +**Aftermath to clean:** alerts that started during the outage sit in `state: 0` ("failed to +start recording"), and the queued MetagameEvent backlog replays when the broker returns, +"ending" alerts with bogus 0/100/0 or 7/5/5 territory. Delete those +`instance_metagame_territories` docs and the matching `instance` rows in +`instance_facility_controls` and `aggregate_instance_*`. Per-alert queues for instances the +aggregator never tracked are left orphaned in Rabbit and need `rabbitmqctl delete_queue`. + +**What catches it now:** every compose service has a healthcheck (rabbit's is +`rabbitmq-diagnostics check_running`, which fails in exactly this state), and +`docker/production/healthcheck.sh` runs the diagnostics itself every minute, requires a +consumer on the API and per-world MetagameEvent queues, and fails if no alert has recorded +for 12 hours. Verified by stopping the broker app in place: the cron check went red within a +minute, Docker's health state ~90s later. + +Related: [[aggregator-health-endpoint-broken]]. diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..078e799 --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1 @@ +{ "autoMemoryDirectory": "~/code/ps2alerts/stack/.claude/memory" } diff --git a/.gitignore b/.gitignore index eed6d7f..0b0e045 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,4 @@ archive/terraform/.auto.tfvars archive/terraform/kube-config archive/terraform/tfplan.out .DS_Store +.claude/settings.local.json diff --git a/README.md b/README.md index 0e95ae6..80e9986 100644 --- a/README.md +++ b/README.md @@ -159,4 +159,9 @@ We use grafana to monitor metrics from within the applications. #### Deployment -Deployment is currently manually done by Maelstromeous. Upon merge into `master`, run `./sync-to-ceres.sh` +Deployment is manual. After merging into `master`, run `./deploy.sh` with an ssh alias named +`ps2alerts` pointing at the production host. It copies the compose file, the Grafana and +Prometheus config and the host healthcheck to the box, splices the host's Erlang cookie into the +compose file, validates it with `docker compose config`, keeps `.bak` copies of what it replaces, +then runs `docker compose up -d`. Secrets (`*.env`, the cookie, `/root/healthcheck.env`) live only +on the host and are never overwritten by the deploy. diff --git a/deploy.sh b/deploy.sh new file mode 100755 index 0000000..1c7322e --- /dev/null +++ b/deploy.sh @@ -0,0 +1,28 @@ +#!/bin/bash +# Deploys the production compose, monitoring config and host healthcheck to the ps2alerts box. +# Secrets never live in this repo: env files and the Erlang cookie stay on the host. +set -euo pipefail +HOST=ps2alerts +REMOTE=/root/docker + +echo "Syncing files to $HOST" +scp -r monitoring/grafana/ $HOST:$REMOTE/files +scp -r monitoring/prometheus/ $HOST:$REMOTE/files/prometheus +scp docker/production/docker-compose.yml $HOST:$REMOTE/docker-compose.yml.new +scp docker/production/healthcheck.sh $HOST:/root/healthcheck.sh.new + +ssh $HOST bash -s <<'REMOTE' +set -euo pipefail +cd /root/docker +cookie=$(grep -oE 'RABBITMQ_ERL_COOKIE: "[^"]+"' docker-compose.yml | head -1) +sed -i "s|RABBITMQ_ERL_COOKIE: \"REDACTED\"|$cookie|" docker-compose.yml.new +docker compose -f docker-compose.yml.new config -q +cp docker-compose.yml docker-compose.yml.bak +mv docker-compose.yml.new docker-compose.yml +bash -n /root/healthcheck.sh.new +cp /root/healthcheck.sh /root/healthcheck.sh.bak +mv /root/healthcheck.sh.new /root/healthcheck.sh +chmod 700 /root/healthcheck.sh +docker compose up -d --remove-orphans +REMOTE +echo "Done." diff --git a/docker/aggregator.env b/docker/aggregator.env index d947e45..440f6fc 100644 --- a/docker/aggregator.env +++ b/docker/aggregator.env @@ -2,14 +2,14 @@ NODE_ENV=production VERSION=12345 CENSUS_ENVIRONMENT=ps2 CENSUS_SERVICE_ID=REDACTED -RABBITMQ_HOST=ps2alerts-rabbitmq +RABBITMQ_HOST=rabbit RABBITMQ_USER=REDACTED RABBITMQ_PASS="REDACTED" RABBITMQ_API_QUEUE="api-queue-production" RABBITMQ_API_QUEUE_DELAY="api-queue-production-delay" -REDIS_HOST=ps2alerts-redis +REDIS_HOST=redis REDIS_DB=0 -INTERNAL_API_HOST="http://ps2alerts-api-rest-production:3000" +INTERNAL_API_HOST="http://api-rest:3000" INTERNAL_API_USER=REDACTED INTERNAL_API_PASS="REDACTED" LOGGER_TRANSPORTS=console diff --git a/docker/api.env b/docker/api.env index 26fe00f..c07a94f 100644 --- a/docker/api.env +++ b/docker/api.env @@ -1,10 +1,10 @@ NODE_ENV=production DB_USER=REDACTED DB_PASS="REDACTED" -DB_HOST="ps2alerts-db" +DB_HOST="db" DB_POOL_SIZE=10 DB_AUTH_SOURCE=REDACTED -REDIS_HOST="ps2alerts-redis" +REDIS_HOST="redis" REDIS_DB="10" AGGREGATOR_ENABLED=false CRON_ENABLED=false diff --git a/docker/collector.env b/docker/collector.env index ebdb5dd..aa84cb2 100644 --- a/docker/collector.env +++ b/docker/collector.env @@ -1,5 +1,5 @@ SERVICE_ID=REDACTED STREAM_ENDPOINT="wss://push.nanite-systems.net/streaming" -RABBITMQ_URL="amqp://REDACTED:REDACTED@ps2alerts-mq:5672" +RABBITMQ_URL="amqp://REDACTED:REDACTED@rabbit:5672" RABBITMQ_COLLECTOR_EXCHANGE_NAME="ps2alerts-census" SUBSCRIBE_EVENTS="Death,FacilityControl,GainExperience,MetagameEvent,VehicleDestroy" diff --git a/docker/production/docker-compose.yml b/docker/production/docker-compose.yml index 405a22d..759e593 100644 --- a/docker/production/docker-compose.yml +++ b/docker/production/docker-compose.yml @@ -3,6 +3,12 @@ services: container_name: website image: maelstromeous/ps2alerts:website-production-latest restart: unless-stopped + healthcheck: + test: ["CMD-SHELL", "curl -fsS -o /dev/null http://127.0.0.1/"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s ports: - "80:80" api-aggregator: @@ -17,6 +23,12 @@ services: - rabbit.env - redis.env restart: unless-stopped + healthcheck: + test: ["CMD-SHELL", "curl -fsS -u \"$$RABBITMQ_USER:$$RABBITMQ_PASS\" http://rabbit:15672/api/queues/%2F/api-queue-production | grep -qE '\"consumers\":[1-9]'"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 120s environment: AGGREGATOR_ENABLED: "true" RABBITMQ_QUEUE: api-queue-production @@ -27,6 +39,12 @@ services: container_name: api-cron image: maelstromeous/ps2alerts:api-production-latest restart: unless-stopped + healthcheck: + test: ["CMD-SHELL", "curl -fsS -o /dev/null http://127.0.0.1:3000/healthcheck"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s depends_on: - db - redis @@ -41,6 +59,12 @@ services: container_name: api-rest image: maelstromeous/ps2alerts:api-production-latest restart: unless-stopped + healthcheck: + test: ["CMD-SHELL", "curl -fsS -o /dev/null http://127.0.0.1:3000/healthcheck"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s depends_on: - db - redis @@ -57,6 +81,12 @@ services: container_name: assets image: maelstromeous/ps2alerts:assets-latest restart: unless-stopped + healthcheck: + test: ["CMD-SHELL", "curl -fsS -o /dev/null http://127.0.0.1/"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s env_file: - collector.env ports: @@ -65,6 +95,12 @@ services: container_name: collector-pc image: ghcr.io/nanite-systems/stream-collector:1.3.2 restart: unless-stopped + healthcheck: + test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:3000/health >/dev/null"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s depends_on: - rabbit env_file: @@ -77,6 +113,12 @@ services: container_name: collector-ps4eu image: ghcr.io/nanite-systems/stream-collector:1.3.2 restart: unless-stopped + healthcheck: + test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:3000/health >/dev/null"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s depends_on: - rabbit env_file: @@ -89,6 +131,12 @@ services: container_name: collector-ps4us image: ghcr.io/nanite-systems/stream-collector:1.3.2 restart: unless-stopped + healthcheck: + test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:3000/health >/dev/null"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s depends_on: - rabbit env_file: @@ -101,6 +149,12 @@ services: container_name: aggregator-pc image: maelstromeous/ps2alerts:aggregator-production-latest restart: unless-stopped + healthcheck: + test: ["CMD-SHELL", "curl -fsS -o /dev/null http://127.0.0.1:$${APP_PORT:-1337}/metrics"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 120s depends_on: - rabbit - redis @@ -117,6 +171,12 @@ services: container_name: aggregator-ps4eu image: maelstromeous/ps2alerts:aggregator-production-latest restart: unless-stopped + healthcheck: + test: ["CMD-SHELL", "curl -fsS -o /dev/null http://127.0.0.1:$${APP_PORT:-1337}/metrics"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 120s depends_on: - rabbit - redis @@ -124,7 +184,7 @@ services: - aggregator.env - redis.env ports: - - "1338:1337" + - "1338:1338" environment: APP_PORT: 1338 CENSUS_ENVIRONMENT: ps2ps4eu @@ -134,6 +194,12 @@ services: container_name: aggregator-ps4us image: maelstromeous/ps2alerts:aggregator-production-latest restart: unless-stopped + healthcheck: + test: ["CMD-SHELL", "curl -fsS -o /dev/null http://127.0.0.1:$${APP_PORT:-1337}/metrics"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 120s depends_on: - rabbit - redis @@ -141,7 +207,7 @@ services: - aggregator.env - redis.env ports: - - "1339:1337" + - "1339:1339" environment: APP_PORT: 1339 CENSUS_ENVIRONMENT: ps2ps4us @@ -149,8 +215,14 @@ services: - ps2alerts db: container_name: db - image: bitnami/mongodb:5.0.14-debian-11-r13 + image: bitnamilegacy/mongodb:5.0.14-debian-11-r31 restart: unless-stopped + healthcheck: + test: ["CMD-SHELL", "mongosh --quiet --eval 'db.adminCommand({ping:1}).ok' | grep -q 1"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 120s env_file: - db.env ports: @@ -161,8 +233,14 @@ services: - ps2alerts rabbit: container_name: rabbit - image: bitnami/rabbitmq:3.10.13 + image: bitnamilegacy/rabbitmq:3.10.13 restart: unless-stopped + healthcheck: + test: ["CMD-SHELL", "rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_local_alarms"] + interval: 30s + timeout: 20s + retries: 3 + start_period: 120s env_file: - rabbit.env environment: @@ -178,6 +256,12 @@ services: container_name: redis image: bitnamilegacy/redis:6.2 restart: unless-stopped + healthcheck: + test: ["CMD-SHELL", "redis-cli -a \"$$REDIS_PASSWORD\" --no-auth-warning ping | grep -q PONG"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s env_file: - redis.env ports: @@ -186,11 +270,16 @@ services: - ./volumes/redis:/bitnami/redis/data networks: - ps2alerts - ps2alerts-prometheus: - container_name: ps2alerts-prom + prometheus: + container_name: prometheus image: prom/prometheus restart: unless-stopped - user: "1001" + healthcheck: + test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:9090/-/healthy >/dev/null"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s ports: - "9090:9090" volumes: @@ -198,11 +287,16 @@ services: - ./volumes/prometheus:/prometheus networks: - ps2alerts - ps2alerts-grafana: - container_name: ps2alerts-grafana + grafana: + container_name: grafana image: grafana/grafana restart: unless-stopped - user: "1001" + healthcheck: + test: ["CMD-SHELL", "curl -fsS -o /dev/null http://127.0.0.1:3000/api/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s ports: - "9091:3000" volumes: diff --git a/docker/production/healthcheck.env.example b/docker/production/healthcheck.env.example new file mode 100644 index 0000000..92f2365 --- /dev/null +++ b/docker/production/healthcheck.env.example @@ -0,0 +1,4 @@ +# Copied to /root/healthcheck.env on the host, mode 600. Never committed with real values. +PING_URL=https://hc-ping.com/ +PUSHOVER_API_TOKEN= +PUSHOVER_USER_KEY= diff --git a/docker/production/healthcheck.sh b/docker/production/healthcheck.sh new file mode 100755 index 0000000..5402977 --- /dev/null +++ b/docker/production/healthcheck.sh @@ -0,0 +1,211 @@ +#!/usr/bin/env bash +# Host-level healthcheck, run from root's crontab every minute. Pings healthchecks.io on +# success, posts the failure list to /fail otherwise. Deployed to /root/healthcheck.sh. +set -o errexit +set -o nounset +set -o pipefail + +# = CONFIGURATION =============================== +ENV_FILE=/root/healthcheck.env # PING_URL, PUSHOVER_API_TOKEN, PUSHOVER_USER_KEY +LOCK_FILE=/var/lock/healthcheck.lock +STATE_FILE=/var/tmp/healthcheck.restarts +MIN_FREE_GB=10 +MIN_FREE_BOOT_MB=50 +EXCLUDE_REGEX='^/(sys|run|dev|proc)' +DOCKER_TIMEOUT=30 +MAX_ALERT_GAP_HOURS=12 # PS2 never goes this long without an alert on some world +ALERT_GAP_NOTIFY_INTERVAL=21600 # seconds between repeat Pushover pings for the same gap +ALERT_GAP_STAMP=/var/tmp/healthcheck.alert-gap-notified +CONTAINERS=( + website assets + api-rest api-cron api-aggregator + collector-pc collector-ps4eu collector-ps4us + aggregator-pc aggregator-ps4eu aggregator-ps4us + db rabbit redis prometheus grafana +) +# Queues that must have a consumer whenever the stack is healthy. +CONSUMED_QUEUES=( + api-queue-production + aggregator-1-MetagameEvent aggregator-10-MetagameEvent aggregator-13-MetagameEvent + aggregator-17-MetagameEvent aggregator-40-MetagameEvent + aggregator-1000-MetagameEvent aggregator-2000-MetagameEvent +) +REDIS_CONTAINER=redis +RECOVERY_SCRIPT=/root/bonk-redis.sh +RECOVERY_TIMEOUT=90 +RECOVERY_INTERVAL=5 + +# shellcheck disable=SC1090 +source "$ENV_FILE" +FAIL_ENDPOINT="${PING_URL}/fail" +FAILURES=() + +# = HELPERS ===================================== +fail_now() { + local message="$1" + echo "[FAIL] ${message}" >&2 + curl -fsS --retry 2 --retry-delay 1 -d "$message" "$FAIL_ENDPOINT" >/dev/null || true + exit 1 +} + +add_failure() { + echo "[FAIL] $1" >&2 + FAILURES+=("$1") +} + +dk() { timeout "$DOCKER_TIMEOUT" docker "$@"; } + +notify_pushover() { + local title="$1" message="$2" + curl -fsS \ + --form-string "token=${PUSHOVER_API_TOKEN}" \ + --form-string "user=${PUSHOVER_USER_KEY}" \ + --form-string "title=${title}" \ + --form-string "message=${message}" \ + --form-string "priority=0" \ + https://api.pushover.net/1/messages.json >/dev/null || true +} + +container_status() { dk inspect --format '{{.State.Status}}' "$1" 2>/dev/null || echo missing; } +container_health() { + dk inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$1" 2>/dev/null || echo missing +} +container_restarts() { dk inspect --format '{{.RestartCount}}' "$1" 2>/dev/null || echo 0; } + +attempt_redis_recovery() { + local reason="$1" + echo "[RECOVERY] Redis issue detected: ${reason}. Running recovery script..." + if ! bash "$RECOVERY_SCRIPT"; then + fail_now "Recovery script '${RECOVERY_SCRIPT}' failed (original issue: ${reason})" + fi + local waited=0 + while (( waited < RECOVERY_TIMEOUT )); do + sleep "$RECOVERY_INTERVAL" + waited=$(( waited + RECOVERY_INTERVAL )) + local status health + status=$(container_status "$REDIS_CONTAINER") + health=$(container_health "$REDIS_CONTAINER") + if [[ "$status" == "running" && ( "$health" == "healthy" || "$health" == "none" ) ]]; then + echo "[RECOVERY] Redis recovered after ${waited}s." + notify_pushover "Redis Auto-Recovery" \ + "Redis recovered automatically after ${waited}s.\nOriginal issue: ${reason}\nHost: $(hostname)" + return 0 + fi + done + fail_now "Redis did not recover within ${RECOVERY_TIMEOUT}s (original issue: ${reason})" +} + +# = LOCK ======================================== +# A redis recovery can outlast the cron interval; never let two runs overlap. +exec 9>"$LOCK_FILE" +if ! flock -n 9; then + echo "another healthcheck run holds the lock, skipping" >&2 + exit 0 +fi + +# = DOCKER DAEMON =============================== +dk info >/dev/null 2>&1 || fail_now "Docker daemon is unreachable" + +# = REDIS RECOVERY ============================== +# Runs first so the generic container check below sees the recovered state. +redis_status=$(container_status "$REDIS_CONTAINER") +redis_health=$(container_health "$REDIS_CONTAINER") +if [[ "$redis_status" == "restarting" ]]; then + attempt_redis_recovery "container in restart loop" +elif [[ "$redis_status" == "running" && "$redis_health" == "unhealthy" ]]; then + attempt_redis_recovery "container unhealthy" +fi + +# = CONTAINERS ================================== +declare -A previous_restarts=() +if [[ -f "$STATE_FILE" ]]; then + while IFS='=' read -r name count; do + [[ -n "$name" ]] && previous_restarts["$name"]="$count" + done < "$STATE_FILE" +fi +: > "${STATE_FILE}.tmp" + +for c in "${CONTAINERS[@]}"; do + status=$(container_status "$c") + health=$(container_health "$c") + restarts=$(container_restarts "$c") + echo "${c}=${restarts}" >> "${STATE_FILE}.tmp" + + if [[ "$status" != "running" ]]; then + add_failure "${c}: not running (status: ${status})" + continue + fi + if [[ "$health" == "unhealthy" ]]; then + add_failure "${c}: unhealthy" + fi + prev=${previous_restarts[$c]:-$restarts} + if (( restarts - prev >= 2 )); then + add_failure "${c}: crash-looping (${prev} -> ${restarts} restarts in the last minute)" + fi +done +mv "${STATE_FILE}.tmp" "$STATE_FILE" + +# = RABBITMQ ==================================== +# "Up" is not enough: the broker app can crash during recovery and leave the container running. +if [[ $(container_status rabbit) == "running" ]]; then + for check in check_running check_local_alarms check_port_connectivity; do + if ! dk exec rabbit rabbitmq-diagnostics -q "$check" >/dev/null 2>&1; then + add_failure "rabbit: rabbitmq-diagnostics ${check} failed" + fi + done + if queue_list=$(dk exec rabbit rabbitmqctl -q list_queues name consumers 2>/dev/null); then + for q in "${CONSUMED_QUEUES[@]}"; do + consumers=$(awk -v q="$q" '$1 == q {print $2; exit}' <<<"$queue_list") + if [[ -z "$consumers" ]]; then + add_failure "rabbit: queue ${q} does not exist" + elif ! [[ "$consumers" =~ ^[0-9]+$ ]] || (( consumers < 1 )); then + add_failure "rabbit: queue ${q} has no consumer" + fi + done + else + add_failure "rabbit: rabbitmqctl list_queues failed" + fi +fi + +# = LAST RECORDED ALERT ========================= +# state 0 is an alert that failed to record, so it does not count as evidence the pipeline works. +last_alert=$(dk exec db sh -c 'mongosh --quiet -u "$MONGODB_USERNAME" -p "$MONGODB_PASSWORD" \ + --authenticationDatabase "$MONGODB_DATABASE" "$MONGODB_DATABASE" --eval \ + "const d = db.instance_metagame_territories.find({state:{\$ne:0}},{timeStarted:1}).sort({timeStarted:-1}).limit(1).next(); print(d ? d.timeStarted.getTime() : 0)"' 2>/dev/null || echo error) +if [[ "$last_alert" =~ ^[0-9]+$ ]]; then + gap_hours=$(( ( $(date +%s) - last_alert / 1000 ) / 3600 )) + if (( gap_hours >= MAX_ALERT_GAP_HOURS )); then + add_failure "no alert recorded for ${gap_hours}h (last: $(date -u -d @$(( last_alert / 1000 )) +%FT%TZ))" + last_notified=$(cat "$ALERT_GAP_STAMP" 2>/dev/null || echo 0) + if (( $(date +%s) - last_notified >= ALERT_GAP_NOTIFY_INTERVAL )); then + notify_pushover "PS2Alerts: no alerts for ${gap_hours}h" \ + "Last recorded alert started $(date -u -d @$(( last_alert / 1000 )) +%FT%TZ). Collectors, rabbit or the aggregators are probably wedged.\nHost: $(hostname)" + date +%s > "$ALERT_GAP_STAMP" + fi + else + rm -f "$ALERT_GAP_STAMP" + fi +else + add_failure "db: could not read last alert time (${last_alert})" +fi + +# = PUBLISHED PORTS ============================= +curl -fsS -m 10 -o /dev/null http://127.0.0.1:81/healthcheck || add_failure "api-rest: /healthcheck on :81 failed" +curl -fsS -m 10 -o /dev/null http://127.0.0.1:80/ || add_failure "website: / on :80 failed" + +# = DISK ======================================== +while read -r mount avail_mb; do + free_mb=${avail_mb%M} + (( free_mb < MIN_FREE_BOOT_MB )) && add_failure "${mount} has only ${free_mb} MB free (threshold: ${MIN_FREE_BOOT_MB} MB)" +done < <(df --output=target,avail -BM | awk -v re="$EXCLUDE_REGEX" 'NR>1 && $1 ~ /^\/boot/ && $1 !~ re') +while read -r mount avail_gb; do + free_gb=${avail_gb%G} + (( free_gb < MIN_FREE_GB )) && add_failure "${mount} has only ${free_gb} GB free (threshold: ${MIN_FREE_GB} GB)" +done < <(df --output=target,avail -BG | awk -v re="$EXCLUDE_REGEX" 'NR>1 && $1 !~ /^\/boot/ && $1 !~ re') + +# = RESULT ====================================== +if (( ${#FAILURES[@]} > 0 )); then + fail_now "$(printf '%s\n' "${FAILURES[@]}")" +fi +curl -fsS --retry 2 --retry-delay 1 "$PING_URL" >/dev/null +exit 0 diff --git a/sync-to-ceres.sh b/sync-to-ceres.sh deleted file mode 100755 index cb223aa..0000000 --- a/sync-to-ceres.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/bash - -echo "Syncing files to Ceres" - -scp -r monitoring/grafana/ server@10.0.5.2:/home/server/docker/ps2alerts/production/files -scp -r monitoring/prometheus/ server@10.0.5.2:/home/server/docker/ps2alerts/production/files/prometheus -ssh server@10.0.5.2 'docker restart ps2alerts-grafana'