fix(health): readiness probe returns 503 when the database is down

[tompscanlan]

Problem

readiness() wrapped the Terminus check in a try/catch and returned a plain object on failure, which NestJS serializes as HTTP 200. So a pod that can't reach Postgres passes its readiness probe:

• a pod that loses the database stays in the Service endpoints and keeps taking traffic
• a maxUnavailable=0 rolling update into a broken DB config replaces both pods and completes "successfully"
• the startup probe points at the same path, so it can't catch a DB-broken start either

The old behavior was observed before the fix: with the database ping failing, GET /health/readiness answered 200 through the real Terminus pipeline (the new HTTP-level spec, run pre-fix, failed with Expected: 503, Received: 200).

Fix

• Let the ServiceUnavailableException propagate — the 503 is how the probe reports "not ready".
• Readiness now gates on the database only. Matrix was in the check alongside the DB ping; once the 503 propagates, a Matrix outage would have pulled every api pod out of Service. A Matrix outage degrades chat, not the API, so it comes out of readiness — /health/matrix still carries that signal. liveness is unchanged (its always-OK behavior is deliberate).

The spec test asserting the error body (should return status error when database is disconnected) encoded the old behavior and is flipped deliberately: it now asserts propagation. New HTTP-level tests cover 503 on DB-down, 200 healthy, and matrix staying out of readiness.

Consumers checked

GlobalExceptionFilter (registered via APP_FILTER in InterceptorsModule) catches the propagated exception, keeps the 503 (getStatus() for any HttpException), and rewrites the body to {statusCode, message, path, timestamp} — the HTTP spec runs through the filter so it asserts the response the probes actually see. The k8s probes, Prometheus blackbox probe (http_2xx module), and ingress health check all read the status code; nothing parses the old error body. test/kubernetes/kubernetes.e2e-spec.ts runs against a healthy environment and still passes.

One side effect to be aware of: each failing readiness hit now also increments unhandled_exceptions_total and error-logs via the filter — a DB-down pod gets loud, which seems right for this failure.

Refs: om-ep1ej