[Maelstromeous]

⚠️ Manual step: replace the queue on the box, then merge both PRs straight away

RabbitMQ won't change an existing queue's arguments, so the durable, lazy api-queue-production has to be swapped for a memory-only one once. Run this just before merging this PR and ps2alerts/api#737:

docker exec rabbit rabbitmqctl delete_queue --if-empty api-queue-production && \
docker exec rabbit rabbitmqctl eval 'rabbit_amqqueue:declare(rabbit_misc:r(<<"/">>, queue, <<"api-queue-production">>), false, false, [{<<"x-message-ttl">>, long, 10800000}], none, <<"rmq-cli">>).'
docker exec rabbit rabbitmqctl -q list_queues name durable arguments messages | grep '^api-queue-production\s'
# expect: api-queue-production  false  [{"x-message-ttl",10800000}]  ...

• --if-empty refuses while messages are waiting and deletes nothing. It normally sits at about 0, so re-run it a moment later.
• Assumes the default vhost /. Add -p <vhost> to both commands (and put the vhost in place of <<"/">>) if production uses another one.
• Skip it and the deploy breaks. The new declaration is refused with PRECONDITION_FAILED - inequivalent arg 'durable', so the new aggregator's API publisher can't connect inside its 5 s boot timeout, and the new api-aggregator can't consume.
• Between the command and the api deploy, nothing consumes the queue, because the delete cancels the old api consumer. Messages wait in memory (3 h TTL) and the old aggregators keep publishing into the new queue without trouble. An old container that reconnects before its own deploy is refused, so merge both PRs promptly.

Rehearsed on a local bitnamilegacy/rabbitmq:3.10.13 holding the production queue arguments. --if-empty refused while 5 messages were waiting, the delete and redeclare left 1,000 delay messages untouched, and both new declarations were then accepted. The old api declaration was refused.

What this does

api-queue-production was durable, lazy and fed persistent messages. RabbitMQ 3.10 therefore wrote every message to disk, even though the api drains the queue to about 0. Every delayed message was written twice: once into its delay queue, and again into api-queue-production after dead-lettering.

• ApiQueue now takes a durable flag. The API queue is declared non-durable with no lazy mode, and its messages are published transient.
• Both delay queues are unchanged: durable, lazy and persistent.
• The declaration has to match perf: declare api-queue-production memory-only, matching the aggregator api#737 exactly, or whichever service declares second is refused.

RabbitMQ 3.10 notes: x-queue-mode: lazy writes every message to disk, transient ones included. That is why lazy mode comes off along with durability. From 3.12 the argument is ignored and every classic queue behaves roughly lazy, so after an upgrade a non-durable queue with transient messages is the only way to keep this off the disk. Classic queues dead-letter at most once, so a dead-letter with no queue to route to is dropped.

Accepted trade-off: after any RabbitMQ restart, api-queue-production doesn't exist until the aggregator or api declares it again. Delay messages that expire before then are dropped. That includes every one that fell due while the broker was down. A local run on 3.10.13 confirmed that restart drops the queue and keeps the delay queues' messages.

Why the delay queues come back empty after a restart

There is no code fix here. The cause is in the broker's storage or in what happens at boot, and the checks below tell which.

No code path deletes them. destroyQueue()/deleteQueue is only reached through QueueAuthority, which only holds the per-alert aggregator-<instance>-<event> queues. ApiQueue.destroy() has no callers. Nothing in either repo purges these queues, and every declaration of them matches.

RabbitMQ keeps them. On a local bitnamilegacy/rabbitmq:3.10.13 with the production arguments, all 70,000 persistent messages in the 91 min delay queue survived three things:

• a graceful restart;
• a SIGKILL;
• a container recreate, with data mounted at /bitnami/rabbitmq/mnesia.

Messages that expired during 30 s of downtime were dead-lettered into api-queue-production at boot, and none were lost.

Two things do empty them:

1. Data that isn't on the bind mount. Bitnami keeps its data behind /bitnami/rabbitmq/mnesia, and the image declares no volume. Mounted anywhere else, the data lives in the container layer. It survives a restart or a reboot, but a recreate wipes every queue: a docker compose down/up, such as the one the stack-recovery script runs. The apps then redeclare the queues empty. Reproduced locally: 5,000 messages survived docker restart, and the queues were gone after a recreate.
2. Downtime longer than the remaining delay. TTL expiry is absolute, so every message that fell due while the broker was down dead-letters at boot. The messages move, but the api then acks and drops any global aggregate whose alert isn't ENDED yet, or whose instance lookup fails. A Mongo error there is reported as "does not exist", which is filtered out of the logs. With this PR, those messages are dropped outright instead (see the trade-off above).

Read-only checks for the box:

docker inspect rabbit --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}created={{.Created}}'
ls <Source>/rabbit@localhost/msg_stores/vhosts        # on the VM, using the Source printed above
docker exec rabbit rabbitmqctl -q eval 'rabbit_mnesia:dir().'

• Cause 1 if the mount source has no rabbit@localhost/msg_stores. Confirm it by checking that created= matches the last stack recycle in the healthcheck log.
• Cause 2 if the source holds live data but api-aggregator logs Received Global Aggregate message for unfinished instance straight after a restart.
• Also worth checking: any setup script on the box that calls the management API, since none is in a repo.

Validation done

• yarn run lint:check and yarn run build pass. The aggregator has no unit test runner; jest follows in a separate PR.
• A local broker run with the built code: the aggregator and api declarations were accepted, direct messages arrived with deliveryMode 1 and delay messages persistent, and after a restart the API queue was gone while the delay queues kept their 1,001 messages.

🤖 Generated with Claude Code