[Maelstromeous]

Manual steps

No manual steps or intervention required.

What this does

Aggregate writes could lose data without any trace. This makes the failures visible and retries the recoverable ones:

• Duplicate-key races. Each message's docs ran as an ordered bulk of updateMany upserts, and a duplicate-key error was caught and reported as success. When two consumers created the same new row at once, the loser's first doc failed and every increment after it in that message was dropped. Upserts now use updateOne; a duplicate-key error on one doc is retried from that doc, and the docs after it still apply. A second failure on the same doc is thrown.
• Failed writes were acknowledged. Any other write error was logged (or not, for per-alert rows) and the message acked, so a Mongo blip lost increments for good. A failed write that provably wrote nothing is now requeued once; if it fails again on redelivery it is logged as an error and dropped, so a bad message cannot loop. A prolonged Mongo outage still ends in dropped messages after that one retry; a durable delayed retry belongs with the delivery work in the next PR.
• Replays never double count. A failed upsert is only requeued when it provably wrote nothing: the first operation failed, or no Mongo server was reachable. A failure after some increments applied is logged as an error and acked, because replaying it would count those increments twice. Message ids with de-duplication, which make every replay safe, are the next PR.
• Lookup and insert failures. A failed alert lookup in a global message used to be reported as "instance does not exist" and dropped; only a genuinely missing alert is dropped now, and a lookup failure is requeued once. Failed inserts (populations, facility control events) are requeued once; those collections have unique indexes and inserts are now unordered, so a replay cannot duplicate rows.
• Double ack on victories. The victories cache bump ran inside the same try as the write. If Redis failed after the message was acked, the catch acked it again, and RabbitMQ answers a double ack by closing the channel, which stops aggregate consumption until a restart. The cache bump now runs after the message is settled.

Pairs with ps2alerts/aggregator#1024, which fixes the keys those writes use. Either can deploy first.

Validation done

tsc --noEmit and ESLint clean. 13 specs cover the retry, requeue and no-replay rules, all passing, alongside the pagination specs from #736, and run in the Tests workflow.

🤖 Generated with Claude Code