fix(cron): stop metrics partition drops from deadlocking with message inserts

[odesenfans]

Problem

Nodes started logging this on 2026-09-01:

aleph.jobs.job_utils: Unexpected error while fetching message
psycopg2.errors.DeadlockDetected: deadlock detected
LINE 1: INSERT INTO messages (item_hash, type, chain, sender, signat...
DETAIL:  Process 8802 waits for RowExclusiveLock on relation 16454 of database 16384; blocked by process 8798.
Process 8798 waits for AccessExclusiveLock on relation 16454 of database 16384; blocked by process 8802.

Relation 16454 is messages. Process 8798 is the metrics_partition cron job.

The metrics tables carry a foreign key to messages (migration 0059), and every monthly partition inherits it. Dropping a partition therefore locks messages twice inside one cron transaction:

1. ALTER TABLE ... DETACH PARTITION creates standalone RI triggers on messages: SHARE ROW EXCLUSIVE.
2. DROP TABLE removes those triggers: ACCESS EXCLUSIVE.

The message processor escalates on messages the same way in one transaction: the existence check in MessageHandler.process takes ACCESS SHARE, then the upsert asks for ROW EXCLUSIVE. ROW EXCLUSIVE conflicts with the cron's SHARE ROW EXCLUSIVE, and ACCESS EXCLUSIVE conflicts with the processor's ACCESS SHARE. Each side holds what the other needs and PostgreSQL aborts one of them. Which one dies is arbitrary.

Retention drops only fire on the first cron tick of a new month (a partition's upper bound has to cross the 12 month cutoff), which is why this surfaced on September 1 and not during the month. The lock_timeout from #1247 does not cover it: the deadlock detector fires after deadlock_timeout (1s by default), long before the 5s cap.

Impact is limited: the message is retried, and when the cron is the victim it already catches the OperationalError and retries next tick. But it fires on every node twice a month (once per metrics table) whenever a drop coincides with an insert, and it looks alarming in the logs.

Fix

• Partition creation and each partition drop now run in their own transaction.
• The drop transaction takes LOCK TABLE messages IN ACCESS EXCLUSIVE MODE as its first statement, before DETACH and DROP. The cron never escalates a lock it already holds on messages, so PostgreSQL lets an in-flight processor transaction upgrade past the queued request and commit, after which the cron proceeds. No cycle can form.
• The wait stays bounded by PARTITION_LOCK_TIMEOUT (5s). A timeout still defers the table to the next tick, as before.

The cost is one brief ACCESS EXCLUSIVE on messages per dropped partition, once a month per metrics table, held for the duration of DETACH + DROP (milliseconds). That lock was already being taken by DROP TABLE before this change; it is just taken first now. Removing it entirely would mean dropping the FK from the metrics tables, which is out of scope here.

Test

test_partition_drop_does_not_deadlock_with_message_insert reproduces the production interleaving through the real job: a thread holds an open read on messages, waits (via pg_locks) until the cron is queued for ACCESS EXCLUSIVE on messages, then inserts a message and commits. Without the fix it fails with the exact DeadlockDetected error from the report; with the fix both the insert and the partition drop succeed.

The existing test_cron_defers_table_when_parent_lock_is_contended still passes, so the lock_timeout deferral from #1247 is preserved.

Verification

PYTHONPATH=src pytest tests/jobs/test_metrics_partition_job.py -q
10 passed

black, isort and ruff are clean on both files. mypy reports no errors in the changed module (the remaining output is the pre-existing missing-stubs noise from other modules).

[foxpatch-aleph]

Correct and well-reasoned fix for a genuine mutual lock-upgrade deadlock. Taking ACCESS EXCLUSIVE on messages as the drop transaction's first statement ensures the cron never escalates a lock it already holds, letting an in-flight processor transaction upgrade past the queued request and commit — the standard remedy for this class of deadlock. I verified by inspection that the scoring-insert path (insert_node_metrics from the post handler) takes its messages locks before touching crn_metrics, so the new lock ordering introduces no cycle there; that _partitions_past_cutoff preserves the old filter semantics; and that lock_timeout still bounds every wait, with the monkeypatch-based deferral test unaffected. I could not run the tests locally (no hatch/Postgres in this sandbox), so the new interleaving test is verified by reading only. Three non-blocking observations below; none affect correctness.

src/aleph/jobs/cron/metrics_partition_job.py (line 114): Non-blocking: to_drop is now listed in one transaction and consumed by later ones. If a partition disappeared in between, DETACH would raise UndefinedTable — a ProgrammingError, which slips past except OperationalError and escapes run() (caught and logged one level up in CronJob.__run_job, then retried next tick). With one cron process per node DB this race is essentially impossible, so it is fine as-is; if you want it airtight, either re-check existence inside _drop_partition or catch ProgrammingError alongside OperationalError and treat it as a deferral.

src/aleph/jobs/cron/metrics_partition_job.py (line 200): Worth noting in the PR text: this ACCESS EXCLUSIVE on messages is held while DETACH waits for the parent's ACCESS EXCLUSIVE, so a concurrent metrics read/write can block all message traffic for up to PARTITION_LOCK_TIMEOUT (5s), not just the milliseconds described. The old code had comparable exposure once DROP's AEL request was queued, and the window is once-a-month per table, so this is acceptable — just not strictly 'milliseconds' under contention.

tests/jobs/test_metrics_partition_job.py (line 413): The processor thread must win the race to acquire ACCESS SHARE on messages before the cron's drop transaction reaches LOCK TABLE. If the cron wins, no not-granted AEL ever appears in pg_locks and the test fails on "cron never reached DROP". In practice the thread's SELECT runs well before the cron finishes its ensure queries, but setting a threading.Event after the SELECT and waiting on it before run_in_executor would make the test deterministic on loaded CI.