Rotate the credential encryption key without losing a row

[haksungjang]

GITHUB_APP_ENCRYPTION_KEY now takes a comma-separated list. The first key encrypts, every key decrypts, so a rotation is: put the new key first, re-encrypt, remove the old one. A deployment with one key has a value containing no comma and reaches the same path it did before, which is why the variable was widened rather than replaced.

The dangerous part is not the re-encryption, it is knowing when it has finished. Removing a key while rows still hold ciphertext written under it makes those rows unreadable permanently, and nothing about the running system looks wrong until something reads one.

So counting what is left and rewriting what is left go through one predicate, is_on_current_key. Not two implementations kept in agreement: one implementation. A Fernet token names no key, so "which key wrote this row" can only be answered by trying to open it; at ten thousand rows that is under a tenth of a second, so the scan is a full one.

MODE=count reports and changes nothing; MODE=rewrite moves rows. Both repeat safely, and rewrite interrupts safely because rows already on the newest key are skipped. The update is conditional on the value that was read, so a row somebody changed mid-pass updates nothing and keeps what they wrote: an application write already used the newest key.

The count also appears in the boot log, and only when more than one key is configured. The person who runs the re-encryption and the person who edits the environment are often not the same person, and the command's output was only ever on the first one's terminal.

scripts/restore.sh runs the same count after a restore. A backup carries ciphertext written under whatever key was current when it was taken, and the runbook states the retention rule that follows: the old key must outlive the backups that predate the rotation.

Counting had to distinguish two states

count originally answered "is this row on an older key". That merges two situations needing opposite actions: a row an older key still opens is work the rewrite does, and a row no configured key opens is a key already removed too early, where running the rewrite cannot help. Reported as one number, an operator runs a command that cannot fix what they have and watches the number not move.

A column that changes keys without a migration

core/encrypted_columns.py lists every column holding ciphertext and the key that opens it, and a contract test fails when the code and the list disagree, or when a column changes keys.

This is not hypothetical. A change in flight moved registry_credentials.password_encrypted from the shared key to a derived subkey with no re-encryption, which would have made every already-stored registry password unreadable. The module it edited carried a docstring forbidding that outcome by a different route, and a docstring cannot fail. Planting the same move here fails with a message naming both keys and saying a re-encryption migration has to come with it.

Verification

The rotation sequence is exercised against a real database, ending with the old key removed and the value still readable, because a test that stopped earlier would not have shown the earlier steps were enough.

Three mutations, each confirmed applied. Removing the conditional from the update fails the race test; making the predicate always report the current key fails four. An earlier version of the race test built the UPDATE itself and survived the first mutation, since the statement it checked was the one it had written; it now drives the service's own pass over a stale value. The implementation later changed from rowcount to RETURNING and the mutations were re-run, because a mutation result is about the implementation it ran against.

Full backend suite green, mypy clean across 927 files, ruff clean, ko-style 0, docs-uat 0 errors.