perf(startup): the welcome-credits backfill blocks every boot for ~100s and grants nothing

[FlyM1ss]

Found while reading prod deploy logs for #502. Measured on the live service, not inferred.

The measurement

From the #506 deploy (srv-d7lbmpjbc2fs73bcr6t0, 2026-09-22):

20:13:52.224  INFO:  Waiting for application startup.
20:15:31.709  [credits] welcome campaign total=241 granted=0 existing=241 failed=0
20:15:31.772  INFO:  Application startup complete.      <- 64 ms later

The entire 99.5 s startup hook is that one call. Everything after it finishes takes 64 ms.

It is not a one-off. Same interval on the last three deploys:

deploy	hook duration
#506 · 2026-09-22 20:13	99.5 s
#501 · 2026-09-22 01:26	110.1 s
#500 · 2026-09-21 17:24	109.0 s

Pre-existing — not introduced by any of those three PRs.

Why it costs that much and returns nothing

credits_service.backfill_default_signup_credits (domain/credits/service.py:181) is a serial loop over every durable account:

user_ids = self.store.list_user_ids()
for user_id in user_ids:
    created = self.grant_default_signup_credits(user_id)

241 users, one round trip each against Neon, ≈ 0.41 s per user. And granted=0 / existing=241: the campaign finished long ago, so every boot re-walks the whole table to discover there is nothing to do.

It is the only awaited call in startup_event (app.py:187). Everything else there — paper baselines, daily leaderboard, the bar-cache sweep/warm — is deliberately a daemon thread. This one is await asyncio.to_thread(...), so it holds the hook open, and the new instance does not bind its port until it returns (==> No open ports detected, continuing to scan... at 20:14:16 is the new instance still inside it).

Impact

• Every deploy is ~100 s longer than it needs to be. Not user-visible today — Render keeps the old instance serving, and the /health 200s through that window are the old process — but it is the whole of the deploy's tail latency.
• A genuine cold boot is user-visible: a restart with no old instance to serve, which is exactly the slow-boot case ops: keep-warm cron and free-tier cold-start copy may be obsolete on Standard #471 notes can still happen after a deploy.
• It grows linearly with signups. 241 users → 100 s. At 1,000 users → ~7 minutes of startup hook, still granting nothing.

This is the same shape as both 2026-09-11 outage burners (the 60 s reaper sweep and the inline per-event recompute in record_server_event): an O(users) operation on a path that runs unconditionally. Those were killed in #485; this one survived because it is on the boot path rather than a request path, so it never showed up in egress or latency graphs.

Options, cheapest first

1. Move it off the await. One line — same daemon-thread treatment as the four things around it. Removes it from deploy latency entirely; the backfill still runs. Does not stop the pointless work.
2. Make it skippable. Gate on a marker (a settings row, or list_user_ids_without_default_grant()), so a completed campaign costs one query instead of 241. This is the real fix — the loop's whole purpose is a one-time migration.
3. Batch it. One INSERT ... ON CONFLICT DO NOTHING over the set rather than a round trip per user. Worth doing anyway if the grant stays a startup concern.

(1) and (2) are independent and both worth having: (1) so a slow backfill can never gate a deploy again, (2) so it stops being slow.

Not in scope here

Whether the welcome campaign should run at boot at all. It reads as a migration that was left wired in after it completed; removing it is a product decision, and options 1–3 are all safe without making it.