Skip to content

fix(auth): send verification codes from the blocking pool, not the async worker - #214

Merged
argszero merged 1 commit into
mainfrom
fix/mail-send-offload-blocking-pool
Sep 13, 2026
Merged

argszero merged 1 commit into
mainfrom
fix/mail-send-offload-blocking-pool

Conversation

@argszero

Copy link
Copy Markdown
Owner

Summary

crate::mail::send_verification_code is blocking I/O (lettre: 3 attempts, a 2 s sleep between them, 15 s timeout per attempt when SMTP accepts a connection and never answers). It was called in place from three unauthenticated handlers, so it occupied a tokio worker thread instead of the blocking pool — while it runs, the whole runtime stops serving (including /healthz, the liveness probe).

Measured before the fix, with a probe injected into src/mail.rs + src/routes/mod.rs and then reverted byte-identically:

reading value
default worker threads 12 (available_parallelism)
1-worker runtime, one register in flight a concurrent request's own sleep(100ms) took 4.2427 s — the process stops serving
12 workers, 14 concurrent registers a sleep(1500ms) took 4.3416 s, accepts=42 (14x3) ⇒ pool saturated
SMTP accepts but never answers (the author's own documented Gmail silent-drop for datacenter IPs) one request held a worker for 49.028 s

grep spawn_blocking across the tree was 0. The author's convention is already stated in the same file — send_code's doc says 「发信(含重试)不能持锁」 (mailing must not block a shared resource) — only the resource it named was the DB mutex; nobody re-checked the worker pool. mail.rs never mentions the async runtime. Impact is availability, not money: 12 concurrent unauthenticated requests (2 on a 2-core box) stall the gateway, and a failed liveness probe restarts the container.

Changes

  • send_code (src/routes/mod.rs) is now async and moves the send into tokio::task::spawn_blocking with owned inputs (cfg clone, email/code as String). A JoinError becomes a 500 rather than being swallowed.
  • The SMTP-failure arm (log + clear_verification + 502) stays outside the blocking task, so it still takes the DB lock on the async path. src/mail.rs is untouched.
  • The three call sites (register, resend-code, forgot-password) now .await.
  • Drive-by (no behaviour change): the DB guards in those handlers had to become explicit block scopes instead of drop(conn). With an await after them, the async-generator Send analysis still counts the MutexGuard as live across the await (future cannot be sent between threads safely), so axum's Handler bound fails to compile. Guards are released at exactly the same points as before.
  • No config/data-structure changes, so config/config.example.toml needs no update.

Related Issue

None — the problem was found by a runtime probe, not an issue report.

Tests

  • cargo test — 216 passed (was 215; +1 new test), 0 failed
  • cargo fmt --check — clean; cargo clippy --all-targets -- -D warnings — clean
  • New test added

New test routes::tests::slow_smtp_send_does_not_stall_the_runtime: runs the router on a 1-worker runtime with a real listener, and from outside the runtime (a std thread, so it cannot be blocked by the runtime itself) issues GET /healthz while a register sits inside the blocking send against a fake SMTP that accepts-then-closes.

  • The measurement window starts at the fake SMTP's first accept (i.e. the register has already entered the send), so the argon2 cost (~0.24 s, a different concern) is not part of the reading.
  • Fixed: healthz_wait = 521 µs. With only the spawn_blocking reverted: healthz_wait = 4.015941458 s, and that test is the only failure (215 passed / 1 failed) — i.e. the assertion discriminates exactly the defect.
  • Companions: accepts == 3 on both trees (the blocking retry path really ran), register still answering 502 with the same body, and a dev-mode control arm (no SMTP ⇒ non-blocking) answering 201 with a ms-scale /healthz.

Checklist

  • Branch naming follows the convention (fix/)
  • Commit message uses Conventional Commits (fix(auth): …)
  • Single responsibility, minimal diff
  • ⛔ No release (the tag is host-authorized only)
  • Not in scope, deliberately: offloading argon2 (bounded ~0.24 s; deliberately placed in C2090), the retry policy / timeout numbers in mail.rs, the 502-vs-201 semantics, the DB lock, the schema, ui/, i18n

…ync worker

`crate::mail::send_verification_code` is blocking I/O: 3 attempts with a
2 s sleep between them, and a 15 s timeout per attempt when SMTP accepts a
connection but never answers. It was called in place from three
unauthenticated handlers (`register`, `resend-code`, `forgot-password`), so
it occupied a tokio **worker** thread instead of the blocking pool — the
whole runtime stops serving while it runs.

Measured (probe injected, then reverted byte-identically): on a 1-worker
runtime one `register` pushed a concurrent request's 100 ms sleep to
4.2427 s; 14 concurrent registers on the default 12 workers pushed it to
4.3416 s; a silently dropping SMTP held one worker for 49.028 s. The whole
tree had zero `spawn_blocking` calls.

`send_code` is now `async` and moves the send into `tokio::task::spawn_blocking`
with owned inputs; a `JoinError` becomes a 500 instead of being swallowed.
The SMTP-failure arm (log + `clear_verification` + 502) stays outside the
blocking task, so it still takes the DB lock on the async path.
`src/mail.rs` is untouched.

The DB guard scopes in the three handlers had to become explicit blocks
rather than `drop(conn)`: with an `await` after them the generator's
`Send` analysis still counts the `MutexGuard` as live across the await.
No behaviour change — the guards are released at the same points.

Test: `slow_smtp_send_does_not_stall_the_runtime` runs the router on a
1-worker runtime with a real listener and, from outside the runtime, issues
`GET /healthz` while a `register` sits inside the blocking send (gated on the
fake SMTP's first accept, so the argon2 cost is not part of the window).
Fixed: 0.5 ms; with only the `spawn_blocking` reverted: 4.016 s and that test
is the only failure (215 passed / 1 failed). Companions: `accepts == 3` on
both trees, a dev-mode non-blocking control arm, and `register` still
answering 502.
@argszero
argszero merged commit a2d5fe1 into main Sep 13, 2026
1 check passed
@argszero
argszero deleted the fix/mail-send-offload-blocking-pool branch September 13, 2026 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant