Skip to content

feat(relay): end a drain once every session has left - #4186

Merged
kixelated merged 2 commits into
quest/m1/drain/READMEfrom
quest/m1/drain/drain-exit
Sep 25, 2026
Merged

kixelated merged 2 commits into
quest/m1/drain/READMEfrom
quest/m1/drain/drain-exit

Conversation

@kixelated

Copy link
Copy Markdown
Collaborator

Completes the drain-exit quest and removes the line's last child, sizing the README [M]. What remains on the line is its own end-to-end test.

What changed

Relay::run no longer sleeps out the whole drain window plus a second. It returns once every established session has left. The drain deadline force-closes stragglers, so the window is still the upper bound. The exit log says which bound ended the drain:

  • drain complete: every session left; exiting with elapsed
  • drain deadline force-closed sessions; exiting with elapsed and forced (the count)
  • drain deadline passed with sessions still open; exiting with forced and open. This is the one-second backstop for a close that never completes and should not happen.

/metrics gains a moq_relay_draining_sessions gauge: the sessions sent a GOAWAY that have not left yet.

How it works: shutdown::Observer now keeps a tally. supervise (QUIC on tokio and io_uring, plus iroh) and the WebSocket handler hold a guard for each established session. drain_session holds a draining guard, and that guard counts the session as forced if it is still open at its deadline. The driver arms its force-close after the GOAWAY goes out, so it never fires before that deadline and every force-close gets counted. The guard also covers the WebSocket path, where the drain future is dropped if the driver ends first. Drain deadlines now use tokio::time::Instant, so the unit tests run on paused time.

Decisions (the user was unavailable, so I took the recommended options)

  • The relay's own outbound cluster sessions are not counted. Nothing sends them a GOAWAY, and nobody's media is held hostage on them. The peer sees them close when the process exits. If they counted, every clustered relay would wait out the full window.
  • Counting starts once a session is established, not at accept. A session still in its handshake when the last established one leaves gets cut off along with the process. With DNS withdrawn first, arrivals should be rare, and they would get a GOAWAY right away anyway. This is documented in doc/bin/relay/config.md.
  • No forced-sessions counter on /metrics. The relay exits right after it force-closes the stragglers, so a scrape would almost never see such a counter move. The count goes in the exit log instead, and the gauge shows the drain's progress up to that point.

Tests

  • a_drain_ends_once_every_session_leaves (rs/moq-relay/tests/shutdown_signal.rs): the drain window is 600s. A one-shot client leaves on its GOAWAY, and /metrics then shows one session still draining while the relay keeps running. When that straggler drops, run returns within 5s. Without this change the test times out.
  • shutdown::tests covers the tally on paused time: drained waits for every session, and only a session still open at the deadline counts as forced.
  • The existing drain tests relied on the one-shot client staying connected after a GOAWAY. Since feat(net): migrate the JS connection on GOAWAY; refuse bad redirects in Rust #4143 it leaves. Each of those tests now also holds a moq-lite-03 straggler: that version has no GOAWAY message, so the peer stays until the deadline. That keeps the tests' "waits out the window" assertions meaningful, and the tests now also exercise the forced path.

API and wire impact

  • Public API (additive): internal::Internal::with_shutdown(shutdown::Observer). The Tally/serve/drained additions on shutdown::Observer are pub(crate).
  • Behavior: Relay::run returns as soon as a drain empties. An embedder that relied on it blocking for the full window will now see it return early.
  • Metrics: new gauge moq_relay_draining_sessions.
  • Wire: none.

Checks: just check scoped to this branch. Main's moq-ffi may not compile until #4157 merges; this PR does not touch it.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 2 commits September 25, 2026 12:21
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Relay::run no longer sleeps out the whole drain window. It returns as soon
as every established session has left (the deadline force-closes the rest)
and logs which bound ended the drain, with the number of sessions the
deadline force-closed. /metrics gains moq_relay_draining_sessions.

The relay's own outbound cluster sessions are not counted: nothing sends
them a GOAWAY, and the peer sees them close when the process exits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review September 25, 2026 19:59
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-25T20:03:13.954873Z c20dd4b Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated
kixelated merged commit 5e04fdb into quest/m1/drain/README Sep 25, 2026
3 checks passed
@kixelated
kixelated deleted the quest/m1/drain/drain-exit branch September 25, 2026 22:28
@kixelated

Copy link
Copy Markdown
Collaborator Author

Landing on quest/m1/drain/README.

Relay::run now returns as soon as every established session has left a drain, instead of sleeping out the whole window. The deadline still force-closes stragglers, the exit log says which bound ended the drain, and /metrics gains moq_relay_draining_sessions. Outbound cluster sessions are not counted. No code changes from review; Check and Test were already green.

(Written by Grok 4.7)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant