-
-
Notifications
You must be signed in to change notification settings - Fork 248
fix(uring): halve the completion queue and lift the relay's memlock limit #4197
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -20,13 +20,17 @@ const SQ_ENTRIES: u32 = 256; | |
|
|
||
| /// Completion queue depth. Every in-flight operation can post a completion | ||
| /// (one per send buffer with GSO on, one per provided receive buffer, the | ||
| /// park futex, transient cancels), so this covers a few sockets at the | ||
| /// default pool ceilings in [`udp::Config`]. Running past it is not fatal: | ||
| /// the kernel backlogs completions (`IORING_FEAT_NODROP`) rather than drop | ||
| /// them. But the backlog is an allocation-per-CQE slow path and it ends any | ||
| /// armed multishot receive, so the CQ is sized to keep it out of steady | ||
| /// state. | ||
| const CQ_ENTRIES: u32 = 4096; | ||
| /// park futex, transient cancels), so this covers one socket at the default | ||
| /// pool ceilings in [`udp::Config`], the one-socket-per-worker layout the | ||
| /// relay runs. Running past it is not fatal: the kernel backlogs completions | ||
| /// (`IORING_FEAT_NODROP`) rather than drop them. But the backlog is an | ||
| /// allocation-per-CQE slow path and it ends any armed multishot receive, so | ||
| /// the CQ is sized to keep it out of steady state. | ||
| /// | ||
| /// No larger: the ring is charged to `RLIMIT_MEMLOCK` at 16 bytes per entry, | ||
| /// most of each worker's footprint, and that budget is shared by every | ||
| /// io_uring the user runs. | ||
| const CQ_ENTRIES: u32 = 2048; | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When two default-configured sockets share a worker, each can contribute 1,024 send and 256 receive completions, so their 2,560 CQEs exceed this new 2,048-entry queue. This is a supported in-repo topology: AGENTS.md reference: AGENTS.md:L51-L51 Useful? React with 👍 / 👎. |
||
|
|
||
| /// Maximum completions copied at once while teardown is deadline-bounded. | ||
| const TEARDOWN_CQE_BATCH: usize = 64; | ||
|
|
@@ -891,14 +895,14 @@ mod tests { | |
|
|
||
| #[test] | ||
| fn cq_covers_the_default_pool_ceilings() { | ||
| // The completion queue must cover at least two sockets at their | ||
| // default pool ceilings (plus the futex), or the kernel's overflow | ||
| // slow path becomes steady state for the workload the ceilings exist | ||
| // to serve. Fails when someone raises the udp defaults without | ||
| // revisiting CQ_ENTRIES. | ||
| // The completion queue must cover a socket at its default pool | ||
| // ceilings (plus the futex), or the kernel's overflow slow path | ||
| // becomes steady state for the workload the ceilings exist to serve. | ||
| // Fails when someone raises the udp defaults without revisiting | ||
| // CQ_ENTRIES. | ||
| let config = udp::Config::default(); | ||
| let per_socket = u32::from(config.tx_buffers_max) + u32::from(config.rx_buffers_max); | ||
| assert!(CQ_ENTRIES > 2 * per_socket, "CQ_ENTRIES fell behind the pool defaults"); | ||
| assert!(CQ_ENTRIES > per_socket, "CQ_ENTRIES fell behind the pool defaults"); | ||
| } | ||
|
|
||
| #[test] | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The official package build at
.github/workflows/moq-relay.yml:58-85invokescargo zigbuild ... -p moq-relaywithout--features io-uring, and that feature is explicitly absent from the defaults inrs/moq-relay/Cargo.toml:28-49, so the binary installed alongside this unit cannot create these workers. This therefore grants an unrelated network-facing process unlimited locked memory without fixing any packaged configuration;systemd-analyze syscall-filter @system-servicealso expands to@memlock, whose group includesmlock,mlock2, andmlockall, so the existing sandbox does not neutralize the raised limit. Either enable io_uring in the packaged binary or keep a bounded limit for this unit.Useful? React with 👍 / 👎.