Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/bin/relay/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ except that `mtu_discovery` (its datagram path sends a fixed payload) and the
three flow-control windows (these workers run fixed ones) are refused under
`io_uring` rather than quietly ignored. Each worker reports its own counters at
[`/metrics`](/bin/relay/http#get-metrics). The kernel charges each worker's
ring (~100 KiB, plus a page per socket) to `RLIMIT_MEMLOCK`, a budget shared by
ring (~56 KiB, plus a page per socket) to `RLIMIT_MEMLOCK`, a budget shared by
every io\_uring the user runs; raise it (`LimitMEMLOCK=` under systemd) if
workers fail to start with a message naming that limit.

Expand Down
3 changes: 3 additions & 0 deletions packaging/moq-relay/moq-relay.service
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ RestartSec=5s
RestartSteps=5
RestartMaxDelaySec=1min
LimitNOFILE=1048576
# io_uring workers charge their rings to RLIMIT_MEMLOCK (~56 KiB each), and hosts still on the old
# 64 KiB default fit one worker at most.
LimitMEMLOCK=infinity

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid unlimited memlock for a build without io_uring

The official package build at .github/workflows/moq-relay.yml:58-85 invokes cargo zigbuild ... -p moq-relay without --features io-uring, and that feature is explicitly absent from the defaults in rs/moq-relay/Cargo.toml:28-49, so the binary installed alongside this unit cannot create these workers. This therefore grants an unrelated network-facing process unlimited locked memory without fixing any packaged configuration; systemd-analyze syscall-filter @system-service also expands to @memlock, whose group includes mlock, mlock2, and mlockall, so the existing sandbox does not neutralize the raised limit. Either enable io_uring in the packaged binary or keep a bounded limit for this unit.

Useful? React with 👍 / 👎.


# Sandboxing. DynamicUser allocates a transient UID, StateDirectory creates
# /var/lib/moq-relay owned by that UID so cert/key/jwk files placed there are
Expand Down
30 changes: 17 additions & 13 deletions rs/moq-uring/src/worker.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,17 @@ const SQ_ENTRIES: u32 = 256;

/// Completion queue depth. Every in-flight operation can post a completion
/// (one per send buffer with GSO on, one per provided receive buffer, the
/// park futex, transient cancels), so this covers a few sockets at the
/// default pool ceilings in [`udp::Config`]. Running past it is not fatal:
/// the kernel backlogs completions (`IORING_FEAT_NODROP`) rather than drop
/// them. But the backlog is an allocation-per-CQE slow path and it ends any
/// armed multishot receive, so the CQ is sized to keep it out of steady
/// state.
const CQ_ENTRIES: u32 = 4096;
/// park futex, transient cancels), so this covers one socket at the default
/// pool ceilings in [`udp::Config`], the one-socket-per-worker layout the
/// relay runs. Running past it is not fatal: the kernel backlogs completions
/// (`IORING_FEAT_NODROP`) rather than drop them. But the backlog is an
/// allocation-per-CQE slow path and it ends any armed multishot receive, so
/// the CQ is sized to keep it out of steady state.
///
/// No larger: the ring is charged to `RLIMIT_MEMLOCK` at 16 bytes per entry,
/// most of each worker's footprint, and that budget is shared by every
/// io_uring the user runs.
const CQ_ENTRIES: u32 = 2048;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve CQ capacity for multiple sockets per worker

When two default-configured sockets share a worker, each can contribute 1,024 send and 256 receive completions, so their 2,560 CQEs exceed this new 2,048-entry queue. This is a supported in-repo topology: rs/moq-uring/benches/session_lite.rs binds both its server and client sockets through the same handle at lines 115-121. Once concurrent sessions grow both pools, the worker enters the allocation-per-CQE overflow path and terminates armed multishot receives, undermining the benchmark and public multi-socket use; retain the two-socket capacity or enforce a one-socket limit and measure that change.

AGENTS.md reference: AGENTS.md:L51-L51

Useful? React with 👍 / 👎.


/// Maximum completions copied at once while teardown is deadline-bounded.
const TEARDOWN_CQE_BATCH: usize = 64;
Expand Down Expand Up @@ -891,14 +895,14 @@ mod tests {

#[test]
fn cq_covers_the_default_pool_ceilings() {
// The completion queue must cover at least two sockets at their
// default pool ceilings (plus the futex), or the kernel's overflow
// slow path becomes steady state for the workload the ceilings exist
// to serve. Fails when someone raises the udp defaults without
// revisiting CQ_ENTRIES.
// The completion queue must cover a socket at its default pool
// ceilings (plus the futex), or the kernel's overflow slow path
// becomes steady state for the workload the ceilings exist to serve.
// Fails when someone raises the udp defaults without revisiting
// CQ_ENTRIES.
let config = udp::Config::default();
let per_socket = u32::from(config.tx_buffers_max) + u32::from(config.rx_buffers_max);
assert!(CQ_ENTRIES > 2 * per_socket, "CQ_ENTRIES fell behind the pool defaults");
assert!(CQ_ENTRIES > per_socket, "CQ_ENTRIES fell behind the pool defaults");
}

#[test]
Expand Down
Loading