From 2ecb8f3e5de4bfbe1b0d1d52aca6297b354d3ac3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 20 Sep 2026 11:23:10 -0700 Subject: [PATCH 1/3] quest(quic): split ECN into the uring fix, a study, and L4S The io_uring UDP path carries no IP_TOS/IPV6_TCLASS and never reads the mark back, so noq disables ECN on the first ACK. That fix needs no fork, so it becomes its own ready quest; the provider and fq_codel measurement follows it; only the Ect1 option and the ecn config knob wait on the fork. Co-Authored-By: Claude Opus 5 --- quest/m2/quic/README.md | 8 +++-- quest/m2/quic/ecn-measure.md | 37 +++++++++++++++++++++++ quest/m2/quic/ecn-uring.md | 47 +++++++++++++++++++++++++++++ quest/m2/quic/ecn.md | 57 +++++++++++++++++------------------- quest/m2/quic/upstream.md | 4 +-- 5 files changed, 119 insertions(+), 34 deletions(-) create mode 100644 quest/m2/quic/ecn-measure.md create mode 100644 quest/m2/quic/ecn-uring.md diff --git a/quest/m2/quic/README.md b/quest/m2/quic/README.md index 02f8e0237b..081ef50ae2 100644 --- a/quest/m2/quic/README.md +++ b/quest/m2/quic/README.md @@ -45,6 +45,10 @@ This is a transport API change, not a MoQ wire change. written down - [Deliver the application close before io_uring teardown](/quest/m2/quic/uring-close.md) - the peer receives the final close when the client immediately stops its worker +- [ECN on the io_uring UDP path](/quest/m2/quic/ecn-uring.md) - the ring's + sends carry ECT(0) and its receives read the mark, matching `noq-udp` +- [Measure ECN on the backbone](/quest/m2/quic/ecn-measure.md) - a written + verdict on marking versus dropping, and whether Linode and OVH keep marks - [Per-stream ACK progress](/quest/m2/quic/ack-progress.md) - the fork reports how far a send stream has been acknowledged and when - [poll_acked in web-transport](/quest/m2/quic/ack-hook.md) - the @@ -67,8 +71,8 @@ This is a transport API change, not a MoQ wire change. first-class crate in the fork over the shared stream state machine - [Careful resume on reconnect](/quest/m2/quic/careful-resume.md) - a redial starts at the previous connection's rate -- [ECN on the backbone](/quest/m2/quic/ecn.md) - relay peers validate and - react to ECN marks +- [L4S on the backbone](/quest/m2/quic/ecn.md) - an ECT(1) option in the + fork, an `ecn` config knob, and a dualpi2 measurement - [Release the stack](/quest/m2/quic/release.md) - publish immutable, consumable versions of the fork and its adapters - [Upstream the fork](/quest/m2/quic/upstream.md) - every general carried diff --git a/quest/m2/quic/ecn-measure.md b/quest/m2/quic/ecn-measure.md new file mode 100644 index 0000000000..83d8bc9b56 --- /dev/null +++ b/quest/m2/quic/ecn-measure.md @@ -0,0 +1,37 @@ +# [S] Measure ECN on the backbone + +## Goal + +A written verdict on classic ECN between relays: whether a marking +bottleneck reduces the rate before its queue overflows instead of after, +with the numbers, and whether Linode's and OVH's networks preserve ECT(0) +and CE between relays. No code lands; the verdict is recorded in +[L4S on the backbone](/quest/m2/quic/ecn.md), which acts on it. + +## Plan + +A manual procedure on Linux, run as root, with the commands and what to +record written here so the dualpi2 run and any later provider re-check +repeat it. + +- Bottleneck: two relays on the io_uring runtime across a netem link with a + rate limit and delay, once with `fq_codel` marking (`ecn` on) and once + with the same qdisc dropping (`noecn`). Measure queueing delay at the + bottleneck, goodput, and loss under a moq-bench media profile. The + question is whether the marking run holds a shorter queue at the same + goodput. +- Providers: between a Linode relay and an OVH relay, `tcpdump -v` on each + end shows the TOS byte of arriving packets. Record whether ECT(0) arrives + intact in both directions and whether any CE appears; ECN is only alive + when both directions preserve the marks, since a stripped return path + means ACKs without counts. Repeat the capture for cross-provider and + same-provider pairs. +- Record the qdisc parameters, the moq-bench profile, the kernel version, + and the tcpdump summaries beside the numbers in the L4S quest's Plan. + If neither provider preserves the marks, say so there: L4S stays off and + the marking response is only a lab result. + +## Required + +- [ECN on the io_uring UDP path](/quest/m2/quic/ecn-uring.md) - the relays + under test must mark diff --git a/quest/m2/quic/ecn-uring.md b/quest/m2/quic/ecn-uring.md new file mode 100644 index 0000000000..0738af91ce --- /dev/null +++ b/quest/m2/quic/ecn-uring.md @@ -0,0 +1,47 @@ +# [S] ECN on the io_uring UDP path + +## Goal + +A relay on the io_uring runtime keeps ECN negotiated the way it already does +on the tokio runtime: its packets leave marked ECT(0), and marks on arriving +packets reach the QUIC stack, so the peer's ACKs carry ECN counts and noq +never disables ECN on a uring-to-uring session. A viewer's session is +unaffected. + +## Plan + +noq-proto already does classic ECN end to end: every path starts with +`sending_ecn` on and marks ECT(0), `process_ecn` validates the peer's ACK +ECN counts, and an ACK without counts disables it. `noq-udp` carries the +mark through `IP_TOS` / `IPV6_TCLASS` on send and reads it back on receive +(`unix.rs:607-620`). + +The gap is ours. `rs/moq-uring/src/udp.rs` builds its own control messages +and carries only `UDP_SEGMENT` on send and `UDP_GRO` on receive: no TOS or +TCLASS out, no `IP_RECVTOS` / `IPV6_RECVTCLASS` in. Nothing in `moq-uring` +mentions ECN, and `udp::TxBuf::send(len, to, segment)` has no slot for the +codepoint, so on the io_uring path the peer never sees a mark, its ACKs +carry no counts, and noq turns ECN off within the first ACK. + +- Send: `TxBuf::send` takes the codepoint beside the segment size and writes + `IP_TOS` (v4) or `IPV6_TCLASS` (v6) into the same control buffer as + `UDP_SEGMENT`, so a GSO train carries the mark on every segment. The noq + and quinn adapter in `src/quic/quinn/endpoint.rs` passes `Transmit::ecn` + through. quiche has no ECN send API and stays untouched. +- Receive: enable `IP_RECVTOS` and `IPV6_RECVTCLASS` on the socket, parse + the TOS or TCLASS control message next to `UDP_GRO`, and fill + `RecvMeta::ecn` for every segment of a GRO batch. +- `moq-uring` is 0.0.1 and unpublished, so the `udp` module's signature + changes on `main`. No config, wire, or doc changes. +- Regression, in `rs/moq-uring/tests`: a socket pair through the worker + sends with ECT(0) and the receiver reads the codepoint back, over v4 and + v6, for a single datagram and for a GSO train; and a packet arriving with + CE reaches `RecvMeta::ecn` as CE. Fails today. noq-proto 1.2 exposes no + ECN state, so the session-level check waits for the fork; the `rs uring` + nightly lane already runs this crate and gates on the kernel floor. + +## Related + +- [Measure ECN on the backbone](/quest/m2/quic/ecn-measure.md) - the study + this fix unblocks +- [L4S on the backbone](/quest/m2/quic/ecn.md) - the fork-side half diff --git a/quest/m2/quic/ecn.md b/quest/m2/quic/ecn.md index 12f91d1837..126b60703a 100644 --- a/quest/m2/quic/ecn.md +++ b/quest/m2/quic/ecn.md @@ -1,45 +1,42 @@ -# [M] ECN on the backbone +# [M] L4S on the backbone ## Goal -Relay-to-relay sessions keep ECN negotiated on the io_uring runtime the way -they already do on the tokio runtime, and a marking network reduces the rate -before a queue overflows instead of after. L4S (ECT(1) with a scalable -response) is measured behind an option. Paths that strip or mangle the marks -fall back to no ECN, and a viewer's session is unaffected. +Relay-to-relay sessions can mark ECT(1) with a scalable response, measured +behind an option that is off by default, and a deployment whose network +mangles marks can turn ECN off explicitly. Paths that strip or mangle the +marks fall back to no ECN, and a viewer's session is unaffected. ## Plan -noq-proto already does classic ECN end to end: every path starts with -`sending_ecn` on and marks ECT(0) (`connection/paths.rs:305`, -`connection/mod.rs:1260`), `process_ecn` validates the peer's ACK ECN counts -and an ACK without counts disables it (`:3146`, `:3101`), and a CE increase -reaches `Controller::on_congestion_event(is_ecn = true)`, which Cubic and -BBR3 both handle. `noq-udp` carries the mark through `IP_TOS` / -`IPV6_TCLASS` on send and reads it back on receive (`unix.rs:607-620`). +Classic ECN is already end to end once the +[io_uring path marks](/quest/m2/quic/ecn-uring.md); this quest is the +fork-side half. noq-proto has no ECN knob: `sending_ecn` is hardcoded on +per path and only an ACK without counts turns it off, so both `off` and +`ect1` need the fork. -The gap is ours. `rs/moq-uring/src/udp.rs` builds its own cmsgs and carries -no TOS or TCLASS on send and no `IP_RECVTOS` / `IPV6_RECVTCLASS` on receive, -so on the io_uring path the peer never sees a mark, its ACKs carry no counts, -and noq turns ECN off within the first ACK. - -- moq-uring: set the codepoint from `Transmit::ecn` in the send cmsg beside - the GSO segment size, enable the receive-side TOS/TCLASS cmsg, and fill - `RecvMeta::ecn`, so the io_uring path matches `noq-udp`. Regression: a - uring-to-uring session reports ECN still enabled after the handshake. - In the fork: an `Ect1` marking option and the accounting to keep the two codepoints apart, so an L4S response (proportional to the CE fraction per RTT, per RFC 9330 to 9332) can be tried as a `Controller` change without a transport change. Off by default. - `moq-tokio`'s `[quic]` section gains `ecn = off | ect0 | ect1`, ect0 by - default (today's behavior on tokio), so a deployment whose network mangles - marks can turn it off explicitly. -- Measure on a netem bottleneck with a marking qdisc (`fq_codel`, then - `dualpi2` for L4S) against the same bottleneck dropping: queueing delay, - goodput, loss. Record whether Linode's and OVH's networks preserve the - marks between relays; if neither does, L4S stays off and the result is - written down. + default (today's behavior on tokio), and `doc/bin/relay/config.md` + documents it in the same PR. +- Measure on the netem bottleneck from the + [ECN study](/quest/m2/quic/ecn-measure.md) with `dualpi2` marking against + the same bottleneck dropping: queueing delay, goodput, loss. The study's + provider verdict decides whether the result matters outside the lab; + if neither Linode nor OVH preserves the marks, L4S stays off and the + result is written down here. +- ECN visibility stays on the wire: the study observes marks with tcpdump, + and exposing per-path ECN state in stats is a later quest if operators + need it. ## Required -- [Fork noq](/quest/m2/quic/fork.md) - the `Ect1` option lives there +- [Fork noq](/quest/m2/quic/fork.md) - the `Ect1` option and the `off` knob + live there +- [ECN on the io_uring UDP path](/quest/m2/quic/ecn-uring.md) - the relays + must mark before a response can be measured +- [Measure ECN on the backbone](/quest/m2/quic/ecn-measure.md) - the + provider verdict this quest acts on diff --git a/quest/m2/quic/upstream.md b/quest/m2/quic/upstream.md index 820b8f5c5c..842c1fd45c 100644 --- a/quest/m2/quic/upstream.md +++ b/quest/m2/quic/upstream.md @@ -28,7 +28,7 @@ it: 3. keep-alive by deadline ([keep-alive](/quest/m2/quic/keep-alive.md)); 4. hierarchical send groups ([scheduler](/quest/m2/quic/scheduler.md)); 5. careful resume as a `Controller` wrapper ([careful resume](/quest/m2/quic/careful-resume.md)); -6. ECN validation and marking ([ECN](/quest/m2/quic/ecn.md)); +6. ECT(1) marking and its accounting ([L4S](/quest/m2/quic/ecn.md)); 7. per-stream deadlines ([deadlines](/quest/m2/quic/deadline.md)); 8. capacity probing by early retransmission ([probe](/quest/m2/quic/probe.md)); 9. the qmux crate over the shared stream state machine ([qmux](/quest/m2/quic/qmux.md)). @@ -50,7 +50,7 @@ offered and answered. - [Keep-alive by deadline](/quest/m2/quic/keep-alive.md) - [Hierarchical stream scheduling](/quest/m2/quic/scheduler.md) - [Careful resume on reconnect](/quest/m2/quic/careful-resume.md) -- [ECN on the backbone](/quest/m2/quic/ecn.md) +- [L4S on the backbone](/quest/m2/quic/ecn.md) - [Per-stream deadlines](/quest/m2/quic/deadline.md) - [Probe by early retransmission](/quest/m2/quic/probe.md) - [qmux on the QUIC stream state machine](/quest/m2/quic/qmux.md) From b139a65e841a41dd04b7ed11f37847d2b1155f0d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 20 Sep 2026 12:23:15 -0700 Subject: [PATCH 2/3] quest(quic): name the real ECN call sites in moq-uring Co-Authored-By: Claude Opus 5 --- quest/m2/quic/ecn-uring.md | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/quest/m2/quic/ecn-uring.md b/quest/m2/quic/ecn-uring.md index 0738af91ce..49073f045e 100644 --- a/quest/m2/quic/ecn-uring.md +++ b/quest/m2/quic/ecn-uring.md @@ -26,17 +26,23 @@ carry no counts, and noq turns ECN off within the first ACK. - Send: `TxBuf::send` takes the codepoint beside the segment size and writes `IP_TOS` (v4) or `IPV6_TCLASS` (v6) into the same control buffer as `UDP_SEGMENT`, so a GSO train carries the mark on every segment. The noq - and quinn adapter in `src/quic/quinn/endpoint.rs` passes `Transmit::ecn` - through. quiche has no ECN send API and stays untouched. -- Receive: enable `IP_RECVTOS` and `IPV6_RECVTCLASS` on the socket, parse - the TOS or TCLASS control message next to `UDP_GRO`, and fill - `RecvMeta::ecn` for every segment of a GRO batch. + and quinn adapter passes `Transmit::ecn` at both call sites, + `rs/moq-uring/src/quic/quinn/endpoint.rs:377` and + `rs/moq-uring/src/quic/quinn/connection.rs:797`. quiche has no ECN send + API, so its three callers (`quiche/endpoint.rs:476`, + `quiche/connection.rs:843`) pass no codepoint and change only to match + the signature. +- Receive: enable `IP_RECVTOS` and `IPV6_RECVTCLASS` on the socket and parse + the TOS or TCLASS control message next to `UDP_GRO`. `udp::Packet` gains + an `ecn` accessor (one mark per completion; a GRO batch shares it), and + the noq and quinn adapter threads it into both `Endpoint::handle` calls at + `quinn/endpoint.rs:240-252`, which pass `None` today. - `moq-uring` is 0.0.1 and unpublished, so the `udp` module's signature changes on `main`. No config, wire, or doc changes. - Regression, in `rs/moq-uring/tests`: a socket pair through the worker - sends with ECT(0) and the receiver reads the codepoint back, over v4 and - v6, for a single datagram and for a GSO train; and a packet arriving with - CE reaches `RecvMeta::ecn` as CE. Fails today. noq-proto 1.2 exposes no + sends with ECT(0) and the receiving `Packet::ecn` reads it back, over v4 + and v6, for a single datagram and for a GSO train; and a datagram sent + with CE arrives as CE. Fails today. noq-proto 1.2 exposes no ECN state, so the session-level check waits for the fork; the `rs uring` nightly lane already runs this crate and gates on the kernel floor. From b6dfad465223a6370ef4d6e314567df6ca0ed7d8 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 20 Sep 2026 12:48:47 -0700 Subject: [PATCH 3/3] quest(quic): two quiche callers, full paths Co-Authored-By: Claude Opus 5 --- quest/m2/quic/ecn-uring.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/quest/m2/quic/ecn-uring.md b/quest/m2/quic/ecn-uring.md index 49073f045e..db39ec0a7a 100644 --- a/quest/m2/quic/ecn-uring.md +++ b/quest/m2/quic/ecn-uring.md @@ -29,9 +29,9 @@ carry no counts, and noq turns ECN off within the first ACK. and quinn adapter passes `Transmit::ecn` at both call sites, `rs/moq-uring/src/quic/quinn/endpoint.rs:377` and `rs/moq-uring/src/quic/quinn/connection.rs:797`. quiche has no ECN send - API, so its three callers (`quiche/endpoint.rs:476`, - `quiche/connection.rs:843`) pass no codepoint and change only to match - the signature. + API, so its two callers (`rs/moq-uring/src/quic/quiche/endpoint.rs:476` + and `rs/moq-uring/src/quic/quiche/connection.rs:843`) pass no codepoint + and change only to match the signature. - Receive: enable `IP_RECVTOS` and `IPV6_RECVTCLASS` on the socket and parse the TOS or TCLASS control message next to `UDP_GRO`. `udp::Packet` gains an `ecn` accessor (one mark per completion; a GRO batch shares it), and