diff --git a/quest/m2/quic/README.md b/quest/m2/quic/README.md index 02f8e0237b..081ef50ae2 100644 --- a/quest/m2/quic/README.md +++ b/quest/m2/quic/README.md @@ -45,6 +45,10 @@ This is a transport API change, not a MoQ wire change. written down - [Deliver the application close before io_uring teardown](/quest/m2/quic/uring-close.md) - the peer receives the final close when the client immediately stops its worker +- [ECN on the io_uring UDP path](/quest/m2/quic/ecn-uring.md) - the ring's + sends carry ECT(0) and its receives read the mark, matching `noq-udp` +- [Measure ECN on the backbone](/quest/m2/quic/ecn-measure.md) - a written + verdict on marking versus dropping, and whether Linode and OVH keep marks - [Per-stream ACK progress](/quest/m2/quic/ack-progress.md) - the fork reports how far a send stream has been acknowledged and when - [poll_acked in web-transport](/quest/m2/quic/ack-hook.md) - the @@ -67,8 +71,8 @@ This is a transport API change, not a MoQ wire change. first-class crate in the fork over the shared stream state machine - [Careful resume on reconnect](/quest/m2/quic/careful-resume.md) - a redial starts at the previous connection's rate -- [ECN on the backbone](/quest/m2/quic/ecn.md) - relay peers validate and - react to ECN marks +- [L4S on the backbone](/quest/m2/quic/ecn.md) - an ECT(1) option in the + fork, an `ecn` config knob, and a dualpi2 measurement - [Release the stack](/quest/m2/quic/release.md) - publish immutable, consumable versions of the fork and its adapters - [Upstream the fork](/quest/m2/quic/upstream.md) - every general carried diff --git a/quest/m2/quic/ecn-measure.md b/quest/m2/quic/ecn-measure.md new file mode 100644 index 0000000000..83d8bc9b56 --- /dev/null +++ b/quest/m2/quic/ecn-measure.md @@ -0,0 +1,37 @@ +# [S] Measure ECN on the backbone + +## Goal + +A written verdict on classic ECN between relays: whether a marking +bottleneck reduces the rate before its queue overflows instead of after, +with the numbers, and whether Linode's and OVH's networks preserve ECT(0) +and CE between relays. No code lands; the verdict is recorded in +[L4S on the backbone](/quest/m2/quic/ecn.md), which acts on it. + +## Plan + +A manual procedure on Linux, run as root, with the commands and what to +record written here so the dualpi2 run and any later provider re-check +repeat it. + +- Bottleneck: two relays on the io_uring runtime across a netem link with a + rate limit and delay, once with `fq_codel` marking (`ecn` on) and once + with the same qdisc dropping (`noecn`). Measure queueing delay at the + bottleneck, goodput, and loss under a moq-bench media profile. The + question is whether the marking run holds a shorter queue at the same + goodput. +- Providers: between a Linode relay and an OVH relay, `tcpdump -v` on each + end shows the TOS byte of arriving packets. Record whether ECT(0) arrives + intact in both directions and whether any CE appears; ECN is only alive + when both directions preserve the marks, since a stripped return path + means ACKs without counts. Repeat the capture for cross-provider and + same-provider pairs. +- Record the qdisc parameters, the moq-bench profile, the kernel version, + and the tcpdump summaries beside the numbers in the L4S quest's Plan. + If neither provider preserves the marks, say so there: L4S stays off and + the marking response is only a lab result. + +## Required + +- [ECN on the io_uring UDP path](/quest/m2/quic/ecn-uring.md) - the relays + under test must mark diff --git a/quest/m2/quic/ecn-uring.md b/quest/m2/quic/ecn-uring.md new file mode 100644 index 0000000000..db39ec0a7a --- /dev/null +++ b/quest/m2/quic/ecn-uring.md @@ -0,0 +1,53 @@ +# [S] ECN on the io_uring UDP path + +## Goal + +A relay on the io_uring runtime keeps ECN negotiated the way it already does +on the tokio runtime: its packets leave marked ECT(0), and marks on arriving +packets reach the QUIC stack, so the peer's ACKs carry ECN counts and noq +never disables ECN on a uring-to-uring session. A viewer's session is +unaffected. + +## Plan + +noq-proto already does classic ECN end to end: every path starts with +`sending_ecn` on and marks ECT(0), `process_ecn` validates the peer's ACK +ECN counts, and an ACK without counts disables it. `noq-udp` carries the +mark through `IP_TOS` / `IPV6_TCLASS` on send and reads it back on receive +(`unix.rs:607-620`). + +The gap is ours. `rs/moq-uring/src/udp.rs` builds its own control messages +and carries only `UDP_SEGMENT` on send and `UDP_GRO` on receive: no TOS or +TCLASS out, no `IP_RECVTOS` / `IPV6_RECVTCLASS` in. Nothing in `moq-uring` +mentions ECN, and `udp::TxBuf::send(len, to, segment)` has no slot for the +codepoint, so on the io_uring path the peer never sees a mark, its ACKs +carry no counts, and noq turns ECN off within the first ACK. + +- Send: `TxBuf::send` takes the codepoint beside the segment size and writes + `IP_TOS` (v4) or `IPV6_TCLASS` (v6) into the same control buffer as + `UDP_SEGMENT`, so a GSO train carries the mark on every segment. The noq + and quinn adapter passes `Transmit::ecn` at both call sites, + `rs/moq-uring/src/quic/quinn/endpoint.rs:377` and + `rs/moq-uring/src/quic/quinn/connection.rs:797`. quiche has no ECN send + API, so its two callers (`rs/moq-uring/src/quic/quiche/endpoint.rs:476` + and `rs/moq-uring/src/quic/quiche/connection.rs:843`) pass no codepoint + and change only to match the signature. +- Receive: enable `IP_RECVTOS` and `IPV6_RECVTCLASS` on the socket and parse + the TOS or TCLASS control message next to `UDP_GRO`. `udp::Packet` gains + an `ecn` accessor (one mark per completion; a GRO batch shares it), and + the noq and quinn adapter threads it into both `Endpoint::handle` calls at + `quinn/endpoint.rs:240-252`, which pass `None` today. +- `moq-uring` is 0.0.1 and unpublished, so the `udp` module's signature + changes on `main`. No config, wire, or doc changes. +- Regression, in `rs/moq-uring/tests`: a socket pair through the worker + sends with ECT(0) and the receiving `Packet::ecn` reads it back, over v4 + and v6, for a single datagram and for a GSO train; and a datagram sent + with CE arrives as CE. Fails today. noq-proto 1.2 exposes no + ECN state, so the session-level check waits for the fork; the `rs uring` + nightly lane already runs this crate and gates on the kernel floor. + +## Related + +- [Measure ECN on the backbone](/quest/m2/quic/ecn-measure.md) - the study + this fix unblocks +- [L4S on the backbone](/quest/m2/quic/ecn.md) - the fork-side half diff --git a/quest/m2/quic/ecn.md b/quest/m2/quic/ecn.md index 12f91d1837..126b60703a 100644 --- a/quest/m2/quic/ecn.md +++ b/quest/m2/quic/ecn.md @@ -1,45 +1,42 @@ -# [M] ECN on the backbone +# [M] L4S on the backbone ## Goal -Relay-to-relay sessions keep ECN negotiated on the io_uring runtime the way -they already do on the tokio runtime, and a marking network reduces the rate -before a queue overflows instead of after. L4S (ECT(1) with a scalable -response) is measured behind an option. Paths that strip or mangle the marks -fall back to no ECN, and a viewer's session is unaffected. +Relay-to-relay sessions can mark ECT(1) with a scalable response, measured +behind an option that is off by default, and a deployment whose network +mangles marks can turn ECN off explicitly. Paths that strip or mangle the +marks fall back to no ECN, and a viewer's session is unaffected. ## Plan -noq-proto already does classic ECN end to end: every path starts with -`sending_ecn` on and marks ECT(0) (`connection/paths.rs:305`, -`connection/mod.rs:1260`), `process_ecn` validates the peer's ACK ECN counts -and an ACK without counts disables it (`:3146`, `:3101`), and a CE increase -reaches `Controller::on_congestion_event(is_ecn = true)`, which Cubic and -BBR3 both handle. `noq-udp` carries the mark through `IP_TOS` / -`IPV6_TCLASS` on send and reads it back on receive (`unix.rs:607-620`). +Classic ECN is already end to end once the +[io_uring path marks](/quest/m2/quic/ecn-uring.md); this quest is the +fork-side half. noq-proto has no ECN knob: `sending_ecn` is hardcoded on +per path and only an ACK without counts turns it off, so both `off` and +`ect1` need the fork. -The gap is ours. `rs/moq-uring/src/udp.rs` builds its own cmsgs and carries -no TOS or TCLASS on send and no `IP_RECVTOS` / `IPV6_RECVTCLASS` on receive, -so on the io_uring path the peer never sees a mark, its ACKs carry no counts, -and noq turns ECN off within the first ACK. - -- moq-uring: set the codepoint from `Transmit::ecn` in the send cmsg beside - the GSO segment size, enable the receive-side TOS/TCLASS cmsg, and fill - `RecvMeta::ecn`, so the io_uring path matches `noq-udp`. Regression: a - uring-to-uring session reports ECN still enabled after the handshake. - In the fork: an `Ect1` marking option and the accounting to keep the two codepoints apart, so an L4S response (proportional to the CE fraction per RTT, per RFC 9330 to 9332) can be tried as a `Controller` change without a transport change. Off by default. - `moq-tokio`'s `[quic]` section gains `ecn = off | ect0 | ect1`, ect0 by - default (today's behavior on tokio), so a deployment whose network mangles - marks can turn it off explicitly. -- Measure on a netem bottleneck with a marking qdisc (`fq_codel`, then - `dualpi2` for L4S) against the same bottleneck dropping: queueing delay, - goodput, loss. Record whether Linode's and OVH's networks preserve the - marks between relays; if neither does, L4S stays off and the result is - written down. + default (today's behavior on tokio), and `doc/bin/relay/config.md` + documents it in the same PR. +- Measure on the netem bottleneck from the + [ECN study](/quest/m2/quic/ecn-measure.md) with `dualpi2` marking against + the same bottleneck dropping: queueing delay, goodput, loss. The study's + provider verdict decides whether the result matters outside the lab; + if neither Linode nor OVH preserves the marks, L4S stays off and the + result is written down here. +- ECN visibility stays on the wire: the study observes marks with tcpdump, + and exposing per-path ECN state in stats is a later quest if operators + need it. ## Required -- [Fork noq](/quest/m2/quic/fork.md) - the `Ect1` option lives there +- [Fork noq](/quest/m2/quic/fork.md) - the `Ect1` option and the `off` knob + live there +- [ECN on the io_uring UDP path](/quest/m2/quic/ecn-uring.md) - the relays + must mark before a response can be measured +- [Measure ECN on the backbone](/quest/m2/quic/ecn-measure.md) - the + provider verdict this quest acts on diff --git a/quest/m2/quic/upstream.md b/quest/m2/quic/upstream.md index 820b8f5c5c..842c1fd45c 100644 --- a/quest/m2/quic/upstream.md +++ b/quest/m2/quic/upstream.md @@ -28,7 +28,7 @@ it: 3. keep-alive by deadline ([keep-alive](/quest/m2/quic/keep-alive.md)); 4. hierarchical send groups ([scheduler](/quest/m2/quic/scheduler.md)); 5. careful resume as a `Controller` wrapper ([careful resume](/quest/m2/quic/careful-resume.md)); -6. ECN validation and marking ([ECN](/quest/m2/quic/ecn.md)); +6. ECT(1) marking and its accounting ([L4S](/quest/m2/quic/ecn.md)); 7. per-stream deadlines ([deadlines](/quest/m2/quic/deadline.md)); 8. capacity probing by early retransmission ([probe](/quest/m2/quic/probe.md)); 9. the qmux crate over the shared stream state machine ([qmux](/quest/m2/quic/qmux.md)). @@ -50,7 +50,7 @@ offered and answered. - [Keep-alive by deadline](/quest/m2/quic/keep-alive.md) - [Hierarchical stream scheduling](/quest/m2/quic/scheduler.md) - [Careful resume on reconnect](/quest/m2/quic/careful-resume.md) -- [ECN on the backbone](/quest/m2/quic/ecn.md) +- [L4S on the backbone](/quest/m2/quic/ecn.md) - [Per-stream deadlines](/quest/m2/quic/deadline.md) - [Probe by early retransmission](/quest/m2/quic/probe.md) - [qmux on the QUIC stream state machine](/quest/m2/quic/qmux.md)