Thread-per-core relay runtime: io-uring + quiche, eBPF connection steering, rename moq-native to moq-tokio

[kixelated]

Motivation

The first big customer plans to use MoQ as a message bus for chat. Egress per connection is tiny, so the cost driver is CPU per connection and CPU per message, not bandwidth. The current relay runs one work-stealing tokio runtime: every packet potentially crosses threads, every wakeup is a candidate context switch, and every UDP datagram is a syscall. The goal is a thread-per-core mode where each connection is pinned to one thread for its whole life, I/O is batched via io-uring, and incoming packets are steered to the owning thread in the kernel.

Media workloads (1:1 and 1:N) must not regress and should also benefit.

What the codebase already gives us (verified)

• moq-net is already runtime-agnostic on dev. refactor(net)!: require the poll-based transport interface #2736 (refactor(net)!: require the poll-based transport interface) drives transports purely through web_transport_trait::poll::{Session, SendStream, RecvStream} (v0.4). moq-net has no tokio dependency; every future is kio-based and a session Driver can be stepped with kio::Waiter from any executor. This part of the plan is confirmed, not speculative.
• The one remaining runtime tie is time. kio::time::Deadline (and moq-net's bandwidth sampling, control-stream timeout, and subscription linger) bottom out in web_async::time, which is tokio's time driver on native and panics outside a tokio runtime. A non-tokio runtime needs a pluggable timer backend in kio.
• Send bounds: on native, moq-net boxes internal futures as Send (MaybeSendBox = BoxFuture); only wasm gets the LocalBoxFuture path. Thread-per-core does not require !Send (pinned Send types work; locks just become uncontended), but !Send internals (Rc, thread-local wakers) are off the table until that cfg split is widened.
• A quiche backend already exists (web-transport-quiche, the quiche feature of moq-native), including raw QUIC. quiche is sans-IO by design: the application owns sockets, timers, and packet pacing, which is exactly the contract an io-uring driver wants.
• moq-bench exists and its F=0 mode (lone JSON keyframe groups) is nearly the chat workload already. It has no CPU-side measurement yet.
• No SO_REUSEPORT anywhere today, only SO_REUSEADDR.

Decisions taken (from design review)

1. Success bar: baseline first. Milestone 0 measures where CPU actually goes today, then we commit to a concrete multiplier (e.g. Nx connections per core). No optimization is judged without a number from the rig.
2. QUIC stack: quiche. Battle-tested at exactly this architecture (thread-per-core, SO_REUSEPORT, eBPF steering) at Cloudflare. noq-proto and quinn-proto remain possible later since everything goes through web-transport-trait, but the io-uring driver is built around quiche.
3. Milestone 1 is thread-per-core tokio, not io-uring. N pinned current_thread runtimes with per-thread SO_REUSEPORT sockets and eBPF steering validates the architecture, keeps timers and WebSocket working, and isolates how much win comes from pinning alone vs io-uring itself.
4. Scope: QUIC first, parity later. The new runtime owns QUIC on :443/UDP only. WebSocket/WSS, TCP, UDS, iroh, and cert reload stay on a tokio runtime in the same process. The plan commits to eventually porting the fallbacks so tokio can be dropped from the relay, but not in v1.
5. Deployment floor: our own infra. Modern kernels (6.x), CAP_BPF available. The io-uring path hard-requires its features and refuses to start otherwise. Self-hosters keep the tokio relay; no fallback detection logic.
6. Send story: decided by data. Ship Milestone 1 with Send types, profile atomic/lock overhead under the chat workload, and only widen the wasm !Send path to native (feature-gated) if it shows up as real CPU.
7. io-uring shape: direct implementation. Monoio is rejected for this work because its public UDP API does not expose the ancillary-data and batching primitives we require. Build the spike directly on the io-uring crate.
8. WebTransport layer: web-transport-proto over quiche. web-transport-quiche is coupled to tokio-quiche and spawns tokio tasks, so it is not the abstraction for the new runtime. Reuse the protocol state machine from web-transport-proto and implement its I/O against sans-I/O quiche.
9. UDP batching is required, not optional polish. The tokio/control implementation must support recvmmsg; the io-uring implementation uses multishot recvmsg with provided buffers as its equivalent. Both must preserve ancillary data and UDP GRO. Egress must use sendmsg with UDP_SEGMENT GSO.
10. Time is an M3 prerequisite. Replace the concrete tokio sleep inside kio::time with an executor-provided deadline backend. Tokio and wasm keep their existing behavior; the direct runtime drives deadlines with timeout SQEs or a measured local timer structure.

Architecture sketch

• N worker threads, each pinned to a core. Each binds its own UDP socket to :443 with SO_REUSEPORT, giving N sockets in one reuseport group.
• An sk_reuseport eBPF program (SK_REUSEPORT + BPF_MAP_TYPE_REUSEPORT_SOCKARRAY) parses the QUIC header and steers by connection ID: server-chosen CIDs encode the owning socket index (a few plaintext bytes plus entropy, standard quiche-style routing).
• Initial packets carry a client-chosen DCID we cannot decode, so they hash to an arbitrary socket. Whichever thread receives the Initial owns the connection forever and issues CIDs that encode its own index. No cross-thread handoff, no migration.
• Retransmitted Initials must reach the same thread before our CID takes effect: steer them by hashing the client-chosen DCID consistently in the eBPF program, so retries land on the thread that saw the first packet.
• Each thread runs: io-uring UDP rx (multishot recvmsg with a provided-buffer ring, the io-uring equivalent of recvmmsg, plus GRO) -> quiche packet processing -> moq-net Drivers polled on a local task queue -> quiche egress -> io-uring UDP tx (sendmsg + UDP_SEGMENT GSO, zerocopy only where it wins). Timers come from timeout SQEs or a measured local timer structure feeding quiche's on_timeout and kio deadlines.
• Fanout (1:N) across threads goes through the existing shared Origin model: cross-thread wakeups are memory traffic, not syscalls. Measured, not assumed, in the benchmarks.
• The same process keeps a small tokio runtime for WebSocket/TCP fallback, stats, signal handling, and cert hot-reload.

Non-goals

• Routing Initials to threads with spare capacity. Accept hash placement; revisit only if the benchmarks show real imbalance.
• Customer/tenant affinity (SNI, token root, path). Invites hot keys; a busy tenant must be able to span cores.
• Full moq-native parity in the new crate for v1 (see decision 4).
• Fallback detection for hardened/exotic environments (see decision 5).

Milestones

Each milestone is a tracking issue with its own PRs; this issue is the epic. Every milestone ends with the same benchmark suite run on the same rig, so the deltas compose into one story.

M0: Measure. Chat + media benchmark suite and the tokio baseline

The whole project is gated on this: if profiling shows CPU goes to crypto or memcpy rather than syscalls and scheduling, the plan changes before we build anything.

• Extend moq-bench with a chat profile: thousands of connections, tiny frames, low per-connection rate, F=0 style groups, plus fanout matrices for 1:1 and 1:N (N spanning small rooms to large ones, e.g. 10 / 1k / 100k subscribers).
• Keep/refresh the media profiles (video-sized frames, keyframe cadence) for 1:1 and 1:N.
• Add CPU measurement to the harness: CPU per connection, CPU per message, context switches (perf stat), syscall share, flame graphs of the relay under each profile. Results in a machine-readable format so runs are diffable.
• A just recipe to run the matrix against a relay; document the reference hardware.
• Exit criteria: baseline numbers for the current multi-thread tokio relay are written down, the dominant CPU sinks are identified, and the target multiplier for the project is committed.

M1: Thread-per-core tokio relay

• Relay mode: N pinned threads, each a current_thread tokio runtime, each with its own SO_REUSEPORT UDP socket on :443.
• The sk_reuseport eBPF steering program and its loader: CID-encoded socket index for established connections, consistent DCID hash for Initials. This program is runtime-agnostic and carries over unchanged to the io-uring milestones.
• CID generation in the quiche backend encodes the thread index (quinn can follow later if we care).
• Wire it behind a relay config flag; the default stays the multi-thread runtime.
• Profile: how much did pinning + no work-stealing buy on the M0 suite? What share of remaining CPU is atomics/locks (decides the !Send question), syscalls (decides how much io-uring can still win), crypto?
• Exit criteria: benchmark delta vs M0 recorded; Send/!Send decision made from the profile; go/no-go on the io-uring investment made from the syscall share.

M2: Crate split. moq-native becomes moq-tokio

• Rename moq-native to moq-tokio (semver break, lands on dev per branch targeting rules).
• Extract the runtime-neutral pieces the new runtime will also need (TLS/cert plumbing, config types, address resolution, the CID codec and eBPF loader from M1) into a shared crate so moq-tokio and the future io-uring crate depend on common code instead of copying it.
• Keep moq-native as a deprecated re-export shim only if it is cheap; otherwise a clean break on dev with migration notes in the PR.
• Docs sweep per the cross-package sync table (doc/lib/rs, examples, justfiles).
• Exit criteria: tree builds with moq-tokio; no behavior change; downstream crates in the workspace migrated.

M3: Prove the direct io-uring data path and runtime prerequisites

Time-boxed. This milestone answers whether the required Linux batching path works and improves the actual quiche workload before the relay integration grows around it. There is no Monoio candidate.

M3a: Replace the native kio::time backend

• Consolidate every moq-net timer and clock read behind kio::time. The initial audit found direct web_async::time use in bandwidth sampling, GOAWAY enforcement, and the lite PROBE interval in addition to existing Deadline users.
• Replace Deadline's concrete web_async::time::Sleep with an executor-selected timer registration. Selection must happen at runtime because tokio fallback work and io-uring QUIC coexist in one process.
• Keep the public surface small: callers continue to own and poll a deadline. Do not pass a timer provider through every moq-net type.
• Preserve tokio paused-clock tests and the wasm backend. Add a no-tokio test executor that proves a moq-net driver can arm, re-arm, cancel, and fire deadlines without entering a tokio runtime.
• Compare timeout SQEs with a local heap/wheel for cancellation churn and pacing precision. The ring is not automatically the winner for thousands of short-lived timers.
• Exit criteria: kio::time::Deadline and a representative moq-net driver run under the local executor with no tokio runtime; timer precision and cancellation behavior are measured and documented.

M3b: UDP batching ablation benchmark

• Build the direct per-thread loop on the io-uring crate. Receive with multishot recvmsg plus a provided-buffer ring, parse source/destination address, ECN, timestamps, and UDP_GRO, and re-arm cleanly after IORING_CQE_F_MORE clears or ENOBUFS.
• Add a recvmmsg receive-batching control to the tokio path. Multishot recvmsg is the io-uring equivalent, but the control is needed to separate receive batching from executor effects.
• Send quiche packet batches with sendmsg and a UDP_SEGMENT control message. Only coalesce packets with the same destination, source, ECN, pacing time, and segment size.
• Run an ablation matrix at a fixed offered load: single receive/single send, receive batching only, GSO only, receive batching + GSO, then the direct io-uring path. Toggle GRO separately. Record actual batch sizes, packets/syscall, ring submissions/completions, drops, ENOBUFS, CPU/message, CPU/Gbps, and latency.
• Test chat-sized datagrams and media-sized traffic. Treat zerocopy as a separate experiment because notification CQEs and buffer lifetime may cost more than the copy for small packets.
• Use a dedicated transport benchmark first, then repeat the winning configuration with the M0 workload. The M0 harness speaks MoQ and cannot directly drive a raw echo server without a transport mode/client.
• Exit criteria: all required socket features work on the deployment kernel and NIC; the ablation attributes the gain to receive batching and/or GSO; the direct path improves quiche CPU at the same offered load. If it does not improve end-to-end CPU, stop before M4.

M3c: quiche plus WebTransport proof

• Drive sans-I/O quiche directly from the M3b loop.
• Put web-transport-proto on top for the HTTP/3 CONNECT handshake and WebTransport session state. Do not depend on web-transport-quiche or tokio-quiche in the new path.
• Accept a session from an existing client and run bidirectional stream and datagram echo through it. Feed quiche timeout and pacing deadlines through the M3a backend.
• Keep the adapter shaped toward web_transport_trait::poll, but do not build the complete public transport crate in the spike.
• Exit criteria: the same benchmark client completes the WebTransport handshake and echo workload on tokio and direct io-uring; CPU, latency, loss, and syscall/ring-operation deltas are recorded on the same hardware.

M4: Integrate the proven io-uring runtime

• New crate (name TBD in the M4 PR, e.g. moq-uring): grow the M3 winner into the production runtime. It owns thread spawning/pinning, sockets, the ring, provided-buffer pools, the local executor, and the timer backend selected in M3a.
• Extract the runtime-neutral TLS/cert configuration, CID codec, eBPF loader, and socket configuration that remain private in moq-tokio. The M2 rename landed, but these shared pieces were not extracted.
• Implement the WebTransport-over-quiche adapter from M3c as web_transport_trait::poll session and stream types, then drive real moq-net Drivers.
• Relay integration behind the same config flag as M1: io-uring threads own QUIC, while the tokio side keeps WebSocket/TCP/UDS, stats, signals, and cert reload. Cluster/upstream QUIC dials happen from the io-uring threads too.
• Exit criteria: relay passes the existing integration and smoke tests in io-uring mode; the full M0 suite records deltas against M1; the required receive batching and send GSO remain enabled and observable.

M5: Hardening and continuous benchmarks

• Nightly benchmark runs on a dedicated box (rig details TBD; cloud CI runners are too noisy for CPU numbers), trended over time, regressions visible per commit range.
• Soak: multi-hour chat + media mixed load, memory stability, CID rotation, connection churn.
• Operational story: qlog in io-uring mode, moq-stats integration, graceful drain/shutdown of pinned threads.
• Rollout on our fleet behind the flag, then default for our deployment. moq-tokio remains the default for everyone else.
• If M1's profile said the !Send refactor pays: land it here (feature-gated LocalBoxFuture path in moq-net, Rc internals, thread-local kio wakers) and re-measure.
• Exit criteria: the M0 target multiplier is met or the gap is explained with a profile; benchmarks run nightly; the chat customer's projected load fits the promised hardware.

Risks and open questions

• WebTransport adapter scope: web-transport-proto supplies protocol state, but the new path still owns the quiche stream/datagram adapter and its web_transport_trait::poll lifecycle. Prove the minimal boundary in M3c before designing the production public API.
• Cross-thread fanout: 1:N where subscribers live on many threads keeps the shared-memory origin model on the hot path. If M0/M1 show contention there, group fanout may need per-thread egress queues. Measured before designed.
• io_uring security posture is fine for our fleet (decision 5) but permanently excludes some self-host environments; the tokio relay remains the answer there.
• eBPF program lifecycle: pinning, upgrade across relay restarts (reuseport group membership changes when threads restart), and observability when steering goes wrong.
• Timer precision and cancellation: quiche pacing wants fine-grained timers, while a timeout SQE per logical deadline may create excessive submission and cancellation traffic. M3a compares the ring with a local timer structure under realistic churn.
• Provided-buffer ownership: multishot receive depends on correct buffer-ring replenishment. Exhaustion, stale CQEs after cancellation, and GRO segment parsing need counters and stress tests.
• Benchmark interpretation: loopback UDP microbenchmarks prove feature support and give directional data, but only the fixed-load quiche and M0 runs decide whether the runtime is better.
• Benchmark rig: hardware not yet chosen (dedicated bare metal vs a fixed cloud metal instance). Owner input welcome on what we can dedicate.
• Fanout matrix ceilings (is 100k subscribers per broadcast a real chat shape, or is 10k the honest max?).

No wire format changes anywhere in this plan, so no drafts/ updates are expected. Relay config/docs updates ride each milestone PR per the cross-package sync table.

(Written by GPT-5.6)

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• Multi-threaded UDP Receiver #679

🔗 Related PRs

#570 - Add WebSocket fallback support [closed]
#928 - (AI) Add support for quiche to moq-native [closed]
#1096 - Switch to qmux with ALPN negotiation and TLS 1.2 [closed]
#1810 - feat(relay): unauthenticated internal listener over qmux (tcp:// + unix://) [merged]
#2161 - feat(moq-native): add quic::Client/quic::Server transport config [merged]
#2423 - refactor!: pre-bump API polish across the release batch [merged]
#2490 - feat(net, relay): GOAWAY with graceful drain and cluster migration [open]
#2500 - fix(net): tear down idle upstream subscriptions [merged]
#2514 - feat(moq-native)!: expand quiche backend parity [merged]
#2556 - fix(net): prefer the newest route so a reconnect takes over immediately [merged]
#2629 - feat(net): implement the MoQ Cluster extension over moq-transport [merged]
#2647 - feat: fail-fast retries: jittered backoff bounded by time, not error type [merged]
#2748 - feat(net): announce namespaces unasked, with a SETUP opt-out [merged]
#2771 - fix(net): serve subscriptions in arrival order so reordered groups aren't dropped [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[kixelated]

Preliminary UDP batching benchmark

I ran the existing noq-udp 1.0.1 throughput benchmark as a kernel feasibility and directional check.

Environment:

• AMD Ryzen 7 5800X, 8 cores / 16 threads
• Linux 7.1.3-200.fc44.x86_64
• IPv6 loopback UDP
• 10 MiB per iteration, 1280-byte segments
• Criterion: 20 samples, 2 second measurement, 1 second warmup

GSO	GRO	recvmmsg	Throughput	Delta vs baseline
off	off	off	171.32 MiB/s	baseline
off	off	on	173.06 MiB/s	+1.0%
off	on	off	175.54 MiB/s	+2.5%
off	on	on	168.62 MiB/s	-1.6%
on	off	off	323.06 MiB/s	+88.6%
on	off	on	370.39 MiB/s	+116.2%
on	on	off	320.51 MiB/s	+87.1%
on	on	on	346.79 MiB/s	+102.4%

What this establishes:

• sendmsg with UDP_SEGMENT GSO works on this kernel and is the dominant low-level win in this test.
• Once GSO creates receive bursts, recvmmsg adds another 14.7% over GSO alone.
• recvmmsg alone is neutral here because the benchmark sends once and immediately drains, leaving little queued work to batch.
• GRO is not a throughput win on loopback in this shape. It remains required for feature parity and must be measured across a real NIC.

This is not the M3 decision benchmark. It measures UDP throughput, not quiche CPU, and loopback avoids the NIC. The next benchmark must hold offered load constant and compare CPU/message and CPU/Gbps for:

1. current tokio-quiche,
2. tokio receive changed from single recvmsg to recvmmsg,
3. direct io-uring multishot recvmsg with provided buffers,
4. each receive path with GSO disabled/enabled and GRO toggled separately.

The code audit also found that current tokio-quiche already sends GSO batches with sendmsg and receives GRO with a single recvmsg. The new path needs to preserve that GSO behavior, add receive batching, and show an end-to-end quiche CPU improvement before M4.

(Written by GPT-5.6)

[kixelated]

Plan: starting moq-uring, and the moq-net/kio API changes it needs

Outcome of a design session on the runtime prerequisites (M3a plus the M4 crate shape), assuming #3003 lands. A few decisions amend the epic text; each amendment is called out.

Decisions

1. Explicit runtime injection, no ambient state. (Amends M3a's "executor-selected timer registration", which implied a thread-local or global driver.) Timers reach moq-net through a Runtime trait passed explicitly at session construction, quinn-style. No thread-locals, no globals, no link-time registration. moq-tokio, moq-uring, and moq-wasm each supply their runtime inside their own connect/accept helpers, so end users never see the parameter.

2. kio sheds time entirely. kio::time and kio::tokio are deleted along with the web-async and tokio dependencies; kio 0.6 is pure waker/channel primitives. Time moves to moq_net::runtime. (kio::time's only consumer is moq-net; the apparent sibling-crate matches were substring hits on "tokio::time".)

3. The Runtime trait.

// moq_net::runtime (Runtime re-exported at the crate root)
pub trait Runtime: Clone {
    type Transport: transport::poll::Session;
    type Deadline: Deadline;

    fn deadline(&self) -> Self::Deadline;
    /// Own the session's protocol machine and poll it to completion.
    fn spawn(&self, machine: SessionMachine<Self>);
}

pub trait Deadline {
    /// Arm, re-arm, or disarm. Synchronous and in-memory.
    fn set(&mut self, at: Option<Instant>);
    /// Ready once elapsed; registers the waiter otherwise. Fused.
    fn poll(&mut self, waiter: &kio::Waiter) -> Poll<()>;
}

• Instant is std::time::Instant on native (wasm gets a same-API shim), so quiche interop is conversion-free and tokio arms via Instant::from_std.
• No Send bound anywhere in the trait. The associated Transport type is what makes conditional Send work without feature flags or companion traits: each impl spawns a concrete machine type whose Send-ness it knows. moq-tokio's transport enum is Send, so plain tokio::spawn keeps working (no LocalSets, no work-stealing sacrifice). moq-uring's machine is !Send and goes on its pinned local queue. moq-wasm wraps the machine in kio::wait and spawn_locals it.
• One runtime = one transport type; moq-tokio already erases its backends into one session enum.
• The public Driver type is removed. Client::connect(runtime, transport) spawns the machine and awaits the establishment signal; Server::accept likewise. Deliberate API break.
• Clone because subsystems store their own handle to mint deadlines; it's a ZST copy or an Rc bump.

4. No clock on the trait. Runtime has no now(). Clock reads use std::time::Instant::now() directly, under one discipline: deadline firings gate protocol behavior, clock reads only annotate (group activity stamps, cache epochs, stats, Timestamp anchors). The few sites that compare now() against armed instants (ietf publisher Refused::offerable, linger idle checks) get refactored to key off deadline-fired state, so virtual-time tests stay coherent without virtualizing the clock.

5. Deterministic test runtime. moq_net::runtime::Test behind a test-runtime feature (moq-net dev-depends on itself with the feature enabled, because cfg(test) items are invisible to the tests/ directory's integration crates). Manual clock + deadline wheel + machine queue: advance(Duration) and advance_to_deadline(), both explicit, no auto-advance. Replaces the tokio::time::pause() magic across moq-net's 67 paused tests; migration is incremental behind a shim.

6. Timers are userspace; one park deadline; never per-deadline SQEs. The uring worker keeps a timer wheel/heap and parks with the earliest deadline via the 6.12 absolute-timeout enter argument (zero SQEs). This is what every serious runtime converged on: monoio explicitly rejected per-timer SQEs over cancel churn, glommio uses a BTreeMap, and quiche hands you exactly one timeout() per connection anyway. That survey settles the M3a "timeout SQEs vs local structure" comparison; the trait is SQE-agnostic if this ever needs revisiting.

7. The !Send plan: sever, de-future, relax. No feature flags; Send-ness flows from types.

• Fact established during review: the model layer (origin/broadcast/track/group/frame) never touches the transport, and every Deadline is session-side. The model stays Send + Sync forever, which cross-thread fanout requires.
• Sever: Session stops holding the transport. close() becomes a kio flag the machine observes, closed() a kio consumer, stats() the machine's last sample (the bandwidth sampler already ticks at 100ms while consumed). The public Session handle is then always Send + Sync over any transport, so moq-ffi (UniFFI requires Send + Sync), the relay, and hang never notice the uring path exists. Mirror of the existing invariant: the machine holds no Session clone, and the Session holds no transport.
• De-future: the ~8 remaining async blocks (protocol setup, session maintenance, a few ietf select arms) become named poll state machines; MaybeSendBox/MaybeSendTask and the last cfg(wasm) split in util.rs are deleted. SessionMachine<R> is Send exactly when R::Transport is.
• Relax: transport::poll::Session drops MaybeSend + MaybeSync, keeps Clone + 'static. Clone stays because concurrent protocol ops each need a handle, and the quiche adapter is structurally Rc<RefCell<Connection>> plus per-stream ids anyway: web_transport_trait::poll streams are owned objects while quiche multiplexes every stream op through one &mut Connection, so interior sharing in the adapter is unavoidable.

8. moq-uring worker, kernel floor 6.12, no fallback. (Amends the spike's 6.1 floor.) 6.12 buys IOU_PBUF_RING_INC, the abs-timer park, min-timeout batched waits, and cheap MSG_RING into DEFER_TASKRUN rings, with one code path. Refuses to start on older kernels with a legible error (default Docker/containerd seccomp blocks io_uring entirely; self-hosters keep moq-tokio).

• Ring per pinned thread: SINGLE_ISSUER | DEFER_TASKRUN | COOP_TASKRUN.
• RX: multishot recvmsg + provided buffer ring + UDP_GRO, with IOU_PBUF_RING_INC sizing (few huge buffers consumed incrementally, matching GRO's 100B-to-64KB completion variance).
• TX: sendmsg + UDP_SEGMENT cmsg on every send (never rely on the socket default; cf. tokio-quiche: always set UDP_SEGMENT cmsg when calling sendmsg() cloudflare/quiche#2060). No zerocopy in v1: it only pays above ~3KB per send and had a GSO+ZC EFAULT history around 6.7. The TX pool owns buffers by id with explicit release, so SENDMSG_ZC's deferred-reclaim NOTIF model can be added later without reshaping the path.
• Packet buffers (pre-encrypt TX staging, post-decrypt RX) come from a pool of io_uring registered buffers owned by the worker loop. quiche's BufFactory sits above crypto and is not that pool; the adapter uses BufFactory<Buf = Bytes> + stream_send_zc so publish payloads flow app -> packetizer without a stream-buffer copy.
• Parking/wakeup: a futex word per worker (FUTEX_WAIT SQE armed while parked; a remote waker sets the atomic and calls futex(2) wake, needing no ring of its own, and the atomic fast path skips the syscall while the worker is awake). MSG_RING is a later prototype.
• Public shape: a Worker per thread that yields accepted sessions to a caller-owned loop, plus local spawn. The relay writes the thread loop and owns pinning and drain policy; no serve-callback runtime.
• WebTransport adapter: web-transport-proto state machine over sans-IO quiche, private to moq-uring until the boundary is proven, as Rc<RefCell> types implementing web_transport_trait::poll (already !Send-ready and &mut self by design).

Sequencing (the breaking series targets dev)

1. PR 1 (dev): kio 0.6 (delete time/tokio) + moq_net::runtime (trait, Instant, Test) + the moq-tokio Runtime impl. Driver removed; connect/accept take the runtime.
2. PR 2 (stacked): moq-uring worker skeleton: ring setup, futex park, timer wheel, provided-buffer RX, GSO TX, local spawn, validated by a raw-quiche echo. This is the blocker-flush milestone.
3. PR series 3 (dev): sever Session, de-future the protocol machines, relax the transport bounds. Landable step by step; sever first, it is independently valuable.
4. PR 4: the WebTransport-over-quiche adapter + real moq-net sessions on the worker (M3c equivalent), then relay integration behind the config flag (M4).

Benchmark obligations are unchanged from the epic: every milestone runs the M0 suite, and the ablation matrix (recv batching / GSO / GRO toggles) rides PR 2's echo harness.

Follow-up ledger (post-v1, rough order of expected value)

• SENDMSG_ZC for GSO trains (media workloads): needs the deferred-reclaim pool; verify the GSO+ZC EFAULT fix on the floor kernel first.
• MSG_RING wakeups replacing futex where the sender owns a ring; payload-carrying wakes.
• min-timeout batched waits (io_uring_submit_and_wait_min_timeout, 6.12) under bursty load.
• RX-side Bytes pooling in the adapter (poll_read_chunk served from a per-thread pool via Bytes::from_owner).
• Session-internal mutex removal: the model keeps kio's Mutex (cross-thread by design); session-local state can shed Arc<Mutex> for Rc<RefCell> once profiling justifies it.
• NAPI busy-poll (6.9) as a tail-latency knob, tuned per deployment.
• zcrx: TCP-only as merged (header split + steered queues); watch for a UDP path, do not design for it.
• Bundles: rejected for unconnected UDP; send bundles carry no per-datagram destination and GRO already batches receive.

Context from the research pass: Cloudflare runs tokio-quiche on plain sendmsg+GSO at millions of rps with no io_uring in production, and the best published io_uring QUIC gain is ~25%. The pinning/steering architecture is expected to carry as much of the win as the ring itself, which the M1-vs-M3 split measures separately.

(Written by Claude Fable 5)

[kixelated]

PR 1 of the plan above is up: #3007 (moq_net::runtime, explicit injection, Driver removed, tokio/wasm/Inline/Test runtimes).

Two amendments to the plan, found during implementation and detailed in the PR description:

1. Runtime keeps a defaulted now(). Relative arming (now + interval) and stamp-derived deadlines must read the virtual clock under a virtual-time test, or an advance leaves every later-armed instant in the past (the bandwidth sampler would fire on every poll). The default is the std clock; only the test runtime overrides it, so real runtimes never think about it.
2. The kio 0.6 time removal moves to PR 2. The model layer also owns deadlines (the origin driver's serve/linger machinery, missed in the earlier audit), and ~80 model tests manipulate the paused tokio clock. PR 1 converts the session side, which is everything a session runtime needs; the follow-up migrates the model and origin::Driver (likely by handing the runtime to the driver at spawn time so origin::Producer::new keeps its signature), then deletes kio::time/kio::tokio.

Sequencing is otherwise unchanged: model migration + kio 0.6 next, then the moq-uring worker skeleton.

(Written by Claude Fable 5)

[kixelated]

Stack update + a scope finding on the de-future step

The runtime series now runs four deep: #3007 (runtime injection) <- #3014 (uring worker) <- #3027 (sever) <- #3028 (kio Task + de-erasure).

Landed by the new PRs:

• feat(net)!: sever Session from the transport #3027 is the sever: Session holds no transport; abort/last-drop, closed(), and stats() relay through a supervisor leg of the machine, and the handle is unconditionally Send + Sync (const-asserted).
• feat(kio): let Tasks hold named machines, unboxing the protocol children #3028 gives kio::Tasks a Task trait so named machines store unboxed, and converts every task set whose occupants were already hand-rolled machines (all of lite, plus ietf's GroupServe). util::poll_task is gone.

The finding: the plan's "de-future the ~8 remaining async blocks" undercounts. For Machine<R> to be Send exactly when R::Transport is, the protocol future must be a nameable type all the way down: a single async block anywhere in the composition forces dyn boxing at the machine, and a box must pick Send or not at compile time per target, which is exactly what a native !Send uring transport cannot satisfy. I checked the alternatives before accepting this: a GAT (type Boxed<F>) cannot add a per-impl F: Send bound, and a hyper-style Executor<F> needs the future type nameable in connect's signature (TAIT is unstable). Conditional-Send erasure is inexpressible in stable Rust, so full de-future is the only sound route, as the plan says, just bigger than it says.

Measured scope: the ietf tree has ~40 non-test async fns across adapter.rs (1.4k lines), publisher.rs (3.0k), subscriber.rs (2.8k), and session.rs (1.2k), and the driver arms await subscriber/publisher/adapter futures, so conversion is forced bottom-up through the deepest code: adapter run loops -> subscriber/publisher bodies (run_broadcast, run_subscribe, handle_stream, namespace machinery) -> session driver arms -> Machine concretization + deleting MaybeSendBox/MaybeSendTask. The lite path is already fully machine-based after #3028.

Options for sequencing, in my preference order:

1. Grind the ietf de-future as its own PR series (several large PRs, mechanical but delicate; the lite conversions are precedent).
2. Ship the uring path lite-only first: PR 4's adapter negotiates moq-lite exclusively, with the machine generic over a per-protocol composition, and the ietf de-future proceeds in parallel without blocking the relay integration. Requires care that the ietf construction path stays behind a Send-capable bound without poisoning connect's signature, which is its own design question.
3. Revisit whether ietf-over-uring matters enough to pay for at all (the relay's cluster links and lite clients are the io_uring beneficiaries; ietf interop peers could stay on the tokio stack indefinitely).

Nothing below #3028 depends on this choice; the stack is green (just check / just test, 3171 tests) through all four PRs.

(Written by Claude Fable 5)