Skip to content
Merged
4 changes: 0 additions & 4 deletions quest/m1/js-restart-keeps-request.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,3 @@ both languages already cover (`a_newer_epoch_leaves_the_broadcast_in_flight`).
Add the test to `js/net/src/origin.test.ts`, failing before the fix.

Public API: behavior only. Wire: none.

## Required

- [Dynamic epoch](/quest/m0/broadcast-epoch/dynamic-epoch.md) - #5141, which corrects the Rust `Dynamic::update` doc this quest aligns JS to
2 changes: 1 addition & 1 deletion quest/m1/test-flakes-2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Public API: none. Wire: none.
## Required

- [Relay restart rebind](/quest/m1/test-flakes-2/relay-restart-rebind.md) - the crash drill restarts on its original UDP address under concurrent load
- [Impaired handshake](/quest/m1/test-flakes-2/impaired-handshake.md) - the impaired cluster drills' clients never time out while connecting
- [Impaired handshake](/quest/m1/test-flakes-2/impaired-handshake.md) - the impaired drills' clients never time out while connecting
- [Interop close code](/quest/m1/test-flakes-2/interop-close-code.md) - the browser close-code test reliably sees `unauthorized` for a refused session
- [TS duration fidelity](/quest/m1/test-flakes-2/ts-duration-fidelity.md) - TS compliance captures the whole round-tripped stream
- [Cluster burst overrun](/quest/m1/test-flakes-2/cluster-burst-overrun.md) - the impaired cluster burst drill always overruns its bottleneck, so it always exercises it
Expand Down
60 changes: 38 additions & 22 deletions quest/m1/test-flakes-2/impaired-handshake.md
Original file line number Diff line number Diff line change
@@ -1,31 +1,47 @@
# [S] Impaired drill clients connect reliably
# [XS] Impaired drill clients connect reliably

## Goal

The impaired cluster drills (`bursts_cross_a_cluster::impaired`,
`bursts_cross_a_flapping_peer::impaired` in `rs/moq-relay/tests/drills.rs`)
never fail while connecting their publisher or subscriber.
The impaired drills (`bursts_cross_a_cluster::impaired`,
`bursts_cross_a_flapping_peer::impaired`, and
`interrupted_publisher_republishes_new_content::impaired` in
`rs/moq-relay/tests/drills.rs`) never fail while connecting their publisher or
subscriber.

## Plan

Measured 2026-10-07 by running the built drill binary in a loop (four tests
in parallel):

- `main` at `fe0113ff2`: 1 of 40 runs failed `publisher connect failed:
Noq(Connection("timed out"))`, seed 3495212678159036442.
- #4972, with both relays back on the default 10s idle timeout: 3 of 40 runs,
`subscriber connect failed: Noq(Connection("timed out"))` twice and once
`failed to exchange h3 settings: ... reset by peer`.

So the relays' idle timeout is not the cause. The likely cause is the drills'
2s client idle timeout (`quic()`) against a handshake over 5% loss and a
100 kbit/s bottleneck: noq arms the idle timer at `max(idle, 3 * PTO)` with
the base PTO, while a backed-off PTO can leave the client silent longer than
that. Confirm with a qlog of a failing seed, then fix it at the cause, without
raising a timeout or retrying the connect.

Public API: none. Wire: none.
Both causes are fixed in the QUIC stack, in `moq-quic` (#5163) and in
moq-dev/noq ([#32](https://github.com/moq-dev/noq/pull/32) and
[#34](https://github.com/moq-dev/noq/pull/34), stacked). The drills still run
on `moq-noq`, and [the switch](/quest/m1/quic/fork/switch.md) is not started,
so what remains is to bump the `moq-noq-proto` pin to a release carrying both
noq PRs. Then loop the impaired drills under load (four to eight in parallel,
a few hundred runs each) and confirm that no connect fails.

The causes, from qlogs of failing runs (2026-10-09). Both are the drills' 2s
client idle timeout meeting a backed-off PTO during the handshake.

1. **A lost Finished waited on the Initial backoff** (`reset by peer`).
RFC 9002 A.11 resets `pto_count` when Initial or Handshake keys are
discarded; noq and quinn did not. The server, which cannot read 1-RTT
before the Finished, idled out first. Fix: reset `pto_count` in
`discard_space`.
2. **Two Initial flights per idle window** (`timed out`). The idle timer was
armed at `max(2s, 3 * 999ms)` = 2997ms, and the second probe is due at
2997ms plus timer slop. Decided 2026-10-09: add a separate
`handshake_idle_timeout` (default 10s, as msquic does) that governs until
the handshake completes. Dead-peer detection on established connections is
unchanged. The rejected options were capping the handshake probe interval
(it touches the PTO deadline's monotonic invariant) and flooring the idle
timer at the backed-off PTO (it slows dead-peer detection).

With the bump, a handshake to an unreachable relay waits 10s instead of about
3s. Check that the crash drill's reconnect loop (`dial()`'s 5s backoff
timeout) still holds.

Public API: none in this repo; `moq-quic` and `moq-noq-proto` gain
`TransportConfig::handshake_idle_timeout`. Wire: none.

## Related

- [Switch](/quest/m1/quic/fork/switch.md) - moves the runtimes from noq to `moq-quic`, whose idle-timer arming a fix may land in instead
- [Switch](/quest/m1/quic/fork/switch.md) - moves the runtimes from noq to `moq-quic`, which already carries both fixes
4 changes: 0 additions & 4 deletions quest/m1/webgpu-safari.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,6 @@ instead of guessing.

Public API: possibly a new `RendererError` value. Wire: none.

## Required

- [WebGPU renderer](/quest/m1/webgpu-renderer.md) - the renderer and `@moq/video` package (#5138) this verifies

## Related

- [WebGPU HDR](/quest/m1/webgpu-hdr.md) - builds on the same renderer
3 changes: 3 additions & 0 deletions rs/moq-quic/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,9 @@ Changes on top of the upstream commit, besides the renames:
A `WouldBlock` partway through the resend makes the caller retry the whole batch, duplicating the datagrams already sent; QUIC drops the duplicates.
Drop it if upstream lands [quinn#2748](https://github.com/quinn-rs/quinn/pull/2748) and we cherry-pick that.
- Apple fast-path fallback preserves the segmentation of batches prepared before the fast path was disabled, including when a private symbol is unavailable.
- Discarding Initial or Handshake keys resets the PTO backoff, as RFC 9002 A.11 does, so a client's Initial backoff no longer delays the retransmission of a lost Finished. Also [moq-dev/noq#32](https://github.com/moq-dev/noq/pull/32); not offered upstream yet.
- `TransportConfig::handshake_idle_timeout` (default 10s, as msquic's `HandshakeIdleTimeoutMs`) governs the idle timer until the handshake completes, and the negotiated idle timeout takes over once established. A 2s idle timeout otherwise expires 3x the initial PTO in, just before the second Initial probe, leaving a client two flights. A server also drops an Initial left waiting for `accept` past it, and a deadline too far out to represent never fires instead of panicking. Also [moq-dev/noq#34](https://github.com/moq-dev/noq/pull/34), whose third change (the handshake idle timeout picks noq's idle-based PTO interval cap) has nothing to apply to here, since quinn's PTO backoff has no such cap.
quinn's `handshake_timeout` test in its async layer expects the handshake to end at the idle timeout, so the switch's import has to set both timeouts there.

Recheck the BBR3 changes when cherry-picking quinn#2481 updates, or if it merges.

Expand Down
17 changes: 17 additions & 0 deletions rs/moq-quic/src/config/transport.rs
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ pub struct TransportConfig {
pub(crate) max_concurrent_bidi_streams: VarInt,
pub(crate) max_concurrent_uni_streams: VarInt,
pub(crate) max_idle_timeout: Option<VarInt>,
pub(crate) handshake_idle_timeout: Duration,
pub(crate) stream_receive_window: VarInt,
pub(crate) receive_window: VarInt,
pub(crate) send_window: u64,
Expand Down Expand Up @@ -104,6 +105,19 @@ impl TransportConfig {
self
}

/// Maximum duration of inactivity to accept before abandoning a handshake.
///
/// Used in place of [`max_idle_timeout`](Self::max_idle_timeout) until the handshake
/// completes, so a short idle timeout meant to detect a dead peer quickly does not also cut
/// a handshake off after a single probe. It applies whether the idle timeout is shorter,
/// longer, or disabled, and whatever idle timeout the peer advertises; the negotiated idle
Comment on lines +110 to +113

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Apply the handshake timeout when rejecting stale Initials

When a server delays Endpoint::start_accept, the stale-Initial check in endpoint.rs:602-607 still uses max_idle_timeout. With a 2s idle timeout and the new 10s handshake timeout, an Initial accepted after 2s is therefore rejected even though this API promises the handshake timeout replaces the idle timeout until establishment. Use handshake_idle_timeout for that check so manually accepted server handshakes receive the configured window.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, fixed in 5e4ce04d. The stale-Initial check now uses handshake_idle_timeout. New test deferred_accept_uses_handshake_idle_timeout: with a 2s idle timeout, an Initial accepted 3s late succeeds and one accepted 10s late is refused with TimedOut. It fails without the change.

(Written by Claude Opus 5.5)

/// timeout takes over once the connection is established. Defaults to 10 seconds, as
/// msquic's `HandshakeIdleTimeoutMs`.
pub fn handshake_idle_timeout(&mut self, value: Duration) -> &mut Self {
self.handshake_idle_timeout = value;
self
}

/// Maximum number of bytes the peer may transmit without acknowledgement on any one stream
/// before becoming blocked.
///
Expand Down Expand Up @@ -383,6 +397,7 @@ impl Default for TransportConfig {
max_concurrent_uni_streams: 100u32.into(),
// 30 second default recommended by RFC 9308 § 3.2
max_idle_timeout: Some(VarInt(30_000)),
handshake_idle_timeout: Duration::from_secs(10),
stream_receive_window: STREAM_RWND.into(),
receive_window: VarInt::MAX,
send_window: (8 * STREAM_RWND).into(),
Expand Down Expand Up @@ -422,6 +437,7 @@ impl fmt::Debug for TransportConfig {
max_concurrent_bidi_streams,
max_concurrent_uni_streams,
max_idle_timeout,
handshake_idle_timeout,
stream_receive_window,
receive_window,
send_window,
Expand Down Expand Up @@ -452,6 +468,7 @@ impl fmt::Debug for TransportConfig {
s.field("max_concurrent_bidi_streams", max_concurrent_bidi_streams)
.field("max_concurrent_uni_streams", max_concurrent_uni_streams)
.field("max_idle_timeout", max_idle_timeout)
.field("handshake_idle_timeout", handshake_idle_timeout)
.field("stream_receive_window", stream_receive_window)
.field("receive_window", receive_window)
.field("send_window", send_window)
Expand Down
23 changes: 16 additions & 7 deletions rs/moq-quic/src/connection/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2116,15 +2116,18 @@ impl Connection {
}

fn reset_idle_timeout(&mut self, now: Instant, space: SpaceId) {
let Some(timeout) = self.idle_timeout else {
return;
let timeout = match &self.state {
state if state.is_closed() => None,
State::Handshake(_) => Some(self.config.handshake_idle_timeout),
_ => self.idle_timeout,
};
if self.state.is_closed() {
self.timers.stop(Timer::Idle);
return;
// A deadline too far out to represent never fires.
let deadline =
timeout.and_then(|timeout| now.checked_add(cmp::max(timeout, 3 * self.pto(space))));
match deadline {
Some(deadline) => self.timers.set(Timer::Idle, deadline),
None => self.timers.stop(Timer::Idle),
}
let dt = cmp::max(timeout, 3 * self.pto(space));
self.timers.set(Timer::Idle, now + dt);
}

fn reset_keep_alive(&mut self, now: Instant) {
Expand Down Expand Up @@ -2360,6 +2363,10 @@ impl Connection {
for packet in sent_packets.into_values() {
self.remove_in_flight(&packet);
}
// Discarding keys is progress, so the backoff from probing the old space
// must not delay the next one (RFC 9002 A.11). A client never resets it on
// an Initial ACK, so this is where its handshake backoff ends.
self.pto_count = 0;
self.set_loss_detection_timer(now)
}

Expand Down Expand Up @@ -2817,6 +2824,8 @@ impl Connection {

self.events.push_back(Event::Connected);
self.state = State::Established;
// Swap the handshake idle timeout for the negotiated one.
self.reset_idle_timeout(now, SpaceId::Data);
trace!("established");
Ok(())
}
Expand Down
15 changes: 7 additions & 8 deletions rs/moq-quic/src/endpoint.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ use thiserror::Error;
use tracing::{debug, error, trace, warn};

use crate::{
Duration, INITIAL_MTU, Instant, MAX_CID_SIZE, MIN_INITIAL_SIZE, RESET_TOKEN_SIZE, ResetToken,
Side, Transmit, TransportConfig, TransportError,
INITIAL_MTU, Instant, MAX_CID_SIZE, MIN_INITIAL_SIZE, RESET_TOKEN_SIZE, ResetToken, Side,
Transmit, TransportConfig, TransportError,
cid_generator::ConnectionIdGenerator,
coding::BufMutExt,
config::{ClientConfig, EndpointConfig, ServerConfig},
Expand Down Expand Up @@ -599,12 +599,11 @@ impl Endpoint {
.clone()
});

if server_config
.transport
.max_idle_timeout
.is_some_and(|timeout| {
incoming.received_at + Duration::from_millis(timeout.into()) <= now
})
// Still handshaking, so the handshake idle timeout applies, not the idle timeout.
if incoming
.received_at
.checked_add(server_config.transport.handshake_idle_timeout)
.is_some_and(|deadline| deadline <= now)
{
debug!("abandoning accept of stale initial");
self.ignore(incoming);
Expand Down
Loading
Loading