From eaf4e0377de0b3eb81c878829098bc5739802082 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 5 Oct 2026 23:02:17 -0700 Subject: [PATCH] quest: re-scope cross-relay bursts as a wait quest plus an impaired two-relay drill The mock two-relay repro showed the burst-start and route-flap losses from #4349 fixed since 0.15.6, but could not produce relay-hop Old resets. Wait on the reporter's re-run against current cdn.moq.pro, and plan a real-QUIC two-relay drill under loss, delay, and flow-control pressure. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 3 +- quest/m1/cluster-routing/README.md | 2 +- quest/m1/cross-relay-bursts.md | 45 +++++++++++++++++++----------- quest/m1/cross-relay-drill.md | 37 ++++++++++++++++++++++++ 4 files changed, 69 insertions(+), 18 deletions(-) create mode 100644 quest/m1/cross-relay-drill.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 39ceb50da8..756194421d 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -57,7 +57,8 @@ blocks. The quests that gated m0 lines moved under them. - [#2991](/quest/m1/2991-net-coalesce-dynamic-tracks-and-preserve-sequences-across.md) - one dynamic producer per track name in both languages, with the sequence namespace surviving a replacement - [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - a lite-07 SUBSCRIBE asks for past and live ranges in either order and replaces FETCH; relays fill misses by range, including over moq-transport - [Live media time](/quest/m1/subscribe-live-time.md) - re-scoped against `set_live`: a lite-07 SUBSCRIBE_OK carries the publisher's current media time only if a reader still needs it -- [Cross-relay bursts](/quest/m1/cross-relay-bursts.md) - bursty small-group tracks cross two relays without lost groups, unanswered FETCHes, or stalls +- [Cross-relay bursts re-run](/quest/m1/cross-relay-bursts.md) - condition: the #4349 reporter re-runs their A/B/C comparison against current cdn.moq.pro +- [Two-relay drill on impaired links](/quest/m1/cross-relay-drill.md) - a bursty small-group track crosses two clustered relays over lossy, delayed, flow-limited QUIC without unanswered FETCHes or `Old` stalls - [Untimed lite-07](/quest/m1/lite-untimed.md) - lite-07 carries an untimed track in both languages; lite-05/06 write send time - [Finalize moq-lite-07](/quest/m1/lite07-finalize.md) - when the maintainer cuts it, lite-07 negotiates as `moq-lite-07` and the next release ships it - [Late lower groups](/quest/m1/lite-late-lower-group.md) - a moq-lite subscriber with a floor receives a group created below the first served one, as moq-transport does diff --git a/quest/m1/cluster-routing/README.md b/quest/m1/cluster-routing/README.md index f71672a766..33969c1b23 100644 --- a/quest/m1/cluster-routing/README.md +++ b/quest/m1/cluster-routing/README.md @@ -146,5 +146,5 @@ Once every child has landed: - [One route cost](/quest/m1/route-cost.md) - Warm and Cold collapse to one static cost - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - the importer half of a redundant pair under one explicit epoch - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - a redundant pair shares one epoch -- [Cross-relay delivery under bursts](/quest/m1/cross-relay-bursts.md) - its #4349 report shows closed broadcasts announced for up to 229 s +- [Cross-relay bursts re-run](/quest/m1/cross-relay-bursts.md) - its #4349 report shows closed broadcasts announced for up to 229 s - [Routing cost domains](/quest/m3/routing-cost-domains.md) - policy and aggregation at boundaries between operators diff --git a/quest/m1/cross-relay-bursts.md b/quest/m1/cross-relay-bursts.md index 3dc14f1030..34741261dc 100644 --- a/quest/m1/cross-relay-bursts.md +++ b/quest/m1/cross-relay-bursts.md @@ -1,25 +1,37 @@ -# [L] Cross-relay delivery under bursts +# [XS] Cross-relay bursts re-run ## Goal -Bursty small-group tracks cross two relays without losing groups or stalling. -On cdn.moq.pro, publishers and a subscriber on different nodes saw groups -never arrive (FETCH for them unanswered for 2 s while the publisher stayed -connected), a subscription opened early deliver only a burst's tail, and -return tracks stall up to 31 s with `Stream(Old)`. The same workload on one -node, or one self-hosted relay, loses nothing. +Condition: the [#4349](https://github.com/moq-dev/moq/issues/4349) reporter +re-runs their A/B/C comparison (different nodes, one node, self-hosted) +against current cdn.moq.pro, and ideally shares their load harness, as asked +in the 2026-10-06 maintainer reply. -## Plan +Check: a new comment on #4349 with the results, noting for each failed FETCH +whether it ended in an error or a timeout. + +Once the condition clears, delete this quest. If the re-run still loses +groups or stalls cross-node, replace it with a concrete repro quest built on +their harness and move the `Closes` entry there; if it comes back clean, +close #4349 in the deletion PR. -Reproduce first, on a local two-relay cluster with the reporter's harness -(offered in the issue) and the relay build cdn.moq.pro ran. Then fix what the -repro shows. Suspects: `Old` expiry or newest-first dropping on the relay hop, -and FETCH not forwarded or answered upstream. A group awaiting its FIN ack -now expires and follows priority updates, which removed one candidate cause. -Keep the repro as a regression test in the cluster tests. +## Plan -The issue's stale and flapping announcements after a clean close are -cluster-routing evidence, not this quest's scope. +Established by a mock two-relay repro (one-frame groups in bursts of 14, a +subscription opened 5 s before the burst, fetch-on-gap with a 2 s deadline, +a flapping peer link), run against the 0.15.6 tree the reporter's clients +used and against `main`: + +- **Fixed.** Burst-start misses (one more group lost per relay hop) and + route-flap drops (about 70 of 560 groups) reproduce only on 0.15.6 and are + gone on moq-net 0.3.8 and later, most likely by #4387. +- **Not reproduced.** Unanswered FETCHes and the 30 s `Stream(Old)` stalls + showed up on no build. The mock models neither loss nor flow control, so + [Two-relay drill on impaired links](/quest/m1/cross-relay-drill.md) covers + that gap. Several 30 s timers of that era have since been removed or + shortened (#4606, #4741). +- **Expected.** Across relays a burst arrives newest-first, as the lite draft + specifies; fetching on every gap at once multiplies FETCHes. ## Closes @@ -27,4 +39,5 @@ cluster-routing evidence, not this quest's scope. ## Related +- [Two-relay drill on impaired links](/quest/m1/cross-relay-drill.md) - looks for the unreproduced FETCH and `Old` stalls without waiting on the reporter - [Routes and announces](/quest/m1/cluster-routing/routes.md) - owns the stale and flapping announcements from the same report diff --git a/quest/m1/cross-relay-drill.md b/quest/m1/cross-relay-drill.md new file mode 100644 index 0000000000..249336d9b5 --- /dev/null +++ b/quest/m1/cross-relay-drill.md @@ -0,0 +1,37 @@ +# [M] Two-relay drill on impaired links + +## Goal + +A real-QUIC test runs a bursty small-group track (one frame per group, bursts +of tens of groups) from a publisher on one relay to a subscriber on a +clustered second relay, with the links impaired by loss, delay, and +flow-control pressure. It shows every group either arriving or failing by +name within a bounded time: no FETCH left unanswered, and no track stalled +by relay-hop `Old` resets. A failure it finds gets fixed at the cause, or a +quest of its own if large. + +The mock two-relay harness behind +[#4349](https://github.com/moq-dev/moq/issues/4349) cannot produce those +resets, so the issue's unanswered FETCHes and 30 s `Stream(Old)` stalls are +still unexplained. + +## Plan + +`rs/moq-relay/tests/drills.rs` already runs single-relay drills over real +QUIC in a loopback and a `moq_shaper` impaired lane, and the cluster tests +(`cluster_idle.rs`, `goaway_cluster.rs`) start peered relays in-process. +Open for the planner: + +- A new drill in that file's lanes or a separate two-relay test, and which + hop or hops (client and peer links) get shaped. +- How to apply flow-control pressure: tight QUIC windows, a rate limit below + the burst rate, or both. +- The subscriber's recovery model: fetch-on-gap with a deadline, as the + reporter's app does, versus waiting and reordering locally. +- Whether a burst plus a flapping peer link joins as a regression for the + 0.15.6 losses fixed in moq-net 0.3.8. + +## Related + +- [Cross-relay bursts re-run](/quest/m1/cross-relay-bursts.md) - waits on the #4349 reporter's re-run against current cdn.moq.pro +- [Routes and announces](/quest/m1/cluster-routing/routes.md) - the cluster routing a flapping peer link exercises