Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ blocks. The quests that gated m0 lines moved under them.
- [#2991](/quest/m1/2991-net-coalesce-dynamic-tracks-and-preserve-sequences-across.md) - one dynamic producer per track name in both languages, with the sequence namespace surviving a replacement
- [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - a lite-07 SUBSCRIBE asks for past and live ranges in either order and replaces FETCH; relays fill misses by range, including over moq-transport
- [Live media time](/quest/m1/subscribe-live-time.md) - re-scoped against `set_live`: a lite-07 SUBSCRIBE_OK carries the publisher's current media time only if a reader still needs it
- [Cross-relay bursts](/quest/m1/cross-relay-bursts.md) - bursty small-group tracks cross two relays without lost groups, unanswered FETCHes, or stalls
- [Cross-relay bursts re-run](/quest/m1/cross-relay-bursts.md) - condition: the #4349 reporter re-runs their A/B/C comparison against current cdn.moq.pro
- [Two-relay drill on impaired links](/quest/m1/cross-relay-drill.md) - a bursty small-group track crosses two clustered relays over lossy, delayed, flow-limited QUIC without unanswered FETCHes or `Old` stalls
- [Untimed lite-07](/quest/m1/lite-untimed.md) - lite-07 carries an untimed track in both languages; lite-05/06 write send time
- [Finalize moq-lite-07](/quest/m1/lite07-finalize.md) - when the maintainer cuts it, lite-07 negotiates as `moq-lite-07` and the next release ships it
- [Late lower groups](/quest/m1/lite-late-lower-group.md) - a moq-lite subscriber with a floor receives a group created below the first served one, as moq-transport does
Expand Down
2 changes: 1 addition & 1 deletion quest/m1/cluster-routing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,5 +146,5 @@ Once every child has landed:
- [One route cost](/quest/m1/route-cost.md) - Warm and Cold collapse to one static cost
- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - the importer half of a redundant pair under one explicit epoch
- [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - a redundant pair shares one epoch
- [Cross-relay delivery under bursts](/quest/m1/cross-relay-bursts.md) - its #4349 report shows closed broadcasts announced for up to 229 s
- [Cross-relay bursts re-run](/quest/m1/cross-relay-bursts.md) - its #4349 report shows closed broadcasts announced for up to 229 s
- [Routing cost domains](/quest/m3/routing-cost-domains.md) - policy and aggregation at boundaries between operators
45 changes: 29 additions & 16 deletions quest/m1/cross-relay-bursts.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,43 @@
# [L] Cross-relay delivery under bursts
# [XS] Cross-relay bursts re-run

## Goal

Bursty small-group tracks cross two relays without losing groups or stalling.
On cdn.moq.pro, publishers and a subscriber on different nodes saw groups
never arrive (FETCH for them unanswered for 2 s while the publisher stayed
connected), a subscription opened early deliver only a burst's tail, and
return tracks stall up to 31 s with `Stream(Old)`. The same workload on one
node, or one self-hosted relay, loses nothing.
Condition: the [#4349](https://github.com/moq-dev/moq/issues/4349) reporter
re-runs their A/B/C comparison (different nodes, one node, self-hosted)
against current cdn.moq.pro, and ideally shares their load harness, as asked
in the 2026-10-06 maintainer reply.

## Plan
Check: a new comment on #4349 with the results, noting for each failed FETCH
whether it ended in an error or a timeout.

Once the condition clears, delete this quest. If the re-run still loses
groups or stalls cross-node, replace it with a concrete repro quest built on
their harness and move the `Closes` entry there; if it comes back clean,
close #4349 in the deletion PR.

Reproduce first, on a local two-relay cluster with the reporter's harness
(offered in the issue) and the relay build cdn.moq.pro ran. Then fix what the
repro shows. Suspects: `Old` expiry or newest-first dropping on the relay hop,
and FETCH not forwarded or answered upstream. A group awaiting its FIN ack
now expires and follows priority updates, which removed one candidate cause.
Keep the repro as a regression test in the cluster tests.
## Plan

The issue's stale and flapping announcements after a clean close are
cluster-routing evidence, not this quest's scope.
Established by a mock two-relay repro (one-frame groups in bursts of 14, a
subscription opened 5 s before the burst, fetch-on-gap with a 2 s deadline,
a flapping peer link), run against the 0.15.6 tree the reporter's clients
used and against `main`:

- **Fixed.** Burst-start misses (one more group lost per relay hop) and
route-flap drops (about 70 of 560 groups) reproduce only on 0.15.6 and are
gone on moq-net 0.3.8 and later, most likely by #4387.
- **Not reproduced.** Unanswered FETCHes and the 30 s `Stream(Old)` stalls
showed up on no build. The mock models neither loss nor flow control, so
[Two-relay drill on impaired links](/quest/m1/cross-relay-drill.md) covers
that gap. Several 30 s timers of that era have since been removed or
shortened (#4606, #4741).
- **Expected.** Across relays a burst arrives newest-first, as the lite draft
specifies; fetching on every gap at once multiplies FETCHes.

## Closes

- [#4349](https://github.com/moq-dev/moq/issues/4349) - close this issue when the quest finishes

## Related

- [Two-relay drill on impaired links](/quest/m1/cross-relay-drill.md) - looks for the unreproduced FETCH and `Old` stalls without waiting on the reporter
- [Routes and announces](/quest/m1/cluster-routing/routes.md) - owns the stale and flapping announcements from the same report
37 changes: 37 additions & 0 deletions quest/m1/cross-relay-drill.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# [M] Two-relay drill on impaired links

## Goal

A real-QUIC test runs a bursty small-group track (one frame per group, bursts
of tens of groups) from a publisher on one relay to a subscriber on a
clustered second relay, with the links impaired by loss, delay, and
flow-control pressure. It shows every group either arriving or failing by
name within a bounded time: no FETCH left unanswered, and no track stalled
by relay-hop `Old` resets. A failure it finds gets fixed at the cause, or a
quest of its own if large.

The mock two-relay harness behind
[#4349](https://github.com/moq-dev/moq/issues/4349) cannot produce those
resets, so the issue's unanswered FETCHes and 30 s `Stream(Old)` stalls are
still unexplained.

## Plan

`rs/moq-relay/tests/drills.rs` already runs single-relay drills over real
QUIC in a loopback and a `moq_shaper` impaired lane, and the cluster tests
(`cluster_idle.rs`, `goaway_cluster.rs`) start peered relays in-process.
Open for the planner:

- A new drill in that file's lanes or a separate two-relay test, and which
hop or hops (client and peer links) get shaped.
- How to apply flow-control pressure: tight QUIC windows, a rate limit below
the burst rate, or both.
- The subscriber's recovery model: fetch-on-gap with a deadline, as the
reporter's app does, versus waiting and reordering locally.
- Whether a burst plus a flapping peer link joins as a regression for the
0.15.6 losses fixed in moq-net 0.3.8.

## Related

- [Cross-relay bursts re-run](/quest/m1/cross-relay-bursts.md) - waits on the #4349 reporter's re-run against current cdn.moq.pro
- [Routes and announces](/quest/m1/cluster-routing/routes.md) - the cluster routing a flapping peer link exercises
Loading