diff --git a/quest/m1/README.md b/quest/m1/README.md index e213823fff..8bbbbb08ce 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -62,7 +62,7 @@ session outcomes, and embedder refusals inside session outcomes. - [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - every stream group in a lite subscription arrives or is dropped by name, and lite-07 drops its stream count for it - [lite-07 Live flag](/quest/m1/lite-live.md) - a separate `Live` field on lite-07 SUBSCRIBE, so merged floors never starve a subscriber; late lower groups build on it - [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - a lite-07 SUBSCRIBE asks for past and live ranges in either order and replaces FETCH; relays fill misses by range, including over moq-transport -- [Cross-relay bursts re-run](/quest/m1/cross-relay-bursts.md) - condition: the #4349 reporter re-runs their A/B/C comparison against current cdn.moq.pro +- [Reproduce cross-node announce flapping](/quest/m1/cross-node-flapping.md) - a clean close reaches a client on another node as one retraction, checked by a 3+ relay drill on main - [Untimed lite-07](/quest/m1/lite-untimed.md) - lite-07 carries an untimed track in both languages; lite-05/06 write send time - [A watch and publish release ships assets()](/quest/m1/assets-release.md) - the release that lets the sites host the worklets - [Dogfood hosted worklets](/quest/m1/dogfood-assets.md) - the moq.pro dashboard hosts the worklets and calls `assets()` after the release diff --git a/quest/m1/cluster-routing/README.md b/quest/m1/cluster-routing/README.md index 9fe66baf32..84d4ea4b0b 100644 --- a/quest/m1/cluster-routing/README.md +++ b/quest/m1/cluster-routing/README.md @@ -143,5 +143,5 @@ Once every child has landed: - [P2P](/quest/m3/p2p/README.md) - browser and native peers that become routing nodes over this layer - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - the importer half of a redundant pair under one explicit epoch - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - a redundant pair shares one epoch -- [Cross-relay bursts re-run](/quest/m1/cross-relay-bursts.md) - its #4349 report shows closed broadcasts announced for up to 229 s +- [Reproduce cross-node announce flapping](/quest/m1/cross-node-flapping.md) - a drill for #4349's post-close announce toggling across nodes, checked on the release and on main - [Routing cost domains](/quest/m3/routing-cost-domains.md) - policy and aggregation at boundaries between operators diff --git a/quest/m1/cross-node-flapping.md b/quest/m1/cross-node-flapping.md new file mode 100644 index 0000000000..caad7bfb07 --- /dev/null +++ b/quest/m1/cross-node-flapping.md @@ -0,0 +1,54 @@ +# [M] Reproduce cross-node announce flapping + +## Goal + +A time-mocked drill with three or more clustered relays, extending +`rs/moq-net/tests/mesh_withdraw.rs` (which already checks one retraction on a +34-node partial mesh) or added beside it: a publisher on one node cleanly closes a broadcast, and a client +watching its prefix on another node sees one retraction, with no +re-announce, for that path. The drill runs against `main` and lands as a +regression test. + +Reported in [#4349](https://github.com/moq-dev/moq/issues/4349) against the +release on cdn.moq.pro (2026-10-11): after a clean close, the subscriber's +announce stream saw the path Announced/Retracted about 1140 times over about +2 s, and subscribes meanwhile answered `internal error` or `unroutable`. +It never happened on one node or on a self-hosted two-relay 0.17.2 cluster. +Loss and FETCH were clean in every run (about 31k FETCHes, all ok). + +## Plan + +- Decided 2026-10-11: `main` likely already handles this (the report ran on + the release), so reproduce before fixing. Run the drill on the + `moq-relay-v0.17.2` tree too, to confirm it catches the reported toggling; + a passing `main` drill counts as evidence only after that positive control + fails on the release. +- Suspected mechanism, unconfirmed: path hunting in today's lite-06 path + vector. A peer's re-announce of a withdrawn prefix clears its withdrawal + (`reannounced` in `rs/moq-net/src/model/origin.rs`) and revives stale + alternate routes through it, so each node's late withdrawal toggles the + path again. A two-relay cluster has no alternates, which fits the report. + #4642's 300 ms hold does not cover it: it delays only updates to a + prefix already delivered, and each toggle is a fresh retract or announce. +- If `main` is clean, land the drill and close #4349 asking the reporter to + re-run once the next release reaches cdn.moq.pro. If `main` still flaps, + fix it at the source in the path vector when the fix is small; otherwise + land the drill as an acceptance test for + [Routes and announces](/quest/m1/cluster-routing/routes.md), keep #4349 + open, and move this `Closes` entry there. +- Not planned here: the reporter's self-hosted two-relay rejoin stall + (`subscribe canceled (idle)`, then `old`, then no state). On 0.17.2 a + rejoin could read a stale cached group as the live edge, which #4741 and + #4914 changed on `main`; revisit it if their re-run on a release + carrying those still stalls. +- Harness for reference: https://github.com/fperex/moq-burst-repro (published + crates only); its traces hold per-session announcement sequences. + +## Closes + +- [#4349](https://github.com/moq-dev/moq/issues/4349) - close this issue when the quest finishes + +## Related + +- [Routes and announces](/quest/m1/cluster-routing/routes.md) - the lite-07 redesign that ends path hunting by construction +- [Dropped sources](/quest/m1/dropped-sources.md) - the `internal error` replies seen during the same churn diff --git a/quest/m1/cross-relay-bursts.md b/quest/m1/cross-relay-bursts.md deleted file mode 100644 index 40e2a18ec0..0000000000 --- a/quest/m1/cross-relay-bursts.md +++ /dev/null @@ -1,55 +0,0 @@ -# [XS] Cross-relay bursts re-run - -## Goal - -Condition: the [#4349](https://github.com/moq-dev/moq/issues/4349) reporter -re-runs their A/B/C comparison (different nodes, one node, self-hosted) -against current cdn.moq.pro, and ideally shares their load harness, as asked -in the 2026-10-06 maintainer reply. - -Check: a new comment on #4349 with the results, noting for each failed FETCH -whether it ended in an error or a timeout. - -Once the condition clears, delete this quest. If the re-run still loses -groups or stalls cross-node, replace it with a concrete repro quest built on -their harness and move the `Closes` entry there; if it comes back clean, -close #4349 in the deletion PR. - -## Plan - -Established by a mock two-relay repro (one-frame groups in bursts of 14, a -subscription opened 5 s before the burst, fetch-on-gap with a 2 s deadline, -a flapping peer link), run against the 0.15.6 tree the reporter's clients -used and against `main`: - -- **Fixed.** Burst-start misses (one more group lost per relay hop) and - route-flap drops (about 70 of 560 groups) reproduce only on 0.15.6 and are - gone on moq-net 0.3.8 and later, most likely by #4387. -- **Not reproduced.** Unanswered FETCHes and the 30 s `Stream(Old)` stalls - showed up on no build, nor in the real-QUIC drill - `rs/moq-relay/tests/drills.rs::bursts_cross_a_cluster` (bursts of 30 - across two clustered relays, every hop with 5% loss, delay, a 100 kbit/s - bottleneck that each burst overflows, and 16-stream QUIC credit): every - group arrives within about 3.5 s, and the occasional live `Old` reset is - usually recovered by FETCH. Not always: on `main` at `fe0113ff2`, 1 of 40 - runs lost group 89 to a live `Old` reset whose FETCH was also refused `old` - (seed 11025055814286623932). Several - 30 s timers of that era have since been removed or shortened (#4606, #4741). - That drill uses a lite peer link; over a `moq-transport-19` peer link, a - related refusal reproduces every run: FETCHes answered `old` for groups the - origin still holds. Both are fixed: a FETCH answered from a group still - arriving over a subscription now asks upstream again when that delivery - gives up on it as `Old`, and the drill gained a moq-transport peer link. -- **Queueing.** FETCHes beyond a session's stream credit wait for it, one - cross-relay round trip per slot, so with tight credit a gap's FETCH can - outlast a 2 s deadline without being lost. -- **Expected.** Across relays a burst arrives newest-first, as the lite draft - specifies; fetching on every gap at once multiplies FETCHes. - -## Closes - -- [#4349](https://github.com/moq-dev/moq/issues/4349) - close this issue when the quest finishes - -## Related - -- [Routes and announces](/quest/m1/cluster-routing/routes.md) - owns the stale and flapping announcements from the same report diff --git a/quest/m1/dropped-sources.md b/quest/m1/dropped-sources.md index 4c20212884..c185cc6188 100644 --- a/quest/m1/dropped-sources.md +++ b/quest/m1/dropped-sources.md @@ -23,6 +23,27 @@ end carries no cause since #4031, so only track errors are in scope. neither document nor test `is_auth` for a stream-scoped Unauthorized. Each should surface Unauthorized. +- Decided 2026-10-11 from [#4349](https://github.com/moq-dev/moq/issues/4349), + where a clean publisher close ended an in-flight subscription with + `internal error` and subscribes during cross-node churn answered the same. + `Dropped` has no arm in `From<&Error> for StreamError`, so it falls through + the `_ => Internal` catch-all to 0x0. + - Fronts end with `Error::Dropped` today (`model/front.rs`: `Retired`, + `Closed`, a resolved front whose route leaves the table, `source_closed`). + Requests and tracks a front actually aborts end with `Unroutable` instead, + or `Cancel` on a local origin `Retired` or `Closed`. + - Tracks in flight are not aborted: `Action::End` in `model/origin.rs` + leaves them on their copy, which drains (ANNOUNCE_END does not disturb + in-flight subscriptions) and ends with the source's own error. Trace that + copy's end for the reported case rather than remapping the front. + - Remove the `_ => Internal` catch-alls in both `From<&Error>` conversions + (`StreamError` and `SessionError`); `Dropped` stays `Internal` + deliberately, since it then marks a real bug. + - Regression: a relay test closes a broadcast under an active subscription + whose source copy ends with a specific non-clean error, and expects that + exact error, never `internal error`; a subscribe against a withdrawn + route expects `Unroutable`. + Public API: none expected; error values consumers observe change. Wire: none. ## Required