From 0d92970fa1a1b8893b97efc4f319717d39700325 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 12:32:06 -0700 Subject: [PATCH 1/3] quest(m1): plan #4349 follow-ups after the reporter's re-run The re-run against cdn.moq.pro clears the cross-relay-bursts condition: no loss and no failed FETCH. What remains is cross-node announce flapping after a clean close, and `internal error` replies from fronts that end on Error::Dropped. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 3 +- quest/m1/cluster-routing/README.md | 2 +- quest/m1/cross-node-flapping.md | 51 +++++++++++++++++++++++++++ quest/m1/cross-relay-bursts.md | 55 ------------------------------ quest/m1/front-abort-codes.md | 42 +++++++++++++++++++++++ 5 files changed, 96 insertions(+), 57 deletions(-) create mode 100644 quest/m1/cross-node-flapping.md delete mode 100644 quest/m1/cross-relay-bursts.md create mode 100644 quest/m1/front-abort-codes.md diff --git a/quest/m1/README.md b/quest/m1/README.md index e213823fff..ce0f2d4c3d 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -62,7 +62,8 @@ session outcomes, and embedder refusals inside session outcomes. - [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - every stream group in a lite subscription arrives or is dropped by name, and lite-07 drops its stream count for it - [lite-07 Live flag](/quest/m1/lite-live.md) - a separate `Live` field on lite-07 SUBSCRIBE, so merged floors never starve a subscriber; late lower groups build on it - [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - a lite-07 SUBSCRIBE asks for past and live ranges in either order and replaces FETCH; relays fill misses by range, including over moq-transport -- [Cross-relay bursts re-run](/quest/m1/cross-relay-bursts.md) - condition: the #4349 reporter re-runs their A/B/C comparison against current cdn.moq.pro +- [Reproduce cross-node announce flapping](/quest/m1/cross-node-flapping.md) - a clean close reaches a client on another node as one retraction, checked by a 3+ relay drill on main +- [Fronts abort instead of dropping](/quest/m1/front-abort-codes.md) - a closed or withdrawn broadcast ends subscriptions with Unroutable or Cancel, never internal error - [Untimed lite-07](/quest/m1/lite-untimed.md) - lite-07 carries an untimed track in both languages; lite-05/06 write send time - [A watch and publish release ships assets()](/quest/m1/assets-release.md) - the release that lets the sites host the worklets - [Dogfood hosted worklets](/quest/m1/dogfood-assets.md) - the moq.pro dashboard hosts the worklets and calls `assets()` after the release diff --git a/quest/m1/cluster-routing/README.md b/quest/m1/cluster-routing/README.md index 9fe66baf32..0828499680 100644 --- a/quest/m1/cluster-routing/README.md +++ b/quest/m1/cluster-routing/README.md @@ -143,5 +143,5 @@ Once every child has landed: - [P2P](/quest/m3/p2p/README.md) - browser and native peers that become routing nodes over this layer - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - the importer half of a redundant pair under one explicit epoch - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - a redundant pair shares one epoch -- [Cross-relay bursts re-run](/quest/m1/cross-relay-bursts.md) - its #4349 report shows closed broadcasts announced for up to 229 s +- [Reproduce cross-node announce flapping](/quest/m1/cross-node-flapping.md) - #4349's post-close announce toggling across nodes, reproduced on main - [Routing cost domains](/quest/m3/routing-cost-domains.md) - policy and aggregation at boundaries between operators diff --git a/quest/m1/cross-node-flapping.md b/quest/m1/cross-node-flapping.md new file mode 100644 index 0000000000..e3a373cf1c --- /dev/null +++ b/quest/m1/cross-node-flapping.md @@ -0,0 +1,51 @@ +# [M] Reproduce cross-node announce flapping + +## Goal + +A time-mocked drill in `rs/moq-relay/tests` with three or more clustered +relays: a publisher on one node cleanly closes a broadcast, and a client +watching its prefix on another node sees one retraction, with no +re-announce, for that path. The drill runs against `main` and lands as a +regression test. + +Reported in [#4349](https://github.com/moq-dev/moq/issues/4349) against the +release on cdn.moq.pro (2026-10-11): after a clean close, the subscriber's +announce stream saw the path Announced/Retracted about 1140 times over about +2 s, and subscribes meanwhile answered `internal error` or `unroutable`. +It never happened on one node or on a self-hosted two-relay 0.17.2 cluster. +Loss and FETCH were clean in every run (about 31k FETCHes, all ok). + +## Plan + +- Decided 2026-10-11: `main` likely already handles this (the report ran on + the release), so reproduce before fixing. Run the drill on the + `moq-relay-v0.17.2` tree too, to confirm it catches the reported toggling. +- Suspected mechanism, unconfirmed: path hunting in today's lite-06 path + vector. A peer's re-announce of a withdrawn prefix clears its withdrawal + (`reannounced` in `rs/moq-net/src/model/origin.rs`) and revives stale + alternate routes through it, so each node's late withdrawal toggles the + path again. A two-relay cluster has no alternates, which fits the report. + #4642's 300 ms hold does not cover it: it delays only updates to a + prefix already delivered, and each toggle is a fresh retract or announce. +- If `main` is clean, land the drill and close #4349 asking the reporter to + re-run once the next release reaches cdn.moq.pro. If `main` still flaps, + fix it at the source in the path vector when the fix is small; otherwise + land the drill as an acceptance test for + [Routes and announces](/quest/m1/cluster-routing/routes.md), keep #4349 + open, and move this `Closes` entry there. +- Not planned here: the reporter's self-hosted two-relay rejoin stall + (`subscribe canceled (idle)`, then `old`, then no state). On 0.17.2 a + rejoin could read a stale cached group as the live edge, which #4741 and + #4914 changed on `main`; revisit it if their re-run on a release + carrying those still stalls. +- Harness for reference: https://github.com/fperex/moq-burst-repro (published + crates only); its traces hold per-session announcement sequences. + +## Closes + +- [#4349](https://github.com/moq-dev/moq/issues/4349) - close this issue when the quest finishes + +## Related + +- [Routes and announces](/quest/m1/cluster-routing/routes.md) - the lite-07 redesign that ends path hunting by construction +- [Fronts abort instead of dropping](/quest/m1/front-abort-codes.md) - the `internal error` replies seen during the same churn diff --git a/quest/m1/cross-relay-bursts.md b/quest/m1/cross-relay-bursts.md deleted file mode 100644 index 40e2a18ec0..0000000000 --- a/quest/m1/cross-relay-bursts.md +++ /dev/null @@ -1,55 +0,0 @@ -# [XS] Cross-relay bursts re-run - -## Goal - -Condition: the [#4349](https://github.com/moq-dev/moq/issues/4349) reporter -re-runs their A/B/C comparison (different nodes, one node, self-hosted) -against current cdn.moq.pro, and ideally shares their load harness, as asked -in the 2026-10-06 maintainer reply. - -Check: a new comment on #4349 with the results, noting for each failed FETCH -whether it ended in an error or a timeout. - -Once the condition clears, delete this quest. If the re-run still loses -groups or stalls cross-node, replace it with a concrete repro quest built on -their harness and move the `Closes` entry there; if it comes back clean, -close #4349 in the deletion PR. - -## Plan - -Established by a mock two-relay repro (one-frame groups in bursts of 14, a -subscription opened 5 s before the burst, fetch-on-gap with a 2 s deadline, -a flapping peer link), run against the 0.15.6 tree the reporter's clients -used and against `main`: - -- **Fixed.** Burst-start misses (one more group lost per relay hop) and - route-flap drops (about 70 of 560 groups) reproduce only on 0.15.6 and are - gone on moq-net 0.3.8 and later, most likely by #4387. -- **Not reproduced.** Unanswered FETCHes and the 30 s `Stream(Old)` stalls - showed up on no build, nor in the real-QUIC drill - `rs/moq-relay/tests/drills.rs::bursts_cross_a_cluster` (bursts of 30 - across two clustered relays, every hop with 5% loss, delay, a 100 kbit/s - bottleneck that each burst overflows, and 16-stream QUIC credit): every - group arrives within about 3.5 s, and the occasional live `Old` reset is - usually recovered by FETCH. Not always: on `main` at `fe0113ff2`, 1 of 40 - runs lost group 89 to a live `Old` reset whose FETCH was also refused `old` - (seed 11025055814286623932). Several - 30 s timers of that era have since been removed or shortened (#4606, #4741). - That drill uses a lite peer link; over a `moq-transport-19` peer link, a - related refusal reproduces every run: FETCHes answered `old` for groups the - origin still holds. Both are fixed: a FETCH answered from a group still - arriving over a subscription now asks upstream again when that delivery - gives up on it as `Old`, and the drill gained a moq-transport peer link. -- **Queueing.** FETCHes beyond a session's stream credit wait for it, one - cross-relay round trip per slot, so with tight credit a gap's FETCH can - outlast a 2 s deadline without being lost. -- **Expected.** Across relays a burst arrives newest-first, as the lite draft - specifies; fetching on every gap at once multiplies FETCHes. - -## Closes - -- [#4349](https://github.com/moq-dev/moq/issues/4349) - close this issue when the quest finishes - -## Related - -- [Routes and announces](/quest/m1/cluster-routing/routes.md) - owns the stale and flapping announcements from the same report diff --git a/quest/m1/front-abort-codes.md b/quest/m1/front-abort-codes.md new file mode 100644 index 0000000000..f91a7fcb4d --- /dev/null +++ b/quest/m1/front-abort-codes.md @@ -0,0 +1,42 @@ +# [S] Fronts abort instead of dropping + +## Goal + +A subscription whose broadcast's source closes, whose route is withdrawn, +or whose origin shuts down ends with a deliberate stream code, never +`internal error`. The local-to-wire error conversions in +`rs/moq-net/src/error.rs` match every `Error` variant explicitly, so a new +variant cannot hide behind a catch-all. + +From [#4349](https://github.com/moq-dev/moq/issues/4349): a clean publisher +close ended the subscriber's in-flight subscription with `internal error`, +and subscribes during cross-node churn answered the same. + +## Plan + +Decided 2026-10-11: + +- `Error::Dropped` means a producer went away without aborting, which is a + bug, not a routine end. Fronts end with it today + (`rs/moq-net/src/model/front.rs`: `Event::Retired`, `Event::Closed`, a + resolved front whose route leaves the table, `source_closed`), and + `Action::End` in `origin.rs` passes it to every unfinished track; it has + no arm in `From<&Error> for StreamError`, so it reaches the wire as + `internal error` (0x0). +- Codes: a closed source or withdrawn route ends with `Unroutable`, the same + answer a fresh subscribe already gets once the broadcast is gone. A local + origin `Retired` or `Closed` ends with `Cancel`. A track the publisher + finished still ends with FIN. +- Remove the `_ => Internal` catch-alls in both `From<&Error>` conversions + (`StreamError` and `SessionError`). `Dropped` stays `Internal` + deliberately, since it now marks a real bug. +- No wire change: every code already exists. Check `js/net` for the same + catch-all only if it has an equivalent conversion. + +Tests: front unit tests assert the new codes in place of +`Action::End { err: Error::Dropped }`, and a relay test closes a broadcast +under an active subscription and expects `Unroutable`, not `internal error`. + +## Related + +- [Reproduce cross-node announce flapping](/quest/m1/cross-node-flapping.md) - the churn where subscribes answered `internal error` From 65f1bb7bddf52ba742530e9d6988aab7adeb7a57 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 12:39:48 -0700 Subject: [PATCH 2/3] quest(m1): fold the abort plan into dropped-sources In-flight subscriptions end with their source copy's error, not the front's, so remapping front ends alone cannot fix #4349's case; the existing dropped-sources quest already owns preserving the real error. The flapping drill extends mesh_withdraw and needs the release tree as a positive control. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 1 - quest/m1/cross-node-flapping.md | 11 +++++---- quest/m1/dropped-sources.md | 20 ++++++++++++++++ quest/m1/front-abort-codes.md | 42 --------------------------------- 4 files changed, 27 insertions(+), 47 deletions(-) delete mode 100644 quest/m1/front-abort-codes.md diff --git a/quest/m1/README.md b/quest/m1/README.md index ce0f2d4c3d..8bbbbb08ce 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -63,7 +63,6 @@ session outcomes, and embedder refusals inside session outcomes. - [lite-07 Live flag](/quest/m1/lite-live.md) - a separate `Live` field on lite-07 SUBSCRIBE, so merged floors never starve a subscriber; late lower groups build on it - [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - a lite-07 SUBSCRIBE asks for past and live ranges in either order and replaces FETCH; relays fill misses by range, including over moq-transport - [Reproduce cross-node announce flapping](/quest/m1/cross-node-flapping.md) - a clean close reaches a client on another node as one retraction, checked by a 3+ relay drill on main -- [Fronts abort instead of dropping](/quest/m1/front-abort-codes.md) - a closed or withdrawn broadcast ends subscriptions with Unroutable or Cancel, never internal error - [Untimed lite-07](/quest/m1/lite-untimed.md) - lite-07 carries an untimed track in both languages; lite-05/06 write send time - [A watch and publish release ships assets()](/quest/m1/assets-release.md) - the release that lets the sites host the worklets - [Dogfood hosted worklets](/quest/m1/dogfood-assets.md) - the moq.pro dashboard hosts the worklets and calls `assets()` after the release diff --git a/quest/m1/cross-node-flapping.md b/quest/m1/cross-node-flapping.md index e3a373cf1c..caad7bfb07 100644 --- a/quest/m1/cross-node-flapping.md +++ b/quest/m1/cross-node-flapping.md @@ -2,8 +2,9 @@ ## Goal -A time-mocked drill in `rs/moq-relay/tests` with three or more clustered -relays: a publisher on one node cleanly closes a broadcast, and a client +A time-mocked drill with three or more clustered relays, extending +`rs/moq-net/tests/mesh_withdraw.rs` (which already checks one retraction on a +34-node partial mesh) or added beside it: a publisher on one node cleanly closes a broadcast, and a client watching its prefix on another node sees one retraction, with no re-announce, for that path. The drill runs against `main` and lands as a regression test. @@ -19,7 +20,9 @@ Loss and FETCH were clean in every run (about 31k FETCHes, all ok). - Decided 2026-10-11: `main` likely already handles this (the report ran on the release), so reproduce before fixing. Run the drill on the - `moq-relay-v0.17.2` tree too, to confirm it catches the reported toggling. + `moq-relay-v0.17.2` tree too, to confirm it catches the reported toggling; + a passing `main` drill counts as evidence only after that positive control + fails on the release. - Suspected mechanism, unconfirmed: path hunting in today's lite-06 path vector. A peer's re-announce of a withdrawn prefix clears its withdrawal (`reannounced` in `rs/moq-net/src/model/origin.rs`) and revives stale @@ -48,4 +51,4 @@ Loss and FETCH were clean in every run (about 31k FETCHes, all ok). ## Related - [Routes and announces](/quest/m1/cluster-routing/routes.md) - the lite-07 redesign that ends path hunting by construction -- [Fronts abort instead of dropping](/quest/m1/front-abort-codes.md) - the `internal error` replies seen during the same churn +- [Dropped sources](/quest/m1/dropped-sources.md) - the `internal error` replies seen during the same churn diff --git a/quest/m1/dropped-sources.md b/quest/m1/dropped-sources.md index 4c20212884..558929a177 100644 --- a/quest/m1/dropped-sources.md +++ b/quest/m1/dropped-sources.md @@ -23,6 +23,26 @@ end carries no cause since #4031, so only track errors are in scope. neither document nor test `is_auth` for a stream-scoped Unauthorized. Each should surface Unauthorized. +- Decided 2026-10-11 from [#4349](https://github.com/moq-dev/moq/issues/4349), + where a clean publisher close ended an in-flight subscription with + `internal error` and subscribes during cross-node churn answered the same. + `Dropped` has no arm in `From<&Error> for StreamError`, so it falls through + the `_ => Internal` catch-all to 0x0. + - Fronts end with `Error::Dropped` today (`model/front.rs`: `Retired`, + `Closed`, a resolved front whose route leaves the table, `source_closed`). + Requests and tracks a front actually aborts end with `Unroutable` instead, + or `Cancel` on a local origin `Retired` or `Closed`. + - Tracks in flight are not aborted: `Action::End` in `model/origin.rs` + leaves them on their copy, which drains (ANNOUNCE_END does not disturb + in-flight subscriptions) and ends with the source's own error. Trace that + copy's end for the reported case rather than remapping the front. + - Remove the `_ => Internal` catch-alls in both `From<&Error>` conversions + (`StreamError` and `SessionError`); `Dropped` stays `Internal` + deliberately, since it then marks a real bug. + - Regression: a relay test closes a broadcast under an active subscription + and expects the source's error, never `internal error`, and a subscribe + against a withdrawn route expects `Unroutable`. + Public API: none expected; error values consumers observe change. Wire: none. ## Required diff --git a/quest/m1/front-abort-codes.md b/quest/m1/front-abort-codes.md deleted file mode 100644 index f91a7fcb4d..0000000000 --- a/quest/m1/front-abort-codes.md +++ /dev/null @@ -1,42 +0,0 @@ -# [S] Fronts abort instead of dropping - -## Goal - -A subscription whose broadcast's source closes, whose route is withdrawn, -or whose origin shuts down ends with a deliberate stream code, never -`internal error`. The local-to-wire error conversions in -`rs/moq-net/src/error.rs` match every `Error` variant explicitly, so a new -variant cannot hide behind a catch-all. - -From [#4349](https://github.com/moq-dev/moq/issues/4349): a clean publisher -close ended the subscriber's in-flight subscription with `internal error`, -and subscribes during cross-node churn answered the same. - -## Plan - -Decided 2026-10-11: - -- `Error::Dropped` means a producer went away without aborting, which is a - bug, not a routine end. Fronts end with it today - (`rs/moq-net/src/model/front.rs`: `Event::Retired`, `Event::Closed`, a - resolved front whose route leaves the table, `source_closed`), and - `Action::End` in `origin.rs` passes it to every unfinished track; it has - no arm in `From<&Error> for StreamError`, so it reaches the wire as - `internal error` (0x0). -- Codes: a closed source or withdrawn route ends with `Unroutable`, the same - answer a fresh subscribe already gets once the broadcast is gone. A local - origin `Retired` or `Closed` ends with `Cancel`. A track the publisher - finished still ends with FIN. -- Remove the `_ => Internal` catch-alls in both `From<&Error>` conversions - (`StreamError` and `SessionError`). `Dropped` stays `Internal` - deliberately, since it now marks a real bug. -- No wire change: every code already exists. Check `js/net` for the same - catch-all only if it has an equivalent conversion. - -Tests: front unit tests assert the new codes in place of -`Action::End { err: Error::Dropped }`, and a relay test closes a broadcast -under an active subscription and expects `Unroutable`, not `internal error`. - -## Related - -- [Reproduce cross-node announce flapping](/quest/m1/cross-node-flapping.md) - the churn where subscribes answered `internal error` From e6fd08a6cbfd4b73c6a1dc9ff522099f6411023f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 12:47:26 -0700 Subject: [PATCH 3/3] quest(m1): tighten the dropped-sources regression and the flapping link Co-Authored-By: Claude Opus 5.5 --- quest/m1/cluster-routing/README.md | 2 +- quest/m1/dropped-sources.md | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/quest/m1/cluster-routing/README.md b/quest/m1/cluster-routing/README.md index 0828499680..84d4ea4b0b 100644 --- a/quest/m1/cluster-routing/README.md +++ b/quest/m1/cluster-routing/README.md @@ -143,5 +143,5 @@ Once every child has landed: - [P2P](/quest/m3/p2p/README.md) - browser and native peers that become routing nodes over this layer - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - the importer half of a redundant pair under one explicit epoch - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - a redundant pair shares one epoch -- [Reproduce cross-node announce flapping](/quest/m1/cross-node-flapping.md) - #4349's post-close announce toggling across nodes, reproduced on main +- [Reproduce cross-node announce flapping](/quest/m1/cross-node-flapping.md) - a drill for #4349's post-close announce toggling across nodes, checked on the release and on main - [Routing cost domains](/quest/m3/routing-cost-domains.md) - policy and aggregation at boundaries between operators diff --git a/quest/m1/dropped-sources.md b/quest/m1/dropped-sources.md index 558929a177..c185cc6188 100644 --- a/quest/m1/dropped-sources.md +++ b/quest/m1/dropped-sources.md @@ -40,8 +40,9 @@ end carries no cause since #4031, so only track errors are in scope. (`StreamError` and `SessionError`); `Dropped` stays `Internal` deliberately, since it then marks a real bug. - Regression: a relay test closes a broadcast under an active subscription - and expects the source's error, never `internal error`, and a subscribe - against a withdrawn route expects `Unroutable`. + whose source copy ends with a specific non-clean error, and expects that + exact error, never `internal error`; a subscribe against a withdrawn + route expects `Unroutable`. Public API: none expected; error values consumers observe change. Wire: none.