diff --git a/quest/m1/README.md b/quest/m1/README.md index d0d397df16..e8ce8e13c6 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -35,7 +35,9 @@ transport, benchmark tooling); worktrees isolate commits, not semantics. - [kio waiter overflow](/quest/m1/kio-waiter-lost.md) - a retained `Waiter` past 8 lists stops adding a duplicate entry to lists it already recorded - [Capture re-anchor](/quest/m1/capture-reanchor.md) - a repeating or restarting device clock never rewinds native capture during a fast backlog drain - [Splice edge cases](/quest/m1/splice-edges.md) - an unstamped successor, a pruned segment's boundary group, and a warm head during a takeover are each handled correctly +- [Resumed groups](/quest/m1/resume-latest.md) - a half-delivered group ends once the new copy is past it, so a group-only reader never parks after a mid-group failover - [Track tail hardening](/quest/m1/track-tail-hardening.md) - Rust and JS wait out a track's tail by the same rules, with the known hang, count, truncation, grace, and memory holes closed +- [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - every stream group in a lite subscription arrives or is dropped by name, and lite-07 drops its stream count for it - [Session death parity](/quest/m1/session-death.md) - a local close ends tracks cleanly in both languages, and JS group readers see the session's error on session death - [moqsrc stop](/quest/m1/moqsrc-stop.md) - moqsrc's stop blocks until its session ends, without deadlocking on a blocked pad push - [More tests under load](/quest/m1/test-flakes-2.md) - the second round of load-only failures, fixed at the cause diff --git a/quest/m1/resume-latest.md b/quest/m1/resume-latest.md new file mode 100644 index 0000000000..f594bfc566 --- /dev/null +++ b/quest/m1/resume-latest.md @@ -0,0 +1,74 @@ +# [S] A resumed group ends once the new copy is past it + +## Goal + +A reader that awaits only the in-flight group's `read_frame` never parks forever +after its source fails mid-group. Once the track fails over to a new copy, the +half-delivered group either completes from that copy or ends with an error, on +every lite version and for local and remote sources alike. The one exception +is a copy that later drops the group it is serving; that waits on +[SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md). + +Two reports, one mechanism: + +- #4392: a publisher aborts a track mid-group over a session. On lite-03/04 the + far subscriber's in-flight `read_frame` stays pending forever; on lite-05/06 + it errors. +- #4365: a publisher aborts a track mid-group and re-creates it under the same + name in one origin. The origin reader parks in `read_frame` while a direct + reader gets the abort. + +## Plan + +Why it hangs: `track_ended` in `rs/moq-net/src/model/front.rs` treats a copy +that dies after delivering as failover and splices a new copy. The resumed +`resume::Group` then waits in `track.poll_peek_group` +(`rs/moq-net/src/model/resume.rs`) on that copy, but never asks it for +anything. Only `resume::Subscriber::apply`, driven by `recv_group`, subscribes +to a new copy. On lite-03/04 the re-query accepts the track without asking the +peer (no TRACK stream), so no SUBSCRIBE goes out and the refusal never comes +back. An in-process copy declares no start, so its peek answers "not yet" +forever. The doc comment on `resume::Group` promises that a group reader alone +picks up the continuation; that promise is what breaks. + +Decided: + +- While a resumed group waits on a copy, it holds a subscription on that copy + with `start = latest`, the same shape as a fresh subscriber. A refusal ends + the group with that error. +- If the copy's latest group is past the resumed group, the group ends with an + error at once. A same-name re-create is a continuation, so group 2 of the old + track ends and group 3 of the new one arrives. No special case for local + broadcasts. +- If the resumed group is the copy's latest, it reads the rest from that + subscription. A copy that later drops it says so with SUBSCRIBE_DROP once + [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) lands; until then that case + waits as today. +- No FETCH, not even where the version supports it. Fetching older groups from + the new copy may come later as its own quest. +- Don't abort the old group eagerly on failover instead: long-lived groups + like a catalog must survive a failover. + +`poll_finished` needs the same subscription as `poll_current`. Drop the +subscription once the wait resolves. + +A prototype of the subscription half (about 20 lines in `resume.rs`) turned +all eight #4392 cases into errors with every moq-net test passing. + +Regression tests, on a paused clock (`#[tokio::test(start_paused = true)]`): + +- The #4392 repro as `rs/moq-net/tests/track_abort_over_session.rs`: abort + mid-group over one and two hops on lite-03 through lite-06; the in-flight + read errors. +- The #4365 repro: abort mid-group 2, re-create at group 3 through the origin; + group 2 ends with an error and group 3 arrives. + +## Closes + +- [#4392](https://github.com/moq-dev/moq/issues/4392) - close this issue when the quest finishes +- [#4365](https://github.com/moq-dev/moq/issues/4365) - close this issue when the quest finishes + +## Related + +- [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - tells a resumed latest group when the new copy dropped it +- [Splice edge cases](/quest/m1/splice-edges.md) - the same splice code and test harness diff --git a/quest/m1/subscribe-drop.md b/quest/m1/subscribe-drop.md new file mode 100644 index 0000000000..3f93210d63 --- /dev/null +++ b/quest/m1/subscribe-drop.md @@ -0,0 +1,50 @@ +# [L] SUBSCRIBE_DROP accounts for every group + +## Goal + +A lite subscriber can tell "not yet" from "never" for every stream group in +its subscription. Each sequence in range either arrives on a Group Stream, is +sent as a datagram, or is named by a SUBSCRIBE_DROP, including sequences the +publisher skipped. Only a lost datagram stays unaccounted. Rust and +`@moq/net` publishers send it on every lite version that has it, and moq-lite-07 +brings it back in place of `Stream Count`. + +## Plan + +Today SUBSCRIBE_DROP is on the wire for lite-03 through lite-06 and the Rust +subscriber accounts for it, but no publisher sends it. lite-07 (still +`moq-lite-07-wip`, unpublished) removed it for a `Stream Count` on +SUBSCRIBE_END (#4224). + +Decided: + +- lite-07 restores SUBSCRIBE_DROP and removes `Stream Count`. With every + sequence accounted for, the count is redundant; one mechanism instead of two. + A reset whose header may be lost is covered by a DROP. +- A reliable reset ([Reliable stream reset](/quest/m1/quic/reliable-reset.md)) + that keeps the stream header acts as a one-group drop, an optimization over + sending the DROP. +- Publishers send SUBSCRIBE_DROP on lite-03 through lite-06 too: for every group + in range they won't deliver (expired, deprioritized, or reset without its + header delivered) and for every explicit gap. Publishers that skip sequences + (`cut` and group discontinuities in the media layers) must mark the gap so + the net layer can drop it. +- Datagram groups stay best effort. A publisher counts a datagram as + delivered, so a lost one leaves an uncovered hole that waits out the tail + grace, as today. +- A resumed group ([Resumed groups](/quest/m1/resume-latest.md)) that is the + new copy's latest ends with the DROP's error when the copy drops it. + +Update `drafts/draft-lcurley-moq-lite.md` (SUBSCRIBE_DROP, SUBSCRIBE_END, the +lite-07 changelog), `doc/concept/moq-lite.md`, and the Rust and JS lite +publishers, subscribers, and tail accounting. Run `just drafts check` and +`just test interop --all`. + +Regression tests: a publisher that expires a group, skips a sequence, and +resets a stream before its header; on each version the subscriber settles +without waiting out the grace. + +## Related + +- [Track tail hardening](/quest/m1/track-tail-hardening.md) - the same tail accounting, in both languages +- [Track tail interop](/quest/m1/track-tail-interop.md) - the Rust-JS proof of the lite-07 drop case diff --git a/quest/m1/track-tail-hardening.md b/quest/m1/track-tail-hardening.md index f1e912491c..74d412bd04 100644 --- a/quest/m1/track-tail-hardening.md +++ b/quest/m1/track-tail-hardening.md @@ -53,9 +53,9 @@ decide in the PR, applying the choice to both languages: and the end waits out the whole grace (JS was reported to wait and Rust not, but Rust's `covers(owed)` reads the same way despite the comment in `route_datagram`; confirm with a test first). Recommendation: no, datagrams - are best effort. On lite-07 the SUBSCRIBE_END stream count (#4224) - settles without looking at sequences, which makes this moot there; older versions keep the - grace as the documented stopgap. + are best effort. [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) decided + the same: datagram groups are not owed, so an uncovered hole waits out the + grace on every version. - **Does a group at or past the declared end abort the track, or only that group?** Rust aborts the track with `ProtocolViolation`; JS aborts the group and ends clean. Recommendation: abort the track in both. The peer @@ -64,7 +64,8 @@ decide in the PR, applying the choice to both languages: permanent gap for the life of the subscription, and so does JS's. A gap older than the grace can no longer be waited for, so folding it in as accounted loses nothing. Recommendation: that, which bounds the ranges by - the gaps inside the grace window; lite-07 needs only the stream count. + the gaps inside the grace window. Once publishers send SUBSCRIBE_DROP for + every gap, only datagram holes remain. Regression tests go in `rs/moq-net/tests/track_tail.rs` (its `hold_unis` mock makes the reorder deterministic) and the JS tail tests. Wire output only @@ -74,4 +75,5 @@ drafts already require. ## Related - [Track tail interop](/quest/m1/track-tail-interop.md) - the Rust-JS check that both sides now agree +- [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - every group is a stream or a drop, replacing lite-07's stream count - [Session death](/quest/m1/session-death.md) - how a tail ends when the session dies under it diff --git a/quest/m1/track-tail-interop.md b/quest/m1/track-tail-interop.md index 4e9bb23e53..01b0a68338 100644 --- a/quest/m1/track-tail-interop.md +++ b/quest/m1/track-tail-interop.md @@ -25,15 +25,14 @@ What stood in the way when the Rust half landed: first frame. It needs a mode that reads a track to its end and reports how it ended and which groups it saw. -Also cover the stream count: +Also cover the drop case: -- On moq-lite-07 add a count case: both subscribers settle on the - SUBSCRIBE_END stream count (#4224), so a group the publisher skipped or - never opened ends the track without waiting out the grace. Folded in from - the lite-count-settle quest, whose local Rust and JS regressions landed; - this Rust-JS case was all it had left. +- On moq-lite-07 add a drop case: both subscribers settle on SUBSCRIBE_DROP, + so a group the publisher skipped or never opened ends the track without + waiting out the grace. lite-07 replaces the SUBSCRIBE_END stream count + (#4224) with it. - The harness exposed a relay start-floor defect: when a newer group arrives - first, earlier in-flight groups can be lost. #4387 fixes it, so the count + first, earlier in-flight groups can be lost. #4387 fixes it, so the drop proof waits on it. QUIC on localhost rarely reorders, so this is a smoke check that the end is @@ -41,8 +40,9 @@ delivered and clean. The ordering race itself stays in the unit tests. ## Required +- [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - publishers name every group they won't deliver, which the lite-07 case checks - #4387 merges: a relayed subscription resolves its start from its source, so earlier in-flight groups are not lost (it adds quest/m1/relay-late-joiner-history.md) ## Related -- [Reliable stream reset](/quest/m1/quic/reliable-reset.md) - makes the count exact by keeping a reset stream's header +- [Reliable stream reset](/quest/m1/quic/reliable-reset.md) - keeps a reset stream's header, so the reset acts as a one-group drop