quest: resumed groups and SUBSCRIBE_DROP (#4392, #4365)

quest/m1/README.md

@@ -35,7 +35,9 @@ transport, benchmark tooling); worktrees isolate commits, not semantics.
 - [kio waiter overflow](/quest/m1/kio-waiter-lost.md) - a retained `Waiter` past 8 lists stops adding a duplicate entry to lists it already recorded
 - [Capture re-anchor](/quest/m1/capture-reanchor.md) - a repeating or restarting device clock never rewinds native capture during a fast backlog drain
 - [Splice edge cases](/quest/m1/splice-edges.md) - an unstamped successor, a pruned segment's boundary group, and a warm head during a takeover are each handled correctly
+- [Resumed groups](/quest/m1/resume-latest.md) - a half-delivered group ends once the new copy is past it, so a group-only reader never parks after a mid-group failover
 - [Track tail hardening](/quest/m1/track-tail-hardening.md) - Rust and JS wait out a track's tail by the same rules, with the known hang, count, truncation, grace, and memory holes closed
+- [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - every stream group in a lite subscription arrives or is dropped by name, and lite-07 drops its stream count for it
 - [Session death parity](/quest/m1/session-death.md) - a local close ends tracks cleanly in both languages, and JS group readers see the session's error on session death
 - [moqsrc stop](/quest/m1/moqsrc-stop.md) - moqsrc's stop blocks until its session ends, without deadlocking on a blocked pad push
 - [More tests under load](/quest/m1/test-flakes-2.md) - the second round of load-only failures, fixed at the cause

quest/m1/resume-latest.md

@@ -0,0 +1,74 @@
+# [S] A resumed group ends once the new copy is past it
+
+## Goal
+
+A reader that awaits only the in-flight group's `read_frame` never parks forever
+after its source fails mid-group. Once the track fails over to a new copy, the
+half-delivered group either completes from that copy or ends with an error, on
+every lite version and for local and remote sources alike. The one exception
+is a copy that later drops the group it is serving; that waits on
+[SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md).
+
+Two reports, one mechanism:
+
+- #4392: a publisher aborts a track mid-group over a session. On lite-03/04 the
+  far subscriber's in-flight `read_frame` stays pending forever; on lite-05/06
+  it errors.
+- #4365: a publisher aborts a track mid-group and re-creates it under the same
+  name in one origin. The origin reader parks in `read_frame` while a direct
+  reader gets the abort.
+
+## Plan
+
+Why it hangs: `track_ended` in `rs/moq-net/src/model/front.rs` treats a copy
+that dies after delivering as failover and splices a new copy. The resumed
+`resume::Group` then waits in `track.poll_peek_group`
+(`rs/moq-net/src/model/resume.rs`) on that copy, but never asks it for
+anything. Only `resume::Subscriber::apply`, driven by `recv_group`, subscribes
+to a new copy. On lite-03/04 the re-query accepts the track without asking the
+peer (no TRACK stream), so no SUBSCRIBE goes out and the refusal never comes
+back. An in-process copy declares no start, so its peek answers "not yet"
+forever. The doc comment on `resume::Group` promises that a group reader alone
+picks up the continuation; that promise is what breaks.
+
+Decided:
+
+- While a resumed group waits on a copy, it holds a subscription on that copy
+  with `start = latest`, the same shape as a fresh subscriber. A refusal ends
+  the group with that error.
+- If the copy's latest group is past the resumed group, the group ends with an
+  error at once. A same-name re-create is a continuation, so group 2 of the old
+  track ends and group 3 of the new one arrives. No special case for local
+  broadcasts.
+- If the resumed group is the copy's latest, it reads the rest from that
+  subscription. A copy that later drops it says so with SUBSCRIBE_DROP once
+  [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) lands; until then that case
+  waits as today.
+- No FETCH, not even where the version supports it. Fetching older groups from
+  the new copy may come later as its own quest.
+- Don't abort the old group eagerly on failover instead: long-lived groups
+  like a catalog must survive a failover.
+
+`poll_finished` needs the same subscription as `poll_current`. Drop the
+subscription once the wait resolves.
+
+A prototype of the subscription half (about 20 lines in `resume.rs`) turned
+all eight #4392 cases into errors with every moq-net test passing.
+
+Regression tests, on a paused clock (`#[tokio::test(start_paused = true)]`):
+
+- The #4392 repro as `rs/moq-net/tests/track_abort_over_session.rs`: abort
+  mid-group over one and two hops on lite-03 through lite-06; the in-flight
+  read errors.
+- The #4365 repro: abort mid-group 2, re-create at group 3 through the origin;
+  group 2 ends with an error and group 3 arrives.
+
+## Closes
+
+- [#4392](https://github.com/moq-dev/moq/issues/4392) - close this issue when the quest finishes
+- [#4365](https://github.com/moq-dev/moq/issues/4365) - close this issue when the quest finishes
+
+## Related
+
+- [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - tells a resumed latest group when the new copy dropped it
+- [Splice edge cases](/quest/m1/splice-edges.md) - the same splice code and test harness

quest/m1/subscribe-drop.md

@@ -0,0 +1,50 @@
+# [L] SUBSCRIBE_DROP accounts for every group
+
+## Goal
+
+A lite subscriber can tell "not yet" from "never" for every stream group in
+its subscription. Each sequence in range either arrives on a Group Stream, is
+sent as a datagram, or is named by a SUBSCRIBE_DROP, including sequences the
+publisher skipped. Only a lost datagram stays unaccounted. Rust and
+`@moq/net` publishers send it on every lite version that has it, and moq-lite-07
+brings it back in place of `Stream Count`.
+
+## Plan
+
+Today SUBSCRIBE_DROP is on the wire for lite-03 through lite-06 and the Rust
+subscriber accounts for it, but no publisher sends it. lite-07 (still
+`moq-lite-07-wip`, unpublished) removed it for a `Stream Count` on
+SUBSCRIBE_END (#4224).
+
+Decided:
+
+- lite-07 restores SUBSCRIBE_DROP and removes `Stream Count`. With every
+  sequence accounted for, the count is redundant; one mechanism instead of two.
+  A reset whose header may be lost is covered by a DROP.
+- A reliable reset ([Reliable stream reset](/quest/m1/quic/reliable-reset.md))
+  that keeps the stream header acts as a one-group drop, an optimization over
+  sending the DROP.
+- Publishers send SUBSCRIBE_DROP on lite-03 through lite-06 too: for every group
+  in range they won't deliver (expired, deprioritized, or reset without its
+  header delivered) and for every explicit gap. Publishers that skip sequences
+  (`cut` and group discontinuities in the media layers) must mark the gap so
+  the net layer can drop it.
+- Datagram groups stay best effort. A publisher counts a datagram as
+  delivered, so a lost one leaves an uncovered hole that waits out the tail
+  grace, as today.
+- A resumed group ([Resumed groups](/quest/m1/resume-latest.md)) that is the
+  new copy's latest ends with the DROP's error when the copy drops it.
+
+Update `drafts/draft-lcurley-moq-lite.md` (SUBSCRIBE_DROP, SUBSCRIBE_END, the
+lite-07 changelog), `doc/concept/moq-lite.md`, and the Rust and JS lite
+publishers, subscribers, and tail accounting. Run `just drafts check` and
+`just test interop --all`.
+
+Regression tests: a publisher that expires a group, skips a sequence, and
+resets a stream before its header; on each version the subscriber settles
+without waiting out the grace.
+
+## Related
+
+- [Track tail hardening](/quest/m1/track-tail-hardening.md) - the same tail accounting, in both languages
+- [Track tail interop](/quest/m1/track-tail-interop.md) - the Rust-JS proof of the lite-07 drop case

quest/m1/track-tail-hardening.md

@@ -53,9 +53,9 @@ decide in the PR, applying the choice to both languages:
   and the end waits out the whole grace (JS was reported to wait and Rust
   not, but Rust's `covers(owed)` reads the same way despite the comment in
   `route_datagram`; confirm with a test first). Recommendation: no, datagrams
-  are best effort. On lite-07 the SUBSCRIBE_END stream count (#4224)
-  settles without looking at sequences, which makes this moot there; older versions keep the
-  grace as the documented stopgap.
+  are best effort. [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) decided
+  the same: datagram groups are not owed, so an uncovered hole waits out the
+  grace on every version.
 - **Does a group at or past the declared end abort the track, or only that
   group?** Rust aborts the track with `ProtocolViolation`; JS aborts the group
   and ends clean. Recommendation: abort the track in both. The peer
@@ -64,7 +64,8 @@ decide in the PR, applying the choice to both languages:
   permanent gap for the life of the subscription, and so does JS's. A gap
   older than the grace can no longer be waited for, so folding it in as
   accounted loses nothing. Recommendation: that, which bounds the ranges by
-  the gaps inside the grace window; lite-07 needs only the stream count.
+  the gaps inside the grace window. Once publishers send SUBSCRIBE_DROP for
+  every gap, only datagram holes remain.
 
 Regression tests go in `rs/moq-net/tests/track_tail.rs` (its `hold_unis`
 mock makes the reorder deterministic) and the JS tail tests. Wire output only
@@ -74,4 +75,5 @@ drafts already require.
 ## Related
 
 - [Track tail interop](/quest/m1/track-tail-interop.md) - the Rust-JS check that both sides now agree
+- [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - every group is a stream or a drop, replacing lite-07's stream count
 - [Session death](/quest/m1/session-death.md) - how a tail ends when the session dies under it

quest/m1/track-tail-interop.md

@@ -25,24 +25,24 @@ What stood in the way when the Rust half landed:
   first frame. It needs a mode that reads a track to its end and reports how it
   ended and which groups it saw.
 
-Also cover the stream count:
+Also cover the drop case:
 
-- On moq-lite-07 add a count case: both subscribers settle on the
-  SUBSCRIBE_END stream count (#4224), so a group the publisher skipped or
-  never opened ends the track without waiting out the grace. Folded in from
-  the lite-count-settle quest, whose local Rust and JS regressions landed;
-  this Rust-JS case was all it had left.
+- On moq-lite-07 add a drop case: both subscribers settle on SUBSCRIBE_DROP,
+  so a group the publisher skipped or never opened ends the track without
+  waiting out the grace. lite-07 replaces the SUBSCRIBE_END stream count
+  (#4224) with it.
 - The harness exposed a relay start-floor defect: when a newer group arrives
-  first, earlier in-flight groups can be lost. #4387 fixes it, so the count
+  first, earlier in-flight groups can be lost. #4387 fixes it, so the drop
   proof waits on it.
 
 QUIC on localhost rarely reorders, so this is a smoke check that the end is
 delivered and clean. The ordering race itself stays in the unit tests.
 
 ## Required
 
+- [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - publishers name every group they won't deliver, which the lite-07 case checks
 - #4387 merges: a relayed subscription resolves its start from its source, so earlier in-flight groups are not lost (it adds quest/m1/relay-late-joiner-history.md)
 
 ## Related
 
-- [Reliable stream reset](/quest/m1/quic/reliable-reset.md) - makes the count exact by keeping a reset stream's header
+- [Reliable stream reset](/quest/m1/quic/reliable-reset.md) - keeps a reset stream's header, so the reset acts as a one-group drop