From 4e5151bd4b1438046acc462eae56141f2f071aa1 Mon Sep 17 00:00:00 2001 From: OneTooMany Date: Mon, 5 Oct 2026 11:40:46 +0200 Subject: [PATCH 1/3] quest: plan untimed failover, object-scope units on drafts 14-16, and a rejoin race Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 3 +- quest/m1/ietf-timestamp-units.md | 48 ++++++++++++-------- quest/m1/js-group-handover.md | 6 ++- quest/m1/js-untimed-model.md | 4 ++ quest/m1/test-flakes-2/README.md | 1 + quest/m1/test-flakes-2/rejoin-idle-race.md | 32 +++++++++++++ quest/m1/untimed-failover.md | 52 ++++++++++++++++++++++ 7 files changed, 126 insertions(+), 20 deletions(-) create mode 100644 quest/m1/test-flakes-2/rejoin-idle-race.md create mode 100644 quest/m1/untimed-failover.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 1a17f595ea..20d901be5c 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -76,6 +76,7 @@ QUIC studies there on that rule. - [Publishing never invents a timestamp](/quest/m1/publish-timestamp.md) - no Rust or binding publish API fills in a timestamp; an untimed payload goes out untimed - [JS publishing never invents a timestamp](/quest/m1/js-publish-timestamp.md) - the same in @moq/json, @moq/flate, and @moq/net - [Untimed model](/quest/m1/untimed-model.md) - moq-net frames and datagrams carry an absent timestamp to every subscriber; no receiver fills in arrival time +- [Untimed failover](/quest/m1/untimed-failover.md) - a resumed group no route continues is given up even when media time can't judge its drift - [JS untimed model](/quest/m1/js-untimed-model.md) - the same in @moq/net - [Untimed lite-07](/quest/m1/lite-untimed.md) - lite-07 encodes an absent timestamp in both languages; lite-05/06 write send time - [Plan: untimed verbatim PES](/quest/m1/plan-ts-pes-untimed.md) - decide how a verbatim TS track carries a PES that has no PTS, then write the implementation quest @@ -85,7 +86,7 @@ QUIC studies there on that rule. - [Draft-20 FETCH](/quest/m1/ietf-fetch-location.md) - a draft-20+ FETCH within one group is served from its LOCATION_FILTER, as older drafts are - [FETCH_OK properties](/quest/m1/fetch-ok-properties.md) - our FETCH_OK carries the track properties SUBSCRIBE_OK does, as draft 16+ requires - [Fetch without SUBSCRIBE](/quest/m1/ietf-fetch-only.md) - a relay fetches from an IETF upstream without subscribing, finished tracks included, with End of Track always reported -- [IETF timestamp units](/quest/m1/ietf-timestamp-units.md) - drafts 14-16 stop sending Timestamp properties they can't give units for +- [IETF timestamp units](/quest/m1/ietf-timestamp-units.md) - drafts 14-16 send an object-scope TIMESCALE beside each Timestamp, so Rust-to-Rust stays timed - [JS IETF datagrams](/quest/m1/js-ietf-datagram.md) - `@moq/net` sends and receives datagram groups over moq-transport, like Rust - [Datagrams are live-only](/quest/m1/datagram-unfetchable.md) - no FETCH, replay to a new subscriber, or cache fill ever returns a datagram group - [#2991](/quest/m1/2991-net-coalesce-dynamic-tracks-and-preserve-sequences-across.md) - one dynamic producer per track name in both languages, with the sequence namespace surviving a replacement diff --git a/quest/m1/ietf-timestamp-units.md b/quest/m1/ietf-timestamp-units.md index cb9343378e..82a92190d1 100644 --- a/quest/m1/ietf-timestamp-units.md +++ b/quest/m1/ietf-timestamp-units.md @@ -1,27 +1,39 @@ -# [XS] IETF drafts 14-16 send no timestamps without units +# [XS] IETF drafts 14-16 send object-scope timestamps ## Goal -On moq-transport drafts 14 through 16, where TIMESCALE can't be sent, the Rust -and JS publishers write no Timestamp object property, as -`drafts/draft-lcurley-moq-timestamp.md` requires ("A publisher that emits -Timestamps MUST send TIMESCALE"). +On moq-transport drafts 14 through 16, where SUBSCRIBE_OK can't carry +TIMESCALE, the Rust publisher writes an object-scope TIMESCALE beside each +Timestamp. A Rust subscriber then reads those objects as timed, so +Rust-to-Rust on those drafts keeps its timeline, and no object carries a +Timestamp without units, as `drafts/draft-lcurley-moq-timestamp.md` +requires. ## Plan -Rust sets `has_extensions: self.timescale.is_some()` -(`rs/moq-net/src/ietf/publisher.rs`), but `Properties::encode` never writes -the TIMESCALE block before draft-17 (`rs/moq-net/src/ietf/properties.rs`), so -drafts 14-16 carry timestamps without units. The Rust subscriber ignores them. -Gate the Timestamp property on a timescale actually sent. Check js/net's -`stamped` flag (`js/net/src/ietf/publisher.ts`) for the same gap. +Decided 2026-10-05 11:39 +0200: send object-scope units rather than stop +sending timestamps, which was this quest's earlier goal. #4822 already +encodes and reads them (`ObjectTime::Object`), and peers ignore object +properties they don't know. The cost is about two bytes per object. -Before landing, check whether an interop peer on draft-14 reads the Timestamp -property without TIMESCALE. If one does, ask whether to keep it. +`ObjectTime::new` (`rs/moq-net/src/ietf/publisher.rs`) picks `Track` whenever +the track has a timescale, but `Properties::encode` never writes TIMESCALE +before draft-17 (`rs/moq-net/src/ietf/properties.rs`). On drafts 14-16, pick +`Object` instead. Test that a timed frame stays timed over IETF 14 and 16, +with and without a relay, in `rs/moq-net/tests/untimed.rs`. -Fix the stale `track::Info::timescale` doc (`rs/moq-net/src/model/track.rs`), -which says IETF always falls back to local milliseconds; draft-17 and later -carry TIMESCALE. +Decided 2026-10-05 11:39 +0200: Rust only. js/net's publisher has the same +gap (`stamped` in `js/net/src/ietf/publisher.ts`). It goes to [JS untimed +model](/quest/m1/js-untimed-model.md), which owns reading object-scope units +in JS. -Wire: stops sending a property peers can't interpret, on published drafts. -Public API: none. Lands on `main`. +Fix the stale `track::Info::timescale` doc (`rs/moq-net/src/model/track.rs`) +if it still says IETF always falls back to local milliseconds. Fix +`doc/concept/standard.md` if it says drafts 14-16 deliver untimed frames. + +Wire: drafts 14-16 objects gain an object-scope TIMESCALE property, which the +timestamp draft already allows. Public API: none. Lands on `main`. + +## Required + +- [Untimed model](/quest/m1/untimed-model.md) - adds object-scope units and their receive path diff --git a/quest/m1/js-group-handover.md b/quest/m1/js-group-handover.md index 8787785850..f3492702de 100644 --- a/quest/m1/js-group-handover.md +++ b/quest/m1/js-group-handover.md @@ -21,7 +21,7 @@ old relay drops its upstream pull, the new relay subscribes upstream from scratch at the live edge. A group boundary that lands inside that window loses the group in flight: about 1 run in 5 at 100 ms groups. -Rust solves this with #4741's single-writer pump (`model/pump.rs`): each track a +Rust solves this with #4741's single-writer pump (`model/resume.rs`): each track a front serves is one producer, a route change subscribes the new route from the first frame the logical track lacks (mid-group), an open group is continued in place, duplicates are dropped by frame index, and the old route @@ -40,6 +40,10 @@ settle along the way: properties differ (timescale, retention, priority, order). Match it. - `js/watch` and `js/hang` consumers that re-subscribe on `active` changes. Check whether they still need to. +- Giving up a resumed group no route continues. Mirror Rust's rule from + [Untimed failover](/quest/m1/untimed-failover.md): when media time can't + judge its drift, give it up once the new route holds a newer group and + nothing can still fill it. Add unit coverage at the origin level against stand-in sessions, then flip `test/drain` to zero budget (drop the resubscribe loop in `drain.ts` and the diff --git a/quest/m1/js-untimed-model.md b/quest/m1/js-untimed-model.md index c5d8dc6a5f..381f0f78b4 100644 --- a/quest/m1/js-untimed-model.md +++ b/quest/m1/js-untimed-model.md @@ -27,6 +27,10 @@ Things to look out for: a track that never declared one must not claim a timeline downstream. - A FETCH keeps timestamps whenever the track or the object gives units. Only objects with neither are untimed. +- On drafts 14-16, where SUBSCRIBE_OK can't carry TIMESCALE, the publisher + writes an object-scope TIMESCALE beside each Timestamp (the `stamped` path + in `js/net/src/ietf/publisher.ts`), as Rust does per [IETF timestamp + units](/quest/m1/ietf-timestamp-units.md). - `Frame.timestamp` and `Datagram.timestamp` become optional. Update callers in js/hang and js/loc (end markers) and anything in js/watch that reads them. diff --git a/quest/m1/test-flakes-2/README.md b/quest/m1/test-flakes-2/README.md index c48c60452b..345570e472 100644 --- a/quest/m1/test-flakes-2/README.md +++ b/quest/m1/test-flakes-2/README.md @@ -39,6 +39,7 @@ Public API: none. Wire: none. - [moq-mux debounce clock](/quest/m1/test-flakes-2/mux-debounce-clock.md) - the TS export debounce test advances on the paused clock - [js/publish audio clock](/quest/m1/test-flakes-2/publish-audio-clock.md) - the audio encoder delay test runs on mock time - [WebSocket paused TLS dial](/quest/m1/test-flakes-2/websocket-paused-tls.md) - the fixed-address WebSocket tests stop pausing the clock over a real dial +- [Rejoin after idle](/quest/m1/test-flakes-2/rejoin-idle-race.md) - an IETF rejoin never gets the stale warm cache first, because the copy goes idle before the cancel ## Related diff --git a/quest/m1/test-flakes-2/rejoin-idle-race.md b/quest/m1/test-flakes-2/rejoin-idle-race.md new file mode 100644 index 0000000000..4437939535 --- /dev/null +++ b/quest/m1/test-flakes-2/rejoin-idle-race.md @@ -0,0 +1,32 @@ +# [S] Rejoin after the copy goes idle + +## Goal + +A reader rejoining a parked IETF track never gets the stale warm cache +first, and moq-tokio `broadcast_rejoin_skips_a_stale_warm_cache` +(`rs/moq-tokio/tests/broadcast.rs`) passes in every full-suite run. + +## Plan + +It failed once under a loaded `just check` ("moq-transport-17: a rejoining +reader was served the stale cache first: group 3") and passed 8/8 alone. + +Likely cause: on idle, the IETF subscriber awaits `cancel_subscribe` before +it calls `track.set_idle()` (`rs/moq-net/src/ietf/subscriber.rs`, around +line 2178 at time of writing). Draft-17 has no UNSUBSCRIBE, so the cancel is +a STOP_SENDING followed by a `close().await` that waits for the peer. +The publisher sees the cancel and its demand drops, which is what the test +waits on, about a round trip before the copy is marked idle. A resubscribe +in that gap finds a copy that still looks live. Its newest group, 3, has no +successor, so it isn't stale, and it is served. + +Decided 2026-10-05 11:39 +0200: fix the product, since a real client +rejoining in that gap gets the stale group too. Reproduce the race first, +then mark the copy idle as soon as `End::Idle` is decided, before the +awaited cancel. The test then waits on an event that comes after idle and +needs no change. Rejected: only making the test wait for the subscriber's +idle, which leaves the window open for clients. Check the other drafts and +lite for the same ordering. Add a deterministic regression test if one is +easy. + +Public API: none. Wire: none. diff --git a/quest/m1/untimed-failover.md b/quest/m1/untimed-failover.md new file mode 100644 index 0000000000..e3c56af23f --- /dev/null +++ b/quest/m1/untimed-failover.md @@ -0,0 +1,52 @@ +# [S] Give up a resumed group drift can't judge + +## Goal + +After a route failover, a resumed group that no remaining route continues is +given up even when its track's media time can't measure how far behind it is, +such as on an all-untimed track. Today a reader holding that group waits until +the track ends. + +## Plan + +Found in #4822. `Recover::poll` (`rs/moq-net/src/model/resume.rs`, around line +854 at time of writing) gives up a group the serving route can't continue +once `poll_stale` reports drift past the reader's `max_age`. `drifted` +(`model/track.rs`) needs a timed live edge and a timed successor (`reach`). +Without both, nothing convicts the group. Only the holder of that +`group::Consumer` blocks: track cursors keep handing out newer groups. The +replaced copy's lease also stays alive. On main this still worked, because +receivers stamped arrival time on untimed frames. + +Decided 2026-10-05 11:39 +0200: + +- A separate quest, not a fix inside #4822, to keep that PR reviewable. It + ships the regression for a short window, which is why this quest is ranked + right after it. +- The rule: when drift can't be measured (no timed live edge, or the + successor is untimed), give up once the serving copy holds a newer group + and no route or recovery fetch can still fill this one. This applies to + mixed tracks too, so there is one rule. No clock is involved. + Rejected: a wall-clock deadline of `max_age` after the stall, which stands + wall time in for media time, the substitution the untimed model removed. + Also rejected: leaving it and documenting it. + +Settle what "nothing pending" means against the recovery fetch +(`a_pending_recovery_fetch_does_not_disable_expiry`), so a fetch about to +fill the group isn't cut short. A FETCH reader has no budget and stays as it +is. + +Test: a model test in `resume.rs` failing over an all-untimed track with an +abandoned group, plus one where the successor is untimed on a timed track. +Both fail without the fix. + +Public API: none. Wire: none. + +## Required + +- [Untimed model](/quest/m1/untimed-model.md) - introduces untimed groups, and the stall + +## Related + +- [JS track handover](/quest/m1/js-group-handover.md) - mirrors this rule in JS +- [Wall-clock age-out](/quest/m1/cache-wall-eviction.md) - retention of untimed groups, not a blocked reader From e429e0458c40c1091451643d09ae8cc569134a44 Mon Sep 17 00:00:00 2001 From: OneTooMany Date: Mon, 5 Oct 2026 11:54:00 +0200 Subject: [PATCH 2/3] quest: js-untimed-model names its drafts 14-16 wire change Co-Authored-By: Claude Opus 5.5 --- quest/m1/js-untimed-model.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/quest/m1/js-untimed-model.md b/quest/m1/js-untimed-model.md index 381f0f78b4..161923bc3e 100644 --- a/quest/m1/js-untimed-model.md +++ b/quest/m1/js-untimed-model.md @@ -38,7 +38,8 @@ Things to look out for: Test: an untimed frame survives a JS subscribe on each receive path. Run `just test interop --all`. -Public API: breaking. Wire: none. +Public API: breaking. Wire: on drafts 14-16, timed objects gain an +object-scope TIMESCALE beside the Timestamp; nothing else changes. ## Related From ba44804b797b84bff875ce4c0d7747faf520d3c7 Mon Sep 17 00:00:00 2001 From: OneTooMany Date: Mon, 5 Oct 2026 12:26:59 +0200 Subject: [PATCH 3/3] quest: leave ietf-timestamp-units to #4822, test failover on a warm cache Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 2 +- quest/m1/ietf-timestamp-units.md | 48 ++++++++++++-------------------- quest/m1/js-untimed-model.md | 5 ++-- quest/m1/untimed-failover.md | 6 +++- 4 files changed, 27 insertions(+), 34 deletions(-) diff --git a/quest/m1/README.md b/quest/m1/README.md index 20d901be5c..7fbf04734d 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -86,7 +86,7 @@ QUIC studies there on that rule. - [Draft-20 FETCH](/quest/m1/ietf-fetch-location.md) - a draft-20+ FETCH within one group is served from its LOCATION_FILTER, as older drafts are - [FETCH_OK properties](/quest/m1/fetch-ok-properties.md) - our FETCH_OK carries the track properties SUBSCRIBE_OK does, as draft 16+ requires - [Fetch without SUBSCRIBE](/quest/m1/ietf-fetch-only.md) - a relay fetches from an IETF upstream without subscribing, finished tracks included, with End of Track always reported -- [IETF timestamp units](/quest/m1/ietf-timestamp-units.md) - drafts 14-16 send an object-scope TIMESCALE beside each Timestamp, so Rust-to-Rust stays timed +- [IETF timestamp units](/quest/m1/ietf-timestamp-units.md) - drafts 14-16 stop sending Timestamp properties they can't give units for - [JS IETF datagrams](/quest/m1/js-ietf-datagram.md) - `@moq/net` sends and receives datagram groups over moq-transport, like Rust - [Datagrams are live-only](/quest/m1/datagram-unfetchable.md) - no FETCH, replay to a new subscriber, or cache fill ever returns a datagram group - [#2991](/quest/m1/2991-net-coalesce-dynamic-tracks-and-preserve-sequences-across.md) - one dynamic producer per track name in both languages, with the sequence namespace surviving a replacement diff --git a/quest/m1/ietf-timestamp-units.md b/quest/m1/ietf-timestamp-units.md index 82a92190d1..cb9343378e 100644 --- a/quest/m1/ietf-timestamp-units.md +++ b/quest/m1/ietf-timestamp-units.md @@ -1,39 +1,27 @@ -# [XS] IETF drafts 14-16 send object-scope timestamps +# [XS] IETF drafts 14-16 send no timestamps without units ## Goal -On moq-transport drafts 14 through 16, where SUBSCRIBE_OK can't carry -TIMESCALE, the Rust publisher writes an object-scope TIMESCALE beside each -Timestamp. A Rust subscriber then reads those objects as timed, so -Rust-to-Rust on those drafts keeps its timeline, and no object carries a -Timestamp without units, as `drafts/draft-lcurley-moq-timestamp.md` -requires. +On moq-transport drafts 14 through 16, where TIMESCALE can't be sent, the Rust +and JS publishers write no Timestamp object property, as +`drafts/draft-lcurley-moq-timestamp.md` requires ("A publisher that emits +Timestamps MUST send TIMESCALE"). ## Plan -Decided 2026-10-05 11:39 +0200: send object-scope units rather than stop -sending timestamps, which was this quest's earlier goal. #4822 already -encodes and reads them (`ObjectTime::Object`), and peers ignore object -properties they don't know. The cost is about two bytes per object. +Rust sets `has_extensions: self.timescale.is_some()` +(`rs/moq-net/src/ietf/publisher.rs`), but `Properties::encode` never writes +the TIMESCALE block before draft-17 (`rs/moq-net/src/ietf/properties.rs`), so +drafts 14-16 carry timestamps without units. The Rust subscriber ignores them. +Gate the Timestamp property on a timescale actually sent. Check js/net's +`stamped` flag (`js/net/src/ietf/publisher.ts`) for the same gap. -`ObjectTime::new` (`rs/moq-net/src/ietf/publisher.rs`) picks `Track` whenever -the track has a timescale, but `Properties::encode` never writes TIMESCALE -before draft-17 (`rs/moq-net/src/ietf/properties.rs`). On drafts 14-16, pick -`Object` instead. Test that a timed frame stays timed over IETF 14 and 16, -with and without a relay, in `rs/moq-net/tests/untimed.rs`. +Before landing, check whether an interop peer on draft-14 reads the Timestamp +property without TIMESCALE. If one does, ask whether to keep it. -Decided 2026-10-05 11:39 +0200: Rust only. js/net's publisher has the same -gap (`stamped` in `js/net/src/ietf/publisher.ts`). It goes to [JS untimed -model](/quest/m1/js-untimed-model.md), which owns reading object-scope units -in JS. +Fix the stale `track::Info::timescale` doc (`rs/moq-net/src/model/track.rs`), +which says IETF always falls back to local milliseconds; draft-17 and later +carry TIMESCALE. -Fix the stale `track::Info::timescale` doc (`rs/moq-net/src/model/track.rs`) -if it still says IETF always falls back to local milliseconds. Fix -`doc/concept/standard.md` if it says drafts 14-16 deliver untimed frames. - -Wire: drafts 14-16 objects gain an object-scope TIMESCALE property, which the -timestamp draft already allows. Public API: none. Lands on `main`. - -## Required - -- [Untimed model](/quest/m1/untimed-model.md) - adds object-scope units and their receive path +Wire: stops sending a property peers can't interpret, on published drafts. +Public API: none. Lands on `main`. diff --git a/quest/m1/js-untimed-model.md b/quest/m1/js-untimed-model.md index 161923bc3e..15e25b840d 100644 --- a/quest/m1/js-untimed-model.md +++ b/quest/m1/js-untimed-model.md @@ -29,8 +29,9 @@ Things to look out for: Only objects with neither are untimed. - On drafts 14-16, where SUBSCRIBE_OK can't carry TIMESCALE, the publisher writes an object-scope TIMESCALE beside each Timestamp (the `stamped` path - in `js/net/src/ietf/publisher.ts`), as Rust does per [IETF timestamp - units](/quest/m1/ietf-timestamp-units.md). + in `js/net/src/ietf/publisher.ts`), as Rust does once #4822 lands. This + replaces the JS half of [IETF timestamp + units](/quest/m1/ietf-timestamp-units.md), which planned to send none there. - `Frame.timestamp` and `Datagram.timestamp` become optional. Update callers in js/hang and js/loc (end markers) and anything in js/watch that reads them. diff --git a/quest/m1/untimed-failover.md b/quest/m1/untimed-failover.md index e3c56af23f..af55674f5a 100644 --- a/quest/m1/untimed-failover.md +++ b/quest/m1/untimed-failover.md @@ -38,7 +38,11 @@ is. Test: a model test in `resume.rs` failing over an all-untimed track with an abandoned group, plus one where the successor is untimed on a timed track. -Both fail without the fix. +Both fail without the fix. Run each against a cold route and against one +whose copy already caches untimed groups past the resumed one, so a cursor +that skips ahead can't hide the stall. #4822 is changing `Cursor::new` so an +explicit start is honoured on untimed tracks; build on that, not on the +older jump to the latest untimed group. Public API: none. Wire: none.