diff --git a/quest/m2/README.md b/quest/m2/README.md index 9aaa4dc6fd..45765ac924 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -105,6 +105,8 @@ waiting on an upstream release waits in [m4](/quest/m4/README.md). - [Windows capture parity](/quest/m2/capture-windows.md) - system audio and a settled app-capture policy - [Linux capture parity](/quest/m2/capture-linux.md) - Wayland window/system-audio capture with explicit display-selection and app-capture limits - [Audio capture time](/quest/m2/audio-capture-time.md) - native audio stamps a buffer's capture instant, not when the driver reads it +- [CMAF frame timestamp](/quest/m2/cmaf-frame-timestamp.md) - CMAF decoders time samples from the moq-lite frame timestamp, using `tfdt` only within the fragment +- [Shared import clock](/quest/m2/shared-clock.md) - an importer joining a clock already in use offsets its PTS instead of moving it, so captures and imports share one timeline - [X11 capture transport](/quest/m2/x11-capture-shm.md) - move X11 capture to shared memory and RandR events instead of a per-frame socket copy - [Egress profile](/quest/m2/quic-egress-profile.md) - measure relay send-path syscalls, pacing bursts, and allocations before optimizing any of them - [SEI separation study](/quest/m2/sei.md) - measure whether separating SEI saves enough, or has a metadata-only consumer, to justify a split diff --git a/quest/m2/cmaf-frame-timestamp.md b/quest/m2/cmaf-frame-timestamp.md new file mode 100644 index 0000000000..42abb97467 --- /dev/null +++ b/quest/m2/cmaf-frame-timestamp.md @@ -0,0 +1,38 @@ +# [S] CMAF decoders time samples from the frame timestamp + +## Goal + +A CMAF fragment decodes at its moq-lite frame timestamp, even when its `tfdt` +says otherwise. Today both decoders take sample times from `tfdt` and ignore +the frame timestamp: `fmp4::decode` in `rs/moq-mux/src/container/fmp4/mod.rs` +and `Format.decode` in `js/hang/src/container/cmaf/format.ts`. A publisher +that moves a passthrough track onto another timeline (an importer joining a +shared clock, ad insertion splicing unrelated PTS bases) would play at the +source's PTS. + +## Plan + +Decided (2026-10-01): the frame timestamp is the broadcast timeline. A +payload's own timestamps are only relative within the frame. Rewriting `tfdt` +on the publisher was rejected: it can force a v0 to v1 box upgrade, which +shifts `trun` data offsets. + +- A sample's time is the frame timestamp plus its PTS offset from the + fragment's first sample. The encoder already stamps the frame with that + first sample's PTS (`fmp4::encode`), so today's output decodes unchanged. +- Rust's decode gets the frame timestamp from the caller. The fMP4 exporter + and moq-hls re-fragment decoded frames, so they follow without changes; + confirm with a test. +- Check the TS exporter's verbatim carriage, which keeps PES PTS inside the + payload. Rewrite those from the frame timestamp, or refuse a mismatch. +- `main`: a behavior fix with no API change. + +Test: in Rust and JS, a fragment whose `tfdt` disagrees with its frame +timestamp decodes at the frame timestamp, with B-frame offsets preserved. + +Public API: none. Wire: none; this states what the frame timestamp already +means. + +## Related + +- [Shared clock](/quest/m2/shared-clock.md) - the first publisher to rely on it diff --git a/quest/m2/shared-clock.md b/quest/m2/shared-clock.md new file mode 100644 index 0000000000..1eb45c0949 --- /dev/null +++ b/quest/m2/shared-clock.md @@ -0,0 +1,68 @@ +# [M] One timeline for every source on a catalog + +## Goal + +Every source sharing a catalog (captures, and any number of container +importers) lands on one forward timeline, whichever starts first. Today the +first importer's first frame re-anchors the catalog clock +(`catalog::Producer::anchor`). A capture that started earlier has copied the +old clock, and a second importer's anchor is a no-op, so it publishes its own +PTS base on a clock it didn't place. Either way the sources drift apart. Following the new clock doesn't help: sources near PTS 0 step it +backwards, and `container::Producer::write` refuses a group below the last one +(`TimestampRewind`), even across a discontinuity. + +## Plan + +Follow-up of [#4668](https://github.com/moq-dev/moq/pull/4668), which moved +data tracks onto the anchored clock. The anchor exists only on `dev`, so this +targets `dev`. + +Decided (2026-10-01): + +- The clock never moves once something stamps with it. The public + `catalog::Producer::clock()` fixes the mapping (sets `anchored`). Crate-internal + readers, like #4668's `Listing`, read the state without fixing it. +- An importer anchors once, on its first frame, and gets back an offset: zero + if it placed the mapping (PTS stays verbatim), `clock.now() - first_pts` + otherwise. Every frame from that importer, on all of its tracks, is shifted + by that offset, so its audio and video stay in sync. Separate importers get + separate offsets. +- Captures read the clock once and never re-check it. Audio already stamps by + sample count from one `clock.now()` per epoch; there is no per-frame clock + read to add. +- Rejected: captures re-read the clock per frame and mark a break when it + changes. A backward anchor would hit `TimestampRewind`, and every audio + buffer would take the catalog lock to watch for an event that should never + happen while it is live. +- The offset rewrites only the moq-lite frame timestamp, never the payload. An + fMP4 passthrough fragment keeps its source `tfdt`, so the two disagree. The + frame timestamp is the broadcast timeline; a payload's timestamps are only + relative within the frame. Ad insertion (splicing sources with unrelated PTS + bases) needs the same rule. +- m2: nothing in-tree reaches it. `moq-cli publish` runs a capture or a stdin + import, never both, and moq-ffi does not expose capture. + +Guidance: + +- The offset is signed: a stream starting at 3600 s on a clock reading 10 s + shifts down. Refuse a frame that would land below zero rather than clamp it. +- A `with_clock` catalog is fixed from the start, so its importers offset too. + Today they publish verbatim PTS on a clock they didn't place. +- Document on `catalog::Producer::clock` that taking the clock fixes it. +- Test: start a synthetic capture, then import an fMP4 starting at PTS 0. Both + tracks advance from the capture's timeline with no rewind. Also cover the + reverse order: an importer first keeps its PTS verbatim. A passthrough + fragment whose `tfdt` disagrees with its frame timestamp decodes at the frame + timestamp, in Rust and JS. + +Public API: no new items. `catalog::Producer::clock()` now fixes the mapping, +and importers no longer publish verbatim PTS when the clock was already taken. +Wire: none. + +## Required + +- [CMAF frame timestamp](/quest/m2/cmaf-frame-timestamp.md) - decoders honour an offset frame timestamp on passthrough tracks + +## Related + +- [Audio capture time](/quest/m2/audio-capture-time.md) - maps audio's capture timeline onto the broadcast clock once per open