From af9e7d1c809943f34a17a13ba8cf2d77a4c12847 Mon Sep 17 00:00:00 2001 From: OneTooMany Date: Thu, 1 Oct 2026 12:40:45 +0200 Subject: [PATCH 1/4] quest: plan captures stamping on the catalog's current clock Co-Authored-By: Claude Opus 5.5 --- quest/m2/README.md | 1 + quest/m2/capture-clock.md | 44 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) create mode 100644 quest/m2/capture-clock.md diff --git a/quest/m2/README.md b/quest/m2/README.md index 9aaa4dc6fd..b25c922181 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -105,6 +105,7 @@ waiting on an upstream release waits in [m4](/quest/m4/README.md). - [Windows capture parity](/quest/m2/capture-windows.md) - system audio and a settled app-capture policy - [Linux capture parity](/quest/m2/capture-linux.md) - Wayland window/system-audio capture with explicit display-selection and app-capture limits - [Audio capture time](/quest/m2/audio-capture-time.md) - native audio stamps a buffer's capture instant, not when the driver reads it +- [Capture clock](/quest/m2/capture-clock.md) - a capture sharing a catalog with an importer stamps on the anchored clock, even when it started first - [X11 capture transport](/quest/m2/x11-capture-shm.md) - move X11 capture to shared memory and RandR events instead of a per-frame socket copy - [Egress profile](/quest/m2/quic-egress-profile.md) - measure relay send-path syscalls, pacing bursts, and allocations before optimizing any of them - [SEI separation study](/quest/m2/sei.md) - measure whether separating SEI saves enough, or has a metadata-only consumer, to justify a split diff --git a/quest/m2/capture-clock.md b/quest/m2/capture-clock.md new file mode 100644 index 0000000000..d481ff665a --- /dev/null +++ b/quest/m2/capture-clock.md @@ -0,0 +1,44 @@ +# [S] Captures stamp on the catalog's current clock + +## Goal + +A moq-audio or moq-video capture sharing a catalog with a container importer +stamps on the clock the importer's first frame anchors, even when the capture +started first. Today both copy `catalog.clock()` once at startup +(`rs/moq-audio/src/encode/capture.rs`, `rs/moq-video/src/encode/producer.rs`), +so after the anchor their frames sit on the catalog's starting clock while the +importer's sit on the anchored one. + +## Plan + +Follow-up of [#4668](https://github.com/moq-dev/moq/pull/4668), which fixed +the same stale copy for JSON and binary data tracks. The first-frame anchor +exists only on `dev`, so this targets `dev`. + +Decided (2026-10-01): + +- Re-read `catalog.clock()` when stamping, rather than make `moq_mux::Clock` a + shared live handle. Same shape as #4668, with no public API change. A shared + handle would also fix copies applications hold, but it breaks a public + `Copy` type and every stamper still needs to detect the break. +- m2: nothing in-tree reaches it. `moq-cli publish` runs a capture or a stdin + import, never both, and moq-ffi does not expose capture. Only a Rust caller + passing `publish_capture` a catalog shared with an importer hits it. +- One quest for both crates: the same pattern and test. + +Guidance: + +- A changed wall mapping is a break in the capture's timeline. Mark it the way + each already marks a stop (video `discontinuity`, audio `reset_epoch`), and + recompute video's per-open `capture_epoch` then, not at the next open. +- Document on `catalog::Producer::clock` that the value is a snapshot an + importer's first frame replaces, for applications that stamp with it. +- Test with a default-clock catalog anchored while a synthetic capture runs. + The existing `mod clock` fixtures pin the clock with `with_clock`, which is + never re-anchored. + +Public API: none. Wire: none. + +## Related + +- [Audio capture time](/quest/m2/audio-capture-time.md) - maps audio's capture timeline onto the broadcast clock once per open, which must recompute on an anchor too From b22db1e457bb58e4a3f42b6a3f54030b89e42162 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 1 Oct 2026 14:50:26 -0700 Subject: [PATCH 2/4] quest: importers offset onto a clock already in use Replaces re-reading the clock per frame: a backward anchor hits TimestampRewind, and audio would lock the catalog on every buffer. Co-Authored-By: Claude Opus 5.5 --- quest/m2/README.md | 2 +- quest/m2/capture-clock.md | 44 ------------------------------ quest/m2/shared-clock.md | 57 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 58 insertions(+), 45 deletions(-) delete mode 100644 quest/m2/capture-clock.md create mode 100644 quest/m2/shared-clock.md diff --git a/quest/m2/README.md b/quest/m2/README.md index b25c922181..358b31a0c8 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -105,7 +105,7 @@ waiting on an upstream release waits in [m4](/quest/m4/README.md). - [Windows capture parity](/quest/m2/capture-windows.md) - system audio and a settled app-capture policy - [Linux capture parity](/quest/m2/capture-linux.md) - Wayland window/system-audio capture with explicit display-selection and app-capture limits - [Audio capture time](/quest/m2/audio-capture-time.md) - native audio stamps a buffer's capture instant, not when the driver reads it -- [Capture clock](/quest/m2/capture-clock.md) - a capture sharing a catalog with an importer stamps on the anchored clock, even when it started first +- [Shared import clock](/quest/m2/shared-clock.md) - an importer joining a clock already in use offsets its PTS instead of moving it, so captures and imports share one timeline - [X11 capture transport](/quest/m2/x11-capture-shm.md) - move X11 capture to shared memory and RandR events instead of a per-frame socket copy - [Egress profile](/quest/m2/quic-egress-profile.md) - measure relay send-path syscalls, pacing bursts, and allocations before optimizing any of them - [SEI separation study](/quest/m2/sei.md) - measure whether separating SEI saves enough, or has a metadata-only consumer, to justify a split diff --git a/quest/m2/capture-clock.md b/quest/m2/capture-clock.md deleted file mode 100644 index d481ff665a..0000000000 --- a/quest/m2/capture-clock.md +++ /dev/null @@ -1,44 +0,0 @@ -# [S] Captures stamp on the catalog's current clock - -## Goal - -A moq-audio or moq-video capture sharing a catalog with a container importer -stamps on the clock the importer's first frame anchors, even when the capture -started first. Today both copy `catalog.clock()` once at startup -(`rs/moq-audio/src/encode/capture.rs`, `rs/moq-video/src/encode/producer.rs`), -so after the anchor their frames sit on the catalog's starting clock while the -importer's sit on the anchored one. - -## Plan - -Follow-up of [#4668](https://github.com/moq-dev/moq/pull/4668), which fixed -the same stale copy for JSON and binary data tracks. The first-frame anchor -exists only on `dev`, so this targets `dev`. - -Decided (2026-10-01): - -- Re-read `catalog.clock()` when stamping, rather than make `moq_mux::Clock` a - shared live handle. Same shape as #4668, with no public API change. A shared - handle would also fix copies applications hold, but it breaks a public - `Copy` type and every stamper still needs to detect the break. -- m2: nothing in-tree reaches it. `moq-cli publish` runs a capture or a stdin - import, never both, and moq-ffi does not expose capture. Only a Rust caller - passing `publish_capture` a catalog shared with an importer hits it. -- One quest for both crates: the same pattern and test. - -Guidance: - -- A changed wall mapping is a break in the capture's timeline. Mark it the way - each already marks a stop (video `discontinuity`, audio `reset_epoch`), and - recompute video's per-open `capture_epoch` then, not at the next open. -- Document on `catalog::Producer::clock` that the value is a snapshot an - importer's first frame replaces, for applications that stamp with it. -- Test with a default-clock catalog anchored while a synthetic capture runs. - The existing `mod clock` fixtures pin the clock with `with_clock`, which is - never re-anchored. - -Public API: none. Wire: none. - -## Related - -- [Audio capture time](/quest/m2/audio-capture-time.md) - maps audio's capture timeline onto the broadcast clock once per open, which must recompute on an anchor too diff --git a/quest/m2/shared-clock.md b/quest/m2/shared-clock.md new file mode 100644 index 0000000000..d9351ea73f --- /dev/null +++ b/quest/m2/shared-clock.md @@ -0,0 +1,57 @@ +# [M] One timeline for every source on a catalog + +## Goal + +Every source sharing a catalog (captures, and any number of container +importers) lands on one forward timeline, whichever starts first. Today the +first importer's first frame re-anchors the catalog clock +(`catalog::Producer::anchor`). A capture that started earlier has copied the +old clock, and a second importer's anchor is a no-op, so it publishes its own +PTS base on a clock it didn't place. Either way the sources drift apart. Following the new clock doesn't help: sources near PTS 0 step it +backwards, and `container::Producer::write` refuses a group below the last one +(`TimestampRewind`), even across a discontinuity. + +## Plan + +Follow-up of [#4668](https://github.com/moq-dev/moq/pull/4668), which moved +data tracks onto the anchored clock. The anchor exists only on `dev`, so this +targets `dev`. + +Decided (2026-10-01): + +- The clock never moves once something stamps with it. The public + `catalog::Producer::clock()` fixes the mapping (sets `anchored`). Crate-internal + readers, like #4668's `Listing`, read the state without fixing it. +- An importer anchors once, on its first frame, and gets back an offset: zero + if it placed the mapping (PTS stays verbatim), `clock.now() - first_pts` + otherwise. Every frame from that importer, on all of its tracks, is shifted + by that offset, so its audio and video stay in sync. Separate importers get + separate offsets. +- Captures read the clock once and never re-check it. Audio already stamps by + sample count from one `clock.now()` per epoch; there is no per-frame clock + read to add. +- Rejected: captures re-read the clock per frame and mark a break when it + changes. A backward anchor would hit `TimestampRewind`, and every audio + buffer would take the catalog lock to watch for an event that should never + happen while it is live. +- m2: nothing in-tree reaches it. `moq-cli publish` runs a capture or a stdin + import, never both, and moq-ffi does not expose capture. + +Guidance: + +- The offset is signed: a stream starting at 3600 s on a clock reading 10 s + shifts down. Refuse a frame that would land below zero rather than clamp it. +- A `with_clock` catalog is fixed from the start, so its importers offset too. + Today they publish verbatim PTS on a clock they didn't place. +- Document on `catalog::Producer::clock` that taking the clock fixes it. +- Test: start a synthetic capture, then import an fMP4 starting at PTS 0. Both + tracks advance from the capture's timeline with no rewind. Also cover the + reverse order: an importer first keeps its PTS verbatim. + +Public API: no new items. `catalog::Producer::clock()` now fixes the mapping, +and importers no longer publish verbatim PTS when the clock was already taken. +Wire: none. + +## Related + +- [Audio capture time](/quest/m2/audio-capture-time.md) - maps audio's capture timeline onto the broadcast clock once per open From b871ba7933f3ab4a0830ff473011515c4c2ad410 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 1 Oct 2026 14:57:10 -0700 Subject: [PATCH 3/4] quest: frame timestamp is the timeline, payload tfdt is relative Co-Authored-By: Claude Opus 5.5 --- quest/m2/shared-clock.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/quest/m2/shared-clock.md b/quest/m2/shared-clock.md index d9351ea73f..31ef9c1dc3 100644 --- a/quest/m2/shared-clock.md +++ b/quest/m2/shared-clock.md @@ -34,6 +34,11 @@ Decided (2026-10-01): changes. A backward anchor would hit `TimestampRewind`, and every audio buffer would take the catalog lock to watch for an event that should never happen while it is live. +- The offset rewrites only the moq-lite frame timestamp, never the payload. An + fMP4 passthrough fragment keeps its source `tfdt`, so the two disagree. The + frame timestamp is the broadcast timeline; a payload's timestamps are only + relative within the frame. Ad insertion (splicing sources with unrelated PTS + bases) needs the same rule. - m2: nothing in-tree reaches it. `moq-cli publish` runs a capture or a stdin import, never both, and moq-ffi does not expose capture. @@ -43,10 +48,17 @@ Guidance: shifts down. Refuse a frame that would land below zero rather than clamp it. - A `with_clock` catalog is fixed from the start, so its importers offset too. Today they publish verbatim PTS on a clock they didn't place. +- Both CMAF decoders stamp samples from `tfdt` and ignore the frame timestamp: + `fmp4::decode` in moq-mux and `Format.decode` in + `js/hang/src/container/cmaf/format.ts`. Rebase them: a sample's time is the + frame timestamp plus its offset from the fragment's first sample. Check the + fMP4 exporter does the same. - Document on `catalog::Producer::clock` that taking the clock fixes it. - Test: start a synthetic capture, then import an fMP4 starting at PTS 0. Both tracks advance from the capture's timeline with no rewind. Also cover the - reverse order: an importer first keeps its PTS verbatim. + reverse order: an importer first keeps its PTS verbatim. A passthrough + fragment whose `tfdt` disagrees with its frame timestamp decodes at the frame + timestamp, in Rust and JS. Public API: no new items. `catalog::Producer::clock()` now fixes the mapping, and importers no longer publish verbatim PTS when the clock was already taken. From f5d52939cc4b32e9f6bc0ddb780158f1b5fa03f7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 1 Oct 2026 15:03:58 -0700 Subject: [PATCH 4/4] quest: plan CMAF decoders timing samples from the frame timestamp Co-Authored-By: Claude Opus 5.5 --- quest/m2/README.md | 1 + quest/m2/cmaf-frame-timestamp.md | 38 ++++++++++++++++++++++++++++++++ quest/m2/shared-clock.md | 9 ++++---- 3 files changed, 43 insertions(+), 5 deletions(-) create mode 100644 quest/m2/cmaf-frame-timestamp.md diff --git a/quest/m2/README.md b/quest/m2/README.md index 358b31a0c8..45765ac924 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -105,6 +105,7 @@ waiting on an upstream release waits in [m4](/quest/m4/README.md). - [Windows capture parity](/quest/m2/capture-windows.md) - system audio and a settled app-capture policy - [Linux capture parity](/quest/m2/capture-linux.md) - Wayland window/system-audio capture with explicit display-selection and app-capture limits - [Audio capture time](/quest/m2/audio-capture-time.md) - native audio stamps a buffer's capture instant, not when the driver reads it +- [CMAF frame timestamp](/quest/m2/cmaf-frame-timestamp.md) - CMAF decoders time samples from the moq-lite frame timestamp, using `tfdt` only within the fragment - [Shared import clock](/quest/m2/shared-clock.md) - an importer joining a clock already in use offsets its PTS instead of moving it, so captures and imports share one timeline - [X11 capture transport](/quest/m2/x11-capture-shm.md) - move X11 capture to shared memory and RandR events instead of a per-frame socket copy - [Egress profile](/quest/m2/quic-egress-profile.md) - measure relay send-path syscalls, pacing bursts, and allocations before optimizing any of them diff --git a/quest/m2/cmaf-frame-timestamp.md b/quest/m2/cmaf-frame-timestamp.md new file mode 100644 index 0000000000..42abb97467 --- /dev/null +++ b/quest/m2/cmaf-frame-timestamp.md @@ -0,0 +1,38 @@ +# [S] CMAF decoders time samples from the frame timestamp + +## Goal + +A CMAF fragment decodes at its moq-lite frame timestamp, even when its `tfdt` +says otherwise. Today both decoders take sample times from `tfdt` and ignore +the frame timestamp: `fmp4::decode` in `rs/moq-mux/src/container/fmp4/mod.rs` +and `Format.decode` in `js/hang/src/container/cmaf/format.ts`. A publisher +that moves a passthrough track onto another timeline (an importer joining a +shared clock, ad insertion splicing unrelated PTS bases) would play at the +source's PTS. + +## Plan + +Decided (2026-10-01): the frame timestamp is the broadcast timeline. A +payload's own timestamps are only relative within the frame. Rewriting `tfdt` +on the publisher was rejected: it can force a v0 to v1 box upgrade, which +shifts `trun` data offsets. + +- A sample's time is the frame timestamp plus its PTS offset from the + fragment's first sample. The encoder already stamps the frame with that + first sample's PTS (`fmp4::encode`), so today's output decodes unchanged. +- Rust's decode gets the frame timestamp from the caller. The fMP4 exporter + and moq-hls re-fragment decoded frames, so they follow without changes; + confirm with a test. +- Check the TS exporter's verbatim carriage, which keeps PES PTS inside the + payload. Rewrite those from the frame timestamp, or refuse a mismatch. +- `main`: a behavior fix with no API change. + +Test: in Rust and JS, a fragment whose `tfdt` disagrees with its frame +timestamp decodes at the frame timestamp, with B-frame offsets preserved. + +Public API: none. Wire: none; this states what the frame timestamp already +means. + +## Related + +- [Shared clock](/quest/m2/shared-clock.md) - the first publisher to rely on it diff --git a/quest/m2/shared-clock.md b/quest/m2/shared-clock.md index 31ef9c1dc3..1eb45c0949 100644 --- a/quest/m2/shared-clock.md +++ b/quest/m2/shared-clock.md @@ -48,11 +48,6 @@ Guidance: shifts down. Refuse a frame that would land below zero rather than clamp it. - A `with_clock` catalog is fixed from the start, so its importers offset too. Today they publish verbatim PTS on a clock they didn't place. -- Both CMAF decoders stamp samples from `tfdt` and ignore the frame timestamp: - `fmp4::decode` in moq-mux and `Format.decode` in - `js/hang/src/container/cmaf/format.ts`. Rebase them: a sample's time is the - frame timestamp plus its offset from the fragment's first sample. Check the - fMP4 exporter does the same. - Document on `catalog::Producer::clock` that taking the clock fixes it. - Test: start a synthetic capture, then import an fMP4 starting at PTS 0. Both tracks advance from the capture's timeline with no rewind. Also cover the @@ -64,6 +59,10 @@ Public API: no new items. `catalog::Producer::clock()` now fixes the mapping, and importers no longer publish verbatim PTS when the clock was already taken. Wire: none. +## Required + +- [CMAF frame timestamp](/quest/m2/cmaf-frame-timestamp.md) - decoders honour an offset frame timestamp on passthrough tracks + ## Related - [Audio capture time](/quest/m2/audio-capture-time.md) - maps audio's capture timeline onto the broadcast clock once per open