diff --git a/quest/m1/archive/README.md b/quest/m1/archive/README.md index b9117707b2..89cd0aedad 100644 --- a/quest/m1/archive/README.md +++ b/quest/m1/archive/README.md @@ -63,6 +63,9 @@ DVR expiry recover from the next checkpoint; `Reader::finish` applies out-of-ban ### Format +[Per-track timelines](/quest/m1/archive/track-timeline/README.md) replaces the +aligned segments below with one timeline per track. + The format is the draft's [Recording section](/drafts/draft-lcurley-moq-hang.md#recording). The application chooses the object prefix, selected tracks, retention, and credentials; `moq-archive` owns the @@ -126,6 +129,8 @@ owned by that prerequisite, not duplicated in archive storage. ## Quests +- [Per-track timelines](/quest/m1/archive/track-timeline/README.md) - every track segments and expires on its own timeline, and HLS is derived from group timestamps at the edge +- [Replay catalog](/quest/m1/archive/replay-catalog.md) - `moq import archive` publishes the recorded catalog live with `store` set, so stock `moq export hls` serves the whole replay - [Browser archive](/quest/m1/archive/browser.md) - the same contract for browser-published broadcasts - [DVR rewind](/quest/m1/archive/dvr.md) - seek through a bounded archive and return to live playback - [Archive proof](/quest/m1/archive/proof.md) - prove persistence ordering, selective reads, exact FETCH replay, and timeline-only HLS generation diff --git a/quest/m1/archive/browser.md b/quest/m1/archive/browser.md index 78a0dd061c..7d08f060cf 100644 --- a/quest/m1/archive/browser.md +++ b/quest/m1/archive/browser.md @@ -9,16 +9,16 @@ layout, and FETCH behavior as native `moq-archive` users. Port the archive contract to the JS packages with memory and OPFS storage. The application explicitly enrolls video, audio, catalog, or arbitrary data tracks; -the archive does not infer them from Hang. The JS timeline already publishes -through the Window (`js/hang/src/container/timeline.ts:134`, `:165`, backed by -`js/json/src/window/`) with pacing tracks and application-driven cuts; add the -deferred commit Rust has (`Producer::deferred`, -`rs/moq-mux/src/timeline.rs:917`). - -Persist one range-named object per track per segment after its groups complete, -then publish the archive timeline record. Match the 19-digit group-bound keys, -ascending delta-encoded IDs, and sequential timeline discovery used by Rust. A typical audio segment contains many -one-group-per-frame audio groups. Match the Rust binary envelope bytes and `.info` property values, per the [Recording section](/drafts/draft-lcurley-moq-hang.md#recording), without inferring catalog-to-group applicability. +the archive does not infer them from Hang. Record against the per-track +timelines from [JS per-track timelines](/quest/m1/archive/track-timeline/js.md). + +Commit each track independently: persist each stored span, including a frame +range of a still-open group, then publish that track's timeline record. Match +the object keys, envelope bytes, `.info` property values, and timeline +discovery the Rust writer uses after +[Rust per-track timelines](/quest/m1/archive/track-timeline/core.md), per the +[Recording section](/drafts/draft-lcurley-moq-hang.md#recording), without +inferring catalog-to-group applicability. [Catalog track identity](/quest/m2/catalog-tracks.md) addresses that separately. Use [JavaScript FETCH](/quest/m1/js-fetch.md)'s on-demand group requests to @@ -34,3 +34,4 @@ remains downstream (moq.pro) work. ## Required - [JavaScript FETCH](/quest/m1/js-fetch.md) - generic on-demand group serving and IETF FETCH support +- [JS per-track timelines](/quest/m1/archive/track-timeline/js.md) - the timeline this archive records against diff --git a/quest/m1/archive/dvr.md b/quest/m1/archive/dvr.md index 096633a809..48959cd37c 100644 --- a/quest/m1/archive/dvr.md +++ b/quest/m1/archive/dvr.md @@ -26,6 +26,15 @@ The reader evicts popped spans from its object cache, but a group it already served stays in `moq_net`'s track cache until the pool reclaims it. Decide whether expiry during a seek needs a group eviction API in `moq-net`. +`moq-hls` reads a timeline from the catalog's own broadcast, and an +`archive.replay` path only marks it non-durable. Decide whether a live +broadcast's exporter follows `replay` to the recording for rewind, or whether +viewers address the replay broadcast directly. + +## Required + +- [Rust per-track timelines](/quest/m1/archive/track-timeline/core.md) - seeks through per-track timelines + ## Closes - [#2275](https://github.com/moq-dev/moq/issues/2275) - close this issue when the quest finishes diff --git a/quest/m1/archive/proof.md b/quest/m1/archive/proof.md index 225449edf4..b24575dbb1 100644 --- a/quest/m1/archive/proof.md +++ b/quest/m1/archive/proof.md @@ -7,8 +7,9 @@ reads, and timeline-only HLS generation from one multi-rendition broadcast. ## Plan -Record explicitly enrolled video, audio, catalog, and non-media tracks. Cut at -aligned keyframe boundaries, including multiple audio groups per segment, then +Record explicitly enrolled video, audio, catalog, and non-media tracks. Cut each +track on its own timeline, including many audio groups per object and a group +split across objects by frame, then replay their original sequences, timestamps, and payloads through `track::Dynamic`. @@ -60,3 +61,7 @@ restart must finish this cleanup before accepting new groups. Finally render and reload HLS playlists while rejecting every media-object GET until a segment URI is requested. A segment request must resolve one object from the replayed timeline without any listing or separate index object. + +## Required + +- [Rust per-track timelines](/quest/m1/archive/track-timeline/core.md) - proves the per-track format, not the aligned one diff --git a/quest/m1/archive/replay-catalog.md b/quest/m1/archive/replay-catalog.md new file mode 100644 index 0000000000..d6f5bb8ec5 --- /dev/null +++ b/quest/m1/archive/replay-catalog.md @@ -0,0 +1,48 @@ +# [S] Replay catalog + +## Goal + +A replayed recording plays as VOD HLS from the stock server: `moq export +archive`, then `moq import archive`, then `moq export hls` lists the whole +recording with no embedder supplying a catalog. The replay broadcast publishes +its recorded catalog live, stamped with the recording's `store` and `version`, +so the exporter treats its timeline as durable and lists past `--window`. A +`--follow` replay grows like an event playlist. + +Choosing which catalog applies to which media group stays with +[Catalog track identity](/quest/m2/catalog-tracks.md). + +## Plan + +Today the reader writes live groups only for timelines; the recorded catalog +is FETCH-only, so a SUBSCRIBE to it on the replay broadcast never sees a group +and `moq-hls` never finds the `archive` entry. The HLS archive tests work around +this by hand-building a catalog. + +- Republish each recorded catalog group live in its timeline's order, so the newest + one is at the live edge and a `--follow` replay picks up catalogs recorded + after it opened. +- Stamp `store` with the URL passed to `import archive`, and `version` with the + recording format. Refuse a URL carrying userinfo so credentials never land in + a catalog. `replay` stays unset: the timelines live on this broadcast. +- Keep the logic in `moq-cli`; `moq-archive` stays catalog-agnostic. Select the + catalog track with the CLI's catalog format as `export archive` does: hang + and hang.z are stamped, MSF is refused. +- The recorded catalog's `archive` entry describes the source's live + timelines, not the recording's, so replace it with the timelines the reader + replays rather than trusting the recorded ones. + +Add a CI test that records a broadcast longer than the default window, replays +it, and asserts the stock exporter lists segment 0 with a durable +`timeShiftBufferDepth`. Consider moving the HLS archive tests onto the +republished catalog instead of their hand-built one. + +Update `doc/bin/cli.md` and `doc/bin/hls.md` inline with an export, import, and +serve example. + +Cover a DVR recording whose catalog outlived its first video segment, and fail +loudly on a recording with no catalog. + +## Required + +- [Rust per-track timelines](/quest/m1/archive/track-timeline/core.md) - a DVR keeps its catalog's newest group diff --git a/quest/m1/archive/track-timeline/README.md b/quest/m1/archive/track-timeline/README.md new file mode 100644 index 0000000000..2b4190e23b --- /dev/null +++ b/quest/m1/archive/track-timeline/README.md @@ -0,0 +1,46 @@ +# Per-track timelines + +## Goal + +Every track carries its own timeline, so tracks segment, commit, and expire +independently. A DVR keeps each track's newest group, such as a catalog that +never changes, and an append-only group stays addressable as frames arrive. An +edge like `moq-hls` derives HLS and DASH from group timestamps, so a publisher +never needs to know about HLS. + +Nothing on `main` has users yet: break the timeline, catalog `archive` entry, +and recording format in place, with no compatibility path, even though hang +0.21 released the current shape. The children merge into this line's branch, +so Rust and JS reach `main` together. + +## Plan + +Today one timeline numbers aligned segments for every track. A DVR pops whole +segments, so a static catalog expires with the first video segment, and the +writer only records a non-pacing group once it completes, so a group that never +closes is never stored. `moq_json::window` pops from the front, so one shared +timeline cannot keep an old catalog record while dropping newer video. + +Decisions: + +- One timeline track per track, live and recorded. `moq-mux` publishes them for + every broadcast; an unsubscribed track costs nothing. +- The catalog's root `archive` entry maps each track to its timeline, including + the catalog track itself. `replay`, `store`, and `version` stay beside it. +- Each track cuts on its own: automatically at a group boundary between a + minimum and maximum duration (roughly 1s and 10s), splitting a long-lived + group by frame at the maximum. Manual cuts stay as an optimization, such as a + video keyframe cutting audio so derived segments need fewer objects. +- A stored object may hold a frame range of a group, not only whole groups. +- HLS and DASH segments are derived at the edge from group timestamps, not + from storage objects. Fetching extra objects is fine when they land in the + reader's cache for the next request. + +## Quests + +- [Rust per-track timelines](/quest/m1/archive/track-timeline/core.md) - the draft, `hang`, `moq-mux`, `moq-archive`, and `moq-hls` move to per-track timelines in one change +- [JS per-track timelines](/quest/m1/archive/track-timeline/js.md) - `@moq/hang` publishes and reads the same per-track timelines as Rust + +## Related + +- [Catalog track identity](/quest/m2/catalog-tracks.md) - the catalog's own timeline gives it timestamps, but which catalog applies to a group stays there diff --git a/quest/m1/archive/track-timeline/core.md b/quest/m1/archive/track-timeline/core.md new file mode 100644 index 0000000000..4651a338f9 --- /dev/null +++ b/quest/m1/archive/track-timeline/core.md @@ -0,0 +1,46 @@ +# [XL] Rust per-track timelines + +## Goal + +The moq-hang draft, `hang`, `moq-mux`, `moq-archive`, and `moq-hls` use one +timeline per track, per the [line's decisions](/quest/m1/archive/track-timeline/README.md). +A DVR recording keeps every track's newest group, an append-only group is +stored as its frames arrive, and `moq-hls` serves aligned HLS and DASH from +live or recorded timelines without the publisher cutting for it. + +## Plan + +These crates compile together, so they change in one PR. Update the draft's +timeline and Recording sections in the same change, in a new draft revision; +most of their rules assume one aligned segment counter (one timeline per +broadcast, cross-track boundaries, whole-segment retention, reading segment N +of track T through record N). + +- **Timeline:** a record describes one stored span of its own track: sequence, + pts, duration, and the group and frame range. Drop cross-track pacing and the + completeness wait; keep manual cuts. Retention pops each track's window + independently and always keeps its newest record. +- **Catalog:** replace the root `archive.track` with a map from track to + timeline, covering the catalog track too. +- **Writer and reader:** commit each track independently, and store frame + ranges so a never-closing group is recorded. Bump the recording `version` and + refuse the old one. Recovery, grace deletion, and FETCH replay follow the + per-track index. +- **HLS and DASH:** take segment boundaries from a reference video rendition + (an audio one when the broadcast has no video), near a target duration, and + number them so every edge and every reload agrees, including after DVR pops. Every other video rendition snaps each + boundary to its nearest group start within a tolerance (around 1s); a + segment with no start in range becomes a gap (`EXT-X-GAP` in HLS), so a + player switching renditions lands on the next real segment. Gaps are a + best-effort fallback: the draft and HLS docs say a publisher wanting HLS + export SHOULD align video GOPs across renditions. Audio and other + renditions take the groups and frames whose timestamps fall in each span, + possibly from more than one object. The reader's cache absorbs the overlap. + +Carry over the existing tests (DVR trim, durable listing, archive replay) and +add a static catalog outliving its first video segment, an append-only group +spanning several objects, audio cut independently of video, renditions that +start at different times, video renditions whose group starts never +coincide, and an audio-only broadcast. + +Update `doc/concept/hang.md` and the HLS and CLI docs this makes stale. diff --git a/quest/m1/archive/track-timeline/js.md b/quest/m1/archive/track-timeline/js.md new file mode 100644 index 0000000000..461ff1cf75 --- /dev/null +++ b/quest/m1/archive/track-timeline/js.md @@ -0,0 +1,17 @@ +# [M] JS per-track timelines + +## Goal + +`@moq/hang` publishes one timeline per track with the same records, cuts, and +catalog `archive` map as Rust, and parses them identically. + +## Plan + +Port the landed Rust shape to `js/hang/src/timeline.ts` and its catalog schema, +replacing the aligned timeline and cross-track pacing rather than keeping both. +Cover the same cut rules and a static catalog outliving other tracks' records, +and check the records against Rust output in the interop suite. + +## Required + +- [Rust per-track timelines](/quest/m1/archive/track-timeline/core.md) - the format this mirrors