-
-
Notifications
You must be signed in to change notification settings - Fork 248
quest(archive): per-track timelines and replay catalog #4168
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
623e960
4320405
264c8d6
5b4dfcf
4c94f32
99f6326
0c835d3
9f9b25e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # [S] Replay catalog | ||
|
|
||
| ## Goal | ||
|
|
||
| A replayed recording plays as VOD HLS from the stock server: `moq export | ||
| archive`, then `moq import archive`, then `moq export hls` lists the whole | ||
| recording with no embedder supplying a catalog. The replay broadcast publishes | ||
| its recorded catalog live, stamped with the recording's `store` and `version`, | ||
| so the exporter treats its timeline as durable and lists past `--window`. A | ||
| `--follow` replay grows like an event playlist. | ||
|
|
||
| Choosing which catalog applies to which media group stays with | ||
| [Catalog track identity](/quest/m2/catalog-tracks.md). | ||
|
|
||
| ## Plan | ||
|
|
||
| Today the reader writes live groups only for timelines; the recorded catalog | ||
| is FETCH-only, so a SUBSCRIBE to it on the replay broadcast never sees a group | ||
| and `moq-hls` never finds the `archive` entry. The HLS archive tests work around | ||
| this by hand-building a catalog. | ||
|
|
||
| - Republish each recorded catalog group live in its timeline's order, so the newest | ||
| one is at the live edge and a `--follow` replay picks up catalogs recorded | ||
| after it opened. | ||
| - Stamp `store` with the URL passed to `import archive`, and `version` with the | ||
| recording format. Refuse a URL carrying userinfo so credentials never land in | ||
| a catalog. `replay` stays unset: the timelines live on this broadcast. | ||
| - Keep the logic in `moq-cli`; `moq-archive` stays catalog-agnostic. Select the | ||
| catalog track with the CLI's catalog format as `export archive` does: hang | ||
| and hang.z are stamped, MSF is refused. | ||
| - The recorded catalog's `archive` entry describes the source's live | ||
| timelines, not the recording's, so replace it with the timelines the reader | ||
| replays rather than trusting the recorded ones. | ||
|
|
||
| Add a CI test that records a broadcast longer than the default window, replays | ||
| it, and asserts the stock exporter lists segment 0 with a durable | ||
| `timeShiftBufferDepth`. Consider moving the HLS archive tests onto the | ||
| republished catalog instead of their hand-built one. | ||
|
|
||
| Update `doc/bin/cli.md` and `doc/bin/hls.md` inline with an export, import, and | ||
| serve example. | ||
|
|
||
| Cover a DVR recording whose catalog outlived its first video segment, and fail | ||
| loudly on a recording with no catalog. | ||
|
|
||
| ## Required | ||
|
|
||
| - [Rust per-track timelines](/quest/m1/archive/track-timeline/core.md) - a DVR keeps its catalog's newest group |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| # Per-track timelines | ||
|
|
||
| ## Goal | ||
|
|
||
| Every track carries its own timeline, so tracks segment, commit, and expire | ||
| independently. A DVR keeps each track's newest group, such as a catalog that | ||
| never changes, and an append-only group stays addressable as frames arrive. An | ||
| edge like `moq-hls` derives HLS and DASH from group timestamps, so a publisher | ||
| never needs to know about HLS. | ||
|
|
||
| Nothing on `main` has users yet: break the timeline, catalog `archive` entry, | ||
| and recording format in place, with no compatibility path, even though hang | ||
| 0.21 released the current shape. The children merge into this line's branch, | ||
| so Rust and JS reach `main` together. | ||
|
|
||
| ## Plan | ||
|
|
||
| Today one timeline numbers aligned segments for every track. A DVR pops whole | ||
| segments, so a static catalog expires with the first video segment, and the | ||
| writer only records a non-pacing group once it completes, so a group that never | ||
| closes is never stored. `moq_json::window` pops from the front, so one shared | ||
| timeline cannot keep an old catalog record while dropping newer video. | ||
|
|
||
| Decisions: | ||
|
|
||
| - One timeline track per track, live and recorded. `moq-mux` publishes them for | ||
| every broadcast; an unsubscribed track costs nothing. | ||
| - The catalog's root `archive` entry maps each track to its timeline, including | ||
| the catalog track itself. `replay`, `store`, and `version` stay beside it. | ||
| - Each track cuts on its own: automatically at a group boundary between a | ||
| minimum and maximum duration (roughly 1s and 10s), splitting a long-lived | ||
| group by frame at the maximum. Manual cuts stay as an optimization, such as a | ||
| video keyframe cutting audio so derived segments need fewer objects. | ||
| - A stored object may hold a frame range of a group, not only whole groups. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The revised browser plan now defers to the Rust writer's keys, but this line still leaves the recording layout as Useful? React with 👍 / 👎. |
||
| - HLS and DASH segments are derived at the edge from group timestamps, not | ||
| from storage objects. Fetching extra objects is fine when they land in the | ||
| reader's cache for the next request. | ||
|
|
||
| ## Quests | ||
|
|
||
| - [Rust per-track timelines](/quest/m1/archive/track-timeline/core.md) - the draft, `hang`, `moq-mux`, `moq-archive`, and `moq-hls` move to per-track timelines in one change | ||
| - [JS per-track timelines](/quest/m1/archive/track-timeline/js.md) - `@moq/hang` publishes and reads the same per-track timelines as Rust | ||
|
|
||
| ## Related | ||
|
|
||
| - [Catalog track identity](/quest/m2/catalog-tracks.md) - the catalog's own timeline gives it timestamps, but which catalog applies to a group stays there | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| # [XL] Rust per-track timelines | ||
|
|
||
| ## Goal | ||
|
|
||
| The moq-hang draft, `hang`, `moq-mux`, `moq-archive`, and `moq-hls` use one | ||
| timeline per track, per the [line's decisions](/quest/m1/archive/track-timeline/README.md). | ||
| A DVR recording keeps every track's newest group, an append-only group is | ||
| stored as its frames arrive, and `moq-hls` serves aligned HLS and DASH from | ||
| live or recorded timelines without the publisher cutting for it. | ||
|
|
||
| ## Plan | ||
|
|
||
| These crates compile together, so they change in one PR. Update the draft's | ||
| timeline and Recording sections in the same change, in a new draft revision; | ||
| most of their rules assume one aligned segment counter (one timeline per | ||
| broadcast, cross-track boundaries, whole-segment retention, reading segment N | ||
| of track T through record N). | ||
|
|
||
| - **Timeline:** a record describes one stored span of its own track: sequence, | ||
| pts, duration, and the group and frame range. Drop cross-track pacing and the | ||
| completeness wait; keep manual cuts. Retention pops each track's window | ||
| independently and always keeps its newest record. | ||
| - **Catalog:** replace the root `archive.track` with a map from track to | ||
| timeline, covering the catalog track too. | ||
| - **Writer and reader:** commit each track independently, and store frame | ||
| ranges so a never-closing group is recorded. Bump the recording `version` and | ||
| refuse the old one. Recovery, grace deletion, and FETCH replay follow the | ||
| per-track index. | ||
|
Comment on lines
+25
to
+28
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When Useful? React with 👍 / 👎. |
||
| - **HLS and DASH:** take segment boundaries from a reference video rendition | ||
| (an audio one when the broadcast has no video), near a target duration, and | ||
| number them so every edge and every reload agrees, including after DVR pops. Every other video rendition snaps each | ||
| boundary to its nearest group start within a tolerance (around 1s); a | ||
|
Comment on lines
+29
to
+32
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When the selected reference rendition is removed, reconfigured, or superseded by a newly added rendition, an existing edge and an edge joining afterward can select different reference timelines and derive different segment boundaries and numbers for the same media. Rendition membership is explicitly dynamic through Useful? React with 👍 / 👎. |
||
| segment with no start in range becomes a gap (`EXT-X-GAP` in HLS), so a | ||
| player switching renditions lands on the next real segment. Gaps are a | ||
| best-effort fallback: the draft and HLS docs say a publisher wanting HLS | ||
| export SHOULD align video GOPs across renditions. Audio and other | ||
| renditions take the groups and frames whose timestamps fall in each span, | ||
| possibly from more than one object. The reader's cache absorbs the overlap. | ||
|
|
||
| Carry over the existing tests (DVR trim, durable listing, archive replay) and | ||
| add a static catalog outliving its first video segment, an append-only group | ||
| spanning several objects, audio cut independently of video, renditions that | ||
| start at different times, video renditions whose group starts never | ||
| coincide, and an audio-only broadcast. | ||
|
|
||
| Update `doc/concept/hang.md` and the HLS and CLI docs this makes stale. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| # [M] JS per-track timelines | ||
|
|
||
| ## Goal | ||
|
|
||
| `@moq/hang` publishes one timeline per track with the same records, cuts, and | ||
| catalog `archive` map as Rust, and parses them identically. | ||
|
|
||
| ## Plan | ||
|
|
||
| Port the landed Rust shape to `js/hang/src/timeline.ts` and its catalog schema, | ||
| replacing the aligned timeline and cross-track pacing rather than keeping both. | ||
|
kixelated marked this conversation as resolved.
|
||
| Cover the same cut rules and a static catalog outliving other tracks' records, | ||
| and check the records against Rust output in the interop suite. | ||
|
|
||
| ## Required | ||
|
|
||
| - [Rust per-track timelines](/quest/m1/archive/track-timeline/core.md) - the format this mirrors | ||
Uh oh!
There was an error while loading. Please reload this page.