Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@ loom = { version = "0.7.2", features = ["futures"] }
# DNS-SD advertisement and browsing for LAN peer discovery (moq-tokio's `mdns` feature).
# `async` awaits the event channel instead of blocking a thread on it.
mdns-sd = { version = "0.21", features = ["async"] }
moq-archive = { version = "0.0.4", path = "rs/moq-archive" }
moq-audio = { version = "0.1.3", path = "rs/moq-audio", default-features = false }
moq-auth = { version = "0.1.1", path = "rs/moq-auth" }
moq-binary = { version = "0.1.3", path = "rs/moq-binary" }
Expand Down
35 changes: 34 additions & 1 deletion doc/bin/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,11 @@ or Docker; see [Install](/setup/install).
| `import` | `capture` | Capture a camera, display, window, or app plus a microphone, and encode natively. |
| `import` | `hls <url>` | Pull a remote HLS playlist. |
| `import` | `rtmp`, `srt`, `rtc` | Accept pushes (`--listen`) or pull from a remote (`--connect`). |
| `import` | `archive <url>` | Replay a recording from an object store. |
| `export` | `fmp4`, `mkv`, `ts`, `flv`, `h264`, `h265` | Write a container to stdout. |
| `export` | `hls --listen` | Serve the broadcast as HLS over HTTP. |
| `export` | `rtmp`, `srt`, `rtc` | Serve plays (`--listen`) or push to a remote (`--connect`). |
| `export` | `archive <url>` | Record the broadcast into an object store. |
| `play` | | Decode and play in a native window with sound. |
| `transcode` | | Publish a just-in-time rendition ladder next to a broadcast. |
| `fetch` | `<track>` | Write one group of a track to stdout. |
Expand Down Expand Up @@ -178,6 +180,36 @@ refuses a listener or cluster flag. It gives up after 30 seconds, as `/fetch`
does, and exits non-zero when the broadcast or group is not found, the relay
refuses, or the deadline passes.

## Archive

```bash
# Record a broadcast until it ends
moq --connect https://relay.example.com/anon --broadcast event.hang export archive s3://recordings/event

# Replay it under another name
moq --connect https://relay.example.com/anon --broadcast event-replay.hang import archive s3://recordings/event
```

`export archive` records one broadcast with
[moq-archive](https://docs.rs/moq-archive), reading its catalog as it changes:
video and audio renditions pace the segments, and the catalog plus every text,
JSON, and binary track are recorded alongside. It refuses a rendition served
from another broadcast, and one that returns after the catalog dropped it. The
stage ends once the broadcast does, and it refuses a store URL that already
holds a recording. `--retention 1h` keeps only the last hour (a DVR),
Comment on lines +198 to +199

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Correct the existing-recording behavior in the CLI docs

The CLI passes the store directly to Writer::new, which now explicitly recovers and resumes an existing recording rather than refusing it. This statement can cause an operator to reuse a prefix expecting a safe failure; if the new source restarts group numbering, the resume floor silently skips its earlier groups. Document the resume and numbering constraints, or restore a CLI-level occupancy check.

AGENTS.md reference: AGENTS.md:L27-L27

Useful? React with 👍 / 👎.

deleting expired objects `--retention-grace` (default 30s) after the timeline
stops advertising them.

`import archive` republishes a recording: the timeline replays as a live track
and every other track's groups are served on request, one object GET per group
range. By default it replays what is stored and ends the timeline there;
`--follow 2s` keeps checking for new segments of a recording still being made.

Store URLs are `file:///absolute/path`, `s3://bucket/prefix`,
`gs://bucket/prefix`, or `az://container/prefix`. Cloud credentials come from
the usual `AWS_*`, `GOOGLE_*`, and `AZURE_*` environment variables. The `s3`,
`gcs`, and `azure` cargo features are on by default.

## Multiple stages

Separate stages with `--` to bridge several broadcasts, or both directions,
Expand All @@ -186,7 +218,8 @@ over one connection:
```bash
moq --connect https://relay.example.com/anon \
import --broadcast event.hang srt --listen 0.0.0.0:9000 \
-- export --broadcast event.hang hls --listen 0.0.0.0:8080
-- export --broadcast event.hang hls --listen 0.0.0.0:8080 \
-- export --broadcast event.hang archive file:///recordings/event
```

## Redundant publishers
Expand Down
16 changes: 15 additions & 1 deletion doc/bin/hls.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,13 +38,27 @@ broadcast by path:
/{broadcast}/{video|audio}/{rendition}/seg/t{pts}.m4s
```

A [`moq-archive`](https://docs.rs/moq-archive) recording replayed through its
`Reader` is served the same way, with no second stored copy. Playlists come
from the replayed timeline alone, and a segment GETs exactly one stored object
of its rendition, so switching renditions never downloads both. An
inline-parameter-set codec with no catalog `description` is the exception:
the first playlist render GETs one keyframe group to build the init segment,
then caches it. Out-of-band configs need no media GET. When the catalog's
`archive` entry names a `store` and no `replay` path, its ranges are durable on
this broadcast, so the playlists list the whole retained timeline and only the
recording's own retention trims them; DASH `timeShiftBufferDepth` is the listed
span. The playlist ends with `EXT-X-ENDLIST` only once the reader's caller
declares the recording finished; the store holds no completion marker.

The init URL carries a hash of its bytes, so a reconfigured rendition gets a
new one. An embedder of the library can also label the publisher's run with
`Broadcaster::set_generation`. Every segment URL then carries it
(`seg/{generation}.{segment}.m4s`), since a restarted publisher reuses segment
numbers for different media.

`--window` sets the playlist duration (default 16 s),
`--window` sets the live playlist duration (default 16 s) and caps segment
`Cache-Control: max-age` for every broadcast,
`--listen-tls-cert`/`--listen-tls-key` or `--listen-tls-generate` serve HTTPS,
and `--cors-origin` opens it to browsers.
H.264/H.265 and AAC/Opus renditions are served. Import handles classic HLS;
Expand Down
9 changes: 9 additions & 0 deletions js/json/src/window/encoder.ts
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,15 @@ export class Encoder<T> {
return this.#start + this.#window.length;
}

/**
* Discard group-local state, so the next edit opens a new group with a header.
*
* Call this whenever the caller closes the current group behind the encoder's back.
*/
reset(): void {
this.#resyncGroup();
}

/** Discard group-local state after an encoded frame did not reach the wire. */
#resyncGroup(): void {
this.#flate = undefined;
Expand Down
18 changes: 18 additions & 0 deletions js/json/src/window/producer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,24 @@ export class Producer<T> {
encoded.commit();
}

/**
* Finish the open group, leaving the next edit to open a replacement with a header.
*
* Idempotent: cutting when no group is open does nothing. A caller that stores complete groups
* cuts after its edits, so every edit so far sits in a group no later frame can extend.
*/
cut(): void {
if (!this.#group) return;

// Reset first: the group closes either way below, and a throw must not leave the encoder
// appending ops to a group that is gone.
this.#encoder.reset();

const group = this.#group;
this.#group = undefined;
group.close();
}

/** Finish the track, closing any open group. */
finish(): void {
if (this.#finished) return;
Expand Down
22 changes: 22 additions & 0 deletions js/json/src/window/window.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,28 @@ test("push and pop round-trip", async () => {
]);
});

test("a cut is invisible to the consumer", async () => {
const live = new Live();
await live.push(0);
live.producer.cut();
live.producer.cut();
await live.push(1);

expect(await live.finish()).toEqual([
{ push: { index: 0, value: { n: 0 } } },
{ push: { index: 1, value: { n: 1 } } },
]);
});

test("reset makes the next edit a header", () => {
const encoder = new Encoder<Rec>({});
encoder.push({ n: 0 }).commit();
expect(encoder.push({ n: 1 }).keyframe).toBe(false);

encoder.reset();
expect(encoder.push({ n: 2 }).keyframe).toBe(true);
});

test("concurrent consumer reads are rejected", async () => {
const track = new Track.Producer("test");
const consumer = new Consumer<Rec>({ track: track.subscribe() });
Expand Down
36 changes: 32 additions & 4 deletions quest/m1/archive/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,41 @@ The segment engine is in `rs/moq-mux/src/timeline.rs`:
- `moq-hls` renders live playlists from the timeline alone and FETCHes media
per HTTP request (`rs/moq-hls/src/export/mod.rs:3-8`). A clean timeline
finish ends every window with `EXT-X-ENDLIST` (:325-327).
- The same exporter serves a recording replayed through `moq_archive::Reader`
with no archive-specific code (`rs/moq-hls/src/export/archive_tests.rs`):
playlists read only the timeline (an inline parameter set also GETs one
keyframe group to build its init), and a segment GETs one object of its
rendition. The caller supplies the catalog.
- A catalog `archive` entry with a `store` and no `replay` path declares its
ranges durable on that broadcast, so the exporter lists the whole retained
timeline and only its pops trim it (`durable` in
`rs/moq-hls/src/export/mod.rs`). DASH `timeShiftBufferDepth` becomes the
listed span, and `--window` still bounds live playlists and caps segment
`max-age`. The catalog already states durability, so no per-broadcast
option or separate server is needed.

`rs/moq-archive` stores the versioned objects on any `object_store::ObjectStore`:
percent-encoded track names, `.info` JSON, the binary envelope, and put/get/list/delete.
`moq_archive::Writer` (`rs/moq-archive/src/writer.rs`) records enrolled tracks
through `Deferred`, omits failed tracks with `Pending::omit`, stores each
segment's timeline groups after `Producer::flush`, and expires DVR segments
with a deletion grace. On a prefix that already holds a recording, it replays the
retained timeline from a checkpoint through `timeline::Producer::resume`, refuses
groups at or below each track's largest stored group, and a DVR deletes
unreferenced group objects one grace period after recovery.
`moq_archive::Reader` (`rs/moq-archive/src/reader/mod.rs`) replays the timeline onto a
supplied `broadcast::Producer` and serves FETCH through `track::Dynamic` with a byte-bounded
object LRU. `Reader::refresh` follows by listing timeline keys after its cursor, so gaps and
DVR expiry recover from the next checkpoint; `Reader::finish` applies out-of-band finality.
`rs/moq-archive/src/proof.rs` records one multi-rendition broadcast end to end: its exact keys and
bytes match on memory, local disk, and an unordered listing, FETCH replays every group exactly,
and a rendition's playback GETs only that rendition's objects.

### Format

[Per-track timelines](/quest/m1/archive/track-timeline/README.md) replaces the
aligned segments below with one timeline per track.

The format is the draft's
[Recording section](/drafts/draft-lcurley-moq-hang.md#recording).
The application chooses the object prefix, selected tracks, retention, and credentials; `moq-archive` owns the
Expand Down Expand Up @@ -103,12 +132,11 @@ owned by that prerequisite, not duplicated in archive storage.

## Quests

- [Recording writer](/quest/m1/archive/writer.md) - feed the segmenter from a `broadcast::Consumer`, store each segment, then commit its record
- [Recording reader](/quest/m1/archive/reader.md) - serve archived FETCH through a supplied `broadcast::Producer`
- [Per-track timelines](/quest/m1/archive/track-timeline/README.md) - every track segments and expires on its own timeline, and HLS is derived from group timestamps at the edge
- [Replay catalog](/quest/m1/archive/replay-catalog.md) - `moq import archive` publishes the recorded catalog live with `store` set, so stock `moq export hls` serves the whole replay
- [Browser archive](/quest/m1/archive/browser.md) - the same contract for browser-published broadcasts
- [Offline archive HLS](/quest/m1/archive/hls.md) - render playlists from the archive timeline and fetch segment media lazily
- [DVR rewind](/quest/m1/archive/dvr.md) - seek through a bounded archive and return to live playback
- [Archive proof](/quest/m1/archive/proof.md) - prove persistence ordering, selective reads, exact FETCH replay, and timeline-only HLS generation
- [DVR timeline pruning](/quest/m1/archive/pruning.md) - a DVR deletes timeline objects no retained checkpoint needs

## Related

Expand Down
Loading