Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions doc/bin/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,8 +136,9 @@ moq ... transcode --rung 720:2500000 --rung 360:600000 --encoder nvenc --decoder

Publishes `cam.hang/transcode.hang` whose catalog references the source's
rendition and adds lower rungs that are decoded and encoded only while someone
watches them. On NVIDIA the whole pipeline stays on the GPU. Requires the
`transcode` feature.
watches them. On NVIDIA the whole pipeline stays on the GPU; `--frames cpu`
forces decoded frames into CPU memory instead of the default `native`.
Requires the `transcode` feature.

The ladder is sized against the source picture and follows it, so a source that
changes resolution mid-stream (a window capture renegotiated by a resize, a
Expand Down
2 changes: 1 addition & 1 deletion doc/lib/rs/moq-video.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ ffmpeg, no GStreamer, no system codec to install.

Highlights:

- **Automatic backend selection**, hardware first. Linux GPU libraries are `dlopen`ed at runtime, so one binary starts anywhere and warns when it falls back to software. openh264 is statically linked as the H.264 fallback; H.265 is hardware-only; AV1 decodes via NVDEC. The VAAPI encoder is compile-verified but not yet validated on hardware.
- **Automatic backend selection**, hardware first. Linux GPU libraries are `dlopen`ed at runtime, so one binary starts anywhere and warns when it falls back to software. openh264 (the default-on `openh264` feature) is statically linked as the H.264 fallback; H.265 is hardware-only; AV1 decodes via NVDEC. The VAAPI encoder is compile-verified but not yet validated on hardware.
- **Publish on demand.** `encode::publish_capture` advertises the track up front and opens the camera only while someone subscribes.
- **GPU ownership where the platform allows.** Matching codec backends consume their native GPU surfaces directly. The renderer imports `CVPixelBuffer` and supported DMA-BUF formats. Linux/NVIDIA producers can import dedicated Vulkan RGBA8 slots into CUDA with timeline-semaphore ordering and completion-driven slot return. Vulkan/CUDA surfaces deliberately have no CPU pixel fallback; other surfaces use the typed `Surface::into_i420()` and configured `Surface::to_rgba(config)` when needed.
- **Live bitrate control** where the selected backend supports it, without forcing a keyframe. An unsupported backend keeps its opening rate.
Expand Down
3 changes: 2 additions & 1 deletion quest/future/audio-loss-recovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
## Goal

Decide whether a concrete MoQ audio consumer benefits from in-band Opus FEC,
with a tested loss/latency policy before exposing a replacement public option.
with a tested loss/latency policy before exposing any additive option on
`encode::Settings`.
This does not block 0.1.

## Plan
Expand Down
7 changes: 4 additions & 3 deletions quest/future/intra-refresh/encode-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,10 @@ without replacing an API after 0.1.
error for `Refresh` (supported or refused, never a silent fallback to
keyframes). The test-only probe backend accepts it so the producer logic is
testable without hardware.
- Group boundaries in refresh mode come from counting: backends report
`keyframe = false` for a sweep start, so the producer marks the first frame
of each cycle by frame count from the last cut and opens the group there.
- Group boundaries in refresh mode come from counting: the splitter sees no
IDR at a sweep start (only a recovery-point SEI where the backend emits
one), so the producer marks the first frame of each cycle by frame count
from the last cut and tells the importer to open the group there.
A backend reports the sweep length it actually configured, which can be
shorter than the cycle (NVENC needs it strictly shorter), and the producer
publishes `warmup` as that length over the framerate.
Expand Down
4 changes: 2 additions & 2 deletions quest/future/intra-refresh/v4l2-refresh.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ claiming a boundary that was not encoded.
`V4L2_CID_MPEG_VIDEO_CYCLIC_INTRA_REFRESH_MB` as macroblocks per frame
derived from the picture size and the cycle) from the sys crate, set
`GOP_SIZE` to zero for an unbounded GOP, and refuse the mode when the driver
rejects both controls, unlike the self-disabling `keyframes` fallback used
for forced keyframes.
rejects both controls, the same way a missing `FORCE_KEY_FRAME` control
refuses cuts with `CutUnsupported` at open.
- A cut in refresh mode restarts the sweep. If the driver cannot honor it,
report rejection through the error path settled in main; this does not
independently require a Result from cut(). Never open a group out of phase
Expand Down
7 changes: 4 additions & 3 deletions quest/future/sip-stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,10 @@ consumable by an embedder.
- The media leg is plain negotiated RTP: `moq-rtc`/str0m is ICE/DTLS-first
and its reusable session internals are crate-private, so it is precedent,
not a base. Consider publicizing moq-rtc's codec bridges rather than
duplicating the RTP-to-hang mapping, and reuse `moq-audio`'s Opus and
resampler for the G.711<->Opus transcode (mono 8 kHz; G.711 companding is
new code, nothing in the repository has it).
duplicating the RTP-to-hang mapping, and reuse `moq-audio`'s Opus codec
for the G.711<->Opus transcode, with rate conversion through
`decode::Output` and `encode::Input` since the resampler is private (mono
8 kHz; G.711 companding is new code, nothing in the repository has it).
- RTP wall-clock normalization off RTCP sender reports, as moq-rtc does.
- The embedder decides paths and auth; the crate's API is
"answer this INVITE, give me the caller as Opus, take Opus to play" plus
Expand Down
4 changes: 2 additions & 2 deletions quest/future/video-vaapi.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ release first.
Four gaps, one external dependency.

**Decode.** The H.264 decoder landed (moq-vaapi 0.0.4, `decode/backend/vaapi.rs`),
with `decode::Config::gpu_frames` exporting DMA-BUF surfaces the renderer
imports without a download. H.265 decode is still missing, so a Linux box
with the default `decode::Config::output` of `Output::Native` handing out
DMA-BUF surfaces the renderer imports without a download. H.265 decode is still missing, so a Linux box
without NVDEC has no hardware path for it.

**The encoder.** Ours is a 111-line CPU-only adapter whose own header says it
Expand Down
5 changes: 2 additions & 3 deletions quest/main/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,8 @@ Their package boundaries are explicit:

The media crates are also 0.0.x, so their changes target main. Adapt callers in
other packages without breaking their published APIs, C layouts, or wire
formats. Do not bump versions as part of these quests. The media review records
when the four crates are ready for a separately requested 0.1 release.
formats. Do not bump versions as part of these quests. The media review
found the four crates ready for a separately requested 0.1 release.

Their package boundaries are explicit:

Expand Down Expand Up @@ -90,7 +90,6 @@ do not add another media abstraction or a renderer crate during stabilization.

- [GPU conversion and NVENC](/quest/main/video-gpu-encode.md) - validate the GPU
conversion, resize and NVENC path for imported frames on NVIDIA hardware
- [Media release review](/quest/main/media-release-review.md) - verify the settled contracts before separately authorizing 0.1 releases

## Related

Expand Down
27 changes: 0 additions & 27 deletions quest/main/media-release-review.md

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -16,37 +16,36 @@ off the same way.

Today audio reserves but never follows. The capture driver takes the
reservation once the layout reveals the encoded rate
(`rs/moq-audio/src/encode/capture.rs:340`, `:471-474`) and nothing reads its
(`rs/moq-audio/src/encode/capture.rs`) and nothing reads its
grant; `Options::bandwidth` documents that
(`rs/moq-audio/src/encode/producer.rs:49-62`). Video already follows through
`rate::Control` (`rs/moq-video/src/encode/producer.rs:467-476`).
(`rs/moq-audio/src/encode/producer.rs`). Video already follows through
`moq_mux::rate::Control` (`rs/moq-video/src/encode/producer.rs`).

- Consume `moq_mux::rate`, keeping one shared policy; this quest adds audio
adaptation without growing a second implementation.
- The follow loop lives in `moq_audio::encode::Producer`, not the capture
driver: `Producer::new` (`producer.rs:255`) already takes `Options` with the
allocator (`:62`), so it reserves the configured bitrate against
driver: `Producer::new` already takes `Options` with the
allocator, so it reserves the configured bitrate against
`self.demand()`, holds a `Reservation::consumer()` plus a
`rate::Control`, and feeds each grant to `Encoder::set_bitrate`
(`rs/moq-audio/src/encode/encoder.rs:411`). Capture and moq-ffi
(`rs/moq-ffi/src/audio.rs:254`) build the Producer, so both adapt without
(`rs/moq-audio/src/encode/encoder.rs`). Capture and moq-ffi
(`rs/moq-ffi/src/audio.rs`) build the Producer, so both adapt without
their own loop; the capture driver's `_reservation` goes away. Public entry
points for a manual ceiling stay `Producer::set_bitrate` (`producer.rs:295`)
points for a manual ceiling stay `Producer::set_bitrate`
and `Encoder::set_bitrate`.
- Floor: `set_opus_bitrate` refuses anything outside
`opus::bitrate_floor(codec_rate, frame_size).max(500)` to
`300_000 * channels` (`encoder.rs:345-346`,
`rs/moq-audio/src/opus.rs:137-143`). `Policy::min` defaults to a tenth of
`300_000 * channels` (`encoder.rs`, `rs/moq-audio/src/opus.rs`). `Policy::min` defaults to a tenth of
the ceiling (`rs/moq-mux/src/rate.rs`); for Opus it is the codec floor, so a
grant below it clamps there and never errors. The reservation's ceiling is
the configured bitrate; only the policy target moves.
- PCM: `pcm::bitrate(sample_rate, channels)` is `pub(crate)`
(`rs/moq-audio/src/pcm.rs:9`), `Config::bitrate` is refused for it
(`encoder.rs:269-272`) and so is `set_bitrate` (`encoder.rs:412-414`). A PCM
(`rs/moq-audio/src/pcm.rs`), `Settings::bitrate` is refused for it
and so is `set_bitrate` (`encoder.rs`). A PCM
Producer reserves its fixed rate and runs no policy. That is the same
reserve-only usage passthrough imports use, so nothing new is added for it.
- Priority is unchanged: `PRIORITY` puts audio at 80 and video at 60
(`rs/hang/src/catalog/priority.rs:21-26`), so the allocator fills audio's
(`rs/hang/src/catalog/priority.rs`), so the allocator fills audio's
reservation before video sees a bit and audio is squeezed only once the link
cannot carry audio alone. Worth doing for that tail, not worth blocking on.

Expand Down
4 changes: 2 additions & 2 deletions quest/next/audio-jitter-target/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,8 @@ the branch still gets wrong is recorded in
the first gap between observed timestamps and the rise is immediate and
unclamped, so a tune-in across a stale group sets the target to seconds.

Native has no jitter buffer at all. `rs/moq-audio`'s decode `Config`
(`rs/moq-audio/src/decode/decoder.rs:60-80`) carries `max_age`, how far
Native has no jitter buffer at all. `rs/moq-audio`'s `decode::Options`
(`rs/moq-audio/src/decode/consumer.rs`) carries `max_age`, how far
playback may drift from the live edge before skipping a stalled group, and
`start`, where to begin on a track that already holds groups. Nothing pads the
buffer against uneven arrivals.
Expand Down
12 changes: 6 additions & 6 deletions quest/next/audio-jitter-target/native.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,18 @@ Video keeps its own path.

## Plan

`rs/moq-audio` has no jitter buffer at all. `decode::Config`
(`rs/moq-audio/src/decode/decoder.rs:60-80`) carries `max_age`, how far
`rs/moq-audio` has no jitter buffer at all. `decode::Options`
(`rs/moq-audio/src/decode/consumer.rs`) carries `max_age`, how far
playback may drift from the live edge before skipping a stalled group, applied
to the subscription and clamped to the track's retention in
`decode/consumer.rs:113`, and `start`, where to begin on a track that already
holds groups. Neither adds latency: the consumer skips only when newer data is
to the subscription and clamped to the track's retention, and `start`, where
to begin on a track that already holds groups; `decode::Config` selects the
backend only. Neither adds latency: the consumer skips only when newer data is
already that far ahead.

- Measure arrivals at the same point the browser does, on the container
consumer before the age budget can skip a group, so both languages estimate
from the same observation.
- Add the knob to `decode::Config`, additive on the `#[non_exhaustive]` struct,
- Add the knob to `decode::Options`, additive on the `#[non_exhaustive]` struct,
so it targets `main`. The spec quest settles the final name and shape before
this implementation starts; `delay` is the recommendation, matching the
browser while `max_age` remains the live-edge skip budget. Do not reopen or
Expand Down
9 changes: 2 additions & 7 deletions quest/next/capture-alsa-link.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,8 @@ vaapi/nvidia pattern and load the system library at runtime instead, falling
through to the next host when it is missing, so a build with the feature on
still links and starts driverless.

The current feature matrix is the contract to change: `capture` and
`playback` pull cpal with ALSA always linked, while
`pipewire = ["cpal/pipewire"]` and `pulseaudio = ["cpal/pulseaudio"]` activate
cpal without either. Make the host flags require `capture` or `playback` so
they never activate cpal alone, and document the supported combinations in
the moq-audio feature table with the same change. If cpal cannot load ALSA at
runtime in-tree, split that half into its own upstream quest holding the cpal
`capture` and `playback` pull cpal with ALSA always linked; the host flags
alone no longer activate it. If cpal cannot load ALSA at runtime in-tree, split that half into its own upstream quest holding the cpal
release as a plain-text `Required` condition, and this quest requires it.

Verify by building in the Nix shell, then running the shipped binary on a
Expand Down
4 changes: 2 additions & 2 deletions quest/next/jitter-flush-clock.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@ clock-free estimates from the media span of each emitted batch.
- Enforce the never-lower rule at the publisher, not only by convention.
`js/publish/src/catalog.ts:25-29` already refuses a decrease and a zero for
audio and video renditions; Rust does not. Add `moq_mux::Error::JitterDecreased`
and return it from `Rendition::set`, `Rendition::update`, and
`Rendition::estimate` (`rs/moq-mux/src/catalog/tracks.rs:444`, `:497`, `:486`)
and return it from `Rendition::set`, `Rendition::replace`, and
`Rendition::estimate` (`rs/moq-mux/src/catalog/tracks.rs`)
when the resolved jitter would drop below what the catalog already carries.
Extend the `MillisCeil` ceiling serialization (`rs/hang/src/catalog/millis.rs`)
to `TextConfig` (`rs/hang/src/catalog/text/mod.rs:124`), which audio and video
Expand Down
19 changes: 10 additions & 9 deletions quest/next/keyframe-trigger.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,20 @@
## Goal

An application publishing through the built-in capture path can ask for a
keyframe. Every encoder backend already forces an IDR and it is tested, but
nothing above the backend can reach it.
keyframe. `Encoder::cut()`, `Sink::cut()`, and the ffi/libmoq `cut` already
force one, refusing with `CutUnsupported` when a backend cannot, but the
turnkey capture paths have no way in.

## Plan

`Encoder::encode` takes `keyframe: bool` and each backend honors it (NVENC via
the `FORCEIDR` picture flag with `repeatSPSPPS` so the IDR carries its
parameter sets, deliberately not `pictureType` which `enablePTD` ignores;
openh264, VAAPI, VideoToolbox and Media Foundation the same way). What is
missing is a caller-facing trigger:
`Backend::encode(frame, cut)` honors a cut (NVENC via the `FORCEIDR` picture
flag with `repeatSPSPPS` so the IDR carries its parameter sets, deliberately
not `pictureType` which `enablePTD` ignores; openh264, VAAPI, VideoToolbox and
Media Foundation the same way) and `can_cut()` answers at open whether it can.
What is missing is a caller-facing trigger on the turnkey paths:

- `publish_capture` forces a keyframe on the first frame and otherwise rides
the backend's GOP cadence, with no way in.
- `publish_capture` relies on every backend opening with a keyframe and
otherwise rides the GOP cadence; its `Options` carry no trigger.
- `js/publish`'s encode path already calls `encoder.encode(frame, { keyFrame })`,
but `lastKeyframe` is a closure-local `let` with no external trigger.
`Config.keyframeInterval` is cadence, not on demand.
Expand Down
4 changes: 2 additions & 2 deletions quest/next/nvenc-recovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ panic, and a rejected rate change leaves the last accepted settings intact.

## Plan

Registration can succeed before mapping fails in safe/buffer.rs, and several
destructors call expect on driver cleanup. Session::reconfigure mutates retained
Registration rollback and non-panicking destructors landed with the ownership
work (#3834, #3835, #3838). Session::reconfigure still mutates retained
bitrate/VBV fields before the driver accepts the change; a rejected zero-rate
update can corrupt the basis of the next proportional update.

Expand Down
2 changes: 1 addition & 1 deletion quest/next/play-audio-rendition-gap.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Worth weighing, and the choice is A/V policy:
largest change to `Playback`.
- Hand the replacement the retired sink rather than opening a new one, so the
ring is already at depth. Keeps one task per kind, but only works when the
rate and channel count match.
sample rate and `Layout` match.

Retiring a rendition mid-track is the case to reproduce: a transcode ladder
resizing under a source that changed resolution is the one that happens in
Expand Down
8 changes: 5 additions & 3 deletions quest/next/qos/stats/encoder-feedback.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,9 @@ prefix. Keyframe requests stay out.

## Plan

- `encode::Config` in `rs/moq-video` and `rs/moq-audio` gains
`feedback: Option<feedback::Consumer>`, a handle built from an
- `encode::Options` in `rs/moq-video` and `rs/moq-audio` (the producer
options, beside `bandwidth`) gains `feedback: Option<feedback::Consumer>`,
a handle built from an
`origin::Consumer` and a prefix: it consumes announcements under the
prefix, keeps one `moq_stats::Consumer<hang::Stats>` per `.stats` broadcast
requesting `<own path>/subscriber.json`, and folds the reports into one
Expand All @@ -22,7 +23,8 @@ prefix. Keyframe requests stay out.
that goes backwards is a restarted viewer and resets that baseline. Viewers
that stop reporting age out on the stats interval, so one stall long ago
never lowers the target forever.
- `rate.rs` takes that signal beside the bandwidth estimate: a stalled share
- `moq_mux::rate::Control` takes that signal beside the bandwidth estimate:
a stalled share
above a threshold steps the target down like a bandwidth drop, recovery
follows the existing attack curve, and the estimate stays the ceiling.
Audio follows the same signal with its narrower ladder.
Expand Down
Loading
Loading