From 14b0f372b51a69c4653d31f887362078e2d1ec44 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 20 Sep 2026 07:10:48 -0700 Subject: [PATCH 1/2] docs(quest): plan media API stabilization before 0.1 Co-Authored-By: GPT-6 --- quest/README.md | 9 +-- quest/m0/README.md | 64 +++++++++++++++++-- quest/m0/audio-aec.md | 24 +++++++ quest/m0/audio-config.md | 41 ++++++++++++ quest/m0/audio-fec.md | 26 ++++++++ quest/m0/audio-playback.md | 22 +++++++ quest/m0/audio-publication.md | 28 ++++++++ quest/m0/media-features.md | 33 ++++++++++ quest/m0/media-rate-policy.md | 26 ++++++++ quest/m0/media-release-review.md | 44 +++++++++++++ quest/m0/nvenc-loading.md | 22 +++++++ quest/m0/nvenc-resources.md | 41 ++++++++++++ quest/m0/video-frames.md | 35 ++++++++++ quest/m0/video-gop.md | 46 +++++++++++++ quest/m0/video-output.md | 33 ++++++++++ quest/m0/video-thread-ownership.md | 27 ++++++++ quest/m0/video-timing.md | 30 +++++++++ quest/m1/README.md | 4 +- ...evc-and-av1-support-in-the-nvidia-codec.md | 50 ++++++++------- ...bandwidth-grant-in-moq-audio-instead-of.md | 19 +++--- quest/m2/README.md | 18 ++++-- quest/m2/audio-buffers.md | 31 +++++++++ quest/m2/audio-capture-format.md | 24 +++++++ quest/m2/audio-codecs/README.md | 13 ++-- quest/m2/audio-codecs/decode-backend.md | 14 ++-- quest/m2/audio-codecs/decode-mediacodec.md | 5 +- quest/m2/audio-codecs/encode-backend.md | 8 +-- quest/m2/audio-codecs/layout.md | 22 ++++--- quest/m2/audio-opus-input.md | 28 ++++++++ quest/m2/capture-ergonomics.md | 24 ++++--- quest/m2/color-model.md | 10 +++ quest/m2/decoded-frames.md | 6 +- quest/m2/intra-refresh/README.md | 4 +- quest/m2/intra-refresh/encode-config.md | 26 ++++---- quest/m2/intra-refresh/v4l2-refresh.md | 15 +++-- quest/m2/ladder/README.md | 5 +- quest/m2/mobile/ffi-video-consumer.md | 1 - quest/m2/nvenc-recovery.md | 28 ++++++++ quest/m2/nvenc-reuse.md | 30 +++++++++ quest/m2/obs-moq-video/adapter.md | 4 ++ quest/m2/obs-moq-video/macos.md | 4 +- quest/m2/obs-moq-video/rate-control.md | 6 +- quest/m2/obs-moq-video/source.md | 2 +- quest/m2/obs-moq-video/windows.md | 4 +- quest/m2/transcode-resources.md | 32 ++++++++++ quest/m2/transcode-source.md | 28 ++++++++ quest/m2/video-render-resources.md | 34 ++++++++++ quest/m3/README.md | 6 +- quest/m3/audio-loss-recovery.md | 28 ++++++++ quest/m3/audio-opus-backend.md | 28 ++++++++ quest/m3/decode-drain.md | 33 ---------- quest/m3/mobile-capture-android.md | 9 +-- quest/m3/mobile-capture-ios.md | 8 ++- quest/m3/video-codec-coverage.md | 34 ++++++++++ 54 files changed, 1041 insertions(+), 155 deletions(-) create mode 100644 quest/m0/audio-aec.md create mode 100644 quest/m0/audio-config.md create mode 100644 quest/m0/audio-fec.md create mode 100644 quest/m0/audio-playback.md create mode 100644 quest/m0/audio-publication.md create mode 100644 quest/m0/media-features.md create mode 100644 quest/m0/media-rate-policy.md create mode 100644 quest/m0/media-release-review.md create mode 100644 quest/m0/nvenc-loading.md create mode 100644 quest/m0/nvenc-resources.md create mode 100644 quest/m0/video-frames.md create mode 100644 quest/m0/video-gop.md create mode 100644 quest/m0/video-output.md create mode 100644 quest/m0/video-thread-ownership.md create mode 100644 quest/m0/video-timing.md create mode 100644 quest/m2/audio-buffers.md create mode 100644 quest/m2/audio-capture-format.md create mode 100644 quest/m2/audio-opus-input.md create mode 100644 quest/m2/nvenc-recovery.md create mode 100644 quest/m2/nvenc-reuse.md create mode 100644 quest/m2/transcode-resources.md create mode 100644 quest/m2/transcode-source.md create mode 100644 quest/m2/video-render-resources.md create mode 100644 quest/m3/audio-loss-recovery.md create mode 100644 quest/m3/audio-opus-backend.md delete mode 100644 quest/m3/decode-drain.md create mode 100644 quest/m3/video-codec-coverage.md diff --git a/quest/README.md b/quest/README.md index d847086e78..1f756f1563 100644 --- a/quest/README.md +++ b/quest/README.md @@ -7,7 +7,9 @@ grouped into milestones ordered by priority. ## Plan -m0 supplies the GPU video path for the Pronto desktop. m1 is the dev branch line. m2 is the next agent +m0 contains the immediate media work: the Pronto desktop GPU path and the four +0.0.x media crate API gates before 0.1. m1 is the dev branch line. m2 is the +next agent wave across reliability, features, performance, and planning. Unsettled quests may stay in m2 for planning; their implementation waits for the required decisions. m3 holds later features, design studies, experiments, @@ -25,9 +27,8 @@ into m2, and opened m4 for the hardware- and partner-gated quests. ## Quests -- [m0: Pronto GPU video](/quest/m0/README.md) - reusable Vulkan/CUDA ownership, - GPU conversion and NVENC for the desktop demo - +- [m0: immediate media](/quest/m0/README.md) - Pronto GPU delivery and the + audio, video, transcode, and NVENC contracts required before 0.1 - [m1: the dev line](/quest/m1/README.md) - the breaking API and wire changes, the merge gates, and the merge itself - [m2: next wave](/quest/m2/README.md) - implementation and planning across diff --git a/quest/m0/README.md b/quest/m0/README.md index a3f5e13efc..ea384587c3 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -1,10 +1,51 @@ -# m0: Pronto GPU video +# m0: immediate media ## Goal -Supply the reusable GPU media support needed to remove raw-pixel CPU transfers -from the Pronto CARLA demo on its Linux/NVIDIA desktop. This is the immediate -priority; the product integration and installation live in moq.pro. +Settle the public contracts of `moq-audio`, `moq-video`, `moq-transcode`, and +`moq-nvenc` before their first 0.1 releases, while supplying the reusable GPU +media support needed to remove raw-pixel CPU transfers from the Pronto CARLA +demo on its Linux/NVIDIA desktop. Product integration and installation live in +moq.pro. + +## Plan + +The Pronto GPU quests remain an independent deliverable within m0. They define +portable Vulkan/CUDA ownership, safe partial NVENC initialization, and a strict +GPU conversion path without changing the scope of the pre-0.1 API audit. + +The four audited crates are 0.0.x, so their changes target main. Adapt callers +in other packages without breaking their published APIs, C layouts, or wire +formats. Do not bump versions as part of these quests. The final review records +when the four crates are ready for a separately requested release. + +The agreed direction is small, honest APIs: typed PCM layouts, rational video +rates, extensible GOP and frame records, explicit ownership, and no knobs that +claim behavior they do not provide. Synchronous codecs remain public and +thread-confined; async sinks own codec execution. Native versus CPU output is a +choice, not a promise that every native backend yields a GPU surface. + +OpenH264 becomes optional but stays enabled by default. Rendering becomes +opt-in; inexpensive native codec defaults remain. Hardware libraries already +loaded at runtime do not need a toolkit dependency added to the build. + +Each quest updates the existing crate documentation, examples, and affected +callers with its change. Contract tests run in CI; feature checks exercise each +crate independently, since workspace feature unification hides missing gates. +Cross-platform compilation and hardware execution are separate evidence. + +The audit was source inspection at `b2ef453e67`, not fresh compilation or +benchmarking. All public module families and their in-tree consumers were +reviewed; the generated NVIDIA ABI was inspected only where wrapper safety +depended on it. Performance concerns remain hypotheses until measured. + +API-preserving implementation, codec additions, allocation work, and hardware +proof remain in the existing backlog. Keep audio's integrated packetizing +Producer and transcode's validated Ladder and coalescing active cursor. Keep +one video Frame/Surface hierarchy and its deliberate native/wgpu type interop; +do not add another media abstraction or a renderer crate during stabilization. +Preserve the documented ability to abort publication after finish unless an +actual ownership fix requires a replacement error-finalization contract. ## Quests @@ -13,6 +54,21 @@ priority; the product integration and installation live in moq.pro. - [NVENC registration rollback](/quest/m0/nvenc-registration.md) - release resources when mapping fails after registration - [GPU conversion and NVENC](/quest/m0/video-gpu-encode.md) - convert, resize and encode imported frames without CPU pixel transfers or fallback +- [NVENC resources](/quest/m0/nvenc-resources.md) - a small safe facade retains submitted resources through completion +- [NVENC loading](/quest/m0/nvenc-loading.md) - unavailable or incompatible drivers return errors instead of panicking +- [Codec threads](/quest/m0/video-thread-ownership.md) - synchronous codec handles cannot escape their owning thread +- [Media features](/quest/m0/media-features.md) - OpenH264 can be excluded, rendering is opt-in, and feature aliases disappear +- [Shared rate policy](/quest/m0/media-rate-policy.md) - the planned public namespace move happens before 0.1 +- [Audio configuration](/quest/m0/audio-config.md) - PCM layout, codec settings, and subscription policy have distinct contracts +- [Audio publication](/quest/m0/audio-publication.md) - callers get demand authority and supported options, not internal transport or resampler machinery +- [Remove ineffective FEC](/quest/m0/audio-fec.md) - no public flag promises redundancy the encoder never emits +- [AEC ownership](/quest/m0/audio-aec.md) - one microphone owns an adaptive canceller and controls remain shareable +- [Playback outcome](/quest/m0/audio-playback.md) - nonblocking writes report accepted and dropped audio +- [Video output](/quest/m0/video-output.md) - codec output and subscription policy are separate, with native or CPU frames +- [Video frames](/quest/m0/video-frames.md) - conversions preserve typed pixels and frame records can grow +- [Video timing](/quest/m0/video-timing.md) - capture time and fractional frame rates survive capture, encoding, and transcode +- [Video GOP](/quest/m0/video-gop.md) - the group contract is extensible before intra-refresh implementation +- [Media release review](/quest/m0/media-release-review.md) - verify the settled contracts before separately authorizing 0.1 releases ## Related diff --git a/quest/m0/audio-aec.md b/quest/m0/audio-aec.md new file mode 100644 index 0000000000..d9b5e5f70b --- /dev/null +++ b/quest/m0/audio-aec.md @@ -0,0 +1,24 @@ +# [M] Exclusive microphone ownership for echo cancellation + +## Goal + +One microphone owns an adaptive echo canceller. Shareable controls cannot be +used to attach the same adaptive state to another microphone, and creating a +second active reference does not silently disable the first. + +## Plan + +Engine::canceller replaces the engine's previous tap, and Canceller clones +share adaptive state despite the documented one-microphone restriction. Split +the exclusive attachment from a cloneable control handle. Refuse conflicting +attachments explicitly, with release on drop and a clear ownership path from +playback reference through capture teardown. + +Keep the current audio-processing implementation. No multi-microphone mixing +or new processing backend is part of this change. Use fake capture/reference +sources to test a second attachment, replacement after drop, control clones, +and teardown while processing. Run the ownership regressions in CI without +real devices, retaining existing signal-quality tests separately. + +Public API: AEC construction, capture attachment, and handle ownership change. +Wire: none. Update speakerphone examples and existing audio documentation. diff --git a/quest/m0/audio-config.md b/quest/m0/audio-config.md new file mode 100644 index 0000000000..32f66ee515 --- /dev/null +++ b/quest/m0/audio-config.md @@ -0,0 +1,41 @@ +# [L] Separate PCM layout, codec settings, and playback policy + +## Goal + +One audio codec-settings type describes what the codec accepts. Source PCM +format/rate/layout and MoQ subscription/output policy are separate, and the +low-level decoder can select a backend without changing its constructor later. + +## Plan + +Encoder::encode accepts codec-sized interleaved f32 at the codec rate, while +its Config also describes source format/rate that only Producer converts. +Options repeats the codec settings. Keep Producer's useful accumulation and +normalization behavior, but compose one settings shape rather than copying a +second set of fields. Keep backend traits private; establish the configuration +and backend-selection entry points with today's implementations first. + +Replace ambiguous Rust channel-count configuration with an extensible Layout +contract. Implement today's modes only. Preserve existing multichannel PCM +passthrough without inventing speaker positions from a count: an unspecified +discrete layout must not silently gain a spatial downmix. Named layouts have +a documented canonical speaker order; unsupported conversions are refused. +Surround codecs and new mixer capabilities remain in the existing backlog. + +Separate codec configuration from Consumer subscription and output conversion +options. Output rate/layout describe decoded samples, not merely catalog claims. +Refuse settings that do not apply to the chosen codec, including DTX on PCM. +Keep Input and settings construction extensible without compatibility aliases. + +Adapt moq-ffi/libmoq internals while preserving their published signatures, +record layouts, channel-count conventions, and sentinel behavior. Tests cover +source/codec rate separation, format conversion, layout mismatch, passthrough, +unsupported settings, and backend refusal in CI. Update existing audio docs. + +Public API: Rust audio configuration, layout, and decoder construction change. +Wire and published binding shapes: unchanged. + +## Related + +- [Audio layouts](/quest/m2/audio-codecs/layout.md) - later named surround layouts and mixing +- [Decode seam](/quest/m2/audio-codecs/decode-backend.md) - later platform selection implementation diff --git a/quest/m0/audio-fec.md b/quest/m0/audio-fec.md new file mode 100644 index 0000000000..ee7ab80899 --- /dev/null +++ b/quest/m0/audio-fec.md @@ -0,0 +1,26 @@ +# [S] Remove the ineffective audio FEC flag + +## Goal + +No moq-audio option claims to enable loss redundancy that the shipped encoder +never emits. A real loss-recovery policy remains separately scoped. + +## Plan + +Config and Options only set OPUS_SET_INBAND_FEC. The pinned unsafe-libopus +initializes expected loss to zero, where its decide_fec returns false. Our +decoder also never requests FEC recovery. The existing test reads the control +flag rather than recovering lost audio, and no production caller enables it. + +Remove the ineffective boolean from both public configuration paths and their +documentation/tests before 0.1. Keep current Opus packetization, DTX, and +ordinary packet-loss concealment behavior. Add a focused configuration test +for the retained options; do not replace the flag with another unproven knob. + +Public API: removes fec from Rust 0.0.x configuration. Wire: no change to the +behavior currently emitted. Real recovery needs loss metadata, sequencing, +decoder policy, and an end-to-end loss fixture before a new API is exposed. + +## Related + +- [Audio loss recovery](/quest/m3/audio-loss-recovery.md) - decide a tested policy when a consumer needs it diff --git a/quest/m0/audio-playback.md b/quest/m0/audio-playback.md new file mode 100644 index 0000000000..52eeca1d0e --- /dev/null +++ b/quest/m0/audio-playback.md @@ -0,0 +1,22 @@ +# [S] Report the outcome of nonblocking playback writes + +## Goal + +A playback writer can distinguish accepted audio from overflow or output-not- +ready drops without blocking the real-time device path. + +## Plan + +Sink::write currently discards the accepted-frame count on overflow and returns +success for output-not-ready. Replace the success value with an extensible +typed outcome that reports accepted and dropped sample frames; name units +explicitly so stereo samples are not counted twice. Keep failure distinct from +intentional real-time dropping, including all-accepted and zero-accepted cases. + +Propagate or deliberately consume the outcome in existing callers without +changing published binding signatures. Test partial acceptance, output not +ready, conversion/resampling units, and closed/error paths with deterministic +device-independent fixtures in CI. Document how callers observe drops without +adding a retry loop that increases latency. + +Public API: playback write's success result changes. Wire: none. diff --git a/quest/m0/audio-publication.md b/quest/m0/audio-publication.md new file mode 100644 index 0000000000..cef94e1fbc --- /dev/null +++ b/quest/m0/audio-publication.md @@ -0,0 +1,28 @@ +# [S] Expose audio publication authority without its internals + +## Goal + +Audio callers can observe demand without bypassing packetization, and the +public surface stops exposing unused resampler mechanics and duplicated +capture-construction arguments. + +## Plan + +Replace Producer::track with demand, matching video. A shared transport Producer +can append or finish outside the audio publisher's framing and padding rules; +the external consumers inspected only need its demand handle. Retain privileged +access privately for capture ownership where necessary. + +Make the standalone Resampler private: no in-tree external consumer needs its +window/timestamp machinery, and its corresponding held-at accessor is already +private. Let publish_capture take the existing PublicationOptions instead of +reconstructing it from five arguments. Reserve extension points on Frame and +Encoded records through their constructors before adding future metadata. + +Preserve arbitrary-sized PCM writes, priming, terminal padding, and the +intentional ability to abort after finish. Adapt bindings and examples without +changing their public signatures. Test demand lifecycle and publication close +behavior through the supported API in CI; update crate examples and rustdoc. + +Public API: remove privileged/helper exports and simplify capture arguments. +Wire: none. diff --git a/quest/m0/media-features.md b/quest/m0/media-features.md new file mode 100644 index 0000000000..b7c345fbe8 --- /dev/null +++ b/quest/m0/media-features.md @@ -0,0 +1,33 @@ +# [M] Optional native compilation and opt-in rendering + +## Goal + +A consumer can build these media crates without OpenH264's C++ compilation or +the rendering stack. Defaults retain a working software H.264 fallback and +the existing inexpensive native backends. + +## Plan + +Make OpenH264 and its sys dependency one optional backend feature, enabled by +default. Remove render from moq-video's defaults. Keep nvidia and mediacodec +defaults and the existing opt-in system-library/libclang features. Remove the +nvenc/nvdec compatibility aliases in video and transcode before 0.1. + +Forward the software-backend choice through transcode and explicitly select +features in workspace consumers, tests, and examples. A hardware-only build +without a usable codec refuses construction; Software and Named selection must +never silently use another class of backend. No mandatory toolchain is added +to moq-nvenc, whose bindings are checked in and drivers are loaded at runtime. + +Audio keeps its pure-Rust AAC default. Make PipeWire/PulseAudio host flags avoid +activating cpal when neither capture nor playback needs it, and document the +capability combination. Do not split encode/decode into more features without +evidence of useful savings. + +Add independent no-default, software-only, native-only, and rendering feature +checks for the affected crates to CI. Inspect dependency graphs to prove +OpenH264/wgpu/cpal are absent when excluded; workspace-wide builds are not +that proof. Preserve codec-independent tests when a backend is disabled. + +Public API: Cargo defaults and supported feature names change. Wire: none. +Update feature tables and build prerequisites with the same change. diff --git a/quest/m0/media-rate-policy.md b/quest/m0/media-rate-policy.md new file mode 100644 index 0000000000..f91a965988 --- /dev/null +++ b/quest/m0/media-rate-policy.md @@ -0,0 +1,26 @@ +# [S] Settle the shared media rate-policy namespace + +## Goal + +The existing video rate policy lives at the shared location already planned +for audio and transcode, before removing its video namespace would break 0.1. +Actual audio/ladder bandwidth adaptation remains deferred. + +## Plan + +The bandwidth quest currently plans to delete the public moq_video::encode::rate +module and move Policy/Control to moq_mux::rate. Perform that API relocation +now, adapt video's existing consumer and docs, and remove the old export +without a compatibility shim. Preserve one policy implementation. + +Move its tests with it and make construction/update agree on bounds: the +current initial target uses max.max(min), although update lowers an excessive +minimum to the ceiling. A policy must never initialize above its ceiling. +Test inverted bounds, initial target, decreases, hysteresis, and recovery in CI. + +Public API: removes the old 0.0.x video namespace and adds the shared policy +under moq-mux without breaking that package's existing exports. Wire: none. + +## Related + +- [Audio bandwidth](/quest/m2/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md) - later adaptation uses the shared policy diff --git a/quest/m0/media-release-review.md b/quest/m0/media-release-review.md new file mode 100644 index 0000000000..4e03875814 --- /dev/null +++ b/quest/m0/media-release-review.md @@ -0,0 +1,44 @@ +# [S] Verify the four media contracts before 0.1 + +## Goal + +Record that the agreed audio, video, transcode, and NVENC API changes have +landed, or return concrete remaining blockers, before a separate version-bump +request. Completing this review does not bump or publish packages. + +## Plan + +Review the final public surface and examples after the prerequisite quests. +Check that deferred backend, surround, refresh, color, and performance work can +use the extension points without another planned replacement of these APIs. +Search the living backlog for obsolete signatures and remove completed work. + +Verify independent crate feature builds, default and backend-free refusal, +ownership/cancellation tests, docs, and migrated callers on the exact release +candidate. Use the Nix check/test recipes and platform CI; record skipped +hardware cases separately from compilation. Confirm package manifests do not +reintroduce mandatory native compilation through feature unification. + +No published FFI/C layout or wire break is authorized by this stabilization +line. Any newly discovered break outside these four 0.0.x crates needs its own +maintainer decision and appropriate branch. No performance claim is accepted +without measurements; unresolved optimization work alone does not block 0.1. + +Public API/wire impact: review only. + +## Required + +- [NVENC resources](/quest/m0/nvenc-resources.md) - safe ownership is enforced +- [NVENC loading](/quest/m0/nvenc-loading.md) - driver refusal is fallible +- [Codec threads](/quest/m0/video-thread-ownership.md) - direct handles have correct auto traits +- [Media features](/quest/m0/media-features.md) - independent build choices work +- [Shared rate policy](/quest/m0/media-rate-policy.md) - later adaptation will not remove a stabilized namespace +- [Audio configuration](/quest/m0/audio-config.md) - PCM and codec contracts are settled +- [Audio publication](/quest/m0/audio-publication.md) - only intended authority is public +- [Remove ineffective FEC](/quest/m0/audio-fec.md) - no unsupported promise remains +- [AEC ownership](/quest/m0/audio-aec.md) - microphone attachment is exclusive +- [Playback outcome](/quest/m0/audio-playback.md) - drops are observable +- [Video output](/quest/m0/video-output.md) - output and subscription contracts are distinct +- [Video frames](/quest/m0/video-frames.md) - pixels and extension points are typed +- [Video timing](/quest/m0/video-timing.md) - timestamps and rates retain their meaning +- [Video GOP](/quest/m0/video-gop.md) - deferred refresh does not replace the group API diff --git a/quest/m0/nvenc-loading.md b/quest/m0/nvenc-loading.md new file mode 100644 index 0000000000..ce0bacf59d --- /dev/null +++ b/quest/m0/nvenc-loading.md @@ -0,0 +1,22 @@ +# [S] Fallible NVENC loading + +## Goal + +Missing libraries, missing entry points, and unsupported NVIDIA driver versions +return errors through codec selection instead of aborting a process. + +## Plan + +`safe/api.rs` initializes the public lazy `ENCODE_API` by panicking. The +moq-video probe only establishes library presence, so an installed but +incompatible driver can still reach a version assertion. Mirror the fallible +loading contract of `cuvid::Api::get`; keep function-table initialization +private to the safe facade, and make public errors extensible. + +Test absent libraries, absent symbols, an old API version, and successful +initialization with an injected loader in normal CI. Automatic selection must +fall through to another eligible backend; an explicitly named NVENC request +must return the reason it cannot open. Do not catch a panic as the design. + +Public API: fallible loader access and extensible error variants in moq-nvenc. +Wire: none. Update its runtime-loading documentation and moq-video errors. diff --git a/quest/m0/nvenc-resources.md b/quest/m0/nvenc-resources.md new file mode 100644 index 0000000000..e6acc58e93 --- /dev/null +++ b/quest/m0/nvenc-resources.md @@ -0,0 +1,41 @@ +# [L] A safe NVENC resource and completion contract + +## Goal + +The safe moq-nvenc facade covers its actual consumers and cannot submit forged +handles, dangling pointer-bearing settings, or buffers that disappear before +the driver finishes. Raw SDK access stays explicitly unsafe. Buffer reuse and +bitrate reconfiguration can coexist without a self-referential wrapper. + +## Plan + +`safe/buffer.rs` exposes implementable input/output traits and a safe generic +registration method taking an arbitrary pointer plus an unrelated marker. +`safe/session.rs` accepts raw picture structs containing pointers, and +`safe/encoder.rs` shallow-copies pointer-bearing configuration. Submission +borrows buffers only for the call, although completion can come later. + +Narrow the safe facade to operations needed by moq-video. Seal handle traits +unless an external implementation is required. Retain the backing allocation, +device/context, registration, and configuration data for their actual use +lifetimes; make escape hatches unsafe with explicit obligations. Completion +must release resources in the right order on success, failure, cancellation, +and session teardown. Avoid recreating the entire SDK as a second codec API. + +Choose owned resource/session relationships that permit reuse and rate changes; +the current Session borrows prohibit mutable reconfiguration while buffers +exist. Pooling itself is deferred. Repeated SDK EOS is permitted and is not +by itself a reason to make that operation terminal. + +Use a test driver boundary to verify pending submissions, delayed completion, +cross-session rejection, partial initialization, and teardown in CI without +an NVIDIA device. Add compile-time coverage that safe callers cannot forge a +handle or free in-flight storage. Keep hardware encode/drain proof separate. + +Public API: narrowing and ownership changes in moq-nvenc, with moq-video +adapted in the same PR. Wire: none. + +## Related + +- [NVENC reuse](/quest/m2/nvenc-reuse.md) - pooling follows the ownership contract +- [NVENC recovery](/quest/m2/nvenc-recovery.md) - failure and reconfiguration behavior beyond the ownership repair diff --git a/quest/m0/video-frames.md b/quest/m0/video-frames.md new file mode 100644 index 0000000000..ba2e8c3758 --- /dev/null +++ b/quest/m0/video-frames.md @@ -0,0 +1,35 @@ +# [M] Typed pixels and one frame-conversion API + +## Goal + +Frame conversion keeps size and color attached to pixels, has one configured +operation per conversion, and leaves room for later packet/frame metadata. + +## Plan + +Collapse resize/resize_with and the RGB/BGRA configured and compatibility +spellings into one operation each. The consuming I420 path must return typed +pixels rather than dropping geometry/color into Bytes; explicit byte extraction +remains available. Use Size where both dimensions are required. Retain one +Frame/Surface hierarchy and existing native platform interoperability. Later +native imports must extend this contract additively; do not stabilize a +construction shape that the already-planned OBS bridges must replace. + +Make frame and encoded-packet records extensible through constructors so later +flags, DTS, or color metadata do not require replacing them. Do not prematurely +add fields for codecs that are not implemented. Preserve backing allocation +ownership through borrowed/owned conversion and delayed consumers. + +Validate dimensions and byte counts with checked arithmetic at every public +constructor/length boundary; an enormous or malformed image returns an error +instead of overflowing before validation. Tests cover extreme dimensions, +same-byte-count transposed images, metadata preservation, and borrowed versus +owned results in CI. Update all consumers and existing conversion docs. + +Public API: conversion signatures/results, construction, and record +extensibility change. Wire and published C layouts: unchanged. + +## Related + +- [Decoded frames](/quest/m2/decoded-frames.md) - later binding ownership reuses this same frame +- [Color model](/quest/m2/color-model.md) - later color/HDR implementation must use the extension points diff --git a/quest/m0/video-gop.md b/quest/m0/video-gop.md new file mode 100644 index 0000000000..face926722 --- /dev/null +++ b/quest/m0/video-gop.md @@ -0,0 +1,46 @@ +# [M] An extensible video group configuration + +## Goal + +The video API expresses group structure without an integer field that must be +replaced when intra refresh arrives. The current keyframe mode keeps working; +new backend modes remain deferred. + +## Plan + +Replace the integer gop with an extensible typed group configuration. Support +the existing keyframe interval first and refuse invalid intervals. Rename the +forced group-boundary request to cut consistently across direct codecs and +async sinks, preserving today's IDR behavior. Do not keep a compatibility alias. + +Settle the rejection point before implementing this quest. Today keyframe() +only queues a request, and backend errors surface from encode(). The deferred +V4L2 refresh plan requires unsupported cuts to be refused, but that does not +by itself require a Result from cut(). Keeping the queued request infallible +and reporting rejection from the next encode is the simpler candidate; a +fallible cut would need a concrete reason for immediate acknowledgement. +The maintainer has not chosen between these contracts yet. Reconcile the +current V4L2 best-effort fallback with the chosen guarantee: it disables +unsupported force-keyframe requests and waits for a scheduled keyframe. Test +that case explicitly rather than assuming every backend honors the request. + +Adapt capture, transcode's group-boundary cuts and eight-second backstop, and +in-tree binding implementations. Preserve externally published binding shapes. +The future refresh mode can add a variant and implement its different grouping +semantics without replacing the field or promising unsupported output today. +Do not depend on catalog warmup or an intra-refresh hardware backend here. + +Tests cover a forced cut, buffered output, repeated group boundaries, invalid +configuration, and consistent direct/Sink behavior in CI. Update current GOP +documentation and leave the deferred refresh quest owning warmup/wire behavior. + +Public API: typed GOP and cut naming in Rust video/transcode callers. Wire: no +change to current keyframe grouping. + +## Required + +- [Video timing](/quest/m0/video-timing.md) - interval defaults derive from the settled rate type + +## Related + +- [Refresh groups](/quest/m2/intra-refresh/encode-config.md) - later refresh variant and grouping implementation diff --git a/quest/m0/video-output.md b/quest/m0/video-output.md new file mode 100644 index 0000000000..85ace40319 --- /dev/null +++ b/quest/m0/video-output.md @@ -0,0 +1,33 @@ +# [M] Separate video decoder output from subscription policy + +## Goal + +Low-level video decode settings describe the codec and its frame output; +Consumer options describe subscription behavior. Callers choose native or CPU +output without backend-specific GPU booleans or redundant acceleration modes. + +## Plan + +Config currently combines backend/resize with start/max_age that bare Decoder +ignores. gpu_frames only affects VAAPI, even though other backends return GPU +surfaces regardless; Acceleration::Gpu behaves like Auto and can fall back. + +Separate those layers. Establish an extensible native-or-CPU output choice: +native permits the backend's natural representation, including CPU, while CPU +produces the documented typed CPU pixels. Conversion is explicit. Retain the +efficient existing default behavior rather than promising GPU residency or +requiring a download accidentally. Remove the redundant GPU preference. +Keep best-effort decoder scaling explicitly identified as a hint, distinct +from the exact-size frame conversion operation. + +Adapt live/fetch transcode paths, rendering, and binding internals. Test config +propagation with CPU and fake-native backends, explicit conversion, a backend +without scaling, and subscription start/max_age separately. Compile the platform +paths through CI and preserve unsupported-backend refusal. + +Public API: decoder/consumer configuration and output policy change. Wire and +published binding layouts: unchanged. Update examples and cancellation docs. + +## Required + +- [Video frames](/quest/m0/video-frames.md) - typed CPU results used by the output contract diff --git a/quest/m0/video-thread-ownership.md b/quest/m0/video-thread-ownership.md new file mode 100644 index 0000000000..a72ee324c0 --- /dev/null +++ b/quest/m0/video-thread-ownership.md @@ -0,0 +1,27 @@ +# [M] Enforce synchronous codec thread ownership + +## Goal + +Public synchronous video Encoder and Decoder handles cannot be moved or dropped +on the wrong thread. Async Sink handles retain their supported worker-owned +execution model. + +## Plan + +The private backend traits require Send, and the Windows implementations justify +unsafe Send by confinement to a Sink worker. Public direct constructors bypass +that confinement. Media Foundation's ComGuard must balance initialization and +destruction on the thread that opened it. + +Keep both synchronous and async APIs. Enforce thread confinement in the type +system instead of documentation or an unsafe Send promise. Construct, use, +and destroy the backend on its worker without requiring the codec itself to +cross threads. Preserve platform frame affinity, including macOS native frames. +Balance successful COM initialization if subsequent MF startup fails. + +Add compile-time ownership coverage and a fake backend recording construction, +calls, cancellation, and destruction threads. Keep the existing rule that a +cancelled queued Sink request poisons further use; document that Consumer +reads inherit it. Wire Windows compilation into the existing platform CI lane. + +Public API: synchronous codec auto-trait guarantees become stricter. Wire: none. diff --git a/quest/m0/video-timing.md b/quest/m0/video-timing.md new file mode 100644 index 0000000000..b97af59933 --- /dev/null +++ b/quest/m0/video-timing.md @@ -0,0 +1,30 @@ +# [L] Capture timestamps and rational video rates + +## Goal + +Capture supplies a timestamped frame, and fractional frame rates survive native +capture, encoder configuration, and transcode without becoming integer fps. + +## Plan + +Capture currently returns only a Surface and publication timestamps it when +dequeued. Camera enumeration exposes a rational Rate, but requested/negotiated +capture and encode use integers; transcode rounds the catalog rate. Preserve +capture time before queueing and use one validated rational rate vocabulary +across these four crates. + +Define the capture clock mapping and preserve source timestamps through resize, +encode, and buffering. Prefer native capture timestamps when available; record +the fallback timestamp at acquisition, never at dequeue. Do not expose unrelated +device clocks as if they were a shared broadcast epoch. Preserve audio/video +clock alignment in the capture publisher and keep unknown rates explicit. + +Test 30000/1001 and 60000/1001, zero denominator, rate overflow, queued capture +delay, monotonic mapping, and transcode catalog/meter propagation without +hardware. Platform fixtures validate native timestamps separately. Keep wire +representations and already-published binding signatures compatible; converting +to their existing representations happens at that boundary with explicit rules. +Update capture/encoding docs and examples with the clock and rate units. + +Public API: capture frame result and video/transcode rate types change. No wire +schema or binding layout change. diff --git a/quest/m1/README.md b/quest/m1/README.md index 3adc43313a..1341080bf6 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -15,7 +15,9 @@ stays here only if it breaks a published API or wire, or gates the merge; names what gates the release after it. Work that is identical on `main`, additive, or targets a `0.0.x` crate lives in [m2](/quest/m2/README.md) even when it builds on dev-only code; it starts on -`main` after the merge. The 2026-09-12 grooming applied that rule to every +`main` after the merge. The four 0.0.x media crates are the explicit exception: +their pre-0.1 contracts live in [m0](/quest/m0/README.md) and can land on main +without the dev merge. The 2026-09-12 grooming applied that rule to every quest here and merged main into dev. The auth API line is here for its request-side break (`mtls=` and the now-required fields) and ranks first because moq.pro adopts the release only once that contract is settled; diff --git a/quest/m2/2147-moq-video-10-bit-hevc-and-av1-support-in-the-nvidia-codec.md b/quest/m2/2147-moq-video-10-bit-hevc-and-av1-support-in-the-nvidia-codec.md index 34de6900b7..1749c006d3 100644 --- a/quest/m2/2147-moq-video-10-bit-hevc-and-av1-support-in-the-nvidia-codec.md +++ b/quest/m2/2147-moq-video-10-bit-hevc-and-av1-support-in-the-nvidia-codec.md @@ -1,38 +1,42 @@ -# [M] moq-video: 10-bit HEVC and AV1 support in the NVIDIA codec path +# [L] Complete NVIDIA Main10 and AV1 encoding support ## Goal -Implement and verify the behavior tracked in [#2147](https://github.com/moq-dev/moq/issues/2147) -within the issue's stated scope and boundaries. +Implement and verify the remaining 10-bit HEVC and AV1 encoding work tracked +in [#2147](https://github.com/moq-dev/moq/issues/2147). NVDEC AV1 decoding and +catalog AV1 types already exist; do not reimplement them. ## Plan -Rescoped during the 2026-08 grooming: AV1 8-bit decode landed on dev (#2178). -Remaining: 10-bit Main10 decode and encode (P016 surfaces, profile plumbing), -and AV1 encode/transcode once Ada+ hardware is available. +Extend the settled frame and NVENC contracts with Main10 surfaces, profile +selection, and accurate codec metadata. Audit byte pitch, plane layout, CPU +download, and P016 input/output together; a codec enum alone does not establish +10-bit support. Preserve color metadata and refuse unsupported conversions. +Do not imply that OpenH264 can decode HEVC or tonemap HDR. -### Issue context +Add AV1 encoding only where the queried NVIDIA device/driver supports it. +Preserve OBU framing and accurate catalog configuration through transcode. +Use existing extensible codec enums; no replacement of the 0.1 core API is +planned. Keep the NVIDIA backend optional and loaded at runtime. -Follow-up to #2145 (NVDEC hardware decode + zero-copy NVDEC -> NVENC transcode). The GPU pipeline currently supports 8-bit 4:2:0 H.264/H.265 only. Two extensions worth doing, probably as separate PRs: +Split Main10 and AV1 implementation into independent PRs if hardware or review +scope warrants it. Validate decoded pixels, bit depth, profile, framing, +resource lifetime, drain, and refusal on unsupported devices. Wire fixtures +and contract tests into CI, and record actual hardware execution separately. +Lack of suitable hardware leaves that implementation unverified, not complete. -#### 10-bit HEVC (Main10) +Public API: additive capabilities on the m0 extension points. Wire: existing +codec signaling, with cross-language fixtures for any metadata change. -- The NVDEC backend rejects `bit_depth_luma_minus8 != 0` today (clean error in the sequence callback). Supporting it means decoding to P016 output surfaces and threading a pixel-format dimension through `frame::cuda::Frame` (pitch is in bytes, but plane layout and the CPU download path assume 8-bit NV12). -- NVENC needs the matching Main10 profile + `NV_ENC_BUFFER_FORMAT_YUV420_10BIT` input, and the catalog codec string must advertise the right profile/level. -- The CPU fallback story needs deciding: openh264 is 8-bit only, so a 10-bit source either has no software fallback (like H.265 already) or gets tonemapped down to 8-bit. +## Required -#### AV1 - -- Decode: NVDEC supports AV1 on Ampere+ (`cudaVideoCodec_AV1` is already in the vendored bindings). Needs a `Codec::Av1` in the decode backend seam, the catalog/container plumbing for AV1 tracks, and AV1 has no Annex-B: the parser takes OBUs directly, so the access-unit prep differs from H.264/H.265. -- Encode: NVENC AV1 exists only on Ada+ (the RTX 3070 Ti dev box can decode AV1 but not encode it), so the first useful shape is AV1 *source* -> H.264/H.265 rungs in `moq-transcode`, not AV1 output. -- The `hang` catalog and `moq-mux` need an AV1 codec entry (`av01.*` codec string, OBU framing) if they don't have one by then; that part rows through the js side per the cross-package sync table. - -Both are additive to the decode/encode `Codec` enums (`#[non_exhaustive]` already), so no breaking changes expected. - -AV1 encode belongs here too: it was waiting on a hardware backend, and the -NVIDIA codec path is that backend. No software AV1 encode, since rav1e is too -slow for real time. +- [Video frames](/quest/m0/video-frames.md) - extensible pixels and frame metadata +- [NVENC resources](/quest/m0/nvenc-resources.md) - safe input and completion ownership ## Closes - [#2147](https://github.com/moq-dev/moq/issues/2147) - close this issue when the quest finishes + +## Related + +- [Codec coverage study](/quest/m3/video-codec-coverage.md) - measure optional software and other native backends separately diff --git a/quest/m2/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md b/quest/m2/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md index 3d88fb5d33..7d40cc1639 100644 --- a/quest/m2/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md +++ b/quest/m2/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md @@ -21,17 +21,12 @@ grant; `Options::bandwidth` documents that (`rs/moq-audio/src/encode/producer.rs:49-62`). Video already follows through `rate::Control` (`rs/moq-video/src/encode/producer.rs:467-476`). -- Move `rs/moq-video/src/encode/rate.rs` (`Policy`, `Control`) to - `moq_mux::rate::Control`. moq-mux is a dependency of both crates - (`rs/moq-audio/Cargo.toml:82`, `rs/moq-video/Cargo.toml:110`); its `pace.rs` - is wall-clock delivery of export frames, a different concern. moq-video is - 0.0.23, so deleting `moq_video::encode::rate` ships on main. Update - moq-video's import (`producer.rs:32`) and the `Options::bandwidth` doc - (`producer.rs:237`). +- Consume `moq_mux::rate` after the m0 namespace move. Keep one shared policy; + this quest adds audio adaptation without removing a stabilized video API. - The follow loop lives in `moq_audio::encode::Producer`, not the capture driver: `Producer::new` (`producer.rs:255`) already takes `Options` with the allocator (`:62`), so it reserves the configured bitrate against - `self.track().demand()`, holds a `Reservation::consumer()` plus a + `self.demand()`, holds a `Reservation::consumer()` plus a `rate::Control`, and feeds each grant to `Encoder::set_bitrate` (`rs/moq-audio/src/encode/encoder.rs:411`). Capture and moq-ffi (`rs/moq-ffi/src/audio.rs:254`) build the Producer, so both adapt without @@ -55,17 +50,21 @@ grant; `Options::bandwidth` documents that reservation before video sees a bit and audio is squeezed only once the link cannot carry audio alone. Worth doing for that tail, not worth blocking on. -Tests: the `Control` unit tests move with the module; an Opus Producer whose +Tests: retain the shared `Control` tests; an Opus Producer whose grant drops below its configured bitrate reports the lower `bitrate()` after one policy step, holds it on a `None` grant, and ramps back when the grant returns; a grant below the Opus floor clamps at the floor; a PCM Producer ignores every grant. -This targets 0.0.x crates and needs the allocator on main, so it starts after +This needs the allocator on main, so it starts after the dev merge. ## Required +- [Shared rate policy](/quest/m0/media-rate-policy.md) - namespace relocation is already complete +- [Audio configuration](/quest/m0/audio-config.md) - the settled codec settings +- [Audio publication](/quest/m0/audio-publication.md) - demand access without transport write authority + - [Merge dev](/quest/m1/merge-dev.md) - builds on dev-only code that reaches `main` with the merge ## Closes diff --git a/quest/m2/README.md b/quest/m2/README.md index b558eb30c8..92eb0b3dfc 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -21,8 +21,10 @@ can act on. Each still carries its own plan and regression test. Work that builds on dev-only code but breaks nothing and gates nothing (the io_uring stream sessions, the perf line, the QUIC worker quests) -also sits here and starts on `main` after the dev merge, as does anything -targeting a `0.0.x` crate. The token SDK default switch explicitly targets a +also sits here and starts on `main` after the dev merge. The four media crates' +pre-0.1 contracts are the explicit exception in [m0](/quest/m0/README.md); +their API-preserving implementation and performance follow-ups remain here. +Other `0.0.x` work stays here. The token SDK default switch explicitly targets a subsequent breaking dev cycle after v1 readers exist; it is not additive M2 work for main. @@ -96,6 +98,10 @@ before format-specific metadata. Unrelated areas can proceed in parallel. - [C# through moq-ffi](/quest/m2/cs/README.md) - generated C# over moq-ffi as a NuGet package with native runtimes - [OBS native codecs](/quest/m2/obs-moq-video/README.md) - remove FFmpeg decoding dependencies, deliver GPU frames, and use native audio/video encoders - [Audio codecs](/quest/m2/audio-codecs/README.md) - platform audio codecs, explicit unsupported cases, and channel layouts up to 7.1 +- [Opus descriptions](/quest/m2/audio-opus-input.md) - validate headers and honor codec clock, pre-skip, and gain +- [Capture formats](/quest/m2/audio-capture-format.md) - unsupported overrides refuse before device open and channel counts cannot wrap +- [NVENC recovery](/quest/m2/nvenc-recovery.md) - partial initialization and rejected rate changes preserve valid state +- [Transcode source](/quest/m2/transcode-source.md) - select a rendition the chosen backend can actually decode - [Keyframe trigger](/quest/m2/keyframe-trigger.md) - an application can ask the built-in capture encoder for a keyframe - [QoS](/quest/m2/qos/README.md) - broadcast health: relay starvation and timeliness histograms, and client stats broadcasts from publishers and viewers - [Drain](/quest/m2/drain/README.md) - relay restarts drain sessions over GOAWAY instead of hard-dropping them @@ -116,6 +122,10 @@ before format-specific metadata. Unrelated areas can proceed in parallel. - [Audio quality harness](/quest/m2/audio-quality-harness/README.md) - a playout latency regression fails a run instead of arriving as a bug report - [Latency ledger](/quest/m2/latency-ledger.md) - a session reports where its end-to-end audio delay went, stage by stage - [Benchmark comparisons](/quest/m2/performance-comparisons.md) - retained evidence, repeated paired runs, and uncertainty for performance claims +- [Audio buffers](/quest/m2/audio-buffers.md) - measure and reduce packetization movement and decoding allocations +- [NVENC reuse](/quest/m2/nvenc-reuse.md) - reuse completed codec resources without weakening ownership +- [Transcode resources](/quest/m2/transcode-resources.md) - measure aggregate threads, codec sessions, retained frames, and probe costs +- [Renderer resources](/quest/m2/video-render-resources.md) - bound surface retention and validate output resources - [#3126](/quest/m2/3126-moq-bench-every-readme-example-fails-to-parse-and.md) - moq-bench reports per-interval latency percentiles so the ramp leaves the steady state - [Relay profiling](/quest/m2/performance-profiles.md) - reproducible CPU and allocation captures under the existing workloads - [Browser benchmarks](/quest/m2/browser-benchmarks.md) - measure JS transport, container, decode, and render costs in an identified browser @@ -143,7 +153,7 @@ before format-specific metadata. Unrelated areas can proceed in parallel. - [#3056](/quest/m2/3056-watch-video-decoder-captures-the-rewind-generation-at.md) - watch: the video decoder resets on a declared discontinuity - [#933](/quest/m2/933-video-rotation-metadata-not-propagated-from-mobile-camera.md) - the catalog rotation follows the live camera's orientation - [#2075](/quest/m2/2075-mirror-catalog-reservation-gating-in-moq-hang-js-hang.md) - @moq/publish gates the first catalog snapshot until every reserved track is described -- [#2848](/quest/m2/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md) - the Opus producer follows its bandwidth grant through `moq_mux::rate::Control`, moved out of moq-video +- [#2848](/quest/m2/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md) - the Opus producer follows its bandwidth grant through the settled `moq_mux::rate::Control` - [Ladder](/quest/m2/ladder/README.md) - a transcode ladder adapts to the uplink it publishes over, instead of encoding every live rung at its ceiling - [LOC duration marker](/quest/m2/loc-duration-marker.md) - LOC producers write the marker once released consumers skip it - [#2278](/quest/m2/2278-watch-absolute-wall-clock-latency-target-for-synchronized.md) - hang: expose the fixed catalog-root broadcast clock without synchronizing library playback to wall time @@ -199,7 +209,7 @@ before format-specific metadata. Unrelated areas can proceed in parallel. - [Ship capture and playback](/quest/m2/cli-packaging.md) - a released moq binary can capture and play, which no distribution currently enables - [Windows capture parity](/quest/m2/capture-windows.md) - system audio and screen cursor capture with a settled app-capture policy - [Linux capture parity](/quest/m2/capture-linux.md) - Wayland window/system-audio capture with explicit display-selection and app-capture limits -- [Plan capture ergonomics](/quest/m2/capture-ergonomics.md) - scope independent crop, audio mixing, and format-validation quests +- [Plan capture ergonomics](/quest/m2/capture-ergonomics.md) - scope independent crop and audio mixing quests - [X11 capture transport](/quest/m2/x11-capture-shm.md) - move X11 capture to shared memory and RandR events instead of a per-frame socket copy - [io_uring flow control](/quest/m2/uring-flow-control-windows.md) - the relay's io_uring workers honor the QUIC flow-control windows instead of refusing them - [Capture frame buffers](/quest/m2/capture-frame-buffers.md) - stop rebuilding a full-frame buffer every tick in the X11 and Windows backends diff --git a/quest/m2/audio-buffers.md b/quest/m2/audio-buffers.md new file mode 100644 index 0000000000..044c558844 --- /dev/null +++ b/quest/m2/audio-buffers.md @@ -0,0 +1,31 @@ +# [M] Measure and reduce audio packetization copies + +## Goal + +Audio packetization and decoding avoid unnecessary per-packet allocation and +front-of-buffer movement without changing the settled frame or codec APIs. + +## Plan + +Producer::publish_full_frames drains the front of a Vec into another Vec for +each packet; Resampler also drains front chunks. Large PCM submissions repeat +that movement. Opus decode allocates space for a maximum-duration packet, then +truncates and converts it into another output buffer. + +Measure allocations, copied samples, CPU, and latency for ordinary 20 ms input, +large caller batches, resampling, and multichannel PCM. Prefer cursors with one +compaction and reusable scratch storage where measurements justify them. Do not +turn borrowed scratch into output whose lifetime ends on the next codec call. + +Use the existing audio quality/benchmark infrastructure. Preserve timestamps, +priming, DTX, partial packets, final padding, and discontinuities in CI regression +fixtures. Report measured changes without hard-coding machine-specific timing +thresholds into unit tests. Public API and wire: unchanged. + +## Required + +- [Audio configuration](/quest/m0/audio-config.md) - optimize the settled PCM/codec boundary + +## Related + +- [Audio quality](/quest/m2/audio-quality-harness/README.md) - end-to-end quality and latency evidence diff --git a/quest/m2/audio-capture-format.md b/quest/m2/audio-capture-format.md new file mode 100644 index 0000000000..3d392f05ef --- /dev/null +++ b/quest/m2/audio-capture-format.md @@ -0,0 +1,24 @@ +# [S] Validate audio capture overrides before opening devices + +## Goal + +Unsupported sample-rate/channel overrides fail with the requested format and +device context, and channel counts never wrap while narrowing to the device API. + +## Plan + +capture::resolve currently overwrites the device default and casts u32 channels +to u16. A value such as 65537 becomes one. Supported ranges are already available +to device enumeration. Validate requested overrides and checked conversions +before opening the stream; make the existing documentation's hint/requirement +wording match the actual contract. Do not silently substitute another format. + +Test unsupported rates, 65537 channels, device ranges, valid default selection, +and backend open failures using fake capabilities in CI. No new capture API, +mixing, crop support, or device backend is required. + +Public API and wire: unchanged; previously malformed requests are refused. + +## Related + +- [Capture ergonomics](/quest/m2/capture-ergonomics.md) - crop and mixing planning remain independent diff --git a/quest/m2/audio-codecs/README.md b/quest/m2/audio-codecs/README.md index eec74c4652..9952ab879e 100644 --- a/quest/m2/audio-codecs/README.md +++ b/quest/m2/audio-codecs/README.md @@ -22,19 +22,22 @@ codec its framework opens, so AC-3, E-AC-3, MP3, and FLAC ride along on the hosts that have them; each still needs a fixture before the backend advertises it. -Channels are a `Layout`, not a count: a closed set of well-known layouts in one +Use the extensible `Layout` contract settled in m0, with well-known layouts in one canonical order, derived from the codec description (AAC channelConfiguration, OpusHead mapping) so the catalog and wire do not change. Every decoder reorders from its codec's native order into that one, so the mixer, the FFI, and OBS never guess where the LFE is. Playback downmixes to whatever the output device -opened. Counts with no standard layout are refused. +opened. Preserve the existing arbitrary-channel PCM passthrough through an +unspecified discrete layout; refuse spatial remixing when speaker positions +are unknown instead of guessing them. Encode mirrors decode: an `encode::backend` seam, `Codec::Aac` meaning AAC-LC at the input's layout, and platform encoders behind it. Opus encode stays mono/stereo. -The layout and seam quests are independent and come first; each platform then -lands as its own decode and encode quest so verification stays per host. The +The core configuration and layout contracts land in m0. These quests implement +surround and backend dispatch on that contract; each platform then lands as +its own decode and encode quest so verification stays per host. The HE-AAC refusal and the PCE parse are defects in what ships today and are ready now. @@ -42,7 +45,7 @@ ready now. - [HE-AAC refusal](/quest/m2/audio-codecs/he-aac-refusal.md) - implicit-SBR HE-AAC over TS is refused instead of half-decoded as the LC core - [AAC PCE](/quest/m2/audio-codecs/aac-pce.md) - a channel_config of 0 parses the program config element instead of guessing stereo -- [Layout](/quest/m2/audio-codecs/layout.md) - a `Layout` type in one canonical order carries up to 7.1 through decode, resample, playback, and the FFI +- [Layout](/quest/m2/audio-codecs/layout.md) - the settled `Layout` carries up to 7.1 through decode, resample, playback, and the FFI - [Decode seam](/quest/m2/audio-codecs/decode-backend.md) - `decode::backend` selects a platform decoder before symphonia, mirroring moq-video - [AudioToolbox decode](/quest/m2/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [Opus surround](/quest/m2/audio-codecs/opus-surround.md) - mapping family 1 decodes on every host through the multistream decoder diff --git a/quest/m2/audio-codecs/decode-backend.md b/quest/m2/audio-codecs/decode-backend.md index bd2abf09c3..e7d6158889 100644 --- a/quest/m2/audio-codecs/decode-backend.md +++ b/quest/m2/audio-codecs/decode-backend.md @@ -12,8 +12,9 @@ documents which codecs each host decodes. Mirror `rs/moq-video/src/decode/backend` in name and shape: a crate-private `Backend` trait (`decode`, `flush`, `name`), an `open(codec, config)` that walks the platform candidates before the software ones and refuses when none -takes the track, and `decode::Kind { Auto, Platform, Software, Named }` on -`decode::Config`. `Decoder::name()` reports what was opened, which the OBS +takes the track. Use the backend-selection configuration and Decoder +constructor settled in m0 rather than adding a conflicting public shape. +`Decoder::name()` reports what was opened, which the OBS stats and `moq play` surface. - The seam is generic over `hang::catalog::AudioCodec`, so a backend advertises @@ -27,8 +28,9 @@ stats and `moq play` surface. catalog said (HE-AAC doubles the rate); `Consumer` already resamples and remixes to the requested output, so that stays the seam's contract. - The `aac` feature keeps gating symphonia. Platform backends are - `cfg(target_os)` like their video counterparts, with `mediacodec` behind the - existing feature. + `cfg(target_os)` like their video counterparts. Audio has no MediaCodec + feature yet; its Android quest introduces the optional dependency following + the settled media build policy. - Docs: `doc/lib/rs/moq-audio.md` gains the backend table `moq-video.md` has, and states the Linux gap. `doc/bin/cli.md` and `doc/bin/obs.md` follow. - Regression: the selection order and `Named` refusal, tested with a stub @@ -36,6 +38,10 @@ stats and `moq play` surface. The FFI does not expose `Kind` until a consumer asks. +## Required + +- [Audio configuration](/quest/m0/audio-config.md) - stable decoder settings and selection entry points + ## Related - [Layout](/quest/m2/audio-codecs/layout.md) - independent; the platform backends need both diff --git a/quest/m2/audio-codecs/decode-mediacodec.md b/quest/m2/audio-codecs/decode-mediacodec.md index 774ad2c46a..c1b342671b 100644 --- a/quest/m2/audio-codecs/decode-mediacodec.md +++ b/quest/m2/audio-codecs/decode-mediacodec.md @@ -9,7 +9,7 @@ device's codec list opens. ## Plan The audio counterpart of `rs/moq-video/src/decode/backend/mediacodec.rs`, -behind the existing `mediacodec` feature and the decode seam, on `target_os +behind a new optional audio `mediacodec` feature and the decode seam, on `target_os = "android"`. - `audio/mp4a-latm` with the catalog description as `csd-0`; the output format @@ -18,7 +18,8 @@ behind the existing `mediacodec` feature and the decode seam, on `target_os - Optional codecs are probed through `AMediaCodecList` at open and advertised only where present. - Fixtures and layout-order tests as in the AudioToolbox quest; runtime proof - on a device or emulator, since no CI runs Android. + on a device or emulator, recorded separately from the existing Android + compile lane. Add the audio feature to that lane. - The binding ships in the moq-ffi Android slice, which is how Kotlin and Dart reach it. diff --git a/quest/m2/audio-codecs/encode-backend.md b/quest/m2/audio-codecs/encode-backend.md index fdd89d325d..d7b67f268f 100644 --- a/quest/m2/audio-codecs/encode-backend.md +++ b/quest/m2/audio-codecs/encode-backend.md @@ -11,10 +11,9 @@ host with no AAC encoder refuses it at construction. Mirror the decode seam: `encode::backend` with a crate-private `Backend` trait (`encode`, `flush`, `set_bitrate`, `name`), an `open(codec, config)` -that walks platform candidates before software ones, and `encode::Kind` on -`encode::Config`. Opus and PCM move behind the trait unchanged and remain the -only software backends: no Rust AAC encoder exists, which is why the platform -quests follow. +that walks platform candidates before software ones, using the public settings +and selection contract settled in m0. Opus and PCM retain their behavior. This +quest adds AAC through platform encoders; no software AAC dependency is selected. - `encode::Codec` gains `Aac`, meaning `mp4a.40.2`, and `as_str` / `FromStr` accept `"aac"`, which is what libmoq's codec string carries. moq-ffi's @@ -38,6 +37,7 @@ quests follow. ## Required +- [Audio configuration](/quest/m0/audio-config.md) - stable PCM/codec settings and selection entry points - [Merge dev](/quest/m1/merge-dev.md) - the required M1 APIs must be available on main before this implementation starts - [Decode seam](/quest/m2/audio-codecs/decode-backend.md) - the naming and shape this mirrors diff --git a/quest/m2/audio-codecs/layout.md b/quest/m2/audio-codecs/layout.md index 94c17350d2..db1f5bcc11 100644 --- a/quest/m2/audio-codecs/layout.md +++ b/quest/m2/audio-codecs/layout.md @@ -9,16 +9,18 @@ Proven with multichannel PCM, the one codec that needs no new decoder. ## Plan -A closed `Layout` enum (mono, stereo, 2.1, quad, 5.0, 5.1, 6.1, 7.1, and the -other AAC channelConfiguration and Opus mapping family 1 entries) in one -canonical order, the SMPTE/WAVE order. Each codec module maps its native +Extend the Layout contract settled in m0 with supported surround layouts +(2.1, quad, 5.0, 5.1, 6.1, 7.1, and the other AAC channelConfiguration and +Opus mapping family 1 entries) in one canonical order, the SMPTE/WAVE order. +Keep the representation extensible. Each codec module maps its native order into it: AAC's `C L R Ls Rs LFE` and Opus's Vorbis `L C R Ls Rs LFE` -both become `L R C LFE Ls Rs`. A count with no standard layout is refused at -construction, so a `Layout` always downmixes. +both become `L R C LFE Ls Rs`. An unspecified discrete PCM layout remains +valid for passthrough but is refused for spatial remixing; never invent speaker +positions from an arbitrary channel count. -- `decode::Config` and the encoder input take a `Layout` where they take a - channel count today; `Frame` stays layout-free since the consumer fixed it - at construction. `Layout::channels()` gives the count. +- Reuse the settled PCM descriptors and codec/consumer settings. This quest + adds supported layout behavior rather than replacing public field types. + Frame stays layout-free since the consumer fixed it at construction. - `resample::remix` becomes a generic remix over layouts: ITU-R BS.775 coefficients for downmix, silence in the extra speakers for upmix, and the existing mono/stereo paths as the two-channel special cases. The resampler is @@ -43,6 +45,10 @@ construction, so a `Layout` always downmixes. Capture stays mono/stereo, and Opus encode stays mapping family 0. +## Required + +- [Audio configuration](/quest/m0/audio-config.md) - extensible layout and current-mode compatibility + ## Related - [Opus surround](/quest/m2/audio-codecs/opus-surround.md) - the first coded multichannel consumer of the layout diff --git a/quest/m2/audio-opus-input.md b/quest/m2/audio-opus-input.md new file mode 100644 index 0000000000..7b3110d633 --- /dev/null +++ b/quest/m2/audio-opus-input.md @@ -0,0 +1,28 @@ +# [M] Honor and validate Opus stream descriptions + +## Goal + +A valid Opus stream decodes using the codec's clock and declared pre-skip/gain; +an explicitly malformed or unsupported description is refused instead of +silently falling back to defaults. + +## Plan + +moq-audio's new_opus ignores description parse failures and uses the OpusHead +original-input-rate field to open a decoder. That metadata can be 44100 or +unknown even though it is not a supported decoder operating rate. Audit the +shared moq-mux Opus description parser as the source of truth, including +version, output gain, channel mapping, and truncation. + +Distinguish an absent optional description from a present invalid one. Preserve +supported mono/stereo and PCM behavior; do not implement surround here or guess +a mapping. Cover input-rate metadata independent of decode rate, unknown rate, +pre-skip, nonzero gain, malformed headers, and unsupported mapping/version with +real Opus fixtures in CI. Verify timestamps as well as decoded sample counts. + +Public API and wire schema: unchanged. Update the existing codec documentation +with supported/refused cases; header-parser changes stay focused on this path. + +## Related + +- [Opus surround](/quest/m2/audio-codecs/opus-surround.md) - adds supported mappings separately diff --git a/quest/m2/capture-ergonomics.md b/quest/m2/capture-ergonomics.md index 263d0543e2..3acd7aaa37 100644 --- a/quest/m2/capture-ergonomics.md +++ b/quest/m2/capture-ergonomics.md @@ -2,16 +2,17 @@ ## Goal -Three papercuts in the capture surface, none blocking but each visible the +Two papercuts in the capture surface, none blocking but each visible the first time someone hits it. This is a planning dispatch. Verify each current limitation against source, -then replace this quest with independently completable crop, audio mixing, -and format-validation quests. Each needs a chosen public API, ownership +then replace this quest with independently completable crop and audio mixing +quests. Each needs a chosen public API, ownership boundary, supported/refused cases, and CI acceptance tests. Ask the maintainer about unsettled crop coordinates and audio clock/mixing policy before coding. -Identify any published API break for M1; do not hold independent validation -work behind crop or mixing design. +Keep additions compatible with the settled m0 capture contracts. Identify any +published API break for a separate maintainer decision; independent format +validation already has its own quest. ## Plan @@ -22,8 +23,11 @@ work behind crop or mixing design. - **Mixing multiple audio devices.** One device, one track. A screen share wanting microphone plus system audio has no way to say so, which is exactly the combination the `System` source makes newly reachable. -- **Format overrides are unvalidated.** A requested sample rate is applied - without checking the device's supported ranges, so a bad combination fails - inside cpal's `build_input_stream` instead of erroring with something that - names the problem. The supported ranges are already enumerable, which is - what `moq devices` reads. + Preserve m0's exclusive AEC microphone ownership and define clock alignment + before sharing processed microphone input. + +## Related + +- [Capture format validation](/quest/m2/audio-capture-format.md) - independent validation work already scoped +- [AEC ownership](/quest/m0/audio-aec.md) - one microphone per adaptive canceller +- [Video timing](/quest/m0/video-timing.md) - the capture clock contract diff --git a/quest/m2/color-model.md b/quest/m2/color-model.md index bf865088aa..9ad3e9321c 100644 --- a/quest/m2/color-model.md +++ b/quest/m2/color-model.md @@ -9,6 +9,12 @@ instead of guessing or reparsing the bitstream. ## Plan +Use the settled m0 frame contract and the existing extensible Color metadata; +do not introduce a second native-frame hierarchy. Fix encoding paths that +warn about a known color mismatch and then label unchanged pixels as the +requested color: convert correctly or refuse. Preserve color information +through CUDA, D3D, and Android paths and test the signaled VUI against pixels. + `rs/hang/src/catalog/video/mod.rs` has carried a bare `// TODO color space` since the config was written. Two codecs already expose colour per their own syntax: `VP9` carries primaries, transfer characteristics, matrix coefficients, and @@ -47,6 +53,10 @@ conflict between VUI and container resolving to the bitstream, a container box that fills a gap the bitstream left unspecified, and an SDR round trip that stays byte-identical. +## Required + +- [Video frames](/quest/m0/video-frames.md) - the extensible frame contract + ## Related - [SEI sidecars](/quest/m3/sei/README.md) - moves SEI out of the video track; diff --git a/quest/m2/decoded-frames.md b/quest/m2/decoded-frames.md index 18bb2f311b..165d4ac1ec 100644 --- a/quest/m2/decoded-frames.md +++ b/quest/m2/decoded-frames.md @@ -38,11 +38,13 @@ adapters own their hardware import proof. Update `moq.h`, affected wrappers, and C/binding documentation; run `just test smoke-full` in CI. Public API: owned frame access and conversion at the binding boundary. Wire: -none. Keep the existing moq-video core API unless a concrete consumer needs a -change. +none. Consume the settled m0 frame/output contracts without replacing them. ## Required +- [Video frames](/quest/m0/video-frames.md) - typed pixels and extensible owned frames +- [Video output](/quest/m0/video-output.md) - explicit native or CPU output + - [Merge dev](/quest/m1/merge-dev.md) - the required M1 APIs must be available on main before this implementation starts ## Related diff --git a/quest/m2/intra-refresh/README.md b/quest/m2/intra-refresh/README.md index 3c139f0511..b02bc8e390 100644 --- a/quest/m2/intra-refresh/README.md +++ b/quest/m2/intra-refresh/README.md @@ -29,7 +29,7 @@ Decisions the quests share: mid-stream skip both decode everything and present nothing until recovery, freezing on the last good frame. A group that opens on a true IDR shows at once. -- The shared encode config replaces the keyframe interval with a `Gop` enum, +- The shared encode config extends the `Gop` contract settled in m0, and a cut in refresh mode starts a new sweep, never an IDR. - H.264 and H.265 only. AV1 and VP9 have no standard gradual refresh signal. WebCodecs has no intra-refresh option, so js/publish is consumer-only here. @@ -42,7 +42,7 @@ Decisions the quests share: - [Consumer warmup](/quest/m2/intra-refresh/consumer-warmup.md) - JS and Rust viewers join `warmup` earlier and withhold display until recovery, except at a true IDR - [H.264 import](/quest/m2/intra-refresh/h264-import.md) - the splitter keeps `recovery_frame_cnt` and import publishes `warmup` from it - [H.265 import](/quest/m2/intra-refresh/h265-import.md) - the splitter reads the recovery-point SEI so an HEVC intra-refresh stream forms groups and publishes `warmup` -- [Encode config](/quest/m2/intra-refresh/encode-config.md) - `gop: Gop` replaces the keyframe interval, `cut()` replaces `keyframe()`, the producer cuts groups per sweep and publishes `warmup` +- [Encode config](/quest/m2/intra-refresh/encode-config.md) - refresh mode extends the settled GOP contract; the producer cuts groups per sweep and publishes `warmup` - [NVENC refresh](/quest/m2/intra-refresh/nvenc-refresh.md) - the NVENC backend encodes refresh mode for H.264 and HEVC - [V4L2 refresh](/quest/m2/intra-refresh/v4l2-refresh.md) - the V4L2 backend encodes refresh mode - [Bindings](/quest/m2/intra-refresh/bindings.md) - ffi, libmoq, and every wrapper expose the `Gop` enum, on `dev` diff --git a/quest/m2/intra-refresh/encode-config.md b/quest/m2/intra-refresh/encode-config.md index ed06f4ce2e..4d7453cc1c 100644 --- a/quest/m2/intra-refresh/encode-config.md +++ b/quest/m2/intra-refresh/encode-config.md @@ -1,4 +1,4 @@ -# [M] Encode config: a Gop enum, cut(), and refresh-mode groups +# [M] Implement refresh-mode groups on the settled GOP contract ## Goal @@ -6,23 +6,19 @@ caller picks keyframes at an interval or intra refresh with a cycle length and cannot ask for both. A forced cut starts a new group in either mode: an IDR with keyframes, a fresh sweep with refresh. In refresh mode the producer opens -a group at every sweep start and publishes `warmup` equal to the cycle. Every +a group at every sweep start and publishes `warmup` as the actual sweep +duration, which can be shorter than the cycle. Every backend that cannot encode refresh mode refuses it when configured, and the -CLI and transcoder expose the choice. moq-video and moq-transcode are `0.0.x`, -so this lands on `main`. +CLI and transcoder expose the choice. This extends the settled m0 contract +without replacing an API after 0.1. ## Plan -- `rs/moq-video/src/encode/encoder.rs`: replace `gop: u32` with `gop: Gop`, - `enum Gop { Keyframe(u32), Refresh(u32) }`, both in frames as today, default - `Keyframe(framerate * 2)`. Keeping the `gop` name and frame unit means every - backend's wiring changes by one match. If `Refresh` reads wrong beside - "keyframe", `Sweep` is the alternative; keep the field name. -- Rename `Encoder::keyframe()` to `cut()` to match the ffi and producer - vocabulary; in `Keyframe` mode it forces an IDR as now, in `Refresh` mode it - asks the backend to restart the sweep. The producer's forced cut on every - (re)open (`rs/moq-video/src/encode/producer.rs`) goes through the same - path. +- Extend the non-exhaustive `Gop` contract from m0 with refresh mode. Keep + the settled frame-count units and `cut()` operation; do not replace the + public config or rename the operation again. A cut in refresh mode asks + the backend to restart the sweep, including the producer's forced cut on + every reopen. - `Backend::encode(frame, keyframe)` in `rs/moq-video/src/encode/backend/mod.rs` becomes `encode(frame, cut)`, and each backend gets the mode at construction. VideoToolbox, openh264, VAAPI, Media Foundation, and MediaCodec return an @@ -46,4 +42,6 @@ so this lands on `main`. ## Required +- [Video GOP](/quest/m0/video-gop.md) - the extensible group contract and cut operation + - [Catalog warmup](/quest/m2/intra-refresh/catalog-warmup.md) - the field the producer publishes diff --git a/quest/m2/intra-refresh/v4l2-refresh.md b/quest/m2/intra-refresh/v4l2-refresh.md index c9385afe4f..33f26a6d8f 100644 --- a/quest/m2/intra-refresh/v4l2-refresh.md +++ b/quest/m2/intra-refresh/v4l2-refresh.md @@ -5,10 +5,10 @@ With `Gop::Refresh`, the V4L2 backend asks the hardware for periodic intra refresh and no IDR after the first frame; a driver that lacks the control refuses the mode. Groups come from the producer's frame count, so a driver -that emits no recovery-point SEI still forms one group per sweep. V4L2 has no -control that restarts a sweep, so a mid-cycle `cut()` is refused on this -backend rather than deferred; the only cut it honours is the implicit one on a -fresh encoder, whose first frame starts the first sweep. +that emits no recovery-point SEI still forms one group per sweep. Periodic +refresh controls do not by themselves guarantee an immediate sweep restart. +Verify the selected driver; refuse an unsupported mid-cycle cut rather than +claiming a boundary that was not encoded. ## Plan @@ -19,9 +19,10 @@ fresh encoder, whose first frame starts the first sweep. `GOP_SIZE` to zero for an unbounded GOP, and refuse the mode when the driver rejects both controls, unlike the self-disabling `keyframes` fallback used for forced keyframes. -- A cut in refresh mode returns an error from this backend: the shared - contract says a cut restarts the sweep, and V4L2 cannot, so it refuses - rather than opening a group out of phase with the refreshed macroblocks. +- A cut in refresh mode restarts the sweep. If the driver cannot honor it, + report rejection through the error path settled in m0; this does not + independently require a Result from cut(). Never open a group out of phase + with the refreshed macroblocks. The producer's forced cut on (re)open does not reach the backend, because a new encoder's first frame is a sweep start by construction; the producer treats it as the cut. The count stays aligned to the driver's period from diff --git a/quest/m2/ladder/README.md b/quest/m2/ladder/README.md index d2b7afdcb0..344a5be242 100644 --- a/quest/m2/ladder/README.md +++ b/quest/m2/ladder/README.md @@ -56,9 +56,8 @@ is successfully applied. Catalog state follows the last target the encoder failure retains the last applied target rather than lying. Start with the shared rate policy's five percent hysteresis, immediate -decreases, and gradual upward ramp (`rate::Control`, moving to -`moq_mux::rate` with -[#2848](/quest/m2/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md)). +decreases, and gradual upward ramp (`moq_mux::rate::Control`, moved before 0.1 +by [Shared rate policy](/quest/m0/media-rate-policy.md)). No second re-entry threshold and no dwell timer until measurements show the catalog state flaps. diff --git a/quest/m2/mobile/ffi-video-consumer.md b/quest/m2/mobile/ffi-video-consumer.md index fb242c4932..3cba131a1b 100644 --- a/quest/m2/mobile/ffi-video-consumer.md +++ b/quest/m2/mobile/ffi-video-consumer.md @@ -39,5 +39,4 @@ This quest must not require Dart artifact enablement, which would create a cycle ## Related -- [Decoder drain](/quest/m3/decode-drain.md) - flush and finish for pipelined decoders this will lean on - [Dart codec parity](/quest/m2/dart-codecs.md) - the one binding that cannot decode until it lands diff --git a/quest/m2/nvenc-recovery.md b/quest/m2/nvenc-recovery.md new file mode 100644 index 0000000000..1cf4271c32 --- /dev/null +++ b/quest/m2/nvenc-recovery.md @@ -0,0 +1,28 @@ +# [M] Recover NVENC initialization and rate-change failures + +## Goal + +NVENC releases every partially initialized resource, destruction does not +panic, and a rejected rate change leaves the last accepted settings intact. + +## Plan + +Registration can succeed before mapping fails in safe/buffer.rs, and several +destructors call expect on driver cleanup. Session::reconfigure mutates retained +bitrate/VBV fields before the driver accepts the change; a rejected zero-rate +update can corrupt the basis of the next proportional update. + +Use the settled resource ownership model and a fake driver to verify rollback +after each acquisition stage, cleanup order, and teardown during an existing +failure. Compute candidate rate settings with checked arithmetic and commit +only on driver success. A following successful update must be based on the +last accepted settings. Preserve fallible explicit operations and non-panicking +Drop without disguising a live operational failure as success. + +Run the failure-injection suite in CI. Do not duplicate cleanup guarantees +already delivered by the ownership quest. Public API and wire: unchanged. + +## Required + +- [NVENC resources](/quest/m0/nvenc-resources.md) - settled lifetime and cleanup ownership +- [NVENC loading](/quest/m0/nvenc-loading.md) - driver failures reach the caller as errors diff --git a/quest/m2/nvenc-reuse.md b/quest/m2/nvenc-reuse.md new file mode 100644 index 0000000000..d7a8f98f26 --- /dev/null +++ b/quest/m2/nvenc-reuse.md @@ -0,0 +1,30 @@ +# [M] Reuse NVENC buffers after completion + +## Goal + +Repeated video encoding avoids unnecessary bitstream/input allocation and CUDA +registration while retaining every resource through its actual completion. + +## Plan + +The moq-video NVENC path allocates/destroys a bitstream per frame and creates an +input buffer or registration; CPU input also creates an interleaved UV buffer. +Measure those costs on representative CPU and CUDA input before choosing pool +depth. Reuse resources internally through the settled moq-nvenc ownership model. + +Bound retained GPU memory, preserve device/context identity, and handle resize, +bitrate changes, delayed output, failure, and teardown. Do not keep a resource +registered against memory that its producer can overwrite or reclaim. + +Test pool ownership and completion with the fake driver in CI; record hardware +latency, throughput, allocations, and registration counts separately. No public +pool controls are needed unless measurements establish a consumer requirement. +Public API and wire: unchanged. + +## Required + +- [NVENC resources](/quest/m0/nvenc-resources.md) - reusable owned resources and completion contract + +## Related + +- [Hardware validation](/quest/m4/video-hardware.md) - device-backed correctness evidence diff --git a/quest/m2/obs-moq-video/adapter.md b/quest/m2/obs-moq-video/adapter.md index 242c0fad9e..10d04d0305 100644 --- a/quest/m2/obs-moq-video/adapter.md +++ b/quest/m2/obs-moq-video/adapter.md @@ -15,6 +15,10 @@ One opt-in Use MoQ encoders choice publishes OBS video and audio through moq-vid ## Required +- [Codec threads](/quest/m0/video-thread-ownership.md) - encoder execution ownership +- [Video frames](/quest/m0/video-frames.md) - owned frame and packet contracts +- [Video timing](/quest/m0/video-timing.md) - rational rates and timestamps + - [OBS migration](/quest/m2/cpp/obs.md) - the plugin is on the generated C++ first - [Encoder presets](/quest/m2/obs-moq-video/presets.md) - common policy - [Audio publishing](/quest/m2/obs-moq-video/audio-publish.md) - both adapters are needed for the combined opt-in UI diff --git a/quest/m2/obs-moq-video/macos.md b/quest/m2/obs-moq-video/macos.md index 1a0302ee4a..ecbcfb909e 100644 --- a/quest/m2/obs-moq-video/macos.md +++ b/quest/m2/obs-moq-video/macos.md @@ -8,9 +8,11 @@ An OBS compositor frame reaches moq-video's VideoToolbox encoder without a GPU-t - Inspect OBS's OpenGL compositor, `encode_texture2`, and mac-videotoolbox input path. Determine whether the output allocation is IOSurface-backed and exportable. OBS's encoder currently copies CPU planes into its own pixel buffer; a CVPixelBuffer in moq-video alone does not remove that upstream readback. - Prefer retained IOSurface/CVPixelBuffer storage in the format VideoToolbox accepts. Otherwise prototype GPU color conversion/blit into an IOSurface-backed NV12 pool. Specify GL/Metal/CoreVideo interop, graphics-context thread affinity, completion fences, and when OBS may recycle the source. -- Reuse `Surface::PixelBuffer` and the moq-video VideoToolbox backend. Retain the destination until encoding completes, including dropped submissions and cancellation. Bound the pool and handle resolution/HDR changes and device failure. +- Reuse the native PixelBuffer surface and the moq-video VideoToolbox backend. Retain the destination until encoding completes, including dropped submissions and cancellation. Bound the pool and handle resolution/HDR changes and device failure. - Verify no CPU readback using GPU/API traces and copy counters. Compare direct import or GPU blit against the CPU baseline at 1080p60 and 4K where supported. Check decoded color bars and moving timestamps, latency percentiles, audio sync, stop/restart, and long-running pool reuse on Apple hardware. ## Required +- [Video frames](/quest/m0/video-frames.md) - settled native ownership and conversion contract + - [Encoder adapter](/quest/m2/obs-moq-video/adapter.md) - frame ownership, queue policy, packet output, and comparison baseline diff --git a/quest/m2/obs-moq-video/rate-control.md b/quest/m2/obs-moq-video/rate-control.md index 1eebac73c9..767dd5315e 100644 --- a/quest/m2/obs-moq-video/rate-control.md +++ b/quest/m2/obs-moq-video/rate-control.md @@ -14,13 +14,15 @@ configured rate. Uses the reservation surface from libmoq (`moq_session_bandwidth`, `moq_bandwidth_reserve`, `moq_reservation_grant`). Apply grants through -the shape `moq_video::encode::rate::Control` uses (drops at once, raises ramp, +the shape `moq_mux::rate::Control` uses (drops at once, raises ramp, hysteresis) rather than pushing every change into `obs_encoder_update`; whether that policy sits in moq-ffi behind the reservation or in the plugin depends on whether a second binding wants it. Verify against a shaped uplink and with `just obs compile` and `just obs test`; document the behaviour in `doc/bin/obs.md`. -<## Required +## Required + +- [Shared rate policy](/quest/m0/media-rate-policy.md) - use its settled shared namespace - [OBS migration](/quest/m2/cpp/obs.md) - the plugin is on the generated C++ first diff --git a/quest/m2/obs-moq-video/source.md b/quest/m2/obs-moq-video/source.md index 13c42501d7..98413f8b17 100644 --- a/quest/m2/obs-moq-video/source.md +++ b/quest/m2/obs-moq-video/source.md @@ -14,7 +14,7 @@ The MoQ source loads and plays supported video without directly linking FFmpeg l - Remove FFmpeg includes, CMake discovery/linkage, unit stubs, compile recipe requirements, and unused swresample linkage. Update OBS build/install docs and `doc/lib/cpp` together. libobs/Qt and native OS/GPU dependencies remain. - Validate new code with decoded pixels and moving timestamps, GPU copy/readback traces, and p50/p95 decode-to-presentation delay. Exercise CPU fallback, unsupported codec, GPU import failure, device loss, repeated start/stop, rendition change, and delayed terminal completion. Verify no AVCodec/AVUtil/SWScale/SWResample imports using platform binary inspection. Load the artifact against the oldest supported OBS release and current stable release, using the repo's supported version policy at implementation time. -<## Required +## Required - [OBS migration](/quest/m2/cpp/obs.md) - the plugin is on the generated C++ before decode changes - [Decoded frame ownership](/quest/m2/decoded-frames.md) - retains the existing frame and defines native-view lifetime before OBS imports it diff --git a/quest/m2/obs-moq-video/windows.md b/quest/m2/obs-moq-video/windows.md index ab640e4b93..1dc375e324 100644 --- a/quest/m2/obs-moq-video/windows.md +++ b/quest/m2/obs-moq-video/windows.md @@ -7,10 +7,12 @@ The moq-video OBS encoder consumes compositor output through D3D11 without CPU s ## Plan - Inspect OBS `encoder_texture` shared handles and `encode_texture2` lock_key/next_key semantics. Record whether the source is packed RGB or split NV12 planes, its adapter LUID, and the lifetime OBS guarantees after the callback returns. -- Reuse `Surface::Texture` where the encoder accepts the same device and format. Otherwise GPU-convert/blit into a bounded encoder-owned NV12 pool before returning the OBS synchronization key. An AddRef alone does not prevent OBS from overwriting pooled pixels. +- Reuse the native D3D11 surface where the encoder accepts the same device and format. Otherwise GPU-convert/blit into a bounded encoder-owned NV12 pool before returning the OBS synchronization key. An AddRef alone does not prevent OBS from overwriting pooled pixels. - Audit keyed mutex/fence sequencing, asynchronous completion, incompatible adapters, software fallback, resolution changes, and device removal. Start with Media Foundation, which accepts D3D11 surfaces. The current NVENC backend is Linux-only and directly imports CUDA surfaces there; Windows NVENC/D3D11 support is separate backend work. Do not infer interoperability from both APIs accepting a texture handle. - Trace readback and GPU copy counts on real Windows hardware. Verify pixels and A/V timestamps with a subscriber, compare latency and utilization to the CPU baseline, and exercise cancellation while textures remain in flight. Include hybrid-GPU and device-mismatch rejection where available. ## Required +- [Video frames](/quest/m0/video-frames.md) - settled native ownership and conversion contract + - [Encoder adapter](/quest/m2/obs-moq-video/adapter.md) - frame ownership, queue policy, packet output, and comparison baseline diff --git a/quest/m2/transcode-resources.md b/quest/m2/transcode-resources.md new file mode 100644 index 0000000000..b43607b33d --- /dev/null +++ b/quest/m2/transcode-resources.md @@ -0,0 +1,32 @@ +# [M] Measure transcode resource and startup costs + +## Goal + +The transcode ladder has measured thread, codec-session, memory, and startup +costs under live and fetched demand, and removes demonstrated overhead without +changing its public API or starving a rung behind another. + +## Plan + +Live rungs already share one decoder. The feed holds 16 items; each rung admits +four concurrent fetch pipelines. Those per-rung limits do not bound aggregate +GPU sessions or codec worker threads across broadcasts. Every newly resolved +rung also probes a temporary real encoder and synthetic frame before catalog +publication. macOS sinks execute inline; other hosts create codec threads. + +Measure increasing broadcast/rung counts, live plus fetch demand, slow encoders, +source resize, and demand churn. Include executor responsiveness, retained GPU +frames, time to first catalog/frame, and cancellation cleanup. Preserve the +honest probed codec description rather than replacing probes with guesses. +Only change scheduling, pooling, or internal budgets after identifying the cost. + +Keep group-boundary drain coverage independent of hardware: a delayed fake +decoder must deliver the tail inside its original live/fetched group. Existing +Consumer EOF tests and hardware-conditional VAAPI tests are not that same proof. +Wire resource/ownership regressions and reproducible benchmarks into CI or the +existing nightly harness. Public API and wire: unchanged. + +## Related + +- [Benchmark comparisons](/quest/m2/performance-comparisons.md) - repeatable measurements +- [Adaptive ladder](/quest/m2/ladder/README.md) - bandwidth control is separate from process resource cost diff --git a/quest/m2/transcode-source.md b/quest/m2/transcode-source.md new file mode 100644 index 0000000000..33c97f380e --- /dev/null +++ b/quest/m2/transcode-source.md @@ -0,0 +1,28 @@ +# [M] Choose a locally decodable transcode source + +## Goal + +A higher-resolution rendition that this host cannot decode does not prevent +transcoding another usable rendition in the same source catalog. + +## Plan + +catalog::choose_source filters by codec syntax and dimensions, then ranks the +largest rendition. It does not establish whether the selected backend can +decode it. On a software-only host, a larger H.265 or AV1 entry can win over a +usable H.264 entry and fail later when demand opens the decoder. + +Reproduce that case with deterministic backend capabilities and preserve the +explicit backend selection policy. Choose among locally supported candidates +or return an actionable refusal when none exists. Avoid repeated encoder or +decoder probes for every catalog edit, and do not turn malformed stream input +into silent fallback after a rendition is already being consumed. + +Test mixed-codec catalogs, an explicitly forced unavailable backend, no usable +candidate, and a catalog update while the selected rendition remains valid. +Keep rendition identity, output names, and existing live/fetch behavior stable. +Run the regressions in CI. Public API and wire: unchanged. + +## Required + +- [Video output](/quest/m0/video-output.md) - use the settled decoder configuration diff --git a/quest/m2/video-render-resources.md b/quest/m2/video-render-resources.md new file mode 100644 index 0000000000..02daac13a8 --- /dev/null +++ b/quest/m2/video-render-resources.md @@ -0,0 +1,34 @@ +# [M] Bound renderer retention and validate output resources + +## Goal + +Video rendering has bounded retained imported surfaces, predictable device-loss +cleanup, and errors for unsupported output configuration through its existing +Result contract. + +## Plan + +The Linux completion worker uses an unbounded channel and waits indefinitely +for submissions while retaining source leases. Renderer output creation checks +nonzero size but forwards format, usage, and dimensions to wgpu without checking +all supported combinations/device limits. Trace both paths before changing +ownership or validation. + +Bound in-flight retention without recycling source pixels before completion; +define cleanup when a device is lost. Validate supported format/usage/size or +route validation failures through the existing error result. Preserve the +documented reusable output-texture alias: a returned handle is overwritten on +the next render, and callers needing independent storage copy it. + +Measure per-frame conversion, bind-group/view allocation, and retained memory. +Optimize only demonstrated costs. Test rejected configurations, delayed +completion, teardown, and device-loss handling in CI with fake completion where +possible; keep native import proof in hardware jobs. Public API and wire: none. + +## Required + +- [Video frames](/quest/m0/video-frames.md) - settled surface/conversion ownership + +## Related + +- [Hardware validation](/quest/m4/video-hardware.md) - real graphics-device proof diff --git a/quest/m3/README.md b/quest/m3/README.md index 158df6eb6d..868b04244d 100644 --- a/quest/m3/README.md +++ b/quest/m3/README.md @@ -21,12 +21,14 @@ worth settling now. Deferral does not by itself abandon a feature. - [Catalog track identity](/quest/m3/catalog-tracks.md) - compare immutable track definitions with explicit catalog-to-group binding - [Mobile ownership](/quest/m3/mobile-ownership.md) - decide whether Rust or platform code owns mobile capture, codecs, and rendering - [iOS capture](/quest/m3/mobile-capture-ios.md) - camera and screen capture if the mobile ownership decision selects Rust -- [Android capture](/quest/m3/mobile-capture-android.md) - NDK/JNI capture and codec backends if the mobile ownership decision selects Rust +- [Android capture](/quest/m3/mobile-capture-android.md) - NDK/JNI capture using the existing codecs if mobile ownership selects Rust - [Mobile completion](/quest/m3/mobile-completion.md) - verify the selected native/mobile path before closing #700 - [Linux OBS GPU input](/quest/m3/obs-linux-gpu.md) - publish OBS compositor frames without CPU readback on a validated Linux graphics/encoder combination - [LiveKit client shim](/quest/m3/livekit-shim.md) - a media compatibility facade over the room SDK -- [Decoder drain](/quest/m3/decode-drain.md) - flush and finish for pipelined decoders, so no picture is lost at a track end or crosses a group boundary +- [Audio loss recovery](/quest/m3/audio-loss-recovery.md) - prove a useful Opus recovery policy before exposing another option +- [Opus implementation](/quest/m3/audio-opus-backend.md) - compare current codec quality, CPU, and optional build costs +- [Video codec coverage](/quest/m3/video-codec-coverage.md) - prioritize remaining native AV1 and portable decoder gaps - [#2819](/quest/m3/2819-moq-video-carry-pipewire-dma-bufs-safely-into-the-vulkan.md) - moq-video: carry PipeWire DMA-BUFs safely into the Vulkan renderer - [Unreal prototype](/quest/m3/unreal.md) - a UE5 module on the C++ package with exceptions disabled, rendering a subscribed broadcast to a texture - [Unity prototype](/quest/m3/unity.md) - the C# package under IL2CPP, playing subscribed audio diff --git a/quest/m3/audio-loss-recovery.md b/quest/m3/audio-loss-recovery.md new file mode 100644 index 0000000000..18beff5be0 --- /dev/null +++ b/quest/m3/audio-loss-recovery.md @@ -0,0 +1,28 @@ +# [M] Decide a tested Opus loss-recovery policy + +## Goal + +Decide whether a concrete MoQ audio consumer benefits from in-band Opus FEC, +with a tested loss/latency policy before exposing a replacement public option. +This does not block 0.1. + +## Plan + +The old boolean supplied no expected-loss percentage and our decoder never +requested FEC recovery. Treat source loss, late-group abandonment, and DTX as +different cases. Define sequencing, expected loss, playout lookahead, recovery +versus concealment, and behavior when the next packet is unavailable. + +Use deterministic dropped-packet fixtures to prove redundancy is emitted and +used, compare audible quality and delay against concealment, and name the +transport scenario where it helps. End with a measured go/no-go and a small +additive policy if justified; do not reintroduce an enable flag tested only by +reading the codec's control value. Implementation fixtures belong in CI. + +Public API: no change during the study; any later policy must fit the extensible +audio settings. Existing wire compatibility must be demonstrated. + +## Related + +- [Remove ineffective FEC](/quest/m0/audio-fec.md) - removes the unsupported promise before 0.1 +- [Audio quality](/quest/m2/audio-quality-harness/README.md) - quality and latency measurements diff --git a/quest/m3/audio-opus-backend.md b/quest/m3/audio-opus-backend.md new file mode 100644 index 0000000000..8872926f34 --- /dev/null +++ b/quest/m3/audio-opus-backend.md @@ -0,0 +1,28 @@ +# [M] Compare current Opus implementations + +## Goal + +Determine whether an optional newer Opus backend improves quality or CPU enough +to justify its build and maintenance cost while retaining the simple Rust path. + +## Plan + +The manifest describes unsafe-libopus as a Rust port of 1.3.1. Compare it with +a currently maintained upstream implementation, including +[Opus 1.6](https://opus-codec.org/demo/opus-1.6/), rather than treating old +unmeasured percentage claims as a decision. Verify the actual versions at +implementation time. + +Measure relevant voice/music rates, frame sizes, DTX, loss behavior, startup, +CPU, and package/toolchain cost. Keep native compilation optional and use the +existing private backend seam; do not add another public configuration surface +just to expose one implementation's controls. A no-go is a valid outcome. + +Retain fixtures and repeatable measurements in the existing CI/nightly audio +harness. Public API and wire: no change for the study; validate compatibility +before a separately scoped backend implementation. + +## Related + +- [Audio configuration](/quest/m0/audio-config.md) - stable settings and backend-selection boundary +- [Audio quality](/quest/m2/audio-quality-harness/README.md) - shared measurement infrastructure diff --git a/quest/m3/decode-drain.md b/quest/m3/decode-drain.md deleted file mode 100644 index d31cc5d136..0000000000 --- a/quest/m3/decode-drain.md +++ /dev/null @@ -1,33 +0,0 @@ -# [M] Drain pipelined decoders at group and track boundaries - -## Goal - -A decoder that holds pictures back (NVDEC, the V4L2 stateful decoder) hands -every one of them over before a track ends, and a transcode's output groups -begin with the picture that began the input group rather than a straggler -from the one before. - -## Plan - -`decode::Backend` has `decode` and `name` and nothing else: there is no way to -tell a backend the stream is pausing or over. A one-in-one-out decoder -(openh264, VideoToolbox) never needs one, but NVDEC and V4L2 both pipeline, so -the last few pictures of a finite stream stay inside the driver, and at a -group boundary in `moq-transcode` the encoder is flushed while the decoder is -not, so pictures from the closing group surface after the next group's first -access unit went in and are encoded ahead of its keyframe. - -Add `flush` and `finish` to `decode::Backend`, mirroring the encode trait and -with the same rule: no default implementation, so a pipelined backend cannot -inherit a no-op and look like it worked. The V4L2 implementation is the -kernel's drain sequence (`V4L2_DEC_CMD_STOP`, dequeue CAPTURE through -`V4L2_BUF_FLAG_LAST`, `V4L2_DEC_CMD_START`), with a source change that lands -mid-drain handled the way `drain_tail` already handles one. NVDEC's is an -end-of-stream packet through the parser. Call `flush` where `moq-transcode` -flushes the encoder and `finish` before a consumer reports the track over, and -cover it with a delayed-backend test that proves no picture is lost or crosses -a group boundary. - -## Related - -- [Embedded video](/quest/m4/video-embedded.md) - the V4L2 decoder this drains diff --git a/quest/m3/mobile-capture-android.md b/quest/m3/mobile-capture-android.md index 480a538516..0f44e6e3a2 100644 --- a/quest/m3/mobile-capture-android.md +++ b/quest/m3/mobile-capture-android.md @@ -7,10 +7,9 @@ camera, MediaProjection for the screen, and MediaCodec for encode and decode. ## Plan -A whole backend family, not a port. Every other platform backend is objc2 or a -C API; this one is NDK and JNI against the Android framework, and MediaCodec's -Surface-in/Surface-out model is natively zero-copy in a shape none of the -existing backends share. +MediaCodec encode/decode already exist in moq-video. Reuse them rather than +planning a second backend family. The remaining capture and native Surface +integration needs NDK/JNI lifecycle, synchronization, and actual device proof. Weigh the cost honestly before starting. `moq-kit` already does this in Kotlin over `moq-ffi`, and raw frames cannot cross the FFI boundary zero-copy, so a @@ -24,6 +23,8 @@ so the mechanism exists. ## Required +- [Video timing](/quest/m0/video-timing.md) - timestamped capture and rational rates + - [Ownership boundary](/quest/m3/mobile-ownership.md) - decides whether an NDK/JNI backend family is worth building ## Related diff --git a/quest/m3/mobile-capture-ios.md b/quest/m3/mobile-capture-ios.md index e58c1182bc..56f487cef1 100644 --- a/quest/m3/mobile-capture-ios.md +++ b/quest/m3/mobile-capture-ios.md @@ -8,9 +8,9 @@ through ReplayKit. ## Plan Not a new codec backend. VideoToolbox already encodes and decodes as the macOS -backend and works on iOS unchanged, and `Surface::PixelBuffer` already carries -a `CVPixelBuffer` zero-copy, so this is capture wiring plus the lifecycle iOS -imposes and macOS does not. +backend. Reuse its native PixelBuffer surface and verify the iOS build and +runtime path rather than assuming desktop behavior. The new work is capture +wiring plus the lifecycle iOS imposes and macOS does not. That lifecycle is the work. Camera and screen access are permission-gated and revocable, an app is suspended and resumed on foreground changes, and @@ -24,6 +24,8 @@ same everywhere. ## Required +- [Video timing](/quest/m0/video-timing.md) - timestamped capture and rational rates + - [Ownership boundary](/quest/m3/mobile-ownership.md) - decides whether Rust owns capture on mobile at all ## Related diff --git a/quest/m3/video-codec-coverage.md b/quest/m3/video-codec-coverage.md new file mode 100644 index 0000000000..f8e37e64e4 --- /dev/null +++ b/quest/m3/video-codec-coverage.md @@ -0,0 +1,34 @@ +# [M] Prioritize remaining native and portable video codec coverage + +## Goal + +Identify which missing decoder/backend coverage serves actual consumers after +the existing platform and NVIDIA work, without adding mandatory build costs or +blocking 0.1. + +## Plan + +Review AV1 decode through Apple VideoToolbox and Windows Media Foundation, +and a portable software AV1 fallback such as +[dav1d](https://code.videolan.org/videolan/dav1d). Establish actual framework, +OS, extension, hardware, format, and fixture requirements; a browser playing +a codec is not proof that our native backend can open it. + +NVIDIA AV1 encode/10-bit, VAAPI expansion, and VP8/VP9 already have quests. +Keep those owners. Windows NVIDIA support needs a demonstrated advantage over +the existing native path. If revisiting software AV1 encoding, measure the +target real-time workload and build cost rather than assuming all presets or +all hardware are equivalent. Optional codec dependencies stay optional. + +Produce a supported/refused matrix and separately completable implementation +quests only for justified gaps. Each selected backend needs decode fixtures, +drain/color/timestamp proof, and a CI or explicit hardware validation lane. +Use the settled Frame/output/codec extension points; no parallel surface API. + +Public API and wire: no changes during this study. + +## Related + +- [NVIDIA formats](/quest/m2/2147-moq-video-10-bit-hevc-and-av1-support-in-the-nvidia-codec.md) - existing AV1 encode and 10-bit scope +- [VAAPI](/quest/m2/video-vaapi.md) - existing Linux codec expansion +- [VP8/VP9](/quest/m2/obs-moq-video/vpx.md) - existing portable decoder scope From bdbab4254ae25c8d118fcc336a99ea999c8752f7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 20 Sep 2026 17:46:10 -0700 Subject: [PATCH 2/2] docs(quest): spell out media package boundaries Record each audited crate's public surface and the additive-only moq-mux plus unchanged binding boundaries. Co-Authored-By: GPT-5 --- quest/m0/README.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/quest/m0/README.md b/quest/m0/README.md index ea384587c3..a1738d9106 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -19,6 +19,22 @@ in other packages without breaking their published APIs, C layouts, or wire formats. Do not bump versions as part of these quests. The final review records when the four crates are ready for a separately requested release. +The package boundaries are explicit: + +- `moq-audio` owns the PCM/layout and codec configuration split, decoder entry + point, publication authority, FEC removal, AEC attachment, playback outcome, + and extensible audio frame and packet construction. +- `moq-video` owns frame conversion and construction, decoder output policy, + synchronous codec thread confinement, capture timestamps and rational rates, + extensible group configuration and `cut` naming, and its feature defaults. +- `moq-transcode` adopts the video rate, group, output, and feature contracts in + its public configuration and observations without adding another media model. +- `moq-nvenc` narrows its safe facade around owned resources and completion, + while loading and incompatibility become fallible public errors. +- Published `moq-mux` gains only the additive shared `rate` namespace. Published + `moq-ffi`, `libmoq`, and language-binding signatures, layouts, and sentinel + behavior remain unchanged while their internals adapt. + The agreed direction is small, honest APIs: typed PCM layouts, rational video rates, extensible GOP and frame records, explicit ownership, and no knobs that claim behavior they do not provide. Synchronous codecs remain public and