diff --git a/doc/lib/c/index.md b/doc/lib/c/index.md index b8425b713e..5a4fcc0e9c 100644 --- a/doc/lib/c/index.md +++ b/doc/lib/c/index.md @@ -35,7 +35,7 @@ and `target/include/moq.h`. - **Threading.** Any function from any thread. Raw publish calls block until the codec takes the frame, which paces a publisher. - **Connection health.** `moq_session_stats()` reports available metrics with per-field validity flags. `moq_session_snapshot()` samples those metrics and the negotiated draft name together from the same connection. Its protocol string is backed by static storage. Both return an offline error between reconnects and leave the destination untouched. `moq_session_bandwidth()` mints an allocator over the send estimate; `moq_bandwidth_reserve` claims a share for an app-owned track, and `moq_encode_video` / `moq_encode_audio` take the same handle so the built-in video encoder follows the grant. - **Raw playback.** Raw audio and video consumers start at the newest cached group when opened, so rebuilding a live decoder skips the retained backlog. -- **Raw audio encode.** `moq_audio_encoder_output.codec` names the codec (only `"opus"` today), and `frame_duration_us` sets the Opus frame length: 2500, 5000, 10000, 20000, 40000, or 60000, with 0 meaning the 20 ms default. +- **Raw audio encode.** `moq_audio_encoder_output.codec` names the codec: `"opus"`, `"pcm"`, or `"aac"`. `frame_duration_us` sets the Opus frame length: 2500, 5000, 10000, 20000, 40000, or 60000, with 0 meaning the codec's default (20 ms for Opus, 1024 samples for AAC). AAC-LC encodes through the platform's encoder, so a host without one refuses it. - **Audio channel layouts.** A `channels` count also names the speaker layout, by the WAVE convention: 1 is mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved front left, front right, center, LFE, back, then side. `moq_decode_audio` remixes to the count you ask for; past 8 channels the samples pass through but can't be remixed. - **Raw decode output.** `moq_video_decoder_output` selects the decoded CPU pixel format (`MOQ_VIDEO_PIXEL_FORMAT_I420` or `_RGBA`) and target size (`width`/`height`, both zero for native; otherwise even and non-zero). Unknown formats and invalid sizes fail `moq_decode_video` before subscribing; accepted requests deliver exactly that layout or fail on the terminal callback. - **Encoded video metadata.** `moq_video_init.hint` is a zero-initialized `moq_video_hint` with `has_*` flags for coded dimensions, bitrate (bits per second), frame rate, and latency preference. Hints seed a video codec track's catalog; detected dimensions take precedence. diff --git a/doc/lib/go/index.md b/doc/lib/go/index.md index 0540cafb90..8a49af5747 100644 --- a/doc/lib/go/index.md +++ b/doc/lib/go/index.md @@ -115,6 +115,12 @@ one: `FetchGroup`/`FetchMediaGroup`, `Dynamic()` with `Requests(ctx)`, `AppendDatagram`/`Datagrams(ctx)`, `SetCatalogSection`, `Demand()` for `Used`/`Unused`, `Session().Stats()`. `moq.IsAuthError` and `moq.IsShutdown` classify errors. `moq.ProtocolError(err)` is the structured protocol failure (scope, verbatim code, kind) when the peer sent one. +`EncodeAudio` encodes raw PCM inside the binding. Its codec is `OpusAudioCodec()` +or `AacAudioCodec()`, and `AudioEncoderOutput.FrameDurationUs` sets the Opus +frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. 0 takes +the codec's own frame, which AAC needs. AAC-LC encodes through the platform's +encoder, so a host without one refuses it. + Audio `Channels` also names the speaker layout, by the WAVE convention: 1 is mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved front left, front right, center, LFE, back, then side. Decoding remixes to the diff --git a/doc/lib/kt/index.md b/doc/lib/kt/index.md index 8f863b942b..7f552b3037 100644 --- a/doc/lib/kt/index.md +++ b/doc/lib/kt/index.md @@ -91,8 +91,11 @@ connection's send estimate; pass it to `encodeVideo` / `encodeAudio` or native side. `encodeAudio` encodes raw PCM inside the binding. Its codec is an object, -`AudioCodec.opus()`, and `AudioEncoderOutput.frameDurationUs` sets the Opus -frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. +`AudioCodec.opus()` or `AudioCodec.aac()`, and +`AudioEncoderOutput.frameDurationUs` sets the Opus frame length: 2500, 5000, +10000, 20000 (the default), 40000, or 60000. 0 takes the codec's own frame, +which AAC needs. AAC-LC encodes through the platform's encoder, so a host +without one refuses it. Audio `channels` also names the speaker layout, by the WAVE convention: 1 is mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved diff --git a/doc/lib/py/index.md b/doc/lib/py/index.md index 5fe4e12de3..1ce6293a5e 100644 --- a/doc/lib/py/index.md +++ b/doc/lib/py/index.md @@ -107,8 +107,11 @@ Each server request reports a `moq.Transport` enum, including QUIC, Iroh, WebSocket, TCP, and Unix sockets. `encode_audio` encodes raw PCM inside the binding. Its codec is an object, -`moq.AudioCodec.opus()`, and `AudioEncoderOutput.frame_duration_us` sets the -Opus frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. +`moq.AudioCodec.opus()` or `moq.AudioCodec.aac()`, and +`AudioEncoderOutput.frame_duration_us` sets the Opus frame length: 2500, 5000, +10000, 20000 (the default), 40000, or 60000. 0 takes the codec's own frame, +which AAC needs. AAC-LC encodes through the platform's encoder, so a host +without one refuses it. Audio `channels` also names the speaker layout, by the WAVE convention: 1 is mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved diff --git a/doc/lib/rs/moq-audio.md b/doc/lib/rs/moq-audio.md index 637343dd61..c76dc1179e 100644 --- a/doc/lib/rs/moq-audio.md +++ b/doc/lib/rs/moq-audio.md @@ -28,7 +28,7 @@ policy. Decoding likewise separates low-level `decode::Config`, PCM | Module | Does | | --- | --- | | `capture` | Microphones via CoreAudio, WASAPI, ALSA (and PipeWire/PulseAudio hosts), plus macOS system audio | -| `encode` | PCM to Opus (with DTX and voice-activity signaling) or raw PCM for the lowest latency | +| `encode` | PCM to Opus (with DTX and voice-activity signaling), raw PCM for the lowest latency, or AAC-LC through a platform encoder | | `decode` | Opus, PCM, and AAC-LC back to PCM, resampled to the rate you want | | `playback` | One output device mixing every track in a call, with click-free volume ramps | | `aec` | Acoustic echo cancellation (a port of WebRTC's), so a laptop with no headset doesn't feed itself back | @@ -48,6 +48,25 @@ its config are refused at construction on every host. HE-AAC signaled only in band plays as its half-rate LC core. Linux has no OS audio decoder, so it will stay that way there. +`encode` selects the same way, through `encode::Settings::kind`, and +`Encoder::name()` reports what opened. + +| Backend | Encodes | Hosts | +| --- | --- | --- | +| `libopus` | Opus, mono or stereo | all | +| `pcm` | PCM | all | + +`encode::Codec::Aac` is AAC-LC (`mp4a.40.2`) at the input's rate and layout: +mono, stereo, 3.0, 4.0, 5.0, 5.1, or 7.1, the layouts with an AAC +channelConfiguration. Frames are 1024 samples, so `Settings::from_input` sets +`frame_duration` to match. The catalog's AudioSpecificConfig is built from the +settings when the track is registered, and since it has no field for the +encoder's delay, packets are stamped that much earlier so the first input +sample still lands at the first timestamp. There is no software AAC encoder, +and no platform encoder is wired in yet, so `Codec::Aac` is refused at +construction on every host for now. Linux has no OS encoder, so it will stay +that way there. + Highlights: - **`encode::Publication`** advertises the track and opens the microphone only while someone listens. Stop, swap devices, and restart without changing the track subscribers know; read a level meter for the UI. diff --git a/doc/lib/swift/index.md b/doc/lib/swift/index.md index dc02120fe7..39ab5e152a 100644 --- a/doc/lib/swift/index.md +++ b/doc/lib/swift/index.md @@ -95,8 +95,11 @@ divides the connection's send estimate; pass it to `encodeVideo` / (scope, verbatim code, kind) when the peer sent one. `encodeAudio` encodes raw PCM inside the binding. Its codec is an object, -`AudioCodec.opus()`, and `AudioEncoderOutput.frameDurationUs` sets the Opus -frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. +`AudioCodec.opus()` or `AudioCodec.aac()`, and +`AudioEncoderOutput.frameDurationUs` sets the Opus frame length: 2500, 5000, +10000, 20000 (the default), 40000, or 60000. 0 takes the codec's own frame, +which AAC needs. AAC-LC encodes through the platform's encoder, so a host +without one refuses it. Audio `channels` also names the speaker layout, by the WAVE convention: 1 is mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved diff --git a/go/wrapper/publish.go b/go/wrapper/publish.go index 6eccc6d8f6..6f46b1ff2d 100644 --- a/go/wrapper/publish.go +++ b/go/wrapper/publish.go @@ -193,7 +193,7 @@ func (b *BroadcastProducer) PublishContainerStream(format ContainerFormat) (*Con // EncodeAudio publishes a raw-audio track with an in-process encoder. // -// Select the codec with OpusAudioCodec (currently the only constructor). +// Select the codec with OpusAudioCodec or AacAudioCodec. // Pass bandwidth to reserve this track's bitrate against the session's // allocator so a co-resident video encoder sizes itself against what is left. func (b *BroadcastProducer) EncodeAudio(name string, input AudioEncoderInput, output AudioEncoderOutput, bandwidth *Bandwidth) (*AudioProducer, error) { diff --git a/go/wrapper/types.go b/go/wrapper/types.go index 40bf4f53ea..3c588ba2df 100644 --- a/go/wrapper/types.go +++ b/go/wrapper/types.go @@ -8,14 +8,14 @@ import ffi "moq.dev/moq-ffi/moq" type ( // Audio describes one audio rendition in a broadcast catalog: codec, sample rate, channel count, and container. Audio = ffi.MoqAudio - // AudioCodec selects the audio encoder codec. Build one with OpusAudioCodec; + // AudioCodec selects the audio encoder codec. Build one with OpusAudioCodec or AacAudioCodec; // adding a codec later adds a constructor, not a breaking enum change. AudioCodec = ffi.MoqAudioCodec // AudioDecoderOutput configures the PCM format, sample rate, and channels DecodeAudio delivers. AudioDecoderOutput = ffi.MoqAudioDecoderOutput // AudioEncoderInput declares the PCM sample format, sample rate, and channel count of frames written to an audio producer. AudioEncoderInput = ffi.MoqAudioEncoderInput - // AudioEncoderOutput configures the Opus encoder: codec, optional sample rate, channels, bitrate, and frame duration. + // AudioEncoderOutput configures the encoder: codec, optional sample rate, channels, bitrate, and frame duration. AudioEncoderOutput = ffi.MoqAudioEncoderOutput // AudioSampleFormat is a raw PCM sample layout, mirroring WebCodecs AudioData.format. AudioSampleFormat = ffi.MoqAudioSampleFormat @@ -184,6 +184,12 @@ func OpusAudioCodec() *AudioCodec { return ffi.MoqAudioCodecOpus() } +// AacAudioCodec selects AAC-LC through the platform's encoder for EncodeAudio. +// A host without one refuses it. Leave FrameDurationUs at 0 for AAC's own frame. +func AacAudioCodec() *AudioCodec { + return ffi.MoqAudioCodecAac() +} + // VideoPixelFormat values: the raw pixel layout fed to the in-process encoder, // and the one the in-process decoder delivers. const ( diff --git a/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt b/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt index 38883f6cda..f59782cf66 100644 --- a/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt +++ b/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt @@ -134,7 +134,7 @@ typealias FetchGroupOptions = uniffi.moq.MoqFetchGroupOptions typealias TrackInfo = uniffi.moq.MoqTrackInfo /** One audio frame: PCM payload bytes plus a presentation timestamp. */ typealias AudioFrame = uniffi.moq.MoqAudioFrame -/** Selects the audio encoder codec. Build one with `AudioCodec.opus()`. */ +/** Selects the audio encoder codec. Build one with `AudioCodec.opus()` or `AudioCodec.aac()`. */ typealias AudioCodec = uniffi.moq.MoqAudioCodec /** A raw PCM sample format, mirroring WebCodecs `AudioData.format`. */ typealias AudioSampleFormat = uniffi.moq.MoqAudioSampleFormat diff --git a/py/moq-rs/README.md b/py/moq-rs/README.md index 6add3e7adb..0b60f1cb92 100644 --- a/py/moq-rs/README.md +++ b/py/moq-rs/README.md @@ -154,7 +154,7 @@ client = moq.Client( - `.publish_audio(format, init, *, label=None, track=None) → MediaProducer`. `init` is required: an OpusHead or AudioSpecificConfig resolves the whole rendition. `track` names the track; otherwise a unique name is derived from the format. - `.publish_video(format, init=b"", *, label=None, hint=None, track=None) → MediaProducer`. `init` may be empty for a format that resolves in band; a `VideoHint` pins catalog fields the stream can't reveal (bitrate) or publishes the catalog before the first keyframe. `track` names the track as in `publish_audio`. - `.encode_video(input, output, *, bandwidth=None) → VideoProducer`. Encode raw `VideoFrame`s inside the binding; `.write(frame)` each one. - - `.encode_audio(name, input, output, *, bandwidth=None) → AudioProducer`. Encode raw PCM `AudioFrame`s; the codec is `output.codec`, e.g. `AudioCodec.opus()`, with `output.frame_duration_us` setting the Opus frame length. + - `.encode_audio(name, input, output, *, bandwidth=None) → AudioProducer`. Encode raw PCM `AudioFrame`s; the codec is `output.codec`, e.g. `AudioCodec.opus()` or `AudioCodec.aac()`, with `output.frame_duration_us` setting the Opus frame length (0 takes the codec's own frame, which AAC needs). - `.finish()` - **`BroadcastDynamic`**. Async source of tracks requested by subscribers. - `await .requested_track() → TrackRequest`. Call `.accept()` on it for a `TrackProducer`, or `.abort(code)` to reject. diff --git a/py/moq-rs/moq/publish.py b/py/moq-rs/moq/publish.py index cbaf29ba13..e7e8bb03af 100644 --- a/py/moq-rs/moq/publish.py +++ b/py/moq-rs/moq/publish.py @@ -749,8 +749,8 @@ def encode_audio( ) -> AudioProducer: """Publish a raw-audio track with an in-process encoder. - Select the codec with ``moq.AudioCodec.opus()`` (currently the only - constructor), placed in ``output``. + Select the codec with ``moq.AudioCodec.opus()`` or + ``moq.AudioCodec.aac()``, placed in ``output``. Pass ``bandwidth`` to reserve this track's bitrate against the session's allocator so a co-resident video encoder sizes itself against what is left. diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index fdeca61705..ee3cca548f 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -45,7 +45,6 @@ its own decode and encode quest so verification stays per host. - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - mapping family 1 decodes on every host through the multistream decoder -- [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - `encode::backend` and `Codec::Aac`, so a native publisher can produce AAC-LC - [AudioToolbox encode](/quest/m1/audio-codecs/encode-audiotoolbox.md) - macOS and iOS encode AAC-LC ## Related diff --git a/quest/m1/audio-codecs/encode-audiotoolbox.md b/quest/m1/audio-codecs/encode-audiotoolbox.md index 2ceda88280..71267e9b63 100644 --- a/quest/m1/audio-codecs/encode-audiotoolbox.md +++ b/quest/m1/audio-codecs/encode-audiotoolbox.md @@ -15,11 +15,15 @@ the encode seam as the platform candidate on macOS and iOS. `kAudioConverterPrimeInfo` gives the delay the timestamps fold in. - Bitrate through `kAudioConverterEncodeBitRate`, updated live where the converter allows. +- The seam assumes one packet per frame. If the converter holds output back, + the backend needs a `flush` and a zero-or-more return, which changes + `Encoder::encode` and so targets `dev`. +- Gate the seam's "AAC refused without a platform encoder" test to hosts + without one. - Regression: a stereo and a 5.1 encode round-trip through the AudioToolbox decoder and through symphonia (stereo only), with timestamps continuous across the priming. ## Required -- [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - the candidate order this backend joins - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - the round-trip regression decodes through it diff --git a/quest/m1/audio-codecs/encode-backend.md b/quest/m1/audio-codecs/encode-backend.md deleted file mode 100644 index 4f73ed98f2..0000000000 --- a/quest/m1/audio-codecs/encode-backend.md +++ /dev/null @@ -1,40 +0,0 @@ -# [M] An encode backend seam and an AAC output codec - -## Goal - -`moq_audio::encode` selects a backend the way `moq_video::encode` does, and -`Codec::Aac` is a valid output: AAC-LC at the input's sample rate and layout, -published with the AudioSpecificConfig description every player expects. A -host with no AAC encoder refuses it at construction. - -## Plan - -Mirror the decode seam (`rs/moq-audio/src/decode/backend`): `encode::backend` with a crate-private `Backend` -trait (`encode`, `flush`, `set_bitrate`, `name`), an `open(codec, config)` -that walks platform candidates before software ones, using the public settings -and selection contract settled in main. Opus and PCM retain their behavior. This -quest adds AAC through platform encoders; no software AAC dependency is selected. - -- `encode::Codec` gains `Aac`, meaning `mp4a.40.2`, and `as_str` / `FromStr` - accept `"aac"`, which is what libmoq's codec string carries. moq-ffi's - immutable codec object gains an `aac()` constructor and conversion; do not - reintroduce a closed binding enum. The generated bindings, hand-written - wrappers, and docs follow the Cross-Package Sync table. -- Catalog emission: `AudioCodec::AAC` with profile 2, `Container::Legacy`, - and the encoder's reported delay folded into timestamps like Opus pre-skip - is today. `Producer` registers the rendition before the first frame is - written, so the ASC `description` is synthesized at construction from the - config with `moq_mux::codec::aac::Config::encode`, never read back from the - backend's first packet. A backend that reports its own header (a magic - cookie, `csd-0`, `MF_MT_USER_DATA`) must produce one equal to the synthesized - ASC, asserted in its tests. -- Frame size is the codec's (1024 samples for AAC), so `frame_duration` is - validated per codec rather than against the Opus table. -- Bitrate updates go through the backend; one that cannot change rate - mid-stream keeps its opening rate, as the video seam documents. -- Regression: the selection order with a stub backend; `Codec::Aac` refused on - a host with no backend; the Opus and PCM paths unchanged. - -## Related - -- [OBS audio publishing](/quest/m1/obs-moq-video/audio-publish.md) - the OBS encoder adapter can offer AAC once this lands diff --git a/quest/m1/obs-moq-video/audio-publish.md b/quest/m1/obs-moq-video/audio-publish.md index ae2543cd35..3b7cf9981a 100644 --- a/quest/m1/obs-moq-video/audio-publish.md +++ b/quest/m1/obs-moq-video/audio-publish.md @@ -8,7 +8,7 @@ MoQ publishing can encode OBS's mixed audio with moq-audio Opus while preserving - Expose `moq_audio::encode::Encoder` through a codec-only moq-ffi type with owned encoder/packet handles, so every binding gets it. The existing raw audio producer combines encoding and publication and must not create a second publication alongside the OBS encoded output. Reuse frame sizing, bitrate updates, catalog configuration, and finish/padding behavior. - Implement the OBS audio encoder interface, including fixed input frame size, mono/stereo PCM conversion, timestamps, codec headers, final padding, and packet release. Ask OBS for the input layout the encoder supports; moq-audio does not implement arbitrary channel remapping. Keep capture/mixing/device ownership in OBS. -- Publish Opus initially. Leave AAC with the existing OBS mode and defer PCM publishing UI, since the output currently declares AAC/Opus. Correct stale binding documentation that describes the raw codec parser as Opus-only if that API is touched. +- Publish Opus initially. `encode::Codec::Aac` lets the adapter offer AAC where a platform encoder exists; until then leave AAC with the existing OBS mode and defer PCM publishing UI, since the output currently declares AAC/Opus. Correct stale binding documentation that describes the raw codec parser as Opus-only if that API is touched. - Apply the shared presets and independent audio bitrate. Test 10/20 ms packetization, frame-size changes at stream boundaries, partial final frames, silence, reconnect, saturation, and stop with pending output. Verify decoded audio and A/V timestamps with a real subscriber. - Land the internal adapter independently; the combined Use MoQ encoders UI becomes available when the video adapter also lands. Do not expose a temporary video/audio mix-and-match product UI. @@ -16,7 +16,3 @@ MoQ publishing can encode OBS's mixed audio with moq-audio Opus while preserving - [OBS migration](/quest/m1/cpp/obs.md) - the plugin is on the generated C++ first - [Encoder presets](/quest/m1/obs-moq-video/presets.md) - shared policy and truthful reporting - -## Related - -- [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - `Codec::Aac` lets the adapter offer AAC instead of leaving it to the OBS encoder mode diff --git a/quest/m2/audio-encode-mediacodec.md b/quest/m2/audio-encode-mediacodec.md index d488ec184b..8af6ad3e51 100644 --- a/quest/m2/audio-encode-mediacodec.md +++ b/quest/m2/audio-encode-mediacodec.md @@ -13,6 +13,9 @@ behind the `mediacodec` feature and the encode seam. - `audio/mp4a-latm` with `KEY_AAC_PROFILE` = LC. The catalog ASC is synthesized at construction per the encode seam, since `csd-0` only arrives with the first output buffer; assert the two match. +- MediaCodec pipelines output, which the one-packet-per-frame seam does not + allow yet: add a `flush` and a zero-or-more return, a change to + `Encoder::encode` that targets `dev`, unless the AudioToolbox quest already did. - Multichannel is device-dependent; probe the encoder's capabilities at open and refuse a layout it does not list. - Round-trip regression through the MediaCodec decoder; runtime proof on a @@ -20,5 +23,4 @@ behind the `mediacodec` feature and the encode seam. ## Required -- [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - the candidate order this backend joins - [MediaCodec decode](/quest/m2/audio-decode-mediacodec.md) - the round-trip regression decodes through it diff --git a/quest/m2/audio-encode-mediafoundation.md b/quest/m2/audio-encode-mediafoundation.md index 8347dd1854..b2bad52972 100644 --- a/quest/m2/audio-encode-mediafoundation.md +++ b/quest/m2/audio-encode-mediafoundation.md @@ -20,7 +20,6 @@ behind the encode seam on Windows. ## Required -- [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - the candidate order this backend joins - [Media Foundation decode](/quest/m2/audio-decode-mediafoundation.md) - the round-trip regression decodes through it ## Related diff --git a/rs/libmoq/src/audio.rs b/rs/libmoq/src/audio.rs index d389abc0fe..21d3b1c225 100644 --- a/rs/libmoq/src/audio.rs +++ b/rs/libmoq/src/audio.rs @@ -79,7 +79,8 @@ pub struct moq_audio_encoder_input { #[repr(C)] #[allow(non_camel_case_types)] pub struct moq_audio_encoder_output { - /// Codec id, UTF-8 (currently only "opus"). + /// Codec id, UTF-8: "opus", "pcm", or "aac". AAC encodes through the + /// platform's encoder, so a host without one refuses it. pub codec: *const c_char, pub codec_len: usize, /// 0 = derive from input. @@ -89,8 +90,8 @@ pub struct moq_audio_encoder_output { /// 0 = libopus default. pub bitrate: u32, /// Encoded frame duration in microseconds. Opus accepts exactly - /// 2500/5000/10000/20000/40000/60000 us. 0 = the 20 ms default, which - /// matches the JS publish path. + /// 2500/5000/10000/20000/40000/60000 us. 0 = the codec's default: 20 ms for + /// Opus, which matches the JS publish path, and 1024 samples for AAC. pub frame_duration_us: u32, } diff --git a/rs/moq-audio/Cargo.toml b/rs/moq-audio/Cargo.toml index a270e8dbde..5980b1bc2c 100644 --- a/rs/moq-audio/Cargo.toml +++ b/rs/moq-audio/Cargo.toml @@ -26,7 +26,7 @@ default = ["aac"] # AAC-LC decode via symphonia. Pure Rust, so it costs no toolchain: the trade # this crate already refused for Opus. Only a subscriber to an ingest-sourced # broadcast needs it (RTMP, SRT/TS, fmp4, and gstreamer all publish AAC, while -# everything this crate encodes is Opus or PCM), so a publish-only build can +# this crate encodes in software is Opus or PCM), so a publish-only build can # drop it and the two crates it pulls. aac = ["dep:symphonia-codec-aac", "dep:symphonia-core"] # Device capture. Microphones go through cpal (pure-Rust: CoreAudio / WASAPI / diff --git a/rs/moq-audio/src/aac.rs b/rs/moq-audio/src/aac.rs index 7c8b948233..52fb66e3dd 100644 --- a/rs/moq-audio/src/aac.rs +++ b/rs/moq-audio/src/aac.rs @@ -1,7 +1,7 @@ //! AAC constraints, the sibling of the `opus` and `pcm` modules. //! -//! Only the decode side exists: there is no Rust AAC encoder, so this crate -//! publishes Opus or PCM and reads AAC that a gateway produced. +//! The decode side. There is no Rust AAC encoder, so `encode` produces AAC only +//! through a platform backend and otherwise reads AAC that a gateway produced. use crate::Error; diff --git a/rs/moq-audio/src/encode/backend/libopus.rs b/rs/moq-audio/src/encode/backend/libopus.rs new file mode 100644 index 0000000000..118c701be4 --- /dev/null +++ b/rs/moq-audio/src/encode/backend/libopus.rs @@ -0,0 +1,207 @@ +//! Opus through libopus 1.3.1, via [`unsafe_libopus`]. + +use bytes::Bytes; +use unsafe_libopus::{ + OPUS_APPLICATION_AUDIO, OPUS_GET_BITRATE_REQUEST, OPUS_GET_LOOKAHEAD_REQUEST, OPUS_OK, OPUS_RESET_STATE, + OPUS_SET_BITRATE_REQUEST, OPUS_SET_DTX_REQUEST, OpusEncoder, opus_encode_float, opus_encoder_create, + opus_encoder_ctl_impl, opus_encoder_destroy, varargs, +}; + +use super::Backend; +use crate::encode::{Encoded, Settings}; +use crate::{Error, opus}; + +pub(super) const NAME: &str = "libopus"; + +/// libopus packet size ceiling per RFC 6716 §3.4. +const MAX_PACKET_BYTES: usize = 4_000; + +pub(super) struct Libopus { + inner: *mut OpusEncoder, + scratch: Vec, + sample_rate: u32, + channels: u32, + frame_size: usize, + bitrate: u64, + lookahead: usize, +} + +// SAFETY: OpusEncoder is heap-allocated state owned exclusively by this +// struct; libopus encoder methods take a single &mut, so a unique owner is +// allowed to move it across threads. +unsafe impl Send for Libopus {} + +impl Libopus { + /// Opens at the settings' rate and layout, which the front end has already + /// checked against what Opus codes. + pub(super) fn open(settings: &Settings) -> Result, Error> { + Ok(Box::new(Self::new(settings)?)) + } + + fn new(settings: &Settings) -> Result { + let sample_rate = settings.sample_rate; + let channels = settings.layout.channels(); + let frame_size = opus::frame_size(sample_rate, settings.frame_duration)?; + + let mut err = 0i32; + // SAFETY: out-pointer `err` is valid; inner is checked for null below. + let inner = unsafe { + opus_encoder_create( + sample_rate as i32, + opus::validate_channels(channels)?, + OPUS_APPLICATION_AUDIO, + &mut err, + ) + }; + if err != OPUS_OK || inner.is_null() { + return Err(opus::error(err, "opus_encoder_create")); + } + + // Owned from here, so an early return below destroys it. + let mut backend = Self { + inner, + scratch: vec![0u8; MAX_PACKET_BYTES], + sample_rate, + channels, + frame_size, + bitrate: 0, + lookahead: 0, + }; + + if let Some(bitrate) = settings.bitrate { + backend.set_rate(bitrate.as_bps())?; + } + backend.set_ctl(OPUS_SET_DTX_REQUEST, i32::from(settings.dtx), "OPUS_SET_DTX")?; + + let bitrate = backend.get_ctl(OPUS_GET_BITRATE_REQUEST, "OPUS_GET_BITRATE")?; + backend.bitrate = u64::try_from(bitrate) + .map_err(|_| Error::Unsupported(format!("Opus reported negative bitrate {bitrate}")))?; + let lookahead = backend.get_ctl(OPUS_GET_LOOKAHEAD_REQUEST, "OPUS_GET_LOOKAHEAD")?; + backend.lookahead = usize::try_from(lookahead) + .map_err(|_| Error::Unsupported(format!("Opus reported negative lookahead {lookahead}")))?; + + Ok(backend) + } + + /// Refuse rates libopus would silently clamp, then apply the rest. + fn set_rate(&mut self, bitrate: u64) -> Result<(), Error> { + let (channels, frame_size) = (self.channels, self.frame_size); + let max = 300_000 * channels as u64; + let min = opus::bitrate_floor(self.sample_rate, frame_size).max(500); + if !(min..=max).contains(&bitrate) { + return Err(Error::Unsupported(format!( + "Opus bitrate must be between {min} and {max} bits per second for {channels} channel(s) at {frame_size} samples, got {bitrate}" + ))); + } + self.set_ctl(OPUS_SET_BITRATE_REQUEST, bitrate as i32, "OPUS_SET_BITRATE") + } + + fn set_ctl(&mut self, request: i32, value: i32, name: &'static str) -> Result<(), Error> { + // SAFETY: `inner` owns a live encoder and each request here expects one i32. + let rc = unsafe { opus_encoder_ctl_impl(self.inner, request, varargs![value]) }; + if rc != OPUS_OK { + return Err(opus::error(rc, name)); + } + Ok(()) + } + + fn get_ctl(&self, request: i32, name: &'static str) -> Result { + let mut value = 0; + // SAFETY: `inner` owns a live encoder and each request here expects one + // valid mutable i32 output. + let rc = unsafe { opus_encoder_ctl_impl(self.inner, request, varargs![&mut value]) }; + if rc != OPUS_OK { + return Err(opus::error(rc, name)); + } + Ok(value) + } +} + +impl Backend for Libopus { + fn encode(&mut self, pcm: &[f32]) -> Result { + // SAFETY: `inner` owns a live OpusEncoder; pcm and scratch slices are + // bounded by the lengths we pass, and the front end sized `pcm` to one frame. + let n = unsafe { + opus_encode_float( + self.inner, + pcm.as_ptr(), + self.frame_size as i32, + self.scratch.as_mut_ptr(), + self.scratch.len() as i32, + ) + }; + if n < 0 { + return Err(opus::error(n, "opus_encode_float")); + } + let payload = Bytes::copy_from_slice(&self.scratch[..n as usize]); + let activity = opus::activity(&payload, false); + Ok(Encoded { payload, activity }) + } + + fn reset(&mut self) { + // SAFETY: `inner` owns a live encoder and OPUS_RESET_STATE takes no arguments. + let rc = unsafe { opus_encoder_ctl_impl(self.inner, OPUS_RESET_STATE, varargs![]) }; + debug_assert_eq!(rc, OPUS_OK, "OPUS_RESET_STATE failed with {rc}"); + } + + fn set_bitrate(&mut self, bitrate: u64) -> Result<(), Error> { + if bitrate != self.bitrate { + self.set_rate(bitrate)?; + self.bitrate = bitrate; + } + Ok(()) + } + + fn bitrate(&self) -> u64 { + self.bitrate + } + + fn delay(&self) -> usize { + self.lookahead + } + + fn name(&self) -> &str { + NAME + } +} + +impl Drop for Libopus { + fn drop(&mut self) { + // SAFETY: `inner` is a live OpusEncoder that nothing else aliases. + unsafe { opus_encoder_destroy(self.inner) }; + } +} + +#[cfg(test)] +mod tests { + use unsafe_libopus::OPUS_GET_DTX_REQUEST; + + use super::*; + + #[test] + fn runtime_bitrate_reaches_libopus() { + let mut backend = Libopus::new(&Settings { + bitrate: Some(moq_net::bandwidth::Rate::from_bps(64_000)), + ..Settings::default() + }) + .unwrap(); + + backend.set_bitrate(32_000).unwrap(); + assert_eq!(backend.bitrate(), 32_000); + assert_eq!( + backend.get_ctl(OPUS_GET_BITRATE_REQUEST, "OPUS_GET_BITRATE").unwrap(), + 32_000 + ); + } + + #[test] + fn applies_dtx_control() { + let backend = Libopus::new(&Settings { + dtx: true, + ..Settings::default() + }) + .unwrap(); + + assert_eq!(backend.get_ctl(OPUS_GET_DTX_REQUEST, "OPUS_GET_DTX").unwrap(), 1); + } +} diff --git a/rs/moq-audio/src/encode/backend/mod.rs b/rs/moq-audio/src/encode/backend/mod.rs new file mode 100644 index 0000000000..1879d49f37 --- /dev/null +++ b/rs/moq-audio/src/encode/backend/mod.rs @@ -0,0 +1,323 @@ +//! Pluggable audio encoder backends. +//! +//! The mirror of the decode backends. [`Backend`] is the seam between the codec +//! and the [`Encoder`](super::Encoder) front end, which owns what every backend +//! of a codec shares: validating [`Settings`](super::Settings) against the +//! codec, framing the input, draining the startup delay at the end, and the +//! catalog entry, whose description is synthesized from the settings rather than +//! read back from the backend. +//! +//! [`open`] tries the platform encoders before the software ones, skipping any +//! that does not emit the codec, and refuses when none opens. The software tier +//! is libopus for Opus and a passthrough for PCM. AAC has no software encoder, +//! so it is only as available as the platform's, and no platform encoder is +//! wired in yet. + +use super::Encoded; +use super::encoder::{Codec, Kind, Settings}; +use crate::Error; + +mod libopus; +mod pcm; + +#[cfg(test)] +pub(crate) mod stub; + +/// An opened encoder: one frame of interleaved `f32` PCM in, one packet out. +/// +/// Input arrives at the settings' rate, in the crate's canonical channel order +/// for the settings' layout; a codec with another native order reorders it here. +/// +/// One packet per frame is part of the contract: the producer stamps packets by +/// counting frames. A codec that pipelines output (MediaCodec) needs a `flush` +/// and a zero-or-more return, which changes `Encoder::encode` too, so that lands +/// with the first backend that needs it rather than as an always-empty method. +pub(crate) trait Backend: Send { + /// Encode exactly one frame of the codec's frame size. + fn encode(&mut self, pcm: &[f32]) -> Result; + + /// Drop codec history so the next frame codes as if it were the first. + fn reset(&mut self); + + /// Retune the live encoder to `bitrate` bits per second, a no-op at the current + /// rate. + /// + /// No default: a backend that can't change rate mid-stream refuses with + /// [`Error::Unsupported`] and keeps its opening rate, rather than inheriting a + /// silent no-op that ignores congestion. + fn set_bitrate(&mut self, bitrate: u64) -> Result<(), Error>; + + /// The current target bitrate in bits per second, as the codec resolved it. + fn bitrate(&self) -> u64; + + /// Frames of codec priming at the start of the decoded stream, at the codec + /// rate: Opus lookahead, AAC encoder delay. + fn delay(&self) -> usize; + + /// The stable lowercase name [`Kind::Named`] selects this backend by. + fn name(&self) -> &str; +} + +/// A backend constructor: its name, the codecs it emits, and an opener. +struct Candidate { + name: &'static str, + codecs: &'static [Codec], + open: fn(&Settings) -> Result, Error>, +} + +/// Operating-system encoders, in priority order. +const PLATFORM: &[Candidate] = &[]; + +const SOFTWARE: &[Candidate] = &[ + Candidate { + name: libopus::NAME, + codecs: &[Codec::Opus], + open: libopus::Libopus::open, + }, + Candidate { + name: pcm::NAME, + codecs: &[Codec::Pcm], + open: pcm::Pcm::open, + }, +]; + +/// Test-only backends, in neither tier so `Auto` and `Software` never pick one: +/// they exist to be asked for by name. +#[cfg(test)] +const NAMED_ONLY: &[Candidate] = &[Candidate { + name: stub::NAME, + codecs: &[Codec::Aac], + open: stub::Stub::open, +}]; + +#[cfg(not(test))] +const NAMED_ONLY: &[Candidate] = &[]; + +/// Open the first backend that emits the codec and accepts the settings. +pub(crate) fn open(settings: &Settings) -> Result, Error> { + select(settings, candidates(&settings.kind, PLATFORM, SOFTWARE)) +} + +/// The candidates `kind` allows, in the order to try them. +/// +/// Takes the tiers as arguments so a test can supply stubs instead of whatever +/// this host compiles in. +fn candidates<'a>(kind: &Kind, platform: &'a [Candidate], software: &'a [Candidate]) -> Vec<&'a Candidate> { + match kind { + Kind::Auto => platform.iter().chain(software).collect(), + Kind::Software => software.iter().collect(), + Kind::Named(name) => platform + .iter() + .chain(software) + .chain(NAMED_ONLY) + .filter(|c| c.name == name) + .collect(), + } +} + +fn select(settings: &Settings, candidates: Vec<&Candidate>) -> Result, Error> { + let codec = settings.codec; + let mut refused = Vec::new(); + + for candidate in candidates { + if !candidate.codecs.contains(&codec) { + continue; + } + match (candidate.open)(settings) { + Ok(backend) => return Ok(backend), + Err(err) => refused.push((candidate.name, err)), + } + } + + // One refusal is the whole answer, so keep its variant. + if refused.len() == 1 { + let (_, err) = refused.remove(0); + return Err(err); + } + if !refused.is_empty() { + let reasons: Vec = refused.iter().map(|(name, err)| format!("{name}: {err}")).collect(); + return Err(Error::Unsupported(reasons.join(", "))); + } + + let available: Vec<&str> = PLATFORM + .iter() + .chain(SOFTWARE) + .filter(|c| c.codecs.contains(&codec)) + .map(|c| c.name) + .collect(); + let available = match available.is_empty() { + true => "none".to_owned(), + false => available.join(", "), + }; + Err(Error::Unsupported(match &settings.kind { + Kind::Named(name) => format!("no audio encoder named {name:?} for {codec} (this build has: {available})"), + Kind::Software => format!("no software {codec} audio encoder (this build has: {available})"), + Kind::Auto => format!("no {codec} audio encoder (this build has: {available})"), + })) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Opens anything it advertises and reports which candidate it came from. + struct Fake(&'static str); + + impl Backend for Fake { + fn encode(&mut self, _pcm: &[f32]) -> Result { + Ok(Encoded::new(bytes::Bytes::new())) + } + + fn reset(&mut self) {} + + fn set_bitrate(&mut self, _bitrate: u64) -> Result<(), Error> { + Ok(()) + } + + fn bitrate(&self) -> u64 { + 0 + } + + fn delay(&self) -> usize { + 0 + } + + fn name(&self) -> &str { + self.0 + } + } + + const PLATFORM_STUB: Candidate = Candidate { + name: "platform", + codecs: &[Codec::Aac], + open: |_| Ok(Box::new(Fake("platform"))), + }; + + /// Compiled in but refusing the settings, like a platform encoder asked for a + /// layout its framework does not open. + const REFUSING: Candidate = Candidate { + name: "refusing", + codecs: &[Codec::Aac], + open: |_| Err(Error::Unsupported("not these settings".into())), + }; + + const SOFTWARE_STUB: Candidate = Candidate { + name: "software", + codecs: &[Codec::Aac], + open: |_| Ok(Box::new(Fake("software"))), + }; + + /// Emits nothing but PCM, so an AAC request never reaches its opener. + const PCM_ONLY: Candidate = Candidate { + name: "pcm-only", + codecs: &[Codec::Pcm], + open: |_| panic!("opened for a codec it does not emit"), + }; + + fn aac(kind: Kind) -> Settings { + Settings { + kind, + ..Settings::from_input(Codec::Aac, &Default::default()) + } + } + + fn pick(kind: Kind, platform: &[Candidate], software: &[Candidate]) -> Result { + let settings = aac(kind); + let backend = select(&settings, candidates(&settings.kind, platform, software))?; + Ok(backend.name().to_owned()) + } + + #[test] + fn auto_prefers_platform() { + let name = pick(Kind::Auto, &[PCM_ONLY, PLATFORM_STUB], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "platform"); + } + + #[test] + fn auto_falls_back_to_software() { + let name = pick(Kind::Auto, &[REFUSING], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "software"); + } + + #[test] + fn software_skips_platform() { + let name = pick(Kind::Software, &[PLATFORM_STUB], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "software"); + } + + #[test] + fn named_forces_one() { + let name = pick( + Kind::Named("software".into()), + &[PLATFORM_STUB], + &[PCM_ONLY, SOFTWARE_STUB], + ) + .unwrap(); + assert_eq!(name, "software"); + } + + /// A named backend that refuses the settings is the answer: nothing else is tried. + #[test] + fn named_refusal_does_not_fall_back() { + let err = pick(Kind::Named("refusing".into()), &[REFUSING], &[SOFTWARE_STUB]).unwrap_err(); + assert!(err.to_string().contains("not these settings"), "{err}"); + } + + #[test] + fn every_refusal_is_reported() { + const ALSO_REFUSING: Candidate = Candidate { + name: "also-refusing", + ..REFUSING + }; + + let err = pick(Kind::Auto, &[REFUSING], &[ALSO_REFUSING]).unwrap_err(); + let message = err.to_string(); + assert!( + message.contains("refusing: ") && message.contains("also-refusing: "), + "{message}" + ); + } + + /// No AAC encoder is wired in outside the test stub, so `Auto` refuses at + /// construction and says there is nothing to fall back to. A platform backend + /// gates this to the hosts without one. + #[test] + fn aac_without_a_platform_encoder_is_refused() { + let err = open(&aac(Kind::Auto)).err().expect("no AAC encoder on this host"); + let message = err.to_string(); + assert!(message.contains("aac") && message.contains("none"), "{message}"); + } + + /// An unknown name says what this build has for the codec instead. + #[test] + fn unknown_name_lists_the_alternatives() { + let settings = Settings { + kind: Kind::Named("opus".into()), + ..Settings::default() + }; + let message = open(&settings) + .err() + .expect("no backend is named after its codec") + .to_string(); + assert!( + message.contains("\"opus\"") && message.contains(libopus::NAME), + "{message}" + ); + } + + #[test] + fn software_backends_open_by_name() { + let settings = Settings { + kind: Kind::Named(libopus::NAME.into()), + ..Settings::default() + }; + assert_eq!(open(&settings).unwrap().name(), libopus::NAME); + + let settings = Settings { + codec: Codec::Pcm, + kind: Kind::Named(pcm::NAME.into()), + ..Settings::default() + }; + assert_eq!(open(&settings).unwrap().name(), pcm::NAME); + } +} diff --git a/rs/moq-audio/src/encode/backend/pcm.rs b/rs/moq-audio/src/encode/backend/pcm.rs new file mode 100644 index 0000000000..04ff1f16e2 --- /dev/null +++ b/rs/moq-audio/src/encode/backend/pcm.rs @@ -0,0 +1,46 @@ +//! Uncompressed little-endian `f32` PCM, which needs no codec at all. + +use super::Backend; +use crate::encode::{Encoded, Settings}; +use crate::{Error, pcm}; + +pub(super) const NAME: &str = "pcm"; + +pub(super) struct Pcm { + bitrate: u64, +} + +impl Pcm { + pub(super) fn open(settings: &Settings) -> Result, Error> { + let bitrate = pcm::bitrate(settings.sample_rate, settings.layout.channels())?; + Ok(Box::new(Self { bitrate })) + } +} + +impl Backend for Pcm { + fn encode(&mut self, pcm: &[f32]) -> Result { + let mut payload = Vec::with_capacity(std::mem::size_of_val(pcm)); + for sample in pcm { + payload.extend_from_slice(&sample.to_le_bytes()); + } + Ok(Encoded::new(payload.into())) + } + + fn reset(&mut self) {} + + fn set_bitrate(&mut self, _bitrate: u64) -> Result<(), Error> { + Err(Error::Unsupported("pcm bitrate is fixed".into())) + } + + fn bitrate(&self) -> u64 { + self.bitrate + } + + fn delay(&self) -> usize { + 0 + } + + fn name(&self) -> &str { + NAME + } +} diff --git a/rs/moq-audio/src/encode/backend/stub.rs b/rs/moq-audio/src/encode/backend/stub.rs new file mode 100644 index 0000000000..c581332c15 --- /dev/null +++ b/rs/moq-audio/src/encode/backend/stub.rs @@ -0,0 +1,59 @@ +//! A stand-in AAC encoder, so the AAC front end is testable on a host with no +//! platform encoder. Selectable only by name, and only in tests. + +use bytes::Bytes; + +use super::Backend; +use crate::Error; +use crate::encode::{Encoded, Settings}; + +pub(crate) const NAME: &str = "stub"; + +/// The AudioToolbox AAC-LC encoder delay, which is what a real backend reports. +pub(crate) const DELAY: usize = 2112; + +/// Emits each frame's index as its payload, and cannot retune. +pub(crate) struct Stub { + bitrate: u64, + frames: u64, +} + +impl Stub { + pub(super) fn open(settings: &Settings) -> Result, Error> { + Ok(Box::new(Self { + bitrate: settings.bitrate.map_or(128_000, |rate| rate.as_bps()), + frames: 0, + })) + } +} + +impl Backend for Stub { + fn encode(&mut self, _pcm: &[f32]) -> Result { + let payload = Bytes::copy_from_slice(&self.frames.to_be_bytes()); + self.frames += 1; + Ok(Encoded::new(payload)) + } + + fn reset(&mut self) { + self.frames = 0; + } + + fn set_bitrate(&mut self, bitrate: u64) -> Result<(), Error> { + match bitrate == self.bitrate { + true => Ok(()), + false => Err(Error::Unsupported("the stub cannot change rate mid-stream".into())), + } + } + + fn bitrate(&self) -> u64 { + self.bitrate + } + + fn delay(&self) -> usize { + DELAY + } + + fn name(&self) -> &str { + NAME + } +} diff --git a/rs/moq-audio/src/encode/encoder.rs b/rs/moq-audio/src/encode/encoder.rs index 90479471db..5f908c8b68 100644 --- a/rs/moq-audio/src/encode/encoder.rs +++ b/rs/moq-audio/src/encode/encoder.rs @@ -1,26 +1,29 @@ //! Audio encoder front end. //! -//! [`Encoder`] dispatches over the closed [`Codec`] set. Opus wraps libopus -//! 1.3.1 via [`unsafe_libopus`], while PCM serializes interleaved `f32` samples -//! directly. +//! [`Encoder`] checks [`Settings`] against the codec, opens a +//! [`Backend`](super::backend::Backend) for it, and owns what every backend of a +//! codec shares: framing, the terminal drain, and the catalog entry. use std::str::FromStr; use std::time::Duration; use bytes::Bytes; -use unsafe_libopus::{ - OPUS_APPLICATION_AUDIO, OPUS_GET_BITRATE_REQUEST, OPUS_GET_LOOKAHEAD_REQUEST, OPUS_OK, OPUS_RESET_STATE, - OPUS_SET_BITRATE_REQUEST, OPUS_SET_DTX_REQUEST, OpusEncoder, opus_encode_float, opus_encoder_create, - opus_encoder_ctl_impl, opus_encoder_destroy, varargs, -}; use super::Encoded; +use super::backend::{self, Backend}; use crate::opus; use crate::pcm; use crate::{Error, Format, Layout}; -/// libopus packet size ceiling per RFC 6716 §3.4. -const MAX_PACKET_BYTES: usize = 4_000; +/// Samples per channel in one AAC-LC frame. +const AAC_FRAME_SIZE: usize = 1024; + +/// The audioObjectType of AAC-LC (ISO 14496-3 Table 1.17), `mp4a.40.2`. +const AAC_LC: u8 = 2; + +/// The widest sample rate an AudioSpecificConfig can name: the escape from the +/// frequency table is a 24-bit field. +const AAC_MAX_SAMPLE_RATE: u32 = 0xFF_FFFF; /// Output audio codec. `#[non_exhaustive]` so new codecs can be added without /// breaking external `match`es. @@ -32,6 +35,9 @@ pub enum Codec { Opus, /// Uncompressed interleaved little-endian IEEE-754 binary32 PCM. Pcm, + /// AAC-LC (`mp4a.40.2`), through the platform's encoder. A host without one + /// refuses it at construction. + Aac, } impl Codec { @@ -41,6 +47,7 @@ impl Codec { match self { Self::Opus => "opus", Self::Pcm => "pcm", + Self::Aac => "aac", } } } @@ -58,11 +65,25 @@ impl FromStr for Codec { match s { "opus" => Ok(Self::Opus), "pcm" => Ok(Self::Pcm), + "aac" => Ok(Self::Aac), other => Err(Error::Unsupported(format!("unknown codec: {other}"))), } } } +/// Encoder backend selection. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +#[non_exhaustive] +pub enum Kind { + /// Prefer a platform encoder, falling back to software. + #[default] + Auto, + /// Require a software backend. + Software, + /// Require a backend by its stable lowercase name: `"libopus"` or `"pcm"`. + Named(String), +} + /// PCM supplied to [`Producer::write`](super::Producer::write). #[derive(Clone, Debug)] #[non_exhaustive] @@ -101,9 +122,12 @@ pub struct Settings { /// Sample rate accepted by the codec. pub sample_rate: u32, /// Layout accepted by the codec. + /// + /// AAC takes the layouts its channelConfiguration names: mono, stereo, 3.0, + /// 4.0, 5.0, 5.1, and 7.1. pub layout: Layout, - /// Bitrate in bits per second. `None` lets Opus pick. PCM requires `None` - /// because its bitrate is fixed by the sample rate and channel count. + /// Bitrate in bits per second. `None` lets the codec pick. PCM requires + /// `None` because its bitrate is fixed by the sample rate and channel count. /// /// Rates too low for Opus to code anything at the chosen /// [`frame_duration`](Self::frame_duration) are rejected. The floor is 1200 @@ -113,8 +137,13 @@ pub struct Settings { /// Enable Opus discontinuous transmission during silence. pub dtx: bool, /// Encoded frame duration. Opus accepts 2.5 / 5 / 10 / 20 / 40 / 60 ms. - /// PCM accepts any duration containing a whole number of samples. + /// PCM accepts any duration containing a whole number of samples. AAC frames + /// are 1024 samples, so it accepts the duration that rounds to that at + /// [`sample_rate`](Self::sample_rate), which [`from_input`](Self::from_input) + /// fills in. pub frame_duration: Duration, + /// Which encoder implementation to use. + pub kind: Kind, } impl Settings { @@ -127,6 +156,7 @@ impl Settings { bitrate: None, dtx: false, frame_duration: Duration::from_millis(20), + kind: Kind::Auto, } } @@ -134,13 +164,72 @@ impl Settings { pub fn from_input(codec: Codec, input: &Input) -> Self { let sample_rate = match codec { Codec::Opus => opus::pick_rate(input.sample_rate), - Codec::Pcm => input.sample_rate, + Codec::Pcm | Codec::Aac => input.sample_rate, + }; + let defaults = Self::new(sample_rate, input.layout); + let frame_duration = match codec { + Codec::Aac => aac_frame_duration(sample_rate), + Codec::Opus | Codec::Pcm => defaults.frame_duration, }; Self { codec, - sample_rate, - layout: input.layout, - ..Self::default() + frame_duration, + ..defaults + } + } + + /// Check the settings against the codec, returning its frame size. + /// + /// Codec rules live here rather than in a backend, so every backend of a + /// codec refuses the same settings. + fn frame_size(&self) -> Result { + self.layout.validate()?; + let (rate, channels) = (self.sample_rate, self.layout.channels()); + + match self.codec { + Codec::Opus => { + opus::validate_rate(rate)?; + if !matches!(self.layout, Layout::Mono | Layout::Stereo) { + return Err(Error::Unsupported("opus requires a named mono or stereo layout".into())); + } + opus::frame_size(rate, self.frame_duration) + } + Codec::Pcm => { + if self.bitrate.is_some() { + return Err(Error::Unsupported( + "pcm bitrate is fixed; leave Settings::bitrate unset".into(), + )); + } + if self.dtx { + return Err(Error::Unsupported( + "pcm does not support discontinuous transmission".into(), + )); + } + if rate == 0 { + return Err(Error::Unsupported("pcm sample rate must be greater than zero".into())); + } + let frame_size = pcm::frame_size(rate, self.frame_duration)?; + pcm::frame_bytes(frame_size, channels)?; + pcm::bitrate(rate, channels)?; + Ok(frame_size) + } + Codec::Aac => { + if self.dtx { + return Err(Error::Unsupported( + "aac does not support discontinuous transmission".into(), + )); + } + aac_config(self)?; + let frames = (self.frame_duration.as_nanos() * u128::from(rate) + 500_000_000) / 1_000_000_000; + if frames != AAC_FRAME_SIZE as u128 { + return Err(Error::Unsupported(format!( + "aac frames are {AAC_FRAME_SIZE} samples, {:?} at {rate} Hz (got {:?})", + aac_frame_duration(rate), + self.frame_duration + ))); + } + Ok(AAC_FRAME_SIZE) + } } } } @@ -151,6 +240,52 @@ impl Default for Settings { } } +/// One AAC frame at `sample_rate`, to the nearest nanosecond. +fn aac_frame_duration(sample_rate: u32) -> Duration { + if sample_rate == 0 { + return Duration::ZERO; + } + let rate = u64::from(sample_rate); + Duration::from_nanos((AAC_FRAME_SIZE as u64 * 1_000_000_000 + rate / 2) / rate) +} + +/// The AudioSpecificConfig fields for AAC-LC at the settings' rate and layout. +/// +/// Only layouts with a channelConfiguration are accepted, since synthesizing +/// one from a bare count would mislabel the rest: config 3 is 3.0 where the +/// count's default layout is 2.1, and 6.1 has no config the encoder writes. +/// 7.1 takes config 7, the one every decoder reads as eight channels. +fn aac_config(settings: &Settings) -> Result { + let layout = settings.layout; + if !matches!( + layout, + Layout::Mono + | Layout::Stereo + | Layout::ThreePointZero + | Layout::FourPointZero + | Layout::FivePointZero + | Layout::FivePointOne + | Layout::SevenPointOne + ) { + return Err(Error::Unsupported(format!( + "aac has no channelConfiguration for {layout:?}; use mono, stereo, 3.0, 4.0, 5.0, 5.1, or 7.1" + ))); + } + + let sample_rate = settings.sample_rate; + if !(1..=AAC_MAX_SAMPLE_RATE).contains(&sample_rate) { + return Err(Error::Unsupported(format!( + "aac sample rate must be between 1 and {AAC_MAX_SAMPLE_RATE} Hz (got {sample_rate})" + ))); + } + + Ok(moq_mux::codec::aac::Config { + profile: AAC_LC, + sample_rate, + channel_count: layout.channels(), + }) +} + /// Audio encoder over codec-sized interleaved `f32` PCM. /// /// Build one with [`Encoder::new`], feed full PCM frames via @@ -158,38 +293,16 @@ impl Default for Settings { /// [`finish`](Self::finish). Publish every packet either call returns and apply /// the terminal [`Finish::discard_padding`] when the container supports it. pub struct Encoder { - backend: Backend, + backend: Box, settings: Settings, - /// Codec sample rate. - codec_rate: u32, - /// Codec channel count. - codec_channels: u32, - /// Current libopus target bitrate. - bitrate: u64, - /// Encoder lookahead expressed in the OpusHead 48 kHz timebase. - pre_skip: u16, - /// Encoder lookahead in codec-rate frames. - lookahead: usize, frame_size: usize, + /// The catalog description, synthesized from the settings at construction so + /// the rendition can be registered before the first packet exists. + description: Option, /// Whether input has reached the codec, since a fresh encoder owes no drain. started: bool, } -enum Backend { - Opus(Opus), - Pcm, -} - -struct Opus { - inner: *mut OpusEncoder, - scratch: Vec, -} - -// SAFETY: OpusEncoder is heap-allocated state owned exclusively by this -// struct; libopus encoder methods take a single &mut, so a unique owner is -// allowed to move it across threads. -unsafe impl Send for Opus {} - /// Packets emitted by [`Encoder::finish`] and the decoded padding at their end. pub struct Finish { packets: Vec, @@ -214,158 +327,40 @@ impl Finish { } impl Encoder { - /// Open an encoder for `settings`. + /// Open an encoder for `settings`, refusing a codec no backend on this host + /// encodes. pub fn new(settings: &Settings) -> Result { - settings.layout.validate()?; - match settings.codec { - Codec::Opus => Self::new_opus(settings.clone()), - Codec::Pcm => Self::new_pcm(settings.clone()), - } - } - - fn new_opus(settings: Settings) -> Result { - let codec_rate = settings.sample_rate; - opus::validate_rate(codec_rate)?; + let frame_size = settings.frame_size()?; + let backend = backend::open(settings)?; - let codec_channels = settings.layout.channels(); - if !matches!(settings.layout, Layout::Mono | Layout::Stereo) { - return Err(Error::Unsupported("opus requires a named mono or stereo layout".into())); - } - let channels = opus::validate_channels(codec_channels)?; - - let frame_size = opus::frame_size(codec_rate, settings.frame_duration)?; - - let mut err = 0i32; - // SAFETY: out-pointer `err` is valid; inner is checked for null below. - let inner = unsafe { opus_encoder_create(codec_rate as i32, channels, OPUS_APPLICATION_AUDIO, &mut err) }; - if err != OPUS_OK || inner.is_null() { - return Err(opus::error(err, "opus_encoder_create")); - } - - let configured = Self::configure_opus(inner, &settings, codec_rate, codec_channels, frame_size); - let (bitrate, lookahead, pre_skip) = match configured { - Ok(configured) => configured, - Err(err) => { - // SAFETY: `inner` was created above and not yet handed out. - unsafe { opus_encoder_destroy(inner) }; - return Err(err); + let description = match settings.codec { + Codec::Opus => { + // OpusHead carries the lookahead in the 48 kHz timebase. + let lookahead = backend.delay() as u64; + let pre_skip = u16::try_from((lookahead * 48_000) / u64::from(settings.sample_rate)) + .map_err(|_| Error::Unsupported(format!("Opus lookahead {lookahead} does not fit in OpusHead")))?; + let head = moq_mux::codec::opus::Config::new(settings.sample_rate, settings.layout.channels()) + .with_pre_skip(pre_skip) + .encode() + .map_err(moq_mux::Error::from)?; + Some(head) } + Codec::Aac => Some(aac_config(settings)?.encode()), + Codec::Pcm => None, }; Ok(Self { - backend: Backend::Opus(Opus { - inner, - scratch: vec![0u8; MAX_PACKET_BYTES], - }), - settings, - codec_rate, - codec_channels, - bitrate, - pre_skip, - lookahead, + backend, + settings: settings.clone(), frame_size, + description, started: false, }) } - fn new_pcm(settings: Settings) -> Result { - if settings.bitrate.is_some() { - return Err(Error::Unsupported( - "pcm bitrate is fixed; leave Settings::bitrate unset".into(), - )); - } - if settings.dtx { - return Err(Error::Unsupported( - "pcm does not support discontinuous transmission".into(), - )); - } - - let codec_rate = settings.sample_rate; - if codec_rate == 0 { - return Err(Error::Unsupported("pcm sample rate must be greater than zero".into())); - } - - let codec_channels = settings.layout.channels(); - if codec_channels == 0 { - return Err(Error::Unsupported("pcm channel count must be greater than zero".into())); - } - let frame_size = pcm::frame_size(codec_rate, settings.frame_duration)?; - pcm::frame_bytes(frame_size, codec_channels)?; - let bitrate = pcm::bitrate(codec_rate, codec_channels)?; - Ok(Self { - backend: Backend::Pcm, - settings, - codec_rate, - codec_channels, - bitrate, - pre_skip: 0, - lookahead: 0, - frame_size, - started: false, - }) - } - - fn configure_opus( - inner: *mut OpusEncoder, - settings: &Settings, - codec_rate: u32, - codec_channels: u32, - frame_size: usize, - ) -> Result<(u64, usize, u16), Error> { - if let Some(bitrate) = settings.bitrate { - Self::set_opus_bitrate(inner, codec_channels, bitrate.as_bps(), codec_rate, frame_size)?; - } - Self::set_opus_ctl(inner, OPUS_SET_DTX_REQUEST, i32::from(settings.dtx), "OPUS_SET_DTX")?; - - let bitrate = Self::get_opus_ctl(inner, OPUS_GET_BITRATE_REQUEST, "OPUS_GET_BITRATE")?; - let bitrate = u64::try_from(bitrate) - .map_err(|_| Error::Unsupported(format!("Opus reported negative bitrate {bitrate}")))?; - let lookahead = Self::get_opus_ctl(inner, OPUS_GET_LOOKAHEAD_REQUEST, "OPUS_GET_LOOKAHEAD")?; - let lookahead = u64::try_from(lookahead) - .map_err(|_| Error::Unsupported(format!("Opus reported negative lookahead {lookahead}")))?; - let pre_skip = u16::try_from((lookahead * 48_000) / codec_rate as u64) - .map_err(|_| Error::Unsupported(format!("Opus lookahead {lookahead} does not fit in OpusHead")))?; - let lookahead = usize::try_from(lookahead) - .map_err(|_| Error::Unsupported(format!("Opus lookahead {lookahead} does not fit in memory")))?; - - Ok((bitrate, lookahead, pre_skip)) - } - - fn set_opus_bitrate( - inner: *mut OpusEncoder, - channels: u32, - bitrate: u64, - codec_rate: u32, - frame_size: usize, - ) -> Result<(), Error> { - let max = 300_000 * channels as u64; - let min = opus::bitrate_floor(codec_rate, frame_size).max(500); - if !(min..=max).contains(&bitrate) { - return Err(Error::Unsupported(format!( - "Opus bitrate must be between {min} and {max} bits per second for {channels} channel(s) at {frame_size} samples, got {bitrate}" - ))); - } - Self::set_opus_ctl(inner, OPUS_SET_BITRATE_REQUEST, bitrate as i32, "OPUS_SET_BITRATE") - } - - fn set_opus_ctl(inner: *mut OpusEncoder, request: i32, value: i32, name: &'static str) -> Result<(), Error> { - // SAFETY: `inner` owns a live encoder and each request here expects one i32. - let rc = unsafe { opus_encoder_ctl_impl(inner, request, varargs![value]) }; - if rc != OPUS_OK { - return Err(opus::error(rc, name)); - } - Ok(()) - } - - fn get_opus_ctl(inner: *mut OpusEncoder, request: i32, name: &'static str) -> Result { - let mut value = 0; - // SAFETY: `inner` owns a live encoder and each request here expects one - // valid mutable i32 output. - let rc = unsafe { opus_encoder_ctl_impl(inner, request, varargs![&mut value]) }; - if rc != OPUS_OK { - return Err(opus::error(rc, name)); - } - Ok(value) + /// The encoder backend name in use, e.g. `"libopus"`. + pub fn name(&self) -> &str { + self.backend.name() } /// The encoder settings, including the latest accepted runtime bitrate. @@ -382,13 +377,13 @@ impl Encoder { /// Sample rate the codec actually runs at, which is /// [`Settings::sample_rate`]. pub fn codec_rate(&self) -> u32 { - self.codec_rate + self.settings.sample_rate } /// Channel count the codec actually runs at, which is /// [`Settings::layout`]'s channel count. pub fn codec_channels(&self) -> u32 { - self.codec_channels + self.settings.layout.channels() } /// Number of samples per channel the codec consumes per call to @@ -399,23 +394,21 @@ impl Encoder { /// Current target bitrate. pub fn bitrate(&self) -> moq_net::bandwidth::Rate { - moq_net::bandwidth::Rate::from_bps(self.bitrate) + moq_net::bandwidth::Rate::from_bps(self.backend.bitrate()) } - /// Retune the live Opus encoder to `bitrate`. + /// Retune the live encoder to `bitrate`. + /// + /// # Errors + /// + /// Returns [`Error::Unsupported`] when the codec's rate is fixed (PCM) or the + /// backend can't change it mid-stream. The encoder keeps running at its + /// opening rate, so a caller driving a control loop should stop adapting + /// rather than stop encoding. pub fn set_bitrate(&mut self, bitrate: moq_net::bandwidth::Rate) -> Result<(), Error> { - let Backend::Opus(opus) = &mut self.backend else { - return Err(Error::Unsupported("pcm bitrate is fixed".into())); - }; - if bitrate.as_bps() != self.bitrate { - Self::set_opus_bitrate( - opus.inner, - self.codec_channels, - bitrate.as_bps(), - self.codec_rate, - self.frame_size, - )?; - self.bitrate = bitrate.as_bps(); + let previous = self.backend.bitrate(); + self.backend.set_bitrate(bitrate.as_bps())?; + if bitrate.as_bps() != previous { self.settings.bitrate = Some(bitrate); } Ok(()) @@ -423,11 +416,7 @@ impl Encoder { /// Drop all codec history so a later epoch cannot emit audio from this one. pub(super) fn reset(&mut self) { - if let Backend::Opus(opus) = &mut self.backend { - // SAFETY: `inner` owns a live encoder and OPUS_RESET_STATE takes no arguments. - let rc = unsafe { opus_encoder_ctl_impl(opus.inner, OPUS_RESET_STATE, varargs![]) }; - debug_assert_eq!(rc, OPUS_OK, "OPUS_RESET_STATE failed with {rc}"); - } + self.backend.reset(); self.started = false; } @@ -436,47 +425,34 @@ impl Encoder { self.started } + /// Codec priming the catalog can't signal, in codec-rate frames, which the + /// producer folds into its timestamps instead. + /// + /// Opus declares its lookahead as OpusHead pre-skip, which the decoder trims, + /// so nothing is folded. An AudioSpecificConfig has no such field, so each + /// AAC packet is stamped that much earlier and the priming lands before the + /// first input sample rather than delaying it. + pub(super) fn folded_delay(&self) -> usize { + match self.settings.codec { + Codec::Aac => self.backend.delay(), + Codec::Opus | Codec::Pcm => 0, + } + } + /// Encode one frame of interleaved `f32` PCM at [`codec_rate`](Self::codec_rate). /// /// `pcm.len()` must equal `frame_size() * codec_channels()`. The /// [`Producer`](super::Producer) handles format conversion and resampling /// before calling this; for direct use, the caller does the same. pub fn encode(&mut self, pcm: &[f32]) -> Result { - let expected = self.frame_size * self.codec_channels as usize; + let expected = self.frame_size * self.codec_channels() as usize; if pcm.len() != expected { return Err(Error::Misaligned { got: std::mem::size_of_val(pcm), expected: expected * std::mem::size_of::(), }); } - let encoded = match &mut self.backend { - Backend::Opus(opus) => { - // SAFETY: `inner` owns a live OpusEncoder; pcm and scratch slices - // are bounded by the lengths we pass. - let n = unsafe { - opus_encode_float( - opus.inner, - pcm.as_ptr(), - self.frame_size as i32, - opus.scratch.as_mut_ptr(), - opus.scratch.len() as i32, - ) - }; - if n < 0 { - return Err(crate::opus::error(n, "opus_encode_float")); - } - let payload = Bytes::copy_from_slice(&opus.scratch[..n as usize]); - let activity = crate::opus::activity(&payload, false); - Encoded { payload, activity } - } - Backend::Pcm => { - let mut payload = Vec::with_capacity(std::mem::size_of_val(pcm)); - for sample in pcm { - payload.extend_from_slice(&sample.to_le_bytes()); - } - Encoded::new(payload.into()) - } - }; + let encoded = self.backend.encode(pcm)?; self.started = true; Ok(encoded) } @@ -496,7 +472,7 @@ impl Encoder { /// Same drain as [`finish`](Self::finish), without consuming the encoder. pub(super) fn drain(&mut self, pcm: &[f32]) -> Result { - let channels = self.codec_channels as usize; + let channels = self.codec_channels() as usize; let frame_samples = self.frame_size * channels; if pcm.len() > frame_samples || !pcm.len().is_multiple_of(channels) { return Err(Error::Misaligned { @@ -529,7 +505,8 @@ impl Encoder { }); } - let drain = self.lookahead.saturating_sub(padding); + let lookahead = self.backend.delay(); + let drain = lookahead.saturating_sub(padding); let silence = vec![0.0; frame_samples]; for _ in 0..drain.div_ceil(self.frame_size) { packets.push(self.encode(&silence)?); @@ -538,7 +515,7 @@ impl Encoder { let discard_padding = packets .len() .saturating_mul(self.frame_size) - .saturating_sub(self.lookahead) + .saturating_sub(lookahead) .saturating_sub(source_frames); Ok(Finish { @@ -549,46 +526,27 @@ impl Encoder { /// hang catalog entry describing this encoder's output stream. pub fn catalog(&self) -> hang::catalog::AudioConfig { - match self.settings.codec { - Codec::Opus => { - // `codec_channels` is validated to mono/stereo at encoder construction, - // so the OpusHead (channel mapping family 0) always encodes. - let head = moq_mux::codec::opus::Config::new(self.codec_rate, self.codec_channels) - .with_pre_skip(self.pre_skip) - .encode() - .expect("opus encoder channels validated to mono/stereo"); - - let mut config = hang::catalog::AudioConfig::new( - hang::catalog::AudioCodec::Opus, - self.codec_rate, - self.codec_channels, - ); - config.bitrate = self.settings.bitrate.map(moq_net::bandwidth::Rate::as_bps); - config.description = Some(head); - config.container = hang::catalog::Container::Legacy; - config - } - Codec::Pcm => { - let mut config = hang::catalog::AudioConfig::new( - hang::catalog::AudioCodec::Pcm, - self.codec_rate, - self.codec_channels, - ); - config.bitrate = Some( - pcm::bitrate(self.codec_rate, self.codec_channels) - .expect("pcm encoder bitrate validated at construction"), - ); - config.container = hang::catalog::Container::Legacy; - config - } - } - } -} + let (rate, channels) = (self.codec_rate(), self.codec_channels()); + let (codec, bitrate): (hang::catalog::AudioCodec, _) = match self.settings.codec { + Codec::Opus => ( + hang::catalog::AudioCodec::Opus, + self.settings.bitrate.map(moq_net::bandwidth::Rate::as_bps), + ), + Codec::Pcm => ( + hang::catalog::AudioCodec::Pcm, + Some(pcm::bitrate(rate, channels).expect("pcm encoder bitrate validated at construction")), + ), + Codec::Aac => ( + hang::catalog::AAC { profile: AAC_LC }.into(), + self.settings.bitrate.map(moq_net::bandwidth::Rate::as_bps), + ), + }; -impl Drop for Opus { - fn drop(&mut self) { - // SAFETY: `inner` is a live OpusEncoder that nothing else aliases. - unsafe { opus_encoder_destroy(self.inner) }; + let mut config = hang::catalog::AudioConfig::new(codec, rate, channels); + config.bitrate = bitrate; + config.description = self.description.clone(); + config.container = hang::catalog::Container::Legacy; + config } } @@ -609,13 +567,6 @@ mod tests { out } - fn opus_inner(encoder: &Encoder) -> *mut OpusEncoder { - let Backend::Opus(opus) = &encoder.backend else { - panic!("expected Opus encoder"); - }; - opus.inner - } - #[test] fn opus_encode_then_decode_keeps_signal_close() { let mut enc = Encoder::new(&Settings { @@ -675,7 +626,6 @@ mod tests { let desc = cfg.description.expect("OpusHead should be present"); assert_eq!(desc.len(), 19); let head = moq_mux::codec::opus::Config::parse(&mut desc.as_ref()).unwrap(); - assert_eq!(head.pre_skip, enc.pre_skip); assert_eq!(head.pre_skip, 312); } @@ -688,7 +638,7 @@ mod tests { let first = dec.decode(&enc.encode(&frame).unwrap().payload).unwrap(); assert_eq!( first.samples.len(), - (enc.frame_size() - enc.pre_skip as usize) * enc.codec_channels() as usize + (enc.frame_size() - enc.backend.delay()) * enc.codec_channels() as usize ); let second = dec.decode(&enc.encode(&frame).unwrap().payload).unwrap(); @@ -757,15 +707,6 @@ mod tests { enc.set_bitrate(moq_net::bandwidth::Rate::from_bps(32_000)).unwrap(); assert_eq!(enc.bitrate(), moq_net::bandwidth::Rate::from_bps(32_000)); assert_eq!(enc.settings().bitrate, Some(moq_net::bandwidth::Rate::from_bps(32_000))); - assert_eq!( - Encoder::get_opus_ctl( - opus_inner(&enc), - unsafe_libopus::OPUS_GET_BITRATE_REQUEST, - "OPUS_GET_BITRATE" - ) - .unwrap(), - 32_000 - ); } #[test] @@ -777,20 +718,6 @@ mod tests { assert_eq!(enc.bitrate(), original); } - #[test] - fn opus_applies_dtx_control() { - let enc = Encoder::new(&Settings { - dtx: true, - ..Settings::default() - }) - .unwrap(); - - assert_eq!( - Encoder::get_opus_ctl(opus_inner(&enc), unsafe_libopus::OPUS_GET_DTX_REQUEST, "OPUS_GET_DTX").unwrap(), - 1 - ); - } - #[test] fn codec_roundtrips_as_str() { assert_eq!(Codec::Opus.as_str(), "opus"); @@ -799,15 +726,20 @@ mod tests { assert_eq!(Codec::Pcm.as_str(), "pcm"); assert_eq!(Codec::Pcm.to_string(), "pcm"); assert_eq!("pcm".parse::().unwrap(), Codec::Pcm); - assert!("aac".parse::().is_err()); + assert_eq!(Codec::Aac.as_str(), "aac"); + assert_eq!(Codec::Aac.to_string(), "aac"); + assert_eq!("aac".parse::().unwrap(), Codec::Aac); + assert!("mp3".parse::().is_err()); } #[test] fn settings_fix_the_codec_rate() { let enc = Encoder::new(&Settings::new(24_000, Layout::Mono)).unwrap(); assert_eq!(enc.codec_rate(), 24_000); - assert_eq!(enc.catalog().sample_rate, 24_000); - assert_eq!(enc.pre_skip, 312); + let catalog = enc.catalog(); + assert_eq!(catalog.sample_rate, 24_000); + let head = moq_mux::codec::opus::Config::parse(&mut catalog.description.unwrap().as_ref()).unwrap(); + assert_eq!(head.pre_skip, 312); } #[test] @@ -906,4 +838,121 @@ mod tests { let settings = Settings::new(48_000, Layout::Discrete(2)); assert!(matches!(Encoder::new(&settings), Err(Error::Unsupported(_)))); } + + /// AAC settings routed to the test stub, since this host has no AAC encoder. + fn aac(layout: Layout) -> Settings { + Settings { + kind: Kind::Named(backend::stub::NAME.into()), + ..Settings::from_input(Codec::Aac, &Input::new(48_000, layout)) + } + } + + /// The ASC is synthesized from the settings, so it exists before any packet. + #[test] + fn aac_catalog_carries_the_synthesized_asc() { + let enc = Encoder::new(&aac(Layout::Stereo)).unwrap(); + assert_eq!(enc.name(), backend::stub::NAME); + assert_eq!(enc.frame_size(), 1024); + + let catalog = enc.catalog(); + assert_eq!(catalog.codec, hang::catalog::AAC { profile: 2 }.into()); + assert_eq!(catalog.codec.to_string(), "mp4a.40.2"); + assert_eq!(catalog.sample_rate, 48_000); + assert_eq!(catalog.channel_count, 2); + assert_eq!(catalog.container, hang::catalog::Container::Legacy); + // AAC-LC (2), 48 kHz (index 3), stereo (config 2). + assert_eq!(catalog.description.as_deref(), Some(&[0x11, 0x90][..])); + } + + #[test] + fn aac_takes_the_layouts_with_a_channel_configuration() { + for (layout, config) in [ + (Layout::Mono, 1), + (Layout::Stereo, 2), + (Layout::ThreePointZero, 3), + (Layout::FourPointZero, 4), + (Layout::FivePointZero, 5), + (Layout::FivePointOne, 6), + (Layout::SevenPointOne, 7), + ] { + let catalog = Encoder::new(&aac(layout)).unwrap().catalog(); + let description = catalog.description.unwrap(); + assert_eq!(description[1] >> 3 & 0xF, config, "{layout:?}"); + assert_eq!(catalog.channel_count, layout.channels(), "{layout:?}"); + } + + for layout in [ + Layout::TwoPointOne, + Layout::Quad, + Layout::SixPointOne, + Layout::Discrete(2), + ] { + assert!( + matches!(Encoder::new(&aac(layout)), Err(Error::Unsupported(_))), + "{layout:?}" + ); + } + } + + /// AAC frames are 1024 samples however the duration is spelled. + #[test] + fn aac_frame_duration_is_the_codecs() { + assert_eq!(aac(Layout::Stereo).frame_duration, Duration::from_nanos(21_333_333)); + let settings = Settings { + frame_duration: Duration::from_micros(21_333), + ..aac(Layout::Stereo) + }; + assert_eq!(Encoder::new(&settings).unwrap().frame_size(), 1024); + + let settings = Settings { + frame_duration: Duration::from_millis(20), + ..aac(Layout::Stereo) + }; + let err = Encoder::new(&settings).err().expect("20 ms is 960 samples"); + assert!(err.to_string().contains("1024"), "{err}"); + } + + #[test] + fn aac_refuses_dtx() { + let settings = Settings { + dtx: true, + ..aac(Layout::Stereo) + }; + assert!(matches!(Encoder::new(&settings), Err(Error::Unsupported(_)))); + } + + /// A backend that can't retune keeps its opening rate. + #[test] + fn fixed_rate_backend_keeps_its_opening_rate() { + let mut enc = Encoder::new(&Settings { + bitrate: Some(moq_net::bandwidth::Rate::from_bps(96_000)), + ..aac(Layout::Stereo) + }) + .unwrap(); + + let err = enc.set_bitrate(moq_net::bandwidth::Rate::from_bps(64_000)); + assert!(matches!(err, Err(Error::Unsupported(_)))); + assert_eq!(enc.bitrate(), moq_net::bandwidth::Rate::from_bps(96_000)); + assert_eq!(enc.settings().bitrate, Some(moq_net::bandwidth::Rate::from_bps(96_000))); + assert_eq!(enc.catalog().bitrate, Some(96_000)); + } + + /// The drain pushes the encoder delay out through whole silent frames. + #[test] + fn aac_finish_drains_the_encoder_delay() { + let mut enc = Encoder::new(&aac(Layout::Mono)).unwrap(); + assert_eq!(enc.folded_delay(), backend::stub::DELAY); + enc.encode(&[0.0; 1024]).unwrap(); + + // 2112 frames of delay take three 1024-frame packets. + let finish = enc.finish(&[]).unwrap(); + assert_eq!(finish.packets().len(), 3); + assert_eq!(finish.discard_padding(), 3 * 1024 - backend::stub::DELAY); + } + + /// Opus signals its lookahead as pre-skip, so the producer folds none of it. + #[test] + fn opus_folds_no_delay() { + assert_eq!(Encoder::new(&Settings::default()).unwrap().folded_delay(), 0); + } } diff --git a/rs/moq-audio/src/encode/mod.rs b/rs/moq-audio/src/encode/mod.rs index 9c918b9744..10c3125e41 100644 --- a/rs/moq-audio/src/encode/mod.rs +++ b/rs/moq-audio/src/encode/mod.rs @@ -1,6 +1,8 @@ //! Encode raw PCM and publish it as a moq audio track. //! -//! The output codec is selected via [`Codec`]. +//! The output codec is selected via [`Codec`], and the implementation behind it +//! via [`Kind`]: a platform encoder when the host has one, software otherwise. +//! AAC has no software encoder, so a host without a platform one refuses it. //! //! Entry points, high to low level: //! - `publish_capture` captures a microphone (or system audio) and publishes @@ -15,6 +17,7 @@ //! `publish_capture` is unlinked above because it only exists with the `capture` //! feature, so a default-feature rustdoc build has nothing to link to. +mod backend; mod encoded; mod encoder; mod producer; @@ -23,7 +26,7 @@ mod producer; mod capture; pub use encoded::Encoded; -pub use encoder::{Codec, Encoder, Finish, Input, Settings}; +pub use encoder::{Codec, Encoder, Finish, Input, Kind, Settings}; pub use producer::{Options, Producer}; #[cfg(feature = "capture")] diff --git a/rs/moq-audio/src/encode/producer.rs b/rs/moq-audio/src/encode/producer.rs index 1b61545141..afb0b73722 100644 --- a/rs/moq-audio/src/encode/producer.rs +++ b/rs/moq-audio/src/encode/producer.rs @@ -366,7 +366,12 @@ impl Producer { let chunk: Vec = self.pending.drain(..frame_samples).collect(); let packet = self.encoder.encode(&chunk)?; - let timestamp = Self::timestamp(epoch_us, self.frames_produced, self.encoder.codec_rate())?; + let timestamp = Self::timestamp( + epoch_us, + self.frames_produced, + self.encoder.folded_delay(), + self.encoder.codec_rate(), + )?; self.frames_produced += self.encoder.frame_size() as u64; self.activity = packet.activity; Self::publish(&mut self.track, packet, timestamp)?; @@ -376,10 +381,18 @@ impl Producer { Ok(()) } - /// PTS of the next frame: the epoch plus the samples emitted since it. - fn timestamp(epoch_us: u64, frames_produced: u64, codec_rate: u32) -> Result { - let offset_us = (frames_produced * 1_000_000) / codec_rate as u64; - Ok(Timestamp::from_micros(epoch_us + offset_us)?) + /// PTS of the frame `frames` samples past the epoch, stamped `delay` samples + /// earlier to fold in codec priming the catalog can't signal. + /// + /// Priming that would land before a zero epoch is stamped at zero instead: it + /// decodes to the codec's warm-up rather than to input, so only its spacing is + /// lost. + fn timestamp(epoch_us: u64, frames: u64, delay: usize, codec_rate: u32) -> Result { + let frames = i128::from(frames) - delay as i128; + let offset_us = (frames * 1_000_000).div_euclid(i128::from(codec_rate)); + let micros = (i128::from(epoch_us) + offset_us).max(0); + let micros = u64::try_from(micros).map_err(|_| moq_net::TimeOverflow)?; + Ok(Timestamp::from_micros(micros)?) } fn publish( @@ -474,8 +487,10 @@ impl Producer { let codec_rate = self.encoder.codec_rate(); let channels = self.encoder.codec_channels() as usize; let source_frames = self.pending.len() / channels; - let start = Self::timestamp(epoch_us, self.frames_produced, codec_rate)?; - let end = Self::timestamp(epoch_us, self.frames_produced + source_frames as u64, codec_rate)?; + let delay = self.encoder.folded_delay(); + let start = Self::timestamp(epoch_us, self.frames_produced, delay, codec_rate)?; + // The source ends where it ends: priming only moves the packets carrying it. + let end = Self::timestamp(epoch_us, self.frames_produced + source_frames as u64, 0, codec_rate)?; let finish = self.encoder.drain(&self.pending)?; let discard_padding = finish.discard_padding(); let packets = finish.into_packets(); @@ -493,7 +508,7 @@ impl Producer { )?; } else { for packet in packets { - let timestamp = Self::timestamp(epoch_us, self.frames_produced, codec_rate)?; + let timestamp = Self::timestamp(epoch_us, self.frames_produced, delay, codec_rate)?; self.activity = packet.activity; Self::publish(&mut self.track, packet, timestamp)?; self.frames_produced += frame_size as u64; @@ -898,6 +913,51 @@ mod tests { assert_eq!(pts, vec![1_000_000]); } + /// AAC can't signal its encoder delay, so each packet is stamped that much + /// earlier and the first input sample still decodes at the epoch. + #[tokio::test] + async fn aac_folds_the_encoder_delay_into_timestamps() { + async fn pts(epoch_us: u64) -> Vec { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); + let consumer = broadcast.consume(); + + let input = Input::new(48_000, Layout::Mono); + let options = Options { + track: Some("audio".to_string()), + settings: Settings { + kind: crate::encode::Kind::Named(crate::encode::backend::stub::NAME.into()), + ..Settings::from_input(crate::encode::Codec::Aac, &input) + }, + ..Options::default() + }; + let mut producer = Producer::new(&mut broadcast, catalog, input, &options).unwrap(); + + let track = consumer + .track("audio") + .unwrap() + .subscribe(moq_net::track::Subscription::default().with_max_age(Duration::from_secs(1))) + .await + .unwrap(); + let mut reader = moq_mux::container::Consumer::new( + track, + moq_mux::container::legacy::Wire(moq_mux::container::Kind::Audio), + ); + + producer.write(&pcm_frame(&[0.1; 4 * 1024], epoch_us)).unwrap(); + let mut pts = Vec::new(); + for _ in 0..4 { + pts.push(reader.read().await.unwrap().expect("a packet").timestamp.as_micros()); + } + pts + } + + // 2112 frames of delay at 48 kHz is 44 ms, rounded down per packet. + assert_eq!(pts(1_000_000).await, vec![956_000, 977_333, 998_666, 1_020_000]); + // Priming before a zero epoch stamps at zero; the input still starts on time. + assert_eq!(pts(0).await, vec![0, 0, 0, 20_000]); + } + /// The encoder needs no correction for the resampler's own delay: it anchors /// the epoch to the first input timestamp and advances by emitted samples, /// while `Resampler::process` drops its startup silence rather than passing it diff --git a/rs/moq-audio/src/lib.rs b/rs/moq-audio/src/lib.rs index b8c7aa9242..92d1c04dff 100644 --- a/rs/moq-audio/src/lib.rs +++ b/rs/moq-audio/src/lib.rs @@ -3,8 +3,9 @@ //! Counterpart to [`moq-video`](https://crates.io/crates/moq-video) for audio //! tracks, and shaped the same way. Sits on top of [`moq_mux`] and [`hang`] and //! adds the missing piece for native callers: Rust-native Opus and uncompressed -//! PCM codecs that turn raw samples into HANG audio tracks and back, plus AAC-LC -//! on the way in, which is what the gateways (RTMP, SRT, HLS, gstreamer) publish. +//! PCM codecs that turn raw samples into HANG audio tracks and back, plus AAC-LC, +//! which is what the gateways (RTMP, SRT, HLS, gstreamer) publish. AAC decodes in +//! software and encodes only through a platform encoder. //! //! - `capture` describes an audio source (`capture::Config`) and grabs buffers //! per platform: a microphone via cpal (CoreAudio / WASAPI / ALSA) everywhere, @@ -26,7 +27,7 @@ //! - [`decode`] subscribes to an encoded track and decodes it back to PCM. //! [`decode::Consumer`] is the mirror of [`encode::Producer`]. It reads AAC-LC //! too, behind the default-on `aac` feature, since a broadcast that came in -//! through a gateway is AAC rather than one of the two codecs we encode. +//! through a gateway is AAC rather than one of the codecs we encode in software. //! - `playback` plays decoded PCM out a speaker. `playback::Engine` owns the //! output device and mixes the `playback::Sink`s registered with it, so one //! device serves every track in a call. Requires the `playback` feature, so diff --git a/rs/moq-ffi/src/audio.rs b/rs/moq-ffi/src/audio.rs index 5e8bea78ce..22c3ba4742 100644 --- a/rs/moq-ffi/src/audio.rs +++ b/rs/moq-ffi/src/audio.rs @@ -48,7 +48,7 @@ impl From for moq_audio::Format { /// Audio codec selection for the encoder. /// /// An immutable object so adding a codec later does not break callers -/// switching over a closed enum. Currently only Opus is available. +/// switching over a closed enum. #[derive(uniffi::Object)] pub struct MoqAudioCodec { inner: moq_audio::encode::Codec, @@ -63,6 +63,16 @@ impl MoqAudioCodec { inner: moq_audio::encode::Codec::Opus, }) } + + /// AAC-LC (`mp4a.40.2`) through the platform's encoder, at the input's rate + /// and layout. A host without one refuses it when the producer is built. + /// Its frames are 1024 samples, so leave `frame_duration_us` at 0. + #[uniffi::constructor] + pub fn aac() -> Arc { + Arc::new(Self { + inner: moq_audio::encode::Codec::Aac, + }) + } } impl MoqAudioCodec { @@ -96,7 +106,7 @@ pub struct MoqAudioEncoderOutput { pub bitrate: Option, /// Encoded frame duration in microseconds. Opus accepts exactly /// 2500/5000/10000/20000/40000/60000 us, and the default 20 ms matches the - /// JS publish path. + /// JS publish path. 0 takes the codec's own frame, which AAC needs. #[uniffi(default = 20000)] pub frame_duration_us: u32, } @@ -290,7 +300,16 @@ impl MoqBroadcastProducer { options.settings.layout = moq_audio::Layout::from_channels(channels)?; } options.settings.bitrate = output.bitrate.map(|bps| moq_net::bandwidth::Rate::from_bps(bps.into())); - options.settings.frame_duration = Duration::from_micros(output.frame_duration_us.into()); + if output.frame_duration_us != 0 { + options.settings.frame_duration = Duration::from_micros(output.frame_duration_us.into()); + } else if output.codec.codec() == moq_audio::encode::Codec::Aac { + // from_input sized this at the input rate. The codec rate may be the + // override above, and AAC's own frame is 1024 samples of that rate. + let mut rated = input.clone(); + rated.sample_rate = options.settings.sample_rate; + options.settings.frame_duration = + moq_audio::encode::Settings::from_input(moq_audio::encode::Codec::Aac, &rated).frame_duration; + } if let Some(bandwidth) = &bandwidth { options.bandwidth = bandwidth.allocator().clone(); } diff --git a/rs/moq-ffi/src/test.rs b/rs/moq-ffi/src/test.rs index 265dee6162..01f10f0e99 100644 --- a/rs/moq-ffi/src/test.rs +++ b/rs/moq-ffi/src/test.rs @@ -512,6 +512,60 @@ async fn raw_audio_frame_durations() { broadcast.finish().unwrap(); } +/// A frame duration of 0 takes the codec's own frame, and AAC, which encodes only +/// through a platform encoder, is refused where there is none. +#[cfg(feature = "audio")] +#[tokio::test] +async fn raw_audio_codec_default_frame() { + use crate::audio::*; + + let broadcast = MoqBroadcastProducer::new().unwrap(); + let input = || MoqAudioEncoderInput { + format: MoqAudioSampleFormat::F32, + sample_rate: 48_000, + channels: 2, + }; + let output = |codec| MoqAudioEncoderOutput { + codec, + sample_rate: None, + channels: None, + bitrate: None, + frame_duration_us: 0, + }; + + let opus = broadcast + .encode_audio("opus".into(), input(), output(MoqAudioCodec::opus()), None) + .unwrap(); + opus.finish().unwrap(); + + assert_eq!(MoqAudioCodec::aac().codec(), moq_audio::encode::Codec::Aac); + let aac = broadcast.encode_audio("aac".into(), input(), output(MoqAudioCodec::aac()), None); + let Err(MoqError::Audio(message)) = aac else { + panic!("no platform AAC encoder on this host"); + }; + assert!(message.contains("no aac audio encoder"), "{message}"); + + // 0 still means 1024 samples after an output-rate override, not the input rate. + let mut resampled = output(MoqAudioCodec::aac()); + resampled.sample_rate = Some(48_000); + let aac = broadcast.encode_audio( + "aac-rate".into(), + MoqAudioEncoderInput { + format: MoqAudioSampleFormat::F32, + sample_rate: 44_100, + channels: 2, + }, + resampled, + None, + ); + let Err(MoqError::Audio(message)) = aac else { + panic!("no platform AAC encoder on this host"); + }; + assert!(message.contains("no aac audio encoder"), "{message}"); + + broadcast.finish().unwrap(); +} + #[tokio::test] async fn raw_track_datagram_roundtrip() { let broadcast = MoqBroadcastProducer::new().unwrap(); diff --git a/swift/Sources/Moq/Aliases.swift b/swift/Sources/Moq/Aliases.swift index dcfef7b25e..c864511cff 100644 --- a/swift/Sources/Moq/Aliases.swift +++ b/swift/Sources/Moq/Aliases.swift @@ -52,7 +52,7 @@ public typealias VideoDecodedFrame = MoqVideoDecodedFrame public typealias AudioDecoderOutput = MoqFFI.MoqAudioDecoderOutput /// A raw PCM sample format, mirroring WebCodecs `AudioData.format`. public typealias AudioSampleFormat = MoqFFI.MoqAudioSampleFormat -/// Selects the audio encoder codec. Build one with `AudioCodec.opus()`. +/// Selects the audio encoder codec. Build one with `AudioCodec.opus()` or `AudioCodec.aac()`. public typealias AudioCodec = MoqFFI.MoqAudioCodec /// One raw video frame: pixels in the configured layout plus a presentation /// timestamp. diff --git a/swift/Sources/Moq/Broadcast.swift b/swift/Sources/Moq/Broadcast.swift index cf52ddba1b..85487ed175 100644 --- a/swift/Sources/Moq/Broadcast.swift +++ b/swift/Sources/Moq/Broadcast.swift @@ -273,7 +273,7 @@ public final class BroadcastProducer: Sendable { /// Open a raw-audio track. PCM written via `AudioProducer.write` is encoded /// inside the FFI boundary per `input`/`output`. Select the codec with - /// `AudioCodec.opus()` (currently the only constructor), placed in `output`. + /// `AudioCodec.opus()` or `AudioCodec.aac()`, placed in `output`. /// /// Pass `bandwidth` to reserve this track's bitrate against the session's /// allocator so a co-resident video encoder sizes itself against what is left.