diff --git a/doc/bin/cli.md b/doc/bin/cli.md index cbd4d9d03a..723b80a3e4 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -87,11 +87,11 @@ moq --connect https://relay.example.com/anon --broadcast my-stream.hang play moq ... play --delay 500ms # trade latency for a jittery link ``` -Decodes H.264, H.265, and AV1 video and Opus, PCM, and AAC-LC audio using -the platform hardware decoder where available. `--video-name` and -`--audio-name` pick a rendition. -HE-AAC signaled only in band (implicit SBR, as over MPEG-TS) plays as its -half-rate AAC-LC core. +Decodes H.264, H.265, and AV1 video using the platform hardware decoder where +available, and Opus, PCM, and AAC-LC (mono or stereo) audio in software. The +log names the decoder each track opened. `--video-name` and `--audio-name` +pick a rendition. HE-AAC signaled only in band (implicit SBR, as over MPEG-TS) +plays as its half-rate AAC-LC core. Playback runs on a clock it owns. `--delay` (default 100 ms) is how far it trails the live edge, which is both the jitter a late frame may absorb and the diff --git a/doc/lib/rs/moq-audio.md b/doc/lib/rs/moq-audio.md index 2d57b4a5bc..637343dd61 100644 --- a/doc/lib/rs/moq-audio.md +++ b/doc/lib/rs/moq-audio.md @@ -33,9 +33,20 @@ policy. Decoding likewise separates low-level `decode::Config`, PCM | `playback` | One output device mixing every track in a call, with click-free volume ramps | | `aec` | Acoustic echo cancellation (a port of WebRTC's), so a laptop with no headset doesn't feed itself back | -AAC decoding refuses HE-AAC its config declares. HE-AAC signaled only in band -(implicit SBR, as over MPEG-TS) goes undetected and plays as its half-rate -AAC-LC core. +`decode` picks a backend per track the way `moq-video` does: a platform decoder +first, then software. `decode::Config::kind` forces one (`Kind::Software`, or +`Kind::Named` with a name below), and `Decoder::name()` reports what opened. + +| Backend | Decodes | Hosts | +| --- | --- | --- | +| `libopus` | Opus, mono or stereo | all | +| `pcm` | PCM | all | +| `symphonia` | AAC-LC, mono or stereo (the default-on `aac` feature) | all | + +No platform decoder is wired in yet, so multichannel AAC and HE-AAC declared in +its config are refused at construction on every host. HE-AAC signaled only in +band plays as its half-rate LC core. Linux has no OS audio decoder, so it will +stay that way there. Highlights: diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index 78912f392c..9b3763d7f1 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -44,7 +44,6 @@ its own decode and encode quest so verification stays per host. ## Quests - [TS export PCE](/quest/m1/audio-codecs/ts-export-pce.md) - a TS export of a PCE-described AAC track writes channel_config 0 and the PCE instead of a count-derived config -- [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - `decode::backend` selects a platform decoder before symphonia, mirroring moq-video - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - mapping family 1 decodes on every host through the multistream decoder - [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - `encode::backend` and `Codec::Aac`, so a native publisher can produce AAC-LC diff --git a/quest/m1/audio-codecs/decode-audiotoolbox.md b/quest/m1/audio-codecs/decode-audiotoolbox.md index e81db0d554..cda8f73b5c 100644 --- a/quest/m1/audio-codecs/decode-audiotoolbox.md +++ b/quest/m1/audio-codecs/decode-audiotoolbox.md @@ -10,9 +10,14 @@ documents as unsupported. ## Plan An `AudioConverter` from the packetized format to interleaved `f32` at the -codec's native rate and layout, behind the decode seam as the first platform -candidate on `target_os = "macos"` and `"ios"`. `objc2-audio-toolbox` is the -binding, alongside the `objc2-core-audio-types` the crate already carries. +codec's native rate and layout, behind the decode seam +(`rs/moq-audio/src/decode/backend`) as the first platform candidate on +`target_os = "macos"` and `"ios"`. `objc2-audio-toolbox` is the binding, +alongside the `objc2-core-audio-types` the crate already carries. + +- With a platform tier in place, `Auto` falling past a refusing platform + decoder to software should warn, as moq-video's `select` does; the seam + only aggregates the refusals into its error today. - Build the `AudioStreamBasicDescription` and magic cookie from the catalog description; the converter reports the output layout, which maps to @@ -20,7 +25,8 @@ binding, alongside the `objc2-core-audio-types` the crate already carries. - HE-AAC: the converter reads SBR in band and reports the doubled rate; the seam passes it through. No config-level guessing. - Priming and remainder: AudioToolbox reports `kAudioConverterPrimeInfo`; - trim it so timestamps line up with symphonia's output on the same stream. + report it as the backend's startup delay, which the front end trims, so + timestamps line up with symphonia's output on the same stream. - Every codec the backend advertises has a fixture and a decode test, and the test asserts the layout order matches the canonical one (the LFE and centre end up where `Layout` says). @@ -29,10 +35,6 @@ binding, alongside the `objc2-core-audio-types` the crate already carries. - Docs: `doc/bin/obs.md` drops the HE-AAC and multichannel caveat on macOS, and the backend table names what this host decodes. -## Required - -- [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the candidate order this backend joins - ## Related - [Media Foundation decode](/quest/m2/audio-decode-mediafoundation.md) - the same shape on Windows diff --git a/quest/m1/audio-codecs/decode-backend.md b/quest/m1/audio-codecs/decode-backend.md deleted file mode 100644 index 271ef84b52..0000000000 --- a/quest/m1/audio-codecs/decode-backend.md +++ /dev/null @@ -1,41 +0,0 @@ -# [M] A decode backend seam that prefers the platform codec - -## Goal - -`moq_audio::decode` selects a backend per codec the way `moq_video::decode` -does: platform first, software fallback, and a `Kind` to force one. Opus, -PCM, and symphonia AAC-LC become the software backends, and the crate -documents which codecs each host decodes. - -## Plan - -Mirror `rs/moq-video/src/decode/backend` in name and shape: a crate-private -`Backend` trait (`decode`, `flush`, `name`), an `open(codec, config)` that -walks the platform candidates before the software ones and refuses when none -takes the track. Use the backend-selection configuration and Decoder -constructor settled in main rather than adding a conflicting public shape. -`Decoder::name()` reports what was opened, which the OBS -stats and `moq play` surface. - -- The seam is generic over `hang::catalog::AudioCodec`, so a backend advertises - the set it opens and the selector asks each in order. Symphonia advertises - AAC-LC mono/stereo only; the platform backends that follow advertise what - their framework opens and has a fixture for. -- Move today's Opus, PCM, and symphonia code behind the trait without changing - behavior. Document per host that symphonia plays implicit-SBR HE-AAC as its - half-rate LC core: finding the in-band SBR element needs a full Huffman walk - of the channel elements, and symphonia detects it internally without - exposing or refusing it. -- A backend's output rate and layout are what it produced, not what the - catalog said (HE-AAC doubles the rate); `Consumer` already resamples and - remixes to the requested output, so that stays the seam's contract. -- The `aac` feature keeps gating symphonia. Platform backends are - `cfg(target_os)` like their video counterparts. Audio has no MediaCodec - feature yet; its Android quest introduces the optional dependency following - the settled media build policy. -- Docs: `doc/lib/rs/moq-audio.md` gains the backend table `moq-video.md` has, - and states the Linux gap. `doc/bin/cli.md` and `doc/bin/obs.md` follow. -- Regression: the selection order and `Named` refusal, tested with a stub - backend like the video seam's `probe`. - -The FFI does not expose `Kind` until a consumer asks. diff --git a/quest/m1/audio-codecs/encode-backend.md b/quest/m1/audio-codecs/encode-backend.md index 5c6dbc1073..4f73ed98f2 100644 --- a/quest/m1/audio-codecs/encode-backend.md +++ b/quest/m1/audio-codecs/encode-backend.md @@ -9,7 +9,7 @@ host with no AAC encoder refuses it at construction. ## Plan -Mirror the decode seam: `encode::backend` with a crate-private `Backend` +Mirror the decode seam (`rs/moq-audio/src/decode/backend`): `encode::backend` with a crate-private `Backend` trait (`encode`, `flush`, `set_bitrate`, `name`), an `open(codec, config)` that walks platform candidates before software ones, using the public settings and selection contract settled in main. Opus and PCM retain their behavior. This @@ -35,10 +35,6 @@ quest adds AAC through platform encoders; no software AAC dependency is selected - Regression: the selection order with a stub backend; `Codec::Aac` refused on a host with no backend; the Opus and PCM paths unchanged. -## Required - -- [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the naming and shape this mirrors - ## Related - [OBS audio publishing](/quest/m1/obs-moq-video/audio-publish.md) - the OBS encoder adapter can offer AAC once this lands diff --git a/quest/m2/audio-decode-mediacodec.md b/quest/m2/audio-decode-mediacodec.md index 5b7df87c71..1446550bf0 100644 --- a/quest/m2/audio-decode-mediacodec.md +++ b/quest/m2/audio-decode-mediacodec.md @@ -23,10 +23,6 @@ behind a new optional audio `mediacodec` feature and the decode seam, on `target - The binding ships in the moq-ffi Android slice, which is how Kotlin and Dart reach it. -## Required - -- [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the candidate order this backend joins - ## Related - [Android capture](/quest/m2/mobile-capture-android.md) - the video MediaCodec family this sits beside diff --git a/quest/m2/audio-decode-mediafoundation.md b/quest/m2/audio-decode-mediafoundation.md index 1da6a8337c..37fc31fca7 100644 --- a/quest/m2/audio-decode-mediafoundation.md +++ b/quest/m2/audio-decode-mediafoundation.md @@ -22,10 +22,6 @@ ones. Behind the decode seam as the first candidate on `target_os = - Verification runs on a Windows host; the per-PR CI only compiles the platform code, and `just rs windows` runs nightly. -## Required - -- [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the candidate order this backend joins - ## Related - [Runtime QA hosts](/quest/m2/runtime-qa-hosts.md) - where the Windows run happens diff --git a/rs/moq-audio/src/decode/backend/libopus.rs b/rs/moq-audio/src/decode/backend/libopus.rs new file mode 100644 index 0000000000..0fa9f78b69 --- /dev/null +++ b/rs/moq-audio/src/decode/backend/libopus.rs @@ -0,0 +1,126 @@ +//! Opus through libopus, the software decoder for every host. + +use unsafe_libopus::{ + OPUS_OK, OPUS_RESET_STATE, OpusDecoder, opus_decode_float, opus_decoder_create, opus_decoder_ctl_impl, + opus_decoder_destroy, varargs, +}; + +use super::Backend; +use crate::decode::Decoded; +use crate::{Error, Layout, opus}; + +pub(super) const NAME: &str = "libopus"; + +/// Opus packets cap at 120 ms (RFC 6716 §2.1.4). +const MAX_FRAME_MS: usize = 120; + +pub(super) struct Libopus { + inner: *mut OpusDecoder, + sample_rate: u32, + layout: Layout, + pre_skip: usize, + max_frame_size: usize, + in_dtx: bool, +} + +// SAFETY: the decoder is owned exclusively and libopus keeps no thread-local state. +unsafe impl Send for Libopus {} + +impl Libopus { + /// Parses the OpusHead `description` if present; falls back to the catalog's + /// declared sample rate / channel count. + pub(super) fn open(catalog: &hang::catalog::AudioConfig) -> Result, Error> { + let (sample_rate, channel_count, pre_skip) = if let Some(desc) = &catalog.description { + let mut buf = desc.as_ref(); + match moq_mux::codec::opus::Config::parse(&mut buf) { + Ok(head) => (head.sample_rate, head.channel_count, head.pre_skip), + Err(_) => (catalog.sample_rate, catalog.channel_count, 0), + } + } else { + (catalog.sample_rate, catalog.channel_count, 0) + }; + + opus::validate_rate(sample_rate)?; + let channels = opus::validate_channels(channel_count)?; + let layout = Layout::from_channels(channel_count)?; + + let mut err = 0i32; + // SAFETY: out-pointer is valid; inner is checked for null below. + let inner = unsafe { opus_decoder_create(sample_rate as i32, channels, &mut err) }; + if err != OPUS_OK || inner.is_null() { + return Err(opus::error(err, "opus_decoder_create")); + } + + Ok(Box::new(Self { + inner, + sample_rate, + layout, + // OpusHead counts pre-skip at 48 kHz whatever rate the decoder runs at. + pre_skip: (pre_skip as usize * sample_rate as usize) / 48_000, + max_frame_size: (sample_rate as usize * MAX_FRAME_MS) / 1000, + in_dtx: false, + })) + } +} + +impl Backend for Libopus { + /// Empty packets invoke packet-loss concealment. Loss during DTX remains + /// classified as DTX, while loss during active audio remains active. + fn decode(&mut self, packet: &[u8]) -> Result { + let channels = self.layout.channels() as usize; + let mut out = vec![0.0f32; self.max_frame_size * channels]; + // SAFETY: `inner` owns a live OpusDecoder; packet/out slices are bounded by + // the lengths we pass. + let samples = unsafe { + opus_decode_float( + &mut *self.inner, + packet.as_ptr(), + packet.len() as i32, + out.as_mut_ptr(), + self.max_frame_size as i32, + 0, + ) + }; + if samples < 0 { + return Err(opus::decode_error(samples)); + } + out.truncate(samples as usize * channels); + + let activity = opus::activity(packet, self.in_dtx); + self.in_dtx = activity.is_dtx(); + Ok(Decoded { samples: out, activity }) + } + + fn reset(&mut self) -> Result<(), Error> { + // SAFETY: `inner` owns a live decoder and OPUS_RESET_STATE takes no arguments. + let rc = unsafe { opus_decoder_ctl_impl(self.inner, OPUS_RESET_STATE, varargs![]) }; + if rc != OPUS_OK { + return Err(opus::error(rc, "OPUS_RESET_STATE")); + } + self.in_dtx = false; + Ok(()) + } + + fn sample_rate(&self) -> u32 { + self.sample_rate + } + + fn layout(&self) -> Layout { + self.layout + } + + fn delay(&self) -> usize { + self.pre_skip + } + + fn name(&self) -> &str { + NAME + } +} + +impl Drop for Libopus { + fn drop(&mut self) { + // SAFETY: `inner` is a live OpusDecoder that nothing else aliases. + unsafe { opus_decoder_destroy(self.inner) }; + } +} diff --git a/rs/moq-audio/src/decode/backend/mod.rs b/rs/moq-audio/src/decode/backend/mod.rs new file mode 100644 index 0000000000..f39bb4c170 --- /dev/null +++ b/rs/moq-audio/src/decode/backend/mod.rs @@ -0,0 +1,303 @@ +//! Pluggable audio decoder backends. +//! +//! The audio mirror of `moq-video`'s decode backends. [`Backend`] is the seam +//! between the codec and the [`Decoder`](super::Decoder) front end, which owns +//! what every codec shares: trimming the startup delay a backend reports. +//! +//! [`open`] tries the platform decoders before the software ones, skipping any +//! that does not advertise the catalog codec, and refuses when none opens the +//! track. The software tier is libopus for Opus, a passthrough for PCM, and +//! symphonia for AAC-LC mono and stereo (behind the `aac` feature). No platform +//! decoder is wired in yet. + +use hang::catalog::{AudioCodec, AudioConfig}; + +use super::Decoded; +use super::decoder::{Config, Kind}; +use crate::{Error, Layout}; + +mod libopus; +mod pcm; +#[cfg(feature = "aac")] +mod symphonia; + +/// An opened decoder: packets in, interleaved `f32` PCM out. +pub(crate) trait Backend: Send { + /// Decode one packet into interleaved samples at [`sample_rate`](Self::sample_rate) + /// and [`layout`](Self::layout), untrimmed. + fn decode(&mut self, packet: &[u8]) -> Result; + + /// Drop codec history after a discontinuity, so the next packet does not + /// predict from audio that is no longer adjacent. + fn reset(&mut self) -> Result<(), Error>; + + /// The rate this backend decodes to, which may differ from the catalog's. + fn sample_rate(&self) -> u32; + + /// The layout this backend decodes to, which may differ from the catalog's. + fn layout(&self) -> Layout; + + /// Frames at the start of the stream that are codec priming, not media. + fn delay(&self) -> usize { + 0 + } + + /// The stable lowercase name [`Kind::Named`] selects this backend by. + fn name(&self) -> &str; +} + +type Open = fn(&AudioConfig) -> Result, Error>; + +/// A backend constructor: its name, the catalog codecs it advertises, and an opener. +struct Candidate { + name: &'static str, + supports: fn(&AudioCodec) -> bool, + open: Open, +} + +/// Operating-system decoders, in priority order. +const PLATFORM: &[Candidate] = &[]; + +const SOFTWARE: &[Candidate] = &[ + Candidate { + name: libopus::NAME, + supports: |codec| matches!(codec, AudioCodec::Opus), + open: libopus::Libopus::open, + }, + Candidate { + name: pcm::NAME, + supports: |codec| matches!(codec, AudioCodec::Pcm), + open: pcm::Pcm::open, + }, + // Claims every AAC profile and refuses at open what it can't decode: the + // profile that matters is the description's, which the catalog string can + // contradict. + #[cfg(feature = "aac")] + Candidate { + name: symphonia::NAME, + supports: |codec| matches!(codec, AudioCodec::AAC(_)), + open: symphonia::Symphonia::open, + }, +]; + +/// Open the first backend that advertises the catalog codec and accepts the track. +pub(crate) fn open(catalog: &AudioConfig, config: &Config) -> Result, Error> { + select(catalog, &config.kind, candidates(&config.kind, PLATFORM, SOFTWARE)) +} + +/// The candidates `kind` allows, in the order to try them. +/// +/// Takes the tiers as arguments so a test can supply stubs instead of whatever +/// this host compiles in. +fn candidates<'a>(kind: &Kind, platform: &'a [Candidate], software: &'a [Candidate]) -> Vec<&'a Candidate> { + match kind { + Kind::Auto => platform.iter().chain(software).collect(), + Kind::Software => software.iter().collect(), + Kind::Named(name) => platform.iter().chain(software).filter(|c| c.name == name).collect(), + } +} + +fn select(catalog: &AudioConfig, kind: &Kind, candidates: Vec<&Candidate>) -> Result, Error> { + let codec = &catalog.codec; + let mut refused = Vec::new(); + + for candidate in candidates { + if !(candidate.supports)(codec) { + continue; + } + match (candidate.open)(catalog) { + Ok(backend) => return Ok(backend), + Err(err) => refused.push((candidate.name, err)), + } + } + + // One refusal is the whole answer, so keep its variant: a malformed + // description stays a container error rather than becoming a string. + if refused.len() == 1 { + let (_, err) = refused.remove(0); + return Err(err); + } + if !refused.is_empty() { + let reasons: Vec = refused.iter().map(|(name, err)| format!("{name}: {err}")).collect(); + return Err(Error::Unsupported(reasons.join(", "))); + } + + match kind { + Kind::Named(name) => { + let available: Vec<&str> = PLATFORM + .iter() + .chain(SOFTWARE) + .filter(|c| (c.supports)(codec)) + .map(|c| c.name) + .collect(); + Err(Error::Unsupported(format!( + "no audio decoder named {name:?} for {codec} (this build has: {})", + available.join(", ") + ))) + } + _ => Err(Error::Unsupported(format!("unsupported audio codec: {codec}"))), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::Activity; + + /// Opens anything it advertises and reports which candidate it came from. + struct Stub(&'static str); + + impl Backend for Stub { + fn decode(&mut self, _packet: &[u8]) -> Result { + Ok(Decoded { + samples: Vec::new(), + activity: Activity::Active, + }) + } + + fn reset(&mut self) -> Result<(), Error> { + Ok(()) + } + + fn sample_rate(&self) -> u32 { + 48_000 + } + + fn layout(&self) -> Layout { + Layout::Stereo + } + + fn name(&self) -> &str { + self.0 + } + } + + const PLATFORM_STUB: Candidate = Candidate { + name: "platform", + supports: |codec| matches!(codec, AudioCodec::Opus), + open: |_| Ok(Box::new(Stub("platform"))), + }; + + /// Compiled in but refusing the track, like a platform decoder asked for a + /// layout its framework does not open. + const REFUSING: Candidate = Candidate { + name: "refusing", + supports: |codec| matches!(codec, AudioCodec::Opus), + open: |_| Err(Error::Unsupported("not this track".into())), + }; + + const SOFTWARE_STUB: Candidate = Candidate { + name: "software", + supports: |codec| matches!(codec, AudioCodec::Opus), + open: |_| Ok(Box::new(Stub("software"))), + }; + + /// Advertises nothing but PCM, so an Opus track never reaches its opener. + const PCM_ONLY: Candidate = Candidate { + name: "pcm-only", + supports: |codec| matches!(codec, AudioCodec::Pcm), + open: |_| panic!("opened for a codec it does not advertise"), + }; + + fn opus() -> AudioConfig { + AudioConfig::new(AudioCodec::Opus, 48_000, 2) + } + + fn pick(kind: Kind, platform: &[Candidate], software: &[Candidate]) -> Result { + let backend = select(&opus(), &kind, candidates(&kind, platform, software))?; + Ok(backend.name().to_owned()) + } + + #[test] + fn auto_prefers_platform() { + let name = pick(Kind::Auto, &[PCM_ONLY, PLATFORM_STUB], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "platform"); + } + + #[test] + fn auto_falls_back_to_software() { + let name = pick(Kind::Auto, &[REFUSING], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "software"); + } + + #[test] + fn software_skips_platform() { + let name = pick(Kind::Software, &[PLATFORM_STUB], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "software"); + } + + #[test] + fn named_forces_one() { + let name = pick( + Kind::Named("software".into()), + &[PLATFORM_STUB], + &[PCM_ONLY, SOFTWARE_STUB], + ) + .unwrap(); + assert_eq!(name, "software"); + } + + /// A named backend that refuses the track is the answer: nothing else is tried. + #[test] + fn named_refusal_does_not_fall_back() { + let err = pick(Kind::Named("refusing".into()), &[REFUSING], &[SOFTWARE_STUB]).unwrap_err(); + assert!(err.to_string().contains("not this track"), "{err}"); + } + + #[test] + fn every_refusal_is_reported() { + const ALSO_REFUSING: Candidate = Candidate { + name: "also-refusing", + ..REFUSING + }; + + let err = pick(Kind::Auto, &[REFUSING], &[ALSO_REFUSING]).unwrap_err(); + let message = err.to_string(); + assert!( + message.contains("refusing: ") && message.contains("also-refusing: "), + "{message}" + ); + } + + /// An unknown name says what this build has for the codec instead. + #[test] + fn unknown_name_lists_the_alternatives() { + let err = open( + &opus(), + &Config { + kind: Kind::Named("opus".into()), + }, + ) + .err() + .expect("no backend is named after its codec"); + let message = err.to_string(); + assert!( + message.contains("\"opus\"") && message.contains(libopus::NAME), + "{message}" + ); + } + + /// Asking for a real backend that does not decode the codec is refused, not + /// quietly swapped for one that does. + #[test] + fn named_backend_for_another_codec_is_refused() { + let config = Config { + kind: Kind::Named(pcm::NAME.into()), + }; + assert!(matches!(open(&opus(), &config), Err(Error::Unsupported(_)))); + } + + #[test] + fn software_backends_open_by_name() { + let pcm = AudioConfig::new(AudioCodec::Pcm, 48_000, 2); + let config = Config { + kind: Kind::Named(pcm::NAME.into()), + }; + assert_eq!(open(&pcm, &config).unwrap().name(), pcm::NAME); + + let config = Config { + kind: Kind::Named(libopus::NAME.into()), + }; + assert_eq!(open(&opus(), &config).unwrap().name(), libopus::NAME); + } +} diff --git a/rs/moq-audio/src/decode/backend/pcm.rs b/rs/moq-audio/src/decode/backend/pcm.rs new file mode 100644 index 0000000000..bead86fb5e --- /dev/null +++ b/rs/moq-audio/src/decode/backend/pcm.rs @@ -0,0 +1,78 @@ +//! Uncompressed little-endian `f32` PCM, described entirely by the catalog. + +use super::Backend; +use crate::decode::Decoded; +use crate::{Activity, Error, Layout, pcm}; + +pub(super) const NAME: &str = "pcm"; + +pub(super) struct Pcm { + sample_rate: u32, + layout: Layout, + bytes_per_frame: usize, +} + +impl Pcm { + /// Uses the catalog's rate and channel count, and requires an absent `description`. + pub(super) fn open(catalog: &hang::catalog::AudioConfig) -> Result, Error> { + if catalog.sample_rate == 0 { + return Err(Error::Unsupported("pcm sample rate must be greater than zero".into())); + } + if catalog.channel_count == 0 { + return Err(Error::Unsupported("pcm channel count must be greater than zero".into())); + } + if catalog.description.is_some() { + return Err(Error::Unsupported("pcm catalog description must be absent".into())); + } + let bitrate = pcm::bitrate(catalog.sample_rate, catalog.channel_count)?; + if catalog.bitrate.is_some_and(|declared| declared != bitrate) { + return Err(Error::Unsupported(format!( + "pcm catalog bitrate must be {bitrate} bits per second" + ))); + } + + Ok(Box::new(Self { + sample_rate: catalog.sample_rate, + layout: Layout::from_channels(catalog.channel_count)?, + bytes_per_frame: pcm::frame_bytes(1, catalog.channel_count)?, + })) + } +} + +impl Backend for Pcm { + fn decode(&mut self, packet: &[u8]) -> Result { + if packet.is_empty() || !packet.len().is_multiple_of(self.bytes_per_frame) { + return Err(Error::Misaligned { + got: packet.len(), + expected: packet.len().max(1).next_multiple_of(self.bytes_per_frame), + }); + } + + let samples = packet + .as_chunks::<{ pcm::BYTES_PER_SAMPLE }>() + .0 + .iter() + .map(|sample| f32::from_le_bytes(*sample)) + .collect(); + Ok(Decoded { + samples, + activity: Activity::Active, + }) + } + + fn reset(&mut self) -> Result<(), Error> { + Ok(()) + } + + fn sample_rate(&self) -> u32 { + self.sample_rate + } + + fn layout(&self) -> Layout { + self.layout + } + + fn name(&self) -> &str { + NAME + } +} diff --git a/rs/moq-audio/src/decode/backend/symphonia.rs b/rs/moq-audio/src/decode/backend/symphonia.rs new file mode 100644 index 0000000000..e2a87c6253 --- /dev/null +++ b/rs/moq-audio/src/decode/backend/symphonia.rs @@ -0,0 +1,106 @@ +//! AAC-LC through symphonia, the pure-Rust fallback for hosts without a +//! platform AAC decoder. + +use symphonia_core::codecs::audio::AudioDecoder; + +use super::Backend; +use crate::decode::Decoded; +use crate::{Activity, Error, Layout, aac}; + +pub(super) const NAME: &str = "symphonia"; + +pub(super) struct Symphonia { + inner: symphonia_codec_aac::AacDecoder, + sample_rate: u32, + layout: Layout, +} + +impl Symphonia { + /// AAC-LC in mono or stereo only, which is what every gateway that feeds + /// this crate publishes. + /// + /// HE-AAC is rejected however its config spells it: leading with SBR or PS + /// (mp4a.40.5 / .29), or leading with LC and declaring SBR in a sync extension + /// after the core. Symphonia decodes no SBR either way, so the alternative is + /// half-rate audio that sounds like a fault rather than an unsupported codec. + /// A stream that signals SBR only in band is indistinguishable from LC in the + /// config, and does decode as the core. + pub(super) fn open(catalog: &hang::catalog::AudioConfig) -> Result, Error> { + use symphonia_core::codecs::audio::well_known::CODEC_ID_AAC; + use symphonia_core::codecs::audio::{AudioCodecParameters, AudioDecoderOptions}; + + let hang::catalog::AudioCodec::AAC(codec) = &catalog.codec else { + return Err(Error::Unsupported(format!("symphonia cannot decode {}", catalog.codec))); + }; + let description = aac::description(catalog, codec.profile)?; + + let mut params = AudioCodecParameters::new(); + params + .for_codec(CODEC_ID_AAC) + .with_extra_data(description.to_vec().into_boxed_slice()); + + let inner = symphonia_codec_aac::AacDecoder::try_new(¶ms, &AudioDecoderOptions::default()) + .map_err(|err| Error::Unsupported(format!("aac decoder: {err}")))?; + + // Resolved by the decoder from the config, so this is what it will emit + // even when the catalog's own fields say otherwise. + let params = inner.codec_params(); + let sample_rate = params + .sample_rate + .ok_or_else(|| Error::Unsupported("aac config declares no sample rate".into()))?; + let channel_count = params + .channels + .as_ref() + .map(|channels| channels.count()) + .ok_or_else(|| Error::Unsupported("aac config declares no channels".into()))?; + + Ok(Box::new(Self { + inner, + sample_rate, + layout: Layout::from_channels(channel_count as u32)?, + })) + } +} + +impl Backend for Symphonia { + fn decode(&mut self, packet: &[u8]) -> Result { + // The packet is a raw AAC frame, not ADTS, so there is nothing to timestamp + // it with here: the container carries the timestamp and the decoder only + // reads the payload. + let packet = symphonia_core::packet::PacketRef::new( + 0, + symphonia_core::units::Timestamp::ZERO, + symphonia_core::units::Duration::ZERO, + packet, + ); + + let decoded = self + .inner + .decode_ref(&packet) + .map_err(|err| Error::Decode(format!("aac: {err}")))?; + + let mut samples = Vec::new(); + decoded.copy_to_vec_interleaved(&mut samples); + Ok(Decoded { + samples, + activity: Activity::Active, + }) + } + + fn reset(&mut self) -> Result<(), Error> { + self.inner.reset(); + Ok(()) + } + + fn sample_rate(&self) -> u32 { + self.sample_rate + } + + fn layout(&self) -> Layout { + self.layout + } + + fn name(&self) -> &str { + NAME + } +} diff --git a/rs/moq-audio/src/decode/consumer.rs b/rs/moq-audio/src/decode/consumer.rs index 5110db70e5..b5f8ccdce8 100644 --- a/rs/moq-audio/src/decode/consumer.rs +++ b/rs/moq-audio/src/decode/consumer.rs @@ -187,6 +187,11 @@ impl Consumer { }) } + /// The decoder backend name in use, e.g. `"libopus"` or `"symphonia"`. + pub fn name(&self) -> &str { + self.decoder.name() + } + /// The options this consumer was built with. pub fn options(&self) -> &Options { &self.options diff --git a/rs/moq-audio/src/decode/decoder.rs b/rs/moq-audio/src/decode/decoder.rs index 5f85d66c1c..9cb9d7954f 100644 --- a/rs/moq-audio/src/decode/decoder.rs +++ b/rs/moq-audio/src/decode/decoder.rs @@ -1,36 +1,24 @@ //! Audio decoder front end. //! -//! Mirror of [`encode::Encoder`](crate::encode::Encoder): dispatches over the -//! catalog codec and produces interleaved `f32` PCM. - -use unsafe_libopus::{ - OPUS_OK, OPUS_RESET_STATE, OpusDecoder, opus_decode_float, opus_decoder_create, opus_decoder_ctl_impl, - opus_decoder_destroy, varargs, -}; - -#[cfg(feature = "aac")] -use symphonia_core::codecs::audio::AudioDecoder; +//! Mirror of [`encode::Encoder`](crate::encode::Encoder): opens a +//! [`Backend`](super::backend::Backend) for the catalog codec and trims its +//! startup delay, producing interleaved `f32` PCM. use super::Decoded; -#[cfg(feature = "aac")] -use crate::aac; -use crate::opus; -use crate::pcm; -use crate::{Activity, Error, Layout}; - -/// Opus packets cap at 120 ms (RFC 6716 §2.1.4). -const MAX_FRAME_MS: usize = 120; +use super::backend::{self, Backend}; +use crate::{Error, Layout}; /// Decoder backend selection. #[derive(Clone, Debug, Default, PartialEq, Eq)] #[non_exhaustive] pub enum Kind { - /// Pick the available backend automatically. + /// Prefer a platform decoder, falling back to software. #[default] Auto, - /// Require the built-in software backend. + /// Require a software backend. Software, - /// Require a backend by its stable lowercase name. + /// Require a backend by its stable lowercase name: `"libopus"`, `"pcm"`, or + /// `"symphonia"`. Named(String), } @@ -54,184 +42,42 @@ impl Config { /// The bring-your-own-payload layer under [`Consumer`](super::Consumer): use it /// when the packets don't come from a plain track subscription. pub struct Decoder { - backend: Backend, - sample_rate: u32, - layout: Layout, + backend: Box, + /// Startup delay in native-rate frames, and how much of it is left to trim. delay: usize, -} - -enum Backend { - Opus(Opus), - Pcm { - bytes_per_frame: usize, - }, - #[cfg(feature = "aac")] - Aac(Box), -} - -struct Opus { - inner: *mut OpusDecoder, - pre_skip_remaining: usize, - max_frame_size: usize, - in_dtx: bool, -} - -// SAFETY: see Encoder. -unsafe impl Send for Opus {} - -/// Boxed in [`Backend`]: the symphonia decoder carries its own filterbank state, -/// which is far larger than the other backends' handles. -#[cfg(feature = "aac")] -struct Aac { - inner: symphonia_codec_aac::AacDecoder, + delay_remaining: usize, } impl Decoder { /// Build a decoder from a catalog [`AudioConfig`](hang::catalog::AudioConfig). /// - /// Parses the OpusHead `description` if present; falls back to the catalog's - /// declared sample rate / channel count. PCM uses those catalog fields - /// directly and requires an absent `description`. + /// Opus parses the OpusHead `description` if present, falling back to the + /// catalog's declared sample rate and channel count. PCM uses those catalog + /// fields directly and requires an absent `description`. AAC reads its + /// AudioSpecificConfig, synthesizing one from the catalog when absent. pub fn new(catalog: &hang::catalog::AudioConfig, config: &Config) -> Result { - let name = match &catalog.codec { - hang::catalog::AudioCodec::Opus => "opus", - hang::catalog::AudioCodec::Pcm => "pcm", - #[cfg(feature = "aac")] - hang::catalog::AudioCodec::AAC(_) => "aac", - codec => return Err(Error::Unsupported(format!("unsupported audio codec: {codec}"))), - }; - match &config.kind { - Kind::Auto | Kind::Software => {} - Kind::Named(requested) if requested == name => {} - Kind::Named(requested) => { - return Err(Error::Unsupported(format!( - "audio decoder backend {requested:?} is unavailable for {name}" - ))); - } - } - match &catalog.codec { - hang::catalog::AudioCodec::Opus => Self::new_opus(catalog), - hang::catalog::AudioCodec::Pcm => Self::new_pcm(catalog), - #[cfg(feature = "aac")] - hang::catalog::AudioCodec::AAC(aac) => Self::new_aac(catalog, aac.profile), - codec => Err(Error::Unsupported(format!("unsupported audio codec: {codec}"))), - } - } - - fn new_opus(catalog: &hang::catalog::AudioConfig) -> Result { - let (sample_rate, channel_count, pre_skip) = if let Some(desc) = &catalog.description { - let mut buf = desc.as_ref(); - match moq_mux::codec::opus::Config::parse(&mut buf) { - Ok(head) => (head.sample_rate, head.channel_count, head.pre_skip), - Err(_) => (catalog.sample_rate, catalog.channel_count, 0), - } - } else { - (catalog.sample_rate, catalog.channel_count, 0) - }; - - opus::validate_rate(sample_rate)?; - let channels = opus::validate_channels(channel_count)?; - - let mut err = 0i32; - // SAFETY: out-pointer is valid; inner is checked for null below. - let inner = unsafe { opus_decoder_create(sample_rate as i32, channels, &mut err) }; - if err != OPUS_OK || inner.is_null() { - return Err(opus::error(err, "opus_decoder_create")); - } - - let max_frame_size = (sample_rate as usize * MAX_FRAME_MS) / 1000; - let pre_skip_remaining = (pre_skip as usize * sample_rate as usize) / 48_000; - + let backend = backend::open(catalog, config)?; + let delay = backend.delay(); Ok(Self { - backend: Backend::Opus(Opus { - inner, - pre_skip_remaining, - max_frame_size, - in_dtx: false, - }), - sample_rate, - layout: Layout::from_channels(channel_count)?, - delay: pre_skip_remaining, + backend, + delay, + delay_remaining: delay, }) } - /// AAC-LC only, which is what every gateway that feeds this crate publishes. - /// - /// HE-AAC is rejected however its config spells it: leading with SBR or PS - /// (mp4a.40.5 / .29), or leading with LC and declaring SBR in a sync extension - /// after the core. Symphonia decodes no SBR either way, so the alternative is - /// half-rate audio that sounds like a fault rather than an unsupported codec. - /// A stream that signals SBR only in band is indistinguishable from LC in the - /// config, and does decode as the core. - #[cfg(feature = "aac")] - fn new_aac(catalog: &hang::catalog::AudioConfig, profile: u8) -> Result { - use symphonia_core::codecs::audio::well_known::CODEC_ID_AAC; - use symphonia_core::codecs::audio::{AudioCodecParameters, AudioDecoderOptions}; - - let description = aac::description(catalog, profile)?; - - let mut params = AudioCodecParameters::new(); - params - .for_codec(CODEC_ID_AAC) - .with_extra_data(description.to_vec().into_boxed_slice()); - - let inner = symphonia_codec_aac::AacDecoder::try_new(¶ms, &AudioDecoderOptions::default()) - .map_err(|err| Error::Unsupported(format!("aac decoder: {err}")))?; - - // Resolved by the decoder from the config, so this is what it will emit - // even when the catalog's own fields say otherwise. - let params = inner.codec_params(); - let sample_rate = params - .sample_rate - .ok_or_else(|| Error::Unsupported("aac config declares no sample rate".into()))?; - let channel_count = params - .channels - .as_ref() - .map(|channels| channels.count()) - .ok_or_else(|| Error::Unsupported("aac config declares no channels".into()))?; - - Ok(Self { - backend: Backend::Aac(Box::new(Aac { inner })), - sample_rate, - layout: Layout::from_channels(channel_count as u32)?, - delay: 0, - }) + /// The decoder backend name in use, e.g. `"libopus"` or `"symphonia"`. + pub fn name(&self) -> &str { + self.backend.name() } - fn new_pcm(catalog: &hang::catalog::AudioConfig) -> Result { - if catalog.sample_rate == 0 { - return Err(Error::Unsupported("pcm sample rate must be greater than zero".into())); - } - if catalog.channel_count == 0 { - return Err(Error::Unsupported("pcm channel count must be greater than zero".into())); - } - if catalog.description.is_some() { - return Err(Error::Unsupported("pcm catalog description must be absent".into())); - } - let bitrate = pcm::bitrate(catalog.sample_rate, catalog.channel_count)?; - if catalog.bitrate.is_some_and(|declared| declared != bitrate) { - return Err(Error::Unsupported(format!( - "pcm catalog bitrate must be {bitrate} bits per second" - ))); - } - let bytes_per_frame = pcm::frame_bytes(1, catalog.channel_count)?; - - Ok(Self { - backend: Backend::Pcm { bytes_per_frame }, - sample_rate: catalog.sample_rate, - layout: Layout::from_channels(catalog.channel_count)?, - delay: 0, - }) - } - - /// The rate the codec decodes at, read from the catalog. + /// The rate the codec decodes at, which may differ from the catalog's. pub fn sample_rate(&self) -> u32 { - self.sample_rate + self.backend.sample_rate() } - /// The PCM layout decoded from the catalog. + /// The PCM layout the codec decodes to. pub fn layout(&self) -> Layout { - self.layout + self.backend.layout() } /// Reset codec history and reapply startup delay for a new discontinuous epoch. @@ -243,27 +89,12 @@ impl Decoder { /// Reapply catalog startup delay for a new playhead epoch without resetting codec prediction. pub(super) fn reapply_delay(&mut self) { - if let Backend::Opus(opus) = &mut self.backend { - opus.pre_skip_remaining = self.delay; - } + self.delay_remaining = self.delay; } /// Reset codec prediction after packet loss without reapplying stream startup delay. pub(super) fn reset_prediction(&mut self) -> Result<(), Error> { - match &mut self.backend { - Backend::Opus(opus) => { - // SAFETY: `inner` owns a live decoder and OPUS_RESET_STATE takes no arguments. - let rc = unsafe { opus_decoder_ctl_impl(opus.inner, OPUS_RESET_STATE, varargs![]) }; - if rc != OPUS_OK { - return Err(crate::opus::error(rc, "OPUS_RESET_STATE")); - } - opus.in_dtx = false; - } - Backend::Pcm { .. } => {} - #[cfg(feature = "aac")] - Backend::Aac(aac) => aac.inner.reset(), - } - Ok(()) + self.backend.reset() } /// How much startup delay is still to be trimmed, in native-rate frames. @@ -272,12 +103,7 @@ impl Decoder { /// it, so a caller tracking where a packet ends has to add back whatever this /// dropped across the call. pub(super) fn delay_remaining(&self) -> usize { - match &self.backend { - Backend::Opus(opus) => opus.pre_skip_remaining, - Backend::Pcm { .. } => 0, - #[cfg(feature = "aac")] - Backend::Aac(_) => 0, - } + self.delay_remaining } /// Decode one packet into interleaved `f32` PCM and report its codec activity. @@ -285,88 +111,14 @@ impl Decoder { /// Empty Opus packets invoke packet-loss concealment. Loss during DTX remains /// classified as DTX, while loss during active audio remains active. pub fn decode(&mut self, packet: &[u8]) -> Result { - match &mut self.backend { - Backend::Opus(opus) => { - let channels = self.layout.channels() as usize; - let mut out = vec![0.0f32; opus.max_frame_size * channels]; - // SAFETY: `inner` owns a live OpusDecoder; packet/out slices are - // bounded by the lengths we pass. - let samples = unsafe { - opus_decode_float( - &mut *opus.inner, - packet.as_ptr(), - packet.len() as i32, - out.as_mut_ptr(), - opus.max_frame_size as i32, - 0, - ) - }; - if samples < 0 { - return Err(crate::opus::decode_error(samples)); - } - out.truncate(samples as usize * channels); - let trim_frames = opus.pre_skip_remaining.min(samples as usize); - if trim_frames > 0 { - let trim_samples = trim_frames * channels; - out.copy_within(trim_samples.., 0); - out.truncate(out.len() - trim_samples); - opus.pre_skip_remaining -= trim_frames; - } - let activity = crate::opus::activity(packet, opus.in_dtx); - opus.in_dtx = activity.is_dtx(); - Ok(Decoded { samples: out, activity }) - } - Backend::Pcm { bytes_per_frame } => { - if packet.is_empty() || !packet.len().is_multiple_of(*bytes_per_frame) { - return Err(Error::Misaligned { - got: packet.len(), - expected: packet.len().max(1).next_multiple_of(*bytes_per_frame), - }); - } - - let out = packet - .as_chunks::<{ pcm::BYTES_PER_SAMPLE }>() - .0 - .iter() - .map(|sample| f32::from_le_bytes([sample[0], sample[1], sample[2], sample[3]])) - .collect(); - Ok(Decoded { - samples: out, - activity: Activity::Active, - }) - } - #[cfg(feature = "aac")] - Backend::Aac(aac) => { - // The packet is a raw AAC frame, not ADTS, so there is nothing to - // timestamp it with here: the container carries the timestamp and the - // decoder only reads the payload. - let packet = symphonia_core::packet::PacketRef::new( - 0, - symphonia_core::units::Timestamp::ZERO, - symphonia_core::units::Duration::ZERO, - packet, - ); - - let decoded = aac - .inner - .decode_ref(&packet) - .map_err(|err| Error::Decode(format!("aac: {err}")))?; - - let mut out = Vec::new(); - decoded.copy_to_vec_interleaved(&mut out); - Ok(Decoded { - samples: out, - activity: Activity::Active, - }) - } + let mut decoded = self.backend.decode(packet)?; + let channels = self.backend.layout().channels() as usize; + let trim = self.delay_remaining.min(decoded.samples.len() / channels); + if trim > 0 { + decoded.samples.drain(..trim * channels); + self.delay_remaining -= trim; } - } -} - -impl Drop for Opus { - fn drop(&mut self) { - // SAFETY: `inner` is a live OpusDecoder that nothing else aliases. - unsafe { opus_decoder_destroy(self.inner) }; + Ok(decoded) } } @@ -405,6 +157,7 @@ mod tests { #[test] fn aac_decodes_a_sine() { let mut decoder = Decoder::new(&aac_catalog(), &Config::default()).unwrap(); + assert_eq!(decoder.name(), "symphonia"); assert_eq!(decoder.sample_rate(), 44_100); assert_eq!(decoder.layout(), Layout::Mono); diff --git a/rs/moq-audio/src/decode/mod.rs b/rs/moq-audio/src/decode/mod.rs index 7e3cbd14a7..6f4500054c 100644 --- a/rs/moq-audio/src/decode/mod.rs +++ b/rs/moq-audio/src/decode/mod.rs @@ -10,8 +10,11 @@ //! [`Decoded`] interleaved `f32` samples. //! //! [`Options`] keeps subscription and output policy separate from the -//! lower-level decoder [`Config`]. +//! lower-level decoder [`Config`], whose [`Kind`] picks the backend: a +//! platform decoder first where one takes the track, then software (libopus, +//! PCM, and symphonia for AAC-LC). +mod backend; mod consumer; mod decoded; mod decoder; diff --git a/rs/moq-cli/src/play/media.rs b/rs/moq-cli/src/play/media.rs index 0c12887e13..c47b3a9c3c 100644 --- a/rs/moq-cli/src/play/media.rs +++ b/rs/moq-cli/src/play/media.rs @@ -224,7 +224,7 @@ impl Media { decode.output.format = moq_audio::Format::F32; match moq_audio::decode::Consumer::new(&rendition, &config, &name, decode).await { Ok(consumer) => { - tracing::info!(track = name, "playing audio rendition"); + tracing::info!(track = name, decoder = consumer.name(), "playing audio rendition"); if engine.is_none() { engine = Some(Engine::open(Default::default()).await?); }