From 576285af1f9fc8ef5817deee73d5c59c3c5f4f82 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 24 Sep 2026 18:13:32 -0700 Subject: [PATCH 01/11] quest: open the audio-codecs line Co-Authored-By: Claude Opus 5.5 From c31063b052ebfebb5728d8312436c322bac6b8dd Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 00:17:26 -0700 Subject: [PATCH 02/11] fix(mux): parse the AAC program config element instead of guessing stereo (#4093) Co-authored-by: Claude Opus 5.5 --- quest/m1/audio-codecs/README.md | 6 +- quest/m1/audio-codecs/aac-pce.md | 34 -- quest/m1/audio-codecs/he-aac-refusal.md | 1 - quest/m1/audio-codecs/layout.md | 5 +- quest/m1/audio-codecs/ts-export-pce.md | 17 + rs/moq-mux/src/codec/aac/mod.rs | 396 ++++++++++++++++-- rs/moq-mux/src/container/ts/adts.rs | 20 +- rs/moq-mux/src/container/ts/import.rs | 20 +- rs/moq-mux/src/container/ts/import_test.rs | 26 ++ .../src/container/ts/test_data/aac_quad.ts | Bin 0 -> 2632 bytes 10 files changed, 436 insertions(+), 89 deletions(-) delete mode 100644 quest/m1/audio-codecs/aac-pce.md create mode 100644 quest/m1/audio-codecs/ts-export-pce.md create mode 100644 rs/moq-mux/src/container/ts/test_data/aac_quad.ts diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index 372157036a..c1366d831b 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -37,14 +37,12 @@ mono/stereo. The core configuration and layout contracts land in main. These quests implement surround and backend dispatch on that contract; each platform then lands as -its own decode and encode quest so verification stays per host. The -HE-AAC refusal and the PCE parse are defects in what ships today and are -ready now. +its own decode and encode quest so verification stays per host. ## Quests - [HE-AAC refusal](/quest/m1/audio-codecs/he-aac-refusal.md) - implicit-SBR HE-AAC over TS is refused instead of half-decoded as the LC core -- [AAC PCE](/quest/m1/audio-codecs/aac-pce.md) - a channel_config of 0 parses the program config element instead of guessing stereo +- [TS export PCE](/quest/m1/audio-codecs/ts-export-pce.md) - a TS export of a PCE-described AAC track writes channel_config 0 and the PCE instead of a count-derived config - [Layout](/quest/m1/audio-codecs/layout.md) - the settled `Layout` carries up to 7.1 through decode, resample, playback, and the FFI - [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - `decode::backend` selects a platform decoder before symphonia, mirroring moq-video - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers diff --git a/quest/m1/audio-codecs/aac-pce.md b/quest/m1/audio-codecs/aac-pce.md deleted file mode 100644 index db99ec93f9..0000000000 --- a/quest/m1/audio-codecs/aac-pce.md +++ /dev/null @@ -1,34 +0,0 @@ -# [S] moq-mux: an AAC channel_config of 0 parses the PCE or refuses the track - -## Goal - -The catalog never invents an AAC channel count. A stream whose -channelConfiguration is 0 gets its count from the program config element it -carries, and a reserved value refuses the track with a clear error. Today -both map to stereo with a warning. - -## Plan - -The ADTS and ASC importers in `rs/moq-mux/src/codec/aac` map -`channel_config == 0` and every value from 8 to 15 to stereo and warn. -Warn-then-continue is banned: supported or refused. - -- ASC: with `channel_config == 0` the GASpecificConfig carries a - `program_config_element`; parse its front, side, back, and LFE element - counts into the channel count (and, once [Layout](/quest/m1/audio-codecs/layout.md) - lands, into a layout). -- ADTS: channel_config 0 means the PCE is in the first raw data block. Parse - it there, once per track, the same way the HE-AAC sniff reads the first - block's fill elements. -- Give every ASC configuration from 8 to 15 an explicit disposition: implement - any supported channel mapping and return `Error::Unsupported` for every - remaining value, including reserved configurations. -- Regression: an ASC fixture with a PCE reports its real count; an ADTS - fixture with an in-band PCE does too; each value from 8 to 15 has its - supported count asserted or is refused; the - common configurations 1 to 7 are unchanged. - -## Related - -- [HE-AAC refusal](/quest/m1/audio-codecs/he-aac-refusal.md) - reads the same first block -- [Layout](/quest/m1/audio-codecs/layout.md) - what the parsed PCE eventually maps to diff --git a/quest/m1/audio-codecs/he-aac-refusal.md b/quest/m1/audio-codecs/he-aac-refusal.md index f7995cc0f7..c3edf29e21 100644 --- a/quest/m1/audio-codecs/he-aac-refusal.md +++ b/quest/m1/audio-codecs/he-aac-refusal.md @@ -31,4 +31,3 @@ does too. ## Related - [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the sniff becomes the software backend's contract -- [AAC PCE](/quest/m1/audio-codecs/aac-pce.md) - the other place the catalog lies about an AAC stream diff --git a/quest/m1/audio-codecs/layout.md b/quest/m1/audio-codecs/layout.md index 2df431c95c..e92dfbbe38 100644 --- a/quest/m1/audio-codecs/layout.md +++ b/quest/m1/audio-codecs/layout.md @@ -16,7 +16,10 @@ Keep the representation extensible. Each codec module maps its native order into it: AAC's `C L R Ls Rs LFE` and Opus's Vorbis `L C R Ls Rs LFE` both become `L R C LFE Ls Rs`. An unspecified discrete PCM layout remains valid for passthrough but is refused for spatial remixing; never invent speaker -positions from an arbitrary channel count. +positions from an arbitrary channel count. An AAC stream with +channelConfiguration 0 takes its layout from the program config element's +front, side, back, and LFE elements, which `moq_mux::codec::aac` already parses +for the channel count. - Reuse the settled PCM descriptors and codec/consumer settings. This quest adds supported layout behavior rather than replacing public field types. diff --git a/quest/m1/audio-codecs/ts-export-pce.md b/quest/m1/audio-codecs/ts-export-pce.md new file mode 100644 index 0000000000..31e9b76b25 --- /dev/null +++ b/quest/m1/audio-codecs/ts-export-pce.md @@ -0,0 +1,17 @@ +# [S] TS export carries an AAC track's program config element + +## Goal + +`moq-mux`'s MPEG-TS export writes an AAC track described by a program config +element (channelConfiguration 0) as ADTS with channel_config 0 and the PCE at +the start of the first raw data block, so a TS round trip keeps the layout. +Today the export derives the ADTS channel_config from the channel count, which +mislabels such a track. + +## Plan + +Take the PCE from the track's AudioSpecificConfig description with the parser +that reads it on import, and write it as ffmpeg does: once, leading the first +raw data block. Tracks with a nonzero channelConfiguration are unchanged. Test +a round trip of the quad fixture from the PCE import (`aac_quad.ts`): import, +export, and import again, asserting the same description and channel count. diff --git a/rs/moq-mux/src/codec/aac/mod.rs b/rs/moq-mux/src/codec/aac/mod.rs index e2477911fc..6d7f7cb67a 100644 --- a/rs/moq-mux/src/codec/aac/mod.rs +++ b/rs/moq-mux/src/codec/aac/mod.rs @@ -27,6 +27,26 @@ pub enum Error { #[error("unsupported sample rate index: {0}")] UnsupportedSampleRateIndex(u8), + + /// The channelConfiguration is a reserved value (8, 9, 10, or 15). + #[error("reserved channelConfiguration: {0}")] + ReservedChannelConfig(u8), + + /// A program config element is used with an object type that has no GASpecificConfig. + #[error("channelConfiguration 0 is unsupported for audioObjectType {0}")] + ProgramConfigUnsupported(u8), + + /// ADTS signals a program config element, but none leads the first raw data block. + #[error("channelConfiguration 0 without a program config element leading the first raw data block")] + ProgramConfigMissing, + + /// The program config element ends before its declared fields. + #[error("program config element truncated")] + ProgramConfigTruncated, + + /// The program config element declares zero channels. + #[error("program config element declares no channels")] + ProgramConfigEmpty, } pub type Result = std::result::Result; @@ -46,19 +66,16 @@ impl Config { /// (object_type == 31), and explicit sample rates (freq_index == 15). The /// fields are bit-packed and not byte-aligned, so a bit reader is required: /// with an explicit 24-bit rate the channelConfiguration lands mid-byte after - /// it. Any SBR/PS extension bits after the core fields are consumed. + /// it. A channelConfiguration of 0 takes the count from the program config element + /// that follows, and a reserved one is refused. Any SBR/PS extension bits after the + /// core fields are consumed. pub fn parse(buf: &mut T) -> Result { if buf.remaining() < 2 { return Err(Error::ConfigTooShort); } let mut reader = BitReader::new(buf); - - // audioObjectType: 5 bits, escaped to 6 more when it reads 31. - let mut object_type = reader.read(5, Error::ConfigTooShort)? as u8; - if object_type == 31 { - object_type = 32 + reader.read(6, Error::ExtendedConfigTooShort)? as u8; - } + let object_type = read_object_type(&mut reader)?; // samplingFrequencyIndex: 4 bits; index 15 means an explicit 24-bit rate follows. let freq_index = reader.read(4, Error::IncompleteConfig)? as u8; @@ -72,7 +89,36 @@ impl Config { // channelConfiguration: 4 bits, immediately after the (possibly explicit) rate. let channel_config = reader.read(4, Error::IncompleteConfig)? as u8; - let channel_count = channel_count_from_config(channel_config); + let channel_count = match channel_config { + 0 => { + // Explicit SBR and PS name their core object type after an extension rate; the + // GASpecificConfig carrying the program config element follows that core type. + let mut core = object_type; + if matches!(object_type, 5 | 29) { + if reader.read(4, Error::IncompleteConfig)? == 15 { + reader.read(24, Error::IncompleteConfig)?; + } + core = read_object_type(&mut reader)?; + if core == 22 { + // extensionChannelConfiguration, only for ER BSAC. + reader.read(4, Error::IncompleteConfig)?; + } + } + if !GENERAL_AUDIO.contains(&core) { + return Err(Error::ProgramConfigUnsupported(core)); + } + + // GASpecificConfig: frameLengthFlag, dependsOnCoreCoder (then a 14-bit + // coreCoderDelay), and extensionFlag precede the element. + reader.read(1, Error::IncompleteConfig)?; + if reader.read(1, Error::IncompleteConfig)? == 1 { + reader.read(14, Error::IncompleteConfig)?; + } + reader.read(1, Error::IncompleteConfig)?; + program_config(&mut reader)? + } + _ => channel_count_from_config(channel_config)?, + }; // AudioSpecificConfig can carry variable-length extensions (SBR, PS, etc.). // We've extracted the essential fields; drain the rest so the buffer is advanced. @@ -132,6 +178,101 @@ impl Config { } } +/// Build the AudioSpecificConfig for a stream that signals its fields per frame, as ADTS does. +/// +/// A `channel_config` of 0 means a program config element leads `block`, the stream's first raw +/// data block; it moves into the config, so the config describes the channels on its own. Any +/// other value leaves `block` unread. An element placed anywhere else is refused: reaching past +/// the channel data needs a full Huffman decode. +pub(crate) fn in_band_config(profile: u8, sample_rate: u32, channel_config: u8, block: &[u8]) -> Result { + let mut out = BitWriter::default(); + out.write(5, u32::from(profile & 0x1F)); + match SAMPLE_RATES.iter().position(|&rate| rate == sample_rate) { + Some(index) => out.write(4, index as u32), + None => { + out.write(4, 15); + out.write(24, sample_rate); + } + } + out.write(4, u32::from(channel_config)); + + if channel_config == 0 { + // GASpecificConfig: frameLengthFlag, dependsOnCoreCoder, and extensionFlag, all clear. + out.write(3, 0); + + let mut block = block; + let mut reader = BitReader::new(&mut block); + if reader.read(3, Error::ProgramConfigMissing)? != ID_PCE { + return Err(Error::ProgramConfigMissing); + } + reader.record = Some(out); + program_config(&mut reader)?; + out = reader.record.take().expect("recording set above"); + } + + Ok(Bytes::from(out.bytes)) +} + +/// The raw data block element ID of a program config element (ISO 14496-3 Table 4.85). +const ID_PCE: u32 = 5; + +/// The audioObjectTypes whose specific config is a GASpecificConfig (ISO 14496-3 §1.6.2.1), the +/// only ones where channelConfiguration 0 means a program config element follows. +const GENERAL_AUDIO: [u8; 12] = [1, 2, 3, 4, 6, 7, 17, 19, 20, 21, 22, 23]; + +/// Read an audioObjectType: 5 bits, escaped to 6 more when it reads 31. +fn read_object_type(reader: &mut BitReader) -> Result { + let object_type = reader.read(5, Error::ConfigTooShort)? as u8; + if object_type == 31 { + return Ok(32 + reader.read(6, Error::ExtendedConfigTooShort)? as u8); + } + Ok(object_type) +} + +/// Walk a program_config_element (ISO 14496-3 Table 4.2) to the number of channels it outputs. +/// +/// Its byte alignment is relative to where the reader started: the AudioSpecificConfig, or the +/// raw data block. +fn program_config(reader: &mut BitReader) -> Result { + let mut read = |n| reader.read(n, Error::ProgramConfigTruncated); + + // element_instance_tag, object_type, sampling_frequency_index. + read(10)?; + let (front, side, back) = (read(4)?, read(4)?, read(4)?); + let (lfe, assoc, cc) = (read(2)?, read(3)?, read(4)?); + // Mono and stereo mixdowns each name an element; a matrix mixdown an index and a flag. + for bits in [4, 4, 3] { + if read(1)? == 1 { + read(bits)?; + } + } + + let mut channels = lfe; + for _ in 0..front + side + back { + // A channel pair element carries two channels, a single channel element one. + channels += 1 + read(1)?; + read(4)?; + } + // Tags of the LFE and data elements, then each coupling channel element's switch flag and tag. + for _ in 0..lfe + assoc { + read(4)?; + } + for _ in 0..cc { + read(5)?; + } + + reader.align(); + let comment = reader.read(8, Error::ProgramConfigTruncated)?; + for _ in 0..comment { + reader.read(8, Error::ProgramConfigTruncated)?; + } + + if channels == 0 { + return Err(Error::ProgramConfigEmpty); + } + Ok(channels) +} + /// The 13 standard AAC sampling frequencies, indexed by samplingFrequencyIndex /// (ISO 14496-3 Table 1.18). Index 15 is the escape for an explicit 24-bit rate. const SAMPLE_RATES: [u32; 13] = [ @@ -147,6 +288,8 @@ struct BitReader<'a, T: Buf> { buf: &'a mut T, current: u8, bits_left: u8, + /// Every bit read is copied here when set, with alignment redone on the writer's own bytes. + record: Option, } impl<'a, T: Buf> BitReader<'a, T> { @@ -155,6 +298,15 @@ impl<'a, T: Buf> BitReader<'a, T> { buf, current: 0, bits_left: 0, + record: None, + } + } + + /// Skip to the next byte boundary. + fn align(&mut self) { + self.bits_left = 0; + if let Some(record) = &mut self.record { + record.align(); } } @@ -172,27 +324,52 @@ impl<'a, T: Buf> BitReader<'a, T> { self.bits_left -= 1; value = (value << 1) | u32::from((self.current >> self.bits_left) & 1); } + if let Some(record) = &mut self.record { + record.write(n, value); + } Ok(value) } } -/// Map an AAC `channel_config` (ISO 14496-3 Table 1.19) to its real channel count. -/// Configs 1..=6 happen to be identity (5.1 has config=6 and 6 channels). Config -/// 7 is 7.1 = 8 channels. Config 0 means "described elsewhere" — we default to -/// stereo. -fn channel_count_from_config(channel_config: u8) -> u32 { - match channel_config { - 1..=6 => channel_config as u32, - 7 => 8, - 0 => { - tracing::warn!("channel_config=0 (program config element) unsupported, defaulting to stereo"); - 2 - } - _ => { - tracing::warn!(channel_config, "unsupported channel config, defaulting to stereo"); - 2 +/// MSB-first bit writer, the inverse of [`BitReader`]. +#[derive(Default)] +struct BitWriter { + bytes: Vec, + /// Bits used in the last byte; 0 when the next write starts a new one. + used: u8, +} + +impl BitWriter { + /// Write the low `n` bits (n <= 32) of `value`, MSB-first. + fn write(&mut self, n: u8, value: u32) { + for i in (0..n).rev() { + if self.used == 0 { + self.bytes.push(0); + } + let bit = ((value >> i) & 1) as u8; + *self.bytes.last_mut().expect("pushed above") |= bit << (7 - self.used); + self.used = (self.used + 1) % 8; } } + + /// Pad the last byte with zeros. + fn align(&mut self) { + self.used = 0; + } +} + +/// Map an AAC `channel_config` (ISO 14496-3 Table 1.19) to its channel count, for every value but +/// 0, which a program config element describes instead. Configs 1..=6 are identity. 7, 12, and 14 +/// are 8 channels (7.1 with wide fronts, rear surrounds, or front heights), 11 is 6.1, and 13 is +/// 22.2. The rest are reserved. +fn channel_count_from_config(channel_config: u8) -> Result { + match channel_config { + 1..=6 => Ok(channel_config as u32), + 7 | 12 | 14 => Ok(8), + 11 => Ok(7), + 13 => Ok(24), + _ => Err(Error::ReservedChannelConfig(channel_config)), + } } /// Inverse of [`channel_count_from_config`]. Defaults to stereo for unsupported @@ -281,10 +458,177 @@ mod tests { assert_eq!(parsed.channel_count, 8, "7.1 surround should round-trip as 8 channels"); } + /// The AudioSpecificConfig ffmpeg 9.0.1 writes for quad (`-af pan=quad|...`, native `aac` + /// encoder, `.m4a`): channelConfiguration 0 and a program config element with two channel + /// pair elements, front and back, then a "Lavc63.1.101" comment and an SBR sync extension. + const FFMPEG_QUAD_ASC: [u8; 24] = [ + 0x11, 0x80, 0x04, 0xC4, 0x04, 0x00, 0x21, 0x10, 0x0C, 0x4C, 0x61, 0x76, 0x63, 0x36, 0x33, 0x2E, 0x31, 0x2E, + 0x31, 0x30, 0x31, 0x56, 0xE5, 0x00, + ]; + + /// Write a program config element with `front`, `side`, and `back` elements (true for a + /// channel pair) and `lfe` LFE elements, starting at the writer's position. + fn write_pce(out: &mut BitWriter, front: &[bool], side: &[bool], back: &[bool], lfe: u32) { + // element_instance_tag, object_type (LC), sampling_frequency_index (48 kHz). + out.write(4, 0); + out.write(2, 1); + out.write(4, 3); + out.write(4, front.len() as u32); + out.write(4, side.len() as u32); + out.write(4, back.len() as u32); + out.write(2, lfe); + // No data or coupling elements, and no mixdowns. + out.write(3, 0); + out.write(4, 0); + out.write(3, 0); + for (tag, &cpe) in front.iter().chain(side).chain(back).enumerate() { + out.write(1, cpe.into()); + out.write(4, tag as u32); + } + for tag in 0..lfe { + out.write(4, tag); + } + out.align(); + // A one-byte comment. + out.write(8, 1); + out.write(8, b'x'.into()); + } + + /// An AudioSpecificConfig with channelConfiguration 0 around [`write_pce`]'s element. + fn pce_asc(object_type: u8, front: &[bool], side: &[bool], back: &[bool], lfe: u32) -> Vec { + let mut out = BitWriter::default(); + out.write(5, object_type.into()); + out.write(4, 3); + out.write(4, 0); + out.write(3, 0); + write_pce(&mut out, front, side, back, lfe); + out.bytes + } + + #[test] + fn parses_ffmpeg_program_config_element() { + let cfg = Config::parse(&mut FFMPEG_QUAD_ASC.as_slice()).unwrap(); + assert_eq!(cfg.profile, 2); + assert_eq!(cfg.sample_rate, 48_000); + assert_eq!(cfg.channel_count, 4, "two channel pair elements"); + } + + #[test] + fn parses_program_config_element_counts() { + // 5.1: a single and a pair in front, a pair in back, and an LFE. + let asc = pce_asc(2, &[false, true], &[], &[true], 1); + assert_eq!(Config::parse(&mut asc.as_slice()).unwrap().channel_count, 6); + + // Side elements count too: 3/2/2 with no LFE. + let asc = pce_asc(2, &[false, true], &[true], &[true], 0); + assert_eq!(Config::parse(&mut asc.as_slice()).unwrap().channel_count, 7); + } + + #[test] + fn parses_program_config_element_behind_explicit_sbr() { + // audioObjectType 5 (SBR), 24 kHz core, channelConfiguration 0, a 48 kHz extension + // rate, then the core type (LC) whose GASpecificConfig carries the element. + let mut out = BitWriter::default(); + out.write(5, 5); + out.write(4, 6); + out.write(4, 0); + out.write(4, 3); + out.write(5, 2); + out.write(3, 0); + write_pce(&mut out, &[true], &[], &[true], 0); + let cfg = Config::parse(&mut out.bytes.as_slice()).unwrap(); + assert_eq!(cfg.profile, 5); + assert_eq!(cfg.channel_count, 4); + } + + #[test] + fn refuses_bad_program_config_elements() { + let asc = pce_asc(2, &[], &[], &[], 0); + assert!(matches!( + Config::parse(&mut asc.as_slice()), + Err(Error::ProgramConfigEmpty) + )); + + let asc = pce_asc(2, &[true], &[], &[], 0); + assert!(matches!( + Config::parse(&mut &asc[..asc.len() - 1]), + Err(Error::ProgramConfigTruncated) + )); + + // ALS (36) describes its channels in its own specific config, not a PCE. + let mut out = BitWriter::default(); + out.write(5, 31); + out.write(6, 4); + out.write(4, 3); + out.write(4, 0); + out.write(8, 0); + assert!(matches!( + Config::parse(&mut out.bytes.as_slice()), + Err(Error::ProgramConfigUnsupported(36)) + )); + } + + #[test] + fn channel_config_dispositions() { + let expected = [1, 2, 3, 4, 5, 6, 8]; + for (config, count) in (1..=7).zip(expected) { + let asc = [0x11, 0x80 | (config << 3)]; + assert_eq!(Config::parse(&mut asc.as_slice()).unwrap().channel_count, count); + } + + for (config, count) in [(11, 7), (12, 8), (13, 24), (14, 8)] { + let asc = [0x11, 0x80 | (config << 3)]; + assert_eq!(Config::parse(&mut asc.as_slice()).unwrap().channel_count, count); + } + + for config in [8, 9, 10, 15] { + let asc = [0x11, 0x80 | (config << 3)]; + assert!(matches!( + Config::parse(&mut asc.as_slice()), + Err(Error::ReservedChannelConfig(c)) if c == config + )); + } + } + + #[test] + fn in_band_config_moves_the_program_config_element() { + // A raw data block leading with ID_PCE, which puts the element 3 bits off the byte grid + // the AudioSpecificConfig puts it on, so the copy has to redo the alignment. + let mut block = BitWriter::default(); + block.write(3, ID_PCE); + write_pce(&mut block, &[false, true], &[], &[true], 1); + // The channel elements that follow are never read. + block.write(8, 0xFF); + + let asc = in_band_config(2, 48_000, 0, &block.bytes).unwrap(); + assert_eq!(asc, pce_asc(2, &[false, true], &[], &[true], 1)); + assert_eq!(Config::parse(&mut asc.as_ref()).unwrap().channel_count, 6); + } + #[test] - fn channel_config_zero_falls_back_to_stereo() { - // Config 0 means "described in PCE" which we don't implement. - assert_eq!(channel_count_from_config(0), 2); + fn in_band_config_refuses_a_block_without_a_leading_pce() { + // ID_CPE first: any element past the channel data is out of reach. + assert!(matches!( + in_band_config(2, 48_000, 0, &[0x20, 0x00]), + Err(Error::ProgramConfigMissing) + )); + assert!(matches!( + in_band_config(2, 48_000, 0, &[]), + Err(Error::ProgramConfigMissing) + )); + } + + #[test] + fn in_band_config_matches_encode() { + // Without a PCE the block is unread and the config is the plain two-byte form. + let asc = in_band_config(2, 44_100, 2, &[]).unwrap(); + let encoded = Config { + profile: 2, + sample_rate: 44_100, + channel_count: 2, + } + .encode(); + assert_eq!(asc, encoded); } #[test] diff --git a/rs/moq-mux/src/container/ts/adts.rs b/rs/moq-mux/src/container/ts/adts.rs index b5a986a2f3..0166c05937 100644 --- a/rs/moq-mux/src/container/ts/adts.rs +++ b/rs/moq-mux/src/container/ts/adts.rs @@ -21,7 +21,8 @@ pub(super) struct Header { /// audioObjectType (ADTS `profile` + 1). AAC-LC is 2. pub object_type: u8, pub sample_rate: u32, - pub channel_count: u32, + /// channelConfiguration; 0 means a program config element in the raw data block describes the channels. + pub channel_config: u8, /// Total access-unit length, header included. pub frame_len: usize, /// Header length: 7 without CRC, 9 with. @@ -51,7 +52,7 @@ impl Header { Ok(Self { object_type: profile + 1, sample_rate, - channel_count: channel_count_from_config(channel_config), + channel_config, frame_len, header_len, }) @@ -96,16 +97,7 @@ fn freq_index_from_rate(sample_rate: u32) -> anyhow::Result { .with_context(|| format!("sample rate {sample_rate} not representable in ADTS")) } -/// Map an AAC `channel_config` (ISO 14496-3 Table 1.19) to a channel count. -fn channel_count_from_config(channel_config: u8) -> u32 { - match channel_config { - 1..=6 => channel_config as u32, - 7 => 8, - _ => 2, - } -} - -/// Inverse of [`channel_count_from_config`]. +/// Map a channel count to an AAC `channel_config` (ISO 14496-3 Table 1.19). fn channel_config_from_count(channel_count: u32) -> u8 { match channel_count { 1..=6 => channel_count as u8, @@ -126,7 +118,7 @@ mod tests { assert_eq!(parsed.object_type, 2); assert_eq!(parsed.sample_rate, 48_000); - assert_eq!(parsed.channel_count, 2); + assert_eq!(parsed.channel_config, 2); assert_eq!(parsed.header_len, 7); assert_eq!(parsed.frame_len, 107, "frame_len includes the 7-byte header"); } @@ -142,7 +134,7 @@ mod tests { fn frame_len_for_5_1() { let header = write_header(2, 44_100, 6, 512).unwrap(); let parsed = Header::parse(&header).unwrap(); - assert_eq!(parsed.channel_count, 6); + assert_eq!(parsed.channel_config, 6); assert_eq!(parsed.sample_rate, 44_100); assert_eq!(parsed.frame_len, 519); } diff --git a/rs/moq-mux/src/container/ts/import.rs b/rs/moq-mux/src/container/ts/import.rs index d811e24cc1..643069c77d 100644 --- a/rs/moq-mux/src/container/ts/import.rs +++ b/rs/moq-mux/src/container/ts/import.rs @@ -2135,21 +2135,23 @@ impl AacStream { let import = match &mut self.import { Some(import) => import, None => { - let config = aac::Config { - profile: header.object_type, - sample_rate: header.sample_rate, - channel_count: header.channel_count, - }; + // Synthesize the AudioSpecificConfig `description` so out-of-band consumers + // (fMP4/MKV export, WebCodecs) can configure the decoder. A channel_config of 0 + // moves the program config element out of this first frame into it. + let asc = aac::in_band_config( + header.object_type, + header.sample_rate, + header.channel_config, + &data[offset + header.header_len..end], + )?; + let mut config = aac::config(&asc)?; + config.container = self.container.clone(); // Consume the reservation held since the PMT: this resolves the gated rendition, // and carries the catalog's declared media retention onto the track. - // The importer synthesizes the AudioSpecificConfig `description` from the config so - // out-of-band consumers (fMP4/MKV export, WebCodecs) can configure the decoder. let reserved = self.reserved.take().expect("aac reservation already consumed"); let track = self .broadcast .unique_track(".aac", reserved.track_info(hang::catalog::PRIORITY.audio))?; - let mut config: hang::catalog::AudioConfig = config.into(); - config.container = self.container.clone(); let aac = aac::Import::new(track, reserved, config)?; self.import.insert(aac) } diff --git a/rs/moq-mux/src/container/ts/import_test.rs b/rs/moq-mux/src/container/ts/import_test.rs index 348f2cf926..d0c48534ac 100644 --- a/rs/moq-mux/src/container/ts/import_test.rs +++ b/rs/moq-mux/src/container/ts/import_test.rs @@ -156,6 +156,32 @@ fn import_ac3_catalog() { assert!(audio.description.is_none(), "verbatim AC-3 needs no description"); } +/// `aac_quad.ts` is an ffmpeg-authored audio-only AAC program in quad, which has no +/// channelConfiguration, so its ADTS headers carry 0 and the first raw data block leads with a +/// program config element. Regenerated with (ffmpeg 9.0.1): +/// `ffmpeg -f lavfi -i sine=frequency=440:sample_rate=48000:duration=0.1 +/// -af "pan=quad|FL=c0|FR=c0|BL=c0|BR=c0" -c:a aac -b:a 128k -f mpegts aac_quad.ts`. +#[test] +fn import_aac_program_config_catalog() { + let data = include_bytes!("test_data/aac_quad.ts"); + let catalog = import_ts(data); + + assert_eq!(catalog.audio.renditions.len(), 1, "expected one AAC track"); + let audio = catalog.audio.renditions.values().next().unwrap(); + assert_eq!(audio.codec.to_string(), "mp4a.40.2"); + assert_eq!(audio.sample_rate, 48_000); + assert_eq!( + audio.channel_count, 4, + "two channel pair elements, not a guessed stereo" + ); + + // The element moved into the description is byte-for-byte what ffmpeg itself writes as the + // AudioSpecificConfig for the same stream in MP4, minus the trailing SBR sync extension. + let mut expected = vec![0x11, 0x80, 0x04, 0xC4, 0x04, 0x00, 0x21, 0x10, 0x0C]; + expected.extend_from_slice(b"Lavc63.1.101"); + assert_eq!(audio.description.as_deref(), Some(expected.as_slice())); +} + /// `opus.ts` is an ffmpeg-authored audio-only Opus program (private stream_type 0x06 /// plus the 'Opus' registration and DVB extension descriptors), generated with: /// `ffmpeg -f lavfi -i sine=frequency=440:sample_rate=48000:duration=0.5 diff --git a/rs/moq-mux/src/container/ts/test_data/aac_quad.ts b/rs/moq-mux/src/container/ts/test_data/aac_quad.ts new file mode 100644 index 0000000000000000000000000000000000000000..17aff4786200068fa22e9b89662f19d2ce14ce7f GIT binary patch literal 2632 zcmd6oc~nzZ8o*x&5@etu7#5`(Ng_s&%_VhE*-QwKxUk4naAXTh6l4I|B#;EqAuL4< z5-JL5hDE0Aj0<5qKn@^P>0Xgo^1D+$nimvDz#4H10Fty* z004lA5yfowCKCWL8~`e4fVp%{P{?MhW3_1kaY4lWmNu3));6QS#wWn(l$NpEVVdK! zLtT>|J-0*HOm*J0JUoLUa8M9t8%V{u?Ge*ho-{ zTWH&EwqJz(p)$HUESAKRMLW>Tc;186b@kH1#g_${^a|ghs{`T<@r1+4DI$h%)x$gY z77@96dxH6UAvt%lqYGu`k*!x~3VFn1%BTM=7yE|YMK!Ii@C^tO*~so%pu0}ouhuE! zY=}HK+?W_XybtdJF(dRKcRjpEkCkedk7CwL+`^BaF>mOKr8lSE{%m#m^@tB2%*BXq zU&uKpQd;?a=(H=3%=!#fMV#nuo-R(fU{R2C#ZV-1Xl+IeTuCNiPL?;q7FLo8@R@~` z3+qWBMpg{YRQ0kC_WA7fDOE-^lT5fJgfHj0C06;Q*7@ZqZrarRYZ}Ii-%ZGX%`Y<_ zwCk$%6gP5AHpFhzs{9`tp7h;wOHSp@vLN>@qGFwDDp-*#ead;CoT9vhT=koL$|hEX zjK%c*;^k8Jj%C4D`fi)9Hn8wu5y!B)ZGHK)Wm1pj<)`ZDd1XY?sv8n|bHkqY3|MgA8S%9Z!Q*+m zB}(X_BfIQcU4Ifu`!t5iz`;)0w1cum#Ii-auT@Mt7<)EeY?{9N!K?K1IQxPg&>|;h zD9fPIzUX3}W?*QyDN`8CKN_m$l%P8A9e!&j32hR;&%4%GCtuVY`myaTqr8?HIa=_w z8wb2Puv&uIS`;rG#Y~QOEiS?*$C0cAN@qs^wmW z@K0s;=cuR49Kgqnk6ZoKb)vsXrnFaDt%;=Xr>M`ZY$cSG zhYv=bHHF*@{XweF)bNr!iHvq}5+-X_YMR+r-O}u^QTNnikZc`4xzfi{fC|=>5|o*p zt4ti~KK#>I_o!Pd+TG%^c9?4MxlBTakgMr+@+-*QP~T(|0b#2qyf3=y+{SMpMtFZ6iR4hr&!OLFlBv<$JP=qx*CEgjjX@yIIg4v}^f! zJMOoiQI;fI$~@j&cm33zfj=q6G98v?Gn^$Hxiym%oQ-626{rGhJFVSq6c_Olp#-%Z z2OqDQW#WzLLh4)16qIO%RWZbmuZ*T%r)c}QF0fk_@W~nz;}FDfs{(b;p9#4eX=6}Z zgW%&rCh1Z(G8IA&SlhwJMCzTafH^6f_^Nwud1;sLd7M$($aY)>R=;lr%*~*F*R-n( zA5)EAa`?8P_~{j1cymneuiW2au4^tvw%SOClkSh}+;4c;lCs(RWo~TCW@CC$P-*P> zTVUt}ubICPzqwy!(M*3dIH@^<5^5o`)({8X2f9tEG}s;f3H6bMr@Z)#1Ielg?a%Mg^7 zevv9P3){2D>Yl3gI`BzOW)b!)LH}MaiN0Im^BXOn9U8jX6isc*>%tC@fJb-k5Cp>V z)FuY5`Lw}HXd+H<2jaO=e^9mhI`F>*mOr@@cHj^CL^3KnFsGdSE90fhzw#JZE%~;P zz4#-LyX|4%|NqT1Is(Ov4(n?QHx*CgmPZp3UAw7VHE6CHJe#<{JWPy?AAcr)C;3L2 zMrp^kI1)zBIokF~8;$DE6TzBGEODD$-5! Qn0E6##`H6}Y$K`j-^lX>rT_o{ literal 0 HcmV?d00001 From a963040c8883c0dd1c5472d4f2ba03bde4572706 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 05:25:47 -0700 Subject: [PATCH 03/11] chore(quest): abandon he-aac-refusal (#4090) Co-authored-by: Claude Opus 5.5 --- doc/bin/cli.md | 2 ++ doc/lib/rs/moq-audio.md | 4 +++ quest/m1/audio-codecs/README.md | 7 +++--- quest/m1/audio-codecs/decode-backend.md | 6 +++-- quest/m1/audio-codecs/he-aac-refusal.md | 33 ------------------------- 5 files changed, 14 insertions(+), 38 deletions(-) delete mode 100644 quest/m1/audio-codecs/he-aac-refusal.md diff --git a/doc/bin/cli.md b/doc/bin/cli.md index 1376686f9d..cbd4d9d03a 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -90,6 +90,8 @@ moq ... play --delay 500ms # trade latency for a jittery link Decodes H.264, H.265, and AV1 video and Opus, PCM, and AAC-LC audio using the platform hardware decoder where available. `--video-name` and `--audio-name` pick a rendition. +HE-AAC signaled only in band (implicit SBR, as over MPEG-TS) plays as its +half-rate AAC-LC core. Playback runs on a clock it owns. `--delay` (default 100 ms) is how far it trails the live edge, which is both the jitter a late frame may absorb and the diff --git a/doc/lib/rs/moq-audio.md b/doc/lib/rs/moq-audio.md index 82f65a3306..5fee2602e7 100644 --- a/doc/lib/rs/moq-audio.md +++ b/doc/lib/rs/moq-audio.md @@ -27,6 +27,10 @@ policy. Decoding likewise separates low-level `decode::Config`, PCM | `playback` | One output device mixing every track in a call, with click-free volume ramps | | `aec` | Acoustic echo cancellation (a port of WebRTC's), so a laptop with no headset doesn't feed itself back | +AAC decoding refuses HE-AAC its config declares. HE-AAC signaled only in band +(implicit SBR, as over MPEG-TS) goes undetected and plays as its half-rate +AAC-LC core. + Highlights: - **`encode::Publication`** advertises the track and opens the microphone only while someone listens. Stop, swap devices, and restart without changing the track subscribers know; read a level meter for the UI. diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index c1366d831b..6682cbc9c2 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -16,8 +16,10 @@ publisher can produce AAC. Platform first, exactly like video: AudioToolbox on macOS and iOS, Media Foundation on Windows, MediaCodec on Android, and symphonia (AAC-LC mono/stereo) as the software fallback that openh264 is for H.264. Linux has no -OS audio decoder, so HE-AAC and multichannel AAC stay refused there, stated in -the docs and rejected at construction. A platform backend claims every catalog +OS audio decoder, so multichannel AAC stays refused there, stated in the docs +and rejected at construction. HE-AAC signaled only in band (implicit SBR, as +over MPEG-TS) plays as its half-rate LC core on symphonia; detecting it needs a +full element walk, so the docs state it instead of refusing it. A platform backend claims every catalog codec its framework opens, so AC-3, E-AC-3, MP3, and FLAC ride along on the hosts that have them; each still needs a fixture before the backend advertises it. @@ -41,7 +43,6 @@ its own decode and encode quest so verification stays per host. ## Quests -- [HE-AAC refusal](/quest/m1/audio-codecs/he-aac-refusal.md) - implicit-SBR HE-AAC over TS is refused instead of half-decoded as the LC core - [TS export PCE](/quest/m1/audio-codecs/ts-export-pce.md) - a TS export of a PCE-described AAC track writes channel_config 0 and the PCE instead of a count-derived config - [Layout](/quest/m1/audio-codecs/layout.md) - the settled `Layout` carries up to 7.1 through decode, resample, playback, and the FFI - [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - `decode::backend` selects a platform decoder before symphonia, mirroring moq-video diff --git a/quest/m1/audio-codecs/decode-backend.md b/quest/m1/audio-codecs/decode-backend.md index ecfb6f64ba..490f5eac84 100644 --- a/quest/m1/audio-codecs/decode-backend.md +++ b/quest/m1/audio-codecs/decode-backend.md @@ -22,8 +22,10 @@ stats and `moq play` surface. AAC-LC mono/stereo only; the platform backends that follow advertise what their framework opens and has a fixture for. - Move today's Opus, PCM, and symphonia code behind the trait without changing - behavior; the HE-AAC sniff from [HE-AAC refusal](/quest/m1/audio-codecs/he-aac-refusal.md) - lands in the symphonia backend. + behavior. Document per host that symphonia plays implicit-SBR HE-AAC as its + half-rate LC core: finding the in-band SBR element needs a full Huffman walk + of the channel elements, and symphonia detects it internally without + exposing or refusing it. - A backend's output rate and layout are what it produced, not what the catalog said (HE-AAC doubles the rate); `Consumer` already resamples and remixes to the requested output, so that stays the seam's contract. diff --git a/quest/m1/audio-codecs/he-aac-refusal.md b/quest/m1/audio-codecs/he-aac-refusal.md deleted file mode 100644 index c3edf29e21..0000000000 --- a/quest/m1/audio-codecs/he-aac-refusal.md +++ /dev/null @@ -1,33 +0,0 @@ -# [S] Refuse implicit-SBR HE-AAC instead of half-decoding it - -## Goal - -An HE-AAC stream that signals AAC-LC and carries SBR only in band is refused -with a clear error, the same way explicitly signaled and backward-compatible -HE-AAC already are. Today it decodes as the LC core at half the sample rate -with no indication anything is wrong. - -## Plan - -ADTS carries a 2-bit profile, so HE-AAC over MPEG-TS (SRT, `moq import ts`) -always arrives as `mp4a.40.2` with a synthesized LC AudioSpecificConfig. The -config-level checks in `rs/moq-audio/src/aac.rs` cannot see it, and the code -says so. The stream itself can: the first raw data block carries an `ID_FIL` -element with `EXT_SBR_DATA` (or `EXT_SBR_DATA_CRC`), and every later frame -does too. - -- Sniff the fill elements of the first packet before handing it to symphonia. - An SBR extension makes the track `Error::Unsupported` with the same wording - as the config-level refusal, naming the host's decoder as the reason. -- Symphonia already walks the element tree, so the sniff is a small parser - over the same syntax, not a decode. It runs once per track. -- Regression: an ADTS fixture with implicit SBR is refused; the existing - explicit and backward-compatible fixtures keep their errors; an LC fixture - with an unrelated fill element still decodes. -- When a platform decoder handles the track ([Decode seam](/quest/m1/audio-codecs/decode-backend.md)), - the sniff belongs to the symphonia backend only; the OS decoders read SBR in - band themselves. - -## Related - -- [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the sniff becomes the software backend's contract From c0c78568c13f59bdd88eba6295adf7f654df04b9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 07:22:21 -0700 Subject: [PATCH 04/11] feat(audio): carry surround layouts through decode, remix, and playback (#4119) Co-authored-by: Claude Opus 5.5 --- doc/lib/c/index.md | 1 + doc/lib/go/index.md | 6 + doc/lib/kt/index.md | 6 + doc/lib/py/index.md | 6 + doc/lib/rs/moq-audio.md | 10 +- doc/lib/swift/index.md | 6 + js/hang/src/util/aac.test.ts | 8 +- js/hang/src/util/aac.ts | 10 +- quest/m1/audio-codecs/README.md | 1 - quest/m1/audio-codecs/decode-audiotoolbox.md | 1 - quest/m1/audio-codecs/decode-backend.md | 4 - quest/m1/audio-codecs/encode-audiotoolbox.md | 1 - quest/m1/audio-codecs/layout.md | 54 ---- quest/m1/audio-codecs/opus-surround.md | 4 - quest/m2/aac-encode-refusal.md | 1 - quest/m2/audio-decode-mediacodec.md | 1 - quest/m2/audio-decode-mediafoundation.md | 1 - quest/m2/audio-encode-mediacodec.md | 1 - quest/m2/audio-encode-mediafoundation.md | 1 - rs/libmoq/src/audio.rs | 7 +- rs/moq-audio/src/aec.rs | 28 +- rs/moq-audio/src/decode/consumer.rs | 16 +- rs/moq-audio/src/encode/capture.rs | 35 +-- rs/moq-audio/src/encode/encoder.rs | 6 +- rs/moq-audio/src/encode/producer.rs | 16 +- rs/moq-audio/src/layout.rs | 132 ++++++++- rs/moq-audio/src/playback.rs | 7 +- rs/moq-audio/src/playback/device.rs | 120 +++++--- rs/moq-audio/src/playback/driver.rs | 30 +- rs/moq-audio/src/playback/mixer.rs | 205 ++++++++------ rs/moq-audio/src/playback/sink.rs | 170 ++++++++---- rs/moq-audio/src/resample.rs | 271 ++++++++++++++++--- rs/moq-audio/tests/roundtrip.rs | 68 +++++ rs/moq-ffi/src/audio.rs | 7 +- 34 files changed, 901 insertions(+), 340 deletions(-) delete mode 100644 quest/m1/audio-codecs/layout.md diff --git a/doc/lib/c/index.md b/doc/lib/c/index.md index 15de234894..69fec60fbd 100644 --- a/doc/lib/c/index.md +++ b/doc/lib/c/index.md @@ -36,6 +36,7 @@ and `target/include/moq.h`. - **Connection health.** `moq_session_stats()` reports available metrics with per-field validity flags. `moq_session_snapshot()` samples those metrics and the negotiated draft name together from the same connection. Its protocol string is backed by static storage. Both return an offline error between reconnects and leave the destination untouched. `moq_session_bandwidth()` mints an allocator over the send estimate; `moq_bandwidth_reserve` claims a share for an app-owned track, and `moq_encode_video` / `moq_encode_audio` take the same handle so the built-in video encoder follows the grant. - **Raw playback.** Raw audio and video consumers start at the newest cached group when opened, so rebuilding a live decoder skips the retained backlog. - **Raw audio encode.** `moq_audio_encoder_output.codec` names the codec (only `"opus"` today), and `frame_duration_us` sets the Opus frame length: 2500, 5000, 10000, 20000, 40000, or 60000, with 0 meaning the 20 ms default. +- **Audio channel layouts.** A `channels` count also names the speaker layout, by the WAVE convention: 1 is mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved front left, front right, center, LFE, back, then side. `moq_decode_audio` remixes to the count you ask for; past 8 channels the samples pass through but can't be remixed. - **Raw decode output.** `moq_video_decoder_output` selects the decoded CPU pixel format (`MOQ_VIDEO_PIXEL_FORMAT_I420` or `_RGBA`) and target size (`width`/`height`, both zero for native; otherwise even and non-zero). Unknown formats and invalid sizes fail `moq_decode_video` before subscribing; accepted requests deliver exactly that layout or fail on the terminal callback. - **Encoded video metadata.** `moq_video_init.hint` is a zero-initialized `moq_video_hint` with `has_*` flags for coded dimensions, bitrate (bits per second), frame rate, and latency preference. Hints seed a video codec track's catalog; detected dimensions take precedence. - **Client config.** A zeroed `moq_client_config` means the defaults for every knob, which is what lets a new one be appended without disturbing callers. Fields cover protocol (`versions`), TLS (`tls_fingerprints`, `tls_roots`, `tls_cert`/`_key`, `tls_host_name`), transport (`bind`, `connect_timeout_us`, the Happy Eyeballs delays, `websocket_enabled`/`_delay_us`), and tuning (reconnect backoff, `quic_*`). Every duration is in microseconds. A knob whose default isn't zero carries a `has_*` flag, so setting `backoff_timeout_us = 0` needs `has_backoff_timeout = true` to mean "retry forever" rather than "use the default". `moq_client_defaults()` reports what a NULL config dials with. diff --git a/doc/lib/go/index.md b/doc/lib/go/index.md index 001f5a50e4..0540cafb90 100644 --- a/doc/lib/go/index.md +++ b/doc/lib/go/index.md @@ -115,6 +115,12 @@ one: `FetchGroup`/`FetchMediaGroup`, `Dynamic()` with `Requests(ctx)`, `AppendDatagram`/`Datagrams(ctx)`, `SetCatalogSection`, `Demand()` for `Used`/`Unused`, `Session().Stats()`. `moq.IsAuthError` and `moq.IsShutdown` classify errors. `moq.ProtocolError(err)` is the structured protocol failure (scope, verbatim code, kind) when the peer sent one. +Audio `Channels` also names the speaker layout, by the WAVE convention: 1 is +mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved +front left, front right, center, LFE, back, then side. Decoding remixes to the +count you ask for; past 8 channels the samples pass through but can't be +remixed. + `DecodeVideo` picks the decoded CPU pixel layout: `VideoDecoderOutput.Format` is I420 when nil, or `VideoPixelFormatRgba` for four bytes a pixel, and every `VideoDecodedFrame` repeats the layout it was decoded to. `Resize` is best diff --git a/doc/lib/kt/index.md b/doc/lib/kt/index.md index a241ad6b0c..8f863b942b 100644 --- a/doc/lib/kt/index.md +++ b/doc/lib/kt/index.md @@ -94,6 +94,12 @@ native side. `AudioCodec.opus()`, and `AudioEncoderOutput.frameDurationUs` sets the Opus frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. +Audio `channels` also names the speaker layout, by the WAVE convention: 1 is +mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved +front left, front right, center, LFE, back, then side. Decoding remixes to the +count you ask for; past 8 channels the samples pass through but can't be +remixed. + `decodeVideo` picks the decoded CPU pixel layout: `VideoDecoderOutput.format` is `VideoPixelFormat.I420` when null, or `VideoPixelFormat.RGBA` for four bytes a pixel, and every frame repeats the layout it was decoded to. `resize` is best diff --git a/doc/lib/py/index.md b/doc/lib/py/index.md index a41cb7255b..5fe4e12de3 100644 --- a/doc/lib/py/index.md +++ b/doc/lib/py/index.md @@ -110,6 +110,12 @@ WebSocket, TCP, and Unix sockets. `moq.AudioCodec.opus()`, and `AudioEncoderOutput.frame_duration_us` sets the Opus frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. +Audio `channels` also names the speaker layout, by the WAVE convention: 1 is +mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved +front left, front right, center, LFE, back, then side. Decoding remixes to the +count you ask for; past 8 channels the samples pass through but can't be +remixed. + `decode_video` picks the decoded CPU pixel layout: `VideoDecoderOutput.format` is `VideoPixelFormat.I420` when unset, or `VideoPixelFormat.RGBA` for four bytes a pixel, and every frame repeats the layout it was decoded to. `resize` diff --git a/doc/lib/rs/moq-audio.md b/doc/lib/rs/moq-audio.md index 5fee2602e7..2d57b4a5bc 100644 --- a/doc/lib/rs/moq-audio.md +++ b/doc/lib/rs/moq-audio.md @@ -13,8 +13,14 @@ the far end. Everything is Rust, so there is no C toolchain, CMake step, or codec to install. `Layout` names speaker meaning separately from a channel count. `Mono` is center, -`Stereo` is left then right, and `Discrete(n)` preserves unnamed channels without -inventing speaker positions. Encoding keeps source PCM in `encode::Input` and +`Stereo` is left then right, and the surround layouts up to `SevenPointOne` +interleave in the SMPTE/WAVE order (front left, front right, center, LFE, back, +side). A catalog carries only a count, which reads as that count's WAVE default +(`Layout::from_channels`: 6 is 5.1, 8 is 7.1). `Discrete(n)` preserves unnamed +channels without inventing speaker positions, so it passes through but never +remixes. Decoding and playback downmix with the ITU-R BS.775 coefficients and +upmix by leaving the extra speakers silent; the playback mix runs in whatever +layout the output device opened. Encoding keeps source PCM in `encode::Input` and codec requirements in `encode::Settings`; `encode::Options` adds publication policy. Decoding likewise separates low-level `decode::Config`, PCM `decode::Output`, and subscription `decode::Options`. diff --git a/doc/lib/swift/index.md b/doc/lib/swift/index.md index a9aa8b3c44..dc02120fe7 100644 --- a/doc/lib/swift/index.md +++ b/doc/lib/swift/index.md @@ -98,6 +98,12 @@ divides the connection's send estimate; pass it to `encodeVideo` / `AudioCodec.opus()`, and `AudioEncoderOutput.frameDurationUs` sets the Opus frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. +Audio `channels` also names the speaker layout, by the WAVE convention: 1 is +mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved +front left, front right, center, LFE, back, then side. Decoding remixes to the +count you ask for; past 8 channels the samples pass through but can't be +remixed. + `decodeVideo` picks the decoded CPU pixel layout: `VideoDecoderOutput.format` is `.i420` when unset, or `.rgba` for four bytes a pixel, and every frame repeats the layout it was decoded to. `resize` is best effort: only NVDEC has a diff --git a/js/hang/src/util/aac.test.ts b/js/hang/src/util/aac.test.ts index c31059ec80..50dfe903ad 100644 --- a/js/hang/src/util/aac.test.ts +++ b/js/hang/src/util/aac.test.ts @@ -48,9 +48,11 @@ describe("audioSpecificConfig", () => { expect(audioSpecificConfig(48000, 8)).toEqual(new Uint8Array([0x11, 0xb8])); }); - // Unsupported channel counts fall back to stereo (config 2). - it("unsupported channel count falls back to stereo", () => { - expect(audioSpecificConfig(48000, 7)).toEqual(audioSpecificConfig(48000, 2)); + // A count with no channel configuration is refused rather than described as stereo. + it("refuses a channel count with no configuration", () => { + for (const count of [0, 7, 9]) { + expect(() => audioSpecificConfig(48000, count)).toThrow(); + } }); // Non-table sample rates use the 5-byte explicit-frequency form (freqIndex 0xF). diff --git a/js/hang/src/util/aac.ts b/js/hang/src/util/aac.ts index f417e4c0bd..ca0cc2fd45 100644 --- a/js/hang/src/util/aac.ts +++ b/js/hang/src/util/aac.ts @@ -45,19 +45,19 @@ export function pickRate(rate: number): number { const AAC_LC = 2; // audioObjectType for AAC-LC // Map a channel count to its AAC channelConfiguration (ISO 14496-3 Table 1.19). Configs 1..=6 are -// identity (5.1 is config 6 / 6 channels); 8 channels is config 7 (7.1). Anything else has no valid -// config, so fall back to stereo (matching the Rust muxer in rs/moq-mux/src/codec/aac). +// identity (5.1 is config 6 / 6 channels); 8 channels is config 7 (7.1). Anything else has no +// config, and describing it as stereo would mislabel every channel past the first two. function channelConfig(channelCount: number): number { - if (channelCount >= 1 && channelCount <= 6) return channelCount; + if (Number.isInteger(channelCount) && channelCount >= 1 && channelCount <= 6) return channelCount; if (channelCount === 8) return 7; - return 2; + throw new Error(`no AAC channel configuration for ${channelCount} channels`); } /** * Build the AAC-LC AudioSpecificConfig that decoders need when frames are raw (no ADTS header). * * Standard sample rates produce the 2-byte form; non-table rates fall back to the 5-byte form - * with an explicit 24-bit frequency. Mirrors the Rust muxer so JS and Rust agree on the bytes. + * with an explicit 24-bit frequency. Throws for a channel count no configuration names. */ export function audioSpecificConfig(sampleRate: number, channelCount: number): Uint8Array { const config = channelConfig(channelCount); diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index 6682cbc9c2..78912f392c 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -44,7 +44,6 @@ its own decode and encode quest so verification stays per host. ## Quests - [TS export PCE](/quest/m1/audio-codecs/ts-export-pce.md) - a TS export of a PCE-described AAC track writes channel_config 0 and the PCE instead of a count-derived config -- [Layout](/quest/m1/audio-codecs/layout.md) - the settled `Layout` carries up to 7.1 through decode, resample, playback, and the FFI - [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - `decode::backend` selects a platform decoder before symphonia, mirroring moq-video - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - mapping family 1 decodes on every host through the multistream decoder diff --git a/quest/m1/audio-codecs/decode-audiotoolbox.md b/quest/m1/audio-codecs/decode-audiotoolbox.md index effbebb809..e81db0d554 100644 --- a/quest/m1/audio-codecs/decode-audiotoolbox.md +++ b/quest/m1/audio-codecs/decode-audiotoolbox.md @@ -32,7 +32,6 @@ binding, alongside the `objc2-core-audio-types` the crate already carries. ## Required - [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the candidate order this backend joins -- [Layout](/quest/m1/audio-codecs/layout.md) - what a multichannel frame is delivered as ## Related diff --git a/quest/m1/audio-codecs/decode-backend.md b/quest/m1/audio-codecs/decode-backend.md index 490f5eac84..271ef84b52 100644 --- a/quest/m1/audio-codecs/decode-backend.md +++ b/quest/m1/audio-codecs/decode-backend.md @@ -39,7 +39,3 @@ stats and `moq play` surface. backend like the video seam's `probe`. The FFI does not expose `Kind` until a consumer asks. - -## Related - -- [Layout](/quest/m1/audio-codecs/layout.md) - independent; the platform backends need both diff --git a/quest/m1/audio-codecs/encode-audiotoolbox.md b/quest/m1/audio-codecs/encode-audiotoolbox.md index 0f86953835..2ceda88280 100644 --- a/quest/m1/audio-codecs/encode-audiotoolbox.md +++ b/quest/m1/audio-codecs/encode-audiotoolbox.md @@ -22,5 +22,4 @@ the encode seam as the platform candidate on macOS and iOS. ## Required - [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - the candidate order this backend joins -- [Layout](/quest/m1/audio-codecs/layout.md) - the input layout the encoder accepts - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - the round-trip regression decodes through it diff --git a/quest/m1/audio-codecs/layout.md b/quest/m1/audio-codecs/layout.md deleted file mode 100644 index e92dfbbe38..0000000000 --- a/quest/m1/audio-codecs/layout.md +++ /dev/null @@ -1,54 +0,0 @@ -# [M] Channel layouts through decode, playback, and the FFI - -## Goal - -`moq-audio` carries up to 7.1 end to end. A decoded frame says which layout -it is in, the playback mixer downmixes it to whatever the device opened, a raw -PCM consumer can ask for a layout, and the FFI and libmoq expose the same. -Proven with multichannel PCM, the one codec that needs no new decoder. - -## Plan - -Extend the Layout contract settled in main with supported surround layouts -(2.1, quad, 5.0, 5.1, 6.1, 7.1, and the other AAC channelConfiguration and -Opus mapping family 1 entries) in one canonical order, the SMPTE/WAVE order. -Keep the representation extensible. Each codec module maps its native -order into it: AAC's `C L R Ls Rs LFE` and Opus's Vorbis `L C R Ls Rs LFE` -both become `L R C LFE Ls Rs`. An unspecified discrete PCM layout remains -valid for passthrough but is refused for spatial remixing; never invent speaker -positions from an arbitrary channel count. An AAC stream with -channelConfiguration 0 takes its layout from the program config element's -front, side, back, and LFE elements, which `moq_mux::codec::aac` already parses -for the channel count. - -- Reuse the settled PCM descriptors and codec/consumer settings. This quest - adds supported layout behavior rather than replacing public field types. - Frame stays layout-free since the consumer fixed it at construction. -- `resample::remix` becomes a generic remix over layouts: ITU-R BS.775 - coefficients for downmix, silence in the extra speakers for upmix, and the - existing mono/stereo paths as the two-channel special cases. The resampler is - already channel-generic. -- Playback: the mix bus takes the device's layout instead of fixed stereo, the - device chooser prefers the widest well-known layout the device offers (a - 5.1 HDMI sink opens at six channels, a headset at two), and each `Sink` - remixes into the bus. Today's "silence past the front pair" fan-out goes. -- moq-ffi and libmoq keep `channels` as a count, and the count means the - default layout for that count (the WAVE convention: 3 is 2.1, 4 is quad, 6 - is 5.1, 8 is 7.1), delivered or accepted in the canonical order. No record, - `repr(C)` struct, or binding changes, so this stays on `main`; only the - Rust API names the layout, and a binding that needs 5.0 rather than 5.1 is - a later additive field. Document the mapping in every binding's audio doc. -- The catalog does not change: `channel_count` already carries what the - description implies, and `js/hang`'s `aac.ts` stops falling back to stereo - for a count it cannot map. -- Regressions: a 5.1 PCM broadcast published through `encode::Producer` and - read back through `decode::Consumer` at 5.1, at stereo, and at mono; the - mixer fed a 5.1 sink into a stereo bus and a stereo sink into a 5.1 bus; the - device chooser picking six channels when offered. - -Capture stays mono/stereo, and Opus encode stays mapping family 0. - -## Related - -- [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - the first coded multichannel consumer of the layout -- [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - the first platform decoder producing more than stereo diff --git a/quest/m1/audio-codecs/opus-surround.md b/quest/m1/audio-codecs/opus-surround.md index de6247ae0c..059552ddde 100644 --- a/quest/m1/audio-codecs/opus-surround.md +++ b/quest/m1/audio-codecs/opus-surround.md @@ -20,7 +20,3 @@ so this is the one multichannel path Linux gets. channels. - Regression: a family-1 5.1 fixture decodes to six channels in canonical order; family 255 is refused at construction. - -## Required - -- [Layout](/quest/m1/audio-codecs/layout.md) - the type the mapping resolves to diff --git a/quest/m2/aac-encode-refusal.md b/quest/m2/aac-encode-refusal.md index 3008bf0702..316e7f72e2 100644 --- a/quest/m2/aac-encode-refusal.md +++ b/quest/m2/aac-encode-refusal.md @@ -19,4 +19,3 @@ every count from 1 to 8 and one beyond. ## Related - [AAC PCE](https://github.com/moq-dev/moq/pull/4093) - the parse half -- [Layout](/quest/m1/audio-codecs/layout.md) - the layout a PCE would be derived from diff --git a/quest/m2/audio-decode-mediacodec.md b/quest/m2/audio-decode-mediacodec.md index 936bc13a32..5b7df87c71 100644 --- a/quest/m2/audio-decode-mediacodec.md +++ b/quest/m2/audio-decode-mediacodec.md @@ -26,7 +26,6 @@ behind a new optional audio `mediacodec` feature and the decode seam, on `target ## Required - [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the candidate order this backend joins -- [Layout](/quest/m1/audio-codecs/layout.md) - what a multichannel frame is delivered as ## Related diff --git a/quest/m2/audio-decode-mediafoundation.md b/quest/m2/audio-decode-mediafoundation.md index 20a2518458..1da6a8337c 100644 --- a/quest/m2/audio-decode-mediafoundation.md +++ b/quest/m2/audio-decode-mediafoundation.md @@ -25,7 +25,6 @@ ones. Behind the decode seam as the first candidate on `target_os = ## Required - [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the candidate order this backend joins -- [Layout](/quest/m1/audio-codecs/layout.md) - what a multichannel frame is delivered as ## Related diff --git a/quest/m2/audio-encode-mediacodec.md b/quest/m2/audio-encode-mediacodec.md index 7bb26674ea..d488ec184b 100644 --- a/quest/m2/audio-encode-mediacodec.md +++ b/quest/m2/audio-encode-mediacodec.md @@ -21,5 +21,4 @@ behind the `mediacodec` feature and the encode seam. ## Required - [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - the candidate order this backend joins -- [Layout](/quest/m1/audio-codecs/layout.md) - the input layout the encoder accepts - [MediaCodec decode](/quest/m2/audio-decode-mediacodec.md) - the round-trip regression decodes through it diff --git a/quest/m2/audio-encode-mediafoundation.md b/quest/m2/audio-encode-mediafoundation.md index b1c3440f6b..8347dd1854 100644 --- a/quest/m2/audio-encode-mediafoundation.md +++ b/quest/m2/audio-encode-mediafoundation.md @@ -21,7 +21,6 @@ behind the encode seam on Windows. ## Required - [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - the candidate order this backend joins -- [Layout](/quest/m1/audio-codecs/layout.md) - the input layout the encoder accepts - [Media Foundation decode](/quest/m2/audio-decode-mediafoundation.md) - the round-trip regression decodes through it ## Related diff --git a/rs/libmoq/src/audio.rs b/rs/libmoq/src/audio.rs index b23a57c18e..d389abc0fe 100644 --- a/rs/libmoq/src/audio.rs +++ b/rs/libmoq/src/audio.rs @@ -67,6 +67,9 @@ pub struct moq_audio_encoder_input { /// `moq_audio_sample_format` discriminant. pub format: u32, pub sample_rate: u32, + /// Interleaved channel count, which also names the speaker layout by the + /// WAVE convention: 1 mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, + /// 8 7.1, in front left, front right, center, LFE, back, side order. pub channels: u32, } @@ -98,7 +101,9 @@ pub struct moq_audio_decoder_output { pub format: u32, /// 0 = deliver at the codec's native sample rate. pub sample_rate: u32, - /// 0 = deliver at the codec's native channel count. + /// 0 = deliver at the codec's native channel count. A count names its + /// layout as `moq_audio_encoder_input.channels` describes, and the decoder + /// remixes to it. pub channels: u32, /// Upper bound on buffering before skipping a stalled group, in /// microseconds. Same congestion-control knob as diff --git a/rs/moq-audio/src/aec.rs b/rs/moq-audio/src/aec.rs index 607d0111ca..da24dda60a 100644 --- a/rs/moq-audio/src/aec.rs +++ b/rs/moq-audio/src/aec.rs @@ -40,7 +40,7 @@ use sonora::config::{EchoCanceller, GainController2, NoiseSuppression, NoiseSupp use sonora::{AudioProcessing, StreamConfig}; use crate::Error; -use crate::playback::{self, BUS_CHANNELS}; +use crate::playback::{self, REFERENCE_CHANNELS}; /// Sample rate the echo reference is resampled to on its way out of the mixer. /// @@ -187,7 +187,7 @@ impl Control { "echo cancellation needs a microphone between 8 and 384 kHz (got {sample_rate})" ))); } - if channels == 0 || channels > BUS_CHANNELS as u32 { + if channels == 0 || channels > REFERENCE_CHANNELS as u32 { return Err(Error::Unsupported(format!( "echo cancellation accepts a mono or stereo microphone (got {channels} channels)" ))); @@ -381,9 +381,9 @@ impl State { self.pending = Vec::with_capacity(headroom); self.processed = Vec::with_capacity(headroom); - self.reference_frame = vec![0.0; REFERENCE_FRAME * BUS_CHANNELS]; - self.render_in = vec![0.0; REFERENCE_FRAME * BUS_CHANNELS]; - self.render_out = vec![0.0; REFERENCE_FRAME * BUS_CHANNELS]; + self.reference_frame = vec![0.0; REFERENCE_FRAME * REFERENCE_CHANNELS]; + self.render_in = vec![0.0; REFERENCE_FRAME * REFERENCE_CHANNELS]; + self.render_out = vec![0.0; REFERENCE_FRAME * REFERENCE_CHANNELS]; self.capture_in = vec![0.0; frame * channels]; self.capture_out = vec![0.0; frame * channels]; @@ -453,7 +453,7 @@ impl State { if let Some(reference) = reference { while reference.available_frames() >= REFERENCE_FRAME { reference.read_interleaved(reference_frame, false); - deinterleave(reference_frame, render_in, BUS_CHANNELS); + deinterleave(reference_frame, render_in, REFERENCE_CHANNELS); let _ = process_render(processor, render_in, render_out); } } @@ -484,7 +484,7 @@ impl State { /// The format the mixer's reference tap arrives in. fn reference_config() -> StreamConfig { - StreamConfig::new(REFERENCE_RATE, BUS_CHANNELS as u16) + StreamConfig::new(REFERENCE_RATE, REFERENCE_CHANNELS as u16) } /// Feed one reference frame to the echo model. @@ -602,7 +602,7 @@ impl Reference { /// canceller reads it back at [`REFERENCE_RATE`]. fn channel(rate: u32) -> (ResamplingProd, ResamplingCons) { resampling_channel::( - BUS_CHANNELS, + REFERENCE_CHANNELS, rate, REFERENCE_RATE, true, @@ -652,7 +652,7 @@ mod tests { /// Push one 10 ms stereo frame of `value` into the tap. fn play(prod: &mut ResamplingProd, value: f32) { - prod.push_interleaved(&vec![value; REFERENCE_FRAME * BUS_CHANNELS]); + prod.push_interleaved(&vec![value; REFERENCE_FRAME * REFERENCE_CHANNELS]); } #[test] @@ -976,10 +976,10 @@ mod tests { fn round(&mut self) -> (f64, f64) { let played: Vec = (0..self.frame).map(|_| self.noise.next()).collect(); - let mut reference = vec![0.0f32; self.frame * BUS_CHANNELS]; + let mut reference = vec![0.0f32; self.frame * REFERENCE_CHANNELS]; for (i, sample) in played.iter().enumerate() { - reference[i * BUS_CHANNELS] = *sample; - reference[i * BUS_CHANNELS + 1] = *sample; + reference[i * REFERENCE_CHANNELS] = *sample; + reference[i * REFERENCE_CHANNELS + 1] = *sample; } self.prod.push_interleaved(&reference); @@ -1112,13 +1112,13 @@ mod tests { let mut tone = Vec::with_capacity(frames * 2 * 4); for frame in 0..frames { let value = (std::f32::consts::TAU * 440.0 * frame as f32 / 48_000.0).sin() * 0.5; - for _ in 0..BUS_CHANNELS { + for _ in 0..REFERENCE_CHANNELS { tone.extend_from_slice(&value.to_le_bytes()); } } let mut energy = 0.0f64; - let mut buf = vec![0.0f32; REFERENCE_FRAME * BUS_CHANNELS]; + let mut buf = vec![0.0f32; REFERENCE_FRAME * REFERENCE_CHANNELS]; for _ in 0..20 { let _ = sink.write(&tone).expect("write"); diff --git a/rs/moq-audio/src/decode/consumer.rs b/rs/moq-audio/src/decode/consumer.rs index 0f7ee37fb6..5110db70e5 100644 --- a/rs/moq-audio/src/decode/consumer.rs +++ b/rs/moq-audio/src/decode/consumer.rs @@ -5,7 +5,7 @@ use std::collections::VecDeque; use bytes::Bytes; use super::decoder::{Config, Decoder}; -use crate::resample::{Resampler, remix, validate_remix}; +use crate::resample::{Remix, Resampler}; use crate::{Activity, Error, Format, Frame, Layout}; /// Where a consumer starts on a track that already holds groups. @@ -63,6 +63,8 @@ pub struct Consumer { decoder: Decoder, track: moq_mux::container::Consumer, resampler: Option, + /// Converts the decoded layout to the output's, when they differ. + remix: Option, options: Options, max_age: std::time::Duration, resolved_sample_rate: u32, @@ -111,7 +113,9 @@ impl Consumer { let decoder = Decoder::new(catalog, &options.decoder)?; let sample_rate = options.output.sample_rate.unwrap_or_else(|| decoder.sample_rate()); let layout = options.output.layout.unwrap_or_else(|| decoder.layout()); - validate_remix(decoder.layout(), layout)?; + let remix = (decoder.layout() != layout) + .then(|| Remix::new(decoder.layout(), layout)) + .transpose()?; let resampler = if sample_rate == decoder.sample_rate() { None @@ -165,6 +169,7 @@ impl Consumer { decoder, track, resampler, + remix, options, max_age, resolved_sample_rate: sample_rate, @@ -438,10 +443,9 @@ impl Consumer { /// Remix and pack decoded PCM into an output frame. fn frame(&self, pcm: Vec, timestamp: moq_net::Timestamp, activity: Activity) -> Result { - let pcm = if self.decoder.layout() == self.resolved_layout { - pcm - } else { - remix(&pcm, self.decoder.layout(), self.resolved_layout)? + let pcm = match &self.remix { + Some(remix) => remix.process(&pcm), + None => pcm, }; let bytes = self diff --git a/rs/moq-audio/src/encode/capture.rs b/rs/moq-audio/src/encode/capture.rs index e166e19f1a..32d89909be 100644 --- a/rs/moq-audio/src/encode/capture.rs +++ b/rs/moq-audio/src/encode/capture.rs @@ -12,7 +12,7 @@ use moq_mux::catalog::hang::CatalogExt; use super::producer::Reserved; use super::{Input, Options, Producer}; use crate::capture; -use crate::resample::{Resampler, remix, validate_remix}; +use crate::resample::{Remix, Resampler}; use crate::{Error, Format, Frame, Layout as PcmLayout}; /// Backoff bounds for reopening a capture source. The quick first retry covers @@ -445,7 +445,7 @@ impl Driver { None => continue, }, }; - let pcm_layout = match PcmLayout::from_channels(layout.channels) { + let pcm_layout = match device_layout(layout.channels) { Ok(layout) => layout, Err(err) => match self.failed(err, track, desired.revision).await { Some(result) => return Some(result), @@ -891,23 +891,29 @@ impl EncoderOutput<'_, E> { } } +/// The layout of a capture device, which reports only a channel count. That names +/// speakers for mono and stereo alone: a six-channel microphone array is not a +/// 5.1 speaker layout, so it stays discrete rather than being remixed as one. +fn device_layout(channels: u32) -> Result { + match channels { + 0..=2 => PcmLayout::from_channels(channels), + channels => Ok(PcmLayout::Discrete(channels)), + } +} + /// Converts one opened stream's native layout into the producer's fixed input /// layout. A new instance per open keeps filter state out of recovery gaps. struct Converter { - input: capture::Layout, - output: capture::Layout, + remix: Option, resampler: Option, anchor_us: Option, } impl Converter { fn new(input: capture::Layout, output: capture::Layout) -> Result { - if input.channels != output.channels { - validate_remix( - PcmLayout::from_channels(input.channels)?, - PcmLayout::from_channels(output.channels)?, - )?; - } + let remix = (input.channels != output.channels) + .then(|| Remix::new(device_layout(input.channels)?, device_layout(output.channels)?)) + .transpose()?; let resampler = if input.sample_rate == output.sample_rate { None @@ -924,8 +930,7 @@ impl Converter { }; Ok(Self { - input, - output, + remix, resampler, anchor_us: None, }) @@ -956,10 +961,8 @@ impl Converter { let data = resampler.process(&samples.data, moq_net::Timestamp::from_micros(timestamp_us)?)?; samples.replace(data); } - if self.input.channels != self.output.channels { - let input = PcmLayout::from_channels(self.input.channels)?; - let output = PcmLayout::from_channels(self.output.channels)?; - let data = remix(&samples.data, input, output)?; + if let Some(remix) = &self.remix { + let data = remix.process(&samples.data); samples.replace(data); } if samples.data.is_empty() { diff --git a/rs/moq-audio/src/encode/encoder.rs b/rs/moq-audio/src/encode/encoder.rs index ca2045a633..90479471db 100644 --- a/rs/moq-audio/src/encode/encoder.rs +++ b/rs/moq-audio/src/encode/encoder.rs @@ -883,8 +883,10 @@ mod tests { assert!(matches!(Encoder::new(&settings), Err(Error::Unsupported(_)))); } + /// The catalog carries only a count, so a discrete layout comes back as the + /// count's default one, with the samples untouched. #[test] - fn pcm_preserves_discrete_multichannel_layout() { + fn pcm_passes_discrete_multichannel_samples_through() { let settings = Settings { codec: Codec::Pcm, ..Settings::new(48_000, Layout::Discrete(3)) @@ -895,7 +897,7 @@ mod tests { let input = [0.1, 0.2, 0.3].repeat(encoder.frame_size()); let output = decoder.decode(&encoder.encode(&input).unwrap().payload).unwrap(); - assert_eq!(decoder.layout(), Layout::Discrete(3)); + assert_eq!(decoder.layout(), Layout::TwoPointOne); assert_eq!(output.samples, input); } diff --git a/rs/moq-audio/src/encode/producer.rs b/rs/moq-audio/src/encode/producer.rs index dcc3d1c956..1b61545141 100644 --- a/rs/moq-audio/src/encode/producer.rs +++ b/rs/moq-audio/src/encode/producer.rs @@ -10,7 +10,7 @@ use moq_net::Timestamp; use super::encoded::Encoded; use super::encoder::{Encoder, Input, Settings}; -use crate::resample::{Resampler, remix, validate_remix}; +use crate::resample::{Remix, Resampler}; use crate::{Activity, Error, Frame}; /// Encode and publication policy for [`Producer`]. @@ -64,6 +64,8 @@ impl Default for Options { pub struct Producer { encoder: Encoder, input: Input, + /// Converts the input layout to the codec's, when they differ. + remix: Option, resampler: Option, track: moq_mux::container::Producer, _ext: std::marker::PhantomData E>, @@ -140,7 +142,9 @@ impl Reserved { /// Separate from [`encode`](Self::encode), which cannot fail, so a layout the /// codec rejects leaves the reservation intact for another input. pub(crate) fn register(&mut self, input: Input, options: &Options) -> Result { - validate_remix(input.layout, options.settings.layout)?; + let remix = (input.layout != options.settings.layout) + .then(|| Remix::new(input.layout, options.settings.layout)) + .transpose()?; let encoder = Encoder::new(&options.settings)?; let resampler = if input.sample_rate == encoder.codec_rate() { @@ -163,6 +167,7 @@ impl Reserved { Ok(Registered { encoder, input, + remix, resampler, }) } @@ -172,6 +177,7 @@ impl Reserved { Producer { encoder: registered.encoder, input: registered.input, + remix: registered.remix, resampler: registered.resampler, track: self.track, _ext: self._ext, @@ -193,6 +199,7 @@ impl Reserved { pub(crate) struct Registered { encoder: Encoder, input: Input, + remix: Option, resampler: Option, } @@ -337,7 +344,10 @@ impl Producer { let input = &self.input; let (format, channels) = (input.format, input.layout.channels()); let pcm = format.as_interleaved_f32(frame.data.as_ref(), channels)?; - let pcm = remix(&pcm, input.layout, self.encoder.settings().layout)?; + let pcm = match &self.remix { + Some(remix) => remix.process(&pcm), + None => pcm.into_owned(), + }; let pcm: Vec = match self.resampler.as_mut() { Some(r) => r.process(&pcm, frame.timestamp)?, None => pcm, diff --git a/rs/moq-audio/src/layout.rs b/rs/moq-audio/src/layout.rs index 0ff4d0d09b..90373ed715 100644 --- a/rs/moq-audio/src/layout.rs +++ b/rs/moq-audio/src/layout.rs @@ -1,6 +1,11 @@ use crate::Error; /// Speaker meaning and interleaving order for PCM channels. +/// +/// Named layouts interleave in the SMPTE/WAVE order (front left, front right, +/// center, LFE, back left, back right, back center, side left, side right), +/// keeping only the speakers the layout has. Every decoder reorders its codec's +/// native order into this one. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] #[non_exhaustive] pub enum Layout { @@ -10,11 +15,47 @@ pub enum Layout { #[default] Stereo, /// Channels with no declared speaker positions, in source order. + /// + /// Passes through unchanged but can't be remixed into another layout. Discrete(u32), + /// 2.1: left, right, LFE. + TwoPointOne, + /// 3.0: left, right, center. + ThreePointZero, + /// Quad: left, right, back left, back right. + Quad, + /// 4.0: left, right, center, back center. + FourPointZero, + /// 5.0: left, right, center, side left, side right. + FivePointZero, + /// 5.1: left, right, center, LFE, side left, side right. + FivePointOne, + /// 6.1: left, right, center, LFE, back center, side left, side right. + SixPointOne, + /// 7.1: left, right, center, LFE, back left, back right, side left, side right. + SevenPointOne, +} + +/// One speaker position, declared in the canonical interleaving order. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum Speaker { + FrontLeft, + FrontRight, + FrontCenter, + Lfe, + BackLeft, + BackRight, + BackCenter, + SideLeft, + SideRight, } impl Layout { - /// Infer today's conventional layout from a channel count. + /// The default layout for a channel count, by the WAVE convention: 1 is mono, + /// 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1. + /// + /// Any other count is [`Discrete`](Self::Discrete), since there is no + /// convention to take speaker positions from. pub fn from_channels(channels: u32) -> Result { match channels { 0 => Err(Error::Unsupported( @@ -22,6 +63,12 @@ impl Layout { )), 1 => Ok(Self::Mono), 2 => Ok(Self::Stereo), + 3 => Ok(Self::TwoPointOne), + 4 => Ok(Self::Quad), + 5 => Ok(Self::FivePointZero), + 6 => Ok(Self::FivePointOne), + 7 => Ok(Self::SixPointOne), + 8 => Ok(Self::SevenPointOne), channels => Ok(Self::Discrete(channels)), } } @@ -29,12 +76,40 @@ impl Layout { /// Number of interleaved channels in this layout. pub fn channels(self) -> u32 { match self { - Self::Mono => 1, - Self::Stereo => 2, Self::Discrete(channels) => channels, + layout => layout.speakers().map_or(0, <[_]>::len) as u32, } } + /// The speaker each channel feeds, in interleaved order, or `None` for a + /// discrete layout. + pub(crate) fn speakers(self) -> Option<&'static [Speaker]> { + use Speaker::*; + + Some(match self { + Self::Mono => &[FrontCenter], + Self::Stereo => &[FrontLeft, FrontRight], + Self::Discrete(_) => return None, + Self::TwoPointOne => &[FrontLeft, FrontRight, Lfe], + Self::ThreePointZero => &[FrontLeft, FrontRight, FrontCenter], + Self::Quad => &[FrontLeft, FrontRight, BackLeft, BackRight], + Self::FourPointZero => &[FrontLeft, FrontRight, FrontCenter, BackCenter], + Self::FivePointZero => &[FrontLeft, FrontRight, FrontCenter, SideLeft, SideRight], + Self::FivePointOne => &[FrontLeft, FrontRight, FrontCenter, Lfe, SideLeft, SideRight], + Self::SixPointOne => &[FrontLeft, FrontRight, FrontCenter, Lfe, BackCenter, SideLeft, SideRight], + Self::SevenPointOne => &[ + FrontLeft, + FrontRight, + FrontCenter, + Lfe, + BackLeft, + BackRight, + SideLeft, + SideRight, + ], + }) + } + pub(crate) fn validate(self) -> Result<(), Error> { if self.channels() == 0 { return Err(Error::Unsupported( @@ -44,3 +119,54 @@ impl Layout { Ok(()) } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn counts_map_to_the_wave_defaults() { + let expected = [ + Layout::Mono, + Layout::Stereo, + Layout::TwoPointOne, + Layout::Quad, + Layout::FivePointZero, + Layout::FivePointOne, + Layout::SixPointOne, + Layout::SevenPointOne, + ]; + for (count, layout) in (1..).zip(expected) { + assert_eq!(Layout::from_channels(count).unwrap(), layout); + assert_eq!(layout.channels(), count); + } + + assert_eq!(Layout::from_channels(9).unwrap(), Layout::Discrete(9)); + assert!(Layout::from_channels(0).is_err()); + } + + /// Canonical order is the WAVE bit order, so every layout's speakers must + /// ascend through [`Speaker`]'s declaration order without repeating. + #[test] + fn speakers_follow_the_canonical_order() { + let layouts = [ + Layout::Mono, + Layout::Stereo, + Layout::TwoPointOne, + Layout::ThreePointZero, + Layout::Quad, + Layout::FourPointZero, + Layout::FivePointZero, + Layout::FivePointOne, + Layout::SixPointOne, + Layout::SevenPointOne, + ]; + for layout in layouts { + let speakers = layout.speakers().unwrap(); + assert!( + speakers.windows(2).all(|pair| (pair[0] as u8) < (pair[1] as u8)), + "{layout:?} is out of order" + ); + } + } +} diff --git a/rs/moq-audio/src/playback.rs b/rs/moq-audio/src/playback.rs index 49e81de354..1f5e514a68 100644 --- a/rs/moq-audio/src/playback.rs +++ b/rs/moq-audio/src/playback.rs @@ -50,7 +50,7 @@ pub use sink::{Control, Input, Sink, Write}; #[cfg(feature = "aec")] pub(crate) use driver::Shared; #[cfg(feature = "aec")] -pub(crate) use mixer::BUS_CHANNELS; +pub(crate) use mixer::REFERENCE_CHANNELS; use crate::Error; @@ -137,13 +137,14 @@ impl Engine { /// Add a stream to the mix, taking PCM in the layout `input` describes. /// /// Independent of the device: several sinks can play at different rates and - /// channel counts, and each is resampled on its way to the mix. One device + /// layouts, and each is resampled and remixed to the device's on its way to + /// the mix. One device /// mixes up to 64 of them, past which this returns an error rather than /// handing back a sink that plays nothing. pub fn sink(&self, input: Input) -> Result { let sink = self .shared - .add(|id, rate| sink::new(id, rate, input, self.shared.clone(), self.handle.clone()))?; + .add(|id, rate, bus| sink::new(id, rate, bus, input, self.shared.clone(), self.handle.clone()))?; // Covers the case where the mixer's command queue was momentarily full, // so a sink is never left silently unmixed. diff --git a/rs/moq-audio/src/playback/device.rs b/rs/moq-audio/src/playback/device.rs index 8d55a28a42..680e6a80f7 100644 --- a/rs/moq-audio/src/playback/device.rs +++ b/rs/moq-audio/src/playback/device.rs @@ -11,10 +11,9 @@ use crate::Error; /// second choice. const RATES: &[u32] = &[48_000, 44_100]; -/// Channel counts to try, best first. The mixer produces stereo, and mono is -/// the only other count worth naming: anything else is a surround layout we -/// would be guessing the speaker order of. -const CHANNELS: &[u16] = &[2, 1]; +/// The widest channel count with a well-known layout. Wider counts name no +/// speaker positions to mix into, so they are never opened. +const MAX_CHANNELS: u16 = 8; /// Sample formats we can write, best first: `f32` is what the mixer produces, /// and the rest are conversions on the way out. @@ -129,16 +128,24 @@ pub(super) fn open(selector: Option<&str>) -> Result { /// Pick the stream format to open `device` with. /// -/// Only considers formats in [`FORMATS`], and prefers in that order: a channel -/// count in [`CHANNELS`], a rate the pipeline already runs at, then a format we -/// write without converting. Failing all of those it takes the highest rate the -/// device supports, since resampling down is kinder than resampling up. +/// Only considers formats in [`FORMATS`] and well-known layouts, and prefers in +/// that order: the device's own channel count, the widest layout, a rate the +/// pipeline already runs at, then a format we write without converting. Failing +/// the rates it takes the highest the device supports, since resampling down is +/// kinder than resampling up. pub(super) fn negotiate(device: &cpal::Device) -> Result { let supported = device .supported_output_configs() .map_err(|err| Error::Playback(format!("cannot enumerate output configs: {err}")))?; - choose(supported).ok_or_else(|| Error::Unsupported("output device offers no sample format we can write".into())) + // What the device is set up for: a 5.1 HDMI sink reports six channels and a + // headset two. Taken first because a sound server's plugin also accepts + // every count up to its limit, where the widest would upmix a headset. + let native = device.default_output_config().ok().map(|config| config.channels()); + + choose(supported, native).ok_or_else(|| { + Error::Unsupported("output device offers no sample format and speaker layout we can write".into()) + }) } /// Pick the best of the stream configurations a device reports. @@ -146,10 +153,14 @@ pub(super) fn negotiate(device: &cpal::Device) -> Result) -> Option { +fn choose( + supported: impl Iterator, + native: Option, +) -> Option { supported .filter(|config| FORMATS.contains(&config.sample_format())) - .min_by_key(preference) + .filter(|config| (1..=MAX_CHANNELS).contains(&config.channels())) + .min_by_key(|config| preference(config, native)) .map(|config| match preferred_rate(&config) { Some(rate) => config.try_with_sample_rate(rate).expect("a rate the range covers"), None => config.with_max_sample_rate(), @@ -164,11 +175,11 @@ fn choose(supported: impl Iterator) -> /// and a pass per channel count that gave up when neither preferred rate matched /// would hand that to the mono device and downmix. The channel count leads /// because a downmix is audible where a resample is not. -fn preference(config: &cpal::SupportedStreamConfigRange) -> (usize, usize, std::cmp::Reverse, usize) { - let channels = CHANNELS - .iter() - .position(|count| *count == config.channels()) - .unwrap_or(CHANNELS.len()); +fn preference( + config: &cpal::SupportedStreamConfigRange, + native: Option, +) -> ((bool, std::cmp::Reverse), usize, std::cmp::Reverse, usize) { + let channels = (native != Some(config.channels()), std::cmp::Reverse(config.channels())); let rate = match preferred_rate(config) { Some(rate) => (RATES.iter().position(|r| *r == rate).expect("from RATES"), rate), // Nothing we asked for, so take the most the device offers: resampling @@ -223,33 +234,72 @@ mod tests { /// the way to a stereo sink. #[test] fn stereo_wins_even_when_the_device_lists_mono_first() { - let chosen = choose([range(1, 48_000, SampleFormat::F32), range(2, 48_000, SampleFormat::F32)].into_iter()) - .expect("a config"); + let chosen = choose( + [range(1, 48_000, SampleFormat::F32), range(2, 48_000, SampleFormat::F32)].into_iter(), + None, + ) + .expect("a config"); assert_eq!(chosen.channels(), 2); } /// Preferring stereo must not refuse a device that has no stereo to offer. #[test] fn mono_is_taken_when_that_is_all_there_is() { - let chosen = choose([range(1, 48_000, SampleFormat::F32)].into_iter()).expect("a config"); + let chosen = choose([range(1, 48_000, SampleFormat::F32)].into_iter(), None).expect("a config"); assert_eq!(chosen.channels(), 1); } - /// Neither preferred count is offered, so the pass that accepts any count - /// has to catch it. Without it a surround-only sink would not open at all. + /// A surround-only sink opens at its own width rather than being refused. #[test] - fn a_count_we_do_not_prefer_still_opens() { - let chosen = choose([range(6, 48_000, SampleFormat::F32)].into_iter()).expect("a config"); + fn a_surround_only_device_opens() { + let chosen = choose([range(6, 48_000, SampleFormat::F32)].into_iter(), None).expect("a config"); assert_eq!(chosen.channels(), 6); } + /// A 5.1 HDMI sink opens at six channels rather than being folded to stereo. + #[test] + fn six_channels_are_chosen_when_offered() { + let offered = || [range(2, 48_000, SampleFormat::F32), range(6, 48_000, SampleFormat::F32)].into_iter(); + assert_eq!(choose(offered(), None).expect("a config").channels(), 6); + assert_eq!(choose(offered(), Some(6)).expect("a config").channels(), 6); + } + + /// A sound server's plugin accepts every count up to its limit, so the + /// device's own count beats the widest: a headset stays at two rather than + /// being upmixed to 7.1 and folded back down by the server. + #[test] + fn the_native_count_beats_the_widest() { + let offered = (1..=8).map(|channels| range(channels, 48_000, SampleFormat::F32)); + assert_eq!(choose(offered, Some(2)).expect("a config").channels(), 2); + } + + /// Past 7.1 there is no convention naming the speakers, so there is nothing + /// to remix into. + #[test] + fn counts_without_a_layout_are_refused() { + assert!(choose([range(12, 48_000, SampleFormat::F32)].into_iter(), Some(12)).is_none()); + let chosen = choose( + [ + range(12, 48_000, SampleFormat::F32), + range(2, 48_000, SampleFormat::F32), + ] + .into_iter(), + Some(12), + ) + .expect("a config"); + assert_eq!(chosen.channels(), 2); + } + /// Rate is preferred within a channel count, not across one: a stereo config /// at an awkward rate beats a mono config at the pipeline's own rate, /// because resampling is inaudible and a downmix is not. #[test] fn channels_outrank_the_sample_rate() { - let chosen = choose([range(1, 48_000, SampleFormat::F32), range(2, 44_100, SampleFormat::F32)].into_iter()) - .expect("a config"); + let chosen = choose( + [range(1, 48_000, SampleFormat::F32), range(2, 44_100, SampleFormat::F32)].into_iter(), + None, + ) + .expect("a config"); assert_eq!((chosen.channels(), chosen.sample_rate()), (2, 44_100)); } @@ -257,8 +307,11 @@ mod tests { /// produces rather than one that costs a conversion. #[test] fn f32_is_preferred_over_a_format_we_convert_to() { - let chosen = choose([range(2, 48_000, SampleFormat::I16), range(2, 48_000, SampleFormat::F32)].into_iter()) - .expect("a config"); + let chosen = choose( + [range(2, 48_000, SampleFormat::I16), range(2, 48_000, SampleFormat::F32)].into_iter(), + None, + ) + .expect("a config"); assert_eq!(chosen.sample_format(), SampleFormat::F32); } @@ -266,8 +319,8 @@ mod tests { /// stream that plays noise. #[test] fn a_device_we_cannot_write_to_is_rejected() { - assert!(choose([range(2, 48_000, SampleFormat::I8)].into_iter()).is_none()); - assert!(choose(std::iter::empty()).is_none()); + assert!(choose([range(2, 48_000, SampleFormat::I8)].into_iter(), None).is_none()); + assert!(choose(std::iter::empty(), None).is_none()); } /// Channels lead even when neither preferred rate is on offer. @@ -278,8 +331,11 @@ mod tests { /// downmix the whole preference exists to avoid. #[test] fn stereo_at_an_awkward_rate_beats_mono_at_a_preferred_one() { - let chosen = choose([range(1, 48_000, SampleFormat::F32), range(2, 96_000, SampleFormat::F32)].into_iter()) - .expect("a config"); + let chosen = choose( + [range(1, 48_000, SampleFormat::F32), range(2, 96_000, SampleFormat::F32)].into_iter(), + None, + ) + .expect("a config"); assert_eq!((chosen.channels(), chosen.sample_rate()), (2, 96_000)); } @@ -293,6 +349,7 @@ mod tests { range(2, 96_000, SampleFormat::F32), ] .into_iter(), + None, ) .expect("a config"); assert_eq!((chosen.channels(), chosen.sample_rate()), (2, 96_000)); @@ -309,6 +366,7 @@ mod tests { range(2, 32_000, SampleFormat::F32), ] .into_iter(), + None, ) .expect("a config"); assert_eq!( diff --git a/rs/moq-audio/src/playback/driver.rs b/rs/moq-audio/src/playback/driver.rs index ce30e3688d..12bbd70fa5 100644 --- a/rs/moq-audio/src/playback/driver.rs +++ b/rs/moq-audio/src/playback/driver.rs @@ -19,7 +19,7 @@ use rand::RngExt; use super::mixer::{self, Mixer}; use super::sink::{Registration, Sink}; -use crate::Error; +use crate::{Error, Layout}; /// Backoff bounds for reopening a device that failed. The first retry is quick because the common /// case is a device that came right back (a USB re-enumerate, a sample-rate change); the ceiling @@ -78,6 +78,9 @@ struct State { /// Rate the device is running at, which is what sinks resample to. Zero /// until the first stream opens. rate: u32, + /// Layout the device is running in, which is what sinks remix to. Stereo + /// stands in until the first stream opens. + layout: Layout, /// Registration channel to the live mixer, replaced every time the stream is /// rebuilt. `None` while no stream is running. mixer: Option>, @@ -108,13 +111,14 @@ struct State { impl Shared { /// Build a sink, register it, and start mixing it. /// - /// `build` is handed the sink's id and the rate its channel should target. + /// `build` is handed the sink's id and the rate and layout its channel should + /// target. /// It runs with no device open too: the registration waits for the next /// restart, so a device that is briefly missing doesn't become an error the /// caller has to retry. pub(super) fn add(&self, build: F) -> Result where - F: FnOnce(u64, u32) -> Result<(Sink, Registration), Error>, + F: FnOnce(u64, u32, Layout) -> Result<(Sink, Registration), Error>, { let mut state = self.state.lock().unwrap(); @@ -131,7 +135,7 @@ impl Shared { // 48 kHz stands in until a device opens and the channel is rebuilt at // the real rate. let rate = if state.rate == 0 { 48_000 } else { state.rate }; - let (sink, mut registration) = build(state.next_id, rate)?; + let (sink, mut registration) = build(state.next_id, rate, state.layout)?; state.next_id += 1; if let Some(mixer) = &state.mixer { @@ -270,11 +274,11 @@ impl Shared { } /// Point every sink at a freshly opened stream: rebuild each channel at - /// `rate` and hand the new consumers to `mixer`. - fn rebind(&self, rate: u32, mixer: SyncSender) { + /// `rate` and `layout` and hand the new consumers to `mixer`. + fn rebind(&self, rate: u32, layout: Layout, mixer: SyncSender) { let mut state = self.state.lock().unwrap(); for sink in &mut state.sinks { - sink.rebuild(rate); + sink.rebuild(rate, layout); sink.attach(&mixer); } @@ -289,6 +293,7 @@ impl Shared { } state.rate = rate; + state.layout = layout; state.mixer = Some(mixer); // The old mixer is gone, and with it every sink it was told about. state.detaching.clear(); @@ -787,7 +792,8 @@ impl Driver { // a full retirement channel is the one case where the mixer has to free // on the audio thread after all. let (retired_tx, retired_rx) = sync_channel(COMMAND_QUEUE); - let mixer = Mixer::new(rx, retired_tx, rate, channels); + let layout = Layout::from_channels(channels as u32)?; + let mixer = Mixer::new(rx, retired_tx, rate, layout)?; let failures = Arc::new(Failures::default()); let reporter = FailureReporter { @@ -799,7 +805,7 @@ impl Driver { .play() .map_err(|err| Error::Playback(format!("cannot start output stream: {err}")))?; - self.shared.rebind(rate, tx); + self.shared.rebind(rate, layout, tx); self.stream = Some(stream); self.failures = Some(failures); // Replaces the previous receiver, dropping anything the old stream @@ -807,7 +813,7 @@ impl Driver { self.retired = Some(retired_rx); self.retry = RETRY_MIN; - tracing::info!(rate, channels, ?format, "opened audio output"); + tracing::info!(rate, ?layout, ?format, "opened audio output"); Ok(()) } @@ -1005,7 +1011,7 @@ mod tests { let handle = Arc::new(super::super::Handle { commands }); let (tx, mixer) = sync_channel(depth); - shared.rebind(48_000, tx); + shared.rebind(48_000, Layout::Stereo, tx); Wired { shared, @@ -1016,7 +1022,7 @@ mod tests { } fn add(shared: &Arc, handle: &Arc) -> Result { - shared.add(|id, rate| sink::new(id, rate, Input::default(), shared.clone(), handle.clone())) + shared.add(|id, rate, bus| sink::new(id, rate, bus, Input::default(), shared.clone(), handle.clone())) } /// Long enough that only a lost wake, rather than a loaded machine, trips a diff --git a/rs/moq-audio/src/playback/mixer.rs b/rs/moq-audio/src/playback/mixer.rs index d3f23124e3..6187f2ca9a 100644 --- a/rs/moq-audio/src/playback/mixer.rs +++ b/rs/moq-audio/src/playback/mixer.rs @@ -19,6 +19,10 @@ use fixed_resample::ResamplingCons; #[cfg(feature = "aec")] use fixed_resample::ResamplingProd; +#[cfg(feature = "aec")] +use crate::resample::Remix; +use crate::{Error, Layout}; + /// Frames mixed per pass. The callback buffer is chunked to this so the scratch /// buffers stay a fixed size no matter what period the device asks for. const CHUNK: usize = 1024; @@ -27,9 +31,10 @@ const CHUNK: usize = 1024; /// inaudible as a click, short enough to feel instant. const RAMP: f32 = 0.003; -/// The mix bus is stereo: sinks resample into it and it fans out to however many -/// channels the device wants. -pub(crate) const BUS_CHANNELS: usize = 2; +/// The echo reference is the mix downmixed to stereo, whatever layout the +/// device plays, so the canceller's model survives a device switch. +#[cfg(feature = "aec")] +pub(crate) const REFERENCE_CHANNELS: usize = 2; /// Sinks one device will mix. The entry list is allocated to this up front and /// never grows, which is what keeps registration off the allocator. @@ -138,36 +143,61 @@ pub(super) struct Mixer { /// Where anything the mixer is done with goes to be dropped, since dropping /// it here would free on the audio thread. retired: SyncSender, - /// Channels the device takes. + /// Channels the device takes, which is also the bus layout's: every sink + /// remixes into it on its way here. channels: usize, /// Per-frame gain step, so any change spans [`RAMP`] regardless of rate. step: f32, - /// Stereo accumulator for one chunk. + /// Accumulator for one chunk, in the device's layout. bus: Vec, - /// Stereo scratch for the sink being read. + /// Scratch for the sink being read, in the device's layout. scratch: Vec, - /// Where echo cancellation reads what was played. Fed the mix after - /// clipping but before it fans out, since that is the signal the speaker - /// gets and therefore the one the microphone hears back. + /// Where echo cancellation reads what was played. Fed the clipped mix, since + /// that is the signal the speaker gets and therefore the one the microphone + /// hears back. #[cfg(feature = "aec")] reference: Option>, + /// Downmixes the bus to the reference's stereo. + #[cfg(feature = "aec")] + downmix: Remix, + /// Stereo scratch for the downmixed reference. + #[cfg(feature = "aec")] + stereo: Vec, } impl Mixer { - /// `rate` and `channels` describe the device, not the sinks: each sink - /// resamples into the bus on its way here. - pub(super) fn new(commands: Receiver, retired: SyncSender, rate: u32, channels: usize) -> Self { - Self { + /// `rate` and `layout` describe the device, not the sinks: each sink + /// resamples and remixes into the bus on its way here. + pub(super) fn new( + commands: Receiver, + retired: SyncSender, + rate: u32, + layout: Layout, + ) -> Result { + // Sinks remix by speaker position, so a bus without any has nothing to + // remix into. + if layout.speakers().is_none() { + return Err(Error::Unsupported(format!( + "cannot mix into an output without speaker positions ({layout:?})" + ))); + } + + let channels = layout.channels() as usize; + Ok(Self { entries: Vec::with_capacity(MAX_SINKS), commands, retired, channels, step: 1.0 / (rate as f32 * RAMP), - bus: vec![0.0; CHUNK * BUS_CHANNELS], - scratch: vec![0.0; CHUNK * BUS_CHANNELS], + bus: vec![0.0; CHUNK * channels], + scratch: vec![0.0; CHUNK * channels], #[cfg(feature = "aec")] reference: None, - } + #[cfg(feature = "aec")] + downmix: Remix::new(layout, Layout::Stereo)?, + #[cfg(feature = "aec")] + stereo: vec![0.0; CHUNK * REFERENCE_CHANNELS], + }) } /// Hand something the mixer is done with back to the driver to drop. @@ -234,6 +264,10 @@ impl Mixer { scratch, #[cfg(feature = "aec")] reference, + #[cfg(feature = "aec")] + downmix, + #[cfg(feature = "aec")] + stereo, .. } = self; let channels = *channels; @@ -242,7 +276,7 @@ impl Mixer { let mut done = 0; while done < total { let frames = (total - done).min(CHUNK); - let samples = frames * BUS_CHANNELS; + let samples = frames * channels; bus[..samples].fill(0.0); @@ -257,13 +291,16 @@ impl Mixer { let mut applied = entry.applied; let mut peak = 0.0f32; - for frame in 0..frames { + for (input, mixed) in scratch[..samples] + .chunks_exact(channels) + .zip(bus[..samples].chunks_exact_mut(channels)) + { applied += (target - applied).clamp(-*step, *step); - let left = scratch[frame * 2] * applied; - let right = scratch[frame * 2 + 1] * applied; - peak = peak.max(left.abs()).max(right.abs()); - bus[frame * 2] += left; - bus[frame * 2 + 1] += right; + for (input, mixed) in input.iter().zip(mixed) { + let sample = input * applied; + peak = peak.max(sample.abs()); + *mixed += sample; + } } entry.applied = applied; @@ -280,26 +317,12 @@ impl Mixer { // useful to do about it from here and nothing may be logged. #[cfg(feature = "aec")] if let Some(reference) = reference.as_mut() { - reference.push_interleaved(&bus[..samples]); + let stereo = &mut stereo[..frames * REFERENCE_CHANNELS]; + downmix.apply(&bus[..samples], stereo); + reference.push_interleaved(stereo); } - let out = &mut out[done * channels..(done + frames) * channels]; - match channels { - 1 => { - for (frame, out) in out.iter_mut().enumerate() { - *out = (bus[frame * 2] + bus[frame * 2 + 1]) * 0.5; - } - } - _ => { - for (frame, out) in out.chunks_exact_mut(channels).enumerate() { - out[0] = bus[frame * 2]; - out[1] = bus[frame * 2 + 1]; - // Surround devices get silence past the front pair, which - // is better than duplicating stereo into the rears. - out[2..].fill(0.0); - } - } - } + out[done * channels..(done + frames) * channels].copy_from_slice(&bus[..samples]); done += frames; } @@ -316,6 +339,9 @@ mod tests { const RATE: u32 = 48_000; + /// The bus most tests mix into. + const STEREO: usize = 2; + /// Frames pushed per test: a tenth of a second, far more than any `fill` /// below drains, so a short read never starves by accident. const FRAMES: usize = RATE as usize / 10; @@ -328,21 +354,23 @@ mod tests { /// audio thread never drops one. retired: Receiver, next: u64, + channels: usize, } impl Harness { - fn new(channels: usize) -> Self { - Self::with_depth(channels, 8) + fn new(layout: Layout) -> Self { + Self::with_depth(layout, 8) } - fn with_depth(channels: usize, depth: usize) -> Self { + fn with_depth(layout: Layout, depth: usize) -> Self { let (commands, rx) = sync_channel(depth); let (retired_tx, retired) = sync_channel(MAX_SINKS); Self { - mixer: Mixer::new(rx, retired_tx, RATE, channels), + mixer: Mixer::new(rx, retired_tx, RATE, layout).unwrap(), commands, retired, next: 0, + channels: layout.channels() as usize, } } @@ -352,7 +380,7 @@ mod tests { /// pushed samples and the assertions. fn add(&mut self, gain: Arc) -> (u64, ResamplingProd) { let (prod, cons) = resampling_channel::( - BUS_CHANNELS, + self.channels, RATE, RATE, true, @@ -389,12 +417,12 @@ mod tests { /// Push `frames` stereo frames of a constant sample. fn push(prod: &mut ResamplingProd, value: f32, frames: usize) { - prod.push_interleaved(&vec![value; frames * BUS_CHANNELS]); + prod.push_interleaved(&vec![value; frames * STEREO]); } #[test] fn discards_writes_until_the_device_reads() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let (_, mut prod) = harness.add(Arc::new(Gain::new())); // Nothing has read yet, so these samples are dropped rather than queued @@ -411,7 +439,7 @@ mod tests { #[test] fn sums_sinks_and_clips() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 2048]; // Three sinks at 0.5 sum to 1.5, which must clip to 1.0. @@ -433,7 +461,7 @@ mod tests { #[test] fn volume_ramps_instead_of_stepping() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 2048]; let gain = Arc::new(Gain::new()); @@ -453,8 +481,8 @@ mod tests { // The whole ramp is monotonic, not just its endpoints. let ramp = (RATE as f32 * RAMP) as usize; assert!(ramp < out.len() / 2, "test buffer is shorter than the ramp"); - for frame in out[..ramp * BUS_CHANNELS] - .as_chunks::() + for frame in out[..ramp * STEREO] + .as_chunks::() .0 .iter() .collect::>() @@ -484,7 +512,7 @@ mod tests { /// device buffer into NaN. #[test] fn output_stays_finite_after_a_non_finite_volume() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 2048]; let gain = Arc::new(Gain::new()); @@ -500,7 +528,7 @@ mod tests { #[test] fn peak_reports_the_loudest_sample_then_resets() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 2048]; let gain = Arc::new(Gain::new()); @@ -517,7 +545,7 @@ mod tests { #[test] fn removed_sinks_stop_mixing() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 2048]; let (id, mut prod) = harness.add(Arc::new(Gain::new())); @@ -536,7 +564,7 @@ mod tests { /// callback would free on the audio thread, so it goes back to the driver. #[test] fn removed_sinks_are_handed_back_rather_than_dropped() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 512]; let (id, _prod) = harness.add(Arc::new(Gain::new())); @@ -561,7 +589,7 @@ mod tests { #[cfg(feature = "aec")] #[test] fn the_echo_reference_gets_the_mix() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 2048]; let (reference, mut tap) = reference_channel(); @@ -574,7 +602,7 @@ mod tests { // does, so that nothing queues up while nobody is listening. That first // read also primes the channel with its configured latency in silence, // which is why the buffer below is read past it rather than at it. - let mut heard = vec![0.0f32; 2048 * BUS_CHANNELS]; + let mut heard = vec![0.0f32; 2048 * REFERENCE_CHANNELS]; tap.read_interleaved(&mut heard, false); for prod in &mut prods { @@ -596,7 +624,7 @@ mod tests { #[cfg(feature = "aec")] #[test] fn a_replaced_echo_reference_is_handed_back() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 512]; harness @@ -629,7 +657,7 @@ mod tests { #[cfg(feature = "aec")] fn reference_channel() -> (ResamplingProd, ResamplingCons) { resampling_channel::( - BUS_CHANNELS, + REFERENCE_CHANNELS, RATE, RATE, true, @@ -645,7 +673,7 @@ mod tests { /// thread, which is why the driver caps registrations at MAX_SINKS. #[test] fn the_entry_list_never_grows() { - let mut harness = Harness::with_depth(2, MAX_SINKS); + let mut harness = Harness::with_depth(Layout::Stereo, MAX_SINKS); let mut out = vec![0.0f32; 512]; let capacity = harness.mixer.entries.capacity(); @@ -660,50 +688,65 @@ mod tests { #[test] fn silence_when_no_sink_is_registered() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![1.0f32; 512]; harness.fill(&mut out); assert!(out.iter().all(|s| *s == 0.0), "callback buffer was not overwritten"); } + /// The bus is the device's own layout, so every speaker of a surround device + /// carries what the sinks put there rather than silence past the front pair. #[test] - fn mono_device_gets_the_stereo_average() { - let mut harness = Harness::new(1); - let mut out = vec![0.0f32; 1024]; + fn a_surround_bus_mixes_every_speaker() { + let mut harness = Harness::new(Layout::FivePointOne); + let mut out = vec![0.0f32; 6 * 512]; let (_, mut prod) = harness.add(Arc::new(Gain::new())); harness.fill(&mut out); - - // Hard left, so a mono device should hear half of it. - let mut samples = vec![0.0f32; FRAMES * BUS_CHANNELS]; - for frame in samples.as_chunks_mut::().0.iter_mut() { - frame[0] = 1.0; - } - prod.push_interleaved(&samples); + let frame = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6]; + prod.push_interleaved(&frame.repeat(FRAMES)); harness.settle(&mut out); - assert!((out[out.len() - 1] - 0.5).abs() < 1e-5, "got {}", out[out.len() - 1]); + + let last = &out[out.len() - 6..]; + for (got, want) in last.iter().zip(frame) { + assert!((got - want).abs() < 1e-5, "got {last:?}"); + } } + /// The canceller models a stereo reference whatever the device plays, so a + /// surround bus reaches it downmixed rather than cut to its front pair. + #[cfg(feature = "aec")] #[test] - fn surround_devices_get_silence_past_the_front_pair() { - let mut harness = Harness::new(6); - let mut out = vec![0.0f32; 6 * 512]; + fn the_echo_reference_downmixes_a_surround_bus() { + let mut harness = Harness::new(Layout::FivePointOne); + let mut out = vec![0.0f32; 6 * 2048]; + let (reference, mut tap) = reference_channel(); + harness.commands.send(Command::Reference(Some(reference))).unwrap(); let (_, mut prod) = harness.add(Arc::new(Gain::new())); harness.fill(&mut out); - push(&mut prod, 1.0, FRAMES); + let mut heard = vec![0.0f32; 2048 * REFERENCE_CHANNELS]; + tap.read_interleaved(&mut heard, false); + + // Center only, which a stereo downmix splits evenly at -3 dB. + prod.push_interleaved(&[0.0, 0.0, 0.5, 0.0, 0.0, 0.0].repeat(FRAMES)); harness.settle(&mut out); + tap.read_interleaved(&mut heard, false); - let last = &out[out.len() - 6..]; - assert!(last[0] > 0.9 && last[1] > 0.9, "front pair was silent"); - assert!(last[2..].iter().all(|s| *s == 0.0), "rear channels were not silent"); + let want = 0.5 * std::f32::consts::FRAC_1_SQRT_2; + let tail = &heard[heard.len() - 64..]; + assert!( + tail.iter().all(|s| (*s - want).abs() < 1e-5), + "the tap saw {:?}, not the downmix", + &tail[..4] + ); } #[test] fn underflow_reads_as_silence_rather_than_stale_samples() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 8192]; let (_, mut prod) = harness.add(Arc::new(Gain::new())); diff --git a/rs/moq-audio/src/playback/sink.rs b/rs/moq-audio/src/playback/sink.rs index 41b8d0f79d..74bed20c19 100644 --- a/rs/moq-audio/src/playback/sink.rs +++ b/rs/moq-audio/src/playback/sink.rs @@ -1,6 +1,5 @@ //! [`Sink`]: one stream of PCM on its way to the speaker. -use std::borrow::Cow; use std::sync::mpsc::{SyncSender, TrySendError}; use std::sync::{Arc, Mutex}; use std::time::Duration; @@ -8,8 +7,8 @@ use std::time::Duration; use fixed_resample::{PushStatus, ResamplingChannelConfig, ResamplingCons, ResamplingProd, resampling_channel}; use super::driver::Shared; -use super::mixer::{self, BUS_CHANNELS, Gain}; -use crate::resample::remix; +use super::mixer::{self, Gain}; +use crate::resample::Remix; use crate::{Error, Format, Layout}; /// Default for [`Input::latency`]: audio buffered between [`Sink::write`] and @@ -28,7 +27,7 @@ const HEADROOM: f64 = 3.0; /// /// The playback counterpart to [`encode::Input`](crate::encode::Input): it /// describes the buffers you hand in, not the device, which is free to run at -/// its own rate and channel count. +/// its own rate and layout. /// /// `#[non_exhaustive]`: construct via [`Input::default`] and set fields, so new /// options can be added without breaking callers. @@ -40,7 +39,8 @@ pub struct Input { /// Samples per second per channel. Resampled to the device rate if they /// differ. pub sample_rate: u32, - /// Speaker meaning and channel order. + /// Speaker meaning and channel order. Remixed to the device's layout, so it + /// must name speaker positions rather than be [`Layout::Discrete`]. pub layout: Layout, /// How much audio to hold between [`Sink::write`] and the speaker (default: @@ -77,9 +77,9 @@ impl Input { if self.sample_rate == 0 { return Err(Error::Unsupported("sample rate must be > 0".into())); } - if !matches!(self.layout, Layout::Mono | Layout::Stereo) { + if self.layout.speakers().is_none() { return Err(Error::Unsupported(format!( - "playback accepts named mono or stereo input (got {:?})", + "playback needs speaker positions to remix (got {:?})", self.layout ))); } @@ -120,13 +120,13 @@ impl Write { /// One stream of PCM being played, mixed with every other sink on the device. /// /// Write decoded samples with [`write`](Self::write) and drop the sink to stop. -/// Writes are cheap and never block on the device: they hand samples to a ring -/// buffer that the audio thread drains on its own clock, resampling to the -/// device rate on the way. +/// Writes are cheap and never block on the device: they remix to the device's +/// layout and hand samples to a ring buffer that the audio thread drains on its +/// own clock, resampling to the device rate on the way. pub struct Sink { id: u64, input: Input, - prod: Arc>>, + channel: Arc>, control: Control, /// Whether the last write overflowed, so a writer that stays ahead of the /// device logs once rather than on every write. @@ -151,17 +151,21 @@ impl Sink { /// The returned [`Write`] counts input sample frames accepted and dropped; /// dropped live audio should be observed for telemetry, not retried. pub fn write(&mut self, samples: &[u8]) -> Result { - let pcm = self - .input - .format - .as_interleaved_f32(samples, self.input.layout.channels())?; - let pcm = match self.input.layout.channels() as usize { - BUS_CHANNELS => pcm, - _ => Cow::Owned(remix(&pcm, self.input.layout, Layout::Stereo)?), + let channels = self.input.layout.channels(); + let pcm = self.input.format.as_interleaved_f32(samples, channels)?; + let requested_sample_frames = pcm.len() / channels as usize; + + let mut channel = self.channel.lock().unwrap(); + let mixed; + let pcm = match &channel.remix { + Some(remix) => { + mixed = remix.process(&pcm); + &mixed + } + None => pcm.as_ref(), }; - let requested_sample_frames = pcm.len() / BUS_CHANNELS; - let accepted_sample_frames = match self.prod.lock().unwrap().push_interleaved(&pcm) { + let accepted_sample_frames = match channel.prod.push_interleaved(pcm) { // OutputNotReady means the device has not read yet, so these samples // are dropped rather than queued to play late. PushStatus::Ok => { @@ -201,7 +205,7 @@ impl Sink { /// climbs when the writer runs ahead, and falls toward zero when it falls /// behind. pub fn buffered(&self) -> Duration { - Duration::from_secs_f64(self.prod.lock().unwrap().occupied_seconds().max(0.0)) + Duration::from_secs_f64(self.channel.lock().unwrap().prod.occupied_seconds().max(0.0)) } /// The PCM layout this sink was built with. @@ -278,15 +282,21 @@ impl Control { } } +/// The ring into the mixer and the remix that fills it, swapped together when +/// the device changes rate or layout. +struct Channel { + prod: ResamplingProd, + /// Converts the sink's layout to the device's, when they differ. + remix: Option, +} + /// A sink as the driver sees it: enough to rebuild its channel when the device /// changes underneath it. pub(super) struct Registration { pub(super) id: u64, - /// The caller's rate, which is the input side of the channel. - rate: u32, - /// The depth the rebuilt channel has to keep, from the caller's [`Input`]. - latency: Duration, - prod: Arc>>, + /// The caller's rate, layout, and latency: the input side of the channel. + input: Input, + channel: Arc>, gain: Arc, /// The consumer waiting to be handed to a mixer. Taken once it is attached, /// and refilled by [`rebuild`](Self::rebuild). @@ -319,34 +329,36 @@ impl Registration { } } - /// Re-create the channel for a device now running at `rate`, swapping the - /// producer the caller's [`Sink`] writes into. - pub(super) fn rebuild(&mut self, rate: u32) { - let (prod, cons) = channel(self.rate, rate, self.latency); - *self.prod.lock().unwrap() = prod; + /// Re-create the channel for a device now running at `rate` in `bus`, + /// swapping the producer the caller's [`Sink`] writes into. + pub(super) fn rebuild(&mut self, rate: u32, bus: Layout) { + let (channel, cons) = channel(&self.input, rate, bus); + *self.channel.lock().unwrap() = channel; self.pending = Some(cons); } } /// Build a sink and its registration. The device may not be open yet, in which -/// case `rate` is a placeholder the driver replaces on the next rebuild. +/// case `rate` and `bus` are placeholders the driver replaces on the next +/// rebuild. pub(super) fn new( id: u64, rate: u32, + bus: Layout, input: Input, shared: Arc, engine: Arc, ) -> Result<(Sink, Registration), Error> { input.validate()?; - let (prod, cons) = channel(input.sample_rate, rate, input.latency); - let prod = Arc::new(Mutex::new(prod)); + let (channel, cons) = self::channel(&input, rate, bus); + let channel = Arc::new(Mutex::new(channel)); let gain = Arc::new(Gain::new()); let sink = Sink { id, input, - prod: prod.clone(), + channel: channel.clone(), control: Control { gain: gain.clone() }, overflowing: false, shared, @@ -355,9 +367,8 @@ pub(super) fn new( let registration = Registration { id, - rate: sink.input.sample_rate, - latency: sink.input.latency, - prod, + input: sink.input.clone(), + channel, gain, pending: Some(cons), }; @@ -365,14 +376,17 @@ pub(super) fn new( Ok((sink, registration)) } -/// The ring buffer between a writer and the audio thread, resampling the -/// caller's rate to the device's. -fn channel(from: u32, to: u32, latency: Duration) -> (ResamplingProd, ResamplingCons) { - let latency = latency.as_secs_f64(); - resampling_channel::( - BUS_CHANNELS, - from, - to, +/// The ring buffer between a writer and the audio thread, remixing the caller's +/// layout to the device's and resampling its rate to the device's. +fn channel(input: &Input, rate: u32, bus: Layout) -> (Channel, ResamplingCons) { + let remix = + (input.layout != bus).then(|| Remix::new(input.layout, bus).expect("sink and bus layouts name their speakers")); + + let latency = input.latency.as_secs_f64(); + let (prod, cons) = resampling_channel::( + bus.channels() as usize, + input.sample_rate, + rate, // We only ever push interleaved, which lets the channel skip its planar // staging buffer. true, @@ -385,7 +399,9 @@ fn channel(from: u32, to: u32, latency: Duration) -> (ResamplingProd, Resam overflow_autocorrect_percent_threshold: Some(75.0), ..Default::default() }, - ) + ); + + (Channel { prod, remix }, cons) } #[cfg(test)] @@ -393,20 +409,65 @@ mod tests { use super::*; fn sink(input: Input, output_rate: u32) -> (Sink, ResamplingCons) { + sink_into(input, output_rate, Layout::Stereo) + } + + fn sink_into(input: Input, output_rate: u32, bus: Layout) -> (Sink, ResamplingCons) { let shared = Arc::new(Shared::default()); let engine = Arc::new(super::super::Handle { commands: super::super::driver::Commands::default(), }); - let (sink, mut registration) = new(0, output_rate, input, shared, engine).unwrap(); + let (sink, mut registration) = new(0, output_rate, bus, input, shared, engine).unwrap(); (sink, registration.pending.take().unwrap()) } + /// Write `frame` repeated as `input` and read back what the bus got. + fn mix(input: Layout, bus: Layout, frame: &[f32]) -> Vec { + let input = Input { + layout: input, + ..Default::default() + }; + let (mut sink, mut cons) = sink_into(input, 48_000, bus); + let channels = bus.channels() as usize; + cons.read_interleaved(&mut vec![0.0; channels], false); + + let pcm: Vec = frame.repeat(4800).iter().flat_map(|s| s.to_le_bytes()).collect(); + assert_eq!(sink.write(&pcm).unwrap().dropped_sample_frames, 0); + + let mut out = vec![0.0; 4800 * channels]; + cons.read_interleaved(&mut out, false); + out[out.len() - channels..].to_vec() + } + + fn close(got: &[f32], want: &[f32]) { + assert_eq!(got.len(), want.len(), "{got:?} vs {want:?}"); + for (g, w) in got.iter().zip(want) { + assert!((g - w).abs() < 1e-4, "{got:?} vs {want:?}"); + } + } + + /// A 5.1 track on a stereo device plays its downmix, center and surrounds + /// folded into the front pair at -3 dB. + #[test] + fn a_surround_sink_downmixes_into_a_stereo_bus() { + let h = std::f32::consts::FRAC_1_SQRT_2; + let got = mix(Layout::FivePointOne, Layout::Stereo, &[0.1, 0.2, 0.3, 0.4, 0.05, 0.06]); + close(&got, &[0.1 + h * 0.3 + h * 0.05, 0.2 + h * 0.3 + h * 0.06]); + } + + /// A stereo track on a 5.1 device plays from the front pair alone. + #[test] + fn a_stereo_sink_fills_the_front_of_a_surround_bus() { + let got = mix(Layout::Stereo, Layout::FivePointOne, &[0.25, 0.75]); + close(&got, &[0.25, 0.75, 0.0, 0.0, 0.0, 0.0]); + } + fn s16(frames: usize, channels: usize) -> Vec { vec![0; frames * channels * 2] } fn ready(cons: &mut ResamplingCons) { - cons.read_interleaved(&mut [0.0; BUS_CHANNELS], false); + cons.read_interleaved(&mut [0.0; 2], false); } #[test] @@ -495,7 +556,7 @@ mod tests { #[test] fn rejects_layouts_it_cannot_mix() { - for layout in [Layout::Discrete(0), Layout::Discrete(6)] { + for layout in [Layout::Discrete(0), Layout::Discrete(2), Layout::Discrete(6)] { let input = Input { layout, ..Default::default() @@ -511,8 +572,13 @@ mod tests { } #[test] - fn accepts_mono_and_stereo() { - for layout in [Layout::Mono, Layout::Stereo] { + fn accepts_named_layouts() { + for layout in [ + Layout::Mono, + Layout::Stereo, + Layout::FivePointOne, + Layout::SevenPointOne, + ] { let input = Input { layout, ..Default::default() diff --git a/rs/moq-audio/src/resample.rs b/rs/moq-audio/src/resample.rs index 173a3b1ba2..402f0b8529 100644 --- a/rs/moq-audio/src/resample.rs +++ b/rs/moq-audio/src/resample.rs @@ -2,14 +2,17 @@ //! //! Wraps [`rubato`] with a small interleaved-`f32` interface so the //! producer/consumer doesn't have to convert to planar on every call. -//! The resampler keeps the channel layout unchanged; [`remix`] converts mono -//! and stereo after sample-rate conversion. +//! The resampler keeps the channel layout unchanged; [`Remix`] converts between +//! layouts after sample-rate conversion. + +use std::f32::consts::FRAC_1_SQRT_2; use rubato::audioadapter_buffers::direct::SequentialSliceOfVecs; use rubato::{ Async, FixedAsync, Resampler as RubatoTrait, SincInterpolationParameters, SincInterpolationType, WindowFunction, }; +use crate::layout::Speaker; use crate::{Error, Layout}; #[derive(Debug, thiserror::Error)] @@ -301,50 +304,147 @@ impl Resampler { } } -/// Convert between known layouts without assigning positions to discrete channels. -pub(crate) fn remix(samples: &[f32], input: Layout, output: Layout) -> Result, Error> { - validate_remix(input, output)?; - match (input, output) { - (input, output) if input == output => Ok(samples.to_vec()), - (Layout::Mono, Layout::Stereo) => { - let mut output = Vec::with_capacity(samples.len() * 2); - for &sample in samples { - output.extend_from_slice(&[sample, sample]); +/// A channel mix from one layout to another, each output channel a weighted sum +/// of the input channels. +/// +/// Refuses to give discrete channels speaker positions, though it will drop +/// them from named channels. +/// +/// Downmixing uses the ITU-R BS.775 coefficients: center and surrounds fold into +/// the front pair at -3 dB and the LFE is dropped. Upmixing leaves the speakers +/// the input lacks silent. Mono is the exception both ways: it plays at full +/// level from both front speakers when there is no center, and a mono output +/// averages the stereo downmix. +pub(crate) struct Remix { + inputs: usize, + outputs: usize, + /// One row of `inputs` weights per output channel. + weights: Vec, +} + +impl Remix { + pub(crate) fn new(input: Layout, output: Layout) -> Result { + input.validate()?; + output.validate()?; + + let (inputs, outputs) = (input.channels() as usize, output.channels() as usize); + // Dropping speaker positions is always safe; inventing them is not. + let unchanged = input == output || output == Layout::Discrete(inputs as u32); + let weights = match (input.speakers(), output.speakers()) { + _ if unchanged => (0..outputs) + .flat_map(|o| (0..inputs).map(move |i| if i == o { 1.0 } else { 0.0 })) + .collect(), + (Some(from), Some(to)) => weights(from, to), + _ => { + return Err(Error::Unsupported(format!( + "cannot convert audio layout {input:?} to {output:?} without speaker positions" + ))); + } + }; + + Ok(Self { + inputs, + outputs, + weights, + }) + } + + /// Mix whole interleaved input frames into `output`, which holds as many + /// frames. Never allocates, so the audio thread can call it. + pub(crate) fn apply(&self, input: &[f32], output: &mut [f32]) { + for (frame, out) in input + .chunks_exact(self.inputs) + .zip(output.chunks_exact_mut(self.outputs)) + { + for (sample, row) in out.iter_mut().zip(self.weights.chunks_exact(self.inputs)) { + *sample = row.iter().zip(frame).map(|(weight, input)| weight * input).sum(); } - Ok(output) } - (Layout::Stereo, Layout::Mono) => Ok(samples - .as_chunks::<2>() - .0 - .iter() - .map(|pair| (pair[0] + pair[1]) * 0.5) - .collect()), - _ => Err(Error::Unsupported(format!( - "cannot convert audio layout {input:?} to {output:?} without speaker positions" - ))), + } + + /// Mix whole interleaved input frames into a new buffer. + pub(crate) fn process(&self, input: &[f32]) -> Vec { + let mut output = vec![0.0; input.len() / self.inputs * self.outputs]; + self.apply(input, &mut output); + output } } -/// Check that [`remix`] can convert between two layouts. -pub(crate) fn validate_remix(input: Layout, output: Layout) -> Result<(), Error> { - input.validate()?; - output.validate()?; - if input == output - || matches!( - (input, output), - (Layout::Mono, Layout::Stereo) | (Layout::Stereo, Layout::Mono) - ) { - return Ok(()); - } - Err(Error::Unsupported(format!( - "cannot convert audio layout {input:?} to {output:?} without speaker positions" - ))) +/// The weights mixing `input` speakers into `output` speakers, one row per output. +fn weights(input: &[Speaker], output: &[Speaker]) -> Vec { + use Speaker::*; + + // The only layout without a front pair. Average the stereo downmix rather + // than invent a center weight for every speaker. + if output == [FrontCenter] && input != [FrontCenter] { + let stereo = weights(input, &[FrontLeft, FrontRight]); + let (left, right) = stereo.split_at(input.len()); + return left.iter().zip(right).map(|(l, r)| (l + r) * 0.5).collect(); + } + + let mut weights = vec![0.0; output.len() * input.len()]; + let has = |speaker| output.contains(&speaker); + + for (i, &speaker) in input.iter().enumerate() { + let mut feed = |to: Speaker, weight: f32| { + if let Some(o) = output.iter().position(|s| *s == to) { + weights[o * input.len() + i] += weight; + } + }; + + if has(speaker) { + feed(speaker, 1.0); + continue; + } + + match speaker { + FrontCenter => { + let weight = if input == [FrontCenter] { 1.0 } else { FRAC_1_SQRT_2 }; + feed(FrontLeft, weight); + feed(FrontRight, weight); + } + Lfe => {} + SideLeft | BackLeft | SideRight | BackRight => { + let (front, side, back) = match speaker { + SideLeft | BackLeft => (FrontLeft, SideLeft, BackLeft), + _ => (FrontRight, SideRight, BackRight), + }; + // Side and back are the same surround to a layout with only one of them. + let other = if speaker == side { back } else { side }; + if has(other) { + feed(other, 1.0); + } else { + feed(front, FRAC_1_SQRT_2); + } + } + BackCenter => { + if has(BackLeft) { + feed(BackLeft, FRAC_1_SQRT_2); + feed(BackRight, FRAC_1_SQRT_2); + } else if has(SideLeft) { + feed(SideLeft, FRAC_1_SQRT_2); + feed(SideRight, FRAC_1_SQRT_2); + } else { + feed(FrontLeft, 0.5); + feed(FrontRight, 0.5); + } + } + // Every output but mono, handled above, has a front pair. + FrontLeft | FrontRight => unreachable!("{output:?} has no front pair"), + } + } + + weights } #[cfg(test)] mod tests { use super::*; + fn remix(samples: &[f32], input: Layout, output: Layout) -> Result, Error> { + Ok(Remix::new(input, output)?.process(samples)) + } + /// `frames` into a stream at `rate`, as a timestamp in the source's own scale. fn at(frames: u64, rate: u64) -> moq_net::Timestamp { moq_net::Timestamp::from_scale(frames, rate).unwrap() @@ -548,4 +648,105 @@ mod tests { [2.0, 3.0] ); } + + /// One frame of 5.1 with a distinct level per speaker, in canonical order. + const FIVE_ONE: [f32; 6] = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6]; + + fn close(got: &[f32], want: &[f32]) { + assert_eq!(got.len(), want.len(), "{got:?} vs {want:?}"); + for (g, w) in got.iter().zip(want) { + assert!((g - w).abs() < 1e-6, "{got:?} vs {want:?}"); + } + } + + #[test] + fn remix_downmixes_five_one_to_stereo_by_bs775() { + let h = FRAC_1_SQRT_2; + let [l, r, c, _lfe, ls, rs] = FIVE_ONE; + close( + &remix(&FIVE_ONE, Layout::FivePointOne, Layout::Stereo).unwrap(), + &[l + h * c + h * ls, r + h * c + h * rs], + ); + } + + #[test] + fn remix_downmixes_five_one_to_mono_through_stereo() { + let stereo = remix(&FIVE_ONE, Layout::FivePointOne, Layout::Stereo).unwrap(); + close( + &remix(&FIVE_ONE, Layout::FivePointOne, Layout::Mono).unwrap(), + &[(stereo[0] + stereo[1]) * 0.5], + ); + } + + #[test] + fn remix_upmixes_stereo_into_the_front_pair() { + close( + &remix(&[0.25, 0.75], Layout::Stereo, Layout::FivePointOne).unwrap(), + &[0.25, 0.75, 0.0, 0.0, 0.0, 0.0], + ); + } + + #[test] + fn remix_upmixes_mono_into_the_center() { + close( + &remix(&[0.5], Layout::Mono, Layout::FivePointOne).unwrap(), + &[0.0, 0.0, 0.5, 0.0, 0.0, 0.0], + ); + // Quad has no center, so mono plays from both fronts as it does in stereo. + close( + &remix(&[0.5], Layout::Mono, Layout::Quad).unwrap(), + &[0.5, 0.5, 0.0, 0.0], + ); + } + + /// 7.1 to 5.1 folds the back pair into the sides at full level, since a 5.1 + /// surround pair is the only surround it has. + #[test] + fn remix_folds_back_into_side_surrounds() { + let seven = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8]; + close( + &remix(&seven, Layout::SevenPointOne, Layout::FivePointOne).unwrap(), + &[0.1, 0.2, 0.3, 0.4, 0.5 + 0.7, 0.6 + 0.8], + ); + } + + #[test] + fn remix_refuses_positions_it_would_invent() { + for (input, output) in [ + (Layout::Discrete(6), Layout::Stereo), + (Layout::Mono, Layout::Discrete(2)), + (Layout::Discrete(0), Layout::Discrete(0)), + ] { + assert!( + matches!(Remix::new(input, output), Err(Error::Unsupported(_))), + "{input:?} -> {output:?}" + ); + } + + assert_eq!( + remix(&[1.0, 2.0, 3.0], Layout::Discrete(3), Layout::Discrete(3)).unwrap(), + [1.0, 2.0, 3.0] + ); + assert_eq!( + remix(&[1.0, 2.0, 3.0], Layout::TwoPointOne, Layout::Discrete(3)).unwrap(), + [1.0, 2.0, 3.0] + ); + assert!(Remix::new(Layout::Discrete(3), Layout::TwoPointOne).is_err()); + } + + /// Every pair of named layouts converts, and the mix is sized to the output. + #[test] + fn remix_converts_between_every_named_layout() { + let layouts: Vec = (1..=8) + .map(|n| Layout::from_channels(n).unwrap()) + .chain([Layout::ThreePointZero, Layout::FourPointZero]) + .collect(); + for &input in &layouts { + for &output in &layouts { + let frame = vec![0.5; input.channels() as usize * 2]; + let mixed = remix(&frame, input, output).unwrap(); + assert_eq!(mixed.len(), output.channels() as usize * 2, "{input:?} -> {output:?}"); + } + } + } } diff --git a/rs/moq-audio/tests/roundtrip.rs b/rs/moq-audio/tests/roundtrip.rs index 570dacac45..55e09bfab3 100644 --- a/rs/moq-audio/tests/roundtrip.rs +++ b/rs/moq-audio/tests/roundtrip.rs @@ -203,3 +203,71 @@ async fn pcm_round_trip_is_lossless() { assert_eq!(decoded, samples); assert!(consumer.read().await.unwrap().is_none()); } + +/// A 5.1 broadcast carries only its channel count in the catalog, which reads +/// back as 5.1 in canonical order and remixes to whatever the subscriber asks +/// for. +#[tokio::test] +async fn pcm_five_one_reads_back_at_any_layout() { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); + let mut catalog_consumer = catalog.consume().unwrap(); + let broadcast_consumer = broadcast.consume(); + + let input = encode::Input::new(48_000, Layout::FivePointOne); + let mut options = encode::Options::default(); + options.track = Some("pcm".to_string()); + options.settings = encode::Settings::new(48_000, Layout::FivePointOne); + options.settings.codec = encode::Codec::Pcm; + + // A level per speaker: left, right, center, LFE, side left, side right. + let speakers = [0.1f32, 0.2, 0.3, 0.4, 0.05, 0.06]; + let mut producer = encode::Producer::new(&mut broadcast, catalog.clone(), input, &options).unwrap(); + producer + .write(&Frame::new( + f32_bytes(&speakers.repeat(960)), + Timestamp::from_micros(0).unwrap(), + )) + .unwrap(); + + let snapshot = catalog_consumer.next().await.unwrap().unwrap(); + let rendition = snapshot.audio.renditions.get("pcm").unwrap(); + assert_eq!(rendition.channel_count, 6); + + let h = std::f32::consts::FRAC_1_SQRT_2; + let left = 0.1 + h * 0.3 + h * 0.05; + let right = 0.2 + h * 0.3 + h * 0.06; + let cases = [ + (None, Layout::FivePointOne, speakers.to_vec()), + (Some(Layout::Stereo), Layout::Stereo, vec![left, right]), + (Some(Layout::Mono), Layout::Mono, vec![(left + right) * 0.5]), + ]; + + let mut consumers = Vec::new(); + for (output, _, _) in &cases { + let mut options = decode::Options::default(); + options.output.layout = *output; + consumers.push( + decode::Consumer::new(&broadcast_consumer, rendition, "pcm", options) + .await + .unwrap(), + ); + } + producer.finish().unwrap(); + + for (mut consumer, (_, layout, frame)) in consumers.into_iter().zip(cases) { + assert_eq!(consumer.layout(), layout); + let decoded = consumer.read().await.unwrap().unwrap(); + let samples: Vec = decoded + .data + .as_chunks::<4>() + .0 + .iter() + .map(|sample| f32::from_le_bytes([sample[0], sample[1], sample[2], sample[3]])) + .collect(); + assert_eq!(samples.len(), 960 * frame.len(), "{layout:?}"); + for (got, want) in samples.iter().zip(frame.iter().cycle()) { + assert!((got - want).abs() < 1e-6, "{layout:?}: got {got}, want {want}"); + } + } +} diff --git a/rs/moq-ffi/src/audio.rs b/rs/moq-ffi/src/audio.rs index 16750c1375..5e8bea78ce 100644 --- a/rs/moq-ffi/src/audio.rs +++ b/rs/moq-ffi/src/audio.rs @@ -76,6 +76,9 @@ impl MoqAudioCodec { pub struct MoqAudioEncoderInput { pub format: MoqAudioSampleFormat, pub sample_rate: u32, + /// Interleaved channel count, which also names the speaker layout by the + /// WAVE convention: 1 mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, + /// 8 7.1, in front left, front right, center, LFE, back, side order. pub channels: u32, } @@ -105,7 +108,9 @@ pub struct MoqAudioDecoderOutput { /// `None` delivers samples at the codec's native rate. #[uniffi(default = None)] pub sample_rate: Option, - /// `None` delivers samples at the codec's native channel count. + /// `None` delivers samples at the codec's native channel count. A count + /// names its layout as [`MoqAudioEncoderInput::channels`] describes, and + /// the decoder remixes to it. #[uniffi(default = None)] pub channels: Option, /// Upper bound on buffering before skipping a stalled group, in From c1cdb920b72a889fc48b7972a23f225e4178bad3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 08:10:26 -0700 Subject: [PATCH 05/11] feat(audio): a decode backend seam that prefers the platform codec (#4131) Co-authored-by: Claude Opus 5.5 --- doc/bin/cli.md | 10 +- doc/lib/rs/moq-audio.md | 17 +- quest/m1/audio-codecs/README.md | 1 - quest/m1/audio-codecs/decode-audiotoolbox.md | 18 +- quest/m1/audio-codecs/decode-backend.md | 41 --- quest/m1/audio-codecs/encode-backend.md | 6 +- quest/m2/audio-decode-mediacodec.md | 4 - quest/m2/audio-decode-mediafoundation.md | 4 - rs/moq-audio/src/decode/backend/libopus.rs | 126 +++++++ rs/moq-audio/src/decode/backend/mod.rs | 303 +++++++++++++++++ rs/moq-audio/src/decode/backend/pcm.rs | 78 +++++ rs/moq-audio/src/decode/backend/symphonia.rs | 106 ++++++ rs/moq-audio/src/decode/consumer.rs | 5 + rs/moq-audio/src/decode/decoder.rs | 325 +++---------------- rs/moq-audio/src/decode/mod.rs | 5 +- rs/moq-cli/src/play/media.rs | 2 +- 16 files changed, 692 insertions(+), 359 deletions(-) delete mode 100644 quest/m1/audio-codecs/decode-backend.md create mode 100644 rs/moq-audio/src/decode/backend/libopus.rs create mode 100644 rs/moq-audio/src/decode/backend/mod.rs create mode 100644 rs/moq-audio/src/decode/backend/pcm.rs create mode 100644 rs/moq-audio/src/decode/backend/symphonia.rs diff --git a/doc/bin/cli.md b/doc/bin/cli.md index cbd4d9d03a..723b80a3e4 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -87,11 +87,11 @@ moq --connect https://relay.example.com/anon --broadcast my-stream.hang play moq ... play --delay 500ms # trade latency for a jittery link ``` -Decodes H.264, H.265, and AV1 video and Opus, PCM, and AAC-LC audio using -the platform hardware decoder where available. `--video-name` and -`--audio-name` pick a rendition. -HE-AAC signaled only in band (implicit SBR, as over MPEG-TS) plays as its -half-rate AAC-LC core. +Decodes H.264, H.265, and AV1 video using the platform hardware decoder where +available, and Opus, PCM, and AAC-LC (mono or stereo) audio in software. The +log names the decoder each track opened. `--video-name` and `--audio-name` +pick a rendition. HE-AAC signaled only in band (implicit SBR, as over MPEG-TS) +plays as its half-rate AAC-LC core. Playback runs on a clock it owns. `--delay` (default 100 ms) is how far it trails the live edge, which is both the jitter a late frame may absorb and the diff --git a/doc/lib/rs/moq-audio.md b/doc/lib/rs/moq-audio.md index 2d57b4a5bc..637343dd61 100644 --- a/doc/lib/rs/moq-audio.md +++ b/doc/lib/rs/moq-audio.md @@ -33,9 +33,20 @@ policy. Decoding likewise separates low-level `decode::Config`, PCM | `playback` | One output device mixing every track in a call, with click-free volume ramps | | `aec` | Acoustic echo cancellation (a port of WebRTC's), so a laptop with no headset doesn't feed itself back | -AAC decoding refuses HE-AAC its config declares. HE-AAC signaled only in band -(implicit SBR, as over MPEG-TS) goes undetected and plays as its half-rate -AAC-LC core. +`decode` picks a backend per track the way `moq-video` does: a platform decoder +first, then software. `decode::Config::kind` forces one (`Kind::Software`, or +`Kind::Named` with a name below), and `Decoder::name()` reports what opened. + +| Backend | Decodes | Hosts | +| --- | --- | --- | +| `libopus` | Opus, mono or stereo | all | +| `pcm` | PCM | all | +| `symphonia` | AAC-LC, mono or stereo (the default-on `aac` feature) | all | + +No platform decoder is wired in yet, so multichannel AAC and HE-AAC declared in +its config are refused at construction on every host. HE-AAC signaled only in +band plays as its half-rate LC core. Linux has no OS audio decoder, so it will +stay that way there. Highlights: diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index 78912f392c..9b3763d7f1 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -44,7 +44,6 @@ its own decode and encode quest so verification stays per host. ## Quests - [TS export PCE](/quest/m1/audio-codecs/ts-export-pce.md) - a TS export of a PCE-described AAC track writes channel_config 0 and the PCE instead of a count-derived config -- [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - `decode::backend` selects a platform decoder before symphonia, mirroring moq-video - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - mapping family 1 decodes on every host through the multistream decoder - [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - `encode::backend` and `Codec::Aac`, so a native publisher can produce AAC-LC diff --git a/quest/m1/audio-codecs/decode-audiotoolbox.md b/quest/m1/audio-codecs/decode-audiotoolbox.md index e81db0d554..cda8f73b5c 100644 --- a/quest/m1/audio-codecs/decode-audiotoolbox.md +++ b/quest/m1/audio-codecs/decode-audiotoolbox.md @@ -10,9 +10,14 @@ documents as unsupported. ## Plan An `AudioConverter` from the packetized format to interleaved `f32` at the -codec's native rate and layout, behind the decode seam as the first platform -candidate on `target_os = "macos"` and `"ios"`. `objc2-audio-toolbox` is the -binding, alongside the `objc2-core-audio-types` the crate already carries. +codec's native rate and layout, behind the decode seam +(`rs/moq-audio/src/decode/backend`) as the first platform candidate on +`target_os = "macos"` and `"ios"`. `objc2-audio-toolbox` is the binding, +alongside the `objc2-core-audio-types` the crate already carries. + +- With a platform tier in place, `Auto` falling past a refusing platform + decoder to software should warn, as moq-video's `select` does; the seam + only aggregates the refusals into its error today. - Build the `AudioStreamBasicDescription` and magic cookie from the catalog description; the converter reports the output layout, which maps to @@ -20,7 +25,8 @@ binding, alongside the `objc2-core-audio-types` the crate already carries. - HE-AAC: the converter reads SBR in band and reports the doubled rate; the seam passes it through. No config-level guessing. - Priming and remainder: AudioToolbox reports `kAudioConverterPrimeInfo`; - trim it so timestamps line up with symphonia's output on the same stream. + report it as the backend's startup delay, which the front end trims, so + timestamps line up with symphonia's output on the same stream. - Every codec the backend advertises has a fixture and a decode test, and the test asserts the layout order matches the canonical one (the LFE and centre end up where `Layout` says). @@ -29,10 +35,6 @@ binding, alongside the `objc2-core-audio-types` the crate already carries. - Docs: `doc/bin/obs.md` drops the HE-AAC and multichannel caveat on macOS, and the backend table names what this host decodes. -## Required - -- [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the candidate order this backend joins - ## Related - [Media Foundation decode](/quest/m2/audio-decode-mediafoundation.md) - the same shape on Windows diff --git a/quest/m1/audio-codecs/decode-backend.md b/quest/m1/audio-codecs/decode-backend.md deleted file mode 100644 index 271ef84b52..0000000000 --- a/quest/m1/audio-codecs/decode-backend.md +++ /dev/null @@ -1,41 +0,0 @@ -# [M] A decode backend seam that prefers the platform codec - -## Goal - -`moq_audio::decode` selects a backend per codec the way `moq_video::decode` -does: platform first, software fallback, and a `Kind` to force one. Opus, -PCM, and symphonia AAC-LC become the software backends, and the crate -documents which codecs each host decodes. - -## Plan - -Mirror `rs/moq-video/src/decode/backend` in name and shape: a crate-private -`Backend` trait (`decode`, `flush`, `name`), an `open(codec, config)` that -walks the platform candidates before the software ones and refuses when none -takes the track. Use the backend-selection configuration and Decoder -constructor settled in main rather than adding a conflicting public shape. -`Decoder::name()` reports what was opened, which the OBS -stats and `moq play` surface. - -- The seam is generic over `hang::catalog::AudioCodec`, so a backend advertises - the set it opens and the selector asks each in order. Symphonia advertises - AAC-LC mono/stereo only; the platform backends that follow advertise what - their framework opens and has a fixture for. -- Move today's Opus, PCM, and symphonia code behind the trait without changing - behavior. Document per host that symphonia plays implicit-SBR HE-AAC as its - half-rate LC core: finding the in-band SBR element needs a full Huffman walk - of the channel elements, and symphonia detects it internally without - exposing or refusing it. -- A backend's output rate and layout are what it produced, not what the - catalog said (HE-AAC doubles the rate); `Consumer` already resamples and - remixes to the requested output, so that stays the seam's contract. -- The `aac` feature keeps gating symphonia. Platform backends are - `cfg(target_os)` like their video counterparts. Audio has no MediaCodec - feature yet; its Android quest introduces the optional dependency following - the settled media build policy. -- Docs: `doc/lib/rs/moq-audio.md` gains the backend table `moq-video.md` has, - and states the Linux gap. `doc/bin/cli.md` and `doc/bin/obs.md` follow. -- Regression: the selection order and `Named` refusal, tested with a stub - backend like the video seam's `probe`. - -The FFI does not expose `Kind` until a consumer asks. diff --git a/quest/m1/audio-codecs/encode-backend.md b/quest/m1/audio-codecs/encode-backend.md index 5c6dbc1073..4f73ed98f2 100644 --- a/quest/m1/audio-codecs/encode-backend.md +++ b/quest/m1/audio-codecs/encode-backend.md @@ -9,7 +9,7 @@ host with no AAC encoder refuses it at construction. ## Plan -Mirror the decode seam: `encode::backend` with a crate-private `Backend` +Mirror the decode seam (`rs/moq-audio/src/decode/backend`): `encode::backend` with a crate-private `Backend` trait (`encode`, `flush`, `set_bitrate`, `name`), an `open(codec, config)` that walks platform candidates before software ones, using the public settings and selection contract settled in main. Opus and PCM retain their behavior. This @@ -35,10 +35,6 @@ quest adds AAC through platform encoders; no software AAC dependency is selected - Regression: the selection order with a stub backend; `Codec::Aac` refused on a host with no backend; the Opus and PCM paths unchanged. -## Required - -- [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the naming and shape this mirrors - ## Related - [OBS audio publishing](/quest/m1/obs-moq-video/audio-publish.md) - the OBS encoder adapter can offer AAC once this lands diff --git a/quest/m2/audio-decode-mediacodec.md b/quest/m2/audio-decode-mediacodec.md index 5b7df87c71..1446550bf0 100644 --- a/quest/m2/audio-decode-mediacodec.md +++ b/quest/m2/audio-decode-mediacodec.md @@ -23,10 +23,6 @@ behind a new optional audio `mediacodec` feature and the decode seam, on `target - The binding ships in the moq-ffi Android slice, which is how Kotlin and Dart reach it. -## Required - -- [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the candidate order this backend joins - ## Related - [Android capture](/quest/m2/mobile-capture-android.md) - the video MediaCodec family this sits beside diff --git a/quest/m2/audio-decode-mediafoundation.md b/quest/m2/audio-decode-mediafoundation.md index 1da6a8337c..37fc31fca7 100644 --- a/quest/m2/audio-decode-mediafoundation.md +++ b/quest/m2/audio-decode-mediafoundation.md @@ -22,10 +22,6 @@ ones. Behind the decode seam as the first candidate on `target_os = - Verification runs on a Windows host; the per-PR CI only compiles the platform code, and `just rs windows` runs nightly. -## Required - -- [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the candidate order this backend joins - ## Related - [Runtime QA hosts](/quest/m2/runtime-qa-hosts.md) - where the Windows run happens diff --git a/rs/moq-audio/src/decode/backend/libopus.rs b/rs/moq-audio/src/decode/backend/libopus.rs new file mode 100644 index 0000000000..0fa9f78b69 --- /dev/null +++ b/rs/moq-audio/src/decode/backend/libopus.rs @@ -0,0 +1,126 @@ +//! Opus through libopus, the software decoder for every host. + +use unsafe_libopus::{ + OPUS_OK, OPUS_RESET_STATE, OpusDecoder, opus_decode_float, opus_decoder_create, opus_decoder_ctl_impl, + opus_decoder_destroy, varargs, +}; + +use super::Backend; +use crate::decode::Decoded; +use crate::{Error, Layout, opus}; + +pub(super) const NAME: &str = "libopus"; + +/// Opus packets cap at 120 ms (RFC 6716 §2.1.4). +const MAX_FRAME_MS: usize = 120; + +pub(super) struct Libopus { + inner: *mut OpusDecoder, + sample_rate: u32, + layout: Layout, + pre_skip: usize, + max_frame_size: usize, + in_dtx: bool, +} + +// SAFETY: the decoder is owned exclusively and libopus keeps no thread-local state. +unsafe impl Send for Libopus {} + +impl Libopus { + /// Parses the OpusHead `description` if present; falls back to the catalog's + /// declared sample rate / channel count. + pub(super) fn open(catalog: &hang::catalog::AudioConfig) -> Result, Error> { + let (sample_rate, channel_count, pre_skip) = if let Some(desc) = &catalog.description { + let mut buf = desc.as_ref(); + match moq_mux::codec::opus::Config::parse(&mut buf) { + Ok(head) => (head.sample_rate, head.channel_count, head.pre_skip), + Err(_) => (catalog.sample_rate, catalog.channel_count, 0), + } + } else { + (catalog.sample_rate, catalog.channel_count, 0) + }; + + opus::validate_rate(sample_rate)?; + let channels = opus::validate_channels(channel_count)?; + let layout = Layout::from_channels(channel_count)?; + + let mut err = 0i32; + // SAFETY: out-pointer is valid; inner is checked for null below. + let inner = unsafe { opus_decoder_create(sample_rate as i32, channels, &mut err) }; + if err != OPUS_OK || inner.is_null() { + return Err(opus::error(err, "opus_decoder_create")); + } + + Ok(Box::new(Self { + inner, + sample_rate, + layout, + // OpusHead counts pre-skip at 48 kHz whatever rate the decoder runs at. + pre_skip: (pre_skip as usize * sample_rate as usize) / 48_000, + max_frame_size: (sample_rate as usize * MAX_FRAME_MS) / 1000, + in_dtx: false, + })) + } +} + +impl Backend for Libopus { + /// Empty packets invoke packet-loss concealment. Loss during DTX remains + /// classified as DTX, while loss during active audio remains active. + fn decode(&mut self, packet: &[u8]) -> Result { + let channels = self.layout.channels() as usize; + let mut out = vec![0.0f32; self.max_frame_size * channels]; + // SAFETY: `inner` owns a live OpusDecoder; packet/out slices are bounded by + // the lengths we pass. + let samples = unsafe { + opus_decode_float( + &mut *self.inner, + packet.as_ptr(), + packet.len() as i32, + out.as_mut_ptr(), + self.max_frame_size as i32, + 0, + ) + }; + if samples < 0 { + return Err(opus::decode_error(samples)); + } + out.truncate(samples as usize * channels); + + let activity = opus::activity(packet, self.in_dtx); + self.in_dtx = activity.is_dtx(); + Ok(Decoded { samples: out, activity }) + } + + fn reset(&mut self) -> Result<(), Error> { + // SAFETY: `inner` owns a live decoder and OPUS_RESET_STATE takes no arguments. + let rc = unsafe { opus_decoder_ctl_impl(self.inner, OPUS_RESET_STATE, varargs![]) }; + if rc != OPUS_OK { + return Err(opus::error(rc, "OPUS_RESET_STATE")); + } + self.in_dtx = false; + Ok(()) + } + + fn sample_rate(&self) -> u32 { + self.sample_rate + } + + fn layout(&self) -> Layout { + self.layout + } + + fn delay(&self) -> usize { + self.pre_skip + } + + fn name(&self) -> &str { + NAME + } +} + +impl Drop for Libopus { + fn drop(&mut self) { + // SAFETY: `inner` is a live OpusDecoder that nothing else aliases. + unsafe { opus_decoder_destroy(self.inner) }; + } +} diff --git a/rs/moq-audio/src/decode/backend/mod.rs b/rs/moq-audio/src/decode/backend/mod.rs new file mode 100644 index 0000000000..f39bb4c170 --- /dev/null +++ b/rs/moq-audio/src/decode/backend/mod.rs @@ -0,0 +1,303 @@ +//! Pluggable audio decoder backends. +//! +//! The audio mirror of `moq-video`'s decode backends. [`Backend`] is the seam +//! between the codec and the [`Decoder`](super::Decoder) front end, which owns +//! what every codec shares: trimming the startup delay a backend reports. +//! +//! [`open`] tries the platform decoders before the software ones, skipping any +//! that does not advertise the catalog codec, and refuses when none opens the +//! track. The software tier is libopus for Opus, a passthrough for PCM, and +//! symphonia for AAC-LC mono and stereo (behind the `aac` feature). No platform +//! decoder is wired in yet. + +use hang::catalog::{AudioCodec, AudioConfig}; + +use super::Decoded; +use super::decoder::{Config, Kind}; +use crate::{Error, Layout}; + +mod libopus; +mod pcm; +#[cfg(feature = "aac")] +mod symphonia; + +/// An opened decoder: packets in, interleaved `f32` PCM out. +pub(crate) trait Backend: Send { + /// Decode one packet into interleaved samples at [`sample_rate`](Self::sample_rate) + /// and [`layout`](Self::layout), untrimmed. + fn decode(&mut self, packet: &[u8]) -> Result; + + /// Drop codec history after a discontinuity, so the next packet does not + /// predict from audio that is no longer adjacent. + fn reset(&mut self) -> Result<(), Error>; + + /// The rate this backend decodes to, which may differ from the catalog's. + fn sample_rate(&self) -> u32; + + /// The layout this backend decodes to, which may differ from the catalog's. + fn layout(&self) -> Layout; + + /// Frames at the start of the stream that are codec priming, not media. + fn delay(&self) -> usize { + 0 + } + + /// The stable lowercase name [`Kind::Named`] selects this backend by. + fn name(&self) -> &str; +} + +type Open = fn(&AudioConfig) -> Result, Error>; + +/// A backend constructor: its name, the catalog codecs it advertises, and an opener. +struct Candidate { + name: &'static str, + supports: fn(&AudioCodec) -> bool, + open: Open, +} + +/// Operating-system decoders, in priority order. +const PLATFORM: &[Candidate] = &[]; + +const SOFTWARE: &[Candidate] = &[ + Candidate { + name: libopus::NAME, + supports: |codec| matches!(codec, AudioCodec::Opus), + open: libopus::Libopus::open, + }, + Candidate { + name: pcm::NAME, + supports: |codec| matches!(codec, AudioCodec::Pcm), + open: pcm::Pcm::open, + }, + // Claims every AAC profile and refuses at open what it can't decode: the + // profile that matters is the description's, which the catalog string can + // contradict. + #[cfg(feature = "aac")] + Candidate { + name: symphonia::NAME, + supports: |codec| matches!(codec, AudioCodec::AAC(_)), + open: symphonia::Symphonia::open, + }, +]; + +/// Open the first backend that advertises the catalog codec and accepts the track. +pub(crate) fn open(catalog: &AudioConfig, config: &Config) -> Result, Error> { + select(catalog, &config.kind, candidates(&config.kind, PLATFORM, SOFTWARE)) +} + +/// The candidates `kind` allows, in the order to try them. +/// +/// Takes the tiers as arguments so a test can supply stubs instead of whatever +/// this host compiles in. +fn candidates<'a>(kind: &Kind, platform: &'a [Candidate], software: &'a [Candidate]) -> Vec<&'a Candidate> { + match kind { + Kind::Auto => platform.iter().chain(software).collect(), + Kind::Software => software.iter().collect(), + Kind::Named(name) => platform.iter().chain(software).filter(|c| c.name == name).collect(), + } +} + +fn select(catalog: &AudioConfig, kind: &Kind, candidates: Vec<&Candidate>) -> Result, Error> { + let codec = &catalog.codec; + let mut refused = Vec::new(); + + for candidate in candidates { + if !(candidate.supports)(codec) { + continue; + } + match (candidate.open)(catalog) { + Ok(backend) => return Ok(backend), + Err(err) => refused.push((candidate.name, err)), + } + } + + // One refusal is the whole answer, so keep its variant: a malformed + // description stays a container error rather than becoming a string. + if refused.len() == 1 { + let (_, err) = refused.remove(0); + return Err(err); + } + if !refused.is_empty() { + let reasons: Vec = refused.iter().map(|(name, err)| format!("{name}: {err}")).collect(); + return Err(Error::Unsupported(reasons.join(", "))); + } + + match kind { + Kind::Named(name) => { + let available: Vec<&str> = PLATFORM + .iter() + .chain(SOFTWARE) + .filter(|c| (c.supports)(codec)) + .map(|c| c.name) + .collect(); + Err(Error::Unsupported(format!( + "no audio decoder named {name:?} for {codec} (this build has: {})", + available.join(", ") + ))) + } + _ => Err(Error::Unsupported(format!("unsupported audio codec: {codec}"))), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::Activity; + + /// Opens anything it advertises and reports which candidate it came from. + struct Stub(&'static str); + + impl Backend for Stub { + fn decode(&mut self, _packet: &[u8]) -> Result { + Ok(Decoded { + samples: Vec::new(), + activity: Activity::Active, + }) + } + + fn reset(&mut self) -> Result<(), Error> { + Ok(()) + } + + fn sample_rate(&self) -> u32 { + 48_000 + } + + fn layout(&self) -> Layout { + Layout::Stereo + } + + fn name(&self) -> &str { + self.0 + } + } + + const PLATFORM_STUB: Candidate = Candidate { + name: "platform", + supports: |codec| matches!(codec, AudioCodec::Opus), + open: |_| Ok(Box::new(Stub("platform"))), + }; + + /// Compiled in but refusing the track, like a platform decoder asked for a + /// layout its framework does not open. + const REFUSING: Candidate = Candidate { + name: "refusing", + supports: |codec| matches!(codec, AudioCodec::Opus), + open: |_| Err(Error::Unsupported("not this track".into())), + }; + + const SOFTWARE_STUB: Candidate = Candidate { + name: "software", + supports: |codec| matches!(codec, AudioCodec::Opus), + open: |_| Ok(Box::new(Stub("software"))), + }; + + /// Advertises nothing but PCM, so an Opus track never reaches its opener. + const PCM_ONLY: Candidate = Candidate { + name: "pcm-only", + supports: |codec| matches!(codec, AudioCodec::Pcm), + open: |_| panic!("opened for a codec it does not advertise"), + }; + + fn opus() -> AudioConfig { + AudioConfig::new(AudioCodec::Opus, 48_000, 2) + } + + fn pick(kind: Kind, platform: &[Candidate], software: &[Candidate]) -> Result { + let backend = select(&opus(), &kind, candidates(&kind, platform, software))?; + Ok(backend.name().to_owned()) + } + + #[test] + fn auto_prefers_platform() { + let name = pick(Kind::Auto, &[PCM_ONLY, PLATFORM_STUB], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "platform"); + } + + #[test] + fn auto_falls_back_to_software() { + let name = pick(Kind::Auto, &[REFUSING], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "software"); + } + + #[test] + fn software_skips_platform() { + let name = pick(Kind::Software, &[PLATFORM_STUB], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "software"); + } + + #[test] + fn named_forces_one() { + let name = pick( + Kind::Named("software".into()), + &[PLATFORM_STUB], + &[PCM_ONLY, SOFTWARE_STUB], + ) + .unwrap(); + assert_eq!(name, "software"); + } + + /// A named backend that refuses the track is the answer: nothing else is tried. + #[test] + fn named_refusal_does_not_fall_back() { + let err = pick(Kind::Named("refusing".into()), &[REFUSING], &[SOFTWARE_STUB]).unwrap_err(); + assert!(err.to_string().contains("not this track"), "{err}"); + } + + #[test] + fn every_refusal_is_reported() { + const ALSO_REFUSING: Candidate = Candidate { + name: "also-refusing", + ..REFUSING + }; + + let err = pick(Kind::Auto, &[REFUSING], &[ALSO_REFUSING]).unwrap_err(); + let message = err.to_string(); + assert!( + message.contains("refusing: ") && message.contains("also-refusing: "), + "{message}" + ); + } + + /// An unknown name says what this build has for the codec instead. + #[test] + fn unknown_name_lists_the_alternatives() { + let err = open( + &opus(), + &Config { + kind: Kind::Named("opus".into()), + }, + ) + .err() + .expect("no backend is named after its codec"); + let message = err.to_string(); + assert!( + message.contains("\"opus\"") && message.contains(libopus::NAME), + "{message}" + ); + } + + /// Asking for a real backend that does not decode the codec is refused, not + /// quietly swapped for one that does. + #[test] + fn named_backend_for_another_codec_is_refused() { + let config = Config { + kind: Kind::Named(pcm::NAME.into()), + }; + assert!(matches!(open(&opus(), &config), Err(Error::Unsupported(_)))); + } + + #[test] + fn software_backends_open_by_name() { + let pcm = AudioConfig::new(AudioCodec::Pcm, 48_000, 2); + let config = Config { + kind: Kind::Named(pcm::NAME.into()), + }; + assert_eq!(open(&pcm, &config).unwrap().name(), pcm::NAME); + + let config = Config { + kind: Kind::Named(libopus::NAME.into()), + }; + assert_eq!(open(&opus(), &config).unwrap().name(), libopus::NAME); + } +} diff --git a/rs/moq-audio/src/decode/backend/pcm.rs b/rs/moq-audio/src/decode/backend/pcm.rs new file mode 100644 index 0000000000..bead86fb5e --- /dev/null +++ b/rs/moq-audio/src/decode/backend/pcm.rs @@ -0,0 +1,78 @@ +//! Uncompressed little-endian `f32` PCM, described entirely by the catalog. + +use super::Backend; +use crate::decode::Decoded; +use crate::{Activity, Error, Layout, pcm}; + +pub(super) const NAME: &str = "pcm"; + +pub(super) struct Pcm { + sample_rate: u32, + layout: Layout, + bytes_per_frame: usize, +} + +impl Pcm { + /// Uses the catalog's rate and channel count, and requires an absent `description`. + pub(super) fn open(catalog: &hang::catalog::AudioConfig) -> Result, Error> { + if catalog.sample_rate == 0 { + return Err(Error::Unsupported("pcm sample rate must be greater than zero".into())); + } + if catalog.channel_count == 0 { + return Err(Error::Unsupported("pcm channel count must be greater than zero".into())); + } + if catalog.description.is_some() { + return Err(Error::Unsupported("pcm catalog description must be absent".into())); + } + let bitrate = pcm::bitrate(catalog.sample_rate, catalog.channel_count)?; + if catalog.bitrate.is_some_and(|declared| declared != bitrate) { + return Err(Error::Unsupported(format!( + "pcm catalog bitrate must be {bitrate} bits per second" + ))); + } + + Ok(Box::new(Self { + sample_rate: catalog.sample_rate, + layout: Layout::from_channels(catalog.channel_count)?, + bytes_per_frame: pcm::frame_bytes(1, catalog.channel_count)?, + })) + } +} + +impl Backend for Pcm { + fn decode(&mut self, packet: &[u8]) -> Result { + if packet.is_empty() || !packet.len().is_multiple_of(self.bytes_per_frame) { + return Err(Error::Misaligned { + got: packet.len(), + expected: packet.len().max(1).next_multiple_of(self.bytes_per_frame), + }); + } + + let samples = packet + .as_chunks::<{ pcm::BYTES_PER_SAMPLE }>() + .0 + .iter() + .map(|sample| f32::from_le_bytes(*sample)) + .collect(); + Ok(Decoded { + samples, + activity: Activity::Active, + }) + } + + fn reset(&mut self) -> Result<(), Error> { + Ok(()) + } + + fn sample_rate(&self) -> u32 { + self.sample_rate + } + + fn layout(&self) -> Layout { + self.layout + } + + fn name(&self) -> &str { + NAME + } +} diff --git a/rs/moq-audio/src/decode/backend/symphonia.rs b/rs/moq-audio/src/decode/backend/symphonia.rs new file mode 100644 index 0000000000..e2a87c6253 --- /dev/null +++ b/rs/moq-audio/src/decode/backend/symphonia.rs @@ -0,0 +1,106 @@ +//! AAC-LC through symphonia, the pure-Rust fallback for hosts without a +//! platform AAC decoder. + +use symphonia_core::codecs::audio::AudioDecoder; + +use super::Backend; +use crate::decode::Decoded; +use crate::{Activity, Error, Layout, aac}; + +pub(super) const NAME: &str = "symphonia"; + +pub(super) struct Symphonia { + inner: symphonia_codec_aac::AacDecoder, + sample_rate: u32, + layout: Layout, +} + +impl Symphonia { + /// AAC-LC in mono or stereo only, which is what every gateway that feeds + /// this crate publishes. + /// + /// HE-AAC is rejected however its config spells it: leading with SBR or PS + /// (mp4a.40.5 / .29), or leading with LC and declaring SBR in a sync extension + /// after the core. Symphonia decodes no SBR either way, so the alternative is + /// half-rate audio that sounds like a fault rather than an unsupported codec. + /// A stream that signals SBR only in band is indistinguishable from LC in the + /// config, and does decode as the core. + pub(super) fn open(catalog: &hang::catalog::AudioConfig) -> Result, Error> { + use symphonia_core::codecs::audio::well_known::CODEC_ID_AAC; + use symphonia_core::codecs::audio::{AudioCodecParameters, AudioDecoderOptions}; + + let hang::catalog::AudioCodec::AAC(codec) = &catalog.codec else { + return Err(Error::Unsupported(format!("symphonia cannot decode {}", catalog.codec))); + }; + let description = aac::description(catalog, codec.profile)?; + + let mut params = AudioCodecParameters::new(); + params + .for_codec(CODEC_ID_AAC) + .with_extra_data(description.to_vec().into_boxed_slice()); + + let inner = symphonia_codec_aac::AacDecoder::try_new(¶ms, &AudioDecoderOptions::default()) + .map_err(|err| Error::Unsupported(format!("aac decoder: {err}")))?; + + // Resolved by the decoder from the config, so this is what it will emit + // even when the catalog's own fields say otherwise. + let params = inner.codec_params(); + let sample_rate = params + .sample_rate + .ok_or_else(|| Error::Unsupported("aac config declares no sample rate".into()))?; + let channel_count = params + .channels + .as_ref() + .map(|channels| channels.count()) + .ok_or_else(|| Error::Unsupported("aac config declares no channels".into()))?; + + Ok(Box::new(Self { + inner, + sample_rate, + layout: Layout::from_channels(channel_count as u32)?, + })) + } +} + +impl Backend for Symphonia { + fn decode(&mut self, packet: &[u8]) -> Result { + // The packet is a raw AAC frame, not ADTS, so there is nothing to timestamp + // it with here: the container carries the timestamp and the decoder only + // reads the payload. + let packet = symphonia_core::packet::PacketRef::new( + 0, + symphonia_core::units::Timestamp::ZERO, + symphonia_core::units::Duration::ZERO, + packet, + ); + + let decoded = self + .inner + .decode_ref(&packet) + .map_err(|err| Error::Decode(format!("aac: {err}")))?; + + let mut samples = Vec::new(); + decoded.copy_to_vec_interleaved(&mut samples); + Ok(Decoded { + samples, + activity: Activity::Active, + }) + } + + fn reset(&mut self) -> Result<(), Error> { + self.inner.reset(); + Ok(()) + } + + fn sample_rate(&self) -> u32 { + self.sample_rate + } + + fn layout(&self) -> Layout { + self.layout + } + + fn name(&self) -> &str { + NAME + } +} diff --git a/rs/moq-audio/src/decode/consumer.rs b/rs/moq-audio/src/decode/consumer.rs index 5110db70e5..b5f8ccdce8 100644 --- a/rs/moq-audio/src/decode/consumer.rs +++ b/rs/moq-audio/src/decode/consumer.rs @@ -187,6 +187,11 @@ impl Consumer { }) } + /// The decoder backend name in use, e.g. `"libopus"` or `"symphonia"`. + pub fn name(&self) -> &str { + self.decoder.name() + } + /// The options this consumer was built with. pub fn options(&self) -> &Options { &self.options diff --git a/rs/moq-audio/src/decode/decoder.rs b/rs/moq-audio/src/decode/decoder.rs index 5f85d66c1c..9cb9d7954f 100644 --- a/rs/moq-audio/src/decode/decoder.rs +++ b/rs/moq-audio/src/decode/decoder.rs @@ -1,36 +1,24 @@ //! Audio decoder front end. //! -//! Mirror of [`encode::Encoder`](crate::encode::Encoder): dispatches over the -//! catalog codec and produces interleaved `f32` PCM. - -use unsafe_libopus::{ - OPUS_OK, OPUS_RESET_STATE, OpusDecoder, opus_decode_float, opus_decoder_create, opus_decoder_ctl_impl, - opus_decoder_destroy, varargs, -}; - -#[cfg(feature = "aac")] -use symphonia_core::codecs::audio::AudioDecoder; +//! Mirror of [`encode::Encoder`](crate::encode::Encoder): opens a +//! [`Backend`](super::backend::Backend) for the catalog codec and trims its +//! startup delay, producing interleaved `f32` PCM. use super::Decoded; -#[cfg(feature = "aac")] -use crate::aac; -use crate::opus; -use crate::pcm; -use crate::{Activity, Error, Layout}; - -/// Opus packets cap at 120 ms (RFC 6716 §2.1.4). -const MAX_FRAME_MS: usize = 120; +use super::backend::{self, Backend}; +use crate::{Error, Layout}; /// Decoder backend selection. #[derive(Clone, Debug, Default, PartialEq, Eq)] #[non_exhaustive] pub enum Kind { - /// Pick the available backend automatically. + /// Prefer a platform decoder, falling back to software. #[default] Auto, - /// Require the built-in software backend. + /// Require a software backend. Software, - /// Require a backend by its stable lowercase name. + /// Require a backend by its stable lowercase name: `"libopus"`, `"pcm"`, or + /// `"symphonia"`. Named(String), } @@ -54,184 +42,42 @@ impl Config { /// The bring-your-own-payload layer under [`Consumer`](super::Consumer): use it /// when the packets don't come from a plain track subscription. pub struct Decoder { - backend: Backend, - sample_rate: u32, - layout: Layout, + backend: Box, + /// Startup delay in native-rate frames, and how much of it is left to trim. delay: usize, -} - -enum Backend { - Opus(Opus), - Pcm { - bytes_per_frame: usize, - }, - #[cfg(feature = "aac")] - Aac(Box), -} - -struct Opus { - inner: *mut OpusDecoder, - pre_skip_remaining: usize, - max_frame_size: usize, - in_dtx: bool, -} - -// SAFETY: see Encoder. -unsafe impl Send for Opus {} - -/// Boxed in [`Backend`]: the symphonia decoder carries its own filterbank state, -/// which is far larger than the other backends' handles. -#[cfg(feature = "aac")] -struct Aac { - inner: symphonia_codec_aac::AacDecoder, + delay_remaining: usize, } impl Decoder { /// Build a decoder from a catalog [`AudioConfig`](hang::catalog::AudioConfig). /// - /// Parses the OpusHead `description` if present; falls back to the catalog's - /// declared sample rate / channel count. PCM uses those catalog fields - /// directly and requires an absent `description`. + /// Opus parses the OpusHead `description` if present, falling back to the + /// catalog's declared sample rate and channel count. PCM uses those catalog + /// fields directly and requires an absent `description`. AAC reads its + /// AudioSpecificConfig, synthesizing one from the catalog when absent. pub fn new(catalog: &hang::catalog::AudioConfig, config: &Config) -> Result { - let name = match &catalog.codec { - hang::catalog::AudioCodec::Opus => "opus", - hang::catalog::AudioCodec::Pcm => "pcm", - #[cfg(feature = "aac")] - hang::catalog::AudioCodec::AAC(_) => "aac", - codec => return Err(Error::Unsupported(format!("unsupported audio codec: {codec}"))), - }; - match &config.kind { - Kind::Auto | Kind::Software => {} - Kind::Named(requested) if requested == name => {} - Kind::Named(requested) => { - return Err(Error::Unsupported(format!( - "audio decoder backend {requested:?} is unavailable for {name}" - ))); - } - } - match &catalog.codec { - hang::catalog::AudioCodec::Opus => Self::new_opus(catalog), - hang::catalog::AudioCodec::Pcm => Self::new_pcm(catalog), - #[cfg(feature = "aac")] - hang::catalog::AudioCodec::AAC(aac) => Self::new_aac(catalog, aac.profile), - codec => Err(Error::Unsupported(format!("unsupported audio codec: {codec}"))), - } - } - - fn new_opus(catalog: &hang::catalog::AudioConfig) -> Result { - let (sample_rate, channel_count, pre_skip) = if let Some(desc) = &catalog.description { - let mut buf = desc.as_ref(); - match moq_mux::codec::opus::Config::parse(&mut buf) { - Ok(head) => (head.sample_rate, head.channel_count, head.pre_skip), - Err(_) => (catalog.sample_rate, catalog.channel_count, 0), - } - } else { - (catalog.sample_rate, catalog.channel_count, 0) - }; - - opus::validate_rate(sample_rate)?; - let channels = opus::validate_channels(channel_count)?; - - let mut err = 0i32; - // SAFETY: out-pointer is valid; inner is checked for null below. - let inner = unsafe { opus_decoder_create(sample_rate as i32, channels, &mut err) }; - if err != OPUS_OK || inner.is_null() { - return Err(opus::error(err, "opus_decoder_create")); - } - - let max_frame_size = (sample_rate as usize * MAX_FRAME_MS) / 1000; - let pre_skip_remaining = (pre_skip as usize * sample_rate as usize) / 48_000; - + let backend = backend::open(catalog, config)?; + let delay = backend.delay(); Ok(Self { - backend: Backend::Opus(Opus { - inner, - pre_skip_remaining, - max_frame_size, - in_dtx: false, - }), - sample_rate, - layout: Layout::from_channels(channel_count)?, - delay: pre_skip_remaining, + backend, + delay, + delay_remaining: delay, }) } - /// AAC-LC only, which is what every gateway that feeds this crate publishes. - /// - /// HE-AAC is rejected however its config spells it: leading with SBR or PS - /// (mp4a.40.5 / .29), or leading with LC and declaring SBR in a sync extension - /// after the core. Symphonia decodes no SBR either way, so the alternative is - /// half-rate audio that sounds like a fault rather than an unsupported codec. - /// A stream that signals SBR only in band is indistinguishable from LC in the - /// config, and does decode as the core. - #[cfg(feature = "aac")] - fn new_aac(catalog: &hang::catalog::AudioConfig, profile: u8) -> Result { - use symphonia_core::codecs::audio::well_known::CODEC_ID_AAC; - use symphonia_core::codecs::audio::{AudioCodecParameters, AudioDecoderOptions}; - - let description = aac::description(catalog, profile)?; - - let mut params = AudioCodecParameters::new(); - params - .for_codec(CODEC_ID_AAC) - .with_extra_data(description.to_vec().into_boxed_slice()); - - let inner = symphonia_codec_aac::AacDecoder::try_new(¶ms, &AudioDecoderOptions::default()) - .map_err(|err| Error::Unsupported(format!("aac decoder: {err}")))?; - - // Resolved by the decoder from the config, so this is what it will emit - // even when the catalog's own fields say otherwise. - let params = inner.codec_params(); - let sample_rate = params - .sample_rate - .ok_or_else(|| Error::Unsupported("aac config declares no sample rate".into()))?; - let channel_count = params - .channels - .as_ref() - .map(|channels| channels.count()) - .ok_or_else(|| Error::Unsupported("aac config declares no channels".into()))?; - - Ok(Self { - backend: Backend::Aac(Box::new(Aac { inner })), - sample_rate, - layout: Layout::from_channels(channel_count as u32)?, - delay: 0, - }) + /// The decoder backend name in use, e.g. `"libopus"` or `"symphonia"`. + pub fn name(&self) -> &str { + self.backend.name() } - fn new_pcm(catalog: &hang::catalog::AudioConfig) -> Result { - if catalog.sample_rate == 0 { - return Err(Error::Unsupported("pcm sample rate must be greater than zero".into())); - } - if catalog.channel_count == 0 { - return Err(Error::Unsupported("pcm channel count must be greater than zero".into())); - } - if catalog.description.is_some() { - return Err(Error::Unsupported("pcm catalog description must be absent".into())); - } - let bitrate = pcm::bitrate(catalog.sample_rate, catalog.channel_count)?; - if catalog.bitrate.is_some_and(|declared| declared != bitrate) { - return Err(Error::Unsupported(format!( - "pcm catalog bitrate must be {bitrate} bits per second" - ))); - } - let bytes_per_frame = pcm::frame_bytes(1, catalog.channel_count)?; - - Ok(Self { - backend: Backend::Pcm { bytes_per_frame }, - sample_rate: catalog.sample_rate, - layout: Layout::from_channels(catalog.channel_count)?, - delay: 0, - }) - } - - /// The rate the codec decodes at, read from the catalog. + /// The rate the codec decodes at, which may differ from the catalog's. pub fn sample_rate(&self) -> u32 { - self.sample_rate + self.backend.sample_rate() } - /// The PCM layout decoded from the catalog. + /// The PCM layout the codec decodes to. pub fn layout(&self) -> Layout { - self.layout + self.backend.layout() } /// Reset codec history and reapply startup delay for a new discontinuous epoch. @@ -243,27 +89,12 @@ impl Decoder { /// Reapply catalog startup delay for a new playhead epoch without resetting codec prediction. pub(super) fn reapply_delay(&mut self) { - if let Backend::Opus(opus) = &mut self.backend { - opus.pre_skip_remaining = self.delay; - } + self.delay_remaining = self.delay; } /// Reset codec prediction after packet loss without reapplying stream startup delay. pub(super) fn reset_prediction(&mut self) -> Result<(), Error> { - match &mut self.backend { - Backend::Opus(opus) => { - // SAFETY: `inner` owns a live decoder and OPUS_RESET_STATE takes no arguments. - let rc = unsafe { opus_decoder_ctl_impl(opus.inner, OPUS_RESET_STATE, varargs![]) }; - if rc != OPUS_OK { - return Err(crate::opus::error(rc, "OPUS_RESET_STATE")); - } - opus.in_dtx = false; - } - Backend::Pcm { .. } => {} - #[cfg(feature = "aac")] - Backend::Aac(aac) => aac.inner.reset(), - } - Ok(()) + self.backend.reset() } /// How much startup delay is still to be trimmed, in native-rate frames. @@ -272,12 +103,7 @@ impl Decoder { /// it, so a caller tracking where a packet ends has to add back whatever this /// dropped across the call. pub(super) fn delay_remaining(&self) -> usize { - match &self.backend { - Backend::Opus(opus) => opus.pre_skip_remaining, - Backend::Pcm { .. } => 0, - #[cfg(feature = "aac")] - Backend::Aac(_) => 0, - } + self.delay_remaining } /// Decode one packet into interleaved `f32` PCM and report its codec activity. @@ -285,88 +111,14 @@ impl Decoder { /// Empty Opus packets invoke packet-loss concealment. Loss during DTX remains /// classified as DTX, while loss during active audio remains active. pub fn decode(&mut self, packet: &[u8]) -> Result { - match &mut self.backend { - Backend::Opus(opus) => { - let channels = self.layout.channels() as usize; - let mut out = vec![0.0f32; opus.max_frame_size * channels]; - // SAFETY: `inner` owns a live OpusDecoder; packet/out slices are - // bounded by the lengths we pass. - let samples = unsafe { - opus_decode_float( - &mut *opus.inner, - packet.as_ptr(), - packet.len() as i32, - out.as_mut_ptr(), - opus.max_frame_size as i32, - 0, - ) - }; - if samples < 0 { - return Err(crate::opus::decode_error(samples)); - } - out.truncate(samples as usize * channels); - let trim_frames = opus.pre_skip_remaining.min(samples as usize); - if trim_frames > 0 { - let trim_samples = trim_frames * channels; - out.copy_within(trim_samples.., 0); - out.truncate(out.len() - trim_samples); - opus.pre_skip_remaining -= trim_frames; - } - let activity = crate::opus::activity(packet, opus.in_dtx); - opus.in_dtx = activity.is_dtx(); - Ok(Decoded { samples: out, activity }) - } - Backend::Pcm { bytes_per_frame } => { - if packet.is_empty() || !packet.len().is_multiple_of(*bytes_per_frame) { - return Err(Error::Misaligned { - got: packet.len(), - expected: packet.len().max(1).next_multiple_of(*bytes_per_frame), - }); - } - - let out = packet - .as_chunks::<{ pcm::BYTES_PER_SAMPLE }>() - .0 - .iter() - .map(|sample| f32::from_le_bytes([sample[0], sample[1], sample[2], sample[3]])) - .collect(); - Ok(Decoded { - samples: out, - activity: Activity::Active, - }) - } - #[cfg(feature = "aac")] - Backend::Aac(aac) => { - // The packet is a raw AAC frame, not ADTS, so there is nothing to - // timestamp it with here: the container carries the timestamp and the - // decoder only reads the payload. - let packet = symphonia_core::packet::PacketRef::new( - 0, - symphonia_core::units::Timestamp::ZERO, - symphonia_core::units::Duration::ZERO, - packet, - ); - - let decoded = aac - .inner - .decode_ref(&packet) - .map_err(|err| Error::Decode(format!("aac: {err}")))?; - - let mut out = Vec::new(); - decoded.copy_to_vec_interleaved(&mut out); - Ok(Decoded { - samples: out, - activity: Activity::Active, - }) - } + let mut decoded = self.backend.decode(packet)?; + let channels = self.backend.layout().channels() as usize; + let trim = self.delay_remaining.min(decoded.samples.len() / channels); + if trim > 0 { + decoded.samples.drain(..trim * channels); + self.delay_remaining -= trim; } - } -} - -impl Drop for Opus { - fn drop(&mut self) { - // SAFETY: `inner` is a live OpusDecoder that nothing else aliases. - unsafe { opus_decoder_destroy(self.inner) }; + Ok(decoded) } } @@ -405,6 +157,7 @@ mod tests { #[test] fn aac_decodes_a_sine() { let mut decoder = Decoder::new(&aac_catalog(), &Config::default()).unwrap(); + assert_eq!(decoder.name(), "symphonia"); assert_eq!(decoder.sample_rate(), 44_100); assert_eq!(decoder.layout(), Layout::Mono); diff --git a/rs/moq-audio/src/decode/mod.rs b/rs/moq-audio/src/decode/mod.rs index 7e3cbd14a7..6f4500054c 100644 --- a/rs/moq-audio/src/decode/mod.rs +++ b/rs/moq-audio/src/decode/mod.rs @@ -10,8 +10,11 @@ //! [`Decoded`] interleaved `f32` samples. //! //! [`Options`] keeps subscription and output policy separate from the -//! lower-level decoder [`Config`]. +//! lower-level decoder [`Config`], whose [`Kind`] picks the backend: a +//! platform decoder first where one takes the track, then software (libopus, +//! PCM, and symphonia for AAC-LC). +mod backend; mod consumer; mod decoded; mod decoder; diff --git a/rs/moq-cli/src/play/media.rs b/rs/moq-cli/src/play/media.rs index 0c12887e13..c47b3a9c3c 100644 --- a/rs/moq-cli/src/play/media.rs +++ b/rs/moq-cli/src/play/media.rs @@ -224,7 +224,7 @@ impl Media { decode.output.format = moq_audio::Format::F32; match moq_audio::decode::Consumer::new(&rendition, &config, &name, decode).await { Ok(consumer) => { - tracing::info!(track = name, "playing audio rendition"); + tracing::info!(track = name, decoder = consumer.name(), "playing audio rendition"); if engine.is_none() { engine = Some(Engine::open(Default::default()).await?); } From 057bdec27fbea878b37f1d59ddcef3aae446aab5 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 16:01:32 -0700 Subject: [PATCH 06/11] fix(mux): TS export carries an AAC track's program config element (#4178) Co-authored-by: Claude Opus 5.5 Co-authored-by: Grok 4.7 --- quest/m1/audio-codecs/README.md | 1 - quest/m1/audio-codecs/ts-export-pce.md | 17 --- rs/moq-ffi/src/binary.rs | 9 +- rs/moq-mux/src/codec/aac/mod.rs | 165 ++++++++++++++------- rs/moq-mux/src/container/ts/adts.rs | 10 +- rs/moq-mux/src/container/ts/export.rs | 39 +++-- rs/moq-mux/src/container/ts/export_test.rs | 59 ++++++++ rs/moq-mux/src/container/ts/import.rs | 8 +- 8 files changed, 215 insertions(+), 93 deletions(-) delete mode 100644 quest/m1/audio-codecs/ts-export-pce.md diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index 9b3763d7f1..fdeca61705 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -43,7 +43,6 @@ its own decode and encode quest so verification stays per host. ## Quests -- [TS export PCE](/quest/m1/audio-codecs/ts-export-pce.md) - a TS export of a PCE-described AAC track writes channel_config 0 and the PCE instead of a count-derived config - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - mapping family 1 decodes on every host through the multistream decoder - [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - `encode::backend` and `Codec::Aac`, so a native publisher can produce AAC-LC diff --git a/quest/m1/audio-codecs/ts-export-pce.md b/quest/m1/audio-codecs/ts-export-pce.md deleted file mode 100644 index 31e9b76b25..0000000000 --- a/quest/m1/audio-codecs/ts-export-pce.md +++ /dev/null @@ -1,17 +0,0 @@ -# [S] TS export carries an AAC track's program config element - -## Goal - -`moq-mux`'s MPEG-TS export writes an AAC track described by a program config -element (channelConfiguration 0) as ADTS with channel_config 0 and the PCE at -the start of the first raw data block, so a TS round trip keeps the layout. -Today the export derives the ADTS channel_config from the channel count, which -mislabels such a track. - -## Plan - -Take the PCE from the track's AudioSpecificConfig description with the parser -that reads it on import, and write it as ffmpeg does: once, leading the first -raw data block. Tracks with a nonzero channelConfiguration are unchanged. Test -a round trip of the quad fixture from the PCE import (`aac_quad.ts`): import, -export, and import again, asserting the same description and channel count. diff --git a/rs/moq-ffi/src/binary.rs b/rs/moq-ffi/src/binary.rs index 8e343bd2ad..cf9e400094 100644 --- a/rs/moq-ffi/src/binary.rs +++ b/rs/moq-ffi/src/binary.rs @@ -8,6 +8,7 @@ use std::sync::Arc; +use moq_mux::binary::Config; use moq_mux::catalog::hang::Extra; use crate::error::MoqError; @@ -25,9 +26,9 @@ pub struct MoqBinaryConfig { pub mime: Option, } -impl From for moq_mux::binary::Config { +impl From for Config { fn from(config: MoqBinaryConfig) -> Self { - let mut out = moq_mux::binary::Config::default().with_compression(config.compression); + let mut out = Config::default().with_compression(config.compression); if let Some(mime) = config.mime { out = out.with_mime(mime); } @@ -48,7 +49,7 @@ impl MoqBroadcastProducer { let _guard = crate::ffi::enter(); self.with_state(|state| { let track = state.broadcast.create_track(name, None)?; - let producer = state.catalog.binary_snapshot(track, config.into())?; + let producer = state.catalog.binary_snapshot(track, Config::from(config))?; Ok(Arc::new(MoqBinarySnapshotProducer { inner: std::sync::Mutex::new(Some(producer)), })) @@ -66,7 +67,7 @@ impl MoqBroadcastProducer { let _guard = crate::ffi::enter(); self.with_state(|state| { let track = state.broadcast.create_track(name, None)?; - let producer = state.catalog.binary_stream(track, config.into())?; + let producer = state.catalog.binary_stream(track, Config::from(config))?; Ok(Arc::new(MoqBinaryStreamProducer { inner: std::sync::Mutex::new(Some(producer)), })) diff --git a/rs/moq-mux/src/codec/aac/mod.rs b/rs/moq-mux/src/codec/aac/mod.rs index 6d7f7cb67a..5ccb24a110 100644 --- a/rs/moq-mux/src/codec/aac/mod.rs +++ b/rs/moq-mux/src/codec/aac/mod.rs @@ -70,53 +70,10 @@ impl Config { /// that follows, and a reserved one is refused. Any SBR/PS extension bits after the /// core fields are consumed. pub fn parse(buf: &mut T) -> Result { - if buf.remaining() < 2 { - return Err(Error::ConfigTooShort); - } - let mut reader = BitReader::new(buf); - let object_type = read_object_type(&mut reader)?; - - // samplingFrequencyIndex: 4 bits; index 15 means an explicit 24-bit rate follows. - let freq_index = reader.read(4, Error::IncompleteConfig)? as u8; - let sample_rate = if freq_index == 15 { - reader.read(24, Error::ExplicitSampleRateTooShort)? - } else { - *SAMPLE_RATES - .get(freq_index as usize) - .ok_or(Error::UnsupportedSampleRateIndex(freq_index))? - }; - - // channelConfiguration: 4 bits, immediately after the (possibly explicit) rate. - let channel_config = reader.read(4, Error::IncompleteConfig)? as u8; + let (object_type, sample_rate, channel_config) = read_header(&mut reader)?; let channel_count = match channel_config { - 0 => { - // Explicit SBR and PS name their core object type after an extension rate; the - // GASpecificConfig carrying the program config element follows that core type. - let mut core = object_type; - if matches!(object_type, 5 | 29) { - if reader.read(4, Error::IncompleteConfig)? == 15 { - reader.read(24, Error::IncompleteConfig)?; - } - core = read_object_type(&mut reader)?; - if core == 22 { - // extensionChannelConfiguration, only for ER BSAC. - reader.read(4, Error::IncompleteConfig)?; - } - } - if !GENERAL_AUDIO.contains(&core) { - return Err(Error::ProgramConfigUnsupported(core)); - } - - // GASpecificConfig: frameLengthFlag, dependsOnCoreCoder (then a 14-bit - // coreCoderDelay), and extensionFlag precede the element. - reader.read(1, Error::IncompleteConfig)?; - if reader.read(1, Error::IncompleteConfig)? == 1 { - reader.read(14, Error::IncompleteConfig)?; - } - reader.read(1, Error::IncompleteConfig)?; - program_config(&mut reader)? - } + 0 => program_config(&mut reader)?, _ => channel_count_from_config(channel_config)?, }; @@ -178,13 +135,65 @@ impl Config { } } +/// Read an AudioSpecificConfig up to its channels: the audioObjectType, sample rate, and +/// channelConfiguration. For a channelConfiguration of 0 the reader stops at the program config +/// element. +fn read_header(reader: &mut BitReader) -> Result<(u8, u32, u8)> { + if reader.buf.remaining() < 2 { + return Err(Error::ConfigTooShort); + } + let object_type = read_object_type(reader)?; + + // samplingFrequencyIndex: 4 bits; index 15 means an explicit 24-bit rate follows. + let freq_index = reader.read(4, Error::IncompleteConfig)? as u8; + let sample_rate = if freq_index == 15 { + reader.read(24, Error::ExplicitSampleRateTooShort)? + } else { + *SAMPLE_RATES + .get(freq_index as usize) + .ok_or(Error::UnsupportedSampleRateIndex(freq_index))? + }; + + // channelConfiguration: 4 bits, immediately after the (possibly explicit) rate. + let channel_config = reader.read(4, Error::IncompleteConfig)? as u8; + if channel_config != 0 { + return Ok((object_type, sample_rate, channel_config)); + } + + // Explicit SBR and PS name their core object type after an extension rate; the + // GASpecificConfig carrying the program config element follows that core type. + let mut core = object_type; + if matches!(object_type, 5 | 29) { + if reader.read(4, Error::IncompleteConfig)? == 15 { + reader.read(24, Error::IncompleteConfig)?; + } + core = read_object_type(reader)?; + if core == 22 { + // extensionChannelConfiguration, only for ER BSAC. + reader.read(4, Error::IncompleteConfig)?; + } + } + if !GENERAL_AUDIO.contains(&core) { + return Err(Error::ProgramConfigUnsupported(core)); + } + + // GASpecificConfig: frameLengthFlag, dependsOnCoreCoder (then a 14-bit + // coreCoderDelay), and extensionFlag precede the element. + reader.read(1, Error::IncompleteConfig)?; + if reader.read(1, Error::IncompleteConfig)? == 1 { + reader.read(14, Error::IncompleteConfig)?; + } + reader.read(1, Error::IncompleteConfig)?; + Ok((object_type, sample_rate, 0)) +} + /// Build the AudioSpecificConfig for a stream that signals its fields per frame, as ADTS does. /// /// A `channel_config` of 0 means a program config element leads `block`, the stream's first raw -/// data block; it moves into the config, so the config describes the channels on its own. Any -/// other value leaves `block` unread. An element placed anywhere else is refused: reaching past -/// the channel data needs a full Huffman decode. -pub(crate) fn in_band_config(profile: u8, sample_rate: u32, channel_config: u8, block: &[u8]) -> Result { +/// data block; it moves into the config, so the config describes the channels on its own, and +/// `block` advances past it. Any other value leaves `block` untouched. An element placed anywhere +/// else is refused: reaching past the channel data needs a full Huffman decode. +pub(crate) fn in_band_config(profile: u8, sample_rate: u32, channel_config: u8, block: &mut &[u8]) -> Result { let mut out = BitWriter::default(); out.write(5, u32::from(profile & 0x1F)); match SAMPLE_RATES.iter().position(|&rate| rate == sample_rate) { @@ -200,8 +209,8 @@ pub(crate) fn in_band_config(profile: u8, sample_rate: u32, channel_config: u8, // GASpecificConfig: frameLengthFlag, dependsOnCoreCoder, and extensionFlag, all clear. out.write(3, 0); - let mut block = block; - let mut reader = BitReader::new(&mut block); + // The element ends byte aligned to the block, so the reader stops exactly past it. + let mut reader = BitReader::new(block); if reader.read(3, Error::ProgramConfigMissing)? != ID_PCE { return Err(Error::ProgramConfigMissing); } @@ -213,6 +222,27 @@ pub(crate) fn in_band_config(profile: u8, sample_rate: u32, channel_config: u8, Ok(Bytes::from(out.bytes)) } +/// Split an AudioSpecificConfig into the channel signaling of a stream that carries it per frame, +/// as ADTS does; the inverse of [`in_band_config`]. +/// +/// Returns the channelConfiguration and, when that is 0, the program config element framed to +/// lead a raw data block: its element ID, then the element aligned to the block. +pub(crate) fn in_band_channels(asc: &[u8]) -> Result<(u8, Option)> { + let mut asc = asc; + let mut reader = BitReader::new(&mut asc); + let (_, _, channel_config) = read_header(&mut reader)?; + if channel_config != 0 { + return Ok((channel_config, None)); + } + + let mut out = BitWriter::default(); + out.write(3, ID_PCE); + reader.record = Some(out); + program_config(&mut reader)?; + let out = reader.record.take().expect("recording set above"); + Ok((0, Some(Bytes::from(out.bytes)))) +} + /// The raw data block element ID of a program config element (ISO 14496-3 Table 4.85). const ID_PCE: u32 = 5; @@ -597,23 +627,48 @@ mod tests { let mut block = BitWriter::default(); block.write(3, ID_PCE); write_pce(&mut block, &[false, true], &[], &[true], 1); + let pce = block.bytes.clone(); // The channel elements that follow are never read. block.write(8, 0xFF); - let asc = in_band_config(2, 48_000, 0, &block.bytes).unwrap(); + let mut rest = block.bytes.as_slice(); + let asc = in_band_config(2, 48_000, 0, &mut rest).unwrap(); assert_eq!(asc, pce_asc(2, &[false, true], &[], &[true], 1)); assert_eq!(Config::parse(&mut asc.as_ref()).unwrap().channel_count, 6); + assert_eq!(rest, [0xFF], "the element moves out of the block"); + + // And back: the element leads the block again, byte for byte. + assert_eq!(in_band_channels(&asc).unwrap(), (0, Some(Bytes::from(pce)))); + } + + #[test] + fn in_band_channels_of_ffmpeg_program_config_element() { + // Round trip ffmpeg's own element through a raw data block and back into a config. The + // trailing SBR sync extension is not part of the element, so it drops. + let (config, pce) = in_band_channels(&FFMPEG_QUAD_ASC).unwrap(); + assert_eq!(config, 0); + let pce = pce.unwrap(); + let mut block = pce.as_ref(); + let asc = in_band_config(2, 48_000, 0, &mut block).unwrap(); + assert!(block.is_empty(), "the element is all that was framed"); + assert_eq!(asc, FFMPEG_QUAD_ASC[..FFMPEG_QUAD_ASC.len() - 3]); + } + + #[test] + fn in_band_channels_without_a_program_config_element() { + assert_eq!(in_band_channels(&[0x11, 0x90]).unwrap(), (2, None)); + assert_eq!(in_band_channels(&[0x11, 0xE0]).unwrap(), (12, None)); } #[test] fn in_band_config_refuses_a_block_without_a_leading_pce() { // ID_CPE first: any element past the channel data is out of reach. assert!(matches!( - in_band_config(2, 48_000, 0, &[0x20, 0x00]), + in_band_config(2, 48_000, 0, &mut [0x20, 0x00].as_slice()), Err(Error::ProgramConfigMissing) )); assert!(matches!( - in_band_config(2, 48_000, 0, &[]), + in_band_config(2, 48_000, 0, &mut [].as_slice()), Err(Error::ProgramConfigMissing) )); } @@ -621,7 +676,7 @@ mod tests { #[test] fn in_band_config_matches_encode() { // Without a PCE the block is unread and the config is the plain two-byte form. - let asc = in_band_config(2, 44_100, 2, &[]).unwrap(); + let asc = in_band_config(2, 44_100, 2, &mut [].as_slice()).unwrap(); let encoded = Config { profile: 2, sample_rate: 44_100, diff --git a/rs/moq-mux/src/container/ts/adts.rs b/rs/moq-mux/src/container/ts/adts.rs index 0166c05937..ac1aec6823 100644 --- a/rs/moq-mux/src/container/ts/adts.rs +++ b/rs/moq-mux/src/container/ts/adts.rs @@ -63,13 +63,17 @@ impl Header { pub(super) fn write_header( object_type: u8, sample_rate: u32, - channel_count: u32, + channel_config: u8, raw_len: usize, ) -> anyhow::Result<[u8; 7]> { // ADTS `profile` is the 2-bit audioObjectType - 1. let profile = object_type.saturating_sub(1) & 0x03; let freq_index = freq_index_from_rate(sample_rate)?; - let channel_config = channel_config_from_count(channel_count); + // ADTS has 3 bits for it; the higher configurations only fit an AudioSpecificConfig. + anyhow::ensure!( + channel_config < 8, + "channelConfiguration {channel_config} not representable in ADTS" + ); let frame_len = raw_len + 7; anyhow::ensure!(frame_len < (1 << 13), "AAC frame too large for ADTS framing"); @@ -98,7 +102,7 @@ fn freq_index_from_rate(sample_rate: u32) -> anyhow::Result { } /// Map a channel count to an AAC `channel_config` (ISO 14496-3 Table 1.19). -fn channel_config_from_count(channel_count: u32) -> u8 { +pub(super) fn channel_config_from_count(channel_count: u32) -> u8 { match channel_count { 1..=6 => channel_count as u8, 8 => 7, diff --git a/rs/moq-mux/src/container/ts/export.rs b/rs/moq-mux/src/container/ts/export.rs index e1380502ae..c53a7c141f 100644 --- a/rs/moq-mux/src/container/ts/export.rs +++ b/rs/moq-mux/src/container/ts/export.rs @@ -254,10 +254,14 @@ impl Track { enum Kind { /// Video carries its TS stream type (H.264 = 0x1B, H.265 = 0x24). Video(StreamType), + /// AAC, framed as ADTS. A `channel_config` of 0 defers the layout to a program config + /// element, which leads the next raw data block written and is then taken. A catalog update + /// rebuilds the kind and so repeats it once, which a decoder tuning in mid-stream welcomes. Aac { object_type: u8, sample_rate: u32, - channel_count: u32, + channel_config: u8, + program_config: Option, }, /// Opus (private stream_type 0x06). Each frame is one Opus packet, prefixed with /// the Opus-in-TS access-unit control header and announced with the 'Opus' @@ -1341,9 +1345,12 @@ impl Export { if self.span_counters.is_none() { self.span_counters = Some(self.counters.clone()); } - let track = self.tracks.get(name).context("missing track")?; + let track = self.tracks.get_mut(name).context("missing track")?; let pid = track.pid; let kind = track.kind.clone(); + if let Kind::Aac { program_config, .. } = &mut track.kind { + program_config.take(); + } let is_video = matches!(kind, Kind::Video(_)); let timestamp = frame.timestamp; let keyframe = frame.keyframe; @@ -1356,11 +1363,15 @@ impl Export { Kind::Aac { object_type, sample_rate, - channel_count, + channel_config, + program_config, } => { - let header = adts::write_header(*object_type, *sample_rate, *channel_count, frame.payload.len())?; - let mut framed = Vec::with_capacity(7 + frame.payload.len()); + let pce = program_config.as_deref().unwrap_or_default(); + let raw_len = pce.len() + frame.payload.len(); + let header = adts::write_header(*object_type, *sample_rate, *channel_config, raw_len)?; + let mut framed = Vec::with_capacity(header.len() + raw_len); framed.extend_from_slice(&header); + framed.extend_from_slice(pce); framed.extend_from_slice(&frame.payload); Some(framed) } @@ -2086,11 +2097,19 @@ fn video_es_payload(stream_type: StreamType, description: Option<&Bytes>, frame: fn audio_kind(config: &AudioConfig, name: &str) -> anyhow::Result { ensure_raw(&config.container, "audio", name)?; match &config.codec { - AudioCodec::AAC(aac) => Ok(Kind::Aac { - object_type: aac.profile, - sample_rate: config.sample_rate, - channel_count: config.channel_count, - }), + AudioCodec::AAC(aac) => { + // The description names the layout exactly; without one, the count is all there is. + let (channel_config, program_config) = match &config.description { + Some(asc) => crate::codec::aac::in_band_channels(asc)?, + None => (adts::channel_config_from_count(config.channel_count), None), + }; + Ok(Kind::Aac { + object_type: aac.profile, + sample_rate: config.sample_rate, + channel_config, + program_config, + }) + } AudioCodec::Mp2 => Ok(Kind::Mp2 { sample_rate: config.sample_rate, }), diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index 19dd6f9fae..00d9294308 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -1425,6 +1425,65 @@ async fn ac3_roundtrip_byte_exact() { assert_eq!(roundtripped, ingested, "AC-3 frames must survive byte-for-byte"); } +/// The first ADTS frame of the first AAC PES: its header and raw data block. +fn first_adts_frame(ts: &[u8]) -> (super::adts::Header, Vec) { + let mut pes = PesPacketReader::new(TsPacketReader::new(Cursor::new(ts))); + let packet = pes.read_pes_packet().unwrap().expect("an AAC PES"); + let header = super::adts::Header::parse(&packet.data).unwrap(); + (header, packet.data[header.header_len..header.frame_len].to_vec()) +} + +/// ffmpeg's quad AAC fixture has no channelConfiguration, so its layout rides in a program +/// config element. Import moves it into the description and export puts it back: channel_config +/// 0 in ADTS and the element leading the first raw data block, exactly as ffmpeg wrote it. +#[tokio::test(start_paused = true)] +async fn aac_program_config_roundtrip() { + let data = include_bytes!("test_data/aac_quad.ts"); + + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); + let mut import = crate::container::ts::Import::new(broadcast, catalog.reserve()); + import.decode(&BytesMut::from(&data[..])).unwrap(); + import.finish().unwrap(); + + let snapshot = catalog.snapshot(); + let (name, audio) = snapshot.audio.renditions.iter().next().expect("an AAC track"); + assert_eq!(audio.channel_count, 4); + let ingested = read_frames(&consumer, name, Kind::Audio).await; + assert!(!ingested.is_empty(), "no AAC frames"); + + let ts = drain(consumer).await; + assert_packet_aligned(&ts); + + let (header, block) = first_adts_frame(&ts); + assert_eq!(header.channel_config, 0, "the layout is not a channelConfiguration"); + assert_eq!( + block, + first_adts_frame(data).1, + "the first raw data block, element and all" + ); + + let mut broadcast2 = moq_net::broadcast::Info::new().produce(); + let consumer2 = broadcast2.consume(); + let catalog2 = crate::catalog::Producer::new(&mut broadcast2, crate::catalog::Config::default()).unwrap(); + let mut import2 = crate::container::ts::Import::new(broadcast2, catalog2.reserve()); + import2.decode(&BytesMut::from(ts.as_ref())).unwrap(); + import2.finish().unwrap(); + + let snapshot2 = catalog2.snapshot(); + let (name2, audio2) = snapshot2 + .audio + .renditions + .iter() + .next() + .expect("round-trip lost the AAC track"); + assert_eq!(audio2.channel_count, 4); + assert_eq!(audio2.description, audio.description); + let roundtripped = read_frames(&consumer2, name2, Kind::Audio).await; + assert_eq!(roundtripped, ingested, "the element is written once, not per frame"); +} + /// The ffmpeg E-AC-3 fixture must survive TS -> MoQ -> TS byte-for-byte in an /// audio-only program; the PMT re-announces ATSC 0x87 with the 'EAC3' /// registration descriptor. diff --git a/rs/moq-mux/src/container/ts/import.rs b/rs/moq-mux/src/container/ts/import.rs index 643069c77d..057d998aa1 100644 --- a/rs/moq-mux/src/container/ts/import.rs +++ b/rs/moq-mux/src/container/ts/import.rs @@ -2132,17 +2132,19 @@ impl AacStream { } }; + let mut block = &data[offset + header.header_len..end]; let import = match &mut self.import { Some(import) => import, None => { // Synthesize the AudioSpecificConfig `description` so out-of-band consumers // (fMP4/MKV export, WebCodecs) can configure the decoder. A channel_config of 0 - // moves the program config element out of this first frame into it. + // moves the program config element out of this first frame into it, as + // ffmpeg's aac_adtstoasc does; the TS export puts it back. let asc = aac::in_band_config( header.object_type, header.sample_rate, header.channel_config, - &data[offset + header.header_len..end], + &mut block, )?; let mut config = aac::config(&asc)?; config.container = self.container.clone(); @@ -2157,7 +2159,7 @@ impl AacStream { } }; - import.decode(&data[offset + header.header_len..end], pts)?; + import.decode(block, pts)?; // Count only completed frames; input gaps and unfinished tails are not a media burst. burst += std::time::Duration::from_nanos((1024_u64 * 1_000_000_000).div_ceil(header.sample_rate as u64)); // The importer accumulates; cut each ADTS frame into its own group (one QUIC stream) From c6ec067760efd46a646ac1b20825bf4a98c008c0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 16:25:21 -0700 Subject: [PATCH 07/11] feat(audio): an encode backend seam and an AAC output codec (#4183) Co-authored-by: Claude Opus 5.5 Co-authored-by: Grok 4.7 --- doc/lib/c/index.md | 2 +- doc/lib/go/index.md | 6 + doc/lib/kt/index.md | 7 +- doc/lib/py/index.md | 7 +- doc/lib/rs/moq-audio.md | 21 +- doc/lib/swift/index.md | 7 +- go/wrapper/publish.go | 2 +- go/wrapper/types.go | 10 +- .../kotlin/dev/moq/Aliases.kt | 2 +- py/moq-rs/README.md | 2 +- py/moq-rs/moq/publish.py | 4 +- quest/m1/audio-codecs/README.md | 1 - quest/m1/audio-codecs/encode-audiotoolbox.md | 6 +- quest/m1/audio-codecs/encode-backend.md | 40 -- quest/m1/obs-moq-video/audio-publish.md | 6 +- quest/m2/audio-encode-mediacodec.md | 4 +- quest/m2/audio-encode-mediafoundation.md | 1 - rs/libmoq/src/audio.rs | 7 +- rs/moq-audio/Cargo.toml | 2 +- rs/moq-audio/src/aac.rs | 4 +- rs/moq-audio/src/encode/backend/libopus.rs | 207 ++++++ rs/moq-audio/src/encode/backend/mod.rs | 323 +++++++++ rs/moq-audio/src/encode/backend/pcm.rs | 46 ++ rs/moq-audio/src/encode/backend/stub.rs | 59 ++ rs/moq-audio/src/encode/encoder.rs | 671 ++++++++++-------- rs/moq-audio/src/encode/mod.rs | 7 +- rs/moq-audio/src/encode/producer.rs | 76 +- rs/moq-audio/src/lib.rs | 7 +- rs/moq-ffi/src/audio.rs | 25 +- rs/moq-ffi/src/test.rs | 54 ++ swift/Sources/Moq/Aliases.swift | 2 +- swift/Sources/Moq/Broadcast.swift | 2 +- 32 files changed, 1221 insertions(+), 399 deletions(-) delete mode 100644 quest/m1/audio-codecs/encode-backend.md create mode 100644 rs/moq-audio/src/encode/backend/libopus.rs create mode 100644 rs/moq-audio/src/encode/backend/mod.rs create mode 100644 rs/moq-audio/src/encode/backend/pcm.rs create mode 100644 rs/moq-audio/src/encode/backend/stub.rs diff --git a/doc/lib/c/index.md b/doc/lib/c/index.md index b8425b713e..5a4fcc0e9c 100644 --- a/doc/lib/c/index.md +++ b/doc/lib/c/index.md @@ -35,7 +35,7 @@ and `target/include/moq.h`. - **Threading.** Any function from any thread. Raw publish calls block until the codec takes the frame, which paces a publisher. - **Connection health.** `moq_session_stats()` reports available metrics with per-field validity flags. `moq_session_snapshot()` samples those metrics and the negotiated draft name together from the same connection. Its protocol string is backed by static storage. Both return an offline error between reconnects and leave the destination untouched. `moq_session_bandwidth()` mints an allocator over the send estimate; `moq_bandwidth_reserve` claims a share for an app-owned track, and `moq_encode_video` / `moq_encode_audio` take the same handle so the built-in video encoder follows the grant. - **Raw playback.** Raw audio and video consumers start at the newest cached group when opened, so rebuilding a live decoder skips the retained backlog. -- **Raw audio encode.** `moq_audio_encoder_output.codec` names the codec (only `"opus"` today), and `frame_duration_us` sets the Opus frame length: 2500, 5000, 10000, 20000, 40000, or 60000, with 0 meaning the 20 ms default. +- **Raw audio encode.** `moq_audio_encoder_output.codec` names the codec: `"opus"`, `"pcm"`, or `"aac"`. `frame_duration_us` sets the Opus frame length: 2500, 5000, 10000, 20000, 40000, or 60000, with 0 meaning the codec's default (20 ms for Opus, 1024 samples for AAC). AAC-LC encodes through the platform's encoder, so a host without one refuses it. - **Audio channel layouts.** A `channels` count also names the speaker layout, by the WAVE convention: 1 is mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved front left, front right, center, LFE, back, then side. `moq_decode_audio` remixes to the count you ask for; past 8 channels the samples pass through but can't be remixed. - **Raw decode output.** `moq_video_decoder_output` selects the decoded CPU pixel format (`MOQ_VIDEO_PIXEL_FORMAT_I420` or `_RGBA`) and target size (`width`/`height`, both zero for native; otherwise even and non-zero). Unknown formats and invalid sizes fail `moq_decode_video` before subscribing; accepted requests deliver exactly that layout or fail on the terminal callback. - **Encoded video metadata.** `moq_video_init.hint` is a zero-initialized `moq_video_hint` with `has_*` flags for coded dimensions, bitrate (bits per second), frame rate, and latency preference. Hints seed a video codec track's catalog; detected dimensions take precedence. diff --git a/doc/lib/go/index.md b/doc/lib/go/index.md index 0540cafb90..8a49af5747 100644 --- a/doc/lib/go/index.md +++ b/doc/lib/go/index.md @@ -115,6 +115,12 @@ one: `FetchGroup`/`FetchMediaGroup`, `Dynamic()` with `Requests(ctx)`, `AppendDatagram`/`Datagrams(ctx)`, `SetCatalogSection`, `Demand()` for `Used`/`Unused`, `Session().Stats()`. `moq.IsAuthError` and `moq.IsShutdown` classify errors. `moq.ProtocolError(err)` is the structured protocol failure (scope, verbatim code, kind) when the peer sent one. +`EncodeAudio` encodes raw PCM inside the binding. Its codec is `OpusAudioCodec()` +or `AacAudioCodec()`, and `AudioEncoderOutput.FrameDurationUs` sets the Opus +frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. 0 takes +the codec's own frame, which AAC needs. AAC-LC encodes through the platform's +encoder, so a host without one refuses it. + Audio `Channels` also names the speaker layout, by the WAVE convention: 1 is mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved front left, front right, center, LFE, back, then side. Decoding remixes to the diff --git a/doc/lib/kt/index.md b/doc/lib/kt/index.md index 8f863b942b..7f552b3037 100644 --- a/doc/lib/kt/index.md +++ b/doc/lib/kt/index.md @@ -91,8 +91,11 @@ connection's send estimate; pass it to `encodeVideo` / `encodeAudio` or native side. `encodeAudio` encodes raw PCM inside the binding. Its codec is an object, -`AudioCodec.opus()`, and `AudioEncoderOutput.frameDurationUs` sets the Opus -frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. +`AudioCodec.opus()` or `AudioCodec.aac()`, and +`AudioEncoderOutput.frameDurationUs` sets the Opus frame length: 2500, 5000, +10000, 20000 (the default), 40000, or 60000. 0 takes the codec's own frame, +which AAC needs. AAC-LC encodes through the platform's encoder, so a host +without one refuses it. Audio `channels` also names the speaker layout, by the WAVE convention: 1 is mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved diff --git a/doc/lib/py/index.md b/doc/lib/py/index.md index 5fe4e12de3..1ce6293a5e 100644 --- a/doc/lib/py/index.md +++ b/doc/lib/py/index.md @@ -107,8 +107,11 @@ Each server request reports a `moq.Transport` enum, including QUIC, Iroh, WebSocket, TCP, and Unix sockets. `encode_audio` encodes raw PCM inside the binding. Its codec is an object, -`moq.AudioCodec.opus()`, and `AudioEncoderOutput.frame_duration_us` sets the -Opus frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. +`moq.AudioCodec.opus()` or `moq.AudioCodec.aac()`, and +`AudioEncoderOutput.frame_duration_us` sets the Opus frame length: 2500, 5000, +10000, 20000 (the default), 40000, or 60000. 0 takes the codec's own frame, +which AAC needs. AAC-LC encodes through the platform's encoder, so a host +without one refuses it. Audio `channels` also names the speaker layout, by the WAVE convention: 1 is mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved diff --git a/doc/lib/rs/moq-audio.md b/doc/lib/rs/moq-audio.md index 637343dd61..c76dc1179e 100644 --- a/doc/lib/rs/moq-audio.md +++ b/doc/lib/rs/moq-audio.md @@ -28,7 +28,7 @@ policy. Decoding likewise separates low-level `decode::Config`, PCM | Module | Does | | --- | --- | | `capture` | Microphones via CoreAudio, WASAPI, ALSA (and PipeWire/PulseAudio hosts), plus macOS system audio | -| `encode` | PCM to Opus (with DTX and voice-activity signaling) or raw PCM for the lowest latency | +| `encode` | PCM to Opus (with DTX and voice-activity signaling), raw PCM for the lowest latency, or AAC-LC through a platform encoder | | `decode` | Opus, PCM, and AAC-LC back to PCM, resampled to the rate you want | | `playback` | One output device mixing every track in a call, with click-free volume ramps | | `aec` | Acoustic echo cancellation (a port of WebRTC's), so a laptop with no headset doesn't feed itself back | @@ -48,6 +48,25 @@ its config are refused at construction on every host. HE-AAC signaled only in band plays as its half-rate LC core. Linux has no OS audio decoder, so it will stay that way there. +`encode` selects the same way, through `encode::Settings::kind`, and +`Encoder::name()` reports what opened. + +| Backend | Encodes | Hosts | +| --- | --- | --- | +| `libopus` | Opus, mono or stereo | all | +| `pcm` | PCM | all | + +`encode::Codec::Aac` is AAC-LC (`mp4a.40.2`) at the input's rate and layout: +mono, stereo, 3.0, 4.0, 5.0, 5.1, or 7.1, the layouts with an AAC +channelConfiguration. Frames are 1024 samples, so `Settings::from_input` sets +`frame_duration` to match. The catalog's AudioSpecificConfig is built from the +settings when the track is registered, and since it has no field for the +encoder's delay, packets are stamped that much earlier so the first input +sample still lands at the first timestamp. There is no software AAC encoder, +and no platform encoder is wired in yet, so `Codec::Aac` is refused at +construction on every host for now. Linux has no OS encoder, so it will stay +that way there. + Highlights: - **`encode::Publication`** advertises the track and opens the microphone only while someone listens. Stop, swap devices, and restart without changing the track subscribers know; read a level meter for the UI. diff --git a/doc/lib/swift/index.md b/doc/lib/swift/index.md index dc02120fe7..39ab5e152a 100644 --- a/doc/lib/swift/index.md +++ b/doc/lib/swift/index.md @@ -95,8 +95,11 @@ divides the connection's send estimate; pass it to `encodeVideo` / (scope, verbatim code, kind) when the peer sent one. `encodeAudio` encodes raw PCM inside the binding. Its codec is an object, -`AudioCodec.opus()`, and `AudioEncoderOutput.frameDurationUs` sets the Opus -frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. +`AudioCodec.opus()` or `AudioCodec.aac()`, and +`AudioEncoderOutput.frameDurationUs` sets the Opus frame length: 2500, 5000, +10000, 20000 (the default), 40000, or 60000. 0 takes the codec's own frame, +which AAC needs. AAC-LC encodes through the platform's encoder, so a host +without one refuses it. Audio `channels` also names the speaker layout, by the WAVE convention: 1 is mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved diff --git a/go/wrapper/publish.go b/go/wrapper/publish.go index 6eccc6d8f6..6f46b1ff2d 100644 --- a/go/wrapper/publish.go +++ b/go/wrapper/publish.go @@ -193,7 +193,7 @@ func (b *BroadcastProducer) PublishContainerStream(format ContainerFormat) (*Con // EncodeAudio publishes a raw-audio track with an in-process encoder. // -// Select the codec with OpusAudioCodec (currently the only constructor). +// Select the codec with OpusAudioCodec or AacAudioCodec. // Pass bandwidth to reserve this track's bitrate against the session's // allocator so a co-resident video encoder sizes itself against what is left. func (b *BroadcastProducer) EncodeAudio(name string, input AudioEncoderInput, output AudioEncoderOutput, bandwidth *Bandwidth) (*AudioProducer, error) { diff --git a/go/wrapper/types.go b/go/wrapper/types.go index 40bf4f53ea..3c588ba2df 100644 --- a/go/wrapper/types.go +++ b/go/wrapper/types.go @@ -8,14 +8,14 @@ import ffi "moq.dev/moq-ffi/moq" type ( // Audio describes one audio rendition in a broadcast catalog: codec, sample rate, channel count, and container. Audio = ffi.MoqAudio - // AudioCodec selects the audio encoder codec. Build one with OpusAudioCodec; + // AudioCodec selects the audio encoder codec. Build one with OpusAudioCodec or AacAudioCodec; // adding a codec later adds a constructor, not a breaking enum change. AudioCodec = ffi.MoqAudioCodec // AudioDecoderOutput configures the PCM format, sample rate, and channels DecodeAudio delivers. AudioDecoderOutput = ffi.MoqAudioDecoderOutput // AudioEncoderInput declares the PCM sample format, sample rate, and channel count of frames written to an audio producer. AudioEncoderInput = ffi.MoqAudioEncoderInput - // AudioEncoderOutput configures the Opus encoder: codec, optional sample rate, channels, bitrate, and frame duration. + // AudioEncoderOutput configures the encoder: codec, optional sample rate, channels, bitrate, and frame duration. AudioEncoderOutput = ffi.MoqAudioEncoderOutput // AudioSampleFormat is a raw PCM sample layout, mirroring WebCodecs AudioData.format. AudioSampleFormat = ffi.MoqAudioSampleFormat @@ -184,6 +184,12 @@ func OpusAudioCodec() *AudioCodec { return ffi.MoqAudioCodecOpus() } +// AacAudioCodec selects AAC-LC through the platform's encoder for EncodeAudio. +// A host without one refuses it. Leave FrameDurationUs at 0 for AAC's own frame. +func AacAudioCodec() *AudioCodec { + return ffi.MoqAudioCodecAac() +} + // VideoPixelFormat values: the raw pixel layout fed to the in-process encoder, // and the one the in-process decoder delivers. const ( diff --git a/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt b/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt index 38883f6cda..f59782cf66 100644 --- a/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt +++ b/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt @@ -134,7 +134,7 @@ typealias FetchGroupOptions = uniffi.moq.MoqFetchGroupOptions typealias TrackInfo = uniffi.moq.MoqTrackInfo /** One audio frame: PCM payload bytes plus a presentation timestamp. */ typealias AudioFrame = uniffi.moq.MoqAudioFrame -/** Selects the audio encoder codec. Build one with `AudioCodec.opus()`. */ +/** Selects the audio encoder codec. Build one with `AudioCodec.opus()` or `AudioCodec.aac()`. */ typealias AudioCodec = uniffi.moq.MoqAudioCodec /** A raw PCM sample format, mirroring WebCodecs `AudioData.format`. */ typealias AudioSampleFormat = uniffi.moq.MoqAudioSampleFormat diff --git a/py/moq-rs/README.md b/py/moq-rs/README.md index 6add3e7adb..0b60f1cb92 100644 --- a/py/moq-rs/README.md +++ b/py/moq-rs/README.md @@ -154,7 +154,7 @@ client = moq.Client( - `.publish_audio(format, init, *, label=None, track=None) → MediaProducer`. `init` is required: an OpusHead or AudioSpecificConfig resolves the whole rendition. `track` names the track; otherwise a unique name is derived from the format. - `.publish_video(format, init=b"", *, label=None, hint=None, track=None) → MediaProducer`. `init` may be empty for a format that resolves in band; a `VideoHint` pins catalog fields the stream can't reveal (bitrate) or publishes the catalog before the first keyframe. `track` names the track as in `publish_audio`. - `.encode_video(input, output, *, bandwidth=None) → VideoProducer`. Encode raw `VideoFrame`s inside the binding; `.write(frame)` each one. - - `.encode_audio(name, input, output, *, bandwidth=None) → AudioProducer`. Encode raw PCM `AudioFrame`s; the codec is `output.codec`, e.g. `AudioCodec.opus()`, with `output.frame_duration_us` setting the Opus frame length. + - `.encode_audio(name, input, output, *, bandwidth=None) → AudioProducer`. Encode raw PCM `AudioFrame`s; the codec is `output.codec`, e.g. `AudioCodec.opus()` or `AudioCodec.aac()`, with `output.frame_duration_us` setting the Opus frame length (0 takes the codec's own frame, which AAC needs). - `.finish()` - **`BroadcastDynamic`**. Async source of tracks requested by subscribers. - `await .requested_track() → TrackRequest`. Call `.accept()` on it for a `TrackProducer`, or `.abort(code)` to reject. diff --git a/py/moq-rs/moq/publish.py b/py/moq-rs/moq/publish.py index cbaf29ba13..e7e8bb03af 100644 --- a/py/moq-rs/moq/publish.py +++ b/py/moq-rs/moq/publish.py @@ -749,8 +749,8 @@ def encode_audio( ) -> AudioProducer: """Publish a raw-audio track with an in-process encoder. - Select the codec with ``moq.AudioCodec.opus()`` (currently the only - constructor), placed in ``output``. + Select the codec with ``moq.AudioCodec.opus()`` or + ``moq.AudioCodec.aac()``, placed in ``output``. Pass ``bandwidth`` to reserve this track's bitrate against the session's allocator so a co-resident video encoder sizes itself against what is left. diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index fdeca61705..ee3cca548f 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -45,7 +45,6 @@ its own decode and encode quest so verification stays per host. - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - mapping family 1 decodes on every host through the multistream decoder -- [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - `encode::backend` and `Codec::Aac`, so a native publisher can produce AAC-LC - [AudioToolbox encode](/quest/m1/audio-codecs/encode-audiotoolbox.md) - macOS and iOS encode AAC-LC ## Related diff --git a/quest/m1/audio-codecs/encode-audiotoolbox.md b/quest/m1/audio-codecs/encode-audiotoolbox.md index 2ceda88280..71267e9b63 100644 --- a/quest/m1/audio-codecs/encode-audiotoolbox.md +++ b/quest/m1/audio-codecs/encode-audiotoolbox.md @@ -15,11 +15,15 @@ the encode seam as the platform candidate on macOS and iOS. `kAudioConverterPrimeInfo` gives the delay the timestamps fold in. - Bitrate through `kAudioConverterEncodeBitRate`, updated live where the converter allows. +- The seam assumes one packet per frame. If the converter holds output back, + the backend needs a `flush` and a zero-or-more return, which changes + `Encoder::encode` and so targets `dev`. +- Gate the seam's "AAC refused without a platform encoder" test to hosts + without one. - Regression: a stereo and a 5.1 encode round-trip through the AudioToolbox decoder and through symphonia (stereo only), with timestamps continuous across the priming. ## Required -- [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - the candidate order this backend joins - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - the round-trip regression decodes through it diff --git a/quest/m1/audio-codecs/encode-backend.md b/quest/m1/audio-codecs/encode-backend.md deleted file mode 100644 index 4f73ed98f2..0000000000 --- a/quest/m1/audio-codecs/encode-backend.md +++ /dev/null @@ -1,40 +0,0 @@ -# [M] An encode backend seam and an AAC output codec - -## Goal - -`moq_audio::encode` selects a backend the way `moq_video::encode` does, and -`Codec::Aac` is a valid output: AAC-LC at the input's sample rate and layout, -published with the AudioSpecificConfig description every player expects. A -host with no AAC encoder refuses it at construction. - -## Plan - -Mirror the decode seam (`rs/moq-audio/src/decode/backend`): `encode::backend` with a crate-private `Backend` -trait (`encode`, `flush`, `set_bitrate`, `name`), an `open(codec, config)` -that walks platform candidates before software ones, using the public settings -and selection contract settled in main. Opus and PCM retain their behavior. This -quest adds AAC through platform encoders; no software AAC dependency is selected. - -- `encode::Codec` gains `Aac`, meaning `mp4a.40.2`, and `as_str` / `FromStr` - accept `"aac"`, which is what libmoq's codec string carries. moq-ffi's - immutable codec object gains an `aac()` constructor and conversion; do not - reintroduce a closed binding enum. The generated bindings, hand-written - wrappers, and docs follow the Cross-Package Sync table. -- Catalog emission: `AudioCodec::AAC` with profile 2, `Container::Legacy`, - and the encoder's reported delay folded into timestamps like Opus pre-skip - is today. `Producer` registers the rendition before the first frame is - written, so the ASC `description` is synthesized at construction from the - config with `moq_mux::codec::aac::Config::encode`, never read back from the - backend's first packet. A backend that reports its own header (a magic - cookie, `csd-0`, `MF_MT_USER_DATA`) must produce one equal to the synthesized - ASC, asserted in its tests. -- Frame size is the codec's (1024 samples for AAC), so `frame_duration` is - validated per codec rather than against the Opus table. -- Bitrate updates go through the backend; one that cannot change rate - mid-stream keeps its opening rate, as the video seam documents. -- Regression: the selection order with a stub backend; `Codec::Aac` refused on - a host with no backend; the Opus and PCM paths unchanged. - -## Related - -- [OBS audio publishing](/quest/m1/obs-moq-video/audio-publish.md) - the OBS encoder adapter can offer AAC once this lands diff --git a/quest/m1/obs-moq-video/audio-publish.md b/quest/m1/obs-moq-video/audio-publish.md index ae2543cd35..3b7cf9981a 100644 --- a/quest/m1/obs-moq-video/audio-publish.md +++ b/quest/m1/obs-moq-video/audio-publish.md @@ -8,7 +8,7 @@ MoQ publishing can encode OBS's mixed audio with moq-audio Opus while preserving - Expose `moq_audio::encode::Encoder` through a codec-only moq-ffi type with owned encoder/packet handles, so every binding gets it. The existing raw audio producer combines encoding and publication and must not create a second publication alongside the OBS encoded output. Reuse frame sizing, bitrate updates, catalog configuration, and finish/padding behavior. - Implement the OBS audio encoder interface, including fixed input frame size, mono/stereo PCM conversion, timestamps, codec headers, final padding, and packet release. Ask OBS for the input layout the encoder supports; moq-audio does not implement arbitrary channel remapping. Keep capture/mixing/device ownership in OBS. -- Publish Opus initially. Leave AAC with the existing OBS mode and defer PCM publishing UI, since the output currently declares AAC/Opus. Correct stale binding documentation that describes the raw codec parser as Opus-only if that API is touched. +- Publish Opus initially. `encode::Codec::Aac` lets the adapter offer AAC where a platform encoder exists; until then leave AAC with the existing OBS mode and defer PCM publishing UI, since the output currently declares AAC/Opus. Correct stale binding documentation that describes the raw codec parser as Opus-only if that API is touched. - Apply the shared presets and independent audio bitrate. Test 10/20 ms packetization, frame-size changes at stream boundaries, partial final frames, silence, reconnect, saturation, and stop with pending output. Verify decoded audio and A/V timestamps with a real subscriber. - Land the internal adapter independently; the combined Use MoQ encoders UI becomes available when the video adapter also lands. Do not expose a temporary video/audio mix-and-match product UI. @@ -16,7 +16,3 @@ MoQ publishing can encode OBS's mixed audio with moq-audio Opus while preserving - [OBS migration](/quest/m1/cpp/obs.md) - the plugin is on the generated C++ first - [Encoder presets](/quest/m1/obs-moq-video/presets.md) - shared policy and truthful reporting - -## Related - -- [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - `Codec::Aac` lets the adapter offer AAC instead of leaving it to the OBS encoder mode diff --git a/quest/m2/audio-encode-mediacodec.md b/quest/m2/audio-encode-mediacodec.md index d488ec184b..8af6ad3e51 100644 --- a/quest/m2/audio-encode-mediacodec.md +++ b/quest/m2/audio-encode-mediacodec.md @@ -13,6 +13,9 @@ behind the `mediacodec` feature and the encode seam. - `audio/mp4a-latm` with `KEY_AAC_PROFILE` = LC. The catalog ASC is synthesized at construction per the encode seam, since `csd-0` only arrives with the first output buffer; assert the two match. +- MediaCodec pipelines output, which the one-packet-per-frame seam does not + allow yet: add a `flush` and a zero-or-more return, a change to + `Encoder::encode` that targets `dev`, unless the AudioToolbox quest already did. - Multichannel is device-dependent; probe the encoder's capabilities at open and refuse a layout it does not list. - Round-trip regression through the MediaCodec decoder; runtime proof on a @@ -20,5 +23,4 @@ behind the `mediacodec` feature and the encode seam. ## Required -- [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - the candidate order this backend joins - [MediaCodec decode](/quest/m2/audio-decode-mediacodec.md) - the round-trip regression decodes through it diff --git a/quest/m2/audio-encode-mediafoundation.md b/quest/m2/audio-encode-mediafoundation.md index 8347dd1854..b2bad52972 100644 --- a/quest/m2/audio-encode-mediafoundation.md +++ b/quest/m2/audio-encode-mediafoundation.md @@ -20,7 +20,6 @@ behind the encode seam on Windows. ## Required -- [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - the candidate order this backend joins - [Media Foundation decode](/quest/m2/audio-decode-mediafoundation.md) - the round-trip regression decodes through it ## Related diff --git a/rs/libmoq/src/audio.rs b/rs/libmoq/src/audio.rs index d389abc0fe..21d3b1c225 100644 --- a/rs/libmoq/src/audio.rs +++ b/rs/libmoq/src/audio.rs @@ -79,7 +79,8 @@ pub struct moq_audio_encoder_input { #[repr(C)] #[allow(non_camel_case_types)] pub struct moq_audio_encoder_output { - /// Codec id, UTF-8 (currently only "opus"). + /// Codec id, UTF-8: "opus", "pcm", or "aac". AAC encodes through the + /// platform's encoder, so a host without one refuses it. pub codec: *const c_char, pub codec_len: usize, /// 0 = derive from input. @@ -89,8 +90,8 @@ pub struct moq_audio_encoder_output { /// 0 = libopus default. pub bitrate: u32, /// Encoded frame duration in microseconds. Opus accepts exactly - /// 2500/5000/10000/20000/40000/60000 us. 0 = the 20 ms default, which - /// matches the JS publish path. + /// 2500/5000/10000/20000/40000/60000 us. 0 = the codec's default: 20 ms for + /// Opus, which matches the JS publish path, and 1024 samples for AAC. pub frame_duration_us: u32, } diff --git a/rs/moq-audio/Cargo.toml b/rs/moq-audio/Cargo.toml index a270e8dbde..5980b1bc2c 100644 --- a/rs/moq-audio/Cargo.toml +++ b/rs/moq-audio/Cargo.toml @@ -26,7 +26,7 @@ default = ["aac"] # AAC-LC decode via symphonia. Pure Rust, so it costs no toolchain: the trade # this crate already refused for Opus. Only a subscriber to an ingest-sourced # broadcast needs it (RTMP, SRT/TS, fmp4, and gstreamer all publish AAC, while -# everything this crate encodes is Opus or PCM), so a publish-only build can +# this crate encodes in software is Opus or PCM), so a publish-only build can # drop it and the two crates it pulls. aac = ["dep:symphonia-codec-aac", "dep:symphonia-core"] # Device capture. Microphones go through cpal (pure-Rust: CoreAudio / WASAPI / diff --git a/rs/moq-audio/src/aac.rs b/rs/moq-audio/src/aac.rs index 7c8b948233..52fb66e3dd 100644 --- a/rs/moq-audio/src/aac.rs +++ b/rs/moq-audio/src/aac.rs @@ -1,7 +1,7 @@ //! AAC constraints, the sibling of the `opus` and `pcm` modules. //! -//! Only the decode side exists: there is no Rust AAC encoder, so this crate -//! publishes Opus or PCM and reads AAC that a gateway produced. +//! The decode side. There is no Rust AAC encoder, so `encode` produces AAC only +//! through a platform backend and otherwise reads AAC that a gateway produced. use crate::Error; diff --git a/rs/moq-audio/src/encode/backend/libopus.rs b/rs/moq-audio/src/encode/backend/libopus.rs new file mode 100644 index 0000000000..118c701be4 --- /dev/null +++ b/rs/moq-audio/src/encode/backend/libopus.rs @@ -0,0 +1,207 @@ +//! Opus through libopus 1.3.1, via [`unsafe_libopus`]. + +use bytes::Bytes; +use unsafe_libopus::{ + OPUS_APPLICATION_AUDIO, OPUS_GET_BITRATE_REQUEST, OPUS_GET_LOOKAHEAD_REQUEST, OPUS_OK, OPUS_RESET_STATE, + OPUS_SET_BITRATE_REQUEST, OPUS_SET_DTX_REQUEST, OpusEncoder, opus_encode_float, opus_encoder_create, + opus_encoder_ctl_impl, opus_encoder_destroy, varargs, +}; + +use super::Backend; +use crate::encode::{Encoded, Settings}; +use crate::{Error, opus}; + +pub(super) const NAME: &str = "libopus"; + +/// libopus packet size ceiling per RFC 6716 §3.4. +const MAX_PACKET_BYTES: usize = 4_000; + +pub(super) struct Libopus { + inner: *mut OpusEncoder, + scratch: Vec, + sample_rate: u32, + channels: u32, + frame_size: usize, + bitrate: u64, + lookahead: usize, +} + +// SAFETY: OpusEncoder is heap-allocated state owned exclusively by this +// struct; libopus encoder methods take a single &mut, so a unique owner is +// allowed to move it across threads. +unsafe impl Send for Libopus {} + +impl Libopus { + /// Opens at the settings' rate and layout, which the front end has already + /// checked against what Opus codes. + pub(super) fn open(settings: &Settings) -> Result, Error> { + Ok(Box::new(Self::new(settings)?)) + } + + fn new(settings: &Settings) -> Result { + let sample_rate = settings.sample_rate; + let channels = settings.layout.channels(); + let frame_size = opus::frame_size(sample_rate, settings.frame_duration)?; + + let mut err = 0i32; + // SAFETY: out-pointer `err` is valid; inner is checked for null below. + let inner = unsafe { + opus_encoder_create( + sample_rate as i32, + opus::validate_channels(channels)?, + OPUS_APPLICATION_AUDIO, + &mut err, + ) + }; + if err != OPUS_OK || inner.is_null() { + return Err(opus::error(err, "opus_encoder_create")); + } + + // Owned from here, so an early return below destroys it. + let mut backend = Self { + inner, + scratch: vec![0u8; MAX_PACKET_BYTES], + sample_rate, + channels, + frame_size, + bitrate: 0, + lookahead: 0, + }; + + if let Some(bitrate) = settings.bitrate { + backend.set_rate(bitrate.as_bps())?; + } + backend.set_ctl(OPUS_SET_DTX_REQUEST, i32::from(settings.dtx), "OPUS_SET_DTX")?; + + let bitrate = backend.get_ctl(OPUS_GET_BITRATE_REQUEST, "OPUS_GET_BITRATE")?; + backend.bitrate = u64::try_from(bitrate) + .map_err(|_| Error::Unsupported(format!("Opus reported negative bitrate {bitrate}")))?; + let lookahead = backend.get_ctl(OPUS_GET_LOOKAHEAD_REQUEST, "OPUS_GET_LOOKAHEAD")?; + backend.lookahead = usize::try_from(lookahead) + .map_err(|_| Error::Unsupported(format!("Opus reported negative lookahead {lookahead}")))?; + + Ok(backend) + } + + /// Refuse rates libopus would silently clamp, then apply the rest. + fn set_rate(&mut self, bitrate: u64) -> Result<(), Error> { + let (channels, frame_size) = (self.channels, self.frame_size); + let max = 300_000 * channels as u64; + let min = opus::bitrate_floor(self.sample_rate, frame_size).max(500); + if !(min..=max).contains(&bitrate) { + return Err(Error::Unsupported(format!( + "Opus bitrate must be between {min} and {max} bits per second for {channels} channel(s) at {frame_size} samples, got {bitrate}" + ))); + } + self.set_ctl(OPUS_SET_BITRATE_REQUEST, bitrate as i32, "OPUS_SET_BITRATE") + } + + fn set_ctl(&mut self, request: i32, value: i32, name: &'static str) -> Result<(), Error> { + // SAFETY: `inner` owns a live encoder and each request here expects one i32. + let rc = unsafe { opus_encoder_ctl_impl(self.inner, request, varargs![value]) }; + if rc != OPUS_OK { + return Err(opus::error(rc, name)); + } + Ok(()) + } + + fn get_ctl(&self, request: i32, name: &'static str) -> Result { + let mut value = 0; + // SAFETY: `inner` owns a live encoder and each request here expects one + // valid mutable i32 output. + let rc = unsafe { opus_encoder_ctl_impl(self.inner, request, varargs![&mut value]) }; + if rc != OPUS_OK { + return Err(opus::error(rc, name)); + } + Ok(value) + } +} + +impl Backend for Libopus { + fn encode(&mut self, pcm: &[f32]) -> Result { + // SAFETY: `inner` owns a live OpusEncoder; pcm and scratch slices are + // bounded by the lengths we pass, and the front end sized `pcm` to one frame. + let n = unsafe { + opus_encode_float( + self.inner, + pcm.as_ptr(), + self.frame_size as i32, + self.scratch.as_mut_ptr(), + self.scratch.len() as i32, + ) + }; + if n < 0 { + return Err(opus::error(n, "opus_encode_float")); + } + let payload = Bytes::copy_from_slice(&self.scratch[..n as usize]); + let activity = opus::activity(&payload, false); + Ok(Encoded { payload, activity }) + } + + fn reset(&mut self) { + // SAFETY: `inner` owns a live encoder and OPUS_RESET_STATE takes no arguments. + let rc = unsafe { opus_encoder_ctl_impl(self.inner, OPUS_RESET_STATE, varargs![]) }; + debug_assert_eq!(rc, OPUS_OK, "OPUS_RESET_STATE failed with {rc}"); + } + + fn set_bitrate(&mut self, bitrate: u64) -> Result<(), Error> { + if bitrate != self.bitrate { + self.set_rate(bitrate)?; + self.bitrate = bitrate; + } + Ok(()) + } + + fn bitrate(&self) -> u64 { + self.bitrate + } + + fn delay(&self) -> usize { + self.lookahead + } + + fn name(&self) -> &str { + NAME + } +} + +impl Drop for Libopus { + fn drop(&mut self) { + // SAFETY: `inner` is a live OpusEncoder that nothing else aliases. + unsafe { opus_encoder_destroy(self.inner) }; + } +} + +#[cfg(test)] +mod tests { + use unsafe_libopus::OPUS_GET_DTX_REQUEST; + + use super::*; + + #[test] + fn runtime_bitrate_reaches_libopus() { + let mut backend = Libopus::new(&Settings { + bitrate: Some(moq_net::bandwidth::Rate::from_bps(64_000)), + ..Settings::default() + }) + .unwrap(); + + backend.set_bitrate(32_000).unwrap(); + assert_eq!(backend.bitrate(), 32_000); + assert_eq!( + backend.get_ctl(OPUS_GET_BITRATE_REQUEST, "OPUS_GET_BITRATE").unwrap(), + 32_000 + ); + } + + #[test] + fn applies_dtx_control() { + let backend = Libopus::new(&Settings { + dtx: true, + ..Settings::default() + }) + .unwrap(); + + assert_eq!(backend.get_ctl(OPUS_GET_DTX_REQUEST, "OPUS_GET_DTX").unwrap(), 1); + } +} diff --git a/rs/moq-audio/src/encode/backend/mod.rs b/rs/moq-audio/src/encode/backend/mod.rs new file mode 100644 index 0000000000..1879d49f37 --- /dev/null +++ b/rs/moq-audio/src/encode/backend/mod.rs @@ -0,0 +1,323 @@ +//! Pluggable audio encoder backends. +//! +//! The mirror of the decode backends. [`Backend`] is the seam between the codec +//! and the [`Encoder`](super::Encoder) front end, which owns what every backend +//! of a codec shares: validating [`Settings`](super::Settings) against the +//! codec, framing the input, draining the startup delay at the end, and the +//! catalog entry, whose description is synthesized from the settings rather than +//! read back from the backend. +//! +//! [`open`] tries the platform encoders before the software ones, skipping any +//! that does not emit the codec, and refuses when none opens. The software tier +//! is libopus for Opus and a passthrough for PCM. AAC has no software encoder, +//! so it is only as available as the platform's, and no platform encoder is +//! wired in yet. + +use super::Encoded; +use super::encoder::{Codec, Kind, Settings}; +use crate::Error; + +mod libopus; +mod pcm; + +#[cfg(test)] +pub(crate) mod stub; + +/// An opened encoder: one frame of interleaved `f32` PCM in, one packet out. +/// +/// Input arrives at the settings' rate, in the crate's canonical channel order +/// for the settings' layout; a codec with another native order reorders it here. +/// +/// One packet per frame is part of the contract: the producer stamps packets by +/// counting frames. A codec that pipelines output (MediaCodec) needs a `flush` +/// and a zero-or-more return, which changes `Encoder::encode` too, so that lands +/// with the first backend that needs it rather than as an always-empty method. +pub(crate) trait Backend: Send { + /// Encode exactly one frame of the codec's frame size. + fn encode(&mut self, pcm: &[f32]) -> Result; + + /// Drop codec history so the next frame codes as if it were the first. + fn reset(&mut self); + + /// Retune the live encoder to `bitrate` bits per second, a no-op at the current + /// rate. + /// + /// No default: a backend that can't change rate mid-stream refuses with + /// [`Error::Unsupported`] and keeps its opening rate, rather than inheriting a + /// silent no-op that ignores congestion. + fn set_bitrate(&mut self, bitrate: u64) -> Result<(), Error>; + + /// The current target bitrate in bits per second, as the codec resolved it. + fn bitrate(&self) -> u64; + + /// Frames of codec priming at the start of the decoded stream, at the codec + /// rate: Opus lookahead, AAC encoder delay. + fn delay(&self) -> usize; + + /// The stable lowercase name [`Kind::Named`] selects this backend by. + fn name(&self) -> &str; +} + +/// A backend constructor: its name, the codecs it emits, and an opener. +struct Candidate { + name: &'static str, + codecs: &'static [Codec], + open: fn(&Settings) -> Result, Error>, +} + +/// Operating-system encoders, in priority order. +const PLATFORM: &[Candidate] = &[]; + +const SOFTWARE: &[Candidate] = &[ + Candidate { + name: libopus::NAME, + codecs: &[Codec::Opus], + open: libopus::Libopus::open, + }, + Candidate { + name: pcm::NAME, + codecs: &[Codec::Pcm], + open: pcm::Pcm::open, + }, +]; + +/// Test-only backends, in neither tier so `Auto` and `Software` never pick one: +/// they exist to be asked for by name. +#[cfg(test)] +const NAMED_ONLY: &[Candidate] = &[Candidate { + name: stub::NAME, + codecs: &[Codec::Aac], + open: stub::Stub::open, +}]; + +#[cfg(not(test))] +const NAMED_ONLY: &[Candidate] = &[]; + +/// Open the first backend that emits the codec and accepts the settings. +pub(crate) fn open(settings: &Settings) -> Result, Error> { + select(settings, candidates(&settings.kind, PLATFORM, SOFTWARE)) +} + +/// The candidates `kind` allows, in the order to try them. +/// +/// Takes the tiers as arguments so a test can supply stubs instead of whatever +/// this host compiles in. +fn candidates<'a>(kind: &Kind, platform: &'a [Candidate], software: &'a [Candidate]) -> Vec<&'a Candidate> { + match kind { + Kind::Auto => platform.iter().chain(software).collect(), + Kind::Software => software.iter().collect(), + Kind::Named(name) => platform + .iter() + .chain(software) + .chain(NAMED_ONLY) + .filter(|c| c.name == name) + .collect(), + } +} + +fn select(settings: &Settings, candidates: Vec<&Candidate>) -> Result, Error> { + let codec = settings.codec; + let mut refused = Vec::new(); + + for candidate in candidates { + if !candidate.codecs.contains(&codec) { + continue; + } + match (candidate.open)(settings) { + Ok(backend) => return Ok(backend), + Err(err) => refused.push((candidate.name, err)), + } + } + + // One refusal is the whole answer, so keep its variant. + if refused.len() == 1 { + let (_, err) = refused.remove(0); + return Err(err); + } + if !refused.is_empty() { + let reasons: Vec = refused.iter().map(|(name, err)| format!("{name}: {err}")).collect(); + return Err(Error::Unsupported(reasons.join(", "))); + } + + let available: Vec<&str> = PLATFORM + .iter() + .chain(SOFTWARE) + .filter(|c| c.codecs.contains(&codec)) + .map(|c| c.name) + .collect(); + let available = match available.is_empty() { + true => "none".to_owned(), + false => available.join(", "), + }; + Err(Error::Unsupported(match &settings.kind { + Kind::Named(name) => format!("no audio encoder named {name:?} for {codec} (this build has: {available})"), + Kind::Software => format!("no software {codec} audio encoder (this build has: {available})"), + Kind::Auto => format!("no {codec} audio encoder (this build has: {available})"), + })) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Opens anything it advertises and reports which candidate it came from. + struct Fake(&'static str); + + impl Backend for Fake { + fn encode(&mut self, _pcm: &[f32]) -> Result { + Ok(Encoded::new(bytes::Bytes::new())) + } + + fn reset(&mut self) {} + + fn set_bitrate(&mut self, _bitrate: u64) -> Result<(), Error> { + Ok(()) + } + + fn bitrate(&self) -> u64 { + 0 + } + + fn delay(&self) -> usize { + 0 + } + + fn name(&self) -> &str { + self.0 + } + } + + const PLATFORM_STUB: Candidate = Candidate { + name: "platform", + codecs: &[Codec::Aac], + open: |_| Ok(Box::new(Fake("platform"))), + }; + + /// Compiled in but refusing the settings, like a platform encoder asked for a + /// layout its framework does not open. + const REFUSING: Candidate = Candidate { + name: "refusing", + codecs: &[Codec::Aac], + open: |_| Err(Error::Unsupported("not these settings".into())), + }; + + const SOFTWARE_STUB: Candidate = Candidate { + name: "software", + codecs: &[Codec::Aac], + open: |_| Ok(Box::new(Fake("software"))), + }; + + /// Emits nothing but PCM, so an AAC request never reaches its opener. + const PCM_ONLY: Candidate = Candidate { + name: "pcm-only", + codecs: &[Codec::Pcm], + open: |_| panic!("opened for a codec it does not emit"), + }; + + fn aac(kind: Kind) -> Settings { + Settings { + kind, + ..Settings::from_input(Codec::Aac, &Default::default()) + } + } + + fn pick(kind: Kind, platform: &[Candidate], software: &[Candidate]) -> Result { + let settings = aac(kind); + let backend = select(&settings, candidates(&settings.kind, platform, software))?; + Ok(backend.name().to_owned()) + } + + #[test] + fn auto_prefers_platform() { + let name = pick(Kind::Auto, &[PCM_ONLY, PLATFORM_STUB], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "platform"); + } + + #[test] + fn auto_falls_back_to_software() { + let name = pick(Kind::Auto, &[REFUSING], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "software"); + } + + #[test] + fn software_skips_platform() { + let name = pick(Kind::Software, &[PLATFORM_STUB], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "software"); + } + + #[test] + fn named_forces_one() { + let name = pick( + Kind::Named("software".into()), + &[PLATFORM_STUB], + &[PCM_ONLY, SOFTWARE_STUB], + ) + .unwrap(); + assert_eq!(name, "software"); + } + + /// A named backend that refuses the settings is the answer: nothing else is tried. + #[test] + fn named_refusal_does_not_fall_back() { + let err = pick(Kind::Named("refusing".into()), &[REFUSING], &[SOFTWARE_STUB]).unwrap_err(); + assert!(err.to_string().contains("not these settings"), "{err}"); + } + + #[test] + fn every_refusal_is_reported() { + const ALSO_REFUSING: Candidate = Candidate { + name: "also-refusing", + ..REFUSING + }; + + let err = pick(Kind::Auto, &[REFUSING], &[ALSO_REFUSING]).unwrap_err(); + let message = err.to_string(); + assert!( + message.contains("refusing: ") && message.contains("also-refusing: "), + "{message}" + ); + } + + /// No AAC encoder is wired in outside the test stub, so `Auto` refuses at + /// construction and says there is nothing to fall back to. A platform backend + /// gates this to the hosts without one. + #[test] + fn aac_without_a_platform_encoder_is_refused() { + let err = open(&aac(Kind::Auto)).err().expect("no AAC encoder on this host"); + let message = err.to_string(); + assert!(message.contains("aac") && message.contains("none"), "{message}"); + } + + /// An unknown name says what this build has for the codec instead. + #[test] + fn unknown_name_lists_the_alternatives() { + let settings = Settings { + kind: Kind::Named("opus".into()), + ..Settings::default() + }; + let message = open(&settings) + .err() + .expect("no backend is named after its codec") + .to_string(); + assert!( + message.contains("\"opus\"") && message.contains(libopus::NAME), + "{message}" + ); + } + + #[test] + fn software_backends_open_by_name() { + let settings = Settings { + kind: Kind::Named(libopus::NAME.into()), + ..Settings::default() + }; + assert_eq!(open(&settings).unwrap().name(), libopus::NAME); + + let settings = Settings { + codec: Codec::Pcm, + kind: Kind::Named(pcm::NAME.into()), + ..Settings::default() + }; + assert_eq!(open(&settings).unwrap().name(), pcm::NAME); + } +} diff --git a/rs/moq-audio/src/encode/backend/pcm.rs b/rs/moq-audio/src/encode/backend/pcm.rs new file mode 100644 index 0000000000..04ff1f16e2 --- /dev/null +++ b/rs/moq-audio/src/encode/backend/pcm.rs @@ -0,0 +1,46 @@ +//! Uncompressed little-endian `f32` PCM, which needs no codec at all. + +use super::Backend; +use crate::encode::{Encoded, Settings}; +use crate::{Error, pcm}; + +pub(super) const NAME: &str = "pcm"; + +pub(super) struct Pcm { + bitrate: u64, +} + +impl Pcm { + pub(super) fn open(settings: &Settings) -> Result, Error> { + let bitrate = pcm::bitrate(settings.sample_rate, settings.layout.channels())?; + Ok(Box::new(Self { bitrate })) + } +} + +impl Backend for Pcm { + fn encode(&mut self, pcm: &[f32]) -> Result { + let mut payload = Vec::with_capacity(std::mem::size_of_val(pcm)); + for sample in pcm { + payload.extend_from_slice(&sample.to_le_bytes()); + } + Ok(Encoded::new(payload.into())) + } + + fn reset(&mut self) {} + + fn set_bitrate(&mut self, _bitrate: u64) -> Result<(), Error> { + Err(Error::Unsupported("pcm bitrate is fixed".into())) + } + + fn bitrate(&self) -> u64 { + self.bitrate + } + + fn delay(&self) -> usize { + 0 + } + + fn name(&self) -> &str { + NAME + } +} diff --git a/rs/moq-audio/src/encode/backend/stub.rs b/rs/moq-audio/src/encode/backend/stub.rs new file mode 100644 index 0000000000..c581332c15 --- /dev/null +++ b/rs/moq-audio/src/encode/backend/stub.rs @@ -0,0 +1,59 @@ +//! A stand-in AAC encoder, so the AAC front end is testable on a host with no +//! platform encoder. Selectable only by name, and only in tests. + +use bytes::Bytes; + +use super::Backend; +use crate::Error; +use crate::encode::{Encoded, Settings}; + +pub(crate) const NAME: &str = "stub"; + +/// The AudioToolbox AAC-LC encoder delay, which is what a real backend reports. +pub(crate) const DELAY: usize = 2112; + +/// Emits each frame's index as its payload, and cannot retune. +pub(crate) struct Stub { + bitrate: u64, + frames: u64, +} + +impl Stub { + pub(super) fn open(settings: &Settings) -> Result, Error> { + Ok(Box::new(Self { + bitrate: settings.bitrate.map_or(128_000, |rate| rate.as_bps()), + frames: 0, + })) + } +} + +impl Backend for Stub { + fn encode(&mut self, _pcm: &[f32]) -> Result { + let payload = Bytes::copy_from_slice(&self.frames.to_be_bytes()); + self.frames += 1; + Ok(Encoded::new(payload)) + } + + fn reset(&mut self) { + self.frames = 0; + } + + fn set_bitrate(&mut self, bitrate: u64) -> Result<(), Error> { + match bitrate == self.bitrate { + true => Ok(()), + false => Err(Error::Unsupported("the stub cannot change rate mid-stream".into())), + } + } + + fn bitrate(&self) -> u64 { + self.bitrate + } + + fn delay(&self) -> usize { + DELAY + } + + fn name(&self) -> &str { + NAME + } +} diff --git a/rs/moq-audio/src/encode/encoder.rs b/rs/moq-audio/src/encode/encoder.rs index 90479471db..5f908c8b68 100644 --- a/rs/moq-audio/src/encode/encoder.rs +++ b/rs/moq-audio/src/encode/encoder.rs @@ -1,26 +1,29 @@ //! Audio encoder front end. //! -//! [`Encoder`] dispatches over the closed [`Codec`] set. Opus wraps libopus -//! 1.3.1 via [`unsafe_libopus`], while PCM serializes interleaved `f32` samples -//! directly. +//! [`Encoder`] checks [`Settings`] against the codec, opens a +//! [`Backend`](super::backend::Backend) for it, and owns what every backend of a +//! codec shares: framing, the terminal drain, and the catalog entry. use std::str::FromStr; use std::time::Duration; use bytes::Bytes; -use unsafe_libopus::{ - OPUS_APPLICATION_AUDIO, OPUS_GET_BITRATE_REQUEST, OPUS_GET_LOOKAHEAD_REQUEST, OPUS_OK, OPUS_RESET_STATE, - OPUS_SET_BITRATE_REQUEST, OPUS_SET_DTX_REQUEST, OpusEncoder, opus_encode_float, opus_encoder_create, - opus_encoder_ctl_impl, opus_encoder_destroy, varargs, -}; use super::Encoded; +use super::backend::{self, Backend}; use crate::opus; use crate::pcm; use crate::{Error, Format, Layout}; -/// libopus packet size ceiling per RFC 6716 §3.4. -const MAX_PACKET_BYTES: usize = 4_000; +/// Samples per channel in one AAC-LC frame. +const AAC_FRAME_SIZE: usize = 1024; + +/// The audioObjectType of AAC-LC (ISO 14496-3 Table 1.17), `mp4a.40.2`. +const AAC_LC: u8 = 2; + +/// The widest sample rate an AudioSpecificConfig can name: the escape from the +/// frequency table is a 24-bit field. +const AAC_MAX_SAMPLE_RATE: u32 = 0xFF_FFFF; /// Output audio codec. `#[non_exhaustive]` so new codecs can be added without /// breaking external `match`es. @@ -32,6 +35,9 @@ pub enum Codec { Opus, /// Uncompressed interleaved little-endian IEEE-754 binary32 PCM. Pcm, + /// AAC-LC (`mp4a.40.2`), through the platform's encoder. A host without one + /// refuses it at construction. + Aac, } impl Codec { @@ -41,6 +47,7 @@ impl Codec { match self { Self::Opus => "opus", Self::Pcm => "pcm", + Self::Aac => "aac", } } } @@ -58,11 +65,25 @@ impl FromStr for Codec { match s { "opus" => Ok(Self::Opus), "pcm" => Ok(Self::Pcm), + "aac" => Ok(Self::Aac), other => Err(Error::Unsupported(format!("unknown codec: {other}"))), } } } +/// Encoder backend selection. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +#[non_exhaustive] +pub enum Kind { + /// Prefer a platform encoder, falling back to software. + #[default] + Auto, + /// Require a software backend. + Software, + /// Require a backend by its stable lowercase name: `"libopus"` or `"pcm"`. + Named(String), +} + /// PCM supplied to [`Producer::write`](super::Producer::write). #[derive(Clone, Debug)] #[non_exhaustive] @@ -101,9 +122,12 @@ pub struct Settings { /// Sample rate accepted by the codec. pub sample_rate: u32, /// Layout accepted by the codec. + /// + /// AAC takes the layouts its channelConfiguration names: mono, stereo, 3.0, + /// 4.0, 5.0, 5.1, and 7.1. pub layout: Layout, - /// Bitrate in bits per second. `None` lets Opus pick. PCM requires `None` - /// because its bitrate is fixed by the sample rate and channel count. + /// Bitrate in bits per second. `None` lets the codec pick. PCM requires + /// `None` because its bitrate is fixed by the sample rate and channel count. /// /// Rates too low for Opus to code anything at the chosen /// [`frame_duration`](Self::frame_duration) are rejected. The floor is 1200 @@ -113,8 +137,13 @@ pub struct Settings { /// Enable Opus discontinuous transmission during silence. pub dtx: bool, /// Encoded frame duration. Opus accepts 2.5 / 5 / 10 / 20 / 40 / 60 ms. - /// PCM accepts any duration containing a whole number of samples. + /// PCM accepts any duration containing a whole number of samples. AAC frames + /// are 1024 samples, so it accepts the duration that rounds to that at + /// [`sample_rate`](Self::sample_rate), which [`from_input`](Self::from_input) + /// fills in. pub frame_duration: Duration, + /// Which encoder implementation to use. + pub kind: Kind, } impl Settings { @@ -127,6 +156,7 @@ impl Settings { bitrate: None, dtx: false, frame_duration: Duration::from_millis(20), + kind: Kind::Auto, } } @@ -134,13 +164,72 @@ impl Settings { pub fn from_input(codec: Codec, input: &Input) -> Self { let sample_rate = match codec { Codec::Opus => opus::pick_rate(input.sample_rate), - Codec::Pcm => input.sample_rate, + Codec::Pcm | Codec::Aac => input.sample_rate, + }; + let defaults = Self::new(sample_rate, input.layout); + let frame_duration = match codec { + Codec::Aac => aac_frame_duration(sample_rate), + Codec::Opus | Codec::Pcm => defaults.frame_duration, }; Self { codec, - sample_rate, - layout: input.layout, - ..Self::default() + frame_duration, + ..defaults + } + } + + /// Check the settings against the codec, returning its frame size. + /// + /// Codec rules live here rather than in a backend, so every backend of a + /// codec refuses the same settings. + fn frame_size(&self) -> Result { + self.layout.validate()?; + let (rate, channels) = (self.sample_rate, self.layout.channels()); + + match self.codec { + Codec::Opus => { + opus::validate_rate(rate)?; + if !matches!(self.layout, Layout::Mono | Layout::Stereo) { + return Err(Error::Unsupported("opus requires a named mono or stereo layout".into())); + } + opus::frame_size(rate, self.frame_duration) + } + Codec::Pcm => { + if self.bitrate.is_some() { + return Err(Error::Unsupported( + "pcm bitrate is fixed; leave Settings::bitrate unset".into(), + )); + } + if self.dtx { + return Err(Error::Unsupported( + "pcm does not support discontinuous transmission".into(), + )); + } + if rate == 0 { + return Err(Error::Unsupported("pcm sample rate must be greater than zero".into())); + } + let frame_size = pcm::frame_size(rate, self.frame_duration)?; + pcm::frame_bytes(frame_size, channels)?; + pcm::bitrate(rate, channels)?; + Ok(frame_size) + } + Codec::Aac => { + if self.dtx { + return Err(Error::Unsupported( + "aac does not support discontinuous transmission".into(), + )); + } + aac_config(self)?; + let frames = (self.frame_duration.as_nanos() * u128::from(rate) + 500_000_000) / 1_000_000_000; + if frames != AAC_FRAME_SIZE as u128 { + return Err(Error::Unsupported(format!( + "aac frames are {AAC_FRAME_SIZE} samples, {:?} at {rate} Hz (got {:?})", + aac_frame_duration(rate), + self.frame_duration + ))); + } + Ok(AAC_FRAME_SIZE) + } } } } @@ -151,6 +240,52 @@ impl Default for Settings { } } +/// One AAC frame at `sample_rate`, to the nearest nanosecond. +fn aac_frame_duration(sample_rate: u32) -> Duration { + if sample_rate == 0 { + return Duration::ZERO; + } + let rate = u64::from(sample_rate); + Duration::from_nanos((AAC_FRAME_SIZE as u64 * 1_000_000_000 + rate / 2) / rate) +} + +/// The AudioSpecificConfig fields for AAC-LC at the settings' rate and layout. +/// +/// Only layouts with a channelConfiguration are accepted, since synthesizing +/// one from a bare count would mislabel the rest: config 3 is 3.0 where the +/// count's default layout is 2.1, and 6.1 has no config the encoder writes. +/// 7.1 takes config 7, the one every decoder reads as eight channels. +fn aac_config(settings: &Settings) -> Result { + let layout = settings.layout; + if !matches!( + layout, + Layout::Mono + | Layout::Stereo + | Layout::ThreePointZero + | Layout::FourPointZero + | Layout::FivePointZero + | Layout::FivePointOne + | Layout::SevenPointOne + ) { + return Err(Error::Unsupported(format!( + "aac has no channelConfiguration for {layout:?}; use mono, stereo, 3.0, 4.0, 5.0, 5.1, or 7.1" + ))); + } + + let sample_rate = settings.sample_rate; + if !(1..=AAC_MAX_SAMPLE_RATE).contains(&sample_rate) { + return Err(Error::Unsupported(format!( + "aac sample rate must be between 1 and {AAC_MAX_SAMPLE_RATE} Hz (got {sample_rate})" + ))); + } + + Ok(moq_mux::codec::aac::Config { + profile: AAC_LC, + sample_rate, + channel_count: layout.channels(), + }) +} + /// Audio encoder over codec-sized interleaved `f32` PCM. /// /// Build one with [`Encoder::new`], feed full PCM frames via @@ -158,38 +293,16 @@ impl Default for Settings { /// [`finish`](Self::finish). Publish every packet either call returns and apply /// the terminal [`Finish::discard_padding`] when the container supports it. pub struct Encoder { - backend: Backend, + backend: Box, settings: Settings, - /// Codec sample rate. - codec_rate: u32, - /// Codec channel count. - codec_channels: u32, - /// Current libopus target bitrate. - bitrate: u64, - /// Encoder lookahead expressed in the OpusHead 48 kHz timebase. - pre_skip: u16, - /// Encoder lookahead in codec-rate frames. - lookahead: usize, frame_size: usize, + /// The catalog description, synthesized from the settings at construction so + /// the rendition can be registered before the first packet exists. + description: Option, /// Whether input has reached the codec, since a fresh encoder owes no drain. started: bool, } -enum Backend { - Opus(Opus), - Pcm, -} - -struct Opus { - inner: *mut OpusEncoder, - scratch: Vec, -} - -// SAFETY: OpusEncoder is heap-allocated state owned exclusively by this -// struct; libopus encoder methods take a single &mut, so a unique owner is -// allowed to move it across threads. -unsafe impl Send for Opus {} - /// Packets emitted by [`Encoder::finish`] and the decoded padding at their end. pub struct Finish { packets: Vec, @@ -214,158 +327,40 @@ impl Finish { } impl Encoder { - /// Open an encoder for `settings`. + /// Open an encoder for `settings`, refusing a codec no backend on this host + /// encodes. pub fn new(settings: &Settings) -> Result { - settings.layout.validate()?; - match settings.codec { - Codec::Opus => Self::new_opus(settings.clone()), - Codec::Pcm => Self::new_pcm(settings.clone()), - } - } - - fn new_opus(settings: Settings) -> Result { - let codec_rate = settings.sample_rate; - opus::validate_rate(codec_rate)?; + let frame_size = settings.frame_size()?; + let backend = backend::open(settings)?; - let codec_channels = settings.layout.channels(); - if !matches!(settings.layout, Layout::Mono | Layout::Stereo) { - return Err(Error::Unsupported("opus requires a named mono or stereo layout".into())); - } - let channels = opus::validate_channels(codec_channels)?; - - let frame_size = opus::frame_size(codec_rate, settings.frame_duration)?; - - let mut err = 0i32; - // SAFETY: out-pointer `err` is valid; inner is checked for null below. - let inner = unsafe { opus_encoder_create(codec_rate as i32, channels, OPUS_APPLICATION_AUDIO, &mut err) }; - if err != OPUS_OK || inner.is_null() { - return Err(opus::error(err, "opus_encoder_create")); - } - - let configured = Self::configure_opus(inner, &settings, codec_rate, codec_channels, frame_size); - let (bitrate, lookahead, pre_skip) = match configured { - Ok(configured) => configured, - Err(err) => { - // SAFETY: `inner` was created above and not yet handed out. - unsafe { opus_encoder_destroy(inner) }; - return Err(err); + let description = match settings.codec { + Codec::Opus => { + // OpusHead carries the lookahead in the 48 kHz timebase. + let lookahead = backend.delay() as u64; + let pre_skip = u16::try_from((lookahead * 48_000) / u64::from(settings.sample_rate)) + .map_err(|_| Error::Unsupported(format!("Opus lookahead {lookahead} does not fit in OpusHead")))?; + let head = moq_mux::codec::opus::Config::new(settings.sample_rate, settings.layout.channels()) + .with_pre_skip(pre_skip) + .encode() + .map_err(moq_mux::Error::from)?; + Some(head) } + Codec::Aac => Some(aac_config(settings)?.encode()), + Codec::Pcm => None, }; Ok(Self { - backend: Backend::Opus(Opus { - inner, - scratch: vec![0u8; MAX_PACKET_BYTES], - }), - settings, - codec_rate, - codec_channels, - bitrate, - pre_skip, - lookahead, + backend, + settings: settings.clone(), frame_size, + description, started: false, }) } - fn new_pcm(settings: Settings) -> Result { - if settings.bitrate.is_some() { - return Err(Error::Unsupported( - "pcm bitrate is fixed; leave Settings::bitrate unset".into(), - )); - } - if settings.dtx { - return Err(Error::Unsupported( - "pcm does not support discontinuous transmission".into(), - )); - } - - let codec_rate = settings.sample_rate; - if codec_rate == 0 { - return Err(Error::Unsupported("pcm sample rate must be greater than zero".into())); - } - - let codec_channels = settings.layout.channels(); - if codec_channels == 0 { - return Err(Error::Unsupported("pcm channel count must be greater than zero".into())); - } - let frame_size = pcm::frame_size(codec_rate, settings.frame_duration)?; - pcm::frame_bytes(frame_size, codec_channels)?; - let bitrate = pcm::bitrate(codec_rate, codec_channels)?; - Ok(Self { - backend: Backend::Pcm, - settings, - codec_rate, - codec_channels, - bitrate, - pre_skip: 0, - lookahead: 0, - frame_size, - started: false, - }) - } - - fn configure_opus( - inner: *mut OpusEncoder, - settings: &Settings, - codec_rate: u32, - codec_channels: u32, - frame_size: usize, - ) -> Result<(u64, usize, u16), Error> { - if let Some(bitrate) = settings.bitrate { - Self::set_opus_bitrate(inner, codec_channels, bitrate.as_bps(), codec_rate, frame_size)?; - } - Self::set_opus_ctl(inner, OPUS_SET_DTX_REQUEST, i32::from(settings.dtx), "OPUS_SET_DTX")?; - - let bitrate = Self::get_opus_ctl(inner, OPUS_GET_BITRATE_REQUEST, "OPUS_GET_BITRATE")?; - let bitrate = u64::try_from(bitrate) - .map_err(|_| Error::Unsupported(format!("Opus reported negative bitrate {bitrate}")))?; - let lookahead = Self::get_opus_ctl(inner, OPUS_GET_LOOKAHEAD_REQUEST, "OPUS_GET_LOOKAHEAD")?; - let lookahead = u64::try_from(lookahead) - .map_err(|_| Error::Unsupported(format!("Opus reported negative lookahead {lookahead}")))?; - let pre_skip = u16::try_from((lookahead * 48_000) / codec_rate as u64) - .map_err(|_| Error::Unsupported(format!("Opus lookahead {lookahead} does not fit in OpusHead")))?; - let lookahead = usize::try_from(lookahead) - .map_err(|_| Error::Unsupported(format!("Opus lookahead {lookahead} does not fit in memory")))?; - - Ok((bitrate, lookahead, pre_skip)) - } - - fn set_opus_bitrate( - inner: *mut OpusEncoder, - channels: u32, - bitrate: u64, - codec_rate: u32, - frame_size: usize, - ) -> Result<(), Error> { - let max = 300_000 * channels as u64; - let min = opus::bitrate_floor(codec_rate, frame_size).max(500); - if !(min..=max).contains(&bitrate) { - return Err(Error::Unsupported(format!( - "Opus bitrate must be between {min} and {max} bits per second for {channels} channel(s) at {frame_size} samples, got {bitrate}" - ))); - } - Self::set_opus_ctl(inner, OPUS_SET_BITRATE_REQUEST, bitrate as i32, "OPUS_SET_BITRATE") - } - - fn set_opus_ctl(inner: *mut OpusEncoder, request: i32, value: i32, name: &'static str) -> Result<(), Error> { - // SAFETY: `inner` owns a live encoder and each request here expects one i32. - let rc = unsafe { opus_encoder_ctl_impl(inner, request, varargs![value]) }; - if rc != OPUS_OK { - return Err(opus::error(rc, name)); - } - Ok(()) - } - - fn get_opus_ctl(inner: *mut OpusEncoder, request: i32, name: &'static str) -> Result { - let mut value = 0; - // SAFETY: `inner` owns a live encoder and each request here expects one - // valid mutable i32 output. - let rc = unsafe { opus_encoder_ctl_impl(inner, request, varargs![&mut value]) }; - if rc != OPUS_OK { - return Err(opus::error(rc, name)); - } - Ok(value) + /// The encoder backend name in use, e.g. `"libopus"`. + pub fn name(&self) -> &str { + self.backend.name() } /// The encoder settings, including the latest accepted runtime bitrate. @@ -382,13 +377,13 @@ impl Encoder { /// Sample rate the codec actually runs at, which is /// [`Settings::sample_rate`]. pub fn codec_rate(&self) -> u32 { - self.codec_rate + self.settings.sample_rate } /// Channel count the codec actually runs at, which is /// [`Settings::layout`]'s channel count. pub fn codec_channels(&self) -> u32 { - self.codec_channels + self.settings.layout.channels() } /// Number of samples per channel the codec consumes per call to @@ -399,23 +394,21 @@ impl Encoder { /// Current target bitrate. pub fn bitrate(&self) -> moq_net::bandwidth::Rate { - moq_net::bandwidth::Rate::from_bps(self.bitrate) + moq_net::bandwidth::Rate::from_bps(self.backend.bitrate()) } - /// Retune the live Opus encoder to `bitrate`. + /// Retune the live encoder to `bitrate`. + /// + /// # Errors + /// + /// Returns [`Error::Unsupported`] when the codec's rate is fixed (PCM) or the + /// backend can't change it mid-stream. The encoder keeps running at its + /// opening rate, so a caller driving a control loop should stop adapting + /// rather than stop encoding. pub fn set_bitrate(&mut self, bitrate: moq_net::bandwidth::Rate) -> Result<(), Error> { - let Backend::Opus(opus) = &mut self.backend else { - return Err(Error::Unsupported("pcm bitrate is fixed".into())); - }; - if bitrate.as_bps() != self.bitrate { - Self::set_opus_bitrate( - opus.inner, - self.codec_channels, - bitrate.as_bps(), - self.codec_rate, - self.frame_size, - )?; - self.bitrate = bitrate.as_bps(); + let previous = self.backend.bitrate(); + self.backend.set_bitrate(bitrate.as_bps())?; + if bitrate.as_bps() != previous { self.settings.bitrate = Some(bitrate); } Ok(()) @@ -423,11 +416,7 @@ impl Encoder { /// Drop all codec history so a later epoch cannot emit audio from this one. pub(super) fn reset(&mut self) { - if let Backend::Opus(opus) = &mut self.backend { - // SAFETY: `inner` owns a live encoder and OPUS_RESET_STATE takes no arguments. - let rc = unsafe { opus_encoder_ctl_impl(opus.inner, OPUS_RESET_STATE, varargs![]) }; - debug_assert_eq!(rc, OPUS_OK, "OPUS_RESET_STATE failed with {rc}"); - } + self.backend.reset(); self.started = false; } @@ -436,47 +425,34 @@ impl Encoder { self.started } + /// Codec priming the catalog can't signal, in codec-rate frames, which the + /// producer folds into its timestamps instead. + /// + /// Opus declares its lookahead as OpusHead pre-skip, which the decoder trims, + /// so nothing is folded. An AudioSpecificConfig has no such field, so each + /// AAC packet is stamped that much earlier and the priming lands before the + /// first input sample rather than delaying it. + pub(super) fn folded_delay(&self) -> usize { + match self.settings.codec { + Codec::Aac => self.backend.delay(), + Codec::Opus | Codec::Pcm => 0, + } + } + /// Encode one frame of interleaved `f32` PCM at [`codec_rate`](Self::codec_rate). /// /// `pcm.len()` must equal `frame_size() * codec_channels()`. The /// [`Producer`](super::Producer) handles format conversion and resampling /// before calling this; for direct use, the caller does the same. pub fn encode(&mut self, pcm: &[f32]) -> Result { - let expected = self.frame_size * self.codec_channels as usize; + let expected = self.frame_size * self.codec_channels() as usize; if pcm.len() != expected { return Err(Error::Misaligned { got: std::mem::size_of_val(pcm), expected: expected * std::mem::size_of::(), }); } - let encoded = match &mut self.backend { - Backend::Opus(opus) => { - // SAFETY: `inner` owns a live OpusEncoder; pcm and scratch slices - // are bounded by the lengths we pass. - let n = unsafe { - opus_encode_float( - opus.inner, - pcm.as_ptr(), - self.frame_size as i32, - opus.scratch.as_mut_ptr(), - opus.scratch.len() as i32, - ) - }; - if n < 0 { - return Err(crate::opus::error(n, "opus_encode_float")); - } - let payload = Bytes::copy_from_slice(&opus.scratch[..n as usize]); - let activity = crate::opus::activity(&payload, false); - Encoded { payload, activity } - } - Backend::Pcm => { - let mut payload = Vec::with_capacity(std::mem::size_of_val(pcm)); - for sample in pcm { - payload.extend_from_slice(&sample.to_le_bytes()); - } - Encoded::new(payload.into()) - } - }; + let encoded = self.backend.encode(pcm)?; self.started = true; Ok(encoded) } @@ -496,7 +472,7 @@ impl Encoder { /// Same drain as [`finish`](Self::finish), without consuming the encoder. pub(super) fn drain(&mut self, pcm: &[f32]) -> Result { - let channels = self.codec_channels as usize; + let channels = self.codec_channels() as usize; let frame_samples = self.frame_size * channels; if pcm.len() > frame_samples || !pcm.len().is_multiple_of(channels) { return Err(Error::Misaligned { @@ -529,7 +505,8 @@ impl Encoder { }); } - let drain = self.lookahead.saturating_sub(padding); + let lookahead = self.backend.delay(); + let drain = lookahead.saturating_sub(padding); let silence = vec![0.0; frame_samples]; for _ in 0..drain.div_ceil(self.frame_size) { packets.push(self.encode(&silence)?); @@ -538,7 +515,7 @@ impl Encoder { let discard_padding = packets .len() .saturating_mul(self.frame_size) - .saturating_sub(self.lookahead) + .saturating_sub(lookahead) .saturating_sub(source_frames); Ok(Finish { @@ -549,46 +526,27 @@ impl Encoder { /// hang catalog entry describing this encoder's output stream. pub fn catalog(&self) -> hang::catalog::AudioConfig { - match self.settings.codec { - Codec::Opus => { - // `codec_channels` is validated to mono/stereo at encoder construction, - // so the OpusHead (channel mapping family 0) always encodes. - let head = moq_mux::codec::opus::Config::new(self.codec_rate, self.codec_channels) - .with_pre_skip(self.pre_skip) - .encode() - .expect("opus encoder channels validated to mono/stereo"); - - let mut config = hang::catalog::AudioConfig::new( - hang::catalog::AudioCodec::Opus, - self.codec_rate, - self.codec_channels, - ); - config.bitrate = self.settings.bitrate.map(moq_net::bandwidth::Rate::as_bps); - config.description = Some(head); - config.container = hang::catalog::Container::Legacy; - config - } - Codec::Pcm => { - let mut config = hang::catalog::AudioConfig::new( - hang::catalog::AudioCodec::Pcm, - self.codec_rate, - self.codec_channels, - ); - config.bitrate = Some( - pcm::bitrate(self.codec_rate, self.codec_channels) - .expect("pcm encoder bitrate validated at construction"), - ); - config.container = hang::catalog::Container::Legacy; - config - } - } - } -} + let (rate, channels) = (self.codec_rate(), self.codec_channels()); + let (codec, bitrate): (hang::catalog::AudioCodec, _) = match self.settings.codec { + Codec::Opus => ( + hang::catalog::AudioCodec::Opus, + self.settings.bitrate.map(moq_net::bandwidth::Rate::as_bps), + ), + Codec::Pcm => ( + hang::catalog::AudioCodec::Pcm, + Some(pcm::bitrate(rate, channels).expect("pcm encoder bitrate validated at construction")), + ), + Codec::Aac => ( + hang::catalog::AAC { profile: AAC_LC }.into(), + self.settings.bitrate.map(moq_net::bandwidth::Rate::as_bps), + ), + }; -impl Drop for Opus { - fn drop(&mut self) { - // SAFETY: `inner` is a live OpusEncoder that nothing else aliases. - unsafe { opus_encoder_destroy(self.inner) }; + let mut config = hang::catalog::AudioConfig::new(codec, rate, channels); + config.bitrate = bitrate; + config.description = self.description.clone(); + config.container = hang::catalog::Container::Legacy; + config } } @@ -609,13 +567,6 @@ mod tests { out } - fn opus_inner(encoder: &Encoder) -> *mut OpusEncoder { - let Backend::Opus(opus) = &encoder.backend else { - panic!("expected Opus encoder"); - }; - opus.inner - } - #[test] fn opus_encode_then_decode_keeps_signal_close() { let mut enc = Encoder::new(&Settings { @@ -675,7 +626,6 @@ mod tests { let desc = cfg.description.expect("OpusHead should be present"); assert_eq!(desc.len(), 19); let head = moq_mux::codec::opus::Config::parse(&mut desc.as_ref()).unwrap(); - assert_eq!(head.pre_skip, enc.pre_skip); assert_eq!(head.pre_skip, 312); } @@ -688,7 +638,7 @@ mod tests { let first = dec.decode(&enc.encode(&frame).unwrap().payload).unwrap(); assert_eq!( first.samples.len(), - (enc.frame_size() - enc.pre_skip as usize) * enc.codec_channels() as usize + (enc.frame_size() - enc.backend.delay()) * enc.codec_channels() as usize ); let second = dec.decode(&enc.encode(&frame).unwrap().payload).unwrap(); @@ -757,15 +707,6 @@ mod tests { enc.set_bitrate(moq_net::bandwidth::Rate::from_bps(32_000)).unwrap(); assert_eq!(enc.bitrate(), moq_net::bandwidth::Rate::from_bps(32_000)); assert_eq!(enc.settings().bitrate, Some(moq_net::bandwidth::Rate::from_bps(32_000))); - assert_eq!( - Encoder::get_opus_ctl( - opus_inner(&enc), - unsafe_libopus::OPUS_GET_BITRATE_REQUEST, - "OPUS_GET_BITRATE" - ) - .unwrap(), - 32_000 - ); } #[test] @@ -777,20 +718,6 @@ mod tests { assert_eq!(enc.bitrate(), original); } - #[test] - fn opus_applies_dtx_control() { - let enc = Encoder::new(&Settings { - dtx: true, - ..Settings::default() - }) - .unwrap(); - - assert_eq!( - Encoder::get_opus_ctl(opus_inner(&enc), unsafe_libopus::OPUS_GET_DTX_REQUEST, "OPUS_GET_DTX").unwrap(), - 1 - ); - } - #[test] fn codec_roundtrips_as_str() { assert_eq!(Codec::Opus.as_str(), "opus"); @@ -799,15 +726,20 @@ mod tests { assert_eq!(Codec::Pcm.as_str(), "pcm"); assert_eq!(Codec::Pcm.to_string(), "pcm"); assert_eq!("pcm".parse::().unwrap(), Codec::Pcm); - assert!("aac".parse::().is_err()); + assert_eq!(Codec::Aac.as_str(), "aac"); + assert_eq!(Codec::Aac.to_string(), "aac"); + assert_eq!("aac".parse::().unwrap(), Codec::Aac); + assert!("mp3".parse::().is_err()); } #[test] fn settings_fix_the_codec_rate() { let enc = Encoder::new(&Settings::new(24_000, Layout::Mono)).unwrap(); assert_eq!(enc.codec_rate(), 24_000); - assert_eq!(enc.catalog().sample_rate, 24_000); - assert_eq!(enc.pre_skip, 312); + let catalog = enc.catalog(); + assert_eq!(catalog.sample_rate, 24_000); + let head = moq_mux::codec::opus::Config::parse(&mut catalog.description.unwrap().as_ref()).unwrap(); + assert_eq!(head.pre_skip, 312); } #[test] @@ -906,4 +838,121 @@ mod tests { let settings = Settings::new(48_000, Layout::Discrete(2)); assert!(matches!(Encoder::new(&settings), Err(Error::Unsupported(_)))); } + + /// AAC settings routed to the test stub, since this host has no AAC encoder. + fn aac(layout: Layout) -> Settings { + Settings { + kind: Kind::Named(backend::stub::NAME.into()), + ..Settings::from_input(Codec::Aac, &Input::new(48_000, layout)) + } + } + + /// The ASC is synthesized from the settings, so it exists before any packet. + #[test] + fn aac_catalog_carries_the_synthesized_asc() { + let enc = Encoder::new(&aac(Layout::Stereo)).unwrap(); + assert_eq!(enc.name(), backend::stub::NAME); + assert_eq!(enc.frame_size(), 1024); + + let catalog = enc.catalog(); + assert_eq!(catalog.codec, hang::catalog::AAC { profile: 2 }.into()); + assert_eq!(catalog.codec.to_string(), "mp4a.40.2"); + assert_eq!(catalog.sample_rate, 48_000); + assert_eq!(catalog.channel_count, 2); + assert_eq!(catalog.container, hang::catalog::Container::Legacy); + // AAC-LC (2), 48 kHz (index 3), stereo (config 2). + assert_eq!(catalog.description.as_deref(), Some(&[0x11, 0x90][..])); + } + + #[test] + fn aac_takes_the_layouts_with_a_channel_configuration() { + for (layout, config) in [ + (Layout::Mono, 1), + (Layout::Stereo, 2), + (Layout::ThreePointZero, 3), + (Layout::FourPointZero, 4), + (Layout::FivePointZero, 5), + (Layout::FivePointOne, 6), + (Layout::SevenPointOne, 7), + ] { + let catalog = Encoder::new(&aac(layout)).unwrap().catalog(); + let description = catalog.description.unwrap(); + assert_eq!(description[1] >> 3 & 0xF, config, "{layout:?}"); + assert_eq!(catalog.channel_count, layout.channels(), "{layout:?}"); + } + + for layout in [ + Layout::TwoPointOne, + Layout::Quad, + Layout::SixPointOne, + Layout::Discrete(2), + ] { + assert!( + matches!(Encoder::new(&aac(layout)), Err(Error::Unsupported(_))), + "{layout:?}" + ); + } + } + + /// AAC frames are 1024 samples however the duration is spelled. + #[test] + fn aac_frame_duration_is_the_codecs() { + assert_eq!(aac(Layout::Stereo).frame_duration, Duration::from_nanos(21_333_333)); + let settings = Settings { + frame_duration: Duration::from_micros(21_333), + ..aac(Layout::Stereo) + }; + assert_eq!(Encoder::new(&settings).unwrap().frame_size(), 1024); + + let settings = Settings { + frame_duration: Duration::from_millis(20), + ..aac(Layout::Stereo) + }; + let err = Encoder::new(&settings).err().expect("20 ms is 960 samples"); + assert!(err.to_string().contains("1024"), "{err}"); + } + + #[test] + fn aac_refuses_dtx() { + let settings = Settings { + dtx: true, + ..aac(Layout::Stereo) + }; + assert!(matches!(Encoder::new(&settings), Err(Error::Unsupported(_)))); + } + + /// A backend that can't retune keeps its opening rate. + #[test] + fn fixed_rate_backend_keeps_its_opening_rate() { + let mut enc = Encoder::new(&Settings { + bitrate: Some(moq_net::bandwidth::Rate::from_bps(96_000)), + ..aac(Layout::Stereo) + }) + .unwrap(); + + let err = enc.set_bitrate(moq_net::bandwidth::Rate::from_bps(64_000)); + assert!(matches!(err, Err(Error::Unsupported(_)))); + assert_eq!(enc.bitrate(), moq_net::bandwidth::Rate::from_bps(96_000)); + assert_eq!(enc.settings().bitrate, Some(moq_net::bandwidth::Rate::from_bps(96_000))); + assert_eq!(enc.catalog().bitrate, Some(96_000)); + } + + /// The drain pushes the encoder delay out through whole silent frames. + #[test] + fn aac_finish_drains_the_encoder_delay() { + let mut enc = Encoder::new(&aac(Layout::Mono)).unwrap(); + assert_eq!(enc.folded_delay(), backend::stub::DELAY); + enc.encode(&[0.0; 1024]).unwrap(); + + // 2112 frames of delay take three 1024-frame packets. + let finish = enc.finish(&[]).unwrap(); + assert_eq!(finish.packets().len(), 3); + assert_eq!(finish.discard_padding(), 3 * 1024 - backend::stub::DELAY); + } + + /// Opus signals its lookahead as pre-skip, so the producer folds none of it. + #[test] + fn opus_folds_no_delay() { + assert_eq!(Encoder::new(&Settings::default()).unwrap().folded_delay(), 0); + } } diff --git a/rs/moq-audio/src/encode/mod.rs b/rs/moq-audio/src/encode/mod.rs index 9c918b9744..10c3125e41 100644 --- a/rs/moq-audio/src/encode/mod.rs +++ b/rs/moq-audio/src/encode/mod.rs @@ -1,6 +1,8 @@ //! Encode raw PCM and publish it as a moq audio track. //! -//! The output codec is selected via [`Codec`]. +//! The output codec is selected via [`Codec`], and the implementation behind it +//! via [`Kind`]: a platform encoder when the host has one, software otherwise. +//! AAC has no software encoder, so a host without a platform one refuses it. //! //! Entry points, high to low level: //! - `publish_capture` captures a microphone (or system audio) and publishes @@ -15,6 +17,7 @@ //! `publish_capture` is unlinked above because it only exists with the `capture` //! feature, so a default-feature rustdoc build has nothing to link to. +mod backend; mod encoded; mod encoder; mod producer; @@ -23,7 +26,7 @@ mod producer; mod capture; pub use encoded::Encoded; -pub use encoder::{Codec, Encoder, Finish, Input, Settings}; +pub use encoder::{Codec, Encoder, Finish, Input, Kind, Settings}; pub use producer::{Options, Producer}; #[cfg(feature = "capture")] diff --git a/rs/moq-audio/src/encode/producer.rs b/rs/moq-audio/src/encode/producer.rs index 1b61545141..afb0b73722 100644 --- a/rs/moq-audio/src/encode/producer.rs +++ b/rs/moq-audio/src/encode/producer.rs @@ -366,7 +366,12 @@ impl Producer { let chunk: Vec = self.pending.drain(..frame_samples).collect(); let packet = self.encoder.encode(&chunk)?; - let timestamp = Self::timestamp(epoch_us, self.frames_produced, self.encoder.codec_rate())?; + let timestamp = Self::timestamp( + epoch_us, + self.frames_produced, + self.encoder.folded_delay(), + self.encoder.codec_rate(), + )?; self.frames_produced += self.encoder.frame_size() as u64; self.activity = packet.activity; Self::publish(&mut self.track, packet, timestamp)?; @@ -376,10 +381,18 @@ impl Producer { Ok(()) } - /// PTS of the next frame: the epoch plus the samples emitted since it. - fn timestamp(epoch_us: u64, frames_produced: u64, codec_rate: u32) -> Result { - let offset_us = (frames_produced * 1_000_000) / codec_rate as u64; - Ok(Timestamp::from_micros(epoch_us + offset_us)?) + /// PTS of the frame `frames` samples past the epoch, stamped `delay` samples + /// earlier to fold in codec priming the catalog can't signal. + /// + /// Priming that would land before a zero epoch is stamped at zero instead: it + /// decodes to the codec's warm-up rather than to input, so only its spacing is + /// lost. + fn timestamp(epoch_us: u64, frames: u64, delay: usize, codec_rate: u32) -> Result { + let frames = i128::from(frames) - delay as i128; + let offset_us = (frames * 1_000_000).div_euclid(i128::from(codec_rate)); + let micros = (i128::from(epoch_us) + offset_us).max(0); + let micros = u64::try_from(micros).map_err(|_| moq_net::TimeOverflow)?; + Ok(Timestamp::from_micros(micros)?) } fn publish( @@ -474,8 +487,10 @@ impl Producer { let codec_rate = self.encoder.codec_rate(); let channels = self.encoder.codec_channels() as usize; let source_frames = self.pending.len() / channels; - let start = Self::timestamp(epoch_us, self.frames_produced, codec_rate)?; - let end = Self::timestamp(epoch_us, self.frames_produced + source_frames as u64, codec_rate)?; + let delay = self.encoder.folded_delay(); + let start = Self::timestamp(epoch_us, self.frames_produced, delay, codec_rate)?; + // The source ends where it ends: priming only moves the packets carrying it. + let end = Self::timestamp(epoch_us, self.frames_produced + source_frames as u64, 0, codec_rate)?; let finish = self.encoder.drain(&self.pending)?; let discard_padding = finish.discard_padding(); let packets = finish.into_packets(); @@ -493,7 +508,7 @@ impl Producer { )?; } else { for packet in packets { - let timestamp = Self::timestamp(epoch_us, self.frames_produced, codec_rate)?; + let timestamp = Self::timestamp(epoch_us, self.frames_produced, delay, codec_rate)?; self.activity = packet.activity; Self::publish(&mut self.track, packet, timestamp)?; self.frames_produced += frame_size as u64; @@ -898,6 +913,51 @@ mod tests { assert_eq!(pts, vec![1_000_000]); } + /// AAC can't signal its encoder delay, so each packet is stamped that much + /// earlier and the first input sample still decodes at the epoch. + #[tokio::test] + async fn aac_folds_the_encoder_delay_into_timestamps() { + async fn pts(epoch_us: u64) -> Vec { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); + let consumer = broadcast.consume(); + + let input = Input::new(48_000, Layout::Mono); + let options = Options { + track: Some("audio".to_string()), + settings: Settings { + kind: crate::encode::Kind::Named(crate::encode::backend::stub::NAME.into()), + ..Settings::from_input(crate::encode::Codec::Aac, &input) + }, + ..Options::default() + }; + let mut producer = Producer::new(&mut broadcast, catalog, input, &options).unwrap(); + + let track = consumer + .track("audio") + .unwrap() + .subscribe(moq_net::track::Subscription::default().with_max_age(Duration::from_secs(1))) + .await + .unwrap(); + let mut reader = moq_mux::container::Consumer::new( + track, + moq_mux::container::legacy::Wire(moq_mux::container::Kind::Audio), + ); + + producer.write(&pcm_frame(&[0.1; 4 * 1024], epoch_us)).unwrap(); + let mut pts = Vec::new(); + for _ in 0..4 { + pts.push(reader.read().await.unwrap().expect("a packet").timestamp.as_micros()); + } + pts + } + + // 2112 frames of delay at 48 kHz is 44 ms, rounded down per packet. + assert_eq!(pts(1_000_000).await, vec![956_000, 977_333, 998_666, 1_020_000]); + // Priming before a zero epoch stamps at zero; the input still starts on time. + assert_eq!(pts(0).await, vec![0, 0, 0, 20_000]); + } + /// The encoder needs no correction for the resampler's own delay: it anchors /// the epoch to the first input timestamp and advances by emitted samples, /// while `Resampler::process` drops its startup silence rather than passing it diff --git a/rs/moq-audio/src/lib.rs b/rs/moq-audio/src/lib.rs index b8c7aa9242..92d1c04dff 100644 --- a/rs/moq-audio/src/lib.rs +++ b/rs/moq-audio/src/lib.rs @@ -3,8 +3,9 @@ //! Counterpart to [`moq-video`](https://crates.io/crates/moq-video) for audio //! tracks, and shaped the same way. Sits on top of [`moq_mux`] and [`hang`] and //! adds the missing piece for native callers: Rust-native Opus and uncompressed -//! PCM codecs that turn raw samples into HANG audio tracks and back, plus AAC-LC -//! on the way in, which is what the gateways (RTMP, SRT, HLS, gstreamer) publish. +//! PCM codecs that turn raw samples into HANG audio tracks and back, plus AAC-LC, +//! which is what the gateways (RTMP, SRT, HLS, gstreamer) publish. AAC decodes in +//! software and encodes only through a platform encoder. //! //! - `capture` describes an audio source (`capture::Config`) and grabs buffers //! per platform: a microphone via cpal (CoreAudio / WASAPI / ALSA) everywhere, @@ -26,7 +27,7 @@ //! - [`decode`] subscribes to an encoded track and decodes it back to PCM. //! [`decode::Consumer`] is the mirror of [`encode::Producer`]. It reads AAC-LC //! too, behind the default-on `aac` feature, since a broadcast that came in -//! through a gateway is AAC rather than one of the two codecs we encode. +//! through a gateway is AAC rather than one of the codecs we encode in software. //! - `playback` plays decoded PCM out a speaker. `playback::Engine` owns the //! output device and mixes the `playback::Sink`s registered with it, so one //! device serves every track in a call. Requires the `playback` feature, so diff --git a/rs/moq-ffi/src/audio.rs b/rs/moq-ffi/src/audio.rs index 5e8bea78ce..22c3ba4742 100644 --- a/rs/moq-ffi/src/audio.rs +++ b/rs/moq-ffi/src/audio.rs @@ -48,7 +48,7 @@ impl From for moq_audio::Format { /// Audio codec selection for the encoder. /// /// An immutable object so adding a codec later does not break callers -/// switching over a closed enum. Currently only Opus is available. +/// switching over a closed enum. #[derive(uniffi::Object)] pub struct MoqAudioCodec { inner: moq_audio::encode::Codec, @@ -63,6 +63,16 @@ impl MoqAudioCodec { inner: moq_audio::encode::Codec::Opus, }) } + + /// AAC-LC (`mp4a.40.2`) through the platform's encoder, at the input's rate + /// and layout. A host without one refuses it when the producer is built. + /// Its frames are 1024 samples, so leave `frame_duration_us` at 0. + #[uniffi::constructor] + pub fn aac() -> Arc { + Arc::new(Self { + inner: moq_audio::encode::Codec::Aac, + }) + } } impl MoqAudioCodec { @@ -96,7 +106,7 @@ pub struct MoqAudioEncoderOutput { pub bitrate: Option, /// Encoded frame duration in microseconds. Opus accepts exactly /// 2500/5000/10000/20000/40000/60000 us, and the default 20 ms matches the - /// JS publish path. + /// JS publish path. 0 takes the codec's own frame, which AAC needs. #[uniffi(default = 20000)] pub frame_duration_us: u32, } @@ -290,7 +300,16 @@ impl MoqBroadcastProducer { options.settings.layout = moq_audio::Layout::from_channels(channels)?; } options.settings.bitrate = output.bitrate.map(|bps| moq_net::bandwidth::Rate::from_bps(bps.into())); - options.settings.frame_duration = Duration::from_micros(output.frame_duration_us.into()); + if output.frame_duration_us != 0 { + options.settings.frame_duration = Duration::from_micros(output.frame_duration_us.into()); + } else if output.codec.codec() == moq_audio::encode::Codec::Aac { + // from_input sized this at the input rate. The codec rate may be the + // override above, and AAC's own frame is 1024 samples of that rate. + let mut rated = input.clone(); + rated.sample_rate = options.settings.sample_rate; + options.settings.frame_duration = + moq_audio::encode::Settings::from_input(moq_audio::encode::Codec::Aac, &rated).frame_duration; + } if let Some(bandwidth) = &bandwidth { options.bandwidth = bandwidth.allocator().clone(); } diff --git a/rs/moq-ffi/src/test.rs b/rs/moq-ffi/src/test.rs index 265dee6162..01f10f0e99 100644 --- a/rs/moq-ffi/src/test.rs +++ b/rs/moq-ffi/src/test.rs @@ -512,6 +512,60 @@ async fn raw_audio_frame_durations() { broadcast.finish().unwrap(); } +/// A frame duration of 0 takes the codec's own frame, and AAC, which encodes only +/// through a platform encoder, is refused where there is none. +#[cfg(feature = "audio")] +#[tokio::test] +async fn raw_audio_codec_default_frame() { + use crate::audio::*; + + let broadcast = MoqBroadcastProducer::new().unwrap(); + let input = || MoqAudioEncoderInput { + format: MoqAudioSampleFormat::F32, + sample_rate: 48_000, + channels: 2, + }; + let output = |codec| MoqAudioEncoderOutput { + codec, + sample_rate: None, + channels: None, + bitrate: None, + frame_duration_us: 0, + }; + + let opus = broadcast + .encode_audio("opus".into(), input(), output(MoqAudioCodec::opus()), None) + .unwrap(); + opus.finish().unwrap(); + + assert_eq!(MoqAudioCodec::aac().codec(), moq_audio::encode::Codec::Aac); + let aac = broadcast.encode_audio("aac".into(), input(), output(MoqAudioCodec::aac()), None); + let Err(MoqError::Audio(message)) = aac else { + panic!("no platform AAC encoder on this host"); + }; + assert!(message.contains("no aac audio encoder"), "{message}"); + + // 0 still means 1024 samples after an output-rate override, not the input rate. + let mut resampled = output(MoqAudioCodec::aac()); + resampled.sample_rate = Some(48_000); + let aac = broadcast.encode_audio( + "aac-rate".into(), + MoqAudioEncoderInput { + format: MoqAudioSampleFormat::F32, + sample_rate: 44_100, + channels: 2, + }, + resampled, + None, + ); + let Err(MoqError::Audio(message)) = aac else { + panic!("no platform AAC encoder on this host"); + }; + assert!(message.contains("no aac audio encoder"), "{message}"); + + broadcast.finish().unwrap(); +} + #[tokio::test] async fn raw_track_datagram_roundtrip() { let broadcast = MoqBroadcastProducer::new().unwrap(); diff --git a/swift/Sources/Moq/Aliases.swift b/swift/Sources/Moq/Aliases.swift index dcfef7b25e..c864511cff 100644 --- a/swift/Sources/Moq/Aliases.swift +++ b/swift/Sources/Moq/Aliases.swift @@ -52,7 +52,7 @@ public typealias VideoDecodedFrame = MoqVideoDecodedFrame public typealias AudioDecoderOutput = MoqFFI.MoqAudioDecoderOutput /// A raw PCM sample format, mirroring WebCodecs `AudioData.format`. public typealias AudioSampleFormat = MoqFFI.MoqAudioSampleFormat -/// Selects the audio encoder codec. Build one with `AudioCodec.opus()`. +/// Selects the audio encoder codec. Build one with `AudioCodec.opus()` or `AudioCodec.aac()`. public typealias AudioCodec = MoqFFI.MoqAudioCodec /// One raw video frame: pixels in the configured layout plus a presentation /// timestamp. diff --git a/swift/Sources/Moq/Broadcast.swift b/swift/Sources/Moq/Broadcast.swift index cf52ddba1b..85487ed175 100644 --- a/swift/Sources/Moq/Broadcast.swift +++ b/swift/Sources/Moq/Broadcast.swift @@ -273,7 +273,7 @@ public final class BroadcastProducer: Sendable { /// Open a raw-audio track. PCM written via `AudioProducer.write` is encoded /// inside the FFI boundary per `input`/`output`. Select the codec with - /// `AudioCodec.opus()` (currently the only constructor), placed in `output`. + /// `AudioCodec.opus()` or `AudioCodec.aac()`, placed in `output`. /// /// Pass `bandwidth` to reserve this track's bitrate against the session's /// allocator so a co-resident video encoder sizes itself against what is left. From 0e8a5e8fe9f304822d6e8d4f677c85eeb82572d3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 17:12:12 -0700 Subject: [PATCH 08/11] quest(audio-codecs): ADTS export refuses what it cannot label (#4202) Co-authored-by: Claude Opus 5.5 --- quest/m1/audio-codecs/README.md | 1 + quest/m1/audio-codecs/adts-refusals.md | 25 +++++++++++++++++++++++++ 2 files changed, 26 insertions(+) create mode 100644 quest/m1/audio-codecs/adts-refusals.md diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index ee3cca548f..b298d3560f 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -45,6 +45,7 @@ its own decode and encode quest so verification stays per host. - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - mapping family 1 decodes on every host through the multistream decoder +- [ADTS refusals](/quest/m1/audio-codecs/adts-refusals.md) - the ADTS writer refuses channel counts and object types it cannot label instead of mislabeling them - [AudioToolbox encode](/quest/m1/audio-codecs/encode-audiotoolbox.md) - macOS and iOS encode AAC-LC ## Related diff --git a/quest/m1/audio-codecs/adts-refusals.md b/quest/m1/audio-codecs/adts-refusals.md new file mode 100644 index 0000000000..c7896e377d --- /dev/null +++ b/quest/m1/audio-codecs/adts-refusals.md @@ -0,0 +1,25 @@ +# [S] ADTS export refuses what it cannot label + +## Goal + +`moq-mux`'s ADTS writer never labels an AAC track with the wrong layout or +object type. A channel count ADTS cannot name is refused, not silently written +as stereo, and an explicit SBR or PS description is written with the object +type ADTS can carry or refused, not masked to two bits into a wrong profile. + +## Plan + +- `channel_config_from_count` falls back to stereo for counts it cannot + represent; refuse instead, the same way #4178 made channelConfiguration 11 to + 14 refuse. +- The ADTS header masks the object type to two bits, so an explicit HE-AAC + (object type 5) or HE-AACv2 (29) description is mislabeled. Decide per case: + signal the backward-compatible AAC-LC core (implicit SBR) when the description + allows it, otherwise refuse. +- Tests for both with real fixtures, checked against ffprobe. + +Public API: none. Wire: none; TS output changes only for inputs it mislabeled. + +## Related + +- [#4178](https://github.com/moq-dev/moq/pull/4178) - the PCE export that found these From 7824c89e37bf5e9aabe6fa136d3a26574cff936d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 21:00:55 -0700 Subject: [PATCH 09/11] feat(audio): decode surround Opus through the multistream decoder (#4177) Co-authored-by: Claude Opus 5.5 Co-authored-by: Grok 4.7 --- doc/lib/rs/moq-audio.md | 6 +- quest/m1/audio-codecs/README.md | 2 +- quest/m1/audio-codecs/opus-surround.md | 22 -- quest/m1/audio-codecs/ts-opus-surround.md | 20 ++ quest/m1/audio-opus-input.md | 4 - rs/moq-audio/src/decode/backend/libopus.rs | 341 +++++++++++++++++++-- rs/moq-audio/src/opus.rs | 199 +++++++++++- rs/moq-mux/src/catalog/msf/consumer.rs | 3 +- rs/moq-mux/src/codec/opus/import.rs | 5 +- rs/moq-mux/src/codec/opus/mod.rs | 206 ++++++++++++- 10 files changed, 743 insertions(+), 65 deletions(-) delete mode 100644 quest/m1/audio-codecs/opus-surround.md create mode 100644 quest/m1/audio-codecs/ts-opus-surround.md diff --git a/doc/lib/rs/moq-audio.md b/doc/lib/rs/moq-audio.md index c76dc1179e..13078570e0 100644 --- a/doc/lib/rs/moq-audio.md +++ b/doc/lib/rs/moq-audio.md @@ -39,14 +39,16 @@ first, then software. `decode::Config::kind` forces one (`Kind::Software`, or | Backend | Decodes | Hosts | | --- | --- | --- | -| `libopus` | Opus, mono or stereo | all | +| `libopus` | Opus, mono or stereo, and surround up to 7.1 (channel mapping family 1) | all | | `pcm` | PCM | all | | `symphonia` | AAC-LC, mono or stereo (the default-on `aac` feature) | all | No platform decoder is wired in yet, so multichannel AAC and HE-AAC declared in its config are refused at construction on every host. HE-AAC signaled only in band plays as its half-rate LC core. Linux has no OS audio decoder, so it will -stay that way there. +stay that way there. Surround Opus is pure Rust, so it is the one multichannel +path every host has. Opus mapping families other than 0 and 1 (ambisonics, and +255's unpositioned channels) are refused, since they declare no speakers. `encode` selects the same way, through `encode::Settings::kind`, and `Encoder::name()` reports what opened. diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index b298d3560f..71b47defcc 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -44,9 +44,9 @@ its own decode and encode quest so verification stays per host. ## Quests - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers -- [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - mapping family 1 decodes on every host through the multistream decoder - [ADTS refusals](/quest/m1/audio-codecs/adts-refusals.md) - the ADTS writer refuses channel counts and object types it cannot label instead of mislabeling them - [AudioToolbox encode](/quest/m1/audio-codecs/encode-audiotoolbox.md) - macOS and iOS encode AAC-LC +- [TS surround Opus](/quest/m1/audio-codecs/ts-opus-surround.md) - a 3 to 8 channel Opus stream from MPEG-TS imports with a family 1 OpusHead that decodes ## Related diff --git a/quest/m1/audio-codecs/opus-surround.md b/quest/m1/audio-codecs/opus-surround.md deleted file mode 100644 index 059552ddde..0000000000 --- a/quest/m1/audio-codecs/opus-surround.md +++ /dev/null @@ -1,22 +0,0 @@ -# [S] Opus surround through the multistream decoder - -## Goal - -An Opus track with channel mapping family 1 (up to 7.1, RFC 7845 §5.1.1) -decodes on every host, delivered in the canonical `Layout` order. Pure Rust, -so this is the one multichannel path Linux gets. - -## Plan - -- `moq_mux::codec::opus::Config` parses the channel mapping (family, stream - count, coupled count, and the mapping table) instead of skipping it. Family - 0 stays mono/stereo; family 1 maps to a `Layout` by channel count; family - 255 and unknown families are refused, since they carry no speaker - assignment. -- The Opus decode backend opens `opus_multistream_decoder_create` from - `unsafe-libopus` when the family is 1, and reorders Vorbis order into the - canonical one on the way out. -- Encode stays family 0; `Config::encode` keeps refusing more than two - channels. -- Regression: a family-1 5.1 fixture decodes to six channels in canonical - order; family 255 is refused at construction. diff --git a/quest/m1/audio-codecs/ts-opus-surround.md b/quest/m1/audio-codecs/ts-opus-surround.md new file mode 100644 index 0000000000..5e6e6be4fc --- /dev/null +++ b/quest/m1/audio-codecs/ts-opus-surround.md @@ -0,0 +1,20 @@ +# [S] Surround Opus from MPEG-TS + +## Goal + +An MPEG-TS Opus stream with 3 to 8 channels imports with an OpusHead that +decodes, so it plays natively like the same stream from Matroska or FLV. + +## Plan + +- The TS importer knows only the `channel_config_code` from the Opus + extension descriptor and builds `opus::Config::new(48_000, channels)`, whose + `encode` refuses more than two channels. The catalog then carries no + description and `moq-audio` refuses the track. +- Codes 3 to 8 mean family 1 with the Vorbis default stream and coupled counts + and mapping (the Vorbis orders of RFC 7845 §5.1.1.2), so the + importer can synthesize the head. That needs a way to build a family 1 + `Config` and have `encode` emit its table; decide whether that is a + constructor on `opus::Mapping` or a TS-local head writer. +- Regression: a 5.1 TS fixture imports with a family 1 description and decodes + to six channels. diff --git a/quest/m1/audio-opus-input.md b/quest/m1/audio-opus-input.md index 9a6b0927b5..c98e60889a 100644 --- a/quest/m1/audio-opus-input.md +++ b/quest/m1/audio-opus-input.md @@ -22,7 +22,3 @@ real Opus fixtures in CI. Verify timestamps as well as decoded sample counts. Public API and wire schema: unchanged. Update the existing codec documentation with supported/refused cases; header-parser changes stay focused on this path. - -## Related - -- [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - adds supported mappings separately diff --git a/rs/moq-audio/src/decode/backend/libopus.rs b/rs/moq-audio/src/decode/backend/libopus.rs index 0fa9f78b69..0d66970c62 100644 --- a/rs/moq-audio/src/decode/backend/libopus.rs +++ b/rs/moq-audio/src/decode/backend/libopus.rs @@ -1,8 +1,11 @@ //! Opus through libopus, the software decoder for every host. +//! +//! Every stream goes through the multistream decoder, which is also how a +//! family 0 (mono/stereo) stream decodes: one stream, coupled when stereo. use unsafe_libopus::{ - OPUS_OK, OPUS_RESET_STATE, OpusDecoder, opus_decode_float, opus_decoder_create, opus_decoder_ctl_impl, - opus_decoder_destroy, varargs, + OPUS_OK, OPUS_RESET_STATE, OpusMSDecoder, opus_multistream_decode_float, opus_multistream_decoder_create, + opus_multistream_decoder_ctl_impl, opus_multistream_decoder_destroy, varargs, }; use super::Backend; @@ -14,10 +17,18 @@ pub(super) const NAME: &str = "libopus"; /// Opus packets cap at 120 ms (RFC 6716 §2.1.4). const MAX_FRAME_MS: usize = 120; +/// The family 0 streams: one stream, coupled when stereo, decoded in order. +const MONO: &[u8] = &[0]; +const STEREO: &[u8] = &[0, 1]; + pub(super) struct Libopus { - inner: *mut OpusDecoder, + inner: *mut OpusMSDecoder, sample_rate: u32, layout: Layout, + /// For each canonical channel, the Vorbis-order channel it comes from, when they differ. + reorder: Option<&'static [usize]>, + /// Opus streams in one packet. Family 0 is one; the rest come from the mapping. + streams: u8, pre_skip: usize, max_frame_size: usize, in_dtx: bool, @@ -27,34 +38,67 @@ pub(super) struct Libopus { unsafe impl Send for Libopus {} impl Libopus { - /// Parses the OpusHead `description` if present; falls back to the catalog's - /// declared sample rate / channel count. + /// Parses the OpusHead `description` when one is present. A missing description + /// falls back to the catalog's sample rate and channel count, which must then + /// be mono or stereo. A description that does not parse is refused: guessing + /// family 0 would decode those packets with the wrong stream layout. + /// + /// Channel mapping family 1 decodes up to 7.1 in the canonical [`Layout`] + /// order; every other family is refused, since none declares speakers. pub(super) fn open(catalog: &hang::catalog::AudioConfig) -> Result, Error> { - let (sample_rate, channel_count, pre_skip) = if let Some(desc) = &catalog.description { - let mut buf = desc.as_ref(); - match moq_mux::codec::opus::Config::parse(&mut buf) { - Ok(head) => (head.sample_rate, head.channel_count, head.pre_skip), - Err(_) => (catalog.sample_rate, catalog.channel_count, 0), - } - } else { - (catalog.sample_rate, catalog.channel_count, 0) + let head = match catalog.description.as_ref() { + Some(desc) => moq_mux::codec::opus::Config::parse(&mut desc.as_ref()) + .map_err(|err| Error::Unsupported(format!("opus description: {err}")))?, + None => moq_mux::codec::opus::Config::new(catalog.sample_rate, catalog.channel_count), }; + let (sample_rate, channel_count, pre_skip) = (head.sample_rate, head.channel_count, head.pre_skip); opus::validate_rate(sample_rate)?; - let channels = opus::validate_channels(channel_count)?; - let layout = Layout::from_channels(channel_count)?; + let (streams, coupled, table, layout, reorder) = match &head.mapping { + None => { + let table = match opus::validate_channels(channel_count)? { + 1 => MONO, + _ => STEREO, + }; + let layout = Layout::from_channels(channel_count)?; + (1, table.len() as i32 - 1, table, layout, None) + } + Some(mapping) if mapping.family() == 1 => { + let (layout, reorder) = vorbis(channel_count)?; + let (streams, coupled) = (mapping.streams() as i32, mapping.coupled() as i32); + (streams, coupled, mapping.table(), layout, reorder) + } + Some(mapping) => { + return Err(Error::Unsupported(format!( + "opus channel mapping family {} declares no speaker positions", + mapping.family() + ))); + } + }; let mut err = 0i32; - // SAFETY: out-pointer is valid; inner is checked for null below. - let inner = unsafe { opus_decoder_create(sample_rate as i32, channels, &mut err) }; + // SAFETY: `table` holds one entry per output channel, the count we pass, + // and the out-pointer is valid; inner is checked for null below. + let inner = unsafe { + opus_multistream_decoder_create( + sample_rate as i32, + table.len() as i32, + streams, + coupled, + table.as_ptr(), + &mut err, + ) + }; if err != OPUS_OK || inner.is_null() { - return Err(opus::error(err, "opus_decoder_create")); + return Err(opus::error(err, "opus_multistream_decoder_create")); } Ok(Box::new(Self { inner, sample_rate, layout, + reorder, + streams: streams as u8, // OpusHead counts pre-skip at 48 kHz whatever rate the decoder runs at. pre_skip: (pre_skip as usize * sample_rate as usize) / 48_000, max_frame_size: (sample_rate as usize * MAX_FRAME_MS) / 1000, @@ -63,17 +107,47 @@ impl Libopus { } } +/// The layout a family 1 stream of `channels` declares (RFC 7845 §5.1.1.2), and +/// how to reorder its Vorbis channel order into that layout's canonical one. +/// +/// Vorbis puts the center between the fronts and the LFE last. Its "rear" +/// pair in 5.0 and 5.1 is the surround pair, which is side in the canonical +/// layouts, while 7.1 has distinct side and rear pairs. +fn vorbis(channels: u32) -> Result<(Layout, Option<&'static [usize]>), Error> { + Ok(match channels { + 1 => (Layout::Mono, None), + 2 => (Layout::Stereo, None), + // L, C, R. + 3 => (Layout::ThreePointZero, Some(&[0, 2, 1])), + // FL, FR, RL, RR. + 4 => (Layout::Quad, None), + // FL, C, FR, RL, RR. + 5 => (Layout::FivePointZero, Some(&[0, 2, 1, 3, 4])), + // FL, C, FR, RL, RR, LFE. + 6 => (Layout::FivePointOne, Some(&[0, 2, 1, 5, 3, 4])), + // FL, C, FR, SL, SR, RC, LFE. + 7 => (Layout::SixPointOne, Some(&[0, 2, 1, 6, 5, 3, 4])), + // FL, C, FR, SL, SR, RL, RR, LFE. + 8 => (Layout::SevenPointOne, Some(&[0, 2, 1, 7, 5, 6, 3, 4])), + other => { + return Err(Error::Unsupported(format!( + "opus channel mapping family 1 has no {other}-channel layout" + ))); + } + }) +} + impl Backend for Libopus { /// Empty packets invoke packet-loss concealment. Loss during DTX remains /// classified as DTX, while loss during active audio remains active. fn decode(&mut self, packet: &[u8]) -> Result { let channels = self.layout.channels() as usize; let mut out = vec![0.0f32; self.max_frame_size * channels]; - // SAFETY: `inner` owns a live OpusDecoder; packet/out slices are bounded by + // SAFETY: `inner` owns a live OpusMSDecoder; packet/out slices are bounded by // the lengths we pass. let samples = unsafe { - opus_decode_float( - &mut *self.inner, + opus_multistream_decode_float( + self.inner, packet.as_ptr(), packet.len() as i32, out.as_mut_ptr(), @@ -86,14 +160,24 @@ impl Backend for Libopus { } out.truncate(samples as usize * channels); - let activity = opus::activity(packet, self.in_dtx); + if let Some(order) = self.reorder { + let mut vorbis = [0.0f32; 8]; + for frame in out.chunks_exact_mut(channels) { + vorbis[..channels].copy_from_slice(frame); + for (sample, &from) in frame.iter_mut().zip(order) { + *sample = vorbis[from]; + } + } + } + + let activity = opus::multistream_activity(packet, self.streams, self.in_dtx); self.in_dtx = activity.is_dtx(); Ok(Decoded { samples: out, activity }) } fn reset(&mut self) -> Result<(), Error> { // SAFETY: `inner` owns a live decoder and OPUS_RESET_STATE takes no arguments. - let rc = unsafe { opus_decoder_ctl_impl(self.inner, OPUS_RESET_STATE, varargs![]) }; + let rc = unsafe { opus_multistream_decoder_ctl_impl(self.inner, OPUS_RESET_STATE, varargs![]) }; if rc != OPUS_OK { return Err(opus::error(rc, "OPUS_RESET_STATE")); } @@ -120,7 +204,214 @@ impl Backend for Libopus { impl Drop for Libopus { fn drop(&mut self) { - // SAFETY: `inner` is a live OpusDecoder that nothing else aliases. - unsafe { opus_decoder_destroy(self.inner) }; + // SAFETY: `inner` is a live OpusMSDecoder that nothing else aliases. + unsafe { opus_multistream_decoder_destroy(self.inner) }; + } +} + +#[cfg(test)] +mod tests { + use unsafe_libopus::{ + OPUS_APPLICATION_AUDIO, opus_multistream_encode_float, opus_multistream_encoder_destroy, + opus_multistream_surround_encoder_create, + }; + + use crate::Error; + use crate::decode::{Config, Decoder}; + use crate::layout::Speaker::{self, *}; + + /// Speakers in Vorbis channel order for each family 1 channel count (RFC 7845 §5.1.1.2). + const VORBIS: [&[Speaker]; 8] = [ + &[FrontCenter], + &[FrontLeft, FrontRight], + &[FrontLeft, FrontCenter, FrontRight], + &[FrontLeft, FrontRight, BackLeft, BackRight], + &[FrontLeft, FrontCenter, FrontRight, SideLeft, SideRight], + &[FrontLeft, FrontCenter, FrontRight, SideLeft, SideRight, Lfe], + &[FrontLeft, FrontCenter, FrontRight, SideLeft, SideRight, BackCenter, Lfe], + &[ + FrontLeft, + FrontCenter, + FrontRight, + SideLeft, + SideRight, + BackLeft, + BackRight, + Lfe, + ], + ]; + + const RATE: usize = 48_000; + const FRAME: usize = 960; + const PACKETS: usize = 15; + + /// A distinct tone per speaker, low for the LFE, which libopus band-limits. + fn tone(speaker: Speaker) -> f32 { + match speaker { + Lfe => 80.0, + other => 400.0 + 300.0 * other as u8 as f32, + } + } + + /// Encode `channels` of Vorbis-ordered tones as a family 1 stream, returning + /// its OpusHead and packets. + fn surround(channels: usize) -> (bytes::Bytes, Vec>) { + let speakers = VORBIS[channels - 1]; + let (mut streams, mut coupled, mut mapping) = (0i32, 0i32, [0u8; 8]); + let mut err = 0i32; + // SAFETY: every out-pointer is valid and `mapping` holds `channels` entries. + let encoder = unsafe { + opus_multistream_surround_encoder_create( + RATE as i32, + channels as i32, + 1, + &mut streams, + &mut coupled, + mapping.as_mut_ptr(), + OPUS_APPLICATION_AUDIO, + &mut err, + ) + }; + assert!(err == 0 && !encoder.is_null(), "encoder create failed: {err}"); + + let packets = (0..PACKETS) + .map(|packet| { + let mut pcm = Vec::with_capacity(FRAME * channels); + for i in packet * FRAME..(packet + 1) * FRAME { + for &speaker in speakers { + let phase = std::f32::consts::TAU * tone(speaker) * i as f32 / RATE as f32; + pcm.push(phase.sin() * 0.5); + } + } + let mut out = vec![0u8; 4000]; + // SAFETY: `encoder` is live, `pcm` holds FRAME frames, and `out` is as long as we say. + let len = unsafe { + opus_multistream_encode_float(encoder, pcm.as_ptr(), FRAME as i32, out.as_mut_ptr(), 4000) + }; + assert!(len > 0, "encode failed: {len}"); + out.truncate(len as usize); + out + }) + .collect(); + // SAFETY: `encoder` is live and not used again. + unsafe { opus_multistream_encoder_destroy(encoder) }; + + let mut head = moq_mux::codec::opus::Config::new(RATE as u32, 2) + .encode() + .unwrap() + .to_vec(); + head[9] = channels as u8; + head[18] = 1; + head.extend_from_slice(&[streams as u8, coupled as u8]); + head.extend_from_slice(&mapping[..channels]); + (head.into(), packets) + } + + fn catalog(head: bytes::Bytes, channels: u32) -> hang::catalog::AudioConfig { + let mut catalog = hang::catalog::AudioConfig::new(hang::catalog::AudioCodec::Opus, 48_000, channels); + catalog.description = Some(head); + catalog + } + + /// Energy of `samples` at `freq`, by the Goertzel recurrence. + fn power(samples: &[f32], freq: f32) -> f32 { + let coeff = 2.0 * (std::f32::consts::TAU * freq / RATE as f32).cos(); + let (mut s1, mut s2) = (0.0f32, 0.0f32); + for &x in samples { + let s = x + coeff * s1 - s2; + s2 = s1; + s1 = s; + } + s1 * s1 + s2 * s2 - coeff * s1 * s2 + } + + /// Every family 1 layout decodes with each canonical channel carrying its + /// own speaker's tone, which pins the Vorbis reorder independently of it. + #[test] + fn family_one_decodes_in_canonical_order() { + for channels in 1..=8 { + let (head, packets) = surround(channels); + let mut decoder = Decoder::new(&catalog(head, channels as u32), &Config::default()).unwrap(); + let layout = decoder.layout(); + // The layout has exactly the Vorbis speakers, which canonical order sorts. + let mut speakers = VORBIS[channels - 1].to_vec(); + speakers.sort_by_key(|&speaker| speaker as u8); + assert_eq!(layout.speakers().unwrap(), speakers, "{channels} channels"); + + let mut pcm = Vec::new(); + for packet in &packets { + pcm.extend(decoder.decode(packet).unwrap().samples); + } + // Skip the encoder's warmup. + let pcm = &pcm[pcm.len() / 2..]; + + let tones: Vec = VORBIS[channels - 1].iter().copied().map(tone).collect(); + for (index, &speaker) in layout.speakers().unwrap().iter().enumerate() { + let channel: Vec = pcm.iter().skip(index).step_by(channels).copied().collect(); + let loudest = tones + .iter() + .copied() + .max_by(|a, b| power(&channel, *a).total_cmp(&power(&channel, *b))) + .unwrap(); + assert_eq!(loudest, tone(speaker), "{channels} channels, {speaker:?} at {index}"); + } + } + } + + /// An all-DTX surround packet is one empty Opus packet per stream. Read as a + /// single stream, the later subpackets look like payload and the span is lost. + #[test] + fn surround_silence_is_dtx() { + let (head, packets) = surround(6); + let mut decoder = Decoder::new(&catalog(head.clone(), 6), &Config::default()).unwrap(); + let mid = decoder.decode(&packets[PACKETS / 2]).unwrap(); + assert!(mid.activity.is_active(), "a coded surround frame must stay active"); + + // Two coupled streams, then two mono, each a 20 ms empty frame. + let dtx = [0xfc, 0x00, 0xfc, 0x00, 0xf8, 0x00, 0xf8]; + let mut decoder = Decoder::new(&catalog(head, 6), &Config::default()).unwrap(); + let decoded = decoder.decode(&dtx).expect("empty multistream packet"); + assert!(decoded.activity.is_dtx(), "all-DTX surround packet read as active"); + assert_eq!(decoded.samples.len() % 6, 0); + } + + /// A description that is present but truncated used to be dropped, and a + /// stereo catalog then opened a family 0 decoder for a family 1 or 255 head. + #[test] + fn malformed_description_is_refused() { + for family in [1u8, 255] { + let mut head = moq_mux::codec::opus::Config::new(48_000, 2).encode().unwrap().to_vec(); + head[18] = family; + let err = Decoder::new(&catalog(head.into(), 2), &Config::default()) + .err() + .expect("refused"); + assert!( + matches!(&err, Error::Unsupported(message) if message.contains("opus description")), + "family {family}: {err}" + ); + } + + let plain = hang::catalog::AudioConfig::new(hang::catalog::AudioCodec::Opus, 48_000, 2); + assert!(Decoder::new(&plain, &Config::default()).is_ok()); + } + + /// Families other than 0 and 1 carry no speaker positions, so there is + /// nothing to put in a layout: refused, not passed through as discrete. + #[test] + fn other_families_are_refused() { + for (family, channels, table) in [(255u8, 2u8, &[2, 0, 0, 1][..]), (2, 4, &[4, 0, 0, 1, 2, 3])] { + let mut head = moq_mux::codec::opus::Config::new(48_000, 2).encode().unwrap().to_vec(); + head[9] = channels; + head[18] = family; + head.extend_from_slice(table); + + let err = Decoder::new(&catalog(head.into(), channels.into()), &Config::default()) + .err() + .expect("refused"); + assert!( + matches!(&err, Error::Unsupported(message) if message.contains("no speaker positions")), + "family {family}: {err}" + ); + } } } diff --git a/rs/moq-audio/src/opus.rs b/rs/moq-audio/src/opus.rs index d87a6055d3..0dd53c4edc 100644 --- a/rs/moq-audio/src/opus.rs +++ b/rs/moq-audio/src/opus.rs @@ -61,7 +61,7 @@ pub(crate) fn decode_error(code: i32) -> Error { if code == unsafe_libopus::OPUS_INVALID_PACKET { return Error::Decode(format!("libopus rejected the packet (code {code})")); } - error(code, "opus_decode_float") + error(code, "opus_multistream_decode_float") } /// Classify a packet, preserving DTX across packet loss. @@ -80,6 +80,171 @@ pub(crate) fn activity(packet: &[u8], in_dtx: bool) -> Activity { } } +/// Classify a multistream packet, preserving DTX across packet loss. +/// +/// Every stream but the last is self-delimited (RFC 6716 framing plus the length +/// libopus writes ahead of that stream's payload). One `opus_packet_parse` of the +/// whole buffer treats the later streams as payload, so an all-DTX surround +/// packet reads as active. DTX means every stream coded nothing. +pub(crate) fn multistream_activity(packet: &[u8], streams: u8, in_dtx: bool) -> Activity { + if packet.is_empty() { + return if in_dtx { Activity::Dtx } else { Activity::Active }; + } + if streams <= 1 { + return activity(packet, in_dtx); + } + if streams_carry_nothing(packet, streams) { + Activity::Dtx + } else { + Activity::Active + } +} + +/// Whether every stream in a multistream packet coded no audio. +fn streams_carry_nothing(mut packet: &[u8], streams: u8) -> bool { + for index in 0..streams { + let last = index + 1 == streams; + if last { + return carries_nothing(packet); + } + let Some((silent, offset)) = self_delimited(packet) else { + return false; + }; + if !silent { + return false; + } + packet = &packet[offset..]; + } + false +} + +/// One self-delimited Opus packet at the front of `data`: whether every frame is +/// empty, and how many bytes it occupies, including its trailing padding. +/// +/// `None` when the bytes are not that packet. The length of the last frame is +/// coded where libopus puts it, before the frame payloads. +fn self_delimited(data: &[u8]) -> Option<(bool, usize)> { + if data.is_empty() { + return None; + } + let framesize = unsafe { unsafe_libopus::opus_packet_get_samples_per_frame(data.as_ptr(), 48_000) }; + if framesize <= 0 { + return None; + } + + let toc = data[0]; + let mut pos = 1usize; + let mut len = data.len() - 1; + let mut last_size = len; + let mut cbr = false; + let mut pad = 0usize; + let mut sizes = [0i32; 48]; + + let count = match toc & 0x3 { + 0 => 1, + 1 => { + cbr = true; + 2 + } + 2 => { + let (bytes, size) = parse_size(data.get(pos..pos.checked_add(len)?)?)?; + len = len.checked_sub(bytes)?; + if size as usize > len { + return None; + } + sizes[0] = size; + pos += bytes; + last_size = len - size as usize; + 2 + } + _ => { + if len < 1 { + return None; + } + let ch = data[pos]; + pos += 1; + len -= 1; + let count = (ch & 0x3f) as usize; + if count == 0 || count > sizes.len() || framesize as usize * count > 5760 { + return None; + } + if ch & 0x40 != 0 { + loop { + if len == 0 { + return None; + } + let p = data[pos] as usize; + pos += 1; + len -= 1; + let tmp = if p == 255 { 254 } else { p }; + len = len.checked_sub(tmp)?; + pad += tmp; + if p != 255 { + break; + } + } + } + cbr = ch & 0x80 == 0; + if !cbr { + last_size = len; + for slot in &mut sizes[..count - 1] { + let (bytes, size) = parse_size(data.get(pos..pos.checked_add(len)?)?)?; + len = len.checked_sub(bytes)?; + if size as usize > len { + return None; + } + *slot = size; + pos += bytes; + last_size = last_size.checked_sub(bytes + size as usize)?; + } + } + count + } + }; + + let (bytes, size) = parse_size(data.get(pos..pos.checked_add(len)?)?)?; + len = len.checked_sub(bytes)?; + if size as usize > len { + return None; + } + sizes[count - 1] = size; + pos += bytes; + if cbr { + if size as usize * count > len { + return None; + } + for slot in &mut sizes[..count - 1] { + *slot = size; + } + } else if bytes + size as usize > last_size { + return None; + } + + for &frame in &sizes[..count] { + pos = pos.checked_add(frame as usize)?; + if pos > data.len() { + return None; + } + } + let offset = pad.checked_add(pos)?; + if offset == 0 || offset > data.len() { + return None; + } + let silent = sizes[..count].iter().all(|&frame| frame == 0); + Some((silent, offset)) +} + +/// The self-delimited length prefix (RFC 6716 §3.2.5), or `None` when it is cut off. +fn parse_size(data: &[u8]) -> Option<(usize, i32)> { + let first = *data.first()?; + if first < 252 { + Some((1, i32::from(first))) + } else { + let second = *data.get(1)?; + Some((2, 4 * i32::from(second) + i32::from(first))) + } +} + /// Whether the sender coded no audio at all for this packet: a bare TOC whose /// every frame is empty. /// @@ -168,6 +333,38 @@ mod tests { assert_eq!(activity(&[], false), Activity::Active); } + /// A surround packet is one self-delimited Opus packet per stream, then a + /// normal packet for the last. DTX is all of them empty; one coded frame is + /// audio. Parsing the buffer as a single stream would read the later + /// subpackets as that frame's payload. + #[test] + fn multistream_activity_requires_every_stream_empty() { + // 5.1: two coupled streams, then two mono. Code 0, empty frame. + let dtx = [0xfc, 0x00, 0xfc, 0x00, 0xf8, 0x00, 0xf8]; + assert_eq!(multistream_activity(&dtx, 4, false), Activity::Dtx); + // A coded frame on the last stream. + let last = [0xfc, 0x00, 0xfc, 0x00, 0xf8, 0x00, 0xf8, 0xaa]; + assert_eq!(multistream_activity(&last, 4, false), Activity::Active); + // A coded frame on an earlier stream. The length byte is the payload size. + let early = [0xfc, 0x01, 0xaa, 0xfc, 0x00, 0xf8, 0x00, 0xf8]; + assert_eq!(multistream_activity(&early, 4, false), Activity::Active); + // Two empty frames in one self-delimited code-1 packet, then a bare TOC. + let cbr = [0xf9, 0x00, 0xf8]; + assert_eq!(multistream_activity(&cbr, 2, false), Activity::Dtx); + let cbr_payload = [0xf9, 0x01, 0xaa, 0xbb, 0xf8]; + assert_eq!(multistream_activity(&cbr_payload, 2, false), Activity::Active); + // Code 3 packs the 60 ms case: three empty frames, self-delimited, then a bare TOC. + let code3 = [0xfb, 0x03, 0x00, 0xf8]; + assert_eq!(multistream_activity(&code3, 2, false), Activity::Dtx); + let code3_payload = [0xfb, 0x03, 0x01, 0xaa, 0xbb, 0xcc, 0xf8]; + assert_eq!(multistream_activity(&code3_payload, 2, false), Activity::Active); + // Loss still carries the previous classification, whatever the layout. + assert_eq!(multistream_activity(&[], 4, true), Activity::Dtx); + assert_eq!(multistream_activity(&[], 4, false), Activity::Active); + // One stream is an ordinary packet. + assert_eq!(multistream_activity(&[0xf8], 1, false), Activity::Dtx); + } + /// A silence run's periodic refresh is an ordinarily coded frame, and a /// speech onset can leave an earlier frame of the same packet empty. Neither /// is distinguishable by framing, so both read active rather than risking diff --git a/rs/moq-mux/src/catalog/msf/consumer.rs b/rs/moq-mux/src/catalog/msf/consumer.rs index 6df191bfff..bbdeb41bef 100644 --- a/rs/moq-mux/src/catalog/msf/consumer.rs +++ b/rs/moq-mux/src/catalog/msf/consumer.rs @@ -615,7 +615,8 @@ mod test { head.push(6); // channel_count (5.1) head.extend_from_slice(&0u16.to_le_bytes()); // pre_skip head.extend_from_slice(&24_000u32.to_le_bytes()); // sample_rate - head.extend_from_slice(&[0, 0, 0]); // output gain (i16) + channel mapping family (1 byte) + head.extend_from_slice(&[0, 0, 1]); // output gain (i16) + channel mapping family 1 + head.extend_from_slice(&[4, 2, 0, 4, 1, 2, 3, 5]); // streams, coupled, Vorbis 5.1 mapping let init_b64 = base64::engine::general_purpose::STANDARD.encode(&head); let mut track = audio_track("audio0", moq_msf::Packaging::Loc); diff --git a/rs/moq-mux/src/codec/opus/import.rs b/rs/moq-mux/src/codec/opus/import.rs index d32c33ae58..69b4631c86 100644 --- a/rs/moq-mux/src/codec/opus/import.rs +++ b/rs/moq-mux/src/codec/opus/import.rs @@ -114,7 +114,10 @@ impl Import { /// Build a catalog config from an OpusHead. Errors on a malformed or empty buffer. pub fn config(init: &[u8]) -> crate::Result { let mut buf = init; - Ok(Config::parse(&mut buf)?.into()) + let mut config: hang::catalog::AudioConfig = Config::parse(&mut buf)?.into(); + // Publish the head as given: re-encoding it would drop a channel mapping table. + config.description = Some(bytes::Bytes::copy_from_slice(init)); + Ok(config) } impl From for hang::catalog::AudioConfig { diff --git a/rs/moq-mux/src/codec/opus/mod.rs b/rs/moq-mux/src/codec/opus/mod.rs index bd1bba4a9b..a73e9227a6 100644 --- a/rs/moq-mux/src/codec/opus/mod.rs +++ b/rs/moq-mux/src/codec/opus/mod.rs @@ -23,10 +23,23 @@ pub enum Error { #[error("invalid OpusHead signature")] InvalidSignature, - /// [`Config::encode`] was asked to emit an OpusHead for a channel count other - /// than mono or stereo; channel mapping family 0 only covers 1 or 2 channels. - #[error("channel mapping family 0 only supports mono/stereo (got {0} channels)")] + /// The channel count is zero, or more than the channel mapping family allows: + /// family 0 covers mono/stereo and family 1 up to eight channels. + #[error("channel mapping family does not allow {0} channels")] UnsupportedChannelCount(u32), + + /// A nonzero channel mapping family without its complete mapping table. + #[error("OpusHead channel mapping table is truncated")] + MappingTableTooShort, + + /// The channel mapping table names no streams, more coupled streams than + /// streams, or a channel index past the decoded streams. + #[error("invalid OpusHead channel mapping table")] + InvalidMappingTable, + + /// [`Config::encode`] only emits channel mapping family 0. + #[error("cannot encode channel mapping family {0}")] + UnsupportedMappingFamily(u8), } pub type Result = std::result::Result; @@ -37,10 +50,85 @@ pub type Result = std::result::Result; pub struct Config { /// Original input sample rate in Hz. pub sample_rate: u32, - /// Number of encoded channels. + /// Number of output channels. pub channel_count: u32, /// Number of decoded 48 kHz samples to discard at stream start. pub pre_skip: u16, + /// The channel mapping table, or `None` for family 0 (mono/stereo, one stream). + pub mapping: Option, +} + +/// An OpusHead channel mapping table (RFC 7845 §5.1.1), present for every family but 0. +#[derive(Clone, Copy, PartialEq, Eq)] +pub struct Mapping { + family: u8, + streams: u8, + coupled: u8, + channels: u8, + // Sized for the most channels any family allows, so `Config` stays `Copy`. + table: [u8; 255], +} + +impl Mapping { + fn parse(buf: &mut T, family: u8, channels: u8) -> Result { + if buf.remaining() < 2 + channels as usize { + return Err(Error::MappingTableTooShort); + } + let streams = buf.get_u8(); + let coupled = buf.get_u8(); + if streams == 0 || coupled > streams || streams as u32 + coupled as u32 > 255 { + return Err(Error::InvalidMappingTable); + } + + let mut table = [0u8; 255]; + for entry in &mut table[..channels as usize] { + *entry = buf.get_u8(); + // 255 marks a silent channel. + if *entry != 255 && *entry >= streams + coupled { + return Err(Error::InvalidMappingTable); + } + } + + Ok(Self { + family, + streams, + coupled, + channels, + table, + }) + } + + /// The channel mapping family: 1 is the Vorbis speaker order, 2 and 3 are + /// ambisonics, and 255 is channels with no declared position. + pub fn family(&self) -> u8 { + self.family + } + + /// Number of Opus streams in each packet. + pub fn streams(&self) -> u8 { + self.streams + } + + /// How many of those streams are coupled (stereo); they come first. + pub fn coupled(&self) -> u8 { + self.coupled + } + + /// The decoded channel feeding each output channel, 255 for silence. + pub fn table(&self) -> &[u8] { + &self.table[..self.channels as usize] + } +} + +impl std::fmt::Debug for Mapping { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Mapping") + .field("family", &self.family) + .field("streams", &self.streams) + .field("coupled", &self.coupled) + .field("table", &self.table()) + .finish() + } } impl Config { @@ -50,6 +138,7 @@ impl Config { sample_rate, channel_count, pre_skip: 0, + mapping: None, } } @@ -61,8 +150,9 @@ impl Config { /// Parse an OpusHead buffer (RFC 7845 §5.1). /// - /// Verifies the magic signature; reads channel count, pre-skip, and sample - /// rate; ignores gain and channel mapping. Any trailing bytes are consumed. + /// Verifies the magic signature; reads channel count, pre-skip, sample rate, + /// and the channel mapping, refusing a channel count the family forbids or an + /// inconsistent mapping table; ignores gain. Any trailing bytes are consumed. pub fn parse(buf: &mut T) -> Result { if buf.remaining() < 19 { return Err(Error::HeadTooShort); @@ -76,8 +166,23 @@ impl Config { let channel_count = buf.get_u8() as u32; let pre_skip = buf.get_u16_le(); let sample_rate = buf.get_u32_le(); + buf.advance(2); // Skip gain until if/when we support it. + let family = buf.get_u8(); + + let max_channels = match family { + 0 => 2, + 1 => 8, + _ => 255, + }; + if channel_count == 0 || channel_count > max_channels { + return Err(Error::UnsupportedChannelCount(channel_count)); + } + + let mapping = match family { + 0 => None, + family => Some(Mapping::parse(buf, family, channel_count as u8)?), + }; - // Skip gain, channel mapping until if/when we support them. if buf.remaining() > 0 { buf.advance(buf.remaining()); } @@ -86,6 +191,7 @@ impl Config { sample_rate, channel_count, pre_skip, + mapping, }) } @@ -95,8 +201,11 @@ impl Config { /// Errors with [`Error::UnsupportedChannelCount`] unless `channel_count` is 1 /// or 2, since mapping family 0 is only defined for mono/stereo per RFC 7845 §5.1. /// Multi-channel streams need family 1 with a channel mapping table, which - /// this helper does not emit. + /// this helper does not emit, so any `mapping` is [`Error::UnsupportedMappingFamily`]. pub fn encode(&self) -> Result { + if let Some(mapping) = &self.mapping { + return Err(Error::UnsupportedMappingFamily(mapping.family)); + } if !(1..=2).contains(&self.channel_count) { return Err(Error::UnsupportedChannelCount(self.channel_count)); } @@ -190,4 +299,85 @@ mod tests { let err = Config::new(48_000, 6).encode().unwrap_err(); assert!(matches!(err, Error::UnsupportedChannelCount(6))); } + + /// A 19-byte head with the given channel count and mapping family. + fn head(channels: u8, family: u8) -> Vec { + let mut head = Config::new(48_000, 2).encode().unwrap().to_vec(); + head[9] = channels; + head[18] = family; + head + } + + #[test] + fn parses_a_mapping_table() { + // 5.1 in family 1: four streams, two coupled, Vorbis order. + let mut bytes = head(6, 1); + bytes.extend_from_slice(&[4, 2, 0, 4, 1, 2, 3, 5]); + let parsed = Config::parse(&mut bytes.as_slice()).unwrap(); + assert_eq!(parsed.channel_count, 6); + + let mapping = parsed.mapping.unwrap(); + assert_eq!(mapping.family(), 1); + assert_eq!(mapping.streams(), 4); + assert_eq!(mapping.coupled(), 2); + assert_eq!(mapping.table(), &[0, 4, 1, 2, 3, 5]); + + // Encode only emits family 0. + assert!(matches!(parsed.encode(), Err(Error::UnsupportedMappingFamily(1)))); + } + + #[test] + fn parse_rejects_channel_counts_the_family_does_not_allow() { + for (channels, family) in [(0, 0), (3, 0), (0, 1), (9, 1), (0, 255)] { + let mut bytes = head(channels, family); + bytes.extend_from_slice(&[1, 0]); + bytes.extend(std::iter::repeat_n(0, channels as usize)); + assert!( + matches!( + Config::parse(&mut bytes.as_slice()), + Err(Error::UnsupportedChannelCount(_)) + ), + "{channels} channels in family {family}" + ); + } + } + + #[test] + fn parse_rejects_a_bad_mapping_table() { + // Family 1 promising a table that is not there, or only half of it. + assert!(matches!( + Config::parse(&mut head(2, 1).as_slice()), + Err(Error::MappingTableTooShort) + )); + let mut short = head(2, 1); + short.extend_from_slice(&[1, 1, 0]); + assert!(matches!( + Config::parse(&mut short.as_slice()), + Err(Error::MappingTableTooShort) + )); + + for table in [ + // No streams. + [0, 0, 0, 1], + // More coupled streams than streams. + [1, 2, 0, 1], + // A channel index past the two decoded channels. + [1, 1, 0, 2], + ] { + let mut bytes = head(2, 1); + bytes.extend_from_slice(&table); + assert!( + matches!(Config::parse(&mut bytes.as_slice()), Err(Error::InvalidMappingTable)), + "{table:?}" + ); + } + + // 255 is a silent channel, not an index. + let mut silent = head(2, 1); + silent.extend_from_slice(&[1, 0, 0, 255]); + assert_eq!( + Config::parse(&mut silent.as_slice()).unwrap().mapping.unwrap().table(), + &[0, 255] + ); + } } From 42582a8cc75c01ec433a2486c7b645118e7cdec4 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 26 Sep 2026 18:10:45 -0700 Subject: [PATCH 10/11] fix(mux): ADTS export refuses what it cannot label (#4283) Co-authored-by: Claude Opus 5.5 --- doc/bin/cli.md | 5 + quest/m1/audio-codecs/README.md | 4 +- quest/m1/audio-codecs/aac-encode-refusals.md | 24 +++ .../audio-codecs/aac-parse-truncated-sbr.md | 21 +++ quest/m1/audio-codecs/adts-refusals.md | 25 --- .../m1/audio-codecs/he-aac-catalog-output.md | 29 +++ rs/moq-mux/src/codec/aac/mod.rs | 168 ++++++++++++------ rs/moq-mux/src/container/ts/adts.rs | 38 +++- rs/moq-mux/src/container/ts/export.rs | 52 +++--- rs/moq-mux/src/container/ts/export_test.rs | 85 +++++++++ .../src/container/ts/test_data/he_aac.flv | Bin 0 -> 3110 bytes .../src/container/ts/test_data/he_aac_v2.flv | Bin 0 -> 3297 bytes 12 files changed, 336 insertions(+), 115 deletions(-) create mode 100644 quest/m1/audio-codecs/aac-encode-refusals.md create mode 100644 quest/m1/audio-codecs/aac-parse-truncated-sbr.md delete mode 100644 quest/m1/audio-codecs/adts-refusals.md create mode 100644 quest/m1/audio-codecs/he-aac-catalog-output.md create mode 100644 rs/moq-mux/src/container/ts/test_data/he_aac.flv create mode 100644 rs/moq-mux/src/container/ts/test_data/he_aac_v2.flv diff --git a/doc/bin/cli.md b/doc/bin/cli.md index 723b80a3e4..76b01f12a7 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -66,6 +66,11 @@ discarding the old mux buffer. The first new clock packet signals the break and stdout pacing re-anchors. Every rendition joins the new program generation; no track is fenced across the marker. +MPEG-TS export frames AAC as ADTS, which labels only the AAC Main, LC, SSR, +and LTP profiles. HE-AAC and HE-AACv2 go out as their AAC-LC core, and decoders +find the SBR and PS in band, as ffmpeg's ADTS output does. A track whose +profile or channel layout ADTS cannot label is refused rather than mislabeled. + A constant-rate MPEG-TS source records its multiplex rate in the catalog (`mpegts.muxRate`, measured off the PCR clock, null stuffing included), and `export ts` pads its output with null packets back to that rate so an IRD or diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index 71b47defcc..762769dc5a 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -44,9 +44,11 @@ its own decode and encode quest so verification stays per host. ## Quests - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers -- [ADTS refusals](/quest/m1/audio-codecs/adts-refusals.md) - the ADTS writer refuses channel counts and object types it cannot label instead of mislabeling them +- [HE-AAC catalog output](/quest/m1/audio-codecs/he-aac-catalog-output.md) - HE-AAC catalog entries name the output rate and layout, not the LC core - [AudioToolbox encode](/quest/m1/audio-codecs/encode-audiotoolbox.md) - macOS and iOS encode AAC-LC - [TS surround Opus](/quest/m1/audio-codecs/ts-opus-surround.md) - a 3 to 8 channel Opus stream from MPEG-TS imports with a family 1 OpusHead that decodes +- [AAC encode refusals](/quest/m1/audio-codecs/aac-encode-refusals.md) - `Config::encode` refuses channel counts it cannot name instead of writing stereo +- [AAC parse truncated SBR](/quest/m1/audio-codecs/aac-parse-truncated-sbr.md) - `Config::parse` refuses SBR and PS configs cut off before their core ## Related diff --git a/quest/m1/audio-codecs/aac-encode-refusals.md b/quest/m1/audio-codecs/aac-encode-refusals.md new file mode 100644 index 0000000000..9a891bb69f --- /dev/null +++ b/quest/m1/audio-codecs/aac-encode-refusals.md @@ -0,0 +1,24 @@ +# [S] AAC config encode refuses unnameable channel counts + +## Goal + +`moq_mux::codec::aac::Config::encode` never writes an AudioSpecificConfig that +names the wrong layout. A channel count no channelConfiguration names is +refused, not logged and written as stereo, the way the ADTS writer refuses it. + +## Plan + +- `encode` returns `Bytes` today, so refusing is a published API break: this + retargets to `dev`. Decide whether it returns a `Result` or whether a + constructor validates the count up front so encoding cannot fail. +- A count a program config element could describe (7, or more than 8) is + still refused unless the caller supplies a layout; guessing speaker + positions is what this removes. +- Callers in `moq-audio` (encoder config, description synthesis) already + validate first; check they keep their error messages. + +Public API: `Config::encode` (or its constructor) changes. Wire: none. + +## Related + +- [#4283](https://github.com/moq-dev/moq/pull/4283) - the ADTS export refusals this mirrors diff --git a/quest/m1/audio-codecs/aac-parse-truncated-sbr.md b/quest/m1/audio-codecs/aac-parse-truncated-sbr.md new file mode 100644 index 0000000000..57f9bf04e6 --- /dev/null +++ b/quest/m1/audio-codecs/aac-parse-truncated-sbr.md @@ -0,0 +1,21 @@ +# [XS] AAC config parse refuses truncated SBR and PS + +## Goal + +`Config::parse` refuses an AudioSpecificConfig that signals SBR or PS +(object type 5 or 29) but ends before its extension rate and core object type, +instead of accepting it as if the core were known. + +## Plan + +- The export path already requires these fields; `parse` stays lenient only + because `Config::encode` writes a two-byte config for those object types. + Once encode refuses or completes them, the leniency has no producer. +- Check the `moq-audio` tests that feed a two-byte HE-AAC config; they should + keep asserting a refusal, just a different one. + +Public API: none beyond a new error case. Wire: none. + +## Required + +- [AAC encode refusals](/quest/m1/audio-codecs/aac-encode-refusals.md) - encode stops producing truncated SBR and PS configs diff --git a/quest/m1/audio-codecs/adts-refusals.md b/quest/m1/audio-codecs/adts-refusals.md deleted file mode 100644 index c7896e377d..0000000000 --- a/quest/m1/audio-codecs/adts-refusals.md +++ /dev/null @@ -1,25 +0,0 @@ -# [S] ADTS export refuses what it cannot label - -## Goal - -`moq-mux`'s ADTS writer never labels an AAC track with the wrong layout or -object type. A channel count ADTS cannot name is refused, not silently written -as stereo, and an explicit SBR or PS description is written with the object -type ADTS can carry or refused, not masked to two bits into a wrong profile. - -## Plan - -- `channel_config_from_count` falls back to stereo for counts it cannot - represent; refuse instead, the same way #4178 made channelConfiguration 11 to - 14 refuse. -- The ADTS header masks the object type to two bits, so an explicit HE-AAC - (object type 5) or HE-AACv2 (29) description is mislabeled. Decide per case: - signal the backward-compatible AAC-LC core (implicit SBR) when the description - allows it, otherwise refuse. -- Tests for both with real fixtures, checked against ffprobe. - -Public API: none. Wire: none; TS output changes only for inputs it mislabeled. - -## Related - -- [#4178](https://github.com/moq-dev/moq/pull/4178) - the PCE export that found these diff --git a/quest/m1/audio-codecs/he-aac-catalog-output.md b/quest/m1/audio-codecs/he-aac-catalog-output.md new file mode 100644 index 0000000000..9032995d6e --- /dev/null +++ b/quest/m1/audio-codecs/he-aac-catalog-output.md @@ -0,0 +1,29 @@ +# [S] HE-AAC catalog entries describe the output + +## Goal + +A catalog entry for HE-AAC or HE-AACv2 names what the stream plays as, for +example 48 kHz stereo, not its AAC-LC core (24 kHz, and mono for v2). Players +and gateways that size buffers or pick a decoder from the catalog see the real +output. + +## Plan + +- `Config::parse` reports the core rate and channel count because those lead + an explicit SBR or PS AudioSpecificConfig. The FLV and MKV importers copy + them into the catalog; fMP4 takes the sample entry instead, so the importers + disagree for the same stream. +- Derive the output from the config: the extension rate under SBR, two + channels under PS. Implicit SBR, found only in band, cannot be known from the + config; keep the documented half-rate behavior for it. +- Consumers that rebuild a config from catalog fields (description synthesis in + `moq-audio`, the MSF path) must not then mistake the output rate for the core. +- Regression: the `fdkaacenc` HE-AAC and HE-AACv2 FLV fixtures under + `rs/moq-mux/src/container/ts/test_data` import as 48 kHz stereo, matching + ffprobe. + +Public API: possibly `Config` fields. Wire: catalog values change for HE-AAC. + +## Related + +- [#4283](https://github.com/moq-dev/moq/pull/4283) - found while labeling HE-AAC for ADTS diff --git a/rs/moq-mux/src/codec/aac/mod.rs b/rs/moq-mux/src/codec/aac/mod.rs index 5ccb24a110..ad46c1773b 100644 --- a/rs/moq-mux/src/codec/aac/mod.rs +++ b/rs/moq-mux/src/codec/aac/mod.rs @@ -73,7 +73,11 @@ impl Config { let mut reader = BitReader::new(buf); let (object_type, sample_rate, channel_config) = read_header(&mut reader)?; let channel_count = match channel_config { - 0 => program_config(&mut reader)?, + 0 => { + let core = read_core(&mut reader, object_type)?; + read_general_audio(&mut reader, core)?; + program_config(&mut reader)? + } _ => channel_count_from_config(channel_config)?, }; @@ -135,9 +139,8 @@ impl Config { } } -/// Read an AudioSpecificConfig up to its channels: the audioObjectType, sample rate, and -/// channelConfiguration. For a channelConfiguration of 0 the reader stops at the program config -/// element. +/// Read an AudioSpecificConfig's leading fields: the audioObjectType, sample rate, and +/// channelConfiguration. fn read_header(reader: &mut BitReader) -> Result<(u8, u32, u8)> { if reader.buf.remaining() < 2 { return Err(Error::ConfigTooShort); @@ -156,35 +159,40 @@ fn read_header(reader: &mut BitReader) -> Result<(u8, u32, u8)> { // channelConfiguration: 4 bits, immediately after the (possibly explicit) rate. let channel_config = reader.read(4, Error::IncompleteConfig)? as u8; - if channel_config != 0 { - return Ok((object_type, sample_rate, channel_config)); - } + Ok((object_type, sample_rate, channel_config)) +} - // Explicit SBR and PS name their core object type after an extension rate; the - // GASpecificConfig carrying the program config element follows that core type. - let mut core = object_type; - if matches!(object_type, 5 | 29) { - if reader.read(4, Error::IncompleteConfig)? == 15 { - reader.read(24, Error::IncompleteConfig)?; - } - core = read_object_type(reader)?; - if core == 22 { - // extensionChannelConfiguration, only for ER BSAC. - reader.read(4, Error::IncompleteConfig)?; - } +/// Read the core audioObjectType, which explicit SBR and PS name after an extension rate. Any +/// other object type is its own core, and the leading sample rate is always the core's. +fn read_core(reader: &mut BitReader, object_type: u8) -> Result { + if !matches!(object_type, 5 | 29) { + return Ok(object_type); } + if reader.read(4, Error::IncompleteConfig)? == 15 { + reader.read(24, Error::IncompleteConfig)?; + } + let core = read_object_type(reader)?; + if core == 22 { + // extensionChannelConfiguration, only for ER BSAC. + reader.read(4, Error::IncompleteConfig)?; + } + Ok(core) +} + +/// Read a GASpecificConfig up to the program config element that a channelConfiguration of 0 +/// puts next, refusing a `core` object type whose specific config is something else. +fn read_general_audio(reader: &mut BitReader, core: u8) -> Result<()> { if !GENERAL_AUDIO.contains(&core) { return Err(Error::ProgramConfigUnsupported(core)); } - // GASpecificConfig: frameLengthFlag, dependsOnCoreCoder (then a 14-bit - // coreCoderDelay), and extensionFlag precede the element. + // frameLengthFlag, dependsOnCoreCoder (then a 14-bit coreCoderDelay), and extensionFlag. reader.read(1, Error::IncompleteConfig)?; if reader.read(1, Error::IncompleteConfig)? == 1 { reader.read(14, Error::IncompleteConfig)?; } reader.read(1, Error::IncompleteConfig)?; - Ok((object_type, sample_rate, 0)) + Ok(()) } /// Build the AudioSpecificConfig for a stream that signals its fields per frame, as ADTS does. @@ -222,25 +230,46 @@ pub(crate) fn in_band_config(profile: u8, sample_rate: u32, channel_config: u8, Ok(Bytes::from(out.bytes)) } -/// Split an AudioSpecificConfig into the channel signaling of a stream that carries it per frame, -/// as ADTS does; the inverse of [`in_band_config`]. -/// -/// Returns the channelConfiguration and, when that is 0, the program config element framed to -/// lead a raw data block: its element ID, then the element aligned to the block. -pub(crate) fn in_band_channels(asc: &[u8]) -> Result<(u8, Option)> { +/// The fields a stream that signals its config per frame, as ADTS does, carries in each header. +#[derive(Clone, Debug, PartialEq)] +pub(crate) struct InBand { + /// The core audioObjectType. Explicit SBR and PS name it under their own, and a decoder finds + /// them again in band, as it would in a stream that never signaled them (implicit signaling). + pub object_type: u8, + /// The core's sample rate, which SBR doubles on output. + pub sample_rate: u32, + pub channel_config: u8, + /// When `channel_config` is 0, the program config element framed to lead a raw data block: + /// its element ID, then the element aligned to the block. + pub program_config: Option, +} + +/// Split an AudioSpecificConfig into what a stream that carries it per frame signals, as ADTS +/// does; the inverse of [`in_band_config`]. +pub(crate) fn in_band(asc: &[u8]) -> Result { let mut asc = asc; let mut reader = BitReader::new(&mut asc); - let (_, _, channel_config) = read_header(&mut reader)?; - if channel_config != 0 { - return Ok((channel_config, None)); - } + let (object_type, sample_rate, channel_config) = read_header(&mut reader)?; + let core = read_core(&mut reader, object_type)?; + + let program_config = match channel_config { + 0 => { + read_general_audio(&mut reader, core)?; + let mut out = BitWriter::default(); + out.write(3, ID_PCE); + reader.record = Some(out); + program_config(&mut reader)?; + Some(Bytes::from(reader.record.take().expect("recording set above").bytes)) + } + _ => None, + }; - let mut out = BitWriter::default(); - out.write(3, ID_PCE); - reader.record = Some(out); - program_config(&mut reader)?; - let out = reader.record.take().expect("recording set above"); - Ok((0, Some(Bytes::from(out.bytes)))) + Ok(InBand { + object_type: core, + sample_rate, + channel_config, + program_config, + }) } /// The raw data block element ID of a program config element (ISO 14496-3 Table 4.85). @@ -554,10 +583,9 @@ mod tests { assert_eq!(Config::parse(&mut asc.as_slice()).unwrap().channel_count, 7); } - #[test] - fn parses_program_config_element_behind_explicit_sbr() { - // audioObjectType 5 (SBR), 24 kHz core, channelConfiguration 0, a 48 kHz extension - // rate, then the core type (LC) whose GASpecificConfig carries the element. + /// audioObjectType 5 (SBR), 24 kHz core, channelConfiguration 0, a 48 kHz extension rate, + /// then the core type (LC) whose GASpecificConfig carries a front and back pair. + fn sbr_pce_asc() -> Vec { let mut out = BitWriter::default(); out.write(5, 5); out.write(4, 6); @@ -566,7 +594,12 @@ mod tests { out.write(5, 2); out.write(3, 0); write_pce(&mut out, &[true], &[], &[true], 0); - let cfg = Config::parse(&mut out.bytes.as_slice()).unwrap(); + out.bytes + } + + #[test] + fn parses_program_config_element_behind_explicit_sbr() { + let cfg = Config::parse(&mut sbr_pce_asc().as_slice()).unwrap(); assert_eq!(cfg.profile, 5); assert_eq!(cfg.channel_count, 4); } @@ -638,16 +671,18 @@ mod tests { assert_eq!(rest, [0xFF], "the element moves out of the block"); // And back: the element leads the block again, byte for byte. - assert_eq!(in_band_channels(&asc).unwrap(), (0, Some(Bytes::from(pce)))); + let in_band = in_band(&asc).unwrap(); + assert_eq!(in_band.channel_config, 0); + assert_eq!(in_band.program_config, Some(Bytes::from(pce))); } #[test] - fn in_band_channels_of_ffmpeg_program_config_element() { + fn in_band_of_ffmpeg_program_config_element() { // Round trip ffmpeg's own element through a raw data block and back into a config. The // trailing SBR sync extension is not part of the element, so it drops. - let (config, pce) = in_band_channels(&FFMPEG_QUAD_ASC).unwrap(); - assert_eq!(config, 0); - let pce = pce.unwrap(); + let in_band = in_band(&FFMPEG_QUAD_ASC).unwrap(); + assert_eq!(in_band.channel_config, 0); + let pce = in_band.program_config.unwrap(); let mut block = pce.as_ref(); let asc = in_band_config(2, 48_000, 0, &mut block).unwrap(); assert!(block.is_empty(), "the element is all that was framed"); @@ -655,9 +690,40 @@ mod tests { } #[test] - fn in_band_channels_without_a_program_config_element() { - assert_eq!(in_band_channels(&[0x11, 0x90]).unwrap(), (2, None)); - assert_eq!(in_band_channels(&[0x11, 0xE0]).unwrap(), (12, None)); + fn in_band_without_a_program_config_element() { + let stereo = in_band(&[0x11, 0x90]).unwrap(); + assert_eq!((stereo.channel_config, stereo.program_config), (2, None)); + assert_eq!(in_band(&[0x11, 0xE0]).unwrap().channel_config, 12); + } + + #[test] + fn in_band_of_explicit_sbr_is_its_lc_core() { + // GStreamer 1.28 `fdkaacenc` at 48 kHz stereo: HE-AAC is SBR over a 24 kHz stereo LC core, + // and HE-AACv2 is PS over a mono one. + for (asc, channel_config) in [([0x2B, 0x11, 0x88, 0x00], 2), ([0xEB, 0x09, 0x88, 0x00], 1)] { + let expected = InBand { + object_type: 2, + sample_rate: 24_000, + channel_config, + program_config: None, + }; + assert_eq!(in_band(&asc).unwrap(), expected); + } + + // A config naming SBR but stopping before its core has no core to name. + assert!(matches!(in_band(&[0x2A, 0x10]), Err(Error::IncompleteConfig))); + } + + #[test] + fn in_band_of_explicit_sbr_keeps_its_program_config_element() { + let in_band = in_band(&sbr_pce_asc()).unwrap(); + assert_eq!(in_band.object_type, 2); + assert_eq!(in_band.sample_rate, 24_000); + assert_eq!(in_band.channel_config, 0); + let pce = in_band.program_config.expect("a program config element"); + let mut block = pce.as_ref(); + let asc = in_band_config(2, 24_000, 0, &mut block).unwrap(); + assert_eq!(Config::parse(&mut asc.as_ref()).unwrap().channel_count, 4); } #[test] diff --git a/rs/moq-mux/src/container/ts/adts.rs b/rs/moq-mux/src/container/ts/adts.rs index ac1aec6823..632947bdce 100644 --- a/rs/moq-mux/src/container/ts/adts.rs +++ b/rs/moq-mux/src/container/ts/adts.rs @@ -66,8 +66,12 @@ pub(super) fn write_header( channel_config: u8, raw_len: usize, ) -> anyhow::Result<[u8; 7]> { - // ADTS `profile` is the 2-bit audioObjectType - 1. - let profile = object_type.saturating_sub(1) & 0x03; + // ADTS `profile` is the 2-bit audioObjectType - 1, so only Main, LC, SSR, and LTP fit. + anyhow::ensure!( + (1..=4).contains(&object_type), + "audioObjectType {object_type} not representable in ADTS" + ); + let profile = object_type - 1; let freq_index = freq_index_from_rate(sample_rate)?; // ADTS has 3 bits for it; the higher configurations only fit an AudioSpecificConfig. anyhow::ensure!( @@ -101,12 +105,13 @@ fn freq_index_from_rate(sample_rate: u32) -> anyhow::Result { .with_context(|| format!("sample rate {sample_rate} not representable in ADTS")) } -/// Map a channel count to an AAC `channel_config` (ISO 14496-3 Table 1.19). -pub(super) fn channel_config_from_count(channel_count: u32) -> u8 { +/// Map a channel count to the ADTS `channel_config` (ISO 14496-3 Table 1.19) naming it, refusing a +/// count that none does rather than guess a layout. +pub(super) fn channel_config_from_count(channel_count: u32) -> anyhow::Result { match channel_count { - 1..=6 => channel_count as u8, - 8 => 7, - _ => 2, + 1..=6 => Ok(channel_count as u8), + 8 => Ok(7), + _ => anyhow::bail!("{channel_count} channels have no ADTS channelConfiguration"), } } @@ -134,6 +139,25 @@ mod tests { assert!(Header::parse(&header).is_err()); } + #[test] + fn write_refuses_object_types_outside_the_profile_field() { + for object_type in [0, 5, 29] { + assert!(write_header(object_type, 48_000, 2, 10).is_err()); + } + for object_type in 1..=4 { + let header = write_header(object_type, 48_000, 2, 10).unwrap(); + assert_eq!(Header::parse(&header).unwrap().object_type, object_type); + } + } + + #[test] + fn channel_counts_without_a_configuration_are_refused() { + assert_eq!(channel_config_from_count(8).unwrap(), 7); + for count in [0, 7, 9, 24] { + assert!(channel_config_from_count(count).is_err()); + } + } + #[test] fn frame_len_for_5_1() { let header = write_header(2, 44_100, 6, 512).unwrap(); diff --git a/rs/moq-mux/src/container/ts/export.rs b/rs/moq-mux/src/container/ts/export.rs index c53a7c141f..07ac0bc49f 100644 --- a/rs/moq-mux/src/container/ts/export.rs +++ b/rs/moq-mux/src/container/ts/export.rs @@ -37,7 +37,7 @@ use moq_net::Timestamp; use crate::catalog::hang::Catalog; use crate::catalog::{CatalogFormat, Stream}; -use crate::codec::annexb; +use crate::codec::{aac, annexb}; use crate::container::{ExportSource, Frame}; use super::adts; @@ -257,12 +257,7 @@ enum Kind { /// AAC, framed as ADTS. A `channel_config` of 0 defers the layout to a program config /// element, which leads the next raw data block written and is then taken. A catalog update /// rebuilds the kind and so repeats it once, which a decoder tuning in mid-stream welcomes. - Aac { - object_type: u8, - sample_rate: u32, - channel_config: u8, - program_config: Option, - }, + Aac(aac::InBand), /// Opus (private stream_type 0x06). Each frame is one Opus packet, prefixed with /// the Opus-in-TS access-unit control header and announced with the 'Opus' /// registration plus DVB extension descriptor. @@ -1185,7 +1180,7 @@ impl Export { tracks.iter().find(|t| { matches!( t.kind, - Kind::Aac { .. } | Kind::Opus { .. } | Kind::Mp2 { .. } | Kind::Ac3 | Kind::Eac3 + Kind::Aac(_) | Kind::Opus { .. } | Kind::Mp2 { .. } | Kind::Ac3 | Kind::Eac3 ) }) }) @@ -1197,7 +1192,7 @@ impl Export { .map(|t| { let stream_type = match &t.kind { Kind::Video(stream_type) => *stream_type, - Kind::Aac { .. } => StreamType::AdtsAac, + Kind::Aac(_) => StreamType::AdtsAac, // Opus rides private-data PES; the registration + extension descriptors // below tell the demuxer it's Opus. Kind::Opus { .. } => StreamType::from_u8(0x06).map_err(anyhow::Error::msg)?, @@ -1348,8 +1343,8 @@ impl Export { let track = self.tracks.get_mut(name).context("missing track")?; let pid = track.pid; let kind = track.kind.clone(); - if let Kind::Aac { program_config, .. } = &mut track.kind { - program_config.take(); + if let Kind::Aac(aac) = &mut track.kind { + aac.program_config.take(); } let is_video = matches!(kind, Kind::Video(_)); let timestamp = frame.timestamp; @@ -1360,15 +1355,10 @@ impl Export { // verbatim streams carry no PES payload; the section is written separately below. let es_payload = match &kind { Kind::Video(stream_type) => Some(video_es_payload(*stream_type, track.source.description(), &frame)?), - Kind::Aac { - object_type, - sample_rate, - channel_config, - program_config, - } => { - let pce = program_config.as_deref().unwrap_or_default(); + Kind::Aac(aac) => { + let pce = aac.program_config.as_deref().unwrap_or_default(); let raw_len = pce.len() + frame.payload.len(); - let header = adts::write_header(*object_type, *sample_rate, *channel_config, raw_len)?; + let header = adts::write_header(aac.object_type, aac.sample_rate, aac.channel_config, raw_len)?; let mut framed = Vec::with_capacity(header.len() + raw_len); framed.extend_from_slice(&header); framed.extend_from_slice(pce); @@ -2097,18 +2087,18 @@ fn video_es_payload(stream_type: StreamType, description: Option<&Bytes>, frame: fn audio_kind(config: &AudioConfig, name: &str) -> anyhow::Result { ensure_raw(&config.container, "audio", name)?; match &config.codec { - AudioCodec::AAC(aac) => { - // The description names the layout exactly; without one, the count is all there is. - let (channel_config, program_config) = match &config.description { - Some(asc) => crate::codec::aac::in_band_channels(asc)?, - None => (adts::channel_config_from_count(config.channel_count), None), - }; - Ok(Kind::Aac { - object_type: aac.profile, - sample_rate: config.sample_rate, - channel_config, - program_config, - }) + AudioCodec::AAC(codec) => { + // The description is exact, and names the LC core under explicit SBR or PS. Without + // one, the catalog is all there is. + Ok(Kind::Aac(match &config.description { + Some(asc) => aac::in_band(asc)?, + None => aac::InBand { + object_type: codec.profile, + sample_rate: config.sample_rate, + channel_config: adts::channel_config_from_count(config.channel_count)?, + program_config: None, + }, + })) } AudioCodec::Mp2 => Ok(Kind::Mp2 { sample_rate: config.sample_rate, diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index 00d9294308..f8847cffc2 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -1484,6 +1484,91 @@ async fn aac_program_config_roundtrip() { assert_eq!(roundtripped, ingested, "the element is written once, not per frame"); } +/// GStreamer 1.28 `fdkaacenc` output, 48 kHz stereo, remuxed to FLV by ffmpeg 9.0.1. Its +/// AudioSpecificConfig signals SBR (and PS for v2) explicitly: object type 5 or 29 over an LC core +/// at 24 kHz, stereo for v1 and mono for v2. ADTS has two bits for the object type, so export +/// labels the LC core at its own rate and layout and leaves SBR and PS to implicit signaling, the +/// header ffmpeg's ADTS muxer writes too. ffprobe reads the result back as HE-AAC or HE-AACv2 at +/// 48 kHz stereo, like the source. +#[tokio::test(start_paused = true)] +async fn aac_explicit_sbr_exports_its_lc_core() { + let fixtures: [(&[u8], u8); 2] = [ + (include_bytes!("test_data/he_aac.flv"), 2), + (include_bytes!("test_data/he_aac_v2.flv"), 1), + ]; + for (data, channel_config) in fixtures { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); + let mut import = crate::container::flv::Import::new(broadcast, catalog.reserve()); + import.decode(data).unwrap(); + import.finish().unwrap(); + + let snapshot = catalog.snapshot(); + let (name, _) = snapshot.audio.renditions.iter().next().expect("an AAC track"); + let ingested = read_frames(&consumer, name, Kind::Audio).await; + assert!(!ingested.is_empty(), "no AAC frames"); + + let ts = drain(consumer).await; + assert_packet_aligned(&ts); + + let (header, block) = first_adts_frame(&ts); + assert_eq!(header.object_type, 2, "the LC core, not a masked SBR or PS"); + assert_eq!(header.sample_rate, 24_000, "the core rate, not the output rate"); + assert_eq!(header.channel_config, channel_config); + assert_eq!(block, ingested[0], "the raw data block is untouched"); + } +} + +/// Without a description, the catalog is all ADTS has to label a track with. A channel count no +/// channelConfiguration names is refused rather than written as stereo, and HE-AAC, whose core +/// rate and layout only a description names, is refused rather than masked to AAC Main. +#[tokio::test(start_paused = true)] +async fn aac_export_refuses_what_adts_cannot_label() { + for (profile, channel_count, refusal) in [(2, 7, "7 channels"), (5, 2, "audioObjectType 5")] { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let mut catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); + + let track = broadcast + .create_track( + broadcast.unique_name(".aac"), + hang::container::track_info(hang::catalog::PRIORITY.audio), + ) + .unwrap(); + let mut cfg = AudioConfig::new(AAC { profile }, 48_000, channel_count); + cfg.container = Container::Legacy; + catalog + .modify() + .unwrap() + .audio + .renditions + .insert(track.name().to_string(), cfg); + + let mut producer = Producer::new(track, HangContainer::Legacy(crate::container::Kind::Data)); + producer + .write(Frame { + timestamp: Timestamp::from_millis(0).unwrap(), + duration: None, + payload: Bytes::from_static(&[0x01, 0x02]), + keyframe: true, + }) + .unwrap(); + producer.finish().unwrap(); + + let mut exporter = Export::new(crate::source::announced(&consumer)).await.unwrap(); + let err = loop { + match tokio::time::timeout(Duration::from_secs(1), exporter.next()).await { + Ok(Ok(Some(_))) => continue, + Ok(Ok(None)) => panic!("export completed; expected a refusal naming {refusal}"), + Ok(Err(e)) => break e, + Err(_) => panic!("export neither errored nor completed"), + } + }; + assert!(err.to_string().contains(refusal), "expected {refusal}, got: {err}"); + } +} + /// The ffmpeg E-AC-3 fixture must survive TS -> MoQ -> TS byte-for-byte in an /// audio-only program; the PMT re-announces ATSC 0x87 with the 'EAC3' /// registration descriptor. diff --git a/rs/moq-mux/src/container/ts/test_data/he_aac.flv b/rs/moq-mux/src/container/ts/test_data/he_aac.flv new file mode 100644 index 0000000000000000000000000000000000000000..5bf778ce3c02107160e456ee1dca1f0b63c0e1b5 GIT binary patch literal 3110 zcmd5;c~BEq9DW-z5D+90P*g0&0wP|KQILWm96}Y3P!$9f2_$BM=8*dmQP4q-;ZhML zU~L>xK`Mw^AP_-X7!fRrNWl=qyFo5f5eVtqBv?D`Uv^;nZQkzNcl_S>e);y>b=%~H zU;zNyuz*kNY*l7@$WBqqDel zI0W`E0L9ToIT`3H+e2t<-T{WH=N0dtJt3@UIESR_0Gc78j9>;AL4dAub|9C|2xQR# z#Y{b52AdYd4c;3NOykfo0L$jBwg!4^76-P4GlD}PF$_RRfDp)K^B~WF0}R#J5DdVh zgJD_43gNP0u?Dz(tOE=cfx=Dd1z-*k0Qah?zAphw2}x+302_-Y8xbOLJusvqZhc}v zR5s4k1xUQ!K!D+m{}-wtyIYWYBLGWLq9q8?qbXirp-^BYQQ_z8FK-v?wC@RPeOoqB z;5c@0c8=qS+_C@V$a;GDuNmt?aoM{u7B{fNH8s-y>NtZmM?%-E*CuCU=z17mY>83d zp_0HS8@2B0@SWS5iqs)&ifl z*h}ZawqX;+LhIUn!l?b)9?|cKa*lL7(_N%VF#{LsfP=n%Z)pWBzx)vf<1L| z(`OUz6+8m_seNDpW??S&+lrWH>uB=KYSpFcc{4ta&rYd0H8(3m?BAfI)A3yGEB5}IvRqR{tlwrEI zO|SxG4S@&`IG>&#xPs>Wa8tHa zF069=s>>3)X-h>kxuIii^DpMlx|iXH%I&D4^<~VmefKlzTFq-%Gn4^f46Zl+;xs@4 zmr_gSGnbiA8J{-Jh(K=XY$+LkdjvkHXW!J7{FJAW@!ZaJ^j59&x)$@-L^<8e%_2Or)2vFu0l$+xyq+OjCAU$9H<*r{-P+zwc1|?S(U$_nL7T zMJkmJ)6%)QPc*hPXCy_svK0F+*vIV|Qg3>MhznS!R@R>dNri1zV zBU=N<-P6X~3~?)-iS6#(yp7%9$=uAh3GH~uE=CXa&A(i_pS%M88ea%&dSfo*w9lvC zJ_`$S5}!+fUkK&Lfruy}N*14D-}uY}MR2>n!|g@`%qL?yDT?i*`IF*#k&1j0ej74e sek{HgxzjrlDRScKv{q%ghGO*PVD`6gE`GJ@yIKUqe*ejeP$I|v1S3oeTL1t6 literal 0 HcmV?d00001 diff --git a/rs/moq-mux/src/container/ts/test_data/he_aac_v2.flv b/rs/moq-mux/src/container/ts/test_data/he_aac_v2.flv new file mode 100644 index 0000000000000000000000000000000000000000..1df313b4099b5f084b889f928d0e7a24784f352c GIT binary patch literal 3297 zcmd5;dpOf;9Djc^8Z}Kxky-T*>-|z3c&+lFB;)Y-W0Lrj{ zX#fnOH3SBzu!7byLTPriP#O*nr~n+D6G97RvVwqhce8GNw~sBr(>Qb{iw>8-83v$O zQ)*CSFjYFvrUeEEFk~}5!P1F;%&?ieppA)a0w}RV86gZ7f`BPfYhMO~R*3hO5Lyr&1F(U?q@@5K$P9vGTt)~R62kz*1gQD40)rvXEddPKS~dm{(8aKr zGTE#^Se63ZR%QT0M&Ls2_yI5m2!OsSjra4xYvlyAtp*1#$_EjIqANJHb#NXc9l@Rl zO2bhq9zP(wAi-itJr96IDA#;Q-Bxglry7O!6ykdA5Q+6Lr0@FU(MP$LK<+I7Y(&{Y z?t~-h?QCv_^W`SpGHt#5F+4})<^RfTjK~sKmgJT>jdc!1uuh0ncC^_sUaMB|c}b}6QA z^=F&boo2sjkL|fvndIQ9TJ6+Serx1TdfUWlAW@U z^FU-qCSNVy)Of$H?$0+PqSxMe*w|x9<{Y;ivCtCnHc#>J89Fs;OFXH+yu-65R0Z`2 zGpGy3h@${lz)lZzH6H*0IsskP__(Cd9=GDTXIoXc*!o1&!MXgyV8nbUk8G7q9c8-za6E(vhcg%Q)@cUe|egtXAcZuZ$v5dt41` zA>qt_LOV2%m*u+<-3rYa67KL$in{I84UTUmE1L&La*UClM48MWa+x7O>MLN_6$sHc)~CI1XsfPGx}$l2@9)iTdhT?Au4VTta^F_;C-z*$ zT*~JNQ9RCaKg+PY`Cz3Y>H#tf%of@$WKJ}A@GLa!c>wt$GmyKfCDu^6LtI%NY5C3_ zsi#t@+j5hSfZ6X)M_3gv=yIjL#Na)y`3rMqxX-TeeN}IOa>mtw)-oOE?}nOj?l5Hp zuqjU&l4Sx;2T8D@5sjPks`+t?JSlyBeIG^Z-Y`ri#pW3h8X}1E%kM~dqHs}F@`b2; zuSy=5E1nt?x@D8glPoH_m6`ylB-tLGD5On-XN;$eTr^gaW@wm_6AbwG4|b1+^CGKw zKW`EwCW_WONl;(0{7wbk#LjLd=czSBE}y42yeH^J2(kt7T8(y?6A2E(^;myU86GEsCz38#c@e8Nm4+bg{8Y~Vwckih$@h?&uPTF) ztBmwLGfnNr3^i-xCl9Mf2kqni{QPcDi8xNnQdn1a0LL?Kx>4aBH_)zx!j4Rk@V9Gx zjrMEXiH-J`mxes-B7l*r%o5V#{arqawwj&~!!kOZ)yoh=AWT=-@lx;(WN$2beDn${ z(7Ajlp8VH>&BK22qvtk`I24FG3njugR4?<6((6SfME&zc4YE&Zle{IX3$kw|X&G%R zYPfo~E(12k`vuvD9PLK*&pQyL0f9fsT4q1F+%vjzmyXxI0_%VKHI+Fsd)K636E4|lK%z_QapPA literal 0 HcmV?d00001 From 437159c8e128c498572335a2a726c8d532b0f96c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 26 Sep 2026 18:47:12 -0700 Subject: [PATCH 11/11] feat(ts): import surround Opus with a family 1 OpusHead (#4282) Co-authored-by: Claude Opus 5.5 --- doc/lib/rs/moq-mux.md | 2 +- quest/m1/audio-codecs/README.md | 4 +- quest/m1/audio-codecs/gst-opus-surround.md | 21 ++ .../m1/audio-codecs/ts-opus-channel-codes.md | 27 +++ .../audio-codecs/ts-opus-export-refusals.md | 23 +++ quest/m1/audio-codecs/ts-opus-surround.md | 20 -- rs/moq-audio/tests/import.rs | 64 +++++++ rs/moq-mux/src/codec/opus/import.rs | 2 +- rs/moq-mux/src/codec/opus/mapping.rs | 179 ++++++++++++++++++ rs/moq-mux/src/codec/opus/mod.rs | 154 ++++++--------- rs/moq-mux/src/container/ts/import.rs | 23 ++- rs/moq-mux/src/container/ts/import_test.rs | 23 +++ .../src/container/ts/test_data/opus_5_1.ts | Bin 0 -> 5452 bytes 13 files changed, 412 insertions(+), 130 deletions(-) create mode 100644 quest/m1/audio-codecs/gst-opus-surround.md create mode 100644 quest/m1/audio-codecs/ts-opus-channel-codes.md create mode 100644 quest/m1/audio-codecs/ts-opus-export-refusals.md delete mode 100644 quest/m1/audio-codecs/ts-opus-surround.md create mode 100644 rs/moq-audio/tests/import.rs create mode 100644 rs/moq-mux/src/codec/opus/mapping.rs create mode 100644 rs/moq-mux/src/container/ts/test_data/opus_5_1.ts diff --git a/doc/lib/rs/moq-mux.md b/doc/lib/rs/moq-mux.md index a3fd8d42e8..4d9f6b115f 100644 --- a/doc/lib/rs/moq-mux.md +++ b/doc/lib/rs/moq-mux.md @@ -14,7 +14,7 @@ Turns existing container formats into hang broadcasts and back. This is what | Format | Import | Export | Notes | | --- | --- | --- | --- | | fMP4 / CMAF | yes | yes | Passthrough as `cmaf` or repackaged as `legacy`. | -| MPEG-TS | yes | yes | H.264/H.265; AAC, MP2, AC-3, E-AC-3; SCTE-35 and subtitle PIDs carried as tracks; service tables round-trip; signalled timebase discontinuities preserved; paced export. | +| MPEG-TS | yes | yes | H.264/H.265; AAC, MP2, AC-3, E-AC-3, Opus up to 7.1; SCTE-35 and subtitle PIDs carried as tracks; service tables round-trip; signalled timebase discontinuities preserved; paced export. | | FLV / RTMP | yes | yes | Legacy H.264 + AAC + MP3, plus enhanced-RTMP HEVC, AV1, VP9, Opus, AC-3, E-AC-3, and multitrack. | | Matroska / WebM | yes | yes | | | Annex-B (H.264, H.265) | yes | yes | Parameter sets extracted to the catalog or re-injected per keyframe. | diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index 762769dc5a..6dc3b3e0d1 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -45,10 +45,12 @@ its own decode and encode quest so verification stays per host. - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [HE-AAC catalog output](/quest/m1/audio-codecs/he-aac-catalog-output.md) - HE-AAC catalog entries name the output rate and layout, not the LC core +- [TS Opus export refusals](/quest/m1/audio-codecs/ts-opus-export-refusals.md) - the TS exporter refuses Opus heads its channel code cannot describe instead of mislabeling them +- [TS Opus channel codes](/quest/m1/audio-codecs/ts-opus-channel-codes.md) - TS Opus with a channel code of 0x81 or above imports with its real head or is refused, never guessed as stereo - [AudioToolbox encode](/quest/m1/audio-codecs/encode-audiotoolbox.md) - macOS and iOS encode AAC-LC -- [TS surround Opus](/quest/m1/audio-codecs/ts-opus-surround.md) - a 3 to 8 channel Opus stream from MPEG-TS imports with a family 1 OpusHead that decodes - [AAC encode refusals](/quest/m1/audio-codecs/aac-encode-refusals.md) - `Config::encode` refuses channel counts it cannot name instead of writing stereo - [AAC parse truncated SBR](/quest/m1/audio-codecs/aac-parse-truncated-sbr.md) - `Config::parse` refuses SBR and PS configs cut off before their core +- [GStreamer surround Opus](/quest/m1/audio-codecs/gst-opus-surround.md) - the moq-gst sink publishes 3 to 8 channel Opus with the OpusHead its caps describe ## Related diff --git a/quest/m1/audio-codecs/gst-opus-surround.md b/quest/m1/audio-codecs/gst-opus-surround.md new file mode 100644 index 0000000000..8b7a195ebc --- /dev/null +++ b/quest/m1/audio-codecs/gst-opus-surround.md @@ -0,0 +1,21 @@ +# [S] Surround Opus from GStreamer + +## Goal + +`moq-gst`'s sink publishes a 3 to 8 channel `audio/x-opus` stream with the +OpusHead its caps describe, so surround Opus from `opusenc` plays natively like +the same stream from Matroska or MPEG-TS. + +## Plan + +- The sink refuses more than two channels because it builds the head from + `channels` and `rate` alone. Multichannel Opus caps also carry + `channel-mapping-family`, `stream-count`, `coupled-count`, and + `channel-mapping`; build the mapping with `opus::Mapping::new`, which + validates them, and refuse caps that omit or contradict them. +- Prefer a `streamheader` OpusHead when the caps carry one, if that path is + simpler and covers the same streams. +- Regression: an `opusenc` 5.1 pipeline publishes a family 1 description that + `moq-audio` decodes to six channels. + +Public API: none. Wire: none. diff --git a/quest/m1/audio-codecs/ts-opus-channel-codes.md b/quest/m1/audio-codecs/ts-opus-channel-codes.md new file mode 100644 index 0000000000..2fbce343d2 --- /dev/null +++ b/quest/m1/audio-codecs/ts-opus-channel-codes.md @@ -0,0 +1,27 @@ +# [S] TS Opus import refuses or parses extended channel codes + +## Goal + +An MPEG-TS Opus stream whose extension descriptor carries a +`channel_config_code` of 0x81 or above imports with its real OpusHead or is +refused. It never decodes as a guessed stereo stream, which misreads every +multistream packet. + +## Plan + +- The importer falls back to stereo for codes it does not know. 0x81 is + specified as an explicitly coded layout (channel count, mapping family, stream + counts, and table in the descriptor), and ffmpeg's muxer also writes + `0x80 | channels` for an alternate table its own demuxer does not read. Check + the Opus-in-TS spec and ffmpeg before deciding which codes to parse. +- Build parsed layouts with the public `opus::Mapping::new`, which validates + the table; refuse any code that is reserved or does not parse, per stream, + without failing the rest of the program. +- Fixtures from a real muxer where one exists; a hand-built descriptor + otherwise, noted as such. + +Public API: none. Wire: none. + +## Related + +- [TS Opus export refusals](/quest/m1/audio-codecs/ts-opus-export-refusals.md) - the exporter side of the same descriptor diff --git a/quest/m1/audio-codecs/ts-opus-export-refusals.md b/quest/m1/audio-codecs/ts-opus-export-refusals.md new file mode 100644 index 0000000000..c501cb032f --- /dev/null +++ b/quest/m1/audio-codecs/ts-opus-export-refusals.md @@ -0,0 +1,23 @@ +# [S] TS Opus export refuses what it cannot label + +## Goal + +`moq-mux`'s MPEG-TS exporter never labels an Opus track with a channel +configuration its packets do not have. A track whose OpusHead the Opus +extension descriptor can describe exports as today; anything else is refused, +not written with a guessed `channel_config_code`. + +## Plan + +- The exporter derives `channel_config_code` from the catalog channel count + alone, clamped to 1..=8, so a family 255 or ambisonics head, a family 1 head + with a non-Vorbis table, or more than eight channels exports mislabeled. + Decide from the parsed description: family 0, and family 1 with the Vorbis + default mapping, keep the plain code; refuse the rest, or write the explicit + layout if the import side of + [TS Opus channel codes](/quest/m1/audio-codecs/ts-opus-channel-codes.md) settles + one both sides agree on. +- A track with no description stays mono or stereo only. +- Tests with real heads, checked against ffprobe where ffmpeg reads the result. + +Public API: none. Wire: none; TS output changes only for inputs it mislabeled. diff --git a/quest/m1/audio-codecs/ts-opus-surround.md b/quest/m1/audio-codecs/ts-opus-surround.md deleted file mode 100644 index 5e6e6be4fc..0000000000 --- a/quest/m1/audio-codecs/ts-opus-surround.md +++ /dev/null @@ -1,20 +0,0 @@ -# [S] Surround Opus from MPEG-TS - -## Goal - -An MPEG-TS Opus stream with 3 to 8 channels imports with an OpusHead that -decodes, so it plays natively like the same stream from Matroska or FLV. - -## Plan - -- The TS importer knows only the `channel_config_code` from the Opus - extension descriptor and builds `opus::Config::new(48_000, channels)`, whose - `encode` refuses more than two channels. The catalog then carries no - description and `moq-audio` refuses the track. -- Codes 3 to 8 mean family 1 with the Vorbis default stream and coupled counts - and mapping (the Vorbis orders of RFC 7845 §5.1.1.2), so the - importer can synthesize the head. That needs a way to build a family 1 - `Config` and have `encode` emit its table; decide whether that is a - constructor on `opus::Mapping` or a TS-local head writer. -- Regression: a 5.1 TS fixture imports with a family 1 description and decodes - to six channels. diff --git a/rs/moq-audio/tests/import.rs b/rs/moq-audio/tests/import.rs new file mode 100644 index 0000000000..010c1976c4 --- /dev/null +++ b/rs/moq-audio/tests/import.rs @@ -0,0 +1,64 @@ +//! Container imports that must decode end to end: a stream `moq-mux` imports +//! plays through [`decode::Consumer`] like the same stream from any container. + +use std::time::Duration; + +use moq_audio::{Format, Layout, decode}; + +/// MPEG-TS names surround Opus only by channel count, so the importer has to +/// synthesize the OpusHead mapping table or the decoder refuses the track. +#[tokio::test] +async fn ts_surround_opus_decodes() { + // A 440 Hz center channel in 5.1; see `import_opus_surround_catalog` in moq-mux. + let data = include_bytes!("../../moq-mux/src/container/ts/test_data/opus_5_1.ts"); + + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let broadcast_consumer = broadcast.consume(); + let catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); + let mut import = moq_mux::container::ts::Import::new(broadcast, catalog.reserve()); + import.decode(&data[..]).unwrap(); + import.finish().unwrap(); + + let snapshot = catalog.snapshot(); + let (name, config) = snapshot.audio.renditions.iter().next().expect("an Opus track"); + + let mut options = decode::Options::default(); + options.output.format = Format::F32; + // The whole file is imported before anything decodes it. + options.max_age = Duration::from_secs(30); + let mut consumer = decode::Consumer::new(&broadcast_consumer, config, name, options) + .await + .unwrap(); + assert_eq!(consumer.layout(), Layout::FivePointOne); + + // Canonical 5.1 order: FL, FR, FC, LFE, SL, SR. + let mut energy = [0.0f64; 6]; + let mut frames = 0; + while let Some(frame) = tokio::time::timeout(Duration::from_secs(5), consumer.read()) + .await + .expect("decoded frame timed out") + .unwrap() + { + let (samples, rest) = frame.data.as_chunks::<4>(); + assert!(rest.is_empty()); + for chunk in samples.as_chunks::<6>().0 { + for (channel, sample) in energy.iter_mut().zip(chunk) { + *channel += f32::from_le_bytes(*sample).powi(2) as f64; + } + frames += 1; + } + } + + assert!(frames > 12_000, "expected most of 0.5 s at 48 kHz, got {frames} frames"); + let center = energy[2]; + assert!( + center / frames as f64 > 0.001, + "center should carry the tone: {energy:?}" + ); + for (channel, &other) in energy.iter().enumerate().filter(|&(channel, _)| channel != 2) { + assert!( + other < center / 100.0, + "channel {channel} should be near silent: {energy:?}" + ); + } +} diff --git a/rs/moq-mux/src/codec/opus/import.rs b/rs/moq-mux/src/codec/opus/import.rs index 69b4631c86..829bef3736 100644 --- a/rs/moq-mux/src/codec/opus/import.rs +++ b/rs/moq-mux/src/codec/opus/import.rs @@ -115,7 +115,7 @@ impl Import { pub fn config(init: &[u8]) -> crate::Result { let mut buf = init; let mut config: hang::catalog::AudioConfig = Config::parse(&mut buf)?.into(); - // Publish the head as given: re-encoding it would drop a channel mapping table. + // Publish the head as given: re-encoding it would drop the output gain. config.description = Some(bytes::Bytes::copy_from_slice(init)); Ok(config) } diff --git a/rs/moq-mux/src/codec/opus/mapping.rs b/rs/moq-mux/src/codec/opus/mapping.rs new file mode 100644 index 0000000000..4157dd5303 --- /dev/null +++ b/rs/moq-mux/src/codec/opus/mapping.rs @@ -0,0 +1,179 @@ +//! The OpusHead channel mapping table (RFC 7845 §5.1.1). + +use bytes::Buf; + +use super::{Error, Result}; + +/// An OpusHead channel mapping table (RFC 7845 §5.1.1), present for every family but 0. +#[derive(Clone, Copy, PartialEq, Eq)] +pub struct Mapping { + family: u8, + streams: u8, + coupled: u8, + channels: u8, + // Sized for the most channels any family allows, so `opus::Config` stays `Copy`. + table: [u8; 255], +} + +/// The fields of a channel mapping table, checked into a [`Mapping`] by [`Mapping::new`]. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Config<'a> { + /// The channel mapping family; 0 has no table and is refused. + pub family: u8, + /// Number of Opus streams in each packet. + pub streams: u8, + /// How many of those streams are coupled (stereo); they come first. + pub coupled: u8, + /// The decoded channel feeding each output channel, 255 for silence. + pub table: &'a [u8], +} + +/// The family 1 stream counts and table for 1 to 8 channels, as libopus and +/// RFC 7845 §5.1.1.2 lay out the Vorbis channel orders. +const VORBIS: [(u8, u8, &[u8]); 8] = [ + (1, 0, &[0]), + (1, 1, &[0, 1]), + (2, 1, &[0, 2, 1]), + (2, 2, &[0, 1, 2, 3]), + (3, 2, &[0, 4, 1, 2, 3]), + (4, 2, &[0, 4, 1, 2, 3, 5]), + (4, 3, &[0, 4, 1, 2, 3, 5, 6]), + (5, 3, &[0, 6, 1, 2, 3, 4, 5, 7]), +]; + +impl Mapping { + /// The family 1 mapping a surround encoder uses for `channels` in Vorbis + /// order, for sources that name only a channel count. + pub(crate) fn vorbis(channels: u8) -> Result { + let (streams, coupled, table) = *channels + .checked_sub(1) + .and_then(|index| VORBIS.get(index as usize)) + .ok_or(Error::UnsupportedChannelCount(channels as u32))?; + Self::new(Config { + family: 1, + streams, + coupled, + table, + }) + } + + /// Check `config` into a mapping with one output channel per table entry. + /// + /// Refuses family 0, a channel count the family does not allow, and a table + /// that names no streams, more coupled streams than streams, or a channel + /// past the decoded ones. + pub fn new(config: Config<'_>) -> Result { + let Config { + family, + streams, + coupled, + table, + } = config; + let max_channels = match family { + 0 => return Err(Error::UnsupportedMappingFamily(0)), + 1 => 8, + _ => 255, + }; + if table.is_empty() || table.len() > max_channels { + return Err(Error::UnsupportedChannelCount(table.len() as u32)); + } + + let decoded = streams as u32 + coupled as u32; + if streams == 0 || coupled > streams || decoded > 255 { + return Err(Error::InvalidMappingTable); + } + if table.iter().any(|&entry| entry != 255 && entry as u32 >= decoded) { + return Err(Error::InvalidMappingTable); + } + + let mut padded = [0u8; 255]; + padded[..table.len()].copy_from_slice(table); + Ok(Self { + family, + streams, + coupled, + channels: table.len() as u8, + table: padded, + }) + } + + pub(super) fn parse(buf: &mut T, family: u8, channels: u8) -> Result { + if buf.remaining() < 2 + channels as usize { + return Err(Error::MappingTableTooShort); + } + let streams = buf.get_u8(); + let coupled = buf.get_u8(); + let mut table = [0u8; 255]; + buf.copy_to_slice(&mut table[..channels as usize]); + Self::new(Config { + family, + streams, + coupled, + table: &table[..channels as usize], + }) + } + + /// The channel mapping family: 1 is the Vorbis speaker order, 2 and 3 are + /// ambisonics, and 255 is channels with no declared position. + pub fn family(&self) -> u8 { + self.family + } + + /// Number of Opus streams in each packet. + pub fn streams(&self) -> u8 { + self.streams + } + + /// How many of those streams are coupled (stereo); they come first. + pub fn coupled(&self) -> u8 { + self.coupled + } + + /// The decoded channel feeding each output channel, 255 for silence. + pub fn table(&self) -> &[u8] { + &self.table[..self.channels as usize] + } +} + +impl std::fmt::Debug for Mapping { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Mapping") + .field("family", &self.family) + .field("streams", &self.streams) + .field("coupled", &self.coupled) + .field("table", &self.table()) + .finish() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn new_validates_the_table() { + fn new(family: u8, streams: u8, coupled: u8, table: &[u8]) -> Result { + Mapping::new(Config { + family, + streams, + coupled, + table, + }) + } + + let mapping = new(255, 2, 0, &[0, 1, 255]).unwrap(); + assert_eq!(mapping.family(), 255); + assert_eq!((mapping.streams(), mapping.coupled()), (2, 0)); + assert_eq!(mapping.table(), &[0, 1, 255]); + + assert!(matches!(new(0, 1, 1, &[0, 1]), Err(Error::UnsupportedMappingFamily(0)))); + assert!(matches!(new(1, 1, 0, &[]), Err(Error::UnsupportedChannelCount(0)))); + assert!(matches!(new(1, 9, 0, &[0; 9]), Err(Error::UnsupportedChannelCount(9)))); + for (streams, coupled, table) in [(0, 0, &[0][..]), (1, 2, &[0]), (1, 1, &[2]), (200, 100, &[0])] { + assert!( + matches!(new(1, streams, coupled, table), Err(Error::InvalidMappingTable)), + "{streams} streams, {coupled} coupled, {table:?}" + ); + } + } +} diff --git a/rs/moq-mux/src/codec/opus/mod.rs b/rs/moq-mux/src/codec/opus/mod.rs index a73e9227a6..749cce7217 100644 --- a/rs/moq-mux/src/codec/opus/mod.rs +++ b/rs/moq-mux/src/codec/opus/mod.rs @@ -4,8 +4,10 @@ //! publishes raw Opus frames (no Ogg framing) to a moq broadcast. mod import; +pub mod mapping; pub use import::*; +pub use mapping::Mapping; use bytes::{Buf, Bytes}; @@ -37,8 +39,8 @@ pub enum Error { #[error("invalid OpusHead channel mapping table")] InvalidMappingTable, - /// [`Config::encode`] only emits channel mapping family 0. - #[error("cannot encode channel mapping family {0}")] + /// [`Mapping::new`] was given family 0, which has no mapping table. + #[error("channel mapping family {0} has no mapping table")] UnsupportedMappingFamily(u8), } @@ -58,79 +60,6 @@ pub struct Config { pub mapping: Option, } -/// An OpusHead channel mapping table (RFC 7845 §5.1.1), present for every family but 0. -#[derive(Clone, Copy, PartialEq, Eq)] -pub struct Mapping { - family: u8, - streams: u8, - coupled: u8, - channels: u8, - // Sized for the most channels any family allows, so `Config` stays `Copy`. - table: [u8; 255], -} - -impl Mapping { - fn parse(buf: &mut T, family: u8, channels: u8) -> Result { - if buf.remaining() < 2 + channels as usize { - return Err(Error::MappingTableTooShort); - } - let streams = buf.get_u8(); - let coupled = buf.get_u8(); - if streams == 0 || coupled > streams || streams as u32 + coupled as u32 > 255 { - return Err(Error::InvalidMappingTable); - } - - let mut table = [0u8; 255]; - for entry in &mut table[..channels as usize] { - *entry = buf.get_u8(); - // 255 marks a silent channel. - if *entry != 255 && *entry >= streams + coupled { - return Err(Error::InvalidMappingTable); - } - } - - Ok(Self { - family, - streams, - coupled, - channels, - table, - }) - } - - /// The channel mapping family: 1 is the Vorbis speaker order, 2 and 3 are - /// ambisonics, and 255 is channels with no declared position. - pub fn family(&self) -> u8 { - self.family - } - - /// Number of Opus streams in each packet. - pub fn streams(&self) -> u8 { - self.streams - } - - /// How many of those streams are coupled (stereo); they come first. - pub fn coupled(&self) -> u8 { - self.coupled - } - - /// The decoded channel feeding each output channel, 255 for silence. - pub fn table(&self) -> &[u8] { - &self.table[..self.channels as usize] - } -} - -impl std::fmt::Debug for Mapping { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - f.debug_struct("Mapping") - .field("family", &self.family) - .field("streams", &self.streams) - .field("coupled", &self.coupled) - .field("table", &self.table()) - .finish() - } -} - impl Config { /// Build a mono/stereo Opus config with no pre-skip. pub fn new(sample_rate: u32, channel_count: u32) -> Self { @@ -169,17 +98,9 @@ impl Config { buf.advance(2); // Skip gain until if/when we support it. let family = buf.get_u8(); - let max_channels = match family { - 0 => 2, - 1 => 8, - _ => 255, - }; - if channel_count == 0 || channel_count > max_channels { - return Err(Error::UnsupportedChannelCount(channel_count)); - } - let mapping = match family { - 0 => None, + 0 if (1..=2).contains(&channel_count) => None, + 0 => return Err(Error::UnsupportedChannelCount(channel_count)), family => Some(Mapping::parse(buf, family, channel_count as u8)?), }; @@ -195,28 +116,35 @@ impl Config { }) } - /// Encode the minimal OpusHead packet (19 bytes; channel mapping family - /// 0 and zero gain). + /// Encode an OpusHead packet (RFC 7845 §5.1) with zero gain, followed by + /// the channel mapping table when there is one. /// - /// Errors with [`Error::UnsupportedChannelCount`] unless `channel_count` is 1 - /// or 2, since mapping family 0 is only defined for mono/stereo per RFC 7845 §5.1. - /// Multi-channel streams need family 1 with a channel mapping table, which - /// this helper does not emit, so any `mapping` is [`Error::UnsupportedMappingFamily`]. + /// Errors with [`Error::UnsupportedChannelCount`] when `channel_count` is not + /// 1 or 2 without a `mapping` (family 0 is only defined for mono/stereo), or + /// is not the mapping's own channel count. pub fn encode(&self) -> Result { - if let Some(mapping) = &self.mapping { - return Err(Error::UnsupportedMappingFamily(mapping.family)); - } - if !(1..=2).contains(&self.channel_count) { + let valid = match &self.mapping { + None => (1..=2).contains(&self.channel_count), + Some(mapping) => self.channel_count == mapping.table().len() as u32, + }; + if !valid { return Err(Error::UnsupportedChannelCount(self.channel_count)); } - let mut head = Vec::with_capacity(19); + + let mut head = Vec::with_capacity(21 + self.channel_count as usize); head.extend_from_slice(b"OpusHead"); head.push(1); // version head.push(self.channel_count as u8); head.extend_from_slice(&self.pre_skip.to_le_bytes()); head.extend_from_slice(&self.sample_rate.to_le_bytes()); head.extend_from_slice(&0i16.to_le_bytes()); // output gain - head.push(0); // channel mapping family (0 = mono/stereo) + match &self.mapping { + None => head.push(0), + Some(mapping) => { + head.extend_from_slice(&[mapping.family(), mapping.streams(), mapping.coupled()]); + head.extend_from_slice(mapping.table()); + } + } Ok(Bytes::from(head)) } } @@ -322,8 +250,36 @@ mod tests { assert_eq!(mapping.coupled(), 2); assert_eq!(mapping.table(), &[0, 4, 1, 2, 3, 5]); - // Encode only emits family 0. - assert!(matches!(parsed.encode(), Err(Error::UnsupportedMappingFamily(1)))); + // The table survives a re-encode. + assert_eq!(parsed.encode().unwrap(), bytes); + } + + #[test] + fn encodes_the_vorbis_mappings() { + for channels in 1..=8u8 { + let mut config = Config::new(48_000, channels as u32); + config.mapping = Some(Mapping::vorbis(channels).unwrap()); + let head = config.encode().unwrap(); + assert_eq!(head.len(), 21 + channels as usize, "{channels} channels"); + assert_eq!( + Config::parse(&mut head.as_ref()).unwrap(), + config, + "{channels} channels" + ); + } + + // 5.1 is the table libopus and ffmpeg write. + let five_one = Mapping::vorbis(6).unwrap(); + assert_eq!((five_one.streams(), five_one.coupled()), (4, 2)); + assert_eq!(five_one.table(), &[0, 4, 1, 2, 3, 5]); + + assert!(matches!(Mapping::vorbis(0), Err(Error::UnsupportedChannelCount(0)))); + assert!(matches!(Mapping::vorbis(9), Err(Error::UnsupportedChannelCount(9)))); + + // The channel count must agree with the table. + let mut config = Config::new(48_000, 5); + config.mapping = Some(five_one); + assert!(matches!(config.encode(), Err(Error::UnsupportedChannelCount(5)))); } #[test] diff --git a/rs/moq-mux/src/container/ts/import.rs b/rs/moq-mux/src/container/ts/import.rs index 057d998aa1..c8ac4d5090 100644 --- a/rs/moq-mux/src/container/ts/import.rs +++ b/rs/moq-mux/src/container/ts/import.rs @@ -497,11 +497,10 @@ impl Import { // by an 'Opus' registration descriptor. Channels and the (always 48 kHz) rate // come from the descriptors, so the importer is built up front. StreamType::Mpeg2PacketizedData if registration_format(descriptors) == Some(*b"Opus") => { - let channel_count = opus_channel_count(descriptors).unwrap_or(2); + let config = opus_config(descriptors)?; let track = self .broadcast .unique_track(".opus", self.catalog.track_info(hang::catalog::PRIORITY.audio))?; - let config = opus::Config::new(48_000, channel_count); let mut config: hang::catalog::AudioConfig = config.into(); config.container = self.container.clone(); Stream::Opus(Box::new(OpusStream { @@ -2342,14 +2341,15 @@ fn registration_format(descriptors: &[mpeg2ts::ts::Descriptor]) -> Option<[u8; 4 .and_then(|s| s.try_into().ok()) } -/// The Opus channel count from the DVB extension descriptor (tag 0x7f, ext tag 0x80). +/// The OpusHead implied by the DVB extension descriptor (tag 0x7f, ext tag 0x80). /// /// `channel_config_code` follows the Opus-in-TS mapping (and ffmpeg's demuxer): 0 is dual -/// mono (decoded as stereo), 1..=8 is the channel count directly. Higher codes (0x81 -/// explicitly-coded layouts, reserved values) aren't supported, so they fall back to the -/// caller's default rather than being read as a raw 129..=255 count. -fn opus_channel_count(descriptors: &[mpeg2ts::ts::Descriptor]) -> Option { - descriptors +/// mono (decoded as stereo), 1..=8 is the channel count directly, with family 0 for mono and +/// stereo and the Vorbis family 1 mapping above that. Higher codes (0x81 explicitly-coded +/// layouts, reserved values) aren't supported, so they fall back to stereo rather than being +/// read as a raw 129..=255 count. +fn opus_config(descriptors: &[mpeg2ts::ts::Descriptor]) -> crate::Result { + let channels = descriptors .iter() .find(|d| d.tag == 0x7f && d.data.first() == Some(&0x80)) .and_then(|d| d.data.get(1)) @@ -2358,6 +2358,13 @@ fn opus_channel_count(descriptors: &[mpeg2ts::ts::Descriptor]) -> Option { 1..=8 => Some(cc as u32), _ => None, }) + .unwrap_or(2); + + let mut config = opus::Config::new(48_000, channels); + if channels > 2 { + config.mapping = Some(opus::Mapping::vorbis(channels as u8)?); + } + Ok(config) } /// Parse one Opus-in-TS access-unit control header, returning `(header_len, payload_size)`. diff --git a/rs/moq-mux/src/container/ts/import_test.rs b/rs/moq-mux/src/container/ts/import_test.rs index d0c48534ac..46f8b2c792 100644 --- a/rs/moq-mux/src/container/ts/import_test.rs +++ b/rs/moq-mux/src/container/ts/import_test.rs @@ -200,6 +200,29 @@ fn import_opus_catalog() { assert_eq!(audio.channel_count, 2); } +/// `opus_5_1.ts` is a 440 Hz center channel in 5.1, which ffmpeg's libopus +/// encodes as family 1 and its muxer labels `channel_config_code` 6: +/// `ffmpeg -f lavfi -i sine=frequency=440:sample_rate=48000:duration=0.5 +/// -ac 6 -c:a libopus -b:a 128k -f mpegts opus_5_1.ts`. The descriptor names only +/// the channel count, so the importer must synthesize the Vorbis mapping table +/// or the track has no OpusHead a decoder accepts. +#[test] +fn import_opus_surround_catalog() { + let data = include_bytes!("test_data/opus_5_1.ts"); + let catalog = import_ts(data); + + assert_eq!(catalog.audio.renditions.len(), 1, "expected one Opus track"); + let audio = catalog.audio.renditions.values().next().unwrap(); + assert_eq!(audio.channel_count, 6); + + let head = crate::codec::opus::Config::parse(&mut audio.description.as_deref().expect("an OpusHead")).unwrap(); + assert_eq!(head.channel_count, 6); + let mapping = head.mapping.expect("a family 1 mapping"); + assert_eq!(mapping.family(), 1); + assert_eq!((mapping.streams(), mapping.coupled()), (4, 2)); + assert_eq!(mapping.table(), &[0, 4, 1, 2, 3, 5]); +} + /// Opus frames from real ffmpeg output must decode: a non-empty run of Opus packets, /// each a plausible size (the control header was stripped, not left in the payload). #[tokio::test(start_paused = true)] diff --git a/rs/moq-mux/src/container/ts/test_data/opus_5_1.ts b/rs/moq-mux/src/container/ts/test_data/opus_5_1.ts new file mode 100644 index 0000000000000000000000000000000000000000..14ad7cb9c04960dcb68052e1a3177667cb971d93 GIT binary patch literal 5452 zcmdUzc{G&k1IOP{_PtEm%FTRI-LqG}PcXW;EyC)31B(xqsZ>HRrtZ&%E#V`9AOS`8@BO7j7sj0vJ;j z08kqMtbx`55IsuFVjGCd+&m!A%bVBE>(@}9lU`~N^hJ}7wSU(KZU~3~K#d@?I{=~r zO0`?-{#_M+*amLH=+03i&FuaLpaNWsFFww#sKAh53`Z2#&9TmfA71=_{sC?TsBzl> zK)668005w7PLqR209ig4x!K}$#w@OmJ) z?M;KG)nKuZy<5CkoO72%M=x&c(%LG9f?^M6obL@9r(==U<>Dn0Bkx_DM!udQXm~A_ z79j*j9nf(pe8xR#a1bDBp1LB_2MatzBWIQB9*aJr5XgUfF!P|aFv^H_wV9a;XQx7O zbt55o`eCTq`Szii(FV}sbmiBX+IJ(~Uy)Pv;aF^Xa}+pOsuic~R~9<%9^K57zxZ;T zrZRMq^M5Cq&Ah7aoo~Ol+8QJ=+?gIrG$?b76blfavgG1!`1J|;ZrH)_aBun72ljrJ za_xg2-hS3ekeOt`=&pVi_-7ZNeIH<~`+(56E3vq%8SkxLWv`_hZk_fM)u<}@LiYlw zzc|`zY4pXMm{oe!(V6Qc3u~Pv#S*{Ii1+IY#c)-06T2$9iCxuLclBDqhO4nmS9y2W zVkk)V;u_+TC~0?Z-!|0fA9U#(D!H&gq*T97ED^snS*qWW>sD)&X+h$a?5!Rxp2iKo zq%IGi(si>vc(&7*EXDVc1V%CewPWX*QH zhX7W*(>LvhqL%Nxpg`EmDe`3^Q!dQ;gGhlqN^xk&PS57*5V>*f)w+$WAWL?TJ}by< z<}sN6e%icL<7|lFkZ|6OUE=9PA|$?ZX(%_x9<|Mah%pqq|ndKkVvX+8N%iA6vfi;qr=ySi-?yQ;nJYM$nXtL7|My+9bsAvSC3IhHaE0}v4PsevVK z144a;@n`OZ`mViY68mGh0-7owyIW2)M)0@2IReAiEFlm)I9B0o~W`^z!|^x4hSSj`|Bb$MPIT%_RL#aS{> zIAmL8pZ^Bsxd-ygpDtUj30~8P^t<=cwBp%3P#-r&8M#e0P< zh5}`;s4AQHfa`k{M*G5wm)cdKEnn*p(epmlPVwO9PQ()gNkucUk4lYLCJsjIi#1s| zsLml_IaO7z5&DusMKV2VN2JRNmhT&-zN@>U`nkh9dnaGs7m0dds$~IR&^VQu;lj>9 zZNPq1>hno~wEp|p5CX!H!Mg-E#pdk=o(1B$8t8O!~ zJBSbLQfEiyW|{fU?0oB5!pFE{dWla;O;iJ~FuFs7|HRe5^c?vcC+XOlO6V2%Ai-@e z*FkHu_#R%{|K!mZILh~==NhVbnavYN>H|To!7xoR`s1`h8W#NW-1*|GBzMJq4G{IH zf%RE2{zr;!VpqjBv8%9kSKC}RTpeM#I#0k*e#cqOFY||dE#6aH`Z~^K6{hf<589Qv zGvDe=Df|@8_t)1H8|dtYaDYS*2e?$B*6?3 zBwoDqVEj9Zt0yI;Q<8LM^lypbo+=ersXeexDQf%24O`T~h0 z9=AYRc*<}F?4N$Z;9c69<5OY?4eQhO@dN5%%guqEg4bWY{Pf)jHOOvlbMvONCmMVd zxI33sGMR*&9g%s*fh;oWcw*J`}1kmID zMhUIhKW(3{d4_gPN?D|~dQv!r(On<<_g&ShJZxQ*`zQjtLYGiqUKvLPIjh%A_lod1 zrhGT|>Phj|Y+t*pgq#J-0P%69J^S699f!3c1$`kNSlcr|N$6~fe^u7hN)NwB(xbc; zVwBJp{|~+JH=+voIVI(c6^AX`Pwy#{B@gb?E2VD$ literal 0 HcmV?d00001