Skip to content

feat(audio): a decode backend seam that prefers the platform codec - #4131

Merged
kixelated merged 2 commits into
quest/m1/audio-codecs/READMEfrom
quest/m1/audio-codecs/decode-backend
Sep 25, 2026
Merged

kixelated merged 2 commits into
quest/m1/audio-codecs/READMEfrom
quest/m1/audio-codecs/decode-backend

Conversation

@kixelated

Copy link
Copy Markdown
Collaborator

Problem

moq_audio::decode::Decoder matched on the catalog codec inline, so there was no place for a platform decoder (AudioToolbox, Media Foundation, MediaCodec) to slot in ahead of the software ones. Kind::Auto and Kind::Software behaved the same, and Kind::Named matched codec names, which stop being unique once a second AAC decoder exists.

Approach

  • decode::backend mirrors moq-video's seam: a crate-private Backend trait (decode, reset, sample_rate, layout, delay, name) and an open that walks PLATFORM then SOFTWARE candidates, skips any that doesn't advertise the catalog AudioCodec, and refuses when none opens the track.
  • The existing Opus, PCM, and symphonia code moved behind the trait without behavior changes. PLATFORM is empty for now.
  • A backend reports the rate and layout it produces, not the catalog's. Consumer already resamples and remixes to the requested output.
  • The front end now trims startup delay for every backend (only Opus pre-skip is non-zero today). AudioToolbox priming reuses that path.
  • A single refusal keeps its error variant, so a malformed description is still Error::Mux. Several refusals join into one Error::Unsupported that names each backend. An unknown name lists what this build has for the codec.
  • moq play logs the audio decoder it opened, like it does for video.
  • Docs: doc/lib/rs/moq-audio.md gains the backend table and the Linux gap. doc/bin/cli.md no longer says audio uses the platform decoder. doc/bin/obs.md is unchanged: the OBS source decodes audio with FFmpeg, not libmoq.
  • Regression tests use stub candidates. They cover Auto preferring platform, falling back past a refusal, Software skipping platform, Named forcing one without falling back, aggregated refusals, and unknown and wrong-codec names.
  • Deletes the quest and its references. decode-audiotoolbox.md now says to warn when Auto falls past a platform refusal, and to report priming as the backend's delay.

Impact

  • decode::Decoder::name() and decode::Consumer::name() are new, mirroring moq_video::decode. Additive.
  • Maintainer call: Kind::Named now takes backend names "libopus", "pcm", "symphonia" instead of the codec names "opus", "pcm", "aac". The old names were never documented, but 0.1.2 accepted them. The type doesn't change, but a caller passing "opus" or "aac" is now refused. I left this as a draft for that decision.
  • Kind::Auto and Kind::Software docs now describe platform-first selection. They behave the same until a platform backend lands.
  • No wire, catalog, or FFI change. The FFI still doesn't expose Kind.

Alternatives

  • Keep the codec names ("opus", "aac"). That has no behavior change, but "aac" becomes ambiguous once AudioToolbox also decodes AAC, and it breaks moq-video's convention of naming by implementation ("openh264"). A hidden alias is ruled out as a compatibility shim.
  • Mirror video's flush: left out, because no backend here buffers output and the consumer never flushes audio. The first platform backend that holds frames can add it along with the consumer wiring.
  • Warn on a fallback past a refusing platform decoder, as video does: left out, because PLATFORM is empty and that path would be dead code. It's noted in the AudioToolbox quest.
  • A public NAMES list like video's: kept private until a consumer, such as CLI completion, needs it.

Follow-ups

  • chore(quest): abandon he-aac-refusal #4090 deletes he-aac-refusal.md on the same line. This PR only drops that file's links to the deleted quest, so resolve the modify/delete conflict by deleting the file.
  • Platform backends: decode-audiotoolbox.md (m1), and Media Foundation and MediaCodec (m2).

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 2 commits September 25, 2026 07:57
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated force-pushed the quest/m1/audio-codecs/decode-backend branch from e35bc4c to 4475984 Compare September 25, 2026 15:04
@kixelated
kixelated marked this pull request as ready for review September 25, 2026 15:04
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-25T15:08:58.781452Z 4475984 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated

Copy link
Copy Markdown
Collaborator Author

MERGE

Positive improvement: this is the right seam for platform audio decoders. Mirroring moq-video's candidate walk (platform then software, Kind to force one, name() for observability) unblocks AudioToolbox / Media Foundation / MediaCodec without another rewrite of the front end. Moving Opus, PCM, and symphonia behind the trait with stub-driven selection tests covers the important cases (prefer platform, fall back past refusal, Software skips platform, Named does not fall back, aggregated refusals, wrong-codec names).

Worth the complexity. The trait surface is small, PLATFORM is empty so there is no dead warn-on-fallback, and delaying a public NAMES list / flush until a consumer needs them keeps the blast radius tight. Backend-reported rate/layout with Consumer remix stays the correct contract for HE-AAC doubling later.

Different approach: keeping codec names for Kind::Named would avoid the soft break the PR already flags, but it becomes ambiguous the moment a second AAC decoder exists and fights the video convention. Prefer the backend names and treat any caller still passing \"opus\" / \"aac\" as the rare breakage the draft note calls out. Prefer merging once that call is accepted.

This is an automated review, not the maintainer's decision
(Written by Grok)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 44759844c3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread rs/moq-audio/src/decode/backend/mod.rs
@kixelated

Copy link
Copy Markdown
Collaborator Author

Rebased onto quest/m1/audio-codecs/README after #4119 (c0c78568) and squash-merging. just check passed locally.

The quest list keeps the TS export PCE entry and drops the finished Layout quest and this seam. layout.md and aac-pce.md stay deleted, and so does the abandoned he-aac-refusal.md. The platform-codec seam stays: decode::backend tries platform candidates before software, and moq play logs the decoder it opened.

Kind::Named takes backend names (libopus, pcm, symphonia). That refuses the undocumented 0.1.2 codec strings opus and aac. Left as-is on this quest line, not retargeted to dev.

Check and Test were still pending. This base has no required checks.

(Written by Grok 4.7)

@kixelated
kixelated merged commit c1cdb92 into quest/m1/audio-codecs/README Sep 25, 2026
3 checks passed
@kixelated
kixelated deleted the quest/m1/audio-codecs/decode-backend branch September 25, 2026 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant