Skip to content

feat(qta): read Apple Voice Memos recordings (#646) - #762

Merged
KnifeLemon merged 1 commit into
KnifeLemon:mainfrom
SeoJCalive:feature/qta-voice-memos
Oct 29, 2026
Merged

KnifeLemon merged 1 commit into
KnifeLemon:mainfrom
SeoJCalive:feature/qta-voice-memos

Conversation

@SeoJCalive

Copy link
Copy Markdown
Collaborator

Closes #646

What it does

  • .qta is a read-only audio format (FormatRegistry: qta, audio, Writable: false). Apple Voice Memos saves some recordings as .qta, which is QuickTime audio, so FFmpeg's mov demuxer reads it and it goes to every audio target (MP3, M4A, WAV and the other 19) in one FFmpeg step, like DSS or APE. Nothing writes .qta.
  • Audio FFmpeg can't decode is skipped. Public reports say iOS 26 memos carry an AAC stereo track and a 4-channel APAC (Apple Positional Audio Codec) track. FFmpeg 9.0.2 knows apple_apac but has no decoder for it, so if that track came first, -map 0:a:0 would fail with no decoder found for: apple_apac. Now MediaInfo keeps only the audio streams FFmpeg can decode and remembers each one's place among the file's audio streams (AudioStreamInfo.Ordinal); every place that mapped 0:a:0 / [0:a:0] (audio targets, multi-track video targets, clips, clip preview audio, joining, live photo) maps that number instead.
    • Which codecs decode comes from ffmpeg -codecs (the D flag) of the same FFmpeg Filee runs, cached per executable like the encoder list. If the query fails or a stream has no codec name, nothing is skipped (today's behaviour).
    • I didn't use "ffprobe has no sample_fmt" as the signal: an AAC track that starts late in an MPEG-TS has no sample_fmt either, yet decodes fine. A test pins that case.
    • Files whose audio all decodes get the same FFmpeg arguments as before (the number is 0, 1, … as now). If no audio decodes, nothing is skipped and the conversion fails with FFmpeg's message as today.
    • Joining copies streams only when every file's chosen track has the same number; otherwise it goes through the existing per-file encode, so a later file's sound isn't lost.
  • Several decodable tracks: audio targets still take the first one, so a recording with a second audio track, a text track or a timecode track converts its first audio track only.
  • Settings schema 55: existing libraries' audio donut learns .qta; a .qta the user put on their own donut stays there. (The issue comment said 49; main had already reached 54.)
  • Docs: docs/ENGINES.md mentions .qta and the skipped APAC track. The README count goes 268 → 269 in all 15 READMEs.
  • No new strings, so no i18n changes.

Not checked with a real recording

I couldn't get a real .qta: Voice Memos recordings I made on both a Mac and an iPhone came out as .m4a, and no sample was attached to the issue. So the APAC part is tested with a file built in the test: a normal two-track AAC QuickTime file whose first track is relabelled apac (sample entry) with its esds box renamed, which ffprobe then reports as apple_apac and FFmpeg refuses to decode, the same error a real APAC track gives. If anyone has an iOS 26 Voice Memos .qta, a quick conversion to MP3 would confirm it.

Tests

  • QtaFormatTests (Core): registry entry and audio profile; the 54 → 55 library step runs once; a library saved at schema 54 learns .qta on the next start and doesn't learn it again after the user removes it; a user's own donut with QTA keeps it.
  • MediaTests (with FFmpeg): a .qta reaches every audio target in one step; a QuickTime AAC .qta converts to MP3, WAV and M4A; the first of several decodable tracks is used; an APAC-first .qta converts to MP3, WAV and M4A from the AAC track (1 channel, 1 s, not the 4-channel 2 s APAC one); an APAC-only file fails with the codec name and writes nothing; a video with AAC then APAC converts to MKV with only the AAC track; the late-start MPEG-TS keeps its first AAC.
  • Argument tests in AudioTracksTests, ClipTests, MediaJoiningTests and LivePhotoCreationTests check the chosen number reaches each mapping, the output option numbers stay 0, 1, …, and joining refuses to copy when the numbers differ; MediaToolRealTests joins two real clips like that and checks the sound lasts the whole result. HardwareEncodingTests parses the codec list and checks a missing or hanging FFmpeg gives up.
  • Each new check was confirmed by breaking what it guards, one change at a time. These included: letting every audio stream through, using the stream's absolute index or 0:a:0 at each mapping, numbering output options by input track, dropping the joining gate, going back to the sample_fmt rule, ignoring the D flag, and treating a failed query as "nothing decodes". The matching test fails every time. The one exception is the ffplay preview call in ClipPreview, which passes the same number but has no test that runs ffplay.

Ran locally (Linux, FFmpeg 9.0.2 from build/fetch-engines.ps1), compared with the same commands before the change:

  • The Release build has the same 15 warnings, and dotnet format --verify-no-changes is clean.
  • Filee.Core.Tests 728 passed and Filee.Cli.Tests 41 passed.
  • Filee.Engines.Tests for the media area (Media, AudioTracks, Clip, LivePhoto, HardwareEncoding): 328 passed, 4 skipped (Windows-only or hardware encoders). The one failure is the same on main (Alac_speex_and_dts_are_written_and_read: this FFmpeg build has no libspeex encoder).
  • Filee.App.Tests 1541 passed / 9 skipped. The one failure is the same on main (Engines_page_shows_versions_descriptions_and_download_speed, which depends on the engines installed on this PC).
  • CLI filee-cli convert of the APAC-first .qta to MP3, WAV and M4A: each exits 0 with one 1-channel 1.0 s audio stream.

Register .qta as a read-only audio format: it is QuickTime audio, so
FFmpeg's mov demuxer reads it and it goes to every audio target in one
step. Settings schema 55 teaches existing audio donuts the extension.

iOS 26 memos are reported to pair an AAC track with a 4-channel APAC
track FFmpeg has no decoder for. MediaInfo now drops audio streams whose
codec lacks the D flag in `ffmpeg -codecs` of the FFmpeg in use (cached
per executable; nothing is dropped when the query fails or no stream
decodes) and keeps each stream's audio ordinal, which every mapping
uses instead of 0:a:0. Files whose audio all decodes get the same
arguments as before. Joining copies only when every file's chosen
ordinal matches.

Closes KnifeLemon#646
@github-actions github-actions Bot added the feature New features label Oct 29, 2026

@KnifeLemon KnifeLemon left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @SeoJCalive! Reading .qta as QuickTime audio, and mapping the first track FFmpeg can actually decode, both look right. Every call site in the diff uses the track's place among all audio tracks, the -codecs parsing skips encoder-only and identify-only entries, and the migration follows the v48/v50/v54 steps. Merging.

A few things for a follow-up, none blocking:

  1. Three places still map 0:a:0, although the description says all of them changed. Today's behaviour, so not a regression:

    • ClipPreview.cs:158: the waveform (ReadLoudnessAsync)
    • ClipPreview.cs:200: SilenceArguments
    • MediaEncoding.Clip.cs:125: clip music [1:a:0] (ClipMusic doesn't carry the track number)

    With an APAC-first file the waveform comes out empty, and "remove silence" or adding it as music fails.

  2. The first ffmpeg -codecs can run on the UI thread: MediaInfo.ProbeAsync calls DecodableCodecsOf synchronously after an await without ConfigureAwait(false). Callers from the clip window (VideoFrames.ProbeAsync, ClipPreview.ReadSoundLengthAsync) can freeze the first open for as long as the query takes.

  3. The codec cache: one timeout caches null for the whole session, and GetOrAdd can start several ffmpeg -codecs at once on a batch. Same pattern as EncodersOf, so fixing both together makes sense.

  4. A stream without codec_name counts as decodable. When the decodable set is known, it could go with the undecodable ones.

  5. docs/ENGINES.md: the audio-track line ("Video to audio takes the first") and Merge videos could mention that tracks FFmpeg can't decode are skipped.

Schema note for whoever merges next: this takes 55, so #705 moves to 56.

@KnifeLemon
KnifeLemon merged commit 42a540f into KnifeLemon:main Oct 29, 2026
9 checks passed
@SeoJCalive

Copy link
Copy Markdown
Collaborator Author

@KnifeLemon Thanks for the review and the merge! You're right on all five, and the description overstated item 1: those three spots still map 0:a:0. I'll do them together in one follow-up PR:

  1. Waveform, silence and clip music: pass the chosen track number through ReadLoudnessAsync, SilenceArguments and ClipMusic, so an APAC-first file gets a waveform and works with "remove silence" and as music.
  2. First -codecs off the UI thread: run the lookup off the caller's context so the clip window doesn't freeze on its first open.
  3. Codec and encoder caches: don't keep a timeout for the whole session, and run one query per executable even when a batch probes at once, for DecodableCodecsOf and EncodersOf both.
  4. Streams without codec_name: treat them as undecodable when the decodable set is known.
  5. docs/ENGINES.md: note in the audio-track line and in Merge videos that tracks FFmpeg can't decode are skipped.

I'm starting on it now.

@SeoJCalive
SeoJCalive deleted the feature/qta-voice-memos branch October 29, 2026 12:56
KnifeLemon added a commit that referenced this pull request Oct 30, 2026
fix(qta): follow-up to the #762 review (track choice in clips, codec query caching)
rohitLovesKrishna pushed a commit to rohitLovesKrishna/Filee that referenced this pull request Oct 30, 2026
Waveform, silence detection and clip music take the first audio track
FFmpeg can decode. The codec and encoder list queries run off the
caller's thread, run once per executable even when callers overlap, and
are asked again after a failure. Audio without a codec name counts as
undecodable when the decodable set is known. ENGINES.md mentions the
skipped tracks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Format request: .qta

2 participants