[Feature Request / Bug]: Support non-WAV / browser audio uploads (WebM/Opus, MP3, OGG) on `/v1/audio/transcriptions` for Open WebUI compatibility

[ampersandru]

Description

Currently, the POST /v1/audio/transcriptions endpoint strictly expects raw WAV PCM audio uploads (RIFF....WAVE header).

When integrating audio.cpp with popular web clients like Open WebUI, transcription fails because modern web browsers (Chrome, Edge, Firefox, Safari) using the HTML5 MediaRecorder API record microphone input in audio/webm;codecs=opus (or audio/ogg). Browsers do not support recording native WAV via MediaRecorder.

As a result, sending microphone recordings from Open WebUI to any ASR model (e.g. parakeet_tdt, granite5asr, whisper) fails with:
Error transcribing chunk: External: 500, message='Internal Server Error', url='http://:8080/v1/audio/transcriptions'

In contrast, clients that buffer and convert audio to WAV beforehand (such as Home Assistant via wyoming_openai or curl -F file=@sample.wav) succeed with exceptional sub-30ms latency.

---

Steps to Reproduce

1. Start audiocpp_server with any ASR model (e.g., parakeet_tdt or granite5asr):

   {
     "models": [
       {
         "id": "parakeet_tdt",
         "family": "parakeet_tdt",
         "path": "/models/parakeet-tdt-0.6b-v3-q8_0.gguf",
         "task": "asr",
         "mode": "offline"
       }
     ]
   }
   

2. In Open WebUI (Admin Settings ➔ Audio):

• Set Speech-to-Text Engine: OpenAI
• Set API Base URL: http://:8080/v1
• Set STT Model: parakeet_tdt

3. Click the microphone button in Open WebUI chat to record speech.
4. Open WebUI fails with HTTP 500.

Root Cause

handle_transcription_multipart in app/server/runtime.cpp checks is_wav_upload_filename and directly parses bytes using engine::audio::read_wav_f32(), rejecting .webm, .mp3, .ogg, and .m4a files.
When browsers send MediaRecorder chunks or blobs (often named audio.webm or blob), the server throws an unhandled exception or returns a 400/500 error instead of decoding the compressed audio stream.

Suggested Fixes & Solutions

Solution 1: Lightweight In-Tree Decoders (Recommended)

Embed header-only C/C++ audio decoders (similar to how ggml or llama.cpp bundle dependencies) so audio.cpp remains self-contained without external runtime dependencies:

miniaudio (Public Domain / MIT, single-file) — handles MP3, FLAC, WAV, and Vorbis decoding natively.
opusfile / libopus — decodes WebM/Ogg Opus streams directly to 16 kHz float32 PCM in memory.

Solution 2: Optional Server-Side Transcoder Bridge

If external tools are acceptable when present, allow audiocpp_server to detect and pipe incoming non-WAV multipart bytes through ffmpeg or libavcodec to convert to 16 kHz mono WAV PCM in memory before running inference.

Solution 3: Graceful Handling of Empty / Incomplete Chunks

When MediaRecorder begins recording, the initial chunk may sometimes be 0 bytes or header-incomplete. Returning {"text": ""} (HTTP 200) instead of throwing an unhandled exception ensures the client does not crash before the full audio is delivered.

[0xShug0]

I see the issue. Let me think about it. This could be part of the longer-term plan.

[ampersandru]

I see the issue. Let me think about it. This could be part of the longer-term plan.

thanks - another possible request, support wyoming protocol for direct compatibility with home assistant. Currently, I have to use https://github.com/roryeckel/wyoming_openai to connect any openai-compatible TTS/STT to home assistant. Having it directly incorporated somehow would be amazing!

[0xShug0]

The question I’m considering is not what to support, but how to support it. The core server should stay lightweight and avoid forcing users to compile unnecessary dependencies, while features that are better suited to the frontend or can be handled there easily should be implemented as extensions in the server. That will likely require some redesign of the server architecture. That’s why MP3 support (https://github.com/0xShug0/audio.cpp/tree/ci/new-feature-mp3) was never merged.

[arty-hlr]

I was trying out audio.cpp with open-webui and came across this issue as well. Maybe an extension would make sense or something optional if you don't want to add it by default to the core server @0xShug0 , and at least an option in the config to support this or that format. The docs can then show that the dependency for the extension needs to be installed to support other audio file types.

[arty-hlr]

In the meantime I wrote a quick proxy to convert to wav, maybe that's also useful for others until the feature is added here :)

https://github.com/arty-hlr/audio_to_wav_cpp

[arty-hlr]

As a side note, it seems that open-webui already converts webm to mp3, so audio.cpp would only need to support mp3 -> wav in this case.

2026-08-27 15:21:58.653 | INFO     | open_webui.routers.audio:transcription:1207 - file.content_type: audio/webm; codecs=opus
2026-08-27 15:21:58.655 | INFO     | open_webui.routers.audio:transcribe:1080 - transcribe: /app/backend/data/cache/audio/transcriptions/fcf757e8-b2cb-488e-8eac-540d64bfc9eb.webm None
2026-08-27 15:21:58.915 | INFO     | open_webui.routers.audio:convert_audio_to_mp3:160 - Converted /app/backend/data/cache/audio/transcriptions/fcf757e8-b2cb-488e-8eac-540d64bfc9eb.webm to /app/backend/data/cache/audio/transcriptions/fcf757e8-b2cb-488e-8eac-540d64bfc9eb.mp3
Chunk paths: ['/app/backend/data/cache/audio/transcriptions/fcf757e8-b2cb-488e-8eac-540d64bfc9eb.mp3']
2026-08-27 15:21:59.270 | INFO     | uvicorn.protocols.http.httptools_impl:send:485 - 192.168.178.22:0 - "POST /api/v1/audio/transcriptions HTTP/1.1" 200

[0xShug0]

In the meantime I wrote a quick proxy to convert to wav, maybe that's also useful for others until the feature is added here :)

https://github.com/arty-hlr/audio_to_wav_cpp

Thanks for the pointer! I'm thinking about the same direction. I feel the easier and clean solution is adding a "middleware" to the server which is opt-in,and decoupled from the server.

[IIIIIllllIIIIIlllll]

A preliminary suggestion: add a compile option to decide whether to enable the decoder, generate a standalone file, and let audiocpp_server check at runtime whether the decoder is present and reference it if so?

[0xShug0]

A preliminary suggestion: add a compile option to decide whether to enable the decoder, generate a standalone file, and let audiocpp_server check at runtime whether the decoder is present and reference it if so?

We definitely need a compile option if we go with the middleware/proxy route. But it should be transparent to the core server. I.e., the server should continue to operate on the same normalized audio input, while the optional middleware handles format detection/decoding before forwarding the request.