qvac-fabric-speech.cpp

On-device speech and audio AI in C++17 on ggml: speech-to-text, speaker diarization, end-of-utterance detection, text-to-speech, voice cloning, sound effects, speech-to-speech, enhancement, and music generation.

Property	Value
Build	Feature-gated CMake superbuild over Whisper and three native engines
Runtime	No Python, PyTorch, NeMo, or ONNX Runtime at inference time
Model formats	Whisper/Silero GGML .bin; Parakeet, TTS, and AudioGen GGUF
Desktop	Linux, macOS, Windows
Mobile	Android arm64-v8a, iOS arm64; model and backend coverage varies by engine
Backends	CPU, Metal, Vulkan, OpenCL, CUDA, WebGPU (Dawn, opt-in; see WebGPU); selected models also support explicit Hexagon placement or optional Apple Core ML sidecars
Shared ggml	One ggml-speech dependency from qvac-ext-ggml@speech

Supported models

The engine guides own model formats, quantization, backend validation, and input requirements. An available ggml backend does not imply every model has been validated on it.

Engine	Model families	Capabilities and limitations
Whisper	tiny, base, small, medium, large v1/v2/v3, large-v3-turbo; Silero v5.1.2/v6.2.0	ASR, multilingual-to-English translation, VAD, tinydiarize speaker turns; metadata-only memory fit from a model or its weightless description; .en checkpoints are English-only and turbo is intended for transcription; tiny/base/small/large-v3-turbo validated on WebGPU
Parakeet	CTC, Unified RNN-T, TDT, TranscribePsy, IndicConformer, realtime EOU, Nemotron 3.5 ASR	Offline, streaming, long-form ASR and end-of-turn detection; IndicConformer CTC requires a language selection; TranscribePsy ships as ready-made GGUF and Core ML files for eight European languages and runs on the Hexagon NPU
Parakeet	Sortformer v1/v2/v2.1, Nemotron 3 Diarization	Speaker diarization and attributed ASR; Sortformer supports up to four speakers, Nemotron up to eight
Parakeet	MOSS-Transcribe-Diarize	One-pass transcription, speakers, timestamps, hotwords and metadata-only memory fit including scheduler storage and CPU scratch; fused CPU encoder attention; CPU/Metal validated on Spanish/Chinese; Metal decode reuses one graph per cache window (29-42% faster decode on M3 Ultra, identical transcripts); explicit CUDA/Vulkan/OpenCL/Metal/Hexagon selection with generated-model CPU parity and quantized checkpoint smoke coverage; English long-form skips spans in both native and reference runs
TTS	Chatterbox Turbo/Multilingual, Supertonic 1/2/3, Parler mini/large/Indic, CosyVoice3, Audio8	Synthesis with preset, described, or cloned voices, depending on the model; streaming support varies by API and CLI; Supertonic 3, Parler mini, CosyVoice3 and Audio8 also validated on WebGPU
TTS	Pocket TTS	English synthesis and incremental CPU/Metal/CUDA/Vulkan/OpenCL streaming; cloning requires encoder-enabled weights
TTS	MOSS-TTS-v1.5 / MOSS-TTSD	Multilingual synthesis, cloning, dialogue and streaming; single-reference dialogue avoids duplicate reference encoding; selective text-control projection on Metal; batched audio-logit readback on Vulkan; CPU/Metal validated; CUDA/Vulkan generated-model parity and synthesis/streaming/cancellation checks (see scoped validation in the TTS guide); explicit Hexagon NPU placement with a q4_0 transformer; numerical reference parity pending
TTS	MOSS-SoundEffect-v2, MOSS-Speech	Sound-effect generation and spoken replies respectively; SoundEffect reuses fixed cross-attention K/V, selects supported Metal DiT flash attention, and runs quantized matrices from ggml's repacked CPU layout, with the cache and retained scheduler peak included in metadata-only memory fit for repeated requests; CPU/Metal generation validated, including a q8_0 GGUF with q4_0 DiT blocks; CUDA/Vulkan generated-model parity and quantized SoundEffect/Speech checkpoint smoke coverage; explicit Hexagon NPU placement for SoundEffect, with reference parity, and for the Speech LM and speech tokenizer
TTS	LavaSR denoiser/enhancer	Speech denoising and bandwidth extension; GPU selection includes ARM Mali for both stages and excludes WebGPU, whose kernels lack an op the graphs need
AudioGen	ACE-Step v15 turbo/sft/base	Music generation and editing; base additionally supports lego stems
AudioGen	MiniMax-Music3	Desktop music generation on CPU or GPU, optional Core ML conditioner projection, and metadata-only memory-fit preflight; unavailable on Android/iOS

MOSS cloning matches upstream reference loudness preprocessing and uses the TTSD-specific dialogue prompt. The TTSD example sets its model-specific sampling parameters explicitly; check generated speech for complete dialogue rather than relying on a successful exit alone. An optional validated cloning runner checks English dialogue, including each requested turn, with local ASR and retries rejected generations before publishing an output WAV. It supports CPU, CUDA, and Vulkan (--gpu --backend vulkan). ASR results that reach the decoding token limit are rejected.

MOSS generation starts at the reference assistant header and converts ISO language codes such as en to the model's language names (English). The full-checkpoint TTS CI gate requires every expected CPU/GPU batch, streaming and reuse output to pass WER <= 0.25, with no baseline-audio exemption for wrong text. See MOSS validation.

CosyVoice3 uses capability-checked flash attention for single-token LM decoding on CUDA and Metal. Supported CUDA backends reuse decode graphs within masked 128-token cache windows, allowing CUDA graph capture. Unsupported backends or head dimensions retain their existing attention path.

CosyVoice3 flow uses F16 key/value operands with F32 accumulation for capability-checked Vulkan flash attention, falling back to F32 operands or regular attention when required.

CosyVoice3 expands bf16 flow weights to f32 on ARM CPUs to avoid scalar bf16 matmuls; f16 remains the recommended ARM flow bundle. Its benchmark supports independent LM, flow and HiFT model overrides for stage profiling.

Nemotron 3 Diarization preparation supports additional weight precisions via convert_nemotron_diarization.py --outtype. CPU/CUDA offline validation covers F16, BF16, Q6_K, Q5_K, Q4_K, and NVFP4. Q4_K/Q6_K also pass offline Metal and Vulkan validation; NVFP4 runs on Vulkan but falls back to CPU matmuls on Metal. Streaming quality varies by precision and backend. See the quantization measurements before selecting a format. MXFP4 failed CPU quality checks.

Experimental Parakeet-Mosaic

Mosaic uses a frozen mixed-precision recipe with ordinary GGML kernels. The shared 486.7 MB Mosaic and 339.6 MB Mosaic Compact also support separate, matching optional Core ML encoder sidecars. The earlier experimental parakeet-affine-v1 format uses integer carriers, with reconstruction by default and opt-in Metal fusion via PARAKEET_AFFINE_FUSED=1. Fusion requires the matching GGML dependency; older dependencies and unsupported fused backends fail explicitly. Offline TDT is the evaluated scope; packed-affine substage profiling is rejected. Recipe receipts and measured limitations are documented in the linked guides.

Fixed TDT Core ML exports now require both mel and mel_valid_mask, and need the updated runtime. Legacy single-input sidecars still load but must be re-exported for padding invariance. See the sidecar compatibility contract.

Apple Core ML sidecars

Optional sidecars accelerate a stage while the rest of the pipeline stays on ggml. They default to disabled and have model-specific input and fallback contracts; see the engine guides before exporting or deploying one.

Engine	Accelerated stage	Guide
Whisper	Encoder	Whisper Core ML
Parakeet	Eligible ASR and Sortformer v2.1 encoder paths; TDT models also run a fixed-window Neural Engine encoder and a batched TDT decoder	Backend and routing contracts, Neural Engine sidecars
Parakeet	Nemotron 3 Diarization and Sortformer v2.1 whole networks	Diarizer sidecars
Parakeet	MOSS-Transcribe-Diarize audio encoder	MOSS-Transcribe-Diarize
TTS	Chatterbox Turbo and Multilingual S3Gen encoder, CFM and HiFT (overlapping the next sentence's T3)	Chatterbox Core ML
TTS	CosyVoice3 speech LM, flow and HiFT neural stages	CosyVoice3 Core ML
TTS	Supertonic vocoder, Audio8 codec synthesis	Supertonic, Audio8
TTS	LavaSR denoiser (batch prediction) and enhancer neural cores, with explicit model paths and compute policies	LavaSR Core ML
TTS	MOSS-TTS/TTSD codec decoder, MOSS-SoundEffect DiT and VAE, MOSS-Speech speech tokenizer	MOSS, SoundEffect, Speech
AudioGen	ACE-Step VAE decoder	AudioGen backends
AudioGen	MiniMax conditioner projection, or explicit full-network Core ML (coreml_profile_dir + coreml_compute_units, macOS 15+)	MiniMax Core ML

MiniMax's full Core ML route is also exposed through its public C++ engine. Build with AUDIOGEN_MINIMAX_COREML=ON and supply both coreml_profile_dir and coreml_compute_units; the separate flag does not enable automatic ACE-Step sidecars. It supports CPU_ONLY, CPU_AND_GPU, CPU_AND_NE, and ALL on macOS 15+. See the full Core ML guide for the matching F16 compiled profile, measurements, and remaining numerical validation.

CosyVoice3 exports HiFT in FP32 to preserve waveform parity. Its memory-fit preflight returns Error with coreml-memory-estimate-unavailable when enabled sidecars are present, rather than undercounting their resident memory. Re-export CosyVoice3 LM sidecars with the current exporter to preserve cache feedback on macOS 15 and 26; see the Core ML guide.

Hexagon NPU (Snapdragon)

Selected models run on the Hexagon HTP accelerator via --backend hexagon (CLI) or the engine's backend = "hexagon" option (API), end to end except for MOSS-Speech, whose reply decoder stays on the CPU or the GPU. Each guide names the validated quantisation tier, the runtime env flags that recover the measured speedup, and how the backend compares to CPU and OpenCL on the same device.

Engine	Models	Guide
ASR	Parakeet CTC and TDT 0.6B (q8_0), TranscribePsy-250M-EU-sub (q8_0), MOSS-Transcribe-Diarize (q8_0, f16)	Parakeet Hexagon, MOSS Hexagon
TTS	Chatterbox Turbo and Multilingual (q4_0, q8_0, f16 or f32), Supertonic 3, Parler-TTS mini, CosyVoice3, Audio8, MOSS-TTS-v1.5 / MOSS-TTSD (q4_0 transformer), MOSS-SoundEffect-v2 (f16, q8_0, or q8_0 with q4_0 DiT blocks), MOSS-Speech LM (q4_0 transformer) and speech tokenizer (f16 codec)	Chatterbox Hexagon, Audio8 Hexagon, MOSS Hexagon, MOSS-SoundEffect Hexagon, MOSS-Speech Hexagon, and the matching sections in the other model guides
AudioGen	ACE-Step v15 turbo (q8_0 DiT; K-quant GGUFs are refused at load)	AudioGen Hexagon

WebGPU (Dawn)

GGML_WEBGPU=ON (the webgpu feature of the speech-cpp / ggml-speech ports) adds ggml's WebGPU backend on Google Dawn, driven natively inside the process; no browser is involved. It registers as WebGPU, ranks with Vulkan and Metal in every engine's automatic GPU selection (CUDA still wins on the same NVIDIA card), and is selected explicitly with --backend webgpu or backend = "webgpu". Build and host requirements are in docs/BUILD.md. Validation on an RTX 5090 (Dawn on Vulkan; 2026-10-20, the LM-based TTS models and large-v3-turbo 2026-10-26), each model against the same CPU run:

Engine	Model	WebGPU result
Whisper	tiny.en, base, small	Validated: same transcripts; encoder 8x (base) to 33x (small) faster than CPU, a 2-minute clip 3.1x faster end to end
Whisper	large-v3-turbo	Validated: same transcripts; a 2-minute clip encodes in 0.27 s against 23.4 s on the CPU, 11x faster end to end
Parakeet	TranscribePsy 250M TDT	Validated: identical transcript, 2.8x faster end to end
TTS	Supertonic 3	Validated: waveform correlation 0.997 with the CPU render, identical transcript
TTS	Parler-TTS mini (f16, q8_0)	Validated: the same Whisper transcripts as the CPU render over four prompts; 3.5x faster end to end
TTS	CosyVoice3 (q8_0 LM, f16 flow and HiFT)	Validated: the same Whisper transcripts as the CPU render over four prompts; 12x faster end to end
TTS	Audio8 (q8_0 and f16 LM, f16 codec)	Validated: the same Whisper transcripts as the CPU render over four prompts; 5x faster end to end
TTS	Pocket, Chatterbox, MOSS, LavaSR	Not tested
AudioGen	ACE-Step, MiniMax-Music3	Not tested; the ACE-Step LM and detokenizer keep their CPU placement on an unmeasured backend

Parler, CosyVoice3 and Audio8 sample their tokens, so their WebGPU and CPU renders diverge after the first differing token the way any two backends do; the transcripts are the check. They rely on ggml-webgpu computing the matmul and flash-attention ops they mark GGML_PREC_F32 with f32 activations and f32 accumulation, which ggml-speech provides from 2026-10-27; with an older ggml-speech their decoders overflow f16 and produce NaN logits, silence or noise.

Getting started

Install a shared speech-branch ggml using docs/BUILD.md, then configure from this checkout with its install prefix:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH=/path/to/ggml-install
cmake --build build --parallel

For a first transcription on a single-config build:

bash third_party/whisper.cpp/models/download-ggml-model.sh base.en
./build/bin/whisper-cli -m third_party/whisper.cpp/models/ggml-base.en.bin \
  -f third_party/whisper.cpp/samples/jfk.wav

The build guide provides Windows and multi-config commands. Use docs/CLI.md to choose a tool, and the engine's model guide to acquire its required weights.

Performance

audio8-codec-bench can decode captured codes on CPU, Hexagon or OpenCL and export WAV/raw F32 PCM while measuring the codec separately from generation. See the fixed-code decoder benchmark.

RTF = inference_time / audio_duration; lower is faster. These are dated measurements with different workloads and timing boundaries, not universal speed guarantees. Full methods, build pins and limitations live in docs/PERFORMANCE.md and the linked engine reports; benchmark coverage lists which model each desktop benchmark family measures and scores.

Task	Model	Recorded measurements
ASR	Parakeet TDT 0.6b v3	September 2026 GPU lanes: short-clip RTF 0.0007–0.0055; method and full table
TTS	Supertonic 3	September 2026 GPU lanes: process wall 0.61–0.82 s, including model load; full table
TTS	Audio8	September 2026 GPU campaign: RTF 0.20–0.65, with later RTX 5090 measurements down to 0.116; full table and follow-up
Music	ACE-Step 1.5	September 2026 CUDA/Vulkan/Metal lanes: generation 1,338–9,016 ms, RTF 0.139–0.957; full table
Music	MiniMax-Music3 f16	September 2026 RTX 5090: two-minute song in 73.9 s CUDA / 84.2 s Vulkan, excluding model load; full table

Memory fit results

Every engine's memory-fit preflight reports the same device fields, so a caller can add up several models that share one device:

• device_name is the ggml device name, as ggml_backend_dev_by_name finds it: CPU, CUDA0, Vulkan0, MTL0, GPUOpenCL, HTP0. Two engines on the same device report the same name.
• device_shares_host_memory is true for a device that draws from host RAM: the CPU, an integrated GPU, Apple Metal, Hexagon and Android's OpenCL GPUs.
• device_free_bytes and device_total_bytes are the device's own figures. A device that shares host memory and reports none of its own, such as Hexagon, carries the CPU's figures.

Use in QVAC

These engines ship inside QVAC as SDK addons consuming the speech-cpp vcpkg port. For JavaScript/TypeScript APIs, Bare runtime integration, and desktop/mobile applications, see QVAC.

QVAC addon	Wraps	speech-cpp features
@qvac/asr-ggml	ASR, diarization, end-of-utterance	whisper, parakeet
@qvac/tts-ggml	Synthesis, voice cloning, enhancement	tts
@qvac/audiogen-ggml	Music generation	audiogen
@qvac/bci-whispercpp	Brain-computer interface transcription	whisper

Licenses

Code and model weights have separate terms. No model weights are shipped in this source tree. Engine license sections and NOTICE files identify sources and model-specific exceptions.

Component	Code	Model weights
Whisper	MIT	OpenAI Whisper MIT; Silero under its model terms
Parakeet	Apache-2.0	NVIDIA Parakeet/Sortformer CC-BY-4.0; EOU NVIDIA Open Model License; Nemotron OpenMDW-1.1; IndicConformer MIT; MOSS-Transcribe-Diarize and TranscribePsy Apache-2.0
TTS	MIT	Chatterbox MIT; Pocket CC-BY-4.0; Supertonic OpenRAIL-M; Parler, CosyVoice3, Audio8, LavaSR and MOSS models Apache-2.0
AudioGen	MIT	ACE-Step 1.5 MIT; Qwen3-Embedding Apache-2.0; MiniMax-Music3 Community License

Documentation

Parakeet's test guide separates model-free unit tests from fixture coverage. Its fixture runner rejects runs with no executed tests and can require specific model-family tests to pass; the opt-in GPU CI lane uses this gate.

Topic	Where
Architecture and repository layout	docs/ARCHITECTURE.md
Building and consuming packages	docs/BUILD.md
CLI selection and examples	docs/CLI.md
Performance overview and reports	docs/PERFORMANCE.md
Whisper models and formats	docs/WHISPER.md
Native Whisper Snapdragon build and validation	docs/WHISPER-HEXAGON.md
Whisper subtree deltas and synchronization	PATCHES.md, docs/UPSTREAM-SYNC.md
ASR, diarization, end-of-utterance	Parakeet
Synthesis, voice cloning, sound effects, speech-to-speech, enhancement	TTS
Music generation and editing	AudioGen
Benchmark quality diagnostics and model preparation	Music alignment guide
Historical development and integration reports	Chatterbox, Supertonic, Pocket, Parakeet, Strix Halo

Audio8 exposes incremental PCM through Engine::synthesize_stream(), with codec-window delivery on ggml and Core ML. See the streaming API and measurement guide.