Harden experimental Qwen3-TTS and unblock Android Vulkan

[leehack]

Current status

The original public-API work has shipped:

• feat: add typed text-to-speech support #330 added typed native TextToSpeechEngine support and the Qwen3-TTS chat-app workflow.
• feat: add WebGPU text-to-speech #341 added capability-gated WebGPU TTS with published bridge assets.
• Update native and Web llama.cpp pins to b10514 #342 and llamadart 0.8.20 updated the packaged native/Web runtimes and recorded representative speech validation, including physical-iPad CPU synthesis.

Qwen3-TTS remains experimental. CPU is the safe Android path.

Primary blocker: Android Vulkan

• Upstream llama.cpp issue Eval bug: [Vulkan] GGML_ASSERT failed in GET_ROWS with view offset (Qwen3-TTS) ggml-org/llama.cpp#26853 remains open.
• Proposed fix vulkan: fall back to CPU for GET_ROWS with misaligned offsets ggml-org/llama.cpp#26854 remains draft and unmerged.
• Published llamadart-native@b10545 does not contain that unmerged fix.

Remaining work

• Consume a published native artifact containing the upstream Vulkan fix.
• Re-run the pinned Qwen3-TTS model/projector on a representative Android device with explicit Vulkan, covering cold and repeated warm synthesis.
• Verify playback/WAV metadata, cancellation, speaker-reference synthesis, unload/reload, memory, latency and thermal behavior.
• Record broader Metal, multilingual, repeated-call and lifecycle evidence before reconsidering the experimental label.

Acceptance gates

• The Vulkan repair is merged upstream and included in an immutable llamadart-native release.
• Explicit Vulkan synthesis completes without abort, tombstone, corrupted audio or silent fallback.
• Cold/warm output, cancellation, cleanup and memory behavior pass on physical Android hardware.
• Unsupported/runtime-skewed paths continue to fail with typed actionable errors and preserve CPU fallback guidance.
• Documentation and the support matrix describe only the validated platform/backend combinations.

Typed STT and remaining speech-to-text validation are tracked separately in #325, #327 and #329.

[leehack]

Evidence update — 2026-08-11

The current investigation strengthens this issue but changes the implementation boundary:

• llamadart now pins llamadart-native@b10356, whose provenance resolves to llama.cpp 0666ad2b2b2452668733729e8b54234f5964643a.
• Published b10356 artifacts actually compile/package the Qwen3-TTS mtmd implementation and export mtmd_gen_audio_* plus all mtmd_helper_gen_audio_* functions. The llama-tts CLI and model files are not packaged.
• The available helper API is explicitly experimental. Current llama.cpp master has already changed its structs and mtmd_helper_gen_audio_step_gen() signature while adding PocketTTS:
  mtmd: support pocket-tts ggml-org/llama.cpp#26871
• Therefore public Dart support must not bind the raw helper as its stable contract. llamadart-native should first own a versioned llama_dart_speech_*/TTS C wrapper covering opaque lifecycle, capability/API version probes, sampler/hidden-state ownership, caller-owned PCM reads, cancellation, bounded backpressure, stable errors, reset, and repeated-call behavior.
• Qwen3-TTS b10356 produces 24 kHz mono PCM or PCM WAV and supports language plus optional speaker-reference audio. Initial real-model validation should start with CPU. Vulkan TTS remains unsupported while upstream issue #26853 is unresolved:
  Eval bug: [Vulkan] GGML_ASSERT failed in GET_ROWS with view offset (Qwen3-TTS) ggml-org/llama.cpp#26853
• Artifact publication and Dart binding/API work should remain separate approval and compatibility boundaries.

Typed STT/Qwen3-ASR work is tracked separately in #325. Shared audio metadata and task/capability semantics should remain compatible across the two APIs, while chat completion and speech output types stay separate.

[leehack]

Android Vulkan follow-up — Pixel 9 Pro, 2026-08-13

PR #330 shipped the typed TTS API and chat-app workflow, but Qwen3-TTS Vulkan remains blocked by upstream runtime correctness.

Exact device evidence:

• Pixel 9 Pro, Android 17.
• llamadart-native@b10356-llamadart.1 / llama.cpp b10356.
• Pinned Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf plus the pinned Q8_0 projector.
• The normal app policy resolved to CPU and synthesized a playable 1.2-second, 24 kHz mono WAV. A warm synthesis of “Hello from llama dart.” took about 54 seconds wall time.
• Forcing Vulkan reached the packaged Vulkan backend but aborted in DartWorker through ggml_abort → init_pushconst_tensor_offsets<vk_op_binary_push_constants> → ggml_backend_sched_graph_compute → clip_encode → mtmd_gen_audio_process → llama_dart_tts_step.
• The device was restored to Auto/CPU after the diagnostic; CPU remains the safe Android path.

This matches upstream llama.cpp #26853. The proposed fix is llama.cpp #26854, which is still open.

Before enabling or claiming Android Vulkan TTS:

• Upstream fix is merged into a tagged llama.cpp revision.
• llamadart-native publishes an artifact containing that revision.
• llamadart updates the native pin and preserves safe CPU fallback/version-skew behavior.
• The same pinned Qwen3-TTS model/projector passes cold and repeated warm synthesis on the Pixel with explicit Vulkan.
• Playback/WAV metadata, cancellation, speaker-reference synthesis, unload/reload, memory, latency, and thermal behavior are recorded.
• No abort, tombstone, corrupted audio, or silent fallback is observed.

Until these gates pass, keep Android Vulkan TTS unavailable/experimental and recommend CPU.

[leehack]

Implementation and upstream status — 2026-08-21

The original public-API design work is implemented:

• feat: add typed text-to-speech support #330 shipped typed native TextToSpeechEngine support and the Qwen3-TTS chat workflow.
• feat: add WebGPU text-to-speech #341 added capability-gated WebGPU TTS with published bridge assets.
• Update native and Web llama.cpp pins to b10514 #342 and llamadart 0.8.20 validated the current packaged native/Web speech surfaces, including physical-iPad CPU synthesis.

This tracker remains open for runtime hardening rather than API design. In particular, Android Vulkan Qwen3-TTS is still blocked:

• upstream issue Eval bug: [Vulkan] GGML_ASSERT failed in GET_ROWS with view offset (Qwen3-TTS) ggml-org/llama.cpp#26853 remains open;
• proposed fix vulkan: fall back to CPU for GET_ROWS with misaligned offsets ggml-org/llama.cpp#26854 remains open and unmerged;
• published llamadart-native@b10545 follows llama.cpp b10545 and does not make that unmerged fix consumable.

Keep CPU as the safe Android path. The remaining gate is a published native artifact containing an upstream Vulkan fix, followed by exact-model cold/warm device validation with no abort, silent fallback, corrupted audio, or lifecycle leak. Broader Metal, speaker-reference, multilingual, cancellation, repeated-call, memory, and thermal evidence should continue to be recorded before the experimental label is reconsidered.