Skip to content

v2: Parakeet and Qwen3-ASR engines, richer reply suggestions, UI fixes - #4

Merged
Yalhash merged 9 commits into
mainfrom
v2
Nov 5, 2026
Merged

Yalhash merged 9 commits into
mainfrom
v2

Conversation

@Yalhash

@Yalhash Yalhash commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

Summary

v2 adds two local speech engines that run without NVIDIA hardware, gives reply suggestions better context plus user-written instructions, and fixes three UI issues found while testing.

Speech engines

  • asr.engine picks the recognizer: whisper (default), parakeet or qwen3. Settings → Speech has a "Speech engine" dropdown that shows only the chosen engine's options.
  • Parakeet TDT 0.6B v3 runs on ONNX Runtime through onnx-asr. It covers 25 European languages and is fast on a CPU (int8). It uses CUDA or DirectML automatically when that ONNX Runtime build is installed. It doesn't report the language it heard, so the transcript carries the locked or configured language.
  • Qwen3-ASR (0.6B default, 1.7B option) runs on PyTorch through qwen-asr. It covers 30 languages, uses float32 on a CPU and bfloat16 on a CUDA GPU, and detects the language.
  • Both are optional extras: uv sync --extra parakeet / --extra qwen3. Each is imported only when its model loads. A missing package is a ConfigError that names the install command. Whisper stays installed because Silero VAD ships inside faster-whisper.

Suggestions

  • Context after the target: when a reply hotkey answers an earlier sentence, the suggester now also gets what was said after it (suggestions.history_after, default 4). history_chars covers both sides, filled nearest-first.
  • Named target: the prompt is a numbered transcript with the target marked >>> and named by number. Later lines are labelled context only, so the reply must still fit the conversation now and must not repeat what the user already said.
  • Extra instructions: suggestions.instructions (Settings → Suggestions → Extra instructions, up to 1000 characters) is added to the system prompt. Fixed rules (reply language, output format, no invented facts) win over it. It applies live, like the profile: changing only the instructions no longer rebuilds the pipeline (config.needs_rebuild / LIVE_FIELDS).

Fixes

  • Error stays visible: a model-load error stayed on screen only for a moment, because the automatic stop replaced it with "Stopped". A fatal stop now keeps the ERROR status and its message; the next Start or a manual Stop clears it.
  • Duplicate status text: "Loading speech model…: loading speech model" now reads "Loading speech model…".
  • Settings layout: tabs pack their rows at the top. The Model + Refresh row no longer stretches to fill short tabs.

Testing

  • Unit, pipeline and adapter tests cover every change. The Parakeet and Qwen3 adapters are tested against stand-ins for their libraries, so they run without the extras installed.
  • UI tests cover the engine selector, the instructions field, and row heights in a tall dialog.
  • Slow tests (pytest -m slow) run the real Parakeet and Qwen3 models on synthesized speech, and skip when the extras aren't installed. CI's models job doesn't install them, so the real models are untested in CI.
  • CI passes on Windows, macOS and Linux (Python 3.10 and 3.12) on the latest commit, including the UI tests.

https://claude.ai/code/session_0178pC5pfGXepbgU7JEpqVUh


Generated by Claude Code

Yalhash and others added 9 commits November 5, 2026 03:33
asr.engine picks the local recognizer: whisper (default), parakeet or qwen3.
Neither new engine needs NVIDIA hardware:

- Parakeet TDT 0.6B v3 via onnx-asr on ONNX Runtime: int8 on the CPU by
  default; uses CUDA or DirectML automatically when that onnxruntime build
  is installed. 25 European languages. It does not report the language, so
  the transcript carries the locked/configured one.
- Qwen3-ASR (0.6B default, 1.7B option) via qwen-asr on PyTorch: float32
  on the CPU, bfloat16 on a CUDA GPU. 30 languages, language detected.

Both are optional extras (uv sync --extra parakeet / --extra qwen3) and
are imported only when their model loads; a missing package is a
ConfigError naming the install command. Settings > Speech gains an engine
selector that shows only the chosen engine's options.

Tests: adapter tests against fake libraries, wiring/config tests, a
settings-dialog test, and slow real-model tests that skip when the
extras are not installed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0178pC5pfGXepbgU7JEpqVUh
When the speech model failed to load, the pipeline stopped capture in the
background, and that stop published a plain STOPPED status, which
replaced the error a moment after it appeared. A fatal stop now leaves
the status at ERROR with its message. ERROR is not a listening state, so
the window still offers Start; the next Start (or a manual Stop) clears it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0178pC5pfGXepbgU7JEpqVUh
The LOADING status already shows "Loading speech model…"; the pipeline
also sent "loading speech model" as its detail, so the window read
"Loading speech model…: loading speech model". The detail is now empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0178pC5pfGXepbgU7JEpqVUh
A reply hotkey can answer an earlier sentence, but the suggester only saw
what came before it, and was told "They just said" as if it were the
latest. Now:

- SuggestionContext.following holds what was said after the target
  (both speakers, finals only), up to suggestions.history_after
  utterances (default 4). The history_chars budget covers both sides,
  filled nearest-first, so the closest context survives a tight budget.
- The prompt is one numbered transcript with the target marked ">>>" and
  named by number ("Suggest replies to message [3]"). When later lines
  exist, they are labelled context only: the reply must answer the
  target, still make sense now, and not repeat what the user already said.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0178pC5pfGXepbgU7JEpqVUh
suggestions.instructions (Settings > Suggestions > Extra instructions,
up to 1000 characters) is added to every suggestion prompt, after the
profile, for things like tone, length, or "don't ask questions back
unless the conversation needs one". The prompt tells the model to follow
them except where they conflict with the fixed rules (reply language,
output format, no invented facts).

Like the profile, it applies live: changing only the instructions no
longer rebuilds the pipeline (which stopped listening and reloaded the
speech model). config.needs_rebuild ignores fields listed in LIVE_FIELDS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0178pC5pfGXepbgU7JEpqVUh
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0178pC5pfGXepbgU7JEpqVUh
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0178pC5pfGXepbgU7JEpqVUh
Each tab set its QFormLayout directly on the page, so when the dialog
(as tall as its tallest tab) had spare height, Qt gave it to rows that
could grow. The Model + Refresh row stretched to fill the Suggestions
and Translation tabs, leaving "Model" far above its field.

Pages now pack their rows at the top with a stretch below, and the
Model + Refresh row has a fixed vertical size. A UI test checks the row
keeps its natural height in a tall dialog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0178pC5pfGXepbgU7JEpqVUh
@Yalhash
Yalhash merged commit 12afcc3 into main Nov 5, 2026
16 checks passed
@Yalhash
Yalhash deleted the v2 branch November 7, 2026 05:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant