Skip to content

Latest commit

 

History

History
251 lines (208 loc) · 15.8 KB

ARCHITECTURE.md

File metadata and controls

251 lines (208 loc) · 15.8 KB

Polyglot v1 architecture

Polyglot v1 is a pipeline of single-purpose stages around a pure-Python core. Imports point inward only, there is one thread bridge to the UI, and only the coordinator writes conversation state. This document describes what is built. The design rationale and the critique of v0 that motivated it are in the "Polyglot v1 — Architecture Design" doc.

Layers

Layers

(Sources: v1-layers.mmd, v1-pipeline.mmd; render with mmdc -i <file> -o <png>.)

Layer Package May import Knows about
Core polyglot.core stdlib, numpy nothing outside itself
Adapters polyglot.adapters core + its own library (lazily) one technology each
Platform polyglot.platform core, ctypes the OS
Config polyglot.config core, pydantic, platformdirs settings and secrets
Presentation polyglot.ui core (presenter is Qt-free; ui.qt adds PyQt6) view model, widgets
Composition wiring, controller, app, __main__ everything which concrete class goes where

The dependency rule is a test. tests/unit/test_architecture.py parses every module's imports and fails the build if the core imports anything but the standard library and numpy, if an adapter or the platform layer imports the UI, or if the UI reaches into adapters. OS checks (sys.platform ==) outside polyglot.platform also fail it.

Module map

src/polyglot/
  __main__.py          CLI: GUI, --demo, --headless, --file, --list-devices
  app.py               Qt composition root: window, presenter, bridge, tray, hotkeys
  wiring.py            config -> adapters -> Pipeline (Qt-free); build_demo() for fakes
  controller.py        pipeline lifecycle; applies settings; rebuilds keeping the conversation
  config.py            AppConfig (pydantic), TOML load/save, env overrides, Secrets
  logging_setup.py     rotating file + stderr; no transcript text by default
  selfcheck.py         --self-check: loads every native dependency (used on packaged builds)
  core/
    models.py          Utterance, Speaker, SpeechSegment, Transcript, Suggestion, ReplyOption, IdSource
    errors.py          ProviderTimeout, ProviderUnavailable, QuotaExceeded, AuthError, ...
    events.py          UtteranceUpdated, SuggestionsReady, PipelineStatus, StageError, EventBus
    ports.py           AudioSource, Resampler, VoiceActivityModel, Transcriber, Translator,
                       Suggester, WindowShield, SecretStore
    ringbuffer.py      preallocated SPSC audio ring buffer with overrun accounting
    segmenter.py       VAD endpointing state machine (pure)
    session.py         conversation state + language lock policy
    profile.py         UserProfile: public details the user shares with the suggester
    context.py         ContextBuilder: Session + UserProfile -> SuggestionContext
    languages.py       code <-> name
    pipeline.py        FrontEnd, AsrQueue, AsrWorker, Coordinator, Pipeline
  adapters/
    audio/             wasapi_loopback, sounddevice_source, playback (WAV/array), resample (soxr)
    vad/               silero (onnxruntime), energy
    asr/               faster_whisper, parakeet (onnx-asr), qwen3 (qwen-asr); picked by asr.engine
    translate/         openai, google
    suggest/           openai (JSON-schema structured output)
    http.py            stdlib HTTPS client + status -> typed error mapping
    secrets.py         keyring store, memory store
    fakes.py           a working fake for every port
    dump.py            optional per-utterance WAV dumps
  platform/            WindowsShield (SetWindowDisplayAffinity), UnsupportedShield
  ui/
    presenter.py       Presenter, RowVM, SuggestionVM, StatusVM, ViewPort, Commands
    qt/                overlay, settings_dialog, bridge, shield_filter, hotkeys, theme.qss

Domain model

Utterance is the unit of work: one continuous stretch of the other person's speech. It is a frozen dataclass. Every change makes a new copy, so a snapshot handed to another thread can never change underneath it.

Status Meaning
LISTENING speech started, no text yet
PARTIAL provisional text (only when partials are enabled)
FINAL endpoint reached, text fixed; error set if translation failed
TRANSLATED translation attached
FAILED transcription failed or the utterance was dropped under overload

Utterance IDs come from one monotonic IdSource and define spoken order. Results are placed by ID, never by arrival time.

Every utterance has a Speaker: OTHER (the person being listened to) or USER. Only OTHER utterances are ever replied to; USER turns (from the user's own microphone, see "The user's microphone" below) are context for the next suggestion.

Suggestion context

ContextBuilder (core, pure, immutable) is the one place that decides what the suggester sees: the target utterance, a window of earlier utterances from both speakers (capped by count, suggestions.history, and by length, suggestions.history_chars; cut from the oldest end, the newest is always kept), the languages, the conversation type and the user's UserProfile. The coordinator owns the builder and the Session, so state stays single-writer; a Suggester only renders the context it is given. A rolling summary would slot in here as one more derived input.

UserProfile is an allowlist of public details (name, role, location, summary, goals, reply tone, languages, skills, experience, education, hobbies, interests, talking points). It lives in the [profile] section of config.toml, so it persists, and a change is applied to the running pipeline without a rebuild. A unit test pins the field list.

Pipeline

Pipeline

Thread Job Must never
audio library callback downmix and copy the block into the ring buffer allocate per sample, log, touch disk
polyglot-frontend resample to 16 kHz, cut 32 ms frames, VAD, segmenter, route transcribe
polyglot-frontend-user (only with the microphone on) the same, for the user's microphone transcribe
polyglot-asr (exactly one) load the model, transcribe segments in ID order run in parallel with itself
polyglot-net-* (4) translation and suggestion calls with timeouts write state
polyglot-coord own Session; apply results; dispatch; publish events block on I/O
Qt main thread receive events through EventBridge; presenter; view (not part of the pipeline)

Endpointing (core/segmenter.py)

Rule Default Config
start: probability ≥ start threshold for N frames 0.5 × 3 frames (~100 ms) vad.start_threshold, vad.start_frames
pre-roll kept before the start 300 ms vad.pre_roll_ms
end: probability < end threshold for this long 0.35 for 600 ms vad.end_threshold, vad.end_silence_ms
force-cut at the quietest frame of the last 2 s 15 s vad.max_utterance_s
discard if less speech than 300 ms vad.min_utterance_ms
partial transcripts every off vad.partial_interval_ms

Gap fill. WASAPI loopback delivers no data at all while nothing is playing. Without help, an utterance whose speaker stops talking would never see its 600 ms of silence. The front end therefore feeds real-time-equivalent silence while an utterance is open and no audio has arrived for 250 ms (audio.gap_fill).

The user's microphone

With audio.user_mic = true the pipeline runs a second, independent audio stream tagged Speaker.USER: its own source (SoundDeviceSource), ring buffer, resampler, VAD instance (VADs keep state) and segmenter, in a second front-end thread. Everything after the segmenter is shared:

  • One IdSource (locked), so IDs still define spoken order across both streams.
  • One AsrQueue and the one ASR worker; the segment carries its speaker through to the Utterance. Both speakers are transcribed with the same language hint (the other person's locked language), because the user is replying in it.
  • One coordinator and Session. User utterances never move the language lock, are not translated, are never the target of a suggestion, and do not mark a suggestion stale. They do appear in the history the suggester sees, and in the window as "You" rows.
  • A microphone that fails (at start or later) is reported as a warning (StageError("audio", ...), "microphone: ...") and listening to the other person continues. The mic is skipped, with a warning, when the main input is the same device, and for replayed files.

Settings has a microphone test (ui/mic_test.py, Qt-free): record 5 s from the chosen device on a worker thread, judge the level (AudioClip.level()), play it back. Recording and playback come from a MicTester injected by the composition root (SoundDeviceMicTester), so the UI never imports an adapter. It refuses to run while listening, because the playback would be captured as the other person.

Known limitation: with speakers instead of headphones the microphone hears the other person too, and those lines would be tagged as the user's. There is no echo cancellation or cross-stream gating yet.

Ordering, staleness and backpressure

  • Every result carries its utterance ID; the coordinator places it by ID.
  • Nothing is dropped for being late. A late translation fills its own row.
  • Suggestions run only on the hotkey, for the latest final utterance by the other person, with the last 6 final utterances (either speaker, within a length budget) and the user profile as context. A reply stays linked to the sentence it answers. If newer sentences arrive first, the presenter marks it "newer sentences since". Pressing the hotkey again supersedes an in-flight request: its result is ignored, since the newer request wins.
  • Older partials are skipped (a newer partial or the final covers the same audio).
  • Ring buffer: 30 s; on overrun the oldest audio is overwritten, counted, and a FALLING_BEHIND status is published.
  • ASR queue: at most 8 waiting finals. When full, the oldest is dropped, marked FAILED(overloaded), and FALLING_BEHIND is published. Latency stays bounded instead of growing for the whole session.
  • Network: hard timeouts (translation 3 s, suggestions 6 s) and one retry with jitter for retryable errors.

Lifecycle

Pipeline.start/stop control listening (source, front end, ASR worker). The coordinator and session outlive them, so suggestions and language commands keep working while stopped. stop finishes and transcribes the utterance in progress. When a settings change needs a new pipeline, the controller passes the old session and ids to the new one, so the conversation continues and IDs keep increasing.

Presentation

Model–View–Presenter with a passive view:

  • EventBridge is the only thread bridge. It subscribes to the bus and re-emits each event as a queued Qt signal, so delivery happens on the GUI thread.
  • Presenter (pure Python, unit-tested) keeps the view model, formats it (language names, error wording, "translating…", "Replying to: …") and turns clicks and hotkeys into Commands.
  • OverlayWindow draws rows by ID, the suggestion panel, a status strip and the capture indicator. It emits intent signals and holds no logic beyond view state: FollowLock (pure, ui/follow.py) decides whether the transcript follows the newest sentence. It starts locked, unlocks when the user scrolls, and re-locks only on the lock button or hotkey. The window scrolls when the scroll range changes, i.e. after the layout has measured the new text.

Platform layer

Concern Windows macOS Linux
System audio WASAPI loopback (PyAudioWPatch), follows the default output BlackHole via SoundDeviceSource any input device
Hide from screen capture SetWindowDisplayAffinity(WDA_EXCLUDEFROMCAPTURE) on every top-level window, on show and on handle change, read back with GetWindowDisplayAffinity not attempted (ScreenCaptureKit ignores NSWindow.sharingType on macOS 15+) not attempted

On Windows builds before 19041 the same call would put a black box in the share, so it is skipped. Whenever the window is not hidden, the footer says "Visible in screen share".

Errors, logging, privacy

  • Ports raise typed errors (core/errors.py). Failures show as failures (row note, status strip) and are never replaced with made-up text.
  • No modal dialogs mid-conversation; the only modal dialog is the quit confirmation while listening.
  • Logs: module loggers, rotating file in the user log directory. INFO carries IDs and timings (utt=3 final asr_ms=412), never transcript text.
  • Translation and suggestions are off until enabled. Only text is sent to providers. Keys live in the OS keyring. Audio dumps are off by default and go to the user data directory.

Testing

Suite What Needs
tests/unit segmenter rules, ring buffer, session, events, AsrQueue, config, presenter, controller, architecture rule nothing
tests/pipeline the threaded pipeline end to end with fakes: pairing under out-of-order completion, single ASR worker, language lock, retries, overload, suggestions (supersede, late, failure), clear, stop/restart, gap fill, resampled stereo input, error reporting nothing
tests/adapters HTTP error mapping, OpenAI/Google/suggester request and response handling, WAV I/O, playback, resampling, VAD, Windows shield (fake user32), keyring store soxr/Silero tests skip if absent
tests/ui overlay rows and panels, bridge delivers on the GUI thread, shield filter covers dialogs, settings dialog, demo-mode app smoke test; real shield on Windows PyQt6 + pytest-qt (offscreen)
tests/integration (-m slow) Silero and Whisper tiny on espeak-synthesized speech, end to end faster-whisper, espeak-ng

Implementation decisions beyond the design doc

Decision Why
HTTP via the standard library behind an HttpClient interface, not the OpenAI SDK or httpx Two REST calls do not justify the SDK's dependency tree; tests inject a fake transport
Config via plain pydantic models plus a small TOML writer, not pydantic-settings / tomli-w The precedence rules are ~30 lines; fewer dependencies
Resampling behind a Resampler port, built per source rate Keeps the core numpy-only and lets tests resample without soxr
Silero loaded from faster-whisper's bundled model, both v4 and v5 interfaces detected No extra download; falls back to the energy VAD with a visible warning if it cannot load
Coordinator outlives start/stop Suggestions and language commands work while not listening
Gap fill in the front end Loopback delivers nothing during silence; utterances must still end
Suggest requests supersede each other (generation counter) One request at a time; the newest press wins
OpenAI model per feature, from a built-in list plus the account's live list (adapters/openai_models.py) Model lineups change faster than releases; the dialog gets the list through an injected catalog, so the UI never imports an adapter
Request options chosen per model family, with a one-step fallback on HTTP 400 naming a parameter Older models take temperature; reasoning models reject it and take reasoning_effort, whose accepted values differ by family
Three local ASR engines behind the one Transcriber port; Parakeet and Qwen3 are optional extras, imported only when their model loads Whisper covers the most languages; Parakeet (ONNX Runtime) is faster and more accurate on a CPU for European languages; Qwen3-ASR covers more languages than Parakeet. None needs NVIDIA hardware, and the default install stays small
Parakeet reports the requested language back; Qwen3's reported language counts as confident (it gives no score) Parakeet has no language output; the lock rule (asr.lock_after agreeing utterances) still guards Qwen3's guesses