Skip to content

Add foreground Windows app voice transcription - #534

Open
power246 wants to merge 1 commit into
OctoSense-org:mainfrom
woshuoduijiushidui:feat/windows-app-voice-transcription-upstream
Open

power246 wants to merge 1 commit into
OctoSense-org:mainfrom
woshuoduijiushidui:feat/windows-app-voice-transcription-upstream

Conversation

@power246

@power246 power246 commented Oct 13, 2026 •

Copy link
Copy Markdown

PR draft for OctoSense-org/OctoSense

Team: 我说对就是队

Compare: main...woshuoduijiushidui:feat/windows-app-voice-transcription-upstream

Title: Add foreground Windows app voice recording and transcription

Why

Installed Windows apps need a short voice-input path. This ports the voice-only source from fork PR #3 onto current upstream main; the fork's later image bridge is excluded.

Approach

  1. Wire Windows microphone permission status/request/revoke into the existing host-owned consent flow. Add native Windows capture to microphone.record_start/status/stop/cancel with foreground and app/heap ownership checks, bounded workers, and app-jail WAV output.
  2. Add Windows model.transcribe_capability and model.transcribe to the shared AI provider service. Validate an app-local WAV, choose the configured MiniMax-family route, upload once using the host-owned key, return only text and duration, and apply the shared call/rate budget plus a daily transcription quota.
  3. Update the pinned Makepad overlay for Windows OS-reported pointer provenance on permission clicks, click activation, and text input focus; update the patch lock and bilingual docs. Extend focused tests.

Supported functionality

  • Foreground installed Windows apps can request microphone consent, record mono 16 kHz PCM16 WAV, poll, stop, cancel, or automatically stop at 30 seconds. The file stays in the caller's storage jail. Cancel or foreground loss discards it.
  • Admitted Windows apps with the model grant can probe ASR availability and transcribe their own WAV using configured MiniMax / MiniMax CN / MiniMaxI. Capability output exposes no provider identity or credential. The transcript is returned for app-side review/editing before sending; see the PantrySteward companion app.
  • Existing macOS/Android recording and playback routes remain supported. Windows playback is not advertised here.

Validation

  • python -X utf8 tools/setup.py --check --cargo: passed.
  • cargo test --locked --offline -q -p octosense-llm-service --lib --test media: 30 library + 30 integration tests passed.
  • cargo test --locked --offline -q -p octosense-shell --features app-hub --lib audio_service::tests: 6 passed.
  • git diff --check: passed. This commit has the same file tree as the reviewed voice-only head of fork PR Revert the makepad repoint — it broke the lockfile and split the graph #3.
  • The earlier task-owned Windows desktop candidate used a real microphone and person-configured MiniMax provider. Manual and automatic stop produced editable transcripts. This physical/provider check was not rerun on the new upstream branch; its source tree is identical.

Known limits and review points

  • ASR is Windows-only and currently needs a MiniMax-family provider; Kimi-only configurations report configured: false. No Windows speaker playback, TTS, background recording, or streaming ASR is added.
  • Transcription accepts a nonempty caller-local mono 16 kHz PCM16 WAV of at most 1 MiB / 30 seconds. The default daily allowance is 300 input seconds per app. Upload failures are not automatically retried.
  • The pinned Windows permission adapter cannot query the OS microphone privacy setting. Host consent/status therefore does not prove audio frames will arrive; device/open/frame failures surface separately.
  • Windows input provenance trusts OS hardware-class pointer messages. UIAccess-injected events can share that classification; remote desktop and other injected-input routes have not been physically verified. This host-wide runtime change deserves explicit review.
  • Automated tests do not establish support for every microphone, privacy configuration, provider account, or release build. No Store/cross-platform ASR acceptance is claimed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant