Skip to content

Add agent-voice: local push-to-talk dictation for coding agents - #1482

Open
mmisbaah wants to merge 9 commits into
CodebuffAI:mainfrom
mmisbaah:feat/agent-voice-app
Open

mmisbaah wants to merge 9 commits into
CodebuffAI:mainfrom
mmisbaah:feat/agent-voice-app

Conversation

@mmisbaah

@mmisbaah mmisbaah commented Oct 5, 2026

Copy link
Copy Markdown

What

A standalone desktop app at Apps/agent-voice/ — Tauri v2 + React 19 + TypeScript, its own bun.lock, deliberately not a root workspace member so the monorepo build and the modified root package.json/bun.lock are untouched.

Hold Ctrl/Cmd+Shift+Space, an always-on-top overlay shows the mic level, whisper.cpp transcribes on-device, and the text is pasted into the window that had focus before recording started. Audio never leaves the machine.

Also included: an MCP server (agent-voice mcp) exposing ask_user_by_voice, so a coding agent can put a question on the overlay and read the spoken answer back.

Tiers

  • Free forever — whisper base/small, 30 min/day, snippets and vocabulary, no account.
  • Pro $59 once (Paddle) — unlimited minutes, large-v3-turbo, per-app AI cleanup modes via BYOK or Ollama, ask_user_by_voice.

Entitlements are Ed25519-signed tokens the app verifies offline; a Cloudflare Worker mints them from Paddle webhooks and caps each license at three devices.

Structure

  • src-tauri/src/ — hotkey (push-to-talk orchestration, free-quota gate), audio (cpal → downmix → 16 kHz), asr + models (whisper-rs, streamed download with sha256/GGML check), text (snippets/vocabulary), paste/focus, usage, entitlement/license, bridge (loopback HTTP for the MCP companion), mcp, updates.
  • src/ — one bundle, two windows chosen by label: settings shell + overlay; Zustand store; Paddle checkout overlay with agentvoice:// deep-link return.
  • worker/ — Cloudflare Worker: /activate, /deactivate, /webhook, /health; HMAC-verified Paddle webhooks; refunds revoke; device cap; rate limiting.
  • docs/entitlement.md — token format, key generation, rotation.

Verification

Check Result
cargo test 72 passed
cargo clippy --all-targets -- -D warnings clean
cargo fmt --check clean
cargo build --bin agent-voice links (25 MB, whisper.cpp bundled)
bun run typecheck (app + worker) clean
bun test 46 passed
bun run build ok
agent-voice mcp over stdin free tier advertises no tools and refuses the call; with a Pro token ask_user_by_voice is listed and invoked
Cross-language token a worker-minted token verifies in the Rust verifier; the same token is pinned as a golden vector in both test suites

Not verified

No end-to-end run with a real microphone/model/GUI, no signed release bundle, and the Paddle + Cloudflare deployment path is documented but not executed against the live accounts.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff noreply@codebuff.com

mmisbaah and others added 8 commits October 5, 2026 11:54
Hold a global hotkey, an always-on-top overlay tracks the mic, whisper.cpp
transcribes on-device and the text is pasted into whatever window had focus
before the recording started. Audio never leaves the machine.

The Rust core covers capture (cpal), resampling, the model catalog and
download manager, snippet/vocabulary post-processing, focus-aware clipboard
paste, a 30 min/day free quota, offline Ed25519 entitlement verification, a
loopback HTTP bridge for the MCP companion, and the `agent-voice mcp` stdio
server exposing ask_user_by_voice. The React front end drives a settings
window plus the overlay from one bundle, selected by window label.

Standalone package with its own bun.lock so the monorepo build and the
root workspaces stay untouched.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
Pro is a one-time purchase, so the only thing standing between a key and a
working app is a small worker: it accepts Paddle webhooks, records which
license keys are active, hands out Ed25519-signed entitlement tokens, and
caps each license at three devices.

Tokens are the same format the Rust app already verifies offline, so the
worker's signing key is the only secret it holds and the app never needs a
network round-trip to know it is Pro. A refund or chargeback revokes the
license and frees the device slots; webhook signatures are checked with
HMAC-SHA256 before anything is written.

KV is used for both license records and a best-effort fixed-window rate
limit on /activate; the eventual-consistency caveat is documented rather
than hidden.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
Users should not have to hunt for a new build, so the Tauri updater plugin
is wired up behind a "Check for updates" action; signatures are verified
against the minisign key baked into the bundle config, and the version
comparison refuses to offer a downgrade.

The docs cover what a first-time user needs (install, hotkey, MCP config,
privacy) and what an operator needs (worker deployment, Paddle webhook
events, entitlement token format and key rotation).

CI runs the frontend typecheck, tests and build, plus fmt, clippy and tests
for the Rust core on Linux and Windows. agent-voice keeps its own workflow
because it is deliberately outside the monorepo workspaces.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
The entitlement token is the one artifact two languages agree on, and until
now nothing enforced that agreement: a drift in either implementation would
have shown up only as "Pro key rejected" in the field.

Both sides now assert the same token, minted by the worker's TypeScript code
with the dev signing key, byte for byte. The Rust test also proves the
embedded dev public key still matches, so the pair cannot rot silently.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
The title tag still read "Tauri + React + Typescript", which is what shows
in the taskbar and in the browser tab during development.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
Paddle Billing issues no license keys and has no refund or chargeback
events, so the fulfillment model the worker assumed could not work on
this platform: there is no license_key_created event to listen for and
nothing for the customer to paste.

The transaction id becomes the license code instead. The worker activates
on transaction.completed and revokes on adjustment.created when the action
is a refund or a chargeback, using an allowlist so a goodwill credit note
or a dispute warning never costs a paying customer their license. The app
reads the code from checkout.completed, accepts it pasted from a receipt,
and shows it with a copy button so a license can move between machines.

The default worker URL becomes the project's own deployment, served from a
custom domain rather than a workers.dev address.
Paddle signs `timestamp:raw body` and carries no event-type header, but
the worker was signing `eventType:body` and rejecting any delivery that
lacked that header. Every real webhook would have failed verification —
the local tests passed only because they reproduced the same wrong
assumption, so nothing caught it until Paddle's own simulator delivered a
genuine signature and got a 400.

Verify over the raw bytes before parsing, read `event_type` from the
payload, compare digests in constant time, and reject a timestamp outside
a five-minute window to blunt replay. The simulator now returns 200 and
the license record is created.
A refund starts as pending_approval, so acting on adjustment.created
alone stripped Pro from a paying customer while Paddle still had the
refund under review — and a rejected refund never came back. Only an
approved refund or chargeback revokes; a rejected one restores the
license, and credits, reversals and chargeback warnings are ignored.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
@codebuff-team

Copy link
Copy Markdown
Contributor

The engineering here looks careful — tests, clippy, a documented entitlement format, device-cap logic — but this isn't a PR a maintainer can "port in" in the sense the mirror process expects. It is an entire new commercial product: a Tauri app with its own payment processor integration (Paddle), its own license-minting backend (a Cloudflare Worker), Ed25519 entitlement tokens, and a $59 paywall baked into src-tauri/src/entitlement.rs and worker/.

The repo explicitly keeps packages/billing/ and packages/bigquery/-style infrastructure out of the public mirror's reach. Standing up a parallel billing/licensing stack in Apps/agent-voice/worker/ and docs/entitlement.md works around that boundary rather than respecting it — monetization, vendor selection (Paddle, Cloudflare), and pricing are product-direction decisions, not something a first PR should introduce unilaterally, however well-implemented.

Separately: at +15238/-0 across 84 files, with its own bun.lock, Cargo.lock, CI workflow, and a secrets/key-rotation story, this is far beyond what a maintainer can review and hand-port in one sitting, even setting the business-model question aside.

If the underlying idea — local push-to-talk dictation via whisper.cpp, pasted into the focused window, with an MCP ask_user_by_voice tool — has merit, I'd suggest proposing it as a free, non-monetized utility first (drop the Paddle/worker/entitlement layer entirely) and scoping it much smaller, so a maintainer can actually evaluate the dictation mechanics (hotkey orchestration, focus capture, audio pipeline) on their own merits.

@codebuff-team codebuff-team added bot:triaged Classified by the community triage bot pr:out-of-scope Touches paths the public mirror does not accept labels Oct 5, 2026
The setup steps said "exactly two events" while listing three, and
bundling adjustment.updated with adjustment.created implied Paddle adds
it for you. It does not — events are ticked individually, so a
destination subscribed only to adjustment.created would never deliver
the event that restores a license after a rejected refund.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:triaged Classified by the community triage bot pr:out-of-scope Touches paths the public mirror does not accept

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants