Skip to content

About

Voice remote for AI agents on macOS. Hold a button, press a shortcut or knock twice: speak, and your words are typed anywhere, sent to Claude Code or Codex with voice approvals, or used to control your Mac safely with per-app access. Speech to text runs on your Mac.

Resources

Security policy

Stars

1 star

Watchers

0 watching

Forks

 
 

Repository files navigation

Xpu Command

M8ven Score

Hold a mouse button, talk to your AI coding agent. A menu-bar app for macOS and Windows: press and hold a side button on any mouse, speak, release. Your words are transcribed on your computer and either typed where your cursor is, or handed to Claude Code or Codex to work on your project, with live status and a spoken summary in your headset when it's done.

Xpu Command settings with the listening pill

Download for macOS · free and open source · speech stays on your Mac

Screenshots

Get started: three steps with a download progress bar Modes: type, send to an agent, or control your Mac
Get started. Three steps, one button each, with a progress bar for the speech model. Three modes. Type where your cursor is, send to Claude Code or Codex, or control your Mac.
App access: Off, See, Control or Ask for each app Memory: notes the assistant knows about you
App access. Every app is Off until you choose See, Control or Ask. Memory. "Remember that Sara is my accountant". Stays on your Mac.
Routines: several actions under one name The status pill: listening, working, allow, done
Routines. Say a name and several actions run at once. The pill. Listening, working, asking before a command, done.

Download

Download Xpu Command for macOS (Apple silicon): open the latest release and get the .dmg file under Assets. On Linux, see Linux below.

Version 0.10.0
Requires macOS 12 or later, a Mac with Apple silicon (M1 or newer)
File Xpu-Command-0.10.0-arm64.dmg (5.3 MB)
SHA-256 cddcd1cff92838d61fae000d366ea48222b21120452bbbc24298b6ccbbb9a7f4

What's new in each version: CHANGELOG.md.

Install

  1. Open the .dmg and drag Xpu Command to Applications.
  2. Open Xpu Command from Applications.
  3. Allow Microphone and Accessibility when asked. Xpu Command lives in the menu bar.
  4. In Settings › Speech to text, download a speech model (148 MB, one time).

"Apple could not verify Xpu Command is free of malware"

This build is not notarized by Apple yet, so macOS asks once. To open it:

  1. Click Done on the message (don't move the app to the Bin).
  2. Open System Settings › Privacy & Security and scroll down to "Xpu Command" was blocked to protect your Mac.
  3. Click Open Anyway, enter your password, and click Open Anyway again.

You only do this once per version. If you prefer Terminal: xattr -dr com.apple.quarantine "/Applications/Xpu Command.app", then open the app normally.

Agent features need Claude Code or Codex installed and signed in. To check your download: shasum -a 256 Xpu-Command-0.10.0-arm64.dmg.

Linux

Every feature works on Linux, with the differences below. Build the packages (.deb, .rpm, AppImage) on Linux with npm ci && npx tauri build, after installing the build libraries (Debian/Ubuntu: libwebkit2gtk-4.1-dev libgtk-3-dev libayatana-appindicator3-dev librsvg2-dev libasound2-dev libspeechd-dev libdbus-1-dev libxkbcommon-dev libxtst-dev libclang-dev cmake pkg-config).

X11 Wayland
Talk button (mouse) Works as installed Add yourself to the input group once: sudo usermod -aG input $USER, then log out and in. The packages ship a udev rule so the press doesn't also click in the app under the pointer.
Keyboard shortcut Works Through the desktop's GlobalShortcuts portal (GNOME 48+, KDE Plasma, Hyprland); the desktop asks you to confirm the keys once.
Dictation typing Works in every app GTK, Qt, Firefox and LibreOffice fields through accessibility. Other apps need wtype (Sway, Hyprland, KDE) or ydotoold.
Control my computer Reads and operates apps through AT-SPI; pointer clicks, drag and pixel clicks also work. Reads and operates apps through AT-SPI; no pixel clicks or drag (Wayland doesn't let apps move the pointer), and an app must be in front for key presses.
  • Accessibility: Settings turns on the desktop's accessibility switch (the same as "assistive technologies"). Chromium and Electron apps show their contents only when started with --force-renderer-accessibility.
  • Web pages are read through accessibility (read_window, click, type_text); Linux browsers have no page-scripting bridge, so read_page isn't offered.
  • The Mac's apps, on Linux: media through MPRIS (Spotify, browsers, any player), volume through PipeWire or PulseAudio, dark mode on GNOME-family and KDE desktops, reminders as notifications Xpu Command keeps and shows on time, events as calendar files your calendar app adds, notes as Markdown in ~/Documents/Notes, email drafts in your mail app, the Trash by the freedesktop.org spec (undo puts items back), and "Shortcuts" as your own scripts in ~/.config/xpu-command/shortcuts.
  • Headset button: Xpu Command registers as a media player, so a Bluetooth headset's play/pause press starts talking.
  • Double knock needs an accelerometer the kernel exposes (convertible laptops and tablets).
  • Secrets go to the Secret Service (GNOME Keyring, KWallet). Logs are in ~/.local/state/xpu-command/logs.
  • Optional helpers: wl-clipboard or xclip (clipboard tools), poppler-utils (reading PDFs), unzip (Word and OpenDocument files), plocate (faster file search).

How it works

mouse button (held) ─► microphone ─► speech-to-text ─► ┬─ Mode A: typed into the focused app (any tool)
  global hook,          in memory       local Whisper   └─ Mode B: Claude Code / Codex on a project folder
  button swallowed      only            or OpenAI          ─► live status pill ─► spoken summary
  • Mode A, "type where my cursor is" works with every tool: a terminal running claude or codex, Cursor, ChatGPT, anything.
  • Mode B, "send to an agent" runs the agent headless in a folder you choose, shows what it's doing (Reading auth.ts, Editing login.py) and reads the result aloud.
    • Voice approvals (Claude Code, "Edit files; ask me before commands", the default): every command is asked in the pill and out loud. Answer "yes", "no" or "always", or press to allow; commands that delete, push, publish or run downloaded code need a spoken "yes".
    • Conversation: after the answer is read aloud it listens briefly for your reply ("now run the tests"); silence or "that's all" ends it.
    • Any other agent: choose Custom and pick its program (Aider, Gemini CLI, your own script). It runs in the project folder, reads the request on stdin, gets XPU_SAFETY (read_only, edit, ask_commands) and XPU_RESUME, and prints plain text or JSON lines ({"type":"status","text":…}, {"type":"session","id":…}, {"type":"done","summary":…}, {"type":"error","message":…}). It runs with your permissions: the safety level is passed, not enforced. Shells and interpreters are refused, so speech can't run as code.
    • Follow-ups continue the same conversation for 30 minutes (Claude --resume, Codex exec resume).
    • Say "stop" to stop the agent, or "new task" to start fresh.
    • Several Claude accounts? Pick the profile (~/.claude, ~/.claude-work, …) in Settings; it's passed as CLAUDE_CONFIG_DIR.
  • Mode C, "control my Mac" (macOS): "open Spotify", "volume 30", "remind me to call the accountant tomorrow at 10", "draft an email to sara@example.com saying I'll be late".
    • Instant commands: simple one-step requests ("open Spotify", "volume 30", "next song", "dark mode on", «اسپاتیفای رو باز کن», «صدا رو بذار روی ۳۰») run at once, about 0.15 s instead of 4 s, without the AI and through the same checks. Anything longer, or a name it doesn't recognize, goes to Claude. Turn it off in Mode › Instant commands.
    • Claude Code plans with only the app's own 27 tools (no shell, no files, no web); the app checks every call, then runs, shows-then-runs, or asks first, by risk.
    • Reminders, events, notes, drafts and quitting apps: shown first, press or knock within 3 s to cancel. Shortcuts: press or knock to allow. Nothing is ever sent, deleted or bought.
    • Works inside apps you allow (Settings › App access: Off / See / Control / Ask each time; Off by default). It reads windows through macOS Accessibility (real buttons and fields, numbered), clicks, types, uses menus and keys; screenshots are a fallback. A glowing outline marks the window it is working on.
    • Every action is in Settings › Activity. Debug: xpu-command --selftest-mac "open calculator".
    • Say "undo that" (or press Undo in Activity) within 30 minutes to reverse the last request: apps it opened are closed, apps it quit reopen, the reminders, events and notes it made are deleted, and volume, dark mode and music go back.
  • Two triggers: a mouse button, or a keyboard shortcut (default ⌃⌥Space) for trackpads. Hold, speak, release.
  • Listening: after a click (toggle mode) or a double knock it sends when you stop talking (1.5 s of quiet) and cancels if you say nothing. Silence is trimmed before transcription, the stop knock is cut off, and Whisper's phantom phrases ("Thank you.") are dropped. Soft sound cues mark start, send and cancel.
  • Memory: say "remember that Sara is my accountant" (or add it in Settings › Memory) and the agent knows it from then on; "forget Sara" removes it. Stored on this Mac only (0600), up to 200 short notes, editable and removable, and given to the agent as facts, never as instructions. Turn it off in Settings › Memory.
  • Routines: several actions under one name. Ask for something, then say "save that as a routine called start my day"; from then on saying "start my day" runs it at once, without waiting for the AI, through the same App access and cancel checks. When you ask for the same actions on different days, Settings › Routines offers to save them. "Undo that" reverses a routine too.
  • Persian (فارسی): pick Persian in Settings › Speech to text (it switches to a multilingual model; Small is best). Answer with «آره»، «نه» or «همیشه», say «بس کن» to stop and «برگردون» to undo. The agent answers in the language you spoke; macOS has no Persian voice, so those answers are shown, not read aloud.
  • AirPods: press once to talk (Settings › Talk button › Press AirPods to talk). Double and triple press still skip tracks in Spotify or Music.
  • Double knock (experimental, MacBooks with Apple silicon): knock twice on the palm rest to start; stop talking or knock again to send. Taps within 1.2 s of typing, clicking or scrolling are ignored. Off by default. Calibrate with src-tauri/target/debug/xpu-command --selftest-knock [low|medium|high|<g>].

Run it (development)

Requirements: Rust 1.80+, Node 20+, CMake (builds whisper.cpp). For Mode B, the claude and/or codex CLI installed and signed in.

npm install
npx tauri dev          # first build compiles whisper.cpp: a few minutes
  1. macOS: grant Accessibility when asked (System Settings › Privacy & Security › Accessibility). In development the grant goes to your terminal app. The talk button starts working within two seconds of the grant.
  2. Open Settings from the menu-bar icon, download the speech model (148 MB, checked against its published SHA-256), pick your button.
  3. Hold the button, speak, release.

Tests and checks:

cd src-tauri && cargo test && cargo clippy --all-targets -- -D warnings   # 126 tests, incl. fake-agent integration tests
cargo audit --deny unsound --deny yanked --ignore RUSTSEC-2024-0429
npm run typecheck
# debug builds only: the pipeline without the GUI
./target/debug/xpu-command --selftest-model base.en
./target/debug/xpu-command --selftest-stt speech.wav
./target/debug/xpu-command --selftest-agent claude /path/to/project "add a test for the parser"
XPU_RESUME=<session id> ./target/debug/xpu-command --selftest-agent claude /path/to/project "what did you change?"
XPU_OVERLAY_DEMO=1 ./target/debug/xpu-command   # cycles the overlay; the frontmost app must not change

Release build: see Releasing.

Releasing (macOS)

scripts/release-macos.sh

Builds the app and a DMG (src-tauri/target/release/bundle/dmg/) and prints its SHA-256. Upload it to a GitHub release.

Without Apple credentials the app is ad-hoc signed, and users see the "Apple could not verify" message once (see Download). With an Apple Developer account, set these and run the same script: the app and DMG are signed with your Developer ID and notarized, so they open without any warning.

export APPLE_SIGNING_IDENTITY="Developer ID Application: Your Name (TEAMID)"
export APPLE_ID="you@example.com"          # your Apple Account
export APPLE_PASSWORD="xxxx-xxxx-xxxx-xxxx" # an app-specific password from account.apple.com
export APPLE_TEAM_ID="TEAMID"

Layout

Path What
src-tauri/src/input/ Global mouse hook: macOS Quartz event tap, Windows WH_MOUSE_LL
src-tauri/src/audio.rs Push-to-talk capture (cpal), resampling to 16 kHz
src-tauri/src/stt/ Local whisper.cpp engine, pinned model catalog and verified download, OpenAI engine
src-tauri/src/sanitize.rs Transcript cleaning (no control characters can reach a terminal) and speakable summaries
src-tauri/src/inject.rs Mode A: typing into the focused app
src-tauri/src/agents/ Mode B: the adapter trait, Claude Code and Codex adapters
src-tauri/src/controller.rs The talk loop and the overlay
src-tauri/src/commands.rs The only commands the settings window may call
src/ Settings window and overlay (TypeScript, no framework)

Status

macOS Linux Windows
Talk button (hook, swallow, ignore injected clicks) built, unit-tested X11 button grab; Wayland evdev + uinput passthrough. Compiled and unit-tested; not yet run on a Linux desktop built, unit-tested logic; not yet run on Windows
Local speech-to-text verified end to end builds (CPU); not yet run needs a Windows build run
Dictation typing and editing in place built AT-SPI, XTest, wtype/ydotool; not yet run built (typing only)
Overlay never takes focus verified: frontmost app unchanged during a full overlay cycle GTK no-focus hints; not yet run SW_SHOWNOACTIVATE + WS_EX_NOACTIVATE; not yet run
Control my computer verified (bench) same tools over AT-SPI, X11, D-Bus; unit-tested, not yet run not available
Mode B Claude Code verified: edits a project, shell refused, follow-up resumes the session shared code; not yet run not yet run
Mode B Codex flags and event format verified on codex-cli 0.167.2 (incl. exec resume under the same sandbox); a signed-in run not yet done
Tests 195 pass, clippy clean 206 pass and clippy clean in a Debian container

Next: a first run on real Linux desktops (GNOME and KDE, X11 and Wayland) and Linux packages on each release, auto-update, notarized installers.

Security: see SECURITY.md.

Roadmap

What's built and what's next: ROADMAP.md.

License

Apache License 2.0: see LICENSE and NOTICE. Privacy: PRIVACY.md. Terms: TERMS.md.

About

Voice remote for AI agents on macOS. Hold a button, press a shortcut or knock twice: speak, and your words are typed anywhere, sent to Claude Code or Codex with voice approvals, or used to control your Mac safely with per-app access. Speech to text runs on your Mac.

Resources

Security policy

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages