virtual talkbox

a virtual talkbox using computer vision to track your mouth movements and shape sound with formant filters. basically: you play notes (midi or qwerty keyboard) and your mouth controls how they sound!

what is this

with a talkbox, you play a synth through a tube into your mouth, and shape the sound with vowel movements. this does the same thing but with a webcam instead of a tube. it uses mediapipe to track your mouth, calculates approximate formant frequencies (F1, F2, F3) based on your mouth shape, and filters a sawtooth wave in real-time. the result sounds like you're "singing" (kinda) the notes you play!

setup

this project uses uv to manage python and dependencies. install it with brew install uv (macOS) or curl -LsSf https://astral.sh/uv/install.sh | sh.

macOS

# audio libraries
brew install portaudio portmidi liblo libsndfile

# creates .venv with python 3.12 and everything installed (pyo builds from source, takes a minute)
uv sync

# run it
uv run main.py

macOS will ask for permission the first time:

• camera - for the mouth tracking
• input monitoring / accessibility - only in qwerty keyboard mode. the keyboard listener is global, so macOS needs to allow your terminal app under system settings > privacy & security > input monitoring. without it, key presses are silently ignored.

Linux

# debian/ubuntu
sudo apt-get install portaudio19-dev libportmidi-dev liblo-dev libsndfile1-dev

uv sync
uv run main.py

Windows

pyo ships pre-built on windows:

uv sync
uv run main.py

the first run downloads mediapipe's face landmarker model (a few MB) into models/.

usage

when you start it up, you'll get a menu to choose your input:

• midi keyboard - if you have one plugged in
• computer keyboard - if you don't

the app will remember your choice for next time. quit with Esc or by closing the video window.

keyboard layout

if you're using qwerty keyboard mode, it's set up like a piano:

black keys:  w e   t y u   o p
white keys: a s d f g h j k l ; '

extra controls:

• z/x - change octave
• c - toggle vibrato
• arrow up/down - pitch bend

heads up: the key listener is global, so keys you type in other windows will play notes too while the app is running.

midi

note on/off play the synth, the pitch wheel bends ±2 semitones, and the mod wheel adds vibrato.

playing

1. press a key to play a note
2. move your mouth while the note is playing
3. experiment with different vowel shapes:
   • "ah" = open mouth
   • "ee" = wide smile
   • "oo" = rounded lips

sound only plays when a note is pressed AND your face is detected

how it works

• face tracking: mediapipe face landmarker for mouth detection
• formant mapping:
  • F1 (270-730hz) controlled by jaw opening
  • F2 (870-2290hz) controlled by lip width
  • F3 (1650-3000hz) combination of both
• audio: pyo synthesizer with supersaw oscillator + formant bandpass filters
• threading: separate threads for video, audio, and input

config

creates a config.json file where you can tweak:

• audio.buffer_size - lower = less latency, higher = more stability
• audio.sample_rate
• video.device_id - which camera to use
• video.flip_horizontal - mirror the preview
• video.fps - camera frame rate (can fall back)
• formants.f1_range / f2_range / f3_range - frequency ranges in hz
• display.show_debug_info - overlay the measured mouth values and formants
• display.window_name

credits

uses mediapipe, pyo, opencv, mido, and pynput