You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(anim): sync talking animation intensity with streamed audio. #61
PR #57 introduces streamed WAV playback through POST /play together with the new talking animation.
The current talking animation is intentionally independent from the audio. Head nods, neck movement, hand gestures, and eye movement are driven by timing and randomness rather than by the sound that is actually being played.
This already looks surprisingly natural, but it can still feel disconnected from the speech. For example, the robot may make a stronger gesture during a quiet part of the clip or remain relatively still during a louder or more expressive phrase.
Since /play already processes PCM samples as they arrive, we have access to the audio signal during playback. We could use that signal to make the speaking animation react to the actual speech.
Proposed change
Experiment with deriving a simple audio activity / amplitude signal from the PCM samples streamed through POST /play, and use it to influence the talking animation.
For example:
calculate a smoothed RMS or peak amplitude over short audio windows
map that signal to animation intensity
use louder / more energetic speech to increase head nod amplitude, gesture frequency, or movement speed
reduce movement during quiet sections and pauses
keep some randomness so the animation does not become a direct mechanical waveform visualization
This should remain part of the built-in talking animation rather than introducing a new client-side animation protocol.
The first version can be deliberately simple. The goal is to test whether loosely synchronizing movement intensity with speech makes the robot feel noticeably more expressive and natural.
Activity
jamro commentedon Oct 10, 2026
Implemented in 611b554