Skip to content

feat(anim): sync talking animation intensity with streamed audio. #61

Closed

Description

@jamro

Area

Firmware

Problem

PR #57 introduces streamed WAV playback through POST /play together with the new talking animation.

The current talking animation is intentionally independent from the audio. Head nods, neck movement, hand gestures, and eye movement are driven by timing and randomness rather than by the sound that is actually being played.

This already looks surprisingly natural, but it can still feel disconnected from the speech. For example, the robot may make a stronger gesture during a quiet part of the clip or remain relatively still during a louder or more expressive phrase.

Since /play already processes PCM samples as they arrive, we have access to the audio signal during playback. We could use that signal to make the speaking animation react to the actual speech.

Proposed change

Experiment with deriving a simple audio activity / amplitude signal from the PCM samples streamed through POST /play, and use it to influence the talking animation.

For example:

  • calculate a smoothed RMS or peak amplitude over short audio windows
  • map that signal to animation intensity
  • use louder / more energetic speech to increase head nod amplitude, gesture frequency, or movement speed
  • reduce movement during quiet sections and pauses
  • keep some randomness so the animation does not become a direct mechanical waveform visualization

This should remain part of the built-in talking animation rather than introducing a new client-side animation protocol.

The first version can be deliberately simple. The goal is to test whether loosely synchronizing movement intensity with speech makes the robot feel noticeably more expressive and natural.

Activity

  1. jamro commented on Oct 10, 2026

    @jamro
    OwnerAuthor

    Implemented in 611b554

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions