feat(audio): add AudioModule for issue #1932#2507
Conversation
Adds mic audio capture and chunked publishing as AudioStamped on an Out stream, mirroring CameraModule. Validated on macOS Apple Silicon at 50 Hz / 20 ms frames with both synthetic (sine tone) and real mic sources. - dimos/msgs/audio_msgs/AudioStamped.py: Python overlay wrapping foxglove_msgs.RawAudio for LCM encode/decode, with from_pcm() and to_numpy() helpers. Flags that builtin_interfaces.Time (not std_msgs.Header) is the wire type, so frame_id is not preserved. - dimos/hardware/sensors/audio/module.py: AudioModule(Module) with AudioConfig(ModuleConfig), async def main() lifecycle, @rpc start/stop, @Skill record_clip. - examples/audio/validate_audio_module.py: LCM round-trip assert + live stream rate/timestamp validation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GuoZhuoRan
requested review from
leshy,
mustafab0,
paul-nechifor and
spomichter
as code owners
Greptile SummaryThis PR introduces a full audio subsystem —
Confidence Score: 3/5The audio pipeline works in the happy path but has a subscription leak that silently accumulates callbacks on re-start, and previously flagged timestamp and thread-safety issues remain unaddressed. The start() methods on SpeechToTextModule, SpeakerModule, and TextToSpeechModule all overwrite their _unsub/_unsubs field without first calling the previous unsubscribe functions. Any restart cycle leaks the old callbacks permanently. dimos/hardware/sensors/audio/module.py — specifically the start() methods of SpeechToTextModule (line 478), SpeakerModule (line 302), and TextToSpeechModule (line 938), and the _on_tts_active handler (line 511). Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Mic as Mic / PortAudio
participant AM as AudioModule
participant STT as SpeechToTextModule
participant TTS as TextToSpeechModule
participant FX as FunVoiceEffectsModule
participant SP as SpeakerModule
Mic->>AM: PCM callback (_sd_callback)
AM->>AM: AudioStamped.from_pcm()
AM-->>STT: audio Out to mic_audio
STT->>STT: VAD + AEC + segment buffer
STT-->>TTS: text Out to speech_text
TTS->>TTS: synthesise (OpenAI / say / pyttsx3)
TTS-->>STT: tts_active_signal, recent_tts_text, tts_reference_audio
TTS-->>FX: tts_audio_raw
FX->>FX: pitch / time-stretch
FX-->>SP: tts_audio
SP->>SP: sounddevice.OutputStream.write()
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Mic as Mic / PortAudio
participant AM as AudioModule
participant STT as SpeechToTextModule
participant TTS as TextToSpeechModule
participant FX as FunVoiceEffectsModule
participant SP as SpeakerModule
Mic->>AM: PCM callback (_sd_callback)
AM->>AM: AudioStamped.from_pcm()
AM-->>STT: audio Out to mic_audio
STT->>STT: VAD + AEC + segment buffer
STT-->>TTS: text Out to speech_text
TTS->>TTS: synthesise (OpenAI / say / pyttsx3)
TTS-->>STT: tts_active_signal, recent_tts_text, tts_reference_audio
TTS-->>FX: tts_audio_raw
FX->>FX: pitch / time-stretch
FX-->>SP: tts_audio
SP->>SP: sounddevice.OutputStream.write()
Reviews (6): Last reviewed commit: "fix(audio): add OpenAI TTS timeout + fai..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| @skill | ||
| def record_clip(self, seconds: float = 1.0) -> bytes: | ||
| """Record and return a clip of raw PCM audio. | ||
|
|
||
| Collects frames from the live audio stream for `seconds` seconds and | ||
| returns them concatenated as raw S16LE PCM bytes. | ||
| """ | ||
| import threading | ||
|
|
||
| buf: list[bytes] = [] | ||
| done = threading.Event() | ||
| collected = [0.0] | ||
|
|
||
| def on_frame(msg: AudioStamped) -> None: | ||
| buf.append(msg.data) | ||
| collected[0] += self.config.frame_ms / 1000.0 | ||
| if collected[0] >= seconds: | ||
| done.set() | ||
|
|
||
| unsub = self.audio.subscribe(on_frame) | ||
| done.wait(timeout=seconds + 2.0) | ||
| unsub() | ||
| return b"".join(buf) |
There was a problem hiding this comment.
record_clip silently returns empty bytes if the module is not running
If record_clip is called before start() or after stop(), no frames will ever arrive, done.wait will time out after seconds + 2.0 seconds, and the method returns b"" with no error or log message. Callers have no way to distinguish a successful empty recording from a misconfigured call. At minimum, a log warning on timeout (or a raised exception) would surface the problem.
| def __repr__(self) -> str: | ||
| n_samples = len(self.data) // (2 if "16" in self.sample_format else 4) | ||
| return ( | ||
| f"AudioStamped(rate={self.sample_rate}, ch={self.channels}, " | ||
| f"fmt={self.sample_format}, samples={n_samples}, ts={self.ts:.6f})" | ||
| ) |
There was a problem hiding this comment.
n_samples heuristic does not divide by self.channels, so for multi-channel audio the repr reports total interleaved samples (e.g. 320 for 20 ms of stereo 16 kHz) rather than samples per channel (160). The existing byte-width check ("16" in self.sample_format) also silently falls back to 4 bytes/sample for any unknown format string, which could produce a nonsensical count.
| def __repr__(self) -> str: | |
| n_samples = len(self.data) // (2 if "16" in self.sample_format else 4) | |
| return ( | |
| f"AudioStamped(rate={self.sample_rate}, ch={self.channels}, " | |
| f"fmt={self.sample_format}, samples={n_samples}, ts={self.ts:.6f})" | |
| ) | |
| def __repr__(self) -> str: | |
| bytes_per_sample = 2 if "16" in self.sample_format else 4 | |
| n_frames = len(self.data) // (bytes_per_sample * self.channels) | |
| return ( | |
| f"AudioStamped(rate={self.sample_rate}, ch={self.channels}, " | |
| f"fmt={self.sample_format}, frames={n_frames}, ts={self.ts:.6f})" | |
| ) |
| audio metadata. Serialises to/from foxglove_msgs.RawAudio on the wire. | ||
| """ | ||
|
|
||
| msg_name = "foxglove_msgs.RawAudio" # wire type used for LCM |
There was a problem hiding this comment.
we don't use foxglove, where does this come from?
There was a problem hiding this comment.
Foxglove is not a new dependency; it's already mirrored into dimos_lcm, and RawAudio is the only audio type in there, so I reused it. Left a note that it's a stand-in pending a native Header-bearing type.
| def lcm_encode(self) -> bytes: | ||
| """Encode to foxglove_msgs.RawAudio wire bytes. | ||
|
|
||
| NOTE: frame_id and seq from self.header are NOT preserved (the wire |
There was a problem hiding this comment.
ros2 header has no seq
why not preserve frame_id?
There was a problem hiding this comment.
My comment is wrong; I will fix this issue. Frame_id does exist, but based on the RawAudio format, it only carries a timestamp; there is no frame_id on the wire type to put in. Preserving a frame_id means adding a header-bearing audio type to dimos-lcm, and we can discuss it today
- Remove all mentions of `seq` (ROS2 std_msgs/Header has no seq field) - Reword frame_id note: dropped because RawAudio has no frame_id field on the wire, not by design choice - Mark foxglove_msgs.RawAudio as a temporary stand-in pending team decision on a native Header-bearing LCM type Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
| ts: float | None = None, | ||
| ) -> AudioStamped: | ||
| """Construct from raw PCM bytes.""" | ||
| t = ts if ts is not None else time.monotonic() |
There was a problem hiding this comment.
from_pcm factory's fallback timestamp uses time.monotonic(), which returns an opaque system-relative counter (seconds since boot) rather than a Unix wall-clock time. Any caller that omits the ts argument — including external consumers of this public API — will create an AudioStamped whose ts field is near 0–86400 rather than near the Unix epoch (~1.7 × 10⁹). This makes Timestamped.dt() return a date in 1970 and breaks cross-stream alignment with any module that uses time.time().
| t = ts if ts is not None else time.monotonic() | |
| t = ts if ts is not None else time.time() |
Add demo_audio blueprint to module.py and regenerate all_blueprints.py so AudioModule is accessible via: dimos run demo-audio (blueprint) dimos run audio-module (standalone module) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
| (TextToSpeechModule, "text", "speech_text"), | ||
| (TextToSpeechModule, "audio", "tts_audio"), | ||
| (SpeakerModule, "audio", "tts_audio"), | ||
| ] | ||
| ) | ||
|
|
||
|
|
||
| demo_audio = autoconnect( | ||
| AudioModule.blueprint(), |
There was a problem hiding this comment.
TextToSpeechModule publishes frames with time.monotonic() timestamps
Every chunk published from _worker_loop uses time.monotonic() as its timestamp. time.monotonic() returns a system-relative counter (seconds since boot), not a Unix wall-clock time. Downstream consumers calling Timestamped.dt() will get dates in 1970, and cross-stream alignment with any module that uses time.time() (e.g., CameraModule) will fail. Replace with time.time() to match the rest of the stack.
2 participants
Adds mic audio capture and chunked publishing as AudioStamped on an Out stream, mirroring CameraModule. Validated on macOS Apple Silicon at 50 Hz / 20 ms frames with both synthetic (sine tone) and real mic sources.
Problem
Closes DIM-XXX
Solution
How to Test
Contributor License Agreement