feat(agent): hear voice messages addressed to the agent

docs/user-guide.md

@@ -140,6 +140,8 @@ Say what goes on a list when you start it ("the farmers' market on Saturday: veg
 
 **Then just say what you need.** "Stacky, we need milk, eggs and toothpaste." Each item lands in its section. Something you have bought before goes back to the list and the section it had, so after the first week nobody has to say where.
 
+**Or say it, or show it.** Hold the mic button and say "Stacky, we need milk and eggs." Stacky's answer starts with what it heard, so a misheard word shows. A photo of a handwritten list works too: send it with a caption that starts with "Stacky," ("Stacky, put this on the shopping list"). In a room with others, a voice message or a photo that does not start with Stacky's name is not for Stacky, the same as typed text. A photo sent first and explained in a second message is not connected yet: put the words in the caption.
+
 **Tick off as you shop.** "Stacky, I got the milk and the bread." You don't have to say which list. A ticked item stays visible for the rest of the day, so "did we get eggs?" still has an answer, and moves to the list's archive overnight. The list then shows only what is still open.
 
 More you can say:
@@ -154,7 +156,7 @@ More you can say:
 
 **A list belongs to a topic.** In a topic room, Stacky uses that topic's lists. To tie an existing list to a room: "Stacky, the supermarket list belongs to this room."
 
-Every list is a page in the [family wiki](#the-family-wiki) and a plain Markdown checklist, so you can also tick items in Obsidian or in Forgejo's editor. Stacky does not hear voice messages or see photos yet: a screenshot of a list has to be typed in for now.
+Every list is a page in the [family wiki](#the-family-wiki) and a plain Markdown checklist, so you can also tick items in Obsidian or in Forgejo's editor.
 
 ---
 

stacklets/agent/config.json

@@ -3,8 +3,16 @@
     "custom": {
       "api_base": "${AGENT_OPENAI_URL}",
       "api_key": "${AGENT_OPENAI_KEY}"
+    },
+    "openai": {
+      "api_base": "${AGENT_WHISPER_URL}",
+      "api_key": "${AGENT_WHISPER_KEY}"
     }
   },
+  "transcription": {
+    "provider": "openai",
+    "model": "${AGENT_WHISPER_MODEL}"
+  },
   "model_presets": {
     "primary": {
       "provider": "custom",

stacklets/agent/docker-compose.yml

@@ -29,6 +29,10 @@ services:
       AGENT_OPENAI_URL: ${AGENT_OPENAI_URL}
       AGENT_OPENAI_KEY: ${AGENT_OPENAI_KEY}
       AGENT_MODEL: ${AGENT_MODEL}
+      AGENT_WHISPER_URL: ${AGENT_WHISPER_URL}
+      AGENT_WHISPER_KEY: ${AGENT_WHISPER_KEY}
+      AGENT_WHISPER_MODEL: ${AGENT_WHISPER_MODEL}
+      AGENT_LANGUAGE: ${AGENT_LANGUAGE}
       AGENT_MATRIX_HOMESERVER: ${AGENT_MATRIX_HOMESERVER}
       AGENT_MATRIX_SERVER_NAME: ${AGENT_MATRIX_SERVER_NAME}
       AGENT_BOT_ID: ${AGENT_BOT_ID}

stacklets/agent/entrypoint.sh

@@ -14,6 +14,19 @@ mkdir -p "$NB/workspace"
 
 cp -f /app/config.json "$NB/config.json"
 
+# The household language goes into the config as a value, not a ${VAR}:
+# nanobot checks transcription.language against ^[a-z]{2,3}$ before it
+# resolves variables, and refuses to start. Two letters, as the archivist
+# sends it; none at all lets Whisper detect the language.
+python3 - "$NB/config.json" "${AGENT_LANGUAGE:-}" <<'PY'
+import json, sys
+path, language = sys.argv[1], sys.argv[2].strip().lower()[:2]
+config = json.load(open(path))
+if language.isalpha():
+    config.setdefault("transcription", {})["language"] = language
+json.dump(config, open(path, "w"), indent=2)
+PY
+
 NAME="${AGENT_NAME:-Stacky}"
 for f in SOUL.md AGENTS.md; do
     [ -f "$NB/workspace/$f" ] || sed "s/__AGENT_NAME__/$NAME/g" "/app/workspace-seed/$f" > "$NB/workspace/$f"
@@ -30,6 +43,10 @@ if [ -d /app/workspace-seed/skills ]; then
     cp -r /app/workspace-seed/skills/. "$NB/workspace/skills/"
 fi
 
+# nanobot transcribes only with a key set; a Whisper on this Mac needs none and
+# ignores whatever it is sent.
+export AGENT_WHISPER_KEY="${AGENT_WHISPER_KEY:-none}"
+
 SECRETS="/setup-state/secrets.toml"
 if [ -f "$SECRETS" ]; then
     AGENT_MATRIX_PASSWORD="$(python3 -c "import tomllib; print(tomllib.load(open('$SECRETS','rb')).get('agent__AGENT_BOT_PASSWORD',''))" 2>/dev/null || true)"

stacklets/agent/runtime/sitecustomize.py

@@ -46,6 +46,11 @@
 8. join_greeting (join_greeting.py) — on being invited, take one turn and
    introduce the room's topic instead of joining in silence.
 
+9. voice_trigger (voice_trigger.py) — a voice message in a shared room goes
+   through nanobot's gate, is transcribed, and reaches the agent only if the
+   transcript addresses it by name, like typed text. The answer starts with
+   what was heard, quoted.
+
 WHY SHIMS AND NOT A FORK
     nanobot 0.3 has a seam for per-turn context (runtime-context providers) and
     keeps a session per Matrix thread, which retired the two shims that needed
@@ -81,6 +86,13 @@
                  `MatrixChannel.client` (nio AsyncClient), `MatrixChannel.config.user_id`
                  — and that `groupPolicy: mention` still routes through
                  `_is_bot_mentioned` (config.json sets that policy)
+    voice_trigger: `MatrixChannel._should_process_message(self, room, event) -> bool` (sync),
+                 `MatrixChannel._handle_message(sender_id, chat_id, content, ..., metadata=)`
+                 with `metadata["event_id"]`, `MatrixChannel._stop_typing_keepalive(room_id,
+                 clear_typing=True)`, `MatrixChannel.is_allowed(sender)`, and that
+                 `_on_media_message` passes the transcript as `[transcription: <text>]`;
+                 `MatrixChannel.send(self, msg: OutboundMessage)` (async, a dataclass whose
+                 `metadata["event_id"]` is the message being answered)
     join_greeting: `nanobot.channels.matrix.runtime.MatrixChannel._on_room_invite(self, room, event)`
                  `MatrixChannel._handle_message(sender_id, chat_id, content, metadata, is_dm)`
 
@@ -272,6 +284,65 @@ def _is_bot_mentioned_or_our_thread(self, event):
     _log.exception("thread-trigger shim could not attach (nanobot internals changed?)")
 
 
+# ── voice_trigger: a voice message is addressed by what was said ─────────────
+# nanobot's gate runs before transcription, and a voice message's text is
+# "Voice message", so in a shared room it never reached the agent. Let it
+# through the gate, and once nanobot has transcribed it, hold the transcript
+# to the rule typed text meets: it has to address the agent by name. See
+# voice_trigger.py.
+try:
+    import dataclasses as _dataclasses_voice
+    import os as _os_voice
+
+    import nanobot.channels.matrix.runtime as _matrix_voice
+    from name_trigger import addressed_by_name as _addressed_by_name_voice
+    from voice_trigger import PendingVoice as _PendingVoice
+    from voice_trigger import is_voice as _is_voice
+    from voice_trigger import quoted as _quoted
+    from voice_trigger import spoken_text as _spoken_text
+
+    _pending_voice = _PendingVoice()
+    _orig_should_process = _matrix_voice.MatrixChannel._should_process_message
+    _orig_handle_voice = _matrix_voice.MatrixChannel._handle_message
+
+    def _should_process_message(self, room, event):
+        if _orig_should_process(self, room, event):
+            return True
+        if not (_is_voice(event) and self.is_allowed(event.sender)):
+            return False
+        _pending_voice.hold(getattr(event, "event_id", "") or "")
+        return True
+
+    async def _handle_message(self, sender_id, chat_id, content, *args, **kwargs):
+        event_id = (kwargs.get("metadata") or {}).get("event_id", "")
+        spoken = _spoken_text(content)
+        if _pending_voice.release(event_id) and not _addressed_by_name_voice(
+                spoken, _os_voice.environ.get("AGENT_NAME", "")):
+            # Not for the agent: no answer, and no typing indicator left on.
+            await self._stop_typing_keepalive(chat_id, clear_typing=True)
+            return None
+        _pending_voice.heard(event_id, spoken)
+        return await _orig_handle_voice(self, sender_id, chat_id, content, *args, **kwargs)
+
+    _orig_send_voice = _matrix_voice.MatrixChannel.send
+
+    async def _send(self, msg):
+        # The answer carries the voice message's event id for threading; the
+        # first one with text gets what was heard in front of it.
+        heard = _pending_voice.answering((msg.metadata or {}).get("event_id", "")) \
+            if msg.content else ""
+        if heard:
+            msg = _dataclasses_voice.replace(msg, content=_quoted(heard, msg.content))
+        return await _orig_send_voice(self, msg)
+
+    _matrix_voice.MatrixChannel._should_process_message = _should_process_message
+    _matrix_voice.MatrixChannel._handle_message = _handle_message
+    _matrix_voice.MatrixChannel.send = _send
+    _log.info("voice-trigger shim active")
+except Exception:
+    _log.exception("voice-trigger shim could not attach (nanobot internals changed?)")
+
+
 # ── join_greeting: say something useful the moment you are invited ───────────
 # Stock nanobot joins an invite silently. In a topic room that silence is the
 # family's first impression of the agent, so it takes one ordinary turn instead

stacklets/agent/runtime/voice_trigger.py

@@ -0,0 +1,98 @@
+"""Is this voice message for the agent? Decided on what was said.
+
+A family member walking through the kitchen says "Stacky, wir brauchen
+Milch und Eier" into the phone. In a room with several people nanobot
+decides whether a message is for the agent before it transcribes
+anything, from the event alone: a pill mention, or (with our name
+trigger) the agent's name in the text. A voice message has neither, its
+text is "Voice message", so in a shared room it never reached the agent.
+
+So a voice message in a shared room is let through nanobot's gate,
+transcribed by nanobot as usual, and only then held to the same rule as
+typed text: the transcript has to address the agent by name. One that
+does not is dropped without an answer, exactly like a typed message
+that does not. In a direct chat, or in a thread the agent is part of,
+nanobot's own gate already lets it through and nothing here applies.
+
+The price is a transcription for every voice message in a shared room
+the agent is in, also the ones meant for someone else. The transcript
+is not kept and not answered.
+
+nanobot hands the transcript to the agent as `[transcription: <text>]`
+after the event's own text (`channels/matrix/runtime.py`,
+`_on_media_message`); `spoken_text` reads it back from there.
+"""
+
+from __future__ import annotations
+
+import re
+from typing import Any
+
+# The Matrix message type of a voice message (and any other audio file).
+VOICE_MSGTYPE = "m.audio"
+
+_TRANSCRIPTION = re.compile(r"\[transcription: (?P<text>.*)\]\s*$", re.DOTALL)
+
+
+def is_voice(event: Any) -> bool:
+    """True for an audio message, the kind phones send as voice messages."""
+    source = getattr(event, "source", None)
+    content = source.get("content") if isinstance(source, dict) else None
+    return isinstance(content, dict) and content.get("msgtype") == VOICE_MSGTYPE
+
+
+def spoken_text(content: str) -> str:
+    """What was said, from the text nanobot built for a voice message; "" if nothing was heard."""
+    match = _TRANSCRIPTION.search(content or "")
+    return match["text"].strip() if match else ""
+
+
+def quoted(heard: str, answer: str) -> str:
+    """The answer to a voice message, after what was heard as a quote.
+
+    The quote is what the agent acted on, so a word Whisper got wrong is
+    visible next to what it caused. Markdown, which reads the same in
+    every household language.
+    """
+    quote = "\n".join(f"> {line}" for line in heard.splitlines() if line.strip())
+    return f"{quote}\n\n{answer}" if quote and answer else answer
+
+
+class PendingVoice:
+    """Voice messages let through the gate, waiting for their transcript.
+
+    The gate (`_should_process_message`) is synchronous and runs before the
+    transcription; the transcript arrives later in `_handle_message`. This
+    remembers, by event id, which messages still have to prove they were
+    addressed to the agent.
+
+        pending = PendingVoice()
+        pending.hold("$event")          # in the gate
+        pending.release("$event")       # True once, in the handler
+
+    It also keeps what was heard in each voice message the agent answers,
+    until the answer goes out (`heard` / `answering`).
+    """
+
+    def __init__(self) -> None:
+        self._held: set[str] = set()
+        self._heard: dict[str, str] = {}
+
+    def heard(self, event_id: str, text: str) -> None:
+        if event_id and text:
+            self._heard[event_id] = text
+
+    def answering(self, event_id: str) -> str:
+        """What was heard in the message this answer is for, once; "" for any other."""
+        return self._heard.pop(event_id, "")
+
+    def hold(self, event_id: str) -> None:
+        if event_id:
+            self._held.add(event_id)
+
+    def release(self, event_id: str) -> bool:
+        """True if this message was held, so its transcript decides."""
+        if event_id in self._held:
+            self._held.discard(event_id)
+            return True
+        return False

stacklets/agent/stacklet.toml

@@ -53,6 +53,12 @@ AGENT_VERBOSE = "1"
 AGENT_OPENAI_URL = "{ai_openai_url_docker}"
 AGENT_OPENAI_KEY = "{ai_openai_key}"
 AGENT_MODEL      = "{ai_default_model}"
+# Speech to text for voice messages: the stack's Whisper, as the archivist uses
+# it, in the household's language.
+AGENT_WHISPER_URL   = "{ai_whisper_url_docker}"
+AGENT_WHISPER_KEY   = "{ai_whisper_key}"
+AGENT_WHISPER_MODEL = "{ai_whisper_model}"
+AGENT_LANGUAGE      = "{language}"
 # Matrix — nanobot logs in as @<agent_handle> against Synapse on the stack network.
 AGENT_MATRIX_HOMESERVER  = "http://stack-messages-synapse:8008"
 AGENT_MATRIX_SERVER_NAME = "{messages_server_name}"

stacklets/agent/workspace/skills/family-memory/SKILL.md

@@ -29,6 +29,11 @@ SOURCES (answer used a vault page, searched or read):
   only pages I read; never shorten|reuse|invent a link; no link line -> title only
   max 3 sources; nothing from vault -> no Sources line
 
+VOICE AND PHOTOS:
+  "[transcription: …]"       -> what they said aloud; act on it like typed text
+     unclear or empty         -> ask; change nothing      # my answer shows the transcript itself
+  photo attached              -> look at it; a handwritten list -> add each item (list not said -> leave list out)
+
 LISTS:
   "what is on X|which lists"     -> exec `stack memory list show [<list>]`; never read_file to show a list
   item op add|tick|untick|remove -> list_edit(list, op, items=[one or more])

tests/unit/stacklets/conftest.py

@@ -137,9 +137,22 @@ def __init__(self):
                     user_id="@stacky-bot:home.local", group_policy="mention",
                 )
                 self.handled = []
+                self.typing_stopped = []
+                self.sent = []
                 self.joined = []
                 self.processed = []
 
+            def is_allowed(self, sender):
+                return True
+
+            async def send(self, msg):
+                # nanobot 0.3 channels/matrix/runtime.py: an OutboundMessage,
+                # whose metadata carries the event id of the message answered.
+                self.sent.append(msg)
+
+            async def _stop_typing_keepalive(self, room_id, clear_typing=False):
+                self.typing_stopped.append(room_id)
+
             def _is_bot_mentioned(self, event):
                 # Stock nanobot: only an autocompleted pill counts.
                 return getattr(event, "pill_mention", False)