From 1df66ad353e118e0feddb339e9fd62050d962a1b Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 12 Oct 2026 19:22:22 +0200 Subject: [PATCH 1/2] feat(agent): hear voice messages addressed to the agent A voice message to Stacky got no answer. In a room with several people nanobot decides whether a message is for the agent before it transcribes it, and a voice message's text is "Voice message", so it never carried the agent's name. Now the agent transcribes voice messages with the stack's Whisper, in the household language, the way the archivist does. In a shared room a voice message is let through the gate and, once transcribed, held to the rule for typed text: it reaches the agent only if what was said starts with the agent's name. Any other gets no answer. In a direct chat or the agent's own thread every voice message counts. The answer starts with what was heard, as a quote, so a word Whisper got wrong shows next to what it caused. Every voice message in a shared room the agent is in is transcribed, also the ones meant for someone else; the transcript is not kept. Refs: #197 --- docs/user-guide.md | 4 +- stacklets/agent/config.json | 8 ++ stacklets/agent/docker-compose.yml | 4 + stacklets/agent/entrypoint.sh | 17 ++++ stacklets/agent/runtime/sitecustomize.py | 71 ++++++++++++++ stacklets/agent/runtime/voice_trigger.py | 98 +++++++++++++++++++ stacklets/agent/stacklet.toml | 6 ++ .../workspace/skills/family-memory/SKILL.md | 4 + tests/unit/stacklets/conftest.py | 13 +++ .../stacklets/test_agent_runtime_shims.py | 87 +++++++++++++++- .../stacklets/test_agent_voice_trigger.py | 43 ++++++++ 11 files changed, 353 insertions(+), 2 deletions(-) create mode 100644 stacklets/agent/runtime/voice_trigger.py create mode 100644 tests/unit/stacklets/test_agent_voice_trigger.py diff --git a/docs/user-guide.md b/docs/user-guide.md index 4902b2c3..f0857053 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -140,6 +140,8 @@ Say what goes on a list when you start it ("the farmers' market on Saturday: veg **Then just say what you need.** "Stacky, we need milk, eggs and toothpaste." Each item lands in its section. Something you have bought before goes back to the list and the section it had, so after the first week nobody has to say where. +**Or say it.** Hold the mic button and say "Stacky, we need milk and eggs." Stacky's answer starts with what it heard, so a misheard word shows. In a room with others, a voice message that does not start with Stacky's name is not for Stacky, the same as typed text. + **Tick off as you shop.** "Stacky, I got the milk and the bread." You don't have to say which list. A ticked item stays visible for the rest of the day, so "did we get eggs?" still has an answer, and moves to the list's archive overnight. The list then shows only what is still open. More you can say: @@ -154,7 +156,7 @@ More you can say: **A list belongs to a topic.** In a topic room, Stacky uses that topic's lists. To tie an existing list to a room: "Stacky, the supermarket list belongs to this room." -Every list is a page in the [family wiki](#the-family-wiki) and a plain Markdown checklist, so you can also tick items in Obsidian or in Forgejo's editor. Stacky does not hear voice messages or see photos yet: a screenshot of a list has to be typed in for now. +Every list is a page in the [family wiki](#the-family-wiki) and a plain Markdown checklist, so you can also tick items in Obsidian or in Forgejo's editor. --- diff --git a/stacklets/agent/config.json b/stacklets/agent/config.json index 01d66208..8de1a671 100644 --- a/stacklets/agent/config.json +++ b/stacklets/agent/config.json @@ -3,8 +3,16 @@ "custom": { "api_base": "${AGENT_OPENAI_URL}", "api_key": "${AGENT_OPENAI_KEY}" + }, + "openai": { + "api_base": "${AGENT_WHISPER_URL}", + "api_key": "${AGENT_WHISPER_KEY}" } }, + "transcription": { + "provider": "openai", + "model": "${AGENT_WHISPER_MODEL}" + }, "model_presets": { "primary": { "provider": "custom", diff --git a/stacklets/agent/docker-compose.yml b/stacklets/agent/docker-compose.yml index 01d02750..8f6816cd 100644 --- a/stacklets/agent/docker-compose.yml +++ b/stacklets/agent/docker-compose.yml @@ -29,6 +29,10 @@ services: AGENT_OPENAI_URL: ${AGENT_OPENAI_URL} AGENT_OPENAI_KEY: ${AGENT_OPENAI_KEY} AGENT_MODEL: ${AGENT_MODEL} + AGENT_WHISPER_URL: ${AGENT_WHISPER_URL} + AGENT_WHISPER_KEY: ${AGENT_WHISPER_KEY} + AGENT_WHISPER_MODEL: ${AGENT_WHISPER_MODEL} + AGENT_LANGUAGE: ${AGENT_LANGUAGE} AGENT_MATRIX_HOMESERVER: ${AGENT_MATRIX_HOMESERVER} AGENT_MATRIX_SERVER_NAME: ${AGENT_MATRIX_SERVER_NAME} AGENT_BOT_ID: ${AGENT_BOT_ID} diff --git a/stacklets/agent/entrypoint.sh b/stacklets/agent/entrypoint.sh index 52e8c607..ac6ebd01 100644 --- a/stacklets/agent/entrypoint.sh +++ b/stacklets/agent/entrypoint.sh @@ -14,6 +14,19 @@ mkdir -p "$NB/workspace" cp -f /app/config.json "$NB/config.json" +# The household language goes into the config as a value, not a ${VAR}: +# nanobot checks transcription.language against ^[a-z]{2,3}$ before it +# resolves variables, and refuses to start. Two letters, as the archivist +# sends it; none at all lets Whisper detect the language. +python3 - "$NB/config.json" "${AGENT_LANGUAGE:-}" <<'PY' +import json, sys +path, language = sys.argv[1], sys.argv[2].strip().lower()[:2] +config = json.load(open(path)) +if language.isalpha(): + config.setdefault("transcription", {})["language"] = language +json.dump(config, open(path, "w"), indent=2) +PY + NAME="${AGENT_NAME:-Stacky}" for f in SOUL.md AGENTS.md; do [ -f "$NB/workspace/$f" ] || sed "s/__AGENT_NAME__/$NAME/g" "/app/workspace-seed/$f" > "$NB/workspace/$f" @@ -30,6 +43,10 @@ if [ -d /app/workspace-seed/skills ]; then cp -r /app/workspace-seed/skills/. "$NB/workspace/skills/" fi +# nanobot transcribes only with a key set; a Whisper on this Mac needs none and +# ignores whatever it is sent. +export AGENT_WHISPER_KEY="${AGENT_WHISPER_KEY:-none}" + SECRETS="/setup-state/secrets.toml" if [ -f "$SECRETS" ]; then AGENT_MATRIX_PASSWORD="$(python3 -c "import tomllib; print(tomllib.load(open('$SECRETS','rb')).get('agent__AGENT_BOT_PASSWORD',''))" 2>/dev/null || true)" diff --git a/stacklets/agent/runtime/sitecustomize.py b/stacklets/agent/runtime/sitecustomize.py index fd5a6fb9..e35c3862 100644 --- a/stacklets/agent/runtime/sitecustomize.py +++ b/stacklets/agent/runtime/sitecustomize.py @@ -46,6 +46,11 @@ 8. join_greeting (join_greeting.py) — on being invited, take one turn and introduce the room's topic instead of joining in silence. +9. voice_trigger (voice_trigger.py) — a voice message in a shared room goes + through nanobot's gate, is transcribed, and reaches the agent only if the + transcript addresses it by name, like typed text. The answer starts with + what was heard, quoted. + WHY SHIMS AND NOT A FORK nanobot 0.3 has a seam for per-turn context (runtime-context providers) and keeps a session per Matrix thread, which retired the two shims that needed @@ -81,6 +86,13 @@ `MatrixChannel.client` (nio AsyncClient), `MatrixChannel.config.user_id` — and that `groupPolicy: mention` still routes through `_is_bot_mentioned` (config.json sets that policy) + voice_trigger: `MatrixChannel._should_process_message(self, room, event) -> bool` (sync), + `MatrixChannel._handle_message(sender_id, chat_id, content, ..., metadata=)` + with `metadata["event_id"]`, `MatrixChannel._stop_typing_keepalive(room_id, + clear_typing=True)`, `MatrixChannel.is_allowed(sender)`, and that + `_on_media_message` passes the transcript as `[transcription: ]`; + `MatrixChannel.send(self, msg: OutboundMessage)` (async, a dataclass whose + `metadata["event_id"]` is the message being answered) join_greeting: `nanobot.channels.matrix.runtime.MatrixChannel._on_room_invite(self, room, event)` `MatrixChannel._handle_message(sender_id, chat_id, content, metadata, is_dm)` @@ -272,6 +284,65 @@ def _is_bot_mentioned_or_our_thread(self, event): _log.exception("thread-trigger shim could not attach (nanobot internals changed?)") +# ── voice_trigger: a voice message is addressed by what was said ───────────── +# nanobot's gate runs before transcription, and a voice message's text is +# "Voice message", so in a shared room it never reached the agent. Let it +# through the gate, and once nanobot has transcribed it, hold the transcript +# to the rule typed text meets: it has to address the agent by name. See +# voice_trigger.py. +try: + import dataclasses as _dataclasses_voice + import os as _os_voice + + import nanobot.channels.matrix.runtime as _matrix_voice + from name_trigger import addressed_by_name as _addressed_by_name_voice + from voice_trigger import PendingVoice as _PendingVoice + from voice_trigger import is_voice as _is_voice + from voice_trigger import quoted as _quoted + from voice_trigger import spoken_text as _spoken_text + + _pending_voice = _PendingVoice() + _orig_should_process = _matrix_voice.MatrixChannel._should_process_message + _orig_handle_voice = _matrix_voice.MatrixChannel._handle_message + + def _should_process_message(self, room, event): + if _orig_should_process(self, room, event): + return True + if not (_is_voice(event) and self.is_allowed(event.sender)): + return False + _pending_voice.hold(getattr(event, "event_id", "") or "") + return True + + async def _handle_message(self, sender_id, chat_id, content, *args, **kwargs): + event_id = (kwargs.get("metadata") or {}).get("event_id", "") + spoken = _spoken_text(content) + if _pending_voice.release(event_id) and not _addressed_by_name_voice( + spoken, _os_voice.environ.get("AGENT_NAME", "")): + # Not for the agent: no answer, and no typing indicator left on. + await self._stop_typing_keepalive(chat_id, clear_typing=True) + return None + _pending_voice.heard(event_id, spoken) + return await _orig_handle_voice(self, sender_id, chat_id, content, *args, **kwargs) + + _orig_send_voice = _matrix_voice.MatrixChannel.send + + async def _send(self, msg): + # The answer carries the voice message's event id for threading; the + # first one with text gets what was heard in front of it. + heard = _pending_voice.answering((msg.metadata or {}).get("event_id", "")) \ + if msg.content else "" + if heard: + msg = _dataclasses_voice.replace(msg, content=_quoted(heard, msg.content)) + return await _orig_send_voice(self, msg) + + _matrix_voice.MatrixChannel._should_process_message = _should_process_message + _matrix_voice.MatrixChannel._handle_message = _handle_message + _matrix_voice.MatrixChannel.send = _send + _log.info("voice-trigger shim active") +except Exception: + _log.exception("voice-trigger shim could not attach (nanobot internals changed?)") + + # ── join_greeting: say something useful the moment you are invited ─────────── # Stock nanobot joins an invite silently. In a topic room that silence is the # family's first impression of the agent, so it takes one ordinary turn instead diff --git a/stacklets/agent/runtime/voice_trigger.py b/stacklets/agent/runtime/voice_trigger.py new file mode 100644 index 00000000..ec65f76a --- /dev/null +++ b/stacklets/agent/runtime/voice_trigger.py @@ -0,0 +1,98 @@ +"""Is this voice message for the agent? Decided on what was said. + +A family member walking through the kitchen says "Stacky, wir brauchen +Milch und Eier" into the phone. In a room with several people nanobot +decides whether a message is for the agent before it transcribes +anything, from the event alone: a pill mention, or (with our name +trigger) the agent's name in the text. A voice message has neither, its +text is "Voice message", so in a shared room it never reached the agent. + +So a voice message in a shared room is let through nanobot's gate, +transcribed by nanobot as usual, and only then held to the same rule as +typed text: the transcript has to address the agent by name. One that +does not is dropped without an answer, exactly like a typed message +that does not. In a direct chat, or in a thread the agent is part of, +nanobot's own gate already lets it through and nothing here applies. + +The price is a transcription for every voice message in a shared room +the agent is in, also the ones meant for someone else. The transcript +is not kept and not answered. + +nanobot hands the transcript to the agent as `[transcription: ]` +after the event's own text (`channels/matrix/runtime.py`, +`_on_media_message`); `spoken_text` reads it back from there. +""" + +from __future__ import annotations + +import re +from typing import Any + +# The Matrix message type of a voice message (and any other audio file). +VOICE_MSGTYPE = "m.audio" + +_TRANSCRIPTION = re.compile(r"\[transcription: (?P.*)\]\s*$", re.DOTALL) + + +def is_voice(event: Any) -> bool: + """True for an audio message, the kind phones send as voice messages.""" + source = getattr(event, "source", None) + content = source.get("content") if isinstance(source, dict) else None + return isinstance(content, dict) and content.get("msgtype") == VOICE_MSGTYPE + + +def spoken_text(content: str) -> str: + """What was said, from the text nanobot built for a voice message; "" if nothing was heard.""" + match = _TRANSCRIPTION.search(content or "") + return match["text"].strip() if match else "" + + +def quoted(heard: str, answer: str) -> str: + """The answer to a voice message, after what was heard as a quote. + + The quote is what the agent acted on, so a word Whisper got wrong is + visible next to what it caused. Markdown, which reads the same in + every household language. + """ + quote = "\n".join(f"> {line}" for line in heard.splitlines() if line.strip()) + return f"{quote}\n\n{answer}" if quote and answer else answer + + +class PendingVoice: + """Voice messages let through the gate, waiting for their transcript. + + The gate (`_should_process_message`) is synchronous and runs before the + transcription; the transcript arrives later in `_handle_message`. This + remembers, by event id, which messages still have to prove they were + addressed to the agent. + + pending = PendingVoice() + pending.hold("$event") # in the gate + pending.release("$event") # True once, in the handler + + It also keeps what was heard in each voice message the agent answers, + until the answer goes out (`heard` / `answering`). + """ + + def __init__(self) -> None: + self._held: set[str] = set() + self._heard: dict[str, str] = {} + + def heard(self, event_id: str, text: str) -> None: + if event_id and text: + self._heard[event_id] = text + + def answering(self, event_id: str) -> str: + """What was heard in the message this answer is for, once; "" for any other.""" + return self._heard.pop(event_id, "") + + def hold(self, event_id: str) -> None: + if event_id: + self._held.add(event_id) + + def release(self, event_id: str) -> bool: + """True if this message was held, so its transcript decides.""" + if event_id in self._held: + self._held.discard(event_id) + return True + return False diff --git a/stacklets/agent/stacklet.toml b/stacklets/agent/stacklet.toml index 21308bb4..b78fd1fb 100644 --- a/stacklets/agent/stacklet.toml +++ b/stacklets/agent/stacklet.toml @@ -53,6 +53,12 @@ AGENT_VERBOSE = "1" AGENT_OPENAI_URL = "{ai_openai_url_docker}" AGENT_OPENAI_KEY = "{ai_openai_key}" AGENT_MODEL = "{ai_default_model}" +# Speech to text for voice messages: the stack's Whisper, as the archivist uses +# it, in the household's language. +AGENT_WHISPER_URL = "{ai_whisper_url_docker}" +AGENT_WHISPER_KEY = "{ai_whisper_key}" +AGENT_WHISPER_MODEL = "{ai_whisper_model}" +AGENT_LANGUAGE = "{language}" # Matrix — nanobot logs in as @ against Synapse on the stack network. AGENT_MATRIX_HOMESERVER = "http://stack-messages-synapse:8008" AGENT_MATRIX_SERVER_NAME = "{messages_server_name}" diff --git a/stacklets/agent/workspace/skills/family-memory/SKILL.md b/stacklets/agent/workspace/skills/family-memory/SKILL.md index 85cd4e8e..ae0d6cd9 100644 --- a/stacklets/agent/workspace/skills/family-memory/SKILL.md +++ b/stacklets/agent/workspace/skills/family-memory/SKILL.md @@ -29,6 +29,10 @@ SOURCES (answer used a vault page, searched or read): only pages I read; never shorten|reuse|invent a link; no link line -> title only max 3 sources; nothing from vault -> no Sources line +VOICE: + "[transcription: …]" -> what they said aloud; act on it like typed text + unclear or empty -> ask; change nothing # my answer shows the transcript itself + LISTS: "what is on X|which lists" -> exec `stack memory list show []`; never read_file to show a list item op add|tick|untick|remove -> list_edit(list, op, items=[one or more]) diff --git a/tests/unit/stacklets/conftest.py b/tests/unit/stacklets/conftest.py index ef748240..35c32419 100644 --- a/tests/unit/stacklets/conftest.py +++ b/tests/unit/stacklets/conftest.py @@ -137,9 +137,22 @@ def __init__(self): user_id="@stacky-bot:home.local", group_policy="mention", ) self.handled = [] + self.typing_stopped = [] + self.sent = [] self.joined = [] self.processed = [] + def is_allowed(self, sender): + return True + + async def send(self, msg): + # nanobot 0.3 channels/matrix/runtime.py: an OutboundMessage, + # whose metadata carries the event id of the message answered. + self.sent.append(msg) + + async def _stop_typing_keepalive(self, room_id, clear_typing=False): + self.typing_stopped.append(room_id) + def _is_bot_mentioned(self, event): # Stock nanobot: only an autocompleted pill counts. return getattr(event, "pill_mention", False) diff --git a/tests/unit/stacklets/test_agent_runtime_shims.py b/tests/unit/stacklets/test_agent_runtime_shims.py index 370d1969..4e7229cd 100644 --- a/tests/unit/stacklets/test_agent_runtime_shims.py +++ b/tests/unit/stacklets/test_agent_runtime_shims.py @@ -26,7 +26,7 @@ SHIMMED_MODULES = ("sitecustomize", "brief", "state_log", "memory_tool", "person_tool", "history_tool", "name_trigger", "thread_trigger", "join_greeting", "vault_write", - "list_tool", "tool_trim") + "list_tool", "tool_trim", "voice_trigger") # The stub nanobot itself lives in conftest as `nanobot_stub`, shared with @@ -284,6 +284,91 @@ def test_an_invite_produces_a_greeting_turn(nanobot): assert turn["metadata"]["room"] == "Topic: Camping" +# ── a voice message is addressed by what was said ──────────────────────── + +def _voice(event_id="$voice"): + class _Event: + sender = "@homer:simpson" + body = "Voice message" + pill_mention = False + source = {"content": {"msgtype": "m.audio", "body": "Voice message"}} + event = _Event() + event.event_id = event_id + return event + + +def _hand_over(channel, transcript, event_id="$voice"): + import asyncio + asyncio.run(channel._handle_message( + sender_id="@homer:simpson", chat_id="!kitchen", media=["/tmp/voice.ogg"], + content=f"Voice message\n[transcription: {transcript}]", + metadata={"event_id": event_id, "room": "Küche"})) + + +def test_a_voice_message_that_names_the_agent_reaches_it(nanobot, monkeypatch): + """In a shared room the agent hears what was said, not "Voice message".""" + monkeypatch.setenv("AGENT_NAME", "Stacky") + channel = nanobot()["nanobot.channels.matrix.runtime"].MatrixChannel() + + assert channel._should_process_message(None, _voice()), "let through to be transcribed" + _hand_over(channel, "Stacky, wir brauchen Milch und Eier.") + + assert [h["content"] for h in channel.handled] == [ + "Voice message\n[transcription: Stacky, wir brauchen Milch und Eier.]"] + + +def test_the_answer_to_a_voice_message_starts_with_what_was_heard(nanobot, monkeypatch): + """A word Whisper got wrong shows next to what the agent did with it.""" + import asyncio + import dataclasses + + @dataclasses.dataclass + class _Outbound: + chat_id: str + content: str + metadata: dict + + monkeypatch.setenv("AGENT_NAME", "Stacky") + channel = nanobot()["nanobot.channels.matrix.runtime"].MatrixChannel() + channel._should_process_message(None, _voice()) + _hand_over(channel, "Stacky please add bread to the shopping list.") + + asyncio.run(channel.send(_Outbound("!kitchen", "Brot steht jetzt auf der Einkaufsliste.", + {"event_id": "$voice"}))) + asyncio.run(channel.send(_Outbound("!kitchen", "Sonst noch was?", {"event_id": "$voice"}))) + + assert [m.content for m in channel.sent] == [ + "> Stacky please add bread to the shopping list.\n\n" + "Brot steht jetzt auf der Einkaufsliste.", + "Sonst noch was?"] + + +def test_a_voice_message_for_someone_else_gets_no_answer(nanobot, monkeypatch): + """The same rule as typed text: no name, no answer, and no typing left on.""" + monkeypatch.setenv("AGENT_NAME", "Stacky") + channel = nanobot()["nanobot.channels.matrix.runtime"].MatrixChannel() + + channel._should_process_message(None, _voice()) + _hand_over(channel, "Marge, holst du Bart vom Training ab?") + + assert channel.handled == [] + assert channel.typing_stopped == ["!kitchen"] + + +def test_a_typed_message_is_not_held_for_a_transcript(nanobot, monkeypatch): + """Only voice gets past the gate unaddressed; text still needs the name.""" + monkeypatch.setenv("AGENT_NAME", "Stacky") + channel = nanobot()["nanobot.channels.matrix.runtime"].MatrixChannel() + + class _Text: + sender = "@homer:simpson" + body = "wir brauchen Milch" + pill_mention = False + source = {"content": {"msgtype": "m.text", "body": "wir brauchen Milch"}} + + assert not channel._should_process_message(None, _Text()) + + # ── failure is contained, and visible ──────────────────────────────────── def test_a_moved_symbol_does_not_take_the_others_down(nanobot): diff --git a/tests/unit/stacklets/test_agent_voice_trigger.py b/tests/unit/stacklets/test_agent_voice_trigger.py new file mode 100644 index 00000000..e81cf621 --- /dev/null +++ b/tests/unit/stacklets/test_agent_voice_trigger.py @@ -0,0 +1,43 @@ +"""Reading a voice message back: is it audio, and what was said. + +nanobot builds the agent's text for a voice message as the event's own +text followed by `[transcription: ]` (`channels/matrix/runtime.py`, +`_on_media_message`, nanobot 0.3.5). The address rule needs the spoken +part alone, because "Voice message" comes first. +""" + +from __future__ import annotations + +import sys +from pathlib import Path +from types import SimpleNamespace + +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "stacklets" / "agent" / "runtime")) + +from voice_trigger import PendingVoice, is_voice, quoted, spoken_text # noqa: E402 + + +def test_what_was_said_is_read_from_after_the_event_text(): + assert spoken_text("Voice message\n[transcription: Stacky, wir brauchen Milch.]") == \ + "Stacky, wir brauchen Milch." + + +def test_a_voice_message_nobody_could_hear_says_nothing(): + assert spoken_text("Voice message\n[Audio: voice.ogg]") == "" + + +def test_only_audio_counts_as_voice(): + audio = SimpleNamespace(source={"content": {"msgtype": "m.audio"}}) + photo = SimpleNamespace(source={"content": {"msgtype": "m.image"}}) + assert is_voice(audio) and not is_voice(photo) and not is_voice(SimpleNamespace()) + + +def test_a_held_message_is_released_once(): + pending = PendingVoice() + pending.hold("$a") + assert pending.release("$a") and not pending.release("$a") and not pending.release("$b") + + +def test_what_was_heard_is_quoted_line_by_line_above_the_answer(): + assert quoted("Stacky, Milch\nund Eier", "Erledigt.") == "> Stacky, Milch\n> und Eier\n\nErledigt." + assert quoted("", "Erledigt.") == "Erledigt." From 0777d8fa9586a01d77cf7e52a01a077c3ccd1f7e Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 12 Oct 2026 19:22:23 +0200 Subject: [PATCH 2/2] feat(agent): read a photo sent to the agent with a caption A photo of a handwritten list, sent with a caption that starts with the agent's name ("Stacky, put this on the shopping list"), now has its items added. The model sees the image; nanobot falls back to a text placeholder for a model without vision. A photo sent first and explained in a second message is not connected: the user guide says to put the words in the caption. Refs: #197 --- docs/user-guide.md | 2 +- stacklets/agent/workspace/skills/family-memory/SKILL.md | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/user-guide.md b/docs/user-guide.md index f0857053..d28dd735 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -140,7 +140,7 @@ Say what goes on a list when you start it ("the farmers' market on Saturday: veg **Then just say what you need.** "Stacky, we need milk, eggs and toothpaste." Each item lands in its section. Something you have bought before goes back to the list and the section it had, so after the first week nobody has to say where. -**Or say it.** Hold the mic button and say "Stacky, we need milk and eggs." Stacky's answer starts with what it heard, so a misheard word shows. In a room with others, a voice message that does not start with Stacky's name is not for Stacky, the same as typed text. +**Or say it, or show it.** Hold the mic button and say "Stacky, we need milk and eggs." Stacky's answer starts with what it heard, so a misheard word shows. A photo of a handwritten list works too: send it with a caption that starts with "Stacky," ("Stacky, put this on the shopping list"). In a room with others, a voice message or a photo that does not start with Stacky's name is not for Stacky, the same as typed text. A photo sent first and explained in a second message is not connected yet: put the words in the caption. **Tick off as you shop.** "Stacky, I got the milk and the bread." You don't have to say which list. A ticked item stays visible for the rest of the day, so "did we get eggs?" still has an answer, and moves to the list's archive overnight. The list then shows only what is still open. diff --git a/stacklets/agent/workspace/skills/family-memory/SKILL.md b/stacklets/agent/workspace/skills/family-memory/SKILL.md index ae0d6cd9..898738e3 100644 --- a/stacklets/agent/workspace/skills/family-memory/SKILL.md +++ b/stacklets/agent/workspace/skills/family-memory/SKILL.md @@ -29,9 +29,10 @@ SOURCES (answer used a vault page, searched or read): only pages I read; never shorten|reuse|invent a link; no link line -> title only max 3 sources; nothing from vault -> no Sources line -VOICE: +VOICE AND PHOTOS: "[transcription: …]" -> what they said aloud; act on it like typed text unclear or empty -> ask; change nothing # my answer shows the transcript itself + photo attached -> look at it; a handwritten list -> add each item (list not said -> leave list out) LISTS: "what is on X|which lists" -> exec `stack memory list show []`; never read_file to show a list