Repository navigation
feat(voice): treat a voice message like any other message - #85
Merged
Merged
Conversation
The installer only checked that the model file existed, so a download the network interrupted was reported as "Whisper model ready" with a green checkmark. whisper-server then loaded 47 of 587 tensors, aborted, and every `stack up ai` afterwards skipped the re-download because the file was still there. Voice transcription stayed broken with no way to recover short of deleting the file by hand. The model is ~1.5 GB, so anything far under that is a download that did not finish: re-fetch it and fail loudly if it is still short. Adds -f to the curl so an HTTP error page is never written out as a model.
A voice memo is cleaned up by the LLM to restore punctuation. Pointed at a Qwen3 with reasoning on, that call came back with a page of the model's own deliberation and no transcript at all, and we wrote it straight into the vault as what somebody had spoken. The audio is the only other copy. Three changes, smallest first: - Ask models not to think out loud. Verified against oMLX that only `chat_template_kwargs.enable_thinking=false` works; reasoning_effort, a top-level enable_thinking, a reasoning object and the /no_think prompt suffix all had no effect. Sent on every completion, because no famstack prompt wants deliberation in the answer. - Pin the cleanup to temperature 0. It is a transformation, not a generation: the same words should always get the same punctuation. - Verify the result instead of trusting it. The prompt permits punctuation, capitalization and line breaks and nothing else, so the letters are an exact invariant. A polish that summarises, truncates, reorders or leaks reasoning fails it and the raw transcript is kept. The check compares letters rather than words on purpose: whisper writes "theres" and "well known", and restoring the apostrophe or hyphen is punctuation. A word-level check rejected every real polish.
Holding the mic button and typing are the same act: putting words in the room. Only the encoding differed, and until now every bot had to know about it. Transcription was a capability each one reached for, which grew an audio branch in the archivist, a second whisper client in the same process, a third transcribe call for batch scans, and a whole bot whose only job was the capability. Now the transport decodes speech before dispatch, beside decryption: below everyone, done once. A handler written for typed text handles speech without knowing speech exists, and every gate a typed message passes — room mode, thread ownership, mention, routing, corrections — applies to it unchanged. Speaking "find the insurance policy" in the documents room now searches, the way typing it always did. The message keeps its identity through the decode: same sender, event id and thread, so replies, reactions and ownership still land on the voice message the family can see. What it gains is provenance, because whisper is lossy in a way a keyboard is not, and the reply quotes the words back so a mishearing is visible without opening the vault. Two judgement calls worth knowing: - A memo counts as deliberately handed over, the way a link or an upload does. The ambient gate that silences long typed messages in a shared room reads length as a stand-in for intent, and it cannot read speech. Nobody holds the mic button for thirty seconds as a turn in a chat. On the rig, without this, voice capture in a normal family room went from always working to never working. - Transcripts are cached on disk, keyed by event id. They are derived data, not a record — the audio on the timeline is the record — but whisper costs minutes of GPU per memo and a backfill over years of recordings must never pay for the same one twice. The raw text is kept beside the polished text so a better model can re-polish without touching the audio again. Removes scribe-bot, whose only job this was. It had been in zero rooms since it shipped: it declares no room, and only bots that declare one get joined anywhere. `stack messages upload` can now send audio as m.audio, so a voice message can be driven from the terminal at all.
Deleting the bot declaration stops the runner launching it, but the Matrix account survives: still joined to whatever room someone invited it to, still listed as a member, answering nothing, forever. There is no deprovisioning path in the framework, so removing Scribe outright would strand exactly the people who had gone looking for it. So it ships one more release as a shell with one behaviour. It explains that voice messages are transcribed automatically now, that nothing is lost and there is nothing to set up, and then it leaves the room. It registers no message handlers and builds no transcriber; the transport does that work now. The sweep runs on every launch rather than once, so a room it could not leave is retried next boot instead of keeping a silent member. A fresh invite gets the same answer, so following an older guide does not leave one behind either. Delete this one release after it ships. The real gap it papers over is that famstack cannot retire a bot at all, which is worth fixing before the next one goes away.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Transcription moves from a capability each bot reached for to a decode in
the transport, beside decryption: below everyone, done once. A handler
written for typed text now handles speech without knowing speech exists,
and every gate a typed message passes applies to it unchanged.
Removes three transcribe call sites, two whisper clients in one process,
and the archivist's audio branches.
Scribe ships one more release as a shell that explains itself and leaves,
because the framework has no way to deprovision a removed bot: deleting
the declaration leaves the Matrix account joined to whatever rooms it was
invited to, answering nothing. Verified on the rig for both the boot
sweep and a fresh invite. That underlying gap is not fixed here and is
worth closing before another bot goes away.
Also in here, both found by running it against a real rig and a real model
rather than stubs:
installer checked existence, not size.
its own deliberation, which we wrote into the vault as what somebody
had said. Reasoning is now switched off per request, the polish is
pinned to temperature 0, and its result is verified against its own
contract instead of trusted.
Known gap, pre-existing shape but newly reachable: an emailed audio
attachment is transcribed and then dropped, because the transcript
arrives as a bot-sent message carrying
dev.famstack.attachment, which_on_textignores.