Repository navigation
fix(stt): end dictation when Transcribe drops the stream - #17925
amadsalmon wants to merge 1 commit into
Conversation
d562e7d to
ac5e6ea
Compare
|
The workflow runs for the current head are waiting for maintainer approval (fork PR, all at |
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] 1989042 Description read[DESCRIPTION-READ] 279ed5cabddd48b06c9ea5a5ab6b9bdcd41eefe0397ebf4aceda250518bb16b5 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check that a healthy session where the user pauses mid-dictation still sends audio frames, so Transcribe's 15-second no-audio refusal can't trigger in normal use. What this change shipsInventory (6 items) — 6 justifiedIntent: when Amazon Transcribe ends a live dictation stream on its own, the dashboard stops showing "Listening" and tells the user dictation stopped. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 1989042 Description read[DESCRIPTION-READ] 565f7e885ebe7cfa1aa38367b7382da96ccf6b7e809ef54d199873104015004a That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
ac5e6ea to
c5906c2
Compare
c5906c2 to
438c1df
Compare
|
self-added: yes
|
|
self-added: no
|
|
self-added: yes
|
|
Ready for maintainer review on On this head, GPT 6.1 and Opus 5.5 report no blocking findings, Design and First Principles pass, and UX skipped (no UI change). All backend shards and E2E pass. The two red jobs are not from this PR (PR Readiness is red only because of them):
Could a maintainer rerun them and review? |
Amazon Transcribe can end its result stream mid-dictation, for example when it refuses an audio event's signature or hears no audio for 15 seconds. The gateway only looked at the result stream at teardown, so the socket stayed open, the dashboard kept showing it was listening and everything said afterwards was lost until the user stopped. Watch the result stream while the session runs. If it ends before the client stops, send stt_session_failed and close the socket, the same way the apple relay already does. The cap and the watcher share one first-claimant record, so only one error frame goes out and the end audit records which one ended the session. Part of kirodotdev#17842
438c1df to
1989042
Compare
|
Taking this over in #19550 since it has been idle; you are credited as co-author on the commits. Thanks for the groundwork. |
|
This PR is not idle. I opened it only five days ago. The issue is that the CI checks are so flaky and unreliable that it makes it hard to keep up with them and get them to pass to green. I would appreciate some more support from the maintainers of this package to make it easier to pass the CI checks, without that support being taking over my PRs completely. |
…s credentials (#19550) * fix(stt): end dictation when Transcribe drops the stream Amazon Transcribe can end the result stream mid-dictation (a refused event signature, 15 s without audio). The gateway only awaited the result stream at teardown, so the socket stayed open and the dashboard kept showing Listening while every later word was lost. A watcher task now waits on the result stream. If it ends before the client stops, it sends an stt_session_failed error frame and closes the socket, the way the apple relay already does. The duration cap and the watcher share one first-claimant record, so only one frame goes out and stt_stream_end records the cause that came first. The new tests bound their reads with default_timeout() from kiro_crew.testing.wait, which replaced _AUDIT_WAIT_TIMEOUT_SECS on main. Supersedes #17925. Co-authored-by: Amad Salmon <amad.salmon@icloud.com> * fix(stt): sign a Transcribe stream with its opening credentials The streaming SDK asks the credential resolver again for every audio event, and each event's signature chains from the opening request's. _ProfileCredentialResolver asked the boto3 session every time, so a profile whose credentials refreshed mid-stream signed the next event with a different key and Transcribe refused it with "signature does not match". The resolver, built once per stream, now resolves once and returns the same credentials on every later call. botocore refreshes credentials with under ten minutes left before handing them out, so the pinned ones outlive the five-minute stream cap. Supersedes #17924. Co-authored-by: Amad Salmon <amad.salmon@icloud.com> --------- Co-authored-by: Amad Salmon <amad.salmon@icloud.com>
|
@amadsalmon Hello Amad, please try to keep the PR updated frequently, the PRs that have no updates in 5 days will be considered as idle from our current system setting. Happy to keep this thing for you, just kind remind for more updates. |
|
Since the related PR has already been merged, I would suggest you to check whether anything more can be added here. Also we could somehow grant you the write permission so that you could get faster loop (I am seeing you have made many contributions before so you deserve this thing). Please let me know if you would like to be added, and I can ask admin to send the invite |
|
Thanks again @amadsalmon for this fix and the live Transcribe verification. The same change has landed on main through #19550 (merge commit 97beb72), with you credited as co-author on its commits. Current main's |
|
@iamwhatever thank you, and sorry for the slow reply. I missed your message. Yes, I'd like write access, if the offer still stands. Most of my time on #9749 and #9750 goes to waiting for fork CI approvals and rerunning fork lanes I can't rerun myself, so it would speed things up a lot. Please ask the admin to send the invite whenever it suits you. Thanks again! |
Problem / Motivation
Goal: When Amazon Transcribe ends a live dictation stream on its own, the dashboard stops showing "Listening" and tells the user dictation stopped.
Amazon Transcribe sometimes ends the result stream mid-dictation: it refuses an audio event ("the request signature we calculated does not match") or hears no audio for 15 seconds. The gateway only looked at the result stream at teardown, so the socket stayed open. The composer kept the red microphone and the Listening pill, and everything said after that point was silently lost until the user stopped.
Why it matters
Anyone dictating on the
transcribeprovider can lose minutes of speech with no sign that anything went wrong. The reporter saw it in 6 of 53 sessions longer than 30 seconds over two days.Not a goal
What changed (motivation → approach → change)
The read loop only wakes on client data, and nothing watched the result stream while it ran, so its end went unnoticed until the user stopped. The
appleprovider already ends the session when its helper fails;transcribenow does the same.A watcher task in the Transcribe branch of
api_ws_stt(dashboard/stt_stream.py) waits on the result stream. If the stream ends before the client stops, the watcher sends anerrorframe withstt_session_failedand closes the socket. The dashboard already handles that frame: it ends capture, keeps the text transcribed so far and shows "The dictation session failed. Try recording again." The watcher is cancelled beforeend_stream()at teardown, so the normal end after a stop is not reported. The 5-minute cap and the watcher share one first-claimant record, so only one error frame goes out andstt_stream_endrecordserrorortimeoutfor whichever came first. The cap, which since #18199 stops the reader and sends its frame after the drain, skips that frame when an early end claimed the session first. The watcher logs one WARNING,Amazon Transcribe ended the stream before the client stopped, which is the only record when the stream ends without an exception; when it ends with one, the existing drain also logs the cause. The stt-streaming spec documents the newerrorcase.Backwards compatibility
Compatible: the wire contract is unchanged. The new frame uses an existing code the dashboard already renders, and a session the user stops ends exactly as before.
Removes nothing: the diff adds a watcher and an error path; no field, frame, code or reader goes away.
Tests
test_a_result_stream_that_ends_early_ends_the_session[raises|returns]: a result stream that fails or simply ends while the client is still connected produces onestt_session_failedframe, a close,stt_stream_end=error, andend_stream()still runs.test_a_clean_stop_is_not_reported_as_an_early_end: after a stop, the stream's normal end sends no frame and auditsok.test_a_cap_landing_during_an_early_end_sends_no_second_frame: holds the early-end frame in flight until the duration cap has run, then asserts the cap sent nothing and the audit stayserror. Deleting the first-claimant check makes it fail.handle_eventsnow waits forend_stream()like the real stream, instead of returning at once; the live run below confirms the real one does._bounded(), so a regression that leaves the server silent fails the test by name in 5 s instead of hanging until the 300 s cap.Determinism: the 10 affected node ids passed 20/20 at
-n0, 10/10 at-n 4 --dist loadgroup --max-worker-restart=2besidetest_stt_stream_cov80.pyandtest_stt_session.py, and a shuffled run (seed 32584).prove.py: PROVEN (the new tests fail with the fix reverted).Manual verification
Ran this branch's
api_ws_stthandler against live Amazon Transcribe (us-east-1): real SDK, real result-stream handler, no stubs, on an in-process aiohttp server with only the consent gate and origin check bypassed. Two sessions of synthetic 16 kHz PCM:stt_stream_endstopreadyonly, noerrorstopokready, thenstt_session_failedat 18 serrorIn the second session Transcribe refused with "Your request timed out because no new audio was received for 15 seconds."; the watcher logged
Amazon Transcribe ended the stream before the client stoppedand the existing drain logged the cause. So the result stream does not end on its own across silent stretches in a healthy session, and an early end reaches the client within a few seconds.Related Issues
Part of #17842
Pattern harvest
Rule candidate: review-prompt
Pattern: a background stream or task whose failure is awaited only in teardown, so the session looks alive after the stream has died
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)