Skip to content

fix(stt): end dictation when Transcribe drops the stream - #17925

Closed
amadsalmon wants to merge 1 commit into
kirodotdev:mainfrom
amadsalmon:fix/transcribe-stream-end-notice
Closed

amadsalmon wants to merge 1 commit into
kirodotdev:mainfrom
amadsalmon:fix/transcribe-stream-end-notice

Conversation

@amadsalmon

@amadsalmon amadsalmon commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When Amazon Transcribe ends a live dictation stream on its own, the dashboard stops showing "Listening" and tells the user dictation stopped.

Amazon Transcribe sometimes ends the result stream mid-dictation: it refuses an audio event ("the request signature we calculated does not match") or hears no audio for 15 seconds. The gateway only looked at the result stream at teardown, so the socket stayed open. The composer kept the red microphone and the Listening pill, and everything said after that point was silently lost until the user stopped.

Why it matters

Anyone dictating on the transcribe provider can lose minutes of speech with no sign that anything went wrong. The reporter saw it in 6 of 53 sessions longer than 30 seconds over two days.

Not a goal

What changed (motivation → approach → change)

The read loop only wakes on client data, and nothing watched the result stream while it ran, so its end went unnoticed until the user stopped. The apple provider already ends the session when its helper fails; transcribe now does the same.

A watcher task in the Transcribe branch of api_ws_stt (dashboard/stt_stream.py) waits on the result stream. If the stream ends before the client stops, the watcher sends an error frame with stt_session_failed and closes the socket. The dashboard already handles that frame: it ends capture, keeps the text transcribed so far and shows "The dictation session failed. Try recording again." The watcher is cancelled before end_stream() at teardown, so the normal end after a stop is not reported. The 5-minute cap and the watcher share one first-claimant record, so only one error frame goes out and stt_stream_end records error or timeout for whichever came first. The cap, which since #18199 stops the reader and sends its frame after the drain, skips that frame when an early end claimed the session first. The watcher logs one WARNING, Amazon Transcribe ended the stream before the client stopped, which is the only record when the stream ends without an exception; when it ends with one, the existing drain also logs the cause. The stt-streaming spec documents the new error case.

Backwards compatibility

Compatible: the wire contract is unchanged. The new frame uses an existing code the dashboard already renders, and a session the user stops ends exactly as before.

Removes nothing: the diff adds a watcher and an error path; no field, frame, code or reader goes away.

Tests

  • test_a_result_stream_that_ends_early_ends_the_session[raises|returns]: a result stream that fails or simply ends while the client is still connected produces one stt_session_failed frame, a close, stt_stream_end = error, and end_stream() still runs.
  • test_a_clean_stop_is_not_reported_as_an_early_end: after a stop, the stream's normal end sends no frame and audits ok.
  • test_a_cap_landing_during_an_early_end_sends_no_second_frame: holds the early-end frame in flight until the duration cap has run, then asserts the cap sent nothing and the audit stays error. Deleting the first-claimant check makes it fail.
  • The Transcribe stubs' handle_events now waits for end_stream() like the real stream, instead of returning at once; the live run below confirms the real one does.
  • Every websocket read the new tests make is bounded by _bounded(), so a regression that leaves the server silent fails the test by name in 5 s instead of hanging until the 300 s cap.

Determinism: the 10 affected node ids passed 20/20 at -n0, 10/10 at -n 4 --dist loadgroup --max-worker-restart=2 beside test_stt_stream_cov80.py and test_stt_session.py, and a shuffled run (seed 32584). prove.py: PROVEN (the new tests fail with the fix reverted).

Manual verification

Ran this branch's api_ws_stt handler against live Amazon Transcribe (us-east-1): real SDK, real result-stream handler, no stubs, on an in-process aiohttp server with only the consent gate and origin check bypassed. Two sessions of synthetic 16 kHz PCM:

Session What it sent Frames the client got Close stt_stream_end
Healthy 50 s: five tone bursts separated by 2-4 s of silence, then stop ready only, no error 0.2 s after stop ok
Forced early end 2 s of tone, then no audio with the socket left open ready, then stt_session_failed at 18 s at 18 s, with the frame error

In the second session Transcribe refused with "Your request timed out because no new audio was received for 15 seconds."; the watcher logged Amazon Transcribe ended the stream before the client stopped and the existing drain logged the cause. So the result stream does not end on its own across silent stretches in a healthy session, and an early end reaches the client within a few seconds.

Related Issues

Part of #17842

Pattern harvest

Rule candidate: review-prompt
Pattern: a background stream or task whose failure is awaited only in teardown, so the session looks alive after the stream has died

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@amadsalmon
amadsalmon requested a review from a team as a code owner October 7, 2026 23:47
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention fork Pull request from a fork (external contributor) labels Oct 7, 2026
@amadsalmon
amadsalmon force-pushed the fix/transcribe-stream-end-notice branch from d562e7d to ac5e6ea Compare October 8, 2026 00:34
@amadsalmon

Copy link
Copy Markdown
Contributor Author

The workflow runs for the current head are waiting for maintainer approval (fork PR, all at action_required). Could a maintainer approve them? This push removes the (#17842) docstring reference that failed the Comment History Gate on the previous head.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 198904290af889f96eea12ead8acf0a8b5e573ae touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 198904290af889f96eea12ead8acf0a8b5e573ae via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 1989042

Description read

[DESCRIPTION-READ] 279ed5cabddd48b06c9ea5a5ab6b9bdcd41eefe0397ebf4aceda250518bb16b5

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 198904290af889f96eea12ead8acf0a8b5e573ae via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check that a healthy session where the user pauses mid-dictation still sends audio frames, so Transcribe's 15-second no-audio refusal can't trigger in normal use.

What this change ships

Inventory (6 items) — 6 justified

Intent: when Amazon Transcribe ends a live dictation stream on its own, the dashboard stops showing "Listening" and tells the user dictation stopped. This is a FIX.

  1. If Transcribe ends the stream early, dictation stops and shows "The dictation session failed" instead of silently dropping speech — justified
  2. The stt_stream_end audit records error for an early-ended stream instead of ok — justified
  3. One WARNING log line when Transcribe ends the stream before the client stops — justified
  4. If the 5-minute cap fires during an early end, it sends no second frame and keeps the first outcome — justified
  5. The stt-streaming spec documents the new error case — justified
  6. The Transcribe test stubs keep the result stream open until end_stream(); the old handler-exception test becomes a two-case test that still checks the failure is logged, not raised — justified

[FIRST-PRINCIPLES-REVIEWED] 1989042

Description read

[DESCRIPTION-READ] 565f7e885ebe7cfa1aa38367b7382da96ccf6b7e809ef54d199873104015004a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 198904290af889f96eea12ead8acf0a8b5e573ae via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1989042

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 198904290af889f96eea12ead8acf0a8b5e573ae via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1989042

@amadsalmon
amadsalmon force-pushed the fix/transcribe-stream-end-notice branch from ac5e6ea to c5906c2 Compare October 8, 2026 01:27
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 8, 2026
@amadsalmon
amadsalmon force-pushed the fix/transcribe-stream-end-notice branch from c5906c2 to 438c1df Compare October 8, 2026 01:32
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 8, 2026
@amadsalmon

amadsalmon commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: every websocket read in the tests this PR adds is bounded by _bounded()

  • BLOCKING test/test_stt_stream.py:977 unbounded websocket receive span=a2d611b8d4fe — fixed in 438c1df

Every client-side ws.receive_json() / ws.receive() and the frames-until-close read in the tests this PR adds now goes through _bounded(), an asyncio.wait_for at _AUDIT_WAIT_TIMEOUT_SECS (5 s). That covers the quoted test and the same shape in test_a_cap_landing_during_an_early_end_sends_no_second_frame and test_a_clean_stop_is_not_reported_as_an_early_end.
Checked: with the watcher removed, test_a_result_stream_that_ends_early_ends_the_session[returns] fails by name with TimeoutError in under 7 s instead of waiting for the 300 s cap.
The 10 affected node ids pass 20/20 at -n0, 10/10 at -n 4 with co-tenants, and a shuffled run.

@amadsalmon

amadsalmon commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: no

  • The stubs encode the premise; nothing live confirms it span=6f31e9a7f79b — fixed (verified live, recorded in the PR description)

Ran this branch's api_ws_stt against live Amazon Transcribe in us-east-1: real SDK and result-stream handler, no stubs, only the consent gate and origin check bypassed.
(a) Healthy session: 50 s of tone bursts separated by 2-4 s of silence, then stop. The client received ready and no error; the socket closed 0.2 s after stop; stt_stream_end = ok. The result stream did not end on its own across the silent stretches.
(b) Forced early end: 2 s of tone, then no audio with the socket open. Transcribe refused ("no new audio was received for 15 seconds"); the client received stt_session_failed and the close at 18 s; stt_stream_end = error.
Both conditions of the Clears-when line are now in the "Manual verification" section of the description.

@amadsalmon

amadsalmon commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • Item 4 — the new warning log is undeclared span=e39e06e03fbc — fixed in 438c1df (description and code comment)

The warning stays. It is the only record when Transcribe ends the result stream without raising: handle_events() then returns normally, so the drain's Transcribe handler task failed never fires and nothing else logs the early end.
The description now declares it: one WARNING from the watcher, plus the drain's existing line when the stream ended with an exception. The code comment that said "logged once, by the drain" now reads "An exception's cause is logged by the drain below".
On the check-before-merge note: a 15 s silent pause already ends the stream today. Transcribe refuses it server-side ("no new audio was received for 15 seconds"); this PR only stops the UI from claiming it is still listening afterwards.

@amadsalmon

amadsalmon commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author

Ready for maintainer review on 198904290, a rebase onto main 401a14be7 that resolves conflicts with #18199, which reworked the duration cap in stt_stream.py. The cap now claims the same first-claimant record as the result-stream watcher, so its frame is skipped when an early end got there first.

On this head, GPT 6.1 and Opus 5.5 report no blocking findings, Design and First Principles pass, and UX skipped (no UI change). All backend shards and E2E pass.

The two red jobs are not from this PR (PR Readiness is red only because of them):

Could a maintainer rerun them and review?

gh run rerun 37986912231 --failed --repo kirodotdev/KiroCrew
gh run rerun 37986912372 --failed --repo kirodotdev/KiroCrew

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 9, 2026
Amazon Transcribe can end its result stream mid-dictation, for example
when it refuses an audio event's signature or hears no audio for 15
seconds. The gateway only looked at the result stream at teardown, so
the socket stayed open, the dashboard kept showing it was listening and
everything said afterwards was lost until the user stopped.

Watch the result stream while the session runs. If it ends before the
client stops, send stt_session_failed and close the socket, the same
way the apple relay already does. The cap and the watcher share one
first-claimant record, so only one error frame goes out and the end
audit records which one ended the session.

Part of kirodotdev#17842
@amadsalmon
amadsalmon force-pushed the fix/transcribe-stream-end-notice branch from 438c1df to 1989042 Compare October 9, 2026 20:25
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention labels Oct 9, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 9, 2026
@buluoray

Copy link
Copy Markdown
Contributor

Taking this over in #19550 since it has been idle; you are credited as co-author on the commits. Thanks for the groundwork.

@amadsalmon

Copy link
Copy Markdown
Contributor Author

@buluoray

This PR is not idle. I opened it only five days ago.
I would like to keep full credit for the fix through both the PR and the commit.

The issue is that the CI checks are so flaky and unreliable that it makes it hard to keep up with them and get them to pass to green.

I would appreciate some more support from the maintainers of this package to make it easier to pass the CI checks, without that support being taking over my PRs completely.

buluoray added a commit that referenced this pull request Oct 13, 2026
…s credentials (#19550)

* fix(stt): end dictation when Transcribe drops the stream

Amazon Transcribe can end the result stream mid-dictation (a refused
event signature, 15 s without audio). The gateway only awaited the
result stream at teardown, so the socket stayed open and the dashboard
kept showing Listening while every later word was lost.

A watcher task now waits on the result stream. If it ends before the
client stops, it sends an stt_session_failed error frame and closes the
socket, the way the apple relay already does. The duration cap and the
watcher share one first-claimant record, so only one frame goes out and
stt_stream_end records the cause that came first.

The new tests bound their reads with default_timeout() from
kiro_crew.testing.wait, which replaced _AUDIT_WAIT_TIMEOUT_SECS on main.

Supersedes #17925.

Co-authored-by: Amad Salmon <amad.salmon@icloud.com>

* fix(stt): sign a Transcribe stream with its opening credentials

The streaming SDK asks the credential resolver again for every audio
event, and each event's signature chains from the opening request's.
_ProfileCredentialResolver asked the boto3 session every time, so a
profile whose credentials refreshed mid-stream signed the next event
with a different key and Transcribe refused it with "signature does not
match".

The resolver, built once per stream, now resolves once and returns the
same credentials on every later call. botocore refreshes credentials
with under ten minutes left before handing them out, so the pinned ones
outlive the five-minute stream cap.

Supersedes #17924.

Co-authored-by: Amad Salmon <amad.salmon@icloud.com>

---------

Co-authored-by: Amad Salmon <amad.salmon@icloud.com>
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 13, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

@amadsalmon Hello Amad, please try to keep the PR updated frequently, the PRs that have no updates in 5 days will be considered as idle from our current system setting. Happy to keep this thing for you, just kind remind for more updates.

@iamwhatever

Copy link
Copy Markdown
Contributor

Since the related PR has already been merged, I would suggest you to check whether anything more can be added here. Also we could somehow grant you the write permission so that you could get faster loop (I am seeing you have made many contributions before so you deserve this thing). Please let me know if you would like to be added, and I can ask admin to send the invite

@iamwhatever

Copy link
Copy Markdown
Contributor

Thanks again @amadsalmon for this fix and the live Transcribe verification. The same change has landed on main through #19550 (merge commit 97beb72), with you credited as co-author on its commits. Current main's src/kiro_crew/dashboard/stt_stream.py has the result-stream watcher (_watch_results), the stt_session_failed frame and the 'Amazon Transcribe ended the stream before the client stopped' warning, and #17842 is closed as completed. Since nothing in this branch is left to add on top of main, I'm closing it. If you spot a gap that #19550 missed, please open a fresh PR against main and we'll review it quickly.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 23, 2026
@amadsalmon

Copy link
Copy Markdown
Contributor Author

@iamwhatever thank you, and sorry for the slow reply. I missed your message. Yes, I'd like write access, if the offer still stands. Most of my time on #9749 and #9750 goes to waiting for fork CI approvals and rerunning fork lanes I can't rerun myself, so it would speed things up a lot. Please ask the admin to send the invite whenever it suits you. Thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants