Skip to content

Add Transcribe Calls guide - #829

Merged
hey-august merged 25 commits into
mainfrom
august-20261022-ai-transcribe
Nov 6, 2026
Merged

hey-august merged 25 commits into
mainfrom
august-20261022-ai-transcribe

Conversation

@hey-august

@hey-august hey-august commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Description

  • New guide: Transcribe calls (/docs/platform/voice/transcribe-calls). A task-first capability guide covering SWML, Relay, and REST. It walks through a first call that sends each utterance to a webhook, then covers reading the webhook events, options, AI summaries (when the session ends and on demand), stopping transcription, and transcribing the whole call after it ends with transcribe, including the calling.call.transcribe state events that also reach status_url. It also links the AI sidecar guide and call streaming.
  • New webhook reference pages for live_transcribe, built from the TypeSpec. There are three new Calling webhooks: utterance, summary, and conversation log. The event name isn't in the request body, so the pages and the guide show how to tell the events apart by channel_data.content_disposition.
  • Fixed the Relay live_transcribe reference pages (Python and TypeScript). Their examples passed {"action": "start", "language": …}, which the platform rejects. Both SDKs forward action unchanged, so it must be an object with one start, summarize, or stop key. stop and summarize also work as bare strings.
  • Relay live_transcribe reference pages now also list start.hints and the engine-dependent vad_silence_ms default, matching the SWML reference.
  • SWML live_transcribe reference: live_events sends final utterances, not partial results; the page now says so. Added a Webhooks section.
  • Cross-links:
    • a card and a rewritten transcription FAQ on the voice overview
    • links to the guide from record-calls, the SWML transcribe reference, and the ai_sidecar reference
    • the llms.txt bullet now points to the guide

Notes for reviewers

  • The TypeScript samples follow the pinned @signalwire/sdk@2.0.5, so they need updating when Install the latest Server SDK in guides instead of pinned versions #908 unpins installs. On 3.x, liveTranscribe(callId, action) and transcribeStop(callId, controlId) take their arguments positionally, and addVerb("transcribe", …) needs { skipValidation: true }. Whichever of this PR and Install the latest Server SDK in guides instead of pinned versions #908 merges second applies Devon's inline suggestions (lines 571, 825, 905, 969, 1070) and unpins the six install lines. Python is unaffected.
  • Not tested on a live call. Both sample servers were run and fed simulated webhook requests; no real call was placed.
  • Not tested on a call that uses connect. The guide says local-caller includes the other party of a connect. That follows from the audio direction.
  • Webhook delivery, verified against the platform: both webhooks are signed (X-SignalWire-Signature) and send credentials in the URL (user:pass@host) as HTTP Basic auth. Retries differ: live events are retried on a connection error or a 5xx response, and post-call status_url requests on any non-2xx response. The guide's samples leave the webhook routes open to stay short, and a tip covers both protections.
  • Unsigned end-of-session event: with live_events off, the end-of-session event can be sent without a signature (cloud-product#22323). The tip tells readers who verify signatures to keep live_events on. Remove that sentence once the fix ships.
  • Python schema validation: the Python SDK can only turn schema validation off for a whole service, so the post-call sample does that (see Bundled SWML schema and builder omit transcribe and transcribe_stop signalwire-python#129).
  • Capability matrix: Call Flow Builder has no transcription nodes, so its column is all "not available". There's no Browser SDK column, because the browser can't start transcription.
  • Spec: the new webhook models reuse the existing AICallInfo envelope. fern/apis/signalwire-rest/openapi.yaml grows by about 400 lines, all of it the three new webhooks.
  • Follow-up, not fixed here: the Relay live_translate reference pages have the same action shape bug.

Preview

  • /docs/platform/voice/transcribe-calls
  • /docs/platform/voice
  • /docs/platform/voice/record-calls
  • /docs/swml/reference/calling/live-transcribe
  • /docs/swml/reference/calling/transcribe
  • /docs/swml/reference/calling/ai-sidecar
  • /docs/server-sdks/reference/python/relay/call/live-transcribe
  • /docs/server-sdks/reference/typescript/relay/call/live-transcribe
  • /docs/apis/rest/webhooks/live-transcribe-utterance-webhook
  • /docs/apis/rest/webhooks/live-transcribe-summary-webhook
  • /docs/apis/rest/webhooks/live-transcribe-conversation-log-webhook

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Code cleanup / refactor

Related Issues

Closes #768.

Testing

Behavior verified against the platform and the API spec.

  • yarn build:specs succeeds after merging main. WebhookPayloadSnippet uses the operationIds, per main's component change.
  • yarn fern-md-check: all MDX files valid.
  • yarn fern-check: only the redirects check fails locally, because it needs FERN_TOKEN.
  • Every code block was validated against the pinned SDKs, signalwire-sdk==3.4.1 and @signalwire/sdk@2.0.5:
    • JSON and YAML parse, and bash passes bash -n.
    • Python passes py_compile and pyflakes.
    • TypeScript passes tsc --strict.
    • The Python and TypeScript SWML builder chains produce exactly the JSON samples beside them.
  • Both sample servers were run and given simulated webhook requests. Each printed the utterance, end-of-session, and post-call transcript events, and the SWML route rejected requests without credentials.
  • Webhook delivery (signing, HTTP Basic auth from URL credentials, and retries) verified against the platform for both the live transcription webhook and the post-call status_url.
  • Internal links checked by fern-check. Every #anchor checked by hand.

Tests

  • Added/updated unit tests
  • Tested manually
  • Tested with live SignalWire credentials (if applicable)

Checklist

  • I have read the CONTRIBUTING guidelines
  • My code follows the project's style guidelines
  • I have added tests for my changes (if applicable)
  • I have updated documentation (if applicable)
  • All existing tests pass

@hey-august hey-august changed the title August 20261022 ai transcribe Add Transcribe Calls guide Oct 22, 2026
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

@hey-august hey-august mentioned this pull request Oct 22, 2026
1 of 13 tasks

@Devon-White Devon-White left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked this against mod_openai main (translate_transcribe.c, webhook.c), mod_infrastructure master (relay_apis.c, relay.c, swml.c, swml_schema.c), prime-rails (the calling.transcript.* callback path), and the published signalwire-sdk 3.4.1 / 3.7.5 and @signalwire/sdk 2.0.5 / 3.7.1 packages. The platform facts are accurate. What needs to change is in the TypeScript samples, plus a few prose rules.

Before merge

  • Coordinate with #908. It unpins Server SDK installs across the guides, and on the latest TypeScript SDK four samples here break. calling.liveTranscribe takes the action positionally, so { action } nests it and the platform rejects it. transcribeStop takes control_id positionally. addVerb("transcribe", …) throws SchemaValidationError unless you pass { skipValidation: true }. Each one has a suggestion inline. Python is unaffected.
  • Language names in prose. Lines 102, 383, 500, 934–936 and 1023 name Python, TypeScript, or per-language methods, which mdx-capability-guides rules out. Suggestions inline.

Should fix

  • Line 885: both stop forms work on every surface. There's no Relay/REST split.
  • Line 628 tip: from source, the end-of-session event can arrive unsigned when live_events is off. Worth confirming before the tip says every request is signed.
  • Line 583: the REST reference example doesn't match the request, so this should be a cURL block.

Verified correct

  • Action shapes and start keys. start needs lang and a direction array (parse_transcribe_json). remote-caller is the read leg and local-caller the write leg. speech_engine is deepgram (default) or google (get_speech_engine). Sending start while a session runs stops it first (openai_transcribe_init → openai_transcribe_stop).
  • Summaries. summarize falls back to the start webhook and the default prompt, and ai_summary_prompt applies only to the end-of-session summary. On failure, a mid-call summary sends Error creating summary, and the end-of-session event omits conversation_summary.
  • Webhook models. All three match what mod_openai sends:
    • content_disposition values, channel_data fields, and microsecond timestamps
    • every utterance field, with silence_before_ms and confidence omitted when zero
    • the top-level confidence (null when the engine reports none)
    • language in conversation_log, and the remote_caller_* / local_caller_* metrics
  • Live webhook delivery. 10 s timeout. A connection error, timeout, or 5xx is retried, a 4xx isn't, and signed requests don't follow redirects. Basic auth comes from URL credentials.
  • transcribe and transcribe_stop. status_url is the only option, and transcribe_result is success or failed. REST calling.transcribe requires control_id. In the post-call callback, params.text is dropped when empty (.presence + .compact in Relay::TranscriptCallbackSerializer) and params.id is present. The callback has a 5 s read timeout (Signalwire::REQUEST_TIMEOUT), and non-2xx responses are retried (raise Signalwire::RetryableError, Sidekiq retry: 2).
  • Relay reference fix. The old {"action": "start", "language": …} example really is rejected: action gets keys outside start,stop,summarize.
  • Links and snippets. Every link target and #anchor resolves. The calling.live_transcribe, calling.transcribe, and calling.transcribe.stop request examples exist, and so do the three new webhook operation IDs.

Couldn't verify: what a live call prints (no live call was placed, per your notes), including whether the TTS prompt and the far side of a connect come through as local-caller. The write-leg direction supports both.

@hey-august

hey-august commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator Author

@Devon-White Thanks! Addressed everything except the 3.x TypeScript changes:

  • Fixed: language names in prose, both stop forms, the cURL block for REST start, and the nits. The Relay reference pages now list hints and the vad_silence_ms default.
  • Unsigned end-of-session event: confirmed in source, and it hits live_translate too. Filed as cloud-product#22323. The tip tells readers who verify signatures to keep live_events on until it's fixed.
  • SDK version pins: your TypeScript suggestions and the unpin land whenever these pins change. I'd like to merge this first and have Install the latest Server SDK in guides instead of pinned versions #908 pick up this page.
  • Since your review: I merged main and switched the payload snippets to operationIds. I also added that transcribe handles only U.S. English and what a failed mid-call summary looks like, and removed the capability matrix.

@hey-august
hey-august merged commit 6e36e7d into main Nov 6, 2026
2 checks passed
@hey-august
hey-august deleted the august-20261022-ai-transcribe branch November 6, 2026 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Transcribe calls with AI

2 participants