fix(asr/tdt): default v3 long-form to the no-mel path (silence-aligned starts) - #869
Conversation
…d starts) Fixes #803's quiet-speech drops near long mid-file silence runs. The issue's proposed fix — trimming a window's declared end back past a trailing dead-silence run — was implemented and validated as not viable: the failing window decodes all-blank (rawNonBlank=0) even when trimmed AND backfilled to a full window of real audio ending in speech, and applying the trim to the silence-aligned path regressed the case it already handled. The actual trigger is fixed-stride window *starts* landing mid-word on quiet speech (the #594 SOS-collapse family), which the no-mel path's silence-aligned chunk starts avoid. ASRConfig.melChunkContext becomes a tri-state (Bool?): nil (the new default) resolves per model version — false on v3, true elsewhere — so v2/110m keep the PR #264 prepend they were validated with, and explicit opt-ins in either direction are preserved. The CLI gains --mel-context to force the prepend back on for v3. Validation (release binary, M-series): - musan librivox-0000 (444s, quiet chapter outro at ~428-438s): outro PRESENT on the new default; DROPPED with --mel-context (old default). - Single-window clips byte-identical to main (layout only differs for multi-window files). - Prior A/B (2026-08-01, main binary): mel vs no-mel is net-neutral on 4x hour-long Earnings-22 (16.69% vs 16.86% aggregate WER, per-file within ±1.5%), and no-mel is better or tied on every v3-supported language on concatenated FLEURS long-form (en 6.38→5.47, de 3.98→3.10, es 5.01→3.13, fr 16.61→15.61, it 4.67→2.64, pt 7.66→5.11).
PocketTTS Smoke Test ✅
Runtime: 0m22s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon. |
Supertonic3 Smoke Test ✅
Runtime: 0m36s Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf. |
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 54.7s diarization time • Test runtime: 3m 7s • 08/19/2026, 12:59 PM EST |
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 114.1s processing • Test runtime: 2m 13s • 08/19/2026, 01:07 PM EST |
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 1m36s • 08/19/2026, 01:05 PM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 14m46s • 08/19/2026, 01:04 PM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 3m 7s • 2026-08-19T16:53:32.782Z |
…ted shim Review follow-up on #869: the tri-state moved to a new stored property melChunkContextOverride, and the old public melChunkContext name remains as a deprecated Bool computed view (explicit setting, or the non-v3 default true, when unset). Existing consumers that read the flag keep compiling; the init parameter label is unchanged so constructor call sites were never affected. Internal pass-throughs read the override directly to preserve nil-ness.
✅ Japanese ASR Benchmark Results (CTC)Status: Passed
✅ Benchmark completed successfully. The TDT Japanese hybrid model (CTC preprocessor/encoder + TDT decoder/joint) is working correctly. View benchmark log |
1 participant
Addresses #803 for the v3 default path. The opt-in mel-context path (
--mel-context/ explicitmelChunkContext: true) and v2/110m — which keep the PR #264 prepend they were validated with — retain the interior-silence limitation; #803 stays open as the tracker for those paths (documented inLongTranscription.md→ Known Limitations).Problem
A sliding window whose fixed-stride boundary interacts with quiet speech near long silence runs decodes degenerately, silently dropping content mid-file (#803's librivox-0000 quiet chapter outro is the acceptance case).
Why not the trim fix the issue sketched
The issue proposed snapping a window's declared end back past a trailing dead-silence run. Implemented and validated — not viable:
rawNonBlank=0, probed inTdtDecoderV3) even when trimmed and backfilled to a full 15s of real audio ending in speech — trailing silence is not the trigger.The real trigger is fixed-stride window starts landing mid-word on quiet speech (the #594 SOS-collapse family). The v3 no-mel path's
silenceAlignedChunkStartsavoids the class — it already recovers the librivox outro onmainbehindmelChunkContext = false.Change
The mel-context setting becomes tri-state, stored as
ASRConfig.melChunkContextOverride: Bool?.nil(new default) resolves per model version at load:falseon v3 (no-mel + silence-aligned starts),trueelsewhere. Explicit opt-ins in either direction are preserved; CLI gains--mel-contextto force the prepend back on.API compatibility: the init parameter label is unchanged (
melChunkContext:— existing constructor call sites compile as-is,Boolpromotes toBool?), and the oldmelChunkContext: Boolproperty remains as a deprecated computed view, so consumers that read the flag keep compiling with a deprecation warning steering them to the tri-state.Validation
--mel-context(old default)main(layout only differs multi-window)Unit test added for the tri-state resolution (
testMelChunkContextAutoResolution). Docs updated (LongTranscription.md: Current Paths table + Known Limitations bullet rewritten with the measured mechanism).🤖 Generated with Claude Code