Fix missing word timestamps for non-English (espeak) voices#484
Merged
Conversation
KPipeline only attaches timed tokens for English (misaki) voices, so /dev/captioned_speech silently returned timestamps: null for Spanish, French, Italian, Hindi and Portuguese. The model does predict a duration for every phoneme in every language (pred_dur) - the audio is rendered from exactly those durations - but the espeak path discarded them. Derive word timestamps for espeak pipelines from pred_dur directly: pred_dur[0] is BOS and pred_dur[1+i] covers phonemes[i], so summing per-character durations and splitting at the phoneme string's spaces yields exact word times. Words espeak expands into several spoken words (e.g. "1863") are reconciled by re-phonemizing per word; if counts still disagree, no timestamps are emitted (previous behavior). English is untouched, and ja/zh (non-espeak G2Ps, no space-separated word groups) deliberately keep the previous behavior. Fixes remsky#331 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
|
Thanks for this fix, LGTM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #331 (and the non-English cases in #314).
Problem
/dev/captioned_speechreturnstimestamps: nullfor every non-English voice (Spanish, French, Italian, Hindi, Portuguese).KPipelineonly attaches timed tokens on the English (misaki) path; the espeak path yieldsResultobjects withtokens=None, soKokoroV1.generate()silently skips timestamp generation.Insight
The model already knows the timing in every language: it predicts a duration for each phoneme (
pred_dur), and the audio is rendered from exactly those durations. The espeak phoneme string keeps spaces between words, so the mapping back to words is recoverable —pred_dur[0]is BOS,pred_dur[1+i]coversphonemes[i], each unit is 2/80 s (same constants asKPipeline.join_timestamps). Summing per-character durations and splitting at spaces yields word timestamps that are exact by construction, since they're the same durations the audio was generated from.What this PR does
_espeak_word_timestamps()and anelifbranch inKokoroV1.generate()that fires only when the pipeline's G2P isEspeakG2Pand the result has no timed tokens. The English path is untouched.1863→ mil ochocientos sesenta y tres) are reconciled by re-phonemizing per word and merging the groups, so the caller still gets one timestamp per input word.Validation
Running in production on a GPU deployment (v0.5.0 base). Example, Spanish
ef_dora:versus
timestamps: nullbefore. Number expansion:"En 1863 el profesor encontro 25 mapas antiguos."returns 8 word timestamps with1863spanning its three spoken words (0.19–2.02 s). English output is byte-identical to before the patch.api/tests/test_kokoro_v1.pypasses: 13 passed.🤖 Generated with Claude Code