Sync fork to upstream v0.6.0#3
Conversation
- Updated GPU dependency from torch==2.8.0+cu129 to torch==2.8.0+cu126 in pyproject.toml - Changed PyTorch CUDA index URL from https://download.pytorch.org/whl/cu129 to https://download.pytorch.org/whl/cu126 - This change ensures compatibility with CUDA 12.6 runtime while maintaining the same PyTorch version (2.8.0)
The finalize block in write_chunk() called output_buffer.getvalue() before container.close(). For OGG/Opus, the final page of audio data is only written to the buffer during close(), causing ~1-2 seconds of audio to be lost. Swap the order: close container first, then read buffer. Fixes: remsky#447
Fixes remsky#453 The generate() and generate_from_tokens() methods performed a full voice tensor round-trip on every TTS request: load from .pt file → deserialize to torch.Tensor → serialize back → write to new temp file. This created ~20-30MB of transient allocations per request that fragmented the Python heap, causing RSS to grow monotonically. Changes: - Add _voice_cache dict to KokoroV1 for in-memory voice tensor caching - Add _get_voice_tensor() helper with cache lookup by path+device - Replace direct load_voice_tensor() calls with cached version - Only save to temp file when the file doesn't already exist - Cache key includes device to handle GPU/CPU switches correctly
Addresses remsky#452 Voice .pt files contain simple tensors and do not require pickle deserialization. Using weights_only=False allows arbitrary code execution through crafted pickle payloads. This change sets weights_only=True on all torch.load() calls for voice tensors, matching the behavior already used for model weight loading in load_model_weights(). Changes: - api/src/core/paths.py: Change load_voice_tensor default from weights_only=False to weights_only=True - api/src/services/tts_service.py: Add weights_only=True to torch.load call in _load_voice
…tial-state-on-apple-silicon Fix test_initial_state() test failure on apple silicon.
Adds gpt-4o-mini-tts
fix: OGG/Opus audio truncation — final page lost in write_chunk finalize
…s-only fix: use weights_only=True for voice tensor torch.load calls
fix: cache voice tensors to prevent per-request memory leak (remsky#453)
fix: downgrade PyTorch CUDA version from cu129 to cu126
…e image with cu126 wheels, compose image naming, add INCLUDE_JAPANESE flag for future build opts build(bake): redirected to *.optimized dockerfiles examples(transcription tests): long form and multilingual whisper transcription reporsts w/ WER scoring thresholds chore: release alignment prep, version WIP bumped to 0.3.0-rc
docs(readme): refresh badges
…eck race, gate latest
…waveform bugfix ci: build caching, build: container naming consistency on local compose
updated README / cleanup
…to align with OpenWebUI
…wnload_model scripts, static release artifact link
…ded cpu/gpu Dockerfiles
… RTX 50-series release image
…inimum toolkit with Blackwell (sm_120) support (CUDA 12.8) Related remsky#306 remsky#320 remsky#389 remsky#443 remsky#365
Adds a -cu128 tag to the -gpu package while keeping the default -gpu image on cu126 for Maxwell/Pascal compatibility. Explicit -cu126 aliases added to allow stable pinning against future default changes. Compose users opt into cu128 by uncommenting preset args block in docker/gpu/docker-compose.yml.
…/cu129) docs(readme): wiki links
…age for tts-client tests (remsky#468)
…e short ciruit on no-up updates to preserve responsiveness on long sessions. (remsky#471)
…y#474) * feat(api): add POST /dev/unload to release model from GPU VRAM Allows homelab deployments to free GPU memory when the TTS service is idle without stopping the container. The model reloads lazily on the next request. - ModelManager.unload(): acquires lock, calls backend.unload(), nulls _backend, then calls torch.cuda.empty_cache() if CUDA is available - ModelManager.generate(): lazy reinit when _backend is None (calls initialize() + load_model()) instead of raising RuntimeError - POST /dev/unload: 200 on success, 503 if manager not initialised, 500 on unexpected error - TTSService.model_manager annotated as Optional[ModelManager] for correct mypy narrowing at the endpoint - Full test coverage in api/tests/test_model_unload.py (11 tests) Closes remsky#473 (partial) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(model-unload): lazy reload on next request after /dev/unload Add ensure_backend() to ModelManager which reinitialises the backend and reloads the model if /dev/unload was called. All three get_backend() call sites in tts_service now await ensure_backend() first, so the first TTS request after an unload reloads the model automatically rather than raising RuntimeError: Backend not initialized. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(model-unload): add double-checked locking to ensure_backend() Addresses review feedback: the original lazy-reload in generate() ran without a lock, so a burst of requests landing while _backend is None could trigger multiple concurrent initialize()/load_model() calls. - ensure_backend(): fast-path check outside lock, then re-check inside _lock before initializing (double-checked locking pattern) - generate(): routes through ensure_backend() instead of inline check, eliminating the duplicate code path - test_ensure_backend_serializes_concurrent_reloads: 5-way concurrent gather confirms only one initialize/load_model cycle fires Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…DME with VRAM reclaim details
…and seeking (remsky#475) Re: discussions/150
-checkout@v4 → v5, setup-buildx-action@v3 → v4 across all workflows. -Ruff formatting pass: quote style, import order, line length.
…io file for 'bf_isabella' (remsky#480)
…emsky#483) Currently /dev/unload is mounted by default with no authentication, so any caller that can reach the API can unload the active model from memory and can possibly force repeated reload cycles (remsky#478). Add an 'allow_dev_unload' setting (default False) and return 403 when disabled, so the endpoint is off by default and can be re-enabled with ALLOW_DEV_UNLOAD=true for trusted deployments.
…ech (remsky#484) KPipeline only attaches timed tokens for English (misaki) voices, so /dev/captioned_speech silently returned timestamps: null for Spanish, French, Italian, Hindi and Portuguese. The model does predict a duration for every phoneme in every language (pred_dur) - the audio is rendered from exactly those durations - but the espeak path discarded them. Derive word timestamps for espeak pipelines from pred_dur directly: pred_dur[0] is BOS and pred_dur[1+i] covers phonemes[i], so summing per-character durations and splitting at the phoneme string's spaces yields exact word times. Words espeak expands into several spoken words (e.g. "1863") are reconciled by re-phonemizing per word; if counts still disagree, no timestamps are emitted (previous behavior). English is untouched, and ja/zh (non-espeak G2Ps, no space-separated word groups) deliberately keep the previous behavior. Fixes remsky#331 Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…E_DEBUG_ENDPOINTS, documentation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request bumps the project version to 0.6.0 and introduces several major features, including a POST /dev/unload endpoint to release the model from GPU VRAM with lazy reloading, opt-in /debug/* endpoints, and word timestamp generation for non-English espeak pipelines derived from phoneme durations. It also optimizes voice tensor loading with in-memory caching, improves Web UI playback and seeking behavior, and adds an integration test suite. The review feedback highlights critical performance optimizations, such as converting the pred_dur PyTorch tensor to a Python list to prevent synchronous device-to-host transfers during timestamp generation, and vectorizing the silence-trimming boundary search with NumPy. Additionally, a minor typo in a user-facing streaming notice was identified for correction.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| try: | ||
| if pred_dur is None or not phonemes or len(pred_dur) < len(phonemes) + 1: | ||
| return None |
There was a problem hiding this comment.
In _espeak_word_timestamps, pred_dur is a PyTorch tensor. Indexing a PyTorch tensor (especially if it resides on a CUDA/GPU device) inside a Python loop (pred_dur[1 + i]) triggers synchronous device-to-host transfers on every iteration. This introduces a massive performance bottleneck. Converting the tensor to a standard Python list using .tolist() before the loop avoids these roundtrips and significantly speeds up execution.
| try: | |
| if pred_dur is None or not phonemes or len(pred_dur) < len(phonemes) + 1: | |
| return None | |
| try: | |
| if pred_dur is None or not phonemes: | |
| return None | |
| if hasattr(pred_dur, "tolist"): | |
| pred_dur = pred_dur.tolist() | |
| if len(pred_dur) < len(phonemes) + 1: | |
| return None |
| non_silent_index_start, non_silent_index_end = None, None | ||
|
|
||
| for X in range(0, len(audio_data)): | ||
| if abs(audio_data[X]) > amplitude_threshold: | ||
| if abs(int(audio_data[X])) > amplitude_threshold: | ||
| non_silent_index_start = X | ||
| break | ||
|
|
||
| for X in range(len(audio_data) - 1, -1, -1): | ||
| if abs(audio_data[X]) > amplitude_threshold: | ||
| if abs(int(audio_data[X])) > amplitude_threshold: | ||
| non_silent_index_end = X | ||
| break |
There was a problem hiding this comment.
Looping through the audio array elements one-by-one in Python to find the non-silent boundaries is highly inefficient and blocks the event loop for large audio inputs. Utilizing vectorized NumPy operations with np.where is significantly faster. To safely handle the potential int16 overflow for -32768 without slow per-element casting, we can cast the entire array to np.int32 before applying np.abs.
| non_silent_index_start, non_silent_index_end = None, None | |
| for X in range(0, len(audio_data)): | |
| if abs(audio_data[X]) > amplitude_threshold: | |
| if abs(int(audio_data[X])) > amplitude_threshold: | |
| non_silent_index_start = X | |
| break | |
| for X in range(len(audio_data) - 1, -1, -1): | |
| if abs(audio_data[X]) > amplitude_threshold: | |
| if abs(int(audio_data[X])) > amplitude_threshold: | |
| non_silent_index_end = X | |
| break | |
| import numpy as np | |
| non_silent_indices = np.where(np.abs(audio_data.astype(np.int32)) > amplitude_threshold)[0] | |
| if non_silent_indices.size > 0: | |
| non_silent_index_start = int(non_silent_indices[0]) | |
| non_silent_index_end = int(non_silent_indices[-1]) | |
| else: | |
| non_silent_index_start, non_silent_index_end = None, None |
| message = `${formatLabel} output will be generated, playback and/or download will be available when generation finishes.`; | ||
| } else if (!this.audioService.supportsMSEMp3()) { | ||
| message = isFirefox | ||
| ? 'Audio streaming is not currently supported in Firefox. Playback and/or download should stilll be available when generation finishes.' |
There was a problem hiding this comment.
There is a minor typo in the user-facing Firefox streaming notice message: stilll should be corrected to still.
| ? 'Audio streaming is not currently supported in Firefox. Playback and/or download should stilll be available when generation finishes.' | |
| ? 'Audio streaming is not currently supported in Firefox. Playback and/or download should still be available when generation finishes.' |
PR Summary by QodoSync fork to upstream v0.6.0 (dev/debug gating, unload, Docker rebuild)
AI Description
Diagram
High-Level Assessment
Files changed (89)
|
Code Review by Qodo
1. Unbounded voice cache
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
| self._voice_cache: Dict[str, torch.Tensor] = {} # Cache voice tensors by path | ||
|
|
||
| async def _get_voice_tensor(self, voice_path: str) -> torch.Tensor: | ||
| """Load voice tensor with in-memory caching to avoid repeated file I/O. | ||
|
|
||
| Args: | ||
| voice_path: Path to the .pt voice file | ||
|
|
||
| Returns: | ||
| Voice tensor on the target device | ||
| """ | ||
| cache_key = f"{voice_path}:{self._device}" | ||
| if cache_key not in self._voice_cache: | ||
| self._voice_cache[cache_key] = await paths.load_voice_tensor( | ||
| voice_path, device=self._device | ||
| ) | ||
| logger.debug(f"Cached voice tensor from {voice_path}") | ||
| return self._voice_cache[cache_key] |
There was a problem hiding this comment.
1. Unbounded voice cache 🐞 Bug ☼ Reliability
KokoroV1 now stores every loaded voice tensor in _voice_cache keyed by voice_path with no eviction, so distinct voice-paths accumulate indefinitely. Because combined voices are materialized to temp files whose names include the full (user-controlled) voice expression (including float weights), clients can generate unbounded unique keys and exhaust CPU/GPU memory over time.
Agent Prompt
### Issue description
`KokoroV1` caches voice tensors in an unbounded dict keyed by `voice_path` (plus device). Since upstream supports combined voices and weights that are incorporated into a temp filename, callers can create an effectively unlimited number of distinct `voice_path` values, causing monotonically increasing CPU/GPU memory usage and eventual OOM.
### Issue Context
- `_voice_cache` retains tensors for the process lifetime unless the model is explicitly unloaded.
- Combined voices are saved under `tempfile.gettempdir()` with filenames derived from the full voice expression string (including weights), so varying weights creates new files/paths.
### Fix Focus Areas
- api/src/inference/kokoro_v1.py[84-109]
- api/src/services/tts_service.py[180-254]
Suggested implementation directions:
1) Replace `_voice_cache` with a bounded LRU (max entries or max bytes). Evict least-recently-used entries.
2) Avoid caching *generated* combined-voice tensors by default (cache only base voices shipped with the service), or canonicalize weights (e.g., round to N decimals) and enforce a max unique combinations per process.
3) Consider tracking and cleaning up combined-voice temp files when they fall out of cache / on a background janitor, so disk usage also remains bounded.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b291abe69e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "id": "gpt-4o-mini-tts", | ||
| "object": "model", | ||
| "created": 1686935002, | ||
| "owned_by": "kokoro", | ||
| }, |
There was a problem hiding this comment.
When a client follows the model ID added here and calls GET /v1/models/gpt-4o-mini-tts, retrieve_model() still uses a separate hard-coded map that omits this ID, so the endpoint returns 404 even though /v1/models advertises it and speech requests accept it via openai_mappings.json. Please add the same entry to retrieval or derive both responses from one model list.
Useful? React with 👍 / 👎.
| WORKDIR /app | ||
|
|
||
| # Copy uv-managed Python interpreter (the venv's bin/python symlinks into here) | ||
| COPY --from=builder /opt/uv-python /opt/uv-python |
There was a problem hiding this comment.
The builder stage installs Python before running uv venv --python 3.10, so uv uses that system interpreter rather than installing a managed one under UV_PYTHON_INSTALL_DIR; in that normal path /opt/uv-python is never created, and this COPY --from=builder /opt/uv-python ... makes every GPU optimized image build fail before the app is copied. Either force a uv-managed Python or drop this copy and rely on the runtime Python already installed in the next stage.
Useful? React with 👍 / 👎.
| if self._backend: | ||
| return |
There was a problem hiding this comment.
When the first request after /dev/unload hits a load error, initialize() has already assigned _backend and load_model() raises without clearing it. Every later call then takes this early return even though the backend's model is still unloaded, so generation keeps failing with Model not loaded instead of retrying the lazy reload; clear _backend on load failure or check is_loaded here.
Useful? React with 👍 / 👎.
10 participants
Mechanical catch-up of the fork's
masterto upstream remsky/Kokoro-FastAPI v0.6.0 (55 commits). No fork-specific changes here — this is upstream's own reviewed code, isolated so the fork's value-add can be reviewed separately on top of it (see the stacked PR "Fork value-add").Notable upstream changes landing with this sync:
/dev/*and/debug/*endpoints (ALLOW_DEV_UNLOAD,ENABLE_DEBUG_ENDPOINTS), voice + timestamp fixes/dev/captioned_speechDockerfile.optimized+docker-bake.hcl)Deploy note: voice aliases
nova/alloy/ash/coral/echonow map tobf_isabella(audible change if used);/v1/audio/voicesreturns objects (?legacy=trueto opt out). Our narration tooling (rippling-lms) usesaf_heart+ ungated/dev/captioned_speech+/health, so it is unaffected — verified.Merge this first, then the value-add PR.