tts-server: WAV voice registration runs reference extraction on the HTTP thread and degrades all later CPU synthesis even with --max-batch 2

[minamikik]

Summary

On the CPU backend with --max-batch 2, registering a cloned voice from a WAV payload leaves a retained caller-local OpenMP helper team on the HTTP handler thread. After that, every synthesis request in the same server process (streaming or buffered, ICL or x-vector-only) drops from ~7.0 to ~2.7 effective cores (about 38% of the configured threads). The trigger is that qt_extract_voice_ref() executes its speaker-encoder and codec graphs synchronously on the HTTP handler, while synthesis compute runs on the batch worker; once both threads hold a libgomp team, the active synthesis team loses most of its parallelism even though the registration helpers stay asleep.

This is related to but distinct from #31: that report is about buffered synthesis on HTTP worker threads accumulating one team per connection when no batch scheduler is active. Here the batch scheduler is active and protects synthesis itself; the remaining unprotected path is registration.

Environment

• qwentts.cpp at a8a7716, CPU backend, OpenMP on, BLAS off
• Intel Core Ultra 5 235 (14 cores, one affinity mask), Linux (WSL2)
• server auto-selected 7 threads; tts-server --max-batch 2
• 0.6B Base Q4_K_M talker + 12hz Q4_K_M tokenizer (also reproduced with 0.6B Q8_0, 1.7B Q4_K_M / Q8_0)
• effective cores measured as process CPU ticks over request wall time, with /proc/<pid>/task sampled every 20 ms for per-task state and ticks

Reproduction

1. Start tts-server --max-batch 2 with a Base checkpoint.
2. Register a cloned voice from a 24 kHz mono s16 WAV (POST /v1/audio/voices with wav_b64, with or without ref_text).
3. Send streaming PCM synthesis requests and measure effective cores.

Task counts follow exact n_threads - 1 arithmetic twice: 21 tasks at startup, +6 after WAV registration (team on the HTTP handler), +6 after the first synthesis (team on the batch worker), stable at 33 afterwards.

scenario	effective cores	ratio /7
WAV registration, ICL synthesis	2.693	0.385
WAV registration, x-vector-only (ref_text empty)	2.684	0.384
WAV registration, ICL, OMP_NUM_THREADS=1	2.730	0.390
WAV registration, ICL, OPENBLAS_NUM_THREADS=1	2.703	0.386
pre-encoded .spk/.rvq registration, same ICL synthesis	6.995	0.999

During a degraded request, the six registration-created helpers sleep in futex_do_wait for the whole request with zero additional CPU ticks; only the batch worker and its six helpers compute, and those seven tasks together reach only ~2.7 effective cores. OMP_NUM_THREADS=1 has no effect because ggml-cpu enters #pragma omp parallel num_threads(n_threads) explicitly; OPENBLAS_NUM_THREADS=1 is a no-op because the build links no BLAS.

The pre-encoded control is the decisive comparison: registering the same reference as .spk/.rvq (extracted out of process with qwen-codec --talker) performs no graph compute in the server, the HTTP-thread team is never created (21 → 21 → 27), and the same ICL synthesis then runs at 6.995 effective cores and identical audio output.

What is not affected (measured)

To keep the fix scoped, the following were explicitly checked and are healthy under --max-batch 2:

• Two simultaneous streaming requests occupy both scheduler slots, but all GGML work for both slots executes on the single long-lived batch worker; the durable task count stays at 27 and no second team appears. Singles measured immediately after a concurrent wave run at 6.99/7 effective cores, both for built-in speakers and for pre-encoded cloned voices.
• Buffered WAV synthesis also goes through qt_synthesize(), which in batch mode only enqueues a job and blocks; compute stays worker-owned (21 → 27 once, then stable, 6.95/7 afterwards).
• Built-in speaker serving cannot reach the extraction path, so it is unaffected unless the process previously accepted a WAV registration.

So the breaking condition is not concurrency, response format, or slot count: it is any GGML CPU graph entered from a caller other than the batch worker master. Each distinct caller that reaches graph compute retains its own n_threads - 1 helper team. At the current pin, WAV registration is the only serving path that does this when max_batch > 1.

Suggested fix

When max_batch > 1, route the backend work of qt_extract_voice_ref() through the existing batch worker as a queued control job (with a completion condition variable), instead of only serializing it under gpu_mu on the caller. gpu_mu serializes the work but does not move it, so the caller still becomes an OpenMP master and retains a team. Routing extraction through the worker preserves the serialization semantics while keeping a single GGML-compute owner thread.

More generally, it would be good to keep the invariant that every request-time GGML compute path under max_batch > 1 (including control/extraction work added in the future) executes on the scheduler worker.

Workaround at the current pin

Pre-extract references out of process with qwen-codec --talker ... and register spk_b64 + rvq_b64 + ref_text instead of wav_b64. This keeps the serving process free of caller-local teams and restores full parallelism.

One open detail: why a fully idle retained team makes another caller's active team spend most of its time in futex waits was not root-caused inside libgomp; the trigger, the healthy control, and the fix location above are established without it (same open question as in #31).