asr-tse

ASR-annotation-conditioned target speaker extraction — a ClearerVoice-Studio fork that replaces lip-video conditioning with time-aligned annotations from a frozen TS-ASR Qwen. See DESIGN.md for the architecture and the exact visual-branch swap.

This machine is a dev/scaffolding box (Apple silicon, no CUDA). Heavy compute (P0 training, P1 annotation with the 30B Qwen, P2 runs) targets remote GPU. What runs here: the CPU fusion smoke tests, on dummy annotations.

Setup

uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python torch numpy pyyaml einops pytest

Smoke test the P2 fusion path (no weights, no data, no GPU)

.venv/bin/python -m pytest -q

Verifies the full swap is shape-correct: annotations at a different frame rate than the extractor fuse without resampling, cue-dropout and re-injection run, and all five ablation configs load.

Layout

asr_tse/
  annotator/   frozen annotation-service contract + stub + on-disk cache (P1)
  fusion/      layer-mix, cross-attention, annotation encoder, drop-in separator (P2)
  config/      base.yaml (3 weight-drop placeholders) + ablations/ (ablation = config)
  tests/       CPU smoke tests
ClearerVoice-Studio/   external reference (git-ignored), adapted from

Weight-drop day

Edit three lines in asr_tse/config/base.yaml (network_annotation.hidden_dim, .frame_rate, .speaker_embedding) and add the real annotator wrapper. Nothing else should need to change.