Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "vox",
"version": "3.1.0",
"version": "3.1.1",
"description": "Agent-facing speech to text and text to speech over Alibaba Model Studio.",
"repository": "https://github.com/celados/vox",
"bin": {
Expand Down
235 changes: 35 additions & 200 deletions src/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,206 +6,41 @@ description: Agent-facing contract for transcription, speaker identity, and spee

# vox

File transcription and TTS through Alibaba Model Studio, with local MLX speaker identification. `vox hear`
uploads a file, transcribes it, and stores a **run**; markdown, subtitles, and
plain text are exports from that run. There is no realtime ASR and no
microphone capture for transcription. Files up to 12 hours are ordinary input.

Do not invent commands. `vox @schema` is the surface.

## First moves

```bash
vox @schema
vox status
```

Provider keys need no setup. Commands that call a provider take an optional
`apiKey` field; omitted, the default chain runs: the env var
(`DASHSCOPE_API_KEY`, `MIMO_API_KEY`) if set, else the celados-agent vault
through latch. To use another key, pass a reference, never paste the key:
`apiKey: { ref: 'env:MY_KEY' }` or `{ ref: 'bw://<item-id>/<field>' }`.
`vox status` resolves both chains and shows, per provider, `available` or what
it `tried` — never key material. Stored runs, `export`, `session.*`,
`speaker.*` and cached `say` read no key.

## Speech to text

```bash
vox hear --file recording.wav
vox hear --file lecture.mp3 --lang zh
vox hear "{ file: 'lecture.mp3', lang: ['zh'] }"
vox hear "{ file: 'meeting.m4a', speakers: true, vocab: 'meeting' }"
vox hear "{ file: 'recording.wav', refresh: true }"

vox session.list
vox session.list --file recording.wav
vox session.remove --sid a3f1c2
vox session.remove --all

vox export --sid a3f1c2 --format md
vox export --sid a3f1c2 --format srt --output out.srt
vox export --sid a3f1c2 --format txt
```

`--lang` is the cheapest quality win on known-language audio. `--vocab` only
biases domain terms; it cannot recover bad audio. Do not split a file to work
around length: splitting truncates words at every cut.

Vendors: `fun` (default, Fun-ASR), `qwen`, or `mimo`. DashScope's `fun` and
`qwen` cap at **12 hours / 2 GB**. MiMo accepts only MP3/WAV with Base64 payloads
up to 10 MB, returns plain text without timestamps or speaker labels, and does
not support Vox vocabularies. Use it for short A/B samples:

```bash
vox hear --file sample.mp3 --model mimo --lang zh
```

Expect roughly a minute per 45 minutes of audio from the DashScope file models.

`hear` always writes a Markdown transcript under `~/.vox/runs/<sid>/transcript.md`
and returns its path. Pass `output` to choose a destination.

## Speaker identity (Apple Silicon)

Prerequisites: `uv` and `ffmpeg` on PATH (`brew install uv ffmpeg`). ASR still
uses DashScope; voice matching runs locally. The first enrollment or identity
request lazily downloads the pinned model (~26 MB) and Python runtime dependencies.
Weights live in `~/.vox/models/`, runtime files in `~/.vox/runtime/`, and download
caches in `~/.vox/cache/uv` and `~/.vox/models/huggingface`. No PyTorch is required.

Enroll one clean solo recording per person (3–30 seconds; longer files use the
first 30 seconds by default). `start` and `end` select seconds in an existing
recording. Both enrollment and matching remove long near-digital silence before
feature extraction; at least 3 seconds must remain. This is an energy gate, not
a speech/noise classifier. Keep enrollment and evaluation recordings separate when checking quality.

```bash
vox speaker.enroll "{ id: 'ethan', name: 'Ethan', file: 'ethan.wav' }"
vox speaker.enroll "{ id: 'deniffer', name: 'Deniffer', file: 'deniffer.wav', start: 10, end: 30 }"
vox speaker.list
vox hear "{ file: 'meeting.m4a', identify: true, output: 'meeting.transcript.md' }"
```

`identify: true` automatically enables ASR speaker separation and matches against
all enrolled people. Limit candidates with `speakerIds: ['ethan', 'deniffer']`.
MiMo cannot identify speakers because it has no speaker labels.

The Markdown includes timestamps and names. Anonymous IDs remain in the stored
JSON; names and decisions are saved separately as `identified.json`. Uncertain
matches keep `Speaker <id>`, rather than forcing a name. IDs are local to each
recording; multiple IDs may match one person. Identity defaults (`threshold: 0.65`,
`margin: 0.1`) are starting points, not a calibrated accuracy guarantee.

```bash
# Re-identify a stored run after enrollment/config changes; no ASR upload or charge.
vox speaker.identify "{ sid: 'a3f1c2', output: 'meeting.transcript.md' }"
vox speaker.config
vox speaker.config "{ threshold: 0.7, margin: 0.1 }"
vox speaker.enroll "{ id: 'ethan', name: 'Ethan', file: 'better.wav', replace: true }"
vox speaker.remove --id ethan
```

`export` uses the latest saved identity snapshot for that run, including for
SRT/VTT/TXT/JSON. Re-run `speaker.identify` to apply changes to the speaker library.
If identity fails after ASR, the anonymous ASR run remains stored; use
`session.list` and `speaker.identify` to resume. No meeting transcript is modified
outside vox's own run or the explicit output destination.

Profiles and matching settings are in `~/.vox/speakers.json` (private permissions).
Source enrollment audio is not copied; only its normalized voiceprint is stored.
Treat these profiles as personal voice data. `voice.*` is still TTS voice cloning;
`speaker.*` is recognition identity. `VOX_HOME` overrides the state directory for
isolated tests. All speaker artifacts use the model version recorded with the
profiles; changing embedding models requires re-enrollment.

## Result envelopes

`hear` returns a YAML envelope. A short transcript is inlined as `text`. A long
one folds to `preview` plus `$hints` naming the `vox export` command. Do not
try to widen the envelope — run the export.

`session.list` is the same records without `text` or `preview`. `export`
without `--output` prints the document itself (not a YAML wrapper).

## Vocabularies

A vocabulary is a YAML file you write directly. There is no create/edit command.

```yaml
# ~/.vox/vocabulary/meeting.yaml
lang: zh
default_weight: 4
words:
百炼: 5
Fun-ASR: 5
赛德克巴莱:
```

```bash
vox vocab.list
vox vocab.prune --dryRun
```

`vox hear --vocab meeting` syncs on demand. Editing the YAML changes the run
id, so the next `hear` re-recognizes.

Account cap: **10 lists shared across models**. One vocabulary used with both
models consumes two slots. Words are at most 15 characters when non-ASCII.
Weight is 1–5; 50 is a super hotword that only `qwen` honours.

## Text to speech

```bash
vox say --text 'Hello world' --voice Cherry
vox say - <<'EOF'
voice: Cherry
text: |
First line.
Slow and clear.
EOF
vox say "{ text: { ref: 'file:script.txt' }, voice: 'Cherry' }"
vox say --text 'Save this' --output ~/Desktop/out.wav
vox voice.list
vox voice.record --file ~/sample.wav --name myvoice
vox voice.remove --voiceId <id>
```

A script already on disk is `text: { ref: 'file:<path>' }` (read verbatim,
relative to the working directory, `~/` is home) — don't paste it back.

System voices: Cherry, Ethan, Chelsie, Serena (zh/en), Dylan (Beijing), Jada
(Shanghai), Sunny (Sichuan). `--output` writes a WAV and does not play.

## Failures

Domain refusals carry `error: DOMAIN_ERROR` and a matchable `code`. Every
failure exits 1 — branch on `code`, not the process status or the detail prose:

`key_unavailable` · `audio_too_large` · `audio_unsupported` ·
`vocab_not_found` · `vocab_quota_exceeded` · `vocab_model_mismatch` ·
`vocab_index_corrupt` · `session_not_found` · `session_ambiguous` ·
`invalid_usage` · `io_error` · `api_error` · `speaker_not_found` ·
`speaker_labels_missing` · `speaker_runtime_unavailable`

A key the chain cannot find fails as `INVALID_INPUT` on `apiKey` (commands that
always call the provider) or `key_unavailable` (commands that read the key only
when a provider is called); both name every reference tried. Report it to a
person rather than hunting for the key yourself. `session_not_found` → `vox session.list`.

## Anti-patterns

| Don't | Do instead | Why |
| --------------------------------------- | ------------------------------------- | ---------------------------------------------- |
| Split a long file into clips | Pass the whole file to `hear` | Cuts drop words and native sentence boundaries |
| Re-read `hear` output for the full hour | `vox export --sid <sid> --format md` | The envelope folds on purpose |
| Treat the envelope as a subtitle file | `export --format srt` or `txt` | Index and content never share stdout |
| Invent a create/edit vocab command | Write `~/.vox/vocabulary/<name>.yaml` | The YAML is the source of truth |
| Put multi-line TTS in a JSON object | `vox say - <<'EOF'` with `text: \|` | Escaping burns tokens and breaks |
| Paste a script file into `text` | `text: { ref: 'file:script.txt' }` | Read as is; no copy in the transcript |
| Paste a key literal into `apiKey` | `apiKey: { ref: 'env:NAME' }` | Literals land in shell history and transcripts |
File transcription and TTS through Alibaba Model Studio, with local MLX speaker identification. `vox hear` uploads a file, transcribes it, and stores a **run**; markdown, subtitles and plain text are exports of that run. No realtime ASR, no microphone transcription. Preflight: `vox status`.

Read the section for your task, then only that slice: `vox @schema .hear`, or several at once with `vox @schema '.{hear,export}'`. Deeper material: `vox @skill references/<file>`.

## Everywhere

- Provider keys need no setup. A command's optional `apiKey` falls back to the env var (`DASHSCOPE_API_KEY`, `MIMO_API_KEY`), then the vault. To use another key pass a reference (`{ ref: 'env:MY_KEY' }`), never the literal: literals land in shell history and transcripts. A key nobody can find is for a person to fix; don't hunt for it.
- Failures exit 1 with `error: DOMAIN_ERROR` and a `code`. Branch on `code`, never on the exit status or detail prose. `session_not_found` → `vox session.list`.
- Stored runs, `export`, `session.*`, `speaker.*` and cached `say` read no key. `VOX_HOME` relocates `~/.vox` for isolated tests.

## Transcribing — `.{hear,session}`

- Pass `--lang` for known-language audio: the cheapest quality win. Pass the whole file; splitting to get under a limit truncates words at every cut. Expect about a minute per 45 minutes of audio.
- `fun` (default) and `qwen` cap at 12 hours / 2 GB. `mimo` takes only MP3/WAV up to 10 MB and returns plain text with no timestamps, speaker labels or vocabularies. Use it for short A/B samples.
- `hear` writes `~/.vox/runs/<sid>/transcript.md` and returns a YAML envelope. A long transcript folds to `preview` plus a `$hints` export command; run that rather than re-reading the envelope. `session.list` is the same records without text.
- The run id hashes audio plus arguments, so repeating a call is a cache hit; `refresh: true` re-recognizes.

## Exporting — `.export`

`export` renders a stored run. Without `--output` stdout is the document itself, not a YAML wrapper. Use `srt`/`vtt`/`txt`/`json` for subtitles and plain text; the envelope is not a subtitle file. Exports use the latest saved speaker-identity snapshot.

## Speaker identity — `.speaker` (and `hear --identify`)

Local, Apple Silicon, needs `uv` and `ffmpeg`. Voiceprints are personal voice data: treat them accordingly. Enrollment rules, matching semantics and storage: `vox @skill references/speaker.md`.

## Vocabularies — `.vocab`

A vocabulary is a YAML file you write at `~/.vox/vocabulary/<name>.yaml`; there is no create or edit command. `hear --vocab <name>` syncs on demand, and editing the file changes the run id. It only biases domain terms; it cannot rescue bad audio. File format and quota: `vox @skill references/vocab.md`.

## Speaking — `.{say,voice,cache}`

- Multi-line text goes as the whole input in YAML (`vox say - <<'EOF'` with `text: |`), not inside a JSON object. A script already on disk is `text: { ref: 'file:<path>' }`; don't paste it back.
- System voices: Cherry, Ethan, Chelsie, Serena (zh/en), Dylan (Beijing), Jada (Shanghai), Sunny (Sichuan). `--output` writes a WAV and does not play.
- `voice.*` is TTS voice cloning; `speaker.*` is recognition identity. They are unrelated.

## Feedback

File issues at [celados/vox](https://github.com/celados/vox/issues) with the
command, the run id, and the YAML error document.
File issues at [celados/vox](https://github.com/celados/vox/issues) with the command, the run id and the YAML error document.
2 changes: 1 addition & 1 deletion src/main.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ test("@skill prints the embedded guide", async () => {
const { stdout, stderr, exitCode } = await run(["@skill"]);
expect(exitCode, stderr).toBe(0);
expect(stdout).toContain("vox @schema");
expect(stdout).toContain("There is no realtime ASR");
expect(stdout).toContain("No realtime ASR");
});

test("status reports each provider key without key material", async () => {
Expand Down
29 changes: 29 additions & 0 deletions src/references/speaker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
type: Guide
title: vox speaker identity
description: Enrollment, matching semantics and storage for vox speaker.* and hear --identify.
---

# Speaker identity (Apple Silicon)

Prerequisites: `uv` and `ffmpeg` on PATH (`brew install uv ffmpeg`). ASR still uses DashScope; voice matching runs locally. The first enrollment or identity request lazily downloads the pinned model (~26 MB) and Python runtime dependencies. Weights live in `~/.vox/models/`, runtime files in `~/.vox/runtime/`, caches in `~/.vox/cache/uv` and `~/.vox/models/huggingface`. No PyTorch needed.

Selectors: `vox @schema '.{speaker,hear}'`.

## Enrolling

Enroll one clean solo recording per person (3–30 seconds; longer files use the first 30 seconds). `start` and `end` select seconds in an existing recording. Enrollment and matching both strip long near-digital silence first, and at least 3 seconds must remain. That is an energy gate, not a speech/noise classifier. Keep enrollment and evaluation recordings separate when checking quality. Re-enroll with `replace: true`.

## Matching

`identify: true` on `hear` turns on ASR speaker separation and matches against all enrolled people; limit candidates with `speakerIds`. MiMo cannot identify speakers (no labels).

The Markdown includes timestamps and names. Anonymous IDs stay in the stored JSON; names and decisions are saved separately as `identified.json`. Uncertain matches keep `Speaker <id>` rather than a forced name. IDs are local to each recording, and several IDs may match one person. `threshold: 0.65` and `margin: 0.1` are starting points, not a calibrated accuracy guarantee.

## Re-identifying

After enrollment or config changes, `speaker.identify` re-matches a stored run with no ASR upload or charge. `export` uses the latest saved identity snapshot, so re-run `speaker.identify` to apply library changes. If identity fails after ASR, the anonymous ASR run remains stored: resume with `session.list` and `speaker.identify`. Nothing outside vox's own run or the explicit `output` is modified.

## Storage

Profiles and settings: `~/.vox/speakers.json` (private permissions). Source enrollment audio is not copied; only the normalized voiceprint is stored. All artifacts use the model version recorded with the profiles; changing embedding models requires re-enrollment.
24 changes: 24 additions & 0 deletions src/references/vocab.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
type: Guide
title: vox vocabularies
description: Hotword YAML format and account quota for vox vocab.* and hear --vocab.
---

# Vocabularies

```yaml
# ~/.vox/vocabulary/meeting.yaml
lang: zh
default_weight: 4
words:
百炼: 5
Fun-ASR: 5
赛德克巴莱:
```

Selector: `vox @schema .vocab`.

- Account cap: 10 lists shared across models. One vocabulary used with both models consumes two slots.
- Non-ASCII words are at most 15 characters. Weight is 1–5; 50 is a super hotword that only `qwen` honours.
- MiMo does not support vocabularies.
- `vocab.prune --dryRun` first: it deletes server-side lists no local YAML claims.
8 changes: 5 additions & 3 deletions src/skill.embed.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
import { pickFiles } from "@celados/argc/skill";

export function embedSkill(): Record<string, string> {
const files = pickFiles(import.meta.dir, ["index.md"]);
const body = files["index.md"];
const { "index.md": body, ...references } = pickFiles(import.meta.dir, [
"index.md",
"references/**/*.md",
]);
if (body === undefined) throw new Error("Embedded skill body is missing index.md");
return { "SKILL.md": body };
return { "SKILL.md": body, ...references };
}
Loading