Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 58 additions & 45 deletions .agents/docs/DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@

Interoperable Signal products (see [issue #10](https://github.com/BreadchainCoop/sigstack-bot/issues/10)):

1. **Voice transcription** — own CVM (`BOT__ROLE=transcription`)
2. **In-chat group translation** — translation CVM (see [`docs/in-chat-translation.md`](../../docs/in-chat-translation.md))
3. **Language Threads** — translation CVM, multilingual main + N sidecars (see [`docs/language-threads.md`](../../docs/language-threads.md))
1. **Voice transcription** — same process (`BOT__ROLE=translation`; NEAR AI Whisper Large V3, not an in-CVM sidecar)
2. **In-chat group translation** — see [`docs/in-chat-translation.md`](../../docs/in-chat-translation.md)
3. **Language Threads** — multilingual main + N sidecars (see [`docs/language-threads.md`](../../docs/language-threads.md))

Architecture overview: [`docs/two-cvm-architecture.md`](../../docs/two-cvm-architecture.md).
Architecture: [`docs/two-cvm-architecture.md`](../../docs/two-cvm-architecture.md) (one Phala CVM, one Signal number). Why STT is remote: [`docs/solutions/architecture-patterns/2026-08-13-cpu-tee-whisper-does-not-scale.md`](../../docs/solutions/architecture-patterns/2026-08-13-cpu-tee-whisper-does-not-scale.md).

Fork legacy removed: general AI chat, tool use (`crates/tools`), x402 payments, in-memory conversation store.

Expand All @@ -20,105 +20,116 @@ Fork legacy removed: general AI chat, tool use (`crates/tools`), x402 payments,
2. **Attestation**: Remote parties verify code via TDX quotes (`!verify`)
3. **Isolation**: Hypervisor/host cannot read TEE memory

### Signal CLI must run in each product TEE
### Signal CLI must run in the product TEE

Signal E2E encryption terminates at Signal CLI. Each product CVM runs its own `signal-api` + `signal-bot` so plaintext only exists in that TEE.
Signal E2E encryption terminates at Signal CLI. Plaintext only exists in this TEE. Voice bytes leave the TEE only as metadata-stripped audio to NEAR AI Whisper.

### Two CVMs, Signal as bus
### One CVM, one phone

- Transcription CVM: Whisper + transcription bot (phone A) — **worker** (voice only; no hub)
- Translation CVM: translation bot (phone B) + NEAR AI — **hub** (menus, translation products, transcription pairing)
- No cross-CVM Docker network. Integration = both bots in the same Signal group.
- Whisper HTTP (`http://whisper-api:9000`) is **intra**-transcription-stack only.

Full hierarchy table: [`docs/two-cvm-architecture.md`](../../docs/two-cvm-architecture.md#bot-hierarchy).
- Same Phala CVM: one `signal-api` + one `signal-bot` (phone B)
- No `whisper-api` sidecar. The bot posts audio to NEAR AI Whisper Large V3 (GPU TEE) and text to NEAR AI chat.
- Per-message `tokio::spawn` so STT HTTP waits never stall other handlers. After STT, in-chat / Language Threads fan out in-process (this number does not receive its own posts).

### What attestation proves / does not prove

| Property | Verified by |
|----------|-------------|
| Code in Intel TDX | TDX quote |
| Exact compose | Compose hash |
| Whisper / bot images | Digests pinned in compose |
| Bot / proxy images | Digests pinned in compose |

Does **not** prove Signal CLI image integrity beyond pinning, or hide network metadata (timing, sizes, phone numbers).
Does **not** prove Signal CLI image integrity beyond pinning, hide network metadata (timing, sizes, phone numbers), or attest **NEAR AI Whisper weights** (`!verify` is this CVM only).

## `BOT__ROLE`

| Role | Handlers | Requires |
|------|----------|----------|
| `transcription` | Voice, `!transcribe*`, `!transcription` menu, `!help-transcription`, `!verify` (worker — no hub `!help` / `!info` / `!privacy`) | Whisper sidecar |
| `translation` | Hub (`!help`, `!info`, `!privacy`, product menus), Language Threads, in-chat, quote `!translate`, `!transcription` pairing, `!verify` | `NEAR_AI__API_KEY` |
| `translation` | Hub (`!help`, `!info`, `!privacy`, product menus), Language Threads, in-chat, voice / `!transcribe*`, `!transcription` menu, quote `!translate`, `!verify` | `NEAR_AI__API_KEY` + `WHISPER__ENABLED=true` |
| `transcription` | **Retired** — process fail-fasts | — |

Fail-fast if role is missing/invalid or required deps are missing.
Fail-fast if role is missing/invalid or required deps are missing. Do not add a third role. Do not drop `BOT__ROLE`.

## Project structure

```
crates/
signal-bot/ # Binary (role-selected handlers)
signal-bot/ # Binary (unified handlers)
signal-bot-core/ # CommandHandler + AppResult
signal-bot-transcription/ # Voice / !transcribe* product crate
whisper-client/
near-ai-client/
whisper-client/ # OpenAI-compatible STT client (NEAR Whisper)
near-ai-client/ # NEAR AI chat + audio transcriptions
signal-client/
dstack-client/
signal-registration-proxy/ # Ops registration helper
docker/
compose.transcription.yaml
compose.translation.yaml
phala.transcription.yaml
phala.translation.yaml
Dockerfile / Dockerfile.whisper / Dockerfile.proxy
compose.translation.yaml # local one-number stack
compose.transcription.yaml # retired stub
phala.translation.yaml # prod one-CVM suite
phala.transcription.yaml # deprecated stub — do not deploy
Dockerfile / Dockerfile.proxy
docs/
two-cvm-architecture.md
voice-transcription.md
language-threads.md
```

## Local dual Compose
`Dockerfile.whisper` is unused on the live path.

## Local Compose

```bash
cp docker/transcription.env.example docker/transcription.env
cp docker/translation.env.example docker/translation.env
# Different SIGNAL_PHONE values; PEER_PHONE on translation = transcription phone;
# NEAR_AI_API_KEY in translation.env
# SIGNAL_PHONE; NEAR_AI_API_KEY (chat + Whisper STT)

docker compose -f docker/compose.transcription.yaml --env-file docker/transcription.env up -d
docker compose -f docker/compose.translation.yaml --env-file docker/translation.env up -d
```

Networks: `sigstack-transcription-internal`, `sigstack-translation-internal`.
Network: `sigstack-translation-internal`.

## Phala deploy

Build `linux/amd64` images, then deploy **two** CVMs @ ~4 GB (`tdx.medium`):
Build `linux/amd64` images (bot + registration proxy only), then **in-place** upgrade the surviving CVM:

```bash
docker buildx build --platform linux/amd64 -t YOUR/signal-bot-tee:latest -f docker/Dockerfile --push .
docker buildx build --platform linux/amd64 -t YOUR/signal-whisper-api:latest -f docker/Dockerfile.whisper --push .
docker buildx build --platform linux/amd64 -t YOUR/signal-registration-proxy:latest -f docker/Dockerfile.proxy --push .

phala deploy … -c docker/phala.transcription.yaml -e docker/phala.transcription.env --wait -t tdx.medium
phala deploy … -c docker/phala.translation.yaml -e docker/phala.translation.env --wait -t tdx.medium
phala deploy --cvm-id 0e82fa77-8b15-4dbd-89c4-9045ab911353 \
-c docker/phala.translation.yaml -e docker/phala.translation.env --wait
```

Env templates: `docker/phala.transcription.env.example`, `docker/phala.translation.env.example`.
Do **not** `phala deploy -n` against the live CVM. Do **not** deploy `docker/phala.transcription.yaml`. Env template: `docker/phala.translation.env.example`.

Encrypted secrets: `SIGNAL_PHONE` (phone B), `NEAR_AI_API_KEY`.

Health: Signal CLI `GET /v1/health` on `signal-api`. Attestation: `!verify <challenge>` (this CVM’s compose, not remote Whisper; one reply).

Do not re-register phone A. Proxy **:8081** only.

Encrypted secrets: phone numbers per CVM; `PEER_PHONE` for pairing; `NEAR_AI_API_KEY` on translation only.
### CVM storage — do not wipe

Health (transcription): Whisper `GET /health` on `:9000`, Signal CLI `GET /v1/health` on `:8080`. Attestation: `!verify <challenge>`.
**In-place upgrades only** once the phone is registered. TEE RAM wipe is expected; **disk volumes are the product identity.**

| Must keep | Volume | Breakage if lost |
|-----------|--------|------------------|
| Signal session (phone B) | `signal-config-translation` | Bot gone from groups until re-register |
| User prefs (`!translate-me-on`, Language Threads) | `group-prefs-translation` → `/data/group_prefs.enc` | Users must re-enable; suite looks broken |

Use `phala deploy --cvm-id 0e82fa77-8b15-4dbd-89c4-9045ab911353`. Do **not** `phala cvms delete` this CVM, create a replacement, rename those volumes, or `down -v` for an image bump. [`scripts/deploy_phala.sh`](../../scripts/deploy_phala.sh) defaults to that `--cvm-id`.

After upgrade, logs should show `Loaded group preferences for N groups` (not `starting fresh` / `TEE deployment may have changed`), and `signal-api` should still list its account.

Canonical table: [`docs/two-cvm-architecture.md` — CVM storage](../../docs/two-cvm-architecture.md#cvm-storage-keep-intact). Agent rule: [`AGENTS.md` — CVM storage](../../AGENTS.md#cvm-storage-do-not-wipe).

## Configuration

| Variable | Notes |
|----------|-------|
| `BOT__ROLE` | `transcription` \| `translation` |
| `BOT__ROLE` | Live value `translation` (`transcription` fail-fasts) |
| `SIGNAL__SERVICE_URL` | Default `http://signal-api:8080` |
| `SIGNAL__PHONE_NUMBER` | Ops phone for this CVM |
| `SIGNAL__PEER_PHONE` | Peer product bot. Translation: invites transcription. Transcription: must be translation phone for auto-join |
| `NEAR_AI__*` | Translation role |
| `WHISPER__*` | Transcription role |
| `SIGNAL__PHONE_NUMBER` | Ops phone for this process |
| `NEAR_AI__*` | Chat + API key for remote Whisper |
| `WHISPER__*` | Required — `SERVICE_URL` is NEAR `/v1`, not `whisper-api:9000` |
| `TRANSLATE_ALL__*` | In-chat translation |
| `GROUP_PREFERENCES__*` | Encrypted group prefs volume |
| `DSTACK__SOCKET_PATH` | `/var/run/dstack.sock` in Phala |
Expand All @@ -130,9 +141,11 @@ cargo test
cargo build --release
```

Before finishing Rust work: `npm run ci` / `pnpm run ci` (never bare `pnpm ci`).

## Registration proxy

Still useful as an **ops** helper on the translation stack (port 8081) to register phone B. Register phone A against the transcription stack’s `signal-api` via `docker compose exec` / curl. Multi-tenant “create your personal AI bot” web UX is out of scope; Stripe client site is issue #10 follow-up.
Ops helper on the one CVM: **:8081** (phone B). Multi-tenant “create your personal AI bot” web UX is out of scope; Stripe client site is issue #10 follow-up.

## Website

Expand Down
4 changes: 3 additions & 1 deletion .cursor/rules/compound-engineering.mdc
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,9 @@ Product docs stay as siblings under `docs/` (architecture, voice, translation).

## Product constraints

- Required env: `BOT__ROLE=transcription|translation`
- Required env: `BOT__ROLE=translation` (`transcription` is retired and fail-fasts)
- Do not reintroduce tools, x402, or general chat paths
- Do not reintroduce a local Whisper sidecar (`whisper-api`); STT is NEAR AI Whisper Large V3 (GPU TEE). Do not put Whisper on a larger CPU TEE as the scale path. See [docs/solutions/architecture-patterns/2026-08-13-cpu-tee-whisper-does-not-scale.md](docs/solutions/architecture-patterns/2026-08-13-cpu-tee-whisper-does-not-scale.md)
- Image digests stay pinned in compose for attestation
- **Keep live CVM storage:** never wipe or replace Phala volumes / the registered Signal phone for a routine TEE upgrade. User prefs (`group-prefs-translation`) and Signal CLI session (`signal-config-translation`) must stay on the existing CVM (`phala deploy --cvm-id`). See [docs/two-cvm-architecture.md](docs/two-cvm-architecture.md#cvm-storage-keep-intact) and [AGENTS.md](AGENTS.md#cvm-storage-do-not-wipe)
- Before finishing Rust work run `npm run ci` / `pnpm run ci` (never bare `pnpm ci`)
18 changes: 18 additions & 0 deletions .cursor/rules/phala-cvm-storage.mdc
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
description: Keep live Phala CVM volumes (Signal phone + user prefs) on TEE upgrades
globs: docker/phala*.yaml,docker/compose.*.yaml,scripts/deploy_phala.sh,scripts/register_phala_phones.sh
alwaysApply: false
---

# CVM storage — do not wipe

Routine TEE / image upgrades must **keep** each live CVM’s named volumes. TEE RAM wipe is expected; disk is the product identity.

- **Signal phone:** `signal-config-translation` on `signal-api` (phone B). Losing it unlinks the bot from every group until re-register (which takes over the number).
- **User prefs:** `group-prefs-translation` → `/data/group_prefs.enc` (`!translate-me-on`, `!translate-all-on`, Language Threads). Losing it forces every user to opt in again.

**Do:** `phala deploy --cvm-id <existing>`. Keep volume names unchanged.

**Do not:** create a replacement CVM, `phala cvms delete`, rename volumes, or `down -v` for an image bump. [`scripts/deploy_phala.sh`](scripts/deploy_phala.sh) is first-create (`-n`), not a safe upgrade of a registered CVM.

Canonical: [docs/two-cvm-architecture.md](docs/two-cvm-architecture.md#cvm-storage-keep-intact). Agent entry: [AGENTS.md](AGENTS.md#cvm-storage-do-not-wipe).
17 changes: 9 additions & 8 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# Required: which product this process runs
# transcription — voice → text (Whisper); no NEAR AI required
# translation — in-chat + Language Threads (NEAR AI required)
# Required: which product this process runs.
# translation — unified bot (hub + voice + in-chat + Language Threads)
# transcription — retired; process fail-fasts (use translation)
BOT__ROLE=translation

# Signal Configuration
SIGNAL__SERVICE_URL=http://signal-api:8080
SIGNAL__PHONE_NUMBER=+1234567890
SIGNAL__POLL_INTERVAL=1s

# NEAR AI Configuration (required when BOT__ROLE=translation)
# NEAR AI Configuration (required: chat + remote Whisper STT)
NEAR_AI__API_KEY=your-api-key-here
NEAR_AI__BASE_URL=https://cloud-api.near.ai/v1
NEAR_AI__MODEL=deepseek-ai/DeepSeek-V4-Flash
Expand All @@ -22,10 +22,11 @@ BOT__GITHUB_REPO=https://github.com/BreadchainCoop/sigstack-bot
# Dstack Configuration (TEE)
DSTACK__SOCKET_PATH=/var/run/dstack.sock

# Whisper (required when BOT__ROLE=transcription; disable on translation stack)
WHISPER__ENABLED=false
WHISPER__SERVICE_URL=http://whisper-api:9000
WHISPER__MODEL=small
# Whisper / STT (required on the unified translation bot)
# SERVICE_URL is NEAR AI /v1 — not a local whisper-api sidecar.
WHISPER__ENABLED=true
WHISPER__SERVICE_URL=https://cloud-api.near.ai/v1
WHISPER__MODEL=openai/whisper-large-v3
WHISPER__TIMEOUT=120s

# Group auto-translate (!translate-on) — translation role
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -54,3 +54,4 @@ credentials.json
# Compound Engineering machine-local config (keep *.example.yaml tracked)
.compound-engineering/*.local.yaml
.compound-engineering/config.local.yaml
docker/ops/
28 changes: 19 additions & 9 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Short entrypoint for coding agents. Humans: see [README.md](README.md).

## Product

TEE-hosted Signal bots for **voice transcription** and **group translation** (not a general AI chat assistant). Two phone numbers / two bots in a Signal group; Signal is the bus — no Docker network between CVMs. Whisper stays on the transcription stack only.
TEE-hosted Signal bot for **voice transcription** and **group translation** (not a general AI chat assistant). **One** phone number / one bot process in a Signal group on **one** Phala CVM. STT is **NEAR AI Whisper Large V3** (GPU TEE) — do not reintroduce an in-CVM Whisper sidecar. See [`docs/solutions/architecture-patterns/2026-08-13-cpu-tee-whisper-does-not-scale.md`](docs/solutions/architecture-patterns/2026-08-13-cpu-tee-whisper-does-not-scale.md).

## Compound Engineering

Expand Down Expand Up @@ -50,32 +50,42 @@ npm run ci # all GitHub Actions gates (fmt + clippy + coverage + c
pnpm run ci # same as above if you use pnpm (NOT `pnpm ci` — that only installs)
npm run prepush # alias of npm run ci (also run by husky pre-push)

cp docker/transcription.env.example docker/transcription.env
cp docker/translation.env.example docker/translation.env
# Two different SIGNAL_PHONE values; NEAR_AI_API_KEY in translation.env
# SIGNAL_PHONE + NEAR_AI_API_KEY (chat + Whisper STT)

docker compose -f docker/compose.transcription.yaml --env-file docker/transcription.env up -d
docker compose -f docker/compose.translation.yaml --env-file docker/translation.env up -d
```

## Read next

| Doc | Why |
|-----|-----|
| [`.agents/docs/DEVELOPMENT.md`](.agents/docs/DEVELOPMENT.md) | TEE trust model, `BOT__ROLE`, Phala dual-CVM ops |
| [`docs/two-cvm-architecture.md`](docs/two-cvm-architecture.md) | Architecture diagram and compose/Phala split |
| [`docs/voice-transcription.md`](docs/voice-transcription.md) | Voice transcription product + pairing |
| [`.agents/docs/DEVELOPMENT.md`](.agents/docs/DEVELOPMENT.md) | TEE trust model, `BOT__ROLE`, Phala one-CVM ops, **CVM volume / Signal identity** |
| [`docs/two-cvm-architecture.md`](docs/two-cvm-architecture.md) | One CVM / one phone, **CVM storage (keep intact)** |
| [`docs/voice-transcription.md`](docs/voice-transcription.md) | Voice transcription product (NEAR Whisper; in-process fan-out) |
| [`docs/solutions/architecture-patterns/2026-08-13-cpu-tee-whisper-does-not-scale.md`](docs/solutions/architecture-patterns/2026-08-13-cpu-tee-whisper-does-not-scale.md) | Why STT is remote; never re-home Whisper in a CPU TEE |
| [`docs/in-chat-translation.md`](docs/in-chat-translation.md) | In-chat (group) bilingual auto/manual translate |
| [`docs/language-threads.md`](docs/language-threads.md) | Language Threads (multilingual main + N sidecars) |
| [`docs/solutions/`](docs/solutions/) | Compounded learnings from prior work |
| [`docs/plans/`](docs/plans/) | CE implementation plans |
| [`.agents/skills/`](.agents/skills/) | Domain skills (Rust, Docker, Stripe) |
| [`.cursor/rules/`](.cursor/rules/) | Cursor project rules (commits, compound loop) |
| [`.cursor/rules/`](.cursor/rules/) | Cursor project rules (commits, compound loop, **CVM storage**) |

## CVM storage (do not wipe)

**Never destroy live Phala volumes or replace the registered translation CVM for a routine upgrade.** Prod is **one** CVM (`0e82fa77-8b15-4dbd-89c4-9045ab911353`). Keep:

1. **Registered Signal phone** (`signal-config-translation` = phone B) — losing the volume unlinks the bot until ops re-registers (and takes over the number).
2. **Encrypted user prefs** (`group-prefs-translation` → `/data/group_prefs.enc`) — `!translate-me-on`, `!translate-all-on`, Language Threads bridges. Losing this forces every user to turn features back on.

Upgrade with `phala deploy --cvm-id 0e82fa77-8b15-4dbd-89c4-9045ab911353`. Do not create a new CVM, rename those volumes, or `down -v`. Do not recreate a transcription CVM or re-register phone A. TEE RAM wipe on reboot is expected; disk volumes are the identity. Details: [docs/two-cvm-architecture.md — CVM storage](docs/two-cvm-architecture.md#cvm-storage-keep-intact).

## Rules of thumb

- Required env: `BOT__ROLE=transcription|translation`
- Required env: `BOT__ROLE=translation` (`transcription` is retired and fail-fasts)
- Do not reintroduce tools, x402, or general chat paths
- Do not reintroduce a local Whisper sidecar; STT is NEAR AI Whisper Large V3
- Image digests stay pinned in compose for attestation
- **Keep CVM volumes and Signal registrations** — in-place Phala upgrades only; see [CVM storage](#cvm-storage-do-not-wipe)
- Commits must pass [commitlint](https://github.com/conventional-changelog/commitlint) (`type: subject`); subject all lowercase, no trailing period, dashes not snake_case — see [`.cursor/rules/commit-messages.mdc`](.cursor/rules/commit-messages.mdc). Run `npm install` or `pnpm install` so husky `commit-msg` / `pre-push` hooks are active
- **CI style gates are not optional.** GitHub Actions (`test.yml` + `commitlint.yml`) fails on fmt, Clippy `-D warnings`, llvm-cov ≥90% lines, and conventional commits. Before finishing Rust work run `npm run ci` / `pnpm run ci` (never bare `pnpm ci`). Husky `pre-push` runs that script; `commit-msg` runs commitlint on each commit. Cursor auto-fmts `.rs` edits and re-prompts on stop if fmt/clippy would fail CI.
Loading
Loading