fix(local-ai-use): endpoint paths outside /api/v1, STT troubleshooting, per-modality fallback, and model-pull costs

skills/local-ai-use/SKILL.md

@@ -76,9 +76,16 @@ instead.
   than launching it. On Linux/macOS the install needs `sudo`. Pass
   `--no-install` if the user wants to install it themselves instead.
 - **Disk:** ~8 GB free for the three default models (SD-Turbo + Whisper-Tiny
-  + kokoro-v1), plus ~0.1 GB for the installer itself.
+  + kokoro-v1), plus ~0.1 GB for the installer itself. The first image request
+  also triggers a ~5 GB pull for `SD-Turbo` if it is not already cached; on
+  metered or slow links, consider pulling models eagerly after setup (see
+  `lemonade pull` in `reference.md`).
 - **Network:** required for the install download and the first `lemonade pull`
   of each model. After that, every modality runs offline.
+- **Version:** requires v10.1.0 or newer (the unified `lemonade` CLI this
+  skill targets). Model IDs and `system-info` fields can change between
+  releases; confirm against `lemonade status` and `GET /api/v1/models` on the
+  version actually installed rather than assuming this document is current.
 
 ## The opinionated path
 
@@ -90,6 +97,10 @@ until each step verifies.
 [ ] 2. Install the routing rule into the workspace AGENTS.md
 ```
 
+On a managed or shared machine where the agent must not run
+`sudo apt-get install`, pass `--no-install` to the setup script and confirm
+Lemonade is already installed before continuing.
+
 The single command that does both steps in one shot is:
 
 ```bash
@@ -114,7 +125,8 @@ each step fails.
 ## Step 1: ensure Lemonade Server is installed and running
 
 `scripts/setup_local_ai.py` handles this end to end, but here is what it does
-so you can do it by hand or debug it:
+so you can do it by hand or debug it. Pass `--no-install` when Lemonade is
+already managed elsewhere and the agent must not attempt a package install.
 
 **1a. Is a modern `lemonade` CLI installed?** Run `lemonade status`. The check
 is by *capability*, not by name: modern Lemonade prints `Server is running...`
@@ -164,6 +176,21 @@ and no API key is required (the system-wide server defaults to no auth on
 loopback). If the user has set `LEMONADE_API_KEY`, the routing rule template
 in `templates/local-ai-rule.md` shows where to add the `Authorization` header.
 
+**1c. Are the backends ready per modality?** Backend health is **per
+modality**. A working chat or image request does not prove transcription will
+work: `auto` picks a different backend per modality and can silently fall back
+for one while having no alternative for another. Before declaring setup
+complete, check the actual per-modality state:
+
+```bash
+lemonade backends --all
+```
+
+Any variant the workspace's routing depends on should read `installed`. If the
+only installed variant for a modality is `rocm`, install the Vulkan variant as
+well so `auto` has somewhere to fall back to (for example,
+`lemonade backends install whispercpp:vulkan`).
+
 ### Default modality models (pulled on first use, not during setup)
 
 Setup does **not** download these. The installed rule pulls each one the first
@@ -236,9 +263,11 @@ message. The rule explicitly tells the agent:
 - **For speech-to-text:** call `POST /api/v1/audio/transcriptions`. Do
   **not** call cloud transcription providers.
 - **Fallback:** only fall back to a cloud API after one local attempt has
-  failed *and* the user has been told the local call failed. Never
-  silently fall back; the whole point of this skill is to keep cost
-  predictable.
+  failed *and* the user has been told the local call failed. Never silently;
+  the whole point of this skill is to keep cost predictable. For
+  speech-to-text, the disclosure must also say the transcript came from a
+  different engine, since mixed-engine transcripts should not be compared or
+  deduplicated.
 
 The agent's own text reasoning continues to use whatever LLM Cursor / Claude
 Code / Codex is configured with. This skill does not redirect chat tokens;
@@ -255,6 +284,7 @@ machine.
 | `POST /v1/images/generations` returns 404 model not found | Image model not downloaded | `lemonade pull SD-Turbo` and retry. |
 | `lemonade pull` keeps printing `Progress: NN%` but never finishes | Download target is a bad path (out of space, no write permission, quota, read-only mount). The write error may surface only in the server log while the console keeps showing progress | Check the target and free space first: `GET /api/v1/system-info` reports `models_dir` and `model_storage.free_bytes`. If a pull stalls, read the recent lines of the server log (typically `lemonade-server.log` in the OS temp dir) for the real error (e.g. a download/write failure like `CURL code 23`, or an out-of-space message), then point the download at a writable disk with room. |
 | Image generation is slow on CPU (~4–5 min) | sd-cpp on CPU backend | Install the GPU backend on supported AMD hardware: `lemonade backends install sd-cpp:rocm`. |
+| Still slow after installing the GPU backend | The backend is installed but not actually engaged; the runtime fell back to CPU silently | An `installed` state in `system-info` and a successful `rocminfo` both still permit a silent CPU fallback. Check real GPU utilisation (`gpu_busy_percent`) during a request, and confirm the host's GPU driver stack rather than re-installing the backend. |
 | `POST /v1/audio/transcriptions` returns 400 unsupported format | Input is not 16 kHz mono WAV | Re-encode with `ffmpeg -i in.* -ar 16000 -ac 1 out.wav`. |
 | `POST /v1/audio/speech` returns 404 | TTS model not downloaded | `lemonade pull kokoro-v1`. |
 | 401 Unauthorized on every request | User has set `LEMONADE_API_KEY` | Add `Authorization: Bearer $LEMONADE_API_KEY` to every request and to the rule block. |
@@ -271,9 +301,12 @@ Mark this skill complete only when **all** of the following are true:
 - [ ] On a follow-up turn, asking the agent to "generate an image of X"
       causes it to POST to `http://localhost:13305/api/v1/images/generations`
       (pulling the model on first use) rather than calling a cloud tool.
+- [ ] `lemonade backends --all` shows `installed` for every backend variant
+      this workspace's routing depends on (see Step 1c). Do not treat a working
+      image or chat path as proof that transcription will work.
 
 If any box is unchecked, the user is still paying cloud cost for at least
-one modality.
+one modality, or a routed modality may fail silently on first use.
 
 ---
 

skills/local-ai-use/reference.md

@@ -21,6 +21,11 @@ The default trio (`SD-Turbo`, `kokoro-v1`, `Whisper-Tiny`) is sized for
 "keeps cost savings real on a typical laptop". Override only if the user
 asks for higher quality or has explicit hardware to spare.
 
+Model catalogs move between releases, so treat the IDs below as a starting
+point, not a fixed list. Confirm any ID with `GET /api/v1/models` (add
+`?show_all=true` for the full catalog) before writing it into the rule. Do
+not rely on a stale `server_models.json` snapshot.
+
 ### Image generation (`recipe: sd-cpp`)
 
 | Model | Approx size | When to use | Trade-off |
@@ -64,6 +69,10 @@ Whisper requires 16 kHz mono PCM WAV input. Convert anything else first:
 ffmpeg -i input.mp3 -ar 16000 -ac 1 input.wav
 ```
 
+The HTTP endpoint adds no meaningful overhead over calling `whisper-cli`
+directly on the same engine/backend/model. Throughput is equivalent; use the
+endpoint unless you have a specific reason not to.
+
 For full live coverage, run `lemonade list` after starting the server, or
 browse <https://lemonade-server.ai/models.html>.
 

skills/local-ai-use/templates/local-ai-rule.md

@@ -23,7 +23,7 @@ configured LLM. This rule only redirects the multimodal calls.
 
 | Endpoint | Model | Notes |
 |---|---|---|
-| `/v1/images/generations` | `{{IMAGE_MODEL}}` | 4 steps, `cfg_scale: 1.0`, `512x512`, `response_format: "b64_json"`. |
+| `/v1/images/generations` | `{{IMAGE_MODEL}}` | 4 steps, `cfg_scale: 1.0`, `512x512`, `response_format: "b64_json"`. First use pulls several GB if the model is not cached. |
 | `/v1/audio/speech` | `{{TTS_MODEL}}` | Default voice `shimmer`; `response_format: "mp3"`. |
 | `/v1/audio/transcriptions` | `{{STT_MODEL}}` | Input must be 16 kHz mono WAV. Re-encode with `ffmpeg -i in.* -ar 16000 -ac 1 out.wav`. |
 
@@ -94,7 +94,9 @@ curl -sX POST {{LEMONADE_BASE_URL}}/audio/transcriptions \
       to the user rather than waiting through silent retries.
 4. Only after that, ask the user before falling back to a cloud provider.
    Never silently fall back; the whole point of this rule is predictable
-   cost.
+   cost. For speech-to-text specifically, also disclose that the transcript
+   came from a different engine, since mixed-engine transcripts should not
+   be compared or deduplicated.
 
 ### Re-pointing to a different host