From 15763a485b60cbb64bf8b3c57f65f53b67e6b8eb Mon Sep 17 00:00:00 2001 From: Matt Elliott Date: Thu, 13 Aug 2026 00:44:39 -0400 Subject: [PATCH] fix(local-ai-app-integration): correct endpoint paths, Linux NPU row, and unpulled-model behaviour MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified against a live `lemonade-server 11.5.2` on a Ryzen AI MAX+ PRO 395 (Strix Halo / gfx1151, XDNA 2 NPU, Linux). Each change below is a reproduction, not a review comment. Endpoint paths (Step 5 client table, reference.md route table): - `@anthropic-ai/sdk` `base_url` must be the bare `http://127.0.0.1:{port}`, not `.../api/v1`. The SDK appends `/v1/messages`, so the documented value requests `/api/v1/v1/messages` and 404s on the first call. Lemonade serves Anthropic Messages at `/v1/messages`, outside the `/api/v1` prefix. Reproduced end-to-end in a real Anthropic-Messages client. - reference.md listed `POST /api/v1/messages`, which 404s. Corrected, and both exceptions to the `/api/v1` pattern are now called out together. - Added reranking: the proxy serves `/api/v1/reranking`. `/v1/rerank` — the spelling used by Jina, Cohere, vLLM, and llama.cpp — 404s on the proxy even though the supervised back-port serves it. - Added a Step 7 row: a 404 body carrying a `path` field is a routing mistake, not a missing model. Linux NPU (Step 2 profile table, reference.md recipe tables): - Removed the "Speech-to-text (Linux NPU) -> whisper-v3-turbo-FLM / flm" row. No NPU backend installs on Linux at 11.5.2: whispercpp:npu -> Requires Windows ryzenai-llm:npu -> Requires Windows flm:npu -> Requires AMD XDNA 2 AMD NPU The last is a device gate that cannot pass on Linux, where `system-info` reports `amd_npu.family: ""` on hardware that is XDNA 2. This also removes a contradiction with the existing "# Windows NPU path only" packaging example. Unpulled models (Step 6, Step 7, verification checklist): - At 11.5.2 an unpulled model does not return an empty 200; the first inference blocks until the download completes. That collides with the mandatory 120s timeout and presents as a hang, not a blank. Both behaviours are now documented, both cured by the same explicit pull, and the pull step is rejustified around latency control rather than silent failure. - `GET /api/v1/models` returns downloaded models only (23 vs 145 catalogued on the test host). Model-name validation needs `?show_all=true`. Also: Step 1 said "record three things" above a four-item list; the shutdown note read as though Windows needs the harsher call, when the real point is that Windows has no graceful equivalent, so the wait is what matters. --- skills/local-ai-app-integration/SKILL.md | 109 +++++++++++++------ skills/local-ai-app-integration/reference.md | 55 ++++++++-- 2 files changed, 117 insertions(+), 47 deletions(-) diff --git a/skills/local-ai-app-integration/SKILL.md b/skills/local-ai-app-integration/SKILL.md index 6b3fa88..abddcc8 100644 --- a/skills/local-ai-app-integration/SKILL.md +++ b/skills/local-ai-app-integration/SKILL.md @@ -57,7 +57,8 @@ Track progress against this checklist. Move on only when each step verifies. > log line at each transition, "nothing happened" is indistinguishable from > "broke at stage 3." Emit one clear line per stage as you build (see > [Step 4](#step-4-add-a-lemond-launcher)); the most common dead-end in this -> integration — a blank result with no error — is invisible without them. +> integration — a first request that hangs or comes back blank — is invisible +> without them. --- @@ -70,7 +71,7 @@ Find every place the app currently calls a cloud AI API. Search the repo for: - `api.openai.com`, `api.anthropic.com`, `localhost:11434` (Ollama) - `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` -Record three things before continuing: +Record four things before continuing: 1. **Client library and language** (e.g., `openai-python`, `openai-node`, `@anthropic-ai/sdk`, `go-openai`, raw `fetch`). @@ -97,7 +98,7 @@ it. | Vision / multimodal chat | `Gemma-4-E2B-it-GGUF` | `llamacpp` | Small multimodal default | | NPU-first on Ryzen AI | `Llama-3.2-3B-Instruct-Hybrid` | `ryzenai-llm` | XDNA2 NPU on Windows | | Speech-to-text (Windows) | `Whisper-Large-v3-Turbo` | `whispercpp` | One model; probe picks NPU → iGPU/dGPU → CPU automatically | -| Speech-to-text (Linux NPU) | `whisper-v3-turbo-FLM` | `flm` | Linux NPU path; falls back to `whispercpp` iGPU/CPU off-NPU | +| Speech-to-text (Linux) | `Whisper-Large-v3-Turbo` | `whispercpp` | iGPU (`vulkan`) or CPU. **No NPU path** — see below | | Text-to-speech | `kokoro-v1` | `kokoro` | CPU-only, low latency | | Image generation | `SDXL-Turbo` | `sd-cpp` | Single-step generation | @@ -117,6 +118,11 @@ the skill will use the NPU backend — otherwise it falls back to `vulkan`. > [reference.md](reference.md#hardware-probing-with-v1system-info) for > per-recipe decision rules. +> **No NPU backend is installable through Lemonade on Linux** — including +> `flm:npu`, and including hosts whose NPU `system-info` reports as available. +> Plan Linux around `vulkan` (iGPU) or `cpu`. Details and the exact refusals +> are in [reference.md](reference.md#speech-to-text). + For more options and tradeoffs, see [reference.md](reference.md). ## Step 3: Place Embeddable Lemonade in the app's tree and install backends @@ -208,7 +214,8 @@ vendor/lemonade/ > **`server_models.json`:** Do not edit or rely on this file. It can be stale. > The only authoritative model list is `GET /api/v1/models` on a running -> `lemond` instance with the backend already installed. +> `lemond` instance with the backend already installed. That call returns +> **downloaded models only** — add `?show_all=true` for the full catalog. **Bundle decisions: pick deliberately** @@ -267,12 +274,12 @@ supervisor. Its only jobs: > [lemond] Healthy on port > [lemond] : installed (or: already installed / install failed) > [lemond] Pulling model ... then: Model ready (or: pull returned ) -> [local] result: (first inference output — empty string here = unpulled model) +> [local] result: (first inference output verbatim — empty or very slow here = unpulled model) > ``` > -> Logging the **first inference result verbatim** is what turns the -> silent-empty failure (Step 6) from a multi-hour mystery into a one-line -> diagnosis. Route these through the app's normal logging so they can be quieted +> Logging the **first inference result verbatim**, with its elapsed time, is +> what turns the unpulled-model failure (Step 6) from a multi-hour mystery into +> a one-line diagnosis. Route these through the app's normal logging so they can be quieted > for release. > **Dev-mode file watchers:** If the app runs with a file watcher (Tauri, @@ -295,7 +302,9 @@ hit that `/api/v1/health` path. Make **three** changes to the app's existing client construction — all three are required, not optional: -1. Set `base_url` to `http://127.0.0.1:{port}/api/v1` +1. Set `base_url` so the SDK's own path suffix resolves correctly — for + OpenAI-compatible clients that is `http://127.0.0.1:{port}/api/v1`; see the + table below, the prefix is **not** the same for every client 2. Set `api_key` to the launcher key 3. **Set the HTTP timeout to 120 seconds** — this is mandatory, not optional @@ -320,13 +329,24 @@ client = OpenAI( For other clients: -| Existing client | New `base_url` | New auth | Timeout | -|---|---|---|---| -| `openai-python` | `http://127.0.0.1:{port}/api/v1` | `api_key=key` | `httpx.Client(timeout=120)` | -| `openai-node` | `http://127.0.0.1:{port}/api/v1` | `apiKey: key` | `timeout: 120000` | -| `@anthropic-ai/sdk` | `http://127.0.0.1:{port}/api/v1` | `apiKey: key` | `timeout: 120000` | -| Raw `fetch` / `requests` | same | `Authorization: Bearer {key}` | set per-request | -| Ollama-compatible code | `http://127.0.0.1:{port}/api/v0` | pass key anyway | 120s | +| Existing client | New `base_url` | Resolves to | New auth | Timeout | +|---|---|---|---|---| +| `openai-python` | `http://127.0.0.1:{port}/api/v1` | `/api/v1/chat/completions` | `api_key=key` | `httpx.Client(timeout=120)` | +| `openai-node` | `http://127.0.0.1:{port}/api/v1` | `/api/v1/chat/completions` | `apiKey: key` | `timeout: 120000` | +| `@anthropic-ai/sdk` | `http://127.0.0.1:{port}` — **no path suffix** | `/v1/messages` | `apiKey: key` | `timeout: 120000` | +| Raw `fetch` / `requests` | full path, per the route table | — | `Authorization: Bearer {key}` | set per-request | +| Ollama-compatible code | `http://127.0.0.1:{port}/api/v0` | `/api/v0/...` | pass key anyway | 120s | + +> **Path prefixes are not uniform — do not assume `/api/v1` for everything.** +> OpenAI-compatible routes live under `/api/v1`, but Anthropic Messages is +> served at **`/v1/messages`, outside that prefix** (`/api/v1/messages` 404s). +> `@anthropic-ai/sdk` appends `/v1/messages` to whatever `base_url` you give +> it, so `.../api/v1` requests `/api/v1/v1/messages` and 404s on the very first +> call — during the cold-start window where an unpulled model or a short +> timeout is the more natural suspect. A 404 whose body carries a `path` field +> is a routing mistake, not a model problem. Verify with one real request +> before wiring up the rest of the app; see +> [reference.md](reference.md#app-facing-use-these-from-the-apps-existing-client). The model identifier on requests stays a Lemonade model name (e.g. `Qwen3-4B-GGUF`), not the cloud name. @@ -365,10 +385,16 @@ step you let lemond do lazily — pulling is not. ### Pull the model so it exists on disk -Lazy-load only loads weights that are **already downloaded**. If the model was -never pulled, the first inference does not error — lemond returns an empty / -blank result with HTTP 200. So after health passes and the backend is -installed, proactively pull the model: +If the model was never pulled, the first inference request **blocks until the +weights finish downloading**. It does not error and it does not return early — +it just takes as long as the download takes, which on a large model over a slow +link is minutes. That collides directly with the 120-second timeout Step 5 +makes mandatory: the client gives up mid-download and the user sees a blank UI +with no error, indistinguishable from a broken integration. + +So after health passes and the backend is installed, proactively pull the +model. Pulling up front is what makes first-run latency **predictable** and +gives you somewhere to hang a progress indicator: ```http POST /api/v1/pull @@ -381,16 +407,21 @@ first user-triggered inference) and log the result. - **Default model** (the one you chose in Step 2): pull it by name as above. - **Custom / user-overridden model:** do not assume it exists. Confirm it is a - real Lemonade model first via `GET /api/v1/models` (the **only** trusted - catalog — see [reference.md](reference.md)), then pull it the same way. A - model appearing in the catalog is **not** proof its weights are downloaded; - a successful pull is. - -> **Silent-empty is almost always an unpulled model.** If inference returns an -> empty string / blank output with no HTTP error, the model was not downloaded. -> Check your pull step before debugging anything else — this is the failure mode -> that wastes the most time. Log the pull result and the first inference result -> (see Step 4) so this is diagnosable from the console, not by guesswork. + real Lemonade model first via `GET /api/v1/models?show_all=true` (the **only** + trusted catalog — see [reference.md](reference.md); without `show_all` the + response lists downloaded models only, so a valid name looks invalid), then + pull it the same way. A model appearing in the catalog is **not** proof its + weights are downloaded; a successful pull is. + +> **If the app skips the pull, first inference inherits the download.** Log the +> pull result and the first inference result (see Step 4) so the console tells +> you which one you are waiting on. If a first request hangs past the timeout +> and a later identical request is fast, the first one was downloading. +> +> Behaviour here has changed across releases: some versions return an empty +> string with HTTP 200 for an unpulled model instead of downloading it. Both +> failure modes are cured by the same explicit pull, which is why it stays a +> mandatory step rather than something to leave to lemond. ### Surface the *whole* setup, not just model load @@ -417,16 +448,21 @@ These are the only failure modes worth handling. Do not over-engineer. | Symptom | Cause | Recovery | |---|---|---| -| **Inference returns empty / blank with HTTP 200, no error** | Model never pulled: backend is installed but weights are absent, so lazy-load has nothing to load | `POST /api/v1/pull` with `{"model":"..."}`, wait for success, retry. Log the pulled result and the first inference result. This is the most common silent failure — see [Step 6](#step-6-health-backend-then-pull-the-model--before-first-inference) | +| **First inference hangs past the timeout, but a later identical request is fast** | Model was never pulled, so the first request blocked while lemond downloaded the weights | Add the explicit `POST /api/v1/pull` step — see [Step 6](#step-6-health-backend-then-pull-the-model--before-first-inference). Raise the client timeout only as a stopgap; a pull-first setup makes the wait bounded and coverable by a progress indicator | +| **Inference returns empty / blank with HTTP 200, no error** | Model never pulled: on some lemond versions an unpulled model yields an empty result rather than a download | Same fix — `POST /api/v1/pull` with `{"model":"..."}`, wait for success, retry. Log the pull result and the first inference result | +| A 404 body contains a `path` field | The **route** does not exist — a client base-URL mistake, not a missing model. Most often `@anthropic-ai/sdk` pointed at `/api/v1`, producing `/api/v1/v1/messages` | Fix `base_url` per the Step 5 table. Anthropic Messages is `/v1/messages`; reranking is `/api/v1/reranking`. A model-not-found 404 has no `path` field | | `POST /api/v1/load` returns 404 / model not found | Model not pulled yet (same root cause as the empty-result row above) | `POST /api/v1/pull` with `{"model": "..."}` then retry `/api/v1/load` | | `POST /api/v1/load` returns 500 with backend error | Backend not installed for this hardware | `GET /api/v1/system-info`, pick a supported backend, `POST /api/v1/install` with `{"recipe": "...", "backend": "..."}`, retry | | Subprocess exits immediately | Port race: another process grabbed the port between `freePort()` and lemond binding | The reference launcher retries with a fresh port automatically (3 attempts) | | `/api/v1/health` never returns 200 | First-run backend extraction is slow on cold disk | Extend timeout to 90s on first launch, 30s after | | HTTP 401 on every request | Forgot the `Authorization: Bearer` header | Audit the client config because Lemonade rejects unauth'd calls when `LEMONADE_API_KEY` is set | -**Shutdown:** On app exit, `proc.terminate()` (Unix) or -`proc.kill()` (Windows). `lemond` flushes config and exits cleanly within a -couple of seconds. Always wait on the process; never orphan it. +**Shutdown:** On app exit, call `proc.terminate()`, then `proc.wait(timeout=5)`, +and only fall back to `proc.kill()` if the wait expires. `lemond` flushes config +and exits cleanly within a couple of seconds, so the wait is the part that +matters. (On POSIX `terminate()` is SIGTERM and `kill()` is SIGKILL; on Windows +Python maps both to `TerminateProcess`, so there is no graceful/forceful +distinction to make there.) Always wait on the process; never orphan it. **Do not** parse `lemond` stdout to detect readiness; use the HTTP `/api/v1/health` probe. Stdout format is not a stable contract. @@ -442,8 +478,9 @@ The integration is done when **all** of these are true: - [ ] `lemond` starts as a subprocess with a fresh API key per launch. - [ ] `GET /api/v1/health` returns 200 within the timeout. - [ ] The default model is pulled (or bundled) before the first inference; a - custom/overridden model is confirmed via `GET /api/v1/models` and then - pulled. A blank result with no error means this step was skipped. + custom/overridden model is confirmed via `GET /api/v1/models?show_all=true` + and then pulled. A first request that hangs for minutes, or returns blank + with no error, means this step was skipped. - [ ] Each lifecycle stage logs a clear line (spawn, health, backend install, model pull, first result) so a failure is diagnosable from the console. - [ ] The existing client's chat / image / speech call returns a valid diff --git a/skills/local-ai-app-integration/reference.md b/skills/local-ai-app-integration/reference.md index 9226d3f..fd06e87 100644 --- a/skills/local-ai-app-integration/reference.md +++ b/skills/local-ai-app-integration/reference.md @@ -35,7 +35,7 @@ hardware-optimized one at first run after a system probe. | Recipe | Backend | Hardware | Notes | |---|---|---|---| -| `flm` | `npu` | XDNA2 NPU | Cannot be packaging-time bundled on Linux. | +| `flm` | `npu` | XDNA2 NPU | Windows only. Cannot be installed on Linux — see below. | | `ryzenai-llm` | `npu` | XDNA2 NPU | Windows only. Best for the Hybrid model family. | ### Speech-to-text @@ -45,7 +45,16 @@ hardware-optimized one at first run after a system probe. | `whispercpp` | `vulkan` | `Whisper-Large-v3-Turbo` | AMD iGPU / dGPU | Windows, Linux | | `whispercpp` | `cpu` | `Whisper-Large-v3-Turbo` | x86_64 CPU | Windows, Linux | | `whispercpp` | `npu` | `Whisper-Large-v3-Turbo` | XDNA2 NPU | Windows | -| `flm` | `npu` | `whisper-v3-turbo-FLM` | XDNA2 NPU | Linux (runtime-install only) | +| `flm` | `npu` | `whisper-v3-turbo-FLM` | XDNA2 NPU | Windows | + +> **There is no NPU path through Lemonade on Linux.** Verified at 11.5.2 on a +> Ryzen AI MAX+ PRO 395 with `devices.amd_npu.available: true`: +> `whispercpp:npu` and `ryzenai-llm:npu` refuse with *"Requires Windows"*, and +> `flm:npu` refuses with *"Requires AMD XDNA 2 AMD NPU"* — a device gate that +> cannot pass, because Linux reports `amd_npu.family: ""`. On Linux, plan for +> `whispercpp:vulkan` (iGPU) or `whispercpp:cpu`. FastFlowLM can drive the NPU +> on Linux as a standalone server, but that is a separate process outside +> lemond's supervision and outside this skill's launcher. ### Text-to-speech @@ -76,7 +85,7 @@ ship a default and document how to override. | Multimodal (vision) chat | `Gemma-4-E2B-it-GGUF` | 2.0 GB | `llamacpp` | | Hybrid NPU chat (Ryzen AI) | `Llama-3.2-3B-Instruct-Hybrid` | 2.0 GB | `ryzenai-llm` | | Speech-to-text | `Whisper-Large-v3-Turbo` | 1.6 GB | `whispercpp` | -| NPU speech-to-text (Ryzen AI) | `whisper-v3-turbo-FLM` | 0.6 GB | `flm` | +| NPU speech-to-text (Ryzen AI, **Windows**) | `whisper-v3-turbo-FLM` | 0.6 GB | `flm` | | Text-to-speech | `kokoro-v1` | 0.3 GB | `kokoro` | | Image generation | `SDXL-Turbo` | 6.9 GB | `sd-cpp` | @@ -87,13 +96,18 @@ model catalog; it can be stale or incomplete. A model only appears in `GET /v1/models` once its backend is installed (see Step 3), so install the backend first or the list will look empty/incomplete. -**Catalogued ≠ downloaded.** A model listed by `GET /v1/models` is *available -to use*, not necessarily present on disk. It must be **pulled** -(`POST /api/v1/pull {"model":"..."}`) before it can serve — until then, -inference returns an empty result with HTTP 200, not an error. The surest -signal that a model is ready is a successful pull, not its presence in the -catalog. See SKILL.md -[Step 6](SKILL.md#step-6-health-backend-then-pull-the-model--before-first-inference). +**`GET /api/v1/models` lists only downloaded models.** For the full registry — +including everything not yet pulled — use `GET /api/v1/models?show_all=true`. +The difference is large (23 local vs 145 catalogued on one test host), so +validating a user-supplied model name against the plain list rejects valid +models. + +**Catalogued ≠ downloaded.** A model listed in the catalog is *available to +use*, not necessarily present on disk. Pull it explicitly +(`POST /api/v1/pull {"model":"..."}`) rather than letting the first inference +trigger the download — see SKILL.md +[Step 6](SKILL.md#step-6-health-backend-then-pull-the-model--before-first-inference) +for why that matters for timeouts. --- @@ -172,7 +186,26 @@ All endpoints require `Authorization: Bearer {key}` when | `POST /api/v1/audio/transcriptions` | OpenAI Whisper-style transcription | | `POST /api/v1/audio/speech` | OpenAI TTS | | `POST /api/v1/images/generations` | OpenAI image generation | -| `POST /api/v1/messages` | Anthropic Messages API | +| `POST /api/v1/reranking` | Reranking — **note the spelling**, see below | +| `POST /v1/messages` | Anthropic Messages API — **not** under `/api/v1`, see below | + +**Two routes break the `/api/v1` pattern.** Everywhere else `/api/v1/` +and `/v1/` are aliases, which makes these two easy to miss: + +| Route | Correct path | What 404s | +|---|---|---| +| Anthropic Messages | `/v1/messages` | `/api/v1/messages` | +| Reranking | `/api/v1/reranking` | `/api/v1/rerank`, `/v1/rerank` | + +`/v1/rerank` is the spelling used by Jina, Cohere, vLLM, and llama.cpp itself, +so a retrieval app written against the convention 404s against the proxy. +Confusingly, the per-model llama.cpp back-port that lemond supervises *does* +serve `/v1/rerank` — but the proxy on `{port}` does not, and the back-port is +on a separate, dynamically assigned port. + +`GET /api/v1/models` lists only **downloaded** models. Add `?show_all=true` for +the full catalog; without it, validating a user-supplied model name against the +plain list produces a false negative for anything not yet pulled. ### Lifecycle (use these from the launcher / supervisor)