fix(local-ai-app-integration): correct endpoint paths, Linux NPU row, and unpulled-model behaviour - #150
Open
AMD-melliott wants to merge 1 commit into
Open
Conversation
… and unpulled-model behaviour Verified against a live `lemonade-server 11.5.2` on a Ryzen AI MAX+ PRO 395 (Strix Halo / gfx1151, XDNA 2 NPU, Linux). Each change below is a reproduction, not a review comment. Endpoint paths (Step 5 client table, reference.md route table): - `@anthropic-ai/sdk` `base_url` must be the bare `http://127.0.0.1:{port}`, not `.../api/v1`. The SDK appends `/v1/messages`, so the documented value requests `/api/v1/v1/messages` and 404s on the first call. Lemonade serves Anthropic Messages at `/v1/messages`, outside the `/api/v1` prefix. Reproduced end-to-end in a real Anthropic-Messages client. - reference.md listed `POST /api/v1/messages`, which 404s. Corrected, and both exceptions to the `/api/v1` pattern are now called out together. - Added reranking: the proxy serves `/api/v1/reranking`. `/v1/rerank` — the spelling used by Jina, Cohere, vLLM, and llama.cpp — 404s on the proxy even though the supervised back-port serves it. - Added a Step 7 row: a 404 body carrying a `path` field is a routing mistake, not a missing model. Linux NPU (Step 2 profile table, reference.md recipe tables): - Removed the "Speech-to-text (Linux NPU) -> whisper-v3-turbo-FLM / flm" row. No NPU backend installs on Linux at 11.5.2: whispercpp:npu -> Requires Windows ryzenai-llm:npu -> Requires Windows flm:npu -> Requires AMD XDNA 2 AMD NPU The last is a device gate that cannot pass on Linux, where `system-info` reports `amd_npu.family: ""` on hardware that is XDNA 2. This also removes a contradiction with the existing "# Windows NPU path only" packaging example. Unpulled models (Step 6, Step 7, verification checklist): - At 11.5.2 an unpulled model does not return an empty 200; the first inference blocks until the download completes. That collides with the mandatory 120s timeout and presents as a hang, not a blank. Both behaviours are now documented, both cured by the same explicit pull, and the pull step is rejustified around latency control rather than silent failure. - `GET /api/v1/models` returns downloaded models only (23 vs 145 catalogued on the test host). Model-name validation needs `?show_all=true`. Also: Step 1 said "record three things" above a four-item list; the shutdown note read as though Windows needs the harsher call, when the real point is that Windows has no graceful equivalent, so the wait is what matters.
Collaborator
|
@AMD-melliott Thanks for the PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Verified against a live
lemonade-server 11.5.2on a Ryzen AI MAX+ PRO 395(Strix Halo / gfx1151, XDNA 2 NPU, Linux).
This PR:
base_url. The documented value pointedbase_urlat.../api/v1, but the SDK appends/v1/messagesitself, so the first callrequested
/api/v1/v1/messagesand 404s. Corrected to the barehttp://127.0.0.1:{port}, since Lemonade serves Anthropic Messages at/v1/messages, outside the/api/v1prefix. Reproduced end-to-end in areal Anthropic-Messages client.
reference.mdroute table.POST /api/v1/messages404s; corrected,and both exceptions to the
/api/v1pattern are now called out together.Also adds reranking: the proxy serves
/api/v1/reranking, while/v1/rerank(the spelling used by Jina, Cohere, vLLM, and llama.cpp)404s on the proxy even though the supervised back-port serves it.
pathfield means a routingmistake, not a missing model.
whisper-v3-turbo-FLM/flm). No NPU backend installs on Linux at 11.5.2 —whispercpp:npuandryzenai-llm:npurequire Windows, andflm:npugates on an NPU familystring that reports empty on Linux even on XDNA 2 hardware. This also
removes a contradiction with the existing "Windows NPU path only" example.
an empty 200; the first inference blocks until the download completes,
which collides with the mandatory 120s timeout and presents as a hang, not
a blank response. Documents both symptoms, notes both are fixed by the
same explicit pull, and reframes the pull step around latency control
rather than silent failure. Also notes
GET /api/v1/modelsreturnsdownloaded models only, so name validation needs
?show_all=true.has no graceful shutdown equivalent (the original wording implied Windows
just needs a harsher kill signal).