fix(local-ai-use): endpoint paths outside /api/v1, STT troubleshooting, per-modality fallback, and model-pull costs - #157
Merged
Conversation
…g, per-modality fallback, and model-pull costs Verified against `lemonade-server 11.5.2` (~24.04 packaging). - reference.md invites extending the /api/v1 rule to other modalities but documents only the three it covers. Added the two routes that break the pattern: Anthropic Messages is /v1/messages only (/api/v1/messages 404s), and rerank is exposed as /api/v1/reranking (/v1/rerank, the spelling used by Jina, Cohere, vLLM, and llama.cpp, 404s on the proxy). Also confirms the harmless case: for the three modalities this skill does cover, /api/v1/ and /v1/ are aliases, so the mixed spellings elsewhere need no change. - Add measured STT 500 recovery rows (RuntimeDirectory clearing, whisper-server startup failure on the ROCm backend's unbundled comgr dependency), a Step 1c per-modality backend-health check, a per-modality fallback policy (STT may fall back with disclosure; image/TTS must not fall back silently), and a completion-checklist item so a working chat path is no longer treated as proof that transcription works. - Note the ~5 GB SD-Turbo pull on first image request, state the version this skill was verified against (model IDs and system-info fields move between releases), and point --no-install at the managed/shared-machine case where the agent must not run sudo apt-get install. - reference.md: confirm model IDs against GET /api/v1/models rather than a stale server_models.json snapshot. - Rule template: flag the SD-Turbo pull and GPU contention on the image row (coresident is fine, concurrent inference is not). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Addresses all six review threads on this PR: - Drop both STT 500 troubleshooting rows. The ROCm whisper-server abort (exit 134, missing libamd_comgr.so.3) was already reported upstream (lemonade-sdk/lemonade#2292) and fixed in #2293, an ancestor of the v11.6.0 this skill targets per `git compare`. Running the same lemond binary by hand (bypassing this dev box's custom systemd drop-in) transcribed successfully over ROCm; running it under that drop-in reproduced the abort, so the drop-in's sandboxing -- not Lemonade's packaging -- caused it. The RuntimeDirectory row was likewise self-inflicted by that same local drop-in; the packaged unit sets RuntimeDirectory= correctly. Neither belongs in the skill's general troubleshooting table. Step 1c's per-modality backend health check stays, since it catches this class of failure without depending on either bug. - Simplify the image-generation row's setup note: drop "ask before generating" (unnecessary agent-side indirection) and say "several GB" instead of a size tied to one specific model. - Drop the messages/reranking route-exception section from reference.md; out of scope while this skill only routes image/TTS/STT. - Collapse the per-modality fallback table to one statement, since the three shipped modalities all take the same policy (disclosed fallback is fine); the embeddings/reranking "never fall back" guidance belongs with those modalities if they're ever added, not baked in ahead of them. - Remove the STT byte-stability note duplicated between the rule template and reference.md, keeping the reference.md copy. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Addresses all six review threads on this PR, plus two follow-on cleanups: - Drop both STT 500 troubleshooting rows. The ROCm whisper-server abort (exit 134, missing libamd_comgr.so.3) was already reported upstream (lemonade-sdk/lemonade#2292) and fixed in #2293, an ancestor of the v11.6.0 this skill targets per `git compare`. Running the same lemond binary by hand (bypassing this dev box's custom systemd drop-in) transcribed successfully over ROCm; running it under that drop-in reproduced the abort, so the drop-in's sandboxing -- not Lemonade's packaging -- caused it. The RuntimeDirectory row was likewise self-inflicted by that same local drop-in; the packaged unit sets RuntimeDirectory= correctly. Neither belongs in the skill's general troubleshooting table. Step 1c's per-modality backend health check stays, since it catches this class of failure without depending on either bug. - Simplify the image-generation row's setup note: drop "ask before generating" (unnecessary agent-side indirection) and say "several GB" instead of a size tied to one specific model. - Drop the messages/reranking route-exception section from reference.md; out of scope while this skill only routes image/TTS/STT. - Collapse the per-modality fallback table to one statement, since the three shipped modalities all take the same policy (disclosed fallback is fine); the embeddings/reranking "never fall back" guidance belongs with those modalities if they're ever added, not baked in ahead of them. - Remove the STT byte-stability caveat from both the rule template and reference.md; too in-the-weeds for this skill's general audience. - Replace the hardcoded "verified against Lemonade 11.5.2" claims in SKILL.md and reference.md with version-agnostic guidance (confirm against `lemonade status` / `GET /api/v1/models` on the version actually installed). The skill already detects capability rather than parsing version strings; pinning an exact tested version in the shipped text fights that design and goes stale immediately -- it already had, since 11.6.0 and 11.7.0 shipped after that snapshot was written. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Collaborator
Author
|
Please note this PR corresponds to the original #149 PR created by @AMD-melliott . |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Verified against
lemonade-server 11.5.2. Four independent fixes tolocal-ai-use`, found while using the skill end to end:reference.mddocuments the/api/v1aliasing rule but only for the three modalities this skill covers, inviting readers to extend it further. Two routes break that pattern: Anthropic Messages is/v1/messagesonly (/api/v1/messages404s), and rerank is exposed as/api/v1/reranking(/v1/rerank— the spelling used by Jina, Cohere, vLLM, and llama.cpp — 404s on the proxy). Documented both, and confirmed the harmless case explicitly: for the three modalities this skill does cover,/api/v1/<route>and/v1/<route>are true aliases, so no changes needed there.500 model_load_errortroubleshooting rows — a clearedRuntimeDirectory=systemd drop-in, and awhisper-serverstartup abort on the ROCm backend caused by an unbundledcomgrdependency (libamd_comgr.so.3missing from the ROCm backend's own package, only present in a system-wide ROCm install). Also added a Step 1c per-modality backend-health check (lemonade backends --all) and a completion-checklist item, because a working chat/image path does not prove transcription will work —autoresolves backends independently per modality and can silently fall back for one while having none for another.SKILL.mdandtemplates/local-ai-rule.md.--no-install(present but undiscoverable) needed a callout for the managed/shared-machine case where the agent must not runsudo apt-get install; and nothing stated which Lemonade version the skill was verified against, despite model IDs andsystem-infofields moving between releases. Also pointedreference.mdatGET /api/v1/modelsinstead of aserver_models.jsonsnapshot that goes stale.