Feat/add serve local GGUF models over an OpenAI/Anthropic-compatible /v1 API

[ChrystianSchutz]

Summary

• Local models are now available over an OpenAI/Anthropic-compatible /v1 API. It supports chat completions, Responses, Anthropic Messages and count_tokens.
• GET /v1/models lists model ids based on file names. Duplicate file names get a folder prefix and, if still needed, a stable hash suffix, so no model is lost. File paths are never exposed.
• Models load on demand. An optional idle unload works like Ollama's keep_alive, and models can also be unloaded through the API.
• API access settings:
  • turn the API on or off
  • optional API key
  • optional LAN listener on a configurable port
• These access settings can still be changed while a model is generating.
• When the client disconnects, the request is cancelled, including while the model is still being looked up.
• Runtime fixes:
  • idle unload and targeted unload wait for the old process to exit before a new model can start
  • unload checks which model is loaded at the moment it acts
• New "Local API" guide in Settings with client examples.
• New docs: docs/LOCAL_API.md and docs/PERFORMANCE.md. docs/GENERATION_BETA.md is renamed to docs/GENERATION.md.

Testing

• npm run check (lint, tsc, unit, API/MCP E2E, client build, Playwright)
• Manual UI check, if relevant

Privacy Checklist

• No real exports committed
• No .lancedb/, .uploads/, temp folders, logs, or screenshots with private data
• Fixtures are synthetic/anonymized
• OpenRouter/off-device behavior is documented when relevant

Notes

• Environment variables: new local API options, see .env.example.
• LanceDB/data impact: none.
• UI screenshots: Settings → Local API panel and guide.