OpenAI compatible LLM endpoint?

[tftranslate]

I just discovered AnythingMCP as a potential MCP for my bank. This sounds great! Before venturing further, I have a question:
I see that for the (optional) AI-baed features you support OpenRouter, OpenAI and Anthropic.
I would much appreciate if you could add "OpenAI compatible" here, i.e. another environment variable setting where I can specify a different endpoint URL than the OpenAI default one (otherwise, the code logic is the same).
Reason: With the advent of platforms such as Strix Halo and DGX Spark, self-hosted models now are capable enough to run tasks similiar to the ones anythingmcp can use without the need for cloud services. In order to connect to a self-hosted llama.cpp, we need to be able to connect to an "OpenAI compatible" endpoint.

[alexuvlab]

OpenAI-compatible endpoints are becoming really important for this. Being able to switch between hosted providers and self-hosted models without changing the integration saves a lot of friction.

[keysersoft]

Sorry for the late reply, this one slipped past me.

Short answer: yes, that's reasonable and it's a small change on our side.

One thing worth clarifying first, since you mentioned a bank: the LLM is only used by the optional Knowledge Graph features (graph enrichment and skill generation). Connectors, tool execution and the MCP server itself never call a model. With KG_LLM_ENABLED=false, which is the default, nothing at all leaves your network.

For the feature itself, today the base URL is decided by KG_LLM_PROVIDER and hardcoded to either api.openai.com or openrouter.ai, and the key comes from OPENAI_API_KEY / OPENROUTER_API_KEY / ANTHROPIC_API_KEY. What I'd add is a KG_LLM_BASE_URL that overrides the OpenAI-compatible branch, and I'd also make the API key optional when that variable is set, because a local llama.cpp usually runs without auth and right now a missing key silently disables the whole thing.

The part I want to test properly before shipping it: we send response_format: {"type":"json_object"} and parse the reply as strict JSON. llama.cpp's server handles that, but not every OpenAI-compatible server does, so I'll make the parsing tolerant instead of assuming it works.

Which server are you planning to run behind it, llama.cpp directly or something like vLLM, and which model? I'd rather test against the same setup you'll actually use before telling you it works.

[tftranslate]

Hi, yes, I was referring to the optional "AI pass". I am working with llama.cpp and gemma4-31b for my main workload and qwen-3.6 35B A3B for automation tasks. Both seem to respond in proper json when asked to, I exploited this in another tool I wrote. Of the two models, Qwen is the more likely candidate I would offer to AnythingMCP. I am considering going to vLLM in the future though. I realize how enabling openAI "compatible" endpoints offers a plethora of configuration possibilities that you possibly cannot test on. Thanks for responding, much appreciated!