Integrations eval (Open-Source-Ai) + optional Ollama profile, LiteLLM recipe

.env.example

@@ -14,7 +14,9 @@ HELLO_ALLOWED_ORIGINS="http://localhost:3000,http://127.0.0.1:3000,http://localh
 # HELLO_MCP_PORT="3030"
 
 # AI provider (OpenAI-compatible). Unset = AI features return 503 NO_AI_PROVIDER.
-# HELLO_AI_PROVIDER=openrouter        # openrouter | kilo | custom
+# HELLO_AI_PROVIDER=openrouter        # openrouter | kilo | ollama | custom
+# Local Ollama (optional, key not needed): HELLO_AI_PROVIDER=ollama HELLO_AI_MODEL=<pulled model>
+# LiteLLM/Bifrost gateway: HELLO_AI_PROVIDER=custom HELLO_AI_BASE_URL=http://127.0.0.1:4000/v1 HELLO_AI_API_KEY=<gateway key>
 # HELLO_AI_API_KEY=                   # OpenRouter or Kilo Code gateway key
 # HELLO_AI_MODEL=google/gemma-4-31b-it:free
 # HELLO_AI_BASE_URL=                  # optional override; required for custom

docs/INTEGRATIONS-EVAL.md

@@ -0,0 +1,74 @@
+# Open-Source-Ai candidates vs HELLO OS requirements
+
+Evaluated 2026-10-21 against HELLO OS `main` @ `972b3e9` and
+[CodesbyFebin/Open-Source-Ai](https://github.com/CodesbyFebin/Open-Source-Ai) `main` @ `15a7ccf`.
+
+**Catalog state checked first:** `.gitmodules` and 70 submodule (gitlink) entries now exist under
+`stacks/`. That is newer than the earlier review that found none. `catalog.json` is dated 2026-10-20.
+HELLO OS vendors none of them: they're upstream projects with their own licenses (some are not MIT,
+for example Trylon Gateway is BUSL-1.1). HELLO only talks to them over network protocols it already
+speaks.
+
+**Adoption rule:** adopt only if it serves a named HELLO requirement, is disabled by default, has an
+offline test, a documented disable path, and adds no dependency to the base install.
+
+| Catalog area (examples) | HELLO OS requirement it would serve | Current HELLO code that covers it | Verdict | Reason |
+| --- | --- | --- | --- | --- |
+| Local models (Ollama; also llama.cpp server, LocalAI, llamafile) | Local-first AI with no cloud account or key | `src/lib/ai/provider.ts` (OpenAI-compatible `chat`) | **Adopt: `ollama` provider profile** | It's one entry in `PROVIDERS` plus "key optional, model required". No new dependency, off unless `HELLO_AI_PROVIDER=ollama`, offline test in `tests/ollama-gateway-profile.test.mjs`. Other OpenAI-compatible local servers work through `custom`. |
+| Gateways (LiteLLM, Bifrost, Portkey) | Routing/fallback across providers behind one endpoint | `custom` provider + `HELLO_AI_BASE_URL`; retries and 429 handling in `chat()` | **Adopt: docs + test only** | It already works through `custom`. The test proves bearer forwarding and that the key stays out of `aiStatus`. No code or dependency added. |
+| Gateways (FreeLLMAPI, OpenCode Free Router, Trylon) | Same as above | Same | Defer | Free-tier aggregators route to third parties with unclear data handling. Trylon is BUSL-1.1. If needed, they're usable through `custom`, but not recommended. |
+| Eval / observability (promptfoo, DeepEval, Langfuse, Phoenix) | Promotion based on measured task results; traces | `src/lib/eval/*` (PR #34), `Trace` and `AuditLog` models, `ai.call` audit rows | Defer | HELLO's own eval loop is the source of truth and runs offline. A Langfuse/Phoenix export adapter needs a redaction design and a fake-collector test first. Revisit once suites grow. |
+| Memory (Mem0, OpenMemory, Letta, Graphiti) | Agent memory, role-scoped recall | `CortexMemory`, `src/lib/knowledge.ts` (filters by role before ranking) | Defer | No failing quality, scale or isolation test shows Cortex is insufficient. An external store would also have to preserve role filtering. |
+| RAG / vector DBs (Qdrant, Chroma, Milvus, Weaviate, LlamaIndex, Haystack, RAGFlow) | Document retrieval at scale | `knowledge.ts`, `semantic-cache.ts`, `similarity.ts` (local trigram cosine) | Defer | No realistic-corpus benchmark exists yet. Add a vector backend only once one shows the local approach failing. |
+| Agent frameworks (LangChain, LangGraph, CrewAI, AutoGen, smolagents, Agno, OpenHuman) | Scheduling and delegation | `scheduler.ts`, `delegation.ts`, approval gate `gate.ts` | Reject (for now) | A second control plane would bypass the wallet caps and approval gate. There's no concrete use case or interoperability test. |
+| Coding agents (OpenHands, Aider, Continue, OpenCode, Cline) | Agent-proposed code changes | `forge.ts`, `CodeMutation`, Wasm `sandbox.ts` | Reject | These need repo writes and command execution. HELLO's policy is that agent code is untrusted and changes come as human-reviewed patches. |
+| Code review (PR-Agent, Open Code Review) | Review of agent patches | None (human review) | Defer | It would post to GitHub (an external side effect). Consider it only as an advisory, approval-gated tool. |
+| Inference serving (vLLM, SGLang, TGI, Triton) | High-throughput self-hosted inference | `custom` provider | Defer | These are OpenAI-compatible, so they already work via `custom`. They need GPUs and aren't part of local quickstart. |
+| Chat UI / app builders (Open WebUI, LibreChat, Dify, Flowise, Langflow, n8n) | None identified | HELLO's own Next.js UI | Reject | Overlap with the HELLO UI and no requirement they serve. |
+| Backend infra (Supabase, PocketBase, Butterbase) | Hosted persistence | Prisma + SQLite/Turso (`db-config.ts`) | Reject | Turso was already chosen for Vercel. A second database layer adds risk with no gain. |
+| App templates / CodesbyFebin collection | None identified | — | Reject | Not runtime components. |
+
+## What was adopted
+
+1. **`ollama` provider profile** (`src/lib/ai/provider.ts`):
+   - Default base URL `http://127.0.0.1:11434/v1`.
+   - `HELLO_AI_API_KEY` is optional; when it's empty, no `Authorization` header is sent.
+   - `HELLO_AI_MODEL` is required, because the hosted default model won't exist locally.
+2. **LiteLLM/Bifrost gateway recipe**: documentation only, through the existing `custom` provider.
+
+Neither was benchmarked. No latency or quality numbers are claimed.
+
+### Enable
+
+```bash
+# Ollama (after `ollama pull <model>`)
+HELLO_AI_PROVIDER=ollama
+HELLO_AI_MODEL=<model you pulled>
+# optional: HELLO_AI_BASE_URL=http://host:11434/v1
+
+# LiteLLM proxy (or Bifrost) as a gateway
+HELLO_AI_PROVIDER=custom
+HELLO_AI_BASE_URL=http://127.0.0.1:4000/v1
+HELLO_AI_API_KEY=<gateway virtual key>
+HELLO_AI_MODEL=<model name as the gateway exposes it>
+```
+
+### Disable / remove
+
+Unset `HELLO_AI_PROVIDER`. AI routes then return `503 NO_AI_PROVIDER`. Nothing else depends on either profile.
+
+### Data handling
+
+- **Ollama:** prompts go only to the configured base URL, which is loopback by default. If you point it at another host, prompts go to that host.
+- **Gateways:** prompts go to the gateway and then to whatever upstreams it's configured for. Review its logging settings.
+- The key lives only in env. It is never stored in Prisma, returned by `aiStatus`, or written to audit rows (those store hashes and the model name).
+
+### Tests
+
+`tests/ollama-gateway-profile.test.mjs` runs against a local mock server with no network:
+- disabled by default
+- Ollama URL, optional key and required model
+- no auth header without a key
+- gateway bearer forwarding, with the key kept out of status
+
+Not tested against a real Ollama or LiteLLM install.

package.json

@@ -28,14 +28,15 @@
     "routes:manifest": "node scripts/gen-route-manifest.mjs",
     "test:design": "node --experimental-strip-types --test tests/design.test.mjs",
     "typecheck": "tsc --noEmit",
-    "test": "npm run test:aml && npm run test:evolution && npm run test:secrets && npm run test:gate && npm run test:scheduler && npm run test:connectors && npm run test:snapshots && npm run test:cache && npm run test:sandbox && npm run test:roadmap && npm run test:consensus && npm run test:truth && npm run test:ai && npm run test:turso && npm run test:eval",
+    "test": "npm run test:aml && npm run test:evolution && npm run test:secrets && npm run test:gate && npm run test:scheduler && npm run test:connectors && npm run test:snapshots && npm run test:cache && npm run test:sandbox && npm run test:roadmap && npm run test:consensus && npm run test:truth && npm run test:ai && npm run test:turso && npm run test:eval && npm run test:ollama-gateway",
     "test:truth": "node --test tests/no-fake-data.test.mjs",
     "test:ai": "prisma db push --skip-generate && node --experimental-strip-types --import ./tests/register.mjs --test tests/ai-provider.test.mjs",
     "test:mcp": "prisma db push --skip-generate && node --experimental-strip-types --import ./tests/register.mjs --test tests/mcp-inprocess.test.mjs",
     "postinstall": "prisma generate",
     "db:turso:sql": "prisma migrate diff --from-empty --to-schema-datamodel prisma/schema.prisma --script > prisma/turso/init.sql",
     "db:turso:apply": "node scripts/turso-apply.mjs",
     "test:turso": "node --experimental-strip-types --import ./tests/register.mjs --test tests/turso-adapter.test.mjs",
+    "test:ollama-gateway": "node --experimental-strip-types --import ./tests/register.mjs --test tests/ollama-gateway-profile.test.mjs",
     "test:eval": "prisma db push --skip-generate && node --experimental-strip-types --import ./tests/register.mjs --test tests/eval-loop.test.mjs",
     "eval:offline": "node --experimental-strip-types --import ./tests/register.mjs scripts/eval-run.mjs"
   },

src/lib/ai/provider.ts

@@ -5,6 +5,8 @@ export const PROVIDERS = {
   openrouter: { baseURL: "https://openrouter.ai/api/v1" },
   // Kilo Code AI Gateway, OpenAI-compatible: https://kilo.ai/docs/gateway
   kilo: { baseURL: "https://ai-gateway.kilo.ai/api/v1" },
+  // Local Ollama (OpenAI-compatible /v1). Optional; no key needed. Disabled unless HELLO_AI_PROVIDER=ollama.
+  ollama: { baseURL: "http://127.0.0.1:11434/v1" },
   custom: { baseURL: "" },
 } as const;
 export type ProviderName = keyof typeof PROVIDERS;
@@ -15,7 +17,7 @@ export const DEFAULT_MODEL = "google/gemma-4-31b-it:free";
 export class NoAIProviderError extends Error {
   code = "NO_AI_PROVIDER" as const;
   constructor(detail?: string) {
-    super(`no AI provider configured${detail ? `: ${detail}` : ""}. Set HELLO_AI_PROVIDER (openrouter|kilo|custom) and HELLO_AI_API_KEY.`);
+    super(`no AI provider configured${detail ? `: ${detail}` : ""}. Set HELLO_AI_PROVIDER (openrouter|kilo|ollama|custom) and HELLO_AI_API_KEY.`);
     this.name = "NoAIProviderError";
   }
 }
@@ -44,7 +46,10 @@ export function resolveAIConfig(env: Env = process.env): { ok: true; config: AIC
   const baseURL = (env.HELLO_AI_BASE_URL?.trim() || PROVIDERS[provider].baseURL).replace(/\/+$/, "");
   if (!baseURL) return { ok: false, reason: "HELLO_AI_BASE_URL is required for provider custom" };
   const apiKey = env.HELLO_AI_API_KEY?.trim() ?? "";
-  if (!apiKey) return { ok: false, reason: "HELLO_AI_API_KEY is not set" };
+  // Ollama ignores auth, so a key is optional there; every other provider needs one.
+  if (!apiKey && provider !== "ollama") return { ok: false, reason: "HELLO_AI_API_KEY is not set" };
+  // Local servers only have the models you pulled, so the hosted default is not a safe guess.
+  if (provider === "ollama" && !env.HELLO_AI_MODEL?.trim()) return { ok: false, reason: "HELLO_AI_MODEL is required for provider ollama (a model you have pulled)" };
   const timeoutMs = Number(env.HELLO_AI_TIMEOUT_MS) > 0 ? Number(env.HELLO_AI_TIMEOUT_MS) : 30_000;
   const maxRetries = Number.isInteger(Number(env.HELLO_AI_MAX_RETRIES)) && Number(env.HELLO_AI_MAX_RETRIES) >= 0 ? Number(env.HELLO_AI_MAX_RETRIES) : 2;
   return { ok: true, config: { provider, baseURL, apiKey, model: env.HELLO_AI_MODEL?.trim() || DEFAULT_MODEL, timeoutMs, maxRetries } };
@@ -96,7 +101,7 @@ export async function chat(messages: ChatMessage[], opts: { config?: AIConfig; m
         signal: ctl.signal,
         headers: {
           "content-type": "application/json",
-          authorization: `Bearer ${config.apiKey}`,
+          ...(config.apiKey ? { authorization: `Bearer ${config.apiKey}` } : {}),
           "x-title": "HELLO OS",
         },
         body: JSON.stringify({ model: config.model, messages, stream: false, max_tokens: opts.maxTokens ?? 1024, temperature: opts.temperature ?? 0 }),

tests/ollama-gateway-profile.test.mjs

@@ -0,0 +1,56 @@
+// Optional local/gateway profiles (Ollama, LiteLLM via custom) against a local
+// mock OpenAI-compatible server. No real Ollama/LiteLLM, no network.
+import { test, before, after } from "node:test";
+import assert from "node:assert/strict";
+import { createServer } from "node:http";
+
+let server, base, last = null;
+before(async () => {
+  server = createServer((req, res) => {
+    let b = ""; req.on("data", (c) => (b += c)); req.on("end", () => {
+      last = { url: req.url, auth: req.headers.authorization, body: JSON.parse(b || "{}") };
+      res.writeHead(200, { "content-type": "application/json" });
+      res.end(JSON.stringify({ model: last.body.model, choices: [{ message: { content: "ok" } }] }));
+    });
+  });
+  await new Promise((r) => server.listen(0, "127.0.0.1", r));
+  base = `http://127.0.0.1:${server.address().port}/v1`;
+});
+after(() => server.close());
+const P = () => import("../src/lib/ai/provider.ts");
+
+test("disabled by default: no provider env means not configured", async () => {
+  const { resolveAIConfig } = await P();
+  assert.equal(resolveAIConfig({}).ok, false);
+});
+
+test("ollama: default base URL is loopback, key optional, model required", async () => {
+  const { resolveAIConfig, PROVIDERS } = await P();
+  assert.equal(PROVIDERS.ollama.baseURL, "http://127.0.0.1:11434/v1");
+  const noModel = resolveAIConfig({ HELLO_AI_PROVIDER: "ollama" });
+  assert.equal(noModel.ok, false);
+  assert.match(noModel.reason, /HELLO_AI_MODEL/);
+  const r = resolveAIConfig({ HELLO_AI_PROVIDER: "ollama", HELLO_AI_MODEL: "llama3.2" });
+  assert.equal(r.ok, true);
+  assert.equal(r.config.apiKey, "");
+});
+
+test("ollama profile: chat sends no authorization header when no key", async () => {
+  const { resolveAIConfig, chat } = await P();
+  const r = resolveAIConfig({ HELLO_AI_PROVIDER: "ollama", HELLO_AI_BASE_URL: base, HELLO_AI_MODEL: "llama3.2", HELLO_AI_MAX_RETRIES: "0" });
+  const out = await chat([{ role: "user", content: "hi" }], { config: r.config });
+  assert.equal(out.output, "ok");
+  assert.equal(last.url, "/v1/chat/completions");
+  assert.equal(last.auth, undefined);
+  assert.equal(last.body.model, "llama3.2");
+});
+
+test("LiteLLM-style gateway via custom: bearer sent, key never in status", async () => {
+  const { resolveAIConfig, chat, aiStatus } = await P();
+  const env = { HELLO_AI_PROVIDER: "custom", HELLO_AI_BASE_URL: base + "/", HELLO_AI_API_KEY: "sk-litellm-test", HELLO_AI_MODEL: "ollama/llama3.2", HELLO_AI_MAX_RETRIES: "0" };
+  assert.equal(resolveAIConfig({ ...env, HELLO_AI_API_KEY: "" }).ok, false, "custom still needs a key");
+  const r = resolveAIConfig(env);
+  await chat([{ role: "user", content: "hi" }], { config: r.config });
+  assert.equal(last.auth, "Bearer sk-litellm-test");
+  assert.equal(JSON.stringify(aiStatus(env)).includes("sk-litellm-test"), false);
+});