[agicy]

Problem

Robonix's model consumers each duplicate endpoint, model, key and feature assumptions. There is no single source of truth for what a model supports, so changing a model means editing every consumer and feature mismatches surface only at runtime.

User-visible impact

A deployment can set ROBONIX_LLM_CATALOG to one YAML/JSON catalog. Each consumer resolves its endpoint from feature and budget requirements at startup, then connects to the provider directly. Without the variable every consumer keeps its existing VLM_*/manifest path; rollout is per service and rollback is unsetting the variable.

Implementation

• Python: robonix_api.llm in pylib/robonix-api; Rust: system/llm-catalog.
• ModelCatalog: validated YAML/JSON ModelSpec list (endpoint, model, api_key_env, features, context, p95 latency, prices, protocol); api_key_env is a variable name, never a key.
• Resolver: hard feature filter → context/latency/cost filter → rank (latency, price, id) → best-first ModelSpec[]. json_schema falls back to json_object only when no strict candidate fits and marks targets downgraded=true.
• Same operations in both languages: ModelCatalog.load, ModelCatalog.model, ModelCatalog.resolve, Requirements.default, resolve_from_env, api_key_for.
• Pilot: catalog selects the VLM endpoint; default features vision,tools,stream; ROBONIX_LLM_* overrides budgets.
• Opt-in consumers: Scene (vision), Memory (json_object / vision), Memsearch (text), VLM Verifier (vision, covering the new before/after and multi-camera paths through the shared VlmConfig.parse), and CI scripts. All keep the legacy path when the catalog is unset.
• No service, proxy, port or request hop; consumers connect directly.

Validation

• PYTHONPATH=pylib/robonix-api python3 -m unittest discover -s pylib/robonix-api/tests -p 'test_llm_*.py' → 18 passed
• Scene focused CI (pytest/numpy/scipy/httpx/Pillow only): 63 passed, 5 skipped; second group 29 passed
• Memory: 14 + 8 passed; Memsearch: 2 passed; Scene catalog: 1 passed
• VLM Verifier: 33 passed after rbnx codegen -p services/verifiers/vlm_verifier --mcp
• Rust: cargo test -p robonix-llm-catalog 3 passed; cargo test -p robonix-pilot 111 passed; cargo test -p robonix-cli 36 passed
• Catalog-only boot (VLM_* unset, only ROBONIX_LLM_CATALOG + provider key): rbnx preflight skipped, Pilot resolved vlm=catalog
• cargo fmt --check, cargo clippy ... -D warnings, cargo metadata --locked, uv lock --check: clean
• Webots office smoke: all components ACTIVE; quick ask OK; exploration produced 41 objects / 75 relations; Pilot 2 requests, 22,281 input / 1,237 output, 81% cache
• DeepSeek two-image smoke: red, blue; simulation shut down with no robonix containers and GPUs back to 1 MiB

Configuration and migration

• ROBONIX_LLM_CATALOG — absolute catalog path; unset means legacy behavior.
• ROBONIX_LLM_FEATURES — Pilot's required model features; defaults to vision,tools,stream.
• Keys are read by consumers from each target's api_key_env; Core never stores keys.
• No new capability contracts, generated code, ports, processes or Python dependencies; uv.lock is unchanged.
• Rollout is per service; rollback is unsetting the variable.

Related issues

None.

Screenshots / terminal output

No CLI, TUI or web UI changes.

[enkerewpo]

Maybe ModelCatalog? Since we use catalog for capabilities before

[agicy]

Got it. I'll rename it to ModelCatalog in both Python and Rust to avoid confusion with the capability catalog.

[agicy]

Renamed.

[enkerewpo]

please don’t use capabilities here and also in other code structures, since we used it as our skill/service/primitive interface. Maybe use feature

[agicy]

Renamed.