Problem statement
Ollama ships in the container, but there's no real path to using it: GPU access into the container isn't guaranteed, container memory may need to scale for the model in use, and there's no guide for running a local model and making it usable across all decks.
What success looks like
A user can run a local model in the shipped container with GPU access working, sized memory, and clear documentation for making it available across every deck.
Impacted repositories
Acceptance criteria
- The container reliably gets GPU access when available.
- Container memory can be sized appropriately for the local model in use.
- A published guide covers running a local model and exposing it across decks.
Related
- See the separate Research item on tuning llama.cpp/Ollama for speed and larger models (MoE offload, etc.) — that informs this epic but isn't a sub-issue of it.
Problem statement
Ollama ships in the container, but there's no real path to using it: GPU access into the container isn't guaranteed, container memory may need to scale for the model in use, and there's no guide for running a local model and making it usable across all decks.
What success looks like
A user can run a local model in the shipped container with GPU access working, sized memory, and clear documentation for making it available across every deck.
Impacted repositories
Acceptance criteria
Related