The question
What's the best way to tune local inference (llama.cpp and/or Ollama) for faster operation and larger models — including MoE offload — on the hardware a typical user's container will actually have?
Background
Ships Ollama in the container today, but first-class local inference (see that epic) needs local inference to actually be fast and capable enough to be worth using, not just present.
How we'll answer it
Compare llama.cpp against Ollama for efficiency; investigate MoE offload and other techniques for running larger models on constrained hardware. Likely output is a guide/video, and possibly a tool or skill that applies tuned settings automatically.
The question
What's the best way to tune local inference (llama.cpp and/or Ollama) for faster operation and larger models — including MoE offload — on the hardware a typical user's container will actually have?
Background
Ships Ollama in the container today, but first-class local inference (see that epic) needs local inference to actually be fast and capable enough to be worth using, not just present.
How we'll answer it
Compare llama.cpp against Ollama for efficiency; investigate MoE offload and other techniques for running larger models on constrained hardware. Likely output is a guide/video, and possibly a tool or skill that applies tuned settings automatically.