[0.3] Build universal multi-backend model capability fusion layer

Goal

Turn Quantum Runtime 0.3 into a model- and engine-neutral inference layer that can run the broadest practical range of local and externally served models while adopting the strongest tested capabilities from multiple mature upstream runtimes.

Quantum Runtime must not become an Ollama fork and must not become a Gemma-specific runtime. Gemma 4 remains an important reference family because it exercises dense, MoE, multimodal, thinking, tool and long-context behavior, but the Runtime contracts must remain generic enough for unrelated model families.

The public Runtime API, model registry, lifecycle contracts and canonical model identity remain owned by Quantum Runtime while execution is delegated through versioned backend adapters.

Universal model principle

The target is not literally every model ever created. A model is usable when Quantum Runtime has either:

a compatible native/local backend adapter for its artifact/architecture, or
an external backend/API adapter that can expose it through the Runtime capability contract.

Clients must not need model-family-specific code merely to select the execution engine.

Target model-family coverage should include, without hardcoding the Runtime to any one family:

Gemma families
Llama families
Qwen families
Mistral/Mixtral families
Phi families
DeepSeek families
GLM families where compatible backends exist
embedding/reranker models
vision-language models
audio-capable models
dense and Mixture-of-Experts architectures
future model families added through capability + backend adapters rather than API rewrites

Model-family-specific parsing or execution quirks belong in model/backend profiles, not in the generic public API.

Initial upstream capability sources

Use the following projects as capability/reference sources where technically and legally compatible:

Ollama: compatibility behavior, model packaging/profile behavior, thinking/tool API behavior, lifecycle experience and adoption fallback
llama.cpp / ggml: portable GGUF execution, CPU/CUDA/ROCm/Vulkan/SYCL backends, dense/MoE execution/offload and low-level local inference
vLLM: high-throughput server execution, MoE, speculative decoding/MTP, KV-cache offload, tensor/pipeline/expert parallelism and modern quantization paths
MLX / MLX-LM: Apple Silicon optimized dense/MoE execution and cache behavior
Hugging Face Transformers: architecture/configuration reference and model metadata compatibility, not necessarily a default production inference backend

Additional backends may be added later if they provide a capability or hardware path not already covered.

Do not copy whole upstream codebases into Quantum Runtime. Prefer adapter/process/library boundaries. Any selectively reused code must have compatible licensing, preserved notices and explicit provenance.

Required Runtime architecture

Quantum Runtime API
        |
canonical model registry
        |
model + backend capability contract
        |
capability router / scheduler
        |
backend interface v1alpha1
   |          |          |          |
llama.cpp   MLX       vLLM       Ollama
portable    Apple     server     adoption/
local       Silicon   GPU        fallback
        |
 future compatible adapters

The same canonical model identity may have multiple backend artifacts. Backend selection must be based on declared capabilities, artifact compatibility, host hardware, policy and tested health rather than application-specific or model-family-specific hardcoding.

Generic model architecture contract

Quantum Runtime must represent architecture characteristics generically, including where applicable:

dense vs MoE architecture
total vs active parameters
routed/shared experts and expert top-k
attention implementation and head dimensions
sliding-window/global attention behavior
context limits and extended-context profiles
tokenizer/chat-template identity
BOS/EOS/system-role behavior
text, image and audio modalities
embedding/reranking capability
function/tool calling
structured-output capability
configurable thinking/reasoning
multimodal projector/encoder requirements
speculative/draft/MTP decoding capability
quantization format and backend compatibility
KV/prompt cache behavior
CPU/GPU/expert offload capability

Unsupported architecture features must be reported explicitly rather than guessed or silently emulated.

Gemma 4 reference profile

Gemma 4 is the initial high-complexity conformance family, not the Runtime default.

The Runtime must explicitly represent and test relevant Gemma 4 features including:

E2B / E4B edge variants
12B and 31B dense variants
26B-A4B Mixture-of-Experts variant
routed/shared expert metadata
per-layer input / PLE behavior
shared KV layers
sliding-window vs global attention
differing attention head dimensions
final-logit softcapping
128K / 256K context profiles
text/image/audio capability differences
native system role
function/tool calling
configurable thinking/reasoning
multimodal projector/encoder requirements
MTP/draft/speculative decoding support where available

Do not assume every Gemma 4 size has the same modality or execution capabilities.

Backend capability contract

The backend contract should expose generic capabilities such as:

architecture.dense
architecture.moe
moe.total_experts
moe.active_experts
moe.shared_experts
moe.expert_offload
moe.expert_parallel
quantization.formats
speculative.mtp
speculative.draft_model
cache.kv_offload
cache.prompt_cache
multimodal.vision
multimodal.audio
embeddings
reranking
reasoning.control
tools.calling
tools.streaming
structured_output

Backends may report unsupported rather than emulating a feature unsafely.

Agent compatibility boundary

Runtime 0.3 must already preserve the primitives required by future agents even though full agent orchestration is tracked separately.

The inference/backend layer must not destroy or flatten:

assistant content vs reasoning/thinking channels
structured tool/function calls
tool call IDs
multiple/parallel tool calls
typed JSON arguments
tool results linked to the originating call
finish/stop reasons
usage/token metadata
streaming deltas for content, reasoning and tool arguments
structured outputs / JSON-schema-constrained responses
cancellation and resumability metadata where the backend supports it

Backend adapters must normalize these primitives into the Quantum Runtime contract rather than leaking backend-specific event shapes to clients.

Full agent/session/tool orchestration, MCP-style tool connectivity and agent execution state are separate from model inference and will be tracked in a dedicated milestone.

Upstream version policy

Being current means latest-known-good, not blindly tracking upstream HEAD.

Add a machine-readable tested-upstream lock/ledger recording, per backend:

upstream project
tested tag or commit
test date
enabled capabilities
known disabled/broken capabilities
license/provenance metadata
supported hardware classes
relevant model-family compatibility notes

A newer upstream release may be evaluated automatically, but must not replace the production-tested pin until the Runtime conformance suite passes.

Universal conformance suite

Create backend-independent tests for:

canonical model metadata parsing
tokenizer/chat-template/BOS/system-role handling
Unicode tokenization/detokenization, including German umlauts and ß
reasoning/thinking true/false response framing
streaming content and reasoning separation
tool calls with strings, booleans, numbers, arrays, objects, null and parallel calls
tool-call ID preservation across streaming and final responses
structured JSON/schema outputs
long-context prefill boundaries
cancellation during prefill and decode
multimodal request capability negotiation
dense vs MoE loading metadata
expert routing configuration validation
quantization capability negotiation
speculative decoding capability/fallback
backend crash/restart reporting
no silent fallback that changes canonical model identity
model-family fixtures from multiple unrelated families so Gemma-specific assumptions cannot leak into the generic contract

Large-weight/hardware tests may run in a separate optional hardware matrix, but protocol and capability regression fixtures belong in normal CI.

Initial backend priority

1. llama.cpp/ggml native adapter

Preferred first portable local backend for Linux and general hardware. It should give Quantum Runtime its first inference path that no longer depends on the Ollama daemon.

2. existing Ollama adapter

Retain as adoption/fallback and compatibility reference. Do not remove the current non-destructive Ollama path.

3. MLX adapter

Apple Silicon optimized path. Keep separate from generic Linux behavior.

4. vLLM adapter

Server/workstation multi-GPU and throughput-oriented path. This may initially be an external managed backend rather than an embedded library.

Stability and regression policy

Known upstream regressions must be representable as backend/model/hardware capability blocks rather than forcing all clients to discover them independently.

Examples:

reasoning disabled but empty content returned
model loads but tokenizer/BOS/chat-template handling corrupts output
MoE expert offload crashes only on one hardware backend
multimodal capability advertised while a backend cannot actually consume the modality
tool-call streaming loses IDs or corrupts JSON arguments
newer backend release regresses a previously verified model path

If a capability is known-bad on a backend/version/model/hardware combination, Quantum Runtime must fail closed or select a tested alternative. It must not silently claim full support.

CoreOS TCI relationship

Quantum Runtime remains model-neutral.

The future Quantum CoreOS TCI profile may target Gemma 4 E4B and receive the best compatible backend automatically, but Gemma 4 must not become the generic Runtime default.

The TCI personality/persona package remains separate from inference-engine selection and separate from generic agent execution state.

Acceptance criteria

Non-goals

vendoring complete upstream runtimes into one monolithic source tree
synchronized upstream version numbers
automatically trusting newest upstream commits
requiring every backend to implement every capability
tying Quantum Runtime to Gemma 4 or any single model family
embedding a full autonomous agent orchestrator into the 0.3 backend layer
moving TCI personality or user memory into the Runtime backend layer

CPU-first execution and tiered memory placement is now tracked as the cross-cutting Runtime 0.3 priority in #11. Backend interface v1alpha1 in this issue must expose enough placement/capability information for CPU-only, hybrid accelerator, MoE expert placement, KV/cache tiering and latest-known-good performance guardrails. Do not design the backend contract around GPU-first assumptions.

Gemma 4 context-policy note from live Turin/Ollama testing (2026-09-04): do not assume generic num_ctx injection is performance-safe for Gemma 4. Public Ollama profiles can accept explicit num_ctx, so this is not a global syntax prohibition. Runtime should instead model context handling as a backend/model capability. For the Ollama Gemma 4 performance baseline, default to backend/model-native context policy unless conformance tests prove an override safe. Represent fields such as context.native_max, context.requested_limit, context.backend_managed, context.override_supported, and context.override_verified. Treat user/app context limits as admission/scheduling constraints where possible rather than blindly forwarding them as backend num_ctx. Add conformance cases for E2B/E4B/12B/26B/31B comparing native/default vs explicit context overrides for correctness, TTFT, RAM/KV footprint, and decode throughput. A model/backend/version combination may block explicit overrides if testing shows regressions.

Live Turin/Ollama benchmark update (2026-09-04): repeated testing across multiple Gemma 4 family models shows that explicitly injecting num_ctx can reduce decode throughput by roughly 17 tokens/s versus leaving context backend/model-managed. Treat this as a measured performance regression class, not just an unverified concern. Quantum Runtime must default Gemma 4 + Ollama context handling to backend-managed/native policy and only forward an explicit context override when that exact backend/model/version/hardware profile has passed conformance + performance validation. Add a regression guard so a validated native/default path cannot be replaced by an override path with a material throughput loss. Preserve user/app context requests as Runtime scheduling/admission limits when possible instead of blindly mapping them to backend num_ctx.

Concrete Turin/Ollama Gemma 4 benchmark evidence from screenshots (2026-09-04): with explicit num_ctx around 16K, decode throughput measured 27.46 tok/s. With no explicit context override, two adjacent runs measured 47.83 tok/s and 47.75 tok/s. That is a +20.29 tok/s improvement versus the explicit-context run, or about 74% higher decode throughput relative to the 27.46 tok/s baseline. Prompt-eval rates do not show the same pattern, so the regression appears especially decode-path sensitive. These screenshots are not yet a fully controlled same-prompt A/B benchmark, but the repeated ~47.8 tok/s no-override result plus prior multi-model testing is strong enough to keep Gemma 4 + Ollama num_ctx overrides blocked by default until a controlled conformance benchmark proves a specific override safe.

Additional live Turin/Ollama validation (2026-09-04): the num_ctx decode-throughput regression has now been reproduced by the user across Gemma 4 E2B, E4B and 26B with the same qualitative result: explicit num_ctx materially lowers decode throughput versus leaving context backend/model-managed. Treat this as a family-level Ollama/Gemma 4 compatibility-performance rule for these validated variants, not a single-model anomaly. Default policy for E2B/E4B/26B should be context.backend_managed=true and context.override_verified=false until a controlled same-prompt A/B benchmark proves a specific override safe. Keep 12B/31B unclassified until tested rather than extrapolating blindly, but add the same regression test matrix for them.

Live Turin/Ollama Gemma 4 tuning result (2026-09-04): best observed baseline across the tested Gemma 4 family uses only the Gemma 4 renderer/parser plus the core sampling parameters below, with no explicit context override and no additional tuning knobs:

RENDERER gemma4
PARSER gemma4
PARAMETER temperature 1.0
PARAMETER top_k 64
PARAMETER top_p 0.95

For the validated E2B/E4B/26B paths, treat this as the current minimal known-good Ollama profile on the Turin reference system. Do not automatically inject num_ctx or extra model parameters into the production profile unless a controlled A/B benchmark shows they improve correctness/performance. Runtime model profiles should distinguish required, known-good, optional, and known-bad/unverified parameters so client defaults cannot accidentally degrade the backend-native fast path.

Downstream production-style validation from the Ember Global Chat setup described in the whitepaper (2026-09-04): after reducing the Gemma 4 Ollama profile to the minimal known-good settings (RENDERER gemma4, PARSER gemma4, temperature 1.0, top_k 64, top_p 0.95) and removing extra tuning such as explicit num_ctx, observed decode throughput increased from a prior fluctuating ~19-20 tok/s to about 26 tok/s. Treat this as additional evidence that minimizing unverified backend parameters can materially improve real application throughput, not just synthetic terminal benchmarks. Keep this result as a downstream regression target for Runtime profile changes. This is an application-level observation and should later be reproduced under a controlled same-prompt benchmark before being used as a formal whitepaper performance claim.

Correction/clarification from live screenshot (2026-09-04): the ~26 tok/s downstream benchmark is specifically the Game Ember workload, not a generic/global-chat label. Screenshot metrics: prompt eval count 2154 tokens, prompt eval duration 12.945608 s, prompt eval rate 166.39 tok/s, eval count 3097 tokens, eval duration 1m58.843473 s, eval rate 26.06 tok/s, total duration 2m11.817734457 s. This is especially useful as a production-like long-response regression target because the run includes the full Game Ember persona/thinking path rather than a tiny synthetic prompt. Keep the previously observed 19–20 tok/s range as the old baseline and use ~26 tok/s as the current observed post-profile-tuning result, pending controlled repeated A/B runs.

Continued Game Ember live-session validation after removing explicit num_ctx (2026-09-04): subsequent turns measured 25.00 tok/s and 24.67 tok/s decode, following the prior 26.06 tok/s run. Across these three sustained Game Ember turns the mean decode throughput is ~25.24 tok/s. This supports the conclusion that the no-override improvement is sustained across a real persona/thinking chat session rather than a one-shot spike. Compared with the previously observed 19-20 tok/s production range, the current sustained band is roughly 24.7-26.1 tok/s. One run also showed 2304 prompt tokens evaluated at 1938.22 tok/s, but treat that prompt-eval figure as warm/session-state dependent and not a cold-prefill baseline. Keep Game Ember as a downstream regression workload for the Gemma 4 minimal known-good profile.

Root-cause clarification from continued live Game Ember / Gemma 4 26B testing (2026-09-04): the throughput gain is not attributable to removing num_ctx alone. The previous Game Ember Modelfile also contained additional tuning controls, including a predict-related setting, and throughput only rose into the sustained ~24.7-26.1 tok/s band after those extra parameters were removed as well. Therefore the correct Runtime rule is broader: the Gemma 4 Ollama fast path should use the minimal known-good profile (RENDERER gemma4, PARSER gemma4, temperature 1.0, top_k 64, top_p 0.95) and must not inject any additional generation/context/predict knobs unless that exact parameter has passed controlled A/B validation for the backend/model/version/hardware profile. Do not record num_ctx as the sole causal parameter. Track individual extras as unverified/known-bad candidates until isolated tests determine their separate effect.

New application-profile constraint captured in #13: E.M.B.E.R. production deployments require an MoE architecture and currently use a 64 GiB DDR5 ECC / 8 physical core (~2.6 GHz reference class) CPU-only baseline, with GPU optional. This must remain a profile-level admission rule rather than a generic Runtime minimum. Backend capability routing in this issue should therefore expose MoE as a first-class capability so CoreUI/Game can ask for architecture.required = moe without hardcoding one model family. For the current Gemma 4 family, 26B A4B is the MoE candidate; other compatible MoE families can satisfy the same contract once conformance/performance-tested. See #13.

Test provenance clarification for the current Gemma 4 performance findings: most of the 5-6 hour manual validation session was executed through Ember CoreUI rather than directly in the Game backend. The Game Ember 26B runs are therefore best treated as downstream confirmation of the same backend/model behavior, while CoreUI is currently the primary real-world regression harness for the minimal-profile/context-policy findings. Keep both in the conformance matrix, but label CoreUI as the main source of the extended test series.

Downstream client alignment update (2026-09-05): the current minimal known-good Gemma 4 + Ollama request policy is now merged into Ember CoreUI and the private downstream Game client. CoreUI PR #26 moved both synchronous and SSE generation onto one shared option policy. Normal generation now sends only temperature=1.0, top_p=0.95, and top_k=64; context, predict, thread, repeat, seed, and stop controls are not injected by default. Legacy/backend-specific controls remain behind an explicit experimental gate plus explicit configuration, and vision/continuation paths no longer force the old context/predict budgets. The private downstream client received the same normal-path policy. This keeps application PHP from hardcoding the exact backend tuning behavior that Runtime 0.3 is expected to own. Specialized diagnostic/task-specific overrides may still exist, but the production default path is minimal.

0.3 implementation update: backend contract foundation released

PR #14 is merged and published as v0.3.0-alpha.1.

This first 0.3 slice lands the architectural boundary required by this issue without pretending the native llama.cpp backend already exists:

quantum.runtime/backend/v1alpha1 with explicit supported, unsupported, conditional, and unknown states
deterministic canonical-model routing across model metadata, artifact/backend candidates, requested capabilities, backend state, and matching model policies
artifact-specific backend/format/role metadata while preserving one canonical model identity
generic dense/MoE metadata, optional total/active parameter and expert topology fields, context-policy metadata, and additional agent-relevant capability flags
/v1/backends, /v1/route, /v1/model-policies, and /v1/upstreams
Gemma 4 + Ollama Turin minimal policy with temperature=1.0, top_k=64, top_p=0.95; num_ctx, predict/thread/repeat/seed/stop controls remain blocked-unverified unless isolated A/B validation proves them safe for the concrete stack
the policy explicitly records that the observed throughput improvement belongs to the complete minimal profile and is not attributed to num_ctx alone
upstream ledger with the current Ollama evidence deliberately marked observed-unpinned because the exact tested upstream version was not captured; it therefore cannot silently qualify as latest-known-good
llama.cpp/ggml recorded as the first planned native portable backend
CPU is represented as a first-class placement capability; GPU/hybrid remain optional/conditional
existing Ollama adoption/compatibility forwarding remains intact for CoreUI/Game

CI and release verification passed, including go vet, go test -race ./..., legal/systemd checks, version checks, linux amd64/arm64 builds, and release archive generation.

Next 0.3 slice

Implement the first native llama.cpp/ggml adapter, then connect host hardware discovery/calibration and the CPU-first/tiered placement policy from #11/#12. The native adapter must conform to the backend contract rather than bypassing it, and canonical model identity must remain stable when both Ollama and native artifacts are available.

Runtime 0.3 implementation update: v0.3.0-alpha.2 has landed via PR #15.

This slice adds the first direct llama.cpp/GGUF execution path through an operator-provided llama-server. When QUANTUM_RUNTIME_BACKEND=llama.cpp is selected, Ollama is no longer in the inference request path; ollama remains the default adoption/fallback mode.

Implemented in this slice:

direct /api/chat -> /v1/chat/completions, /api/generate -> /v1/completions, and embed -> /v1/embeddings compatibility translation
SSE -> Ollama NDJSON streaming translation, including preservation of upstream reasoning text as message.thinking
strict configured model/API-alias matching, with no silent canonical-model substitution
separate optional llama-server API key; Runtime bearer credentials are not reused upstream
only the validated temperature, top_p, and top_k request knobs are normalized; context/predict/thread/repeat/seed/stop settings are not injected
fail-closed behavior for vision, tool calls/tool history, explicit reasoning control (think:true and think:false), and structured output until those semantics are normalized end to end
protocol/unit tests plus the full Runtime verification suite on the branch, PR, and merged main

The upstream ledger intentionally remains observed-unpinned. This release proves the adapter/protocol boundary in CI, not a real llama.cpp + model + hardware conformance result. No llama.cpp tag/commit is therefore promoted to tested/latest-known-good yet.

Next 0.3 slice: host hardware discovery/calibration and CPU-first tiered RAM/VRAM/NVMe placement (#11/#12), followed by real multi-backend selection/fallback and a Turin/model conformance run. Issue #8 remains open.

Runtime 0.3.0-alpha.3 implementation update

PR #16 has been merged and v0.3.0-alpha.3 published.

This slice adds the first generic host/resource foundation behind the multi-backend router:

quantum.runtime/host-profile/v1alpha1 Linux discovery for CPU topology/features, NUMA, RAM/huge pages, block/NVMe devices, and visible accelerator/VRAM metadata
explicit bounded host calibration with physical-core and logical/SMT worker candidates
quantum.runtime/placement-plan/v1alpha1
CPU/RAM candidate first; visible GPU does not automatically win
hybrid RAM/VRAM candidate only after CPU-only capacity fails and acceleration is explicitly allowed
explicit NVMe cold tier only; hot execution state is never silently spilled to disk
/v1/host, /v1/host/calibrate, /v1/placement

The placement result is intentionally still a pre-activation capacity plan, not a throughput claim. Next slices still need real backend/model prefill+decode conformance, GGUF/KV size derivation, NUMA/thread affinity, persistent calibration, and applying a validated placement to backend activation.

Refs #11, #12, #13.

Runtime 0.3.0-alpha.4 is now merged/released via PR #17. This slice composes the backend/model contract from #8 with application-profile admission and guest-aware resource limits:

canonical model identity remains independent of the application profile
ember-production requires generic architecture_class=moe; it does not string-match or hardcode Gemma as the Runtime default
Gemma 4 26B A4B remains the current Gemma-family MoE reference candidate while unrelated MoE manifests can satisfy the same profile contract
admission fails closed for unsupported architecture/capacity and reports missing provider evidence such as ECC as unknown rather than guessing
KVM/process affinity and cgroup cpusets are exposed separately from host CPU identity, so an EPYC 9645 model name cannot imply 96 usable guest cores
/v1/benchmark-plan now defines repeatable worker-count experiments without inventing throughput

The next #8/#11 integration step is actual backend/model/quantization/context performance evidence and GGUF-derived resource sizing, then feeding those measured results back into routing/placement. No real-model throughput was invented in alpha.4.

Navigation Menu

Repository navigation

[0.3] Build universal multi-backend model capability fusion layer #8

Goal

Universal model principle

Initial upstream capability sources

Required Runtime architecture

Generic model architecture contract

Gemma 4 reference profile

Backend capability contract

Agent compatibility boundary

Upstream version policy

Universal conformance suite

Initial backend priority

1. llama.cpp/ggml native adapter

2. existing Ollama adapter

3. MLX adapter

4. vLLM adapter

Stability and regression policy

CoreOS TCI relationship

Acceptance criteria

Non-goals

0.3 implementation update: backend contract foundation released

Next 0.3 slice

Runtime 0.3.0-alpha.3 implementation update

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[0.3] Build universal multi-backend model capability fusion layer #8

Description

Goal

Universal model principle

Initial upstream capability sources

Required Runtime architecture

Generic model architecture contract

Gemma 4 reference profile

Backend capability contract

Agent compatibility boundary

Upstream version policy

Universal conformance suite

Initial backend priority

1. llama.cpp/ggml native adapter

2. existing Ollama adapter

3. MLX adapter

4. vLLM adapter

Stability and regression policy

CoreOS TCI relationship

Acceptance criteria

Non-goals

Activity

pschildgen87-code commented on Sep 4, 2026

pschildgen87-code commented on Sep 4, 2026

pschildgen87-code commented on Sep 4, 2026

pschildgen87-code commented on Sep 4, 2026

pschildgen87-code commented on Sep 4, 2026

pschildgen87-code commented on Sep 4, 2026

pschildgen87-code commented on Sep 4, 2026

pschildgen87-code commented on Sep 4, 2026

pschildgen87-code commented on Sep 4, 2026

pschildgen87-code commented on Sep 4, 2026

pschildgen87-code commented on Sep 4, 2026

pschildgen87-code commented on Sep 4, 2026

pschildgen87-code commented on Sep 5, 2026

pschildgen87-code commented on Sep 5, 2026

0.3 implementation update: backend contract foundation released

Next 0.3 slice

pschildgen87-code commented on Sep 5, 2026

pschildgen87-code commented on Sep 5, 2026

Runtime 0.3.0-alpha.3 implementation update

pschildgen87-code commented on Sep 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions