[0.3] Make CPU-first execution and tiered memory placement the Runtime default #11

Goal

Make CPU-only execution a first-class performance target and universal compatibility baseline for Quantum Runtime, while treating GPUs/accelerators as optional performance tiers rather than hard requirements.

The default architectural principle is CPU-first, but not as a naive reactive migration strategy. Quantum Runtime should evaluate a CPU/RAM execution plan first, benchmark or predict whether it can satisfy the requested model/workload policy, and choose any GPU/hybrid placement before model activation when acceleration is needed.

A request must remain runnable on supported CPU-only systems when the selected backend/model combination supports CPU execution.

Core principle

CPU is mandatory baseline / control plane
GPU is optional accelerator
NVMe/filesystem is an explicit cold/offload tier

Do not begin generation on CPU and then migrate arbitrary live tensors to GPU only after latency degrades. Mid-generation migration is expensive, backend-specific and often unsafe. Placement should normally be decided before load or at explicit backend-supported paging boundaries.

CPU-only path

For systems with no GPU:

CPU compute
   |
RAM hot working set
   |
file-backed model mapping / page cache
   |
optional NVMe-backed cold weights / prefix-cache tier

There is no VRAM tier on these hosts.

CPU-only behavior should prefer explicit Runtime memory policy over relying on uncontrolled OS swap. A model that exceeds practical RAM bandwidth/capacity should degrade predictably, use bounded mmap/file-backed paging where supported, or report that the requested performance profile cannot be met. It must not thrash the host blindly.

GPU / hybrid path

When an accelerator exists, the planner may choose a hybrid placement such as:

CPU scheduler/control + host RAM
          |
GPU VRAM hot compute/KV/model tensors
          |
pinned host RAM overflow/offload tier
          |
NVMe/filesystem secondary cold tier

Important rules:

• CPU/RAM remains the gateway and authoritative host tier.
• VRAM should contain the hottest tensors/KV blocks that materially benefit latency/throughput.
• Overflow from VRAM should prefer pinned/system RAM before filesystem storage.
• Filesystem/NVMe should be a secondary/cold tier for data that can tolerate higher latency, such as cold model pages or cache blocks where the backend supports it.
• Do not assume direct GPU-to-disk transfer unless a reviewed backend explicitly supports and tests it.
• Active intermediate activations must not be casually spilled to disk.
• Hybrid placement must be benchmarked; a partial GPU offload that performs worse than CPU-only should be rejected by policy.

Separate memory classes

The planner must track these separately because they have different locality/performance requirements:

• model weights
• MoE expert weights
• KV cache
• prefix/prompt cache
• multimodal/projector weights
• temporary activations/workspaces
• tokenizer/templates/metadata

Do not expose a single ambiguous memory_offload=true flag.

CPU optimization targets

Quantum Runtime should detect and exploit host CPU capabilities through the selected backend, including where available:

x86/x86-64

• SSE4/AVX/AVX2
• AVX-VNNI
• AVX-512 variants/VNNI/BF16
• vendor-optimized matrix kernels such as oneMKL/OpenBLAS/BLIS where beneficial
• ZenDNN-style AMD EPYC optimization where supported

ARM

• NEON/dot-product
• int8mm
• SVE/SME
• KleidiAI-style optimized kernels where supported

Topology

• physical-core vs SMT awareness
• NUMA topology and memory locality
• thread pinning/affinity where useful
• memory-channel saturation awareness
• separate prefill and decode tuning
• batch/ubatch calibration

A backend may expose further architecture-specific kernels without changing the public Runtime API.

Auto-tuning / host calibration

Add a machine-readable host performance profile built from hardware discovery plus a short bounded calibration benchmark.

Example data:

cpu.arch
cpu.features[]
cpu.physical_cores
cpu.logical_cores
numa.nodes
memory.total
memory.bandwidth_class
storage.type
storage.measured_read_bandwidth
accelerators[]

Calibration should test representative operations rather than trusting hardware names alone.

For each backend/model profile, the Runtime may retain tested placement data such as:

cpu_only.tokens_per_second
cpu_only.prefill_tokens_per_second
hybrid.tokens_per_second
hybrid.vram_bytes
hybrid.host_ram_bytes
hybrid.offload_bytes

The scheduler can then choose the latest-known-good placement instead of guessing every startup.

Placement policy

A normalized placement decision should consider:

• model artifact/quantization size
• architecture and active parameters
• context length
• expected KV-cache size
• requested concurrency
• requested latency/throughput class
• CPU/RAM benchmark data
• GPU/NPU availability and free memory
• backend capability support
• memory/offload transfer cost
• current host pressure
• operator policy

Recommended policy shape:

1. Construct CPU-only candidate.
2. If it fits and satisfies policy, use CPU-only.
3. If not and a compatible accelerator exists, construct hybrid/GPU candidate.
4. Prefer the measured faster candidate, not automatically the GPU candidate.
5. If accelerator memory is insufficient, use reviewed VRAM -> host RAM offload.
6. If supported and policy permits, use host RAM -> NVMe secondary cold/cache tier.
7. If no candidate can meet the requested profile, return an explicit capacity/performance decision instead of host thrashing.

MoE-specific placement

MoE models are a primary use case for CPU-first/hybrid execution because total parameters may be much larger than active parameters.

Represent and benchmark:

• shared experts
• routed expert count
• active top-k experts
• CPU-resident experts
• accelerator-resident experts
• expert prefetch where supported
• model-family/backend-specific expert placement

Do not assume partial MoE GPU offload is always faster. The Runtime conformance/performance ledger must be able to block a known-slow split on a hardware/backend combination.

KV and cache tiering

Treat cache tiering separately from model-weight offload.

Where supported, use a hierarchy such as:

accelerator KV cache
      <-> pinned host RAM primary offload
      <-> filesystem/NVMe secondary tier

Secondary tiers must not be advertised when the backend cannot safely promote/demote blocks.

CPU-only systems may use RAM plus an optional filesystem-backed prefix/cache tier, but active decode state should not be pushed to disk by default if it makes generation unusably slow.

Disk policy

NVMe offload is an explicit Runtime feature, not generic swap.

Requirements:

• dedicated configurable directory/device
• free-space reservation and hard limits
• bounded cache size
• cleanup/recovery journal
• checksum/integrity metadata where persistence matters
• no user secrets in filenames/paths
• optional encryption policy for persistent cache tiers
• SSD-wear awareness and metrics
• disable or reduce disk tier on slow rotational storage by default

Backend fusion direction

The Runtime may combine ideas/capabilities from mature backends while keeping a stable Quantum contract.

Examples to evaluate through the latest-known-good policy:

• llama.cpp/ggml for portable CPU execution, mmap, quantized CPU kernels, partial GPU/MoE placement and NUMA controls
• vLLM for host-memory weight offload and hierarchical KV-cache tiering
• OpenVINO/oneDNN/oneMKL paths for Intel CPU-specific acceleration where they outperform the generic path
• ZenDNN/vendor paths for supported AMD server CPUs
• MLX for Apple unified-memory systems
• other reviewed backends when they provide a proven CPU/offload capability

Do not vendor unrelated full runtimes into one monolith. Capability adoption happens through adapters, tested components or clearly attributed compatible code.

Latest-known-good performance policy

CPU performance regressions are release blockers even if a newer upstream backend has more features.

The tested-upstream ledger from Runtime 0.3 must include performance guardrails for CPU-only and hybrid profiles.

A backend update should not replace the production pin if representative CPU-only throughput or correctness regresses beyond an explicit tolerance.

Metrics

Expose enough local metrics to explain placement without leaking prompts:

• selected backend
• selected placement profile
• CPU threads used
• accelerator layers/tensors/expert placement
• model RAM/VRAM footprint
• KV/cache footprint by tier
• page/cache hit metrics where available
• prefill throughput
• decode throughput
• transfer/offload bandwidth and stalls
• filesystem tier reads/writes
• fallback reason

Acceptance criteria

• CPU-only is a supported first-class execution profile in the backend contract
• placement planner evaluates a CPU-only candidate before optional acceleration
• no GPU is required for Runtime startup or generic model management
• accelerator placement is chosen before model activation rather than naive mid-stream migration
• Runtime can prefer CPU-only when hybrid/GPU is measured slower
• model weights, MoE experts, KV cache, prefix cache and activations have separate placement classes
• CPU feature/topology detection and bounded calibration profile exist
• NUMA/thread/batch tuning can be expressed without model-family API hacks
• hybrid path supports reviewed VRAM -> host RAM overflow where the backend supports it
• filesystem/NVMe is an explicit secondary/cold tier, not uncontrolled OS swap
• CPU-only mmap/file-backed model behavior is documented and bounded
• disk-tier size, free-space and cleanup policy are enforced
• CPU/MoE performance regressions can block an upstream update
• placement and fallback decisions are observable through metrics/status
• normal CI remains hardware-independent with performance fixtures/mocks; optional hardware benchmarks are documented

Relationship to other Runtime milestones

This is a cross-cutting 0.3 priority and should shape the backend interface in #8 rather than being bolted on after GPU-centric design.

Agent compatibility (#9) and generative media (#10) should consume the same host resource/placement primitives later, but CPU-first text/model inference remains the primary optimization target.

Non-goals

• requiring a GPU
• automatically preferring GPU merely because one exists
• relying on OS swap as the memory scheduler
• blindly spilling active inference state to disk
• moving arbitrary live workloads between CPU/GPU without backend support
• hiding severe capacity/performance degradation from clients
• sacrificing CPU-only performance to track the newest upstream release