docs/FEATURES.md compares this engine against llama.cpp on eleven memory axes and
carries no row for hybrid CPU/GPU expert placement. llama.cpp's -cmoe and -ncmoe
keep the routed-expert tensors in system RAM and run the expert MLPs on the CPU while
attention, dense projections, router and norms stay on the GPU. This engine has no
equivalent. ENG-HYBRID-PLACEMENT owns the capability, its spec is committed at
.agents/specs/hybrid-placement.md, and the row
is READY with no implementation.
The nearest existing row is "Routed-expert weight streaming from disk", which records
llama.cpp as mmap only. That cell is correct for the row it sits in, and it is why
the gap is invisible: streaming moves weights toward the compute, placement moves
compute toward the weights, and the table only has a row for the first one. A reader
comparing the two engines on a discrete consumer GPU cannot find the answer to the
question they arrived with.
The gap is not academic. #149 is the
highest-upvoted request from the r/LocalLLaMA launch thread, and public reports of
125B-class MoE checkpoints served on a single 24 GB card run on exactly this
mechanism.
Scope of this issue is the projection only: one row in the "KV cache and memory"
table, naming the state honestly and pointing at the owning row. It adds no product
code and it does not move ENG-HYBRID-PLACEMENT out of READY.
Owning row: ENG-HYBRID-PLACEMENT.
docs/FEATURES.mdcompares this engine against llama.cpp on eleven memory axes andcarries no row for hybrid CPU/GPU expert placement. llama.cpp's
-cmoeand-ncmoekeep the routed-expert tensors in system RAM and run the expert MLPs on the CPU while
attention, dense projections, router and norms stay on the GPU. This engine has no
equivalent.
ENG-HYBRID-PLACEMENTowns the capability, its spec is committed at.agents/specs/hybrid-placement.md, and the rowis
READYwith no implementation.The nearest existing row is "Routed-expert weight streaming from disk", which records
llama.cpp as
mmap only. That cell is correct for the row it sits in, and it is whythe gap is invisible: streaming moves weights toward the compute, placement moves
compute toward the weights, and the table only has a row for the first one. A reader
comparing the two engines on a discrete consumer GPU cannot find the answer to the
question they arrived with.
The gap is not academic. #149 is the
highest-upvoted request from the r/LocalLLaMA launch thread, and public reports of
125B-class MoE checkpoints served on a single 24 GB card run on exactly this
mechanism.
Scope of this issue is the projection only: one row in the "KV cache and memory"
table, naming the state honestly and pointing at the owning row. It adds no product
code and it does not move
ENG-HYBRID-PLACEMENTout ofREADY.Owning row:
ENG-HYBRID-PLACEMENT.