Performance of llama.cpp with CPU extremely slow with build b10429 and later

[phattja]

llama-b10428.log
llama-b10437.log

Summary

After updating to llama.cpp build b10429 (and later builds), pure CPU inference performance has become extremely slow compared to previous versions.

Example:

• Model: Nemotron-3.5-Lightning-30B-A3B
• Before b10429: 28–30 t/s
• After b10429: 2–3 t/s

Environment

• OS: Debian Linux 13
• CPU: Intel 8444H 64 Cores
• RAM: 384GB
• Build: b10429 and newer
• Previous good build: b10428 or earlier
• Backend: CPU only

Environment Variable

LLAMA_ARG_BATCH_SIZE=512
LLAMA_ARG_CPU_MOE=enabled
LLAMA_ARG_CTX_SIZE=0
LLAMA_ARG_EMBEDDINGS=disabled
LLAMA_ARG_FIT=off
LLAMA_ARG_FLASH_ATTN=on
LLAMA_ARG_FREQUENCY_PENALTY=0.0
LLAMA_ARG_JINJA=enabled
LLAMA_ARG_LOAD_MODE=none
LLAMA_ARG_MIN_P=0
LLAMA_ARG_MODELS_MAX=2
LLAMA_ARG_N_GPU_LAYERS=0
LLAMA_ARG_NO_WARMUP=1
LLAMA_ARG_NUMA=numactl
LLAMA_ARG_PARALLEL=1
LLAMA_ARG_PRESENCE_PENALTY=0.0
LLAMA_ARG_REASONING=on
LLAMA_ARG_REASONING_PRESERVE=on
LLAMA_ARG_REPEAT_PENALTY=1.08
LLAMA_ARG_RERANKING=disabled
LLAMA_ARG_TEMPERATURE=0.85
LLAMA_ARG_THREADS=48
LLAMA_ARG_TOOLS=all
LLAMA_ARG_TOP_K=20
LLAMA_ARG_TOP_P=0.95

Command used

./llama-b10437/llama-server -m ./models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-UD-Q4_K_XL.gguf

Is there any specific additional setting or flag required for builds b10429 and later?

[hoangperry]

Your logs point at a specific suspect, and it's not your configuration. I diffed them: identical system_info (48/64 threads, AVX512+AMX all present, REPACK on), identical tensor placement — yet eval drops from ~26–27 t/s to ~2.3–2.5 t/s and prompt eval from ~191 t/s to ~80 t/s. Same setup, ~10x slower decode: that's a code-path change, and your bisect boundary makes it precise.

Build b10429 contains exactly one commit: 77918caf3 — "server: allow accessing /metrics and /slots during llama_decode() (#27041)". That change lets HTTP endpoint handling interleave with decode instead of waiting for it. Two details from your own log make you unusually exposed to it: the server spins up 63 HTTP threads on your 64-core box, and you run with tools/MCP enabled — if anything in your stack polls /metrics, /slots, or /health during generation (an MCP client, a dashboard, Prometheus scraping), each poll now contends with the 48 decode threads instead of queueing behind them.

Suggestions:

1. Confirm the pin: test b10429 itself — if it's slow and b10428 is fast, the attribution is airtight (your logs currently show b10428 vs b10437).
2. Test the mechanism: run b10437 with any pollers disabled (no --metrics, nothing hitting /slots, MCP/tools off) and see whether decode speed returns; also try shrinking the HTTP thread pool.
3. File it as an issue referencing PR server: allow accessing /metrics and /slots during llama_decode() #27041 with your two logs attached — a 10x CPU decode regression tied to a one-commit build range, with logs, is exactly the report the maintainers act on quickly.

Pinning b10428 is the right workaround meanwhile.

[sylvesterkaczmarek]

I don't see any new required flag after b10429. Since b10428 gives ~30 t/s and b10429 drops to ~3 t/s with the same setup, this looks much more like a regression. I'd bisect between those two builds and report the first bad commit.

[sauravsingla]

Hi everyone,

I’m working on an open-source CPU inference project called MemVanta and would value independent performance-engineering feedback from this community.

We have opened a specific optimization challenge:

Can we increase LLM inference throughput while preserving the existing low-memory constraint?

I’m particularly interested in experiments around:

• Q4/Q8 kernels
• SIMD/vectorization
• memory bandwidth and cache behavior
• threading / CPU affinity
• quantized matrix multiplication
• prefetching and memory layout

The important constraint is that throughput should not be improved simply by materially increasing peak memory.

We have published a reproducibility protocol requiring same-machine baseline vs optimized measurements, peak RSS, tokens/sec, hardware configuration and exact commands.

Challenge:
sauravsingla/MemVanta#11

Positive and negative results are equally valuable. If anyone sees an obvious bottleneck, optimization direction, or flaw in the benchmarking methodology, I would particularly appreciate the technical feedback.

Thanks!