Skip to content

Bump llama.cpp to a691b2345, release v0.8.68 - #124

Merged
nyo16 merged 1 commit into
masterfrom
bump-llama-a691b2345
Nov 8, 2026
Merged

nyo16 merged 1 commit into
masterfrom
bump-llama-a691b2345

Conversation

@nyo16

@nyo16 nyo16 commented Nov 8, 2026

Copy link
Copy Markdown
Owner

llama.cpp 785322d17 → a691b2345 (b12205), 172 commits. Upstream versions: llama.cpp 0.9.0, ggml 0.29.0.

API: the bump needed no NIF change. llama.h adds moe_cache_cpu_share (default 0; the binding does not set the MoE cache). common.h drops common_get_path_from_env and changes common_prompt_checkpoint's buffers; the NIF uses neither. ggml.h drops ggml_rope_yarn_corr_dims (not called). ggml-backend.h and llama-ext.h only add. llama_model_default_params is unchanged. RPC protocol stays at 8.0.1 and GGML_OP_COUNT at 102.

Decision port: llama-vocab.cpp and server-decision.* are untouched; "Ported at" is now a691b2345. scripts/decision_compare.exs against llama-server at the same commit (Metal, -np 1): ALL MATCH on tinylaya, tinyopenjev, d1-2B, d1-3B, d1-omni-600M, vllm-decision2-Kai, vllm-d3-edge, h2o-lightning-4b and Clef Flash.

Binding changes:

  • MTP.init/2 no longer refuses a DFlash/DSpark drafter that is missing one of the target's devices. Upstream #31635 is the same change as the draft-ctx-other-devices fork branch. On two Sparks, a drafter on CUDA0 alone gives the same text and rate as one loaded on both devices.

  • MTP.init/2 warns when a DFlash target's last device is an RPC device. Since #31506 the drafter reads its features from the target's output device. Qwen3.8-27B NVFP4 split over two Sparks:

    target :devices plain DFlash 2
    ["CUDA0", "RPC0"], v0.8.67 13.75 28.1
    ["CUDA0", "RPC0"], v0.8.68 13.7 7.9
    ["RPC0", "CUDA0"], v0.8.68 14.3 32.5–33.0

    The docs and scripts/spark/e2e.exs now list the local device last.

  • Fix: an LLAMA_RPC=1 build did not compile on macOS since v0.8.67 (SOCK_CLOEXEC).

Known issue (documented, not coded around): split_mode: :tensor with an RPC device segfaults the VM on the first decode. Upstream's recursive RPC graph serialization overflows the default 320 KB stack of a dirty CPU scheduler. +sssdcpu 256 fixes it; :layer is unaffected. Upstream llama-bench (gpt-oss-120b, two Sparks): -sm tensor 15.2 tok/s against -sm layer 50.4, so the advice to use :layer across hosts stands.

Notable upstream changes:

  • Unified-KV flash attention over per-sequence rows (#29510). This is the Server default and Decision prompt sharing. LLAMA_KV_ROWS_DISABLE turns it off.
  • Quantized-KV on-device state copy fix (#30541).
  • KV shift fix (#31683).
  • CPU F16 matmuls take F32 activations (#31854).
  • Gemma 4 E2B/E4B prefill pruning (#31356).
  • NVFP4 MMQ fusions (#31296).
  • NCCL for multi-node tensor parallelism (#28967).
  • New architectures: Kimi-K3.1-Pro, Ripple, xing4_0, Metis-1.6, Clover.

Verified:

  • M1 Max (Metal):
    • default suite on a default and an LLAMA_RPC=1 build;
    • :smoke, :embeddings (Qwen3-Embedding, embeddinggemma-2), :slow, :mtp, :tools, :decision, :decision_clef;
    • :mtp_sidecar (Qwen3.8-27B) and vm_exit_test.exs;
    • format, credo --strict and dialyzer clean;
    • the Hex source build clones a691b2345 and compiles.
  • Two DGX Sparks (CUDA, RPC build, verify-build-flags.sh clean):
    • default suite and the same tags, plus :dflash and bge-small;
    • vm_exit_test.exs, :rpc_live and e2e.exs two-node (12/12);
    • bench/spark_spec.exs unchanged: plain 14.21, MTP 26.55, DFlash 44.24.
  • Mac + one Spark over the LAN (layer split): runs, about 17–26 tok/s on Qwen3.5-9B against 42 on the Mac alone.
  • :mtp_cancel is still the known race (2/4 abort, record updated).

Fork branches: rpc-add-server-no-abort and rpc-client-spin-wait still apply cleanly. cuda-mmq-src1-reuse conflicts with #31296 and #31047.

Upstream defects #1 and #3 are unchanged, and #2 has been re-measured; the re-check is in docs/release-guide.md.

172 commits from 785322d17 (b12032+1) to b12205. No NIF change was needed
for the bump; decision.cpp is "Ported at" a691b2345 with no hunk to port.
decision_compare.exs: ALL MATCH on 9 models against llama-server.

- MTP.init/2 drops the DFlash/DSpark device check: upstream #31635 gives the
  draft context the target's backends, verified on two Sparks.
- MTP.init/2 warns when a DFlash target's output device is an RPC device:
  since #31506 the drafter's features live there (7.9 tok/s with
  ["CUDA0", "RPC0"], 32.5 with ["RPC0", "CUDA0"]).
- Fix the macOS LLAMA_RPC=1 build (SOCK_CLOEXEC, since v0.8.67).
- Document the :tensor-over-RPC dirty-scheduler stack overflow (+sssdcpu 256)
  and the re-measured -sm tensor numbers.
@nyo16
nyo16 merged commit ad24f90 into master Nov 8, 2026
9 checks passed
@nyo16
nyo16 deleted the bump-llama-a691b2345 branch November 8, 2026 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant