Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,79 @@
# Changelog

## v0.8.66

llama.cpp bumped to `785322d17`
([`b12032`](https://github.com/ggml-org/llama.cpp/releases/tag/b12032)+1),
31 commits from `418f905a4` (b11986+16). No change was needed in
`llama_nif.cpp`. Of the headers the NIF includes only `common/common.h`
changed, removing `fs_path_to_utf8` (#31423), which the NIF does not call;
`llama.h`, `ggml.h`, `ggml-backend.h`, `ggml-rpc.h`, `chat.h`, `sampling.h`,
`speculative.h` and `src/llama-ext.h` are untouched, as are
`src/llama-vocab.cpp` (the special-token partition `input_tokenize.cpp`
mirrors) and `src/llama-context.cpp`. `llama_model_default_params` /
`llama_context_default_params` are unchanged. RPC protocol stays at 8.0.1
(`GGML_OP_COUNT` 102). Upstream versions: llama.cpp 0.8.0, ggml 0.28.0.

### Changed

- `decision.cpp` ports #30710 and is now "Ported at" `785322d17`. Upstream caps
a shared decision prefix at the first read position of every question
(`server_decision_group_tasks`), so each question's outputs stay in its own
last batch when questions share option tokens (#30482). The port already
capped it there for embedding readouts; label readouts with a read position
are now capped too.
- Upstream, from the 31 commits:
- Recurrent memory refuses a partial rollback past the snapshots its last
ubatch wrote, instead of restoring a stale state (#31368). A ubatch longer
than `n_rs_seq + 1` tokens now only allows a full reset. That is the path
`memory_seq_rm` already reports and the Server already handles for hybrid
models.
- KV rotation constants stay on the GPU (#31073), and layers that reuse
another layer's KV cache rotate correctly (#31406).
- Gemma 3n / Gemma 4 create `token_embd` before the tied output (#31394).
- Reasoning-budget forcing ends on an end tag from the prompt (#31374).
- The qwen3-coder chat parser accepts a think block with
`reasoning_format: none` (#31380).
- LFM3 tensor split and an LFM short-conv reshape (#31431, #30831).
- Metal's quantize kernels compile with `fp math_mode(safe)`, which changes
rounding when writing a Q4_0/Q5_0/Q8_0 KV cache (#31120).
- OpenCL, SYCL, Hexagon and Windows build work.
- `test/mtp_model_test.exs` records `:mtp_cancel` at this commit: 3 of 4 runs
abort (SIGSEGV), the fourth fails with `unknown exception thrown within NIF`.
This is the same known race as before.

Upstream defects (`docs/release-guide.md`): #1, #2 and #3 are unchanged by
source diff. `ggml-cpu/CMakeLists.txt`, `ggml-rpc` and `ggml-cuda.cu` were
not touched. #2's tp=2 re-measurement on Spark is still owed.

Verified on macOS, M1 Max, at `785322d17`, Metal with residency sets on, every
run exiting 0 unless noted:

- default **436 passed, 197 excluded**;
- `--include smoke --include embeddings --include slow --include mtp`
(Qwen3.5-0.8B-UD-Q4_K_XL, Qwen3-Embedding-0.6B-f16,
Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL) **588 passed**;
- CI's set (SmolLM2-135M, bge-small, tinylaya, tinyopenjev, Qwen3.5-0.8B)
**604 passed**;
- `--include embeddings` on embeddinggemma-2-Q8_0 **447 passed**;
- `--include decision_clef` on Clef Flash Q8_0 **440 passed**;
- `--include tools` on Llama-3.2-1B-Instruct 446 of 447 (the known round
trip);
- `--include mtp_sidecar` on the Qwen 3.8 27B pair **442 passed, 6
skipped**;
- CPU backend: `--include tools` on Qwen3.5-0.8B **447 passed**,
`test/vm_exit_test.exs` **3 passed**.

`scripts/decision_compare.exs` against upstream `llama-server` built from the
same commit with Metal (`-np 1`): tinylaya, tinyopenjev, d1-2B, d1-3B,
d1-omni-600M, vllm-decision2-Kai-0.6B, vllm-d3-edge, Clef Flash (Q8_0) and
h2o-lightning-4b (Q4_K_M) **ALL MATCH**, with identical token counts. Every
checked answer agrees within 1e-3, and laya, clef and h2o-lightning agree
exactly. The `sharing` column is unchanged (d1-3B 6.1e-4, Kai 6.4e-4,
nothing else). Format, credo `--strict` and dialyzer (0 errors) clean; the Hex
source build clones `785322d17` and compiles. Not run this time: `rpc_live`
(no `ggml-rpc` change) and granite-switch.

## v0.8.65

Fixes the Metal abort at VM exit with a model still referenced (#116).
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ endif
# Pinned llama.cpp commit, used when vendor/llama.cpp has to be cloned. MUST
# match the vendor/llama.cpp submodule; bump both together, see
# docs/release-guide.md. Override to build the NIF against another revision.
LLAMA_COMMIT ?= 418f905a4efe910d2a75636bbe65d48eca38790a
LLAMA_COMMIT ?= 785322d17f2d51e5185cf996a1f4e46d87640841

# The commit actually on disk. A submodule can be bumped without LLAMA_COMMIT
# following it, and the build has to key off what is really there.
Expand Down
11 changes: 8 additions & 3 deletions c_src/llama_cpp_ex/decision.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
// of tools/server/server-context.cpp (send_decision, the batch layout rules and
// the /v1/systemone handler). See decision.h for scope.
//
// Ported at 418f905a4. Functions that are upstream's carry its name in a
// Ported at 785322d17. Functions that are upstream's carry its name in a
// trailing comment so a bump can diff them one by one; everything below the
// "evaluation" banner replaces server slots and is ours.

Expand Down Expand Up @@ -2103,8 +2103,9 @@ struct Engine::Impl {
}

// Tokens every task starts with, capped so each task keeps at least one
// token of its own (server_decision_group_tasks) and so no read position
// falls inside the prefix, where it would not be an output.
// token of its own and stops at its first read position
// (server_decision_group_tasks, #30710), and so no output of the last batch
// falls inside the prefix.
int32_t shared_prefix(const std::vector<decision_task> & tasks) const {
if (!can_share_prompt() || tasks.size() < 2 || llama_n_seq_max(ctx) < 2) {
return 0;
Expand All @@ -2117,6 +2118,10 @@ struct Engine::Impl {
n_common++;
}
n_shared = std::min({n_shared, n_common, task.tokens.size() - 1});
// upstream casts pos_first() to size_t, so -1 (nothing read) caps nothing
if (task.pos_first() >= 0) {
n_shared = std::min(n_shared, (size_t) task.pos_first());
}
n_shared = std::min(n_shared, (size_t) first_batched(task));
}
return (int32_t) n_shared;
Expand Down
6 changes: 6 additions & 0 deletions docs/release-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,12 @@ everything else. The `ggml-cuda.cu` diff is the MMQ MoE gate/up/GLU fusion
(#31188), RDNA4_M (#31373) and the batched copy hooks (#30881), none of it near
`ggml_backend_cuda_comm_init`.

Re-checked at `785322d17` (b12032+1), covering the 31 commits from
`418f905a4`: #1, #2 and #3 are as above. `ggml-cpu/CMakeLists.txt`,
`ggml-rpc`, `ggml-rpc.h`, `ggml.h` and `ggml-cuda.cu` were not touched, so the
RPC protocol and op ids are unchanged (#31424 removes `fs_path_to_utf8` from
`tools/rpc`, not from the backend).

| # | Upstream defect | Our workaround | Still needed? |
|---|---|---|---|
| 1 | `GGML_NATIVE=ON` makes ggml's `-mcpu=native` probe resolve to **base ARMv8-A** on Cortex-X925/A725 with GCC 13.3 — silently, with a soft CMake warning and exit 0. Costs every `sdot`/`smmla`/SVE kernel. | `LLAMA_CPU_ARM_ARCH` + `LLAMA_CUDA_ARCH` in the `Makefile`, which must be set together. See [DGX Spark](dgx-spark.md) and [Cross-Platform Builds](cross-platform-builds.md). | `scripts/spark/verify-build-flags.sh` on an aarch64 host. If a default build (no `LLAMA_CPU_ARM_ARCH`) now reports non-zero `sdot`/`smmla`, upstream fixed the probe. |
Expand Down
2 changes: 1 addition & 1 deletion mix.exs
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ end
defmodule LlamaCppEx.MixProject do
use Mix.Project

@version "0.8.65"
@version "0.8.66"
@source_url "https://github.com/nyo16/llama_cpp_ex"

def project do
Expand Down
4 changes: 3 additions & 1 deletion test/mtp_model_test.exs
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,9 @@ defmodule LlamaCppEx.MTPCancelTest do
# machine as above, so the abort is not new in 23b0202a1; nothing between
# v0.8.49 and v0.8.56 ran this tag. At a0e6dc738 (b11630+9) four runs aborted
# 2/4 (exit 134); the other two failed with the same `unknown exception` and
# `"prompt decode failed: code=-2"` as above.
# `"prompt decode failed: code=-2"` as above. At 785322d17 (b12032+1) four
# runs aborted 3/4 (SIGSEGV, exit 139) and the fourth failed with the same
# `unknown exception`.
#
# It therefore still lives in its own module carrying only `:mtp_cancel`, and
# deliberately *not* `:mtp`: `--include` beats `--exclude` in ExUnit, so a
Expand Down
2 changes: 1 addition & 1 deletion vendor/llama.cpp
Submodule llama.cpp updated 111 files