diff --git a/CHANGELOG.md b/CHANGELOG.md index 3789ef2..309afd3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,79 @@ # Changelog +## v0.8.66 + +llama.cpp bumped to `785322d17` +([`b12032`](https://github.com/ggml-org/llama.cpp/releases/tag/b12032)+1), +31 commits from `418f905a4` (b11986+16). No change was needed in +`llama_nif.cpp`. Of the headers the NIF includes only `common/common.h` +changed, removing `fs_path_to_utf8` (#31423), which the NIF does not call; +`llama.h`, `ggml.h`, `ggml-backend.h`, `ggml-rpc.h`, `chat.h`, `sampling.h`, +`speculative.h` and `src/llama-ext.h` are untouched, as are +`src/llama-vocab.cpp` (the special-token partition `input_tokenize.cpp` +mirrors) and `src/llama-context.cpp`. `llama_model_default_params` / +`llama_context_default_params` are unchanged. RPC protocol stays at 8.0.1 +(`GGML_OP_COUNT` 102). Upstream versions: llama.cpp 0.8.0, ggml 0.28.0. + +### Changed + +- `decision.cpp` ports #30710 and is now "Ported at" `785322d17`. Upstream caps + a shared decision prefix at the first read position of every question + (`server_decision_group_tasks`), so each question's outputs stay in its own + last batch when questions share option tokens (#30482). The port already + capped it there for embedding readouts; label readouts with a read position + are now capped too. +- Upstream, from the 31 commits: + - Recurrent memory refuses a partial rollback past the snapshots its last + ubatch wrote, instead of restoring a stale state (#31368). A ubatch longer + than `n_rs_seq + 1` tokens now only allows a full reset. That is the path + `memory_seq_rm` already reports and the Server already handles for hybrid + models. + - KV rotation constants stay on the GPU (#31073), and layers that reuse + another layer's KV cache rotate correctly (#31406). + - Gemma 3n / Gemma 4 create `token_embd` before the tied output (#31394). + - Reasoning-budget forcing ends on an end tag from the prompt (#31374). + - The qwen3-coder chat parser accepts a think block with + `reasoning_format: none` (#31380). + - LFM3 tensor split and an LFM short-conv reshape (#31431, #30831). + - Metal's quantize kernels compile with `fp math_mode(safe)`, which changes + rounding when writing a Q4_0/Q5_0/Q8_0 KV cache (#31120). + - OpenCL, SYCL, Hexagon and Windows build work. +- `test/mtp_model_test.exs` records `:mtp_cancel` at this commit: 3 of 4 runs + abort (SIGSEGV), the fourth fails with `unknown exception thrown within NIF`. + This is the same known race as before. + +Upstream defects (`docs/release-guide.md`): #1, #2 and #3 are unchanged by +source diff. `ggml-cpu/CMakeLists.txt`, `ggml-rpc` and `ggml-cuda.cu` were +not touched. #2's tp=2 re-measurement on Spark is still owed. + +Verified on macOS, M1 Max, at `785322d17`, Metal with residency sets on, every +run exiting 0 unless noted: + +- default **436 passed, 197 excluded**; +- `--include smoke --include embeddings --include slow --include mtp` + (Qwen3.5-0.8B-UD-Q4_K_XL, Qwen3-Embedding-0.6B-f16, + Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL) **588 passed**; +- CI's set (SmolLM2-135M, bge-small, tinylaya, tinyopenjev, Qwen3.5-0.8B) + **604 passed**; +- `--include embeddings` on embeddinggemma-2-Q8_0 **447 passed**; +- `--include decision_clef` on Clef Flash Q8_0 **440 passed**; +- `--include tools` on Llama-3.2-1B-Instruct 446 of 447 (the known round + trip); +- `--include mtp_sidecar` on the Qwen 3.8 27B pair **442 passed, 6 + skipped**; +- CPU backend: `--include tools` on Qwen3.5-0.8B **447 passed**, + `test/vm_exit_test.exs` **3 passed**. + +`scripts/decision_compare.exs` against upstream `llama-server` built from the +same commit with Metal (`-np 1`): tinylaya, tinyopenjev, d1-2B, d1-3B, +d1-omni-600M, vllm-decision2-Kai-0.6B, vllm-d3-edge, Clef Flash (Q8_0) and +h2o-lightning-4b (Q4_K_M) **ALL MATCH**, with identical token counts. Every +checked answer agrees within 1e-3, and laya, clef and h2o-lightning agree +exactly. The `sharing` column is unchanged (d1-3B 6.1e-4, Kai 6.4e-4, +nothing else). Format, credo `--strict` and dialyzer (0 errors) clean; the Hex +source build clones `785322d17` and compiles. Not run this time: `rpc_live` +(no `ggml-rpc` change) and granite-switch. + ## v0.8.65 Fixes the Metal abort at VM exit with a model still referenced (#116). diff --git a/Makefile b/Makefile index 070c4a6..a4a93cd 100644 --- a/Makefile +++ b/Makefile @@ -36,7 +36,7 @@ endif # Pinned llama.cpp commit, used when vendor/llama.cpp has to be cloned. MUST # match the vendor/llama.cpp submodule; bump both together, see # docs/release-guide.md. Override to build the NIF against another revision. -LLAMA_COMMIT ?= 418f905a4efe910d2a75636bbe65d48eca38790a +LLAMA_COMMIT ?= 785322d17f2d51e5185cf996a1f4e46d87640841 # The commit actually on disk. A submodule can be bumped without LLAMA_COMMIT # following it, and the build has to key off what is really there. diff --git a/c_src/llama_cpp_ex/decision.cpp b/c_src/llama_cpp_ex/decision.cpp index 28f7e8c..8642d0f 100644 --- a/c_src/llama_cpp_ex/decision.cpp +++ b/c_src/llama_cpp_ex/decision.cpp @@ -2,7 +2,7 @@ // of tools/server/server-context.cpp (send_decision, the batch layout rules and // the /v1/systemone handler). See decision.h for scope. // -// Ported at 418f905a4. Functions that are upstream's carry its name in a +// Ported at 785322d17. Functions that are upstream's carry its name in a // trailing comment so a bump can diff them one by one; everything below the // "evaluation" banner replaces server slots and is ours. @@ -2103,8 +2103,9 @@ struct Engine::Impl { } // Tokens every task starts with, capped so each task keeps at least one - // token of its own (server_decision_group_tasks) and so no read position - // falls inside the prefix, where it would not be an output. + // token of its own and stops at its first read position + // (server_decision_group_tasks, #30710), and so no output of the last batch + // falls inside the prefix. int32_t shared_prefix(const std::vector & tasks) const { if (!can_share_prompt() || tasks.size() < 2 || llama_n_seq_max(ctx) < 2) { return 0; @@ -2117,6 +2118,10 @@ struct Engine::Impl { n_common++; } n_shared = std::min({n_shared, n_common, task.tokens.size() - 1}); + // upstream casts pos_first() to size_t, so -1 (nothing read) caps nothing + if (task.pos_first() >= 0) { + n_shared = std::min(n_shared, (size_t) task.pos_first()); + } n_shared = std::min(n_shared, (size_t) first_batched(task)); } return (int32_t) n_shared; diff --git a/docs/release-guide.md b/docs/release-guide.md index 2f2a0e2..1fd6c5b 100644 --- a/docs/release-guide.md +++ b/docs/release-guide.md @@ -332,6 +332,12 @@ everything else. The `ggml-cuda.cu` diff is the MMQ MoE gate/up/GLU fusion (#31188), RDNA4_M (#31373) and the batched copy hooks (#30881), none of it near `ggml_backend_cuda_comm_init`. +Re-checked at `785322d17` (b12032+1), covering the 31 commits from +`418f905a4`: #1, #2 and #3 are as above. `ggml-cpu/CMakeLists.txt`, +`ggml-rpc`, `ggml-rpc.h`, `ggml.h` and `ggml-cuda.cu` were not touched, so the +RPC protocol and op ids are unchanged (#31424 removes `fs_path_to_utf8` from +`tools/rpc`, not from the backend). + | # | Upstream defect | Our workaround | Still needed? | |---|---|---|---| | 1 | `GGML_NATIVE=ON` makes ggml's `-mcpu=native` probe resolve to **base ARMv8-A** on Cortex-X925/A725 with GCC 13.3 — silently, with a soft CMake warning and exit 0. Costs every `sdot`/`smmla`/SVE kernel. | `LLAMA_CPU_ARM_ARCH` + `LLAMA_CUDA_ARCH` in the `Makefile`, which must be set together. See [DGX Spark](dgx-spark.md) and [Cross-Platform Builds](cross-platform-builds.md). | `scripts/spark/verify-build-flags.sh` on an aarch64 host. If a default build (no `LLAMA_CPU_ARM_ARCH`) now reports non-zero `sdot`/`smmla`, upstream fixed the probe. | diff --git a/mix.exs b/mix.exs index c213e24..f99f3d7 100644 --- a/mix.exs +++ b/mix.exs @@ -133,7 +133,7 @@ end defmodule LlamaCppEx.MixProject do use Mix.Project - @version "0.8.65" + @version "0.8.66" @source_url "https://github.com/nyo16/llama_cpp_ex" def project do diff --git a/test/mtp_model_test.exs b/test/mtp_model_test.exs index 547f839..8aeee78 100644 --- a/test/mtp_model_test.exs +++ b/test/mtp_model_test.exs @@ -169,7 +169,9 @@ defmodule LlamaCppEx.MTPCancelTest do # machine as above, so the abort is not new in 23b0202a1; nothing between # v0.8.49 and v0.8.56 ran this tag. At a0e6dc738 (b11630+9) four runs aborted # 2/4 (exit 134); the other two failed with the same `unknown exception` and - # `"prompt decode failed: code=-2"` as above. + # `"prompt decode failed: code=-2"` as above. At 785322d17 (b12032+1) four + # runs aborted 3/4 (SIGSEGV, exit 139) and the fourth failed with the same + # `unknown exception`. # # It therefore still lives in its own module carrying only `:mtp_cancel`, and # deliberately *not* `:mtp`: `--include` beats `--exclude` in ExUnit, so a diff --git a/vendor/llama.cpp b/vendor/llama.cpp index 418f905..785322d 160000 --- a/vendor/llama.cpp +++ b/vendor/llama.cpp @@ -1 +1 @@ -Subproject commit 418f905a4efe910d2a75636bbe65d48eca38790a +Subproject commit 785322d17f2d51e5185cf996a1f4e46d87640841