Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 118 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,123 @@
# Changelog

## v0.8.68

llama.cpp bumped to `a691b2345`
([`b12205`](https://github.com/ggml-org/llama.cpp/releases/tag/b12205)),
172 commits from `785322d17` (b12032+1). Upstream versions: llama.cpp 0.9.0,
ggml 0.29.0. The bump needed no NIF change. Of the headers the NIF includes:
`llama.h` adds `moe_cache_cpu_share` to `llama_context_params` (default `0`,
which keeps the old behaviour; the binding does not set the MoE cache);
`common.h` drops `common_get_path_from_env` and the Windows `utf8_to_wstring`,
neither called, and changes `common_prompt_checkpoint`'s buffers, which the
NIF does not use; `ggml.h` drops `ggml_rope_yarn_corr_dims` (not called);
`ggml-backend.h` and `src/llama-ext.h` only add. `llama_model_default_params`
is unchanged. `src/llama-vocab.cpp` and `tools/server/server-decision.*` are
untouched, so `input_tokenize.cpp` and the decision port need no change.

Checked on an M1 Max (Metal): the default suite on a default build and on an
`LLAMA_RPC=1` build; `:smoke`, `:embeddings` (Qwen3-Embedding and
embeddinggemma-2), `:slow`, `:mtp`, `:tools`, `:decision`, `:decision_clef`,
`:mtp_sidecar` (Qwen3.8-27B) and `test/vm_exit_test.exs`; and
`scripts/decision_compare.exs` — ALL MATCH on 9 models against llama-server at
the same commit, bit-identical on every request the script checks. On two DGX
Sparks (GB10, CUDA 13.0, `LLAMA_RPC=1` build, `verify-build-flags.sh` clean):
the default suite, the same model tags plus `:dflash` and bge-small
embeddings, `vm_exit_test.exs`, `:rpc_live` against the other node, and
`scripts/spark/e2e.exs two-node` (12/12). Single-node speculative decode on
Qwen3.8-27B NVFP4 is unchanged (`bench/spark_spec.exs`: plain 14.21, MTP
`n_draft: 2` 26.55, DFlash 2 `n_draft: 7` 44.24 tok/s, against 14.28 / 26.76 /
44.22 at v0.8.67).

### Changed

- `LlamaCppEx.MTP.init/2` no longer refuses a DFlash/DSpark drafter that is
not loaded on every device the target uses. Upstream #31635 (the same change
as the `draft-ctx-other-devices` branch of `nyo16/llama.cpp`) gives the
draft context a backend for each of the target's devices. On two Sparks a
drafter on `CUDA0` alone beside a target on `CUDA0` + `RPC0` decodes the
same text at the same rate as one loaded on both.
- `LlamaCppEx.MTP.init/2` logs a warning when a DFlash/DSpark target's last
device is an RPC device. Upstream now keeps the target features a DFlash
drafter reads in a buffer on the target's output device (#31506), so with
the output layer on the worker every verify ships them over the network.
Qwen3.8-27B NVFP4 split 50/50 over two Sparks, DFlash 2 `n_draft: 7`:
7.9 tok/s with `devices: ["CUDA0", "RPC0"]` (28.1 at v0.8.67, plain 13.7),
**32.5–33.0** with `["RPC0", "CUDA0"]`, which also lifts plain decode to
14.3. `LLAMA_DFLASH_DEV=0` restores host copies (28.2–28.7). The `MTP`
docs, `docs/dgx-spark.md` and `scripts/spark/e2e.exs` now list the local
device last.
- `decision.cpp` is now "Ported at" `a691b2345`; no hunk needed porting.
- `test/mtp_model_test.exs` records `:mtp_cancel` at this commit: 2 of 4 runs
abort (exit 134, `GGML_ASSERT` in `common_batch` and in `ggml-backend.cpp`),
the other two pass. The race it documents is unchanged.
- `docs/dgx-spark.md` and `docs/release-guide.md`: of the `nyo16/llama.cpp`
branches, `draft-ctx-other-devices` is upstream; `rpc-add-server-no-abort`
and `rpc-client-spin-wait` are not and still apply; `cuda-mmq-src1-reuse`
conflicts with #31296 and #31047.

### Fixed

- An `LLAMA_RPC=1` build failed to compile on macOS since v0.8.67:
`rpc_probe_hello` opened its socket with `SOCK_CLOEXEC`, which macOS does
not have. It sets `FD_CLOEXEC` with `fcntl` instead.

### Known issue

- `split_mode: :tensor` with an RPC device in `:devices` segfaults the VM on
the first decode at this llama.cpp commit: the RPC client serializes the
meta backend's graph with a recursive walk that overflows a dirty CPU
scheduler's default 320 KB stack (Qwen3-8B over two Sparks; upstream's
`llama-bench` runs on an 8 MB main thread and is not affected). Start the
VM with `+sssdcpu 256` if you use it. `:layer`, the mode the docs recommend
across hosts, is not affected. Measured with `llama-bench` from the
submodule (gpt-oss-120b, two Sparks), `-sm tensor` decodes at 15.2 tok/s
against 50.4 for `-sm layer` (tensor was 8.9 at `785322d17`).

### Upstream

- From the 172 commits:
- Flash attention over a unified KV cache attends to each sequence's own
cells (`ggml_flash_attn_ext_rows`, #29510) instead of the whole cache with
a dense mask. `LlamaCppEx.Server` uses a unified cache by default, and
`LlamaCppEx.Decision` shares a prompt through one. `LLAMA_KV_ROWS_DISABLE`
turns it off. RPC: the op travels as `GGML_OP_FLASH_ATTN_EXT` with an
extra source and the protocol version did not change, so run workers and
clients from the same release.
- On-device sequence state copies of a quantized KV cache copied only part
of each range (#30541). The MTP loop's checkpoint rollback saves state
with `LLAMA_STATE_SEQ_FLAGS_ON_DEVICE`, so it was affected with `:type_k`
/ `:type_v` set to a quantized type.
- A context with `ctx_other` gets backends for the other model's devices,
and the Gemma 4 assistant follows its target's flash-attention setting
(#31635). DFlash copies hidden features on the device (#31506). MTP saves
and restores its hidden state with a checkpoint (#30141).
- KV shifting fix and per-model RoPE (#31683); a cache that shares another's
cells no longer reports 0 KV rows before its sequences have any (#31570).
- The CPU backend computes F16 matmuls with F32 activations (#31854), so CPU
results move slightly; Metal follows the reference precision of
`ggml_prec_set_acc` (#31851).
- Gemma 4 E2B/E4B prompt processing prunes rows that need no logits
(#31356).
- Chat: invalid UTF-8 in PEG-parsed tool call arguments is sanitized
(#31861); a constraint type is allowed after gpt-oss's final channel
(#31556); jinja implements `map('filter')` (#24033), and `upper`, `lower`,
`strip`, `title`, `capitalize` and `join` now reject unexpected arguments.
- New architectures: Kimi-K3.1-Pro (#31779), Ripple (#31692), xing4_0
(#31589), Metis-1.6 (#31134), Clover (#31306); NVFP4 LFM3 (#31608) and
NVFP4 scales in qwen2's FFN (#31657).
- Metal: GROUP_NORM fix (#31852), non-contiguous copy fix (#31581),
`mul_mm` for 32 ≤ K < 64 (#30982), fusable node groups kept in one command
buffer (#31688), DeepSeek-V4 and GLM MoE routing fused (#31615).
- CUDA: NVFP4 gate/up/GLU fused in MMQ (#31296), linearized MoE layout for
MMQ/MMF (#31047), a Q8_1 correction-sum fix (#31247), NaN sanitizing in MoE
selection (#31005), graphs no longer evicted just before use (#30662).
- RPC: NCCL for multi-node tensor parallelism (#28967) — the RPC server's
communicator uses `ggml_backend_comm_init_rank` when its backend has it.
Not measured on two Sparks for this release.
- Apple host buffers use `vm_allocate` (#31437). Vulkan, SYCL, OpenCL,
Hexagon, WebGPU and zDNN work.

## v0.8.67

llama.cpp stays at `785322d17`. **Behaviour change:** `LlamaCppEx.Model.load/2`
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ endif
# Pinned llama.cpp commit, used when vendor/llama.cpp has to be cloned. MUST
# match the vendor/llama.cpp submodule; bump both together, see
# docs/release-guide.md. Override to build the NIF against another revision.
LLAMA_COMMIT ?= 785322d17f2d51e5185cf996a1f4e46d87640841
LLAMA_COMMIT ?= a691b2345844ce1b430201fc5fd404ce062e05f2

# The commit actually on disk. A submodule can be bumped without LLAMA_COMMIT
# following it, and the build has to key off what is really there.
Expand Down
7 changes: 5 additions & 2 deletions bench/spark_tensor_split.exs
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,12 @@ Code.require_file("helpers.exs", __DIR__)
# scripts/spark/remote.sh --env MIX_ENV=bench spark-1 \
# mix run bench/spark_tensor_split.exs local
#
# # 6.2 — two nodes, Meta device over [CUDA0, RPC0]
# # 6.2 — two nodes, Meta device over [CUDA0, RPC0]. Since a691b2345 the RPC
# # client's recursive graph serialization overflows a dirty scheduler's
# # default stack on the tensor leg (VM segfault), hence +sssdcpu.
# scripts/spark/rpc-worker.sh start spark-2
# scripts/spark/remote.sh --env MIX_ENV=bench --env LLAMA_RPC=1 spark-1 \
# scripts/spark/remote.sh --env MIX_ENV=bench --env LLAMA_RPC=1 \
# --env ELIXIR_ERL_OPTIONS="+sssdcpu 256" spark-1 \
# mix run bench/spark_tensor_split.exs remote
#
# "tp=2" is vLLM vocabulary. In llama.cpp at b10362 it means
Expand Down
2 changes: 1 addition & 1 deletion c_src/llama_cpp_ex/decision.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
// of tools/server/server-context.cpp (send_decision, the batch layout rules and
// the /v1/systemone handler). See decision.h for scope.
//
// Ported at 785322d17. Functions that are upstream's carry its name in a
// Ported at a691b2345. Functions that are upstream's carry its name in a
// trailing comment so a bump can diff them one by one; everything below the
// "evaluation" banner replaces server slots and is ours.

Expand Down
8 changes: 3 additions & 5 deletions c_src/llama_cpp_ex/llama_nif.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -690,8 +690,10 @@ bool rpc_probe_hello(const std::string& endpoint, int timeout_ms) {

bool ok = false;
for (addrinfo* ai = res; ai && !ok; ai = ai->ai_next) {
int fd = socket(ai->ai_family, ai->ai_socktype | SOCK_CLOEXEC, ai->ai_protocol);
// SOCK_CLOEXEC is Linux/BSD only; macOS has no such socket() flag.
int fd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);
if (fd < 0) continue;
fcntl(fd, F_SETFD, FD_CLOEXEC);

int flags = fcntl(fd, F_GETFL, 0);
fcntl(fd, F_SETFL, flags | O_NONBLOCK);
Expand Down Expand Up @@ -1083,10 +1085,6 @@ int64_t model_n_layer_nextn(ErlNifEnv* env, fine::ResourcePtr<LlamaModel> model)
LLAMA_NIF(model_n_layer_nextn, 0);

// Names of the devices the model's layers were placed on, in placement order.
// LlamaCppEx.MTP uses it to refuse a DFlash drafter that cannot reach the
// target's embeddings and LM head (shared via ctx_other): a draft context
// without the device holding them hits "pre-allocated tensor ... in a buffer
// that cannot run the operation", a GGML_ABORT.
std::vector<std::string> model_devices(ErlNifEnv* env, fine::ResourcePtr<LlamaModel> model) {
std::vector<std::string> names;
const int32_t n = llama_model_n_devices(model->model);
Expand Down
62 changes: 47 additions & 15 deletions docs/dgx-spark.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,19 @@ inferred from spec sheets; where a number differs from the marketing figure, the
measurement wins and the reason is given.

The work behind this page produced four llama.cpp branches on
`nyo16/llama.cpp`, each based on `785322d17`, the vendored commit. None of them
is in the vendored tree. A fifth, `spark-all`, merges all four.
`nyo16/llama.cpp`, each based on `785322d17`, the commit vendored through
v0.8.67. A fifth, `spark-all`, merges all four. v0.8.68 vendors `a691b2345`,
which carries `draft-ctx-other-devices` as upstream #31635 (`e853ec9a8`, the
same change). The two RPC branches still apply to it unchanged;
`cuda-mmq-src1-reuse` conflicts with upstream's NVFP4 gate/up/GLU MMQ fusion
(#31296) and MoE layout change (#31047) and needs a rebase.

| branch | effect on GB10 | section |
|---|---|---|
| `cuda-mmq-src1-reuse` | NVFP4 prefill +19–20% (src1 reuse, weight-scale fold, SWIGLU in MMQ); gpt-oss-120b prefill +8% (expert biases) | Qwen3.8-27B NVFP4, gpt-oss-120b prefill |
| `rpc-client-spin-wait` | two-node decode +30% unpinned (gpt-oss-120b 44 → 57.6 tok/s, single node 59) | The cause, and a fix |
| `rpc-add-server-no-abort` | an unreachable worker returns an error instead of aborting | Five things, #1 |
| `draft-ctx-other-devices` | DFlash with the target split over RPC | Speculative decoding across two nodes |
| `draft-ctx-other-devices` | DFlash with the target split over RPC (upstream as #31635 since `a691b2345`) | Speculative decoding across two nodes |

## The machine

Expand Down Expand Up @@ -951,19 +955,39 @@ harness above, so acceptance is not comparable with the one-node table):
| DFlash 2 `n_draft: 7` | **22.2–22.4** | 27% | **27.8–27.9** (29%) |

DFlash still pays 1.6× across two nodes, and 1.9× with the RPC client fix. With
the fix, plain decode over two nodes beats one node's 14.3. The drafter must be
loaded on every device the target uses, because it computes with the target's
embeddings and LM head: with the target on `["CUDA0", "RPC0"]` and the drafter on `CUDA0`
alone, upstream aborts the process ("pre-allocated tensor (output.weight) in
a buffer (RPC0) that cannot run the operation") — `llama-server
--spec-draft-device CUDA0` does it too. `MTP.init/2` refuses that pairing;
load the drafter with the target's `:devices` and `tensor_split: [1.0, 0.0]`
to keep its own layers local.

The upstream fix is branch `draft-ctx-other-devices` on `nyo16/llama.cpp`: a
the fix, plain decode over two nodes beats one node's 14.3. Up to v0.8.67 the
drafter had to be loaded on every device the target uses, because it computes
with the target's embeddings and LM head: with the target on `["CUDA0", "RPC0"]`
and the drafter on `CUDA0` alone, upstream aborted the process ("pre-allocated
tensor (output.weight) in a buffer (RPC0) that cannot run the operation"), and
`MTP.init/2` refused that pairing. The fix, first on branch
`draft-ctx-other-devices` of `nyo16/llama.cpp`, is upstream as #31635: a
context with `ctx_other` gets a backend for each of the other model's devices.
With it, that same `llama-server` command runs, at 26.4 tok/s (566 drafted,
173 accepted) on one 256-token completion.
Since v0.8.68 the drafter can sit on `CUDA0` alone, and decodes the same text
at the same rate as one loaded on both devices.

#### Put the local device last

At `a691b2345` DFlash reads the target's layer features from a device buffer
(#31506) instead of host copies, and that buffer lives on the target's
*output* device. Listed `["CUDA0", "RPC0"]`, the output layer is on the worker,
so every verify ships the features over the fabric. Same Qwen3.8-27B NVFP4
50/50 split, one 256-token greedy chat completion ("Explain how a hash map
handles collisions…"), three runs, client on the A725 cores, decode rate from
first to last token:

| target `:devices` | build | plain | DFlash 2 `n_draft: 7` | verify ms/iter |
|---|---|---|---|---|
| `["CUDA0", "RPC0"]` | v0.8.67 (`785322d17`) | 13.75 | 27.4–28.1 | 94–96 |
| `["CUDA0", "RPC0"]` | v0.8.68 (`a691b2345`) | 13.7 | **7.8–7.9** | 395–400 |
| `["CUDA0", "RPC0"]`, `LLAMA_DFLASH_DEV=0` | v0.8.68 | 13.75 | 28.2–28.7 | 93–95 |
| **`["RPC0", "CUDA0"]`** | v0.8.68 | **14.3** | **32.5–33.0** | 86–88 |

With the local device last the buffer stays on `CUDA0` and DFlash is faster
than before the change; plain decode gains 4% too, since the logits no longer
cross the fabric. `MTP.init/2` logs a warning for a DFlash/DSpark target whose
last device is an RPC device. `LLAMA_DFLASH_DEV=0` is upstream's switch back
to host copies. Acceptance is 34.5% in every row.

### Concurrency across two nodes

Expand Down Expand Up @@ -1051,6 +1075,14 @@ communicator's own spin left blocking, decodes at **24.1** with pp512 1553.
That is 2.7× better, and still less than half of a layer split's 57.6. The
verdict below stands.

At `a691b2345` (v0.8.68), same `llama-bench` run from the submodule, client on
the A725 cores against a worker from the same commit: `-sm tensor` decodes at
15.2 tok/s with pp512 850, `-sm layer` at 50.4 with pp512 1737. #28967 lets an
RPC worker's pairwise communicator use NCCL through
`ggml_backend_comm_init_rank`, but the client's `CUDA0` beside `RPC0` still
cannot. Through the binding, `:tensor` over RPC also needs a larger dirty
scheduler stack at this commit (`+sssdcpu 256`); see `LlamaCppEx.Model`.

## Why, and why the comm-mode knob cannot help

`ggml_backend_cuda_comm_init` returns `nullptr` the moment **any** member backend
Expand Down
35 changes: 35 additions & 0 deletions docs/release-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,41 @@ Re-checked at `785322d17` (b12032+1), covering the 31 commits from
RPC protocol and op ids are unchanged (#31424 removes `fs_path_to_utf8` from
`tools/rpc`, not from the backend).

Re-checked at `a691b2345` (b12205), covering the 172 commits from
`785322d17`: #1 and #3 are as above — `ggml-cpu/CMakeLists.txt` was not
touched and `ggml_backend_rpc_start_server` keeps its `void` signature. #2
stands: #28967 adds `ggml_backend_comm_init_rank` (NCCL in the CUDA backend)
and the RPC server's pairwise communicator uses it when the worker's backend
has it, but `ggml_backend_cuda_comm_init` still returns `nullptr` on a
non-CUDA member. Measured with upstream's `llama-bench` built from the
submodule (gpt-oss-120b, `-fa 1 -p 512 -n 64 -r 3`, client on the A725
cores): `-sm layer` pp512 1737, tg64 50.4; `-sm tensor` pp512 850, tg64 15.2,
against 1135 / 8.9 at `785322d17`. Better, still 30% of a layer split.

Through the binding, `:tensor` with an RPC device in the set segfaults the VM
at this commit: the RPC client's `serialize_graph` walks the meta backend's
graph with the recursive `add_tensor`, which overflows a dirty CPU
scheduler's default 320 KB stack (40 kilowords) on Qwen3-8B. `llama-bench`
runs it on an 8 MB main thread and does not notice. `+sssdcpu 256` (2 MB) is
enough; `bench/spark_tensor_split.exs remote` needs
`ELIXIR_ERL_OPTIONS="+sssdcpu 256"`. A layer split is unaffected. Re-check
on a bump by running that bench without the flag.

The RPC protocol stays 8.0.1 and `GGML_OP_COUNT` 102, yet two graph changes
ride on existing ops: `ggml_flash_attn_ext_rows` (#29510) is
`GGML_OP_FLASH_ATTN_EXT` with K/V row indices in `src[5]`, used when the KV
cache is unified, flash attention is on and the device passes the op probe
(`LLAMA_KV_ROWS_DISABLE` turns it off), and RoPE gains a `truncate`
op param (#30206). The RPC client's `supports_op` refuses only
`truncate=false` RoPE, so a worker from v0.8.67 or older is offered
`kv_rows` attention it does not know to read. Run worker and client from the
same release.

Of the `nyo16/llama.cpp` branches, `draft-ctx-other-devices` is upstream as
#31635 (`e853ec9a8`, the same change), `rpc-add-server-no-abort` and
`rpc-client-spin-wait` still apply unchanged, and `cuda-mmq-src1-reuse`
conflicts with #31296 and #31047.

| # | Upstream defect | Our workaround | Still needed? |
|---|---|---|---|
| 1 | `GGML_NATIVE=ON` makes ggml's `-mcpu=native` probe resolve to **base ARMv8-A** on Cortex-X925/A725 with GCC 13.3 — silently, with a soft CMake warning and exit 0. Costs every `sdot`/`smmla`/SVE kernel. | `LLAMA_CPU_ARM_ARCH` + `LLAMA_CUDA_ARCH` in the `Makefile`, which must be set together. See [DGX Spark](dgx-spark.md) and [Cross-Platform Builds](cross-platform-builds.md). | `scripts/spark/verify-build-flags.sh` on an aarch64 host. If a default build (no `LLAMA_CPU_ARM_ARCH`) now reports non-zero `sdot`/`smmla`, upstream fixed the probe. |
Expand Down
5 changes: 5 additions & 0 deletions lib/llama_cpp_ex/model.ex
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,11 @@ defmodule LlamaCppEx.Model do
> disables backend sampling, and its CUDA all-reduce is `ncclCommInitAll` —
> a single-process, all-local-GPUs API. **It cannot span hosts**, so it is not
> "tp=2 across two machines". See `docs/dgx-spark.md` for the measurements.
> With an RPC device in the set it runs, far slower than `:layer`, and at
> llama.cpp `a691b2345` its first decode overflows a dirty scheduler's
> default stack (320 KB) while the RPC backend serializes the graph, which
> kills the VM with a segfault. Start the VM with `+sssdcpu 256` (2 MB,
> e.g. `ELIXIR_ERL_OPTIONS="+sssdcpu 256"`) if you run it anyway.

> #### Load mode {: .info}
>
Expand Down
Loading