Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -51,3 +51,12 @@ RPC, so the bump that brought it in (v0.8.5x) passed.
- `rsync -a` preserves source mtimes: after editing a file on a node, a sync
that restores an older-mtime copy leaves make's `.o` newer than its source.
`touch` the source before rebuilding.

## Update (llama.cpp 9daa0aa22, v0.8.69)

Upstream #30415 (`6122b566b`) makes `rpc_dispatcher::start` return false
instead of aborting, so `ggml_backend_rpc_add_server` returns `nullptr` for an
unreachable, unparseable or mismatched endpoint again. The probe stays: upstream
connects with no timeout and answers an already-registered endpoint from
`reg_map` without contacting it. The protocol is 10.0.0 now; the HELLO layout
did not change, so only the `static_assert` moved.
90 changes: 90 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,95 @@
# Changelog

## v0.8.69

llama.cpp bumped to `9daa0aa22` (b12361+7), 163 commits from `a691b2345`
(b12205). Upstream versions: llama.cpp **1.0.0**, ggml **1.0.0** (#32190,
ggml/1690). The NIF needed one change, in the RPC probe (below). #30565 and
#29801 prune deprecated API: every `DEPRECATED` function in `llama.h`
(`llama_load_model_from_file`, `llama_new_context_with_model`,
`llama_token_*`, `llama_n_vocab`, the old state/session calls,
`llama_set_warmup`, `llama_sampler_init_grammar_lazy`), the
`defrag_thold` field of `llama_context_params`, and the `ggml_add1`,
`ggml_diag_mask_*`, `ggml_rope_custom*` and `*_set_prec` ops; the NIF calls
none of them. The NextN/DFlash queries move from `src/llama-ext.h` into
`llama.h` unchanged. `common.h` and `chat.h` only add.
`llama_model_default_params` is unchanged and `llama_context_default_params`
only loses `defrag_thold`. `tokenizer_st_partition` and the BOS/EOS rules in
`src/llama-vocab.cpp` are untouched (the diff only deletes the deprecated
wrappers), and so are `tools/server/server-decision.*`, so
`input_tokenize.cpp` and the decision port need no change.

Checked on an M1 Max (Metal): the default suite on a default build and on an
`LLAMA_RPC=1` build, plus `:rpc_live` against a local `RPC.Server` worker;
`:smoke`, `:embeddings` (Qwen3-Embedding and
embeddinggemma-2), `:slow`, `:mtp` (Qwen3.6-27B), `:tools`, `:decision`,
`:decision_clef`, `:mtp_sidecar` (Qwen3.8-27B) and `test/vm_exit_test.exs`;
and `scripts/decision_compare.exs` — ALL MATCH on 9 models (tinylaya,
tinyopenjev, Clef Flash, d1-3B, d1-omni-600M, d1-2B, vllm-decision2 Kai,
vllm-d3-edge, h2o-lightning-4b) against llama-server at the same commit. Not
run on a DGX Spark for this release.

### Changed

- **RPC protocol 10.0.0** (was 8.0.1): #29413 takes it to 9 and the ggml
prune (#29801) to 10, removing three ops so every later op id moves. A
worker and a client must both run v0.8.69; against an older worker
`LlamaCppEx.RPC.add_server/1` and `ping/1` return `{:error, :unreachable}`.
The HELLO layout did not change, so the NIF's probe only moves its
`static_assert` to 10.
- Upstream registration no longer aborts on an unreachable, unparseable or
mismatched endpoint (#30415). `LlamaCppEx.RPC` keeps its own pre-flight
HELLO: upstream's connect has no timeout, and a registered endpoint is
answered from upstream's cache without being contacted, so the probe is
still what lets `ping/1` notice a peer that went away. The `RPC` docs,
`docs/dgx-spark.md` and `docs/release-guide.md` say so; of the
`nyo16/llama.cpp` branches, `rpc-add-server-no-abort` is superseded and
`rpc-client-spin-wait` now conflicts with #29413 and needs a rebase.
- `decision.cpp` is now "Ported at" `9daa0aa22`; no hunk needed porting.
- `test/mtp_model_test.exs` records `:mtp_cancel` at this commit: 4 of 7 runs
abort (exit 134, `GGML_ASSERT(logits != nullptr)` in `common_sampler_sample`
from the MTP drafter), the other three pass. The race it documents is
unchanged.

### Known issue

- `split_mode: :tensor` with an RPC device in `:devices` still needs
`+sssdcpu 256`: the RPC client's recursive graph serializer is unchanged
(see v0.8.68). Not re-measured.

### Upstream

- From the 163 commits:
- MTP reads the target's hidden states from a device buffer (#31927); the
NextN API is public in `llama.h`.
- FP8 (E4M3) KV cache (#28392) and OCP FP8 on CUDA (#28413). `:type_k` /
`:type_v` do not offer it yet.
- The non-fused gated delta net path and the ubatch-dependent graph nodes
are gone (#32078).
- The backend scheduler is reused when its parameters are compatible
(#32009) and graph results are reused across reserves (#32089); unused
graph metadata pages are no longer touched (#32043); inputs that are not
read are not uploaded (#31419).
- Every sampling chain is marked as a graph output so its tensors are not
aliased (#32015).
- CPU flash attention accumulates V in F32 (#29810); the 4x4 SIMD GEMM tile
on ARM NEON (#32128). CPU results move slightly.
- Grammar matching is faster through left factoring (#29557).
- Chat: Kimi K2 Horizon closer at medium/low effort (#30335), Gemma 4
schema rules rebuilt (#31915), Muse Spark honours `response_format`
(#32017), no lazy grammar when `response_format` is combined with tools
(#31956), control characters escaped in tool argument strings (#31937),
`allOf` with plain types fixed in JSON-schema grammars (#31834); jinja
supports chained comparisons and argument unpacking (#32072).
- Metal: out-of-bounds fixes in the FA output (#31662) and `SOLVE_TRI`
(#32103); `ggml_tri` selects instead of scaling (#30711).
- CUDA: few-row MMQ for batch sizes 2–24 (#31533), NVFP4 weight scale fused
in the MMQ epilogue (#26048), BF16 `RMS_NORM` (#32135), SM103 SASS in
non-native builds (#30099), deterministic tied top-k (#31084).
- The iGPU lazy-mode fallback applies only when mmap is off (#31957); the
default iGPU load path is unchanged.
- Vulkan, SYCL, OpenCL, Hexagon, WebGPU, zDNN, SpacemiT and mtmd work.

## v0.8.68

llama.cpp bumped to `a691b2345`
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ endif
# Pinned llama.cpp commit, used when vendor/llama.cpp has to be cloned. MUST
# match the vendor/llama.cpp submodule; bump both together, see
# docs/release-guide.md. Override to build the NIF against another revision.
LLAMA_COMMIT ?= a691b2345844ce1b430201fc5fd404ce062e05f2
LLAMA_COMMIT ?= 9daa0aa22b8aa0143344d7549dee1f0e32f048bb

# The commit actually on disk. A submodule can be bumped without LLAMA_COMMIT
# following it, and the build has to key off what is really there.
Expand Down
2 changes: 1 addition & 1 deletion c_src/llama_cpp_ex/decision.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
// of tools/server/server-context.cpp (send_decision, the batch layout rules and
// the /v1/systemone handler). See decision.h for scope.
//
// Ported at a691b2345. Functions that are upstream's carry its name in a
// Ported at 9daa0aa22. Functions that are upstream's carry its name in a
// trailing comment so a bump can diff them one by one; everything below the
// "evaluation" banner replaces server slots and is ours.

Expand Down
26 changes: 14 additions & 12 deletions c_src/llama_cpp_ex/llama_nif.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -531,13 +531,15 @@ LLAMA_NIF(backend_free, 0);
// 1. RPC_STATUS_ASSERT is GGML_ABORT (ggml-rpc.cpp:30). Any peer crash,
// network failure or malformed response terminates the OS process — the
// whole BEAM. There is no error return, no retry, no reconnect.
// 2. Since #18626 (d0132a680) registration aborts too: the connect and the
// HELLO moved into rpc_dispatcher::start, which GGML_ABORTs on an
// unreachable endpoint, an unparseable one, or a version mismatch where
// ggml_backend_rpc_add_server used to return nullptr. rpc_add_server
// therefore probes the endpoint itself first (rpc_probe_hello), so a dead
// or mismatched node is still *detectable before load* and only *fatal
// during it*.
// 2. Registration used to abort too: #18626 (d0132a680) moved the connect
// and the HELLO into rpc_dispatcher::start, which GGML_ABORTed on an
// unreachable, unparseable or version-mismatched endpoint. #30415
// (6122b566b) made those failures return nullptr again, but upstream
// still connects with no timeout and memoizes a registered endpoint
// forever, so a peer that has since gone away is only found mid-load,
// where it aborts. rpc_add_server therefore still probes the endpoint
// itself first (rpc_probe_hello, 5 s), so a dead or mismatched node is
// *detectable before load* and only *fatal during it*.

#ifdef GGML_USE_RPC

Expand Down Expand Up @@ -622,7 +624,7 @@ bool rpc_wait_until_listening(const std::string& host, const std::string& port,
// the major version (the server's "client needs upgrade to protocol v%d.x"),
// so a major bump stops the build here instead of failing every probe at run
// time.
static_assert(RPC_PROTO_MAJOR_VERSION == 8,
static_assert(RPC_PROTO_MAJOR_VERSION == 10,
"RPC protocol major changed: re-check rpc_probe_hello against "
"rpc_msg_hello_req/rsp and RPC_CMD_HELLO in ggml-rpc.cpp");

Expand Down Expand Up @@ -775,16 +777,16 @@ rpc_add_server(ErlNifEnv* env, std::string endpoint) {
#else
size_t before = ggml_backend_dev_count();

// ggml_backend_rpc_add_server aborts the VM on an unreachable or
// mismatched endpoint (see the top of this section), so check first.
// Upstream connects with no timeout and answers a memoized endpoint without
// contacting it (see the top of this section), so check first.
if (!rpc_probe_hello(endpoint, 5000)) {
return fine::Error(atoms::unreachable);
}

ggml_backend_reg_t reg = ggml_backend_rpc_add_server(endpoint.c_str());
if (!reg) {
// A worker reporting zero devices still comes back as nullptr, and
// ggml_backend_register silently no-ops on nullptr — so without this
// An unreachable worker, or one reporting zero devices, comes back as
// nullptr, and ggml_backend_register silently no-ops on nullptr — so without this
// check that node simply vanishes and the model loads onto the wrong
// devices.
return fine::Error(atoms::unreachable);
Expand Down
27 changes: 16 additions & 11 deletions docs/dgx-spark.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,19 @@ The work behind this page produced four llama.cpp branches on
`nyo16/llama.cpp`, each based on `785322d17`, the commit vendored through
v0.8.67. A fifth, `spark-all`, merges all four. v0.8.68 vendors `a691b2345`,
which carries `draft-ctx-other-devices` as upstream #31635 (`e853ec9a8`, the
same change). The two RPC branches still apply to it unchanged;
`cuda-mmq-src1-reuse` conflicts with upstream's NVFP4 gate/up/GLU MMQ fusion
(#31296) and MoE layout change (#31047) and needs a rebase.
same change). v0.8.69 vendors `9daa0aa22`, where #30415 (`6122b566b`) makes
an unreachable endpoint fail registration instead of aborting, the effect of
`rpc-add-server-no-abort` by a different patch. `rpc-client-spin-wait` no
longer applies: it conflicts in `ggml-rpc.cpp` with #29413 (`cf92c1323`, RPC
load times) and needs a rebase, as does `cuda-mmq-src1-reuse`, which
conflicts with upstream's NVFP4 gate/up/GLU MMQ fusion (#31296) and MoE
layout change (#31047).

| branch | effect on GB10 | section |
|---|---|---|
| `cuda-mmq-src1-reuse` | NVFP4 prefill +19–20% (src1 reuse, weight-scale fold, SWIGLU in MMQ); gpt-oss-120b prefill +8% (expert biases) | Qwen3.8-27B NVFP4, gpt-oss-120b prefill |
| `rpc-client-spin-wait` | two-node decode +30% unpinned (gpt-oss-120b 44 → 57.6 tok/s, single node 59) | The cause, and a fix |
| `rpc-add-server-no-abort` | an unreachable worker returns an error instead of aborting | Five things, #1 |
| `rpc-add-server-no-abort` | an unreachable worker returns an error instead of aborting (upstream as #30415 since `9daa0aa22`) | Five things, #1 |
| `draft-ctx-other-devices` | DFlash with the target split over RPC (upstream as #31635 since `a691b2345`) | Speculative decoding across two nodes |

## The machine
Expand Down Expand Up @@ -656,13 +660,14 @@ wrong devices" into `{:error, :unreachable}`. After the load, treat the VM as th
unit of restart. Real fault isolation means putting the RPC client in a separate
OS process, which is a different architecture.

Upstream's registration aborts too since #18626 — an unreachable endpoint, a
listener that is not an RPC worker, a version mismatch and a malformed
`host:port` all reach `GGML_ABORT` in `rpc_dispatcher::start`. So the NIF
connects and exchanges the protocol HELLO itself before registering, with a
5-second timeout. That is what keeps `ping/1` a check rather than a crash on
this llama.cpp; the matching upstream fix is branch `rpc-add-server-no-abort`
on `nyo16/llama.cpp`.
From #18626 until #30415 (v0.8.68 and older), upstream's registration aborted
too — an unreachable endpoint, a listener that is not an RPC worker, a version
mismatch and a malformed `host:port` all reached `GGML_ABORT` in
`rpc_dispatcher::start`. #30415 turns them into a failed registration, but
upstream still connects with no timeout and answers a registered endpoint from
its cache without contacting it. So the NIF still connects and exchanges the
protocol HELLO itself before registering, with a 5-second timeout; that is what
makes `ping/1` report a peer that has gone away since it was registered.

### 2. `devices/0` order is NOT `tensor_split` order

Expand Down
32 changes: 29 additions & 3 deletions docs/release-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -368,10 +368,36 @@ op param (#30206). The RPC client's `supports_op` refuses only
`kv_rows` attention it does not know to read. Run worker and client from the
same release.

Re-checked at `9daa0aa22` (b12361+7), covering the 163 commits from
`a691b2345`, as a source diff: #1, #2 and #3 are as above.
`ggml-cpu/CMakeLists.txt` was not touched. The `ggml-cuda.cu` diff is MMQ,
FP8 (#28413), graph and pool work, none of it near
`ggml_backend_cuda_comm_init`. `ggml_backend_rpc_start_server` was only
reformatted and still returns `void`. The RPC client's recursive
`serialize_graph`/`add_tensor` is unchanged, so the `+sssdcpu 256` note
above still applies (not re-measured: no Spark for this bump).

The RPC protocol goes 8.0.1 → **10.0.0**: #29413 (`cf92c1323`, load times)
takes it to 9, and #29801 (`29dbbad2b`,
ggml prune) removes `GGML_OP_ADD1`, `GGML_OP_DIAG_MASK_INF` and
`GGML_OP_DIAG_MASK_ZERO` (`GGML_OP_COUNT` 102 → 99, every later op
renumbers) and takes it to 10. The handshake refuses a major mismatch, so a
v0.8.68 worker and a v0.8.69 client report `{:error, :unreachable}` rather
than misread each other. The `HELLO` request and response layouts are
unchanged, so `rpc_probe_hello` only needed its `static_assert` moved to 10.
#30415 (`6122b566b`) makes `rpc_dispatcher::start` return false instead of
`GGML_ABORT` on a bad endpoint, a failed connect or a failed HELLO, so
`ggml_backend_rpc_add_server` returns `nullptr` again. The NIF's probe stays:
upstream's connect has no timeout, and a registered endpoint is answered from
`reg_map` without being contacted, so only the probe notices a peer that went
away after registration.

Of the `nyo16/llama.cpp` branches, `draft-ctx-other-devices` is upstream as
#31635 (`e853ec9a8`, the same change), `rpc-add-server-no-abort` and
`rpc-client-spin-wait` still apply unchanged, and `cuda-mmq-src1-reuse`
conflicts with #31296 and #31047.
#31635 (`e853ec9a8`, the same change), `rpc-add-server-no-abort` is
superseded by #30415 (a different patch with the same effect), and
`rpc-client-spin-wait` (still not upstream: the dispatcher only spins while a
communicator holds it) conflicts with #29413 and needs a rebase, as does
`cuda-mmq-src1-reuse`, which conflicts with #31296 and #31047.

| # | Upstream defect | Our workaround | Still needed? |
|---|---|---|---|
Expand Down
21 changes: 12 additions & 9 deletions lib/llama_cpp_ex/rpc.ex
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,13 @@ defmodule LlamaCppEx.RPC do
> to catch.
>
> Registration is the exception and the reason this module is shaped the way it
> is. Upstream's registration aborts too (since llama.cpp #18626), so
> `add_server/1` first connects and exchanges the protocol HELLO itself: an
> unreachable or incompatible endpoint is reported as `{:error, :unreachable}`
> rather than aborting, so a two-node setup can be *checked* before a model
> load and is only *fatal* during one. Check with `ping/1` before you load;
> is. `add_server/1` first connects and exchanges the protocol HELLO itself,
> with a 5-second timeout: an unreachable or incompatible endpoint is reported
> as `{:error, :unreachable}`, so a two-node setup can be *checked* before a
> model load and is only *fatal* during one. (Upstream's own registration
> aborted on such an endpoint from llama.cpp #18626 until #30415; it still
> connects with no timeout and never re-contacts an endpoint it has already
> registered.) Check with `ping/1` before you load;
> treat the VM as the unit of restart afterwards. Real fault isolation would
> mean running the RPC client in a separate OS process, which is a different
> architecture.
Expand Down Expand Up @@ -114,10 +116,11 @@ defmodule LlamaCppEx.RPC do
* `{:error, :rpc_unsupported}` — the NIF was built without `LLAMA_RPC=1`.
* `{:error, :unreachable}` — nothing answered within 5 seconds, the
endpoint is not `host:port`, the peer's RPC protocol major/minor did not
match ours, or the worker reported no devices. Upstream aborts the VM on
the first three, and silently ignores the last one at registration, so
this check is the only thing standing between you and a crash or a model
that quietly loads onto the wrong devices.
match ours, or the worker reported no devices. Upstream returns no
registration for all four but waits on the operating system's connect
timeout, and `ggml_backend_register` silently ignores a missing
registration, so without this check the model would quietly load onto
the wrong devices.
"""
@spec add_server(endpoint()) :: {:ok, non_neg_integer()} | {:error, error()}
def add_server(endpoint) when is_binary(endpoint) do
Expand Down
2 changes: 1 addition & 1 deletion mix.exs
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ end
defmodule LlamaCppEx.MixProject do
use Mix.Project

@version "0.8.68"
@version "0.8.69"
@source_url "https://github.com/nyo16/llama_cpp_ex"

def project do
Expand Down
6 changes: 5 additions & 1 deletion test/mtp_model_test.exs
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,11 @@ defmodule LlamaCppEx.MTPCancelTest do
# runs aborted 3/4 (SIGSEGV, exit 139) and the fourth failed with the same
# `unknown exception`. At a691b2345 (b12205) four runs aborted 2/4 (exit 134:
# `GGML_ASSERT(idx == i - off)` in `common_batch`, `GGML_ASSERT(backend)` in
# `ggml-backend.cpp`) and the other two passed.
# `ggml-backend.cpp`) and the other two passed. At 9daa0aa22 (b12361+7), with
# Qwen3.6-27B (the 35B-A3B was not on the machine), seven runs aborted 4/7
# (exit 134, every time `GGML_ASSERT(logits != nullptr)` in
# `common/sampling.cpp`, reached from the MTP drafter's
# `common_sampler_sample`) and the other three passed.
#
# It therefore still lives in its own module carrying only `:mtp_cancel`, and
# deliberately *not* `:mtp`: `--include` beats `--exclude` in ExUnit, so a
Expand Down
9 changes: 5 additions & 4 deletions test/rpc_test.exs
Original file line number Diff line number Diff line change
Expand Up @@ -77,10 +77,11 @@ defmodule LlamaCppEx.RPCTest do
end

test "a listener that is not an RPC worker is an error, never a crash" do
# Since llama.cpp #18626 upstream's own handshake aborts the process when
# the peer accepts and then answers wrongly, so this is the case the NIF's
# pre-flight HELLO exists for: one peer closes after reading the request,
# the other answers with bytes that are not a HELLO response.
# From llama.cpp #18626 until #30415 upstream's own handshake aborted the
# process when the peer accepted and then answered wrongly, so this is the
# case the NIF's pre-flight HELLO was written for: one peer closes after
# reading the request, the other answers with bytes that are not a HELLO
# response.
for reply <- [:close, "HTTP/1.1 400 Bad Request\r\n\r\n"] do
{:ok, listen} = :gen_tcp.listen(0, [:binary, active: false, ip: {127, 0, 0, 1}])
{:ok, port} = :inet.port(listen)
Expand Down
2 changes: 1 addition & 1 deletion vendor/llama.cpp
Submodule llama.cpp updated 741 files