diff --git a/.claude/solutions/phoenix-issues/rpc-add-server-aborts-vm-dispatcher-20261102.md b/.claude/solutions/phoenix-issues/rpc-add-server-aborts-vm-dispatcher-20261102.md index a73b210..c68eedc 100644 --- a/.claude/solutions/phoenix-issues/rpc-add-server-aborts-vm-dispatcher-20261102.md +++ b/.claude/solutions/phoenix-issues/rpc-add-server-aborts-vm-dispatcher-20261102.md @@ -51,3 +51,12 @@ RPC, so the bump that brought it in (v0.8.5x) passed. - `rsync -a` preserves source mtimes: after editing a file on a node, a sync that restores an older-mtime copy leaves make's `.o` newer than its source. `touch` the source before rebuilding. + +## Update (llama.cpp 9daa0aa22, v0.8.69) + +Upstream #30415 (`6122b566b`) makes `rpc_dispatcher::start` return false +instead of aborting, so `ggml_backend_rpc_add_server` returns `nullptr` for an +unreachable, unparseable or mismatched endpoint again. The probe stays: upstream +connects with no timeout and answers an already-registered endpoint from +`reg_map` without contacting it. The protocol is 10.0.0 now; the HELLO layout +did not change, so only the `static_assert` moved. diff --git a/CHANGELOG.md b/CHANGELOG.md index 7828045..c18ae84 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,95 @@ # Changelog +## v0.8.69 + +llama.cpp bumped to `9daa0aa22` (b12361+7), 163 commits from `a691b2345` +(b12205). Upstream versions: llama.cpp **1.0.0**, ggml **1.0.0** (#32190, +ggml/1690). The NIF needed one change, in the RPC probe (below). #30565 and +#29801 prune deprecated API: every `DEPRECATED` function in `llama.h` +(`llama_load_model_from_file`, `llama_new_context_with_model`, +`llama_token_*`, `llama_n_vocab`, the old state/session calls, +`llama_set_warmup`, `llama_sampler_init_grammar_lazy`), the +`defrag_thold` field of `llama_context_params`, and the `ggml_add1`, +`ggml_diag_mask_*`, `ggml_rope_custom*` and `*_set_prec` ops; the NIF calls +none of them. The NextN/DFlash queries move from `src/llama-ext.h` into +`llama.h` unchanged. `common.h` and `chat.h` only add. +`llama_model_default_params` is unchanged and `llama_context_default_params` +only loses `defrag_thold`. `tokenizer_st_partition` and the BOS/EOS rules in +`src/llama-vocab.cpp` are untouched (the diff only deletes the deprecated +wrappers), and so are `tools/server/server-decision.*`, so +`input_tokenize.cpp` and the decision port need no change. + +Checked on an M1 Max (Metal): the default suite on a default build and on an +`LLAMA_RPC=1` build, plus `:rpc_live` against a local `RPC.Server` worker; +`:smoke`, `:embeddings` (Qwen3-Embedding and +embeddinggemma-2), `:slow`, `:mtp` (Qwen3.6-27B), `:tools`, `:decision`, +`:decision_clef`, `:mtp_sidecar` (Qwen3.8-27B) and `test/vm_exit_test.exs`; +and `scripts/decision_compare.exs` — ALL MATCH on 9 models (tinylaya, +tinyopenjev, Clef Flash, d1-3B, d1-omni-600M, d1-2B, vllm-decision2 Kai, +vllm-d3-edge, h2o-lightning-4b) against llama-server at the same commit. Not +run on a DGX Spark for this release. + +### Changed + +- **RPC protocol 10.0.0** (was 8.0.1): #29413 takes it to 9 and the ggml + prune (#29801) to 10, removing three ops so every later op id moves. A + worker and a client must both run v0.8.69; against an older worker + `LlamaCppEx.RPC.add_server/1` and `ping/1` return `{:error, :unreachable}`. + The HELLO layout did not change, so the NIF's probe only moves its + `static_assert` to 10. +- Upstream registration no longer aborts on an unreachable, unparseable or + mismatched endpoint (#30415). `LlamaCppEx.RPC` keeps its own pre-flight + HELLO: upstream's connect has no timeout, and a registered endpoint is + answered from upstream's cache without being contacted, so the probe is + still what lets `ping/1` notice a peer that went away. The `RPC` docs, + `docs/dgx-spark.md` and `docs/release-guide.md` say so; of the + `nyo16/llama.cpp` branches, `rpc-add-server-no-abort` is superseded and + `rpc-client-spin-wait` now conflicts with #29413 and needs a rebase. +- `decision.cpp` is now "Ported at" `9daa0aa22`; no hunk needed porting. +- `test/mtp_model_test.exs` records `:mtp_cancel` at this commit: 4 of 7 runs + abort (exit 134, `GGML_ASSERT(logits != nullptr)` in `common_sampler_sample` + from the MTP drafter), the other three pass. The race it documents is + unchanged. + +### Known issue + +- `split_mode: :tensor` with an RPC device in `:devices` still needs + `+sssdcpu 256`: the RPC client's recursive graph serializer is unchanged + (see v0.8.68). Not re-measured. + +### Upstream + +- From the 163 commits: + - MTP reads the target's hidden states from a device buffer (#31927); the + NextN API is public in `llama.h`. + - FP8 (E4M3) KV cache (#28392) and OCP FP8 on CUDA (#28413). `:type_k` / + `:type_v` do not offer it yet. + - The non-fused gated delta net path and the ubatch-dependent graph nodes + are gone (#32078). + - The backend scheduler is reused when its parameters are compatible + (#32009) and graph results are reused across reserves (#32089); unused + graph metadata pages are no longer touched (#32043); inputs that are not + read are not uploaded (#31419). + - Every sampling chain is marked as a graph output so its tensors are not + aliased (#32015). + - CPU flash attention accumulates V in F32 (#29810); the 4x4 SIMD GEMM tile + on ARM NEON (#32128). CPU results move slightly. + - Grammar matching is faster through left factoring (#29557). + - Chat: Kimi K2 Horizon closer at medium/low effort (#30335), Gemma 4 + schema rules rebuilt (#31915), Muse Spark honours `response_format` + (#32017), no lazy grammar when `response_format` is combined with tools + (#31956), control characters escaped in tool argument strings (#31937), + `allOf` with plain types fixed in JSON-schema grammars (#31834); jinja + supports chained comparisons and argument unpacking (#32072). + - Metal: out-of-bounds fixes in the FA output (#31662) and `SOLVE_TRI` + (#32103); `ggml_tri` selects instead of scaling (#30711). + - CUDA: few-row MMQ for batch sizes 2–24 (#31533), NVFP4 weight scale fused + in the MMQ epilogue (#26048), BF16 `RMS_NORM` (#32135), SM103 SASS in + non-native builds (#30099), deterministic tied top-k (#31084). + - The iGPU lazy-mode fallback applies only when mmap is off (#31957); the + default iGPU load path is unchanged. + - Vulkan, SYCL, OpenCL, Hexagon, WebGPU, zDNN, SpacemiT and mtmd work. + ## v0.8.68 llama.cpp bumped to `a691b2345` diff --git a/Makefile b/Makefile index dce8271..686e257 100644 --- a/Makefile +++ b/Makefile @@ -36,7 +36,7 @@ endif # Pinned llama.cpp commit, used when vendor/llama.cpp has to be cloned. MUST # match the vendor/llama.cpp submodule; bump both together, see # docs/release-guide.md. Override to build the NIF against another revision. -LLAMA_COMMIT ?= a691b2345844ce1b430201fc5fd404ce062e05f2 +LLAMA_COMMIT ?= 9daa0aa22b8aa0143344d7549dee1f0e32f048bb # The commit actually on disk. A submodule can be bumped without LLAMA_COMMIT # following it, and the build has to key off what is really there. diff --git a/c_src/llama_cpp_ex/decision.cpp b/c_src/llama_cpp_ex/decision.cpp index bcd2f1a..103a987 100644 --- a/c_src/llama_cpp_ex/decision.cpp +++ b/c_src/llama_cpp_ex/decision.cpp @@ -2,7 +2,7 @@ // of tools/server/server-context.cpp (send_decision, the batch layout rules and // the /v1/systemone handler). See decision.h for scope. // -// Ported at a691b2345. Functions that are upstream's carry its name in a +// Ported at 9daa0aa22. Functions that are upstream's carry its name in a // trailing comment so a bump can diff them one by one; everything below the // "evaluation" banner replaces server slots and is ours. diff --git a/c_src/llama_cpp_ex/llama_nif.cpp b/c_src/llama_cpp_ex/llama_nif.cpp index 645547e..68bfa66 100644 --- a/c_src/llama_cpp_ex/llama_nif.cpp +++ b/c_src/llama_cpp_ex/llama_nif.cpp @@ -531,13 +531,15 @@ LLAMA_NIF(backend_free, 0); // 1. RPC_STATUS_ASSERT is GGML_ABORT (ggml-rpc.cpp:30). Any peer crash, // network failure or malformed response terminates the OS process — the // whole BEAM. There is no error return, no retry, no reconnect. -// 2. Since #18626 (d0132a680) registration aborts too: the connect and the -// HELLO moved into rpc_dispatcher::start, which GGML_ABORTs on an -// unreachable endpoint, an unparseable one, or a version mismatch where -// ggml_backend_rpc_add_server used to return nullptr. rpc_add_server -// therefore probes the endpoint itself first (rpc_probe_hello), so a dead -// or mismatched node is still *detectable before load* and only *fatal -// during it*. +// 2. Registration used to abort too: #18626 (d0132a680) moved the connect +// and the HELLO into rpc_dispatcher::start, which GGML_ABORTed on an +// unreachable, unparseable or version-mismatched endpoint. #30415 +// (6122b566b) made those failures return nullptr again, but upstream +// still connects with no timeout and memoizes a registered endpoint +// forever, so a peer that has since gone away is only found mid-load, +// where it aborts. rpc_add_server therefore still probes the endpoint +// itself first (rpc_probe_hello, 5 s), so a dead or mismatched node is +// *detectable before load* and only *fatal during it*. #ifdef GGML_USE_RPC @@ -622,7 +624,7 @@ bool rpc_wait_until_listening(const std::string& host, const std::string& port, // the major version (the server's "client needs upgrade to protocol v%d.x"), // so a major bump stops the build here instead of failing every probe at run // time. -static_assert(RPC_PROTO_MAJOR_VERSION == 8, +static_assert(RPC_PROTO_MAJOR_VERSION == 10, "RPC protocol major changed: re-check rpc_probe_hello against " "rpc_msg_hello_req/rsp and RPC_CMD_HELLO in ggml-rpc.cpp"); @@ -775,16 +777,16 @@ rpc_add_server(ErlNifEnv* env, std::string endpoint) { #else size_t before = ggml_backend_dev_count(); - // ggml_backend_rpc_add_server aborts the VM on an unreachable or - // mismatched endpoint (see the top of this section), so check first. + // Upstream connects with no timeout and answers a memoized endpoint without + // contacting it (see the top of this section), so check first. if (!rpc_probe_hello(endpoint, 5000)) { return fine::Error(atoms::unreachable); } ggml_backend_reg_t reg = ggml_backend_rpc_add_server(endpoint.c_str()); if (!reg) { - // A worker reporting zero devices still comes back as nullptr, and - // ggml_backend_register silently no-ops on nullptr — so without this + // An unreachable worker, or one reporting zero devices, comes back as + // nullptr, and ggml_backend_register silently no-ops on nullptr — so without this // check that node simply vanishes and the model loads onto the wrong // devices. return fine::Error(atoms::unreachable); diff --git a/docs/dgx-spark.md b/docs/dgx-spark.md index 6bc25a9..fdb37e5 100644 --- a/docs/dgx-spark.md +++ b/docs/dgx-spark.md @@ -9,15 +9,19 @@ The work behind this page produced four llama.cpp branches on `nyo16/llama.cpp`, each based on `785322d17`, the commit vendored through v0.8.67. A fifth, `spark-all`, merges all four. v0.8.68 vendors `a691b2345`, which carries `draft-ctx-other-devices` as upstream #31635 (`e853ec9a8`, the -same change). The two RPC branches still apply to it unchanged; -`cuda-mmq-src1-reuse` conflicts with upstream's NVFP4 gate/up/GLU MMQ fusion -(#31296) and MoE layout change (#31047) and needs a rebase. +same change). v0.8.69 vendors `9daa0aa22`, where #30415 (`6122b566b`) makes +an unreachable endpoint fail registration instead of aborting, the effect of +`rpc-add-server-no-abort` by a different patch. `rpc-client-spin-wait` no +longer applies: it conflicts in `ggml-rpc.cpp` with #29413 (`cf92c1323`, RPC +load times) and needs a rebase, as does `cuda-mmq-src1-reuse`, which +conflicts with upstream's NVFP4 gate/up/GLU MMQ fusion (#31296) and MoE +layout change (#31047). | branch | effect on GB10 | section | |---|---|---| | `cuda-mmq-src1-reuse` | NVFP4 prefill +19–20% (src1 reuse, weight-scale fold, SWIGLU in MMQ); gpt-oss-120b prefill +8% (expert biases) | Qwen3.8-27B NVFP4, gpt-oss-120b prefill | | `rpc-client-spin-wait` | two-node decode +30% unpinned (gpt-oss-120b 44 → 57.6 tok/s, single node 59) | The cause, and a fix | -| `rpc-add-server-no-abort` | an unreachable worker returns an error instead of aborting | Five things, #1 | +| `rpc-add-server-no-abort` | an unreachable worker returns an error instead of aborting (upstream as #30415 since `9daa0aa22`) | Five things, #1 | | `draft-ctx-other-devices` | DFlash with the target split over RPC (upstream as #31635 since `a691b2345`) | Speculative decoding across two nodes | ## The machine @@ -656,13 +660,14 @@ wrong devices" into `{:error, :unreachable}`. After the load, treat the VM as th unit of restart. Real fault isolation means putting the RPC client in a separate OS process, which is a different architecture. -Upstream's registration aborts too since #18626 — an unreachable endpoint, a -listener that is not an RPC worker, a version mismatch and a malformed -`host:port` all reach `GGML_ABORT` in `rpc_dispatcher::start`. So the NIF -connects and exchanges the protocol HELLO itself before registering, with a -5-second timeout. That is what keeps `ping/1` a check rather than a crash on -this llama.cpp; the matching upstream fix is branch `rpc-add-server-no-abort` -on `nyo16/llama.cpp`. +From #18626 until #30415 (v0.8.68 and older), upstream's registration aborted +too — an unreachable endpoint, a listener that is not an RPC worker, a version +mismatch and a malformed `host:port` all reached `GGML_ABORT` in +`rpc_dispatcher::start`. #30415 turns them into a failed registration, but +upstream still connects with no timeout and answers a registered endpoint from +its cache without contacting it. So the NIF still connects and exchanges the +protocol HELLO itself before registering, with a 5-second timeout; that is what +makes `ping/1` report a peer that has gone away since it was registered. ### 2. `devices/0` order is NOT `tensor_split` order diff --git a/docs/release-guide.md b/docs/release-guide.md index 686edc9..234cc72 100644 --- a/docs/release-guide.md +++ b/docs/release-guide.md @@ -368,10 +368,36 @@ op param (#30206). The RPC client's `supports_op` refuses only `kv_rows` attention it does not know to read. Run worker and client from the same release. +Re-checked at `9daa0aa22` (b12361+7), covering the 163 commits from +`a691b2345`, as a source diff: #1, #2 and #3 are as above. +`ggml-cpu/CMakeLists.txt` was not touched. The `ggml-cuda.cu` diff is MMQ, +FP8 (#28413), graph and pool work, none of it near +`ggml_backend_cuda_comm_init`. `ggml_backend_rpc_start_server` was only +reformatted and still returns `void`. The RPC client's recursive +`serialize_graph`/`add_tensor` is unchanged, so the `+sssdcpu 256` note +above still applies (not re-measured: no Spark for this bump). + +The RPC protocol goes 8.0.1 → **10.0.0**: #29413 (`cf92c1323`, load times) +takes it to 9, and #29801 (`29dbbad2b`, +ggml prune) removes `GGML_OP_ADD1`, `GGML_OP_DIAG_MASK_INF` and +`GGML_OP_DIAG_MASK_ZERO` (`GGML_OP_COUNT` 102 → 99, every later op +renumbers) and takes it to 10. The handshake refuses a major mismatch, so a +v0.8.68 worker and a v0.8.69 client report `{:error, :unreachable}` rather +than misread each other. The `HELLO` request and response layouts are +unchanged, so `rpc_probe_hello` only needed its `static_assert` moved to 10. +#30415 (`6122b566b`) makes `rpc_dispatcher::start` return false instead of +`GGML_ABORT` on a bad endpoint, a failed connect or a failed HELLO, so +`ggml_backend_rpc_add_server` returns `nullptr` again. The NIF's probe stays: +upstream's connect has no timeout, and a registered endpoint is answered from +`reg_map` without being contacted, so only the probe notices a peer that went +away after registration. + Of the `nyo16/llama.cpp` branches, `draft-ctx-other-devices` is upstream as -#31635 (`e853ec9a8`, the same change), `rpc-add-server-no-abort` and -`rpc-client-spin-wait` still apply unchanged, and `cuda-mmq-src1-reuse` -conflicts with #31296 and #31047. +#31635 (`e853ec9a8`, the same change), `rpc-add-server-no-abort` is +superseded by #30415 (a different patch with the same effect), and +`rpc-client-spin-wait` (still not upstream: the dispatcher only spins while a +communicator holds it) conflicts with #29413 and needs a rebase, as does +`cuda-mmq-src1-reuse`, which conflicts with #31296 and #31047. | # | Upstream defect | Our workaround | Still needed? | |---|---|---|---| diff --git a/lib/llama_cpp_ex/rpc.ex b/lib/llama_cpp_ex/rpc.ex index c2228dc..2212078 100644 --- a/lib/llama_cpp_ex/rpc.ex +++ b/lib/llama_cpp_ex/rpc.ex @@ -39,11 +39,13 @@ defmodule LlamaCppEx.RPC do > to catch. > > Registration is the exception and the reason this module is shaped the way it - > is. Upstream's registration aborts too (since llama.cpp #18626), so - > `add_server/1` first connects and exchanges the protocol HELLO itself: an - > unreachable or incompatible endpoint is reported as `{:error, :unreachable}` - > rather than aborting, so a two-node setup can be *checked* before a model - > load and is only *fatal* during one. Check with `ping/1` before you load; + > is. `add_server/1` first connects and exchanges the protocol HELLO itself, + > with a 5-second timeout: an unreachable or incompatible endpoint is reported + > as `{:error, :unreachable}`, so a two-node setup can be *checked* before a + > model load and is only *fatal* during one. (Upstream's own registration + > aborted on such an endpoint from llama.cpp #18626 until #30415; it still + > connects with no timeout and never re-contacts an endpoint it has already + > registered.) Check with `ping/1` before you load; > treat the VM as the unit of restart afterwards. Real fault isolation would > mean running the RPC client in a separate OS process, which is a different > architecture. @@ -114,10 +116,11 @@ defmodule LlamaCppEx.RPC do * `{:error, :rpc_unsupported}` — the NIF was built without `LLAMA_RPC=1`. * `{:error, :unreachable}` — nothing answered within 5 seconds, the endpoint is not `host:port`, the peer's RPC protocol major/minor did not - match ours, or the worker reported no devices. Upstream aborts the VM on - the first three, and silently ignores the last one at registration, so - this check is the only thing standing between you and a crash or a model - that quietly loads onto the wrong devices. + match ours, or the worker reported no devices. Upstream returns no + registration for all four but waits on the operating system's connect + timeout, and `ggml_backend_register` silently ignores a missing + registration, so without this check the model would quietly load onto + the wrong devices. """ @spec add_server(endpoint()) :: {:ok, non_neg_integer()} | {:error, error()} def add_server(endpoint) when is_binary(endpoint) do diff --git a/mix.exs b/mix.exs index adbde62..df99b7b 100644 --- a/mix.exs +++ b/mix.exs @@ -133,7 +133,7 @@ end defmodule LlamaCppEx.MixProject do use Mix.Project - @version "0.8.68" + @version "0.8.69" @source_url "https://github.com/nyo16/llama_cpp_ex" def project do diff --git a/test/mtp_model_test.exs b/test/mtp_model_test.exs index b0cf9e0..76ab342 100644 --- a/test/mtp_model_test.exs +++ b/test/mtp_model_test.exs @@ -173,7 +173,11 @@ defmodule LlamaCppEx.MTPCancelTest do # runs aborted 3/4 (SIGSEGV, exit 139) and the fourth failed with the same # `unknown exception`. At a691b2345 (b12205) four runs aborted 2/4 (exit 134: # `GGML_ASSERT(idx == i - off)` in `common_batch`, `GGML_ASSERT(backend)` in - # `ggml-backend.cpp`) and the other two passed. + # `ggml-backend.cpp`) and the other two passed. At 9daa0aa22 (b12361+7), with + # Qwen3.6-27B (the 35B-A3B was not on the machine), seven runs aborted 4/7 + # (exit 134, every time `GGML_ASSERT(logits != nullptr)` in + # `common/sampling.cpp`, reached from the MTP drafter's + # `common_sampler_sample`) and the other three passed. # # It therefore still lives in its own module carrying only `:mtp_cancel`, and # deliberately *not* `:mtp`: `--include` beats `--exclude` in ExUnit, so a diff --git a/test/rpc_test.exs b/test/rpc_test.exs index 8909e6f..09b514c 100644 --- a/test/rpc_test.exs +++ b/test/rpc_test.exs @@ -77,10 +77,11 @@ defmodule LlamaCppEx.RPCTest do end test "a listener that is not an RPC worker is an error, never a crash" do - # Since llama.cpp #18626 upstream's own handshake aborts the process when - # the peer accepts and then answers wrongly, so this is the case the NIF's - # pre-flight HELLO exists for: one peer closes after reading the request, - # the other answers with bytes that are not a HELLO response. + # From llama.cpp #18626 until #30415 upstream's own handshake aborted the + # process when the peer accepted and then answered wrongly, so this is the + # case the NIF's pre-flight HELLO was written for: one peer closes after + # reading the request, the other answers with bytes that are not a HELLO + # response. for reply <- [:close, "HTTP/1.1 400 Bad Request\r\n\r\n"] do {:ok, listen} = :gen_tcp.listen(0, [:binary, active: false, ip: {127, 0, 0, 1}]) {:ok, port} = :inet.port(listen) diff --git a/vendor/llama.cpp b/vendor/llama.cpp index a691b23..9daa0aa 160000 --- a/vendor/llama.cpp +++ b/vendor/llama.cpp @@ -1 +1 @@ -Subproject commit a691b2345844ce1b430201fc5fd404ce062e05f2 +Subproject commit 9daa0aa22b8aa0143344d7549dee1f0e32f048bb