From 68b05037ab2184076ed3c6d6ab204a610b6dcfa1 Mon Sep 17 00:00:00 2001 From: Niko Maroulis Date: Sun, 8 Nov 2026 08:51:04 -0500 Subject: [PATCH] Bump llama.cpp to a691b2345, release v0.8.68 172 commits from 785322d17 (b12032+1) to b12205. No NIF change was needed for the bump; decision.cpp is "Ported at" a691b2345 with no hunk to port. decision_compare.exs: ALL MATCH on 9 models against llama-server. - MTP.init/2 drops the DFlash/DSpark device check: upstream #31635 gives the draft context the target's backends, verified on two Sparks. - MTP.init/2 warns when a DFlash target's output device is an RPC device: since #31506 the drafter's features live there (7.9 tok/s with ["CUDA0", "RPC0"], 32.5 with ["RPC0", "CUDA0"]). - Fix the macOS LLAMA_RPC=1 build (SOCK_CLOEXEC, since v0.8.67). - Document the :tensor-over-RPC dirty-scheduler stack overflow (+sssdcpu 256) and the re-measured -sm tensor numbers. --- CHANGELOG.md | 118 +++++++++++++++++++++++++++++++ Makefile | 2 +- bench/spark_tensor_split.exs | 7 +- c_src/llama_cpp_ex/decision.cpp | 2 +- c_src/llama_cpp_ex/llama_nif.cpp | 8 +-- docs/dgx-spark.md | 62 ++++++++++++---- docs/release-guide.md | 35 +++++++++ lib/llama_cpp_ex/model.ex | 5 ++ lib/llama_cpp_ex/mtp.ex | 65 ++++++++++------- mix.exs | 2 +- scripts/spark/e2e.exs | 54 +++++++------- test/mtp_model_test.exs | 4 +- vendor/llama.cpp | 2 +- 13 files changed, 289 insertions(+), 77 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 218e9af..8ac3247 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,123 @@ # Changelog +## v0.8.68 + +llama.cpp bumped to `a691b2345` +([`b12205`](https://github.com/ggml-org/llama.cpp/releases/tag/b12205)), +172 commits from `785322d17` (b12032+1). Upstream versions: llama.cpp 0.9.0, +ggml 0.29.0. The bump needed no NIF change. Of the headers the NIF includes: +`llama.h` adds `moe_cache_cpu_share` to `llama_context_params` (default `0`, +which keeps the old behaviour; the binding does not set the MoE cache); +`common.h` drops `common_get_path_from_env` and the Windows `utf8_to_wstring`, +neither called, and changes `common_prompt_checkpoint`'s buffers, which the +NIF does not use; `ggml.h` drops `ggml_rope_yarn_corr_dims` (not called); +`ggml-backend.h` and `src/llama-ext.h` only add. `llama_model_default_params` +is unchanged. `src/llama-vocab.cpp` and `tools/server/server-decision.*` are +untouched, so `input_tokenize.cpp` and the decision port need no change. + +Checked on an M1 Max (Metal): the default suite on a default build and on an +`LLAMA_RPC=1` build; `:smoke`, `:embeddings` (Qwen3-Embedding and +embeddinggemma-2), `:slow`, `:mtp`, `:tools`, `:decision`, `:decision_clef`, +`:mtp_sidecar` (Qwen3.8-27B) and `test/vm_exit_test.exs`; and +`scripts/decision_compare.exs` — ALL MATCH on 9 models against llama-server at +the same commit, bit-identical on every request the script checks. On two DGX +Sparks (GB10, CUDA 13.0, `LLAMA_RPC=1` build, `verify-build-flags.sh` clean): +the default suite, the same model tags plus `:dflash` and bge-small +embeddings, `vm_exit_test.exs`, `:rpc_live` against the other node, and +`scripts/spark/e2e.exs two-node` (12/12). Single-node speculative decode on +Qwen3.8-27B NVFP4 is unchanged (`bench/spark_spec.exs`: plain 14.21, MTP +`n_draft: 2` 26.55, DFlash 2 `n_draft: 7` 44.24 tok/s, against 14.28 / 26.76 / +44.22 at v0.8.67). + +### Changed + +- `LlamaCppEx.MTP.init/2` no longer refuses a DFlash/DSpark drafter that is + not loaded on every device the target uses. Upstream #31635 (the same change + as the `draft-ctx-other-devices` branch of `nyo16/llama.cpp`) gives the + draft context a backend for each of the target's devices. On two Sparks a + drafter on `CUDA0` alone beside a target on `CUDA0` + `RPC0` decodes the + same text at the same rate as one loaded on both. +- `LlamaCppEx.MTP.init/2` logs a warning when a DFlash/DSpark target's last + device is an RPC device. Upstream now keeps the target features a DFlash + drafter reads in a buffer on the target's output device (#31506), so with + the output layer on the worker every verify ships them over the network. + Qwen3.8-27B NVFP4 split 50/50 over two Sparks, DFlash 2 `n_draft: 7`: + 7.9 tok/s with `devices: ["CUDA0", "RPC0"]` (28.1 at v0.8.67, plain 13.7), + **32.5–33.0** with `["RPC0", "CUDA0"]`, which also lifts plain decode to + 14.3. `LLAMA_DFLASH_DEV=0` restores host copies (28.2–28.7). The `MTP` + docs, `docs/dgx-spark.md` and `scripts/spark/e2e.exs` now list the local + device last. +- `decision.cpp` is now "Ported at" `a691b2345`; no hunk needed porting. +- `test/mtp_model_test.exs` records `:mtp_cancel` at this commit: 2 of 4 runs + abort (exit 134, `GGML_ASSERT` in `common_batch` and in `ggml-backend.cpp`), + the other two pass. The race it documents is unchanged. +- `docs/dgx-spark.md` and `docs/release-guide.md`: of the `nyo16/llama.cpp` + branches, `draft-ctx-other-devices` is upstream; `rpc-add-server-no-abort` + and `rpc-client-spin-wait` are not and still apply; `cuda-mmq-src1-reuse` + conflicts with #31296 and #31047. + +### Fixed + +- An `LLAMA_RPC=1` build failed to compile on macOS since v0.8.67: + `rpc_probe_hello` opened its socket with `SOCK_CLOEXEC`, which macOS does + not have. It sets `FD_CLOEXEC` with `fcntl` instead. + +### Known issue + +- `split_mode: :tensor` with an RPC device in `:devices` segfaults the VM on + the first decode at this llama.cpp commit: the RPC client serializes the + meta backend's graph with a recursive walk that overflows a dirty CPU + scheduler's default 320 KB stack (Qwen3-8B over two Sparks; upstream's + `llama-bench` runs on an 8 MB main thread and is not affected). Start the + VM with `+sssdcpu 256` if you use it. `:layer`, the mode the docs recommend + across hosts, is not affected. Measured with `llama-bench` from the + submodule (gpt-oss-120b, two Sparks), `-sm tensor` decodes at 15.2 tok/s + against 50.4 for `-sm layer` (tensor was 8.9 at `785322d17`). + +### Upstream + +- From the 172 commits: + - Flash attention over a unified KV cache attends to each sequence's own + cells (`ggml_flash_attn_ext_rows`, #29510) instead of the whole cache with + a dense mask. `LlamaCppEx.Server` uses a unified cache by default, and + `LlamaCppEx.Decision` shares a prompt through one. `LLAMA_KV_ROWS_DISABLE` + turns it off. RPC: the op travels as `GGML_OP_FLASH_ATTN_EXT` with an + extra source and the protocol version did not change, so run workers and + clients from the same release. + - On-device sequence state copies of a quantized KV cache copied only part + of each range (#30541). The MTP loop's checkpoint rollback saves state + with `LLAMA_STATE_SEQ_FLAGS_ON_DEVICE`, so it was affected with `:type_k` + / `:type_v` set to a quantized type. + - A context with `ctx_other` gets backends for the other model's devices, + and the Gemma 4 assistant follows its target's flash-attention setting + (#31635). DFlash copies hidden features on the device (#31506). MTP saves + and restores its hidden state with a checkpoint (#30141). + - KV shifting fix and per-model RoPE (#31683); a cache that shares another's + cells no longer reports 0 KV rows before its sequences have any (#31570). + - The CPU backend computes F16 matmuls with F32 activations (#31854), so CPU + results move slightly; Metal follows the reference precision of + `ggml_prec_set_acc` (#31851). + - Gemma 4 E2B/E4B prompt processing prunes rows that need no logits + (#31356). + - Chat: invalid UTF-8 in PEG-parsed tool call arguments is sanitized + (#31861); a constraint type is allowed after gpt-oss's final channel + (#31556); jinja implements `map('filter')` (#24033), and `upper`, `lower`, + `strip`, `title`, `capitalize` and `join` now reject unexpected arguments. + - New architectures: Kimi-K3.1-Pro (#31779), Ripple (#31692), xing4_0 + (#31589), Metis-1.6 (#31134), Clover (#31306); NVFP4 LFM3 (#31608) and + NVFP4 scales in qwen2's FFN (#31657). + - Metal: GROUP_NORM fix (#31852), non-contiguous copy fix (#31581), + `mul_mm` for 32 ≤ K < 64 (#30982), fusable node groups kept in one command + buffer (#31688), DeepSeek-V4 and GLM MoE routing fused (#31615). + - CUDA: NVFP4 gate/up/GLU fused in MMQ (#31296), linearized MoE layout for + MMQ/MMF (#31047), a Q8_1 correction-sum fix (#31247), NaN sanitizing in MoE + selection (#31005), graphs no longer evicted just before use (#30662). + - RPC: NCCL for multi-node tensor parallelism (#28967) — the RPC server's + communicator uses `ggml_backend_comm_init_rank` when its backend has it. + Not measured on two Sparks for this release. + - Apple host buffers use `vm_allocate` (#31437). Vulkan, SYCL, OpenCL, + Hexagon, WebGPU and zDNN work. + ## v0.8.67 llama.cpp stays at `785322d17`. **Behaviour change:** `LlamaCppEx.Model.load/2` diff --git a/Makefile b/Makefile index a4a93cd..dce8271 100644 --- a/Makefile +++ b/Makefile @@ -36,7 +36,7 @@ endif # Pinned llama.cpp commit, used when vendor/llama.cpp has to be cloned. MUST # match the vendor/llama.cpp submodule; bump both together, see # docs/release-guide.md. Override to build the NIF against another revision. -LLAMA_COMMIT ?= 785322d17f2d51e5185cf996a1f4e46d87640841 +LLAMA_COMMIT ?= a691b2345844ce1b430201fc5fd404ce062e05f2 # The commit actually on disk. A submodule can be bumped without LLAMA_COMMIT # following it, and the build has to key off what is really there. diff --git a/bench/spark_tensor_split.exs b/bench/spark_tensor_split.exs index 169f840..1294490 100644 --- a/bench/spark_tensor_split.exs +++ b/bench/spark_tensor_split.exs @@ -6,9 +6,12 @@ Code.require_file("helpers.exs", __DIR__) # scripts/spark/remote.sh --env MIX_ENV=bench spark-1 \ # mix run bench/spark_tensor_split.exs local # -# # 6.2 — two nodes, Meta device over [CUDA0, RPC0] +# # 6.2 — two nodes, Meta device over [CUDA0, RPC0]. Since a691b2345 the RPC +# # client's recursive graph serialization overflows a dirty scheduler's +# # default stack on the tensor leg (VM segfault), hence +sssdcpu. # scripts/spark/rpc-worker.sh start spark-2 -# scripts/spark/remote.sh --env MIX_ENV=bench --env LLAMA_RPC=1 spark-1 \ +# scripts/spark/remote.sh --env MIX_ENV=bench --env LLAMA_RPC=1 \ +# --env ELIXIR_ERL_OPTIONS="+sssdcpu 256" spark-1 \ # mix run bench/spark_tensor_split.exs remote # # "tp=2" is vLLM vocabulary. In llama.cpp at b10362 it means diff --git a/c_src/llama_cpp_ex/decision.cpp b/c_src/llama_cpp_ex/decision.cpp index 8642d0f..bcd2f1a 100644 --- a/c_src/llama_cpp_ex/decision.cpp +++ b/c_src/llama_cpp_ex/decision.cpp @@ -2,7 +2,7 @@ // of tools/server/server-context.cpp (send_decision, the batch layout rules and // the /v1/systemone handler). See decision.h for scope. // -// Ported at 785322d17. Functions that are upstream's carry its name in a +// Ported at a691b2345. Functions that are upstream's carry its name in a // trailing comment so a bump can diff them one by one; everything below the // "evaluation" banner replaces server slots and is ours. diff --git a/c_src/llama_cpp_ex/llama_nif.cpp b/c_src/llama_cpp_ex/llama_nif.cpp index 454aa0f..645547e 100644 --- a/c_src/llama_cpp_ex/llama_nif.cpp +++ b/c_src/llama_cpp_ex/llama_nif.cpp @@ -690,8 +690,10 @@ bool rpc_probe_hello(const std::string& endpoint, int timeout_ms) { bool ok = false; for (addrinfo* ai = res; ai && !ok; ai = ai->ai_next) { - int fd = socket(ai->ai_family, ai->ai_socktype | SOCK_CLOEXEC, ai->ai_protocol); + // SOCK_CLOEXEC is Linux/BSD only; macOS has no such socket() flag. + int fd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol); if (fd < 0) continue; + fcntl(fd, F_SETFD, FD_CLOEXEC); int flags = fcntl(fd, F_GETFL, 0); fcntl(fd, F_SETFL, flags | O_NONBLOCK); @@ -1083,10 +1085,6 @@ int64_t model_n_layer_nextn(ErlNifEnv* env, fine::ResourcePtr model) LLAMA_NIF(model_n_layer_nextn, 0); // Names of the devices the model's layers were placed on, in placement order. -// LlamaCppEx.MTP uses it to refuse a DFlash drafter that cannot reach the -// target's embeddings and LM head (shared via ctx_other): a draft context -// without the device holding them hits "pre-allocated tensor ... in a buffer -// that cannot run the operation", a GGML_ABORT. std::vector model_devices(ErlNifEnv* env, fine::ResourcePtr model) { std::vector names; const int32_t n = llama_model_n_devices(model->model); diff --git a/docs/dgx-spark.md b/docs/dgx-spark.md index c986e77..6bc25a9 100644 --- a/docs/dgx-spark.md +++ b/docs/dgx-spark.md @@ -6,15 +6,19 @@ inferred from spec sheets; where a number differs from the marketing figure, the measurement wins and the reason is given. The work behind this page produced four llama.cpp branches on -`nyo16/llama.cpp`, each based on `785322d17`, the vendored commit. None of them -is in the vendored tree. A fifth, `spark-all`, merges all four. +`nyo16/llama.cpp`, each based on `785322d17`, the commit vendored through +v0.8.67. A fifth, `spark-all`, merges all four. v0.8.68 vendors `a691b2345`, +which carries `draft-ctx-other-devices` as upstream #31635 (`e853ec9a8`, the +same change). The two RPC branches still apply to it unchanged; +`cuda-mmq-src1-reuse` conflicts with upstream's NVFP4 gate/up/GLU MMQ fusion +(#31296) and MoE layout change (#31047) and needs a rebase. | branch | effect on GB10 | section | |---|---|---| | `cuda-mmq-src1-reuse` | NVFP4 prefill +19–20% (src1 reuse, weight-scale fold, SWIGLU in MMQ); gpt-oss-120b prefill +8% (expert biases) | Qwen3.8-27B NVFP4, gpt-oss-120b prefill | | `rpc-client-spin-wait` | two-node decode +30% unpinned (gpt-oss-120b 44 → 57.6 tok/s, single node 59) | The cause, and a fix | | `rpc-add-server-no-abort` | an unreachable worker returns an error instead of aborting | Five things, #1 | -| `draft-ctx-other-devices` | DFlash with the target split over RPC | Speculative decoding across two nodes | +| `draft-ctx-other-devices` | DFlash with the target split over RPC (upstream as #31635 since `a691b2345`) | Speculative decoding across two nodes | ## The machine @@ -951,19 +955,39 @@ harness above, so acceptance is not comparable with the one-node table): | DFlash 2 `n_draft: 7` | **22.2–22.4** | 27% | **27.8–27.9** (29%) | DFlash still pays 1.6× across two nodes, and 1.9× with the RPC client fix. With -the fix, plain decode over two nodes beats one node's 14.3. The drafter must be -loaded on every device the target uses, because it computes with the target's -embeddings and LM head: with the target on `["CUDA0", "RPC0"]` and the drafter on `CUDA0` -alone, upstream aborts the process ("pre-allocated tensor (output.weight) in -a buffer (RPC0) that cannot run the operation") — `llama-server ---spec-draft-device CUDA0` does it too. `MTP.init/2` refuses that pairing; -load the drafter with the target's `:devices` and `tensor_split: [1.0, 0.0]` -to keep its own layers local. - -The upstream fix is branch `draft-ctx-other-devices` on `nyo16/llama.cpp`: a +the fix, plain decode over two nodes beats one node's 14.3. Up to v0.8.67 the +drafter had to be loaded on every device the target uses, because it computes +with the target's embeddings and LM head: with the target on `["CUDA0", "RPC0"]` +and the drafter on `CUDA0` alone, upstream aborted the process ("pre-allocated +tensor (output.weight) in a buffer (RPC0) that cannot run the operation"), and +`MTP.init/2` refused that pairing. The fix, first on branch +`draft-ctx-other-devices` of `nyo16/llama.cpp`, is upstream as #31635: a context with `ctx_other` gets a backend for each of the other model's devices. -With it, that same `llama-server` command runs, at 26.4 tok/s (566 drafted, -173 accepted) on one 256-token completion. +Since v0.8.68 the drafter can sit on `CUDA0` alone, and decodes the same text +at the same rate as one loaded on both devices. + +#### Put the local device last + +At `a691b2345` DFlash reads the target's layer features from a device buffer +(#31506) instead of host copies, and that buffer lives on the target's +*output* device. Listed `["CUDA0", "RPC0"]`, the output layer is on the worker, +so every verify ships the features over the fabric. Same Qwen3.8-27B NVFP4 +50/50 split, one 256-token greedy chat completion ("Explain how a hash map +handles collisions…"), three runs, client on the A725 cores, decode rate from +first to last token: + +| target `:devices` | build | plain | DFlash 2 `n_draft: 7` | verify ms/iter | +|---|---|---|---|---| +| `["CUDA0", "RPC0"]` | v0.8.67 (`785322d17`) | 13.75 | 27.4–28.1 | 94–96 | +| `["CUDA0", "RPC0"]` | v0.8.68 (`a691b2345`) | 13.7 | **7.8–7.9** | 395–400 | +| `["CUDA0", "RPC0"]`, `LLAMA_DFLASH_DEV=0` | v0.8.68 | 13.75 | 28.2–28.7 | 93–95 | +| **`["RPC0", "CUDA0"]`** | v0.8.68 | **14.3** | **32.5–33.0** | 86–88 | + +With the local device last the buffer stays on `CUDA0` and DFlash is faster +than before the change; plain decode gains 4% too, since the logits no longer +cross the fabric. `MTP.init/2` logs a warning for a DFlash/DSpark target whose +last device is an RPC device. `LLAMA_DFLASH_DEV=0` is upstream's switch back +to host copies. Acceptance is 34.5% in every row. ### Concurrency across two nodes @@ -1051,6 +1075,14 @@ communicator's own spin left blocking, decodes at **24.1** with pp512 1553. That is 2.7× better, and still less than half of a layer split's 57.6. The verdict below stands. +At `a691b2345` (v0.8.68), same `llama-bench` run from the submodule, client on +the A725 cores against a worker from the same commit: `-sm tensor` decodes at +15.2 tok/s with pp512 850, `-sm layer` at 50.4 with pp512 1737. #28967 lets an +RPC worker's pairwise communicator use NCCL through +`ggml_backend_comm_init_rank`, but the client's `CUDA0` beside `RPC0` still +cannot. Through the binding, `:tensor` over RPC also needs a larger dirty +scheduler stack at this commit (`+sssdcpu 256`); see `LlamaCppEx.Model`. + ## Why, and why the comm-mode knob cannot help `ggml_backend_cuda_comm_init` returns `nullptr` the moment **any** member backend diff --git a/docs/release-guide.md b/docs/release-guide.md index 1fd6c5b..bd4f347 100644 --- a/docs/release-guide.md +++ b/docs/release-guide.md @@ -338,6 +338,41 @@ Re-checked at `785322d17` (b12032+1), covering the 31 commits from RPC protocol and op ids are unchanged (#31424 removes `fs_path_to_utf8` from `tools/rpc`, not from the backend). +Re-checked at `a691b2345` (b12205), covering the 172 commits from +`785322d17`: #1 and #3 are as above — `ggml-cpu/CMakeLists.txt` was not +touched and `ggml_backend_rpc_start_server` keeps its `void` signature. #2 +stands: #28967 adds `ggml_backend_comm_init_rank` (NCCL in the CUDA backend) +and the RPC server's pairwise communicator uses it when the worker's backend +has it, but `ggml_backend_cuda_comm_init` still returns `nullptr` on a +non-CUDA member. Measured with upstream's `llama-bench` built from the +submodule (gpt-oss-120b, `-fa 1 -p 512 -n 64 -r 3`, client on the A725 +cores): `-sm layer` pp512 1737, tg64 50.4; `-sm tensor` pp512 850, tg64 15.2, +against 1135 / 8.9 at `785322d17`. Better, still 30% of a layer split. + +Through the binding, `:tensor` with an RPC device in the set segfaults the VM +at this commit: the RPC client's `serialize_graph` walks the meta backend's +graph with the recursive `add_tensor`, which overflows a dirty CPU +scheduler's default 320 KB stack (40 kilowords) on Qwen3-8B. `llama-bench` +runs it on an 8 MB main thread and does not notice. `+sssdcpu 256` (2 MB) is +enough; `bench/spark_tensor_split.exs remote` needs +`ELIXIR_ERL_OPTIONS="+sssdcpu 256"`. A layer split is unaffected. Re-check +on a bump by running that bench without the flag. + +The RPC protocol stays 8.0.1 and `GGML_OP_COUNT` 102, yet two graph changes +ride on existing ops: `ggml_flash_attn_ext_rows` (#29510) is +`GGML_OP_FLASH_ATTN_EXT` with K/V row indices in `src[5]`, used when the KV +cache is unified, flash attention is on and the device passes the op probe +(`LLAMA_KV_ROWS_DISABLE` turns it off), and RoPE gains a `truncate` +op param (#30206). The RPC client's `supports_op` refuses only +`truncate=false` RoPE, so a worker from v0.8.67 or older is offered +`kv_rows` attention it does not know to read. Run worker and client from the +same release. + +Of the `nyo16/llama.cpp` branches, `draft-ctx-other-devices` is upstream as +#31635 (`e853ec9a8`, the same change), `rpc-add-server-no-abort` and +`rpc-client-spin-wait` still apply unchanged, and `cuda-mmq-src1-reuse` +conflicts with #31296 and #31047. + | # | Upstream defect | Our workaround | Still needed? | |---|---|---|---| | 1 | `GGML_NATIVE=ON` makes ggml's `-mcpu=native` probe resolve to **base ARMv8-A** on Cortex-X925/A725 with GCC 13.3 — silently, with a soft CMake warning and exit 0. Costs every `sdot`/`smmla`/SVE kernel. | `LLAMA_CPU_ARM_ARCH` + `LLAMA_CUDA_ARCH` in the `Makefile`, which must be set together. See [DGX Spark](dgx-spark.md) and [Cross-Platform Builds](cross-platform-builds.md). | `scripts/spark/verify-build-flags.sh` on an aarch64 host. If a default build (no `LLAMA_CPU_ARM_ARCH`) now reports non-zero `sdot`/`smmla`, upstream fixed the probe. | diff --git a/lib/llama_cpp_ex/model.ex b/lib/llama_cpp_ex/model.ex index 7b647a6..55ec187 100644 --- a/lib/llama_cpp_ex/model.ex +++ b/lib/llama_cpp_ex/model.ex @@ -98,6 +98,11 @@ defmodule LlamaCppEx.Model do > disables backend sampling, and its CUDA all-reduce is `ncclCommInitAll` — > a single-process, all-local-GPUs API. **It cannot span hosts**, so it is not > "tp=2 across two machines". See `docs/dgx-spark.md` for the measurements. + > With an RPC device in the set it runs, far slower than `:layer`, and at + > llama.cpp `a691b2345` its first decode overflows a dirty scheduler's + > default stack (320 KB) while the RPC backend serializes the graph, which + > kills the VM with a segfault. Start the VM with `+sssdcpu 256` (2 MB, + > e.g. `ELIXIR_ERL_OPTIONS="+sssdcpu 256"`) if you run it anyway. > #### Load mode {: .info} > diff --git a/lib/llama_cpp_ex/mtp.ex b/lib/llama_cpp_ex/mtp.ex index 98dc9d0..59f87d7 100644 --- a/lib/llama_cpp_ex/mtp.ex +++ b/lib/llama_cpp_ex/mtp.ex @@ -92,6 +92,7 @@ defmodule LlamaCppEx.MTP do """ alias LlamaCppEx.{Context, Model, Sampler, Tokenizer} + require Logger @enforce_keys [:main_ctx, :mtp_ctx, :spec_ref, :n_draft] defstruct [:main_ctx, :mtp_ctx, :spec_ref, :n_draft] @@ -129,12 +130,16 @@ defmodule LlamaCppEx.MTP do * `:dspark` - a DSpark drafter (DFlash plus a Markov head), passed as `:draft_model`. * `:draft_model` - A separate `LlamaCppEx.Model` holding the drafter. - Required for `:dflash` and `:dspark`, and loaded on every device the - target uses (`LlamaCppEx.Model.devices/1`): a DFlash drafter computes - with the target's embeddings and LM head. For a target split across an - RPC worker, load the drafter with the same `:devices` and a - `:tensor_split` that keeps its layers local. For `:mtp` it is the sidecar head - for checkpoints that ship one rather than embedding it (Qwen 3.8: + Required for `:dflash` and `:dspark`. A DFlash drafter computes with the + target's embeddings and LM head; its context gets a backend for every + device the target uses, so the drafter can sit on a subset of them + (e.g. `CUDA0` alone beside a target split onto an RPC worker). The + drafter reads the target's features from the target's output device, + so a target split onto an RPC worker should list the local device last + (`devices: ["RPC0", "CUDA0"]`); with the RPC device last, every verify + ships them over the network and DFlash decodes slower than plain + (`init/2` logs a warning). For `:mtp` it is the sidecar head for + checkpoints that ship one rather than embedding it (Qwen 3.8: `Qwen3.8-27B-Q4_K_M.gguf` plus `mtp-Qwen3.8-27B-Q4_0.gguf`; Gemma4 E4B: a `gemma4` target plus a `gemma4-assistant` head), loaded with `load_mtp: true`. Defaults to `nil`, meaning the MTP head is expected @@ -159,10 +164,33 @@ defmodule LlamaCppEx.MTP do with :ok <- validate_n_draft(n_draft), {:ok, head_model} <- validate_drafter(type, model, draft_model) do + warn_remote_output(type, model) do_init(model, head_model, opts, n_draft, type) end end + # Upstream (#31506) keeps the target-layer features a DFlash/DSpark drafter + # reads in a buffer on the target's output device. When that device is an RPC + # worker, every verify moves the features over the network: Qwen3.8-27B NVFP4 + # split over two GB10s decoded at 7.9 tok/s with ["CUDA0", "RPC0"] against + # 32.5 with ["RPC0", "CUDA0"], plain decode 13.7. + defp warn_remote_output(type, model) when type in [:dflash, :dspark] do + last = List.last(Model.devices(model)) + + if last && last in Enum.map(LlamaCppEx.RPC.devices(), & &1.name) do + Logger.warning( + "LlamaCppEx.MTP: the target's output layer is on RPC device #{last}, so " <> + "#{type} reads the target's features over the network and decodes several " <> + "times slower than plain. List the local device last in the target's " <> + ":devices (e.g. [\"RPC0\", \"CUDA0\"])." + ) + end + + :ok + end + + defp warn_remote_output(_type, _model), do: :ok + defp validate_n_draft(n) when is_integer(n) and n > 0, do: :ok defp validate_n_draft(_), do: {:error, ":n_draft must be a positive integer"} @@ -172,25 +200,12 @@ defmodule LlamaCppEx.MTP do # in the NIF, where the model's target-layer metadata is readable: upstream's # constructor asserts on it, so it has to be refused before init. # - # The device check has to happen here, before the draft context exists. A - # DFlash drafter computes with the target's token embeddings and LM head - # (shared through ctx_other), so its context needs a backend for every device - # the target placed weights on. A target split onto an RPC worker with the - # drafter on CUDA0 alone aborted the VM with "pre-allocated tensor - # (output.weight) in a buffer (RPC0) that cannot run the operation". - defp validate_drafter(type, target, %Model{} = draft) when type in [:dflash, :dspark] do - case Model.devices(target) -- Model.devices(draft) do - [] -> - {:ok, draft} - - missing -> - {:error, - "the drafter reads the target's embeddings and LM head, so it must be loaded " <> - "on every device the target uses; it is missing #{Enum.join(missing, ", ")}. " <> - "Load it with the target's :devices (a :tensor_split can still keep its " <> - "own layers on one of them)"} - end - end + # Device placement is not checked. Before upstream #31635 (vendored since + # a691b2345) a drafter missing one of the target's devices aborted the VM + # ("pre-allocated tensor (output.weight) in a buffer (RPC0) that cannot run + # the operation"); the draft context now gets the target's backends too. + defp validate_drafter(type, _target, %Model{} = draft) when type in [:dflash, :dspark], + do: {:ok, draft} defp validate_drafter(type, _target, nil) when type in [:dflash, :dspark], do: {:error, "type: #{inspect(type)} needs the drafter GGUF passed as draft_model:"} diff --git a/mix.exs b/mix.exs index 046f112..adbde62 100644 --- a/mix.exs +++ b/mix.exs @@ -133,7 +133,7 @@ end defmodule LlamaCppEx.MixProject do use Mix.Project - @version "0.8.67" + @version "0.8.68" @source_url "https://github.com/nyo16/llama_cpp_ex" def project do diff --git a/scripts/spark/e2e.exs b/scripts/spark/e2e.exs index 78b71bd..594d908 100644 --- a/scripts/spark/e2e.exs +++ b/scripts/spark/e2e.exs @@ -357,42 +357,27 @@ case System.argv() do "models/z-lab/Qwen3.8-27B-DFlash2-GGUF/main/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" ) + # Local device last, so the output layer and the DFlash feature buffer + # stay on CUDA0 (see LlamaCppEx.MTP's :draft_model docs). {:ok, target} = LlamaCppEx.Model.load(big, - devices: ["CUDA0", "RPC0"], + devices: ["RPC0", "CUDA0"], split_mode: :layer, tensor_split: [0.5, 0.5], n_gpu_layers: -1 ) - E2E.step("DFlash drafter on CUDA0 alone is refused, not aborted", fn -> - {:ok, local_only} = LlamaCppEx.Model.load(drafter, devices: ["CUDA0"], n_gpu_layers: -1) - - case LlamaCppEx.MTP.init(target, type: :dflash, draft_model: local_only, n_ctx: 2048) do - {:error, msg} -> if msg =~ "RPC0", do: {:ok, "names RPC0"}, else: {:error, msg} - other -> {:error, inspect(other)} - end - end) - - E2E.step("DFlash over the split target", fn -> - {:ok, d} = - LlamaCppEx.Model.load(drafter, - devices: ["CUDA0", "RPC0"], - split_mode: :layer, - tensor_split: [1.0, 0.0], - n_gpu_layers: -1 - ) + {:ok, prompt} = + LlamaCppEx.Chat.apply_template( + target, + [%{role: "user", content: "What is the capital of France? One word."}], + enable_thinking: false + ) + dflash = fn d -> {:ok, spec} = LlamaCppEx.MTP.init(target, type: :dflash, draft_model: d, n_draft: 7, n_ctx: 2048) - {:ok, prompt} = - LlamaCppEx.Chat.apply_template( - target, - [%{role: "user", content: "What is the capital of France? One word."}], - enable_thinking: false - ) - {:ok, text} = LlamaCppEx.MTP.generate(spec, prompt, max_tokens: 8, temp: 0.0) stats = LlamaCppEx.MTP.stats(spec) @@ -401,6 +386,25 @@ case System.argv() do {:ok, "#{inspect(String.trim(text))}, #{stats.drafts_accepted}/#{stats.drafts_generated} drafts"}, else: {:error, inspect(text)} + end + + # Upstream #31635: the draft context gets a backend for every device of the + # target, so a drafter on CUDA0 alone no longer aborts the VM. + E2E.step("DFlash, drafter on CUDA0 alone", fn -> + {:ok, local_only} = LlamaCppEx.Model.load(drafter, devices: ["CUDA0"], n_gpu_layers: -1) + dflash.(local_only) + end) + + E2E.step("DFlash, drafter on the target's devices", fn -> + {:ok, d} = + LlamaCppEx.Model.load(drafter, + devices: ["CUDA0", "RPC0"], + split_mode: :layer, + tensor_split: [1.0, 0.0], + n_gpu_layers: -1 + ) + + dflash.(d) end) GenServer.stop(local) diff --git a/test/mtp_model_test.exs b/test/mtp_model_test.exs index d88ce9e..b0cf9e0 100644 --- a/test/mtp_model_test.exs +++ b/test/mtp_model_test.exs @@ -171,7 +171,9 @@ defmodule LlamaCppEx.MTPCancelTest do # 2/4 (exit 134); the other two failed with the same `unknown exception` and # `"prompt decode failed: code=-2"` as above. At 785322d17 (b12032+1) four # runs aborted 3/4 (SIGSEGV, exit 139) and the fourth failed with the same - # `unknown exception`. + # `unknown exception`. At a691b2345 (b12205) four runs aborted 2/4 (exit 134: + # `GGML_ASSERT(idx == i - off)` in `common_batch`, `GGML_ASSERT(backend)` in + # `ggml-backend.cpp`) and the other two passed. # # It therefore still lives in its own module carrying only `:mtp_cancel`, and # deliberately *not* `:mtp`: `--include` beats `--exclude` in ExUnit, so a diff --git a/vendor/llama.cpp b/vendor/llama.cpp index 785322d..a691b23 160000 --- a/vendor/llama.cpp +++ b/vendor/llama.cpp @@ -1 +1 @@ -Subproject commit 785322d17f2d51e5185cf996a1f4e46d87640841 +Subproject commit a691b2345844ce1b430201fc5fd404ce062e05f2