diff --git a/.claude/plans/release-v0-8-64/plan.md b/.claude/plans/release-v0-8-64/plan.md new file mode 100644 index 0000000..795528e --- /dev/null +++ b/.claude/plans/release-v0-8-64/plan.md @@ -0,0 +1,155 @@ +# Plan: Release v0.8.64 (no llama.cpp bump) + +**Depth**: quick · **Status**: completed · **Date**: 2026-10-31 → 2026-11-01 + +## Goal + +Publish v0.8.64 to Hex with precompiled NIFs for all 8 targets. It ships the +changes merged after v0.8.63 (CHANGELOG "Unreleased"), with llama.cpp staying +at `418f905a4`: + +- `LlamaCppEx.set_log_level/1`, `config :llama_cpp_ex, log_level:`, DEBUG + filtered by default (#114) +- `:tools` tag in CI against pinned Qwen3.5-0.8B (#114) +- `elixir_make ~> 0.10.1` (#114); req `~> 0.7` allowed, dep lock updates (#113) +- lock: decimal 3.2.0 / ecto 3.14.2 (#114) + +Done means: release not a draft, 8 assets, `checksum.exs` on master, Hex +0.8.64, and the `/tmp` end-to-end smoke passing against the published +package with the `~/Downloads` models. + +## Decisions + +- **No llama.cpp bump** (user). There are 22 upstream commits on master; + they belong in a separate bump release. So: no header diff, no + decision-port diff, no tokenizer-partition re-diff, no + `decision_compare.exs` rerun. No decision code changed since v0.8.63, which + was compared ALL MATCH. +- **Merge #115 first** (user). It is docs only and `.claude/` and `CLAUDE.md` + are not in the Hex `files` list, so the package is unaffected. +- **Release via PR then tag**, as for v0.8.60–v0.8.63; the version bump and + changelog go through CI before the tag. + +## Phase 1 — Prepare master + +- [x] [git] Merge #115 (squash, delete branch); `git pull` master — first + Inference smoke failed on a pre-existing test race (next item); re-run + green; merged as 3b63b52 +- [x] [test] Fix flaky `ServerSmokeTest` "halting a stream early …": + `attach_collector/1` was a global handler; it caught the previous + test's lingering server's `:done` (39 prompt tokens, `max_tokens`), + which `finish_slot/3` emits after replying. Collectors in + `server_smoke_test.exs` and the telemetry test in `llama_cpp_ex_test.exs` + now pin `metadata.server`; test-only, no CHANGELOG entry +- [x] [git] Branch `release-v0.8.64` from master (3b63b52) +- [x] [config] `mix.exs` `@version "0.8.63"` → `"0.8.64"` +- [x] [docs] CHANGELOG: rename `## Unreleased` → `## v0.8.64`, add an + intro line ("llama.cpp stays at `418f905a4`"), keep the existing + Added/Changed entries; #115 is docs only and gets no entry +- [x] [verify] `mix compile --warnings-as-errors`, `mix format --check-formatted`, + `mix credo --strict`, `mix dialyzer` (0 errors) — all clean + +## Phase 2 — Local test matrix (Metal, `GGML_METAL_NO_RESIDENCY=1`) + +Unset `LLAMA_BACKEND` only where a precompiled path is under test; here +everything is a local build (`LLAMA_BACKEND=metal`). + +- [x] [test] default: `mix test` — 436 passed, 194 excluded +- [x] [test] CI's set: `--include smoke --include embeddings --include slow + --include decision --include tools` with SmolLM2, bge-small, tinylaya, + tinyopenjev, Qwen3.5-0.8B — 601 passed, 29 excluded +- [x] [test] release-guide set: `--include smoke --include embeddings + --include slow --include mtp` with Qwen3.5-0.8B, Qwen3-Embedding-0.6B-f16, + Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL — 585 passed, 45 excluded +- [x] [test] `--include embeddings` with embeddinggemma-2-Q8_0 — 447 passed +- [x] [test] `--include decision_clef` with Cloudflare_clef-flash-Q8_0 — 440 + passed (4/4 clef) +- [x] [test] `--include tools` with Llama-3.2-1B-Instruct (expect 10/11: + the round trip calls the tool again; documented, not a regression) — + 446/447, the round-trip test as expected; Qwen3.5 on Metal 447 +- [x] [test] `--include mtp_sidecar` with Qwen3.8-27B-Q4_K_M + + mtp-Qwen3.8-27B-Q4_0 (E4B pair not on disk, so that module skips) — + 442 passed, 6 skipped +- [x] [test] LLAMA_BACKEND=cpu `--include tools` (what CI runs) — 447 passed +- [x] [verify] Hex source build: `mix hex.build`, unpack into a temp dir, + `LLAMA_BACKEND=cpu mix compile`, confirm `vendor/llama.cpp` HEAD == + `418f905a4` and `input_tokenize.{h,cpp}` are in the tarball — 332K + package, build fetched `418f905a4`, NIF built +- [x] [docs] CHANGELOG v0.8.64: add the "Verified on …" paragraph with the + exact counts from this phase (format as in v0.8.63) + +## Phase 3 — PR and CI + +- [x] [git] Commit "Release v0.8.64", push, open PR — #117 (2 commits: + e081cbf test fix, 008f7bf release) +- [x] [ci] Watch CI: Setup, Tests, Credo, Dialyzer, Formatting, Inference + smoke (now including `:tools`), CUDA link cu12/cu13 — all green + (run 41250391316) +- [x] [ci] Confirm in the Inference smoke log: `Including tags: [... :tools]` + and 0 failures; no `xref: [exclude` annotation — 630 tests, 0 failures; + no xref annotations +- [x] [git] Merge PR (ask the user before merging), pull master — user chose + squash; 4f47d41 + +## Phase 4 — Tag and publish + +- [x] [release] `git tag v0.8.64` on the merge commit; `git push origin v0.8.64` +- [x] [release] Watch `Precompile NIFs`: 2× macos-15 Metal (log shows + `minos: 14.0`), 2× CPU, 4× CUDA (cu12/cu13 legs take ~50–110 min), + then "Checksum and publish the release" and "Publish to Hex" — run + 41255669230 all green in 85 min; both Metal legs `minos: 14.0` +- [x] [release] Verify: `gh release view v0.8.64` → not draft, 8 assets; + master has "Update checksum.exs for v0.8.64"; `mix hex.info llama_cpp_ex` + lists 0.8.64 — draft=false, 8 assets, 1041099, `~> 0.8.64` +- [x] [git] `git pull` master (picks up the checksum commit) + +## Phase 5 — End-to-end smoke from Hex (`/tmp`, `~/Downloads`) + +- [x] [e2e] Fresh project: `rm -rf /tmp/llama_e2e_0864 && mix new`, deps + `{:llama_cpp_ex, "== 0.8.64"}`, `{:req, "~> 0.7"}` +- [x] [e2e] `env -u LLAMA_BACKEND mix compile` → "Downloading precompiled NIF" + for `nif-2.18-aarch64-apple-darwin-0.8.64`, checksum passes, `otool` + minos 14.0; `mix.lock` resolves elixir_make 0.10.1 — all as expected + (httpc `max_header_size` notice, known) +- [x] [e2e] Copy `/tmp/llama_e2e/e2e.exs`; add log-level checks: + `set_log_level(:none)` before `init/0` → a load + tool call writes 0 + stderr lines; default level → the tool call writes no "awaiting trigger" + line; invalid level raises `ArgumentError` — child VM via + `loglevel.exs`, stderr to file +- [x] [e2e] Run with `GGML_METAL_NO_RESIDENCY=1`: all checks PASS (generate, + stream, chat, json_schema, #108 ×3 incl. server and Llama 3.2, #110 ×2, + #111 ×6, embeddings, #109 ×3 incl. Clef noul > 0.5, log level ×3) — + 23 passed, 0 failed; clef noul 0.98; :none 0 stderr lines, default 198 +- [x] [e2e] Optional: repeat compile on a fresh project with + `LLAMA_BACKEND=cpu` to confirm the Hex source build from the published + tarball — built at `418f905`, CPU generate " Paris." + +## Verification summary + +| Gate | Command / evidence | +|---|---| +| Compile/lint | `mix compile --warnings-as-errors`, format, credo --strict, dialyzer | +| Local tests | Phase 2 matrix, counts recorded in CHANGELOG | +| CI | PR checks all green, Inference smoke ran `:tools` | +| Release | workflow green, release published with 8 assets, Hex 0.8.64 | +| Published package | `/tmp` e2e: precompiled download + all checks PASS | + +## Iron Law check + +No application code changes in this plan (version, changelog only). Iron Law +#27 (no special-parsed tokenization of rendered prompts) is unaffected; the +e2e #108/#109 checks re-verify it on the published artifact. + +## Risks + +- **CUDA legs are slow** (47–110 min) and can hit runner capacity; the + release stays a draft until all 8 legs pass, so nothing partial reaches Hex. +- **macos-15 arm64 capacity**: the workflow annotated queueing delays for + v0.8.63, so expect waits. +- **Flaky small-model tool tests on CI CPU**: Qwen3.5-0.8B passed 11/11 on CPU + locally and in CI once. If CI flakes, investigate before re-running; + don't loosen assertions to get green. +- **Local Hex auth expired** (`mix hex.user auth`): this only matters for a + manual publish, since the workflow publishes with its own key. +- **Tag hygiene**: move a tag only while the release is still a draft and + Hex has nothing (see `.claude/solutions/deployment-issues/macos-14-runner-brownout-precompile-20261031.md`). diff --git a/.claude/plans/release-v0-8-64/scratchpad.md b/.claude/plans/release-v0-8-64/scratchpad.md new file mode 100644 index 0000000..5c36c1d --- /dev/null +++ b/.claude/plans/release-v0-8-64/scratchpad.md @@ -0,0 +1,34 @@ +# Scratchpad: release v0.8.64 + +## DECISION + +- No llama.cpp bump (user, 2026-10-31). Upstream master was 22 commits + ahead of `418f905a4` (head `ddf48f85c`); do that as its own release with + the full release-guide diff (header, decision port, tokenizer partition). +- Merge #115 (docs only) before tagging (user). +- No research agents spawned: the process is fixed by docs/release-guide.md + and was exercised end to end for v0.8.63 the same day. +- Skip decision_compare.exs: no decision or tokenizer code changed since + v0.8.63, which was ALL MATCH on 9 models. + +## Known, not regressions + +- Llama-3.2-1B-Instruct `:tools` round trip: calls the tool again (10/11). +- OTP 29.0.3 locally: `Invalid option {max_header_size,65536} ignored` during + the NIF download is expected (httpc predates the option). +- "Found packages with security advisories" should be gone after #114 + (decimal 3.2.0); if it reappears, check `mix deps.get` output. + +## DEAD-END + +(none yet) + +## Flake found on #115 CI (2026-10-31) + +`ServerSmokeTest` "halting a stream early cancels generation" got another +server's `:done` (prompt 39, `max_tokens`) — the preceding chat_completion +test's server, still alive after its test exited (ExUnit doesn't wait for +linked processes), emitting telemetry after `GenServer.reply`. Fixed by +pinning `metadata.server` in the collectors. Not reproducible locally on Metal +with seed 489492 (timing); evidence is the CI log ordering (handler attach at +54.311, stale done logged 54.314). diff --git a/.claude/solutions/phoenix-issues/gb10-forced-mmap-slow-decode-load-mode-20261102.md b/.claude/solutions/phoenix-issues/gb10-forced-mmap-slow-decode-load-mode-20261102.md new file mode 100644 index 0000000..c9515ea --- /dev/null +++ b/.claude/solutions/phoenix-issues/gb10-forced-mmap-slow-decode-load-mode-20261102.md @@ -0,0 +1,61 @@ +--- +module: "LlamaCppEx.Model (lib/llama_cpp_ex/model.ex), model_load NIF" +date: "2026-11-02" +problem_type: performance +component: model_loading +symptoms: + - "gpt-oss-120b decodes at ~48.5 tok/s through the binding on DGX Spark (GB10) while llama-bench and llama-server on the same build do ~57-59" + - "the gap is per process: about one binding load in five decodes fast (57), and repeated runs inside one process never change state" + - "nsys shows the same kernels (mul_mat_vec_q MXFP4/Q8_0) running ~22% longer in the slow processes" + - "n_ctx, n_threads, sampler chain, BEAM busy-wait, ASLR, page-cache eviction: no effect" +root_cause: "Model.load/2 defaulted :use_mmap to true, which the NIF mapped to an explicit LLAMA_LOAD_MODE_MMAP. Upstream's default is LLAMA_LOAD_MODE_AUTO (#26081), which reads the file instead of mapping it when a device reports caps.mmap_support == false — ggml-cuda reports that for an integrated GPU, and GB10 is one. Forcing mmap overrode that and made the slow placement the common case." +severity: high +tags: [cuda, gb10, dgx-spark, igpu, mmap, load-mode, performance, benchmark-noise] +related_solutions: [] +--- + +# Forced mmap on GB10 costs ~15% decode (load mode) + +## Symptoms + +Binding vs upstream on one Spark, gpt-oss-120b MXFP4, 128 greedy tokens: +`LlamaCppEx.stream/3` 48.5 tok/s, `llama-bench` tg128 58.9, `llama-server` +56.9. Qwen3-8B showed no gap, which hid it at first. + +## Investigation + +1. Binding overhead ruled out first: on Qwen3-8B the binding matched + `llama-bench` and `llama-server` within 1-2%, single stream and 8-way. +2. Per-process bimodality: the same script gave 48.5 in most processes and + 57 in a few; inside one process every context agreed. +3. `nsys profile -t cuda --cuda-graph-trace=node` on both: identical kernel + list and launch shapes, but the memory-bound matvecs ~22% slower in the + slow processes. So GPU-side, not CPU, sampling or messaging. +4. Ruled out: `n_threads`, sampler chain, `n_ctx`/`swa_full`, BEAM `+sbwt none`, + `setarch -R`, THP (weights are not in RSS), page-cache eviction via + `posix_fadvise(DONTNEED)`. +5. `use_mmap: false` and `use_direct_io: true`: 6/6 processes fast. Upstream's + `llama_model_default_params().load_mode` is `AUTO`; `src/llama-model.cpp` + turns mmap off under AUTO when a device lacks `mmap_support`, and + `ggml-cuda.cu` sets that false for `GGML_BACKEND_DEVICE_TYPE_IGPU`. + +## Fix + +`Model.load/2` resolves the three flags itself (`resolve_load_mode/3`) and +passes the enum to the NIF; with none of them given the result is +`LLAMA_LOAD_MODE_AUTO`. Explicit flags resolve exactly as before. Metal, CPU +and discrete CUDA report `mmap_support`, so AUTO maps there as the old +default did. After: 5/5 processes at 57.0-57.2 tok/s; forced mmap still +bimodal (48.5 / 57). + +## Lessons + +- Never override an upstream default with a binding default that looks + equivalent — `use_mmap: true` was "the default" only until upstream grew a + smarter one. +- On GB10, a single decode measurement is one sample of a bimodal process. + The ~17% dip also appeared occasionally in `llama-bench` itself and on the + RPC worker per client session, independent of mmap; re-run before + believing a number. +- Test: `test/option_forwarding_test.exs` "load-mode flags resolve to + llama.cpp's enum, auto when none is given". diff --git a/.claude/solutions/phoenix-issues/rpc-add-server-aborts-vm-dispatcher-20261102.md b/.claude/solutions/phoenix-issues/rpc-add-server-aborts-vm-dispatcher-20261102.md new file mode 100644 index 0000000..a73b210 --- /dev/null +++ b/.claude/solutions/phoenix-issues/rpc-add-server-aborts-vm-dispatcher-20261102.md @@ -0,0 +1,53 @@ +--- +module: "LlamaCppEx.RPC / rpc_add_server NIF (c_src/llama_cpp_ex/llama_nif.cpp)" +date: "2026-11-02" +problem_type: runtime_error +component: rpc +symptoms: + - "ggml-rpc.cpp:653: Failed to connect to 127.0.0.1:1, then SIGABRT (exit 134) from mix test on an LLAMA_RPC=1 build" + - "RPC.add_server/1, RPC.ping/1 or Model.load(path, rpc_servers: [...]) with a dead endpoint kills the VM instead of returning {:error, :unreachable}" + - "default (non-RPC) builds and macOS CI never show it: they answer :rpc_unsupported without touching ggml-rpc" +root_cause: "llama.cpp #18626 (d0132a680, 'rpc : implement event and async backend APIs') moved the connect and HELLO into rpc_dispatcher::start, which GGML_ABORTs on a parse failure, connect failure, HELLO failure or version mismatch. ggml_backend_rpc_add_server used to return nullptr for all of them. negotiate_hello also RPC_STATUS_ASSERTs when a non-RPC peer answers." +severity: high +tags: [rpc, ggml-rpc, abort, nif, upstream-regression, dgx-spark] +related_solutions: [] +--- + +# RPC registration aborts the VM since llama.cpp #18626 + +## Symptoms + +First `mix test` on a CUDA + `LLAMA_RPC=1` build on a DGX Spark aborted +within seconds at `rpc_test.exs`'s `RPC.add_server("127.0.0.1:1")`. The +published 0.8.66 does the same when built with `LLAMA_RPC=1` +(`mix deps.compile llama_cpp_ex --force` with `LLAMA_RPC=1 LLAMA_BACKEND=cuda`, +then `RPC.add_server("127.0.0.1:1")` → exit 134). + +## Investigation + +`git log -S "Failed to connect to %s" -- ggml/src/ggml-rpc/ggml-rpc.cpp` +pointed at d0132a680; its diff replaces every `return nullptr` in the old +`get_socket` with `GGML_ABORT` in `rpc_dispatcher::start`. CI never builds +RPC, so the bump that brought it in (v0.8.5x) passed. + +## Fix + +- NIF: `rpc_probe_hello/2` connects (IPv4, first-':' parse, `std::stoi`, + exactly as upstream) with a 5 s timeout and sends a TCP-only HELLO (zeroed + conn caps, so the worker never starts an RDMA upgrade for the probe), then + checks major/minor like `negotiate_hello`. Anything short of that is + `{:error, :unreachable}` before `ggml_backend_rpc_add_server` runs. The wire + layout is pinned by `static_assert(RPC_PROTO_MAJOR_VERSION == 8)`. +- Upstream: branch `rpc-add-server-no-abort` on `nyo16/llama.cpp` makes + `start()` return false, adds `try_get_dispatcher()` for registration and + keeps aborting only on established connections. +- Tests: `test/rpc_test.exs` — a listener that closes, one that answers HTTP, + and unparseable endpoints; each aborted the VM with the probe disabled. + +## Lessons + +- An RPC build is a different artifact; run `mix test` on one after every + llama.cpp bump that touches `ggml-rpc`, not only `--include rpc_live`. +- `rsync -a` preserves source mtimes: after editing a file on a node, a sync + that restores an older-mtime copy leaves make's `.o` newer than its source. + `touch` the source before rebuilding. diff --git a/.claude/solutions/phoenix-issues/rpc-two-node-decode-bands-client-wakeups-20261102.md b/.claude/solutions/phoenix-issues/rpc-two-node-decode-bands-client-wakeups-20261102.md new file mode 100644 index 0000000..290c129 --- /dev/null +++ b/.claude/solutions/phoenix-issues/rpc-two-node-decode-bands-client-wakeups-20261102.md @@ -0,0 +1,54 @@ +--- +module: "ggml-rpc client (vendor/llama.cpp/ggml/src/ggml-rpc/ggml-rpc.cpp), two-node layer split" +date: "2026-11-02" +problem_type: performance +component: rpc +symptoms: + - "Two-node (RPC layer split) decode sits in one of two bands per client process: gpt-oss-120b ~44 or ~53 tok/s via llama-bench, ~38 or ~47 via the binding" + - "The same worker serves consecutive clients at different speeds; restarting the worker does not settle it" + - "Kernel times (nsys) are identical in both bands on both nodes; the extra ~4 ms per token is CPU-side" +root_cause: "Every RPC command is two thread handoffs: the caller blocks on a std::future while the dispatcher thread, woken from its queue, does the network round trip and wakes the caller. The dispatcher only busy-spins while a -sm tensor communicator holds it (busy_spin_acquire), so for a layer split each handoff is a futex wake of an idle core. On GB10's Cortex-X925 cores those wakes are slow enough to cost ~4 ms per token; on the A725 cores they are not (why is not established)." +severity: medium +tags: [rpc, ggml-rpc, dgx-spark, gb10, two-node, decode, cpu-affinity, latency] +related_solutions: [rpc-add-server-aborts-vm-dispatcher-20261102] +--- + +# Two-node decode bands: RPC client thread wake-ups + +## Symptoms + +`llama-bench --rpc ... -dev CUDA0/RPC0 -n 128` on two DGX Sparks gave 44 or 53 +tok/s for gpt-oss-120b on different runs (single node 59). The docs had +blamed the worker for the spread. + +## Investigation + +- `nsys` on the worker in a fast and a slow session: same kernel times (118 + µs per MXFP4 expert matvec, 2.7 ms LM head), same per-token + `cudaStreamSynchronize` total. Same on the client. So not GPU, not clocks. +- `taskset` matrix (worker cores x client cores, upstream worker restarted + per run): the client on the A725 cores (0-4,10-14) was always fast; on the + X925 cores (5-9,15-19) slow; unpinned mostly slow. `--poll 0`, `-t 1`, + BEAM `+sbwt none` changed nothing. +- An env-gated hack spinning the dispatcher queue and the callers' future + waits gave 57.6 on any cores: the wake-ups were the cost. + +## Fix + +- Upstream patch, `nyo16/llama.cpp` branch `rpc-client-spin-wait` (merged + into `spark-all`): spin the dispatcher whenever a command arrived in the last + `RPC_BUSY_SPIN_IDLE_TIME` (100 ms), and spin callers on their future for up + to that long before blocking; `busy_spin_acquire/release` removed. 57.3-57.6 + tok/s on X925, A725 or unpinned; prefill unchanged. +- Without the patch (vendored llama.cpp): run the client on the A725 cores, + `scripts/spark/remote.sh --little-cores` (+25% two-node decode through the + binding, 37-38 -> 47 tok/s). + +## Lessons + +- A latency microbenchmark (1-byte RDMA ping-pong favours the X925) did not + predict the end-to-end result; measure the decode loop itself. +- A bimodal per-process number with identical kernel times is a CPU-side + handoff, not the GPU. Profile both nodes before blaming either. +- `pkill -f ` over ssh matches the ssh shell's own command line and + kills it (exit 255); use `pkill -x` or a pid. diff --git a/CHANGELOG.md b/CHANGELOG.md index 309afd3..1018623 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,96 @@ # Changelog +## Unreleased + +Validated on two DGX Sparks (GB10, CUDA 13.0): the full suite with every +model tag on CUDA, an end-to-end run with three models resident, a model +layer-split across both nodes beside a local one, and the published 0.8.66 +package source-built from a fresh project. See `docs/dgx-spark.md` for the +numbers behind the entries below. + +### Added + +- `LlamaCppEx.MTP.init/2` takes `type: :dflash` or `type: :dspark` with the + drafter passed as `:draft_model`, for upstream's DFlash / DFlash 2 + block-diffusion drafters and DSpark. On Qwen3.8-27B NVFP4 on GB10, DFlash 2 + decodes at 3.1× plain (44.2 against 14.3 tok/s), where the model's MTP head + reaches 2.0×; on an M1 Max, 2.5× on the Q4_K_M build. A drafter that is not + a DFlash checkpoint, or reads layers the target does not have, is refused + with an error: upstream asserts on it. `n_draft` is clamped to the drafter's + block size and `%MTP{}.n_draft` reports the clamped value. +- `LlamaCppEx.Model.devices/1`: the devices a model's layers were placed on. + `MTP.init/2` uses it to refuse a DFlash/DSpark drafter that is not loaded on + every device the target uses — the drafter computes with the target's + embeddings and LM head, and a target split onto an RPC worker with the + drafter on `CUDA0` alone aborted the VM. Loaded with the target's + `:devices`, DFlash runs over a two-node split (Qwen3.8-27B NVFP4: 22.3 + tok/s against 14.0 plain). Upstream fix (the draft context gets backends + for the target's devices): `nyo16/llama.cpp` branch + `draft-ctx-other-devices`. + +### Fixed + +- `LlamaCppEx.Model.load/2` no longer forces `mmap`. `:use_mmap` defaulted to + `true`, which mapped to an explicit `LLAMA_LOAD_MODE_MMAP` and overrode + upstream's default `auto` — the mode that skips mmap on integrated GPUs + (#26081). On DGX Spark (GB10) a forced mmap left most gpt-oss-120b loads + decoding at 48.5 tok/s instead of 57 (+17.6%; an `nsys` profile shows the + same kernels running ~22% longer), and about one load in five came out fast, + which is why single benchmark runs disagreed. With `:use_mmap`, + `:use_mlock` and `:use_direct_io` all unset the mode is now `auto`; any of + them given resolves as before. Metal, CPU and discrete CUDA devices support + mmap, so `auto` maps there exactly as the old default did. The NIF's + `model_load` now takes the resolved mode instead of three booleans. +- `LlamaCppEx.Server`'s prefix-instability warning and + `[:llama_cpp_ex, :server, :prefix_instability]` event fired when the same + prompt came back (a retry, a regenerate, a benchmark loop) once the cached + generation made the history long enough — the whole prompt matched, which + is a perfect cache hit. They now fire only when the prompt itself diverges + from the cached history. +- `LlamaCppEx.RPC.add_server/1`, `ping/1` and `Model.load/2`'s + `:rpc_servers` aborted the VM on an RPC build when the endpoint was + unreachable, unparseable, not an RPC worker, or spoke another protocol + version. Upstream's registration has aborted on all four since #18626 + moved the connect into `rpc_dispatcher::start`; the NIF now connects and + exchanges the HELLO itself first (5 s timeout) and returns + `{:error, :unreachable}`. The default build (no `LLAMA_RPC`) was never + affected. The upstream fix is on `nyo16/llama.cpp` branch + `rpc-add-server-no-abort`. +- MTP on a hybrid target (Qwen 3.6 / 3.8: recurrent layers beside attention) + no longer saves the recurrent state every iteration and re-decodes the + accepted prefix on every partial accept. `MTP.init/2` creates the target + context with one recurrent snapshot per draft position, as upstream's server + does, so a rejected draft rolls back natively. GB10, `n_draft: 2`: + Qwen3.8-27B NVFP4 21.1 → 26.8 tok/s, Qwen3.6-35B-A3B 1.42× plain where the + docs had it at 0.95×. M1 Max: Qwen3.8-27B with its sidecar head went from a + net loss at every `n_draft` to 1.96× at `n_draft: 3`. +- After a checkpoint rollback the re-decoded prefix is now fed to the + drafter, which had been left with a gap in its context; DFlash acceptance + was ~3% without it. + +### Changed + +- `LlamaCppEx.Context`'s `:n_rs_seq` docs, `LlamaCppEx.MTP`'s hybrid-model + note and `stats/1`'s `:ckpt` description describe the snapshot rollback. +- RPC docs: `GGML_RPC_NO_RDMA` is upstream's runtime switch to keep a peer on + TCP (`scripts/spark/rpc-worker.sh --tcp` now uses it), and layer split over + RPC is pipelined: the RPC backend implements async compute and events + (#18626), so llama.cpp enables pipeline parallelism across the nodes. +- Two-node docs: run the client on the GB10's Cortex-A725 cores + (`taskset -c 0-4,10-14`, new `scripts/spark/remote.sh --little-cores`). + Unpinned or on the X925 cores, decode over RPC sits in a band ~25% slower: + gpt-oss-120b layer-split decoded 37–38 tok/s against 47 through the binding, + 44 against 54 through `llama-bench`. The decode spread the docs blamed on the + worker came from the client's placement: ggml's RPC client hands every + command between two threads, and those wake-ups are slow on the X925 + cores. Upstream fix on `nyo16/llama.cpp` branch `rpc-client-spin-wait`: + 57.6 tok/s on any cores through `llama-bench` (single node 59), 50.6–50.8 + through the binding unpinned. +- `test/embeddings_test.exs`: `embed_batch` is compared with per-text `embed` + by cosine similarity (> 0.999) instead of a 0.001 elementwise bound, which + CUDA missed (0.0027 on bge-small) because a batch runs different kernels. + Swapped rows still fail it (cos 0.41). + ## v0.8.66 llama.cpp bumped to `785322d17` diff --git a/CLAUDE.md b/CLAUDE.md index 283b4ce..0e1b4ba 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -71,3 +71,14 @@ release problem. - A test-side `:telemetry.attach` filters on an identity the test owns (`%{server: ^server}`, a ref) and detaches in `on_exit`. A linked server can outlive its test and emit after replying. + +## RPC and performance + +- After a llama.cpp bump, run the default `mix test` on an `LLAMA_RPC=1` build + too (a DGX Spark has one): CI never builds RPC, and upstream's registration + started aborting on a dead endpoint without any test noticing (#18626). +- Do not override an upstream default with a binding default that looks + equivalent. `Model.load/2` leaves the load mode `auto` unless the caller + sets a flag; a forced `mmap` cost ~15% decode on GB10. +- On a DGX Spark a decode figure is one sample: a load can land ~17% slow for + its lifetime. Re-run in a fresh process before comparing. diff --git a/README.md b/README.md index 5ec8dc2..1af48db 100644 --- a/README.md +++ b/README.md @@ -1006,84 +1006,73 @@ The manager estimates footprint from GGUF size (plus a coarse KV-cache estimate Model cleanup is garbage-collection based. `unload/1` stops the backing server (dropping its context and model references) and forces a GC. Because reclamation is by GC, **any caller still holding a `%LlamaCppEx.Model{}` obtained via `fetch_model/1` keeps the model alive** past `unload/1` — prefer id-based routing and avoid holding raw refs. -## Speculative decoding (MTP) - -Multi-Token Prediction speculative decoding (upstream PR [#22673](https://github.com/ggml-org/llama.cpp/pull/22673)) drafts several tokens at once via a head shipped inside the same GGUF as the target model. Upstream llama-server reports ~2x speedup at ~75% draft acceptance on Qwen 3.6. - -> **Performance note: Apple Silicon.** The upstream 2× claim is from NVIDIA datacenter GPUs, where a batched verify decode costs ~1.2× a single-token decode. On Apple Silicon (Metal), a 4-wide verify costs ~2.4× a single decode, which cancels MTP's iteration savings. We measured upstream's own `llama-server --spec-type draft-mtp` on M1 Max: **39.80 tok/s with MTP vs 39.14 tok/s plain** on Qwen 3.6 35B-A3B (1.02×) — i.e. effectively zero speedup from the reference implementation itself. This matches the pattern in upstream [#23011](https://github.com/ggml-org/llama.cpp/issues/23011); a Metal MTP optimization is tracked in [#23114](https://github.com/ggml-org/llama.cpp/pull/23114). -> -> **Tuning for Apple Silicon:** use `n_draft: 1`. With one draft per iteration the verify batch is only 2-wide (much cheaper on Metal) and acceptance jumps to ~79% on Qwen 3.6 35B-A3B. Our measurements on M1 Max with `n_draft: 1`: -> - Qwen 3.6 35B-A3B-MTP (hybrid MoE): plain 39.5 → MTP **44.0 tok/s (1.11×)** -> - Qwen 3.6 27B (dense): plain 10.7 → MTP **10.6 tok/s (~1.0×, neutral)** -> -> Larger `n_draft` hurts on Metal because verify cost grows faster than acceptance benefit. - -> **Performance note: NVIDIA GB10 (DGX Spark).** MTP does pay here, but nothing -> like the upstream 2×, and the best `n_draft` is not 3. Qwen3.6-35B-A3B -> UD-Q4_K_XL from the `-MTP` build, 128-token greedy generations, plain and MTP -> interleaved in one process so drift hits both arms equally (n=11 each): -> -> | | median tok/s | range | vs plain | -> |---|---|---|---| -> | plain | 61.4 | 61.2–62.4 | — | -> | MTP `n_draft: 2` | **71.2** | 62.5–75.5 | **+16%** | -> -> The ranges do not overlap — MTP's slowest run beat plain's fastest — so the -> gain is real despite MTP being the noisier arm by an order of magnitude. -> -> Sweeping `n_draft` on the same model, though, puts the optimum at 2 rather -> than 3, and the engine's own counters say why: -> -> | `n_draft` | acceptance | tokens/iteration | tok/s | -> |---|---|---|---| -> | 2 | 68.5% | 2.38 | 63.0 | -> | 3 | 57.2% | 2.73 | 52.8 | -> -> Going from 2 to 3 buys 15% more tokens per iteration and pays 31% more drafting -> plus 10% more verify for them, because the third draft position is the one -> least likely to be accepted. Marginal acceptance decays faster than marginal -> cost, so the extra draft loses money. In a five-run sweep `n_draft: 3` came out -> 2% *below* plain and `n_draft: 4` 14% below. -> -> So the shape matches Apple Silicon even though the cause differs: on Metal the -> wide verify is expensive, while GB10 is a unified-memory part whose MoE decode -> is memory-bandwidth bound, and a wider verify reads more expert weights per -> step. Both end up wanting a narrower draft than a datacenter GPU does. Treat -> `n_draft: 3` as the datacenter default the upstream 2× assumes, not as a value -> that transfers. - -> **Performance note: Qwen 3.8 27B (hybrid SSM, sidecar head).** This one is -> shaped by a cost the models above do not pay. Qwen 3.8 puts 48 SSM layers -> beside 16 attention ones, and a recurrent layer cannot be rolled back to an -> arbitrary position, so every speculative iteration snapshots and restores the -> whole recurrent state — 150 MiB of it at these sizes. `stats/1` reports that -> separately as `timing_us.ckpt`. Q4_K_M target + Q4_0 sidecar head, 120-token -> greedy generations: -> -> | `n_draft` | acceptance | M1 Max (Metal) | GB10 (DGX Spark) | -> |---|---|---|---| -> | 1 | 75.0% | 0.89× | **1.24×** | -> | 2 | 54–59% | 0.66× | 1.17× | -> | 3 | 40–44% | 0.68× | 1.09× | -> | 4 | 32% | — | 0.96× | -> | 5 | 30% | 0.56× | — | -> -> On Metal MTP is a net loss at every draft length: `ckpt` alone was 1.8 s of a -> 12.5 s run at `n_draft: 1` and 6.9 s of 16.3 s at `n_draft: 3`. On GB10 the -> same snapshot is cheap enough that `n_draft: 1` wins, and — unlike the MoE -> above, whose optimum was 2 — the optimum here is 1, monotonically decreasing -> after it. Measure `ckpt` against `total` before trusting speculation on any -> hybrid model. -> -> Both arms of the GB10 column are warm-cache numbers. A first run off cold page -> cache reads ~19 GB and reported a 4.18 tok/s baseline against 10.83 tok/s warm, -> which inverts the comparison entirely. - -### Other speculative types (EAGLE-3, DFlash, n-gram) - -Upstream llama.cpp implements more speculative types behind the same `common_speculative` API — `draft-eagle3`, `draft-dflash` (block-diffusion drafting via a separate drafter GGUF), and several n-gram self-speculation modes, plus `--spec-default`, which stacks n-gram speculation on top of a model-based drafter. **This binding currently exposes only MTP**: `MTP.init/2` pins `COMMON_SPECULATIVE_TYPE_DRAFT_MTP`, so the other types and the combinations are not reachable from here. The draft *model* is no longer tied to the target, though — see `:draft_model` below for the target/sidecar split. - -> **DFlash status (July 2026, llama.cpp b9932).** DFlash runs end-to-end on Metal via upstream `llama-cli`/`llama-server`, but we measured it *slower* than plain decoding on Apple Silicon at small target sizes: Qwen3.5-4B target + z-lab 0.6B drafter on M4 Max reached 42 tok/s with DFlash vs 85 tok/s plain (greedy sampling; 30% draft acceptance, mean accepted run 2.8 — and stochastic sampling at `temp 0.8` collapses acceptance to ~7%). The Metal economics are the same as the MTP note above (wide verify batches are expensive), and the community drafter-GGUF conversions are still churning: of three third-party Qwen 4B drafter repos tested, only one loads with current upstream (the others hit the `dflash-draft` arch mismatch [#25116](https://github.com/ggml-org/llama.cpp/issues/25116) or lack the `target_layers` metadata added by the conversion refactor [#25110](https://github.com/ggml-org/llama.cpp/pull/25110)). Worth revisiting when the drafter format settles; the natural entry point is a `spec_type` + drafter-model option on `speculative_init`. +## Speculative decoding (MTP, DFlash) + +`LlamaCppEx.MTP` drives upstream's `common_speculative` loop: a drafter proposes +several tokens, the target verifies them in one batched decode, and greedy +output is the target's own (up to the batch-width rounding noted in the +`MTPSidecarTest` comments — upstream's server diverges from its own plain decode +at the same places). Three drafters are supported through `:type`: + +- `:mtp` (default) — a Multi-Token Prediction head (upstream PR + [#22673](https://github.com/ggml-org/llama.cpp/pull/22673)), inside the + target GGUF or in a sidecar passed as `:draft_model`. +- `:dflash` — a DFlash / DFlash 2 block-diffusion drafter: a separate small + GGUF (architecture `dflash`) that drafts a whole block in one pass from the + target's hidden states. Pass it as `:draft_model`. +- `:dspark` — DSpark (DFlash plus a Markov head), the same way. + +```elixir +{:ok, target} = LlamaCppEx.load_model("Qwen3.8-27B-Q4_K_M.gguf", n_gpu_layers: -1) +{:ok, drafter} = LlamaCppEx.load_model("Qwen3.8-27B-DFlash2-Q4_K_M.gguf", n_gpu_layers: -1) +{:ok, spec} = LlamaCppEx.MTP.init(target, type: :dflash, draft_model: drafter, n_draft: 7) +``` + +**Measured.** 256-token greedy chat completions (thinking off), four prompts +run twice, median decode rate; `bench/spark_spec.exs` reproduces it on either +machine: + +| Qwen3.8-27B (dense, hybrid SSM) | GB10, NVFP4 MID-HIGH | M1 Max, Q4_K_M + sidecar head | +|---|---|---| +| plain | 14.3 tok/s | 12.7 tok/s | +| MTP `n_draft: 2` | 26.8 (1.87×, 76% accepted) | 22.1 (1.75×) | +| MTP `n_draft: 3` | 28.6 (2.00×) | 24.8 (1.96×) | +| DFlash 2 `n_draft: 7` | **44.2 (3.10×, 42% accepted)** | **31.8 (2.53×)** | + +| Qwen3.6-35B-A3B (MoE, hybrid SSM) | GB10, UD-Q4_K_M | M1 Max, UD-Q4_K_XL | +|---|---|---| +| plain | 68.3 tok/s | 55.4 tok/s | +| MTP `n_draft: 1` | 86.9 (1.27×) | 69.3 (1.25×) | +| MTP `n_draft: 2` | **96.7 (1.42×)** | **74.5 (1.34×)** | +| MTP `n_draft: 3` | 91.6 (1.34×) | 73.5 (1.33×) | + +- **DFlash 2 is the fastest drafter where one exists for the model** — 3.1× on + GB10, against 2.0× for the same checkpoint's MTP head and 1.8× for DSpark + (`magnitudedev/Qwen3.8-27B-DSpark-GGUF`, measured through upstream's + `llama-server`). Its `n_draft` caps at the drafter's block size minus one (7 + for these), which is also where it measured best. +- **MTP wants `n_draft: 2–3` on a dense model and 2 on a sparse MoE**, where + only ~3B parameters move per token and a wider verify reads more experts. +- Within 2–5% of upstream's `llama-server` with the same drafters, on GB10. + +These numbers postdate a rollback fix: hybrid targets used to save their whole +recurrent state every iteration and re-decode the accepted prefix on every +partial accept, which made MTP a net loss on Apple Silicon and held GB10 to +~1.5×. `MTP.init/2` now gives the target context one recurrent snapshot per +draft position (`n_rs_seq`), as upstream's server does, and `stats/1`'s +`timing_us.ckpt` stays zero. + +### Other speculative types (EAGLE-3, n-gram) + +Upstream also implements `draft-eagle3` and several n-gram self-speculation +modes behind the same API, plus `--spec-default`, which stacks n-gram +speculation on top of a model-based drafter. Those are not exposed here. + +### Drafters + +- [`z-lab/Qwen3.8-27B-DFlash2-GGUF`](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF) + — DFlash 2 for Qwen3.8-27B (Q4_K_M 1.1 GB). Pairs with any quantization of + the target, NVFP4 included. ### Models with MTP heads @@ -1298,10 +1287,14 @@ end) `LlamaCppEx.MTP.init/2`: + * `:type` — `:mtp` (default), `:dflash` or `:dspark`. The last two need the + drafter GGUF as `:draft_model`. + * `:draft_model` — a separate `%LlamaCppEx.Model{}`: a DFlash/DSpark drafter, + or an MTP sidecar head loaded with `load_mtp: true`. * `:n_draft` — draft tokens proposed per iteration (default `3`). The optimum - is hardware-specific and worth measuring rather than assuming: `1` on Apple - Silicon, `2` on GB10 (where `3` measured *slower* than no speculation at - all), `2–4` on datacenter NVIDIA. See the two performance notes above. + is model- and hardware-shaped, so measure it: for MTP, `2–3` on a dense + model and `2` on a sparse MoE on both GB10 and M1 Max; for DFlash 2, the + drafter's block size minus one (`7`), which is also its cap. * `:n_ctx`, `:n_threads`, `:flash_attn`, `:type_k`/`:type_v`, `:offload_kqv`, … — any `LlamaCppEx.Context` option; applied to both target and draft contexts. `LlamaCppEx.MTP.stream/3`: @@ -1347,45 +1340,40 @@ New `qwen35moe` architecture with Gated Delta Net (hybrid linear/full attention) ### CUDA: NVIDIA DGX Spark (GB10) -Same model and quantization as the M1 Max row above, so the two are directly -comparable. GB10 (`sm_121a`, aarch64, 128 GB unified), CUDA 13.0.2, driver -580.173.02, llama.cpp b10280, source build with `LLAMA_BACKEND=cuda`. - -| Model | Quantization | Tokens/sec (GB10) | Tokens/sec (M1 Max) | -|-------|-------------|-------------------|---------------------| -| Qwen3.6-35B-A3B | UD-Q4_K_XL | **62.1** | 43.8 | - -128-token generation, `temp: 0.0`, `n_gpu_layers: -1`. Median of 5 runs after a -discarded warm-up: 61.7 / 62.0 / 62.1 / 62.2 / 62.2 t/s — a 0.9% spread, so the -1.42x over M1 Max is well outside the noise. All 41 layers offload; the model -takes 20 799 MiB of device memory. - -Two notes on method, both learned the hard way: - -- **Each run uses a distinct prompt.** Repeating one prompt hits the context - reuse path and reports a throughput the engine never achieved. -- **Tokens are counted by re-encoding the output**, not by counting stream - chunks — a chunk is not a token, and under speculative decoding it can carry - several. - -The first call after load is discarded: it pays CUDA graph capture and the -allocator's first-touch layout, and is not representative of steady state. - -With MTP speculative decoding, using the separate `-MTP` build of the same model -(the plain UD-Q4_K_XL carries no MTP head — `MTP.init/2` now says so rather than -failing with a bare context error): - -| | median tok/s | vs plain | -|---|---|---| -| plain | 61.4 | — | -| `n_draft: 2` | **71.2** | +16% | -| `n_draft: 3` | 61.1 | −2% | -| `n_draft: 4` | 53.8 | −14% | - -`n_draft: 2` is the optimum on this hardware, not the documented default of 3. -See the GB10 performance note under [Speculative decoding -(MTP)](#speculative-decoding-mtp) for the acceptance and timing counters behind -that. +GB10 (`sm_121a`, aarch64, 128 GB unified), CUDA 13.0, driver 580.173.02, +llama.cpp `785322d17`, source build with `LLAMA_BACKEND=cuda`. Single-stream +decode through `LlamaCppEx.stream/3`, 128 greedy tokens, against upstream's +`llama-bench` on the same build: + +| Model | Quantization | llama_cpp_ex | llama-bench tg128 | +|-------|-------------|--------------|-------------------| +| Qwen3-8B | Q4_K_M | 44.1 | 43.9 | +| Qwen3-30B-A3B | Q4_K_M | 91.4 | 92.7 | +| Qwen3.6-35B-A3B | UD-Q4_K_M | 68.3 | 69.8 | +| Qwen3.8-27B | Q4_K_M | 11.3 | 11.3 | +| Qwen3.8-27B | NVFP4 (MID-HIGH) | 14.3 | 14.4 | +| gpt-oss-120b | MXFP4 | 57.0 | 58.8–59.0 | + +With speculative decoding the same machine does 44.2 tok/s on Qwen3.8-27B +NVFP4 (DFlash 2) and 96.7 on Qwen3.6-35B-A3B (MTP, `n_draft: 2`); see +[Speculative decoding](#speculative-decoding-mtp-dflash). Concurrency, prefill +and two-node numbers are in [docs/dgx-spark.md](docs/dgx-spark.md). + +Three notes, all learned the hard way: + +- **Each run uses a distinct prompt, or prompt caching off.** Repeating one + prompt hits the context reuse path and reports a throughput the engine never + achieved. +- **Measure decode more than once per load.** On GB10 a load can land ~17% + slower for its whole lifetime (gpt-oss-120b: 48.5 against 57 tok/s), and a + forced `use_mmap: true` made that the common case — leave the load-mode + options unset. +- **Over two nodes, pin the client to the Cortex-A725 cores** + (`taskset -c 0-4,10-14`). Unpinned or on the X925 cores, decode over RPC + usually lands in a band ~25% slower: gpt-oss-120b layer-split decoded + 37–38 tok/s unpinned against 47 pinned, with the same prefill. The cause is + thread wake-ups in ggml's RPC client. A fix on `nyo16/llama.cpp` + (`rpc-client-spin-wait`) gives 50.6–50.8 unpinned, against 55 single-node. ### Single-sequence generation (Qwen3-4B Q4_K_M) diff --git a/bench/spark_overhead.exs b/bench/spark_overhead.exs new file mode 100644 index 0000000..b6d322b --- /dev/null +++ b/bench/spark_overhead.exs @@ -0,0 +1,149 @@ +# Binding overhead against upstream's own tools, on one model. +# +# scripts/spark/remote.sh --env MIX_ENV=bench --env LLAMA_MODEL_PATH= \ +# [--env SPARK_RPC_ENDPOINT=10.100.64.2:50052] \ +# --big-cores spark-1 mix run bench/spark_overhead.exs +# +# Compare with, from the same llama.cpp commit: +# +# llama-bench -m -fa 1 -p 512 -n 128 -r 3 +# llama-batched-bench -m -fa 1 -c 16384 -npp 128 -ntg 128 -npl 1,4,8 +# +# Three decode paths, each timed from the first streamed token to the last so +# prompt processing and context creation stay out of the number: +# +# stream LlamaCppEx.stream/3 -- the C++ generate_tokens loop, one sequence +# server1 LlamaCppEx.Server, n_parallel 1 -- the batch_eval_sample tick loop +# serverN LlamaCppEx.Server, n_parallel N, N concurrent streams -- aggregate +# +# llama-bench's tg does no sampling and sends nothing anywhere, so the gap to +# `stream` is the whole binding cost per token: sampler chain, token_to_piece, +# enif_send and the BEAM receive. + +model_path = System.get_env("LLAMA_MODEL_PATH") || raise "Set LLAMA_MODEL_PATH" +n_gen = String.to_integer(System.get_env("SPARK_N_GEN") || "128") +samples = String.to_integer(System.get_env("SPARK_SAMPLES") || "3") + +parallel = + (System.get_env("SPARK_PARALLEL") || "4,8") + |> String.split(",", trim: true) + |> Enum.map(&String.to_integer/1) + +# Base-model friendly and long-running under greedy decoding; the chat template +# is deliberately not applied so every path sees the same tokens. +prompt = + "Write a detailed, multi-chapter history of the Roman Empire, starting with " <> + "the founding of Rome and continuing through the fall of Constantinople.\n\n" + +defmodule Overhead do + def median([_ | _] = xs) do + s = Enum.sort(xs) + n = length(s) + + if rem(n, 2) == 1, + do: Enum.at(s, div(n, 2)), + else: (Enum.at(s, div(n, 2) - 1) + Enum.at(s, div(n, 2))) / 2 + end + + # Consumes an enumerable of chunks and returns {tokens_after_first, seconds}. + def time_decode(enum) do + {count, first, last} = + Enum.reduce(enum, {0, nil, nil}, fn + {:error, reason}, _ -> + raise "generation failed: #{inspect(reason)}" + + chunk, {n, first, _} when is_binary(chunk) or is_map(chunk) -> + now = System.monotonic_time(:microsecond) + {n + 1, first || now, now} + + _other, acc -> + acc + end) + + {count - 1, (last - first) / 1_000_000} + end + + def tps({n, s}) when s > 0, do: n / s +end + +:ok = LlamaCppEx.init() + +# SPARK_RPC_ENDPOINT=host:port splits every load 50/50 with that worker, so the +# same table measures two nodes. +placement = + case System.get_env("SPARK_RPC_ENDPOINT") do + nil -> + [] + + endpoint -> + [ + rpc_servers: [endpoint], + devices: ["CUDA0", "RPC0"], + split_mode: :layer, + tensor_split: [0.5, 0.5] + ] + end + +{:ok, model} = LlamaCppEx.load_model(model_path, [n_gpu_layers: -1] ++ placement) + +IO.puts("model: #{Path.basename(model_path)} n_gen: #{n_gen} samples: #{samples}\n") +IO.puts("| path | streams | tokens/stream | per-stream t/s | aggregate t/s |") +IO.puts("|---|---|---|---|---|") + +gen_opts = [max_tokens: n_gen, temp: 0.0, n_ctx: 4096] + +# Warm-up: kernel selection and CUDA graph capture happen on the first run. +model |> LlamaCppEx.stream(prompt, gen_opts) |> Stream.run() + +runs = for _ <- 1..samples, do: Overhead.time_decode(LlamaCppEx.stream(model, prompt, gen_opts)) +tps = Overhead.median(Enum.map(runs, &Overhead.tps/1)) +{n, _} = hd(runs) +IO.puts("| stream | 1 | #{n} | #{Float.round(tps, 2)} | #{Float.round(tps, 2)} |") + +for n_par <- [1 | parallel] do + {:ok, server} = + LlamaCppEx.Server.start_link( + [ + model_path: model_path, + n_gpu_layers: -1, + n_parallel: n_par, + n_ctx: 4096 * n_par, + temp: 0.0, + cache_prompt: false + ] ++ placement + ) + + # start_link returns before the load; a generate call queues behind it. + {:ok, _} = LlamaCppEx.Server.generate(server, prompt, max_tokens: 8) + + one_round = fn -> + t0 = System.monotonic_time(:microsecond) + + results = + 1..n_par + |> Task.async_stream( + fn _ -> + Overhead.time_decode(LlamaCppEx.Server.stream(server, prompt, max_tokens: n_gen)) + end, + max_concurrency: n_par, + timeout: :infinity + ) + |> Enum.map(fn {:ok, r} -> r end) + + wall = (System.monotonic_time(:microsecond) - t0) / 1_000_000 + per = Overhead.median(Enum.map(results, &Overhead.tps/1)) + total = results |> Enum.map(&elem(&1, 0)) |> Enum.sum() + {per, total / wall, results |> hd() |> elem(0)} + end + + rounds = for _ <- 1..samples, do: one_round.() + per = Overhead.median(Enum.map(rounds, &elem(&1, 0))) + agg = Overhead.median(Enum.map(rounds, &elem(&1, 1))) + toks = rounds |> hd() |> elem(2) + + IO.puts( + "| server#{n_par} | #{n_par} | #{toks} | #{Float.round(per, 2)} | #{Float.round(agg, 2)} |" + ) + + GenServer.stop(server) +end diff --git a/bench/spark_spec.exs b/bench/spark_spec.exs new file mode 100644 index 0000000..7a8f30c --- /dev/null +++ b/bench/spark_spec.exs @@ -0,0 +1,140 @@ +# Plain decode against every speculative drafter, through LlamaCppEx.MTP. +# +# scripts/spark/remote.sh --env MIX_ENV=bench --big-cores spark-1 \ +# mix run bench/spark_spec.exs [config ...] +# +# A config is `plain`, `mtp:` (the head inside the target GGUF), +# `mtp::`, `dflash::` or +# `dspark::`. With none given: +# plain, mtp:1..4. Each config runs the same four chat-templated prompts twice, +# greedy, 256 tokens, thinking off, and reports the median decode rate (first +# token to last, so prefill is excluded) and the draft acceptance. +# +# Greedy speculative decoding is lossless, so every config's text is also +# compared with the plain run's: a drafter that changes the output is a bug, +# whatever it does to the rate. Same prompts as the llama-server runs in +# docs/dgx-spark.md, for a like-for-like comparison. + +alias LlamaCppEx.{Chat, MTP} + +[target | configs] = System.argv() +configs = if configs == [], do: ["plain", "mtp:1", "mtp:2", "mtp:3", "mtp:4"], else: configs +n_gen = 256 + +prompts = [ + "Write a Python function that parses an ISO-8601 date string and returns a datetime, with docstring and error handling.", + "Explain step by step how to compute the derivative of x^3 * sin(x).", + "Write a short essay about the history of the printing press.", + "List ten tips for writing maintainable Elixir code, one sentence each." +] + +LlamaCppEx.set_log_level(:error) +:ok = LlamaCppEx.init() +{:ok, model} = LlamaCppEx.load_model(target, n_gpu_layers: -1, load_mtp: true) + +templated = + Enum.map(prompts, fn p -> + {:ok, text} = + Chat.apply_template(model, [%{role: "user", content: p}], enable_thinking: false) + + text + end) + +defmodule Spec do + def median(xs) do + s = Enum.sort(xs) + n = length(s) + + if rem(n, 2) == 1, + do: Enum.at(s, div(n, 2)), + else: (Enum.at(s, div(n, 2) - 1) + Enum.at(s, div(n, 2))) / 2 + end + + # {text, tokens_after_first / seconds} + def run(enum) do + {pieces, first, last, n} = + Enum.reduce(enum, {[], nil, nil, 0}, fn + {:error, reason}, _ -> + raise "generation failed: #{inspect(reason)}" + + piece, {acc, f, _, n} when is_binary(piece) -> + now = System.monotonic_time(:microsecond) + {[piece | acc], f || now, now, n + 1} + end) + + {pieces |> Enum.reverse() |> IO.iodata_to_binary(), + (n - 1) * 1_000_000 / max(last - first, 1)} + end +end + +opts = [max_tokens: n_gen, temp: 0.0] + +IO.puts("target: #{Path.basename(target)}\n") + +IO.puts( + "| config | median t/s | range | vs plain | acceptance | output == plain | ms/iter draft / verify / ckpt / sample / other |" +) + +IO.puts("|---|---|---|---|---|---|---|") + +{reference, _} = + Enum.reduce(configs, {nil, nil}, fn config, {ref_texts, ref_rate} -> + {runner, cleanup, label} = + case String.split(config, ":", parts: 3) do + ["plain"] -> + {fn prompt -> + Spec.run(LlamaCppEx.stream(model, prompt, Keyword.put(opts, :n_ctx, 4096))) + end, fn -> nil end, "plain"} + + [type, n | rest] -> + draft = + case rest do + [path] -> + elem(LlamaCppEx.load_model(path, n_gpu_layers: -1, load_mtp: type == "mtp"), 1) + + [] -> + nil + end + + {:ok, mtp} = + MTP.init(model, + type: String.to_existing_atom(type), + draft_model: draft, + n_draft: String.to_integer(n), + n_ctx: 4096 + ) + + {fn prompt -> Spec.run(MTP.stream(mtp, prompt, opts)) end, fn -> MTP.stats(mtp) end, + "#{type} n=#{mtp.n_draft}"} + end + + _warm = runner.(hd(templated)) + results = for _ <- 1..2, prompt <- templated, do: runner.(prompt) + texts = results |> Enum.take(length(templated)) |> Enum.map(&elem(&1, 0)) + rates = Enum.map(results, &elem(&1, 1)) + med = Spec.median(rates) + + {acc, split} = + case cleanup.() do + %{acceptance_rate: r, iters: iters, timing_us: t} -> + per = fn key -> Float.round(Map.fetch!(t, key) / max(iters, 1) / 1000, 2) end + + {"#{Float.round(r * 100, 1)}%", + Enum.map_join([:draft, :verify, :ckpt, :sample, :other], " / ", &per.(&1))} + + _ -> + {"-", "-"} + end + + same = if ref_texts, do: to_string(texts == ref_texts), else: "-" + vs = if ref_rate, do: "#{Float.round(med / ref_rate, 2)}x", else: "-" + + IO.puts( + "| #{label} | #{Float.round(med, 2)} | #{Float.round(Enum.min(rates), 1)}-#{Float.round(Enum.max(rates), 1)} | #{vs} | #{acc} | #{same} | #{split} |" + ) + + :erlang.garbage_collect() + {ref_texts || texts, ref_rate || med} + end) + +_ = reference diff --git a/bench/spark_two_node.exs b/bench/spark_two_node.exs index 516ef65..ac77f5f 100644 --- a/bench/spark_two_node.exs +++ b/bench/spark_two_node.exs @@ -3,9 +3,13 @@ Code.require_file("helpers.exs", __DIR__) # Two-node measurement suite for a pair of DGX Sparks. # # scripts/spark/rpc-worker.sh start spark-2 -# scripts/spark/remote.sh --env MIX_ENV=bench --env LLAMA_RPC=1 spark-1 \ +# scripts/spark/remote.sh --little-cores --env MIX_ENV=bench --env LLAMA_RPC=1 spark-1 \ # mix run bench/spark_two_node.exs # +# --little-cores runs this client on the Cortex-A725 cores, where two-node decode +# is ~25% faster than in the band an unpinned BEAM usually lands in +# (docs/dgx-spark.md, "Pin the client to the little cores"). +# # Benches, each answering one question: # # b1 RPC overhead, controlled. A model that FITS one node, run single-node @@ -15,10 +19,12 @@ Code.require_file("helpers.exs", __DIR__) # b2 The actual point. A model that does NOT fit one node: two-node RPC # versus single-node paging weights off NVMe. This is the number that # justifies the second Spark, and it is allowed to come out negative. -# b3 RDMA versus TCP, in tokens. There is no runtime switch, so TCP is -# forced with GGML_RDMA_DEV set to a device that does not exist. -# b4 Batching amortisation. Per-token RTT is fixed per *graph*, not per -# token, so n_parallel should amortise it. Measure, do not assume. +# b3 RDMA versus TCP, in tokens. One run per transport; see the b3 clause +# for the worker flags (`--tcp` sets GGML_RPC_NO_RDMA, `--link lan`). +# +# Concurrency across two nodes is bench/spark_overhead.exs with +# SPARK_RPC_ENDPOINT set: it runs N streams at once, which a slot count alone +# does not. # # Every run reports worker RSS before and after: upstream's worker is reported # to grow and never release, and RSS is a column here rather than an afterthought. @@ -248,12 +254,12 @@ case which do "b1" -> # Controlled: the model fits one node, so anything the second node costs is # pure RPC overhead rather than a memory benefit. - path = models["120b"] + path = System.get_env("SPARK_TWO_NODE_MODEL") || models["120b"] File.exists?(path) || raise "missing #{path}" single = TwoNode.measure( - "120b single-node", + "#{Path.basename(path, ".gguf")} single-node", path, [n_parallel: 1, n_ctx: n_ctx, cache_prompt: false], prompt_tokens, @@ -265,7 +271,7 @@ case which do two = TwoNode.measure( - "120b two-node 50/50", + "#{Path.basename(path, ".gguf")} two-node 50/50", path, [ n_parallel: 1, @@ -339,18 +345,19 @@ case which do TwoNode.table(single ++ [two]) "b3" -> - # Transport A/B. Run this twice: once normally, once with the worker AND this - # process started under GGML_RDMA_DEV=nonexistent. There is no runtime - # switch — that variable and a -DGGML_RPC_RDMA=OFF build are the only levers. - path = models["120b"] + # Transport A/B. Run once per transport: normally (RDMA on the fabric), with + # GGML_RPC_NO_RDMA=1 set on the worker (`rpc-worker.sh start --tcp`) for TCP + # over the same 200 GbE link, and with the worker on the 10 GbE LAN + # (`--link lan`) for the management network. SPARK_TRANSPORT labels the row. + path = System.get_env("SPARK_TWO_NODE_MODEL") || models["120b"] register.() {local, remote} = device_names.() - transport = if System.get_env("GGML_RDMA_DEV") in [nil, ""], do: "RDMA", else: "TCP (forced)" + transport = System.get_env("SPARK_TRANSPORT") || "RDMA" row = TwoNode.measure( - "120b two-node over #{transport}", + "#{Path.basename(path, ".gguf")} two-node over #{transport}", path, [ n_parallel: 1, @@ -366,34 +373,7 @@ case which do TwoNode.table([row]) - "b4" -> - # Per-token RTT is fixed per graph, not per token, so a bigger batch should - # amortise it. - path = models["120b"] - register.() - {local, remote} = device_names.() - - rows = - for n_parallel <- [1, 4, 8] do - TwoNode.measure( - "120b two-node n_parallel=#{n_parallel}", - path, - [ - n_parallel: n_parallel, - n_ctx: n_ctx * n_parallel, - cache_prompt: false, - devices: [local, remote], - split_mode: :layer, - tensor_split: [0.5, 0.5] - ], - prompt_tokens, - worker_node - ) - end - - TwoNode.table(rows) - other -> - IO.puts(:stderr, "unknown bench #{inspect(other)}; expected b1, b2, b3 or b4") + IO.puts(:stderr, "unknown bench #{inspect(other)}; expected b1, b2 or b3") System.halt(2) end diff --git a/c_src/llama_cpp_ex/llama_nif.cpp b/c_src/llama_cpp_ex/llama_nif.cpp index 599e67e..454aa0f 100644 --- a/c_src/llama_cpp_ex/llama_nif.cpp +++ b/c_src/llama_cpp_ex/llama_nif.cpp @@ -6,6 +6,9 @@ #include "log.h" #include "sampling.h" #include "speculative.h" +// Staging API: llama_model_target_layer_ids{,_n}, to validate a DFlash drafter +// before upstream's constructor asserts on it. decision.cpp includes it the same way. +#include "../../vendor/llama.cpp/src/llama-ext.h" #include #include #include @@ -26,8 +29,12 @@ // with only -I flags. #ifdef GGML_USE_RPC #include +#include #include +#include +#include #include +#include #include #include #endif @@ -524,10 +531,13 @@ LLAMA_NIF(backend_free, 0); // 1. RPC_STATUS_ASSERT is GGML_ABORT (ggml-rpc.cpp:30). Any peer crash, // network failure or malformed response terminates the OS process — the // whole BEAM. There is no error return, no retry, no reconnect. -// 2. Registration, by contrast, is safe: an unreachable endpoint makes -// ggml_backend_rpc_add_server return nullptr. So failures are *detectable -// before load* and *fatal during it*, and that asymmetry is why -// rpc_add_server reports rather than logs. +// 2. Since #18626 (d0132a680) registration aborts too: the connect and the +// HELLO moved into rpc_dispatcher::start, which GGML_ABORTs on an +// unreachable endpoint, an unparseable one, or a version mismatch where +// ggml_backend_rpc_add_server used to return nullptr. rpc_add_server +// therefore probes the endpoint itself first (rpc_probe_hello), so a dead +// or mismatched node is still *detectable before load* and only *fatal +// during it*. #ifdef GGML_USE_RPC @@ -607,6 +617,122 @@ bool rpc_wait_until_listening(const std::string& host, const std::string& port, return false; } +// rpc_probe_hello speaks the HELLO exchange by hand, so it has to track the +// wire layout in ggml-rpc.cpp. Upstream changes that layout only together with +// the major version (the server's "client needs upgrade to protocol v%d.x"), +// so a major bump stops the build here instead of failing every probe at run +// time. +static_assert(RPC_PROTO_MAJOR_VERSION == 8, + "RPC protocol major changed: re-check rpc_probe_hello against " + "rpc_msg_hello_req/rsp and RPC_CMD_HELLO in ggml-rpc.cpp"); + +constexpr uint8_t kRpcCmdHello = 14; // static_assert'ed stable upstream +constexpr size_t kRpcConnCapsSize = 24; // transport.h RPC_CONN_CAPS_SIZE +// rpc_msg_hello_rsp: major, minor, patch, padding, conn_caps[24] +constexpr size_t kRpcHelloRspSize = 4 + kRpcConnCapsSize; + +bool rpc_send_all(int fd, const void* data, size_t len) { + const auto* p = static_cast(data); + while (len > 0) { + ssize_t n = send(fd, p, len, MSG_NOSIGNAL); + if (n <= 0) { + if (n < 0 && errno == EINTR) continue; + return false; + } + p += n; + len -= static_cast(n); + } + return true; +} + +bool rpc_recv_all(int fd, void* data, size_t len) { + auto* p = static_cast(data); + while (len > 0) { + ssize_t n = recv(fd, p, len, 0); + if (n <= 0) { + if (n < 0 && errno == EINTR) continue; + return false; + } + p += n; + len -= static_cast(n); + } + return true; +} + +// Connects to the endpoint and performs a TCP-only HELLO (no transport +// capabilities advertised, so the worker never starts an RDMA upgrade for it), +// then closes. True when the peer answered with a protocol the client side +// accepts — the same test negotiate_hello applies. +// +// The endpoint is parsed and resolved exactly as rpc_dispatcher::start does +// (first ':', std::stoi, IPv4 only), because anything this accepts goes on to +// that function, where every failure is an abort. A worker that dies between +// the probe and the registration still aborts; that window is upstream's. +bool rpc_probe_hello(const std::string& endpoint, int timeout_ms) { + size_t colon = endpoint.find(':'); + if (colon == std::string::npos) return false; + std::string host = endpoint.substr(0, colon); + int port; + try { + port = std::stoi(endpoint.substr(colon + 1)); + } catch (...) { + return false; + } + if (port <= 0 || port > 65535) return false; + + addrinfo hints{}; + hints.ai_family = AF_INET; + hints.ai_socktype = SOCK_STREAM; + addrinfo* res = nullptr; + if (getaddrinfo(host.c_str(), std::to_string(port).c_str(), &hints, &res) != 0 || !res) { + return false; + } + + bool ok = false; + for (addrinfo* ai = res; ai && !ok; ai = ai->ai_next) { + int fd = socket(ai->ai_family, ai->ai_socktype | SOCK_CLOEXEC, ai->ai_protocol); + if (fd < 0) continue; + + int flags = fcntl(fd, F_GETFL, 0); + fcntl(fd, F_SETFL, flags | O_NONBLOCK); + int rc = connect(fd, ai->ai_addr, ai->ai_addrlen); + if (rc != 0 && errno == EINPROGRESS) { + pollfd pfd{fd, POLLOUT, 0}; + int err = 0; + socklen_t len = sizeof(err); + rc = (poll(&pfd, 1, timeout_ms) == 1 && + getsockopt(fd, SOL_SOCKET, SO_ERROR, &err, &len) == 0 && err == 0) + ? 0 : -1; + } + if (rc != 0) { + close(fd); + continue; + } + fcntl(fd, F_SETFL, flags); + timeval tv{timeout_ms / 1000, (timeout_ms % 1000) * 1000}; + setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv)); + setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &tv, sizeof(tv)); + + // | cmd (1) | request_size (8) | conn_caps (24, zeroed) | + uint8_t req[1 + sizeof(uint64_t) + kRpcConnCapsSize] = {}; + req[0] = kRpcCmdHello; + const uint64_t req_size = kRpcConnCapsSize; + std::memcpy(req + 1, &req_size, sizeof(req_size)); + + uint64_t rsp_size = 0; + uint8_t rsp[kRpcHelloRspSize] = {}; + ok = rpc_send_all(fd, req, sizeof(req)) && + rpc_recv_all(fd, &rsp_size, sizeof(rsp_size)) && + rsp_size == kRpcHelloRspSize && + rpc_recv_all(fd, rsp, sizeof(rsp)) && + rsp[0] == RPC_PROTO_MAJOR_VERSION && + rsp[1] <= RPC_PROTO_MINOR_VERSION; + close(fd); + } + freeaddrinfo(res); + return ok; +} + } // namespace #endif // GGML_USE_RPC @@ -636,7 +762,9 @@ LLAMA_NIF(rpc_supported, 0); // the remote devices. // // Returns the number of devices the endpoint contributed. Upstream memoizes per -// endpoint, so a repeat call is a no-op and returns 0 added. +// endpoint, so a repeat call is a no-op and returns 0 added — but the probe +// still runs, so a registered peer that has since gone away reports +// :unreachable. std::variant, fine::Error> rpc_add_server(ErlNifEnv* env, std::string endpoint) { #ifndef GGML_USE_RPC @@ -645,12 +773,18 @@ rpc_add_server(ErlNifEnv* env, std::string endpoint) { #else size_t before = ggml_backend_dev_count(); + // ggml_backend_rpc_add_server aborts the VM on an unreachable or + // mismatched endpoint (see the top of this section), so check first. + if (!rpc_probe_hello(endpoint, 5000)) { + return fine::Error(atoms::unreachable); + } + ggml_backend_reg_t reg = ggml_backend_rpc_add_server(endpoint.c_str()); if (!reg) { - // Both an unreachable endpoint and a HELLO major/minor mismatch collapse - // to nullptr here, and ggml_backend_register silently no-ops on nullptr - // — so without this check a dead or mismatched node simply vanishes and - // the model loads onto the wrong devices. + // A worker reporting zero devices still comes back as nullptr, and + // ggml_backend_register silently no-ops on nullptr — so without this + // check that node simply vanishes and the model loads onto the wrong + // devices. return fine::Error(atoms::unreachable); } @@ -822,25 +956,23 @@ LLAMA_NIF(device_list, ERL_NIF_DIRTY_JOB_IO_BOUND); // --- Model --- std::variant>, fine::Error> -model_load(ErlNifEnv* env, std::string path, int64_t n_gpu_layers, bool use_mmap, +model_load(ErlNifEnv* env, std::string path, int64_t n_gpu_layers, int64_t load_mode, int64_t main_gpu, int64_t split_mode, std::vector tensor_split, - bool use_mlock, bool use_direct_io, bool vocab_only, bool check_tensors, + bool vocab_only, bool check_tensors, bool load_mtp, std::vector device_names) { + if (load_mode < LLAMA_LOAD_MODE_AUTO || load_mode > LLAMA_LOAD_MODE_DIRECT_IO) { + return fine::Error("invalid load_mode " + std::to_string(load_mode)); + } auto params = llama_model_default_params(); params.n_gpu_layers = static_cast(n_gpu_layers); params.main_gpu = static_cast(main_gpu); params.split_mode = static_cast(split_mode); - // Upstream collapsed the use_mmap/use_mlock/use_direct_io booleans into a - // single llama_load_mode enum. Direct I/O still wins over everything else, - // but mlock no longer implies mmap: b10173 redefined LLAMA_LOAD_MODE_MLOCK - // as mlock *without* mmap and added LLAMA_LOAD_MODE_MMAP_MLOCK for the - // combination, so :use_mlock and :use_mmap are both honoured rather than - // the former silently forcing a memory map. - params.load_mode = use_direct_io ? LLAMA_LOAD_MODE_DIRECT_IO - : use_mlock && use_mmap ? LLAMA_LOAD_MODE_MMAP_MLOCK - : use_mlock ? LLAMA_LOAD_MODE_MLOCK - : use_mmap ? LLAMA_LOAD_MODE_MMAP - : LLAMA_LOAD_MODE_NONE; + // Resolved from :use_mmap / :use_mlock / :use_direct_io by + // LlamaCppEx.Model.load/2. LLAMA_LOAD_MODE_AUTO (none of them given) is + // upstream's default and the one that matters on an iGPU: it skips mmap + // there (#26081), and an explicit mmap on GB10 left most loads decoding + // ~15% slower. + params.load_mode = static_cast(load_mode); params.vocab_only = vocab_only; params.check_tensors = check_tensors; // Upstream defaults this to false so that non-speculative callers do not pay @@ -950,6 +1082,22 @@ int64_t model_n_layer_nextn(ErlNifEnv* env, fine::ResourcePtr model) } LLAMA_NIF(model_n_layer_nextn, 0); +// Names of the devices the model's layers were placed on, in placement order. +// LlamaCppEx.MTP uses it to refuse a DFlash drafter that cannot reach the +// target's embeddings and LM head (shared via ctx_other): a draft context +// without the device holding them hits "pre-allocated tensor ... in a buffer +// that cannot run the operation", a GGML_ABORT. +std::vector model_devices(ErlNifEnv* env, fine::ResourcePtr model) { + std::vector names; + const int32_t n = llama_model_n_devices(model->model); + for (int32_t i = 0; i < n; i++) { + ggml_backend_dev_t dev = llama_model_get_device(model->model, i); + if (dev) names.emplace_back(ggml_backend_dev_name(dev)); + } + return names; +} +LLAMA_NIF(model_devices, 0); + std::string model_desc(ErlNifEnv* env, fine::ResourcePtr model) { char buf[256]; llama_model_desc(model->model, buf, sizeof(buf)); @@ -2651,14 +2799,15 @@ chat_session_parse(ErlNifEnv* env, fine::ResourcePtr session, // Dirty: re-parses the whole reply, so its cost grows with the reply. LLAMA_NIF(chat_session_parse, ERL_NIF_DIRTY_JOB_CPU_BOUND); -// --- Speculative decoding (MTP) --- +// --- Speculative decoding (MTP, DFlash, DSpark) --- std::variant>, fine::Error> speculative_init( ErlNifEnv* env, fine::ResourcePtr ctx_tgt, fine::ResourcePtr ctx_dft, - int64_t n_draft) + int64_t n_draft, + fine::Atom type) { if (n_draft <= 0) { return fine::Error(std::string("n_draft must be > 0")); @@ -2667,22 +2816,66 @@ speculative_init( return fine::Error(kNoLogitsMessage); } + common_speculative_type spec_type; + const std::string type_name = type.to_string(); + if (type_name == "mtp") { + spec_type = COMMON_SPECULATIVE_TYPE_DRAFT_MTP; + } else if (type_name == "dflash") { + spec_type = COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH; + } else if (type_name == "dspark") { + spec_type = COMMON_SPECULATIVE_TYPE_DRAFT_DSPARK; + } else { + return fine::Error("unknown speculative type :" + type_name); + } + + const llama_model* model_tgt = llama_get_model(ctx_tgt->ctx); + const llama_model* model_dft = llama_get_model(ctx_dft->ctx); + + // The DFlash/DSpark constructor GGML_ASSERTs that the draft names the + // target layers it reads, and it reads them by index from the target: a + // draft that is not a DFlash checkpoint, or one trained for a different + // target, aborts the VM instead of failing. Check both here. + if (spec_type != COMMON_SPECULATIVE_TYPE_DRAFT_MTP) { + const uint32_t n_ids = llama_model_target_layer_ids_n(model_dft); + if (n_ids == 0) { + return fine::Error(std::string( + "draft model is not a DFlash/DSpark drafter (it names no target layers)")); + } + const int32_t* ids = llama_model_target_layer_ids(model_dft); + const int32_t n_layer_tgt = llama_model_n_layer(model_tgt); + for (uint32_t k = 0; k < n_ids; k++) { + if (ids[k] < 0 || ids[k] >= n_layer_tgt) { + return fine::Error("draft model reads target layer " + std::to_string(ids[k]) + + ", but the target has " + std::to_string(n_layer_tgt) + + " layers; the drafter belongs to a different model"); + } + } + if (!common_speculative_are_compatible(model_tgt, model_dft)) { + return fine::Error(std::string( + "draft and target vocabularies are incompatible; the drafter belongs to a different model")); + } + } + // Probe partial-rollback support on the target context BEFORE // common_speculative_init. Two reasons: // 1. common_context_can_seq_rm clears the context's KV memory as a // side effect (see common.h:904). // 2. common_speculative_init's MTP impl calls - // llama_set_embeddings_pre_norm(ctx_tgt, true) in its constructor. - // Probing afterwards would clobber that flag and the MTP head - // would see garbage hidden states (acceptance drops to ~5%). - // Dense attention-only models report PART → partial seq_rm is native, - // skip the per-iter checkpoint path. Hybrid models (Qwen 3.6 MoE with - // GDN layers) report FULL → checkpoint every iteration. + // llama_set_embeddings_pre_norm(ctx_tgt, true) in its constructor, + // and the DFlash impl turns on the target-layer extraction there. + // Probing afterwards would clobber those flags and the draft would + // see garbage hidden states (MTP acceptance drops to ~5%). + // Dense attention-only models report PART → partial seq_rm is native. + // A hybrid target (Qwen 3.6/3.8: GDN layers beside attention) reports RS + // when it was created with n_rs_seq > 0 — it keeps that many recurrent + // snapshots and rolls back natively up to that depth, which is how + // upstream's server runs it (n_rs_seq = n_draft) — and FULL without them, + // which costs a state save every iteration plus a restore and re-decode on + // every partial accept. needs_ckpt is settled below, once n_draft is. const auto rm_type_tgt = common_context_can_seq_rm(ctx_tgt->ctx); - const bool needs_ckpt = (rm_type_tgt == COMMON_CONTEXT_SEQ_RM_TYPE_FULL); common_params_speculative params; - params.types = { COMMON_SPECULATIVE_TYPE_DRAFT_MTP }; + params.types = { spec_type }; params.draft.n_max = static_cast(n_draft); params.draft.ctx_tgt = ctx_tgt->ctx; params.draft.ctx_dft = ctx_dft->ctx; @@ -2696,10 +2889,25 @@ speculative_init( if (!spec) { return fine::Error(std::string( - "common_speculative_init returned null — does the model contain MTP heads " - "and does the draft context have ctx_type=:mtp with n_rs_seq>0?")); + "common_speculative_init returned null — does the draft context match " + "the speculative type (ctx_type: :mtp for MTP heads)?")); } + // The DFlash impl clamps n_max to the drafter's trained block size; keep + // what it settled on so the loop never asks for more. + const int32_t n_max = common_speculative_n_max(spec); + if (n_max > 0 && n_max < n_draft) { + n_draft = n_max; + } + + // Snapshots cover a verify batch of up to n_rs_seq + 1 tokens, i.e. a draft + // of up to n_rs_seq; past that the RS path is not guaranteed and the loop + // falls back to checkpoints, like upstream's server. + const bool needs_ckpt = + rm_type_tgt == COMMON_CONTEXT_SEQ_RM_TYPE_FULL || + (rm_type_tgt == COMMON_CONTEXT_SEQ_RM_TYPE_RS && + static_cast(n_draft) > llama_n_rs_seq(ctx_tgt->ctx)); + auto res = fine::make_resource( spec, std::move(ctx_tgt), std::move(ctx_dft), static_cast(n_draft), needs_ckpt); @@ -3207,6 +3415,15 @@ fine::Ok<> generate_mtp_tokens( // then — including it here would duplicate it in the context. // `sampled` sits at prompt[size - n_accepted_total - 1], since // the accept loop pushed n_accepted_total tokens after it. + // + // The draft context was rolled back to n_past above, so the + // drafter has lost these positions too: hand the re-decode to + // common_speculative_process exactly as the verify batch was, + // or the next draft is conditioned on a hole. A DFlash drafter + // reads the target's hidden states only through process, and + // without this its acceptance collapsed to ~3% on a hybrid + // target. Every row asks for output, as the verify batch did, + // because the drafter's extraction follows the output rows. if (n_accepted_total > 0) { llama_batch redo = llama_batch_init(n_accepted_total, 0, 1); BatchFreeGuard redo_guard(redo); @@ -3215,13 +3432,17 @@ fine::Ok<> generate_mtp_tokens( prompt[prompt.size() - n_accepted_total - 1 + i]; batch_push(redo, tok, n_past + static_cast(i), seq_id, - /*logits=*/ i == n_accepted_total - 1); + /*logits=*/ true); } int ret = decode_tracked(*sp.ctx_tgt, redo); if (ret != 0) { send_error("rollback re-decode failed: code=" + std::to_string(ret)); return fine::Ok(); } + if (!speculative_process(sp.spec, spec_batch, redo)) { + send_error("common_speculative_process failed after rollback"); + return fine::Ok(); + } } } else { // Dense model: native partial seq_rm trims the unaccepted diff --git a/docs/dgx-spark.md b/docs/dgx-spark.md index 664fe17..c986e77 100644 --- a/docs/dgx-spark.md +++ b/docs/dgx-spark.md @@ -5,6 +5,17 @@ across two nodes. Everything here is measured on the hardware rather than inferred from spec sheets; where a number differs from the marketing figure, the measurement wins and the reason is given. +The work behind this page produced four llama.cpp branches on +`nyo16/llama.cpp`, each based on `785322d17`, the vendored commit. None of them +is in the vendored tree. A fifth, `spark-all`, merges all four. + +| branch | effect on GB10 | section | +|---|---|---| +| `cuda-mmq-src1-reuse` | NVFP4 prefill +19–20% (src1 reuse, weight-scale fold, SWIGLU in MMQ); gpt-oss-120b prefill +8% (expert biases) | Qwen3.8-27B NVFP4, gpt-oss-120b prefill | +| `rpc-client-spin-wait` | two-node decode +30% unpinned (gpt-oss-120b 44 → 57.6 tok/s, single node 59) | The cause, and a fix | +| `rpc-add-server-no-abort` | an unreachable worker returns an error instead of aborting | Five things, #1 | +| `draft-ctx-other-devices` | DFlash with the target split over RPC | Speculative decoding across two nodes | + ## The machine | | | @@ -25,9 +36,10 @@ Use `free -h`. `LlamaCppEx.devices/0` reports it correctly (130.7 GB total on `:igpu` for the same reason. **The CPU clusters are interleaved.** The performance cores are **5-9, 15-19**; -the efficiency cores are **0-4, 10-14**. `taskset -c 0-9` therefore pins a job -entirely to little cores. Measured on a 1-byte RDMA ping-pong: p50 21.5 µs on -cpu19 versus 29.1 µs on cpu0. +the efficiency cores are **0-4, 10-14**. `taskset -c 0-9` therefore gives a job +five of each, not the X925 cluster. Measured on a 1-byte RDMA ping-pong: p50 +21.5 µs on cpu19 versus 29.1 µs on cpu0. Even so, a two-node *client* decodes +25% faster on the A725 cores (see "Pin the client to the little cores"). **`ping` lies by three orders of magnitude.** ICMP RTT over the 200 Gb/s direct link reads ~1.2 ms. That is cpuidle exit latency (LPI-3 is 433 µs), not the wire @@ -138,7 +150,7 @@ LLAMA_CUDA_ARCH=121a-real \ `remote.sh` exports all three, so on a Spark this is just `mix compile`. -### Why `LLAMA_CPU_ARM_ARCH` is not optional +### Why `LLAMA_CPU_ARM_ARCH` matters, and when it does not ggml probes the host with `-mcpu=native` when `GGML_NATIVE` is on, which is the default. On GB10 with GCC 13.3 that probe **fails silently**: the compiler @@ -154,6 +166,21 @@ Measured on the emitted `libggml-cpu.a`: Those are the Q4/Q8 quantized matmul kernels. +They run only when the CPU does the math: CPU-only builds, `n_gpu_layers` +below the layer count, MoE experts kept on the CPU. With every layer on the +GPU the flag changes nothing measurable. A fresh project depending on the +published package (no `LLAMA_*` variables, so `-mcpu=native`, CUDA picked up +as `121a-real` automatically) against this repo's tuned build, Qwen3.6-35B-A3B +fully offloaded on spark-2, median of 5: + +| build | TTFT, ~2k-token prompt | decode | +|---|---|---| +| `{:llama_cpp_ex, "0.8.66"}` from Hex, defaults | 698.0 ms | 66.78 tok/s | +| `LLAMA_CPU_ARM_ARCH` + `LLAMA_CUDA_ARCH` | 702.8 ms | 66.49 tok/s | + +So a Hex user on a Spark gets full GPU speed with no flags; set them for +CPU-side work. + ### Why `LLAMA_CUDA_ARCH` must accompany it Naming the CPU architecture requires `GGML_NATIVE=OFF`, and with native off @@ -234,19 +261,36 @@ clean A/B for what speculative decoding buys on this hardware. See ## Running on ONE Spark -Full numbers and methodology in -`bench/results/v0.8.43-dgx-spark-baseline.md`, and the two-node numbers in -`bench/results/v0.8.43-dgx-spark-two-node.md`. -The short version: - -| model | prefill (pp) | decode (tg) | -|---|---|---| -| Qwen3-8B Q4_K_M | 3500–4331 t/s | 40.5 t/s | -| Qwen3-30B-A3B Q4_K_M | 3287 t/s | 90.9 t/s | - -Three of the four figures beat the published single-Spark references; the fourth -is 7% under. **Prefill is this machine's strong suit** and decode is bandwidth- -bound, so quote them separately or you describe neither. +Measured at `785322d17` with every layer on the GPU. Prefill and the +single-stream decode reference are `llama-bench` (`-fa 1`, median of 3); +the rest is the binding through `bench/spark_overhead.exs` — `stream` is +`LlamaCppEx.stream/3`, the others `LlamaCppEx.Server` with that many +concurrent streams (per-stream decode, then aggregate): + +| model | size | pp512 | pp4096 | tg128 (upstream) | `stream` | server ×1 | server ×4 | server ×8 | +|---|---|---|---|---|---|---|---|---| +| Qwen3-8B Q4_K_M | 4.7 GiB | 3623 | 3500 | 43.9 | 44.1 | 43.7 | 39.0 / **153** | 36.4 / **284** | +| Qwen3-30B-A3B Q4_K_M | 17.3 GiB | 3282 | 3198 | 92.7 | 91.4 | 88.6 | 57.8 / **224** | 34.9 / **272** | +| Qwen3.6-35B-A3B UD-Q4_K_M | 20.6 GiB | 2918 | 2823 | 69.8 | 68.3 | 63.0 | 43.0 / **165** | 31.5 / **245** | +| Qwen3.8-27B Q4_K_M | 17.7 GiB | 1004 | 1008 | 11.3 | 11.3 | 11.2 | 9.9 / **39** | 8.4 / **66** | +| Qwen3.8-27B NVFP4 MID-HIGH | 15.7 GiB | 1278 | 1275 | 14.4 | 14.3 | 14.3 | 12.2 / **48** | 10.0 / **78** | +| gpt-oss-120b MXFP4 | 59.0 GiB | 1923 | 1990 | 49.2–59.0 | 57.0 | 55.9 | 23.8 / **91** | 13.5 / **105** | + +**Prefill is this machine's strong suit** and decode is bandwidth-bound, so +quote them separately or you describe neither. The dense models scale almost +linearly with concurrency (8 streams of Qwen3-8B cost 17% per stream for 6.4× +the aggregate); the MoE models scale less, because each additional stream +activates different experts and the step reads more weight. The binding's +single-stream decode is within 2% of `llama-bench` everywhere. + +> #### Measure decode more than once per load {: .warning} +> +> gpt-oss-120b's `llama-bench` tg128 came out at 49.2 in one run of a +> six-model sweep and 58.8–59.0 in fourteen others; the same ~17% dip +> appeared per process, with no change in clocks, page cache or memory +> fragmentation that would explain it. Forcing `mmap` made it the common +> case (see the load-mode row below). Treat any single decode figure on +> this machine as one sample. ### Settings that matter @@ -265,10 +309,11 @@ And the settings that do **not** matter, each of which looks like it should: |---|---| | `flash_attn` | leave `:auto` — it is already on, and `:disabled` costs 24% of prefill | | `type_k` / `type_v` | leave f16. Quantizing the KV cache **loses** 2–7% and buys nothing: there is no separate VRAM to free | -| `use_mlock`, `use_direct_io` | leave off. Both cost 3–4%; "pinned in RAM" and "resident on the GPU" are the same DRAM here | +| `use_mmap`, `use_mlock`, `use_direct_io` | **leave all three unset.** Upstream's `auto` then reads the file instead of mapping it on this iGPU. A forced `use_mmap: true` — the binding's default before this run — left most gpt-oss-120b loads decoding at 48.5 tok/s against 57 with `auto`, `use_mmap: false` or `use_direct_io: true`; about one forced-mmap load in five came out fast, so single-load comparisons from before (this row used to say mlock and dio cost 3–4%) are not reliable | | `n_batch`, `n_ubatch` | no measurable effect; the default is fine | +| `GGML_CUDA_GRAPH_OPT=1` (env) | upstream's experimental concurrent-stream graph optimisation: **+2%** decode on gpt-oss-120b (60.1 against 59.0, three runs each), neutral on Qwen3-8B, Qwen3.6-35B-A3B and Qwen3.8-27B NVFP4 | | cpuidle / `idle=poll` | **no effect**, single-node or two-node. See below | -| `taskset -c 5-9,15-19` | hygiene, not speed. But never use `-c 0-9` — those are the little cores | +| `taskset` | single node: hygiene, not speed. Two nodes: run the **client** on the A725 cores, `taskset -c 0-4,10-14`, for **+25% decode** (see "Pin the client to the little cores"). Note `-c 0-9` is half little cores, not the X925s | ### The cpuidle story, and why you can ignore it @@ -285,80 +330,197 @@ C-state and there is no exit latency to avoid. **So do not go asking for `idle=poll` on the kernel cmdline.** It was the one thing this work expected to need root for, and the measurement retired it. -## Qwen3.6 and speculative decoding (MTP) +## The binding costs nothing measurable -Two shapes of the current generation, each measured with and without the -Multi-Token Prediction head. The MTP repos are the *same weights* plus the head, -so this is a clean A/B rather than a comparison across quantizations. +Same model, same llama.cpp commit (`785322d17`), upstream's own tools against +`bench/spark_overhead.exs`. Qwen3-8B Q4_K_M, 128 greedy tokens, decode timed +from the first token to the last: -256-token greedy generations through the chat template, **median of 5** with the -range alongside. Five samples matter here: MTP is the noisier arm, and a single -run per setting is not enough to tell a real 1.6× from a lucky draw. The plain -model is freed before the MTP arm runs — leaving ~20 GB of unrelated weights -resident cost the MTP arm about 10% on this unified-memory part, which is -exactly the kind of confound that flips a conclusion. - -### Qwen3.6-27B Q4_K_M — dense - -| config | decode t/s | range | vs baseline | acceptance | +| path | streams | upstream | llama_cpp_ex | +|---|---|---|---| +| single stream (`llama-bench` tg128 / `LlamaCppEx.stream/3`) | 1 | 44.0 tok/s | 43.5 tok/s | +| server (`llama-server -np 8` / `Server`, `n_parallel: 8`) | 8 | 286 tok/s aggregate | 286 tok/s aggregate | + +`llama-batched-bench` reaches 311 tok/s at 8 streams because it never samples: +every server has to wait for the logits, sample on the CPU and submit the next +step, and that serial dependency is the 8%. Upstream's `--backend-sampling` +(sampling on the GPU) measured within 1% of CPU sampling here, so it is not +exposed. Tuning on this machine is therefore model and drafter choice, not the +binding. + +The same holds for MoE decode. gpt-oss-120b, 256 greedy tokens, five fresh +processes: `LlamaCppEx.stream/3` gives 57.0–57.3 tok/s with the default load +mode, `use_mmap: false` or `use_direct_io: true` alike, against 57.8 for +`llama-bench` tg256. (Under `nsys --cuda-graph-trace=node` the binding drops +to 55.5–55.9 and its expert matvecs read 123 µs against `llama-bench`'s 118 µs. +That gap is the profiler's, so do not compare kernel times across processes +profiled this way.) + +## Qwen3.8-27B NVFP4 + +`esatapedico/Qwen3.8-27B-NVFP4-MTP-GGUF`, `MID-HIGH` tier: every attention and +FFN weight in native NVFP4 (GGML type 40), Q8_0 heads, the MTP head embedded. +Against the Q4_K_M build of the same model, `llama-bench`: + +| | size | pp512 | pp2048 | tg128 | |---|---|---|---|---| -| no MTP head | 11.59 | 11.5–11.6 | — | — | -| MTP `n_draft: 1` | 16.88 | 16.9–16.9 | 1.46× | 86.9% | -| MTP `n_draft: 2` | 18.38 | 18.2–18.4 | 1.59× | 76.4% | -| **MTP `n_draft: 3`** | **18.65** | 18.4–18.7 | **1.61×** | 68.2% | -| MTP `n_draft: 4` | 17.42 | 17.3–17.5 | 1.50× | 57.1% | - -No range overlaps the baseline: **MTP is worth 1.6× on the dense model**, and -`n_draft` 2 and 3 are within noise of each other. - -### Qwen3.6-35B-A3B UD-Q4_K_M — MoE - -| config | decode t/s | range | vs baseline | acceptance | +| Q4_K_M | 17.7 GiB | 1015 | 1023 | 11.21 | +| NVFP4 MID-HIGH | 15.7 GiB | 1236 | 1265 | **14.19** | + +NVFP4 is +22% prefill and +27% decode. Decode reads 15.7 GiB per token at +14.2 tok/s — 240 GB/s, 88% of GB10's 273 GB/s — so it is bandwidth-bound and +there is little left to take from the kernels. Prefill runs W4A4 on the +Blackwell FP4 MMA (`mma.sync ... kind::mxf4nvf4`); an `nsys` profile of pp2048 +puts 53% of GPU time in that matmul, 9% in activation quantization and ~23% in +element-wise ops (SwiGLU, gating muls, adds, the GDN conv-state concat). + +Part of that 9% is waste: projections that read the same activations (Q/K/V, +gate/up, the GDN input projections) quantize them once each — 4450 +`quantize_mmq_nvfp4` launches for a 2048-token prefill. Branch +`cuda-mmq-src1-reuse` on `nyo16/llama.cpp` keeps the last quantized MMQ +input and reuses it until something overwrites the source: 2540 launches, +**+2–2.6% prefill** on Qwen3.8-27B NVFP4 (1279 → 1312 tok/s at pp512), +0.5–1% +on the Q4_K models, perplexity bit-identical. + +The per-tensor NVFP4 weight scale is the next cheap win. llama.cpp builds each +NVFP4 projection as `mul(mul_mat(w, x), w_s)` with a scalar `w_s`. Decode +(MMVQ) already fuses that multiply, but batched MMQ runs it as a separate +broadcast kernel over the output: 224 launches and ~14% of pp512 GPU time. The +second commit on the same branch folds `w_s` into the per-token activation +scale that the NVFP4 write-back already applies. + +The third commit removes the SWIGLU output. An FFN down projection reads +`swiglu(gate, up)`, the widest activation in the model, which the GLU kernel +writes and the quantizer then reads twice. The fused quantizer reads `gate` and +`up` and quantizes `silu(gate) * up` directly. It uses the same expression as +the GLU kernel, so the quantized blocks are bit-identical. It needs 512 threads +per row: at the stock 128 the fused kernel is latency-bound and gains nothing. + +| `llama-bench -fa 1` | pp512 | pp2048 | pp8192 | tg128 | +|---|---|---|---|---| +| Qwen3.8-27B NVFP4, upstream | 1284 | 1297 | 1250 | 14.35 | +| + src1 reuse + scale fold | 1411 | 1422 | 1373 | 14.31 | +| + SWIGLU quantized in MMQ | **1541** | **1552** | **1488** | 14.30 | +| Qwen3-8B Q4_K_M, upstream | 3577 | 3565 | 3214 | 43.97 | +| + src1 reuse (no NVFP4 fusions apply) | 3626 | 3611 | 3237 | 44.07 | + +Together, the three commits give **+19–20% NVFP4 prefill**, with decode +unchanged. Only the scale fold changes numerics, because it moves where the +scale is rounded in. Over 8×512 tokens, its KL divergence against upstream is +0.051. Upstream's native FP4 path differs from itself by about the same amount +(0.052) when only `-ub 256` changes. With Q8 activations +(`GGML_CUDA_MMQ_PREC=q8`), the divergence is 0.0007. The SWIGLU fusion leaves +it at exactly 0.050913. The new `test-backend-ops` case, `MUL_MAT_SCALE`, +compares the fused graphs against the CPU backend. It fails when the fold is +deliberately off by 2× or the silu is dropped. None of these patches are in +the vendored llama.cpp; all three are upstream candidates. + +## gpt-oss-120b prefill: the expert biases + +gpt-oss adds a per-expert bias after every expert projection: an `ADD_ID` after +the up, gate and down `MUL_MAT_ID`s. Decode fuses these into MMVQ, but prefill +made three passes over the `[n_ff, n_expert_used, n_tokens]` projections. That +was 7% of pp2048 GPU time. A fourth commit on the same branch moves the +up and gate biases into the GLU kernel and the down bias into the +weighted expert reduction. It keeps the unfused order of float operations, +so the KL divergence against upstream is 0.000000 with 100% same top token: + +| gpt-oss-120b, `llama-bench -fa 1 -r 5` | pp512 | pp2048 | pp8192 | tg128 | |---|---|---|---|---| -| no MTP head | 65.36 | 65.1–65.8 | — | — | -| MTP `n_draft: 1` | 67.56 | 67.0–67.7 | 1.03× | 81.0% | -| MTP `n_draft: 2` | 62.29 | 61.4–62.3 | 0.95× | 67.2% | -| MTP `n_draft: 3` | 63.28 | 63.0–63.5 | 0.97× | 64.9% | -| MTP `n_draft: 4` | 47.66 | 44.8–48.6 | 0.73× | 40.9% | - -Essentially neutral at best, and a loss past `n_draft: 1`. - -> The README reports **+16%** at `n_draft: 2` for Qwen3.6-35B-A3B on GB10, from -> an interleaved n=11 run on **UD-Q4_K_XL**. This measurement is UD-Q4_K_**M**, -> and the draft acceptance rates agree closely (67.2% here versus 68.5% there at -> `n_draft: 2`) while the throughput economics do not. Take the quantization as -> the likely difference and measure your own before relying on either number. - -### The rule this gives you - -**MTP pays on dense models and roughly breaks even on sparse MoE.** The mechanism -is the one that makes this chip interesting: speculative decoding spends compute -(a batched verification pass) to save memory bandwidth (sequential decode steps). -On the dense 27B every token reads all 27B of weights, so that trade is strongly -favourable — 1.6×. On the 35B-A3B only ~3B parameters move per token, decode is -already cheap, and the draft-and-verify overhead eats the gain. - -Two more things the numbers say: - -- **The best `n_draft` is model-shaped, so measure it.** On the dense model 3 is - best and 2 is within noise; on the MoE anything above 1 loses. There is no - single default that transfers, which is also the README's conclusion for - Metal. -- **Acceptance decays fast with draft depth** — 87% → 76% → 68% → 57% on the - dense model — so past the sweet spot you pay twice: wasted draft compute and a - longer verification batch. +| upstream | 2037 | 2080 | 1993 | 58.07 | +| branch (src1 reuse + bias fusions) | **2209** | **2244** | **2147** | 58.02 | + +That is +8% prefill with decode unchanged. The full CUDA `test-backend-ops` +suite passes on the branch (19246/19246). Models with nothing to fuse stay +level. On Qwen3.6-35B-A3B and Qwen3-30B-A3B (Q4_K_M), pp512 / pp2048 / tg128 +read 2956 / 2896 / 69.9 → 2940 / 2918 / 69.8 and 3288 / 3309 / 92.6 → 3302 / +3353 / 92.6. + +### Through the binding + +`nyo16/llama.cpp` branch `spark-all` is `785322d17`, the vendored commit, plus +these four CUDA commits, the `draft-ctx-other-devices` and +`rpc-add-server-no-abort` fixes, and the RPC client spin +(`rpc-client-spin-wait`, see the two-node section). The prefill numbers below +predate the RPC commit, which does not touch single-node runs. To test it, +copy the checkout on spark-1 and replace `vendor/llama.cpp` with the branch +tree. Then remove the +`_build/*/lib/llama_cpp_ex/obj/llama_build-*` directories and run +`mix compile`. In that build, `Context.decode/2` of a 2048-token prompt +(`n_ubatch: 512`, flash attention, median of 5) gives: + +| | vendored | `spark-all` | +|---|---|---| +| Qwen3.8-27B NVFP4 | 1304 tok/s | **1560 tok/s** (+20%) | +| gpt-oss-120b MXFP4 | 2352 tok/s | **2583 tok/s** (+10%) | + +`scripts/spark/e2e.exs` passes 12/12 on that build, both `single` and `two-node`. So do the test +suite's `smoke`, `embeddings`, `tools` and `decision` tags (602 tests), `mtp` and +`dflash` (458), `rpc_live` against a worker, `vm_exit_test.exs`, and +`rpc_check.exs` over the fabric (VM exit 0). `embed_batch matches per-text +embed` used to fail on CUDA with bge-small-en-v1.5 Q8_0, on the vendored tree +as well: batched and per-text embeddings differed by up to 0.0027 per element +against a 0.001 bound tuned on CPU, because a batch runs different kernels. +The test now compares cosine similarity (> 0.999; CUDA gives 0.99988, swapped +rows 0.41–0.70, the wrong pooling ≤ 0.93) and passes on CUDA with bge-small, +embeddinggemma and Qwen3-Embedding, and on CPU. Speculative +decoding on Qwen3.8-27B NVFP4 (`bench/spark_spec.exs`) runs at plain 14.3, MTP +`n_draft: 2` 26.5 and DFlash 2 `n_draft: 7` 45.9 tok/s, against 14.3 / 26.8 / +44.2 vendored. No source change in the binding is needed for the swap, but a +release would have to vendor the branch or wait for upstream. + +## Speculative decoding: MTP, DFlash 2, DSpark + +256-token greedy chat completions (thinking off), four prompts run twice, +median decode rate. Upstream is `llama-server` with `--spec-type`; the binding +is `LlamaCppEx.MTP` (`bench/spark_spec.exs`): + +| Qwen3.8-27B NVFP4 | upstream | llama_cpp_ex | acceptance | +|---|---|---|---| +| plain | 14.17 | 14.28 | — | +| MTP `n_draft: 1` | 21.22 | — | 83% | +| MTP `n_draft: 2` | 27.27 | 26.76 | 76% | +| MTP `n_draft: 3` | 29.16 | 28.63 | 65% | +| MTP `n_draft: 4` | 29.71 | — | 56% | +| DSpark `n_draft: 7` | 25.74 | — | 22% | +| **DFlash 2 `n_draft: 7`** | **46.60** | **44.22** | 42% | + +DFlash 2 (`z-lab/Qwen3.8-27B-DFlash2-GGUF`, a 1.1 GB drafter) is **3.1×** +plain, against 2.0× for the checkpoint's own MTP head. Its `n_draft` is capped +at the drafter's block size minus one; asking for 10 measured the same as 7. +DSpark (`magnitudedev/Qwen3.8-27B-DSpark-GGUF`) lost to MTP here. + +| Qwen3.6-35B-A3B UD-Q4_K_M (MoE) | llama_cpp_ex | vs plain | acceptance | +|---|---|---|---| +| plain | 68.27 | — | — | +| MTP `n_draft: 1` | 86.86 | 1.27× | 86% | +| MTP `n_draft: 2` | **96.69** | **1.42×** | 77% | +| MTP `n_draft: 3` | 91.58 | 1.34× | 66% | -```elixir -{:ok, model} = LlamaCppEx.Model.load(mtp_path, n_gpu_layers: 99, load_mtp: true) -{:ok, session} = LlamaCppEx.MTP.init(model, n_draft: 3, n_ctx: 4096) -{:ok, text} = LlamaCppEx.MTP.generate(session, prompt, max_tokens: 256) -``` +Speculation spends compute (a wider verify) to save bandwidth (fewer +sequential decodes), and this chip has compute to spare: it pays on the dense +model and still pays on the MoE, where only ~3B parameters move per token, but +the MoE's optimum is narrower. -Reproduce: +> #### Earlier numbers on this page were held down by the binding {: .info} +> +> Before this run the MoE measured *neutral* (0.95–1.03×) and the dense 27B +> 1.6×. Both models are hybrid (GDN layers beside attention), and the binding +> rolled a rejected draft back by saving the whole recurrent state every +> iteration, then restoring it and re-decoding the accepted prefix on every +> partial accept. Upstream's server instead gives the target context one +> recurrent snapshot per draft position (`n_rs_seq`) and rolls back natively; +> `LlamaCppEx.MTP.init/2` now does the same. Qwen3.8-27B NVFP4 at +> `n_draft: 2` went from 21.1 to 26.8 tok/s. + +Greedy speculative output is not byte-identical to plain greedy decode over 256 +tokens, and that holds for upstream too: `llama-server` with DFlash 2 or MTP +diverges from its own plain run on two of the four prompts, where the verify +batch's wider matmul rounds a near-tie the other way. ```bash scripts/spark/remote.sh --env MIX_ENV=bench --big-cores spark-1 \ - mix run bench/spark_mtp.exs + mix run bench/spark_spec.exs plain mtp:2 dflash:7: ``` > #### Qwen3.6 instruct checkpoints need the chat template {: .warning} @@ -382,13 +544,18 @@ than the marketing: - **Capacity: yes.** A model that does not fit in 130 GB runs. Nothing else on this pair will run it at all. -- **Speed: no, but also not the loss you would expect.** Pipeline parallelism is - *disabled* whenever an RPC device participates — the RPC backend reports - `async = false, events = false` and llama.cpp checks that before enabling - pipelining — so the two nodes execute **sequentially**. For a model that fits - on one node, the second node measured within a few percent either way - (see B1 below), because the sequential penalty and the halved per-node - bandwidth pressure roughly cancel. +- **Speed: prefill yes, decode roughly even.** Since llama.cpp #18626, the RPC + backend implements async compute and events, so llama.cpp enables + **pipeline parallelism** across an RPC split. It used to refuse, and the + nodes ran one after the other. A prompt's micro-batches now overlap across + the two nodes: prefill measured **+18–26%** for gpt-oss-120b and +22% for + Qwen3.8-27B NVFP4 against one node. A single decode stream has nothing to + overlap. On the vendored llama.cpp, it pays thread wake-ups in the RPC + client on every command. For gpt-oss-120b, that costs 8% through + `llama-bench` and ~15% through the binding with the client on the A725 + cores, and 25–30% left unpinned. With the RPC client fix on `spark-all`, it + costs 2–3%, and the dense NVFP4 model decodes faster on two nodes than on + one (see the control below). - **Tensor parallelism ("tp=2"): technically yes, practically no.** It runs and it is correct, and it is 2.7× slower on decode. See the verdict section. @@ -485,6 +652,14 @@ wrong devices" into `{:error, :unreachable}`. After the load, treat the VM as th unit of restart. Real fault isolation means putting the RPC client in a separate OS process, which is a different architecture. +Upstream's registration aborts too since #18626 — an unreachable endpoint, a +listener that is not an RPC worker, a version mismatch and a malformed +`host:port` all reach `GGML_ABORT` in `rpc_dispatcher::start`. So the NIF +connects and exchanges the protocol HELLO itself before registering, with a +5-second timeout. That is what keeps `ping/1` a check rather than a crash on +this llama.cpp; the matching upstream fix is branch `rpc-add-server-no-abort` +on `nyo16/llama.cpp`. + ### 2. `devices/0` order is NOT `tensor_split` order Two device lists exist and they disagree: @@ -554,17 +729,23 @@ This is load-bearing, not hygiene. ### The headline: a model that does not fit on one node Qwen3-235B-A22B Q4_K_M, **142.1 GB** of weights against 130.0 GB of unified -memory per node. 512-token prompt, 32 decode steps. +memory per node. 32 decode steps. -| run | load s | TTFT ms | prefill t/s | decode t/s | worker RSS | -|---|---|---|---|---|---| -| single-node, mmap overflow | — | — | — | — | **global OOM, killed `avahi-daemon`** | -| two-node 50/50, RDMA | 538.4 | 1281.9 | 423.5 | **13.69** | 344 → 580 MiB | +| run | prompt | load s | TTFT ms | prefill t/s | decode t/s | worker RSS | +|---|---|---|---|---|---|---| +| single-node, mmap overflow | — | — | — | — | — | **global OOM, killed `avahi-daemon`** | +| two-node 50/50, RDMA, b10362 (sequential) | 512 | 538.4 | 1281.9 | 423.5 | 13.69 | 344 → 580 MiB | +| two-node 50/50, RDMA, `785322d17` (pipelined) | 1024 | 575.2 | 1432.5 | **760.3** | 11.67 | 348 → 628 MiB | There is no percentage to quote here, and that is the point: on one Spark this model does not run slowly, it takes the machine's memory out from under the OOM -killer. On two it runs at **13.7 tokens/s**, which is a usable interactive speed -for a 235B model. +killer. On two it runs at **12–14 tokens/s**, which is a usable interactive speed +for a 235B model, and with pipelining its prefill is 1.8× what it was. (The +decode gap between the two rows is the client wake-up band described under +the control below.) Through `llama-bench` (`-p 512 -n 64`, upstream worker +restarted per run), the vendored client decodes at 13.35 tok/s and the +`spark-all` client with the RPC fix at **14.27**, with the same pp512 (475, +479). The cold load is nine minutes, because ~71 GB of weights cross the fabric. Budget for it, keep the worker's tensor cache on, and do not restart casually. @@ -575,43 +756,176 @@ for it, keep the worker's tensor cache on, and do not restart casually. ### The control: a model that does fit gpt-oss-120b MXFP4 (63.4 GB — fits one node, so this isolates RPC cost from -memory benefit), 1024-token prompt, 64 decode steps. +memory benefit) at `785322d17`, after the load-mode fix. Binding rows: +`bench/spark_two_node.exs`, 1024-token prompt, 128 decode steps, median of 7, +two sessions each. Upstream rows: `llama-bench --rpc ... -dev CUDA0/RPC0` +against the same worker. | run | load s | TTFT ms | prefill t/s | decode t/s | |---|---|---|---|---| -| single-node | 64.8 | 563.6 | 1889.0 | 46.50 | -| two-node 50/50, RDMA | 151.3 | 590.4 | 1797.6 | **48.14** | - -Decode came out **+3.5% on two nodes** for a model that fits on one. That is not -what "the nodes run sequentially" predicts, and it is worth understanding before -reading too much into it: splitting halves each node's per-token weight traffic, -and on a bandwidth-bound part that relief roughly cancels the sequential -penalty. Repeat runs put both configurations in the 45–49 t/s band, so the -honest summary is **"no material difference"**, not "two nodes are faster". - -Load time is the real cost: **2.3× worse cold**, because ~30 GB of weights cross -the network. The worker's content-addressed tensor cache (`-c`, on by default in -`rpc-worker.sh`) took a warm load from 151 s to 139 s — much less than hoped, -because the client still reads and hashes every tensor locally to check the -cache; only the transfer is skipped. - -### RDMA versus TCP - -There is no runtime switch. Transport selection is silent auto-negotiation with -no env var and no endpoint scheme; the only levers are `GGML_RDMA_DEV` pointing -at a device that does not exist, or a `LLAMA_RPC_RDMA=0` build. - -| transport | TTFT ms | prefill t/s | decode t/s | +| single node | 30–52 | 491–495 | 2151–2168 | 53.3–53.8 | +| two nodes 50/50, RDMA | 105–112 | 418–429 | 2552–2614 | 36.2–37.6 | +| `llama-bench`, single node | — | — | 2046 (pp1024) | 58.8–59.0 | +| `llama-bench`, two nodes, fresh worker | — | — | 2460 (pp1024) | 50.4, 41.5, 50.5 (three clients in a row) | + +**Prefill is 18–26% faster on two nodes** and TTFT ~14% lower: with the RPC +backend async (#18626), llama.cpp pipelines the prompt's micro-batches across +the nodes, where it used to run them one node after the other. **Decode is +slower** — a single stream has nothing to overlap and pays a network round +trip per token on top of the same total compute: 14% at best through +`llama-bench`, ~30% through the binding's server, with the client unpinned. +The spread on the two-node rows (one worker served three `llama-bench` clients +in a row at 50.4, 41.5 and 50.5 tok/s) was not the worker's, as this page used +to say. It comes from which cluster the *client* runs on, and pinning it +removes most of the decode cost; see the next section. + +The worker binary matters less than that. `LlamaCppEx.stream/3` over +two nodes, three fresh contexts per session, worker restarted between +sessions: this library's `LlamaCppEx.RPC.Server` decoded at 38–48 tok/s +(sessions at ~38 and ~38/47), upstream's `ggml-rpc-server` +(`rpc-worker.sh --upstream`) at 40–51 (~50 and ~40). Both fall into the same +two bands. + +Load time is still the cost — 2–3× one node — because ~30 GB of weights +cross the network and the client still reads and hashes every tensor to check +the worker's cache. + +### Pin the client to the little cores + +Two-node decode lands in one of two bands, ~44 or ~53 tok/s through +`llama-bench`, and the client's CPU cluster picks the band. The setup was +gpt-oss-120b over RDMA, upstream's worker restarted before every run, and +`llama-bench -dev CUDA0/RPC0 -n 128 -r 3`. Single-node is 59: + +| worker on | client on | decode t/s | +|---|---|---| +| X925 `5-9,15-19` | X925 `5-9,15-19` | 44.2, 44.5, 44.6, 44.1 | +| X925 | unpinned | 44.3, 53.2 (and ~44 in five more unpinned runs) | +| A725 `0-4,10-14` | unpinned | 44.0, 44.5 | +| X925 | **A725 `0-4,10-14`** | **54.0, 54.0, 53.9** | +| A725 | **A725** | **53.5** | +| A725 | X925 | 53.1 | + +The client's cluster decides the band. With the client on the A725 cores, +every run lands in the fast band. Unpinned, 14 of 18 `llama-bench` runs landed +in the slow band, the same band as X925 pinning. llama-bench's own CPU +threads do not explain it: `--poll 0` and `-t 1` change nothing. On the +worker, kernel times are identical in fast and slow sessions (`nsys`: 118 µs +per expert matvec, 2.7 ms for the LM head), and the client's slow-band expert +matvecs run at the same 118 µs. The extra ~4 ms per token is spent on the CPU +and in the network wait; the next section has the cause. + +The binding follows the same rule. `bench/spark_two_node.exs b1` with the +BEAM under `taskset -c 0-4,10-14` (`scripts/spark/remote.sh --little-cores`), +binding worker on its default cores, two sessions each: + +| client BEAM | single-node decode | two-node prefill | two-node decode | |---|---|---|---| -| RDMA | 595.7 | 1784.0 | 46.05 | -| TCP (forced) | 805.8 | 1310.5 | 40.90 | -| | **+35%** | **−27%** | **−11%** | - -So RDMA is worth real tokens, and if RDMA ever wedges (upstream issue #24813, -closed as stale one week before our pinned commit), TCP is a working fallback -that costs about a tenth of decode. - -To confirm which one you got, there is exactly one signal — the worker's log with +| unpinned | 54.1, 55.0 | 2597, 2599 | 37.4, 38.5 | +| **A725 cores** | 55.0, 55.1 | 2587, 2593 | **47.1, 46.6** | + +That is **+25% two-node decode** at no cost to prefill or single-node decode. +On the vendored llama.cpp, **run the client of a two-node setup on the A725 +cores.** + +### The cause, and a fix that makes pinning unnecessary + +Every RPC command the client sends is two thread handoffs. The calling thread +queues it and blocks on a future. The RPC dispatcher thread wakes from its +queue, does the network round trip, then wakes the caller. The dispatcher +already busy-spins on its queue, but only while a `-sm tensor` communicator +holds it. For a layer split, each handoff is a futex wake of an idle core, +and a token takes several commands. On the X925 cores those wakes add up to +the ~4 ms per token: removing them, below, recovers it. Why the A725 cores +wake faster is not established. + +Branch `rpc-client-spin-wait` on `nyo16/llama.cpp` (also merged into +`spark-all`) spins both sides whenever a command arrived in the last 100 ms: +the dispatcher on its queue, and the caller on its future, up to 100 ms +before blocking. An idle client still sleeps. A `llama-server` holding an +RPC-split model used 0 CPU ticks in the 10 s after a request, against 2 +unpatched. The cost is paid while generating. On a 512-token gpt-oss-120b +completion over two nodes, `llama-server` used 1.99 cores on average against +0.89 unpatched, and decoded at 55.7 tok/s against 48.4. That is one extra core +for +15%. Both spins are needed. With an X925 client, `llama-bench` decodes at +44.2 with neither, 52.4 with the dispatcher spin only, 54.1 with the caller +spin only and 57.5 with both. A 30-minute soak of `llama-server` on that +client (`-np 4`, six concurrent clients, random 16/64/256-token completions, +gpt-oss-120b over RDMA) served 1419 completions and 146k tokens with no +errors, and exited 0. Upstream's worker grew from 332 to 1565 MiB RSS over it. +The same through the binding (`spark-all` vendored, `LlamaCppEx.Server` with +`n_parallel: 4`, six concurrent callers, 20 minutes) served 892 completions +with no errors, and the VM exited 0. +With that client and the same unpatched worker: + +| client | `llama-bench` decode, before | after | +|---|---|---| +| X925 cores | 44.2–44.6 | **57.6** | +| unpinned | 42.3–44.5 (4 of 18 runs at 52–53) | **57.4, 57.6** | +| A725 cores | 53.5–54.0 | **57.3** | +| single node | 59.0 | — | + +Prefill is unchanged (pp1024 2487–2513). Two-node decode now costs 2–3% against +one node, wherever the client runs. Through the binding (`spark-all` vendored, +BEAM unpinned, `b1`, two sessions), two-node decode is **50.6 and 50.8 tok/s** +against 54.7 and 55.2 single-node, where the vendored build gives 37.4–38.5 +unpinned. Two-node prefill rises to 2731–2750 with the CUDA commits. + +On the dense model, a two-node split no longer costs any decode. For +Qwen3.8-27B NVFP4 (`llama-bench -p 512 -n 128`, RDMA, fresh upstream worker), +the unpatched client decodes at 13.86 tok/s over two nodes. `spark-all` +decodes at **14.85**, against 14.35 on one node. pp512 is a single +micro-batch, so it does not pipeline, and it stays below one node: 1370 +against 1539. + +With the wake-ups gone, the transport becomes visible again. With the +`spark-all` client and gpt-oss-120b, decode is **57.6 tok/s over RDMA, 53.0 over +TCP on the same 200 GbE link, and 51.8 over the 10 GbE LAN**. The next +section's "no measurable difference" result was measured under the ~4 ms +wake-up cost, which hid the transport. + +With the client fixed, the worker binary shows up. Against the same patched +`llama-bench` client, this library's `LlamaCppEx.RPC.Server` decodes at +53.8–54.0 and upstream's `ggml-rpc-server` at 57.6, four restarts each. The +binding client gives the same split: 50.6–50.8 against 53.9. Both workers run +the same server code from the same tree and build flags. Several things change +nothing: BEAM scheduler flags (`+sbwt none`, `+S 1:1`), no super carriers +(`+MMscs 0 +MMmcs 0`), running the worker on the A725 cores instead of the +X925s, and an explicit `-ts 1/1` split (53.7 against 57.4), which rules out the +layers moving with a different free-memory report. The server thread's context +switches and CPU time match the upstream worker's (1032 / 199 / 25.9 s against +1036 / 190 / 25.2 s over a 256-token run). GPU clocks are the same in both, +2411 MHz. Kernel times under `nsys` do +not settle it, because profiling slows the BEAM process more than a bare +binary (one-node section). The cause is not established. For the last 7% of +two-node decode, run `rpc-worker.sh --upstream`. + +### RDMA versus TCP, and the 10 GbE LAN + +`GGML_RPC_NO_RDMA=1` on either peer keeps the connection on TCP +(`rpc-worker.sh start spark-2 --tcp`); `--link lan` binds the worker to the +10 GbE management NIC instead of the fabric. Same 120b model and harness, +worker restarted with its tensor cache off for every run so every byte crosses +the link, two sessions per transport: + +| transport | load s | TTFT ms | prefill t/s | decode t/s | +|---|---|---|---|---| +| RDMA, 200 GbE fabric | 105–112 | 418–429 | 2552–2614 | 36.2–37.6 | +| TCP, 200 GbE fabric | 89–91 | 419–422 | 2593–2615 | 36.9–37.6 | +| TCP, 10 GbE LAN | 76–77 | 422–429 | 2540–2585 | 38.8–39.1 | + +**On the vendored llama.cpp, you lose nothing measurable without RDMA**, and +nothing on the 10 GbE LAN either. TTFT, prefill and decode overlap across the +three transports. Load time did not favour RDMA either: the LAN was quickest in +this session, and RDMA loaded in 54 s in an earlier one. So load time is +dominated by the client reading and hashing the file, not by the wire. +Per-token traffic across a layer split is one activation row. The −27% +prefill / −11% decode TCP penalty measured at b10362 is gone. **That changes +once the RPC client's wake-ups are fixed** (previous section): decode then +reads 57.6 over RDMA, 53.0 over TCP on the fabric and 51.8 on the LAN. Leave +RDMA on. A TCP fallback, even over the LAN, is still not a cliff. + +To confirm which one you got, look at the worker's log with `GGML_RPC_DEBUG=1`: ```bash @@ -621,19 +935,58 @@ scripts/spark/rpc-worker.sh logs spark-2 200 | grep -E 'RDMA|transport' # RDMA activated: qpn=33437->33437 mtu=4096 rx_depth=24 ``` -`RDMA activate failed, staying on TCP` is the line that means you are measuring -the slow path. +`RDMA activate failed, staying on TCP`, or no `RDMA activated` line at all, is +TCP. + +### Speculative decoding across two nodes + +Qwen3.8-27B NVFP4 split 50/50 over the fabric, one 256-token greedy chat +completion, three runs each through `LlamaCppEx.MTP` (not the four-prompt +harness above, so acceptance is not comparable with the one-node table): + +| config | decode t/s | acceptance | `spark-all` client, upstream worker | +|---|---|---|---| +| plain | 14.02–14.03 | — | **14.79–14.81** | +| MTP `n_draft: 3` | 17.7–19.7 | 50% | **24.8** (53%) | +| DFlash 2 `n_draft: 7` | **22.2–22.4** | 27% | **27.8–27.9** (29%) | + +DFlash still pays 1.6× across two nodes, and 1.9× with the RPC client fix. With +the fix, plain decode over two nodes beats one node's 14.3. The drafter must be +loaded on every device the target uses, because it computes with the target's +embeddings and LM head: with the target on `["CUDA0", "RPC0"]` and the drafter on `CUDA0` +alone, upstream aborts the process ("pre-allocated tensor (output.weight) in +a buffer (RPC0) that cannot run the operation") — `llama-server +--spec-draft-device CUDA0` does it too. `MTP.init/2` refuses that pairing; +load the drafter with the target's `:devices` and `tensor_split: [1.0, 0.0]` +to keep its own layers local. + +The upstream fix is branch `draft-ctx-other-devices` on `nyo16/llama.cpp`: a +context with `ctx_other` gets a backend for each of the other model's devices. +With it, that same `llama-server` command runs, at 26.4 tok/s (566 drafted, +173 accepted) on one 256-token completion. ### Concurrency across two nodes -| `n_parallel` | decode t/s per request | -|---|---| -| 1 | 49.48 | -| 4 | 44.71 | -| 8 | 44.25 | +gpt-oss-120b through `bench/spark_overhead.exs` with `SPARK_RPC_ENDPOINT` +set (every load split 50/50 over the fabric), against the same bench on one +node; per-stream decode, then aggregate: -Per-token RTT is fixed per *graph*, not per token, so batching amortises it -well: 8 concurrent requests cost 11% of per-request decode for 8× the aggregate. +| streams | one node | two nodes | two nodes, `spark-all` client | +|---|---|---|---| +| 1 (`stream/3`) | 57.0 | 47.8 | **56.1** | +| 1 (server) | 55.9 | 45.7 | **55.2** | +| 4 | 23.8 / **91** | 18.2 / **70** | 23.9 / **93** | +| 8 | 13.5 / **105** | 11.8 / **92** | 12.8 / **99** | + +On the vendored client, two nodes keep 77–88% of one node's throughput at +every concurrency, so the per-token round trip does not grow with the batch. +The last column is the `spark-all` build, with the RPC client fix, against +upstream's worker, the BEAM on the X925 cores (`--big-cores`). It keeps +95–102%. (This section used to quote 49.5 / 44.7 / 44.3 for `n_parallel` +1 / 4 / 8 from `spark_two_node.exs b4`, which allocates that many slots but +sends one request. So it measured load-to-load variation rather than +concurrency.) A MoE does not batch well on either: each extra stream routes +to different experts. ### The decode fast path is holding @@ -689,6 +1042,15 @@ Essentially free. So the Meta device is not the problem. Now two nodes: It runs, and the output is byte-identical to the layer-split reference. It is **2.7× slower on decode and 23× slower on prefill**. +At `785322d17` the RPC backend has its own pairwise communicator for this +(#26610, `ggml_backend_rpc_comm_init`), so the next section describes the older +commit. Re-measured on gpt-oss-120b (`llama-bench -sm tensor -p 512 -n 64`, +fresh upstream worker), the vendored client decodes at 8.9 tok/s with pp512 +1135. The `spark-all` client, whose RPC fix also spins the callers that the +communicator's own spin left blocking, decodes at **24.1** with pp512 1553. +That is 2.7× better, and still less than half of a layer split's 57.6. The +verdict below stands. + ## Why, and why the comm-mode knob cannot help `ggml_backend_cuda_comm_init` returns `nullptr` the moment **any** member backend @@ -717,8 +1079,11 @@ only cross-host mode that is both correct and fast, and it buys capacity. cannot span hosts as designed. Across hosts it silently falls back to a generic path that is 2.7× slower. It is the right tool for several GPUs in **one** box, which a Spark does not have. -- Layer split over RPC buys **capacity, not speed**. If your model fits in - 130 GB, one Spark is the answer. +- Layer split over RPC buys **capacity** first. On the vendored llama.cpp it + also buys +18–26% prefill and costs 9–30% decode, depending on where the + client runs. With the `rpc-client-spin-wait` fix, decode costs 2–3% on + gpt-oss-120b and nothing on the dense NVFP4 model. For a model that fits in + 130 GB, a second Spark is a prefill and throughput option, not a requirement. Re-check these claims against a future llama.cpp bump using upstream commits `d6f303004` (`-sm tensor`), `adb541a6a` and `91fef9536`. diff --git a/docs/multi-gpu.md b/docs/multi-gpu.md index b5eff73..4d94c56 100644 --- a/docs/multi-gpu.md +++ b/docs/multi-gpu.md @@ -147,13 +147,15 @@ of a device being a network away — it will budget a remote device as if it wer local VRAM. The numbers are not nonsense, but do not read more into them than that. -### It buys capacity, not speed - -Pipeline parallelism is disabled whenever an RPC device participates: the RPC -backend reports `async = false, events = false`, and llama.cpp checks that before -enabling pipelining. The two nodes therefore execute **sequentially**. A model -that already fits on one machine gains nothing from a second one. See -[DGX Spark](dgx-spark.md) for the measurements. +### It buys capacity, and prefill, not decode + +Since llama.cpp #18626 the RPC backend implements async compute and events, so +llama.cpp enables pipeline parallelism across an RPC layer split: a prompt's +micro-batches overlap across the nodes, and prefill measured 18–26% faster on +two DGX Sparks than on one. A single decode stream has nothing to overlap and +pays a network round trip per token, so it is slower than on one machine. A +model that already fits on one machine gains prefill from a second one, and +loses some decode. See [DGX Spark](dgx-spark.md) for the measurements. > #### A peer failure aborts the VM {: .error} > diff --git a/docs/performance.md b/docs/performance.md index 296f93f..030f831 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -276,17 +276,20 @@ Additional options when loading models: ```elixir {:ok, model} = LlamaCppEx.load_model("model.gguf", n_gpu_layers: -1, # Offload all layers to GPU - use_mmap: true, # Memory-map file (default, faster loading) - use_mlock: true, # Pin in RAM (prevent swapping); implies use_mmap + use_mmap: true, # Memory-map the file (unset: llama.cpp decides) + use_mlock: true, # Pin in RAM (prevent swapping) use_direct_io: false, # Bypass page cache check_tensors: true # Validate tensor data (debugging) ) ``` These three flags resolve to llama.cpp's single `load_mode` enum, with -`use_direct_io` > `use_mlock` > `use_mmap` precedence (`dio`, `mlock`, `mmap`, -or `none` when all are false). Only one mode is ever active, so combining them -is not additive. +`use_direct_io` > `use_mlock` > `use_mmap` precedence (`dio`, `mmap_mlock` or +`mlock`, `mmap`, `none`). Only one mode is ever active, so combining them is +not additive. With none of them given the mode is upstream's `auto`, which +memory-maps except on integrated GPUs, where it reads the file instead; on +DGX Spark's GB10 that is worth ~15% of decode on large models against a forced +`use_mmap: true`. Leave them unset unless you have a reason. ## Complete Optimization Example diff --git a/gguf_ex/_build/dev/lib/gguf_ex/.mix/compile.app_cache b/gguf_ex/_build/dev/lib/gguf_ex/.mix/compile.app_cache new file mode 100644 index 0000000..6ea8115 Binary files /dev/null and b/gguf_ex/_build/dev/lib/gguf_ex/.mix/compile.app_cache differ diff --git a/gguf_ex/_build/dev/lib/gguf_ex/.mix/compile.elixir b/gguf_ex/_build/dev/lib/gguf_ex/.mix/compile.elixir new file mode 100644 index 0000000..07d3657 Binary files /dev/null and b/gguf_ex/_build/dev/lib/gguf_ex/.mix/compile.elixir differ diff --git a/gguf_ex/_build/dev/lib/gguf_ex/.mix/compile.elixir_scm b/gguf_ex/_build/dev/lib/gguf_ex/.mix/compile.elixir_scm new file mode 100644 index 0000000..cb3dbfd Binary files /dev/null and b/gguf_ex/_build/dev/lib/gguf_ex/.mix/compile.elixir_scm differ diff --git a/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.Collectable.beam b/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.Collectable.beam new file mode 100644 index 0000000..e3f8ed4 Binary files /dev/null and b/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.Collectable.beam differ diff --git a/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.Enumerable.beam b/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.Enumerable.beam new file mode 100644 index 0000000..b4eef6e Binary files /dev/null and b/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.Enumerable.beam differ diff --git a/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.IEx.Info.beam b/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.IEx.Info.beam new file mode 100644 index 0000000..f34f29f Binary files /dev/null and b/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.IEx.Info.beam differ diff --git a/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.Inspect.beam b/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.Inspect.beam new file mode 100644 index 0000000..e8d5e61 Binary files /dev/null and b/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.Inspect.beam differ diff --git a/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.JSON.Encoder.beam b/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.JSON.Encoder.beam new file mode 100644 index 0000000..295690b Binary files /dev/null and b/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.JSON.Encoder.beam differ diff --git a/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.List.Chars.beam b/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.List.Chars.beam new file mode 100644 index 0000000..dd3d534 Binary files /dev/null and b/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.List.Chars.beam differ diff --git a/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.String.Chars.beam b/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.String.Chars.beam new file mode 100644 index 0000000..1f6d065 Binary files /dev/null and b/gguf_ex/_build/dev/lib/gguf_ex/consolidated/Elixir.String.Chars.beam differ diff --git a/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.ExpertMap.beam b/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.ExpertMap.beam new file mode 100644 index 0000000..cf61098 Binary files /dev/null and b/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.ExpertMap.beam differ diff --git a/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.File.beam b/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.File.beam new file mode 100644 index 0000000..bba49b7 Binary files /dev/null and b/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.File.beam differ diff --git a/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.Parser.beam b/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.Parser.beam new file mode 100644 index 0000000..bc2d288 Binary files /dev/null and b/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.Parser.beam differ diff --git a/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.Tensor.beam b/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.Tensor.beam new file mode 100644 index 0000000..54a996c Binary files /dev/null and b/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.Tensor.beam differ diff --git a/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.Types.beam b/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.Types.beam new file mode 100644 index 0000000..02b09a2 Binary files /dev/null and b/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.Types.beam differ diff --git a/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.beam b/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.beam new file mode 100644 index 0000000..2ab0100 Binary files /dev/null and b/gguf_ex/_build/dev/lib/gguf_ex/ebin/Elixir.GgufEx.beam differ diff --git a/gguf_ex/_build/dev/lib/gguf_ex/ebin/gguf_ex.app b/gguf_ex/_build/dev/lib/gguf_ex/ebin/gguf_ex.app new file mode 100644 index 0000000..e38c734 --- /dev/null +++ b/gguf_ex/_build/dev/lib/gguf_ex/ebin/gguf_ex.app @@ -0,0 +1 @@ +{application,gguf_ex,[{modules,['Elixir.GgufEx','Elixir.GgufEx.ExpertMap','Elixir.GgufEx.File','Elixir.GgufEx.Parser','Elixir.GgufEx.Tensor','Elixir.GgufEx.Types']},{optional_applications,[]},{applications,[kernel,stdlib,elixir,logger]},{description,"Pure-Elixir GGUF (v2/v3) parser: metadata, tensor infos, and absolute tensor byte layout without loading tensor data."},{registered,[]},{vsn,"0.1.0"}]}. \ No newline at end of file diff --git a/gguf_ex/_build/test/lib/gguf_ex/.mix/.mix_test_failures b/gguf_ex/_build/test/lib/gguf_ex/.mix/.mix_test_failures new file mode 100644 index 0000000..002e9d7 Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/.mix/.mix_test_failures differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/.mix/compile.app_cache b/gguf_ex/_build/test/lib/gguf_ex/.mix/compile.app_cache new file mode 100644 index 0000000..6ea8115 Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/.mix/compile.app_cache differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/.mix/compile.elixir b/gguf_ex/_build/test/lib/gguf_ex/.mix/compile.elixir new file mode 100644 index 0000000..3a31e59 Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/.mix/compile.elixir differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/.mix/compile.elixir_scm b/gguf_ex/_build/test/lib/gguf_ex/.mix/compile.elixir_scm new file mode 100644 index 0000000..cb3dbfd Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/.mix/compile.elixir_scm differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.Collectable.beam b/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.Collectable.beam new file mode 100644 index 0000000..e3f8ed4 Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.Collectable.beam differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.Enumerable.beam b/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.Enumerable.beam new file mode 100644 index 0000000..b4eef6e Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.Enumerable.beam differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.IEx.Info.beam b/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.IEx.Info.beam new file mode 100644 index 0000000..f34f29f Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.IEx.Info.beam differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.Inspect.beam b/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.Inspect.beam new file mode 100644 index 0000000..e8d5e61 Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.Inspect.beam differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.JSON.Encoder.beam b/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.JSON.Encoder.beam new file mode 100644 index 0000000..295690b Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.JSON.Encoder.beam differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.List.Chars.beam b/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.List.Chars.beam new file mode 100644 index 0000000..dd3d534 Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.List.Chars.beam differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.String.Chars.beam b/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.String.Chars.beam new file mode 100644 index 0000000..1f6d065 Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/consolidated/Elixir.String.Chars.beam differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.ExpertMap.beam b/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.ExpertMap.beam new file mode 100644 index 0000000..cf61098 Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.ExpertMap.beam differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.File.beam b/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.File.beam new file mode 100644 index 0000000..bba49b7 Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.File.beam differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.Parser.beam b/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.Parser.beam new file mode 100644 index 0000000..bc2d288 Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.Parser.beam differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.Tensor.beam b/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.Tensor.beam new file mode 100644 index 0000000..54a996c Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.Tensor.beam differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.TestSupport.Builder.beam b/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.TestSupport.Builder.beam new file mode 100644 index 0000000..2563534 Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.TestSupport.Builder.beam differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.Types.beam b/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.Types.beam new file mode 100644 index 0000000..2d74fe5 Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.Types.beam differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.beam b/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.beam new file mode 100644 index 0000000..2ab0100 Binary files /dev/null and b/gguf_ex/_build/test/lib/gguf_ex/ebin/Elixir.GgufEx.beam differ diff --git a/gguf_ex/_build/test/lib/gguf_ex/ebin/gguf_ex.app b/gguf_ex/_build/test/lib/gguf_ex/ebin/gguf_ex.app new file mode 100644 index 0000000..ded8731 --- /dev/null +++ b/gguf_ex/_build/test/lib/gguf_ex/ebin/gguf_ex.app @@ -0,0 +1 @@ +{application,gguf_ex,[{modules,['Elixir.GgufEx','Elixir.GgufEx.ExpertMap','Elixir.GgufEx.File','Elixir.GgufEx.Parser','Elixir.GgufEx.Tensor','Elixir.GgufEx.TestSupport.Builder','Elixir.GgufEx.Types']},{optional_applications,[]},{applications,[kernel,stdlib,elixir,logger]},{description,"Pure-Elixir GGUF (v2/v3) parser: metadata, tensor infos, and absolute tensor byte layout without loading tensor data."},{registered,[]},{vsn,"0.1.0"}]}. \ No newline at end of file diff --git a/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_1.gguf b/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_1.gguf new file mode 100644 index 0000000..e7b9301 Binary files /dev/null and b/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_1.gguf differ diff --git a/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_128.gguf b/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_128.gguf new file mode 100644 index 0000000..e43766a Binary files /dev/null and b/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_128.gguf differ diff --git a/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_16.gguf b/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_16.gguf new file mode 100644 index 0000000..75babb1 Binary files /dev/null and b/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_16.gguf differ diff --git a/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_4096.gguf b/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_4096.gguf new file mode 100644 index 0000000..219bd49 Binary files /dev/null and b/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_4096.gguf differ diff --git a/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_64.gguf b/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_64.gguf new file mode 100644 index 0000000..289b472 Binary files /dev/null and b/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_64.gguf differ diff --git a/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_8.gguf b/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_8.gguf new file mode 100644 index 0000000..686b03f Binary files /dev/null and b/gguf_ex/tmp/GgufExTest/test-alignment-handling-data-offset-and-tensor-offsets-respect-non-default-alignments-bbd0715e/align_8.gguf differ diff --git a/gguf_ex/tmp/GgufExTest/test-expert_map-1-MoE-metadata-without-fused-tensors-4b69cbab/hollow.gguf b/gguf_ex/tmp/GgufExTest/test-expert_map-1-MoE-metadata-without-fused-tensors-4b69cbab/hollow.gguf new file mode 100644 index 0000000..35c57c0 Binary files /dev/null and b/gguf_ex/tmp/GgufExTest/test-expert_map-1-MoE-metadata-without-fused-tensors-4b69cbab/hollow.gguf differ diff --git a/gguf_ex/tmp/GgufExTest/test-expert_map-1-accepts-a-path-directly-d6d2446e/moe2.gguf b/gguf_ex/tmp/GgufExTest/test-expert_map-1-accepts-a-path-directly-d6d2446e/moe2.gguf new file mode 100644 index 0000000..813ea4a Binary files /dev/null and b/gguf_ex/tmp/GgufExTest/test-expert_map-1-accepts-a-path-directly-d6d2446e/moe2.gguf differ diff --git a/gguf_ex/tmp/GgufExTest/test-expert_map-1-dense-model-returns---error,--not_moe--8120b491/dense.gguf b/gguf_ex/tmp/GgufExTest/test-expert_map-1-dense-model-returns---error,--not_moe--8120b491/dense.gguf new file mode 100644 index 0000000..f23d9e0 Binary files /dev/null and b/gguf_ex/tmp/GgufExTest/test-expert_map-1-dense-model-returns---error,--not_moe--8120b491/dense.gguf differ diff --git a/gguf_ex/tmp/GgufExTest/test-expert_map-1-expert-dim-mismatch-is-an-error-34f1fdcd/mismatch.gguf b/gguf_ex/tmp/GgufExTest/test-expert_map-1-expert-dim-mismatch-is-an-error-34f1fdcd/mismatch.gguf new file mode 100644 index 0000000..15c6b97 Binary files /dev/null and b/gguf_ex/tmp/GgufExTest/test-expert_map-1-expert-dim-mismatch-is-an-error-34f1fdcd/mismatch.gguf differ diff --git a/gguf_ex/tmp/GgufExTest/test-expert_map-1-expert_count-=-0-is-dense-b177585f/dense0.gguf b/gguf_ex/tmp/GgufExTest/test-expert_map-1-expert_count-=-0-is-dense-b177585f/dense0.gguf new file mode 100644 index 0000000..fa1296a Binary files /dev/null and b/gguf_ex/tmp/GgufExTest/test-expert_map-1-expert_count-=-0-is-dense-b177585f/dense0.gguf differ diff --git a/gguf_ex/tmp/GgufExTest/test-expert_map-1-maps--layer,-expert--to-contiguous-per-projection-ranges-9c6a7ebb/moe.gguf b/gguf_ex/tmp/GgufExTest/test-expert_map-1-maps--layer,-expert--to-contiguous-per-projection-ranges-9c6a7ebb/moe.gguf new file mode 100644 index 0000000..6b1aa6c Binary files /dev/null and b/gguf_ex/tmp/GgufExTest/test-expert_map-1-maps--layer,-expert--to-contiguous-per-projection-ranges-9c6a7ebb/moe.gguf differ diff --git a/gguf_ex/tmp/GgufExTest/test-parse-1-round-trip-never-reads-tensor-data--header-present,-data-region-is-a-hole-479f9980/sparse.gguf b/gguf_ex/tmp/GgufExTest/test-parse-1-round-trip-never-reads-tensor-data--header-present,-data-region-is-a-hole-479f9980/sparse.gguf new file mode 100644 index 0000000..73e206b Binary files /dev/null and b/gguf_ex/tmp/GgufExTest/test-parse-1-round-trip-never-reads-tensor-data--header-present,-data-region-is-a-hole-479f9980/sparse.gguf differ diff --git a/gguf_ex/tmp/GgufExTest/test-parse-1-round-trip-never-reads-tensor-data--header-present,-data-region-is-a-hole-479f9980/whole.gguf b/gguf_ex/tmp/GgufExTest/test-parse-1-round-trip-never-reads-tensor-data--header-present,-data-region-is-a-hole-479f9980/whole.gguf new file mode 100644 index 0000000..73e206b Binary files /dev/null and b/gguf_ex/tmp/GgufExTest/test-parse-1-round-trip-never-reads-tensor-data--header-present,-data-region-is-a-hole-479f9980/whole.gguf differ diff --git a/gguf_ex/tmp/GgufExTest/test-parse-1-round-trip-parses-GGUF-v2-bc68948d/v2.gguf b/gguf_ex/tmp/GgufExTest/test-parse-1-round-trip-parses-GGUF-v2-bc68948d/v2.gguf new file mode 100644 index 0000000..3198a45 Binary files /dev/null and b/gguf_ex/tmp/GgufExTest/test-parse-1-round-trip-parses-GGUF-v2-bc68948d/v2.gguf differ diff --git a/gguf_ex/tmp/GgufExTest/test-parse-1-round-trip-parses-a-v3-file-with-all-KV-value-types-and-two-tensors-5c283608/basic.gguf b/gguf_ex/tmp/GgufExTest/test-parse-1-round-trip-parses-a-v3-file-with-all-KV-value-types-and-two-tensors-5c283608/basic.gguf new file mode 100644 index 0000000..06d7d7c Binary files /dev/null and b/gguf_ex/tmp/GgufExTest/test-parse-1-round-trip-parses-a-v3-file-with-all-KV-value-types-and-two-tensors-5c283608/basic.gguf differ diff --git a/lib/llama_cpp_ex/context.ex b/lib/llama_cpp_ex/context.ex index 97c8fea..495a986 100644 --- a/lib/llama_cpp_ex/context.ex +++ b/lib/llama_cpp_ex/context.ex @@ -145,10 +145,11 @@ defmodule LlamaCppEx.Context do with a separate `:default` context to drive multi-token-prediction speculative decoding via `LlamaCppEx.MTP`. * `:n_rs_seq` - Number of recurrent-state snapshots per sequence to retain for - partial rollback of speculative drafts. `0` (default) disables rollback. For an - MTP draft context, use `0` — the MTP implementation handles rollback - internally via cached hidden states (`pending_h` / `verify_h`), not - recurrent-state snapshots. + partial rollback of speculative drafts. `0` (default) disables rollback. + `LlamaCppEx.MTP.init/2` gives the target context one per draft position, + as upstream's server does; the draft context gets `0` — the drafters + handle their own rollback (MTP via cached hidden states, DFlash by + re-injecting target features), not via recurrent-state snapshots. * `:ctx_other` - An existing `%Context{}` whose raw pointer is passed as `llama_context_params.ctx_other`. Optional; omitted or `nil` is `nullptr`. `gemma4-assistant` requires it at construction. `MTP.init/2` diff --git a/lib/llama_cpp_ex/model.ex b/lib/llama_cpp_ex/model.ex index 91b7c63..7b647a6 100644 --- a/lib/llama_cpp_ex/model.ex +++ b/lib/llama_cpp_ex/model.ex @@ -47,15 +47,17 @@ defmodule LlamaCppEx.Model do * `:n_gpu_layers` - Number of layers to offload to GPU. Use `-1` for all layers. Defaults to `99` (offload all layers). - * `:use_mmap` - Whether to memory-map the model file. Defaults to `true`. + * `:use_mmap` - Memory-map the model file (`true`) or read it (`false`). + Unset by default, which leaves the choice to llama.cpp — see "Load mode" + below. * `:main_gpu` - GPU device index for single-GPU mode. Defaults to `0`. * `:split_mode` - How to split the model across devices: `:none`, `:layer`, `:row` or `:tensor`. Defaults to `:none`. Only `:none` and `:layer` are generally usable at this llama.cpp version — see the note below. * `:tensor_split` - List of floats specifying the proportion of work per GPU (e.g. `[0.5, 0.5]` for two GPUs). Defaults to `[]`. - * `:use_mlock` - Pin model memory in RAM to prevent swapping. Implies `:use_mmap`. - Defaults to `false`. + * `:use_mlock` - Pin model memory in RAM to prevent swapping. Combined with + `:use_mmap` unless that is `false`. Defaults to `false`. * `:use_direct_io` - Bypass page cache when loading (takes precedence over mmap). Defaults to `false`. * `:vocab_only` - Load vocabulary and metadata only, skip weights. Defaults to `false`. @@ -101,10 +103,15 @@ defmodule LlamaCppEx.Model do > > llama.cpp collapsed its three loading booleans into one `load_mode` enum, so > these options resolve to a single mode. `:use_direct_io` takes precedence - > over everything and selects `dio`; otherwise `:use_mlock` and `:use_mmap` - > combine — both true selects `mmap_mlock`, `:use_mlock` alone selects `mlock` - > (read into anonymous memory, no mapping), `:use_mmap` alone selects `mmap`, - > and all false selects `none`. + > over everything and selects `dio`; otherwise `:use_mlock` selects + > `mmap_mlock`, or `mlock` (read into anonymous memory, no mapping) with + > `use_mmap: false`; `use_mmap: true` alone selects `mmap` and + > `use_mmap: false` alone `none`. With none of the three given, the mode is + > upstream's `auto`: mmap, except on devices that do not support it — an + > integrated GPU such as DGX Spark's GB10 — where the file is read instead + > (llama.cpp #26081). On GB10 a forced `mmap` left most loads of + > gpt-oss-120b decoding ~15% slower (48.5 against 57 tok/s), so leave these + > unset unless you have measured otherwise. > #### Untrusted models {: .warning} > @@ -126,12 +133,17 @@ defmodule LlamaCppEx.Model do @spec load(String.t(), keyword()) :: {:ok, t()} | {:error, String.t()} def load(path, opts \\ []) do n_gpu_layers = Keyword.get(opts, :n_gpu_layers, 99) - use_mmap = Keyword.get(opts, :use_mmap, true) + + load_mode = + resolve_load_mode( + Keyword.get(opts, :use_mmap), + Keyword.get(opts, :use_mlock, false), + Keyword.get(opts, :use_direct_io, false) + ) + main_gpu = Keyword.get(opts, :main_gpu, 0) split_mode = Keyword.get(opts, :split_mode, :none) |> encode_split_mode() tensor_split = Keyword.get(opts, :tensor_split, []) - use_mlock = Keyword.get(opts, :use_mlock, false) - use_direct_io = Keyword.get(opts, :use_direct_io, false) vocab_only = Keyword.get(opts, :vocab_only, false) check_tensors = Keyword.get(opts, :check_tensors, false) load_mtp = Keyword.get(opts, :load_mtp, false) @@ -151,12 +163,10 @@ defmodule LlamaCppEx.Model do LlamaCppEx.NIF.model_load( path, n_gpu_layers, - use_mmap, + load_mode, main_gpu, split_mode, tensor_split, - use_mlock, - use_direct_io, vocab_only, check_tensors, load_mtp, @@ -166,6 +176,23 @@ defmodule LlamaCppEx.Model do end end + # llama_load_mode values (llama.h). `nil` for :use_mmap means the caller did + # not choose, and only then is the result upstream's AUTO. + @load_mode_auto -1 + @load_mode_none 0 + @load_mode_mmap 1 + @load_mode_mlock 2 + @load_mode_mmap_mlock 3 + @load_mode_direct_io 4 + + @doc false + def resolve_load_mode(_use_mmap, _use_mlock, true), do: @load_mode_direct_io + def resolve_load_mode(false, true, _), do: @load_mode_mlock + def resolve_load_mode(_use_mmap, true, _), do: @load_mode_mmap_mlock + def resolve_load_mode(true, _, _), do: @load_mode_mmap + def resolve_load_mode(false, _, _), do: @load_mode_none + def resolve_load_mode(nil, _, _), do: @load_mode_auto + defp register_rpc_servers([]), do: :ok defp register_rpc_servers(endpoints) do @@ -223,6 +250,14 @@ defmodule LlamaCppEx.Model do @spec n_layer_nextn(t()) :: non_neg_integer() def n_layer_nextn(%__MODULE__{ref: ref}), do: LlamaCppEx.NIF.model_n_layer_nextn(ref) + @doc """ + Names of the devices the model's layers were placed on, in placement order — + e.g. `["CUDA0", "RPC0"]` for a model split across a local GPU and an RPC + worker. Empty for a CPU-only load. + """ + @spec devices(t()) :: [String.t()] + def devices(%__MODULE__{ref: ref}), do: LlamaCppEx.NIF.model_devices(ref) + @doc "Returns a human-readable description of the model." @spec desc(t()) :: String.t() def desc(%__MODULE__{ref: ref}), do: LlamaCppEx.NIF.model_desc(ref) diff --git a/lib/llama_cpp_ex/mtp.ex b/lib/llama_cpp_ex/mtp.ex index d5b1123..98dc9d0 100644 --- a/lib/llama_cpp_ex/mtp.ex +++ b/lib/llama_cpp_ex/mtp.ex @@ -52,17 +52,19 @@ defmodule LlamaCppEx.MTP do target's token embeddings); Qwen's constructor leaves `cparams.ctx_other` as `nullptr`. - > #### Speculation is not always a win on hybrid models {: .warning} + > #### Hybrid models roll back on recurrent snapshots {: .info} > > A model that mixes recurrent (SSM) layers with attention ones — Qwen 3.8 is > 48 SSM layers to 16 attention layers — cannot roll back part of a sequence - > natively, so every speculative iteration snapshots and restores the whole - > recurrent state. That state is over 100 MiB at Qwen 3.8's sizes, and the - > cost lands in `stats/1`'s `timing_us.ckpt`. Measured on an M1 Max (Metal, - > Q4_K_M target + Q4_0 head), MTP was a net *slowdown* at every draft length: - > 0.89x at `n_draft: 1` (75% acceptance) falling to 0.56x at `n_draft: 5` - > (30%). Check `timing_us.ckpt` against `timing_us.total` before assuming - > speculation is helping, and prefer small `n_draft` when it is not. + > from its KV cache alone. `init/2` therefore creates the target context with + > one recurrent-state snapshot per draft position (`n_rs_seq: n_draft`), as + > upstream's server does, and a rejected draft is rolled back natively. Before + > that, every iteration saved the whole recurrent state (over 100 MiB at Qwen + > 3.8's sizes) and every partial accept restored it and re-decoded the + > accepted prefix: on GB10 that held Qwen3.8-27B NVFP4 to 21 tok/s at + > `n_draft: 2` where the snapshot path does 27, and on an M1 Max it made MTP a + > net slowdown. `stats/1`'s `timing_us.ckpt` is that save/restore, and stays + > zero unless `n_draft` exceeds the snapshots. Upstream currently requires `n_parallel = 1` for MTP. This module reflects that — a single MTP session decodes one sequence at a time. Reuse the same @@ -110,22 +112,38 @@ defmodule LlamaCppEx.MTP do end @doc """ - Initializes an MTP speculative session: builds the target context, the MTP - draft context (`ctx_type: :mtp`), and the underlying `common_speculative` - state. + Initializes a speculative session: builds the target context, the draft + context, and the underlying `common_speculative` state. ## Options - * `:draft_model` - A separate `LlamaCppEx.Model` holding the MTP head, for - checkpoints that ship it as a sidecar GGUF rather than inside the target - file (Qwen 3.8: `Qwen3.8-27B-Q4_K_M.gguf` plus `mtp-Qwen3.8-27B-Q4_0.gguf`; - Gemma4 E4B: a `gemma4` target plus a `gemma4-assistant` head). It must be - loaded with `load_mtp: true`. - Defaults to `nil`, meaning the head is expected inside the target model - and the draft context is built against it. + * `:type` - The drafter. Defaults to `:mtp`. + * `:mtp` - a Multi-Token Prediction head, inside the target GGUF or in a + sidecar passed as `:draft_model`. The draft context is built with + `ctx_type: :mtp`. + * `:dflash` - a DFlash / DFlash 2 block-diffusion drafter (GGUF + architecture `dflash`), passed as `:draft_model`. It drafts a whole + block in one pass from the target's hidden states, and is the fastest + option measured on GB10: 3.3× plain decode on Qwen3.8-27B NVFP4 against + 2.1× for its MTP head (see `docs/dgx-spark.md`). + * `:dspark` - a DSpark drafter (DFlash plus a Markov head), passed as + `:draft_model`. + * `:draft_model` - A separate `LlamaCppEx.Model` holding the drafter. + Required for `:dflash` and `:dspark`, and loaded on every device the + target uses (`LlamaCppEx.Model.devices/1`): a DFlash drafter computes + with the target's embeddings and LM head. For a target split across an + RPC worker, load the drafter with the same `:devices` and a + `:tensor_split` that keeps its layers local. For `:mtp` it is the sidecar head + for checkpoints that ship one rather than embedding it (Qwen 3.8: + `Qwen3.8-27B-Q4_K_M.gguf` plus `mtp-Qwen3.8-27B-Q4_0.gguf`; Gemma4 E4B: + a `gemma4` target plus a `gemma4-assistant` head), loaded with + `load_mtp: true`. Defaults to `nil`, meaning the MTP head is expected + inside the target model and the draft context is built against it. * `:n_draft` - Max draft tokens generated per iteration. Defaults to `3`. Larger values mean fewer model forward passes but lower per-iteration - acceptance; 2–4 is the sweet spot in practice. + acceptance; 2–4 is the sweet spot for MTP in practice. A DFlash drafter + caps it at its trained block size minus one (7 for Qwen3.8 DFlash 2), + and that is also where it measured fastest. * `:n_ctx` - Context size for both contexts. Defaults to `2048`. * Any `Context` tuning option (e.g. `:n_threads`, `:flash_attn`, `:type_k`/`:type_v`, `:offload_kqv`). Applied to both contexts. @@ -137,16 +155,52 @@ defmodule LlamaCppEx.MTP do def init(%Model{} = model, opts \\ []) do n_draft = Keyword.get(opts, :n_draft, 3) draft_model = Keyword.get(opts, :draft_model) + type = Keyword.get(opts, :type, :mtp) with :ok <- validate_n_draft(n_draft), - {:ok, head_model} <- validate_head(model, draft_model) do - do_init(model, head_model, opts, n_draft) + {:ok, head_model} <- validate_drafter(type, model, draft_model) do + do_init(model, head_model, opts, n_draft, type) end end defp validate_n_draft(n) when is_integer(n) and n > 0, do: :ok defp validate_n_draft(_), do: {:error, ":n_draft must be a positive integer"} + defp validate_drafter(:mtp, target, draft), do: validate_head(target, draft) + + # Whether the drafter really is a DFlash checkpoint for this target is checked + # in the NIF, where the model's target-layer metadata is readable: upstream's + # constructor asserts on it, so it has to be refused before init. + # + # The device check has to happen here, before the draft context exists. A + # DFlash drafter computes with the target's token embeddings and LM head + # (shared through ctx_other), so its context needs a backend for every device + # the target placed weights on. A target split onto an RPC worker with the + # drafter on CUDA0 alone aborted the VM with "pre-allocated tensor + # (output.weight) in a buffer (RPC0) that cannot run the operation". + defp validate_drafter(type, target, %Model{} = draft) when type in [:dflash, :dspark] do + case Model.devices(target) -- Model.devices(draft) do + [] -> + {:ok, draft} + + missing -> + {:error, + "the drafter reads the target's embeddings and LM head, so it must be loaded " <> + "on every device the target uses; it is missing #{Enum.join(missing, ", ")}. " <> + "Load it with the target's :devices (a :tensor_split can still keep its " <> + "own layers on one of them)"} + end + end + + defp validate_drafter(type, _target, nil) when type in [:dflash, :dspark], + do: {:error, "type: #{inspect(type)} needs the drafter GGUF passed as draft_model:"} + + defp validate_drafter(type, _target, other) when type in [:dflash, :dspark], + do: {:error, ":draft_model must be a LlamaCppEx.Model, got: #{inspect(other)}"} + + defp validate_drafter(type, _target, _draft), + do: {:error, ":type must be :mtp, :dflash or :dspark, got: #{inspect(type)}"} + # Which model the draft context is built from, and whether it can actually # serve as one. Two shapes reach this: the head inside the target GGUF (no # `:draft_model`), and the head in a sidecar GGUF alongside it. @@ -216,25 +270,38 @@ defmodule LlamaCppEx.MTP do {:error, ":draft_model must be a LlamaCppEx.Model, got: #{inspect(other)}"} end - defp do_init(model, head_model, opts, n_draft) do + defp do_init(model, head_model, opts, n_draft, type) do base_ctx_opts = forwardable_context_opts(opts) - main_opts = Keyword.merge(base_ctx_opts, ctx_type: :default) - # Match upstream server: MTP draft context is created with n_rs_seq=0. - # The MTP impl handles state rollback internally via cached hidden - # states (pending_h / verify_h), not via recurrent-state snapshots. - draft_opts = Keyword.merge(base_ctx_opts, ctx_type: :mtp, n_rs_seq: 0) + + # Recurrent-state snapshots on the target, one per draft position, as + # upstream's server sizes them (common_params_speculative.need_n_rs_seq). + # A hybrid target (GDN layers beside attention) then rolls a rejected draft + # back natively instead of saving its state every iteration and re-decoding + # the accepted prefix on every partial accept: measured on Qwen3.8-27B + # NVFP4 with DFlash 2, that re-decode was the largest cost in the loop. A + # DFlash drafter clamps n_draft to its block size (at most 16) after init, + # so the snapshots are capped there rather than sized from what was asked. + n_rs_seq = if type == :mtp, do: n_draft, else: min(n_draft, 16) + main_opts = Keyword.merge(base_ctx_opts, ctx_type: :default, n_rs_seq: n_rs_seq) + # Match upstream (common_speculative_init_result): the draft context is + # created with n_rs_seq=0, as an MTP context only for an MTP head. The + # drafters handle rollback themselves (MTP via cached hidden states, + # DFlash by re-injecting target features), not via recurrent snapshots. + draft_ctx_type = if type == :mtp, do: :mtp, else: :default + draft_opts = Keyword.merge(base_ctx_opts, ctx_type: draft_ctx_type, n_rs_seq: 0) with {:ok, main_ctx} <- Context.create(model, main_opts), {:ok, mtp_ctx} <- Context.create(head_model, Keyword.merge(draft_opts, ctx_other: main_ctx)), {:ok, spec_ref} <- - LlamaCppEx.NIF.speculative_init(main_ctx.ref, mtp_ctx.ref, n_draft) do + LlamaCppEx.NIF.speculative_init(main_ctx.ref, mtp_ctx.ref, n_draft, type) do {:ok, %__MODULE__{ main_ctx: main_ctx, mtp_ctx: mtp_ctx, spec_ref: spec_ref, - n_draft: n_draft + # A DFlash drafter clamps n_draft to its block size; report what runs. + n_draft: LlamaCppEx.NIF.speculative_stats(spec_ref).n_draft }} end end @@ -404,11 +471,11 @@ defmodule LlamaCppEx.MTP do * `:tokens_emitted` - tokens streamed back to the caller * `:tokens_per_sec` - throughput over the active generation window * `:timing_us` - `%{draft: μs, verify: μs, sample: μs, ckpt: μs, other: μs, - total: μs}`. `:ckpt` is the recurrent-state save/restore that only hybrid - models pay and is zero elsewhere; on Qwen 3.8 it is large enough to decide - whether speculation helps at all. `:other` is whatever falls outside the - named buckets, dominated on Metal by GPU-sync waits from the previous - iteration's async verify decode. + total: μs}`. `:ckpt` is the recurrent-state save/restore a hybrid target + pays when it has fewer recurrent snapshots than `n_draft` (never, for a + session built by `init/2`), and is zero elsewhere. `:other` is whatever + falls outside the named buckets, dominated on Metal by GPU-sync waits + from the previous iteration's async verify decode. * `:n_draft` - max draft length configured at init Counters are cumulative across all `stream/3` / `generate/3` calls on this diff --git a/lib/llama_cpp_ex/nif.ex b/lib/llama_cpp_ex/nif.ex index ef0c2ec..046917d 100644 --- a/lib/llama_cpp_ex/nif.ex +++ b/lib/llama_cpp_ex/nif.ex @@ -39,12 +39,10 @@ defmodule LlamaCppEx.NIF do def model_load( _path, _n_gpu_layers, - _use_mmap, + _load_mode, _main_gpu, _split_mode, _tensor_split, - _use_mlock, - _use_direct_io, _vocab_only, _check_tensors, _load_mtp, @@ -58,6 +56,7 @@ defmodule LlamaCppEx.NIF do def model_n_layer_nextn(_model), do: :erlang.nif_error(:not_loaded) def model_desc(_model), do: :erlang.nif_error(:not_loaded) def model_size(_model), do: :erlang.nif_error(:not_loaded) + def model_devices(_model), do: :erlang.nif_error(:not_loaded) def model_n_params(_model), do: :erlang.nif_error(:not_loaded) def model_chat_template(_model), do: :erlang.nif_error(:not_loaded) @@ -177,7 +176,7 @@ defmodule LlamaCppEx.NIF do do: :erlang.nif_error(:not_loaded) # Speculative decoding (MTP) - def speculative_init(_ctx_tgt, _ctx_dft, _n_draft), do: :erlang.nif_error(:not_loaded) + def speculative_init(_ctx_tgt, _ctx_dft, _n_draft, _type), do: :erlang.nif_error(:not_loaded) def speculative_stats(_spec), do: :erlang.nif_error(:not_loaded) def speculative_print_stats(_spec), do: :erlang.nif_error(:not_loaded) diff --git a/lib/llama_cpp_ex/rpc.ex b/lib/llama_cpp_ex/rpc.ex index 06ac544..c2228dc 100644 --- a/lib/llama_cpp_ex/rpc.ex +++ b/lib/llama_cpp_ex/rpc.ex @@ -24,9 +24,10 @@ defmodule LlamaCppEx.RPC do LLAMA_RPC=1 LLAMA_BACKEND=cuda mix compile - On Linux the transport auto-negotiates RDMA when both peers have a usable HCA, - which is a build-time choice (`LLAMA_RPC_RDMA`, default on) with no runtime - switch. See `LlamaCppEx.RPC.Server` for how to tell which transport you got. + On Linux the transport auto-negotiates RDMA when both peers have a usable HCA + (`LLAMA_RPC_RDMA`, default on). Setting `GGML_RPC_NO_RDMA` in either peer's + environment keeps that peer on TCP. See `LlamaCppEx.RPC.Server` for how to + tell which transport you got. > #### A peer failure kills the VM {: .error} > @@ -38,11 +39,14 @@ defmodule LlamaCppEx.RPC do > to catch. > > Registration is the exception and the reason this module is shaped the way it - > is: an unreachable endpoint is reported as `{:error, :unreachable}` rather - > than aborting, so a two-node setup can be *checked* before a model load and - > is only *fatal* during one. Check with `ping/1` before you load; treat the VM - > as the unit of restart afterwards. Real fault isolation would mean running - > the RPC client in a separate OS process, which is a different architecture. + > is. Upstream's registration aborts too (since llama.cpp #18626), so + > `add_server/1` first connects and exchanges the protocol HELLO itself: an + > unreachable or incompatible endpoint is reported as `{:error, :unreachable}` + > rather than aborting, so a two-node setup can be *checked* before a model + > load and is only *fatal* during one. Check with `ping/1` before you load; + > treat the VM as the unit of restart afterwards. Real fault isolation would + > mean running the RPC client in a separate OS process, which is a different + > architecture. ## Two device orderings, and they disagree @@ -96,7 +100,8 @@ defmodule LlamaCppEx.RPC do Returns the number of devices the endpoint contributed. Idempotent: upstream memoizes the registration per endpoint, so registering the same endpoint twice - succeeds and reports `0` added the second time. + succeeds and reports `0` added the second time. Every call still connects and + exchanges the HELLO first, with a 5-second timeout. Must be called **before** `LlamaCppEx.Model.load/2`, because tensor placement is computed from the devices that exist at load time. @@ -107,11 +112,12 @@ defmodule LlamaCppEx.RPC do ## Errors * `{:error, :rpc_unsupported}` — the NIF was built without `LLAMA_RPC=1`. - * `{:error, :unreachable}` — nothing answered, or the peer's RPC protocol - major/minor did not match ours. Upstream collapses both to a null - registration, and a null registration is silently ignored by - `ggml_backend_register`, so this check is the only thing standing between - you and a model that quietly loads onto the wrong devices. + * `{:error, :unreachable}` — nothing answered within 5 seconds, the + endpoint is not `host:port`, the peer's RPC protocol major/minor did not + match ours, or the worker reported no devices. Upstream aborts the VM on + the first three, and silently ignores the last one at registration, so + this check is the only thing standing between you and a crash or a model + that quietly loads onto the wrong devices. """ @spec add_server(endpoint()) :: {:ok, non_neg_integer()} | {:error, error()} def add_server(endpoint) when is_binary(endpoint) do @@ -160,13 +166,13 @@ defmodule LlamaCppEx.RPC do Reports whether an endpoint is reachable and speaks a compatible protocol. This is `add_server/1` under a name that says what it is good for. It has the - same side effect — a reachable endpoint stays registered — and it is **not** a - health check you can repeat to monitor a peer: the answer is memoized after the - first success, and once a model is loaded a dead peer aborts the VM rather than - failing a probe. + same side effect — a reachable endpoint stays registered. Each call connects + and exchanges the HELLO again, so it does report a peer that has gone away + since registering; but once a model is loaded a dead peer aborts the VM on + the next RPC command, before any probe gets a say. - Use it once, before loading, to turn "the model landed on the wrong devices" - into a clear error. + Use it before loading, to turn "the model landed on the wrong devices" or a + VM abort into a clear error. """ @spec ping(endpoint()) :: :ok | {:error, error()} def ping(endpoint) when is_binary(endpoint) do diff --git a/lib/llama_cpp_ex/rpc/server.ex b/lib/llama_cpp_ex/rpc/server.ex index bce51f6..f8a895e 100644 --- a/lib/llama_cpp_ex/rpc/server.ex +++ b/lib/llama_cpp_ex/rpc/server.ex @@ -30,11 +30,12 @@ defmodule LlamaCppEx.RPC.Server do ## Which transport did I get? - Transport selection is silent auto-negotiation. There is no env var, no - endpoint scheme and no return value that tells you. The only signal is the - worker's own log with `GGML_RPC_DEBUG=1`: the absence of - `RDMA activate failed, staying on TCP` means RDMA is live. To force TCP for an - A/B, set `GGML_RDMA_DEV` to a device name that does not exist. + Transport selection is auto-negotiation in the HELLO, with no endpoint scheme + and no return value that tells you. The signal is the worker's own log with + `GGML_RPC_DEBUG=1`: `RDMA activated` means RDMA is live, `RDMA activate + failed, staying on TCP` means it is not. To force TCP for an A/B, set + `GGML_RPC_NO_RDMA=1` in the worker's environment (or the client's): a peer + with it set advertises no RDMA capability, so the connection stays on TCP. ## This process cannot stop the server diff --git a/lib/llama_cpp_ex/server.ex b/lib/llama_cpp_ex/server.ex index c682421..09303aa 100644 --- a/lib/llama_cpp_ex/server.ex +++ b/lib/llama_cpp_ex/server.ex @@ -314,8 +314,9 @@ defmodule LlamaCppEx.Server do `:main_gpu`, `:split_mode`, `:tensor_split`, `:use_mmap`, `:use_mlock`, `:use_direct_io`, `:check_tensors` and `:rpc_servers`. The three load flags collapse into llama.cpp's single `load_mode`: `:use_direct_io` wins - outright, otherwise `:use_mlock` and `:use_mmap` combine; see - `LlamaCppEx.Model.load/2`. `:rpc_servers` registers remote endpoints before + outright, otherwise `:use_mlock` and `:use_mmap` combine, and with none + given llama.cpp chooses; see `LlamaCppEx.Model.load/2`. `:rpc_servers` + registers remote endpoints before the load so their devices can hold part of the model — see `LlamaCppEx.RPC`, including the caveat that a peer failure aborts the VM. * GenServer options like `:name`. @@ -1155,7 +1156,7 @@ defmodule LlamaCppEx.Server do {raw_match, own_match} = own_cache_match(state, slot, tokens, n_tokens, cache_prompt?, scope, lcp) - state = maybe_warn_prefix_instability(state, seq_id, slot, raw_match, cache_prompt?) + state = maybe_warn_prefix_instability(state, seq_id, slot, raw_match, n_tokens, cache_prompt?) {state, n_match} = resolve_prefix_cache(state, seq_id, tokens, own_match, cache_prompt?, scope) @@ -1367,9 +1368,15 @@ defmodule LlamaCppEx.Server do # turns (e.g. Qwen thinking-strip re-rendering) — every such request pays a # near-full re-prefill that the caller probably believes is cached. Emit # telemetry per occurrence and warn in the log once per server. - defp maybe_warn_prefix_instability(state, seq_id, slot, raw_match, cache_prompt?) do + # + # Only when the prompt itself diverges from the cache: own_cache_match caps + # the match at n_tokens - 1, so a prompt that is wholly cached (the same + # prompt again, a regenerate) matches exactly that and is a perfect hit, not + # instability, however long the cached history behind it. + defp maybe_warn_prefix_instability(state, seq_id, slot, raw_match, n_tokens, cache_prompt?) do suspicious? = - cache_prompt? and slot.cached_pos >= @prefix_instability_min_cached and + cache_prompt? and raw_match < n_tokens - 1 and + slot.cached_pos >= @prefix_instability_min_cached and raw_match > slot.cached_pos * 0.1 and raw_match < slot.cached_pos * 0.5 if suspicious? do diff --git a/scripts/spark/e2e.exs b/scripts/spark/e2e.exs new file mode 100644 index 0000000..78b71bd --- /dev/null +++ b/scripts/spark/e2e.exs @@ -0,0 +1,414 @@ +# End-to-end check of the CUDA build on a Spark, single node or across two. +# +# # one node: two chat models and an embedding model resident at once +# scripts/spark/remote.sh --env MIX_ENV=test --env LLAMA_RPC=1 spark-1 \ +# mix run scripts/spark/e2e.exs single +# +# # two nodes: a model layer-split over the fabric next to a local one +# scripts/spark/rpc-worker.sh start spark-2 +# scripts/spark/remote.sh --env MIX_ENV=test --env LLAMA_RPC=1 spark-1 \ +# mix run scripts/spark/e2e.exs two-node 10.100.64.2:50052 +# +# Every check asserts on what the model said, not only on {:ok, _}: a model +# that loads onto the wrong device, or a split that corrupts activations, +# still returns {:ok, text}. Exits non-zero on any failure. +# +# Models default to the files on spark-1 and can be overridden: +# SPARK_E2E_BIG Qwen3.8-27B NVFP4 (dense, hybrid GDN, MTP head) +# SPARK_E2E_MOE Qwen3-30B-A3B Q4_K_M +# SPARK_E2E_EMBED Qwen3-Embedding-0.6B +# SPARK_E2E_TOOLS a model whose template supports tools + +home = System.user_home!() + +big = + System.get_env("SPARK_E2E_BIG") || + Path.join( + home, + "models/esatapedico/Qwen3.8-27B-NVFP4-MTP-GGUF/main/Qwen3.8-27B-NVFP4-MTP-MID-HIGH.gguf" + ) + +moe = + System.get_env("SPARK_E2E_MOE") || + Path.join(home, "models/Qwen/Qwen3-30B-A3B-GGUF/main/Qwen3-30B-A3B-Q4_K_M.gguf") + +embed = + System.get_env("SPARK_E2E_EMBED") || + Path.join(home, "models/test/Qwen3-Embedding-0.6B-f16.gguf") + +defmodule E2E do + alias LlamaCppEx.ModelManager + + def start, do: Agent.start_link(fn -> [] end, name: __MODULE__) + + def step(label, fun) do + IO.write(" " <> String.pad_trailing(label, 52)) + t0 = System.monotonic_time(:millisecond) + + outcome = + try do + fun.() + rescue + e -> {:error, Exception.message(e)} + catch + kind, reason -> {:error, "#{kind}: #{inspect(reason)}"} + end + + ms = System.monotonic_time(:millisecond) - t0 + + case outcome do + {:ok, detail} -> + IO.puts("PASS #{detail} (#{ms} ms)") + Agent.update(__MODULE__, &[:ok | &1]) + + {:error, detail} -> + IO.puts("FAIL #{detail}") + Agent.update(__MODULE__, &[:error | &1]) + end + end + + def finish do + results = Agent.get(__MODULE__, & &1) + failed = Enum.count(results, &(&1 == :error)) + + if failed == 0 do + IO.puts("\nall #{length(results)} checks passed") + else + IO.puts("\n#{failed} of #{length(results)} checks FAILED") + System.halt(1) + end + end + + def content({:ok, %{choices: [%{message: message} | _]}}), do: message.content || "" + def content(other), do: raise("unexpected reply: #{inspect(other, limit: 5)}") + + # Factual prompts with one checkable answer each. Thinking is off so the + # answer is in `content`, and the token budget leaves room for a sentence. + def ask(target, question, expect, opts \\ []) do + reply = + target + |> chat(question, opts) + |> content() + + if reply =~ expect, + do: {:ok, inspect(String.slice(String.trim(reply), 0, 40))}, + else: {:error, "expected #{inspect(expect)} in #{inspect(reply)}"} + end + + defp chat({:manager, id}, question, opts), + do: ModelManager.chat_completion(id, [%{role: "user", content: question}], chat_opts(opts)) + + defp chat(server, question, opts), + do: LlamaCppEx.chat_completion(server, [%{role: "user", content: question}], chat_opts(opts)) + + defp chat_opts(opts), + do: Keyword.merge([max_tokens: 64, temp: 0.0, enable_thinking: false], opts) + + def questions do + [ + {"What is the capital of France? One word.", ~r/Paris/i}, + {"What is 12 times 12? Reply with the number only.", ~r/144/}, + {"Which planet is known as the Red Planet? One word.", ~r/Mars/i}, + {"What is the chemical symbol for gold? Reply with the symbol only.", ~r/\bAu\b/} + ] + end + + def concurrent(target, n) do + qs = questions() |> Stream.cycle() |> Enum.take(n) + + results = + qs + |> Task.async_stream(fn {q, e} -> ask(target, q, e) end, + max_concurrency: n, + timeout: 300_000 + ) + |> Enum.map(fn {:ok, r} -> r end) + + case Enum.reject(results, &match?({:ok, _}, &1)) do + [] -> {:ok, "#{n}/#{n} correct"} + bad -> {:error, inspect(bad)} + end + end + + def stream_text(server, question) do + chunks = + server + |> LlamaCppEx.stream_chat_completion([%{role: "user", content: question}], + max_tokens: 64, + temp: 0.0, + enable_thinking: false + ) + |> Enum.to_list() + + text = + chunks + |> Enum.flat_map(fn + %{choices: [%{delta: %{content: c}} | _]} when is_binary(c) -> [c] + _ -> [] + end) + |> Enum.join() + + {length(chunks), text} + end + + def cosine(a, b) do + dot = Enum.zip_with(a, b, &(&1 * &2)) |> Enum.sum() + norm = fn v -> :math.sqrt(Enum.reduce(v, 0.0, &(&1 * &1 + &2))) end + dot / (norm.(a) * norm.(b)) + end + + def mem_gb do + {out, 0} = System.cmd("free", ["-b"]) + [_, total, used | _] = out |> String.split("\n") |> Enum.at(1) |> String.split() + + "#{Float.round(String.to_integer(used) / 1.0e9, 1)} / #{Float.round(String.to_integer(total) / 1.0e9, 1)} GB used" + end +end + +{:ok, _} = E2E.start() +LlamaCppEx.set_log_level(:error) +:ok = LlamaCppEx.init() + +case System.argv() do + ["single" | _] -> + IO.puts("\n== single node: two chat models + one embedding model resident\n") + {:ok, _} = LlamaCppEx.ModelSupervisor.start_link(memory_budget: :infinity) + alias LlamaCppEx.ModelManager + + E2E.step("CUDA0 is the GPU and reports unified memory", fn -> + case Enum.find(LlamaCppEx.devices(), &(&1.backend == "CUDA")) do + %{name: "CUDA0", type: :igpu} = d -> + {:ok, "#{d.description}, #{div(d.memory_total, 1_000_000_000)} GB"} + + other -> + {:error, inspect(other)} + end + end) + + E2E.step("load NVFP4 27B (server, n_parallel 4)", fn -> + {:ok, _} = + ModelManager.load("big", {:path, big}, n_gpu_layers: -1, n_parallel: 4, n_ctx: 16_384) + + {:ok, Path.basename(big)} + end) + + E2E.step("load Qwen3-30B-A3B MoE (server, n_parallel 4)", fn -> + {:ok, _} = + ModelManager.load("moe", {:path, moe}, n_gpu_layers: -1, n_parallel: 4, n_ctx: 16_384) + + {:ok, Path.basename(moe)} + end) + + E2E.step("load embedding model (direct)", fn -> + {:ok, _} = + ModelManager.load("embed", {:path, embed}, capabilities: [:embed], n_gpu_layers: -1) + + {:ok, Path.basename(embed)} + end) + + E2E.step("three models resident", fn -> + ids = ModelManager.list() |> Enum.map(& &1.id) |> Enum.sort() + if ids == ["big", "embed", "moe"], do: {:ok, E2E.mem_gb()}, else: {:error, inspect(ids)} + end) + + for {q, e} <- E2E.questions() |> Enum.take(2), id <- ["big", "moe"] do + E2E.step("#{id}: #{String.slice(q, 0, 30)}", fn -> E2E.ask({:manager, id}, q, e) end) + end + + E2E.step("both chat models at once, 8 requests each", fn -> + [a, b] = + Task.await_many( + [ + Task.async(fn -> E2E.concurrent({:manager, "big"}, 8) end), + Task.async(fn -> E2E.concurrent({:manager, "moe"}, 8) end) + ], + 600_000 + ) + + case {a, b} do + {{:ok, x}, {:ok, y}} -> {:ok, "big #{x}, moe #{y}"} + _ -> {:error, inspect({a, b})} + end + end) + + E2E.step("embeddings rank a paraphrase above an unrelated text", fn -> + {:ok, a} = ModelManager.embed("embed", "The cat sat on the mat.") + {:ok, b} = ModelManager.embed("embed", "A cat was sitting on a rug.") + {:ok, c} = ModelManager.embed("embed", "Quarterly revenue grew by eight percent.") + {sim, far} = {E2E.cosine(a, b), E2E.cosine(a, c)} + + if sim > far, + do: {:ok, "#{Float.round(sim, 3)} > #{Float.round(far, 3)}"}, + else: {:error, "#{sim} <= #{far}"} + end) + + E2E.step("unload one model, the other keeps serving", fn -> + :ok = ModelManager.unload("moe") + E2E.ask({:manager, "big"}, "What is the capital of France? One word.", ~r/Paris/i) + end) + + ["two-node", endpoint | _] -> + IO.puts("\n== two nodes over #{endpoint}: split model + local model\n") + + E2E.step("worker answers the HELLO", fn -> + case LlamaCppEx.RPC.ping(endpoint) do + :ok -> {:ok, inspect(Enum.map(LlamaCppEx.RPC.devices(), & &1.name))} + other -> {:error, inspect(other)} + end + end) + + {:ok, split} = + LlamaCppEx.Server.start_link( + model_path: big, + rpc_servers: [endpoint], + devices: ["CUDA0", "RPC0"], + split_mode: :layer, + tensor_split: [0.5, 0.5], + n_gpu_layers: -1, + n_parallel: 4, + n_ctx: 16_384 + ) + + E2E.step("NVFP4 27B loads split across CUDA0 + RPC0", fn -> + {:ok, _} = LlamaCppEx.Server.generate(split, "Hello", max_tokens: 1) + {:ok, E2E.mem_gb()} + end) + + {:ok, local} = + LlamaCppEx.Server.start_link( + model_path: moe, + devices: ["CUDA0"], + n_gpu_layers: -1, + n_parallel: 4, + n_ctx: 16_384 + ) + + E2E.step("Qwen3-30B-A3B loads locally beside it", fn -> + {:ok, _} = LlamaCppEx.Server.generate(local, "Hello", max_tokens: 1) + {:ok, E2E.mem_gb()} + end) + + for {q, e} <- E2E.questions() do + E2E.step("split: #{String.slice(q, 0, 36)}", fn -> E2E.ask(split, q, e) end) + end + + E2E.step("split: streaming chat completion", fn -> + {n, text} = E2E.stream_text(split, "What is the capital of France? One word.") + + if text =~ ~r/Paris/i, + do: {:ok, "#{n} chunks, #{inspect(String.trim(text))}"}, + else: {:error, inspect(text)} + end) + + E2E.step("split + local at once, 8 requests each", fn -> + [a, b] = + Task.await_many( + [ + Task.async(fn -> E2E.concurrent(split, 8) end), + Task.async(fn -> E2E.concurrent(local, 8) end) + ], + 600_000 + ) + + case {a, b} do + {{:ok, x}, {:ok, y}} -> {:ok, "split #{x}, local #{y}"} + _ -> {:error, inspect({a, b})} + end + end) + + E2E.step("split: tool call", fn -> + tools = [ + %{ + type: "function", + function: %{ + name: "get_weather", + description: "Get the current weather for a city", + parameters: %{ + type: "object", + properties: %{city: %{type: "string"}}, + required: ["city"] + } + } + } + ] + + {:ok, %{choices: [%{message: msg, finish_reason: reason} | _]}} = + LlamaCppEx.chat_completion( + split, + [%{role: "user", content: "What's the weather in Paris right now?"}], + tools: tools, + max_tokens: 512, + temp: 0.0 + ) + + case msg do + %{tool_calls: [%{function: %{name: "get_weather", arguments: args}} | _]} -> + if args =~ ~r/Paris/i, do: {:ok, "#{reason}: #{args}"}, else: {:error, args} + + other -> + {:error, inspect(other, limit: 8)} + end + end) + + drafter = + System.get_env("SPARK_E2E_DFLASH") || + Path.join( + home, + "models/z-lab/Qwen3.8-27B-DFlash2-GGUF/main/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" + ) + + {:ok, target} = + LlamaCppEx.Model.load(big, + devices: ["CUDA0", "RPC0"], + split_mode: :layer, + tensor_split: [0.5, 0.5], + n_gpu_layers: -1 + ) + + E2E.step("DFlash drafter on CUDA0 alone is refused, not aborted", fn -> + {:ok, local_only} = LlamaCppEx.Model.load(drafter, devices: ["CUDA0"], n_gpu_layers: -1) + + case LlamaCppEx.MTP.init(target, type: :dflash, draft_model: local_only, n_ctx: 2048) do + {:error, msg} -> if msg =~ "RPC0", do: {:ok, "names RPC0"}, else: {:error, msg} + other -> {:error, inspect(other)} + end + end) + + E2E.step("DFlash over the split target", fn -> + {:ok, d} = + LlamaCppEx.Model.load(drafter, + devices: ["CUDA0", "RPC0"], + split_mode: :layer, + tensor_split: [1.0, 0.0], + n_gpu_layers: -1 + ) + + {:ok, spec} = + LlamaCppEx.MTP.init(target, type: :dflash, draft_model: d, n_draft: 7, n_ctx: 2048) + + {:ok, prompt} = + LlamaCppEx.Chat.apply_template( + target, + [%{role: "user", content: "What is the capital of France? One word."}], + enable_thinking: false + ) + + {:ok, text} = LlamaCppEx.MTP.generate(spec, prompt, max_tokens: 8, temp: 0.0) + stats = LlamaCppEx.MTP.stats(spec) + + if text =~ ~r/Paris/i, + do: + {:ok, + "#{inspect(String.trim(text))}, #{stats.drafts_accepted}/#{stats.drafts_generated} drafts"}, + else: {:error, inspect(text)} + end) + + GenServer.stop(local) + GenServer.stop(split) + + _ -> + IO.puts("usage: mix run scripts/spark/e2e.exs single | two-node ") + System.halt(2) +end + +E2E.finish() diff --git a/scripts/spark/lib.sh b/scripts/spark/lib.sh index 69e1e0a..efcf887 100755 --- a/scripts/spark/lib.sh +++ b/scripts/spark/lib.sh @@ -18,12 +18,18 @@ SPARK_NODES=(spark-1 spark-2) # a GID against the socket's local address, so one peer connection can only ever # use one link. Link 0 is the one everything uses; link 1 is listed so the fact # sheet can prove both trained. +# +# `lan` is the 10 GbE management NIC (enP7s7, MTU 1500, no RDMA-capable HCA on +# that subnet). It exists only as the slow arm of a transport comparison: a +# two-node run belongs on the fabric. spark_fabric_ip() { case "$1:${2:-0}" in spark-1:0) printf '10.100.64.1' ;; spark-1:1) printf '10.100.65.1' ;; + spark-1:lan) printf '10.0.20.26' ;; spark-2:0) printf '10.100.64.2' ;; spark-2:1) printf '10.100.65.2' ;; + spark-2:lan) printf '10.0.20.25' ;; *) return 1 ;; esac } @@ -55,9 +61,11 @@ SPARK_TOOLCHAIN_DONOR="${SPARK_TOOLCHAIN_DONOR:-spark-1}" # --- CPU topology ------------------------------------------------------------ # The clusters are INTERLEAVED. X925 (performance) is 5-9,15-19 and A725 -# (efficiency) is 0-4,10-14, so the obvious `taskset -c 0-9` pins a job entirely -# to little cores. Measured: 1-byte RDMA ping-pong p50 21.5 us on cpu19 versus -# 29.1 us on cpu0. +# (efficiency) is 0-4,10-14, so the obvious `taskset -c 0-9` gets five of each. +# Measured: 1-byte RDMA ping-pong p50 21.5 us on cpu19 versus 29.1 us on cpu0 -- +# yet a two-node *client* decodes ~25% faster on the A725 cores (gpt-oss-120b, +# 37-38 -> 47 tok/s through the binding; docs/dgx-spark.md, "Pin the client to +# the little cores"). remote.sh --little-cores does that. SPARK_BIG_CORES="5-9,15-19" SPARK_LITTLE_CORES="0-4,10-14" SPARK_BIG_CORE_COUNT=10 diff --git a/scripts/spark/remote.sh b/scripts/spark/remote.sh index 63807a1..0dcf013 100755 --- a/scripts/spark/remote.sh +++ b/scripts/spark/remote.sh @@ -26,6 +26,8 @@ Options: (default: the synced repo; "-" for $HOME) --env K=V extra environment variable, repeatable --big-cores wrap the command in `taskset -c 5-9,15-19` (Cortex-X925) + --little-cores wrap the command in `taskset -c 0-4,10-14` (Cortex-A725): + where a two-node client decodes ~25% faster --tty allocate a remote tty --print print the remote script instead of running it USAGE @@ -35,6 +37,7 @@ login=0 tty=0 print_only=0 big_cores=0 +little_cores=0 forward_agent=0 dir="$SPARK_REMOTE_DIR" extra_env=() @@ -45,6 +48,7 @@ while [ $# -gt 0 ]; do --tty) tty=1; shift ;; --print) print_only=1; shift ;; --big-cores) big_cores=1; shift ;; + --little-cores) little_cores=1; shift ;; --forward-agent) forward_agent=1; shift ;; --dir) dir="${2:?--dir needs a path}"; shift 2 ;; --env) extra_env+=("${2:?--env needs K=V}"); shift 2 ;; @@ -60,8 +64,10 @@ done node=$1; shift spark_is_node "$node" || spark_die "unknown node '$node' (known: ${SPARK_NODES[*]})" +[ "$big_cores" -eq 1 ] && [ "$little_cores" -eq 1 ] && spark_die "--big-cores and --little-cores are exclusive" prefix=() [ "$big_cores" -eq 1 ] && prefix=(taskset -c "$SPARK_BIG_CORES") +[ "$little_cores" -eq 1 ] && prefix=(taskset -c "$SPARK_LITTLE_CORES") # Assemble the remote script. Every word is quoted here rather than left to # ssh's own re-splitting, which joins its arguments with spaces and would mangle diff --git a/scripts/spark/rpc-worker.sh b/scripts/spark/rpc-worker.sh index 3c337dd..5955b9d 100755 --- a/scripts/spark/rpc-worker.sh +++ b/scripts/spark/rpc-worker.sh @@ -56,10 +56,10 @@ cache=1 # legitimately holds tens of GB of weights, so an absolute default would be # either useless or a footgun; this catches the leak, not the workload. rss_fraction=0.92 -# Force the TCP path. There is no runtime switch for the transport: selection is -# silent auto-negotiation, and GGML_RDMA_DEV pointing at a device that does not -# exist is the only lever short of a -DGGML_RPC_RDMA=OFF build. Both ends have -# to agree, so pass --tcp here and GGML_RDMA_DEV to the client too. +# Force the TCP path with GGML_RPC_NO_RDMA, upstream's runtime switch: a peer +# with it set advertises no RDMA capability in the HELLO, so the connection +# stays on TCP whatever the other end can do. Setting it on the worker alone is +# enough; the client can set it too. tcp=0 # `logs` takes a bare line count. lines=100 @@ -137,7 +137,7 @@ exec taskset -c $SPARK_BIG_CORES mix run scripts/spark/rpc_worker.exs" # trees coexist and switching back is a cache hit. printf 'export LLAMA_RPC=1\n' [ "$debug" -eq 1 ] && printf 'export GGML_RPC_DEBUG=1\n' - [ "$tcp" -eq 1 ] && printf 'export GGML_RDMA_DEV=no-such-hca\n' + [ "$tcp" -eq 1 ] && printf 'export GGML_RPC_NO_RDMA=1\n' printf 'cd "$HOME/%s"\n' "$SPARK_REMOTE_DIR" printf '%s\n' "$inner" ) diff --git a/scripts/spark/rpc_check.exs b/scripts/spark/rpc_check.exs index 69ec8be..fbdc661 100644 --- a/scripts/spark/rpc_check.exs +++ b/scripts/spark/rpc_check.exs @@ -7,8 +7,9 @@ # scripts/spark/rpc-worker.sh logs spark-2 # # Set GGML_RPC_DEBUG=1 for the client side of the transport negotiation. The -# only way to know whether you got RDMA or TCP is that log: selection is silent -# auto-negotiation with no env var, no endpoint scheme and no return value. +# only way to know whether you got RDMA or TCP is that log: selection is +# auto-negotiation with no endpoint scheme and no return value +# (GGML_RPC_NO_RDMA=1 on either peer forces TCP). endpoint = case System.argv() do diff --git a/test/embeddings_test.exs b/test/embeddings_test.exs index 20ecaa1..13915ce 100644 --- a/test/embeddings_test.exs +++ b/test/embeddings_test.exs @@ -41,6 +41,13 @@ defmodule LlamaCppEx.EmbeddingsTest do # Guards the multi-sequence batch refactor: batched embeddings (single # context, many sequences) must match one-context-per-text, including when # grouping splits the batch (max_batch_sequences forces multiple groups). + # + # Compared by cosine similarity, not elementwise. A batch runs different + # kernels than a single text on GPU backends (CUDA: up to 0.0027 per element + # with bge-small, cos 0.99988), so an absolute bound tuned on CPU fails there. + # The bugs this guards against are far away: rows swapped between sequences + # give the cosine of two different texts (at most 0.70 on bge-small, + # embeddinggemma and Qwen3-Embedding), and the wrong pooling at most 0.93. test "embed_batch matches per-text embed (incl. multi-group)", %{emb_model: em} do texts = [ "Elixir is a functional programming language.", @@ -60,8 +67,8 @@ defmodule LlamaCppEx.EmbeddingsTest do assert {:ok, grouped} = LlamaCppEx.embed_batch(em, texts, max_batch_sequences: 2) assert length(batched) == length(texts) - assert max_elementwise_diff(batched, reference) < 1.0e-3 - assert max_elementwise_diff(grouped, reference) < 1.0e-3 + assert min_pairwise_cosine(batched, reference) > 0.999 + assert min_pairwise_cosine(grouped, reference) > 0.999 end # Regression: llama_context reserves its prompt graph for n_ubatch rounded up @@ -78,11 +85,16 @@ defmodule LlamaCppEx.EmbeddingsTest do assert Enum.all?(vectors, &(length(&1) == LlamaCppEx.Model.n_embd_out(em))) end - defp max_elementwise_diff(a, b) do - Enum.zip(a, b) + # Lowest cosine similarity between the i-th vectors of two equally long lists. + defp min_pairwise_cosine(a, b) do + a + |> Enum.zip(b) |> Enum.map(fn {va, vb} -> - Enum.zip(va, vb) |> Enum.map(fn {x, y} -> abs(x - y) end) |> Enum.max() + dot = va |> Enum.zip(vb) |> Enum.reduce(0.0, fn {x, y}, acc -> acc + x * y end) + dot / (norm(va) * norm(vb)) end) - |> Enum.max() + |> Enum.min() end + + defp norm(v), do: :math.sqrt(Enum.reduce(v, 0.0, &(&1 * &1 + &2))) end diff --git a/test/llama_cpp_ex_test.exs b/test/llama_cpp_ex_test.exs index 8a63926..243ad26 100644 --- a/test/llama_cpp_ex_test.exs +++ b/test/llama_cpp_ex_test.exs @@ -1092,12 +1092,14 @@ defmodule LlamaCppExTest do %{model: model, mtp: mtp} end - test "contexts use no recurrent-state rollback slots", %{mtp: mtp} do - # Matching upstream server, the draft ctx is created with n_rs_seq=0 — - # MTP rolls back via cached hidden states (pending_h / verify_h), not - # recurrent-state snapshots. Both contexts report 0. + test "the target keeps a recurrent snapshot per draft position, the draft none", %{ + mtp: mtp + } do + # Matching upstream (need_n_rs_seq / common_speculative_init_result): the + # target rolls a rejected draft back on n_draft recurrent snapshots, and + # the draft ctx has none — MTP rolls back via cached hidden states. assert LlamaCppEx.Context.n_rs_seq(mtp.mtp_ctx) == 0 - assert LlamaCppEx.Context.n_rs_seq(mtp.main_ctx) == 0 + assert LlamaCppEx.Context.n_rs_seq(mtp.main_ctx) == mtp.n_draft end test "stream produces text and a sensible acceptance rate", %{mtp: mtp} do diff --git a/test/mtp_model_test.exs b/test/mtp_model_test.exs index 8aeee78..d88ce9e 100644 --- a/test/mtp_model_test.exs +++ b/test/mtp_model_test.exs @@ -305,10 +305,18 @@ defmodule LlamaCppEx.MTPSidecarTest do end # A hybrid target (Qwen 3.8: SSM layers beside attention ones) cannot roll back - # part of a sequence natively, so the loop snapshots the recurrent state every - # iteration. That cost is the difference between speculation paying off and not, - # so it gets its own bucket rather than hiding inside :other. - test "timing_us reports a ckpt bucket", %{session: session} do + # part of a sequence from its KV cache alone. init/2 gives it one recurrent + # snapshot per draft position, as upstream's server does, so a rejected draft + # rolls back natively. Without them the loop saves the recurrent state every + # iteration and re-decodes the accepted prefix on every partial accept, which + # measured 21 tok/s against 27 at n_draft 2 on Qwen3.8-27B NVFP4 — the :ckpt + # bucket is where that path would show. + test "a hybrid target rolls back on recurrent snapshots, not checkpoints", %{ + session: session + } do + assert LlamaCppEx.Context.n_rs_seq(session.main_ctx) == session.n_draft + assert LlamaCppEx.Context.n_rs_seq(session.mtp_ctx) == 0 + assert {:ok, _} = MTP.generate(session, "Count to ten:", max_tokens: 24, temp: 0.0) timing = MTP.stats(session).timing_us @@ -317,8 +325,7 @@ defmodule LlamaCppEx.MTPSidecarTest do assert Map.has_key?(timing, key), "timing_us is missing #{inspect(key)}" end - assert timing.ckpt > 0, - "a hybrid target should have paid for at least one recurrent-state snapshot" + assert timing.ckpt == 0, "the loop fell back to recurrent-state checkpoints" # The named buckets are carved out of total, never billed twice on top of it. assert timing.draft + timing.verify + timing.sample + timing.ckpt <= timing.total @@ -347,6 +354,88 @@ defmodule LlamaCppEx.MTPSidecarTest do end end +defmodule LlamaCppEx.DFlashTest do + # DFlash block-diffusion drafting through the same session API, type: :dflash. + # The drafter is a separate `dflash`-architecture GGUF trained against one + # target: it reads that target's hidden states at fixed layer indices. + # + # LLAMA_SMOKE_DFLASH_MODEL=/path/to/Qwen3.8-27B-Q4_K_M.gguf \ + # LLAMA_SMOKE_DFLASH_DRAFT_MODEL=/path/to/Qwen3.8-27B-DFlash2-Q4_K_M.gguf \ + # mix test --include dflash + # + # async: false and one session for the module, as for the MTP modules above. + use ExUnit.Case, async: false + + alias LlamaCppEx.MTP + + @moduletag :dflash + @moduletag timeout: 300_000 + + setup_all do + :ok = LlamaCppEx.init() + + {:ok, target} = + LlamaCppEx.load_model(LlamaCppEx.TestModels.path!(:dflash), n_gpu_layers: -1) + + {:ok, draft} = + LlamaCppEx.load_model(LlamaCppEx.TestModels.path!(:dflash_draft), n_gpu_layers: -1) + + # Ask for more than any DFlash block holds, to see it clamped. + {:ok, session} = MTP.init(target, type: :dflash, draft_model: draft, n_ctx: 2048, n_draft: 64) + + %{target: target, draft: draft, session: session} + end + + test "n_draft is clamped to the drafter's block size", %{session: session} do + # The loop sizes its verify batch from this; asking upstream for more than + # the block holds would draft nothing past it. + assert session.n_draft < 64 + assert session.n_draft == MTP.stats(session).n_draft + end + + test "generate/3 drafts and the target accepts some of it", %{session: session} do + before = MTP.stats(session) + + assert {:ok, text} = + MTP.generate(session, "List the days of the week, comma separated:", + max_tokens: 32, + temp: 0.0 + ) + + assert text =~ "Monday" + + now = MTP.stats(session) + assert now.drafts_generated > before.drafts_generated + + assert now.drafts_accepted > before.drafts_accepted, + "no draft accepted: the drafter is not seeing the target's hidden states" + end + + test "greedy output matches plain greedy decode on the target", %{ + target: target, + session: session + } do + # Short, for the near-tie reason MTPSidecarTest gives. + prompt = "The capital of France is" + opts = [max_tokens: 8, temp: 0.0] + + assert {:ok, spec} = MTP.generate(session, prompt, opts) + assert {:ok, plain} = LlamaCppEx.generate(target, prompt, opts ++ [n_ctx: 2048]) + assert spec == plain + end + + test "a model that is not a DFlash drafter is refused, not aborted", %{target: target} do + # Upstream's constructor GGML_ASSERTs on a drafter with no target layers. + assert {:error, message} = MTP.init(target, type: :dflash, draft_model: target, n_ctx: 512) + assert message =~ "not a DFlash/DSpark drafter" + end + + test "the drafter cannot stand in for an MTP head", %{target: target, draft: draft} do + assert {:error, message} = MTP.init(target, draft_model: draft, n_ctx: 512) + assert message =~ "load_mtp: true" + end +end + defmodule LlamaCppEx.MTPE4BSidecarTest do # Gemma4 E4B's target/draft split: the target GGUF is `gemma4` with zero nextn # layers, and a separate `mtp-gemma-4-*.gguf` is a `gemma4-assistant` sidecar @@ -442,8 +531,8 @@ defmodule LlamaCppEx.MTPE4BSidecarTest do assert now.tokens_emitted > before.tokens_emitted end - # E4B is dense, not hybrid — ckpt stays 0. The Qwen sidecar test pins - # ckpt > 0 for the hybrid case. + # E4B is dense, not hybrid — ckpt stays 0. The Qwen sidecar test pins that a + # hybrid target stays off the checkpoint path too. test "timing_us reports the expected buckets", %{target: target, session: session} do {:ok, prompt} = LlamaCppEx.Chat.apply_template(target, [%{role: "user", content: "Count to ten:"}], diff --git a/test/mtp_test.exs b/test/mtp_test.exs index 12aedd9..f7cbda0 100644 --- a/test/mtp_test.exs +++ b/test/mtp_test.exs @@ -30,6 +30,32 @@ defmodule LlamaCppEx.MTPTest do end end + describe "init/2 :type validation" do + @unloaded %LlamaCppEx.Model{ref: nil, load_mtp: true} + + test "a DFlash or DSpark drafter must be passed as :draft_model" do + # Without one there is nothing to build the draft context from; the MTP + # fallback (draft context against the target) would hand upstream's DFlash + # constructor a model with no target layers, which it asserts on. + for type <- [:dflash, :dspark] do + assert {:error, message} = MTP.init(@unloaded, type: type) + assert message =~ "draft_model" + end + end + + test "the drafter must be a Model" do + assert {:error, message} = MTP.init(@unloaded, type: :dflash, draft_model: "/path.gguf") + assert message =~ "must be a LlamaCppEx.Model" + end + + test "an unknown type is refused before any context exists" do + for type <- [:eagle3, "dflash", nil] do + assert {:error, message} = MTP.init(@unloaded, type: type) + assert message =~ ":type must be :mtp, :dflash or :dspark" + end + end + end + describe "init/2 requires a model loaded with load_mtp: true" do # Regression test for a silent break: upstream #26296 made the MTP head's # tensors opt-in at load time via a flag defaulting to false. Nothing on the diff --git a/test/option_forwarding_test.exs b/test/option_forwarding_test.exs index 51e39b9..7389977 100644 --- a/test/option_forwarding_test.exs +++ b/test/option_forwarding_test.exs @@ -235,6 +235,29 @@ defmodule LlamaCppEx.OptionForwardingTest do assert key in Model.tuning_option_keys() end end + + test "load-mode flags resolve to llama.cpp's enum, auto when none is given" do + # Values from llama.h's llama_load_mode. Unset must stay AUTO: upstream + # then skips mmap on an integrated GPU (#26081), and a forced mmap on GB10 + # left most loads decoding ~15% slower. + auto = -1 + none = 0 + mmap = 1 + mlock = 2 + mmap_mlock = 3 + dio = 4 + + assert Model.resolve_load_mode(nil, false, false) == auto + assert Model.resolve_load_mode(true, false, false) == mmap + assert Model.resolve_load_mode(false, false, false) == none + assert Model.resolve_load_mode(nil, true, false) == mmap_mlock + assert Model.resolve_load_mode(true, true, false) == mmap_mlock + assert Model.resolve_load_mode(false, true, false) == mlock + + for use_mmap <- [nil, true, false], use_mlock <- [true, false] do + assert Model.resolve_load_mode(use_mmap, use_mlock, true) == dio + end + end end describe "callers do not keep their own copies" do diff --git a/test/rpc_test.exs b/test/rpc_test.exs index 7e7eba3..8909e6f 100644 --- a/test/rpc_test.exs +++ b/test/rpc_test.exs @@ -75,6 +75,39 @@ defmodule LlamaCppEx.RPCTest do test "ping/1 reports the same refusal" do assert RPC.ping("127.0.0.1:1") == {:error, expected_refusal()} end + + test "a listener that is not an RPC worker is an error, never a crash" do + # Since llama.cpp #18626 upstream's own handshake aborts the process when + # the peer accepts and then answers wrongly, so this is the case the NIF's + # pre-flight HELLO exists for: one peer closes after reading the request, + # the other answers with bytes that are not a HELLO response. + for reply <- [:close, "HTTP/1.1 400 Bad Request\r\n\r\n"] do + {:ok, listen} = :gen_tcp.listen(0, [:binary, active: false, ip: {127, 0, 0, 1}]) + {:ok, port} = :inet.port(listen) + + acceptor = + Task.async(fn -> + {:ok, sock} = :gen_tcp.accept(listen, 10_000) + _ = :gen_tcp.recv(sock, 0, 5_000) + if is_binary(reply), do: :gen_tcp.send(sock, reply) + :gen_tcp.close(sock) + end) + + endpoint = "127.0.0.1:#{port}" + assert RPC.add_server(endpoint) == {:error, expected_refusal()} + + # A non-RPC build never connects, so nobody is accepted; do not wait. + if RPC.supported?(), do: Task.await(acceptor), else: Task.shutdown(acceptor, :brutal_kill) + :gen_tcp.close(listen) + end + end + + test "an endpoint that is not host:port is an error, never a crash" do + # Upstream parses with std::stoi and GGML_ABORTs when that fails. + for endpoint <- ["no-port-here", "127.0.0.1:", "127.0.0.1:http"] do + assert RPC.add_server(endpoint) == {:error, expected_refusal()} + end + end end describe "add_servers/1" do diff --git a/test/server_smoke_test.exs b/test/server_smoke_test.exs index e606d41..0d7beb0 100644 --- a/test/server_smoke_test.exs +++ b/test/server_smoke_test.exs @@ -78,6 +78,40 @@ defmodule LlamaCppEx.ServerSmokeTest do end end + describe "the prefix-instability heuristic" do + # It warns that a chat template is rewriting history, so it must fire on a + # prompt that diverges from the cached history early, and never on a prompt + # that is wholly cached — the same prompt again used to trip it as soon as + # the generated tail made the cached history long enough. + @shared "The capital of France is Paris. The capital of Germany is Berlin. " <> + "The capital of Spain is Madrid. The capital of Portugal is Lisbon. " + + test "is silent when the same prompt comes back" do + server = start_server(n_parallel: 1) + attach_collector(server, [:llama_cpp_ex, :server, :prefix_instability]) + + {:ok, _} = Server.generate(server, @shared <> "The capital of Italy is", max_tokens: 96) + {:ok, _} = Server.generate(server, @shared <> "The capital of Italy is", max_tokens: 8) + + refute_receive {:telemetry, _, _}, 500 + end + + test "fires when the prompt diverges from the cached history early" do + server = start_server(n_parallel: 1) + attach_collector(server, [:llama_cpp_ex, :server, :prefix_instability]) + + {:ok, _} = Server.generate(server, @shared <> "The capital of Italy is", max_tokens: 96) + + {:ok, _} = + Server.generate(server, @shared <> "Name three rivers in each of those countries.", + max_tokens: 8 + ) + + {%{matched_tokens: matched, cached_tokens: cached}, _} = next_telemetry() + assert matched < cached + end + end + test "empty token list errors immediately on both call types" do server = start_server([]) diff --git a/test/support/test_models.exs b/test/support/test_models.exs index 1b8819d..da2d12e 100644 --- a/test/support/test_models.exs +++ b/test/support/test_models.exs @@ -29,6 +29,12 @@ defmodule LlamaCppEx.TestModels do # Its own env var because the two files are provisioned independently. mtp_e4b_draft: {"LLAMA_SMOKE_MTP_E4B_DRAFT_MODEL", "a Gemma4 E4B MTP sidecar (gemma4-assistant)"}, + # A DFlash / DFlash 2 drafter and the target it was trained for, e.g. + # Qwen3.8-27B-Q4_K_M.gguf with z-lab/Qwen3.8-27B-DFlash2-GGUF. The target + # is its own var even though it is usually the :mtp file, because a DFlash + # drafter reads specific target layers and only pairs with its own model. + dflash: {"LLAMA_SMOKE_DFLASH_MODEL", "a DFlash target"}, + dflash_draft: {"LLAMA_SMOKE_DFLASH_DRAFT_MODEL", "a DFlash drafter"}, # Decision models, one per readout: laya reads the embeddings output, # openjev reads label logits. Upstream's own test models # (ggml-org/tinylaya-for-testing-gguf, ggml-org/tinyopenjev-for-testing-gguf) @@ -52,6 +58,8 @@ defmodule LlamaCppEx.TestModels do | :mtp_draft | :mtp_e4b | :mtp_e4b_draft + | :dflash + | :dflash_draft | :decision_laya | :decision_openjev | :decision_clef diff --git a/test/test_helper.exs b/test/test_helper.exs index 90c371a..57b98ed 100644 --- a/test/test_helper.exs +++ b/test/test_helper.exs @@ -26,6 +26,10 @@ # (3 of 4 runs) and did not abort; by b11479 it aborts again # (4 of 6 runs at 23b0202a1). Either way it does not belong in # a green run. See test/mtp_model_test.exs. +# :dflash — DFlash speculative decoding (MTP.init type: :dflash); needs +# LLAMA_SMOKE_DFLASH_MODEL and LLAMA_SMOKE_DFLASH_DRAFT_MODEL, +# a target and the `dflash` drafter trained for it (e.g. +# Qwen3.8-27B-Q4_K_M and z-lab/Qwen3.8-27B-DFlash2-GGUF). # :slow — long-running comparison matrices (F16 vs Q8_0 KV cache); # needs LLAMA_SMOKE_GEN_MODEL # :decision — decision models (LlamaCppEx.Decision); needs both @@ -120,6 +124,7 @@ ExUnit.start( :mtp_sidecar, :decision, :decision_clef, + :dflash, :rpc_live ] )