Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
568af25
RPC: probe the endpoint before registering, upstream registration now…
nyo16 Nov 2, 2026
1871c2f
RPC: tests for non-RPC peers and bad endpoints; spark scripts: LAN li…
nyo16 Nov 2, 2026
0981568
spark: end-to-end check script, single node with three models and two…
nyo16 Nov 2, 2026
499b492
MTP: DFlash/DSpark drafters via type:, RS-snapshot rollback on hybrid…
nyo16 Nov 2, 2026
24036b7
docs: GB10 speculative decoding, NVFP4, binding overhead, two-node pi…
nyo16 Nov 2, 2026
615a054
Model.load: default to upstream's auto load mode (no forced mmap on i…
nyo16 Nov 2, 2026
f6220fb
Server: prefix-instability warning only when the prompt diverges from…
nyo16 Nov 2, 2026
9fbd9f5
docs: single-Spark table, two-node control and transport rerun after …
nyo16 Nov 2, 2026
77db4a6
README: GB10 benchmark table, MTP options
nyo16 Nov 2, 2026
a3b7421
docs: two-node worker binary A/B
nyo16 Nov 2, 2026
80c0691
Solution docs: GB10 forced mmap, RPC registration abort; CLAUDE.md rules
nyo16 Nov 2, 2026
95bf175
MTP: refuse a DFlash drafter missing the target's devices (VM abort o…
nyo16 Nov 2, 2026
ea73213
docs: speculative decoding across two nodes
nyo16 Nov 2, 2026
9ff1edc
bench: two-node concurrency through spark_overhead (SPARK_RPC_ENDPOIN…
nyo16 Nov 2, 2026
e8abbad
docs: upstream fix for DFlash drafter on fewer devices; GRAPH_OPT row
nyo16 Nov 2, 2026
7802e84
docs: MMQ src1 reuse upstream candidate
nyo16 Nov 2, 2026
a9f78c1
docs: NVFP4 MMQ weight-scale fold upstream candidate
nyo16 Nov 2, 2026
12e6a17
docs: NVFP4 SWIGLU-in-MMQ fusion, +19-20% prefill
nyo16 Nov 2, 2026
fba36f8
docs: gpt-oss expert-bias fusion results
nyo16 Nov 2, 2026
125ad5d
docs: binding prefill with the spark-all llama.cpp branch
nyo16 Nov 2, 2026
9b208df
docs: pin a two-node client to the A725 cores (+25% RPC decode)
nyo16 Nov 2, 2026
760bcef
docs: RPC client wake-ups explain the two-node decode bands; fix branch
nyo16 Nov 2, 2026
6bf8d10
docs: worker binary gap once the RPC client is fixed
nyo16 Nov 2, 2026
925dfa7
docs: spec decoding on the spark-all build
nyo16 Nov 2, 2026
7a70232
solutions: two-node decode bands come from RPC client wake-ups
nyo16 Nov 2, 2026
59c6220
docs: with the RPC client fixed, dense two-node decode beats one node…
nyo16 Nov 2, 2026
7532fe7
docs: two-node speed summary reflects the RPC client fix
nyo16 Nov 2, 2026
9eb0f7c
docs: 235B two-node decode with the RPC client fix
nyo16 Nov 2, 2026
a269871
docs: index of the llama.cpp branches
nyo16 Nov 2, 2026
f540d2d
docs: two-node concurrency with the RPC client fix
nyo16 Nov 2, 2026
22c4d44
docs: two-node speculative decoding with the RPC client fix
nyo16 Nov 2, 2026
ffcecb1
docs: -sm tensor across two nodes with the RPC client fix
nyo16 Nov 2, 2026
60627e4
docs: test coverage of the spark-all build
nyo16 Nov 2, 2026
f88bb23
docs: note the bge-small embed_batch tolerance miss on CUDA
nyo16 Nov 2, 2026
8fccfb5
docs: bge-small CUDA gap is backend numerics, not batching
nyo16 Nov 2, 2026
d1164d5
docs: MoE expert matvec runs ~4% slower inside the BEAM
nyo16 Nov 2, 2026
81554a3
docs: correct the MoE kernel-gap claims (nsys artifact); binding sing…
nyo16 Nov 2, 2026
3d84f14
docs: the CUDA branch is neutral on Q4_K MoE models
nyo16 Nov 2, 2026
8c6ca4b
docs: two-node e2e on the spark-all build
nyo16 Nov 2, 2026
a4f064b
docs: the RPC spin leaves an idle client idle
nyo16 Nov 2, 2026
8deb8fb
docs: CPU cost of the RPC client spin while generating
nyo16 Nov 2, 2026
5d91acb
docs: both RPC spins are needed
nyo16 Nov 2, 2026
19755d2
docs: state exactly which nsys comparisons were made
nyo16 Nov 2, 2026
ec79013
docs: count unpinned runs instead of guessing the scheduler
nyo16 Nov 2, 2026
1cc5158
docs: soak result, unpinned run count, two-node verdict with the fix
nyo16 Nov 2, 2026
4ba4265
docs: binding two-node soak with the RPC client fix
nyo16 Nov 2, 2026
51c640c
docs: two-node MTP with the RPC client fix
nyo16 Nov 2, 2026
3877c56
docs: more ruled-out causes for the binding worker gap
nyo16 Nov 2, 2026
0eec112
test: compare embed_batch with per-text embed by cosine similarity
nyo16 Nov 2, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
155 changes: 155 additions & 0 deletions .claude/plans/release-v0-8-64/plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
# Plan: Release v0.8.64 (no llama.cpp bump)

**Depth**: quick · **Status**: completed · **Date**: 2026-10-31 → 2026-11-01

## Goal

Publish v0.8.64 to Hex with precompiled NIFs for all 8 targets. It ships the
changes merged after v0.8.63 (CHANGELOG "Unreleased"), with llama.cpp staying
at `418f905a4`:

- `LlamaCppEx.set_log_level/1`, `config :llama_cpp_ex, log_level:`, DEBUG
filtered by default (#114)
- `:tools` tag in CI against pinned Qwen3.5-0.8B (#114)
- `elixir_make ~> 0.10.1` (#114); req `~> 0.7` allowed, dep lock updates (#113)
- lock: decimal 3.2.0 / ecto 3.14.2 (#114)

Done means: release not a draft, 8 assets, `checksum.exs` on master, Hex
0.8.64, and the `/tmp` end-to-end smoke passing against the published
package with the `~/Downloads` models.

## Decisions

- **No llama.cpp bump** (user). There are 22 upstream commits on master;
they belong in a separate bump release. So: no header diff, no
decision-port diff, no tokenizer-partition re-diff, no
`decision_compare.exs` rerun. No decision code changed since v0.8.63, which
was compared ALL MATCH.
- **Merge #115 first** (user). It is docs only and `.claude/` and `CLAUDE.md`
are not in the Hex `files` list, so the package is unaffected.
- **Release via PR then tag**, as for v0.8.60–v0.8.63; the version bump and
changelog go through CI before the tag.

## Phase 1 — Prepare master

- [x] [git] Merge #115 (squash, delete branch); `git pull` master — first
Inference smoke failed on a pre-existing test race (next item); re-run
green; merged as 3b63b52
- [x] [test] Fix flaky `ServerSmokeTest` "halting a stream early …":
`attach_collector/1` was a global handler; it caught the previous
test's lingering server's `:done` (39 prompt tokens, `max_tokens`),
which `finish_slot/3` emits after replying. Collectors in
`server_smoke_test.exs` and the telemetry test in `llama_cpp_ex_test.exs`
now pin `metadata.server`; test-only, no CHANGELOG entry
- [x] [git] Branch `release-v0.8.64` from master (3b63b52)
- [x] [config] `mix.exs` `@version "0.8.63"` → `"0.8.64"`
- [x] [docs] CHANGELOG: rename `## Unreleased` → `## v0.8.64`, add an
intro line ("llama.cpp stays at `418f905a4`"), keep the existing
Added/Changed entries; #115 is docs only and gets no entry
- [x] [verify] `mix compile --warnings-as-errors`, `mix format --check-formatted`,
`mix credo --strict`, `mix dialyzer` (0 errors) — all clean

## Phase 2 — Local test matrix (Metal, `GGML_METAL_NO_RESIDENCY=1`)

Unset `LLAMA_BACKEND` only where a precompiled path is under test; here
everything is a local build (`LLAMA_BACKEND=metal`).

- [x] [test] default: `mix test` — 436 passed, 194 excluded
- [x] [test] CI's set: `--include smoke --include embeddings --include slow
--include decision --include tools` with SmolLM2, bge-small, tinylaya,
tinyopenjev, Qwen3.5-0.8B — 601 passed, 29 excluded
- [x] [test] release-guide set: `--include smoke --include embeddings
--include slow --include mtp` with Qwen3.5-0.8B, Qwen3-Embedding-0.6B-f16,
Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL — 585 passed, 45 excluded
- [x] [test] `--include embeddings` with embeddinggemma-2-Q8_0 — 447 passed
- [x] [test] `--include decision_clef` with Cloudflare_clef-flash-Q8_0 — 440
passed (4/4 clef)
- [x] [test] `--include tools` with Llama-3.2-1B-Instruct (expect 10/11:
the round trip calls the tool again; documented, not a regression) —
446/447, the round-trip test as expected; Qwen3.5 on Metal 447
- [x] [test] `--include mtp_sidecar` with Qwen3.8-27B-Q4_K_M +
mtp-Qwen3.8-27B-Q4_0 (E4B pair not on disk, so that module skips) —
442 passed, 6 skipped
- [x] [test] LLAMA_BACKEND=cpu `--include tools` (what CI runs) — 447 passed
- [x] [verify] Hex source build: `mix hex.build`, unpack into a temp dir,
`LLAMA_BACKEND=cpu mix compile`, confirm `vendor/llama.cpp` HEAD ==
`418f905a4` and `input_tokenize.{h,cpp}` are in the tarball — 332K
package, build fetched `418f905a4`, NIF built
- [x] [docs] CHANGELOG v0.8.64: add the "Verified on …" paragraph with the
exact counts from this phase (format as in v0.8.63)

## Phase 3 — PR and CI

- [x] [git] Commit "Release v0.8.64", push, open PR — #117 (2 commits:
e081cbf test fix, 008f7bf release)
- [x] [ci] Watch CI: Setup, Tests, Credo, Dialyzer, Formatting, Inference
smoke (now including `:tools`), CUDA link cu12/cu13 — all green
(run 41250391316)
- [x] [ci] Confirm in the Inference smoke log: `Including tags: [... :tools]`
and 0 failures; no `xref: [exclude` annotation — 630 tests, 0 failures;
no xref annotations
- [x] [git] Merge PR (ask the user before merging), pull master — user chose
squash; 4f47d41

## Phase 4 — Tag and publish

- [x] [release] `git tag v0.8.64` on the merge commit; `git push origin v0.8.64`
- [x] [release] Watch `Precompile NIFs`: 2× macos-15 Metal (log shows
`minos: 14.0`), 2× CPU, 4× CUDA (cu12/cu13 legs take ~50–110 min),
then "Checksum and publish the release" and "Publish to Hex" — run
41255669230 all green in 85 min; both Metal legs `minos: 14.0`
- [x] [release] Verify: `gh release view v0.8.64` → not draft, 8 assets;
master has "Update checksum.exs for v0.8.64"; `mix hex.info llama_cpp_ex`
lists 0.8.64 — draft=false, 8 assets, 1041099, `~> 0.8.64`
- [x] [git] `git pull` master (picks up the checksum commit)

## Phase 5 — End-to-end smoke from Hex (`/tmp`, `~/Downloads`)

- [x] [e2e] Fresh project: `rm -rf /tmp/llama_e2e_0864 && mix new`, deps
`{:llama_cpp_ex, "== 0.8.64"}`, `{:req, "~> 0.7"}`
- [x] [e2e] `env -u LLAMA_BACKEND mix compile` → "Downloading precompiled NIF"
for `nif-2.18-aarch64-apple-darwin-0.8.64`, checksum passes, `otool`
minos 14.0; `mix.lock` resolves elixir_make 0.10.1 — all as expected
(httpc `max_header_size` notice, known)
- [x] [e2e] Copy `/tmp/llama_e2e/e2e.exs`; add log-level checks:
`set_log_level(:none)` before `init/0` → a load + tool call writes 0
stderr lines; default level → the tool call writes no "awaiting trigger"
line; invalid level raises `ArgumentError` — child VM via
`loglevel.exs`, stderr to file
- [x] [e2e] Run with `GGML_METAL_NO_RESIDENCY=1`: all checks PASS (generate,
stream, chat, json_schema, #108 ×3 incl. server and Llama 3.2, #110 ×2,
#111 ×6, embeddings, #109 ×3 incl. Clef noul > 0.5, log level ×3) —
23 passed, 0 failed; clef noul 0.98; :none 0 stderr lines, default 198
- [x] [e2e] Optional: repeat compile on a fresh project with
`LLAMA_BACKEND=cpu` to confirm the Hex source build from the published
tarball — built at `418f905`, CPU generate " Paris."

## Verification summary

| Gate | Command / evidence |
|---|---|
| Compile/lint | `mix compile --warnings-as-errors`, format, credo --strict, dialyzer |
| Local tests | Phase 2 matrix, counts recorded in CHANGELOG |
| CI | PR checks all green, Inference smoke ran `:tools` |
| Release | workflow green, release published with 8 assets, Hex 0.8.64 |
| Published package | `/tmp` e2e: precompiled download + all checks PASS |

## Iron Law check

No application code changes in this plan (version, changelog only). Iron Law
#27 (no special-parsed tokenization of rendered prompts) is unaffected; the
e2e #108/#109 checks re-verify it on the published artifact.

## Risks

- **CUDA legs are slow** (47–110 min) and can hit runner capacity; the
release stays a draft until all 8 legs pass, so nothing partial reaches Hex.
- **macos-15 arm64 capacity**: the workflow annotated queueing delays for
v0.8.63, so expect waits.
- **Flaky small-model tool tests on CI CPU**: Qwen3.5-0.8B passed 11/11 on CPU
locally and in CI once. If CI flakes, investigate before re-running;
don't loosen assertions to get green.
- **Local Hex auth expired** (`mix hex.user auth`): this only matters for a
manual publish, since the workflow publishes with its own key.
- **Tag hygiene**: move a tag only while the release is still a draft and
Hex has nothing (see `.claude/solutions/deployment-issues/macos-14-runner-brownout-precompile-20261031.md`).
34 changes: 34 additions & 0 deletions .claude/plans/release-v0-8-64/scratchpad.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Scratchpad: release v0.8.64

## DECISION

- No llama.cpp bump (user, 2026-10-31). Upstream master was 22 commits
ahead of `418f905a4` (head `ddf48f85c`); do that as its own release with
the full release-guide diff (header, decision port, tokenizer partition).
- Merge #115 (docs only) before tagging (user).
- No research agents spawned: the process is fixed by docs/release-guide.md
and was exercised end to end for v0.8.63 the same day.
- Skip decision_compare.exs: no decision or tokenizer code changed since
v0.8.63, which was ALL MATCH on 9 models.

## Known, not regressions

- Llama-3.2-1B-Instruct `:tools` round trip: calls the tool again (10/11).
- OTP 29.0.3 locally: `Invalid option {max_header_size,65536} ignored` during
the NIF download is expected (httpc predates the option).
- "Found packages with security advisories" should be gone after #114
(decimal 3.2.0); if it reappears, check `mix deps.get` output.

## DEAD-END

(none yet)

## Flake found on #115 CI (2026-10-31)

`ServerSmokeTest` "halting a stream early cancels generation" got another
server's `:done` (prompt 39, `max_tokens`) — the preceding chat_completion
test's server, still alive after its test exited (ExUnit doesn't wait for
linked processes), emitting telemetry after `GenServer.reply`. Fixed by
pinning `metadata.server` in the collectors. Not reproducible locally on Metal
with seed 489492 (timing); evidence is the CI log ordering (handler attach at
54.311, stale done logged 54.314).
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
module: "LlamaCppEx.Model (lib/llama_cpp_ex/model.ex), model_load NIF"
date: "2026-11-02"
problem_type: performance
component: model_loading
symptoms:
- "gpt-oss-120b decodes at ~48.5 tok/s through the binding on DGX Spark (GB10) while llama-bench and llama-server on the same build do ~57-59"
- "the gap is per process: about one binding load in five decodes fast (57), and repeated runs inside one process never change state"
- "nsys shows the same kernels (mul_mat_vec_q MXFP4/Q8_0) running ~22% longer in the slow processes"
- "n_ctx, n_threads, sampler chain, BEAM busy-wait, ASLR, page-cache eviction: no effect"
root_cause: "Model.load/2 defaulted :use_mmap to true, which the NIF mapped to an explicit LLAMA_LOAD_MODE_MMAP. Upstream's default is LLAMA_LOAD_MODE_AUTO (#26081), which reads the file instead of mapping it when a device reports caps.mmap_support == false — ggml-cuda reports that for an integrated GPU, and GB10 is one. Forcing mmap overrode that and made the slow placement the common case."
severity: high
tags: [cuda, gb10, dgx-spark, igpu, mmap, load-mode, performance, benchmark-noise]
related_solutions: []
---

# Forced mmap on GB10 costs ~15% decode (load mode)

## Symptoms

Binding vs upstream on one Spark, gpt-oss-120b MXFP4, 128 greedy tokens:
`LlamaCppEx.stream/3` 48.5 tok/s, `llama-bench` tg128 58.9, `llama-server`
56.9. Qwen3-8B showed no gap, which hid it at first.

## Investigation

1. Binding overhead ruled out first: on Qwen3-8B the binding matched
`llama-bench` and `llama-server` within 1-2%, single stream and 8-way.
2. Per-process bimodality: the same script gave 48.5 in most processes and
57 in a few; inside one process every context agreed.
3. `nsys profile -t cuda --cuda-graph-trace=node` on both: identical kernel
list and launch shapes, but the memory-bound matvecs ~22% slower in the
slow processes. So GPU-side, not CPU, sampling or messaging.
4. Ruled out: `n_threads`, sampler chain, `n_ctx`/`swa_full`, BEAM `+sbwt none`,
`setarch -R`, THP (weights are not in RSS), page-cache eviction via
`posix_fadvise(DONTNEED)`.
5. `use_mmap: false` and `use_direct_io: true`: 6/6 processes fast. Upstream's
`llama_model_default_params().load_mode` is `AUTO`; `src/llama-model.cpp`
turns mmap off under AUTO when a device lacks `mmap_support`, and
`ggml-cuda.cu` sets that false for `GGML_BACKEND_DEVICE_TYPE_IGPU`.

## Fix

`Model.load/2` resolves the three flags itself (`resolve_load_mode/3`) and
passes the enum to the NIF; with none of them given the result is
`LLAMA_LOAD_MODE_AUTO`. Explicit flags resolve exactly as before. Metal, CPU
and discrete CUDA report `mmap_support`, so AUTO maps there as the old
default did. After: 5/5 processes at 57.0-57.2 tok/s; forced mmap still
bimodal (48.5 / 57).

## Lessons

- Never override an upstream default with a binding default that looks
equivalent — `use_mmap: true` was "the default" only until upstream grew a
smarter one.
- On GB10, a single decode measurement is one sample of a bimodal process.
The ~17% dip also appeared occasionally in `llama-bench` itself and on the
RPC worker per client session, independent of mmap; re-run before
believing a number.
- Test: `test/option_forwarding_test.exs` "load-mode flags resolve to
llama.cpp's enum, auto when none is given".
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
---
module: "LlamaCppEx.RPC / rpc_add_server NIF (c_src/llama_cpp_ex/llama_nif.cpp)"
date: "2026-11-02"
problem_type: runtime_error
component: rpc
symptoms:
- "ggml-rpc.cpp:653: Failed to connect to 127.0.0.1:1, then SIGABRT (exit 134) from mix test on an LLAMA_RPC=1 build"
- "RPC.add_server/1, RPC.ping/1 or Model.load(path, rpc_servers: [...]) with a dead endpoint kills the VM instead of returning {:error, :unreachable}"
- "default (non-RPC) builds and macOS CI never show it: they answer :rpc_unsupported without touching ggml-rpc"
root_cause: "llama.cpp #18626 (d0132a680, 'rpc : implement event and async backend APIs') moved the connect and HELLO into rpc_dispatcher::start, which GGML_ABORTs on a parse failure, connect failure, HELLO failure or version mismatch. ggml_backend_rpc_add_server used to return nullptr for all of them. negotiate_hello also RPC_STATUS_ASSERTs when a non-RPC peer answers."
severity: high
tags: [rpc, ggml-rpc, abort, nif, upstream-regression, dgx-spark]
related_solutions: []
---

# RPC registration aborts the VM since llama.cpp #18626

## Symptoms

First `mix test` on a CUDA + `LLAMA_RPC=1` build on a DGX Spark aborted
within seconds at `rpc_test.exs`'s `RPC.add_server("127.0.0.1:1")`. The
published 0.8.66 does the same when built with `LLAMA_RPC=1`
(`mix deps.compile llama_cpp_ex --force` with `LLAMA_RPC=1 LLAMA_BACKEND=cuda`,
then `RPC.add_server("127.0.0.1:1")` → exit 134).

## Investigation

`git log -S "Failed to connect to %s" -- ggml/src/ggml-rpc/ggml-rpc.cpp`
pointed at d0132a680; its diff replaces every `return nullptr` in the old
`get_socket` with `GGML_ABORT` in `rpc_dispatcher::start`. CI never builds
RPC, so the bump that brought it in (v0.8.5x) passed.

## Fix

- NIF: `rpc_probe_hello/2` connects (IPv4, first-':' parse, `std::stoi`,
exactly as upstream) with a 5 s timeout and sends a TCP-only HELLO (zeroed
conn caps, so the worker never starts an RDMA upgrade for the probe), then
checks major/minor like `negotiate_hello`. Anything short of that is
`{:error, :unreachable}` before `ggml_backend_rpc_add_server` runs. The wire
layout is pinned by `static_assert(RPC_PROTO_MAJOR_VERSION == 8)`.
- Upstream: branch `rpc-add-server-no-abort` on `nyo16/llama.cpp` makes
`start()` return false, adds `try_get_dispatcher()` for registration and
keeps aborting only on established connections.
- Tests: `test/rpc_test.exs` — a listener that closes, one that answers HTTP,
and unparseable endpoints; each aborted the VM with the probe disabled.

## Lessons

- An RPC build is a different artifact; run `mix test` on one after every
llama.cpp bump that touches `ggml-rpc`, not only `--include rpc_live`.
- `rsync -a` preserves source mtimes: after editing a file on a node, a sync
that restores an older-mtime copy leaves make's `.o` newer than its source.
`touch` the source before rebuilding.
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
---
module: "ggml-rpc client (vendor/llama.cpp/ggml/src/ggml-rpc/ggml-rpc.cpp), two-node layer split"
date: "2026-11-02"
problem_type: performance
component: rpc
symptoms:
- "Two-node (RPC layer split) decode sits in one of two bands per client process: gpt-oss-120b ~44 or ~53 tok/s via llama-bench, ~38 or ~47 via the binding"
- "The same worker serves consecutive clients at different speeds; restarting the worker does not settle it"
- "Kernel times (nsys) are identical in both bands on both nodes; the extra ~4 ms per token is CPU-side"
root_cause: "Every RPC command is two thread handoffs: the caller blocks on a std::future while the dispatcher thread, woken from its queue, does the network round trip and wakes the caller. The dispatcher only busy-spins while a -sm tensor communicator holds it (busy_spin_acquire), so for a layer split each handoff is a futex wake of an idle core. On GB10's Cortex-X925 cores those wakes are slow enough to cost ~4 ms per token; on the A725 cores they are not (why is not established)."
severity: medium
tags: [rpc, ggml-rpc, dgx-spark, gb10, two-node, decode, cpu-affinity, latency]
related_solutions: [rpc-add-server-aborts-vm-dispatcher-20261102]
---

# Two-node decode bands: RPC client thread wake-ups

## Symptoms

`llama-bench --rpc ... -dev CUDA0/RPC0 -n 128` on two DGX Sparks gave 44 or 53
tok/s for gpt-oss-120b on different runs (single node 59). The docs had
blamed the worker for the spread.

## Investigation

- `nsys` on the worker in a fast and a slow session: same kernel times (118
µs per MXFP4 expert matvec, 2.7 ms LM head), same per-token
`cudaStreamSynchronize` total. Same on the client. So not GPU, not clocks.
- `taskset` matrix (worker cores x client cores, upstream worker restarted
per run): the client on the A725 cores (0-4,10-14) was always fast; on the
X925 cores (5-9,15-19) slow; unpinned mostly slow. `--poll 0`, `-t 1`,
BEAM `+sbwt none` changed nothing.
- An env-gated hack spinning the dispatcher queue and the callers' future
waits gave 57.6 on any cores: the wake-ups were the cost.

## Fix

- Upstream patch, `nyo16/llama.cpp` branch `rpc-client-spin-wait` (merged
into `spark-all`): spin the dispatcher whenever a command arrived in the last
`RPC_BUSY_SPIN_IDLE_TIME` (100 ms), and spin callers on their future for up
to that long before blocking; `busy_spin_acquire/release` removed. 57.3-57.6
tok/s on X925, A725 or unpinned; prefill unchanged.
- Without the patch (vendored llama.cpp): run the client on the A725 cores,
`scripts/spark/remote.sh --little-cores` (+25% two-node decode through the
binding, 37-38 -> 47 tok/s).

## Lessons

- A latency microbenchmark (1-byte RDMA ping-pong favours the X925) did not
predict the end-to-end result; measure the decode loop itself.
- A bimodal per-process number with identical kernel times is a CPU-side
handoff, not the GPU. Profile both nodes before blaming either.
- `pkill -f <pattern>` over ssh matches the ssh shell's own command line and
kills it (exit 255); use `pkill -x` or a pid.
Loading