Skip to content

DGX Spark CUDA validation: load mode, MTP rollback, DFlash/DSpark, RPC fixes - #122

Merged
nyo16 merged 49 commits into
masterfrom
spark-cuda-validation
Nov 2, 2026
Merged

nyo16 merged 49 commits into
masterfrom
spark-cuda-validation

Conversation

@nyo16

@nyo16 nyo16 commented Nov 2, 2026

Copy link
Copy Markdown
Owner

Validation of the CUDA build on two DGX Sparks (GB10, CUDA 13.0), and the binding fixes it turned up. llama.cpp is unchanged at 785322d17; nothing here depends on a llama.cpp fork.

Fixes and additions

Change Effect
Model.load/2 leaves the load mode to llama.cpp (auto) instead of forcing use_mmap: true gpt-oss-120b decode on GB10 up to +17%; forced mmap left most loads in a ~17% slower band. Behaviour change: pass use_mmap: true explicitly to keep the old mode.
MTP on hybrid targets (Qwen 3.6/3.8) gets one recurrent snapshot per draft position and rolls back natively Qwen3.8-27B NVFP4 n_draft: 2 21.1 → 26.8 tok/s; Qwen3.8 on M1 Max from a net loss to 1.96×
MTP.init/2 takes type: :dflash / :dspark with :draft_model DFlash 2 on Qwen3.8-27B NVFP4: 44.2 tok/s, 3.1× plain
Model.devices/1; MTP.init/2 refuses a DFlash/DSpark drafter that is not on every device the target uses {:error, _} instead of a VM abort when the target is split over RPC
RPC.add_server probes the endpoint (HELLO) before calling ggml An unreachable worker returns {:error, :unreachable}; on an LLAMA_RPC=1 build, 0.8.66 aborts the VM
Server prefix-instability false positive fixed smoke tests added
embed_batch test compares by cosine similarity (> 0.999) It failed on every CUDA run with bge-small (0.0027 per element against a 0.001 bound tuned on CPU); swapped rows still fail it
scripts/spark/*: e2e.exs, remote.sh --little-cores, bench fixes Two-node client on the A725 cores: +25% decode over RPC

docs/dgx-spark.md has the measurements: single and two-node benchmarks, MTP/DFlash/DSpark, RDMA against TCP against the 10 GbE LAN, and llama.cpp patches proposed for upstream (labelled as not in the vendored tree).

The NIF signatures of model_load and speculative_init changed, so this needs a version bump and fresh precompiled NIFs; no re-tag.

Verification

  • Mac (Metal): default suite 442 passed.
  • spark-1 (CUDA + RPC): full tagged suite, rpc_test, mtp / mtp_sidecar / dflash, vm_exit_test (exit 0 also with an RPC-split model).
  • scripts/spark/e2e.exs single (three models resident) and two-node (layer split over the fabric beside a local model, DFlash refusal and DFlash over the split): 12/12 each.
  • Embeddings test: passes on CUDA with bge-small, embeddinggemma and Qwen3-Embedding, and on CPU.
  • Published 0.8.66 smoke-built from a fresh /tmp project on spark-2.

nyo16 added 30 commits November 1, 2026 21:21
… targets, re-inject features after checkpoint rollback; spec bench
…pelining and transport; README speculative section; changelog
…ver RPC splits); Model.devices/1; two-node DFlash e2e
…T); drop b4, which never ran concurrent requests
Two-node decode sits in one of two bands and the client's CPU cluster picks
it. gpt-oss-120b over RDMA: llama-bench 44 on the X925 cores or unpinned,
54 on the A725 cores (single-node 59); the binding 37-38 unpinned, 47
pinned, same prefill. Kernel times are identical in both bands on both
nodes, so the cost is CPU-side and in the network wait; why the X925
cluster is slower here is open.

remote.sh gains --little-cores; the topology notes no longer claim
taskset -c 0-9 is all little cores (it is five of each).
nyo16 added 19 commits November 2, 2026 14:24
The 0.001 elementwise bound was tuned on CPU. On CUDA a batch runs
different kernels than a single text, and bge-small differs by up to
0.0027 per element (cos 0.99988), so the test failed on every CUDA run
with the CI model. Cosine > 0.999 passes on CUDA with bge-small,
embeddinggemma and Qwen3-Embedding and on CPU, and still fails on the bugs
it guards: rows swapped between sequences (cos 0.41-0.70) and the wrong
pooling (<= 0.93).
@nyo16
nyo16 merged commit 350a7a5 into master Nov 2, 2026
9 checks passed
@nyo16 nyo16 mentioned this pull request Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant