Repository navigation
DGX Spark CUDA validation: load mode, MTP rollback, DFlash/DSpark, RPC fixes - #122
Merged
Merged
Conversation
… aborts on failure
…nk, GGML_RPC_NO_RDMA; overhead bench
… targets, re-inject features after checkpoint rollback; spec bench
…pelining and transport; README speculative section; changelog
…GPUs, +17% decode on GB10); docs
…load-mode fix, pipelining
…ver RPC splits); Model.devices/1; two-node DFlash e2e
…T); drop b4, which never ran concurrent requests
Two-node decode sits in one of two bands and the client's CPU cluster picks it. gpt-oss-120b over RDMA: llama-bench 44 on the X925 cores or unpinned, 54 on the A725 cores (single-node 59); the binding 37-38 unpinned, 47 pinned, same prefill. Kernel times are identical in both bands on both nodes, so the cost is CPU-side and in the network wait; why the X925 cluster is slower here is open. remote.sh gains --little-cores; the topology notes no longer claim taskset -c 0-9 is all little cores (it is five of each).
…le-node MoE decode within 1%
The 0.001 elementwise bound was tuned on CPU. On CUDA a batch runs different kernels than a single text, and bge-small differs by up to 0.0027 per element (cos 0.99988), so the test failed on every CUDA run with the CI model. Cosine > 0.999 passes on CUDA with bge-small, embeddinggemma and Qwen3-Embedding and on CPU, and still fails on the bugs it guards: rows swapped between sequences (cos 0.41-0.70) and the wrong pooling (<= 0.93).
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Validation of the CUDA build on two DGX Sparks (GB10, CUDA 13.0), and the binding fixes it turned up. llama.cpp is unchanged at
785322d17; nothing here depends on a llama.cpp fork.Fixes and additions
Model.load/2leaves the load mode to llama.cpp (auto) instead of forcinguse_mmap: trueuse_mmap: trueexplicitly to keep the old mode.n_draft: 221.1 → 26.8 tok/s; Qwen3.8 on M1 Max from a net loss to 1.96×MTP.init/2takestype: :dflash/:dsparkwith:draft_modelModel.devices/1;MTP.init/2refuses a DFlash/DSpark drafter that is not on every device the target uses{:error, _}instead of a VM abort when the target is split over RPCRPC.add_serverprobes the endpoint (HELLO) before calling ggml{:error, :unreachable}; on anLLAMA_RPC=1build, 0.8.66 aborts the VMembed_batchtest compares by cosine similarity (> 0.999)scripts/spark/*:e2e.exs,remote.sh --little-cores, bench fixesdocs/dgx-spark.mdhas the measurements: single and two-node benchmarks, MTP/DFlash/DSpark, RDMA against TCP against the 10 GbE LAN, and llama.cpp patches proposed for upstream (labelled as not in the vendored tree).The NIF signatures of
model_loadandspeculative_initchanged, so this needs a version bump and fresh precompiled NIFs; no re-tag.Verification
rpc_test,mtp/mtp_sidecar/dflash,vm_exit_test(exit 0 also with an RPC-split model).scripts/spark/e2e.exs single(three models resident) andtwo-node(layer split over the fabric beside a local model, DFlash refusal and DFlash over the split): 12/12 each./tmpproject on spark-2.