Repository navigation
ft bench bw picks hybrid on Ampere, but offload is 8.3x faster (DeepSeek-V4-Flash, 2x RTX 3090) #151
Description
ft bench bw picks hybrid on Ampere, but offload is 8.3x faster (DeepSeek-V4-Flash, 2x RTX 3090)
Summary
On a 2x RTX 3090 (SM86) host, ft bench bw measures the CPU MoE kernel at 56.2 GB/s vs 12.3 GB/s for PCIe, reports a 4.57x ratio, and selects hybrid. In actual serving, hybrid delivers 0.67 tok/s while forcing --moe-backend offload on the same machine and model delivers 5.58 tok/s.
The auto-selected backend is 8.3x slower than the one it rejected.
The calibrated CPU number does not survive contact with serving: at 0.67 tok/s and ~3.07 GB of expert weights per token, the effective CPU-MoE rate is roughly 2 GB/s, about 28x below what ft bench bw measured for the same kernel, dtype and workload minutes earlier.
Environment
| FreeToken | 0.1.2 (PyPI, freetoken[accel]) |
| Model | deepseek-ai/DeepSeek-V4-Flash-0731 (fp8 dense, expert_dtype: fp4) |
| GPU | 2x RTX 3090 24GB, SM86, driver 595.84 |
| CPU | Intel Xeon Gold 6252 @ 2.10GHz, avx512f, 46 cores available to the container |
| RAM | 503 GB |
| CUDA / torch | 13.0 / 2.11.0+cu130 |
| OS | Linux x86_64 (container) |
Reproduction
ft bench bw --model dsv4 -o bwprofile.json
ft serve --model-path /path/to/DeepSeek-V4-Flash-0731 \
--tp-size 2 --moe-backend hybrid --moe-cache-auto \
--host 0.0.0.0 --port 1919
# -> 0.67 tok/s
ft serve --model-path /path/to/DeepSeek-V4-Flash-0731 \
--tp-size 2 --moe-backend offload --moe-cache-auto \
--host 0.0.0.0 --port 1919
# -> 5.58 tok/sDecode measured over a streamed 60-token completion, first token excluded, after a warmup request.
ft bench bw output
host 82d6485125bb gpu cuda:0 (NVIDIA GeForce RTX 3090) cpu 24c/24t
ceilings: CPU STREAM read 62.3 | PCIe linear H2D 12.3 D2H 13.2 GB/s (threshold 2.0x)
dsv4 H=4096 I=2048 E=256 top_k=6
format expert CPU-MoE PCIe-gather CPU/PCIe backend
ds_fp4 12.75 MB 56.2 GB/s 12.3 GB/s 4.57x hybrid
overlapped: CPU-MoE 44.8 + PCIe 11.3 GB/s -> hybrid fetches 20.1% of misses
Measured results
| backend | CPU threads | decode | TTFT (short prompt) |
|---|---|---|---|
hybrid (auto-selected) |
23 | 0.58 tok/s | 5.8 s |
hybrid |
44 | 0.67 tok/s | 3.5 s |
offload |
n/a | 5.58 tok/s | 2.5 s |
offload, --memory-ratio 0.95 |
n/a | 5.95 tok/s | 2.4 s |
The last row is included to show the ceiling is not cache-related: raising memory-ratio from the 0.9 default grew the expert pool from 624 to 779 slots (5.7% -> 7.1% residency) for only +6.6% throughput. Expert-cache residency cannot be pushed much further on this host — the ~24 GB of non-expert weights (167 GB total minus 143 GB of experts) already occupy roughly half of the 48 GB of VRAM, leaving ~10-12 GB for all pools combined.
Doubling the CPU pool from 23 to 44 threads yields only +15%, which is itself evidence that the CPU path is not thread-bound but limited by something else — most likely per-layer synchronisation. With top_k=6 over 43 MoE layers there are 258 expert activations per token, each a separate small GEMV with its own sync point, whereas the benchmark measures a tight loop over one expert.
Note on TP=2 and #62
All runs use --tp-size 2, and #62 documents that OffloadMoeCache is TP-unaware, making TP=2 a regression for offloaded MoE. That applies here too, so the absolute numbers below are pessimistic.
TP=1 is not available as a control on this hardware: the non-expert weights are ~25 GB in total (~12.7 GB per rank at TP=2, confirmed by VRAM accounting), which does not fit a single 24 GB card alongside any cache. Anyone reproducing on >=32 GB cards can and should check TP=1.
The hybrid vs offload comparison is unaffected: both ran at TP=2 with identical flags, so the 8.3x gap is attributable to the backend alone. If anything, #62 means both figures would rise together at TP=1.
Config was otherwise identical across runs (--moe-cache-auto, cache resolved to 624 slots / 5.7% residency in every case), so the difference is attributable to the backend alone.
Why this matters
The wrong call is made silently. --moe-backend auto resolves to hybrid here and produces the same 0.67 tok/s; nothing in the logs or in ft ctl stats points at the backend as the cause. Without a reason to try offload by hand, the reasonable conclusion is that the engine is unusable on this hardware, when in fact it is 8.3x faster one flag away.
Suggestions
- Validate the calibration against a short end-to-end decode. After picking a backend, run a few tokens through it and compare achieved throughput to the predicted rate; if the gap exceeds some factor, fall back and warn.
- Make the benchmark representative of serving. Measure the CPU path across a full layer sweep with the real synchronisation pattern (258 activations/token), not a single-expert loop. A per-activation fixed cost would then show up in the model.
- Expose runtime cache/backend effectiveness.
ft ctl statsreportsdecode_tpsbut not expert cache hit rate or the realised CPU-vs-PCIe split. Those would make this diagnosable from the outside, and would also cover the Desktop-UI side of [Feature Request]: Add real-time expert cache hit rate and PCIe transfer telemetry to Desktop UI and CLI #76. - Consider raising the selection threshold for
hybridon architectures without native FP4, or excluding it there until the CPU-side cost model accounts for per-activation overhead.
Secondary issues found in the same session
-lnccl link failure with --tp-size > 1. The JIT build of freetoken__pynccl.so fails on a clean uv pip install "freetoken[accel]":
/usr/bin/ld: cannot find -lnccl: No such file or directory
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
RuntimeError: ninja exited with status 1
ERROR Backend worker is gone and cannot be restarted; stopping the API server
torch ships nvidia/nccl/lib/libnccl.so.2 without the libnccl.so devel symlink that -lnccl requires. Workaround:
NCCLDIR=$VIRTUAL_ENV/lib/python3.12/site-packages/nvidia/nccl/lib
ln -sf libnccl.so.2 $NCCLDIR/libnccl.so
export LIBRARY_PATH="$NCCLDIR:$LIBRARY_PATH"The build could resolve the NCCL path from torch.utils.cpp_extension / the installed nvidia-nccl-* package instead of relying on a system -lnccl.
--moe-cache-size accepts a value that makes startup fail. ft ctl cache reported the moe pool as resizable to 256..828, but --moe-cache-size 828 aborts the backend:
ValueError: DSV4 KV budget -284406580 bytes cannot fit the minimal pool
(31 pages incl. the window working-set floor 31); raise memory_ratio or
lower max_running_req/max_seq_len
The advertised upper bound does not account for the KV/SWA floor, so the value the tool itself reports as the maximum is not actually selectable. Possibly the same budget-solve area as #111 and #150.
Kernel cache directory is not created. Every start recompiles kernels:
freetoken/models/deepseek_v4/weight.py:70: UserWarning: Specified kernel cache
directory could not be created! This disables kernel caching
What worked
For balance: the model itself runs correctly on SM86. fp8 dense weights and fp4 experts are dequantised at runtime with no kernel errors, dsv4_sparse attention is selected automatically, and output quality is fine. The only hard blocker to getting it up was the NCCL symlink above. The problem is backend selection, not Ampere support as such.
Activity
ascorb12 commentedon Aug 24, 2026
Ran the same hybrid vs offload comparison on different hardware and cant reproduce it, they come out the same here.
Gemma-4-26B-A4B-it bf16 (47 GB), 32,768 ctx, single 4090:
Same prompt, same 32k pool, only the backend flag changed.
ft bench bwpicked hybrid for bf16 here at a 2.27x ratio and that turned out fine, no big gap either direction.Setup is 7800X3D, single RTX 4090 (SM89), 63 GB RAM, WSL2 Ubuntu 24.04, built from main, CUDA 13.0, driver 610.88. The CPU MoE executor only got 6 threads pinned here vs your 46 cores.
Fair warning on how much this is worth, its a different model and a different dtype than yours (bf16 Gemma vs fp4 DeepSeek), one GPU instead of two, and the weights are loading off /mnt/c which is slow. So its not a clean refutation, just a data point that the backend pick isnt wrong everywhere.
The part that looks different between our boxes is the CPU side, not the backend pick. Yours benched 56.2 GB/s and delivered ~2 GB/s serving. Ours benched 58.1 GB/s for bf16 and the serving speed lines up with it, no collapse. Might be worth checking whether the container is capping CPU or if its a NUMA thing across those 46 cores, since thats where the 28x gap is showing up rather than in the backend selection itself.
gdevenyi commentedon Sep 5, 2026
Same mispick on Ada, so this is not Ampere-specific.
Box: 2 x RTX 6000 Ada 48 GB (sm_89), PCIe 4 x16, no NVLink, 2 x Xeon Gold 6526Y, 503 GB RAM. Model: Qwen3.8-Flash-Next NVFP4 (
qwen4_exp, 512 routed experts, top-10, 48 MoE layers).ft bench bwrecommends hybrid; measured, offload is about 2x faster single-stream and we serve on--moe-backend offloadpermanently because of it. Smaller than your 8.3x on DeepSeek-V4-Flash, same sign.The mechanism looks the same as the one #278 identifies: the verdict is computed against a generic expert size rather than the served model's. This model's experts are small (
moe_intermediate_size640, NVFP4, so ~1.23 MB per expert per rank at TP=2) and it routes 10 of 512 per token, which is precisely the regime where the hybrid backend's per-layer CPU dispatch latency dominates whatever PCIe it saves. The bandwidth ratio the benchmark measures is real; it just is not the quantity that decides for this shape.Two things I would add from here, both of which argue for #278's direction over a threshold tweak:
--moe-backend autois the default, so this is what a new user gets. On this box the default configuration is the slow one, silently. Both the auto-pick andft bench bw's printed recommendation should agree with the model actually being served.Feedback with the Ada numbers is on #278.
🤖 Generated with Claude Code
https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt
dustfeather commentedon Sep 12, 2026
Adding demand-side telemetry from a third box rather than another "auto mispicks" confirmation, since that part looks settled.
Box: RTX 3070 (sm_86), single GPU, no TP, PCIe gen 3 x16, i7-9700K (8c/8t, AVX2 only, no AVX-512/VNNI), 54.9 GiB DDR4. FreeToken 0.1.2,
--moe-backend offload, Qwen3.6-35B-A3B FP8, context 16384. (WSL2, but GPU passthrough is healthy and not a factor in any of the below.)Sampled with
nvidia-smi dmon -s putduring steady-state decode:Active weight read per token, computed from
config.jsonrather than the model card:STREAM read on this host measures 24.9 GB/s (not DDR4-3200's 51 GB/s theoretical); gen-3 x16 delivers ~12.5 GB/s practical. So:
Measured decode: 13.1 tok/s, with the link at ~78% of practical. That is at the PCIe ceiling — the GPU expert cache carries the small excess above it — while host RAM sits at under 40% of its measured bandwidth.
Why this bears on suggestion #2.
ft bench bwweighs a calibrated CPU-MoE figure against a PCIe figure, but on theoffloadpath the PCIe side is what actually binds, and here it is demonstrably saturated while the CPU side is idle — 7 of 8 cores doing nothing at all. A calibration that reads 56.2 GB/s on the CPU side and concludes it is 4.57× better is measuring a rate the serving path never achieves. Live rx/tx during decode would be a cheap discriminator to add to the calibration, and it is the number that would have predicted the right backend on every box in this thread.The one-pegged-core signature @ascorb12 describes reproduces identically here — at a very different core count, with no TP and no sharding involved — which suggests it is inherent to the path rather than a thread-count or shard-size artifact.
One negative result, so nobody else spends time on it: I suspected the pegged core was
RLIMIT_MEMLOCKstarvation (WSL2 defaults to 65536 KB, and it is a soft limit, raisable viaprlimit). Lifting it changed decode from 12.8 to 13.1 tok/s — about 2% — and did not change the one-core signature. That is not the cause.Happy to run specific instrumentation on this box if it would help narrow the calibration work.
dustfeather commentedon Sep 12, 2026
Correction to my comment above: the bytes/token figure omits the shared expert.
config.jsoncarriesshared_expert_intermediate_size: 512alongsidenum_experts_per_tok: 8— a shared expert the same size as a routed one, active on every token. My formula counted only the 8 routed:Which figure belongs in a ceiling depends on what is being bounded. All 1.132 GB is computed every token, so that is the right number for CPU-side FLOPs. But the shared expert is needed on every token, so a cache of any sensible policy keeps it resident and it never crosses PCIe — leaving the routed 1.007 GB as the only weight that can miss and stream. So the ~12.4 tok/s PCIe ceiling I quoted stands, though for a reason I had not actually stated.
The cleaner derivation avoids the residency question altogether, since the link traffic is directly measurable:
That independently reproduces a ~24% hit rate I had measured another way, and it explains what the original framing could not: 13.1 tok/s is above the 12.4 tok/s pure-streaming ceiling, which only cache hits account for.
None of this changes the point of the original comment — the link is still at ~78% of practical while 7 of 8 cores sit idle, and that is still the demand-side evidence that
offloadis correctly bandwidth-bound where the calibration's CPU figure suggests otherwise. Flagging it because a 12.5% error in a bytes/token figure is exactly the kind of thing that would propagate if anyone reused the formula, and the shared-expert term is missing from the version of it that gets quoted around.A-XIANGQAQ commentedon Sep 24, 2026
Third independent box, same sign — Ada, single RTX 4090 (sm_89).
Setup: RTX 4090 48 GB (sm_89), PCIe 4.0 x16, 38-core Xeon (AVX512F + AVX512-VNNI), 247 GB RAM, FreeToken 0.1.3 from PyPI,
RadixArk/Qwen3.8-Flash-Next-NVFP4(126 GiB, NVFP4 routed experts), single GPU, KV pool 265216 tokens.ft bench bw --dtype nvfp4 --cpu-threads 36recommends hybrid, by a wide margin:What serving actually measures — same prompts, 256 output tokens, streamed, client-side timing, idle server, no other traffic:
--moe-strategyhybrid is 27% slower on decode (ITL p50 21.0 ms vs 15.0 ms). Prefill is unchanged. Under hybrid the GPU stays busy (73–100%) and the CPU only reaches 1–18%, so this is not CPU saturation — the split itself costs us.
Two more data points from this box:
ft bench bw --model qwen3.6-moe --formats nvfp4(1.69 MB experts): CPU-MoE 94.5 vs PCIe 25.3 = 3.74x — still “hybrid”, and the CPU kernel already lost 30% just from the smaller geometry (135.6 → 94.5 GB/s).Workaround we settled on: pin
--moe-strategy offload. Worth calling out for anyone leaving it onauto: once a profile exists,autosilently resolves tohybrid, so the flag has to be pinned rather than left to default.