Offloaded MoE ignores tensor parallelism: TP=2 is a regression, and the fix is not expert parallelism

Summary

--tensor-parallel-size 2 is currently a regression for offloaded MoE: measured 15.63 tok/s against 18.00 at TP=1 on DeepSeek-V4-Flash (2x RTX 6000 Ada, no NVLink). Two independent single-GPU servers give 2.05x instead, so the hardware scales fine — the engine does not.

I went looking for expert parallelism and concluded that EP is the wrong fix, and the right one is already half-built.

Why TP=2 loses today

The resident MoE path shards by TP:

intermediate_size_per_partition = div_even(intermediate_size, tp_size)
if allocate_experts:
    self._alloc_resident_experts(intermediate_size_per_partition)

OffloadMoELayer passes allocate_experts=False and takes its weights from OffloadMoeCache, which has no TP awareness at all — no tp_size/tp_rank in offload_cache.py, expert_banks.py or models/deepseek_v4/moe.py. Every rank therefore loads and caches whole, unsharded experts.

Confirmed at runtime: moe_cache_size=2477 per rank at TP=2 versus 2559 at TP=1. The cache did not grow. Both ranks stream the same experts over their own PCIe link, so aggregate PCIe traffic doubles for identical work, and then the layer pays an all-reduce over PCIe. Only attention and dense shard usefully, and this model is MoE-dominated.

Why not expert parallelism proper

Sharding experts across ranks (rank r owns e % world == r) needs all-to-all dispatch and combine per MoE layer. Three problems, in order of severity:

CUDA-graph capture. Decode graphs are captured at static shapes. All-to-all counts depend on routing, so they vary per step. Workable by padding to the worst case (bs * top_k tokens per rank), but that is real added machinery in the hot path.
Load imbalance. Real MoE routing is skewed; a rank owning hot experts stalls the others every layer.
New collectives in a codebase that currently only needs all_reduce.

The fix that fits: TP-shard the offload banks

Split the intermediate dimension, exactly as the resident path already does. Rank r holds I/tp of every expert.

Cache doubles. A slot is half the bytes, so the same VRAM holds ~2x as many distinct experts. Residency here goes from 2559/11008 = 23% to roughly 46%.
PCIe halves per rank. Each rank fetches only its half of a missed expert, over its own link — two links, half the bytes each.
Host RAM is unchanged. 71.5 GiB per rank instead of 143 GiB replicated, so TP=2 costs no more host memory than TP=1 (today it costs double).
No new collectives. Intermediate-dim sharding makes the down projection produce partial sums, and _maybe_all_reduce already combines them. The communication is bs * H * 2 bytes per layer — 8 KiB at bs=1, negligible.
CUDA graphs unaffected. Shapes stay static.

Extrapolating the measured residency curve (25% -> 50% residency was worth +59% on a comparable model), plus the halved per-rank gather, this looks like 1.4-1.8x on this box — versus 0.87x today.

What it touches

The work is almost entirely in loading, not in the kernels — they already accept whatever I they are handed.

Per-format bank slicing at load (moe/expert_banks.py). The awkward part: gate_up is [2I, H] with gate at rows [0, I) and up at [I, 2I), so a rank's slice is two disjoint row ranges, not one. down is [H, I], a column slice.
Divisibility. I/tp must stay a multiple of each format's block size — 128 for fp8_block, 32 for the fp4 scale blocks, 32 for q4_0. DSV4 at I=2048, tp=2 gives 1024, which is fine, but this needs a guard rather than an assumption.
mxfp4 is transposed (blocks_t [K//2, N]), so its slice is along N and the scale tensor slices differently again. This format has already broken two general assumptions in recent work; it should be last, or explicitly unsupported at first.
Thread I_local through OffloadMoeCache construction in engine.py and the _BANK_SCHEMAS shape asserts.
The CPU MoE / hybrid path takes I from the cache, so it follows for free — but --moe-backend hybrid under TP needs its own check.

Suggested staging

bf16 only, TP=2, behind a guard that refuses unsupported formats. Prove the cache doubles and measure end to end.
Add ds_fp4 and nvfp4 (row-major, straightforward slices).
fp8_block and q4_0.
mxfp4 last, or leave it TP=1-only.

Measurement notes

ft bench decode cannot test this: LLM.__init__ hardcodes tp_info=DistributedInfo(0, 1). Either that gets a parameter or the comparison has to go through ft serve.
A server binds PORT and PORT+1 (a multiprocessing.spawn child), so multi-instance comparisons need spaced ports.
FreeToken JIT-links its own pynccl against a system NCCL; without libnccl-dev installed, TP fails at ld: cannot find -lnccl even though torch bundles libnccl.so.2. Falling back to torch's copy would remove a sharp edge.

We are building the intermediate-dimension shard you describe here, on a different model
(qwen4_exp / Qwen3.8-Flash-Next) and on two AMD R9700s. Adding what we have so far, because your
central claim — that the cache grows and no new collectives are needed — is holding up in
implementation, and because two things bit us that are not obvious from reading the tree.

Your "the cache doubles" argument checks out, and by more than double

Sharding the intermediate dimension per rank, both ranks build 206/206 weights with no shape
mismatch, and the expert cache resolves to:

	TP=1	TP=2 (per rank)
MoE cache slots	5,960	16,306
KV pages / tokens	1720 / 110,080	1720 / 110,080
K + V	2.60 GiB	1.34 GiB per rank
free VRAM after init	3.18 GiB	3.17 GiB

⭐ 2.74x the slots at the same VRAM. Your extrapolation from the residency curve assumed roughly
2x; on this model the slot is small enough relative to the page geometry that it does better.

✅ And the parts you predicted would be free were free. layers/moe.py:94-97 already all-reduces at
tp_size > 1, the fused kernels are dense-in/dense-out, so no all-to-all exists or is needed.
norm_topk_prob defaults True and the router is replicated, so a plain SUM all-reduce is
algebraically exact — not an approximation we are tolerating.

What the reduction costs, measured

We priced the collective before building anything, because it is the dominant risk in the whole case.
On two R9700s at the real decode shape ([1, 2560] bf16 = 5,120 bytes, 48 layers):

28.6–41.9 µs per call ⇒ 1.34–2.01 ms per token
RCCL 2.30.4 chose via P2P/IPC across the root complex, no host staging, despite no XGMI on this
board

⚠️ Two caveats on how to read that. It is a standalone measurement of the transport — our model
load has not yet reached its own collective, so this is what the reduction costs, not yet what the
design costs. And it is one collective per layer; a design that also shards the attention mixers
pays two, which is where our own numbers now sit.

⚠️ We also found prefill is compute-bound at full chunks on this hardware, not PCIe-bound. So the
second card's prefill win is much smaller than the residency argument suggests — around 8 % at 100k
context on our box. The case for TP here is a decode case, and it should be argued as one.

Two things that will bite anyone trying this on AMD

1. ⛔ Two ranks cannot share one copy of anything pinned, and it is a driver fact, not an allocator
one. svm_range_new() reserves size << PAGE_SHIFT per process against a single driver-global
counter, with no physical-page and no BO identity anywhere in the accounting
(kfd_svm.c:335-352, amdgpu_amdkfd_gpuvm.c:169-268). Two processes registering the same physical
pages are charged twice. The dma-buf escape is closed — "Only VRAM and GTT BOs are supported".

⇒ on ROCm the design must shard, never share. Which is an argument for your axis: a sharded
bank is genuinely half the bytes per process, where a shared one is billed as if it were not. We
established this from the driver source and have not measured it; the ~60-line probe that would
is written but unspent.

2. ⚠️ Ranks are spawned, not forked. server/launch.py:158 sets
mp.set_start_method("spawn", force=True), and the banks are mmap.mmap(-1, n) —
MAP_SHARED|MAP_ANONYMOUS, so shareable by fork but unreachable by name. Any design that hoped to
attach one copy from a sibling process needs those mappings named first. Sharding sidesteps this
entirely, which is another point in its favour.

Where our implementation actually is

Both ranks load and initialise. We are working through host RAM: the expert banks total 111.00 GiB
whether we run TP=1 or TP=2 (55.50 GiB per rank), against 122 GiB on this machine, so we are
tight in a way that is a property of our box rather than of your design. On a host with more RAM the
same branch has much more room.

⛔ Not yet cleared, and we are not claiming otherwise: the model's own collective at tp_size = 2,
CUDA-graph capture under the sharded shape, whether the QSA sparse-attention backend accepts a
per-rank head subset (its indexer is LinearReplicated and emits all heads), and the KFD pin ceiling.

⚠️ One thing worth flagging for qwen4_exp specifically, since it is not DSV4: the TP=1 only gate
in its weight loader is load-bearing. qkv_proj and the GDN in_proj are
LinearColParallelMerged and shard whatever you choose, so the offload MoE cannot be sharded in
isolation on this model — mixers come with it. That may be particular to this architecture, but it is
worth knowing before someone scopes "shard the MoE only" as a small change. #29 is the tracking issue
for that gate.

The question

Is the intermediate-dimension shard the axis you would accept for the offload MoE path?

The reduction is already in the tree and the fused kernels are dense-in/dense-out, so the change looks
confined to the loader and to rank-awareness in moe/, which currently has none. We would rather find
that out before polishing a PR against an axis you do not want.

⚠️ Two references for anyone weighing this: a third party measured PR #104's replicated-bank offload
TP=2 path at 3.1 tok/s against 94 at TP=1, which is the strongest argument that replicated banks
are the wrong axis; and #324 on this tracker runs the same model on two RTX 5090s as two separate
engines, because one model across both is not available. The demand is not AMD-specific.

Platform

AMD Radeon AI PRO R9700 x2 (gfx1201), Linux, 122 GiB host RAM · PyTorch 2.11.0+rocm7.14.0, HIP
7.14.60850, RCCL 2.30.4 · FreeToken main@4b94bdc3 + #132 + #133 + #134 + our own sharding work ·
BLOCK_N clamped to 32 in the QSA attend kernel (filed separately), which does not touch any number
above.

Following up on my comment above with the verdict, because the four things I listed as "not yet
cleared" are now cleared: the intermediate-dimension shard serves, and on this hardware it is worth
about +34 % decode and −40 % TTFT.

It serves

Two AMD R9700s (gfx1201), qwen4_exp / Qwen3.8-Flash-Next NVFP4, offload MoE, --tp-size 2.
Against the same box's two single-card controls, full-box, steady state:

	TP=2 (this work)	vs the same-backend TP=1 `disk` row	vs the TP=1 row we actually deploy
decode 10k / 100k (tok/s)	35.01 ± 0.08 / 34.63 ± 0.02	18.98 / 19.43	26.10 / 26.35
TTFT 10k / 100k (s)	5.68 / 53.1	9.59 / 88.8	9.39 / 89.0

Both ranks build 206/206 weights with no shape mismatch; CUDA-graph capture completes in 18–20 s;
the QSA sparse-attention backend does accept a per-rank head subset. At 110k the cache resolves
to 16,306 slots / 1,720 pages per rank — the 2.74x-at-equal-VRAM your "the cache doubles"
argument predicted, and then some.

⭐ Context is where it pays most. At --kv-reserve-tokens 262144 we get 14,876 slots / 4,096
pages, KV 3.19 GiB per rank, and decode costs only −0.9 % / −1.7 % against the 110k steady
state. The same jump on our TP=1 row costs −7.4 %. ⇒ the second card does not just raise the
throughput, it makes long context roughly free.

PR #311 is what unblocked this, and it is worth knowing why

We were stuck. At TP=2 both ranks loaded and then died in CUDA-graph capture at 0.38 GiB
MemAvailable — the pinned expert banks total the same 111.42 GiB at TP=1 and TP=2, and a
second rank's overhead was all it took on a 122 GiB host.

#311 (stream the PLE n-gram table from disk, merged 2026-09-01) removed the one bank nobody was
touching. --ple-backend disk pins 63.74 GiB instead of 111.42 and leaves 51.4 GiB of
MemAvailable at ready instead of 5.3 — and returned byte-identical greedy text on our TP=1 control.
That turned a 2.1 GiB deficit into ~45 GiB of margin, and the two-card path went from "cannot capture"
to "serves" with no change of our own to the memory design.

⇒ if anyone reads #62 and concludes the offload-MoE TP path needs a host with more RAM: on this model
it needed #311, which is already merged.

Vision replicates cleanly at TP=2, and it is charged somewhere non-obvious

We since put the vision tower and TP=2 in one process. The ViT is LinearReplicated throughout — no
name in it matches the _TP_SHARDED suffix table — so the 333-tensor, 897,862,112 B BF16 tower
replicates whole on both ranks, and it answers correctly from the image on a two-rank load.

⚠ The tower is paid for in MoE cache slots, not out of free VRAM. It enters weights_bytes, which
is subtracted from the same budget moe_cache_size is computed from: the identical 262k config goes
14,876 → 14,221 slots per rank, and 655 slots x ~1.33 MiB = 0.848 GiB against the 0.836 GiB
tower. Free-VRAM-after-init is bit-identical either way. Worth knowing before someone sizes a
vision + TP=2 config by looking at free VRAM and concludes the tower is free.

The three we did not close

1. ⚠ Greedy text at TP=2 differs from TP=1, and I cannot tell you whether that is numerics or a
bug. All three of our prompts diverge (first divergence at char 9 / 123 / 207), byte-identical
across three TP=2 loads, so it is systematic rather than a race. A SUM all-reduce over a replicated
router is algebraically exact, so reduction order is the benign explanation — but I have not
demonstrated that, and I am not claiming the two configurations serve the same model bit-for-bit.

⛔ I cannot discriminate it, because the API refuses logprobs ("logprobs is not supported") —
which is exactly what PR #224 implements. ⭐ If #224 lands, this becomes a one-afternoon
question: dump raw log_softmax at the first divergent position on both configurations and see
whether the top-2 gap is inside bf16 reduction noise (numerics) or wide (a defect on the sharded
path). That is the one thing I would ask for from this thread.

⚠ Caveat I should state rather than have found for me: our image predates #329, so these runs used
the approximate top-k/top-p kernel. We pinned top_k:1, top_p:1, so this is argmax and #329's
boundary-tie clipping is an unlikely cause across three prompts — but it is not excluded, and a
re-run on current main is the clean test.

2. ⚠ Roughly one TP=2 load in three comes up ~35 % slower, and nothing in the log distinguishes
them. 2 of 7 cold loads decoded at 22.9 tok/s instead of ~35, on a byte-identical command,
with identical moe_cache_size, identical relay handshake, no swap activity, and a fresh Triton
cache in one of the fast cases (so it is not JIT warmth). A row in that state serves **12 % under
the single-card row it replaces while looking perfectly healthy. Mechanism unidentified — flagging it
in case the shape is familiar to you, and because it means no one should bank a single TP=2 load as
a benchmark, including us.

3. ⚠ Cold load is ~18–20 minutes at TP=2 (1077–1190 s to ready), against ~3 minutes at TP=1. Both
ranks read all 206 shards, and the read is CPU-bound and strided rather than I/O-bound. Fine for a
long-lived server, painful for anything that swaps models.

⚠ Separately, at tp_size > 1 the first request after start is lost in the rank-0→rank-N ZeroMQ
PUB/SUB relay — a classic slow-joiner race — which wedges the server rather than erroring. Filed as
#364 with a handshake fix; not repeated here.

Platform

AMD Radeon AI PRO R9700 x2 (gfx1201), Linux, 122 GiB host RAM · PyTorch 2.11.0+rocm7.14.0, HIP
7.14.60850, RCCL 2.30.4 · FreeToken main@4b94bdc3 + #132/#133/#134 + #311 + our sharding work ·
BLOCK_N clamped to 32 in the QSA attend kernel (filed as #349), which does not touch any number
above · half-box controls where stated, otherwise full-box. No quality claim — we have no
fidelity instrument on this machine, which is exactly why item 1 above is open.

feat(models): TP support for qwen4_exp dense weights (Qwen3.8)

Follow-up on the "18-20 min cold load" I listed above as unresolved: that is closed. It was not the expert read being slow, it was OpenMP barrier spin-wait — both ranks default to one intra-op thread per logical CPU, so TP=2 on a 6-core box runs 24 threads on 6 cores. Clamping per rank takes the load from 1175 s to 61 s.

Filed separately as #371, since it is a load-path bug rather than anything to do with this thread's design question.

Same hardware here — 2 x RTX 6000 Ada, no NVLink, PCIe 4 x16 — and your diagnosis holds up. I built the fix you describe for a different model and it converts TP=2 from a regression into a win, so this is a data point that the "shard the offload cache, don't do expert parallelism" call was the right one.

Model is Qwen3.8-Flash-Next (qwen4_exp, 512 routed experts, top-10, 48 MoE layers) rather than DeepSeek-V4-Flash, but the backend is the same OffloadMoELayer + OffloadMoeCache path and the same "every rank caches whole, unsharded experts" behaviour you found. The work is #385: models/nvfp4_banks.py gains a _Placer that shards each expert bank along the intermediate axis per rank, so rank r only ever loads and caches its own slice.

Measured on the production checkpoint, offload backend, same box:

arm	single-stream	8 concurrent	expert residency
TP=1, one card	59 tok/s	137	37%
two independent TP=1 servers	—	266	37% each
TP=2 with sharded banks (#385)	88.6-89.8	299-325	94.5-100%
TP=2, plus FP8 dense + fp8 KV + FP8 lm_head	105.6	352.3	71.7%*

* residency drops in the last row only because that configuration spends the freed VRAM on an 8 x 262,144-token KV pool instead of expert slots; decode is flat from 100% down to ~70% residency on this model, so it is a deliberate trade for context, not a loss.

The important part is where the win comes from, and it is exactly what you predicted: the gain is residency, not the dense/attention sharding. Halving the expert bytes each rank must cache roughly doubles what fits, and on this model that moved residency from 37% to ~95%+ — which is worth far more than the all-reduce over PCIe costs. Sharding the dense side is almost noise by comparison. That is also why aggregate scales better than single-stream (2.2x vs 1.5x): the second card removes PCIe expert traffic rather than adding compute.

On the three objections to expert parallelism, for what it is worth from the same hardware: (1) is the one that would have hurt us most — this model's decode path is a single captured CUDA graph per batch size, and routing-dependent all-to-all counts would have meant padding to bs * top_k on the hot path for a model whose whole decode step is 9.9 ms. Not worth it when bank sharding gets residency to ~100% with only all_reduce.

One caveat if anyone ports the same change to another model: the bank sharding has to agree with the weight loader's view of the split, per rank, or the buffers silently mismatch. That is most of the diff in #385, not the placement itself.

🤖 Generated with Claude Code

https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt

Thanks for the numbers — and for building it. You opened this thread with a measurement, so what
follows is what our box adds to it rather than a first outside check. Three things: a residency
point your two boxes skip over, a correction to something we posted here ourselves, and the ask
you have already half-answered for us on #224.

1. Your extrapolation holds at a residency point neither of your boxes sits at

Your opening post put the sub-threshold case at 23 % to roughly 46 % on DeepSeek-V4-Flash and
predicted 1.4–1.8×; #385 then measured 37 % to 94.5–100 % on this checkpoint and 1.5×
single-stream. We land between the two, on the same checkpoint as #385.

Two R9700s, 32 GiB each. The expert banks stay host-resident — 31.65 GiB per rank, from
smaps — and what the shard moves is how much of them fits in the on-card cache:

	TP=1	TP=2, per rank
MoE cache slots at a 110k reserve	5,960	16,306
residency, derived	24 %	66 %

That row is arithmetic, not an instrument reading: 512 routed experts over 48 MoE layers = 24,576
expert-layer slices, and at TP=2 a slot holds half a slice. So a third of the bank is still not
resident and still gathers over PCIe — and the shard is worth taking anyway. These are the same
numbers as
our verdict comment above,
laid out as absolutes so the three arms can be read against each other:

	TP=2, full-box	TP=1 `disk`, same backend, half-box	TP=1 `pinned`, the row we deploy, half-box
decode at 10k (tok/s)	35.01 ± 0.08	18.98 ± 0.27	26.10 ± 0.35
decode at 100k (tok/s)	34.63 ± 0.02	19.43 ± 0.18	26.35 ± 0.11
TTFT at 10k (s)	5.68	9.59	9.39
TTFT at 100k (s)	53.1	88.8	89.0
`n` at 10k / 100k	8/10, 2/3	9/10, 2/3	8/10, 2/3

⇒ 1.84× the same-backend control at 24 % → 66 % residency, which is your 1.4–1.8× band coming in
at its top on a bigger residency step. n is after a zlib-compressibility filter on the generated
text, since a degenerate repetition loop reads as fast decode. One running request throughout, tree
c4b96fc1. Decode here is client-side (completion_tokens / (end − TTFT)), which is not the
instrument §2 uses.

Read that as the fast state, not as the row's number — §2 is why that distinction exists. The
same command an hour earlier read 22.88 / 22.66.

Separately, a load at a 262,144-token reserve resolves to 14,876 slots / 4,096 pages per rank
and decodes 34.68 at 10k and 34.03 at 100k, against 35.01 / 34.63 for the same TP=2 row at the
110k reserve. The same move on our deployed TP=1 row — measured separately, on a later build —
takes 10k decode from 25.62 to 23.73. That is the part of your case we would push hardest: on a
box that cannot reach full residency, the second card still makes long context nearly free.

Attribution, because it matters for anyone comparing: every number above belongs to one
tree — upstream 4b94bdc3 plus our patches 0001..0009, whose build asserts tree c4b96fc1 —
with the PLE (the model's n-gram table) streamed from disk rather than pinned, text-only, before
we had vision. It is not the tree we deploy today, and no quality claim is attached to any of it;
we have no fidelity instrument on this box.

2. A correction to our own item 2 above — and then a correction to the correction

In that same verdict comment we wrote that "roughly one TP=2 load in three comes up ~35 % slower."
That was wrong twice over, and it is worth putting the current state on the record because you will
meet it.

It is not a property of the load. A load that came up fast can step to the slow state
mid-life and stay there, and a reload can land slow from the first step. So "one load in three
comes up slow" describes the symptom from too far away.
We then thought we had a deterministic trigger — a multi-chunk image prefill. That is refuted
too. On 2026-09-08 the row stepped up between two consecutive ordinary text requests, with
no image anywhere in the body and no reload: request 53 decoded at 21.93, request 54 at
32.69, then 32.68 and 33.83. The first image in that conversation arrived at request 56, two
requests after the step.

What we can say, from the engine's own gen throughput lines over a three-hour window of our own
serving traffic — n = 2,716 fast and 465 slow decode-step samples, full-box:

The two states are ~22.5 and ~33 tok/s, and a row sits in one or the other. Context does not
select them: the same container ran 40 consecutive turns flat at 22.4–22.7 tok/s from 4k to 169k
of context.
The difference is per-step latency, not tokens per step: 40.08 vs 40.38 tokens per Decode
line, against a line interval of 1.215 s fast and 1.817 s slow. scheduler/status.py emits that
line every decode_log_interval forward passes (_decode_forward_count % interval), so equal
tokens per line is equal tokens per step and the whole gap is in the interval. ⇒ it is not
speculation acceptance. Per step that is 30.16 ms against 45.00 ms — derived from the median
rate in each state, not read off a timer.
Δ = 14.85 ms is additive and flat across a 5.6× span of context — +15.02 ms over the
25k–60k band, +14.59 over 60k–100k, +15.14 over 100k–140k. 140k is where the comparison stops: we
have no slow-state samples above it. A cost that does not scale with context is not attention, not
KV bandwidth and not cache size — it is a fixed per-forward-pass overhead. The GPU reads 97 % busy
in both states while memory-busy falls 31 % to 20 %, so the card is occupied and moving less.

Two rows, not one, which is the strongest thing here. The fast samples are all one TP=2 row (our
vision row, tree f0e5f318). Roughly half the slow samples are that same row and the other half are
a second TP=2 row on an older build with no vision code, a different --memory-ratio, a
different prefill chunk and 41 % more cache slots, which the operator started mid-window. It came up
slow from its first decode sample and read a median of 22.37 against the vision row's 22.4. So
the slow state is not vision, not the ratio, not the chunk size and not the bank.

Cause unidentified, and we are not claiming one. Two candidates and why neither is settled:

The per-layer all_reduce — right shape, constant per step, independent of sequence length,
and we run it through torch/RCCL rather than pynccl. But our own measurement above prices that
collective at 1.34–2.01 ms per token, an order below the 14.85 ms gap, so the transport cost
cannot be it. Rank skew — one rank waiting in the collective — would fit; we have not tested it.
dSpark: no verify CUDA graph for batch=3 (capture list 1,2,4, no padding) -> 6x per-step latency cliff; also no request cancellation #173's shape, a captured-graph fallback that turns into a fixed per-step cliff with the GPU
still busy. Its trigger cannot be ours (bs=3 on the dspark verify path; we run
--max-running-requests 1 and no speculation), but we have not verified that our decode is
replaying its captured graph in both states, and that is the next thing we would look at.

Ruled out and tempting: free VRAM is not the lever, and it is not context decay. And we have never
checked a TP=1 row for a two-state split, so nothing here says the split is TP-specific.

One instrument caveat: these are scheduler-side numbers, and #306 reports client wall-clock
diverging from that line at long context. That does not touch a scheduler-versus-scheduler
comparison, but it does mean these are the engine's tokens per second and not a client's.

If you see two decode plateaus on #385, the single number we would most like is whether your gap is
also ~15 ms/step, flat in context. That would tell us whether this is ours (ROCm,
--disable-pynccl, the offload path) or the design's.

3. On the greedy divergence — a refinement, not a disagreement

#385's Correctness section reads "the TP=2 server diverges from itself at the same points on a
second pass (bf16 atomics in the expert kernels), so that is run-to-run noise rather than a sharding
error." On our stack it is narrower than that, and the difference is cheap to test.

We sent each of three prompts three times on one load and hashed the bytes — full-box, tree
f0e5f318, the same build as §2. Pass b == c, byte-identical, both with the same cache hit. And
across two independent loads of the same row, each of the nine sends matched its counterpart on
the other load — pass a to pass a, b to b, c to c, for all three prompts. ⇒ the row is
deterministic given cache state; it does not disagree with itself run to run. What does differ
is a cold prefill versus a cache-resumed one for the same prompt — and the only prompt that
diverged is the only one with a large prefix hit (9,920 of 9,952 tokens). The one with a 64-token
hit agreed.

Which is why the phrasing in #385 caught our eye: a first pass is cold and a second pass of the same
prompt is cache-resumed, so "diverges from itself on a second pass" and "cold differs from resumed"
are the same observation until something separates them. The cheap discriminator is a third
pass, with the cache in the same state as the second: if b == c, the divergence is cold-vs-resumed
and not bf16 atomics, which is a much smaller and more fixable thing than reduction-order numerics.
Worth doing before anyone writes off TP=2 numerics.

Two limits on ours: n=2 on one prompt, and we have not established why a resumed prefill
lands on different bytes — it is a question of its own, not a conclusion. Our per-prompt pattern is
also not yours: the large-hit prompt diverged here, your two short ones diverged there. And the
caveat we already owed on this thread still stands — our tree predates #329, so these runs use
the approximate top-k/top-p kernel. We pin top_k: 1, top_p: 1, so a sampling change is an unlikely
explanation, but it is not excluded.

4. The ask is still `logprobs` — and you have already narrowed it for us

Our verdict comment above asked for PR #224
so we could compare distributions instead of bytes. Your #224 comment narrows what that buys — chat
logprobs fail closed under a reasoning parser — and it applies to us too: the flag defaults to
auto and the inference maps qwen4_exp to the qwen3 parser, so our serve command runs one
without asking for it.

Worth saying so the fail-close does not read as a blocker for this thread: the string our verdict
comment quoted ("logprobs is not supported") is the legacy completions path's, which is the
path the PR's own error message points at — and the one prompt that actually diverges here is
already a /v1/completions send. So #224 as it stands puts the test within reach: dump
log_softmax at the first divergent position on both configurations and see whether the top-2 gap
is inside bf16 reduction noise or wide. Whether the divergence survives that comparison is exactly
what we do not know yet. It is still open, still clean, and still the one thing we would ask this
thread for.

Platform

2 × AMD Radeon AI PRO R9700 (gfx1201, 32 GiB), Linux, ROCm 7.14, torch 2.11, 6 physical cores /
122 GiB host RAM. qwen4_exp / Qwen3.8-Flash-Next NVFP4, --moe-backend offload,
--ple-backend disk, --tp-size 2 --disable-pynccl, one running request.

Box labelling, since this box has two cards: a full-box number uses both R9700s at
--tp-size 2; a half-box number is a single-card TP=1 control. Every number above is labelled
in the line it appears in.

Our tree is upstream 4b94bdc3 plus a local patch series. §1's numbers are on the build that
asserts tree c4b96fc1 (0001..0009, text-only); §2's and §3's are on f0e5f318, the
deployed vision build, except for the second row named in §2, which was an older build still. The
QSA prefill tile is halved until 2 · head_dim · block_n · itemsize fits a 60 KiB LDS budget
(#349) — at this checkpoint's head_dim = 256 in bf16 that lands on BLOCK_N = 32, and
block_n = 64 does not run here at all, so every TTFT number above is on the clamped tile and the
clamp's cost is unmeasurable rather than zero.

No quality claim anywhere — we have no fidelity instrument on this box, which is exactly why §4 is
still open.

Closing this. It was written with heavy AI assistance, and the maintainers have indicated that do not want such contributions.

Navigation Menu

Repository navigation

Offloaded MoE ignores tensor parallelism: TP=2 is a regression, and the fix is not expert parallelism #62

Summary

Why TP=2 loses today

Why not expert parallelism proper

The fix that fits: TP-shard the offload banks

What it touches

Suggested staging

Measurement notes

Your "the cache doubles" argument checks out, and by more than double

What the reduction costs, measured

Two things that will bite anyone trying this on AMD

Where our implementation actually is

The question

Platform

It serves

PR #311 is what unblocked this, and it is worth knowing why

Vision replicates cleanly at TP=2, and it is charged somewhere non-obvious

The three we did not close

Platform

1. Your extrapolation holds at a residency point neither of your boxes sits at

2. A correction to our own item 2 above — and then a correction to the correction

3. On the greedy divergence — a refinement, not a disagreement

4. The ask is still `logprobs` — and you have already narrowed it for us

Platform

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Offloaded MoE ignores tensor parallelism: TP=2 is a regression, and the fix is not expert parallelism #62

Description

Summary

Why TP=2 loses today

Why not expert parallelism proper

The fix that fits: TP-shard the offload banks

What it touches

Suggested staging

Measurement notes

Activity

lukascechovic commented on Sep 2, 2026

Your "the cache doubles" argument checks out, and by more than double

What the reduction costs, measured

Two things that will bite anyone trying this on AMD

Where our implementation actually is

The question

Platform

lukascechovic commented on Sep 3, 2026

It serves

PR #311 is what unblocked this, and it is worth knowing why

Vision replicates cleanly at TP=2, and it is charged somewhere non-obvious

The three we did not close

Platform

lukascechovic commented on Sep 3, 2026

gdevenyi commented on Sep 5, 2026

lukascechovic commented on Sep 10, 2026

1. Your extrapolation holds at a residency point neither of your boxes sits at

2. A correction to our own item 2 above — and then a correction to the correction

3. On the greedy divergence — a refinement, not a disagreement

4. The ask is still logprobs — and you have already narrowed it for us

Platform

gdevenyi commented on Sep 14, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

4. The ask is still `logprobs` — and you have already narrowed it for us