hybrid MoE strategy 20-40x slower than offload on an offload-heavy box (24 GB card, PCIe gen 3) #436

Activity

1. jason-fxz commented on Sep 11, 2026

   

   jason-fxz

   on Sep 11, 2026

   Last edited by jason-fxz

   Collaborator

   More actions

   Thanks for the write-up. Two things would help us narrow this down:

   1. The ft bench bw output for this box (or the profile file it wrote, ~/.cache/freetoken/benchbw.json). The 27.8% fetch split comes from it, and we would like to see the CPU / PCIe numbers behind that.
   2. The full startup log of a hybrid launch on b16877b, up to API server is ready to serve. The lines we are after are the ones from CPU MoE executor ready: ... (isa / threads), whether cpu-moe flag handshake unavailable: ... appears, and PLE disk backend: ..., wait-sync|launch-gating.

   For reference, the same tree and checkpoint on an H100, with --moe-cache-rate 0.152 to match your 3741-slot residency, decodes at 56 tok/s on hybrid vs 50 tok/s on offload, so this is specific to something on your platform rather than the hybrid path itself.
2. gberasmus87 commented on Sep 11, 2026

   

   gberasmus87

   on Sep 11, 2026

   Author

   More actions

   All three, plus one comparison that I think narrows it a long way.

   1. ~/.cache/freetoken/benchbw.json

   Written 2026-08-25T00:23:21+00:00, version 4. Ceilings and the nvfp4 entry (the one this checkpoint's experts use):

   "cpu": { "physical_cores": 16, "threads_used": 16 },
   "threshold": 2.0,
   "ceilings": {
     "cpu_stream_read_gbs": 31.75,
     "pcie_linear_h2d_gbs": 14.21,
     "pcie_linear_d2h_gbs": 14.34
   },
   "dtypes": { "bf16": "hybrid", "nvfp4": "hybrid", "fp8_block": "offload",
               "mxfp4_triton": "offload", "ds_fp4": "hybrid" },
   "dtype_kernels": {
     "nvfp4": {
       "expert_bytes": 7974912, "synth_experts": 128,
       "cpu_moe_gbs": 29.97, "cpu_moe_isa": "avx2", "isa_sweep": null,
       "pcie_gather_gbs": 13.09,
       "cpu_moe_overlap_gbs": 21.89,
       "pcie_gather_overlap_gbs": 8.41,
       "ratio": 2.29, "recommended": "hybrid", "note": null
     }
   }

   ratio 2.29 against threshold 2.0 is where the hybrid recommendation and the 27.8% split come from. isa_sweep is null on every entry. Happy to attach the whole file if useful — bf16 is ratio 2.352, ds_fp4 2.329, mxfp4_triton 1.38 -> offload, fp8_block has "CPU MoE has no fp8_block weight path; hybrid unavailable".

   Note the date: this profile predates the problem and both builds below read the same one, so it is a constant across the comparison rather than a variable.

   2. Startup, hybrid on b16877b, to "ready to serve"

   Resolved config: moe_strategy='hybrid', attention_backend='qsa_sparse', cache_type='hybrid_radix', page_size=64
   Free memory before loading model: 23.13 GiB
   PLE disk backend: io_uring, O_DIRECT, wait-sync
   MoE experts: nvfp4 via triton
   expert banks: slow path (parallel build)
   --moe-cache-auto resolved moe_cache_size=3741 num_pages=1025 (prefill_overlap=True)
   --moe-hybrid-max-fetch auto: fetching 27.8% of each decode step's expert misses over PCIe (benched PCIe/CPU bandwidth ratio), the rest on the CPU
   torch intra-op threads: 16 -> 1 (cores reserved for the pinned CPU MoE pool)
   CPU MoE executor ready: threads=15 (pinned to cores 0..14) isa=avx2 fmt=nvfp4 H=2560 I=640 experts=512 layers=48 top_k=10 act=silu max_tokens=4
   Allocating 65600 tokens for KV cache, K + V = 1.55 GiB
   Free memory after initialization: 2.18 GiB
   Start capturing CUDA graphs with sizes: [1, 2, 4]
   Free GPU memory after capturing CUDA graphs: 2.03 GiB
   Scheduler is idle, waiting for new reqs...
   API server is ready to serve on 127.0.0.1:1919
   

   PLE disk backend is io_uring, O_DIRECT, **wait-sync**.

   3. cpu-moe flag handshake unavailable

   Never appears. journalctl | grep -ci "cpu-moe flag handshake" over every run since 2026-09-09 returns 0.

   The comparison I think matters

   My 33.5 tok/s baseline is v0.1.2 + local patches. I pulled its startup log and put it beside the b16877b one. Every logged parameter is identical:

   	v0.1.2 (33.5 tok/s)	b16877b (0.5 tok/s)
   strategy / backend / cache	hybrid, qsa_sparse, hybrid_radix, page 64	same
   moe_cache_auto	3741 slots, num_pages=1025	same
   hybrid split	27.8% PCIe	same
   torch intra-op	16 -> 1	same
   CPU MoE executor	threads=15, cores 0..14, isa=avx2, fmt=nvfp4	same
   PLE backend	io_uring, O_DIRECT, wait-sync	same

   Same box, same checkpoint, same flags, same cached bench profile, same resolved everything — 60x apart. So whatever this is, it is not the split, the ISA, the thread layout, the cache sizing or the PLE backend being resolved differently. It is in the code between the two.

   Two other things that rule out the obvious:

   • PLE disk backend is wait-sync on the offload run too, and that run is fine at 23 tok/s. So the PLE path is not the differentiator between the two strategies on this box.
   • The box is an EPYC 7302, 16 physical cores / 32 threads. torch intra-op threads: 16 -> 1 with 15 cores pinned to the MoE pool leaves torch exactly one thread. I suspected that, but the working build does exactly the same thing, so it is not the cause on its own — though it may be what makes this platform sensitive to a change that is free on a box with more cores.

   Given your H100 numbers (56 hybrid vs 50 offload at matched residency), agreed that it is platform-specific rather than the hybrid path as such. The window is v0.1.2..b16877b. I am happy to bisect it here — roughly 34 commits at a ~5 minute install-plus-load cycle, so a few hours — or to run any specific commit or instrumented build you would rather point at first. Say which and I will get you numbers.