Repository navigation
[strata] Edge0-145B NVFP4 prefill hangs RTX PRO 6000 (sm_120) #200
Description
Hi, trying to get Edge0-145B-NVFP4 running on Linux and I can't get a single token out of it.
Setup:
RTX PRO 6000 Blackwell Server Edition 96GB (sm_120), GPU passthrough into a KVM VM, 32 vCPU, 80GB RAM
Ubuntu 26.04, driver 595.84, CUDA 13.2.1, gcc 15.2
commit b7900cb, built from source since there's no Linux package. strata built with -DCMAKE_CUDA_ARCHITECTURES=120, edge0-infer with the cpu preset
The model loads fine (90-135s, ~58GB VRAM) but every request gets stuck on "reading the prompt" and after 60s the engine dies with exit code -6 and restarts. Doesn't matter what the prompt is, even a 54 token "Say hi" does it.
[strata] reading the prompt: 125 of 130 tokens, 11 s so far
...
[strata] reading the prompt: 125 of 130 tokens, 61 s so far
[strata] the engine stopped unexpectedly (exit code -6). The usual cause is running out of RAM ...
[strata] done: 0 tokens in 68 s (no speed) (error, cancel=False)
It's not RAM, nothing in dmesg, no Xid, and the GPU never went above 72C. So I guess the -6 is just the watchdog and the RAM hint is a bit misleading.
Looks a lot like #199 so I applied that patch and rebuilt. Still hangs, same spot. Then tried STRATA_LATENT_FUSED_DU=0, also still hangs. One thing that tripped me up there: setting it as an env var does nothing because the model.json that ships with 145B-NVFP4 has "STRATA_LATENT_FUSED_DU": "1" in its env block and that wins. Had to change it in model.json.
I couldn't get the actual strata stderr, even with --verbose. Looks like it goes to a temp dir that gets deleted when it exits. If you tell me the best way to run it so the log stays around I can try again.
Couple of questions:
is sm_120 supposed to work with 145B NVFP4 on Linux right now?
should I be building with 120a or 120f instead of 120?
Also small thing, the full edge0-infer build fails on gcc 15 because a few tests are missing #include (slots_test.cpp, edge0_35b/loading_test.cpp, edge0_35b/assets_test.cpp). Building just the edge0-infer target works.
Thanks!
Activity
linyubupa commentedon Nov 14, 2026
Thanks for the detailed report — that's genuinely useful.
Short answers first:
sm_120 on Linux is supported. We build and test 12.0 ourselves on RTX 5090 D (same compute capability as the PRO 6000) including 145B NVFP4, so this isn't a fundamental arch gap. -DCMAKE_CUDA_ARCHITECTURES=120 is the correct flag; don't use 120a/120f.
Your env-var finding is right. The shipped model.json carries an env block that is applied to the engine process and wins over your shell environment, so STRATA_LATENT_FUSED_DU has to be changed in model.json (as you did). A shell export does nothing there. Good catch.
The -6 is a SIGABRT during prefill; "running out of RAM" is just the watchdog's generic hint, not a diagnosis. To see what actually aborted we need the engine's own stderr. In serve mode it isn't written to a throwaway temp dir — the engine inherits edge0-infer's stderr, so capture everything with:
edge0-infer serve --model
--device cuda 2>&1 | tee serve.log(edge0-infer chat --model
2>&1 | tee chat.log also works; chat prints the engine log path at startup and keeps it on failure.) Paste the last ~50 lines around the crash — the CUDA error / assert line is what pinpoints this.Since it loads fine (~58 GB VRAM) and dies only in NVFP4 prefill inside a KVM passthrough VM, the two things I want to rule out from that log are a pinned-memory allocation failing under the VM, and a CUDA runtime abort specific to the NVFP4 prefill path.
Also, thanks for the gcc 15 report — the three test files are missing #include ; we'll fix that. Building only edge0-infer is fine meanwhile.