Qwen3.8-27B-NVFP4 fails to load — RuntimeError: CUDA driver error: device not ready #139

Summary

Loading Qwen3.8-27B-NVFP4 via ft serve fails deterministically during weight loading (consistently around 11–38% progress) with RuntimeError: CUDA driver error: device not ready. This occurs inside FreeToken's weight-materialization code, in two closely related locations depending on the run. The failure is fully reproducible and independent of --memory-ratio, torch/CUDA version, driver version, and filesystem source (native ext4 vs. WSL /mnt/c mount). A known-good MoE checkpoint (Qwen3.6-35B-A3B-NVFP4) runs flawlessly for hours on the identical hardware/software stack, isolating the bug to this specific checkpoint/architecture.

Hardware

• GPU: NVIDIA GeForce RTX 5060 Laptop (Blackwell, compute capability sm_120), 8GB VRAM
• Driver (Windows host): NVIDIA 610.88 (KMD), NVIDIA-SMI 610.57.01, CUDA UMD 13.3
• OS: Windows 11 with WSL2 (Ubuntu); model also tested via the native FreeToken Windows desktop app
• Storage tested: both native WSL ext4 (~/models/...) and cross-mounted NTFS (/mnt/c/Users/.../.freetoken/models/...) — same result

Software versions tested (all combinations reproduce the bug identically)

torch.cuda.get_arch_list() confirmed sm_120 present in all working configurations. sgl_kernel import (from sgl_kernel import fused_add_rmsnorm, rmsnorm) verified successful before each test.

Model

• Checkpoint: Qwen3.8-27B-NVFP4 (dense architecture — confirmed by FreeToken's own detection log: qwen3_5 is a dense model (no routed experts))
• Downloaded via FreeToken Desktop's model catalog (added in Desktop v0.2.0-beta.14, released Aug 23 2026)
• Total checkpoint size: 18.8 GiB (per FTW loading progress bar)

Steps to reproduce

source ~/.venv/bin/activate
ft serve --model ~/models/Qwen3.8-27B-NVFP4 --memory-ratio 0.6 --port 1919 --host 127.0.0.1

Also reproduced with --memory-ratio 0.75 / 0.85 / 0.95, and with/without --moe-backend offload --moe-cpu-layers 0.8 explicitly set — these flags are correctly ignored for a dense model per FreeToken's own auto resolution logic (per docs/models.md: "auto — dense models always resolve to fused"), and have no effect on the crash either way.

Observed behavior

Server starts, model config loads, dense-model detection logs correctly, weight loading begins via the FTW format, then crashes partway through with no recovery:

[core|rank=0] INFO     Auto-selected attention backend: fi
[core|rank=0] INFO     Resolved config: attention_backend='fi', cache_type='hybrid_radix', page_size=1
[core|rank=0] INFO     Free memory before loading model: 6.86 GiB
Loading weights (FTW):  37%|██████████████████████████▍     | 6.98G/18.8G [00:03<00:06, 1.93GB/s]
Process freetoken-TP0-scheduler:
[FrontendAPI] ERROR    Backend supervisor: RuntimeError: CUDA driver error: device not ready

Traceback variant A (most common, ~11–38% progress)

Traceback (most recent call last):
  File ".../multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File ".../multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "freetoken/server/launch.py", line 74, in _run_scheduler
    scheduler = Scheduler(args)
  File "freetoken/scheduler/scheduler.py", line 64, in __init__
    self.engine = Engine(config)
  File "freetoken/engine/engine.py", line 323, in __init__
    self.model.load_state_dict(self._load_weight_state_dict(config))
  File "freetoken/engine/engine.py", line 459, in _load_weight_state_dict
    return _materialize_loaded_weight_state_dict(
  File "freetoken/engine/engine.py", line 282, in _materialize_loaded_weight_state_dict
    state_dict[key] = weight.to(device=device, dtype=expected.dtype)
RuntimeError: CUDA driver error: device not ready

Traceback variant B (observed on native-WSL-path runs, ~11% progress)

  File "freetoken/engine/engine.py", line 277, in _materialize_loaded_weight_state_dict
    for key, weight in weights:
  File "freetoken/models/weight.py", line 251, in load_weight
    yield from iter_weights(
  File "freetoken/models/qwen3_5_moe/weight.py", line 279, in iter_weights
    fused = _try_fuse(name, tensor, fuse_buf)
  File "freetoken/models/qwen3_5_moe/weight.py", line 168, in _try_fuse
    return key, torch.cat([slots[i] for i in range(len(parts))], dim=0)
RuntimeError: CUDA driver error: device not ready

Note: Variant B is notable because Qwen3.8-27B is correctly logged as dense (qwen3_5 is a dense model (no routed experts); ignoring MoE settings), yet the crash occurs inside qwen3_5_moe/weight.py's _try_fuse function — MoE-specific expert-fusion logic (torch.cat over expert "slots") — which appears to still execute on this dense checkpoint's tensor layout despite the correct dense-model detection upstream.

Variables ruled out (systematic elimination)

Control test — confirms hardware/environment is healthy

The identical hardware, driver, torch/CUDA stack, and FreeToken installation successfully runs Qwen3.6-35B-A3B-NVFP4 (genuine MoE model) for multi-hour sessions without error, sustaining ~17–22 tokens/sec generation throughput, correct MoE expert offload behavior, and correct KV-cache/context caching (confirmed reuse of 38k+ cached tokens across turns). This rules out any GPU, driver, WSL, or general FreeToken-installation problem — the issue is isolated specifically to loading Qwen3.8-27B-NVFP4.

Suspected root cause

Given:

1. The crash is deterministic (same progress %, same timing, same GPU state) rather than exhibiting the intermittent/timing-sensitive behavior typical of true race conditions or driver-state issues.
2. It's fully independent of memory pressure (reproduces identically even at --memory-ratio 0.6).
3. One crash variant occurs explicitly inside MoE expert-fusion code (qwen3_5_moe/weight.py::_try_fuse) despite correct dense-model detection.

...this strongly suggests a checkpoint-loading logic bug specific to the Qwen3.8-27B architecture within freetoken/models/qwen3_5_moe/, likely related to incomplete separation between the dense and MoE code paths for this model family, and/or mishandling of this checkpoint's NVFP4 tensor packing during the .to(device=...) transfer step. The generic "CUDA driver error: device not ready" message is likely a misleading downstream symptom of a malformed tensor operation (bad shape, dtype, or device-placement state) rather than an actual GPU/driver-level fault.

Possibly related: PR #104 (feat(models): support TP for qwen3_5_moe, opened Aug 23 2026) touches this same file and was under active development the same day this checkpoint was added to the model catalog (Desktop v0.2.0-beta.14, also Aug 23 2026) — the timing suggests dense-model support for this architecture may not yet be fully stabilized.

Suggested fix direction

Verify that qwen3_5_moe/weight.py's dense-model loading path fully bypasses expert-fusion logic (_try_fuse) rather than only skipping the CLI-flag-driven MoE backend selection — the dense-detection warning currently logs correctly but doesn't appear to gate this specific code path.