ft ctl cache: growing the MoE pool while shrinking KV OOMs and rolls back on a nearly full card #5

Environment: RTX 5080 16 GB (sm_120), AMD Ryzen 9 9950X, 123 GB RAM, Ubuntu 24.04 (kernel 7.1.9-xanmod), driver 610.57.04, CUDA 13.3, torch 2.11.0+cu130, triton 3.6.0, FreeToken 0.1.2 editable. Checkpoint: nvidia/Qwen3.8-Flash-Next-NVFP4.

Branch: 5080deploy @ 150c95d (main e0886cc + FlashML-org#408).

Server: ft serve --model ~/models/Qwen3.8-Flash-Next-NVFP4 --kv-cache-dtype nvfp4 --num-tokens 262144 --max-seq-len-override 262144 --max-running-requests 1 --cuda-graph-max-bs 1 --moe-cache-size 512 --disable-moe-prefill-overlap --memory-ratio 0.95 --max-prefill-length 4096

ft ctl cache while idle:

state=serving, 4.1 GiB allocated (rebuild budget 4.8 GiB)
  moe    512 slots (lru, 2.1%)   1.3 GiB    512..1843
  kv     262144 tok (4096 x 64)  1.9 GiB    64..648194
  mamba  8 slots                 882.3 MiB  4..44

Both resizes below target a total at or below what is already allocated, and both are rolled back:

$ ft ctl cache --moe 1024 --kv 65536 --wait 300        # ~2.6 + 0.48 + 0.88 = ~3.96 GiB
HTTP 503: rebuild failed and was rolled back: OutOfMemoryError('CUDA out of memory. Tried to allocate 800.00 MiB. GPU 0 has a total capacity of 15.51 GiB of which 511.50 MiB is free. ...')

$ ft ctl cache --moe 1400 --kv 8192 --mamba 4 --wait 300   # ~3.56 + 0.06 + 0.44 = ~4.06 GiB
HTTP 503: rebuild failed and was rolled back: OutOfMemoryError('CUDA out of memory. Tried to allocate 274.00 MiB. GPU 0 has a total capacity of 15.51 GiB of which 91.50 MiB is free. ...')

The rollback works (the server kept serving at the old sizes). But the advertised "resizable to" ranges and rebuild budget suggest these fit. The OOM sizes suggest the MoE pool grows before the shrinking KV/mamba pools release their memory, so on a nearly full card the ranges aren't reachable live. Restarting with --moe-cache-size 1280 --num-tokens 32768 boots fine.

Frees-before-allocates ordering, or ranges that account for transient double residency, would make ft ctl cache usable here.

Related

• Related upstream: feat: Automatic KV/MoE Laddering for decode speed vs context-length trade off ( upto 33% faster decode )  FlashML-org/FreeToken#300 (automatic KV/MoE ladder) trades expert slots for KV through the same runtime rebuild, so it would hit the same allocate-before-free ordering on a nearly full card. No existing upstream issue found for the rebuild OOM itself.