fix(engine): ft ctl cache live resize: OOM during rebuild breaks rollback and wedges the server #526

What happens

A live cache rebuild (ft ctl cache --kv ... --mamba ...) that runs out of VRAM mid-rebuild fails to roll back and leaves the server answering every request with:

{"error":"server unavailable: maintenance failed (restart required)"}

Log tail:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.79 GiB ...
HTTP 503: OutOfMemoryError(...); rollback to the prior geometry also failed:
AttributeError("'NoneType' object has no attribute 'shape'")

Repro

RTX 4070 Ti SUPER 16 GB, Qwen3.8-Flash-Next (qwen4_exp), KV fp8 (PR #354 branch), serving with the pool near full. Then:

ft ctl cache --kv 260000 --mamba 12 --wait 300

The request needs more than the free budget, the rebuild OOMs partway, and the rollback path hits NoneType.shape. Reproducible 2/2 here. Smaller resizes that fit the budget work fine and report "old cache kept, still serving" correctly, so the guard works when the allocation fits; the failure is specific to an OOM partway through.

Expected

A failed live resize should never wedge the server. Two options: preflight the allocation (try_alloc on a dry run), or make the rollback tolerate a partially built pool.

Workaround we run with

Geometry changes go through the serve script and a restart, never through the live path when the GPU is near full.

Versions

FreeToken branch work (main + #354), driver 580.173.02, CUDA 13, Ubuntu 24.04.