Repository navigation
Draining ComfyUI kills a llama-swap request that ComfyUI itself sent #121
Description
What happened
I asked the QwenChat panel in ComfyUI to edit an image. QwenChat runs inside the ComfyUI process and calls llama-swap for NInfer-HauHauCS-27B. llama-swap started NInfer under berth run --priority foreground, berthd drained ComfyUI's lease, and the gateway stopped the ComfyUI server. That kill took down the process holding the open chat request. llama-swap logged the request as 499 after 4.6 ms, NInfer finished loading with nobody waiting on it, and the browser got HTTP 502 from the gateway on :8188.
Journal, 2026-10-21 14:00:49 MDT:
berthd msg=acquire lease=NInfer-HauHauCS-27B-2 engine=NInfer-HauHauCS-27B priority=foreground
berthd msg=draining lease=comfyui-1 engine=comfyui for=NInfer-HauHauCS-27B
llama-swap "POST /v1/chat/completions HTTP/1.1" 499 0 "Python-urllib/3.13" 4.609392ms
gateway berth: draining comfyui: NInfer-HauHauCS-27B (foreground) needs the card
gateway berth: ComfyUI queue empty; handing off
gateway berth: ComfyUI models unloaded; stopping the server
gateway msg="gateway engine exited" engine=comfyui code=143
berthd msg=granted lease=NInfer-HauHauCS-27B-2 waited=825ms
Python-urllib/3.13 is ComfyUI's interpreter, so the request that needed the card came from the server Berth stopped.
Cause
With keep_comfyui_up = false, a drain stops ComfyUI outright. I set that on 2026-10-18 so an idle ComfyUI stops holding 504 MiB of CUDA context beside a model. Berth treats a drain caused by an outside client the same as one caused by ComfyUI's own process. Under that setting, any ComfyUI node that calls a llama-swap model (QwenChat, prompt expanders) kills its own server.
Workaround
keep_comfyui_up = true. ComfyUI unloads its models through /free, stays up, and the chat request completes. You pay for it with the idle context the setting was meant to free.
Possible fixes
- Give ComfyUI nodes a way to ask Berth for a keep-up release before they call llama-swap, for example a berthd endpoint that releases the
comfyuilease with keep-up semantics. QwenChat would call it first. The drain that follows finds no ComfyUI lease and leaves the server alone. - Keep-up drain plus an idle stop: after a keep-up handoff, stop ComfyUI if it takes no new lease and runs no prompt for N minutes. That frees the 504 MiB later without killing in-flight work.
Either one lets keep_comfyui_up = false stay the default for outside requests.
Done when
- A QwenChat call to a llama-swap model from a leased ComfyUI returns a reply, and ComfyUI is still running afterwards.
- An outside llama-swap request that drains ComfyUI still frees its VRAM.
Activity
T-Crypt commentedon Oct 21, 2026
Fixed in #125. An engine entry now takes
park_ttl, its own park ttl in place of[park] ttl. For this setup:A drain from an outside llama-swap request unloads ComfyUI's models through
/freeand leaves the server up, so a node's chat request finishes. Ten minutes later Berth stops the server and frees the 504 MiB context. Nothing in it is specific to a node. The docs section is ComfyUI nodes that call other engines. It ships in the next release.