Repository navigation
park_ttl: an engine's own park ttl, so ComfyUI nodes that call other engines finish - #125
Merged
Merged
Conversation
A ComfyUI node can call another engine (a chat panel calling a llama-swap model), and that call can drain ComfyUI's own lease. With keep_comfyui_up off, the drain stops the server and kills the request that asked for the card. With it on, ComfyUI stays up and its CUDA context stays on the card for the node's [park] ttl, 30 minutes by default. park_ttl on an engine entry replaces [park] ttl for that engine. Keep-up plus a short park_ttl lets the node's request finish and frees the context minutes later. It rides the acquire and re-attach messages like min_turn. Fixes #121.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #121.
A ComfyUI node can do work outside ComfyUI's queue. In #121 a chat panel inside ComfyUI called a llama-swap model, and that call drained ComfyUI's own lease. With
keep_comfyui_upoff, the drain stopped the server and killed the request that had asked for the card. With it on, ComfyUI stays up and the request finishes, but its 504 MiB CUDA context stays on the card for[park] ttl, which is a node-wide setting, 30 minutes by default.park_ttlon an[[engine]]entry replaces[park] ttlfor that engine. Keep-up plus a shortpark_ttllets the node's request finish and frees the context minutes later. Nothing in it knows about ComfyUI nodes. It works for any parked engine, and an entry without it behaves as before.park.ParkedgainsTTL, andEvictuses it when it is set.park_ttl. It rides OpAcquire and OpReattach asmin_turndoes, so a re-attach keeps it, and gateway starts pick it up fromberth.tomlthroughberth run.berth config checkrefuses a negativepark_ttl.docs/gateway.mdgains a section on ComfyUI nodes that call other engines, with the config to use.contrib/INTEGRATION.mdpoints to it.Proof:
TestEvict(an engine's own ttl beats the node's, both ways) andTestParkedEngineOwnTTLfail withEvictignoring the per-engine ttl and pass with it. Config and options tests cover the key.go test ./..., Windows build and vet, golangci-lint (0 issues).mkdocs build --strictpasses.