Skip to content

bug (memory leak & OOM) New comfy aimdo (comfy compiler pr) conflicts with New spare attention node PR (from kijay) - MINIMAX H3 VIDEO GENERATION #16144

Closed

Description

@bhusann

Custom Node Testing

Expected Behavior

New version of Aimdo (0.5.2) is working great for me, it uses my vram with full utilization and i observed less disk activity during generation which is nice.

Actual Behavior

OOM and crash when i use this with new Blocksparse attention node from kijay (int 8 sol attention)

i guess its a memory leak. the aimdo records like cuda graph like thing and replays, but when we use it with this sparse attention node, i GUESS it sees this as different path and creates more branches for each step and then finally oom. but if i bypass Blocksparse attention node it works correcly

Steps to Reproduce

my specs : rtx 4060 (8gb vram) + 16gb ram.
latest comfy ui pulled from github. Torch 2.13 + cuda 13.0

launch command: taskset -c 0-12 /home/turbo/programs/ComfyUI/compyy/bin/python main.py
--listen
--port 8191
--preview-method none
--disable-auto-launch
--disable-api-nodes
--vram-headroom 1 --reserve-vram 1
--force-channels-last --use-sage-attention --lowvram

i use kijay's 4bit w4a8 fl2va minimax model + path sage attention node + model preview (without tiny vae) + Blocksparse attention. NO lora, NO extra speed up nodes used.

WITHOUT Blocksparse attention: even at 13th and 17th step its like 8gb full but it continues, NO oom occurs and actually finishs the generation.

Image Image

WITH BlockSparse attention:
at step 1 (spare attention still NO active)

Image

after step 5: ( sparse attention node activates) and the memory increases gradually and then OOMs

Image Image Image

Debug Logs

Main log: 
~/programs/ComfyUI master* 13m 37s
❯ taskset -c 0-12 /home/turbo/programs/ComfyUI/compyy/bin/python main.py \
          --listen \
          --port 8191 \
          --preview-method none \
          --disable-auto-launch \
          --disable-api-nodes \
          --vram-headroom 1 --reserve-vram 1 \
          --force-channels-last --use-sage-attention --lowvram
[INFO] setup plugin alembic.autogenerate.schemas
[INFO] setup plugin alembic.autogenerate.tables
[INFO] setup plugin alembic.autogenerate.types
[INFO] setup plugin alembic.autogenerate.constraints
[INFO] setup plugin alembic.autogenerate.defaults
[INFO] setup plugin alembic.autogenerate.comments
/home/turbo/programs/ComfyUI/compyy/lib/python3.14/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.6.3) or chardet (7.2.0)/charset_normalizer (3.4.6) doesn't match a supported version!
  warnings.warn(
[START] Security scan
[INFO] [ComfyUI-Manager] Using `uv` as Python module for pip operations.
Using Python 3.14.7 environment at: compyy
[DONE] Security scan
## ComfyUI-Manager: installing dependencies done.
** ComfyUI startup time: 2026-09-06 13:50:33.114
** Platform: Linux
** Python version: 3.14.7 (main, Aug 14 2026, 06:38:32) [GCC 16.2.1 20260810]
** Python executable: /home/turbo/programs/ComfyUI/compyy/bin/python
** ComfyUI Path: /home/turbo/programs/ComfyUI
** ComfyUI Base Folder Path: /home/turbo/programs/ComfyUI
** User directory: /home/turbo/programs/ComfyUI/user
** ComfyUI-Manager config path: /home/turbo/programs/ComfyUI/user/__manager/config.ini
** Log path: /home/turbo/programs/ComfyUI/user/comfyui.log
Using Python 3.14.7 environment at: compyy
Using Python 3.14.7 environment at: compyy
[INFO]
Prestartup times for custom nodes:
[INFO]    0.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/rgthree-comfy
[INFO]    1.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/comfyui-manager
[INFO]
[INFO] Found comfy_kitchen backend eager: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope1_', 'apply_rope_', 'apply_rope_split_half', 'apply_rope_split_half1', 'apply_rope_split_half1_', 'apply_rope_split_half_', 'convrot_w4a4_linear', 'dequantize_convrot_w4a4_weight', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_embedding', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_mxfp8', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'dequantize_w4a8_int8_weight', 'gemv_awq_w4a16', 'int8_linear', 'na3d', 'prepare_int4_weight_for_int8_linear', 'quantize_and_rotate_rowwise', 'quantize_convrot_w4a4_weight', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'quantize_w4a8_int8_weight', 'rms_adaln', 'rms_rope', 'rms_rope1', 'rms_rope1_', 'rms_rope_', 'rms_rope_split_half', 'rms_rope_split_half1', 'rms_rope_split_half1_', 'rms_rope_split_half_', 'rotate_int8_convrot_weight', 'scaled_mm_mxfp8', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'sol_attn', 'stochastic_rounding_fp8', 'w4a8_int8_linear']}
[INFO] Found comfy_kitchen backend cuda: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope1_', 'apply_rope_', 'apply_rope_split_half', 'apply_rope_split_half1', 'apply_rope_split_half1_', 'apply_rope_split_half_', 'convrot_w4a4_linear', 'dequantize_convrot_w4a4_weight', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'dequantize_w4a8_int8_weight', 'gemv_awq_w4a16', 'int8_linear', 'na3d', 'prepare_int4_weight_for_int8_linear', 'quantize_and_rotate_rowwise', 'quantize_convrot_w4a4_weight', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'quantize_w4a8_int8_weight', 'rms_adaln', 'rms_rope', 'rms_rope1', 'rms_rope1_', 'rms_rope_', 'rms_rope_split_half', 'rms_rope_split_half1', 'rms_rope_split_half1_', 'rms_rope_split_half_', 'rotate_int8_convrot_weight', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'sol_attn', 'stochastic_rounding_fp8', 'w4a8_int8_linear']}
[INFO] Found comfy_kitchen backend hip: {'available': False, 'disabled': False, 'unavailable_reason': 'PyTorch ROCm/HIP runtime not available', 'capabilities': []}
[INFO] Found comfy_kitchen backend triton: {'available': True, 'disabled': True, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope1_', 'apply_rope_', 'apply_rope_split_half', 'apply_rope_split_half1', 'apply_rope_split_half1_', 'apply_rope_split_half_', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'int8_linear', 'na3d', 'quantize_and_rotate_rowwise', 'quantize_int8_rowwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'rms_adaln', 'rms_rope', 'rms_rope1', 'rms_rope1_', 'rms_rope_', 'rms_rope_split_half', 'rms_rope_split_half1', 'rms_rope_split_half1_', 'rms_rope_split_half_', 'w4a8_int8_linear']}
[INFO] Checkpoint files will always be loaded safely.
[INFO] Total VRAM 7834 MB, total RAM 15687 MB
[INFO] pytorch version: 2.13.0+cu130
[INFO] Set vram state to: LOW_VRAM
[INFO] Device: cuda:0 NVIDIA GeForce RTX 4060 Laptop GPU : cudaMallocAsync
[INFO] Using async weight offloading with 2 streams
[INFO] Enabled pinned memory 12549.0
[INFO] Using sage attention
[INFO] aimdo: /project/src/control.c:276:INFO:comfy-aimdo inited for GPU: NVIDIA GeForce RTX 4060 Laptop GPU (VRAM: 7834 MB)
[INFO] DynamicVRAM support detected and enabled
[INFO] Python version: 3.14.7 (main, Aug 14 2026, 06:38:32) [GCC 16.2.1 20260810]
[INFO] ComfyUI version: 0.34.0
[INFO] comfy-aimdo version: 0.5.2
[INFO] comfy-kitchen version: 0.2.33
[INFO] comfyui-frontend-package version: 1.51.9
[INFO] comfyui-workflow-templates version: 0.11.55
[INFO] comfyui-embedded-docs version: 0.5.11
[INFO] comfy-kitchen version: 0.2.33
[INFO] comfy-aimdo version: 0.5.2
[INFO] [Prompt Server] web root: /home/turbo/programs/ComfyUI/compyy/lib/python3.14/site-packages/comfyui_frontend_package/static
[INFO] Asset seeder disabled
[INFO] No OpenGL_accelerate module loaded: Acceleration disabled
[INFO] [KJ PreviewOverride] --disable-api-nodes blocks blob: video, using WebP for animated previews.
[INFO] ### Loading: ComfyUI-Manager (V3.41)
[INFO] [ComfyUI-Manager] network_mode: public
[INFO] [ComfyUI-Manager] ComfyUI per-queue preview override detected (PR #11261). Manager's preview method feature is disabled. Use ComfyUI's --preview-method CLI option or 'Settings > Execution > Live preview method'.
[INFO] ### ComfyUI Version: v0.34.0-62-g15eb748b | Released on '2026-09-06'
[INFO] ComfyUI-Env-Manager v0.1.0 routes registered
[krea2edit] nodes v1.2.5 loaded
[INFO] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
[INFO] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json

[rgthree-comfy] Loaded 48 fantastic nodes. 🎉

[rgthree-comfy] ComfyUI's new Node 2.0 rendering may be incompatible with some rgthree-comfy nodes and features, breaking some rendering as well as losing the ability to access a node's properties (a vital part of many nodes). It also appears to run MUCH more slowly spiking CPU usage and causing jankiness and unresponsiveness, especially with large workflows. Personally I am not planning to use the new Nodes 2.0 and, unfortunately, am not able to invest the time to investigate and overhaul rgthree-comfy where needed. If you have issues when Nodes 2.0 is enabled, I'd urge you to switch it off as well and join me in hoping ComfyUI is not planning to deprecate the existing, stable canvas rendering all together.

[INFO] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
[INFO]
Import times for custom nodes:
[INFO]    0.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/websocket_image_save.py
[INFO]    0.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/image_router.py
[INFO]    0.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/ComfyUI-PromptBuilder
[INFO]    0.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/ComfyUI-MemoryVisualization
[INFO]    0.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/tiled.py
[INFO]    0.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/comfyui_memory_cleanup
[INFO]    0.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/SLcondition
[INFO]    0.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/ComfyUI-Inpaint-CropAndStitch
[INFO]    0.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/comfyui-krea2edit
[INFO]    0.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/ComfyUI-Env-Manager
[INFO]    0.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/ComfyUI-Simple_Readable_Metadata-SG
[INFO]    0.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/ComfyUI-CacheDiT
[INFO]    0.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/comfyui-custom-scripts-proper
[INFO]    0.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/ComfyUI-SolAttn_triton
[INFO]    0.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/rgthree-comfy
[INFO]    0.0 seconds: /home/turbo/programs/ComfyUI/custom_nodes/ComfyUI-KJNodes
[INFO]    0.1 seconds: /home/turbo/programs/ComfyUI/custom_nodes/gguf
[INFO]    0.1 seconds: /home/turbo/programs/ComfyUI/custom_nodes/comfyui-manager
[INFO]    0.1 seconds: /home/turbo/programs/ComfyUI/custom_nodes/ComfyUI-Spectrum-MiniMax-H3
[INFO]    0.3 seconds: /home/turbo/programs/ComfyUI/custom_nodes/comfyui_nvidia_rtx_nodes
[INFO]    0.5 seconds: /home/turbo/programs/ComfyUI/custom_nodes/comfyui-videohelpersuite-proper
[INFO]
[INFO] Context impl SQLiteImpl.
[INFO] Will assume non-transactional DDL.
[INFO] Using RAM pressure cache.
[INFO] Starting server

[INFO] To see the GUI go to: http://0.0.0.0:8191
[INFO] To see the GUI go to: http://[::]:8191





Without BlockSparse attention node: 

INFO] got prompt
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.float16
[INFO] Found quantization metadata version 1
[INFO] Using MixedPrecisionOps for text encoder
[INFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.float32
[INFO] Found quantization metadata version 1
[INFO] Detected mixed precision quantization
[INFO] Using mixed precision operations
[INFO] Native ops: convrot_w4a4, float8_e5m2, asym_w4a8_int8, int8_tensorwise, float8_e4m3fn , emulated ops: nvfp4, mxfp8
[INFO] model weight dtype torch.bfloat16, manual cast: torch.bfloat16
[INFO] model_type FLOW_AV
[INFO] Using sage attention mode: auto
[INFO] Requested to load MiniMaxH3TEModel_
[INFO] Model MiniMaxH3TEModel_ prepared for dynamic VRAM loading. 14956MB Staged. 0 patches attached. Force pre-loaded 410 weights: 4572 KB.
[INFO] Requested to load MiniMaxH3VideoVAE
[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.
VRAM清理完成 [卸载模型: True, 清空缓存: True]
[INFO] Requested to load MiniMaxH3
[INFO] 0 models unloaded.
[INFO] Model MiniMaxH3 prepared for dynamic VRAM loading. 11956MB Staged. 0 patches attached. Force pre-loaded 210 weights: 1175 KB.
  5%|████▉                                                                                              | 1/20 [01:15<23:56, 75.62s/it,  Model Initialization complete!  ][WARNING] [KJ PreviewOverride] tiny VAE decode failed, falling back: Allocation on device 0 would exceed allowed memory. (out of memory)
Currently allocated     : 1022.01 MiB
Requested               : 430.31 MiB
Device limit            : 7.65 GiB
Free (according to CUDA): 10.06 MiB
PyTorch limit (set by user-supplied memory fraction)
                        : 17179869184.00 GiB
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [24:09<00:00, 72.47s/it]
[INFO] Comfy model compiler graph breaks: 1
[INFO] Requested to load MiniMaxH3AudioVAE
[INFO] Model MiniMaxH3AudioVAE prepared for dynamic VRAM loading. 576MB Staged. 0 patches attached. Force pre-loaded 401 weights: 539 KB.
[INFO] Requested to load MiniMaxH3VideoVAE
[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.
[INFO] Prompt executed in 00:27:27
[INFO] Using RAM pressure cache.


With BlockSparse attention NODE: 
[INFO] got prompt
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.float16
[INFO] Found quantization metadata version 1
[INFO] Using MixedPrecisionOps for text encoder
[INFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.float32
[INFO] Found quantization metadata version 1
[INFO] Detected mixed precision quantization
[INFO] Using mixed precision operations
[INFO] Native ops: convrot_w4a4, float8_e5m2, asym_w4a8_int8, int8_tensorwise, float8_e4m3fn , emulated ops: nvfp4, mxfp8
[INFO] model weight dtype torch.bfloat16, manual cast: torch.bfloat16
[INFO] model_type FLOW_AV
[INFO] Using sage attention mode: auto
[INFO] Requested to load MiniMaxH3TEModel_
[INFO] Model MiniMaxH3TEModel_ prepared for dynamic VRAM loading. 14956MB Staged. 0 patches attached. Force pre-loaded 410 weights: 4572 KB.
[INFO] Requested to load MiniMaxH3VideoVAE
[INFO] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.
VRAM清理完成 [卸载模型: True, 清空缓存: True]
[INFO] Requested to load MiniMaxH3
[INFO] 0 models unloaded.
[INFO] Model MiniMaxH3 prepared for dynamic VRAM loading. 11956MB Staged. 0 patches attached. Force pre-loaded 210 weights: 1175 KB.
  0%|                                                                                                                  | 0/20 [00:00<?, ?it/s,   Model Initializing ...  ][INFO] BlockSparseAttention: dense (38968 tokens): sigma 1 outside the start/end window
  5%|████▉                                                                                              | 1/20 [01:13<23:12, 73.26s/it,  Model Initialization complete!  ][INFO] BlockSparseAttention: dense (38968 tokens): sigma 0.996 outside the start/end window
 10%|█████████████▍                                                                                                                        | 2/20 [02:24<21:44, 72.45s/it][INFO] BlockSparseAttention: dense (38968 tokens): sigma 0.991 outside the start/end window
 15%|████████████████████                                                                                                                  | 3/20 [03:37<20:29, 72.32s/it][INFO] BlockSparseAttention: dense (38968 tokens): sigma 0.986 outside the start/end window
 20%|██████████████████████████▊                                                                                                           | 4/20 [04:49<19:17, 72.36s/it][INFO] BlockSparseAttention: dense (38968 tokens): sigma 0.98 outside the start/end window
 25%|█████████████████████████████████▌                                                                                                    | 5/20 [06:01<18:04, 72.33s/it][INFO] BlockSparseAttention: sparse producer path: 38968 tokens, sinks (0, 36)/(22, 36)
 40%|█████████████████████████████████████████████████████▌                                                                                | 8/20 [08:58<13:27, 67.27s/it]
[INFO] Comfy model compiler graph breaks: 188
[ERROR] !!! Exception during processing !!! Allocation on device 0 would exceed allowed memory. (out of memory)
Currently allocated     : 1.05 GiB
Requested               : 1.66 MiB
Device limit            : 7.65 GiB
Free (according to CUDA): 15.75 MiB
PyTorch limit (set by user-supplied memory fraction)
                        : 17179869184.00 GiB
[ERROR] Traceback (most recent call last):
  File "/home/turbo/programs/ComfyUI/execution.py", line 545, in execute
    output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/execution.py", line 344, in get_output_data
    return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/execution.py", line 318, in _async_map_node_over_list
    await process_inputs(input_dict, i)
  File "/home/turbo/programs/ComfyUI/execution.py", line 306, in process_inputs
    result = f(**inputs)
  File "/home/turbo/programs/ComfyUI/comfy_api/internal/__init__.py", line 149, in wrapped_func
    return method(locked_class, **inputs)
  File "/home/turbo/programs/ComfyUI/comfy_api/latest/_io.py", line 2046, in EXECUTE_NORMALIZED
    to_return = cls.execute(*args, **kwargs)
  File "/home/turbo/programs/ComfyUI/comfy_extras/nodes_custom_sampler.py", line 1055, in execute
    samples = guider.sample(noise.generate_noise(latent), latent_image, sampler, sigmas, denoise_mask=noise_mask, callback=callback, disable_pbar=disable_pbar, seed=noise.seed)
  File "/home/turbo/programs/ComfyUI/comfy/samplers.py", line 1335, in sample
    output = executor.execute(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes)
  File "/home/turbo/programs/ComfyUI/comfy/patcher_extension.py", line 114, in execute
    return self.wrappers[self.idx](self, *args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/custom_nodes/ComfyUI-KJNodes/nodes/preview_override_node.py", line 685, in __call__
    return executor(noise, latent_image, sampler, sigmas, denoise_mask, new_callback, disable_pbar, seed, latent_shapes=latent_shapes)
  File "/home/turbo/programs/ComfyUI/comfy/patcher_extension.py", line 106, in __call__
    return new_executor.execute(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/comfy/patcher_extension.py", line 113, in execute
    return self.original(*args, **kwargs)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/comfy/samplers.py", line 1262, in outer_sample
    output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes)
  File "/home/turbo/programs/ComfyUI/comfy/samplers.py", line 1237, in inner_sample
    samples = executor.execute(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
  File "/home/turbo/programs/ComfyUI/comfy/patcher_extension.py", line 113, in execute
    return self.original(*args, **kwargs)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/comfy/samplers.py", line 1005, in sample
    samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
  File "/home/turbo/programs/ComfyUI/compyy/lib/python3.14/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
  File "/home/turbo/programs/ComfyUI/comfy/k_diffusion/sampling.py", line 1528, in sample_res_multistep
    return res_multistep(model, x, sigmas, extra_args=extra_args, callback=callback, disable=disable, s_noise=s_noise, noise_sampler=noise_sampler, eta=0., cfg_pp=False)
  File "/home/turbo/programs/ComfyUI/compyy/lib/python3.14/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
  File "/home/turbo/programs/ComfyUI/comfy/k_diffusion/sampling.py", line 1486, in res_multistep
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "/home/turbo/programs/ComfyUI/comfy/samplers.py", line 640, in __call__
    out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
  File "/home/turbo/programs/ComfyUI/comfy/samplers.py", line 1208, in __call__
    return self.outer_predict_noise(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/comfy/samplers.py", line 1215, in outer_predict_noise
    ).execute(x, timestep, model_options, seed)
      ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/comfy/patcher_extension.py", line 113, in execute
    return self.original(*args, **kwargs)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/comfy/samplers.py", line 1218, in predict_noise
    return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
  File "/home/turbo/programs/ComfyUI/comfy/samplers.py", line 620, in sampling_function
    out = calc_cond_batch(model, conds, x, timestep, model_options)
  File "/home/turbo/programs/ComfyUI/comfy/samplers.py", line 211, in calc_cond_batch
    return _calc_cond_batch_outer(model, conds, x_in, timestep, model_options)
  File "/home/turbo/programs/ComfyUI/comfy/samplers.py", line 219, in _calc_cond_batch_outer
    return executor.execute(model, conds, x_in, timestep, model_options)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/comfy/patcher_extension.py", line 113, in execute
    return self.original(*args, **kwargs)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/comfy/samplers.py", line 335, in _calc_cond_batch
    output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks)
             ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/comfy/model_base.py", line 208, in apply_model
    return comfy.patcher_extension.WrapperExecutor.new_class_executor(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    ...<2 lines>...
        comfy.patcher_extension.get_all_wrappers(comfy.patcher_extension.WrappersMP.APPLY_MODEL, transformer_options)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    ).execute(x, t, c_concat, c_crossattn, control, transformer_options, **kwargs)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/comfy/patcher_extension.py", line 113, in execute
    return self.original(*args, **kwargs)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/comfy/model_base.py", line 252, in _apply_model
    model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds)
  File "/home/turbo/programs/ComfyUI/compyy/lib/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/compyy/lib/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/turbo/programs/ComfyUI/comfy/ldm/minimax/model.py", line 577, in forward
    graph_out = comfy.patcher_extension.WrapperExecutor.new_class_executor(
    ...<2 lines>...
        comfy.patcher_extension.get_all_wrappers(comfy.patcher_extension.WrappersMP.DIFFUSION_MODEL, transformer_options)
    ).execute(x, timestep, context, transformer_options, minimax_payload=minimax_payload,
              denoise_mask=denoise_mask, audio_denoise_mask=audio_denoise_mask, **kwargs)
  File "/home/turbo/programs/ComfyUI/comfy/patcher_extension.py", line 113, in execute
    return self.original(*args, **kwargs)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/comfy/ldm/minimax/model.py", line 753, in _forward
    h = blocks_replace[("double_block", i)](
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        {"img": h, "t_emb": t_emb, "mod_segments": mod_segments, "rope_freqs": rope_freqs,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
         "layout": layout, "transformer_options": transformer_options},
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        {"original_block": block_wrap})["img"]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/comfy_extras/nodes_sparse_attention.py", line 303, in block_patch
    return extra["original_block"](args)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/turbo/programs/ComfyUI/comfy/ldm/minimax/model.py", line 751, in block_wrap
    return {"img": block(args["img"], args["t_emb"], args["mod_segments"], args["rope_freqs"],
                   ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                         transformer_options=args["transformer_options"], attention=args.get("attention"))}
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/compyy/lib/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/compyy/lib/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/turbo/programs/ComfyUI/comfy/ldm/minimax/model.py", line 293, in forward
    shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = self.adaln_proj(t_emb)
                                                                     ~~~~~~~~~~~~~~~^^^^^^^
  File "/home/turbo/programs/ComfyUI/compyy/lib/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/compyy/lib/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/turbo/programs/ComfyUI/comfy/ldm/minimax/model.py", line 225, in forward
    x = self.linear(nn.functional.silu(t_emb) if self.apply_silu else t_emb)
  File "/home/turbo/programs/ComfyUI/compyy/lib/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/compyy/lib/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/turbo/programs/ComfyUI/comfy/ops.py", line 1414, in forward
    output = self.forward_comfy_cast_weights(
        input,
    ...<2 lines>...
        weight_only_quant=weight_only_quant,
    )
  File "/home/turbo/programs/ComfyUI/comfy/ops.py", line 1338, in forward_comfy_cast_weights
    with CastBiasWeightContext(
         ~~~~~~~~~~~~~~~~~~~~~^
        self,
        ^^^^^
    ...<3 lines>...
        want_requant=want_requant,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    ) as (weight, bias):
    ^
  File "/home/turbo/programs/ComfyUI/comfy/ops.py", line 466, in __init__
    self.state = (None, None) if self.slf is None else cast_bias_weight(*args, **kwargs)
                                                       ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/comfy/ops.py", line 380, in cast_bias_weight
    weight, bias = resolve_cast_module_with_vbar(s, dtype, device, bias_dtype, compute_dtype, want_requant)
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/turbo/programs/ComfyUI/comfy/ops.py", line 277, in resolve_cast_module_with_vbar
    cast_dest = prefetch["cast_dest"] if prefetch["cast_dest"] is not None else torch.empty((comfy.memory_management.vram_aligned_size(prefetch["cast_geometry"]),), dtype=torch.uint8, device=device)
                                                                                ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: Allocation on device 0 would exceed allowed memory. (out of memory)
Currently allocated     : 1.05 GiB
Requested               : 1.66 MiB
Device limit            : 7.65 GiB
Free (according to CUDA): 15.75 MiB
PyTorch limit (set by user-supplied memory fraction)
                        : 17179869184.00 GiB

[INFO] Memory summary:
|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      | 209664 KiB |   1368 MiB |      0 B   |      0 B   |
|       from large pool |      0 KiB |      0 MiB |      0 B   |      0 B   |
|       from small pool |      0 KiB |      0 MiB |      0 B   |      0 B   |
|---------------------------------------------------------------------------|
| Active memory         | 209664 KiB |   1368 MiB |      0 B   |      0 B   |
|       from large pool |      0 KiB |      0 MiB |      0 B   |      0 B   |
|       from small pool |      0 KiB |      0 MiB |      0 B   |      0 B   |
|---------------------------------------------------------------------------|
| Requested memory      |      0 B   |      0 B   |      0 B   |      0 B   |
|       from large pool |      0 B   |      0 B   |      0 B   |      0 B   |
|       from small pool |      0 B   |      0 B   |      0 B   |      0 B   |
|---------------------------------------------------------------------------|
| GPU reserved memory   | 229376 KiB |   1696 MiB |      0 B   |      0 B   |
|       from large pool |      0 KiB |      0 MiB |      0 B   |      0 B   |
|       from small pool |      0 KiB |      0 MiB |      0 B   |      0 B   |
|---------------------------------------------------------------------------|
| Non-releasable memory |      0 B   |      0 B   |      0 B   |      0 B   |
|       from large pool |      0 B   |      0 B   |      0 B   |      0 B   |
|       from small pool |      0 B   |      0 B   |      0 B   |      0 B   |
|---------------------------------------------------------------------------|
| Allocations           |       0    |       0    |       0    |       0    |
|       from large pool |       0    |       0    |       0    |       0    |
|       from small pool |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Active allocs         |       0    |       0    |       0    |       0    |
|       from large pool |       0    |       0    |       0    |       0    |
|       from small pool |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| GPU reserved segments |       0    |       0    |       0    |       0    |
|       from large pool |       0    |       0    |       0    |       0    |
|       from small pool |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |       0    |       0    |       0    |       0    |
|       from large pool |       0    |       0    |       0    |       0    |
|       from small pool |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Oversize allocations  |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Oversize GPU segments |       0    |       0    |       0    |       0    |
|===========================================================================|

[ERROR] Got an OOM, unloading all loaded models.
[INFO] Prompt executed in 582.40 seconds
[INFO] Using RAM pressure cache.

sparse attention works it reduces 72s per step to 44 seconds per step but a minor conflict issue or memory leak causes it to OOM

i have test it 3 times with same launch commands and clearing cache.

Other

i am not a developer, take my assumption with grain of salt, but this is the problem i face, i try this by node before official mergeing had same issue exists and after the official merge today the problem exists.

aimdo latest version is not a problem its working but when its combined with block sparse attention its memory leak and oom (thats my guess).

Image

Activity

  1. added
    Potential BugUser is reporting a bug. This should be tested.
    on Sep 6, 2026
  2. changed the title [-]bug (memory leak & OOM) New comfy aimdo (comfy compiler pr) conflicts with New spare attention node PR (from kijay)[/-] [+]bug (memory leak & OOM) New comfy aimdo (comfy compiler pr) conflicts with New spare attention node PR (from kijay) - MINIMAX MODEL[/+] on Sep 6, 2026
  3. changed the title [-]bug (memory leak & OOM) New comfy aimdo (comfy compiler pr) conflicts with New spare attention node PR (from kijay) - MINIMAX MODEL[/-] [+]bug (memory leak & OOM) New comfy aimdo (comfy compiler pr) conflicts with New spare attention node PR (from kijay) - MINIMAX H3 VIDEO GENERATION[/+] on Sep 6, 2026
  4. LukeG89 commented on Sep 6, 2026

    @LukeG89

    Ping @rattus128

    Also having this issue, when the Sparse Attention node starts its job, the memory utilization increases. Also tried to make the node starts at the beginning (start_percent=0) to test if the memory estimation might work better, but the memory utilization increased over time and after a few steps got OOM as well.

    Did the same test with comfy compiler disabled (using the arg --disable-comfy-compiler) and the memory is heavely underutilized in comparison (5.7GB out of 8GB) but the speed generation seems quite similar and no OOM. No issues with start_percent=0.2 as well, no memory leak in both cases.

    In all my tests I've also enabled comfy kitchen attention with the ModelAttentionBackend node and also used MiniMax H3 Chunk FeedForward from KJNodes.

  5. self-assigned this
    on Sep 6, 2026
  6. comfyui-wiki commented on Sep 6, 2026

    @comfyui-wiki
    Member

    Not sure if it's related; another user also reported a similar issue #16150

  7. bhusann commented on Sep 6, 2026

    @bhusann
    Author

    Not sure if it's related; another user also reported a similar issue #16150

    #16150 (comment)

  8. bhusann commented on Sep 6, 2026

    @bhusann
    Author

    #16148

    This PR fixed the issue

  9. Djanghost commented on Sep 7, 2026

    @Djanghost

    Ping @rattus128

    Also having this issue, when the Sparse Attention node starts its job, the memory utilization increases. Also tried to make the node starts at the beginning (start_percent=0) to test if the memory estimation might work better, but the memory utilization increased over time and after a few steps got OOM as well.

    Did the same test with comfy compiler disabled (using the arg --disable-comfy-compiler) and the memory is heavely underutilized in comparison (5.7GB out of 8GB) but the speed generation seems quite similar and no OOM. No issues with start_percent=0.2 as well, no memory leak in both cases.

    In all my tests I've also enabled comfy kitchen attention with the ModelAttentionBackend node and also used MiniMax H3 Chunk FeedForward from KJNodes.

    @rattus128 this is still not fixed unfortunatly. VRAM usage isn't as good as it used to be just before the Kijai's PR was merged. It quickly runs OOM when you adjust the “start” / ‘end’ values of the “Block Sparse Attention” node. The same render at only 277 frames ends up with an OOM with my setup if I set the “end” to 0.9, for example, but it will complete if I set it to 1. I personally use Sol-Attn, RTX 4090 and 128 GB of RAM. I also get the nodes comfy kitchen attention with the ModelAttentionBackend node and MiniMax H3 Chunk FeedForward like @LukeG89

  10. Djanghost commented on Sep 7, 2026

    @Djanghost

    Details:
    The graph-break flood is still reproducible after a99d1f9, on a 24 GB card — so this isn't limited to low-VRAM setups.

    Setup: Windows, RTX 4090 24 GB, BlockSparseAttention, ComfyUI at master including a99d1f9, comfy-kitchen 0.2.33, comfy-aimdo 0.5.2.

    My workflow samples twice at different token counts, so a second VSA plan is built mid-run. That is exactly the case CodeRabbit flagged during review of #16148 and which was merged unaddressed: patch.vsa_rope_freqs() allocates and caches a padded tensor, and the call sits outside pause_malloc_graph(). A new (rope_freqs, plan) pair therefore still produces a rogue allocation.

    As merged: 156 and 199 graph breaks, followed by an OOM requesting 1.36 GiB with 2.82 GiB free — fragmentation, not exhaustion. Reducing the token count of the first sampling brought it down to 15 and 13 and the run completed, which pointed at the plan pair rather than raw load.

    Applying the reviewer's suggestion — moving freqs = patch.vsa_rope_freqs(rope_freqs, plan) inside the pause_malloc_graph() scope, right after n = plan["n"], and removing it from the if patch.vsa: block below:

    12 and 9 graph breaks, at a higher token count than the run that failed. No leak, run completes.

    So the remaining path is real and the one-line move fixes it in practice. Happy to test a proper patch if one lands.

  11. SveSop commented on Sep 7, 2026

    @SveSop

    Atleast its not only me. Struggled with this all day yesterday after updating to comfy-aimdo=0.5.2 + the comfy-compiler. VRAM usage is a lot higher - do not mind that tho, as i would rather use 98% than 80%, (4070Ti Super), BUT, it seems that there is some strange spillover/leak that causes at times things to grind to a complete halt (120+ s/it).

    I had ZERO issues like this prior to aimdo 0.5.2 (was on 0.4.15).

    Using: --disable-comfy-compiler i am using < 88% as opposed to > 98% and occasional OOM/stalling issues.

  12. jackytop commented on Sep 10, 2026

    @jackytop

    still having same problem. crashing at almost last step.
    Latest update.

  13. lonelystarCX commented on Oct 22, 2026

    @lonelystarCX

    Related data point for the "graph breaks jump when the sparse-attention node is in the graph" observation here.

    On a MiniMax-H3 graph with a block-sparse attention patch (Spark-H3) plus a 4-tile tiled high-res pass, we see the same shape — graph breaks climbing at forward boundaries, rogue counter stepping 48 → 54 → 104 → 154 — but in our case it does not end in OOM, it ends in RuntimeError: aimdo memory compile error from malloc_graph_pause().

    Why: malloc_graph_pop() severs every allocation still alive in the scope being popped and registers it as a rogue candidate; those are only adopted at the root pop by finalize_rogues(). If the application frees one of them earlier, graph_failed() sets g->failed permanently and every later API call returns false. So "breaks/rogues climbing" and "the graph refuses to accept work" are two ends of the same severing mechanic.

    Details, evidence and a patch: Comfy-Org/comfy-aimdo#129 (PR Comfy-Org/comfy-aimdo#130). If that direction lands it would be interesting to see whether the 188-break case still OOMs or just completes — the early-free poisoning may be masking the real break-count behaviour.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Potential BugUser is reporting a bug. This should be tested.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions