Skip to content

Silent prompt-conditioning corruption in reference-image edits after the 2026-10-18 update (Qwen-Image-2.1 + INT8-ConvRot Qwen3-VL text encoder, 8 GB GPU, dynamic VRAM) #17187

Open

Description

@Markgatcha

Summary

After updating to master 8f04fd6e (comfyui_version.py = 0.42.0, pulled by Comfy Desktop on 2026-10-18 19:14 EDT), my Qwen-Image-2.1 reference-image edit workflow started silently ignoring the positive prompt: instead of applying the requested edit, the output became a near-copy of the reference image. No error, no OOM, no crash — just wrong conditioning.

Reverting to bab80ef7 (0.41.0) immediately restores correct behaviour. comfy-kitchen 0.2.39 is ruled out — evidence below. This looks like the dynamic-VRAM prefetch / malloc-graph cleanup area.

Environment

  • Windows 11 (10.0.26220.9739), Comfy Desktop
  • NVIDIA GeForce RTX 5050 Laptop GPU, 8151 MiB, driver 617.42, compute capability 12.0
  • 15.7 GB RAM
  • Python 3.13.12, torch 2.14.1+cu130 (CUDA 13.0)
  • Launch args: --use-ck-attention --cache-lru 0 --enable-manager --enable-assets (dynamic VRAM enabled — no --disable-dynamic-vram)
  • Text encoder: qwen3vl_8b_int8_convrot.safetensors (INT8, int8_tensorwise + convrot, convrot_groupsize: 256) — 8916 MB staged
  • Diffusion model: qwen-image-2.1-turbo-Q5_0.gguf (GGUF, loaded in sd.cpp compatibility mode)
  • Working: bab80ef7 (0.41.0) + comfy-kitchen 0.2.38 + numpy 2.5.3 + frontend 1.55.17
  • Broken: 8f04fd6e (0.42.0) + comfy-kitchen 0.2.39 + numpy 2.5.4 + frontend 1.55.18

Symptom details

  • Runs on 0.41.0 before the update: 13 consecutive correct edits.
  • On 8f04fd6e: 3 runs fine, then 6/6 consecutive runs (21:09 → 23:24) failed — clothing/objects preserved exactly as in the reference, i.e. the edit was not applied at all.
  • Identical inputs give different results run-to-run: same seed 593103825223053, same reference image, same prompt → one run partially applied the edit (22:25), another applied nothing (23:24). Four other runs with different seeds produced near-identical copies of the reference (mean pixel delta 5–6/255).
  • Once in the bad state it stays bad until the ComfyUI process is restarted.
  • Sampler is the turbo preset (8 steps, cfg 1.0), so the prompt is the only guidance; with degraded conditioning the model simply re-renders the reference.

This looks like per-run corruption of the streamed text-encoder weights / conditioning rather than a deterministic model or prompt bug.

Log evidence (broken build)

Requested to load QwenImage21TEModel_
Model QwenImage21TEModel_ prepared for dynamic VRAM loading. 8916MB Staged. 0 patches attached. Force pre-loaded 255 weights: 849 KB.
Model QwenImage21TEModel_ prepared for dynamic VRAM loading. 8916MB Staged. 0 patches attached. Force pre-loaded 255 weights: 849 KB.   <-- re-prepared 22 s later, same prompt
loaded partially; 4462.31 MB usable, 4428.41 MB loaded, 407.00 MB offloaded, 33.00 MB buffer reserved, lowvram patches: 0
Using RAM pressure cache.
Comfy model compiler graph breaks: 2, rogues: 0
Warning: This gguf model file is loaded in compatibility mode 'sd.cpp' [arch:qwen_image21]
Using Comfy Kitchen attention

Only ~4.4–4.6 GB of the 8151 MiB is usable (the rest is held by other desktop apps), while the text encoder alone stages 8916 MB — roughly 2× the available VRAM. Partial staging / prefetch churn is therefore guaranteed on every queue.

Ruled out

1. comfy-kitchen 0.2.39 (the third-party package that was bumped in the same update) — not the cause:

  • v0.2.39 contains exactly one functional change: PR chained ksampler/ksampleradvanced nodes with random seed after every gen checked should generate the same seed #255 "Fuse input activations into the W4A8/W6A8 quantizer" — comfy_kitchen/backends/cuda/__init__.py (+63/−21) plus tests.
  • Downloading both wheels from PyPI and comparing the compiled extensions: _C.abi3.pyd is byte-identical except 6 bytes (3 in the PE COFF timestamp, 3 in .rdata). All 36.5 MB of code — .text and the 32 MB .nv_fatb/.nvFatBi sections holding the embedded SASS/PTX — match exactly, so no kernel changed between 0.2.38 and 0.2.39.
  • Python-side, the only changed functions are w4a8_int8_linear, w4a8_packed_linear, _w4a8_linear_chunked, _convrot_int8_fused_shared_memory_bytes plus two new w4a8 helpers. W4A8/W6A8 are 4/6-bit weight formats; my text encoder is INT8 int8_tensorwise + convrot and never reaches the w4a8_* code.
  • The changed shared-memory predictor only matters where it flips a "does the fused ConvRot INT8 kernel fit?" decision. Measured on this GPU via the CUDA driver API (no context created): MAX_SHARED_MEMORY_PER_BLOCK = 49152, MAX_SHARED_MEMORY_PER_BLOCK_OPTIN = 101376. For K=4096 the old estimate is 49152 B and the new one is 24576 B — both below 101376, so the kernel selection is identical on this device.

2. Not the prompt / model / reference / cache node / custom nodes:

  • Prompt text is unchanged across the failing runs (only ~2-character diffs), so it is not prompt wording.
  • Model files (qwen-image-2.1-turbo-Q5_0.gguf, text encoder) and the reference image have mtimes older than both a failing and a working run.
  • The QwenImage21Cache node is set to device: "off" (which forces the full-recompute path), and the failure occurs with it off.
  • No custom node file was modified in that window (no custom node updates were installed alongside the update).
  • numpy/frontend version bumps have no bearing on conditioning.

What actually changed in my update range (bab80ef7..8f04fd6e)

Only four commits touch anything on this workflow's code path:

commit subject effect
b3522383 Fix text generation not properly cleaning itself up (#17169) comfy/text_encoders/llama.py: adds finally: cleanup_prefetch_queues()
7db08fe5 model_prefetch: syncrhonize all graph cleanups (#17178) comfy/model_prefetch.py (+2 lines)
d4056d50 CPU device doesn't support fp8, nvfp4, mxfp8 or int8 (#17170) CPU-only guards
872eb6a3 add support for loading embeddings models onto Qwen3.5 (#17159) different text-encoder family

Everything else in the 19-file diff (naf.py, minimax/moss models, PDD detection, api nodes, webcam, feature flags) is not used by this workflow.

Note that my broken build already contains #17178's synchronize(), so that fix alone does not cover this case — and v0.42.1 (which is 0.42.0 + #17178) would not fix it either. I did not bisect further because the remaining candidates are all in the prefetch/compiler machinery that I would rather not guess at.

Related open items

Ask

  1. Is this silent-corruption variant expected to be covered by Fix cleanup after partial dynamic-VRAM prefetch failures #17177 / Fix MTPHead unpack crash in prefetch queue cleanup (CORE-644) #17180, or should I bisect further?
  2. Can this fail loudly instead of silently producing a corrupt edit? The outputs looked plausible; I only caught it by comparing against the reference.
  3. I can re-run the broken build with --disable-dynamic-vram --lowvram (text encoder on CPU) to confirm the dynamic-VRAM staging path is the trigger, and report back — say the word.

Workaround for others hitting this

Revert to the previous build (Comfy Desktop's rollback) — 0.41.0 is correct. Freeing the ~3.5 GB of this 8 GB card that other apps are holding (so the 8.9 GB text encoder is not ~2× oversubscribed) is also likely to help.

Happy to attach a minimal reproduction workflow on request — I left it out because it is a private user workflow.

Activity

  1. hiroki-abe-58 commented on Oct 21, 2026

    @hiroki-abe-58

    I tried to reproduce this on the same software stack. The build is master d52a7a5; the only commits after your 8f04fd6 are workflow-template bumps and a top-p sampling fix. Versions: comfy-aimdo 0.5.6, comfy-kitchen 0.2.39, torch 2.14.1+cu130. The text encoder is Comfy-Org's qwen3vl_8b_int8_convrot.safetensors, on Windows 11 with a Blackwell card (RTX 5090, memory limited as below).

    The graph is a two-reference Qwen-Image 2.1 edit at cfg 1, with QwenImage21Cache device off, queued repeatedly in one session. Every run re-encodes the same prompt and images; only the negative prompt changes, which is inert at cfg 1. A small test node hashes the positive conditioning tensors.

    DiT memory limit runs positive conditioning image
    Comfy-Org qwen_image_2.1_int8_convrot none 3 identical identical
    same another process holding 25.5 GiB (3.9 GiB free) 10 identical identical
    same --vram-headroom 27 --use-ck-attention (dynamic VRAM held to 3.56 GiB) 10 identical identical
    unsloth qwen-image-2.1-turbo-Q5_K_S.gguf via ComfyUI-GGUF, 8 steps --use-ck-attention --reserve-vram 25.5 --vram-headroom 27: the DiT is "loaded partially; 3257 MB usable, 3206 MB loaded, 1279 MB offloaded" and partially unloaded for the text encoder on every run, as in your log 10 identical identical
    same same + --cache-ram 40 63 10 identical identical

    The conditioning hash was also the same across all configurations, the unconstrained one included, and every edit was correct. So the INT8 text encoder under dynamic VRAM, with far less free VRAM than its 8.9 GB, did not reproduce this here.

    Things I could not match:

    • Your Q5_0 file. It loads in 'sd.cpp' compatibility mode; the unsloth file does not.
    • Genuinely low host RAM (15.7 GB).
    • The laptop GPU.

    The --disable-dynamic-vram run you offered would show whether the dynamic-VRAM path is involved at all.

  2. hiroki-abe-58 commented on Oct 21, 2026

    @hiroki-abe-58

    Follow-up: this looks like #16900, which has a fix in PR #16901. That race needs available RAM to fall below the pin budget. pin_memory then hands another weight's pin to the new weight while a host-to-device copy from that pin may still be queued. With 15.7 GB of RAM and the 8.9 GB text encoder streaming, your machine is in that state. PR #16901's own test is a stock Qwen-Image 2.1 two-image edit with the same symptom: same seed, different results, nothing logged.

    On my 63 GB machine, the runs above never reached that path: there were 0 _steal_pin calls. That would explain why they stayed identical. I then forced the pin budget to fail, as the PR's repro does, which gave 85 real steals over 10 runs. The 5090 still did not hit the race window, with or without the PR. The PR also added no time there: 8.1–8.2 s per run either way.

    Running your workflow with --disable-pinned-memory (the reference used in #16900), or with PR #16901 applied, would show whether this is the same bug.

  3. Mahone722 commented on Nov 14, 2026

    @Mahone722

    Adding an independent reproduction with a different signature — this one is deterministic, which I think rules the pin-race theory out on my machine.

    Environment: Windows 11, RTX 4060 Laptop (8 GB VRAM, 8187 MiB), 15.7 GB RAM, ComfyUI 0.45.2 (Comfy Desktop install). Models: qwen_image_2.1_turbo_int8_convrot.safetensors (7,256,787,829 bytes), qwen3vl_8b_int8_convrot.safetensors (9,350,798,360 bytes), qwen_image_2.1_vae_bf16.safetensors (675,509,688 bytes); byte counts match the Comfy-Org repo listing.

    Graph: UNETLoader -> QwenImage21Cache (auto/default) -> CFGGuider -> SamplerCustomAdvanced (euler, 8-step turbo ManualSigmas) -> VAEDecode -> SaveImage. TextEncodeQwenImage21 with vae connected and one images.image_1; the reference is a 1024x1024 RGB PNG.

    Symptom: identity, pose and framing come out accurate, but the whole frame is covered in broadband coloured speckle. The same weights doing text-to-image with no reference image are perfectly clean.

    Controls that do not change the noise

    Memory-path test: identical weights, reference and seed, against three servers — default, --disable-dynamic-vram, --disable-pinned-memory. All three outputs are bit-identical (mean abs diff 0.000/255, channel extrema (0,0)). A race would not reproduce byte-for-byte across different memory paths, so at least on this machine the failure looks deterministic rather than #16900-style.

    Where it seems to sit: disconnecting vae from TextEncodeQwenImage21 (keep_vision path, so the image conditions only through the text encoder) produces a pure noise texture instead of a noisy edit. That points at the image-token conditioning rather than the DiT/VAE/sampler, since text-to-image is clean.

    Also checked the encoder file itself: model.visual holds 351 tensors, all BF16, with zero _scale / comfy_quant entries — so INT8-ConvRot does not touch the vision tower in this file, which argues against quantization damage there.

    Happy to attach the minimal API-format graph JSON if it is useful.

  4. hiroki-abe-58 commented on Nov 14, 2026

    @hiroki-abe-58

    Thanks for the detailed controls, @Mahone722. Your reported bit-identical outputs across default, --disable-dynamic-vram, and --disable-pinned-memory give a useful distinction from the intermittent symptom discussed above. My earlier #16900/#16901 suggestion was a hypothesis for that earlier report, not a confirmed diagnosis; my RTX 5090 tests did not reproduce the race even with real pin steals.

    Please attach the minimal API-format graph JSON you offered, together with the exact ComfyUI core commit and a public or synthetic reference image and prompt that reproduce the speckle. Please remove private paths or inputs. Keeping the seed and full sigma schedule in the graph would make the comparison reproducible.

    The clean text-to-image result and the reference-path difference help narrow the investigation, but they do not yet establish which component is responsible. I have not run a new reproduction on your 0.45.2 setup.

    AI-assisted triage; the new results above are your reported observations, not independently verified tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions