Skip to content

Add FLUX 3 Image Klein support via ComfyUI CUDA #58

Open

Description

@pirumpi

Proposal

Research date: 2026-10-16. Reviewed Tapioca main 6a1e2b80bf4fd3420ea6b7b9838b876d7a05e632.

Add optional FLUX 3 Image Klein distilled INT8 image generation and editing via an isolated managed ComfyUI CUDA image adapter. This is an integration proposal, not existing Tapioca support. No model weights were downloaded or executed in this audit.

Identity and upstream evidence

Released October 15: BFL announcement, official model card, official ComfyUI package.

5.6B image transformer plus a separate Qwen3-VL-4B text encoder and Flux2 VAE. ComfyUI integration #17064 implements five early blocks per input mode followed by 28 joint blocks; distilled checkpoints use flow-map target-time conditioning. Use Euler, 16 steps, CFG 1 for the distilled variant, not the base model's sampling settings. Prompt expansion must produce a full scene caption; editing expansion must see the reference image.

Exact artifacts and access

Official Hub snapshot: black-forest-labs/flux-3-image-klein-comfy at 5c9b714851dc072b04a23ef6d2d36421f5ccb3a0.

Path Bytes Hub-reported SHA-256
diffusion_models/flux3_image_klein_sd_int8_convrot.safetensors 5827874779 847d8933548afd93df4af2cac2498e70891bf28d9dae06d695f536992e168fe2
text_encoders/qwen3vl_4b_int8_convrot.safetensors 4858559240 32de113cbf8282b02cbff0794b637b9d6676da09314a1db5a30bcea9bf4ee01c
vae/flux2-vae.safetensors 336211292 868fe7b343cc8f3a19dbcfcafbc3d5f888802be3f89bd81b65b3621a066ce8f3
text_encoders/qwen3.5_9b_flux3_prompt_upsampler_int8_convrot.safetensors 9748113996 cb8cb9b9dcd2569509de755dca54d96fd93a16a7aabfd2f4051f0a144230620f

Approximately 20.77 GB for this four-file selection. Also preserve the snapshot's LICENSE.md, notices and reviewed workflow assets. Metadata comes from the public Hub API; full hashes still require verification after authorized download. The repository is auto-gated and requires user acceptance/contact sharing plus authenticated downloads. Anonymous raw-file requests returned 401; no gate was accepted or bypassed.

License representation

FLUX Kommunity License v1.0, not Apache/MIT for the transformer. It permits specified noncommercial use and commercial output generation for qualifying users below $5M aggregate annualized revenue including affiliates, subject to all terms. It is not blanket commercial redistribution/fine-tuning permission. Section 2(e) requires content filtering or output review before dissemination, plus legally required AI disclosures. Preserve license and attribution; show restrictions and require acknowledgement/eligibility. Do not bundle or rehost weights. Qwen encoder/upsampler have separate Apache-2.0 notices. Make any non-qualifying use require the user's separate BFL agreement. License compliance is a shipping acceptance gate, not satisfied merely by a catalog URL.

Runtime and platform boundary

ComfyUI v0.41.0, commit 98cc0bf84b7cb1383654c5b402e7612d875d6d22, contains #17064 (20b46065ef7f34fa21e8abebe8d0cca90a904d6d). Ancestry checked: integration is 63 commits after Tapioca's current ComfyUI pin b0b743566f65daafc423b4fea8a2fbda94b3384a; release is two commits after integration.

Tapioca currently manages ComfyUI in video runtime. An image adapter and isolated cache identity are needed; a catalog addition alone will not work. Do not replace the H3 runtime silently. Diffusers 0.41.0 is not established as a loader for this new model; its official example uses newer mainline modular code.

Initial qualification: Windows/Linux NVIDIA CUDA. Do not advertise CPU, Apple Metal/MPS/MLX, Vulkan, DirectML, ONNX or llama.cpp support. NVFP4/Blackwell and base-training variants are outside initial scope.

Hardware

The official Comfy package documents INT8 on 8GB RTX 3070-class GPUs with offload; that is publisher guidance, not a Tapioca measurement. Reference-code BF16/FP8 memory figures are a different execution path. Qualify 8/12/24GB CUDA tiers at 1024 pixels with staged model residency, 32/64GB host-RAM test targets, and measure peak memory including prompt expansion. These RAM values are proposed test targets, not proven minimums. Do not promise 4K or ten-reference operation based on a 1MP test.

Implementation and acceptance

  1. Add an isolated, integrity-pinned ComfyUI image runtime and backend routing. Pin kernel/dependency versions, workflow JSON and all assets; preserve existing H3 and image models.
  2. Implement authenticated four-artifact installation, eligibility/acknowledgement UI, notices and filtering/review requirements. Test rejected access, expired token, interrupted/resumed download, corruption and offline reuse.
  3. Start with T2I and one-reference editing. Include local prompt expansion with visible expanded captions; never silently call a cloud LLM. Editing templates need a separate image-aware expansion step. Reject unsupported multi-reference input until qualified.
  4. Test workflow wiring, distilled sampler/time schedule, seed/dimensions, conditioning, progress, cancellation, output paths and clear OOM errors.
  5. Run real fixed-seed CUDA generation/editing against official workflow results; inspect nonblank finite output, requested edit, dimensions and latency/memory on each enabled OS/hardware tier. Test fresh install and cache upgrades.
  6. Run catalog validation/checksum, model-watch tests, catalog/adapter tests and affected runtime/desktop tests. Publish catalog/support claims only after these pass.

Risks and non-authoritative field evidence

New shared kernels and memory-management code, prompt expansion overhead, changed content outside the edit region, gated access and custom license obligations.

Reddit release discussion and 4070S editing comparison informed discovery and test selection. The latter reports 1MP/16-step INT8 generation around ten seconds; it is anecdotal, not a Tapioca benchmark.

Deduplication: searched all open/closed issues, Flux/FLUX 3/Klein/ComfyUI and the proposed title; inspected historical #29 and separate Ming integration #45. No existing issue tracks this model. No product files, runtime pins or defaults changed.

Activity

  1. pirumpi commented on Oct 18, 2026

    @pirumpi
    ContributorAuthor

    Material compatibility update — 2026-10-18

    The proposed INT8 prompt-upsample path now has a concrete upstream CUDA stability risk that must be included in qualification: ComfyUI #17115 remains open.

    • The initial report uses the same qwen3.5_9b_flux3_prompt_upsampler_int8_convrot.safetensors selected above on Windows / RTX 3090 Ti (sm_86), ComfyUI 0.41.0, torch 2.13.0+cu130. TextGenerate with mtp=auto reportedly crashes with illegal memory access; mtp=off reportedly completes. A second exception during cleanup can kill prompt_worker while HTTP remains responsive, leaving subsequent jobs pending.
    • New Ampere evidence reports a similar failure after the MTP rewrite #17007, on a different MOSS Qwen3.5 W4A8 encoder. Disabling CUDA graphs alone did not help; disabling the Comfy compiler did. Do not assume a newer commit or W4A8 conversion automatically resolves the risk.
    • Blackwell counter-evidence reports four successful runs with the exact FLUX INT8 encoder on RTX 5090/current master. That does not qualify Ampere or prove a fix.
    • Later comments report MOSS cancellation crashes and dynamic-VRAM workarounds. These are upstream field reports, not failures independently reproduced in Tapioca. No product weights were downloaded or executed in today's audit.

    Required additions to implementation/acceptance scope

    1. Set and regression-test the prompt upsampler's MTP policy explicitly. Treat mtp=off as a workaround to qualify, not a universal fix; test compiler/dynamic-VRAM settings on each supported GPU tier.
    2. Exercise the exact pinned INT8 artifacts on Windows and Linux Ampere as well as newer CUDA hardware, including repeated requests, prompt expansion, cancellation during expansion/sampling, and the next queued request after failure.
    3. Detect a dead/stalled Comfy worker independently of HTTP liveness. Fail pending work clearly and recycle the isolated runtime after an unrecoverable CUDA error; do not leave jobs pending forever.
    4. Record peak memory and all torch/CUDA/kernel/compiler versions. Keep platform support disabled until that platform passes; do not advertise the 5090 result as 3090 qualification.

    Also found a Reddit announcement about revised FLUX templates/v0.41.3. Treat it as discovery only: the web-rendered GitHub tag page exists, but the GitHub release API returned 404 for v0.41.3 and still listed v0.41.0 as latest during this audit. Resolve the exact tag/commit and workflow assets before changing pins; no release upgrade is asserted here.

    Deduplication: searched all open/closed Tapioca issues for FLUX, ComfyUI/MTP, CUDA illegal memory and prompt_worker, and inspected this issue. This changes #58's validation scope; it is not a separate Tapioca bug claim or a routine daily-status comment. No code/catalog/runtime changes.

  2. pirumpi commented on Oct 19, 2026

    @pirumpi
    ContributorAuthor

    Material readiness update — 2026-10-19

    Two verified upstream changes affect this integration's qualification plan. No support claim or product change is made here.

    CUDA: a released fix is now available to retest

    ComfyUI v0.42.1, published October 19, resolves to 63be2f9b82ad0dac88a7ceaa28a9cf3154125a18. Its release notes describe fixes for MOSS template crashes. The v0.42.0…v0.42.1 diff adds synchronization before dropping inactive malloc graphs in comfy/model_prefetch.py (release-branch fix 3b0e87ffe94ef8e6782f343fb7d89b4f51b42dfc). Upstream PR #17178 explicitly links #17115.

    This is a candidate fix, not confirmation that Ampere failures are resolved: #17115 is still open, the PR author says they never reproduced the issue, and listed regressions were on Linux RTX 5090 / 96 GB host RAM. Retain yesterday's exact-INT8, Windows/Linux Ampere, cancellation, repeated-request and dead-worker recovery tests. Qualify an isolated image runtime containing this fix instead of blindly upgrading the existing H3 environment.

    Apple silicon: a separate upstream implementation now exists

    MFlux v.0.23.0, commit 63cec2b005bbb255447e0b42521c5cc6d6d5121c, contains FLUX 3 port #878. It provides Flux3Klein / Flux3KleinEdit, dedicated generation/edit CLI commands, 4/6/8-bit quantization, 16-step distilled inference and up to ten references. This changes the earlier absence of an established Mac path, but does not mean Tapioca supports it: current main's MFlux requirement is mflux>=0.16,<0.20.

    MFlux's revision map pins distilled black-forest-labs/flux-3-image-klein at 8454017f9ebaada30c998fccd9f67a80c7580f68. The Hub API confirms these exact files at that snapshot (not the Comfy single-file bundle):

    File Bytes Hub SHA-256
    transformer/diffusion_pytorch_model-00001-of-00002.safetensors 9907529352 4f01de41470090586d7a25aa4aec93987724eae45db45b68de2b3f566f438a8b
    transformer/diffusion_pytorch_model-00002-of-00002.safetensors 1189878064 6f6217ab2c7ce5ef177ec202f79d4c2ddab78111c4645030c8828ec615d5f02c
    text_encoder/model-00001-of-00002.safetensors 4967229296 30a01a0556622645a3cce87b655bbbbbc1f170c196099f1b666c93202c3339a9
    text_encoder/model-00002-of-00002.safetensors 3908490048 046296a2a387efb43b0c997d5833c789604d168834f6e0d3064bf7bb13d002a6
    vae/diffusion_pytorch_model.safetensors 336213556 d64f3a68e1cc4f9f4e29b6e0da38a0204fe9a49f2d4053f0ec1fa1ca02f9c4b5

    Also inventory the two shard indexes, component configs, tokenizer files/chat template, license and notices from the same revision. Roughly 20.31 GB of tensor files must be acquired before local quantization; a 5 GB saved Q4 model does not mean a 5 GB initial download. Access remains auto-gated and FLUX Kommunity v1.0 obligations still apply; Apache-2.0 applies separately to the Qwen text encoder. No gate was accepted and no weights were downloaded.

    Upstream hardware evidence: on an M4 Pro with 64 GB unified memory, 1024² distilled inference reports peak memory 18.5 GB Q4, 20.6 GB Q6, 22.8 GB Q8, 30.7 GB BF16. These are upstream measurements, not minimum-RAM guarantees or Tapioca tests. Do not recommend a 16 GB Mac based on these figures. The small golden tests do not qualify ten-reference or high-resolution use.

    Important gaps: MFlux does not port BFL's prompt upsampler or safety classifier. Start any Mac qualification with explicit full-scene captions and one-reference editing; represent the missing prompt enhancement clearly, preserve the license's output-review/filtering requirements, and never substitute a cloud LLM silently. Add backend routing, isolated versioned cache, authenticated full-snapshot download and quantize/save/reload tests; reject unsupported negative prompts and handle distilled guidance accurately. Run existing FLUX2/Krea regressions before altering their shared MFlux environment. CPU/Windows/Linux Vulkan, DirectML and ONNX are not established by this Mac implementation.

    Recommendation: retain this single model issue, first retest the released CUDA cleanup fix, and consider Apple-silicon qualification as an optional separate implementation phase. Do not mark either path supported until its own artifact, license, installation and real-generation tests pass.

    Deduplication: searched all open/closed Tapioca issues for FLUX, MFlux, MOSS and ComfyUI cleanup; inspected #58 and related #34/#45/#48. No duplicate issue created. The Reddit reports linked in the original issue and October 18 update remain non-authoritative field evidence; conclusions above come from upstream code, release notes and Hub metadata.

  3. pirumpi commented on Oct 24, 2026

    @pirumpi
    ContributorAuthor

    Material backend-option update — 2026-10-24

    The previously mainline-only Diffusers FLUX 3 path now has a stable release: v0.42.0, published October 23, resolves to 95f9b8925c34469ccac829866d7ad4e3fca1bd89. Its tagged FLUX 3 module exports Flux3ModularPipeline and Flux3DistilledModularPipeline. Release notes describe integrated prompt upsampling, reference-image editing and BFL FP8r/NVFP4 loading.

    This creates a released alternative to evaluate, not a reason to silently replace this issue's ComfyUI INT8 plan or assert support for its four-file bundle. Tapioca main still pins Diffusers 0.41.0. The new path uses ModularPipeline.from_pretrained(...) plus load_components(...), not a drop-in ordinary pipeline call. It requires separate wrapper/component/offload and artifact-layout qualification.

    If selecting this alternative, first inventory and pin its exact component snapshot, tokenizer/configs, upsampler and quantization dependencies; do not assume Comfy INT8 ConvRot artifacts load through it. 0.42.0 requires Python >=3.11 and huggingface-hub >=1.32,<3, so use an isolated versioned runtime and fresh/upgrade installation tests. Existing gating, FLUX Kommunity obligations, full-scene/image-aware prompt expansion and actual CUDA memory/output qualification remain mandatory. This does not qualify MPS, DirectML, Vulkan or ONNX, nor resolve the separate Comfy CUDA stability tests already listed.

    Searched all open/closed issues for FLUX, Diffusers and the runtime requirement, and inspected this issue's existing updates. Keeping this as one model integration issue, with no duplicate or automatic backend decision. No code, pins, catalog entries or weights were changed; no inference qualification is claimed. This update relies on official release/tagged source rather than Reddit performance claims.

  4. pirumpi commented on Nov 4, 2026

    @pirumpi
    ContributorAuthor

    November 4, 2026 — ComfyUI 0.44.0 fixes a cleanup error that could hide the original exception when a Qwen3.5 MTP group remained in the prefetch queue. The RTX 3090 Ti illegal-access report is still open. Release · Cleanup fix · Ampere report

    For the FLUX 3 smoke test, run qwen3.5_9b_flux3_prompt_upsampler_int8_convrot.safetensors on an Ampere GPU with MTP disabled and on auto. Interrupt a job in each mode, check that the logs preserve the original error, then submit a second job to see whether the worker recovers. Tapioca hardware verification is still pending.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions