Skip to content

MiniMax H3: token refiner gets fp32 q/k beside a bf16 v (sage refuses it), and preprocess_text_embeds drops transformer_options #17713

Open

Description

@aksugat

MiniMax H3: the token refiner hands attention fp32 q/k beside a bf16 v, and preprocess_text_embeds drops transformer_options

ComfyUI 0.45.0 (a592c0a65), torch 2.16.0a0+rocm10.3.0a20261110 (gfx1101), sageattention 2.2.0, launched with --use-sage-attention. Both points below are still true on master (c5751c8).

Symptom

Sampling a MiniMax H3 image model logs this once to four times per image, and those calls run on pytorch attention instead of sage:

Error running sage attention: All tensors must have the same dtype., using pytorch attention instead.

Instrumenting the except in attention_sage:

q=torch.float32 k=torch.float32 v=torch.bfloat16
shapes (1, 56, 10, 128) and (1, 56, 1, 128), tensor_layout HND
transformer_options = {}          # empty
stack: model_base.apply_model -> TokenRefiner.forward (comfy/ldm/minimax/model.py:298)
       -> RefinerBlock.forward (:283) -> Attention.forward (:207) -> wrap_attn

On some shapes ComfyUI's own fallback raises as well instead of degrading — attention_pytorch calls F.scaled_dot_product_attention, which needs one dtype too:

Expected query, key, and value to have the same dtype, but got query.dtype: float ... value.dtype: c10::BFloat16

Two separate causes

  1. transformer_options never reach the refiner. comfy/model_base.py calls diffusion_model.preprocess_text_embeds(...) without them (the call is where kwargs is in scope), and preprocess_text_embeds (comfy/ldm/minimax/model.py:552) calls self.token_refiner(...) with no options either, so Attention gets the default {}. No attention override, block patch or hook applies inside the refiner — including the per-run attention overrides trainers install.

  2. The no-rope branch leaves q/k in the norms' dtype. Attention.forward's else branch (:201-202) does q = self.q_norm(...), k = self.k_norm(...) while v (:184) comes straight off the qkv split. When the norm weight is fp32 (the refiner is built in fp32 by sd's model detection in some paths), F.rms_norm promotes and q/k are fp32 beside a bf16 v. The rope branch keeps q's dtype because it is in place, which is why only the refiner is affected.

Suggested fixes

  • Thread transformer_options through preprocess_text_embeds and its comfy/model_base.py callers (it is available at each call site). That alone lets existing overrides fix it, and is right regardless.
  • Independently, keep q/k in the activation dtype in the no-rope branch (cast the norm output, or build the norms in the activation dtype), so the model does not depend on an override to be sampleable.

How it was found

A training run surfaced it in its sample renders; it reproduces in plain sampling of the same official checkpoint with --use-sage-attention on ROCm. A downstream workaround (cast q/k to v's dtype at the model's optimized_attention entry point) is in Nynxz/ZenTrainer#73 commit 3 — a workaround for this, not the right home for it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions