Skip to content

ESMFold2: fold(lm_dropout=0) does not turn the LM dropout off #418

Open

Description

@daylight-00

The fold() docstring says lm_dropout defaults to 0.3 and that 0/None disables it, but 0 leaves the checkpoint's own rate in place: _lm_dropout_context returns early for any falsy value. The released biohub/ESMFold2 and biohub/ESMFold2-Fast configs ship lm_encoder.lm_dropout = 0.25 with per_loop_lm_dropout = True, so a fold asked for no dropout still drops 25 % of the LM pair channel in every loop:

from esm.models.esmfold2.processor import _lm_dropout_context

with _lm_dropout_context(model, 0):
    model.config.lm_encoder.lm_dropout  # 0.25, not 0

The helper's own docstring says "None/0 is a no-op", so the two docstrings disagree about what 0 should mean. For anyone using lm_dropout=0 as a control, the fold is not the dropout-free one it reads as.

What I would expect: 0 disables the dropout for the call and the config is restored afterwards, None keeps the checkpoint's setting, and a positive value overrides it, on both the release and the experimental architecture. If you would rather keep 0 as a no-op, the fold() docstring needs the fix instead.

I have the first version with tests on both architectures on a branch (diff) and can open it as a PR if that is the behaviour you want.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions