Skip to content

Fix residual gradient in non-learnable LayerNorm backward - #449

Open
rocketbastard wants to merge 2 commits into
NVIDIA:mainfrom
rocketbastard:main
Open

Fix residual gradient in non-learnable LayerNorm backward#449
rocketbastard wants to merge 2 commits into
NVIDIA:mainfrom
rocketbastard:main

Conversation

@rocketbastard

Copy link
Copy Markdown

Description

Fix residual gradient accumulation in the non-learnable LayerNorm backward path.

  • Fuse dx_accumulate into the _layer_norm_bwd_dx Triton kernel.
  • Remove the separate PyTorch dx.add_() operation.
  • Add focused numerical unit tests for the LayerNorm operator.

Supersedes #447 because the original fork was deleted.
Fixes #446

Testing

The added unit tests passed successfully in the local GPU environment.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Move dx_accumulate handling into the Triton backward kernel and add focused numerical unit tests.
@greptile-apps

greptile-apps Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR fuses residual-gradient accumulation into the non-learnable Triton LayerNorm backward kernel and adds focused numerical coverage.

  • Adds an optional residual-gradient pointer and row stride to the backward kernel.
  • Eliminates the separate post-kernel accumulation operation.
  • Tests float32 and bfloat16 behavior across residual and non-residual cases with padded row strides.

Confidence Score: 4/5

The kernel change appears sound, but the new regression test's unresolved source-tree import should be fixed before merging so it can be collected reliably.

The numerical kernel path preserves the intended residual addition, while the new test imports a package that is neither rooted at the repository level nor made visible from the test module.

Files Needing Attention: examples/tests/commons/test_triton_layer_norm.py

Important Files Changed

Filename Overview
examples/hstu/ops/triton_ops/triton_layer_norm.py Correctly loads and adds the optional residual gradient inside the non-learnable backward kernel before storing the result.
examples/tests/commons/test_triton_layer_norm.py Adds useful numerical coverage, but its HSTU-local import is unresolved when the test is collected from its new commons test location.

Reviews (1): Last reviewed commit: "Add gradient accumulation support in lay..." | Re-trigger Greptile

Comment on lines +7 to +9
from ops.triton_ops.triton_layer_norm import (
triton_weighted_layer_norm_bwd,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 HSTU test import is unresolved

When pytest collects this test from the repository root or the examples/tests hierarchy without an external PYTHONPATH, ops is not importable because it resides under examples/hstu, causing collection to fail with ModuleNotFoundError.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]Non-learnable LayerNorm backward drops residual gradient

1 participant