Fix residual gradient in non-learnable LayerNorm backward - #449
Open
rocketbastard wants to merge 2 commits into
Open
Fix residual gradient in non-learnable LayerNorm backward#449rocketbastard wants to merge 2 commits into
rocketbastard wants to merge 2 commits into
Conversation
Move dx_accumulate handling into the Triton backward kernel and add focused numerical unit tests.
Contributor
Greptile SummaryThe PR fuses residual-gradient accumulation into the non-learnable Triton LayerNorm backward kernel and adds focused numerical coverage.
Confidence Score: 4/5The kernel change appears sound, but the new regression test's unresolved source-tree import should be fixed before merging so it can be collected reliably. The numerical kernel path preserves the intended residual addition, while the new test imports a package that is neither rooted at the repository level nor made visible from the test module. Files Needing Attention: examples/tests/commons/test_triton_layer_norm.py Important Files Changed
Reviews (1): Last reviewed commit: "Add gradient accumulation support in lay..." | Re-trigger Greptile |
Comment on lines
+7
to
+9
| from ops.triton_ops.triton_layer_norm import ( | ||
| triton_weighted_layer_norm_bwd, | ||
| ) |
Contributor
There was a problem hiding this comment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fix residual gradient accumulation in the non-learnable LayerNorm backward path.
dx_accumulateinto the_layer_norm_bwd_dxTriton kernel.dx.add_()operation.Supersedes #447 because the original fork was deleted.
Fixes #446
Testing
The added unit tests passed successfully in the local GPU environment.
Checklist