Add Gemma 4 E2B MLP quantization and multi-pass sweep configuration. - #375
Open
copybara-service[bot] wants to merge 1 commit into
Open
copybara-service[bot] wants to merge 1 commit into
copybara-service[bot] wants to merge 1 commit into
Conversation
- Support YAML sweep specifications in xm_launch.py with macro expansion, dry-run support, and per-trial output directories. - Support hierarchical scaling and multi-pass quantization configurations for MLP layers in MaxText (base.yml, types.py, quantizations.py). - Add gemma4_e2b_mlp_quant_sweep.yaml defining 13 parallel trials on Gemma 4 E2B (Ghostfish 2x2x1): * Trial 1: Baseline (unquantized BF16) * Trial 2: 1-pass mxfp8_16 on all MLP tensors (flat scales) * Trial 3: 1-pass mxfp8_16 on all MLP tensors + hierarchical scaling on weight gradients (drhs_mlp_grad) * Trial 4: 1-pass mxint8 on all MLP tensors (flat scales) * Trial 5: 1-pass mxint8 on all MLP tensors + hierarchical scaling on weight gradients * Trial 6: 3-pass triangular mxfp8_16 (all MLP, flat scales) * Trial 7: 3-pass triangular mxfp8_16 + hierarchical scaling on weight gradients * Trial 8: 4-pass full-cross mxfp8_16 (all MLP, flat scales) * Trial 9: 4-pass full-cross mxfp8_16 + hierarchical scaling on weight gradients * Trial 10: 3-pass triangular mxint8 (all MLP, flat scales) * Trial 11: 3-pass triangular mxint8 + hierarchical scaling on weight gradients * Trial 12: 4-pass full-cross mxint8 (all MLP, flat scales) * Trial 13: 4-pass full-cross mxint8 + hierarchical scaling on weight gradients - Add launch_gemma4_e2b_mlp_quant_sweep.sh using pretraining reference settings from cl/980100775 (Grain dataset loading, Ghostfish 2x2x1, 10k steps, BS=4, GA=16, 1.05M tokens/step, /cns/yo-d/ storage). PiperOrigin-RevId: 980111536
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Gemma 4 E2B MLP quantization and multi-pass sweep configuration.