The default DeepSpeed preset zero2.json sets allgather_bucket_size: "auto", which neither transformers nor accelerate fills, so ZeRO-2 aborts (CR-A16-007) #410

Activity

1. cemililik

   added this to the v0.12.0 — Wave 1: public contract and documented usage milestone on Oct 9, 2026
2. cemililik

   added

   bugSomething isn't workingSomething isn't working

   area: packaging-ciCode area: packaging-ciCode area: packaging-ci

   severity: highWrong behaviour or a false claim users rely onWrong behaviour or a false claim users rely on

   source: review-2026-10Found by the October 2026 code and documentation reviewFound by the October 2026 code and documentation review

   wave: 1Roadmap wave 1: high severity, small effortRoadmap wave 1: high severity, small effort

   on Oct 9, 2026
3. cemililik

   added this to ForgeLM Project Planon Oct 9, 2026
4. cemililik

   added a parent issue on Oct 9, 2026

   • [Epic] D1-5 · Training, merging and export correctness #852
5. cemililik

   added a commit that references this issue on Oct 18, 2026

   docs: webhooks, DeepSpeed presets, first-run output tree and --offlin…

   ...

   700b6d6
6. cemililik commented on Oct 22, 2026

   

   cemililik

   on Oct 22, 2026

   CollaboratorAuthor

   More actions

   Where to start

   Thanks for taking a look. Claim it by starting a comment with /assign so nobody duplicates the work.

   The bug. configs/deepspeed/zero2.json sets "allgather_bucket_size": "auto" (line 36). transformers fills only reduce_bucket_size, stage3_prefetch_bucket_size and stage3_param_persistence_threshold, and accelerate raises a ValueError for any other "auto" left in the config. zero2 is also the default preset (_resolve_deepspeed_config in forgelm/trainer.py), so distributed.strategy: deepspeed aborts by default. zero3.json and zero3_offload.json are fine.

   What to change (decided).

   1. In zero2.json, set "allgather_bucket_size": 500000000. That is DeepSpeed's own default; keep the key.
   2. Add a CPU-only test to the class TestDeepSpeedConfigResolution in tests/test_distributed.py, next to test_all_presets_use_auto_values.
      • Parametrize it over zero2, zero3 and zero3_offload.
      • Collect the dotted path of every "auto" value in the JSON, and assert that the set is a subset of the keys transformers and accelerate fill. Take that allowlist from HfTrainerDeepSpeedConfig.trainer_config_process / trainer_config_finalize in transformers/integrations/deepspeed.py:
        • train_batch_size, train_micro_batch_size_per_gpu, gradient_accumulation_steps, gradient_clipping;
        • fp16.enabled, bf16.enabled;
        • optimizer.params.{lr,betas,eps,weight_decay};
        • scheduler.params.{warmup_min_lr,warmup_max_lr,warmup_num_steps,total_num_steps};
        • zero_optimization.reduce_bucket_size, zero_optimization.stage3_prefetch_bucket_size, zero_optimization.stage3_param_persistence_threshold.
      • Name the preset and the offending key in the assertion message.
      • Check that it fails on development (it should report zero_optimization.allgather_bucket_size for zero2) and passes after your change.
   3. Remove the now-stale notes, deleting text only (no new Turkish to write):
      • the "Known issue (The default DeepSpeed preset zero2.json sets allgather_bucket_size: "auto", which neither transformers nor accelerate fills, so ZeRO-2 aborts (CR-A16-007) #410)" paragraph from docs/reference/distributed_training.md (about line 60) and the matching "Bilinen sorun" paragraph from docs/reference/distributed_training-tr.md;
      • the clause that calls the zero2 preset broken from line 98 of docs/usermanuals/en/training/distributed.md and of docs/usermanuals/tr/training/distributed.md.
   4. Add a ### Fixed line under ## [Unreleased] in CHANGELOG.md: the default zero2 preset no longer aborts ZeRO-2 start-up.

   Not in scope: shipping the presets in the wheel. That belongs to a Phase 16 step; please do not link it from the PR. A real ZeRO-2 run needs GPUs, which you do not need here: say in the PR that you verified the config contract on CPU.

   pytest tests/test_distributed.py --no-cov
   python3 tools/check_bilingual_parity.py --strict && python3 tools/check_bilingual_code_blocks.py --strict
   python3 tools/check_usermanual_self_contained.py --strict && python3 tools/check_tr_links_prefer_mirror.py --strict

   Open the PR against development with Fixes #410 (finding CR-A16-007).
7. cemililik

   added

   good first issueGood for newcomersGood for newcomers

   on Oct 22, 2026