Repository navigation
[Triage] Code does not match docs/usermanuals/en/training/distributed.md: Launching through accelerate launch makes "DeepSpeed's parameter-partitioning wrapper initialise before the mode… (CR-A04-092) #822
Description
Needs triage. Reviewers disagreed on this finding, so it is not counted in the backlog. Decide whether it is a real defect: if yes, set severity and wave; if not, close it with the reason.
CR-A04-092 — Code does not match docs/usermanuals/en/training/distributed.md: Launching through accelerate launch makes "DeepSpeed's parameter-partitioning wrapper initialise before the model loads"; the `zero3…
severity Low · confidence disputed · single reviewer
What happens. If ZeRO-3 partitioned loading is part of promise F-008, the TrainingArguments/HfDeepSpeedConfig must exist before from_pretrained. Needs verification on multi-GPU hardware.
Failure scenario. The document states: Launching through accelerate launch makes "DeepSpeed's parameter-partitioning wrapper initialise before the model loads"; the zero3_offload preset "fits 70B on 8×24 GB".. Observed in the code: ForgeLM loads the model (get_model_and_tokenizer, cli/_training.py:141) before it builds the TrainingArguments that carry deepspeed= (cli/_training.py:148; trainer.py:592-598). ForgeLM creates no HfDeepSpeedConfig or zero.Init before the load (no occurrence in forgelm/). Whether a launcher-level setting changes this could not be established offline; with ForgeLM's code order, ZeRO-3 init-time partitioning does not appear to happen, which would undercut the 70B sizing claim.
Where.
Evidence.
- forgelm/cli/_training.py:141-148; forgelm/trainer.py:592-598
Recommended fix. Pending verification, soften to: "ForgeLM loads the model before DeepSpeed initialises; verify host RAM for very large models."
Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.
Activity
cemililik commentedon Oct 20, 2026
Confirmed on development: the model is loaded in forgelm/cli/_training.py before the TrainingArguments carrying deepspeed= are built, and nothing in forgelm/ creates an HfDeepSpeedConfig or zero.Init beforehand.
We also checked the launcher side:
accelerate launchonly exports environment variables, and in a fresh process with those set transformers still reports ZeRO-3 as disabled at from_pretrained time. So the sentence 'launch through accelerate launch so partitioning initialises before the model loads' (distributed.md:139, English and Turkish) is not true for ForgeLM today. The 70B-on-8x24GB sizing sentence was already removed in #997.Two steps stay open here: correct the manual wording now (extra-small), and decide separately whether to build the DeepSpeed config before the model load (needs a multi-GPU host; related: #77).
Triage result:
bug,documentation,area: trainer,severity: medium,wave: 3. Medium rather than high because the sizing claim is gone and the cost is host RAM during loading, which has a workaround.Triage round 1, checked against
developmenton 2026-10-20.