Distributed training: Multi-GPU scope and long-context guidance vs ZeRO++, DeepSpeed-Ulysses / ALST sequence parallelism and context parallelism (TrainingArguments.parallelism_config) (aging) (TECH-T04-009) #489

Wave 2 · theme D2-7: Next-release technology updates · highest severity Low · 2 finding(s) · effort not estimated

Summary

The README claim is not false. ForgeLM, however, cannot express the now-documented upstream path for >32K multi-GPU training, and a custom DeepSpeed JSON cannot enable Ulysses without parallelism_config. The 64K row points users at ZeRO-3, which shards parameters, gradients and optimizer state but not activations; for 7B QLoRA, activations dominate.

Primary finding

TECH-T04-009 — Distributed training: Multi-GPU scope and long-context guidance vs ZeRO++, DeepSpeed-Ulysses / ALST sequence parallelism and context parallelism (TrainingArguments.parallelism_config) (aging)

severity Next release

Item. Multi-GPU scope and long-context guidance vs ZeRO++, DeepSpeed-Ulysses / ALST sequence parallelism and context parallelism (TrainingArguments.parallelism_config) · verdict: aging · urgency: Next release

State in ForgeLM. README.md:156 lists 'DeepSpeed ZeRO-2/3, FSDP, multi-GPU'. DistributedConfig (config.py:750-778, extra='forbid') has no sequence/context/tensor-parallel or parallelism_config field, and forgelm/ never references parallelism_config. docs/usermanuals/en/training/long-context.md:78 (and the TR mirror at :78) says '64K | 40 GB+ | Multi-GPU territory; ZeRO-3 helps' for a 7B QLoRA run.

State of the ecosystem on 2026-10-01. transformers v5.0.0 (2026-01-26) integrated ALST/Ulysses. The v5.18.0 docs have a 'Ulysses sequence parallelism' page: it requires Accelerate v1.12.0 and at least 2 GPUs, is configured via ParallelismConfig(sp_backend='deepspeed', sp_size) passed to TrainingArguments.parallelism_config, and notes that TiledMLP and activation offload are not in Transformers. accelerate v1.15.0 documents sequence parallelism and context parallelism. The ALST paper (arXiv 2506.13996) reports an out-of-box HF OOM above 32K, 500K tokens on one H100 and 3.7M on 8xH100. ZeRO++ is maintained (a fix landed in 0.19.5) but is an opt-in multi-node communication optimisation, not a long-context technique.

Impact. The README claim is not false. ForgeLM, however, cannot express the now-documented upstream path for >32K multi-GPU training, and a custom DeepSpeed JSON cannot enable Ulysses without parallelism_config. The 64K row points users at ZeRO-3, which shards parameters, gradients and optimizer state but not activations; for 7B QLoRA, activations dominate.

Recommendation. Correct long-context.md:78 in EN and TR to name sequence or context parallelism and to say that ForgeLM does not expose it. Add one README sentence: the scope is ZeRO/FSDP data-parallel sharding; ZeRO++ is reachable only via custom DeepSpeed JSON; Ulysses SP/CP is not wired. Decide on an opt-in distributed.parallelism block (sp_size, sp_backend) mapped to TrainingArguments.parallelism_config, with accelerate>=1.12, as a roadmap item.

Sources.

• Transformers v5.18.0 docs: Ulysses sequence parallelism (accelerate>=1.12, >=2 GPUs, TrainingArguments.parallelism_conf… (2026-09-30)
• Accelerate v1.15.0 docs: Sequence parallel (Ulysses SP; see also context parallelism) (2026-09-09)
• arXiv 2506.13996: Arctic Long Sequence Training (ALST) (2025-06-16)
• Transformers GitHub releases v4.55-v5.5 (v5.0.0 ALST/Ulysses integration) (2026-01-26)
• DeepSpeed GitHub releases v0.18.8-v0.19.7 (release notes) (2026-09-16)
• DeepSpeed ZeRO++ tutorial (2026-10-01)

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

DOC-BLEGAL-243 — docs/usermanuals/en/training/long-context.md: misleading — [IS-T04-10] '| 64K | 40 GB+ | Multi-GPU territory; ZeRO-3 helps. |'

severity Low

Where. docs/usermanuals/en/training/long-context.md:78 (Turkish mirror: docs/usermanuals/tr/training/long-context.md:78) · type: misleading

The document says. [IS-T04-10] '| 64K | 40 GB+ | Multi-GPU territory; ZeRO-3 helps. |'

What the code does. ZeRO-3 shards parameters, gradients and optimizer state, not activations, which dominate a 64K-token run. The upstream long-context path is Ulysses sequence parallelism or context parallelism via TrainingArguments.parallelism_config (accelerate >= 1.12, at least 2 GPUs), which ForgeLM cannot configure.

User impact. Points users at a technique that does not address the memory bottleneck of long-context training.

Proposed correction. | 64K | 40 GB+ | Multi-GPU territory; needs sequence/context parallelism (Ulysses / ParallelismConfig), which ForgeLM does not configure yet. ZeRO-3 does not shard activations. |

Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

• Each statement is re-checked against the code at fix time.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}