deepspeed>=0.14.0 (uncapped) with torch>=2.4,<3 admits pairs whose JIT ops cannot build. With the default CUDA-13 torch 2.12-2.14 wheels, the zero2/zero3 presets need deepspeed>=0… (CR-A16-084) #531

Wave 2 · theme D2-7: Next-release technology updates · highest severity Low · 2 finding(s) · effort small

Summary

pyproject.toml:116 declares deepspeed>=0.14.0 with no cap, and pyproject.toml:46 admits torch up to 2.14.1. DeepSpeed declares only torch>=2.0.0. transformers and accelerate enforce only deepspeed>=0.9.3 (transformers dependency_versions_table.py:12 in the lower and latest venvs). ForgeLM's own check, trainer.py:564-589, tests importability and never the version. All three presets use optimizer type AdamW. DeepSpeed's engine (v0.19.7 engine.py:2383-2413) maps that to DeepSpeedCPUAdam when the optimizer is offloaded (zero3_offload) and to FusedAdam otherwise (zero2, zero3, and the default preset). Both are JIT-compiled ops. torch>=2.12 headers require C++20 (#error 'C++20 or later compatible compiler is required to use PyTorch', DeepSpeed issue #​8235). DeepSpeed fixed this in two steps. (a) PR #​8238 (merged 2026-08-09, in v0.19.5) set -std=c++20 in OpBuilder.cxx_args for torch>=2.12. Every release from 0.14.0 to 0.19.4 therefore fails the CPU op build (CPUAdam, zero3_offload) and the .cpp frontend of the CUDA ops. (b) The nvcc path still passed -std=c++17 on CUDA 13 in v0.19.4, v0.19.5 and v0.19.6 (builder.py: if cuda_major == 12 and cuda_minor >= 5). PR #​8422 (merged 2026-09-06,…

Primary finding

CR-A16-084 — deepspeed>=0.14.0 (uncapped) with torch>=2.4,<3 admits pairs whose JIT ops cannot build. With the default CUDA-13 torch 2.12-2.14 wheels, the zero2/zero3 presets need deepspeed>=0.19.7, not 0.19.5

severity Low · reachability documented-opt-in · intent intended-but-harmful · confidence likely (static only) · independently confirmed · not reproduced (code trace)

What happens. pyproject.toml:116 declares deepspeed>=0.14.0 with no cap, and pyproject.toml:46 admits torch up to 2.14.1. DeepSpeed declares only torch>=2.0.0. transformers and accelerate enforce only deepspeed>=0.9.3 (transformers dependency_versions_table.py:12 in the lower and latest venvs). ForgeLM's own check, trainer.py:564-589, tests importability and never the version. All three presets use optimizer type AdamW. DeepSpeed's engine (v0.19.7 engine.py:2383-2413) maps that to DeepSpeedCPUAdam when the optimizer is offloaded (zero3_offload) and to FusedAdam otherwise (zero2, zero3, and the default preset). Both are JIT-compiled ops. torch>=2.12 headers require C++20 (#error 'C++20 or later compatible compiler is required to use PyTorch', DeepSpeed issue #​8235). DeepSpeed fixed this in two steps. (a) PR #​8238 (merged 2026-08-09, in v0.19.5) set -std=c++20 in OpBuilder.cxx_args for torch>=2.12. Every release from 0.14.0 to 0.19.4 therefore fails the CPU op build (CPUAdam, zero3_offload) and the .cpp frontend of the CUDA ops. (b) The nvcc path still passed -std=c++17 on CUDA 13 in v0.19.4, v0.19.5 and v0.19.6 (builder.py: if cuda_major == 12 and cuda_minor >= 5). PR #​8422 (merged 2026-09-06, listed in the v0.19.7 notes of 2026-09-16) extended the condition to CUDA 13+. The default PyPI Linux wheels of torch 2.12.0 (2026-05-13), 2.13.0 and 2.14.1 depend on nvidia-*-cu13. DeepSpeed's assert_no_cuda_mismatch requires a toolkit of the same CUDA major and has no 13.x minor-mismatch entry. The FusedAdam build of zero2/zero3 against a default torch 2.12-2.14 therefore needs deepspeed>=0.19.7. This corrects T04's 0.19.5 boundary, which holds only for the CPU path. Any environment resolved between 2026-05-13 and 2026-09-16 at the latest DeepSpeed of the day (0.19.0-0.19.6) with the latest…

Where.

• pyproject.toml:116
• pyproject.toml:46
• forgelm/trainer.py:564-589
• forgelm/trainer.py:591-599
• configs/deepspeed/zero2.json:13-21
• configs/deepspeed/zero3.json:13-21
• configs/deepspeed/zero3_offload.json:13-21
• forgelm/trainer.py:617-660
• docs/usermanuals/en/getting-started/installation.md:86
• forgelm/cli/subcommands/_doctor.py:101

Evidence.

• Upstream sources fetched read-only from GitHub and PyPI: DeepSpeed PR #​8238 diff (op_builder/builder.py cxx_args); PR #​8422 body ('on CUDA 13.x it falls back to -std=c++17. PyTorch >= 2.12 headers require C++20 ... build fails'); builder.py at v0.19.4:869-876, v0.19.5:893 and v0.19.6:893 (CUDA 12.5+ only), v0.19.7:902-906 (or cuda_major >= 13); v0.19.6 builder.py:104-134 (assert_no_cuda_mismatch, no 13 entry); v0.19.7 engine.py:2383-2413; PyPI torch 2.12.0/2.13.0/2.14.1 requires_dist list nvi…

Recommended fix. Raise the floor to deepspeed>=0.19.7 (it covers both the cxx and the CUDA-13 nvcc fixes; 0.19.6+ is also needed for the macOS claim), or cap torch per DeepSpeed band. In _check_deepspeed_available, compare deepspeed.version and torch.version.cuda and raise a ConfigError naming the needed version before model load. Add a CI lane that installs DeepSpeed at the floor and builds/loads FusedAdamBuilder and CPUAdamBuilder.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

TECH-T04-004 — Distributed training: DeepSpeed dependency range: deepspeed>=0.14.0; sys_platform != 'win32' (extra [distributed]): floor, torch compatibility, platform marker (range-unsound)

severity Next release

Item. DeepSpeed dependency range: deepspeed>=0.14.0; sys_platform != 'win32' (extra [distributed]): floor, torch compatibility, platform marker · verdict: range-unsound · urgency: Next release

State in ForgeLM. pyproject.toml:116 declares deepspeed>=0.14.0; sys_platform != 'win32' with no upper bound. The core pins are torch>=2.4.0,<3.0.0 (pyproject.toml:46) and transformers>=5.3.0,<6.0.0 (pyproject.toml:53). No review venv (lower/extras/latest) installs DeepSpeed, so no CI lane exercises any DeepSpeed version. ForgeLM imports deepspeed only to check that it is available (trainer.py:582) and lets transformers drive it.

State of the ecosystem on 2026-10-01. The latest release is deepspeed 0.19.7 (PyPI 2026-09-16). It is published as an sdist only, as has every release since 0.16.6 (2025-04-16). requires_dist is torch>=2.0.0 with no upper bound, and the PyPI classifiers list Python 3.8-3.12. The floor 0.14.0 dates from 2024-03-08. PR #​8238 (merged 2026-08-09, shipped in 0.19.5 on 2026-08-10) switched op builds to -std=c++20 because PyTorch 2.12+ headers require C++20, so earlier DeepSpeed cannot compile its C++/CUDA ops against torch 2.12-2.14.1 (torch 2.12.0 was released 2026-05-13). Apple Silicon (MPS) support exists only from 0.19.6. The HF stack enforces only deepspeed>=0.9.3 (transformers dependency table; accelerate's check), so nothing upstream rejects an old DeepSpeed. The exact torch matrix that the DeepSpeed CI tests could not be found.

Impact. pyproject.toml:116, together with pyproject.toml:46, admits failing pairs: deepspeed 0.14.0-0.19.4 with torch 2.12-2.14. These surface as an op-build crash at the first fused or CPU Adam use (the presets' AdamW block, and the CPUAdam that zero3_offload uses) after config validation has passed, i.e. exit 2 rather than a config error. A fresh resolve picks 0.19.7, so default installs are unaffected today. Lockfiles and constrained environments are exposed. The macOS claim in doctor (separate item) also silently needs >=0.19.6.

Recommendation. Raise the floor to deepspeed>=0.19.5 (or >=0.19.6 if macOS stays in the doctor text). Add an install lane that installs DeepSpeed at the floor and at latest and builds TrainingArguments(deepspeed=) against the lower and latest transformers. Keep the win32 marker, but fix its stated reason (see the platform-claims item).

Sources.

• PyPI JSON: deepspeed (0.19.7 latest, sdist only; per-release files incl. win_amd64 wheels for 0.14.5, 0.15.0, 0.16.3-0.… (2026-09-16)
• DeepSpeed PR #8238: Fix compilation error for torch 2.12+ (use -std=c++20) (2026-08-09)
• DeepSpeed GitHub releases v0.18.8-v0.19.7 (release notes) (2026-09-16)
• DeepSpeed v0.19.7 README (Requirements; Windows section) (2026-09-16)
• PyPI JSON: torch (2.14.1 latest; 2.12.0 2026-05-13) (2026-09-30)
• transformers v5.18.0 dependency_versions_table.py (torch>=2.5, peft>=0.20.0, accelerate>=1.1.0, deepspeed>=0.9.3) (2026-09-30)

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}