DeepSpeed presets (zero2/zero3/zero3_offload) are not packaged in the wheel or image, so preset names fail on every pip/Docker install (CR-A02-004) #110

Wave 0 · theme D0-8: Packaging and delivery promises · highest severity High · 16 finding(s) · effort small

Summary

An operator runs pip install forgelm[distributed], writes the documented distributed: {strategy: deepspeed, deepspeed_config: zero2} and starts training on a multi-GPU host. The model loads, then ForgeTrainer raises ConfigError and the run exits 1 ('Configuration error'). The docs promise a built-in preset, so the operator is misled. In CI, exit 1 (config error) is read as 'fix the YAML', but the YAML is correct.

Primary finding

CR-A02-004 — DeepSpeed presets (zero2/zero3/zero3_offload) are not packaged in the wheel or image, so preset names fail on every pip/Docker install

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN OPS-01 — docs/roadmap/phase-16-trust-surface-hardening.md:642-650,661-662 (S12 open)

What happens. _resolve_deepspeed_config (trainer.py:617-659) resolves preset names, and the default, to configs/deepspeed/*.json under the parent of the forgelm package and then the CWD. pyproject package-data ships nothing under configs/ (MANIFEST.in adds it to sdist only, .dockerignore excludes it), so installed copies raise ConfigError at trainer construction or silently use a CWD-relative file; tests only resolve from the source tree.

Where.

• forgelm/trainer.py:617-664
• pyproject.toml:250-278 (packages.find and package-data omit configs/deepspeed)
• MANIFEST.in:5 (sdist-only)
• configs/deepspeed/zero2.json
• docs/reference/distributed_training.md:52
• Dockerfile:97
• forgelm/trainer.py:632-659 (confirmed)
• forgelm/trainer.py:595
• docs/reference/configuration.md:295
• docs/usermanuals/en/training/distributed.md:69

Evidence.

• Installed lane: ForgeTrainer._resolve_deepspeed_config(None|'zero2'|'zero3'|'zero3_offload') raises ConfigError "DeepSpeed preset 'zero2' not found at (local review artefact)" (logs/ds-train-stub.err, resolve.py output). Snapshot lane: the same four calls return /configs/deepspeed/*.json. Controls: a custom path resolves, and a CWD holding configs/deepspeed/zero2.json resolves. grep configs RECORD in the wheel's dist-info finds nothing. Evidence in outputs/.../logs and resolve.py.
• ds_repro/probe_ds.py in the installed-wheel lane ($RT/installed has no configs/): from an empty CWD, 'zero2', None and 'zero3_offload' -> ConfigError not found at $RT/installed/configs/deepspeed/.... From a CWD that contains configs/deepspeed/zero2.json -> returns 'configs/deepspeed/zero2.json' (the planted file) with no log of the path. Snapshot lane -> resolves to snapshot/configs/deepspeed/*.json.
• Installed-wheel lane (fpy -C $RT/installed), ForgeTrainer.new()._resolve_deepspeed_config('zero2'): from an empty CWD it raises ConfigError "DeepSpeed preset 'zero2' not found at $RT/installed/configs/deepspeed/zero2.json"; from a CWD with a planted configs/deepspeed/zero2.json (nvme_path=(local review artefact)) it returns 'configs/deepspeed/zero2.json'. pyproject package-data lists only forgelm.templates, forgelm (py.typed), forgelm.safety_prompts and forgelm.wizard.

Recommended fix. Ship the presets as package data, for example move them to forgelm/deepspeed_presets/*.json (or forgelm/configs/deepspeed/), add them to [tool.setuptools.package-data], resolve them with importlib.resources, and keep the CWD fallback. Add a wheel-content test or CI step that builds the wheel and resolves every preset from it. Have --dry-run resolve the preset so the failure appears before a model load.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

• TECH-T04-002 (technology, Now) — Distributed training: Built-in DeepSpeed presets 'zero2'/'zero3'/'zero3_offload' are not shipped in the wheel (configs/deepspeed/ lies outside the forgelm package) (incorrect-claim)
  severity Now
  Item. Built-in DeepSpeed presets 'zero2'/'zero3'/'zero3_offload' are not shipped in the wheel (configs/deepspeed/ lies outside the forgelm package) · verdict: incorrect-claim · urgency: Now
  State in ForgeLM. trainer.py:632-659 maps the preset names to configs/deepspeed/.json and resolves them against dirname(dirname(trainer.py)), which is site-packages for a wheel install, and then against the CWD. An omitted deepspeed_config defaults to 'zero2' (trainer.py:638-639). pyproject.toml:250-252 packages only forgelm, the package-data table (pyproject.toml:257-278) lists no configs, and MANIFEST.in:5 (re…
  State of the ecosystem on 2026-10-01. The published PyPI wheel forgelm-0.11.0-py3-none-any.whl (uploaded 2026-07-21; 117 files) contains no configs/ entry, yet its own trainer.py still resolves configs/deepspeed presets. The installed 0.11.1rc1 build is the same. Reproduced on the installed lane: ForgeTrainer._resolve_deepspeed_config(None, p) raises ConfigError for each of zero2, zero3 and zero3_offload. The setuptools docs state th…
  Recommendation. Move the presets into the package (for example forgelm/configs/deepspeed/*.json listed in [tool.setuptools.package-data]) and load them via importlib.resources, materialising a path for TrainingArguments. Add a wheel-install test that resolves all three presets. Correct troubleshooting.md: the error is a ConfigError, and a pip install has no project root.
  Sources.

  • PyPI JSON: forgelm (0.11.0 latest, wheel + sdist uploaded 2026-07-21) (2026-07-21)
  • Published wheel forgelm-0.11.0-py3-none-any.whl (117 files, no configs/deepspeed/*.json) (2026-07-21)

• DOC-BFIELD-002 (docs, High) — forgelm/config.py: false claim — distributed.deepspeed_config: filesystem path to a DS JSON OR preset name (zero2, zero3, zero3_offload).
  severity High
  Where. forgelm/config.py:759-762 · type: false claim
  The document says. distributed.deepspeed_config: filesystem path to a DS JSON OR preset name (zero2, zero3, zero3_offload).
  What the code does. The presets live in configs/deepspeed/*.json at the repository root. They are not package data (pyproject.toml:257-275 lists only forgelm.templates, py.typed, safety_prompts, wizard defaults). _resolve_deepspeed_config (trainer.py:618-660) looks for /configs/deepspeed/.json and then ./configs/deepspeed/.json. On the installed-wheel lane both _resolve_deepspeed_config('z…
  Code references. forgelm/trainer.py:618-660, pyproject.toml:257-275, configs/deepspeed/zero2.json
  Proposed correction. Move the presets under forgelm/ (package data) and resolve them with importlib.resources, or document that preset names need a source checkout.

• DOC-D1B01-007 (docs, High) — docs/reference/configuration.md: misleading — distributed.deepspeed_config: "Preset ("zero2", "zero3", "zero3_offload") or JSON path"
  severity High
  Where. docs/reference/configuration.md:295 (Turkish mirror: docs/reference/configuration-tr.md:318) · type: misleading
  The document says. distributed.deepspeed_config: "Preset ("zero2", "zero3", "zero3_offload") or JSON path"
  What the code does. Presets are looked up at /configs/deepspeed/*.json, then at ./configs/deepspeed/ in the CWD. configs/ is not package data, so in a pip install every preset raises ConfigError, as does the implicit zero2 used when deepspeed_config is null. Reproduced in the installed-wheel lane: all three presets failed with "DeepSpeed preset '...' not found at .../installed/conf…
  Code references. forgelm/trainer.py:617-660, pyproject.toml:257-278
  Proposed correction. Fix packaging: ship the presets as package data and resolve them via importlib.resources. Temporary doc note: "Preset names resolve only from a source checkout (or a configs/deepspeed/ directory in the working directory); with a pip install, point deepspeed_config at a JSON file."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D1B03-003 (docs, High) — docs/reference/distributed_training.md: false claim — ForgeLM provides three built-in DeepSpeed presets; set deepspeed_config to zero2 / zero3 / zero3_offload.
  severity High
  Where. docs/reference/distributed_training.md:52,60-69 (also :16-23, :64-69) (Turkish mirror: docs/reference/distributed_training-tr.md (same 'Presets' / 'DeepSpeed Configuration' sections)) · type: false claim
  The document says. ForgeLM provides three built-in DeepSpeed presets; set deepspeed_config to zero2 / zero3 / zero3_offload.
  What the code does. Presets resolve to configs/deepspeed/*.json relative to dirname(dirname(trainer.py)) or the CWD (forgelm/trainer.py:632-654). pyproject package-data ships only templates, py.typed, safety_prompts and wizard defaults, so the wheel has no configs/ directory. Reproduced on the installed wheel: ForgeTrainer._resolve_deepspeed_config(None,'zero2') raises ConfigError 'DeepSpeed preset zero2 not found a…
  Code references. forgelm/trainer.py:632-672, pyproject.toml:244-280, MANIFEST.in:5
  Proposed correction. Until the code is fixed, add under 'Presets': '> Note: the presets ship in the source tree (configs/deepspeed/) and are not included in the pip wheel or the Docker image. With a pip install, copy the JSON files and set deepspeed_config to the file path.'
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B06-006 (docs, High) — docs/guides/enterprise_deployment.md: broken example — distributed.deepspeed_config: "zero2" # or "zero3", "zero3_offload" presets.
  severity High
  Where. docs/guides/enterprise_deployment.md:119-129 · type: broken example
  The document says. distributed.deepspeed_config: "zero2" # or "zero3", "zero3_offload" presets.
  What the code does. Presets resolve to /configs/deepspeed/.json, or ./configs/deepspeed relative to the CWD. configs/ is not package data in the wheel (pyproject package-data lists only templates, py.typed, safety_prompts and the wizard defaults). From the installed-wheel lane all three presets raised ConfigError 'DeepSpeed preset … not found'. The Docker image (pip install ., WO…
  Code references. forgelm/trainer.py:617-669, pyproject.toml:254-278, Dockerfile:97-107
  Proposed correction. Fix the code: ship configs/deepspeed/*.json as package data (e.g. forgelm/deepspeed_presets) and resolve via importlib.resources. Until then, tell pip/Docker users to pass a JSON path or run from a source checkout.

• DOC-D3B04A-007 (docs, High) — docs/usermanuals/en/reference/configuration.md: false claim — deepspeed_config accepts 'a filesystem path to a DeepSpeed JSON, or one of the built-in preset names zero2, zero3, zero3_offload'.
  severity High
  Where. docs/usermanuals/en/reference/configuration.md:221, 229 (Turkish mirror: docs/usermanuals/tr/reference/configuration.md:222, 230) · type: false claim
  The document says. deepspeed_config accepts 'a filesystem path to a DeepSpeed JSON, or one of the built-in preset names zero2, zero3, zero3_offload'.
  What the code does. Presets resolve to dirname(dirname(forgelm/trainer.py))/configs/deepspeed/.json, falling back to ./configs/deepspeed in the CWD. The configs/ directory is not package data, so wheel installs lack the presets. Reproduced against the installed-wheel lane: _resolve_deepspeed_config(None, 'zero2') raised ConfigError "DeepSpeed preset 'zero2' not found at …/installed/configs/deepspeed/zero2.js…
  Code references. forgelm/trainer.py:617-668, pyproject.toml:257-278, configs/deepspeed/zero2.json
  Proposed correction. Add: "The presets are read from a configs/deepspeed/ directory next to the package or in the current working directory; a plain pip install does not ship them, so point deepspeed_config at a JSON file (copy one from the repository's configs/deepspeed/) unless you run from a source checkout."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B06-008 (docs, High) — docs/usermanuals/en/training/distributed.md: false claim — deepspeed_config accepts the preset names zero2 / zero3 / zero3_offload.
  severity High
  Where. docs/usermanuals/en/training/distributed.md:69,92,98 (Turkish mirror: docs/usermanuals/tr/training/distributed.md:92) · type: false claim
  The document says. deepspeed_config accepts the preset names zero2 / zero3 / zero3_offload.
  What the code does. The presets resolve to <parent of forgelm/>/configs/deepspeed/<name>.json, with a fallback to ./configs/deepspeed/<name>.json relative to the CWD (forgelm/trainer.py:632-660). configs/deepspeed/ is not package data: pyproject.toml:257-277 ships only templates, py.typed, safety_prompts and wizard defaults. On the installed wheel, _resolve_deepspeed_config raises ConfigError: DeepSpeed pre… **Code references.** [forgelm/trainer.py:617-660](https://github.com/HodeTech/ForgeLM/blob/cab2947/forgelm/trainer.py#L617-L660), [pyproject.toml:257-277](https://github.com/HodeTech/ForgeLM/blob/cab2947/pyproject.toml#L257-L277), [configs/deepspeed/zero2.json](https://github.com/HodeTech/ForgeLM/blob/cab2947/configs/deepspeed/zero2.json) **Proposed correction.** Fix the code (bundle the presets in the wheel). Until then add: "Preset names resolve only from a source checkout; with a pip install, point deepspeed_config` at a JSON file path."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D4B02-001 (docs, High) — config_template.yaml: broken example — Launch with: torchrun --nproc_per_node=N forgelm --config my_config.yaml
  severity High
  Where. config_template.yaml:164 · type: broken example
  The document says. Launch with: torchrun --nproc_per_node=N forgelm --config my_config.yaml
  What the code does. torch.distributed.run.config_from_args builds the command [sys.executable, '-u', training_script, ...] unless --no-python or -m is given, so this line runs python -u forgelm --config ..., which fails with "can't open file '.../forgelm'". It only works by accident from a source checkout, where a forgelm/ package directory sits in the cwd. Reproduced: python -u forgelm --config ct.yaml --dry-run… **Code references.** [torch/distributed/run.py:839](https://github.com/HodeTech/ForgeLM/blob/cab2947/torch/distributed/run.py#L839), [forgelm/main.py`](https://github.com/HodeTech/ForgeLM/blob/cab2947/forgelm/__main__.py)
  Proposed correction. # Launch with: torchrun --nproc_per_node=N -m forgelm --config my_config.yaml

• DOC-D4B02-002 (docs, High) — config_template.yaml: false claim — deepspeed_config: "zero2" # preset: "zero2", "zero3", "zero3_offload" or path to JSON
  severity High
  Where. config_template.yaml:167 · type: false claim
  The document says. deepspeed_config: "zero2" # preset: "zero2", "zero3", "zero3_offload" or path to JSON
  What the code does. The presets are resolved as /configs/deepspeed/.json, falling back to ./configs/deepspeed/ in the cwd (trainer.py:632-660). configs/deepspeed/ sits at the repository root and is not packaged: pyproject.toml:250-252 packages only forgelm*, and package-data lists no deepspeed JSON. In the installed-wheel lane, ForgeTrainer._resolve_deepspeed_config(None,'zero2')…
  Code references. forgelm/trainer.py:617-660, pyproject.toml:250-252, pyproject.toml:257-270, configs/deepspeed/zero2.json
  Proposed correction. Fix the code (ship the presets as package data). Temporary doc note: "# Presets resolve only from a source checkout or a cwd containing configs/deepspeed/; on a pip install give a path to a DeepSpeed JSON file."

• DOC-BLEGAL-148 (docs, Medium) — docs/usermanuals/en/training/distributed.md: misleading — [IS-T04-09] 'The zero3_offload preset fits 70B on 8×24 GB by offloading optimiser state and parameters to CPU.'
  severity Medium
  Where. docs/usermanuals/en/training/distributed.md:98 (Turkish mirror: docs/usermanuals/tr/training/distributed.md:98) · type: misleading
  The document says. [IS-T04-09] 'The zero3_offload preset fits 70B on 8×24 GB by offloading optimiser state and parameters to CPU.'
  What the code does. In transformers 5.18, ZeRO-3 init (zero.Init) applies only if the HfDeepSpeedConfig weakref exists before the model loads. ForgeLM loads the model first, and accelerate launch creates no weakref, so each rank first materialises the full model in CPU RAM (reasoned from source, about 140 GB per rank for 70B bf16; not measured). Separately, the zero2/zero3/zero3_offload presets are not shipped in th…
  Proposed correction. Remove the '70B on 8×24 GB' sizing claim, or qualify it: the preset is only available in a source checkout; because ForgeLM loads the model before DeepSpeed initialises, each rank needs host RAM for the full model (≈140 GB for 70B in bf16) before partitioning.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D1B03-008 (docs, Medium) — docs/reference/distributed_training.md: false claim — ForgeLM provides three built-in DeepSpeed presets; set deepspeed_config to one of these names
  severity Medium
  Where. docs/reference/distributed_training.md:52 (Turkish mirror: docs/reference/distributed_training-tr.md) · type: false claim
  The document says. ForgeLM provides three built-in DeepSpeed presets; set deepspeed_config to one of these names
  What the code does. Under a wheel install the preset names raise ConfigError (exit 1), because configs/deepspeed/*.json is not packaged. They work only in a source checkout or when the CWD contains configs/deepspeed/.
  Code references. forgelm/trainer.py:632-654, pyproject.toml:257-278
  Proposed correction. Fix the packaging (F-PKG-1). Until then, say that presets need a source checkout and that pip users must pass a JSON path.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D4B01-023 (docs, Medium) — README.md: misleading — "DeepSpeed ZeRO-2/3"
  severity Medium
  Where. README.md:156 · type: misleading
  The document says. "DeepSpeed ZeRO-2/3"
  What the code does. The preset JSONs (zero2/zero3/zero3_offload) live in the repository's configs/deepspeed/, which the wheel does not ship. Package-data covers only forgelm.* paths, and the installed lane has no configs/. _resolve_deepspeed_config looks beside the package and then in the CWD, so after pip install forgelm a preset name, or the default None (which resolves to zero2), raises ConfigError (exit 1).…
  Code references. forgelm/trainer.py:633-658, pyproject.toml:257-278, MANIFEST.in:5
  Proposed correction. Code: ship configs/deepspeed as package data and resolve it via importlib.resources. Doc, until then: "DeepSpeed ZeRO-2/3 (from a PyPI install, point distributed.deepspeed_config at your own JSON)."

• DOC-D4B02-009 (docs, Medium) — config_template.yaml: false claim — Authentication can be provided via HUGGINGFACE_TOKEN env var, an interactive prompt, or hardcoded here (not recommended).
  severity Medium
  Where. config_template.yaml:173-176 · type: false claim
  The document says. Authentication can be provided via HUGGINGFACE_TOKEN env var, an interactive prompt, or hardcoded here (not recommended).
  What the code does. setup_authentication uses auth.hf_token, then the HUGGINGFACE_TOKEN env var, then the token files ~/.cache/huggingface/token and /.huggingface/token. If none is found it logs a warning and continues. No interactive prompt exists anywhere, and the token-file fallback is not mentioned.
  Code references. forgelm/utils.py:15-55, forgelm/cli/_training.py:137
  Proposed correction. # Authentication: auth.hf_token here (not recommended), else the HUGGINGFACE_TOKEN env var, else the local token file (/.cache/huggingface/token or ~/.huggingface/token). There is no interactive prompt.

• DOC-D2B05-040 (docs, Low) — docs/guides/troubleshooting.md: wrong detail — FileNotFoundError: DeepSpeed preset 'zero2' not found
  severity Low
  Where. docs/guides/troubleshooting.md:343-345 · type: wrong detail
  The document says. FileNotFoundError: DeepSpeed preset 'zero2' not found
  What the code does. The trainer raises ConfigError (exit 1) with "DeepSpeed preset 'zero2' not found at ...". The remedy is correct because the presets are not shipped in the wheel's package-data.
  Code references. forgelm/trainer.py:617-669, pyproject.toml:257-279
  Proposed correction. ConfigError: DeepSpeed preset 'zero2' not found at ... (exit 1). Presets ship only in the source checkout's configs/deepspeed/.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D4B02-013 (docs, Low) — config_template.yaml: wrong detail — Distributed training (multi-GPU). Requires: pip install forgelm[distributed]
  severity Low
  Where. config_template.yaml:163 · type: wrong detail
  The document says. Distributed training (multi-GPU). Requires: pip install forgelm[distributed]
  What the code does. The distributed extra installs only DeepSpeed (not on Windows). The FSDP strategy needs no extra.
  Code references. pyproject.toml (distributed extra: deepspeed>=0.14.0; sys_platform != 'win32')
  Proposed correction. # Distributed training (multi-GPU). DeepSpeed requires: pip install forgelm[distributed]; FSDP needs no extra.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.
• The release note names the contract this fix restores.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}