FSDP wiring uses the deprecated string form; no_shard/hybrid_shard raise and backward_prefetch/auto_wrap are ignored under transformers' FSDP2 default (CR-A02-006) #104

Wave 0 · theme D0-7: Dependency ranges and security floors · highest severity High · 4 finding(s) · effort medium

Summary

An operator selects fsdp_strategy: hybrid_shard for multi-node, as the field description suggests: --dry-run passes and the run exits 2 at trainer construction. An operator who disables auto-wrap or changes the prefetch policy gets the default behaviour with no warning.

Primary finding

CR-A02-006 — FSDP wiring uses the deprecated string form; no_shard/hybrid_shard raise and backward_prefetch/auto_wrap are ignored under transformers' FSDP2 default

severity High · reachability documented-opt-in · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. _apply_distributed_config builds the legacy kwargs['fsdp'] = ' [auto_wrap] [offload]' string plus fsdp_config {backward_prefetch, state_dict_type} without a version key (trainer.py:601-612). On transformers versions inside the range where FSDP2 is default, no_shard/hybrid_shard raise ValueError at TrainingArguments build (exit 2), backward_prefetch is unused, auto_wrap=false has no effect, and the string form is slated for removal in v5.20.

Where.

• forgelm/trainer.py:600-611 (the fsdp branch of _apply_distributed_config; kwargs['fsdp'] string at 607, fsdp_config at 608-611)
• forgelm/config.py:771-774 (fsdp_backward_prefetch)
• forgelm/trainer.py:602
• forgelm/trainer.py:607
• forgelm/trainer.py:608
• forgelm/config.py:763
• forgelm/trainer.py:598-612
• docs/reference/configuration.md:296,299
• pyproject.toml:52 (transformers>=5.3.0,<6.0.0)
• forgelm/trainer.py:530-533

Evidence.

• scripts/fsdp_probe.py, real SFTConfig, transformers 5.13.0: default -> plugin {fsdp_version 2, auto_wrap_policy TRANSFORMER_BASED_WRAP, state_dict_type FULL_STATE_DICT, reshard_after_forward True}, no backward_prefetch; no_shard and hybrid_shard -> ValueError fsdp=... is only available with FSDP1; fsdp_auto_wrap False with backward_post -> plugin still TRANSFORMER_BASED_WRAP and still no backward_prefetch.
• probe_dist.py, all 16 strategy x offload x prefetch x state-dict combinations: on transformers 5.13.0 and 5.18.0, full_shard and shard_grad_op are accepted (fsdp=True, reshard_after_forward True/False, offload mapped to cpu_offload) and no_shard and hybrid_shard raise ValueError; on 5.3.0 all are accepted. transformers training_args.py at tags v5.4.0 to v5.9.0 has no FSDP1-only check; v5.10.0 (PyPI upload 2026-06-03) is the first with it (lines 2830-2838).
• ValueError 'fsdp=hybrid_shard is only available with FSDP1' (5.13 and latest); fsdp_plugin_args lacks backward_prefetch under FSDP2.

Recommended fix. Build fsdp=True plus an explicit FSDP2 fsdp_config (reshard_after_forward, cpu_offload, auto_wrap_policy). Restrict or translate the strategies FSDP2 cannot express, and reject or remap backward_prefetch/auto_wrap so no knob is silently dead.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

CR-A02-045 — distributed.fsdp_state_dict_type is silently ignored on transformers 5.3-5.9 (inside the declared range)

severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · not reproduced (code trace)

What happens. forgelm/trainer.py:608-611 passes fsdp_config={'backward_prefetch', 'state_dict_type'}. In transformers 5.3.0 (lower venv training_args.py:2730-2785), the FSDP plugin args are built without reading fsdp_config['state_dict_type'], so accelerate's default applies. A SHARDED_STATE_DICT request (config.py:775-778) is dropped with no warning, while ForgeLM's config, and any artefact that records it, reports the requested format. 5.13.0 (:2760) and 5.18.0 (:2810) do read it. The upstream tag v5.9.0 has no state_dict_type line in training_args.py; v5.10.0 has it.

Failure scenario. A multi-GPU FSDP run on transformers 5.3-5.9 with SHARDED_STATE_DICT writes full checkpoints, or the reverse, with no signal.

Where.

• forgelm/trainer.py:607-611
• forgelm/config.py:775-778
• pyproject.toml:53

Evidence.

• grep of training_args.py: lower (5.3.0) has backward_prefetch at :2754/2763 and no state_dict_type; extras :2760 and latest :2810 have fsdp_plugin_args['state_dict_type']. Upstream raw training_args.py fetched: v5.8.0/v5.9.0 have no 'state_dict_type'; v5.10.0/v5.11.0 contain it (exact first version inferred from tag fetches).

Recommended fix. Raise the transformers floor to >=5.10 for FSDP, or set FSDP_STATE_DICT_TYPE and warn when the installed version ignores the key.

TECH-T02-004 — Core Hugging Face stack and runtime dependencies: TrainingArguments kwargs: FSDP string API (' [auto_wrap] [offload]') + fsdp_config {backward_prefetch, state_dict_type}, and the other kwar…

severity Now

Item. TrainingArguments kwargs: FSDP string API (' [auto_wrap] [offload]') + fsdp_config {backward_prefetch, state_dict_type}, and the other kwargs/imports (eval_strategy, optim, use_cpu, neftune_noise_alpha, optim_target_modules/optim_args, deepspeed, dtype=) · verdict: deprecated-in-use · urgency: Now

State in ForgeLM. trainer.py:601-612 builds fsdp=' [auto_wrap] [offload]' and fsdp_config={'backward_prefetch', 'state_dict_type'}. config.py:763 accepts full_shard, shard_grad_op, no_shard and hybrid_shard, and fsdp_state_dict_type defaults to FULL_STATE_DICT (config.py:774-776). trainer.py:496-521 also passes eval_strategy, optim, use_cpu and the other kwargs.

State of the ecosystem on 2026-10-01. transformers moved the FSDP default from FSDP1 to FSDP2 between 5.8.0 (get('version', 1)) and 5.10.0 (get('version', 2)), per the cached raw training_args.py. Since then, a string or list fsdp value logs that it 'will be removed in Transformers v5.20', and FSDP1 is deprecated for the same release. The FSDP2 docs use fsdp=True plus fsdp_config. My fsdp_acc run results: - 5.18.0 with accelerate 1.15.0: full_shard and shard_grad_op build an FSDP2 plugin with backward_prefetch=None (the option is ignored); no_shard and hybrid_shard raise ValueError 'only available with FSDP1'; state_dict_type is honoured. - 5.3.0: all four strategy strings produce FSDP1 arguments, but with the declared floor accelerate 1.4.0 the plugin cannot be built at all (TypeError: unexpected keyword 'fsdp_version'). The other kwargs and imports in my api_check run (5.3.0 and 5.18.0): eval_strategy, optim, use_cpu, neftune_noise_alpha, optim_target_modules, optim_args, deepspeed, fsdp and fsdp_config are all TrainingArguments fields, and AutoModelForCausalLM, AutoModelForImageTextToText, AutoProcessor, BitsAndBytesConfig, TextIteratorStreamer, pipeline and EarlyStoppingCallback all import. evaluation_strategy is gone, so ForgeLM's eval_strategy is right. The exception is warmup_ratio, covered in its own item. PyPI shows a new transformers minor every 2-3 weeks (5.15.0 on 08-10, 5.16.0 on 08-26, 5.17.0 on 09-09, 5.18.0 on 09-30).

Impact. FSDP is broken across the declared range: - On transformers >=5.10, including the project venv's 5.13.0 and a fresh 5.18.0 install, no_shard and hybrid_shard (documented values) crash at TrainingArguments construction with exit 2, not exit 1. - fsdp_backward_prefetch is silently ignored. - On the floor pair (transformers 5.3.0 + accelerate 1.4.0), no FSDP run can start. - The string form disappears in 5.20, which the <6 cap admits; at the current release pace that is likely 1-2 releases away. The non-FSDP kwargs listed here need no change.

Recommendation. Emit fsdp=True with fsdp_config {version: 2, reshard_after_forward (full_shard maps to True, shard_grad_op to False), cpu_offload, state_dict_type, auto_wrap_policy}. Reject no_shard, hybrid_shard and backward_prefetch at config time, or set version=1 explicitly with a deprecation notice. Raise the accelerate floor (accelerate item). Until the migration lands, cap transformers <5.20. Add a TrainingArguments plus FSDP-plugin construction test on the floor and latest lanes.

Sources.

• transformers v5.18.0 training_args.py (FSDP2 default, v5.20 removal notice) (2026-09-30)
• transformers v5.10.0 training_args.py (FSDP2 default) (2026-06-03)
• transformers v5.8.0 training_args.py (FSDP1 default) (2026-05-05)
• Transformers docs: FSDP2 (fsdp=True + fsdp_config) (2026-10-01)

TECH-T04-003 — Distributed training: FSDP wiring: legacy string fsdp='<full_shard|shard_grad_op|no_shard|hybrid_shard> [auto_wrap] [offload]' (FSDP1 semantics) with fsdp_config {backward_prefetch, state_dict_type}…

severity Now

Item. FSDP wiring: legacy string fsdp='<full_shard|shard_grad_op|no_shard|hybrid_shard> [auto_wrap] [offload]' (FSDP1 semantics) with fsdp_config {backward_prefetch, state_dict_type} and no 'version' · verdict: deprecated-in-use · urgency: Now

State in ForgeLM. trainer.py:601-612 builds a space-joined fsdp string and passes fsdp_config {backward_prefetch, state_dict_type} without 'version'. config.py:763-778 exposes four strategies (no_shard and hybrid_shard included; hybrid_shard is described as the multi-node option), fsdp_backward_prefetch (backward_pre|backward_post) and fsdp_state_dict_type (FULL|SHARDED). No test builds TrainingArguments from these fields.

State of the ecosystem on 2026-10-01. transformers v5.10.0 (2026-06-03) changed the fsdp_config version default from 1 to 2 (v5.9.0 still read get('version', 1)). From v5.10.0 through 5.18.0 and main, passing fsdp as a string or list, and FSDP1 itself, log 'deprecated and will be removed in Transformers v5.20'. 5.18.0 (2026-09-30) is the latest release, and v5.20 has no published date. My ForgeTrainer probe on transformers 5.13.0 and 5.18.0 showed: - full_shard and shard_grad_op map to FSDP2 reshard_after_forward True and False. - no_shard and hybrid_shard raise ValueError('fsdp=hybrid_shard is only available with FSDP1 ...'). - backward_prefetch reaches the plugin as None for both values. On 5.3.0 all four strategies work as FSDP1 and backward_prefetch is honoured. The accelerate FSDP1-vs-FSDP2 guide marks backward_prefetch as REMOVED, since FSDP2 always prefetches like BACKWARD_PRE, and keeps state_dict_type: FULL and SHARDED are valid and LOCAL is obsolete. Accelerate still supports FSDP1, with no removal date. PyTorch 2.14.1 has not deprecated the FSDP1 class. FSDP2 needs torch>=2.6 (accelerate FSDP2_PYTORCH_VERSION), while ForgeLM allows torch>=2.4.

Impact. The default resolve (transformers 5.18.0) already breaks two of the four documented strategies at TrainingArguments construction. Those configs pass --dry-run and then fail as a training error. fsdp_backward_prefetch is a silent no-op: backward_post is accepted and ignored. Every FSDP config will fail once transformers v5.20 drops the string form, which the <6.0.0 pin admits. torch 2.4/2.5 combined with transformers>=5.10 cannot start FSDP2. Affected docs: configuration.md:296-300, distributed_training.md:115-134, usermanuals/{en,tr}/training/distributed.md:114 and config_template.yaml:169.

Recommendation. Pass fsdp=True with an explicit fsdp_config: version 2, reshard_after_forward (full_shard maps to True, shard_grad_op to False), cpu_offload, auto_wrap_policy and state_dict_type. Map hybrid_shard to an FSDP2 2-D mesh (ParallelismConfig dp_replicate/dp_shard) or reject it with a ConfigError, and drop no_shard in favour of plain DDP. Deprecate fsdp_backward_prefetch, warning when it is set to backward_post. Keep fsdp_state_dict_type. Gate FSDP on torch>=2.6. Add a test that builds TrainingArguments for every DistributedConfig variant on the lower and latest transformers.

Sources.

• transformers v5.18.0 training_args.py (FSDP plugin args, legacy string translation, ZeRO init tip, FSDP checkpointing w… (2026-09-30)
• transformers v5.10.0 training_args.py (fsdp_config version default 2; FSDP1 and string fsdp deprecated) (2026-06-03)
• transformers v5.9.0 training_args.py (fsdp_config version default 1) (2026-05-20)
• transformers main training_args.py (still 'removed in Transformers v5.20') (2026-10-01)
• Accelerate v1.15.0: FSDP1 vs FSDP2 (backward_prefetch REMOVED; state_dict_type kept) (2026-09-09)
• accelerate main utils/dataclasses.py (FSDP plugin; backward_prefetch not supported in FSDP2) (2026-10-01)

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.
• The release note names the contract this fix restores.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}