With distributed.strategy fsdp on FSDP1 (transformers' default at the declared floor 5.3.0) the safety and judge gates cannot generate: every prompt fails with "'weight' must be 2… (CR-A02-026) #107

Wave 0 · theme D0-7: Dependency ranges and security floors · highest severity High · 1 finding(s) · effort medium

Summary

transformers 5.3.x, 4 GPUs, distributed.strategy fsdp, safety enabled, auto_revert:true. Training completes, every probe generation errors, safety records 'no usable evidence', the model is kept, the judge averages scores of empty answers, and the run exits 0. CI ships a model whose safety was never measured.

Primary finding

CR-A02-026 — With distributed.strategy fsdp on FSDP1 (transformers' default at the declared floor 5.3.0) the safety and judge gates cannot generate: every prompt fails with "'weight' must be 2-D", safety becomes…

severity High · reachability documented-opt-in · intent contract-violation · confidence confirmed · independently confirmed · reproduced (plumbing only)

What happens. ForgeLM passes the legacy string fsdp='full_shard auto_wrap' plus fsdp_config without a version (trainer.py:601-611). In transformers 5.3.0 (pyproject floor 'transformers>=5.3.0', pyproject.toml:53) the default FSDP version is 1 (venvs/lower transformers/training_args.py:2752). Versions 5.13.0 and 5.18.0 default to 2 (training_args.py:2734 and :2784). The Trainer replaces self.model with the FSDP root (transformers 5.3.0 trainer.py:1609-1612, 5.13 :1618-1621), and its register_fsdp_forward_method(self.model, 'generate') is a no-op for FSDP1 (torch/distributed/fsdp/_fully_shard/_fully_shard.py:775-777). The gates call self.trainer.model.generate (trainer.py:1942 -> safety/_generate.py:71/29; trainer.py:1987 -> judge.py:429/392) outside FSDP's forward, so root-owned parameters (embed_tokens, norm, lm_head) stay as 1-D sharded views. Probe (fsdp1_probe.py): 2-rank CPU gloo, FSDP1 root with transformer auto-wrap, FULL_SHARD and use_orig_params=True (transformers' FSDP1 defaults). After one root forward, the embed weight shape is (9723968,). ForgeLM's _generate_safety_responses returns 10/10 GeneratedResponse errors "RuntimeError: 'weight' must be 2-D" on both ranks (consistent, no hang). In ForgeLM the 100% unscored set is a protocol failure (_MAX_UNSCORED_RATIO=0.5, safety/_gates.py:36,90), so evaluation_completed=False. _apply_safety_result then keeps the model whatever auto_revert says (trainer.py:1458-1460), and success stays True. A single-process check confirmed that this path exits 0: guard-load-failure scenario, auto_revert:true gives exit 0, envelope success:true, safety.passed:false. The judge's _generate_response turns each failure into '' (judge.py:394-401), which feeds the already-reported empty-answer scoring. The benchmark is unaffected because HFLM calls th…

Where.

• forgelm/trainer.py:601-612
• forgelm/trainer.py:1942
• forgelm/trainer.py:1987
• forgelm/safety/_generate.py:71
• forgelm/safety/_generate.py:29
• forgelm/judge.py:429
• forgelm/judge.py:394-401
• forgelm/trainer.py:1458-1460
• forgelm/safety/_gates.py:36
• forgelm/trainer.py:1605

Evidence.

• outputs/a review run/a1-W70/fsdp1.log: 'PROBE1 rank=0 ... embed_weight_shape=(9723968,)'; 'safety-gen on FSDP1 root: n=10 errors=10 first_error="RuntimeError: 'weight' must be 2-D"'. The single-process exit-0 check is in tmp/.../a1-W70/single/envelope.json.

Recommended fix. Generate inside FSDP.summon_full_params(model, writeback=False) for FSDP1, or unwrap for generation the way TRL's unwrap_model_for_generation does. Alternatively pin fsdp_config['version']=2 when it is supported. Detect a gate whose generations all failed and fail the run (non-zero) under auto_revert:true rather than keep and exit 0.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.
• The release note names the contract this fix restores.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}