TRL trainers and alignment methods: ORPO and SimPO as methods in 2026 practice (aging) (TECH-T01-006) #487

Wave 2 · theme D2-7: Next-release technology updates · highest severity Next release · 1 finding(s) · effort not estimated

Summary

Keeping ORPO and SimPO as niche options is defensible. The docs present them as front-line methods, without the upstream experimental status or the mixed 2024-2026 evidence. Separately, both paths are broken today (items above).

Primary finding

TECH-T01-006 — TRL trainers and alignment methods: ORPO and SimPO as methods in 2026 practice (aging)

severity Next release

Item. ORPO and SimPO as methods in 2026 practice · verdict: aging · urgency: Next release

State in ForgeLM. README.md:153 lists six trainers on equal footing. docs/guides/alignment.md:111 says 'SimPO outperforms DPO at 7B+ scale (+6.4 points on AlpacaEval 2)'. docs/usermanuals/en/training/orpo.md:20 says 'ORPO is the only alignment method that works well directly from a base model'. Neither page says that upstream holds ORPO and SimPO as experimental.

State of the ecosystem on 2026-10-01. TRL keeps ORPO and CPO/SimPO under trl.experimental, outside the stable trainer set of DPO, GRPO, KTO, RLOO, Reward, SFT and Distillation (v1.14.1 _toctree). They are still maintained (fixes in v1.0.0, v1.8.0 and v1.9.0), and they survived the v1.14.0 removal of six experimental trainers. SimPO's length normalisation now also exists in stable DPO as loss_type='sigmoid_norm'. Tülu 3 (arXiv 2411.15124, Section 5.4, Table 18) found that 'only length-normalized DPO outperformed our base checkpoint overall'; its SimPO rows (51.8, 52.9) scored below the SFT base (55.7). A March 2026 controlled study (arXiv 2603.19335) found none of 20 DPO variants significantly better than vanilla DPO. SimPO was significantly worse at 1.5B (−11.5 pp) but best at 7B on GSM8K, which is a scale-dependent ranking inversion. No 2026 primary source treats ORPO as state of the art. The SimPO '+6.4 on AlpacaEval 2' figure is the 2024 paper's own 'up to 6.4 points' claim (quoted in TRL cpo_trainer.md:67).

Impact. Keeping ORPO and SimPO as niche options is defensible. The docs present them as front-line methods, without the upstream experimental status or the mixed 2024-2026 evidence. Separately, both paths are broken today (items above).

Recommendation. Label ORPO and SimPO 'experimental upstream' in the README and manuals. Recommend DPO (or length-normalised DPO) as the default offline preference method. Attribute the SimPO figure to the 2024 paper as 'up to'. Soften or source the ORPO 'only method from a base model' claim.

Sources.

• TRL v1.14.1 docs TOC (CPO, ORPO under Experimental; stable trainer list) (2026-09-29)
• Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales (arXiv 2603.19335) (2026-03-19)
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training (Section 5.4, Table 18) (2024-11-22 (v1; v5 revision date not recorded))
• TRL docs: Experimental (stability contract and removal policy) (2026-10-01)
• huggingface/trl GitHub releases (v1.14.0 six experimental trainers removed; CPO/ORPO fixes in v1.8.0/v1.9.0) (2026-09-25)

Acceptance criteria

• Each statement is re-checked against the code at fix time.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}