Repository navigation
ORPO and SimPO import ORPOConfig/ORPOTrainer/CPOConfig/CPOTrainer from top-level trl, which no trl 1.x in the declared range exports (CR-A02-013) #106
Description
Wave 0 · theme D0-7: Dependency ranges and security floors · highest severity High · 28 finding(s) · effort medium
Summary
An operator sets trainer_type: simpo or orpo as the docs show, on any trl from 1.0.0 to 1.14.1. --dry-run says 'config is valid'. The real run exits 2 after loading the model and dataset.
Primary finding
CR-A02-013 — ORPO and SimPO import ORPOConfig/ORPOTrainer/CPOConfig/CPOTrainer from top-level trl, which no trl 1.x in the declared range exports
severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN GAP-03 — docs/roadmap/phase-16-trust-surface-hardening.md:382-400,868-870 (S4 open)
What happens. forgelm/trainer.py:723 and :737 (_get_training_args_for_type) and :1037 and :1047 (_build_simple_trl_trainer) import ORPOConfig, CPOConfig, ORPOTrainer and CPOTrainer from the top-level trl package. In every trl that pyproject.toml:59 (>=1.0.0,<2.0.0) admits, these names exist only under trl.experimental.orpo and trl.experimental.cpo. trainer_type orpo and simpo therefore raise ImportError at trainer construction (exit 2) after --dry-run, which never imports trl, has reported the config valid.
Where.
forgelm/trainer.py:722-727, 736-745, 1034-1049forgelm/trainer.py:737(confirmed)forgelm/trainer.py:1047(confirmed)pyproject.toml:59(confirmed)tests/test_trainer_max_length.py:50-55(skip-on-AttributeError masks the break)tests/test_trainer_max_length.py:43tests/test_trainer_max_length.py:52pyproject.toml:61forgelm/config.py:461tests/test_trainer_max_length.py:55
Evidence.
- runs/simpo-dir|simpo-file|orpo-dir|orpo-file exit 2: ImportError: cannot import name 'CPOConfig' / 'ORPOConfig' from 'trl'. Probe: hasattr(trl,'CPOConfig') False in trl 1.0.0/1.7.1/1.14.1, while trl.experimental.cpo/orpo expose both classes. With imports switched to trl.experimental (patch.diff) simpo/orpo exit 0 in main, latest and shimmed lower venvs.
- scripts/trl_probe.py in the default (trl 1.7.1), lower (trl 1.0.0) and latest (trl 1.14.1) lanes: getattr(trl,'ORPOConfig'|'CPOConfig'|'ORPOTrainer'|'CPOTrainer') raises AttributeError, while trl.experimental.orpo and trl.experimental.cpo expose all four. CLI runs: runs/orpo -> exit 2
cannot import name 'ORPOConfig' from 'trl'; runs/simpo -> exit 2cannot import name 'CPOConfig' from 'trl'; runs/orpo.latest -> the same exit 2. runs-patched/orpo and runs-patched/simpo, with the imports switc… - probe_trl.py on all three lanes: trl.ORPOConfig, ORPOTrainer, CPOConfig and CPOTrainer each raise AttributeError; trl.experimental.orpo and trl.experimental.cpo import OK. Default-lane e2e: cfg-orpo.yaml EXIT=2 "ImportError: cannot import name 'ORPOConfig' from 'trl'" at trainer.py:723; cfg-simpo.yaml EXIT=2 at trainer.py:737. With the experimental symbols patched onto trl (patch_run.py): orpo and simpo EXIT=0 on the default lane, and also on the lower (with the peft workaround) and latest (wit…
Recommended fix. Import from trl.experimental.cpo / trl.experimental.orpo with a version-aware fallback, or narrow the trl pin. Replace the skip in test_trainer_max_length with a hard failure.
Other findings with the same root cause
These close with the same fix; update them in the same pull request.
-
CR-A02-014 (code, High) — GRPO always fails: _apply_max_length passes max_prompt_length to GRPOConfig, which no trl 1.x accepts
severity High · reachabilitydefault· intentcontract-violation· confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN GAP-03 — docs/roadmap/phase-16-trust-surface-hardening.md:406-409 (S4 open)
Failure scenario. Any GRPO run, including the bundled grpo-math quickstart template, exits 2 at trainer construction after model and dataset loading.
Where.forgelm/trainer.py:780(confirmed),tests/test_trainer_max_length.py:74-97,pyproject.toml:59,forgelm/trainer.py:671-682
Recommended fix. Pass max_prompt_length only when the GRPOConfig signature has it, the way the SFT branch already checks max_length at trainer.py:704. Test against the real GRPOConfig signature. -
CR-A02-015 (code, High) — warmup_ratio is always passed to TrainingArguments-derived configs; transformers 5.15+ inside the <6.0.0 cap removed it, so every trainer fails on a fresh install
severity High · reachabilitydefault· intentcontract-violation· confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN GAP-03 — docs/roadmap/phase-16-trust-surface-hardening.md:406-409 (S4 open)
Failure scenario. pip install forgelm resolves transformers 5.18.x. Every training run exits 2. --dry-run (dry-latest-sft) and doctor still report success.
Where.forgelm/trainer.py:503(confirmed),pyproject.toml:53(confirmed),forgelm/config.py:492-494,config_template.yaml:40
Recommended fix. Map warmup_ratio to the float warmup_steps when the TrainingArguments signature lacks warmup_ratio, or cap transformers below the release that removed it. Add a CI job against the newest in-range dependencies. Research question: the exact transformers release that removed warmup_ratio (5.13.0 has it, 5.18.0 does not). -
CR-A02-028 (code, High) — Within the declared TRL range, orpo and simpo raise ImportError, GRPO raises TypeError, and (trl 1.14.1) every trainer type raises TypeError, so NEFTune can never take effect on them
severity High · reachabilitydefault· intentcontract-violation· confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN GAP-03 — docs/roadmap/phase-16-trust-surface-hardening.md:382-400,406-409 (S4 open)
Failure scenario. An operator following the docs and template for orpo, simpo or grpo on any in-range TRL gets an ImportError or TypeError at trainer build, after the model has loaded. On trl 1.14.1 with transformers 5.18.0 nothing trains.
Where.forgelm/trainer.py:737,forgelm/trainer.py:765(via _apply_max_length forgelm/trainer.py:672-683; GRPOConfig(**kwargs) at :775),forgelm/trainer.py:503,forgelm/trainer.py:1047
Recommended fix. Import ORPO/CPO from trl.experimental with a fallback, drop max_prompt_length for GRPO when the signature lacks it, map warmup_ratio to warmup_steps when absent, and add a signature-driven test per trainer type on the lower and latest venvs. Raise the torch floor to match peft 0.19.0, or gate it. -
CR-A16-022 (code, Medium) — Test suite and nightly dependency-range jobs cannot detect trainer API breakage: trl configs/trainers are mocked or skipped and no real trainer ever runs
severity Medium · reachabilitydefault· intentintended-but-harmful· confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN GAP-03 — docs/roadmap/phase-16-trust-surface-hardening.md:406-409 (S4 open)
Failure scenario. An upstream API removal inside the declared range (transformers 5.18 warmup_ratio) ships with a green nightly, and users find it first.
Where.pyproject.toml:184,.github/workflows/nightly.yml:52,.github/workflows/nightly.yml:244,tests/test_trainer_max_length.py:55
Recommended fix. Add an unmocked smoke test: one max_steps=1 run per trainer_type on a tiny model, run in both nightly range jobs. Turn DeprecationWarning from transformers, trl and peft into errors or a report, not an ignore. -
CR-A01-081 (code, Low) — training.warmup_ratio: 1.0 (allowed by le=1) becomes one warmup step on transformers 5.x, not 100% of training
severity Low · reachabilitydefault· intentintended-but-harmful· confidence confirmed · independently confirmed · reproduced end to end
Failure scenario. An operator asks for full-length warmup and gets one warmup step with no warning.
Where.forgelm/config.py:492,forgelm/trainer.py:503
Recommended fix. Uselt=1on the field, or compute warmup_steps = ceil(ratio * total_steps) in ForgeLM before passing an int. -
TECH-T01-001 (technology, Now) — TRL trainers and alignment methods: trl ORPOConfig/ORPOTrainer (trainer_type='orpo') (range-unsound)
severity Now · already tracked: KNOWN-OPEN GAP-03 docs/roadmap/phase-16-trust-surface-hardening.md:382-400,868-870
Item. trl ORPOConfig/ORPOTrainer (trainer_type='orpo') · verdict:range-unsound· urgency: Now
State in ForgeLM. forgelm/trainer.py:723 runsfrom trl import ORPOConfigand :1037 runsfrom trl import ORPOTrainer, passing beta=orpo_beta and max_length=model.max_length. pyproject.toml:59 declares trl>=1.0.0,<2.0.0. README.md:153, docs/reference/architecture.md:125 and docs/reference/library_api_reference.md:54 (and the -tr mirror) advertise ORPO. Roadmap Phase 16 S4 / C-10 (docs/roadmap/phase-16-trust-surf…
State of the ecosystem on 2026-10-01. TRL moved ORPO to trl.experimental in v0.26.0 (2025-12-09, PR #4480) and kept a FutureWarning shim at top level. v0.29.0 (2026-02-25, PR #5050) removed that shim. On trl 1.0.0, 1.7.1 and 1.14.1 (the latest, 2026-09-29), getattr(trl,'ORPOConfig'/'ORPOTrainer') raises AttributeError. trl.experimental.orpo imports and accepts beta and max_length. TRL's experimental contract says such code 'may chang…
Recommendation. Ship the C-10 resolver now: try the top-level name, then trl.experimental.orpo, then raise an actionable ConfigError that names trl.version. Make a missing advertised class fail the test instead of skipping. Label ORPO 'experimental upstream' in the README and docs. Because experimental code can disappear in a patch release, add a CI job on the newest resolvable trl that builds the real confi…
Sources. -
TECH-T01-002 (technology, Now) — TRL trainers and alignment methods: transformers TrainingArguments.warmup_ratio (passed into every TRL config) (range-unsound)
severity Now
Item. transformers TrainingArguments.warmup_ratio (passed into every TRL config) · verdict:range-unsound· urgency: Now
State in ForgeLM. forgelm/trainer.py:503 puts 'warmup_ratio' into the common kwargs of every TRL config. training.warmup_ratio defaults to 0.1 (forgelm/config.py:492-494). pyproject.toml:53 declares transformers>=5.3.0,<6.0.0.
State of the ecosystem on 2026-10-01. transformers 5.3.0 and 5.13.0 still accept warmup_ratio, warn that it 'is deprecated and will be removed in v5.2. Usewarmup_stepsinstead', and copy it into warmup_steps (training_args.py 5.3.0:1471-1473; 5.13.0:1485-1487). transformers PR #46917 'Remove deprecated training args andis_fastproperty' (merged 2026-07-22) deleted the warmup_ratio field. It shipped in v5.15.0 (2026-08-10), and…
Recommendation. Map training.warmup_ratio to warmup_steps=, which is valid as a float ratio across the declared 5.3-5.18 range, or cap transformers<5.15 until that ships. Add a nightly job on the newest resolvable stack that builds the real TRL configs.
Sources. -
TECH-T01-003 (technology, Now) — TRL trainers and alignment methods: Test and CI coverage of the TRL trainer surface (tests/test_trainer_max_length.py, nightly lower-bound job) (incorrect-claim)
severity Now · already tracked: KNOWN-OPEN GAP-03 docs/roadmap/phase-16-trust-surface-hardening.md:406-411,920-926
Item. Test and CI coverage of the TRL trainer surface (tests/test_trainer_max_length.py, nightly lower-bound job) · verdict:incorrect-claim· urgency: Now
State in ForgeLM. tests/test_trainer_max_length.py:52-55 calls pytest.skip() when trl.ORPOConfig or trl.CPOConfig is missing. :72-97 replaces GRPOConfig with FakeGRPOConfig and asserts max_prompt_length, and the module docstring (:7) claims 'GRPO must honour it as the prompt cap'. .github/workflows/nightly.yml:244-255 pins trl==1.0.0 and transformers==5.3.0 and runs the same suite. The other nightly job (:52) inst…
State of the ecosystem on 2026-10-01. Arbiter runs: the project venv (trl 1.7.1, transformers 5.13.0) gives '3 passed, 2 skipped' for tests/test_trainer_max_length.py. The seven TRL-related test modules (test_trainer_sft_config, test_trainer_grpo_config, test_grpo_reward, test_trainer_max_length, test_grpo_format_reward, test_trainer, test_oom_recovery) give '135 passed, 2 skipped' on both the floor (trl 1.0.0, transformers 5.3.0) an…
Recommendation. Build the real TRL config and trainer classes in tests, with no fakes and no skips, at the floor and at the newest resolvable versions. A missing advertised class must fail. This is already the roadmap S4 exit criterion; extend it to the warmup_ratio and max_prompt_length failures, which S4 does not name.
Sources. -
TECH-T01-004 (technology, Now) — TRL trainers and alignment methods: trl CPOConfig/CPOTrainer used for SimPO (loss_type='simpo', cpo_alpha=0, simpo_gamma) (range-unsound)
severity Now · already tracked: KNOWN-OPEN GAP-03 docs/roadmap/phase-16-trust-surface-hardening.md:382-400
Item. trl CPOConfig/CPOTrainer used for SimPO (loss_type='simpo', cpo_alpha=0, simpo_gamma) · verdict:range-unsound· urgency: Now
State in ForgeLM. forgelm/trainer.py:737 runsfrom trl import CPOConfigwith beta=simpo_beta, cpo_alpha=0.0, simpo_gamma and max_length, and sets loss_type='simpo' at :743. forgelm/trainer.py:1047 runsfrom trl import CPOTrainer. The comment at :739 says 'SimPO is implemented via CPOTrainer with loss_type="simpo" in TRL'.
State of the ecosystem on 2026-10-01. TRL moved CPO to trl.experimental in v0.26.0 (2025-12-09, PR #4470). v0.29.0 (2026-02-25, PR #5046) removed the deprecated top-level path. On 1.0.0, 1.7.1 and 1.14.1, getattr(trl,'CPOConfig'/'CPOTrainer') raises AttributeError, while trl.experimental.cpo.CPOConfig accepts every kwarg ForgeLM passes (loss_type, cpo_alpha, simpo_gamma, beta, max_length). The TRL v1.14.1 docs still give SimPO as los…
Recommendation. Import from trl.experimental.cpo through the C-10 resolver and keep the kwargs unchanged. Do not set TRL_EXPERIMENTAL_SILENCE. Fix the 'sigmoid loss' wording in simpo.md (EN and TR). Plan a documented fallback in case CPO leaves experimental. DPO sigmoid_norm is not reference-free, so present it as a different method, not as SimPO.
Sources. -
TECH-T01-005 (technology, Now) — TRL trainers and alignment methods: trl GRPOConfig/GRPOTrainer (num_generations, max_completion_length, max_prompt_length, reward_funcs) (range-unsound)
severity Now
Item. trl GRPOConfig/GRPOTrainer (num_generations, max_completion_length, max_prompt_length, reward_funcs) · verdict:range-unsound· urgency: Now
State in ForgeLM. forgelm/trainer.py:755-779 builds GRPOConfig with num_generations and max_completion_length. At :765 it calls self._apply_max_length(kwargs, 'max_prompt_length'), and the comment at :762-764 cites 'TRL's 512max_prompt_lengthdefault'. tests/test_trainer_max_length.py:72-97 asserts the kwarg against a FakeGRPOConfig.
State of the ecosystem on 2026-10-01. GRPOConfig.max_prompt_length was deprecated in v0.26.0. The help text at grpo_config.py@v0.26.0:298 and the warning at :869 say it 'will be removed in version 0.28.0. You should instead filter your dataset before training to ensure that prompts do not exceed your desired length'. It is still present in 0.27.0 and absent from 0.28.0 (2026-02-10) onward. The 0.28.0 notes do not announce the GRPO re…
Recommendation. Stop passing max_prompt_length. If a prompt cap is wanted, tokenise the GRPO prompts and filter or left-truncate them to model.max_length before building the trainer, as TRL advises, and record the dropped or truncated count in the manifest. Optionally set vllm_max_model_length when vLLM is used. Replace the fake-config test with one that builds the real GRPOConfig at the floor and the newest ver…
Sources. -
TECH-T02-001 (technology, Now) — Core Hugging Face stack and runtime dependencies: TrainingArguments warmup_ratio passed unconditionally (forgelm/trainer.py:503) (outdated)
severity Now
Item. TrainingArguments warmup_ratio passed unconditionally (forgelm/trainer.py:503) · verdict:outdated· urgency: Now
State in ForgeLM. _get_common_training_kwargs (trainer.py:444-521) always puts 'warmup_ratio' into the kwargs used for SFTConfig, DPOConfig, ORPOConfig, CPOConfig, KTOConfig and GRPOConfig, with no signature check. config.py:492 and docs/reference/configuration.md:76 document training.warmup_ratio.
State of the ecosystem on 2026-10-01. transformers 5.14.0 training_args.py:1443 still has warmup_ratio, with a warning at :1487-1489 that it 'will be removed in v5.2' and that its value is copied into warmup_steps. In 5.15.0 and 5.18.0 the field does not exist (0 occurrences), and warmup_steps is typed float (5.15.0:788). MIGRATION_GUIDE_V5.md:627 says 'warmup_ratio -> use warmup_step instead'. My warmup_repro run calls snapshot Forg…
Recommendation. Map training.warmup_ratio onto warmup_steps (as a float) in trainer.py. warmup_steps already accepts floats at the 5.3.0 floor, as the run showed (warmup_steps=0.1). Until that ships, cap transformers<5.15. Add a latest-lane CI job that builds every TRL config type.
Sources. -
TECH-T04-001 (technology, Now) — Distributed training: Adjacent: training.warmup_ratio passed to TRL/TrainingArguments configs (removed in transformers v5.15.0) under the transformers>=5.3.0,<6.0.0 pin (range-unsound)
severity Now
Item. Adjacent: training.warmup_ratio passed to TRL/TrainingArguments configs (removed in transformers v5.15.0) under the transformers>=5.3.0,<6.0.0 pin · verdict:range-unsound· urgency: Now
State in ForgeLM. trainer.py:503 unconditionally passes 'warmup_ratio' into the kwargs used for SFTConfig, DPOConfig, ORPOConfig, CPOConfig, KTOConfig and GRPOConfig (trainer.py:685-760), with no signature filtering. pyproject.toml:53 permits transformers>=5.3.0,<6.0.0, and trl>=1.0.0,<2.0.0.
State of the ecosystem on 2026-10-01. warmup_ratio is present, as a deprecated field, in transformers v5.14.0 (2026-07-15) and absent from v5.15.0 (2026-08-10) through v5.18.0 (2026-09-30), where only warmup_steps (int, or float in [0,1)) remains. Reproduced through ForgeTrainer._get_training_args_for_type() with config_template.yaml: on transformers 5.18.0 with trl 1.14.1 it raises TypeError: SFTConfig.init() got an unexpected k…
Recommendation. Map training.warmup_ratio to warmup_steps (a float in [0,1)) at trainer.py:503, keeping the YAML field. Add a CI lane on the latest transformers/trl that builds every trainer config, and until then cap transformers<5.15 as a stop-gap.
Sources. -
DOC-D1B01-011 (docs, High) — docs/reference/library_api_reference.md: broken example — ForgeTrainer(config: ForgeConfig) is the constructor; ref example 3 and the guide lifecycle call ForgeTrainer(config) then .train()
severity High
Where.docs/reference/library_api_reference.md:54,175,242 (and -tr mirror); docs/guides/library_api.md:94 · type: broken example
The document says. ForgeTrainer(config: ForgeConfig) is the constructor; ref example 3 and the guide lifecycle call ForgeTrainer(config) then .train()
What the code does. ForgeTrainer.init(self, model, tokenizer, config, dataset); ForgeTrainer(config) raises TypeError (missing 3 required positional arguments). The caller must run get_model_and_tokenizer(config) and prepare_dataset(config, tokenizer) first.
Code references.forgelm/trainer.py:340,forgelm/cli/_training.py:140-148
Proposed correction. Document the real 4-argument signature and show the get_model_and_tokenizer/prepare_dataset steps in example 3 and the guide; fix the TR mirrors.
Turkish mirror. the Turkish page has the same error; fix both in the same PR -
DOC-D1B01-012 (docs, High) — docs/reference/library_api_reference.md: broken example —
forgelm.ForgeTrainer(Stable) signature "ForgeTrainer(config: ForgeConfig)". Also :175 exampletrainer = ForgeTrainer(config), :242tr… severity **High** **Where.** [docs/reference/library_api_reference.md:54](https://github.com/HodeTech/ForgeLM/blob/cab2947/docs/reference/library_api_reference.md#L54) (Turkish mirror: [docs/reference/library_api_reference-tr.md:54](https://github.com/HodeTech/ForgeLM/blob/cab2947/docs/reference/library_api_reference-tr.md#L54), :175, :242) · type: broken example **The document says.**forgelm.ForgeTrainer(Stable) signature "ForgeTrainer(config: ForgeConfig)". Also :175 exampletrainer = ForgeTrainer(config), :242trainer = forgelm.ForgeTrainer(config), and :55 "Heavy deps (torch,transformers,trl) load only when this method is called." **What the code does.** The real signature is ForgeTrainer(model, tokenizer, config, dataset) (trainer.py:340), as recorded in tests/_data/api_signatures_1.2.0.json (also 1.0.0 and 1.1.0). The CLI loads the model and tokenizer, prepares the dataset, then calls ForgeTrainer(model=..., tokenizer=..., config=..., dataset=...) (cli/_training.py:139-147). From the signature, ForgeTrainer(config) raises TypeError for three mi… **Code references.** [forgelm/trainer.py:340-371](https://github.com/HodeTech/ForgeLM/blob/cab2947/forgelm/trainer.py#L340-L371), [tests/_data/api_signatures_1.2.0.json](https://github.com/HodeTech/ForgeLM/blob/cab2947/tests/_data/api_signatures_1.2.0.json), [forgelm/cli/_training.py:139-147](https://github.com/HodeTech/ForgeLM/blob/cab2947/forgelm/cli/_training.py#L139-L147) **Proposed correction.** Signature row: "ForgeTrainer(model, tokenizer, config: ForgeConfig, dataset: Dict[str, Any])". Example 3:model, tokenizer = get_model_and_tokenizer(config);dataset = prepare_dataset(config, tokenizer);trainer = ForgeTrainer(model=model, tokenizer=tokenizer, config=config, dataset=dataset)`. Drop the claim that heavy deps load only in tra…
Turkish mirror. the Turkish page has the same error; fix both in the same PR -
DOC-D2B09-007 (docs, High) — docs/product_strategy.md: false claim — 6 trainer methods (SFT/DPO/SimPO/KTO/ORPO/GRPO) in one tool
severity High · already tracked: KNOWN-OPEN GAP-03 — docs/roadmap/phase-16-trust-surface-hardening.md:385-390,868-870
Where.docs/product_strategy.md:74(Turkish mirror:docs/product_strategy-tr.md:74) · type: false claim
The document says. 6 trainer methods (SFT/DPO/SimPO/KTO/ORPO/GRPO) in one tool
What the code does. ORPO and SimPO importORPOTrainerandCPOTrainerfrom the top-leveltrlnamespace. On trl 1.7.1, which is inside the declared range trl>=1.0.0,<2.0.0, both imports raise ImportError (reproduced); the classes live only under trl.experimental. Phase 16 S4 / decision C-10 is still open.
Code references.forgelm/trainer.py:1036-1049,pyproject.toml:59,docs/roadmap/risks-and-decisions.md:190
Proposed correction. Fix the code (C-10 resolver through trl.experimental). Until then add a temporary note: "ORPO and SimPO currently fail to import on trl>=1.0 (tracked as Phase 16 S4)."
Turkish mirror. the Turkish page has the same error; fix both in the same PR -
DOC-D3B05-002 (docs, High) — docs/usermanuals/en/concepts/alignment-overview.md: false claim — ForgeLM gives you all six (SFT, DPO, SimPO, KTO, ORPO, GRPO) behind one declarative interface
severity High · already tracked: KNOWN-OPEN GAP-03 docs/roadmap/phase-16-trust-surface-hardening.md:385-388
Where.docs/usermanuals/en/concepts/alignment-overview.md:8(Turkish mirror:docs/usermanuals/tr/concepts/alignment-overview.md:8) · type: false claim
The document says. ForgeLM gives you all six (SFT, DPO, SimPO, KTO, ORPO, GRPO) behind one declarative interface
What the code does. All six values pass schema validation (config.py:461-464), but only SFT, DPO and KTO can train. With every TRL in the pinned range (pyproject.toml:59 trl>=1.0.0,<2.0.0; checked 1.0.0, 1.7.1, 1.14.1), ORPO stops atfrom trl import ORPOConfig(trainer.py:723) and SimPO atfrom trl import CPOConfig(trainer.py:737) with ImportError, because TRL 1.x moved both to trl.experimental. GRPO stops at G…
Code references.forgelm/trainer.py:723,forgelm/trainer.py:737,forgelm/trainer.py:765,forgelm/trainer.py:1037
Proposed correction. Fix the code. Import ORPOConfig/ORPOTrainer and CPOConfig/CPOTrainer from trl.experimental.orpo / trl.experimental.cpo, with a version-guarded fallback. Drop or rename the GRPOmax_prompt_lengthkwarg. Add a real-TRL smoke test per trainer. Until then, add a warning here: "ORPO, SimPO and GRPO currently fail at trainer construction with TRL 1.x;…
Turkish mirror. the Turkish page has the same error; fix both in the same PR -
DOC-D3B05-017 (docs, High) — docs/usermanuals/en/training/orpo.md: broken example — Quick example
trainer_type: "orpo"trains an ORPO model
severity High · already tracked: KNOWN-OPEN GAP-03 docs/roadmap/phase-16-trust-surface-hardening.md:382-392
Where.docs/usermanuals/en/training/orpo.md:23-46(Turkish mirror:docs/usermanuals/tr/training/orpo.md:23-46) · type: broken example
The document says. Quick exampletrainer_type: "orpo"trains an ORPO model
What the code does. The config validates (dry-run exit 0), but training fails atfrom trl import ORPOConfig(trainer.py:723) with ImportError ('cannot import name ORPOConfig from trl') on TRL 1.0.0/1.7.1/1.14.1. ORPO now lives in trl.experimental.orpo. A real run exited 2.
Code references.forgelm/trainer.py:723,forgelm/trainer.py:1037,pyproject.toml:59
Proposed correction. Fix the code (import from trl.experimental.orpo with a fallback). Add a temporary warning: "ORPO currently fails with TRL 1.x (ImportError: ORPOConfig)."
Turkish mirror. the Turkish page has the same error; fix both in the same PR -
DOC-D3B05-019 (docs, High) — docs/usermanuals/en/training/simpo.md: broken example — Quick example
trainer_type: "simpo"trains a SimPO model
severity High · already tracked: KNOWN-OPEN GAP-03 docs/roadmap/phase-16-trust-surface-hardening.md:382-392
Where.docs/usermanuals/en/training/simpo.md:22-46(Turkish mirror:docs/usermanuals/tr/training/simpo.md:22-46) · type: broken example
The document says. Quick exampletrainer_type: "simpo"trains a SimPO model
What the code does. The config validates (dry-run exit 0), but training fails atfrom trl import CPOConfig(trainer.py:737) with ImportError on TRL 1.0.0, 1.7.1 and 1.14.1, all inside the pinned range (pyproject.toml:59). CPO now lives in trl.experimental.cpo. A real run exited 2.
Code references.forgelm/trainer.py:737,forgelm/trainer.py:1047,pyproject.toml:59
Proposed correction. Fix the code (import CPOConfig/CPOTrainer from trl.experimental.cpo with a fallback). Add a temporary warning at the top: "SimPO currently fails with TRL 1.x (ImportError: CPOConfig)."
Turkish mirror. the Turkish page has the same error; fix both in the same PR -
DOC-D4B05-001 (docs, High) — notebooks/grpo_reasoning.ipynb: broken example — After the dry-run passes,
!forgelm --config grpo_config.yamltrains a GRPO model and writes ./grpo_checkpoints/final_model
severity High
Where.notebooks/grpo_reasoning.ipynb:132(Turkish mirror:forgelm/trainer.py:765) · type: broken example
The document says. After the dry-run passes,!forgelm --config grpo_config.yamltrains a GRPO model and writes ./grpo_checkpoints/final_model
What the code does. The training step crashes before the first step. forgelm/trainer.py:765 callsself._apply_max_length(kwargs, "max_prompt_length")and trainer.py:780 returnsGRPOConfig(**kwargs). TRL's GRPOConfig has nomax_prompt_lengthparameter on trl 1.0.0 (the pyproject lower bound) or 1.7.1, so construction raisesTypeError: GRPOConfig.__init__() got an unexpected keyword argument 'max_prompt_length'… **Code references.** [forgelm/trainer.py:765](https://github.com/HodeTech/ForgeLM/blob/cab2947/forgelm/trainer.py#L765), [forgelm/trainer.py:780](https://github.com/HodeTech/ForgeLM/blob/cab2947/forgelm/trainer.py#L780), [tests/test_trainer_max_length.py:77-96](https://github.com/HodeTech/ForgeLM/blob/cab2947/tests/test_trainer_max_length.py#L77-L96), [pyproject.toml:59`](https://github.com/HodeTech/ForgeLM/blob/cab2947/pyproject.toml#L59) (trl>=1.0.0,<2.0.0)
Proposed correction. Fix the code (see rationale). Until it is fixed, add a note above Step 6: 'Known issue: GRPO training currently fails with TypeError (max_prompt_length) on TRL >= 1.0; use the SFT, DPO or KTO notebooks until a fixed ForgeLM release.' -
DOC-D4B05-003 (docs, High) — notebooks/quickstart_sft.ipynb: broken example — After
pip install 'forgelm[qlora]==0.11.0'(transformers/trl unpinned) the!forgelm --config my_config.yamltraining cell runs
severity High
Where.notebooks/quickstart_sft.ipynb:27-28, :201 (Turkish mirror:same cell in dpo_alignment, kto_binary_feedback, grpo_reasoning, galore_memory_optimization, multi_dataset) · type: broken example
The document says. Afterpip install 'forgelm[qlora]==0.11.0'(transformers/trl unpinned) the!forgelm --config my_config.yamltraining cell runs
What the code does. pyproject allows transformers>=5.3.0,<6 and trl>=1.0.0,<2. ForgeLM builds every TRL config withwarmup_ratio(trainer.py:503). In the review harness's 'latest' venv (transformers 5.18.0, trl 1.14.1) every trainer type fails withTypeError: SFTConfig.__init__() got an unexpected keyword argument 'warmup_ratio'(real CLI run of this notebook's config, tiny-qwen; also DPO/KTO/GRPO/GaLore/multi c…
Code references.forgelm/trainer.py:503,pyproject.toml:53-59
Proposed correction. Fix the code (replace warmup_ratio with warmup_steps or probe the signature) and release; until then pin'transformers<5.18'next to the forgelm pin in the Step 1 install cell of every notebook. -
DOC-BLEGAL-152 (docs, Medium) — notebooks/grpo_reasoning.ipynb: broken example — !pip install -q --no-cache-dir 'forgelm[qlora]==0.11.0' bitsandbytes (the notebook then documents and runs the GRPO flow)
severity Medium
Where.notebooks/grpo_reasoning.ipynbcell 1 · type: broken example
The document says. !pip install -q --no-cache-dir 'forgelm[qlora]==0.11.0' bitsandbytes (the notebook then documents and runs the GRPO flow)
What the code does. forgelm 0.11.0 is on PyPI (uploaded 2026-07-21) and is the latest release, but it is not the snapshot's code: trainer.py, config.py, safety/*, judge.py, webhook.py and others differ. Its trainer.py:467 passes warmup_ratio unconditionally. transformers removed TrainingArguments.warmup_ratio in v5.15.0, and a fresh resolve within forgelm's transformers>=5.3,<6 pin, or the Colab image (transformers…
Proposed correction. Pintransformers<5.15alongside forgelm==0.11.0 until a release drops warmup_ratio, and note that the notebook runs the 0.11.0 release. -
DOC-BLEGAL-154 (docs, Medium) — notebooks/quickstart_sft.ipynb: broken example — !pip install -q --no-cache-dir 'forgelm[qlora]==0.11.0' (quickstart SFT run)
severity Medium
Where.notebooks/quickstart_sft.ipynbcell 1 · type: broken example
The document says. !pip install -q --no-cache-dir 'forgelm[qlora]==0.11.0' (quickstart SFT run)
What the code does. forgelm 0.11.0 is published and is the latest release, but it differs from the snapshot code. Its trainer passes warmup_ratio unconditionally (wheel trainer.py:467), and SFTConfig on transformers >= 5.15 (Colab: 5.18.0) raises TypeError on warmup_ratio.
Proposed correction. Add 'transformers>=5.3,<5.15' to the install line until a fixed release ships. -
DOC-D1B01-024 (docs, Medium) — docs/reference/library_api_reference.md: stale — Canonical TrainResult fields list
severity Medium
Where.docs/reference/library_api_reference.md:56· type: stale
The document says. Canonical TrainResult fields list
What the code does. TrainResult also has awaiting_approval, run_id and config_hash; awaiting_approval is the authoritative discriminator for 'staged pending human approval' (CLI exit 4).
Code references.forgelm/results.py:34-52
Proposed correction. Add the three fields and note that success=True with awaiting_approval=True means staged.
Turkish mirror. the Turkish page has the same error; fix both in the same PR -
DOC-D3B05-043 (docs, Medium) — docs/usermanuals/en/training/simpo.md: wrong detail — TRL's CPO/SimPO trainer runs with library defaults (sigmoid loss, length normalisation always on, no label smoothing)
severity Medium
Where.docs/usermanuals/en/training/simpo.md:62(Turkish mirror:docs/usermanuals/tr/training/simpo.md:63) · type: wrong detail
The document says. TRL's CPO/SimPO trainer runs with library defaults (sigmoid loss, length normalisation always on, no label smoothing)
What the code does. ForgeLM sets loss_type='simpo' and cpo_alpha=0.0 explicitly (trainer.py:736-745). TRL's CPOConfig defaults would be loss_type 'sigmoid' and cpo_alpha 1.0.
Code references.forgelm/trainer.py:734-745
Proposed correction. Say: "ForgeLM fixesloss_type=\"simpo\"andcpo_alpha=0(pure SimPO, no NLL term); label smoothing stays at TRL's default 0."
Turkish mirror. the Turkish page has no such statement -
DOC-D4B01-022 (docs, Medium) — README.md: misleading — "on CUDA OOM the batch is halved and gradient accumulation doubled, preserving the effective batch, then the step retries"
severity Medium
Where.README.md:154· type: misleading
The document says. "on CUDA OOM the batch is halved and gradient accumulation doubled, preserving the effective batch, then the step retries"
What the code does. This is opt-in:training.oom_recoverydefaults to False, and without it the OOM is re-raised. When enabled, the retry rebuilds the trainer and re-enterstrain()from step 0 with a fresh optimizer and scheduler (or from the explicit --resume checkpoint). The failing step is not retried.
Code references.forgelm/config.py:598-600,forgelm/trainer.py:1196-1207,forgelm/trainer.py:1229-1235,forgelm/trainer.py:1262-1270
Proposed correction. "withtraining.oom_recovery: true(off by default), a CUDA OOM halves the batch and doubles gradient accumulation (effective batch preserved) and restarts training from step 0 (or the --resume checkpoint) with a fresh optimizer" -
DOC-D1B01-045 (docs, Low) — docs/reference/library_api_reference.md: wrong detail —
ForgeTrainer.trainsignature "train() -> TrainResult"
severity Low
Where.docs/reference/library_api_reference.md:55(Turkish mirror:docs/reference/library_api_reference-tr.md:55) · type: wrong detail
The document says.ForgeTrainer.trainsignature "train() -> TrainResult"
What the code does. The method is train(self, resume_from_checkpoint: Optional[str] = None) -> TrainResult.
Code references.forgelm/trainer.py:1661
Proposed correction. "train(resume_from_checkpoint: Optional[str] = None) -> TrainResult"
Turkish mirror. the Turkish page has the same error; fix both in the same PR -
DOC-D1B01-046 (docs, Low) — docs/reference/library_api_reference.md: stale — TrainResult canonical fields (per forgelm/results.py): success ... staging_path, resource_usage
severity Low
Where.docs/reference/library_api_reference.md:56(Turkish mirror:docs/reference/library_api_reference-tr.md:56) · type: stale
The document says. TrainResult canonical fields (per forgelm/results.py): success ... staging_path, resource_usage
What the code does. The field list also includes awaiting_approval, run_id and config_hash. awaiting_approval is the discriminator the CLI uses for exit 4.
Code references.forgelm/results.py,tests/_data/api_signatures_1.2.0.json,forgelm/cli/_training.py:156-161
Proposed correction. Addawaiting_approval: bool,run_id: Optional[str],config_hash: Optional[str].
Turkish mirror. the Turkish page has the same error; fix both in the same PR
Acceptance criteria
- A regression test reproduces the failure scenario and asserts the corrected behaviour.
- Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
- Every finding in this issue is resolved or explicitly closed with a reason.
- The release note names the contract this fix restores.
Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.
Activity
fix: resolve ORPO and SimPO classes across TRL export paths
fix: resolve dynamic TRL exports without a dir precondition
fix: pass warmup through the supported Transformers parameter
fix: keep full-run warmup fractional at a ratio of one
fix: restore training compatibility and model artifact integrity (#934)
cemililik commentedon Oct 17, 2026
Progress on
development— verified 2026-10-18 at6c55900.Fixed:
CR-A02-013(withTECH-T01-001,TECH-T01-004) — ORPO and SimPO resolve their trainer and config classes through a compatibility resolver instead of fixed import names (e0eb32d,1b800a5;test_real_installed_preference_symbols)CR-A02-015,CR-A01-081(withTECH-T01-002,TECH-T02-001,TECH-T04-001) — warmup goes through the supportedwarmup_stepsargument and a ratio of 1.0 is clamped below 1 (9218b4a,39d328c;tests/test_trainer.pybuilds every trainer type's arguments through realTrainingArguments)DOC-D3B05-017,DOC-D3B05-019,DOC-D3B05-043— the ORPO and SimPO pages (EN/TR) carry the experimental-API caveat and the corrected SimPOloss_type/cpo_alphawording (e0eb32d)DOC-D4B01-022— the README out-of-memory claim is gone (5971d58)Partly fixed:
CR-A02-028— ORPO, SimPO and warmup are fixed; the GRPOTypeErrorbelow remainsCR-A16-022,TECH-T01-003— real ORPO and SimPO construction now runs on CPU and in the nightly jobs; there is still no per-trainer-type smoke test for SFT, DPO, KTO and GRPO, andfilterwarningsstill ignoresDeprecationWarningStill open, most important first:
CR-A02-014(withTECH-T01-005) — GRPO still receivesmax_prompt_length(forgelm/trainer.py,_apply_max_length(kwargs, "max_prompt_length")), whichGRPOConfigdoes not define in TRL 1.7.1; its only test uses a fake configDOC-D2B09-007,DOC-D3B05-002(the "all six trainers" claims), the notebooks (DOC-D4B05-001,-003,DOC-BLEGAL-152,-154) and the library API reference (DOC-D1B01-011,-012,-024,-045,-046)Everything else in this issue is unchanged.