training.rope_scaling (as documented and as the wizard emits it) crashes every model load on transformers 5.x, while the validator and --dry-run accept it (CR-A01-005) #103

Wave 0 · theme D0-7: Dependency ranges and security floors · highest severity High · 6 finding(s) · effort medium

Summary

An operator follows the docs or the wizard (which emits {'type': ..., 'factor': max_length/base_context}) to extend the context. Dry-run passes in CI, then the training job fails at model load (exit 2).

Primary finding

CR-A01-005 — training.rope_scaling (as documented and as the wizard emits it) crashes every model load on transformers 5.x, while the validator and --dry-run accept it

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. _build_model_kwargs forwards training.rope_scaling verbatim as from_pretrained(rope_scaling=...) (forgelm/model.py:463-466). In transformers 5.x this replaces config.rope_parameters wholesale without standardisation, so rope_theta is lost and the legacy 'type' key is not mapped to 'rope_type'. {type: ...} raises KeyError 'rope_type'; {rope_type: ...} raises TypeError (NoneType ** Tensor). _validate_rope_scaling (config.py:638-711) accepts both shapes, as well as NaN/inf factors. The wizard's rope collector (wizard/_collectors.py:565-602) and the docs emit the failing shape.

Where.

• forgelm/config.py:572-581 (confirmed)
• forgelm/config.py:638-711 (confirmed)
• forgelm/model.py:463-466 (the verbatim kwarg forward; line 585 is the from_pretrained call that raises)
• forgelm/wizard/_collectors.py:602 (emitting return)
• forgelm/config.py:638-700
• config_template.yaml:89-91 (confirmed)
• docs/usermanuals/en/training/long-context.md:48-51
• docs/reference/configuration.md:126 (confirmed)
• forgelm/model.py:584 (from_pretrained sink; VLM sink at 582)
• forgelm/config.py:638-712 (confirmed)

Evidence.

• Direct from_pretrained(tiny-qwen, rope_scaling=...): transformers 5.3.0 (lower venv), 5.13.0 (default) and 5.18.0 (latest) all give: {'type':'linear','factor':4.0} -> KeyError 'rope_type' (modeling_qwen2.py:61); {'rope_type':'linear', factor 4.0/nan/inf/0.0001} -> TypeError unsupported operand NoneType ** Tensor. CLI: cfg_rope.yaml (sft + rope_scaling {type: linear, factor: 2.0}): --dry-run exit 0; real run exit 2 with the KeyError traceback. Schema probe: factor nan/inf and longrope short_fact…
• Ran scr/rope.py (get_model_and_tokenizer on env/tiny-qwen with rope_scaling {type: linear, factor: 4.0}) for load_in_4bit 0 and 1: KeyError: 'rope_type' at modeling_qwen2.py:61. With {rope_type: linear, factor: 4.0}: TypeError at modeling_rope_utils.py:178. Reproduced on transformers 5.3.0 (lower venv), 5.13.0 and 5.18.0 (latest venv). Control (scr/rope2.py): adding rope_theta to the dict loads correctly with rope_type 'linear'.
• Called forgelm.model.get_model_and_tokenizer with a ForgeConfig on tiny-qwen (snapshot lane, transformers 5.13.0). rope_scaling {'type':'linear','factor':4.0} raised KeyError 'rope_type'. {'rope_type':'yarn','factor':4.0} raised TypeError: unsupported operand type(s) for ** or pow(): 'NoneType' and 'Tensor'. A direct AutoModelForCausalLM.from_pretrained(tiny-qwen, rope_scaling=X) gives the same failures for linear, dynamic and yarn, in both key spellings, on transformers 5.3.0 (floor), 5.13.0 a…

Recommended fix. Merge the operator's dict into the loaded config's rope_parameters: load AutoConfig, update rope_parameters with rope_type (mapped from type) plus the factor keys while keeping rope_theta, then pass config=. Add an integration test that loads a real tiny model with rope_scaling. Add math.isfinite to _check_factor, and finiteness and positivity checks to the longrope list items.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

• TECH-T02-003 (technology, Now) — Core Hugging Face stack and runtime dependencies: transformers rope_scaling kwarg passed to from_pretrained (forgelm/model.py:463-466) and the training.rope_scaling schema/docs (deprecated-in-use)
  severity Now
  Item. transformers rope_scaling kwarg passed to from_pretrained (forgelm/model.py:463-466) and the training.rope_scaling schema/docs · verdict: deprecated-in-use · urgency: Now
  State in ForgeLM. forgelm/model.py:463-466 forwards training.rope_scaling unchanged as from_pretrained(rope_scaling=...). The validator (config.py:638-711) accepts {type|rope_type, factor} without rope_theta, and config.py:579 and :655-657 call type/rope_type interchangeable aliases. The examples at config_template.yaml:89-91, docs/reference/configuration.md:126 (TR :124) and docs/usermanuals/en/training/long-cont…
  State of the ecosystem on 2026-10-01. transformers 5 stores every RoPE setting, rope_theta and rope_type included, in config.rope_parameters (MIGRATION_GUIDE_V5.md:513). rope_scaling survives only as a backward-compat alias whose setter replaces rope_parameters wholesale (5.18.0 configuration_utils.py:585-590). It emits no deprecation warning, and no removal version is announced. Both researchers reproduced the failure on tiny-qwen w…
  Recommendation. Stop passing rope_scaling in model.py. Load AutoConfig, merge the user dict into cfg.rope_parameters (map type to rope_type and keep rope_theta), then pass config= or rope_parameters=. Fix the examples and the alias wording in config.py. Add a tiny-qwen load test on both the floor and the latest transformers.
  Sources.

  • Transformers v5 migration guide (rope_parameters, line 513) (2026-10-01)
  • transformers 5.18.0 wheel (rope_scaling BC alias) (2026-09-30)

• DOC-BFIELD-001 (docs, High) — forgelm/config.py: broken example — training.rope_scaling: linear/dynamic/yarn take a scalar factor (e.g. {"type": "linear", "factor": 4.0}); longrope takes short_factor/long_factor; the tr…
  severity High
  Where. forgelm/config.py:572-581 · type: broken example
  The document says. training.rope_scaling: linear/dynamic/yarn take a scalar factor (e.g. {"type": "linear", "factor": 4.0}); longrope takes short_factor/long_factor; the transformers-5 rope_type key is accepted as an alias for type.
  What the code does. Every shape the description documents, and the validator accepts (config.py:683-711), crashes the model load. model.py:463-466 passes the dict to from_pretrained as the rope_scaling= kwarg, which on transformers 5.x replaces the config's whole rope_parameters. The Qwen2 rotary embedding then finds no rope_type/rope_theta. Reproduced with forgelm.model.get_model_and_tokenizer on the tiny-qwe…
  Code references. forgelm/model.py:463-466, forgelm/config.py:683-711, pyproject.toml:53
  Proposed correction. Fix the consumer: merge into config.rope_parameters and canonicalise rope_type. Add an integration test that loads a tiny model with each documented shape.

• DOC-BLEGAL-002 (docs, High) — config_template.yaml: broken example — Commented example rope_scaling: {type: "linear", factor: 4.0}
  severity High
  Where. config_template.yaml:89-91 · type: broken example
  The document says. Commented example rope_scaling: {type: "linear", factor: 4.0}
  What the code does. transformers 5 stores RoPE settings in config.rope_parameters. The rope_scaling alias replaces them wholesale, so {'type':'linear','factor':4.0} raises KeyError 'rope_type' at model init, and {'rope_type':'linear','factor':4.0} raises TypeError because rope_theta is lost. Both researchers reproduced this on tiny-qwen with 5.3.0, 5.13.0 and 5.18.0. Only a full rope_parameters dict works.
  Proposed correction. Replace with a working transformers-5 form (full rope_parameters including rope_type and rope_theta), or have ForgeLM merge the keys into the model's existing rope_parameters, and update the example to match.

• DOC-BLEGAL-015 (docs, High) — docs/reference/configuration.md: broken example — rope_scaling: "RoPE scaling method dict ({"type": "linear", "factor": 2.0} etc.)"
  severity High
  Where. docs/reference/configuration.md:126 (Turkish mirror: docs/reference/configuration-tr.md:124) · type: broken example
  The document says. rope_scaling: "RoPE scaling method dict ({"type": "linear", "factor": 2.0} etc.)"
  What the code does. Under transformers 5.x the rope_scaling alias replaces rope_parameters wholesale. {'type':'linear',...} raises KeyError 'rope_type' at model init (reproduced on 5.3.0, 5.13.0 and 5.18.0).
  Proposed correction. Document a working transformers-5 form (full rope_parameters with rope_type and rope_theta) or fix the code to merge into existing rope_parameters.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-BLEGAL-037 (docs, High) — docs/usermanuals/en/training/long-context.md: broken example — Qwen/Qwen2.5-7B-Instruct example with "rope_scaling: {type: linear, factor: 4.0}" and "sliding_window_attention: 4096 # process in 4K wi…
  severity High
  Where. docs/usermanuals/en/training/long-context.md:48-51 (Turkish mirror: docs/usermanuals/tr/training/long-context.md:48-51) · type: broken example
  The document says. Qwen/Qwen2.5-7B-Instruct example with "rope_scaling: {type: linear, factor: 4.0}" and "sliding_window_attention: 4096 # process in 4K windows"
  What the code does. The rope_scaling block crashes at model load (KeyError 'rope_type') on transformers 5.3.0-5.18.0. On Qwen2 the sliding-window override does nothing: layer_types stay all full_attention (arbiter sw_check repro on 5.3.0 and 5.18.0).
  Proposed correction. Use a working rope_parameters form and drop sliding_window_attention from the Qwen example (or switch the example to a Mistral-style model), with a note that Qwen2/3 ignore it.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.
• The release note names the contract this fix restores.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}