Declared lm-eval floor 0.4.4 is nominal under ForgeLM's own datasets>=4.7 floor: documented tasks rely on trust_remote_code dataset scripts there, humaneval does not exist at 0.4.… (CR-A16-060) #527

Wave 2 · theme D2-7: Next-release technology updates · highest severity Low · 2 finding(s) · effort trivial

Summary

A deployer installs forgelm[eval] with lm-eval pinned to the declared floor 0.4.4 (constraints file, reproducibility pin) and configures tasks: [hellaswag, mmlu, humaneval]. humaneval is an unknown task, and the script-based datasets fail to load under datasets>=4.7. simple_evaluate raises, the gate reports passed=False, and with auto_revert the trained model is deleted.

Primary finding

CR-A16-060 — Declared lm-eval floor 0.4.4 is nominal under ForgeLM's own datasets>=4.7 floor: documented tasks rely on trust_remote_code dataset scripts there, humaneval does not exist at 0.4.4, and 0.4.9.1 is un…

severity Low · reachability documented-opt-in · intent contract-violation · confidence likely (static only) · independently confirmed · not reproduced (code trace)

What happens. pyproject.toml:84 declares lm-eval>=0.4.4,<1.0.0, while core requires datasets>=4.7.0 (line 55). (1) The 0.4.4 task configs for three of the tasks the user manual lists (docs/usermanuals/en/evaluation/benchmarks.md:39-47) load legacy Hub datasets with dataset_kwargs: trust_remote_code: true: hellaswag (tasks/hellaswag/hellaswag.yaml:24, dataset_path 'hellaswag'), winogrande (tasks/winogrande/default.yaml:19, 'winogrande') and mmlu (tasks/mmlu/default/_default_template_yaml:17, 'hails/mmlu_no_train'). In all; 43 task YAMLs at 0.4.4 set it. Under datasets 4.7.0, 5.0.0 and 5.0.1, load_dataset only logs 'trust_remote_code is not supported anymore' and drops the kwarg, so any of those datasets that still needs a loading script cannot load. The lm-eval 0.4.9.1 release notes say: 'Pinned datasets < 4.0 temporarily to maintain compatibility with trust_remote_code'. From 0.4.9.1 the same tasks moved to Rowan/hellaswag, allenai/winogrande and cais/mmlu. (2) lm-eval 0.4.4 has no humaneval task at all; it first appears in this range by 0.4.9.1. Yet the docs list humaneval as supported (separate from the already-reported confirm_run_unsafe_code issue on newer versions). (3) lm-eval 0.4.9.1 itself requires datasets<4.0, so it can never be installed alongside ForgeLM. A floor-pinned install therefore yields a gate whose documented tasks fail at simple_evaluate time; through the already-reported infrastructure-failure path that becomes passed=False and an auto-revert.

Where.

• pyproject.toml:84
• pyproject.toml:55
• docs/usermanuals/en/evaluation/benchmarks.md:41-47
• forgelm/benchmark.py:187-190
• requirements-eval.txt:1
• docs/usermanuals/tr/evaluation/benchmarks.md:47
• docs/usermanuals/en/deployment/model-merging.md:88
• .github/workflows/nightly.yml:216-242

Evidence.

• Wheels downloaded from PyPI and unzipped (sha256 verified): lm_eval-0.4.4 37f1ab84..., 0.4.9.1 315e5a3a..., 0.4.12 02971ff6..., 0.4.13 5daaa197.... grep shows hellaswag/winogrande/mmlu trust_remote_code at 0.4.4, a 'humaneval' task directory in 0.4.9.1/0.4.12/0.4.13 but not in 0.4.4, and dataset_path values changing as described. PyPI JSON requires_dist for 0.4.9.1 is 'datasets<4.0,>=2.16.0'. datasets.load source (4.7.0/5.0.0/5.0.1) pops trust_remote_code and calls logger.error. The GitHub rele…

Recommended fix. Raise the lm-eval floor to the first release whose built-in task configs need no dataset scripts and are compatible with datasets>=4 (0.4.9.2 or later per its metadata; verify), and exclude 0.4.9.1.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

TECH-T05-003 — Evaluation and safety models: lm-eval>=0.4.4,<1.0.0 (extra [eval], pyproject.toml:84) against transformers>=5.3.0,<6.0.0 (range-unsound)

severity Next release

Item. lm-eval>=0.4.4,<1.0.0 (extra [eval], pyproject.toml:84) against transformers>=5.3.0,<6.0.0 · verdict: range-unsound · urgency: Next release

State in ForgeLM. pyproject.toml:84 declares lm-eval>=0.4.4,<1.0.0 and pyproject.toml:53 declares transformers>=5.3.0,<6.0.0. forgelm/benchmark.py:172-173 runs import lm_eval and from lm_eval.models.huggingface import HFLM outside any try. _check_lm_eval_available (benchmark.py:27-36) catches only ImportError.

State of the ecosystem on 2026-10-01. No 1.0 exists. The latest stable is 0.4.13 (2026-08-31). 0.5.0.dev1 (2026-05-11) is a pre-release, which pip skips by default. lm-eval's own requirement is 'transformers>=4.1; extra == "hf"', so a resolver may pair any 0.4.x with transformers 5. In v0.4.5 and v0.4.9.2, lm_eval/models/init.py eagerly imports hf_vlms. In v0.4.9.2, hf_vlms.py:37 sets AUTO_MODEL_CLASS = transformers.AutoModelForVision2Seq. The arbiter checked that hasattr(transformers, 'AutoModelForVision2Seq') is False on transformers 5.3.0, 5.13.0 and 5.18.0. v0.4.10 made model imports lazy and 'Removed deprecated AutoModelForVision2Seq' (#​3522).

Impact. A fresh install resolves to 0.4.13 and works. A pre-existing environment or a constraints file that holds lm-eval 0.4.5-0.4.9.2 makes from lm_eval.models.huggingface import HFLM raise AttributeError at package import. That fails the benchmark step closed, outside BenchmarkResult. The declared floor therefore admits releases that cannot import under the pinned transformers. This is from reading the source; those versions were not installed (no pip). The 0.4.4 floor itself was not checked. The <1.0.0 cap would admit a 0.5.0 final, which the dev series suggests changes APIs.

Recommendation. Raise the floor to lm-eval>=0.4.10 and exercise it in the lower K-3 lane. Consider capping at <0.5 until 0.5.0 final has been evaluated. Log the installed lm_eval version in the benchmark audit event, since 0.4.13's few-shot leakage fix shifts scores.

Sources.

• lm_eval/models/hf_vlms.py @ v0.4.9.2 (2025-11-26)
• lm_eval/models/init.py @ v0.4.9.2 (2025-11-26)
• lm_eval/models/init.py @ v0.4.5 (2024-10-08)
• lm_eval/models/init.py @ v0.4.10 (lazy MODEL_MAPPING) (2026-01-27)
• PyPI JSON: lm-eval (2026-08-31)
• lm-evaluation-harness GitHub releases (2026-08-31 (latest))

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}