[Triage] cache-tasks depends on lm_eval.tasks.get_task_dict, deprecated in lm-eval 0.4.12+, with the declared cap at <1.0.0 (CR-A15-062) #824

Needs triage. Reviewers disagreed on this finding, so it is not counted in the backlog. Decide whether it is a real defect: if yes, set severity and wave; if not, close it with the reason.

CR-A15-062 — cache-tasks depends on lm_eval.tasks.get_task_dict, deprecated in lm-eval 0.4.12+, with the declared cap at <1.0.0

severity Low · confidence disputed · single reviewer · not reproduced (code trace)

What happens. In lm-eval 0.4.13 (2026-08-31), get_task_dict is wrapped in @​deprecated('Use TaskManager.load() instead'). The 0.4.12 release notes call it, together with load_task_or_group, a 'deprecated shim' returning the legacy nested shape. TaskManager.load now returns a flat {tasks, groups} dict. Each call emits a DeprecationWarning, and the shim can be removed in a later 0.x or 0.5 release, which the <1.0.0 cap still allows. The Task and ConfigurableGroup shapes also moved (ConfigurableGroup is now a deprecated wrapper), and _prepare_one_task's duck-typing of .dataset was written against the old shape. The eval extra declares lm-eval>=0.4.4, but 0.4.4 through 0.4.8 construct tasks with datasets.load_dataset(..., trust_remote_code=...), which datasets>=4 (core floor 4.7.0) no longer accepts. Whether the floor combination works was not tested.

Where.

• forgelm/cli/subcommands/_cache.py:409
• forgelm/cli/subcommands/_cache.py:453
• pyproject.toml:84

Evidence.

• lm-eval v0.4.13 lm_eval/tasks/init.py:136 '@​deprecated("get_task_dict is deprecated. Use TaskManager.load() instead.")'. GitHub release v0.4.12 body (TaskManager refactor). PyPI requires_dist: 0.4.9.1 pins datasets<4.0; 0.4.4 to 0.4.8 only require datasets>=2.16.0.

Recommended fix. Prefer TaskManager().load(...) where available, with the legacy path as a fallback. Raise the eval floor to a release known to work with datasets>=4, and add a CI job against lm-eval latest.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}