Repository navigation
[Triage] Site and CLI help name lm-eval task 'truthfulqa', which does not exist as a task name in lm-eval 0.4.x (CR-A18-099) #830
Description
Needs triage. Reviewers disagreed on this finding, so it is not counted in the backlog. Decide whether it is a real defect: if yes, set severity and wave; if not, close it with the reason.
CR-A18-099 — Site and CLI help name lm-eval task 'truthfulqa', which does not exist as a task name in lm-eval 0.4.x
severity Low · confidence disputed · single reviewer · not reproduced (code trace)
What happens. The landing demo, the wizard benchmark hint in all 6 locales, and forgelm/cli/_parser.py help text use 'truthfulqa' as an lm-eval task. In lm-evaluation-harness v0.4.x the tasks are truthfulqa_mc1, truthfulqa_mc2 and truthfulqa_gen. The project's own TR user manual uses truthfulqa_mc1. The one-line status of this check is that it rests on my knowledge of lm-eval, and I could not confirm it against the installed package here.
Failure scenario. A user copies the demo task list or follows the hint. lm-eval raises a task-not-found error at benchmark time, after training has finished, and the evaluation gate does not run.
Where.
Evidence.
- site/index.html:574; site/js/translations.js lines 1739, 2134, 2529, 2924, 3319, 3714 (wizard.benchmark.tasks.hint); forgelm/cli/_parser.py:1038 and 1046. The check against lm-eval task registries was by recollection of v0.4.4 through main, not by running them.
Recommended fix. Use truthfulqa_mc2 (or _mc1) in examples and hints, and validate task names against lm-eval's registry in a test.
Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.
Activity
cemililik commentedon Oct 20, 2026
Thanks for flagging this. I checked the lm-evaluation-harness source at v0.4.4 and v0.4.13: there is no truthfulqa.yaml, but truthfulqa_gen, truthfulqa_mc1 and truthfulqa_mc2 each declare the tag 'truthfulqa', and the task manager resolves tag names by expanding them to those three tasks.
So 'truthfulqa' is accepted by lm-eval and no task-not-found error occurs; the name is a tag rather than a task.
What is real is the consequence: it runs three tasks, no 'truthfulqa' score key is produced, and truthfulqa_gen is scored through a bleu_acc fallback. That is tracked in #120, where the examples can move to explicit names such as truthfulqa_mc2.
Closing as not a defect; #823 reaches the same conclusion.
Triage round 1, checked against
developmenton 2026-10-20.