Benchmark gate averages whatever rows lm-eval returned instead of the configured tasks: tasks without an acc metric or missing from the results are dropped, and group/subtask rows… (CR-A06-004) #120

Wave 1 · theme D1-1: Correctness of the evaluation gates · highest severity High · 15 finding(s) · effort medium

Summary

tasks [gsm8k, arc_easy] with min_score 0.5: gsm8k has no acc key and is silently skipped, so a model that collapsed on gsm8k is promoted on the strength of arc_easy alone. The audit log and envelope report passed=true.

Primary finding

CR-A06-004 — Benchmark gate averages whatever rows lm-eval returned instead of the configured tasks: tasks without an acc metric or missing from the results are dropped, and group/subtask rows are weighted as tas…

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end · already tracked: CLAIMED-FIXED-STILL-PRESENT CORE-03 — docs/roadmap/phase-16-trust-surface-hardening.md:290,310-313,878 (S2 done); CHANGELOG.md:26-33

What happens. _parse_results (forgelm/benchmark.py:66-97) iterates raw_results.items() and skips any task for which _extract_task_score (:43-63, acc/acc_norm keys only) returns None. run_benchmark averages the survivors unweighted (:195) and compares that average with min_score (:223). The configured tasks list is never reconciled with the results. A configured task with exact_match/pass@k metrics (gsm8k, humaneval) or with no result row is silently excluded, so the gate passes on the rest, against the docstring (:71-74) and the docs' 'mean across all listed tasks'. lm-eval group, subgroup and subtask rows are averaged as separate tasks too, so a large group such as mmlu dominates the mean. If every task is dropped, scores={} gives average 0.0.

Where.

• forgelm/benchmark.py:83-86
• forgelm/benchmark.py:193-195 (confirmed)
• forgelm/benchmark.py:72-74
• forgelm/benchmark.py:83-96
• forgelm/benchmark.py:229
• docs/usermanuals/en/evaluation/benchmarks.md:18-19
• docs/usermanuals/en/evaluation/benchmarks.md:53
• docs/usermanuals/en/evaluation/benchmarks.md:123
• forgelm/benchmark.py:43-63 (confirmed)
• forgelm/benchmark.py:71-74

Evidence.

• B-4 (bench-missing-task, min_score 0.55, auto_revert true): exit 0, benchmark.evaluation_completed passed=true average 0.6 scores {arc_easy: 0.6}; benchmark_results.json lists tasks [arc_easy, hellaswag] but scores only arc_easy. B-5 (bench-stderr-only): passed on hellaswag alone (0.4 >= 0.3).
• runs/bench_group (custom simple_evaluate answer shaped like lm-eval 0.4 group output): exit 0, benchmark {scores: arc_easy 0.2, mmlu 0.7, mmlu_humanities 0.7, mmlu_formal_logic 0.7, mmlu_stem 0.7, mmlu_abstract_algebra 0.7; average 0.6167; passed true}.
• runs/bench_noacc: passed, scores {arc_easy: 0.6}, warning 'gsm8k: no accuracy metric found'. runs/bench_missing (bench-missing-task, min 0.5): passed, no warning. runs/bench_allnoacc_nomin: passed, scores {}, average 0.0. runs/bench_allnoacc_min (min 0.2, auto_revert): exit 3, model deleted, error 'Average benchmark score (0.0000) is below minimum threshold (0.2000).'

Recommended fix. Treat every configured task that yields no usable score as invalid and fail the gate, as is already done for non-finite scores. Record the missing tasks in failure_reason and in benchmark_results.json.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

• CR-A15-002 (code, High) — cache-tasks mishandles lm-eval group tasks (mmlu, truthfulqa): nested {ConfigurableGroup: {...}} entries are not traversed and the non-str key crashes ', '.join with TypeError
  severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced with stubs
  Failure scenario. Following the guide, forgelm cache-tasks --tasks "hellaswag,arc_easy,truthfulqa,mmlu" --output-format json -q | tee cache-tasks.json; jq -e '.success' crashes with a traceback, gives jq empty input and exits 1. The documented CI recipe cannot succeed for MMLU.
  Where. forgelm/cli/subcommands/_cache.py:469-470 (top-level-only iteration), forgelm/cli/subcommands/_cache.py:526 (', '.join TypeError), forgelm/cli/subcommands/_cache.py:526, forgelm/cli/subcommands/_cache.py:89 (latent)
  Recommended fix. Flatten nested group dicts into leaf tasks before preparing them, report each name as str(key), compare the requested names with the returned rows, and add a test whose get_task_dict returns the real group shape.

• CR-A06-030 (code, Medium) — Docs claim ForgeLM ships Turkish benchmark templates mmlu_tr and belebele_tr; nothing ships
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A Turkish project configures tasks: [mmlu_tr, belebele_tr] trusting the manual. If lm-eval has no such task names, the gate fails after a full training run and, with auto_revert, the model is deleted.
  Where. docs/usermanuals/en/evaluation/benchmarks.md:49, docs/usermanuals/tr/evaluation/benchmarks.md:49, forgelm/benchmark.py:188
  Recommended fix. Remove the sentence, or point to the real upstream lm-eval task names.

• CR-A15-009 (code, Medium) — cache-tasks maps every get_task_dict failure, including dataset download, network, auth and disk-full errors, to exit 1 'Unknown or invalid task name'
  severity Medium · reachability documented-opt-in · intent contract-violation · confidence confirmed · independently confirmed · not reproduced (code trace)
  Failure scenario. A transient HF 5xx while staging hellaswag gives exit 1 and 'Unknown or invalid task name in [...]'. CI treats it as an operator typo and does not retry.
  Where. forgelm/cli/subcommands/_cache.py:468-484, forgelm/cli/subcommands/_cache.py:25-31, docs/reference/cache_subcommands.md:62-66, docs/usermanuals/en/reference/json-output.md:637-640
  Recommended fix. Narrow the config-error mapping to lookup errors and send other exceptions to EXIT_TRAINING_ERROR with a message that tells transport failures apart.

• CR-A01-079 (code, Low) — Promise F-067 'per-task floor thresholds' has no implementation or test; only a mean min_score exists
  severity Low · reachability default · intent intended-and-acceptable · confidence likely · independently confirmed · not reproduced (code trace)
  Failure scenario. An operator who reads 'per-task floors' in the promise list assumes each task is individually gated.
  Where. forgelm/config.py:881, forgelm/benchmark.py:223, docs/usermanuals/en/evaluation/benchmarks.md:123, site/features.html:214
  Recommended fix. Align the promise text with the shipped mean-only gate, or implement per-task floors with tests.

• CR-A06-050 (code, Low) — Benchmark never passes confirm_run_unsafe_code, so humaneval, which the docs list as supported, always fails the gate on current lm-eval
  severity Low · reachability default · intent contract-violation · confidence likely · independently confirmed · not reproduced (code trace)
  Where. forgelm/benchmark.py:188, docs/usermanuals/en/evaluation/benchmarks.md:47, docs/usermanuals/en/deployment/model-merging.md:88
  Recommended fix. Either remove humaneval (and other code-execution tasks) from the docs, or add an explicit opt-in config field that is forwarded as confirm_run_unsafe_code and documented as running untrusted generated code.

• CR-A06-074 (code, Low) — 'truthfulqa' expands to three equally weighted entries; truthfulqa_gen is scored by bleu_acc via the substring fallback, and no 'truthfulqa' score key exists
  severity Low · reachability default · intent intended-but-harmful · confidence likely · single reviewer · not reproduced (code trace)
  Where. forgelm/benchmark.py:40, forgelm/benchmark.py:57, forgelm/benchmark.py:193, forgelm/benchmark.py:195
  Recommended fix. Document that 'truthfulqa' means three tasks and name explicit tasks (truthfulqa_mc2 is the usual headline metric). Optionally make _extract_task_score task-aware, or reject the fallback substring match for tasks with no acc/acc_norm key, so the gen proxy is not averaged as accuracy.

• CR-A06-078 (code, Info) — Benchmark run crashes on non-zero local ranks with lm-eval >=0.4.12, which returns None from simple_evaluate
  severity Info · reachability undocumented · intent intended-but-harmful · confidence likely · single reviewer · reproduced with stubs
  Where. forgelm/benchmark.py:188, forgelm/benchmark.py:193
  Recommended fix. Run the benchmark on the main process only, or treat a None return as 'not the reporting rank' explicitly, and keep the results access inside the guarded block.

• TECH-T05-004 (technology, Next release) — Evaluation and safety models: Example benchmark tasks arc_easy / hellaswag / mmlu / truthfulqa and the mean-accuracy gate (aging)
  severity Next release
  Item. Example benchmark tasks arc_easy / hellaswag / mmlu / truthfulqa and the mean-accuracy gate · verdict: aging · urgency: Next release
  State in ForgeLM. BenchmarkConfig.tasks defaults to [] (forgelm/config.py:872); the description, config_template.yaml:186-189 and docs/usermanuals/en/evaluation/benchmarks.md:18, :58, :78, :94 use hellaswag/arc_easy/truthfulqa/mmlu, and the cache-tasks help uses hellaswag,arc_easy,truthfulqa (_cache.py:401). benchmark.py:40-61 picks acc_norm/acc and otherwise any key containing 'acc' that is not a stderr. benchmar…
  State of the ecosystem on 2026-10-01. HF's Open LLM Leaderboard v2 blog (published Jun 26, 2024 according to the cached copy) says 'models are now reaching baseline human performance on HellaSwag, MMLU, and ARC, a phenomenon called saturation'. It flags GSM8K and TruthfulQA contamination and introduces IFEval, BBH, MATH lvl 5, GPQA, MuSR and MMLU-Pro. HF retired the leaderboard on 2025-03-13 because it was 'slowly becoming obsolete'.…
  Recommendation. Keep the four as an explicitly labelled regression smoke set. Fix the docs: use truthfulqa_mc2 rather than the tag, show real result keys, and remove or replace the mmlu_tr/belebele_tr claim with turkishmmlu/belebele_tur_Latn. Average group aggregates only, or only the requested task names. Offer an optional harder preset (leaderboard_ifeval/bbh/gpqa/mmlu_pro/musr/math_hard) once the metric extra…
  Sources.

  • Performances are plateauing, let's make the leaderboard steep again (Open LLM Leaderboard v2 blog) (2024-06-26 (published, per cached page metadata))
  • It's been a wild ride, folks :) (end of the Open LLM Leaderboard) (2025-03-13)

• TECH-T05-010 (technology, Next release) — Evaluation and safety models: lm-eval API usage: simple_evaluate(model=HFLM(pretrained=, ...)) in benchmark.py and lm_eval.tasks.get_task_dict in cache-tasks (deprecated-in-use)
  severity Next release
  Item. lm-eval API usage: simple_evaluate(model=HFLM(pretrained=, ...)) in benchmark.py and lm_eval.tasks.get_task_dict in cache-tasks · verdict: deprecated-in-use · urgency: Next release
  State in ForgeLM. forgelm/benchmark.py:172-188 imports HFLM from lm_eval.models.huggingface and calls lm_eval.simple_evaluate(model=lm_obj, tasks, limit, num_fewshot). forgelm/cli/subcommands/_cache.py:409 imports get_task_dict; :453 calls get_task_dict(task_names); :469-470 iterates over .items() and passes each value to _prepare_one_task (:555-590). _prepare_one_task treats the value as one Task and reads a .dat…
  State of the ecosystem on 2026-10-01. The latest lm-eval is 0.4.13 (PyPI upload 2026-08-31). simple_evaluate is still lazily exported with model: str|LM. HFLM still accepts a pre-built model as pretrained. get_task_dict is decorated @​deprecated('get_task_dict is deprecated. Use TaskManager.load() instead.') at lm_eval/tasks/init.py:136 in v0.4.13. The v0.4.12 release notes (2026-05-11) say TaskManager.load returns a flat {tasks,…
  Recommendation. Move cache-tasks to TaskManager().load(...) and walk the flat {tasks, groups} result, or recurse into nested group dicts. Add a test with a group task (mmlu) and a tag (truthfulqa). Keep the HFLM import path. Record the installed lm_eval version in the benchmark audit event, as the cross-checker suggested.
  Sources.

  • PyPI JSON: lm-eval (2026-08-31 (0.4.13 upload))
  • lm-evaluation-harness GitHub releases (v0.4.6-v0.4.13 notes) (2026-08-31 (latest))

• DOC-D3B08-020 (docs, High) — docs/usermanuals/en/evaluation/benchmarks.md: false claim — 'ForgeLM ships templates for mmlu_tr and belebele_tr adapted to Turkish-specific tasks.'
  severity High
  Where. docs/usermanuals/en/evaluation/benchmarks.md:49 (Turkish mirror: docs/usermanuals/tr/evaluation/benchmarks.md:49) · type: false claim
  The document says. 'ForgeLM ships templates for mmlu_tr and belebele_tr adapted to Turkish-specific tasks.'
  What the code does. Neither name appears anywhere in the snapshot outside the two benchmark manual pages: no code, template, task file or config (rg over the snapshot excluding docs/ returns nothing).
  Code references. forgelm/benchmark.py:141-188
  Proposed correction. Delete the sentence, or replace it with: "ForgeLM ships no Turkish task templates; any Turkish task registered in your installed lm-evaluation-harness can be listed in tasks."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B08-021 (docs, High) — docs/usermanuals/en/evaluation/benchmarks.md: false claim — min_score is the mean across all listed tasks; a task with no accuracy metric drags the average down
  severity High
  Where. docs/usermanuals/en/evaluation/benchmarks.md:53 (and forgelm/benchmark.py:72-74 docstring) · type: false claim
  The document says. min_score is the mean across all listed tasks; a task with no accuracy metric drags the average down
  What the code does. Tasks without an accuracy metric or absent from results are dropped; the mean covers only reporting tasks
  Code references. forgelm/benchmark.py:83-86,195
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B08-022 (docs, High) — docs/usermanuals/en/evaluation/benchmarks.md: broken example — baseline.yaml (model + evaluation.benchmark only) works with forgelm --config baseline.yaml --benchmark-only ....
  severity High
  Where. docs/usermanuals/en/evaluation/benchmarks.md:73-83 (Turkish mirror: docs/usermanuals/tr/evaluation/benchmarks.md:73) · type: broken example
  The document says. baseline.yaml (model + evaluation.benchmark only) works with forgelm --config baseline.yaml --benchmark-only ....
  What the code does. ForgeConfig requires lora, training and data blocks (config.py:1607-1612). Every config-driven invocation validates the full schema first (cli/_dispatch.py:242). Reproduced: the command exits 1 with 3 'Field required' errors.
  Code references. forgelm/config.py:1604-1612, forgelm/cli/_dispatch.py:242, forgelm/cli/_no_train_modes.py:60-68
  Proposed correction. Add the required blocks (e.g. lora: {r: 8}, training: {output_dir: ./baseline-run}, data: {dataset_name_or_path: ./data/train.jsonl}), or say: "add this evaluation.benchmark block to an existing training config".
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B08-051 (docs, Medium) — docs/usermanuals/en/evaluation/benchmarks.md: misleading — 'Anything in lm-evaluation-harness works', listing gsm8k and humaneval, and the gate compares the mean across all listed tasks (lines 25,…
  severity Medium
  Where. docs/usermanuals/en/evaluation/benchmarks.md:37-47 (Turkish mirror: docs/usermanuals/tr/evaluation/benchmarks.md:35) · type: misleading
  The document says. 'Anything in lm-evaluation-harness works', listing gsm8k and humaneval, and the gate compares the mean across all listed tasks (lines 25, 123).
  What the code does. Only accuracy-style metrics are extracted: acc_norm/acc, or any key containing 'acc' (benchmark.py:40-63). A task without one is dropped from scores and from the average with only a warning (benchmark.py:83-86,195). The average is over tasks that produced an acc-type metric, not over the listed tasks.
  Code references. forgelm/benchmark.py:40-63, forgelm/benchmark.py:83-97, forgelm/benchmark.py:195
  Proposed correction. "Only tasks reporting an accuracy metric (acc_norm / acc) are scored; tasks reporting other metrics (e.g. exact_match, pass@k) are skipped with a warning and do not count toward the average." Remove gsm8k/humaneval from the table or annotate them.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B08-080 (docs, Low) — docs/usermanuals/en/evaluation/benchmarks.md: broken example — --benchmark-only prints {"hellaswag": 0.61, "arc_easy": 0.75, ...}.
  severity Low
  Where. docs/usermanuals/en/evaluation/benchmarks.md:84 (Turkish mirror: docs/usermanuals/tr/evaluation/benchmarks.md:84) · type: broken example
  The document says. --benchmark-only prints {"hellaswag": 0.61, "arc_easy": 0.75, ...}.
  What the code does. Text mode logs 'Benchmark Results:' then ' task: 0.6100' lines and the Average. JSON mode prints {success, model_path, benchmark: {scores, average, passed}, failure_reason?} (_no_train_modes.py:101-122).
  Code references. forgelm/cli/_no_train_modes.py:101-125
  Proposed correction. Show the real text log lines, or the JSON envelope from --output-format json.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}