Site advertises per-task benchmark floors (and related gate overclaims) while only the task average is gated (CR-A18-001) #128

Wave 1 · theme D1-1: Correctness of the evaluation gates · highest severity High · 1 finding(s) · effort small

Summary

A compliance reader relies on the Art. 15 page to believe every benchmark task is floored. A task regression masked by the average ships.

Primary finding

CR-A18-001 — Site advertises per-task benchmark floors (and related gate overclaims) while only the task average is gated

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN DOCS-02/DOCS-14/DOCS-15/DOCS-16 (S15 commit 3 site claims; split not public) — docs/roadmap/phase-16-trust-surface-hardening.md:771-772,776-778

What happens. features.eval.bench.body (features.html:214, translations.js:970 + locales), home.cap.bench.body (translations.js:379) and compliance.art15.body (compliance.html:184, translations.js:3865 + locales) state 'per-task floor ... auto-revert if any drops'. BenchmarkConfig has a single min_score ('Minimum average accuracy', config.py:881-886); benchmark.py averages all tasks (195) and compares only average_score < min_score (223). A single collapsed task hidden by the mean passes the gate. C-1410 additionally bundles the hero-terminal audit claim, the dual-control claim and the static hero fallback.

Where.

• site/features.html:214 (confirmed)
• site/js/translations.js:970 (confirmed)
• site/compliance.html:184 (confirmed)
• site/js/translations.js:3880 (compliance.iso.change.body 'dual-controlled', EN)
• forgelm/config.py:881-886 (single min_score, 'Minimum average accuracy')
• forgelm/benchmark.py:223
• site/js/translations.js:379 (confirmed)
• forgelm/config.py:881
• forgelm/benchmark.py:195 (average), forgelm/benchmark.py:223 (only threshold comparison)
• site/index.html:259-269 (hero terminal shows a data audit step inside a forgelm --config run)

Evidence.

• rg 'Per-task floor' site/ (lines cited). BenchmarkConfig has no per-task field.
• benchmark.py:223 elif min_score is not None and average_score < min_score: is the only threshold comparison.
• Final EN values were read by loading translations.js in node (tr.js): features.eval.bench.body, compliance.art15.body and compliance.iso.change.body as quoted. benchmark.py:223 compares average_score < min_score. The trainer contains no audit_dataset call (rg over forgelm/ outside data_audit/).

Recommended fix. Correct the site copy in all locales, or implement per-task floors (e.g. min_task_scores dict).

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}