Benchmark gate: optional valid-task fraction floor, so one flaky lm-eval task does not fail a multi-task run — deferred until a real run reports a flaky-task false failure #914

Deferred work · F-W20260729-CORE-03-floor · severity Low · cost ~2-3 h · recorded in docs/roadmap/risks-and-decisions.md:258

Summary

Phase 16 S2 (decision C-1) made the benchmark gate fail when any lm-eval task returns a non-finite or out-of-range score, so under auto_revert: true one flaky task in a ten-task run deletes the model. A min_valid_fraction-style floor mirroring evaluation.llm_judge.min_valid_fraction was deferred so the strict form's real cost is known before a second threshold is added to a gate that used to pass NaN.

Condition or budget

Revisit if a real run reports a flaky-task false failure. Not met as of 2026-10-09.

State today (re-checked 2026-10-09)

BenchmarkConfig has no valid-fraction field; run_benchmark fails the whole gate on any invalid task score, and its comment records the floor as considered and deferred.

Related

#118 (no "could not evaluate" outcome for infrastructure failures) and #120 (would make missing or no-accuracy tasks fail closed too).

Acceptance criteria

• An opt-in floor excludes a minority of unusable task scores and reports them, while the default stays strict.
• The row is removed from the deferral table and the CHANGELOG names its ID (the table's removal contract).

_{Recorded in docs/roadmap/risks-and-decisions.md (read at f94595f) and re-checked against the code on 2026-10-09. The row leaves that table only when the fix is on main, the CHANGELOG names its ID and any promised guard exists.}