Judge and benchmark gates have no 'could not evaluate' outcome: infrastructure failures become passed=False, so auto_revert deletes the model and the run exits 3 (CR-A02-016) #118

Wave 1 · theme D1-1: Correctness of the evaluation gates · highest severity High · 1 finding(s) · effort medium

Summary

A rotated OpenAI key or a provider outage during the post-train judge pass irreversibly deletes a model that nothing measured as bad.

Primary finding

CR-A02-016 — Judge and benchmark gates have no 'could not evaluate' outcome: infrastructure failures become passed=False, so auto_revert deletes the model and the run exits 3

severity High · reachability documented-opt-in · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end · already tracked: PARTIALLY-FIXED CORE-04 — docs/roadmap/phase-16-trust-surface-hardening.md:376-380 (S3 checkbox open, delivered by cab2947); CHANGELOG.md:67-74

What happens. JudgeResult and BenchmarkResult carry no evaluation_completed discriminator. run_judge_evaluation returns passed=False for a missing eval dataset, judge load failure, HttpSafetyError, JudgeAuthError and all-None or insufficient scores (judge.py:600-669, 812-830); run_benchmark turns any HFLM/simple_evaluate exception into passed=False (benchmark.py:177-191). _apply_judge_result and _apply_benchmark_result (trainer.py:1402-1411, 1484-1494) send every passed=False to _revert_model under auto_revert, and --benchmark-only exits 3 (cli/_no_train_modes.py:124-125), unlike the safety gate's no-evidence carve-out.

Where.

• forgelm/trainer.py:1486-1494 (revert at 1492)
• forgelm/judge.py:650-663 (HttpSafetyError / JudgeAuthError)
• forgelm/judge.py:812-830 (returns at 815, 830)
• forgelm/trainer.py:1486-1493
• forgelm/trainer.py:1402-1412 (revert at 1409)
• forgelm/judge.py:600-663 (passed=False returns at 602, 608, 621, 633, 656, 663)
• forgelm/judge.py:628-633 (local judge load failure)
• forgelm/benchmark.py:177-191 (returns at 181, 191)
• forgelm/trainer.py:1458 (safety carve-out that judge/benchmark lack)
• forgelm/trainer.py:373-394

Evidence.

• J-A5 (HTTP 401 after one call), J-A8 (SSRF refusal, no request sent), J-L2 (local judge load failure), J-A7 (all non-finite), J-A4 (3/5 valid): each ends in exit 3 with model.reverted reason=judge and final_model deleted.
• runs/judge_missing used FIX train_judge_local.yaml with auto_revert: true and eval_dataset pointing at a non-existent file. Result: EXIT=3 and envelope reverted: true, error 'Eval dataset not found: ...'. The audit log has judge.evaluation_completed then model.reverted, and final_model/ was deleted, so no judge ever ran. Copies: outputs/.../runs/judge_missing_run.log, judge_missing_cfg.yaml.
• runs/judge_429_revert (judge-api-429): exit 3, envelope error 'No valid judge scores (all 5/5 parses/requests failed).', final_model absent, audit model.reverted reason judge. runs/judge_missing_ds (eval_dataset 'eval_prompts.jsonl', auto_revert): exit 3, 'Judge eval dataset not found', 'Deleting generated artifacts', final_model absent.

Recommended fix. Add an evaluation_completed-style flag to JudgeResult for no-evidence outcomes and keep the model (fail without reverting), as the safety gate does. Document the policy in auto-revert.md.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}