Judge turns a failed fine-tuned-model generation into an empty answer and scores it as valid evidence (CR-A06-007) #122

Wave 1 · theme D1-1: Correctness of the evaluation gates · highest severity High · 1 finding(s) · effort medium

Summary

The model under test raises on every generate() (CUDA device-side assert, or a bnb model moved to CPU). judge_results.json shows 5/5 valid scores. With judge score 8 the gate passes (exit 0). With judge score 1 and auto_revert, the model is deleted with 'Average judge score (1.00) below minimum (5.00)'.

Primary finding

CR-A06-007 — Judge turns a failed fine-tuned-model generation into an empty answer and scores it as valid evidence

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced with stubs · already tracked: PARTIALLY-FIXED CORE-04 — docs/roadmap/phase-16-trust-surface-hardening.md:356-380 (S3 checkbox open, delivered by cab2947); CHANGELOG.md:49-54,67-74

What happens. _generate_response (forgelm/judge.py:384-401) and the batched OOM/per-prompt fallback _generate_batch_with_oom_retry (judge.py:432-454) return '' when generation raises. _score_eval_prompts (judge.py:714-777) sends '' to the judge as a real answer and records the score as valid, with judge_failed false. The min_valid_fraction floor (judge.py:810-830) counts only judge parse failures, so a run in which every generation failed gets a full-strength verdict: a pass with a lenient judge, or a 'below minimum' failure that auto_revert acts on. The safety package's GeneratedResponse marker has no judge equivalent.

Where.

• forgelm/judge.py:404-454
• forgelm/judge.py:432-454 (confirmed)
• forgelm/judge.py:714-777 (confirmed)
• forgelm/judge.py:810-830
• forgelm/judge.py:810-819
• forgelm/judge.py:716-777
• forgelm/judge.py:394
• forgelm/judge.py:401
• forgelm/judge.py:441
• forgelm/judge.py:442

Evidence.

• runs/judge_mutfail_hi (mut-generation-failure + judge-local-valid): 6 generate failures logged, judge_results.json 5 details score 8.0 judge_failed false, passed true, exit 0. runs/judge_mutfail_lo (mut-generation-failure + judge score 1, auto_revert): exit 3, reverted, final_model deleted, model.reverted reason judge.
• r7: scenario mut-generation-failure + judge-api-valid. Log: 'Judge-generation batch failed ... retrying per-prompt' plus 5x 'Failed to generate response: CUDA error'. judge_results.json: average_score 8.0, passed true, 5 details each judge_failed false. EXIT=0. The judge reply is scripted (constant 8), so the pass outcome depends on that stub. The plumbing (failure invisible to the evidence floor) is real.
• judge.py:401 returns '' on failure. _score_eval_prompts (:716-777) has no way to tell it apart from an empty answer, and failure_count only counts judge-score failures.

Recommended fix. Carry the generation error like safety's GeneratedResponse, skip judging failed generations, count them as invalid evidence (score None, judge_failed/generation_failed true) and include them in the min_valid_fraction floor.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}