Judge user manual documents judge_report.json and fields the code never writes (and other judge doc drift) (CR-A06-009) #124

Wave 1 · theme D1-1: Correctness of the evaluation gates · highest severity High · 1 finding(s) · effort small

Summary

A CI step that parses the documented judge_report.json finds no file. The judge_results.json it does find for a failed run looks contradictory.

Primary finding

CR-A06-009 — Judge user manual documents judge_report.json and fields the code never writes (and other judge doc drift)

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. docs/usermanuals/en|tr/evaluation/judge.md documents <output_dir>/judge_report.json with n_prompts/mean_score/per_prompt. The code writes <checkpoint_dir>/judge/judge_results.json with average_score/num_prompts/details (judge.py:505-546; trainer.py:1985). The same pages also misdescribe batch_size and the default eval_dataset.

Where.

• docs/usermanuals/en/evaluation/judge.md:30 (batch_size comment)
• docs/usermanuals/tr/evaluation/judge.md:44-61
• forgelm/judge.py:530-534
• forgelm/trainer.py:1985 (output_dir = checkpoint_dir/judge)
• docs/reference/configuration.md:239
• docs/usermanuals/en/evaluation/judge.md:28
• forgelm/judge.py:714-738
• forgelm/config.py:1102-1106
• forgelm/trainer.py:396
• forgelm/judge.py:~700 is within _score_eval_prompts; the per-prompt judge call is at about lines 725-730

Evidence.

• runs/judge_mixed scripted-out__train_judge_api__judge__judge_results.json: {average_score 9.0, min_score 5.0, passed false, num_prompts 5, details[...]} with no reason.
• 10 prompts with batch_size 4 produced generation batches [4,4,2] and 10 separate judge calls; _save_judge_results writes judge_results.json.

Recommended fix. Rewrite the Output section from the real schema, and add failure_reason and valid_count to judge_results.json and the audit payload.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}