The judge verdict (and, in the compliance bundle, the benchmark verdict and every gate threshold) never reaches compliance_report.json, annex_iv_metadata.json, training_manifest.y… (CR-A01-022) #76

Wave 1 · theme D1-1: Correctness of the evaluation gates · highest severity High · 1 finding(s) · effort small

Summary

The shipped default auto_revert: false and an LLM judge with min_score 9 average 8.0. The run exits 0. The envelope's judge block is {"average_score": 8.0} with no passed key and no error. The compliance bundle the auditor and the Article 14 approver are told to review ('Review results in

Primary finding

CR-A01-022 — The judge verdict (and, in the compliance bundle, the benchmark verdict and every gate threshold) never reaches compliance_report.json, annex_iv_metadata.json, training_manifest.yaml or the training…

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced with stubs

What happens. TrainResult has no judge_passed field (forgelm/results.py:26-28), and _apply_judge_result stores only average_score and details (forgelm/trainer.py:1478-1480). Every artefact built from TrainResult therefore drops the judge verdict: - the compliance export: judge_dict = {'average_score': ...} at forgelm/trainer.py:2043-2045, copied into evaluation_results.llm_judge at forgelm/compliance.py:1622-1623; - the training JSON envelope: output['judge'] = {'average_score': ...} at forgelm/cli/_result.py:52-53. The benchmark verdict is on TrainResult (benchmark_passed) and in the envelope. The compliance export still drops it: benchmark_dict = {'scores', 'average'} at forgelm/trainer.py:2046-2048. No threshold (min_score, max_safety_regression) is recorded in any compliance artefact. The safety block carries 'passed' but not evaluation_completed or failure_reason (forgelm/trainer.py:2034-2042). For the unevaluated case, safety/safe_ratio=0.0 is presented as a measured performance metric in Annex IV (forgelm/compliance.py:1859), in training_manifest.yaml (forgelm/compliance.py:2186-2190) and in deployer_instructions.md (forgelm/compliance.py:1736-1738). On the kept path success stays True, so the envelope's 'error' key, which carries the gate reason, is never emitted (forgelm/cli/_result.py:69-73). The Article 14 approver sees no verdict either. human_approval.required carries reason='require_human_approval=true' and metrics only (forgelm/trainer.py:1544-1552). The docs' approvals example (docs/usermanuals/en/reference/json-output.md:209-211) shows the reason 'post-train safety eval below threshold'. The docs tell operators to rely on these blocks: - docs/usermanuals/en/reference/exit-codes.md:116: "the benchmark / safety / judge block in the JSON output carries `*_passed:…

Where.

• forgelm/results.py:27-29
• forgelm/trainer.py:1478-1480
• forgelm/trainer.py:2032-2048
• forgelm/compliance.py:1620-1625
• forgelm/compliance.py:1859
• forgelm/compliance.py:2186-2190
• forgelm/compliance.py:1736-1738
• forgelm/cli/_result.py:69-73
• forgelm/trainer.py:1544-1552
• forgelm/compliance.py:1491-1627

Evidence.

• Two runs with identical scripted scores: judge 8.0, benchmark arc_easy 0.6 / hellaswag 0.4 (average 0.5), safety all safe. - c0: judge min_score 5.0, benchmark min_score 0.3; all gates pass. - c3: judge min_score 9.0, benchmark min_score 0.7, auto_revert false; the audit log has benchmark.evaluation_completed passed=false and judge.evaluation_completed passed=false. In both runs: - compliance_report.json evaluation_results excluding metrics is byte-identical: {"safety": {"passed": true, ...}, "…

Recommended fix. Add judge_passed to TrainResult and set it in _apply_judge_result. Emit 'passed' plus the threshold (min_score / max_safety_regression) in every gate block: envelope, compliance_report evaluation_results, and an Annex IV §7 'test results' sub-block. Add evaluation_completed and failure_reason to the safety block. Put the kept-gate reason in the envelope even when success is True (for example a 'gate_failures' list). Include gate verdicts in the human_approval.required event and the approvals listing. Do not present safe_ratio=0.0 from an unevaluated run as a performance metric.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}