A configured eval-loss gate is silently skipped when no eval_loss exists (always for GRPO, and without a validation split) (CR-A02-008) #117

Wave 1 · theme D1-1: Correctness of the evaluation gates · highest severity High · 1 finding(s) · effort small

Summary

An operator configures auto_revert plus max_acceptable_loss on a GRPO stage and believes it is guarded. The gate never runs, the audit log has no decision record, and the model is promoted.

Primary finding

CR-A02-008 — A configured eval-loss gate is silently skipped when no eval_loss exists (always for GRPO, and without a validation split)

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced (plumbing only)

What happens. execute_evaluation_checks returns True with only a logger warning and no audit decision when there is no validation split (trainer.py:802-805) or no eval_loss in metrics (trainer.py:872-874). The post-train evaluate() is skipped for GRPO (trainer.py:1587-1588), so max_acceptable_loss/baseline_loss with auto_revert never gate GRPO; no config validator rejects the combination and tests assert the pass.

Where.

• forgelm/trainer.py:765 (max_prompt_length passed to GRPOConfig: separate defect that blocks GRPO today)
• forgelm/trainer.py:873 (eval_loss-missing skip; the return is at 874)
• forgelm/trainer.py:1587 (GRPO evaluate() skip; the comment is at 1583-1586)
• forgelm/trainer.py:413-421
• forgelm/trainer.py:1587-1588 (confirmed)
• forgelm/trainer.py:871-874
• forgelm/trainer.py:801-805
• forgelm/trainer.py:785-795
• forgelm/trainer.py:802
• forgelm/trainer.py:872

Evidence.

• runs-patched/grpo_gate (patched copy, auto_revert true, max_acceptable_loss 0.001) -> exit 0, success true, 0 loss_gate events, stderr WARNING 'eval_loss not found in metrics. Skipping evaluation checks.'. Control runs/missing_ar (SFT, scripted evalloss-missing) -> exit 2 KeyError metric_for_best_model, i.e. fail-closed for non-GRPO.
• Stubbed CLI run (grpo, auto_revert=true, max_acceptable_loss=0.1): WARNING 'eval_loss not found in metrics. Skipping evaluation checks.', EXIT 0. Audit: pipeline.initialized, training.started, pipeline.completed{success:true}, with no loss_gate event. Control (sft, same thresholds, eval_loss 0.5): EXIT 3 reverted.
• Stub (runD/g.py): ForgeTrainer.new with a real ForgeConfig(trainer_type='grpo', evaluation={auto_revert: True, max_acceptable_loss: 0.5}), a dataset with validation, metrics={'train_loss': 99.0}. execute_evaluation_checks -> True with 0 audit events. The same with trainer_type sft and no validation split -> True with 0 audit events. An end-to-end GRPO run is not possible on trl 1.7.1 (GRPOConfig TypeError, per O-W03).

Recommended fix. Reject (or warn at config load about) loss-gate thresholds with trainer_type grpo. When a configured gate cannot be evaluated, emit loss_gate_completed with passed=null and a reason instead of returning silently.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}