Wizard emits evaluation.safety (default on) without test_prompts and llm_judge without eval_dataset; the cwd-relative schema defaults are not shipped, so the gates cannot run (CR-A18-002) #129

Wave 1 · theme D1-1: Correctness of the evaluation gates · highest severity High · 2 finding(s) · effort small

Summary

A user follows the web wizard with safety evaluation on, trains, and runs from a directory without safety_prompts.jsonl. The safety gate silently cannot evaluate and the run finishes with the gate unevaluated. The user believes a safety gate protected the model. For high-risk tier selections, where the wizard forces safety on, the compliance evidence is a failed or incomplete safety record.

Primary finding

CR-A18-002 — Wizard emits evaluation.safety (default on) without test_prompts and llm_judge without eval_dataset; the cwd-relative schema defaults are not shipped, so the gates cannot run

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced (plumbing only)

What happens. buildYaml (wizard.js:711-729, 752-760) omits evaluation.safety.test_prompts and evaluation.llm_judge.eval_dataset, so the schema defaults 'safety_prompts.jsonl' (config.py:937-939) and 'eval_prompts.jsonl' (config.py:1102) apply. run_safety_evaluation returns passed=False, evaluation_completed=False on a missing file (safety/_orchestrator.py:195-207) and run_judge_evaluation fails 'Eval dataset not found' (judge.py:600-602), which with the wizard's default auto_revert deletes the model. The CLI wizard resolves the bundled default_probes.jsonl instead (wizard/_collectors.py:55-88).

Where.

• site/js/wizard.js:716-722
• site/js/wizard.js:752-760 (confirmed)
• site/js/wizard.js:205-206
• forgelm/config.py:937-939
• forgelm/config.py:1102
• forgelm/safety/_orchestrator.py:195-207
• forgelm/judge.py:600-602
• forgelm/trainer.py:1449-1460
• forgelm/trainer.py:1486-1493
• forgelm/wizard/_collectors.py:60-87

Evidence.

• fpy h/judge_missing.py: run_judge_evaluation(eval_dataset_path='eval_prompts.jsonl') -> passed=False reason='Eval dataset not found: eval_prompts.jsonl'; the defaults YAML (h/yamls/defaults.yaml) loads cleanly with safety.enabled true and no test_prompts.
• site/js/wizard.js:711-729 (block emitted, no test_prompts key); forgelm/config.py:937-939 (default 'safety_prompts.jsonl'); forgelm/safety/_orchestrator.py:195-206 (missing file returns passed=False, evaluation_completed=False); forgelm/trainer.py:1450-1460 (model kept, no revert when evaluation_completed is False); forgelm/wizard/_collectors.py:55-88 (CLI wizard resolves bundled probes). All 45 generated YAML variants pass load_config and 4 passed --dry-run, so the failure is only at run time.

Recommended fix. Emit test_prompts/eval_dataset explicitly with a visible input and a hint (or point safety at the bundled probes the way the CLI wizard does), and warn when judge is enabled with auto_revert. Separately (judge area): make a missing judge eval set an infrastructure failure that withholds revert, as safety already does.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

DOC-D4B09-007 — site/js/translations.js: broken example — The wizard's Llama Guard toggle: 'After training, ForgeLM samples adversarial test prompts and checks the fine-tuned model's responses'. It is on by default…

severity High

Where. site/js/translations.js:1700-1701 (wizard.eval.safety.hint/.detail); web wizard default state safetyEval=true (site/js/wizard.js:203-206), YAML emission site/js/wizard.js:71… · type: broken example

The document says. The wizard's Llama Guard toggle: 'After training, ForgeLM samples adversarial test prompts and checks the fine-tuned model's responses'. It is on by default and forced on for high-risk.

What the code does. The wizard emits evaluation.safety without test_prompts, so the schema default 'safety_prompts.jsonl' (forgelm/config.py:938) applies. ForgeLM ships no such file at that path. The bundled prompts (forgelm/safety_prompts/default_probes.jsonl) are used only by 'safety-eval --default-probes' and by the CLI wizard (forgelm/wizard/_collectors.py:64-87). With the file missing, the post-training gate returns evaluation_completed=False and passed=False (forgelm/safety/_orchestrator.py:195-207). The model is kept because revert is withheld without evidence, but the run reports a failed safety gate after all the GPU time is spent. --dry-run does not catch it.

Code references. site/js/wizard.js:717-736, forgelm/config.py:938, forgelm/safety/_orchestrator.py:195-207, forgelm/wizard/_collectors.py:64-87

User impact. An accept-all-defaults wizard YAML, and every high-risk one, cannot complete its safety gate unless the user creates safety_prompts.jsonl. The text implies prompts are built in.

Proposed correction. Write: 'After training, ForgeLM runs the prompts in evaluation.safety.test_prompts (default ./safety_prompts.jsonl, which you must create; the bundled set is available via forgelm safety-eval --default-probes) through the fine-tuned model and classifies the replies.'

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}