Manual claims forgelm audit flags chosen == rejected preference rows (with fabricated sample output); no such check exists (CR-MISC-006) #397

Wave 1 · theme D1-4: Data audit, PII/secret detection and ingestion · highest severity High · 1 finding(s) · effort small

Summary

A DPO run converges to ~0 loss because of degenerate pairs. The user follows the troubleshooting advice, runs forgelm audit, gets a clean result, and keeps the corrupted dataset.

Primary finding

CR-MISC-006 — Manual claims forgelm audit flags chosen == rejected preference rows (with fabricated sample output); no such check exists

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. Four EN pages and their TR mirrors promise a chosen == rejected check, including a sample output line '⚠ 12 chosen-rejected identical rows'. The audit has no format-specific checks, as data/audit.md:122 itself admits, so the docs contradict each other. The DPO troubleshooting advice (dpo.md:208) sends users to a check that cannot fire.

Where.

• docs/usermanuals/en/concepts/data-formats.md:69
• docs/usermanuals/en/concepts/data-formats.md:161-166
• docs/usermanuals/en/training/dpo.md:103
• docs/usermanuals/en/training/dpo.md:145
• docs/usermanuals/en/training/dpo.md:208
• docs/usermanuals/tr/concepts/data-formats.md:69
• docs/usermanuals/tr/training/dpo.md:94
• docs/usermanuals/tr/training/dpo.md:133
• docs/usermanuals/en/data/audit.md:122 (correct; the contradicting page)
• docs/usermanuals/tr/concepts/data-formats.md:162

Evidence.

• Audit of 16 preference rows, 8 with chosen == rejected: exit 0, and nothing in the summary or report flags them.

Recommended fix. Implement a cheap preference-pair check (exact and normalised chosen == rejected count in the report), or remove the claim and the fabricated sample output from all mirrors.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}