Training run does not audit the dataset (PII, cross-split overlap) despite the first-run transcript showing it (CR-A07-022) #387

Wave 1 · theme D1-4: Data audit, PII/secret detection and ingestion · highest severity High · 1 finding(s) · effort small

Summary

An operator follows the first-run guide and assumes PII and leakage checks ran as part of training. Training on a dataset with PII and train/validation leakage exits 0. The only signal is a WARNING and a compliance.governance_section_missing audit event.

Primary finding

CR-A07-022 — Training run does not audit the dataset (PII, cross-split overlap) despite the first-run transcript showing it

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. first-run.md:139-142 (EN and TR) shows forgelm --config auditing the dataset and printing PII flags and cross-split overlap. The training path never calls the data_audit package. It only inlines an existing data_audit_report.json from training.output_dir, and when that file is absent it logs a warning advising a separate forgelm audit run. A dataset with an email, a phone number, a card number and a train/validation duplicate trained to exit 0 with no PII or overlap output. The same transcript also shows steps the SFT-only customer-support template cannot produce: a DPO preference pass, benchmarks, Llama Guard, and an Annex IV path under artifacts/.

Where.

• docs/usermanuals/en/getting-started/first-run.md:139-149 (the transcript's audit line is 141-142, the DPO line is 143)
• docs/usermanuals/tr/getting-started/first-run.md:137-146
• forgelm/compliance.py:668-704
• forgelm/trainer.py:2095
• docs/usermanuals/en/getting-started/first-run.md:161-172
• forgelm/templates/customer-support/config.yaml:26
• docs/usermanuals/tr/getting-started/first-run.md:167-169
• forgelm/trainer.py:1505
• forgelm/cli/_training.py:148-149
• site/index.html:261-262

Evidence.

• run1/run.log: no PII or overlap lines; line 42 reads 'No data_audit_report.json at .../out/data_audit_report.json — governance report will lack the Article 10 data-quality section. Run forgelm audit <dataset> --output ... before training'. rg finds no audit_dataset or detect_pii call outside forgelm/data_audit/.

Recommended fix. Replace the transcript with real output of the template and add an explicit forgelm audit step before training, or add an opt-in config key that runs the audit in-run.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}