Site and docs present cross-split leakage as a gate ('refuses to certify a leaky split'), but forgelm audit exits 0 at 100% leakage (CR-A11-017) #391

Wave 1 · theme D1-4: Data audit, PII/secret detection and ingestion · highest severity High · 1 finding(s) · effort small

Summary

train.jsonl has 12 rows and validation.jsonl has 4 rows copied from train. forgelm audit exits 0 and the summary says '2 clean split(s)'. A CI job that relies on the documented refusal passes.

Primary finding

CR-A11-017 — Site and docs present cross-split leakage as a gate ('refuses to certify a leaky split'), but forgelm audit exits 0 at 100% leakage

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. forgelm audit gates only critical secrets and critical-tier PII (_audit.py:236-253, 317-321); leakage is reported in notes and the envelope with exit 0 and success:true, and the summary labels leaking splits 'clean' (_summary.py:102-118,435-447), while leakage.md:10, troubleshooting.md:100, data_preparation.md:10 and the site promise a leakage gate.

Where.

• docs/usermanuals/en/operations/troubleshooting.md:100-102
• docs/usermanuals/en/data/leakage.md:10
• docs/usermanuals/tr/data/leakage.md:10
• docs/usermanuals/tr/operations/troubleshooting.md:100-102
• forgelm/cli/subcommands/_audit.py:254
• forgelm/cli/subcommands/_audit.py:249-253
• forgelm/cli/subcommands/_audit.py:317-320
• site/js/translations.js:996,1101,1206,1285,1364,1443 (features.data.leak.body in all six locales)
• site/js/translations.js:3855,3914,4016,4059,4102 (compliance.art10.body 'leakage gate'; DE 3973 is accurate)
• docs/reference/data_preparation.md:10

Evidence.

• _audit.py:9-15 lists the exit codes, and none relates to leakage. Dynamic run: exit 0 with leak_rate_validation=1.0.
• aud/leaky -> EXIT=0, success True, leak_rate_validation 1.0.
• A validation split that is 100% copied from train gives EXIT=0, success true, and leak_rate_validation 1.0.

Recommended fix. Either add an opt-in leakage gate, such as --fail-on-leakage that exits 3, or rewrite troubleshooting.md:100-104 and leakage.md:10 (EN and TR) to match leakage.md:44 (reported, not gated; branch on the JSON with jq).

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}