Docs say a follow-up forgelm audit verifies redaction and catches what masking missed, but the audit runs the same regexes with no normalisation and flags none of the 75 leaked… (CR-A11-019) #393

Wave 1 · theme D1-4: Data audit, PII/secret detection and ingestion · highest severity High · 1 finding(s) · effort small

Summary

An operator follows the documented OCR-then-mask-then-audit recipe. The audit reports zero PII, and the operator takes that as proof that redaction worked, so a corpus containing IBANs, cards, emails and keys is published or trained on.

Primary finding

CR-A11-019 — Docs say a follow-up forgelm audit verifies redaction and catches what masking missed, but the audit runs the same regexes with no normalisation and flags none of the 75 leaked rows

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. Three claims: - docs/guides/ingestion.md:107-109: 'Audit your output afterwards with forgelm audit to verify.' - docs/guides/ingestion.md:670-671: 'The audit step verifies redaction worked: any remaining PII flag in data_audit_report.json is a row that escaped masking.' - docs/usermanuals/en/data/pii-masking.md:59: 'The audit step catches what masking missed'. The audit aggregator calls detect_pii and detect_secrets (forgelm/data_audit/_aggregator.py:147,159), which use the same _PII_PATTERNS and _SECRET_PATTERNS dicts as mask_pii and mask_secrets, with no normalisation. By construction it cannot see any span the masker missed. Only the reverse direction of the guide's sentence holds: a flag does mean an escaped row, but no flag does not mean clean. Also, docs/usermanuals/en/data/all-mask.md:43 justifies secrets-first ordering by saying JWTs 'overlap with email-like substrings'. JWT base64url (_secrets.py:83-88) cannot contain '@', which the email pattern requires (_pii_regex.py:39). The code's own docstring (forgelm/ingestion.py:2110-2113) says the current regex sets show no measurable cross-detector overlap. This extends the already-reported locale-table item with new evidence: the FR row's 'E.164 + French formats' does not hold either. 'Call +33 1 23 45 67 89' is not detected at all.

Where.

• docs/guides/ingestion.md:107-109
• docs/guides/ingestion.md:670-671
• docs/usermanuals/en/data/pii-masking.md:59
• docs/usermanuals/en/data/all-mask.md:43
• forgelm/data_audit/_aggregator.py:159
• docs/guides/ingestion-tr.md:686
• docs/usermanuals/tr/data/pii-masking.md:59
• forgelm/data_audit/_pii_regex.py:37
• forgelm/data_audit/_pii_regex.py:237-305
• forgelm/data_audit/_pii_regex.py:298

Evidence.

• e2e: fpy --venv extras -m forgelm ingest e2e/in --all-mask --output-format json -> EXIT 0, success true, pii {credit_card 3, email 4, iban 4, phone 6}, secrets {aws 2, jwt 1}; 75 of 95 output rows still contain a full value. audit_dataset(e2e/out.jsonl) -> pii_summary {}, secrets_summary {}, pii_severity total 0, worst_tier None (e2e_audit_report.json). mask_pii('Call +33 1 23 45 67 89 now.') -> unchanged, {}.

Recommended fix. Reword: the audit re-runs the same detectors, so it confirms only that no detectable span remains, not that redaction is complete. List the known blind spots (line wraps, invisible characters, NBSP, fullwidth or ligature glyphs, IBAN followed by uppercase text). Drop or correct the JWT/email overlap rationale in all-mask.md. Mirror the changes in the -tr pages.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}