Envelope cross_split_leakage_pairs lists every compared split pair, not leaking pairs, so the documented jq leakage gate fails on every clean multi-split corpus (CR-A11-009) #479

Wave 2 · theme D2-5: Data audit, masking and ingestion: medium effort · highest severity High · 3 finding(s) · effort not estimated

Summary

The document states: jq gate '(.cross_split_leakage_pairs | length) == 0 and ...' — 'Verified against a real envelope: on a clean corpus this gate passes'. Observed in the code: cross_split_leakage_pairs lists every compared split pair, whatever the leak count. On a clean 2-split corpus it is ['train__validation'] with leaked_rows 0, so the gate evaluates false and jq exits 1. It passes only on single-file corpora.

Primary finding

CR-A11-009 — Envelope cross_split_leakage_pairs lists every compared split pair, not leaking pairs, so the documented jq leakage gate fails on every clean multi-split corpus

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed

What happens. _audit.py:288-299 fills cross_split_leakage_pairs with every key of the cross_split_overlap pairs dict built in _orchestrator.py:118-139, regardless of leaked row counts, while json-output.md and audit.md document it as overlap pairs that are empty when clean.

Where.

• forgelm/cli/subcommands/_audit.py:299
• forgelm/data_audit/_orchestrator.py:130-139
• forgelm/cli/subcommands/_audit.py:288
• forgelm/data_audit/_orchestrator.py:118-139 (confirmed)
• forgelm/data_audit/_orchestrator.py:51-81
• docs/usermanuals/en/reference/json-output.md:302,323,338
• docs/usermanuals/en/reference/cli.md:356
• docs/usermanuals/en/data/audit.md:228-235
• docs/usermanuals/tr/reference/json-output.md:302,323,341
• docs/usermanuals/tr/reference/cli.md:357

Evidence.

• forgelm/cli/subcommands/_audit.py:299; forgelm/data_audit/_orchestrator.py:118-137
• forgelm/cli/subcommands/_audit.py:288; forgelm/data_audit/_orchestrator.py:118-139

Recommended fix. Either emit only pairs with leaked_rows>0 in cross_split_leakage_pairs, or change the recipe to test leak counts from the on-disk report

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

DOC-D3B03-006 — docs/usermanuals/en/reference/json-output.md: broken example — cross_split_leakage_pairs lists train/eval overlap pairs and is an empty list when clean. The recommended CI gate is `(.cross_split_leak…

severity High

Where. docs/usermanuals/en/reference/json-output.md:323,334-341 (Turkish mirror: docs/usermanuals/tr/reference/json-output.md:323,341) · type: broken example

The document says. cross_split_leakage_pairs lists train/eval overlap pairs and is an empty list when clean. The recommended CI gate is (.cross_split_leakage_pairs | length) == 0.

What the code does. The envelope lists the key of every split pair that was compared, whether or not any row leaked. I audited a clean train+validation directory: the on-disk report showed leaked_rows 0 and the envelope still returned ['train__validation']. The documented jq gate therefore fails on every clean multi-split corpus and cannot measure leakage. Real leak counts are only in the on-disk report's cross_split_overlap.pairs[].leaked_rows_in_.

Code references. forgelm/cli/subcommands/_audit.py:288, forgelm/data_audit/_orchestrator.py:118-139

User impact. A CI gate copied from the page fails on every clean train/validation corpus, and operators are pushed to delete the leakage gate.

Proposed correction. Fix the code so the list contains only pairs with leaked_rows_in_* > 0 (or add the leak counts to the envelope). Until then, correct the row to: "names of split pairs that were compared (non-empty for any multi-split corpus); leak counts are only in the on-disk report", and change the jq example accordingly.

Turkish mirror. the Turkish page has the same error; fix both in the same PR

DOC-D3B07-020 — docs/usermanuals/en/data/audit.md: broken example — jq gate '(.cross_split_leakage_pairs | length) == 0 and ...' — 'Verified against a real envelope: on a clean corpus this gate passes'

severity Medium

Where. docs/usermanuals/en/data/audit.md:226-236 · type: broken example

The document says. jq gate '(.cross_split_leakage_pairs | length) == 0 and ...' — 'Verified against a real envelope: on a clean corpus this gate passes'

What the code does. cross_split_leakage_pairs lists every compared split pair, whatever the leak count. On a clean 2-split corpus it is ['train__validation'] with leaked_rows 0, so the gate evaluates false and jq exits 1. It passes only on single-file corpora.

Code references. forgelm/cli/subcommands/_audit.py:299, forgelm/data_audit/_orchestrator.py:118-137

User impact. A CI gate copied from the manual is permanently red on every multi-split corpus and gets disabled

Proposed correction. Either emit only pairs with leaked_rows>0 in cross_split_leakage_pairs, or change the recipe to test leak counts from the on-disk report

Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}