Near-duplicate user manual overstates what threshold 3 catches and documents report fields that do not exist (the dedup removal recipe cannot run) (CR-A11-014) #388

Wave 1 · theme D1-4: Data audit, PII/secret detection and ingestion · highest severity High · 1 finding(s) · effort small

Summary

An operator copies the documented jq commands to drop duplicates. The first prints nothing and the second errors, so no duplicates are removed. An operator who relies on the threshold table assumes typo-level duplicates are caught at the default when roughly half are missed.

Primary finding

CR-A11-014 — Near-duplicate user manual overstates what threshold 3 catches and documents report fields that do not exist (the dedup removal recipe cannot run)

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. usermanuals/en/data/deduplication.md (TR mirror identical in structure) makes claims that measurement contradicts, even for English. (a) :54-59 says the default threshold 3 catches 'Send your CV to ali@example.com' vs '... or call us'. The measured simhash Hamming distance is 17 with xxhash and 9 with BLAKE2b, so it is not caught under either backend. (b) :39 says thresholds 1-2 catch 'trivial-edit duplicates'. For a single-character typo in a 200-char English row, only 40% (xxhash) or 54.5% (BLAKE2b) of edits fall within distance 3, with median 4 or 3; fewer fall within 2. The 'Hello!' vs 'Hello.' example works only because punctuation is not tokenised. (c) :14 claims simhash recall is 'Exact within Hamming threshold'. That is true of the LSH-vs-brute search (verified), but the table presents it as duplicate recall. (d) :25 jq '.near_duplicates[]' and :97 jq '[.near_duplicate_summary.pairs[].row_b]' reference fields the report never contains: near_duplicate_summary holds only method / pairs_per_split / hamming_threshold, and there is no top-level near_duplicates key. The documented duplicate-removal recipe therefore fails ('Cannot iterate over null'), and the audit emits no per-pair row indices at all.

Where.

• docs/usermanuals/en/data/deduplication.md:39
• docs/usermanuals/en/data/deduplication.md:25
• docs/usermanuals/en/data/deduplication.md:54-59
• docs/usermanuals/en/data/deduplication.md:91-102
• docs/usermanuals/tr/data/deduplication.md:25-27
• docs/usermanuals/tr/data/deduplication.md:97
• forgelm/data_audit/_summary.py:303-321
• forgelm/data_audit/_aggregator.py:248-262
• docs/usermanuals/tr/data/deduplication.md:54-59

Evidence.

• exp_dedup200.py ('doc ex2 hamming 17'); exp_backend.py (xxhash 17, blake2b 9); en_200 frac_le3 0.4 (xxhash) / 0.545 (blake2b). Actual report: near_duplicate_summary = {'method':'simhash','pairs_per_split':{'train':1,'test':0},'hamming_threshold':3}; 'near_duplicates' in report == False (e2e_en_simhash_report.json). find_near_duplicates equals _find_near_duplicates_brute for thresholds 0,1,3,7,15 on 400 random fingerprints.

Recommended fix. Replace the examples with measured ones and state that recall depends on row length and token count. Either emit per-split pair indices (bounded) in the report or remove the jq recipe. Keep the 'exact' wording to 'LSH search is exact relative to brute force'.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}