TR mirror drift in docs/usermanuals/en/data/deduplication.md: Both algorithms are streaming - they don't load the whole dataset into memory. A 10M-row corpus dedupes in a few minutes on a laptop CPU. (DOC-TR12-002) #348

Wave 1 · theme D1-10: Documentation-only fixes · highest severity High · 1 finding(s) · effort docs fix

Summary

Neither: rows are read as a stream but one 64-bit fingerprint (simhash) or one MinHash object (128 perms, ~1 KB) per row is retained for LSH/leakage, so memory grows linearly with rows.

Primary finding

DOC-TR12-002 — TR mirror drift in docs/usermanuals/en/data/deduplication.md: Both algorithms are streaming - they don't load the whole dataset into memory. A 10M-row corpus dedupes in a few minutes on a laptop CPU.

severity High · already tracked: KNOWN-OPEN TRUST-12|TRUST-06|TRUST-09|TRUST-11 docs/roadmap/phase-16-trust-surface-hardening.md:593-599

Where. EN docs/usermanuals/en/data/deduplication.md:85 · TR docs/usermanuals/tr/data/deduplication.md:85

English text. Both algorithms are streaming - they don't load the whole dataset into memory. A 10M-row corpus dedupes in a few minutes on a laptop CPU.

Turkish text. İki algoritma da streaming — tüm dataset'i belleğe yüklemez. 10M-satır corpus laptop CPU'sunda birkaç dakikada deduplike edilir.

Which is correct. Neither: rows are read as a stream but one 64-bit fingerprint (simhash) or one MinHash object (128 perms, ~1 KB) per row is retained for LSH/leakage, so memory grows linearly with rows.

Proposed correction (Turkish text for the -tr page). Satırlar akış olarak okunur, ancak near-dup ve sızıntı için her satırın imzası (simhash: 8 bayt; MinHash: ~1 KB) bellekte tutulur; bellek satır sayısıyla doğrusal artar. 10M satır için süre/bellek garantisi verilmez.

Acceptance criteria

• Each statement is re-checked against the code at fix time.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}