[Phase 16] S10 — Data-layer fidelity, scale and privacy #896

Roadmap · Phase 16, step S10 · size L · 7 unit(s) · planned in docs/roadmap/phase-16-trust-surface-hardening.md:583-611

Summary

Two threads that share one report schema. The data layer reports a clean or complete result it did not compute — dropped Presidio rows, a no-op script-sanity check, silent decode replacement, a duplicate scan that cannot finish — and a declared data intent is silently not honoured (multimodal.text_column is dead; mix_ratio does not produce the requested blend). CORE-13 is the same layer's privacy contract: a synthetic empty-response WARNING logs raw prompt content.

Units

Plan

• find_near_duplicates and find_near_duplicates_minhash are in data_audit.__all__, so the pair-list shape is public API and the fix must be additive.
• Group on the exact fingerprint before LSH, so k identical rows contribute k(k-1)/2 arithmetically instead of materialising k(k-1)/2 tuples.

Exit criteria

• 5,000 identical records still yield exactly 12,497,500 pairs with subquadratic peak memory under tracemalloc, and a property test holds count_… == len(find_…) across random inputs, including the brute-force fallback.
• An every-row Presidio failure yields coverage=0, incomplete=true.
• tr / TR / Tr normalise byte-identically.
• A single invalid byte in a long file reports decode_replacements=1.
• A text_column absent from the dataset fails early.
• Realised mix ratios and row counts appear in the result artefact.
• A caplog test proves known PII tokens appear in no log record, while the warning still identifies the prompt by index and length.

How to deliver

• One root-cause family, committed on its own; then an Opus review round and a Sonnet review round, each with verified findings fixed and committed before the next step begins.
• A reproducer that is red before the fix and green after; the full gauntlet from CLAUDE.md passes under ./.venv/bin/python.
• Every English documentation edit ships with its Turkish mirror in the same commit; a new or renamed audit event gets its EN and TR catalog rows in the same PR.
• A step that grows a deferred module carries its budget_history justification in the same diff; a widened guard rolls out non-strict first, then flips.
• Commit bodies cite Absorbs F-W20260729-…; the CHANGELOG [Unreleased] section gets an entry only for a change a user or library consumer can observe (decision C-20).

_{Planned in docs/roadmap/phase-16-trust-surface-hardening.md (read at f94595f). Unit IDs (F-W20260729-…) come from the 2026-07-29/30 full-project review; the issue numbers next to them are the October 2026 review's records of the same defects.}