Language-detection user manual documents a per-row outlier report that does not exist (language_outliers, language_distribution, expected, 'mixed', <50-char threshold) (CR-A11-018) #392

Wave 1 · theme D1-4: Data audit, PII/secret detection and ingestion · highest severity High · 1 finding(s) · effort small

Summary

An operator auditing a Turkish medical corpus runs the page's jq command to list the rows that slipped in as English. The command prints null, and there is no row-level language data anywhere in the report. Only top-3 counts over a 200-row sample exist.

Primary finding

CR-A11-018 — Language-detection user manual documents a per-row outlier report that does not exist (language_outliers, language_distribution, expected, 'mixed', <50-char threshold)

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. docs/usermanuals/en/data/language-detection.md:19-28 tells users to run jq '.language_outliers[:3]' audit/data_audit_report.json and shows per-row objects {row, detected, expected, snippet}. :42-55 shows a report block with language_distribution fractions, language_outliers: 17 and expected: "tr". :8 and :19 say ForgeLM 'detects per-row language' and 'lists indices of outliers'. :14-16 shows a summary line ('✓ language: 99.2% tr ... 12 rows in en ... 5 rows in mixed'). :34 says rows under 50 characters become unknown. The TR mirror tr/data/language-detection.md:12-56 is identical in substance. The code does something narrower. _aggregator.py:287-289 adds only splits.<split>.languages_top3, a list of {code, count}, computed by _compute_top_languages (_streaming.py:245-255) over a 200-row uniform reservoir (_LANG_SAMPLE_SIZE, _streaming.py:226). _detect_language returns 'unknown' when len(text) < 20 (_streaming.py:25-27), not 50. langdetect never returns 'mixed', and no row indices, expected language or distribution fractions are produced. The text summary prints languages (top-3): en=N, ... (_summary.py:389-391). rg for language_outliers|language_distribution over forgelm/, tests/ and site/ finds no hits. A real report's top-level keys are croissant, cross_split_overlap, generated_at, near_duplicate_summary, notes, pii_severity, pii_summary, quality_summary, secrets_summary, source_input, source_path, splits, total_samples. The documented jq is not a gate (no -e), so the recipe just prints null. Operators who rely on the page get no per-row language triage and cannot find the 'accidental English rows' the page promises. This is not covered by the existing finding on languages_top3's shape, which is about a different page's per-split example.

Where.

• docs/usermanuals/en/data/language-detection.md:42-55
• docs/usermanuals/en/data/language-detection.md:12-28
• docs/usermanuals/en/data/language-detection.md:34
• docs/usermanuals/tr/data/language-detection.md:8-68
• forgelm/data_audit/_streaming.py:25-27
• forgelm/data_audit/_streaming.py:235-245
• forgelm/data_audit/_streaming.py:245-255
• forgelm/data_audit/_aggregator.py:287-289
• forgelm/data_audit/_summary.py:389-391
• docs/usermanuals/en/data/language-detection.md:64 (also documents nonexistent min_chars)

Evidence.

• jq keys on out/clean/data_audit_report.json shows no language_* key at top level; per-split keys are columns, languages_top3, near_duplicate_pairs, null_or_empty_count, null_or_empty_rate, quality_samples_evaluated, quality_samples_flagged, sample_count, simhash_distinct, text_length. rg 'language_outliers|language_distribution' in forgelm/, tests/ and site/ returned nothing. langdetect is not installed in any fpy lane (find_spec returned None in snapshot/extras/latest/lower), so the detector…

Recommended fix. Rewrite both mirrors to describe splits.<name>.languages_top3 (a list of {code, count} over a 200-row reservoir sample, with 'unknown' below 20 chars or without langdetect). Replace the jq example with jq '.splits | map_values(.languages_top3)'. Remove the outlier list, 'mixed', 'expected' and the 50-char claim, or file them as planned work.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}