Without langdetect every row is silently reported as language 'unknown'; the documented install hint is never emitted (CR-A11-027) #398

Wave 1 · theme D1-4: Data audit, PII/secret detection and ingestion · highest severity High · 7 finding(s) · effort small

Summary

A Turkish-corpus operator checks the language distribution, sees 'unknown=100%' with no hint, and either ignores the check or misreads it, so the 'supposed to be Turkish but 12% English' class of bug is never caught.

Primary finding

CR-A11-027 — Without langdetect every row is silently reported as language 'unknown'; the documented install hint is never emitted

severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. _detect_language (forgelm/data_audit/_streaming.py:25-41) catches ImportError and returns 'unknown' with no log or note, so languages_top3 becomes [{'code':'unknown','count':N}]; langdetect is only in the [ingestion] extra, and the 'unknown (install forgelm[ingestion])' marker promised at docs/guides/data_audit.md:615 exists nowhere in code.

Where.

• forgelm/data_audit/_streaming.py:25-41 (confirmed)
• forgelm/data_audit/_streaming.py:235-245
• docs/guides/data_audit.md:615 (confirmed)
• docs/usermanuals/en/data/language-detection.md:34
• forgelm/data_audit/_streaming.py:200-212
• forgelm/data_audit/_streaming.py:23-41
• pyproject.toml:140-147
• docs/guides/data_audit-tr.md:625
• README.md:128-130

Evidence.

• forgelm audit env/datasets/sft.jsonl on a bare install -> report languages_top3 [{'code': 'unknown', 'count': 16}]; text summary 'languages (top-3): unknown=16'; no warning in stderr.
• Reproduced in the default lane, where langdetect is not installed (importlib find_spec False; it is also absent from the extras, lower and latest venvs). fpy -m forgelm audit env/datasets/sft.jsonl --output out7 gives exit 0, splits.train.languages_top3 == [{'code': 'unknown', 'count': 16}], no note mentioning language, and nothing about language on stderr.
• extras venv (no langdetect): audit of env/datasets/sft gave languages_top3 [{'code': 'unknown', 'count': 12}] and no language-related note. langdetect 1.0.9 itself works on Python 3.13 (en and tr detected, DetectorFactory.seed=0 deterministic).

Recommended fix. Detect langdetect once per audit. When it is absent, omit languages_top3 or emit the documented hint string, and add a note or warning.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

• TECH-T06-006 (technology, Next release) — Data-audit methods: langdetect>=1.0.9,<2.0.0 (language ID for audit top-3 languages) (at-risk)
  severity Next release
  Item. langdetect>=1.0.9,<2.0.0 (language ID for audit top-3 languages) · verdict: at-risk · urgency: Next release
  State in ForgeLM. pyproject.toml:146 pins langdetect>=1.0.9,<2.0.0. forgelm/data_audit/_streaming.py:25-41 sets DetectorFactory.seed=0, returns 'unknown' when len(text) < 20, and returns 'unknown' on ImportError. docs/usermanuals/en/data/language-detection.md:32 and :34 (and the TR mirror) call it a CLD2 port and give a 50-character threshold.
  State of the ecosystem on 2026-10-01. The last PyPI release is 1.0.9 (2021-05-07). The GitHub repo is not archived, but its last push was 2025-03-03 and it has 72 open issues, including a request for a Python 3 wheel (2026-07-17, per the cross-check). The README says it ports Nakatani Shuyo's language-detection library (version from 03/03/2014) and supports 55 languages. Alternatives: DataTrove's LanguageFilter defaults to backend 'f…
  Recommendation. Fix language-detection.md:32 and :34 in EN and TR. Add a pluggable backend (GlotLID or fastText lid.176 from an offline model file, checking the licence; lingua only once the Python floor allows it) behind the existing ImportError fallback, and record the backend in the report.
  Sources.

  • PyPI JSON: langdetect (1.0.9, 2021-05-07) (2021-05-07)
  • GitHub API: Mimino666/langdetect (not archived, last push 2025-03-03, 72 open issues) (2026-10-01T13:50:14Z)

• TECH-T07-007 (technology, Watch) — Ingestion parsers: langdetect>=1.0.9,<2.0.0 (audit top-3 language stat) (aging)
  severity Watch
  Item. langdetect>=1.0.9,<2.0.0 (audit top-3 language stat) · verdict: aging · urgency: Watch
  State in ForgeLM. pyproject.toml:146. Used in forgelm/data_audit/_streaming.py:28-40 behind except ImportError, so the feature is optional. Not installed in any K-3 venv.
  State of the ecosystem on 2026-10-01. 1.0.9 (2021-05-07) is still the latest release. The only artefacts are a py2-none-any wheel and an sdist built from setup.py, with no pyproject.toml. It depends on six, its classifiers stop at 3.8, and it has no advisories. The repo is not archived; last push 2025-03-03, 72 open issues. Both researchers ran the 1.0.9 source on Python 3.13.14 and it detects en and tr. Python 3.14 is untested.
  Recommendation. Keep it optional. Document that it installs from sdist. Add 3.14 to CI for the extra before claiming 3.14, and pick a maintained replacement or a script-based fallback if the build path breaks.
  Sources.

  • PyPI JSON langdetect (1.0.9, py2 wheel + sdist) (2021-05-07)
  • GitHub API repo Mimino666/langdetect (pushed 2025-03-03) (2025-03-03)

• DOC-D3B07-010 (docs, High) — docs/usermanuals/en/data/language-detection.md: false claim — jq '.language_outliers[:3]' → per-row outliers; report has language_distribution, language_outliers, expected
  severity High
  Where. docs/usermanuals/en/data/language-detection.md:19-27,44-55 (Turkish mirror: docs/usermanuals/tr/data/language-detection.md:19-27,44-55) · type: false claim
  The document says. jq '.language_outliers[:3]' → per-row outliers; report has language_distribution, language_outliers, expected
  What the code does. No such keys exist. The only language output is splits..languages_top3, a list of up to 3 {code, count} objects from a 200-row sample. There are no row indices, no outlier detection, no 'mixed' class and no expected language.
  Code references. forgelm/data_audit/_streaming.py:235-245, forgelm/data_audit/_aggregator.py:287-289, forgelm/data_audit/_types.py:122-152
  Proposed correction. Rewrite the page: the audit reports splits.<name>.languages_top3 = [{"code": "tr", "count": 196}, …] from a 200-row uniform sample per split; there is no per-row output, outlier list, mixed class or expected-language setting.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B07-033 (docs, Medium) — docs/usermanuals/en/data/language-detection.md: misleading — Language detection is automatic and always-on in forgelm audit.
  severity Medium
  Where. docs/usermanuals/en/data/language-detection.md:38 (Turkish mirror: docs/usermanuals/tr/data/language-detection.md:38) · type: misleading
  The document says. Language detection is automatic and always-on in forgelm audit.
  What the code does. It needs langdetect, which ships in the [ingestion] extra. When langdetect is missing every sample is silently 'unknown' (verified in an env without langdetect: languages (top-3): unknown=1).
  Code references. forgelm/data_audit/_streaming.py:28-41, pyproject.toml:141-148
  Proposed correction. Language detection runs automatically when langdetect is installed (pip install 'forgelm[ingestion]'); without it every row reports unknown.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-BLEGAL-223 (docs, Low) — docs/usermanuals/en/data/language-detection.md: wrong detail — Uses langdetect (a pure-Python port of Google's CLD2). Supports 55+ languages out of the box.
  severity Low
  Where. docs/usermanuals/en/data/language-detection.md:32 (Turkish mirror: docs/usermanuals/tr/data/language-detection.md:32) · type: wrong detail
  The document says. Uses langdetect (a pure-Python port of Google's CLD2). Supports 55+ languages out of the box.
  What the code does. langdetect ports Nakatani Shuyo's Java language-detection library (version of 03/03/2014), not CLD2 (Chromium's Compact Language Detector 2, a separate library), and lists exactly 55 languages. Its last release was 1.0.9 (2021-05-07). The ~1 ms/row figure could not be verified.
  Proposed correction. 'Uses langdetect (a Python port of Nakatani Shuyo's Java language-detection library). Supports 55 languages.' TR: 'langdetect kullanır (Nakatani Shuyo'nun Java language-detection kütüphanesinin Python port'u). 55 dili destekler.'
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B07-070 (docs, Low) — docs/usermanuals/en/data/language-detection.md: wrong detail — For very short rows (<50 characters) ... ForgeLM marks those as unknown
  severity Low
  Where. docs/usermanuals/en/data/language-detection.md:34 (Turkish mirror: docs/usermanuals/tr/data/language-detection.md:34) · type: wrong detail
  The document says. For very short rows (<50 characters) ... ForgeLM marks those as unknown
  What the code does. The cutoff is 20 characters.
  Code references. forgelm/data_audit/_streaming.py:25-27
  Proposed correction. Rows shorter than 20 characters are marked unknown.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}