Forgelm audit lets ValueError escape as a raw traceback (exit 1, no JSON envelope, no report) for --jaccard-threshold >= 0.99 and the advertised --pii-ml-language tr (CR-A11-023) #365

Wave 1 · theme D1-11: Other high-severity packages · highest severity High · 5 finding(s) · effort small

Summary

forgelm audit data/train.jsonl --output-format json --dedup-method minhash --jaccard-threshold 1.0 -> stderr Traceback ... ValueError: The number of bands are too small (b < 2); stdout 0 bytes; exit 1; the output dir is never created. A CI step parsing stdout JSON breaks. The same happens for --pii-ml --pii-ml-language tr when presidio is installed.

Primary finding

CR-A11-023 — Forgelm audit lets ValueError escape as a raw traceback (exit 1, no JSON envelope, no report) for --jaccard-threshold >= 0.99 and the advertised --pii-ml-language tr

severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN C33-CLI-ERROR-ENVELOPE — docs/roadmap/phase-16-trust-surface-hardening.md:497-508 (S7 open)

What happens. _run_data_audit (forgelm/cli/subcommands/_audit.py:199-231) maps only OSError and ImportError. Two documented inputs raise ValueError: --jaccard-threshold is validated to [0,1] (_argparse_types.py:55-56, _orchestrator.py:299-306) but datasketch MinHashLSH(num_perm=128) rejects >= 0.99 at _minhash.py:66 after the full scan; 'tr', suggested by --help/_orchestrator.py:269, is absent from _SPACY_MODEL_FOR_LANGUAGE so _get_presidio_analyzer raises (_pii_ml.py:101-110) and _require_presidio re-wraps only OSError.

Where.

• forgelm/cli/subcommands/_audit.py:199-231 (confirmed)
• forgelm/data_audit/_minhash.py:66 (confirmed)
• forgelm/cli/_parser.py:566-572 (the --jaccard-threshold arg is at about 563-572)
• forgelm/cli/_parser.py:619-628
• forgelm/data_audit/_pii_ml.py:101-110 (confirmed)
• forgelm/data_audit/_pii_ml.py:151-163 (confirmed)
• forgelm/data_audit/_orchestrator.py:269
• forgelm/data_audit/_aggregator.py:154-156
• forgelm/data_audit/_orchestrator.py:299-306 (confirmed)
• forgelm/cli/subcommands/_audit.py:212-231

Evidence.

• CLI run in the snapshot lane: EXIT=1, o1.txt 0 bytes, ls out1 -> No such file; the stderr tail shows datasketch/lsh.py line 180 ValueError. A datasketch 1.10.0 probe: thresholds 0.9-0.98 ok, 0.99/0.995/1.0 ValueError. With a stub (_optional._HAS_PRESIDIO=True), _run_data_audit(..., 'json', enable_pii_ml=True, pii_ml_language='tr') -> UNCAUGHT ValueError 'No default spaCy model registered for Presidio language 'tr''.
• datasketch 1.10.0: the largest threshold MinHashLSH accepts at num_perm=128 is 0.98. forgelm audit env/datasets/sft --dedup-method minhash --jaccard-threshold 0.99 --output-format json -> ValueError traceback, exit 1, empty stdout, no out_mh directory.
• Reproduced. fpy --venv extras -m forgelm audit env/datasets/sft.jsonl --output out4 --pii-ml --pii-ml-language tr --output-format json gives exit 1, empty stdout, traceback ending 'ValueError: No default spaCy model registered for Presidio language 'tr'' at _pii_ml.py:103, no out4. fpy -m forgelm audit env/datasets/sft --dedup-method minhash --jaccard-threshold 0.99 --output-format json gives exit 1, stdout 0 bytes, 'ValueError: The number of bands are too small (b < 2)' from datasketch/lsh…

Recommended fix. Catch ValueError in _run_data_audit and route it to EXIT_CONFIG_ERROR with the JSON envelope. Pre-validate the LSH threshold (or clamp it, or document the real upper bound) before building MinHashLSH. Remove 'tr' from the help and docstrings, or map it to a real model.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

• TECH-T06-010 (technology, Next release) — Data-audit methods: datasketch>=1.6.0,<2.0.0 (extra [ingestion-scale]) (aging)
  severity Next release
  Item. datasketch>=1.6.0,<2.0.0 (extra [ingestion-scale]) · verdict: aging · urgency: Next release
  State in ForgeLM. pyproject.toml:156 pins datasketch>=1.6.0,<2.0.0; 1.10.0 is installed in the project venv. forgelm/data_audit/_types.py:110-114 says 128 is the datasketch default num_perm. docs/guides/data_audit.md:301-302 says only 'Pin num_perm if you need cross-run determinism'.
  State of the ecosystem on 2026-10-01. datasketch is actively maintained. 2.0.0 (2026-07-05) makes 'affine32' the default MinHash permutation scheme, which fixes an over-estimation bias (issue #​212). Its hash values differ from earlier versions, and scheme='legacy' restores interoperability. 1.10.0 (2026-04-17) had already changed the hash to unsigned 32-bit and requires Python >= 3.10. In 2.0.0 the num_perm default is still 128.
  Recommendation. Add a datasketch 2.x CI lane and move the cap to <3. Record the datasketch version and permutation scheme in near_duplicate_summary. Change the guidance to 'pin the datasketch version and num_perm'.
  Sources.

  • PyPI JSON: datasketch (2.0.0, 2026-07-05) (2026-07-05)
  • datasketch GitHub releases (v2.0.0 affine32; v1.10.0 unsigned 32-bit hash) (2026-07-05)

• DOC-D4B06-001 (docs, High) — notebooks/data_curation.ipynb: broken example — Pass --pii-ml-language tr (and download the matching spaCy model) to audit Turkish corpora.
  severity High
  Where. notebooks/data_curation.ipynb cell 19 (markdown, What's next / --pii-ml bullet) · type: broken example
  The document says. Pass --pii-ml-language tr (and download the matching spaCy model) to audit Turkish corpora.
  What the code does. 'tr' is not a registered Presidio language: forgelm/data_audit/_pii_ml.py:36-51 (_SPACY_MODEL_FOR_LANGUAGE) has no 'tr' entry (the comment at :47-50 names Turkish as lacking a dedicated model). _get_presidio_analyzer raises ValueError at :103 ('No default spaCy model registered for Presidio language tr ... use xx'). Reproduced by patching _HAS_PRESIDIO=True and calling _require_presidio('tr'). Th…
  Code references. forgelm/data_audit/_pii_ml.py:36-51, forgelm/data_audit/_pii_ml.py:103, forgelm/data_audit/_orchestrator.py:316, forgelm/cli/subcommands/_audit.py:198-232
  Proposed correction. Replace the Turkish sentence: '--pii-ml-language accepts de, en, es, fr, it, ja, ko, nl, pl, pt, ru, zh and xx (multilingual fallback, python -m spacy download xx_ent_wiki_sm). Turkish has no dedicated model today; use --pii-ml-language xx.'

• DOC-D2B01-003 (docs, Medium) — docs/guides/data_audit.md: wrong detail — _PII_PATTERNS iteration order governs both detection priority and mask precedence; when a span could match two categories the first/narrower one wins and t…
  severity Medium
  Where. docs/guides/data_audit.md:234-241 (Turkish mirror: docs/guides/data_audit-tr.md:132-139) · type: wrong detail
  The document says. _PII_PATTERNS iteration order governs both detection priority and mask precedence; when a span could match two categories the first/narrower one wins and the span is replaced before the next pattern sees it.
  What the code does. First-match-wins holds only in mask_pii. detect_pii, which feeds pii_summary, pii_severity and the gate, runs every pattern over the original text, so one span can be counted under two categories. Reproduced: detect_pii('Call me at +90 10000000146 tomorrow') returns {'tr_id': 1, 'phone': 1}, while mask_pii(..., return_counts=True) counts only {'tr_id': 1}.
  Code references. forgelm/data_audit/_pii_regex.py:254-261, forgelm/data_audit/_pii_regex.py:298-307
  Proposed correction. Rewrite as: "Mask precedence follows _PII_PATTERNS order (first/narrower pattern wins and the span is replaced before later patterns run). Detection counts (detect_pii, pii_summary) are computed per pattern over the original text, so a span matching two patterns (e.g. a TC Kimlik number written after a +CC prefix) is counted in both catego…
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B01-038 (docs, Low) — docs/guides/data_audit.md: misleading — Pin num_perm if you need cross-run determinism (MinHash section, in a CLI-centred discussion).
  severity Low
  Where. docs/guides/data_audit.md:297-306 (Turkish mirror: docs/guides/data_audit-tr.md) · type: misleading
  The document says. Pin num_perm if you need cross-run determinism (MinHash section, in a CLI-centred discussion).
  What the code does. num_perm is not a CLI flag. It is only the Python argument minhash_num_perm (default 128). The CLI always runs 128.
  Code references. forgelm/data_audit/_orchestrator.py:234, forgelm/cli/subcommands/_audit.py:157-171
  Proposed correction. Add: 'num_perm is fixed at 128 on the CLI; to change it, call audit_dataset(..., minhash_num_perm=N).'
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}