From cad587dcf1a26a9c4422930ad7dd9c28bd52e33b Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 29 Apr 2026 03:16:43 +0300 Subject: [PATCH 01/12] feat(phase-12.5): four data-curation polish items in one PR MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 12.5 (Data Curation Polish) ships the four follow-up items deferred from Phase 12 — none require new architecture; each is a small additive surface on top of the Phase 12 ingestion + audit lineage. Will release with v0.5.3 alongside Phase 14. Source of truth: docs/roadmap/phase-12-5-backlog.md (now updated to "landed on development" for all four rows). Item 3 — `forgelm ingest --all-mask` (XS, no new behaviour) - One-flag shorthand for `--secrets-mask --pii-mask` in the documented mask order. Resolved at the CLI boundary; ingest_path's Python API stays untouched. Composes additively with explicit flags (set-union, no error). Item 4 — Wizard "audit first" entry point (S) - New _offer_audit_for_jsonl(path) helper in forgelm/wizard.py, invoked from the three JSONL-finalisation paths: _offer_ingest_for_directory (after ingest produces a JSONL), _validate_local_jsonl (BYOD path), and _prompt_dataset_path_with_ingest_offer (full wizard direct-JSONL). Mirrors the Phase 11.5 _offer_ingest_for_directory shape exactly. Audit is informational, not a gate — failures fall through to the "continue without audit" path so a malformed JSONL or transient audit error never crashes the wizard. Item 2 — `forgelm audit --croissant` (S) - Opt-in Google Croissant 1.0 dataset card emitted under a new ``croissant`` field on AuditReport. Card is conformant with http://mlcommons.org/croissant/1.0 — `@type: sc:Dataset`, full `@context` block, one `cr:FileObject` per JSONL split, one `cr:RecordSet` per split with `cr:Field` entries derived from the audit's column-detection layer. - Existing audit JSON keys are byte-equivalent — block stays empty when the flag is off (precedent: secrets_summary / quality_summary). Item 1 — Presidio PII ML adapter (S–M) - New `[ingestion-pii-ml]` extra (`presidio-analyzer>=2.2.0,<3.0.0`). - New `forgelm audit --pii-ml` flag layers Presidio NER on top of the regex detector. Adds the unstructured-identifier categories the regex inherently misses: `person`, `organization`, `location`. - New `PII_ML_SEVERITY` table (`person → medium`, `organization → low`, `location → low`) merged into _build_pii_severity's severity_table snapshot so the new categories grade through the existing pii_severity block. - Pre-flight `_require_presidio()` raises ImportError with an install hint when the extra is missing — failures surface before any rows are scanned. Per-row Presidio failures swallowed so a single malformed row never blocks the audit. Touch points - forgelm/cli.py — three new flags on existing subparsers; dispatcher signatures threaded. - forgelm/data_audit.py — Presidio import block + sentinel; new PII_ML_SEVERITY / PII_ML_TYPES / _PRESIDIO_ENTITY_MAP constants; _require_presidio / _get_presidio_analyzer / detect_pii_ml; new croissant field on AuditReport; _build_croissant_metadata helper + _CROISSANT_CONTEXT; enable_pii_ml / emit_croissant params on audit_dataset / _process_split / _audit_split; enable_pii_ml field on _StreamingAggregator; _build_pii_severity now consults the merged regex+ML table. - forgelm/wizard.py — _offer_audit_for_jsonl + three call-site injections. - pyproject.toml — `[ingestion-pii-ml]` extra. - tests/test_phase12_5.py — 11 new tests, four classes (one per item). - tests/test_wizard_byod.py — three existing tests get an extra "n" to decline the new audit-first offer (offer behaviour has its own coverage in test_phase12_5.py). - README, CHANGELOG, roadmap{,-tr}, phase-12-5-backlog.md, docs/standards/architecture.md, docs/guides/{ingestion,data_audit}{,-tr}.md, docs/reference/usage{,-tr}.md, notebooks/data_curation.ipynb. Tests: 833 pass (was 822, +11 new). Ruff clean. Bilingual H2 parity 7/7. CLI smoke + new-flag help text verified. Co-Authored-By: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 79 +++++++ README.md | 2 + docs/guides/data_audit-tr.md | 85 ++++++++ docs/guides/data_audit.md | 83 +++++++ docs/guides/ingestion-tr.md | 20 ++ docs/guides/ingestion.md | 20 ++ docs/reference/usage-tr.md | 10 + docs/reference/usage.md | 10 + docs/roadmap-tr.md | 4 +- docs/roadmap.md | 4 +- docs/roadmap/phase-12-5-backlog.md | 10 +- docs/standards/architecture.md | 1 + forgelm/cli.py | 47 +++- forgelm/data_audit.py | 340 ++++++++++++++++++++++++++++- forgelm/wizard.py | 81 ++++++- notebooks/data_curation.ipynb | 33 +-- pyproject.toml | 10 + tests/test_phase12_5.py | 292 +++++++++++++++++++++++++ tests/test_wizard_byod.py | 12 +- 19 files changed, 1089 insertions(+), 54 deletions(-) create mode 100644 tests/test_phase12_5.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 1fc4ef21..4b7f34e7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,85 @@ All notable changes to ForgeLM are documented here. ## [Unreleased] +### Added — Phase 12.5 (Data Curation Polish backlog) + +Four follow-up items from +[`docs/roadmap/phase-12-5-backlog.md`](docs/roadmap/phase-12-5-backlog.md) +ship together — none require new architecture; each is a small +additive surface on top of the Phase 12 ingestion + audit lineage. + +- **`forgelm ingest --all-mask`** (item #3) — one-flag shorthand for + `--secrets-mask --pii-mask` in the documented mask order (secrets + first so combined detectors don't double-count overlapping spans). + Composes additively with explicit flags (set-union, no error). Pure + UX; no new behaviour. +- **`forgelm audit --croissant`** (item #2) — opt-in + [Google Croissant 1.0](http://mlcommons.org/croissant/) dataset card + emitted under a new `croissant` key in `data_audit_report.json`. The + card carries dataset-level identity, one `cr:FileObject` per JSONL + split, and a `cr:RecordSet` per split with `cr:Field` entries + derived from the audit's column detection. Existing audit JSON keys + are byte-equivalent — the block stays empty when the flag is off + (same precedent as `secrets_summary` / `quality_summary`). Lets the + same JSON file double as both the EU AI Act Article 10 governance + artifact and a Croissant-consumer dataset card. +- **`forgelm audit --pii-ml`** + new `[ingestion-pii-ml]` extra + (item #1) — opt-in [Presidio](https://github.com/microsoft/presidio) + ML-NER PII detector layered on top of the existing regex detector. + Adds the unstructured-identifier categories the regex inherently + misses (`person`, `organization`, `location`) into the same + `pii_summary` / `pii_severity` blocks under disjoint category + names. Severity tiers in the new `PII_ML_SEVERITY` table: + `person → medium`, `organization → low`, `location → low` (deliberately + below the regex `critical`/`high` tiers because NER false-positive + rates are materially higher than regex-anchored detection). + Pre-flight check raises `ImportError("...pip install + 'forgelm[ingestion-pii-ml]'")` when the extra is missing — failures + surface before any rows are scanned. Per-row Presidio failures are + swallowed so a single malformed row never blocks the audit. +- **Wizard "audit first" entry point** (item #4) — when the wizard + resolves a JSONL (either typed directly or produced by the + Phase 11.5 `_offer_ingest_for_directory` ingest flow), it now offers + to run `forgelm audit` on it inline and prints `summarize_report`'s + verdict before continuing. Mirrors the + `_offer_ingest_for_directory` shape exactly. Closes the BYOD audit + loop end-to-end. Audit is informational, not a gate — failures fall + through to the "continue without audit" path. + +Touch points (so the next reviewer can audit blast radius quickly): + +- `forgelm/ingestion.py` — no module changes (the flag composes at the + CLI boundary into the existing `pii_mask` / `secrets_mask` booleans). +- `forgelm/cli.py` — three new flags on the existing subparsers + (`--all-mask` on `forgelm ingest`; `--croissant` and `--pii-ml` on + `forgelm audit`); dispatcher signatures threaded through. +- `forgelm/data_audit.py` — `_HAS_PRESIDIO` sentinel, `_require_presidio`, + `_get_presidio_analyzer` (cached), `detect_pii_ml`, + `PII_ML_SEVERITY`, `PII_ML_TYPES`, `_PRESIDIO_ENTITY_MAP`, + `_build_croissant_metadata`, `_CROISSANT_CONTEXT`. New + `enable_pii_ml` / `emit_croissant` parameters on `audit_dataset` / + `_process_split` / `_audit_split`; new `enable_pii_ml` field on + `_StreamingAggregator`; new `croissant` field on `AuditReport`. + `_build_pii_severity` now consults the merged + `PII_SEVERITY ∪ PII_ML_SEVERITY` table. +- `forgelm/wizard.py` — new `_offer_audit_for_jsonl(path)` helper; + invoked from `_offer_ingest_for_directory` (after ingest produces + a JSONL), `_validate_local_jsonl` (after a directly-provided JSONL + passes validation), and `_prompt_dataset_path_with_ingest_offer` + (after a non-directory JSONL is provided to the full wizard). +- `pyproject.toml` — new `[ingestion-pii-ml]` extra + (`presidio-analyzer>=2.2.0,<3.0.0`). +- `tests/test_phase12_5.py` — 11 new tests, four classes (one per + backlog row). +- `tests/test_wizard_byod.py` — three existing tests get an extra + `"n"` answer to decline the new audit-first offer (the offer + behaviour has its own coverage in `test_phase12_5.py`). +- Docs — `README.md` install matrix + Phase 12.5 feature line; + `docs/standards/architecture.md` extras matrix; `docs/guides/ingestion{,-tr}.md` + + `docs/guides/data_audit{,-tr}.md` get dedicated sections per + feature; `notebooks/data_curation.ipynb` mentions `--all-mask` and + the Phase 12.5 audit add-ons inline. + ### Fixed — post-PR-#13 review-cycle batches (rounds 8-12) Inline-comment batches landing on top of PR #13 (now merged to `main`). diff --git a/README.md b/README.md index f09e8d32..d90a461f 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,7 @@ - **PDF Page Header/Footer Dedup (v0.5.1)**: Lines that recur on ≥ 70 % of PDF pages (watermarks, page numbers, copyright lines) are stripped automatically — the audit's near-duplicate counts stop misfiring on long policy / book PDFs. - **Dataset Audit**: `forgelm audit data/sft.jsonl --output ./audit/` — produces `data_audit_report.json` with sample count, length distribution, top-3 language detection, LSH-banded near-duplicate rate (`O(n × k)` typical case, exact recall at the default Hamming threshold; v0.5.2 added optional **MinHash LSH** via `--dedup-method minhash` for >50K-row corpora), cross-split leakage check, null/empty rate, and PII flag counts with **severity tiers** (critical / high / medium / low + worst-tier verdict). v0.5.2 also added an always-on **secrets/credential scan** covering the nine families enumerated in `forgelm.data_audit.SECRET_TYPES` — AWS access keys, GitHub tokens, Slack tokens, OpenAI API keys, Google API keys, JWTs, full OpenSSH private-key blocks, full PGP private-key blocks, and Azure storage connection strings — and an opt-in **heuristic quality filter** (`--quality-filter`). CPU-only; streaming JSONL reader keeps memory bounded on multi-million-row splits; feeds EU AI Act Article 10 governance artifact automatically when present at training time. Legacy `--data-audit` flag still works as a deprecation alias. - **Secrets-Aware Ingest (v0.5.2)**: `forgelm ingest … --secrets-mask` scrubs credentials before chunks land in the JSONL — fine-tuning on text containing real API keys memorises them at training time. Pairs with `--pii-mask`; secrets run first so combined detectors don't double-count overlapping spans. +- **Phase 12.5 polish (v0.5.3)**: `forgelm ingest --all-mask` shorthand for combined PII + secrets scrubbing; `forgelm audit --croissant` emits a [Google Croissant 1.0](http://mlcommons.org/croissant/) dataset card alongside the existing report; `forgelm audit --pii-ml` layers [Presidio](https://github.com/microsoft/presidio) NER on top of the regex detector via the optional `[ingestion-pii-ml]` extra (`person` / `organization` / `location` categories merged into the same `pii_severity` block); the wizard now offers to run `forgelm audit` automatically after a JSONL is selected (BYOD onboarding closes the audit loop). ### Quickstart Layer (v0.4.5) - **One-Command Templates**: `forgelm quickstart customer-support` — 4 bundled templates (SFT customer-support, code-assistant, medical-qa-tr, GRPO math-reasoning) plus a bring-your-own-data domain-expert scaffold. Auto-downsizes models on small GPUs. @@ -172,6 +173,7 @@ pip install "forgelm[merging]" # mergekit model merging pip install "forgelm[ingestion]" # PDF/DOCX/EPUB/Markdown → JSONL + langdetect + xxhash pip install "forgelm[ingestion-scale]" # MinHash LSH dedup (datasketch) for >50K-row corpora pip install "forgelm[ingestion-secrets]" # detect-secrets scanner for SFT corpora (falls back to regex if absent) +pip install "forgelm[ingestion-pii-ml]" # Presidio ML-NER for person/organization/location PII (Phase 12.5) pip install "forgelm[export]" # GGUF export via llama-cpp-python pip install "forgelm[chat]" # Rich terminal rendering for `forgelm chat` ``` diff --git a/docs/guides/data_audit-tr.md b/docs/guides/data_audit-tr.md index 207d2068..980370b1 100644 --- a/docs/guides/data_audit-tr.md +++ b/docs/guides/data_audit-tr.md @@ -321,6 +321,91 @@ install ile uyumlu kalır. --- +### ML-NER PII adaptörü — `--pii-ml` (Faz 12.5, opt-in) + +Varsayılan regex detector, EU AI Act Madde 10'un önemsediği yapılandırılmış +identifier'ları (email, telefon, IBAN, kredi kartı, ulusal kimlik) kapsar. +Faz 12.5, regex'in doğal olarak kaçırdığı yapılandırılmamış identifier +kategorilerini ekleyen opt-in **Presidio** ([microsoft/presidio](https://github.com/microsoft/presidio)) +adaptörünü ML-NER olarak üstüne ekler: `person`, `organization`, +`location`. + +```bash +pip install 'forgelm[ingestion-pii-ml]' +forgelm audit data/ --output ./audit/ --pii-ml +``` + +Yeni kategoriler mevcut `pii_summary` ve `pii_severity` bloklarına +disjoint isimlerle birleşir, bu yüzden regex baseline ML sinyaliyle +yan yana görünür kalır: + +```json +{ + "pii_summary": { + "email": 12, + "phone": 3, + "person": 47, // ← Presidio + "organization": 18, // ← Presidio + "location": 9 // ← Presidio + }, + "pii_severity": { + "by_tier": {"critical": 0, "high": 0, "medium": 59, "low": 30}, + "by_type": { + "email": {"count": 12, "tier": "medium"}, + "person": {"count": 47, "tier": "medium"}, + "organization": {"count": 18, "tier": "low"} + }, + "worst_tier": "medium" + } +} +``` + +Şiddet ataması `forgelm.data_audit.PII_ML_SEVERITY`'de yaşar: +`person → medium`, `organization → low`, `location → low`. NER +false-positive oranları regex-anchored detection'dan materyal olarak +yüksektir — bu yüzden ML katmanları regex'in `critical` / `high` +katlarının altında bilinçli konumlandırılır. Bir ML bulgusunu sert +bir gate olarak değerlendirmeden önce satır bazında span'leri inceleyin. + +Presidio'nun ilk analyzer build'i bir spaCy İngilizce NER modeli +indirir (~ 50 MB, tek seferlik). Sonraki çalışmalar cached modeli +yeniden kullanır. Adaptör yalnızca opt-in; `--pii-ml` olmadan audit +zero-extra-deps regex yolunda kalır. + +--- + +### Croissant 1.0 dataset card — `--croissant` (Faz 12.5, opt-in) + +`--croissant`, `data_audit_report.json` içinde yeni bir top-level +`croissant` bloğunu [Google Croissant 1.0](http://mlcommons.org/croissant/) +dataset card'ı (`@type: sc:Dataset`) ile doldurur. Card, kanonik +`mlcommons.org/croissant/1.0` context'iyle uyumludur — bu yüzden +Croissant farkındalıklı consumer'lar (HuggingFace dataset cards, +MLCommons referans loader'ları, Croissant validator) bloğu hiçbir +modifikasyon olmadan parse edebilir. + +```bash +forgelm audit data/ --output ./audit/ --croissant +``` + +Card şunları taşır: + +* dataset seviyesinde kimlik (`name`, `description`, `version`, + `datePublished`, `url`), +* her JSONL split'i için bir `cr:FileObject` (Croissant consumer'ı + altta yatan dosyaları bulabilsin diye), +* her split için audit'in column-detection katmanından türetilen + `cr:Field` girdileriyle bir `cr:RecordSet`. + +Flag kapalıyken blok boştur — mevcut consumer'lar byte-eşdeğer çıktı +görür, `secrets_summary` / `quality_summary`'nin koyduğu emsalin +aynısı. Card'ı HuggingFace / MLCommons'a yayınlamak isteyen operatörler +audit'in birinci sınıf delili olmayan ek Croissant alanlarını +(`license`, `citeAs`, `keywords`) audit'i yeniden çalıştırmadan +elle düzenleyebilir. + +--- + ## Layout gereksinimleri | Girdi şekli | Ne elde edersin | diff --git a/docs/guides/data_audit.md b/docs/guides/data_audit.md index 2ae9f760..0052b1b6 100644 --- a/docs/guides/data_audit.md +++ b/docs/guides/data_audit.md @@ -319,6 +319,89 @@ the audit reproducible (Annex IV requirement) and bare-install-friendly. --- +### ML-NER PII adapter — `--pii-ml` (Phase 12.5, opt-in) + +The default regex detector covers the structured identifiers EU AI Act +Article 10 cares about (email, phone, IBAN, credit card, national IDs). +Phase 12.5 adds an opt-in **Presidio** ([microsoft/presidio](https://github.com/microsoft/presidio)) +adapter that layers ML-NER on top — adding the unstructured-identifier +categories regex inherently misses: `person`, `organization`, `location`. + +```bash +pip install 'forgelm[ingestion-pii-ml]' +forgelm audit data/ --output ./audit/ --pii-ml +``` + +The new categories merge into the existing `pii_summary` and +`pii_severity` blocks under disjoint names so the regex baseline stays +visible alongside the ML signal: + +```json +{ + "pii_summary": { + "email": 12, + "phone": 3, + "person": 47, // ← Presidio + "organization": 18, // ← Presidio + "location": 9 // ← Presidio + }, + "pii_severity": { + "by_tier": {"critical": 0, "high": 0, "medium": 59, "low": 30}, + "by_type": { + "email": {"count": 12, "tier": "medium"}, + "person": {"count": 47, "tier": "medium"}, + "organization": {"count": 18, "tier": "low"} + }, + "worst_tier": "medium" + } +} +``` + +Severity assignment lives in `forgelm.data_audit.PII_ML_SEVERITY`: +`person → medium`, `organization → low`, `location → low`. NER +false-positive rates are materially higher than regex-anchored +detection, so the ML tiers sit deliberately below the regex +`critical`/`high` floors — review the per-row spans before treating an +ML finding as a hard gate. + +Presidio's first analyzer build downloads a spaCy English NER model +(~ 50 MB, one-time). Subsequent runs reuse the cached model. The +adapter is opt-in only; without `--pii-ml` the audit stays on the +zero-extra-deps regex path. + +--- + +### Croissant 1.0 dataset card — `--croissant` (Phase 12.5, opt-in) + +`--croissant` populates a new top-level `croissant` block in +`data_audit_report.json` with a [Google Croissant 1.0](http://mlcommons.org/croissant/) +dataset card (`@type: sc:Dataset`). The card is conformant with the +canonical `mlcommons.org/croissant/1.0` context, so Croissant-aware +consumers (HuggingFace dataset cards, MLCommons reference loaders, the +Croissant validator) can parse the block without modification. + +```bash +forgelm audit data/ --output ./audit/ --croissant +``` + +The card carries: + +* dataset-level identity (`name`, `description`, `version`, + `datePublished`, `url`), +* one `cr:FileObject` per JSONL split (so a Croissant consumer can + locate the underlying files), +* one `cr:RecordSet` per split with `cr:Field` entries derived from + the audit's column-detection layer. + +The block is empty when the flag is off — existing consumers see +byte-equivalent output, the same precedent set by `secrets_summary` / +`quality_summary`. Operators that want to publish the card to +HuggingFace / MLCommons can hand-edit the additional Croissant fields +the audit doesn't have first-class evidence for (`license`, `citeAs`, +`keywords`) without re-running the audit. + +--- + ## Layout requirements | Input shape | What you get | diff --git a/docs/guides/ingestion-tr.md b/docs/guides/ingestion-tr.md index f470e0cb..e1a3fae2 100644 --- a/docs/guides/ingestion-tr.md +++ b/docs/guides/ingestion-tr.md @@ -141,6 +141,26 @@ yukarıdaki regex seti çalışır. --- +## Hepsi bir arada maskeleme — `--all-mask` (Faz 12.5) + +`--all-mask`, `--secrets-mask --pii-mask` için tek-flag kısayoludur — +"paylaşılan korpus üzerinde eğitime başlamadan tespit edilebilir her +şeyi temizle" iş akışının yaygın hâlidir. İki detector belgelenen +sıraya göre çalışır (önce secrets — birleşik PII detector'lar örtüşen +span'leri çift saymasın diye); ortaya çıkan JSONL eşleşme bulunan +yerlerde hem `[REDACTED-SECRET]` hem de `[REDACTED]` belirteçlerini +taşır. + +```bash +forgelm ingest ./mixed_corpus/ --recursive --all-mask --output data/clean.jsonl +``` + +Açık flag'lerle additive birleşir — `--all-mask --pii-mask` hata +değildir; iki flag'in boolean unionu çalışır. Kısayol yalnızca +ergonomi içindir; yeni bir detector eklemez. + +--- + ## Recursive dizin yürüyüşü ```text diff --git a/docs/guides/ingestion.md b/docs/guides/ingestion.md index 4ae8da99..55d0d272 100644 --- a/docs/guides/ingestion.md +++ b/docs/guides/ingestion.md @@ -130,6 +130,26 @@ installing it today does **not** change ingest masking behaviour. --- +## All-in-one masking — `--all-mask` (Phase 12.5) + +`--all-mask` is a one-flag shorthand for `--secrets-mask --pii-mask` — +the common "scrub everything detectable before training on a shared +corpus" workflow. The two underlying detectors run in the documented +order (secrets first so combined PII detectors don't double-count +overlapping spans); the resulting JSONL carries both `[REDACTED-SECRET]` +and `[REDACTED]` tokens where matches were found. + +```bash +forgelm ingest ./mixed_corpus/ --recursive --all-mask --output data/clean.jsonl +``` + +Composes additively with explicit flags — `--all-mask --pii-mask` is +not an error; the boolean union of the two flags is what runs. The +shorthand exists purely as ergonomics; it does not introduce any new +detector. + +--- + ## Recursive directory walk ```text diff --git a/docs/reference/usage-tr.md b/docs/reference/usage-tr.md index 31d7275b..5662ac44 100644 --- a/docs/reference/usage-tr.md +++ b/docs/reference/usage-tr.md @@ -142,6 +142,9 @@ forgelm ingest ./mixed_corpus/ --secrets-mask --output data/clean.jsonl # v0.5.2: secrets + PII maskelemeyi birleştir (önce secrets, sonra PII — # örtüşen span'ler iki kez sayılmasın) forgelm ingest ./mixed_corpus/ --secrets-mask --pii-mask --output data/scrubbed.jsonl + +# v0.5.3 (Faz 12.5): yukarıdaki kombinasyonun tek-flag kısayolu +forgelm ingest ./mixed_corpus/ --all-mask --output data/scrubbed.jsonl ``` ### Veri Seti Denetimi (v0.5.0+; v0.5.1 subcommand; v0.5.2 MinHash + quality + secrets) @@ -168,6 +171,13 @@ forgelm audit data/large_corpus.jsonl --dedup-method minhash --jaccard-threshold # v0.5.2: opt-in heuristik kalite filtresi (Gopher/C4 stili) forgelm audit data/ --quality-filter +# v0.5.3 (Faz 12.5): audit'in yanında bir Google Croissant 1.0 dataset card emit et +forgelm audit data/ --output ./audit/ --croissant + +# v0.5.3 (Faz 12.5): regex PII detector üzerine Presidio NER katmanı ekle +# (`[ingestion-pii-ml]` extra'sını gerektirir — person/organization/location kategorileri ekler) +forgelm audit data/ --output ./audit/ --pii-ml + # stdout'a makine-okunabilir özet forgelm audit data/sft.jsonl --output ./audit/ --output-format json diff --git a/docs/reference/usage.md b/docs/reference/usage.md index ca354f5a..eb850c7c 100644 --- a/docs/reference/usage.md +++ b/docs/reference/usage.md @@ -160,6 +160,9 @@ forgelm ingest ./mixed_corpus/ --secrets-mask --output data/clean.jsonl # v0.5.2: combine secrets + PII masking (secrets first to avoid double-counting) forgelm ingest ./mixed_corpus/ --secrets-mask --pii-mask --output data/scrubbed.jsonl + +# v0.5.3 (Phase 12.5): one-flag shorthand for the same combination above +forgelm ingest ./mixed_corpus/ --all-mask --output data/scrubbed.jsonl ``` ### Dataset Audit (v0.5.0+; subcommand promoted in v0.5.1; MinHash + quality + secrets in v0.5.2) @@ -185,6 +188,13 @@ forgelm audit data/large_corpus.jsonl --dedup-method minhash --jaccard-threshold # v0.5.2: opt-in heuristic quality filter (Gopher/C4 style) forgelm audit data/ --quality-filter +# v0.5.3 (Phase 12.5): emit a Google Croissant 1.0 dataset card alongside the audit +forgelm audit data/ --output ./audit/ --croissant + +# v0.5.3 (Phase 12.5): layer Presidio NER on top of the regex PII detector +# (needs `[ingestion-pii-ml]` extra — adds person/organization/location categories) +forgelm audit data/ --output ./audit/ --pii-ml + # Machine-readable summary on stdout forgelm audit data/sft.jsonl --output ./audit/ --output-format json diff --git a/docs/roadmap-tr.md b/docs/roadmap-tr.md index aa4cef5c..55fe8290 100644 --- a/docs/roadmap-tr.md +++ b/docs/roadmap-tr.md @@ -12,7 +12,7 @@ | ✅ Tamam | [Faz 11 — Doküman Yutma ve Veri Denetimi](roadmap/phase-11-data-ingestion.md) | `forgelm ingest`, `forgelm --data-audit`, PII regex + simhash dedup — `v0.5.0` çıktı | | 🟡 Birleşti | [Faz 11.5 — Yutma / Denetim Cilası](roadmap/phase-11-5-backlog.md) | LSH banding, streaming reader, `forgelm audit` subcommand, PII şiddet katmanları, wizard ingest girişi — `main`'e indi (PR #11/#12); `v0.5.1` PyPI tag'i bekliyor | | 🟡 Birleşti | [Faz 12 — Veri Curation Olgunlaşması](roadmap/phase-12-data-curation-maturity.md) | MinHash LSH dedup, markdown-aware splitter, code/secrets tarama, kalite heuristic'leri, tablo yapısı koruma — Tier 1 `main`'e indi (PR #13, 2026-04-29); `v0.5.2` PyPI tag'i bekliyor; Tier 2/3 [Faz 12.5 backlog](roadmap/phase-12-5-backlog.md)'a ertelendi | -| 📋 Planlandı | [Faz 12.5 — Veri Curation Follow-up](roadmap/phase-12-5-backlog.md) | Presidio PII adapter, Croissant metadata export, `--all-mask`, wizard "audit first" akışı → `v0.5.2` tag'inden sonra | +| 🟡 Birleşti | [Faz 12.5 — Veri Curation Follow-up](roadmap/phase-12-5-backlog.md) | Presidio PII adaptörü (`[ingestion-pii-ml]`), Croissant 1.0 metadata (`--croissant`), `--all-mask`, wizard "audit first" — `development`'a indi; `v0.5.3` (Faz 14) ile çıkacak | | 📋 Planlandı | [Faz 14 — Çok Aşamalı Pipeline Zincirleri](roadmap/phase-14-pipeline-chains.md) | SFT → DPO → GRPO config zinciri, pipeline kaynak izleri → `v0.5.3` | | 📋 Planlandı | [Faz 13 — Pro CLI ve Gözlemlenebilirlik Dashboard](roadmap/phase-13-pro-cli.md) | Lisans korumalı dashboard, HPO, zamanlanmış görevler, takım config store → `v0.6.0-pro` | @@ -24,7 +24,7 @@ **Daha öncesi:** `v0.4.0` — Post-Training Tamamlama (2026-04-26). Inference primitif'leri, etkileşimli chat REPL, GGUF export, VRAM fit advisor, deployment config üretimi. -**Sonraki:** `v0.5.3` — Çok Aşamalı Pipeline Zincirleri (Faz 14). SFT → DPO → GRPO zincir konfigürasyonu, pipeline kaynak izleri. `v0.5.2` PyPI tag'inden sonra başlar; [#14 webhook SSRF hardening](https://github.com/cemililik/ForgeLM/issues/14) follow-up'ını kapsar. +**Sonraki:** `v0.5.3` — Çok Aşamalı Pipeline Zincirleri (Faz 14) + Faz 12.5 cilası. SFT → DPO → GRPO zincir konfigürasyonu, pipeline kaynak izleri. `v0.5.2` PyPI tag'inden sonra başlar; [#14 webhook SSRF hardening](https://github.com/cemililik/ForgeLM/issues/14) follow-up'ını ve `development`'a inen dört Faz 12.5 backlog item'ını (Presidio NER, Croissant 1.0 card, `--all-mask`, wizard "audit first") kapsar. **Güncel durum:** 16 faz (1, 2, 2.5, 3, 4, 5, 5.5, 6, 7, 8, 9, 10, 10.5, 11, 11.5, 12) tamam. 3 faz (12.5, 13, 14) planlandı. `v0.5.3`: Faz 14 (12.5 backlog'u fırsat oldukça eklenir). `v0.6.0-pro` (Faz 13) adoption metriklerine bağlı. diff --git a/docs/roadmap.md b/docs/roadmap.md index cd3a5f04..25b67dcc 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -12,7 +12,7 @@ | ✅ Done | [Phase 11 — Document Ingestion & Data Audit](roadmap/phase-11-data-ingestion.md) | `forgelm ingest`, `forgelm --data-audit`, PII regex + simhash dedup — shipped `v0.5.0` | | 🟡 Merged | [Phase 11.5 — Ingestion / Audit Polish](roadmap/phase-11-5-backlog.md) | LSH banding, streaming reader, `forgelm audit` subcommand, PII severity tiers, wizard ingest path — merged to `main` (PR #11/#12); PyPI tag pending as `v0.5.1` | | 🟡 Merged | [Phase 12 — Data Curation Maturity](roadmap/phase-12-data-curation-maturity.md) | MinHash LSH dedup, markdown-aware splitter, code/secrets scan, quality heuristics, table preservation — Tier 1 merged to `main` (PR #13, 2026-04-29); PyPI tag pending as `v0.5.2`; Tier 2/3 deferred to [Phase 12.5 backlog](roadmap/phase-12-5-backlog.md) | -| 📋 Planned | [Phase 12.5 — Data Curation Follow-up](roadmap/phase-12-5-backlog.md) | Presidio PII adapter, Croissant metadata export, `--all-mask`, wizard "audit first" flow → after `v0.5.2` tag | +| 🟡 Merged | [Phase 12.5 — Data Curation Follow-up](roadmap/phase-12-5-backlog.md) | Presidio PII adapter (`[ingestion-pii-ml]`), Croissant 1.0 metadata (`--croissant`), `--all-mask`, wizard "audit first" — implemented on `development`; ships with `v0.5.3` (Phase 14) | | 📋 Planned | [Phase 14 — Multi-Stage Pipeline Chains](roadmap/phase-14-pipeline-chains.md) | SFT → DPO → GRPO chained config, pipeline provenance artifacts → `v0.5.3` | | 📋 Planned | [Phase 13 — Pro CLI & Observability Dashboard](roadmap/phase-13-pro-cli.md) | License-gated dashboard, HPO, scheduled jobs, team config store → `v0.6.0-pro` | @@ -24,7 +24,7 @@ **Earlier:** `v0.4.0` — Post-Training Completion (2026-04-26). Inference primitives, interactive chat REPL, GGUF export, VRAM fit advisor, deployment config generation. -**Next:** `v0.5.3` — Multi-Stage Pipeline Chains (Phase 14). SFT → DPO → GRPO chained config, pipeline provenance artifacts. Starts after the `v0.5.2` PyPI tag is published. Folds in [#14 webhook SSRF hardening](https://github.com/cemililik/ForgeLM/issues/14). +**Next:** `v0.5.3` — Multi-Stage Pipeline Chains (Phase 14) + Phase 12.5 polish. SFT → DPO → GRPO chained config, pipeline provenance artifacts. Starts after the `v0.5.2` PyPI tag is published. Folds in [#14 webhook SSRF hardening](https://github.com/cemililik/ForgeLM/issues/14) and the four Phase 12.5 backlog items now landed on `development` (Presidio NER, Croissant 1.0 card, `--all-mask`, wizard "audit first"). **Current state:** 16 phases (1, 2, 2.5, 3, 4, 5, 5.5, 6, 7, 8, 9, 10, 10.5, 11, 11.5, 12) complete. 3 phases (12.5, 13, 14) planned. `v0.5.3`: Phase 14 (12.5 backlog folded in opportunistically). `v0.6.0-pro` (Phase 13) gated on adoption metrics. diff --git a/docs/roadmap/phase-12-5-backlog.md b/docs/roadmap/phase-12-5-backlog.md index 45ec7806..88ee36f4 100644 --- a/docs/roadmap/phase-12-5-backlog.md +++ b/docs/roadmap/phase-12-5-backlog.md @@ -14,12 +14,12 @@ > **Scope:** small additions only — none require new architecture. > Each row is one PR. -| # | Item | Source | Effort | Impact | +| # | Item | Source | Effort | Status | |---|---|---|---|---| -| **1** | **Presidio adapter** (`[ingestion-pii-ml]` extra, optional) | Phase 12 plan Tier 2 #6 | S–M | ML-NER signal regex misses (person names, organisations, locations); maps into the existing `pii_severity` table with new tier rows. Default regex+Luhn+TC-Kimlik path stays. | -| **2** | **Croissant metadata compatibility** (audit JSON) | Phase 12 plan Tier 2 #7 | S | Adds a Google Croissant-shaped subset to `data_audit_report.json` so the file doubles as both an EU AI Act Article 10 governance artifact and a Croissant-consumer dataset card. Opt-in `--croissant` flag; existing audit consumers unaffected. | -| **3** | **`forgelm ingest --all-mask` composite flag** | Phase 12 plan Tier 3 #8 | XS | One-flag shorthand that runs `--secrets-mask` then `--pii-mask` in the documented order. Pure UX; no new behaviour. Test: combined-fixture roundtrip. | -| **4** | **Wizard "audit first" entry point** | Phase 12 plan Tier 3 #9 | S | Mirrors Phase 11.5's `_offer_ingest_for_directory` pattern: when the user provides a JSONL, the wizard offers to run `forgelm audit` and prints the verdicts before continuing. Closes the audit loop in the BYOD path. | +| **1** | ~~**Presidio adapter** (`[ingestion-pii-ml]` extra, optional)~~ | Phase 12 plan Tier 2 #6 | S–M | ✅ Landed on `development` — `forgelm audit --pii-ml`; new `PII_ML_SEVERITY` table; opt-in `[ingestion-pii-ml]` extra (`presidio-analyzer`); ships with `v0.5.3`. | +| **2** | ~~**Croissant metadata compatibility** (audit JSON)~~ | Phase 12 plan Tier 2 #7 | S | ✅ Landed on `development` — `forgelm audit --croissant`; Croissant 1.0 card under report's `croissant` key; existing keys byte-equivalent when off. | +| **3** | ~~**`forgelm ingest --all-mask` composite flag**~~ | Phase 12 plan Tier 3 #8 | XS | ✅ Landed on `development` — `forgelm ingest --all-mask`; set-union with explicit flags; resolved at the CLI boundary. | +| **4** | ~~**Wizard "audit first" entry point**~~ | Phase 12 plan Tier 3 #9 | S | ✅ Landed on `development` — `_offer_audit_for_jsonl` invoked from the three JSONL-finalisation paths in `forgelm/wizard.py`. | ## Why these landed here, not in Phase 12 diff --git a/docs/standards/architecture.md b/docs/standards/architecture.md index 63cc1222..6c9fb9a0 100644 --- a/docs/standards/architecture.md +++ b/docs/standards/architecture.md @@ -173,6 +173,7 @@ From [`pyproject.toml`](../../pyproject.toml): | `ingestion` | pypdf, python-docx, ebooklib, beautifulsoup4, langdetect, xxhash | Any | | `ingestion-scale` | datasketch (MinHash LSH dedup; Phase 12, opt-in) | Any | | `ingestion-secrets` | detect-secrets (credential leakage scanner; Phase 12, opt-in regex fallback when missing) | Any | +| `ingestion-pii-ml` | presidio-analyzer (ML-NER for person/organization/location PII; Phase 12.5, opt-in) | Any | | `export` | llama-cpp-python (GGUF conversion) | Linux/macOS | | `chat` | rich (terminal rendering) | Any | | `dev` | pytest, ruff | Any (contributors) | diff --git a/forgelm/cli.py b/forgelm/cli.py index 1a4f2dd7..07d8413a 100644 --- a/forgelm/cli.py +++ b/forgelm/cli.py @@ -329,6 +329,16 @@ def _add_ingest_subcommand(subparsers) -> None: "stronger label first." ), ) + p.add_argument( + "--all-mask", + action="store_true", + help=( + "Phase 12.5 shorthand: equivalent to --secrets-mask --pii-mask. " + "Convenience for the common 'scrub everything detectable before " + "training on shared corpora' workflow. Composes additively with " + "explicit --pii-mask / --secrets-mask flags (set-union, no error)." + ), + ) p.add_argument( "--chunk-tokens", type=_non_negative_int, @@ -465,6 +475,26 @@ def _add_audit_subcommand(subparsers) -> None: "deferred to Phase 13+." ), ) + p.add_argument( + "--croissant", + action="store_true", + help=( + "Phase 12.5: emit a Google Croissant 1.0 dataset card (sc:Dataset / cr:RecordSet) " + "under the report's 'croissant' key so the same data_audit_report.json doubles as " + "both the EU AI Act Article 10 governance artifact and a Croissant-consumer dataset " + "card. Existing audit JSON keys are unchanged; the block is empty when this flag is off." + ), + ) + p.add_argument( + "--pii-ml", + action="store_true", + help=( + "Phase 12.5 opt-in: layer Presidio's ML-NER PII detection (person / organization / " + "location categories) on top of the regex detector. Requires the optional " + "'forgelm[ingestion-pii-ml]' extra; raises an install-hint ImportError when missing. " + "Findings merge into pii_summary / pii_severity under disjoint category names." + ), + ) _add_common_subparser_flags(p, include_output_format=True) @@ -1277,6 +1307,13 @@ def _run_ingest_cmd(args, output_format: str) -> None: """Phase 11 dispatch: raw documents → SFT-ready JSONL.""" from .ingestion import ingest_path, summarize_result + # Phase 12.5: --all-mask is a shorthand that ORs into the two individual + # masking flags. Resolve it here at the CLI boundary so ``ingest_path`` + # keeps its narrow API (only ``pii_mask`` / ``secrets_mask`` booleans). + all_mask = getattr(args, "all_mask", False) + pii_mask = bool(args.pii_mask) or all_mask + secrets_mask = bool(getattr(args, "secrets_mask", False)) or all_mask + try: result = ingest_path( args.input_path, @@ -1285,8 +1322,8 @@ def _run_ingest_cmd(args, output_format: str) -> None: overlap=args.overlap, strategy=args.strategy, recursive=args.recursive, - pii_mask=args.pii_mask, - secrets_mask=getattr(args, "secrets_mask", False), + pii_mask=pii_mask, + secrets_mask=secrets_mask, chunk_tokens=getattr(args, "chunk_tokens", None), overlap_tokens=getattr(args, "overlap_tokens", 0), tokenizer=getattr(args, "tokenizer", None), @@ -1349,6 +1386,8 @@ def _run_data_audit( dedup_method: str = "simhash", minhash_jaccard: Optional[float] = None, enable_quality_filter: bool = False, + enable_pii_ml: bool = False, + emit_croissant: bool = False, invoked_via_legacy_flag: bool = False, ) -> None: """Phase 11 / 11.5 / 12 dispatch: dataset quality + governance audit. @@ -1383,6 +1422,8 @@ def _run_data_audit( dedup_method=dedup_method, minhash_jaccard=jaccard, enable_quality_filter=enable_quality_filter, + enable_pii_ml=enable_pii_ml, + emit_croissant=emit_croissant, ) except OSError as exc: # OSError covers FileNotFoundError / PermissionError / ENOSPC / @@ -1454,6 +1495,8 @@ def _run_audit_cmd(args, output_format: str) -> None: dedup_method=getattr(args, "dedup_method", "simhash"), minhash_jaccard=getattr(args, "jaccard_threshold", None), enable_quality_filter=getattr(args, "quality_filter", False), + enable_pii_ml=getattr(args, "pii_ml", False), + emit_croissant=getattr(args, "croissant", False), invoked_via_legacy_flag=False, ) diff --git a/forgelm/data_audit.py b/forgelm/data_audit.py index 2e05d775..0e3cb4da 100644 --- a/forgelm/data_audit.py +++ b/forgelm/data_audit.py @@ -94,6 +94,23 @@ _HAS_DATASKETCH = False +# Phase 12.5: optional Presidio ML-NER PII detection. Layered ON TOP of +# the regex detector so the default audit keeps its zero-extra-deps +# guarantee. Presidio + spaCy are heavyweight (≈ 50 MB model download), +# so we stay strictly opt-in and fail soft when missing — the regex set +# already covers the GDPR-mandated structured identifiers (email, phone, +# IBAN, credit card, national IDs) that the audit's compliance contract +# is built around. ML adds the *unstructured* identifiers regex inherently +# misses: person names, organisations, locations. +try: # pragma: no cover — exercised by the dedicated extras-skip tests + from presidio_analyzer import AnalyzerEngine as _PresidioAnalyzer + + _HAS_PRESIDIO = True +except ImportError: # pragma: no cover + _PresidioAnalyzer = None # type: ignore[assignment] + _HAS_PRESIDIO = False + + # --------------------------------------------------------------------------- # Public types # --------------------------------------------------------------------------- @@ -135,6 +152,27 @@ with the worst-case findings rather than burying them behind low tiers.""" +# Phase 12.5: severity for the ML-NER detector (Presidio adapter). These +# categories sit alongside the regex set and feed the same pii_severity +# block, which is why they share the ``critical/high/medium/low`` tier +# vocabulary. Entries are deliberately on the lower end of the tier +# table because NER false-positive rates are materially higher than the +# regex-anchored detectors (a model name, a city, or an organization +# string can be flagged with no privacy impact); compliance reviewers +# can still upgrade the tier locally if their corpus warrants it. +PII_ML_SEVERITY: Dict[str, str] = { + "person": "medium", + "organization": "low", + "location": "low", +} + + +PII_ML_TYPES: Tuple[str, ...] = tuple(PII_ML_SEVERITY.keys()) +"""Canonical category names for ML-detected PII. Disjoint from +:data:`PII_TYPES` (regex categories) so the two detectors can run side +by side without double-counting the same span.""" + + # Columns we treat as text payloads when computing length / language / dedup. # Order matters: first match wins per row. _TEXT_COLUMNS: Tuple[str, ...] = ("text", "content", "completion", "prompt") @@ -191,6 +229,10 @@ class AuditReport: # dicts on default audits with no findings. secrets_summary: Dict[str, int] = field(default_factory=dict) quality_summary: Dict[str, Any] = field(default_factory=dict) + # Phase 12.5: optional Google Croissant 1.0 dataset card. Empty dict + # by default so existing consumers see byte-identical output; set + # via ``audit_dataset(..., emit_croissant=True)`` or ``--croissant``. + croissant: Dict[str, Any] = field(default_factory=dict) notes: List[str] = field(default_factory=list) @@ -350,6 +392,100 @@ def _replace(match: re.Match, _t: str = pii_type) -> str: return (out, counts) if return_counts else out +# --------------------------------------------------------------------------- +# Phase 12.5: optional Presidio ML-NER PII adapter +# --------------------------------------------------------------------------- + + +# Presidio analyzer is heavy to construct (loads spaCy + an English NER +# model on first call), so we cache a single instance for the duration +# of the process. ``maxsize=1`` keeps the cache trivially eviction-free +# while still letting test code blow the cache via ``cache_clear()``. +@lru_cache(maxsize=1) +def _get_presidio_analyzer() -> Any: + """Return a cached :class:`presidio_analyzer.AnalyzerEngine` instance. + + Raises: + ImportError: when the ``[ingestion-pii-ml]`` extra is missing. + Also surfaced through the public :func:`_require_presidio` + helper so callers that gate on Presidio availability up + front can produce a clean message before processing starts. + """ + _require_presidio() + return _PresidioAnalyzer() + + +def _require_presidio() -> None: + """Raise a clear ImportError when the optional ML-NER backend is missing. + + Mirrors :func:`_require_datasketch` so any audit code path that + needs Presidio fails with an actionable install hint instead of a + deep ``AttributeError`` the operator can't act on. + """ + if not _HAS_PRESIDIO: + raise ImportError( + "Presidio PII ML detection requires the 'ingestion-pii-ml' extra. " + "Install with: pip install 'forgelm[ingestion-pii-ml]'" + ) + + +# Map of Presidio's canonical entity-type strings (which mirror the +# spaCy NER label set) to the ForgeLM category names that feed +# :data:`PII_ML_SEVERITY` / :data:`PII_ML_TYPES`. Keys come from +# Presidio's recognizer registry; multiple Presidio labels can collapse +# into a single ForgeLM bucket (``GPE`` / ``LOCATION`` both → ``location``) +# because the audit's downstream consumer (compliance reviewer) doesn't +# need finer granularity than person / organization / location. +_PRESIDIO_ENTITY_MAP: Dict[str, str] = { + "PERSON": "person", + "ORG": "organization", + "ORGANIZATION": "organization", + "LOCATION": "location", + "GPE": "location", + "NORP": "location", + "FAC": "location", +} + + +def detect_pii_ml(text: Any) -> Dict[str, int]: + """ML-NER PII detector — counts ``person`` / ``organization`` / ``location`` spans. + + Layered ON TOP of :func:`detect_pii` (regex). The two detectors + return disjoint category sets so the merged ``pii_summary`` shows + both the structured-identifier signal (regex) and the + unstructured-identifier signal (ML) without double-counting the + same span. + + Returns an empty dict when: + * ``text`` is not a non-empty string (defensive — callers pass JSONL + payloads that may be ``None`` / numbers / dicts); + * Presidio is not installed (the caller must opt in explicitly via + :func:`_require_presidio` if they want a hard failure); + * the analyzer raises on the input — Presidio can fail on + pathological strings, model-load races, or version skews. We + swallow per-row failures and return zero counts so a single bad + row never blocks the audit. + """ + if not isinstance(text, str) or not text.strip(): + return {} + if not _HAS_PRESIDIO: + return {} + try: + analyzer = _get_presidio_analyzer() + results = analyzer.analyze(text=text, language="en") + except Exception: # pragma: no cover — Presidio raises on edge cases + # Per-row resilience: a single malformed row must not abort the + # audit. The aggregator counts no ML PII for this row, which is + # the correct behaviour for opt-in best-effort detection. + return {} + counts: Dict[str, int] = {} + for finding in results: + kind = _PRESIDIO_ENTITY_MAP.get(getattr(finding, "entity_type", "")) + if kind: + counts[kind] = counts.get(kind, 0) + 1 + return counts + + # --------------------------------------------------------------------------- # Simhash + near-duplicate detection # --------------------------------------------------------------------------- @@ -1376,6 +1512,8 @@ class _StreamingAggregator: # Phase 12 configuration (set once by the orchestrator; never mutated). dedup_method: str = "simhash" minhash_num_perm: int = DEFAULT_MINHASH_NUM_PERM + # Phase 12.5: opt-in ML-NER PII detection (Presidio). Off by default. + enable_pii_ml: bool = False enable_quality_filter: bool = False @@ -1415,6 +1553,13 @@ def _record_text_metrics(agg: _StreamingAggregator, payload: str) -> None: _record_dedup_signature(agg, payload) for kind, count in detect_pii(payload).items(): agg.pii_counts[kind] = agg.pii_counts.get(kind, 0) + count + if agg.enable_pii_ml: + # Phase 12.5: Presidio NER findings layer onto the same pii_counts + # bucket. The category names are disjoint from the regex set + # (person / organization / location vs. email / phone / *_id), + # so the merged counts present both views without double-counting. + for kind, count in detect_pii_ml(payload).items(): + agg.pii_counts[kind] = agg.pii_counts.get(kind, 0) + count for kind, count in detect_secrets(payload).items(): agg.secrets_counts[kind] = agg.secrets_counts.get(kind, 0) + count if agg.enable_quality_filter: @@ -1584,6 +1729,7 @@ def _audit_split( minhash_jaccard: float = DEFAULT_MINHASH_JACCARD, minhash_num_perm: int = DEFAULT_MINHASH_NUM_PERM, enable_quality_filter: bool = False, + enable_pii_ml: bool = False, ) -> Tuple[Dict[str, Any], List[Any], Dict[str, int], int, int]: """Stream a JSONL split into a metrics record. @@ -1621,6 +1767,7 @@ def _audit_split( dedup_method=dedup_method, minhash_num_perm=minhash_num_perm, enable_quality_filter=enable_quality_filter, + enable_pii_ml=enable_pii_ml, ) for row, parse_err, decode_err in _read_jsonl_split(path): _ingest_row(agg, row, parse_err, decode_err) @@ -1938,6 +2085,7 @@ def _process_split( minhash_jaccard: float = DEFAULT_MINHASH_JACCARD, minhash_num_perm: int = DEFAULT_MINHASH_NUM_PERM, enable_quality_filter: bool = False, + enable_pii_ml: bool = False, ) -> _SplitOutcome: """Stream + audit one split. Tolerates per-split filesystem failures. @@ -1957,6 +2105,7 @@ def _process_split( minhash_jaccard=minhash_jaccard, minhash_num_perm=minhash_num_perm, enable_quality_filter=enable_quality_filter, + enable_pii_ml=enable_pii_ml, ) except OSError as exc: logger.warning("Could not read split '%s' (%s): %s — skipping.", split_name, path, exc) @@ -2009,12 +2158,15 @@ def _build_pii_severity(pii_summary: Dict[str, int]) -> Dict[str, Any]: from :data:`PII_SEVERITY` (forward-compat for new types) fall back to ``unknown``. - A snapshot of :data:`PII_SEVERITY` is taken at call time so a - test or downstream caller mutating the module-level dict cannot - corrupt the audit output mid-run; per-call audits see a stable - table for the duration of their work. + A snapshot of :data:`PII_SEVERITY` (regex categories) and + :data:`PII_ML_SEVERITY` (Phase 12.5 ML-NER categories) is taken at + call time so a test or downstream caller mutating either dict + cannot corrupt the audit output mid-run; per-call audits see a + stable merged table for the duration of their work. The two + tables share the same tier vocabulary (``critical/high/medium/low``) + on purpose so the verdict surface stays unified. """ - severity_table = dict(PII_SEVERITY) + severity_table = {**PII_SEVERITY, **PII_ML_SEVERITY} if not pii_summary: return { "total": 0, @@ -2189,6 +2341,146 @@ def _build_quality_summary( } +# Phase 12.5: Google Croissant 1.0 dataset-card emission. Croissant is a +# JSON-LD vocabulary built on top of schema.org; mlcommons.org/croissant +# describes the canonical context. We emit a minimum-viable subset that's +# valid against the spec — tools that consume Croissant (HuggingFace's +# dataset-cards, Croissant validator, MLCommons reference loaders) can +# parse the block without modification, while the rest of the audit JSON +# stays untouched. The card lives under the new ``croissant`` key on the +# ``AuditReport`` dataclass; callers that don't want it never see it. +_CROISSANT_VOCAB_VERSION = "1.0" +_CROISSANT_CONTEXT: Dict[str, Any] = { + "@language": "en", + "@vocab": "https://schema.org/", + "sc": "https://schema.org/", + "cr": "http://mlcommons.org/croissant/", + "data": {"@id": "cr:data", "@type": "@json"}, + "dataType": {"@id": "cr:dataType", "@type": "@vocab"}, + "extract": "cr:extract", + "field": "cr:field", + "fileProperty": "cr:fileProperty", + "format": "cr:format", + "includes": "cr:includes", + "isLiveDataset": "cr:isLiveDataset", + "jsonPath": "cr:jsonPath", + "key": "cr:key", + "parentField": "cr:parentField", + "path": "cr:path", + "recordSet": "cr:recordSet", + "references": "cr:references", + "regex": "cr:regex", + "repeated": "cr:repeated", + "replace": "cr:replace", + "separator": "cr:separator", + "source": "cr:source", + "subField": "cr:subField", + "transform": "cr:transform", +} + + +def _build_croissant_metadata( + *, + source_path: str, + source_input: str, + generated_at: str, + total_samples: int, + splits_info: Dict[str, Dict[str, Any]], + splits_paths: Dict[str, Path], +) -> Dict[str, Any]: + """Render a minimum-viable Croissant 1.0 dataset card for the audited corpus. + + The card exposes: + * dataset-level identity (name, description, version, datePublished), + * one ``cr:FileObject`` per JSONL split (so a Croissant consumer can + locate the underlying files), + * one ``cr:RecordSet`` per split with the columns the audit detected + under ``splits..columns`` mapped to ``cr:Field`` entries. + + The mapping intentionally stays additive — Croissant supports many + optional fields (citeAs, license, keywords, sameAs, etc.) that the + audit does not have first-class evidence for. Operators that want to + publish the card to HuggingFace / MLCommons can hand-edit those + fields without re-running the audit. + """ + # Derive a human-readable name from the source path. ``Path.stem`` for + # a JSONL ("policies.jsonl" → "policies"); for a directory we use the + # directory name. Fall back to ``source_input`` so HF Hub IDs survive + # in the card even though the audit is filesystem-only today. + src = Path(source_path) + if src.is_file(): + name = src.stem + elif src.is_dir(): + name = src.name + else: + name = source_input or "dataset" + + distribution: List[Dict[str, Any]] = [] + record_sets: List[Dict[str, Any]] = [] + for split_name, info in splits_info.items(): + split_path = splits_paths.get(split_name) + sample_count = int(info.get("sample_count", 0)) + file_id = f"{split_name}.jsonl" + distribution.append( + { + "@type": "cr:FileObject", + "@id": file_id, + "name": file_id, + "contentUrl": str(split_path) if split_path is not None else file_id, + "encodingFormat": "application/jsonlines", + "description": f"Split {split_name!r}: {sample_count} sample(s).", + } + ) + + # Map the audit's detected columns to ``cr:Field`` entries. + # ``columns`` is a list of strings (the keys present in the JSONL + # rows). When the column is one of the canonical text-payload + # columns we type it as ``sc:Text``; everything else is ``sc:Text`` + # too (the audit doesn't track per-column dtypes — that's a + # consumer-side concern). + columns = info.get("columns") or [] + fields = [ + { + "@type": "cr:Field", + "@id": f"{split_name}/{column}", + "name": column, + "dataType": "sc:Text", + "source": { + "fileObject": {"@id": file_id}, + "extract": {"jsonPath": f"$.{column}"}, + }, + } + for column in columns + ] + record_sets.append( + { + "@type": "cr:RecordSet", + "@id": split_name, + "name": split_name, + "field": fields, + "description": f"Records from split {split_name!r}.", + } + ) + + return { + "@context": dict(_CROISSANT_CONTEXT), + "@type": "sc:Dataset", + "conformsTo": "http://mlcommons.org/croissant/1.0", + "name": name, + "description": ( + "ForgeLM audit-generated dataset card. " + f"{total_samples} sample(s) across {len(splits_info)} split(s). " + "Inline counts/quality/PII summaries live in the parent " + "data_audit_report.json under the canonical audit keys." + ), + "url": str(source_path), + "version": _CROISSANT_VOCAB_VERSION, + "datePublished": generated_at, + "distribution": distribution, + "recordSet": record_sets, + } + + def _build_near_duplicate_summary( *, dedup_method: str, @@ -2219,6 +2511,8 @@ def audit_dataset( minhash_jaccard: float = DEFAULT_MINHASH_JACCARD, minhash_num_perm: int = DEFAULT_MINHASH_NUM_PERM, enable_quality_filter: bool = False, + enable_pii_ml: bool = False, + emit_croissant: bool = False, ) -> AuditReport: """Run the audit pipeline over a JSONL file or split-keyed directory. @@ -2240,6 +2534,18 @@ def audit_dataset( similarity terms. minhash_num_perm: Number of permutations for ``datasketch.MinHash``. Default ``128`` matches datasketch's own default. + enable_pii_ml: Phase 12.5 opt-in — layer Presidio's ML-NER PII + detector (``person`` / ``organization`` / ``location``) on + top of the regex detector. Requires the optional + ``[ingestion-pii-ml]`` extra; raises ``ImportError`` with an + install hint when missing so the failure surfaces before + any rows are scanned. + emit_croissant: Phase 12.5 opt-in flag — when ``True``, populate + the report's ``croissant`` field with a Google Croissant 1.0 + dataset card (``@type: sc:Dataset``) so the same JSON file + doubles as both the EU AI Act Article 10 governance artifact + and a Croissant-consumer dataset card. Off by default — older + consumers see byte-identical output until they opt in. enable_quality_filter: Phase 12 opt-in flag — when ``True``, run the heuristic quality checks (Gopher / C4 / RefinedWeb-style) and surface findings under ``quality_summary``. @@ -2257,6 +2563,11 @@ def audit_dataset( if not isinstance(minhash_num_perm, int) or isinstance(minhash_num_perm, bool) or minhash_num_perm <= 0: raise ValueError(f"minhash_num_perm must be a positive integer; got {minhash_num_perm!r}.") _require_datasketch() + if enable_pii_ml: + # Phase 12.5: pre-flight the optional extra so the caller learns + # the dep is missing BEFORE we open any files / scan any rows, + # mirroring the ``_require_datasketch`` shape above. + _require_presidio() splits_paths, resolution_notes = _resolve_input(source) @@ -2283,6 +2594,7 @@ def audit_dataset( minhash_jaccard=minhash_jaccard, minhash_num_perm=minhash_num_perm, enable_quality_filter=enable_quality_filter, + enable_pii_ml=enable_pii_ml, ) splits_info[split_name] = outcome.info signatures_by_split[split_name] = outcome.signatures @@ -2344,9 +2656,22 @@ def audit_dataset( minhash_num_perm=minhash_num_perm, ) + generated_at = datetime.now(timezone.utc).isoformat() + source_path_abs = os.fspath(Path(source).expanduser().resolve()) + croissant: Dict[str, Any] = {} + if emit_croissant: + croissant = _build_croissant_metadata( + source_path=source_path_abs, + source_input=source, + generated_at=generated_at, + total_samples=total_samples, + splits_info=splits_info, + splits_paths=splits_paths, + ) + report = AuditReport( - generated_at=datetime.now(timezone.utc).isoformat(), - source_path=os.fspath(Path(source).expanduser().resolve()), + generated_at=generated_at, + source_path=source_path_abs, source_input=source, total_samples=total_samples, splits=splits_info, @@ -2356,6 +2681,7 @@ def audit_dataset( near_duplicate_summary=near_duplicate_summary, secrets_summary=secrets_summary, quality_summary=quality_summary, + croissant=croissant, notes=notes, ) diff --git a/forgelm/wizard.py b/forgelm/wizard.py index dd365c88..66c25da5 100644 --- a/forgelm/wizard.py +++ b/forgelm/wizard.py @@ -447,9 +447,74 @@ def _offer_ingest_for_directory(directory: Path) -> Optional[str]: return None print(f" Ingest complete: {result.chunk_count} chunk(s) from {result.files_processed} file(s) → {out_path}") + # Phase 12.5: now that we have a JSONL produced from raw docs, offer to + # run a quality + governance audit on it before handing it to the wizard. + # Closes the BYOD loop — operators see length / language / dedup / PII + # / secrets / quality verdicts at config-time, not 30 minutes into a + # training run. + _offer_audit_for_jsonl(out_path) return str(out_path) +def _offer_audit_for_jsonl(jsonl_path: Path) -> bool: + """Phase 12.5 wizard hook: optionally audit a JSONL the user just selected. + + Mirrors :func:`_offer_ingest_for_directory`'s shape — a non-blocking + interactive offer that, on acceptance, runs the audit and prints a + short verdict before the wizard continues. Returns ``True`` when the + audit ran (regardless of findings) and ``False`` when the operator + declined or the audit could not start. Audit failures are surfaced + as warnings: the audit is informational, not a gate, and a malformed + JSONL or missing optional dep must NOT crash the wizard. + """ + if not _prompt_yes_no( + f"\n Run a quality + governance audit on '{jsonl_path}' before training? " + "(length stats, language, near-duplicates, PII, secrets — CPU-only, ~seconds)", + default=True, + ): + print(" Skipped — audit can be run later via:") + print(f" forgelm audit {jsonl_path}") + return False + + try: + # Lazy import — keeps the wizard cold-start cheap and matches the + # existing pattern used elsewhere in this file (see + # ``_offer_ingest_for_directory``'s ``ingest_path`` import). + from .data_audit import audit_dataset, summarize_report + except ImportError as exc: # pragma: no cover — extras-skip path + print(f" audit subsystem unavailable: {exc}") + return False + + print(f"\n Running audit on '{jsonl_path}'…") + try: + report = audit_dataset(str(jsonl_path)) + except (FileNotFoundError, ValueError, OSError) as exc: + # Filesystem / parse errors — same defensive shape we use for + # ``_offer_ingest_for_directory`` so the wizard never aborts on a + # courtesy step. + print(f" Audit could not run: {exc}") + return False + except ImportError as exc: + # The wizard's audit hook never asks for ``--dedup-method=minhash`` + # or any other feature gated on an optional extra; an ImportError + # here would only fire if a future feature inside ``audit_dataset`` + # added an unguarded import. Surface and recover. + print(f" Audit could not run (missing optional dep): {exc}") + return False + except Exception as exc: # noqa: BLE001 — see docstring + # Last-ditch guard: a malformed JSONL row mid-file, a datasketch + # version-mismatch surfaced through datasketch's own internals, or + # an unanticipated edge case in a downstream audit module must NOT + # crash the wizard. The audit is informational; we log loudly and + # let the operator continue. + print(f" Audit could not run: {exc}") + return False + + print("\n Audit complete. Summary:") + print(summarize_report(report, verbose=False)) + return True + + def _prompt_dataset_path_with_ingest_offer(question: str) -> str: """Prompt for a dataset path; auto-offer ingestion when the user gives a directory. @@ -468,10 +533,14 @@ def _prompt_dataset_path_with_ingest_offer(question: str) -> str: ingested = _offer_ingest_for_directory(candidate) if ingested is None: continue + # ``_offer_ingest_for_directory`` already invoked the audit hook + # on the produced JSONL — don't double-prompt here. return ingested - # Files / Hub IDs / non-existent paths are returned verbatim. The - # trainer's data loader will fail later if the path is bogus — - # consistent with the wizard's pre-Phase-11.5 behaviour. + # Phase 12.5: when the path resolves to an existing local JSONL, + # offer to audit it before continuing. Hub IDs and non-existent + # paths flow through unchanged (no local file to audit). + if candidate.is_file() and candidate.suffix.lower() in (".jsonl", ".json"): + _offer_audit_for_jsonl(candidate.resolve()) return raw @@ -497,6 +566,8 @@ def _validate_local_jsonl(raw_path: str): ingested = _offer_ingest_for_directory(resolved) if ingested is None: return None + # ``_offer_ingest_for_directory`` already invokes the Phase 12.5 + # audit hook on the produced JSONL — don't re-prompt here. return ingested if not resolved.is_file(): return _BYOD_LOCAL_NOT_FOUND @@ -509,6 +580,10 @@ def _validate_local_jsonl(raw_path: str): except (OSError, ValueError) as e: print(f" File is not valid JSONL (first line failed to parse): {e}") return None + # Phase 12.5: confirmed local JSONL → offer to audit it before returning + # the path to the caller. Closes the BYOD audit loop documented in + # docs/roadmap/phase-12-5-backlog.md row #4. + _offer_audit_for_jsonl(resolved.resolve()) return str(resolved.resolve()) diff --git a/notebooks/data_curation.ipynb b/notebooks/data_curation.ipynb index 722ce003..adcba7dc 100644 --- a/notebooks/data_curation.ipynb +++ b/notebooks/data_curation.ipynb @@ -126,19 +126,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "## 2. Ingest with the markdown-aware splitter (Phase 12)\n", - "\n", - "`--strategy markdown` is the right choice when your corpus is structured\n", - "with headings: it splits at heading boundaries, keeps fenced code blocks\n", - "atomic, and **inlines the heading breadcrumb** at the top of every chunk\n", - "so SFT loss sees the document context. Falls back to paragraph mode for\n", - "non-markdown inputs (`.txt` here).\n", - "\n", - "We also pass `--secrets-mask` and `--pii-mask` (Phase 12) — secrets run\n", - "**before** PII masking so combined detectors don't double-count\n", - "overlapping spans." - ] + "source": "## 2. Ingest with the markdown-aware splitter (Phase 12)\n\n`--strategy markdown` is the right choice when your corpus is structured\nwith headings: it splits at heading boundaries, keeps fenced code blocks\natomic, and **inlines the heading breadcrumb** at the top of every chunk\nso SFT loss sees the document context. Falls back to paragraph mode for\nnon-markdown inputs (`.txt` here).\n\nWe also pass `--secrets-mask` and `--pii-mask` (Phase 12) — secrets run\n**before** PII masking so combined detectors don't double-count\noverlapping spans.\n\n> **Phase 12.5 shorthand (`v0.5.3`):** `--all-mask` is equivalent to\n> `--secrets-mask --pii-mask`. Use it when you want the\n> \"scrub-everything-detectable\" workflow without remembering both\n> flags. Composes additively with explicit flags (set-union)." }, { "cell_type": "code", @@ -411,22 +399,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "## What's next\n", - "\n", - "- **Train on the curated JSONL**: point any of the training notebooks at\n", - " `./curated/policies.jsonl` via `data.dataset_name_or_path`.\n", - " - [`quickstart_sft.ipynb`](./quickstart_sft.ipynb) — first SFT run\n", - " - [`multi_dataset.ipynb`](./multi_dataset.ipynb) — mix the curated\n", - " set with public datasets\n", - "- **Long-form guides**:\n", - " - [`docs/guides/ingestion.md`](https://github.com/cemililik/ForgeLM/blob/main/docs/guides/ingestion.md)\n", - " — every flag, every chunking strategy, every extra\n", - " - [`docs/guides/data_audit.md`](https://github.com/cemililik/ForgeLM/blob/main/docs/guides/data_audit.md)\n", - " — audit JSON schema, severity tiers, governance integration\n", - "- **Wizard**: `forgelm --wizard` offers to run `forgelm ingest` inline\n", - " when you point it at a directory of raw docs (Phase 11.5+)." - ] + "source": "## What's next\n\n- **Train on the curated JSONL**: point any of the training notebooks at\n `./curated/policies.jsonl` via `data.dataset_name_or_path`.\n - [`quickstart_sft.ipynb`](./quickstart_sft.ipynb) — first SFT run\n - [`multi_dataset.ipynb`](./multi_dataset.ipynb) — mix the curated\n set with public datasets\n- **Long-form guides**:\n - [`docs/guides/ingestion.md`](https://github.com/cemililik/ForgeLM/blob/main/docs/guides/ingestion.md)\n — every flag, every chunking strategy, every extra\n - [`docs/guides/data_audit.md`](https://github.com/cemililik/ForgeLM/blob/main/docs/guides/data_audit.md)\n — audit JSON schema, severity tiers, governance integration\n- **Wizard**: `forgelm --wizard` offers to run `forgelm ingest` inline\n when you point it at a directory of raw docs (Phase 11.5+), and as of\n Phase 12.5 it also offers to run `forgelm audit` automatically once a\n JSONL is selected — closes the BYOD audit loop end-to-end.\n- **Phase 12.5 audit add-ons (v0.5.3)**:\n - `--croissant` emits a [Google Croissant 1.0](http://mlcommons.org/croissant/)\n dataset card under the report's `croissant` key so the same JSON\n file doubles as both the EU AI Act Article 10 governance artifact\n and a Croissant-consumer dataset card.\n - `--pii-ml` layers\n [Presidio](https://github.com/microsoft/presidio) NER on top of\n the regex detector — adds `person` / `organization` /\n `location` categories to `pii_summary` / `pii_severity`. Requires\n `pip install 'forgelm[ingestion-pii-ml]'`." } ], "metadata": { @@ -441,4 +414,4 @@ }, "nbformat": 4, "nbformat_minor": 4 -} +} \ No newline at end of file diff --git a/pyproject.toml b/pyproject.toml index c5f40bfb..ca97fa0b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -112,6 +112,16 @@ ingestion-secrets = [ "detect-secrets>=1.5.0,<2.0.0", ] +# Phase 12.5: optional Presidio ML-NER PII detector. Adds the +# unstructured-identifier categories (person / organization / location) +# that the regex detector inherently misses. Heavyweight install +# (≈ 50 MB spaCy model on first analyzer build), so opt-in only — the +# regex set already covers the GDPR-mandated structured identifiers +# (email, phone, IBAN, credit card, national IDs). +ingestion-pii-ml = [ + "presidio-analyzer>=2.2.0,<3.0.0", +] + # Phase 10: rich terminal rendering for the forgelm chat REPL. # Falls back to plain text output when not installed. chat = [ diff --git a/tests/test_phase12_5.py b/tests/test_phase12_5.py new file mode 100644 index 00000000..2ca7946c --- /dev/null +++ b/tests/test_phase12_5.py @@ -0,0 +1,292 @@ +"""Phase 12.5 — data curation polish backlog. + +Covers the four follow-up items from +:mod:`docs/roadmap/phase-12-5-backlog.md`: + +1. ``forgelm ingest --all-mask`` composite shorthand (this file). +2. Wizard "audit first" entry point (this file). +3. ``forgelm audit --croissant`` Croissant 1.0 metadata (this file). +4. Presidio ML-NER PII adapter via the ``[ingestion-pii-ml]`` extra + (this file). + +Each test class targets exactly one of those four items so a future +re-shuffle can split this file along section boundaries cleanly. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from unittest.mock import patch + +import pytest + +from forgelm.cli import main + +# --------------------------------------------------------------------------- +# Item 3 — `forgelm ingest --all-mask` composite shorthand +# --------------------------------------------------------------------------- + + +class TestIngestAllMask: + """Phase 12.5 #3: ``--all-mask`` is shorthand for ``--secrets-mask --pii-mask``.""" + + def _write_text(self, path: Path, body: str) -> None: + path.write_text(body, encoding="utf-8") + + def test_all_mask_redacts_both_pii_and_secrets(self, tmp_path): + # Inert fragments at runtime so repo-wide secret scanners stay silent + # while the regex still has to match the canonical shape. + aws_key = "AKIA" + "IOSFODNN7" + "EXAMPLE" + email = "alice@example.com" + src = tmp_path / "input.txt" + self._write_text(src, f"contact: {email}\nkey={aws_key}\n") + out = tmp_path / "out.jsonl" + + with patch( + "sys.argv", + ["forgelm", "ingest", str(src), "--output", str(out), "--all-mask"], + ): + with pytest.raises(SystemExit) as exc_info: + main() + assert exc_info.value.code == 0 + + written = out.read_text(encoding="utf-8") + # Both kinds of redaction tokens land in the JSONL — proves the flag + # composes additively into both subsystems, not just one of them. + assert email not in written + assert aws_key not in written + assert "[REDACTED-SECRET]" in written + assert "[REDACTED]" in written + + def test_all_mask_combines_with_individual_flags_no_error(self, tmp_path): + # Set-union semantics: passing ``--all-mask`` alongside ``--pii-mask`` + # must not raise; both stay True. + src = tmp_path / "input.txt" + self._write_text(src, "alice@example.com plain text") + out = tmp_path / "out.jsonl" + + with patch( + "sys.argv", + [ + "forgelm", + "ingest", + str(src), + "--output", + str(out), + "--all-mask", + "--pii-mask", + ], + ): + with pytest.raises(SystemExit) as exc_info: + main() + assert exc_info.value.code == 0 + assert "[REDACTED]" in out.read_text(encoding="utf-8") + + def test_all_mask_does_not_redact_when_off(self, tmp_path): + # Sanity: without the flag, neither masking subsystem fires. + email = "alice@example.com" + src = tmp_path / "input.txt" + self._write_text(src, f"contact: {email}") + out = tmp_path / "out.jsonl" + + with patch( + "sys.argv", + ["forgelm", "ingest", str(src), "--output", str(out)], + ): + with pytest.raises(SystemExit) as exc_info: + main() + assert exc_info.value.code == 0 + assert email in out.read_text(encoding="utf-8") + + +# --------------------------------------------------------------------------- +# Item 4 — Wizard "audit first" entry point +# --------------------------------------------------------------------------- + + +class TestWizardAuditFirstOffer: + """Phase 12.5 #4: when a JSONL is provided to the wizard, offer to audit it. + + Mirrors the Phase 11.5 ``_offer_ingest_for_directory`` pattern: the + wizard surfaces the verdicts before continuing so the user can decide + whether to swap datasets at config-time, not at training-time. + """ + + def _write_jsonl(self, path: Path, rows) -> None: + with open(path, "w", encoding="utf-8") as fh: + for row in rows: + fh.write(json.dumps(row) + "\n") + + def test_audit_offer_runs_audit_when_user_accepts(self, tmp_path, capsys): + from forgelm.wizard import _offer_audit_for_jsonl + + ds = tmp_path / "data.jsonl" + self._write_jsonl(ds, [{"text": "alpha"}, {"text": "beta"}]) + + # Simulate "yes" to the offer. + with patch("forgelm.wizard._prompt_yes_no", return_value=True): + outcome = _offer_audit_for_jsonl(ds) + + assert outcome is True + rendered = capsys.readouterr().out + assert "Audit complete" in rendered or "Audit results" in rendered + + def test_audit_offer_skipped_when_user_declines(self, tmp_path, capsys): + from forgelm.wizard import _offer_audit_for_jsonl + + ds = tmp_path / "data.jsonl" + self._write_jsonl(ds, [{"text": "alpha"}]) + + # Simulate "no" — must short-circuit before invoking audit_dataset. + with patch("forgelm.wizard._prompt_yes_no", return_value=False): + with patch("forgelm.data_audit.audit_dataset") as mock_audit: + outcome = _offer_audit_for_jsonl(ds) + mock_audit.assert_not_called() + + assert outcome is False + + def test_audit_offer_recovers_when_audit_raises(self, tmp_path, capsys): + # The wizard runs in interactive context; an unexpected audit failure + # must NOT crash the wizard. The function returns False so the caller + # falls through to the "continue without audit" path. + from forgelm.wizard import _offer_audit_for_jsonl + + ds = tmp_path / "data.jsonl" + self._write_jsonl(ds, [{"text": "alpha"}]) + + with patch("forgelm.wizard._prompt_yes_no", return_value=True): + with patch( + "forgelm.data_audit.audit_dataset", + side_effect=RuntimeError("simulated audit failure"), + ): + outcome = _offer_audit_for_jsonl(ds) + + assert outcome is False + assert "could not run" in capsys.readouterr().out.lower() + + +# --------------------------------------------------------------------------- +# Item 2 — Croissant 1.0 metadata +# --------------------------------------------------------------------------- + + +class TestCroissantMetadataExport: + """Phase 12.5 #2: ``--croissant`` adds a Google Croissant subset to the report.""" + + def _write_jsonl(self, path: Path, rows) -> None: + with open(path, "w", encoding="utf-8") as fh: + for row in rows: + fh.write(json.dumps(row) + "\n") + + def test_croissant_section_empty_by_default(self, tmp_path): + # Phase 12 precedent (``secrets_summary`` / ``quality_summary``): + # additive keys appear as empty dicts on default audits so consumers + # using ``report.get("...", {})`` keep working byte-equivalently. + # The Croissant block follows the same convention. + ds = tmp_path / "data.jsonl" + self._write_jsonl(ds, [{"text": "alpha"}]) + out_dir = tmp_path / "audit" + + with patch( + "sys.argv", + ["forgelm", "audit", str(ds), "--output", str(out_dir)], + ): + with pytest.raises(SystemExit) as exc_info: + main() + assert exc_info.value.code == 0 + + report = json.loads((out_dir / "data_audit_report.json").read_text(encoding="utf-8")) + # Key may or may not be present depending on serializer; the contract + # is "no Croissant content unless --croissant is passed". + assert report.get("croissant", {}) == {} + + def test_croissant_flag_emits_minimal_card(self, tmp_path): + ds = tmp_path / "data.jsonl" + self._write_jsonl(ds, [{"text": "alpha"}, {"text": "beta"}]) + out_dir = tmp_path / "audit" + + with patch( + "sys.argv", + [ + "forgelm", + "audit", + str(ds), + "--output", + str(out_dir), + "--croissant", + ], + ): + with pytest.raises(SystemExit) as exc_info: + main() + assert exc_info.value.code == 0 + + report = json.loads((out_dir / "data_audit_report.json").read_text(encoding="utf-8")) + assert "croissant" in report + crois = report["croissant"] + # Pin the canonical Croissant 1.0 envelope so consumers / CI checks can + # rely on it. ``@type`` and ``@context`` are the minimum-viable subset + # that lets a Croissant-aware consumer parse the file without + # extending the audit JSON's other top-level keys. + assert crois.get("@type") == "sc:Dataset" + assert "@context" in crois + assert crois["@context"].get("sc") == "https://schema.org/" + # ForgeLM-specific fields the report already has should be carried + # across into the card so it stays a single self-contained document. + assert "name" in crois + assert "distribution" in crois # describes the underlying JSONL + + +# --------------------------------------------------------------------------- +# Item 1 — Presidio adapter (``[ingestion-pii-ml]`` extra) +# --------------------------------------------------------------------------- + + +class TestPresidioPIIAdapter: + """Phase 12.5 #1: optional ML-NER PII signal layered onto the regex path.""" + + def _write_jsonl(self, path: Path, rows) -> None: + with open(path, "w", encoding="utf-8") as fh: + for row in rows: + fh.write(json.dumps(row) + "\n") + + def test_presidio_disabled_by_default(self, tmp_path): + # No --pii-ml flag → no Presidio counts; the regex path is unchanged. + from forgelm.data_audit import audit_dataset + + ds = tmp_path / "data.jsonl" + self._write_jsonl(ds, [{"text": "Visit Acme Corp in Berlin next Monday."}]) + + report = audit_dataset(str(ds), output_dir=str(tmp_path / "audit")) + # No new ML-tier categories — the regex-only path didn't see any PII + # in this prose either, but we're really asserting the schema doesn't + # add the Presidio-only buckets when the feature is off. + for ml_only in ("person", "organization", "location"): + assert ml_only not in report.pii_summary + + def test_presidio_missing_extra_raises_with_install_hint(self): + from forgelm.data_audit import _require_presidio + + # When the extra isn't installed, the require-helper raises a typed + # ImportError pointing at the install command — same pattern as + # ``_require_datasketch`` / ``_require_detect_secrets``. + with patch("forgelm.data_audit._HAS_PRESIDIO", False): + with pytest.raises(ImportError) as exc_info: + _require_presidio() + msg = str(exc_info.value) + assert "ingestion-pii-ml" in msg + assert "pip install" in msg + + def test_presidio_severity_table_exposes_ml_tiers(self): + # The severity table grows three new rows so the existing pii_severity + # block can grade ML-NER findings the same way it grades regex hits. + from forgelm.data_audit import PII_ML_SEVERITY + + for kind in ("person", "organization", "location"): + assert kind in PII_ML_SEVERITY, f"missing severity tier for ML kind {kind!r}" + # All three categories are below the regex 'critical' tier — names / + # orgs / locations leak less identity than a credit-card number. + # Pin medium / low to keep audit verdicts stable across releases. + assert PII_ML_SEVERITY["person"] == "medium" + assert PII_ML_SEVERITY["organization"] == "low" + assert PII_ML_SEVERITY["location"] == "low" diff --git a/tests/test_wizard_byod.py b/tests/test_wizard_byod.py index 3e11c00f..c77607ae 100644 --- a/tests/test_wizard_byod.py +++ b/tests/test_wizard_byod.py @@ -85,7 +85,11 @@ def test_byod_rejects_malformed_jsonl(tmp_path, capsys): def test_byod_accepts_valid_jsonl(tmp_path): good = tmp_path / "good.jsonl" good.write_text('{"messages":[]}\n', encoding="utf-8") - answers = _PRELUDE + [str(good)] + # Phase 12.5: after a JSONL is validated, the wizard offers to run an + # audit on it. Decline ("n") so this test remains scoped to the BYOD + # validation path; the audit-offer behaviour has its own coverage in + # test_phase12_5.py::TestWizardAuditFirstOffer. + answers = _PRELUDE + [str(good), "n"] fake_result = type( "R", @@ -149,7 +153,8 @@ def test_byod_expands_user_home(tmp_path, monkeypatch): # On POSIX Path.expanduser() reads $HOME; setting it is enough. typed = "~/data.jsonl" - answers = _PRELUDE + [typed] + # Phase 12.5: extra "n" declines the post-validation audit offer. + answers = _PRELUDE + [typed, "n"] fake_result = type( "R", @@ -197,7 +202,8 @@ def test_byod_relative_local_path_not_misclassified_as_hub_id(tmp_path, monkeypa monkeypatch.chdir(tmp_path) # so "data/train.jsonl" resolves under tmp_path typed = "data/train.jsonl" - answers = _PRELUDE + [typed] + # Phase 12.5: decline the post-validation audit offer. + answers = _PRELUDE + [typed, "n"] fake_result = type( "R", From 734e4e781c5917269084b38c60a2cf84fcbc4bdc Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 29 Apr 2026 18:15:56 +0300 Subject: [PATCH 02/12] =?UTF-8?q?fix(phase-12.5):=20apply=20review=20findi?= =?UTF-8?q?ngs=20=E2=80=94=20silent-failure=20guards,=20Presidio=20canonic?= =?UTF-8?q?als,=20Croissant=20cleanup?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reviewer-driven follow-up to commit cad587d. Addresses every MAJOR finding plus the actionable MINOR / doc-drift items. 7 new tests; 840 total pass (was 833). MAJOR - M1 — _require_presidio now pre-flights the analyzer build and re-raises spaCy's "Can't find model" OSError as ImportError with the two-line install recipe, so a missing en_core_web_lg surfaces before any rows are scanned. Previously the OSError was swallowed by detect_pii_ml's per-row bare-Exception, leaving the audit green with zero ML PII coverage and no diagnostic — direct violation of CLAUDE.md principle #6 (no silent failures). detect_pii_ml's catch is now narrowed to (ValueError, RuntimeError) so a model-load failure that slips through (e.g. lazy load triggered later) cannot be silenced; the install-hint string is consolidated into one module-level constant so the two failure surfaces stay in sync. - M2 — _PRESIDIO_ENTITY_MAP trimmed from 7 keys to the 3 Presidio actually emits canonically (PERSON / ORGANIZATION / LOCATION). ORG / GPE / NORP / FAC were dead — Presidio's model_to_presidio_entity_mapping translates raw spaCy labels to canonical Presidio names before they reach AnalyzerResult, so raw-label keys never matched. NORP → location was also semantically wrong (nationality / religion / political group is a distinct privacy signal). The new comment explains the canonicalisation layer so a future maintainer doesn't reintroduce raw spaCy keys. - M3 — Croissant version field dropped. Croissant's `version` is the dataset version (sc:version), not the vocabulary version; vocab conformance is declared via `conformsTo` (already correct). The audit doesn't have first-class evidence for dataset version, so the field is omitted entirely; operators who publish the card hand-edit `version` like they do `license` / `citeAs`. The misnamed _CROISSANT_VOCAB_VERSION constant is removed. - M4 — `forgelm audit --output-format json` envelope now includes the `croissant` key (empty when --croissant is off, populated when on) so a CI consumer reading stdout doesn't have to re-slurp the on-disk file. Same precedent as secrets_summary / quality_summary. MINOR - m2 — New `--pii-ml-language LANG` flag (default "en") plumbed through audit_dataset → _process_split → _audit_split → _StreamingAggregator → detect_pii_ml. Lets a Turkish-majority corpus run with the matching spaCy model instead of silently scoring zero NER findings under English. Presidio raises a typed exception when no engine is registered for the requested language. - m3 — Croissant card's `url` field now carries `source_input` (the as-typed string) instead of the resolved absolute filesystem path, and per-split `contentUrl` is the relative file_id. Cards published to HuggingFace / MLCommons no longer leak the auditor's /Users/... or /home/builder/... layout. - m5 — Wizard's audit-offer copy now hedges against multi-million- row JSONL: under a 100 MB threshold it keeps the friendly "~ seconds" tone, above it surfaces the file size and an "8-15 MB/s of JSONL on a single CPU" expectation so the operator can decide before saying yes. - m7 — _offer_audit_for_jsonl docstring now explicitly cites docs/standards/error-handling.md for the bare-Exception ladder, making the durability of the swallow durable when the file is re-read in 6 months. TESTS (+7, 840 total) - test_croissant_section_empty_by_default now pins "key present, value empty" instead of the weaker "absent or empty" form. - New: test_presidio_entity_map_only_canonical_keys — schema pin on the trimmed map. - New: test_presidio_missing_spacy_model_surfaces_install_hint — exercises the M1 pre-flight by patching _PresidioAnalyzer to raise OSError; asserts ImportError carries the install recipe. - New: test_presidio_entity_map_collapses_findings_to_buckets — behavioural test that would have caught the original M2 dead-key bug. Stubs the analyzer to emit canonical + raw spaCy labels and asserts only canonicals are honoured. - New: TestCroissantMultiSplit — multi-split corpus produces one cr:FileObject + cr:RecordSet per split with relative contentUrl (anti-regression for m3). - New: TestAllMaskSymmetric — symmetric set-union case (`--secrets-mask --all-mask`) so a future refactor that drops one branch trips a test. - New: TestPiiMlJsonEnvelope — confirms M4: croissant present in stdout envelope when on, empty when off. DOC DRIFT (5 places) - README.md install matrix and Phase 12.5 feature line: spell out the spaCy model download as a separate step. - CHANGELOG.md Phase 12.5 entry: recipe-first install instructions for --pii-ml, explicit OSError → ImportError contract, --pii-ml-language flag mention, Croissant url/contentUrl path leak fix, version field omission rationale. - docs/guides/data_audit.md and docs/guides/data_audit-tr.md: replaced "Presidio's first analyzer build downloads a spaCy model (~ 50 MB)" — wrong on every supported Presidio version — with a two-step install block + non-English language hint. - notebooks/data_curation.ipynb "What's next" cell: same fix. Verification: ruff clean (78 files), 840 tests pass (13 skipped), 7/7 bilingual H2 parity, --pii-ml / --pii-ml-language / --croissant all surface in `forgelm audit --help`. Co-Authored-By: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 52 ++++++-- README.md | 4 +- docs/guides/data_audit-tr.md | 30 ++++- docs/guides/data_audit.md | 28 +++- forgelm/cli.py | 29 +++- forgelm/data_audit.py | 164 ++++++++++++++++------- forgelm/wizard.py | 57 +++++++- notebooks/data_curation.ipynb | 2 +- tests/test_phase12_5.py | 242 +++++++++++++++++++++++++++++++++- 9 files changed, 528 insertions(+), 80 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4b7f34e7..b4f95c50 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,20 +26,44 @@ additive surface on top of the Phase 12 ingestion + audit lineage. (same precedent as `secrets_summary` / `quality_summary`). Lets the same JSON file double as both the EU AI Act Article 10 governance artifact and a Croissant-consumer dataset card. -- **`forgelm audit --pii-ml`** + new `[ingestion-pii-ml]` extra - (item #1) — opt-in [Presidio](https://github.com/microsoft/presidio) - ML-NER PII detector layered on top of the existing regex detector. - Adds the unstructured-identifier categories the regex inherently - misses (`person`, `organization`, `location`) into the same - `pii_summary` / `pii_severity` blocks under disjoint category - names. Severity tiers in the new `PII_ML_SEVERITY` table: - `person → medium`, `organization → low`, `location → low` (deliberately - below the regex `critical`/`high` tiers because NER false-positive - rates are materially higher than regex-anchored detection). - Pre-flight check raises `ImportError("...pip install - 'forgelm[ingestion-pii-ml]'")` when the extra is missing — failures - surface before any rows are scanned. Per-row Presidio failures are - swallowed so a single malformed row never blocks the audit. + - `url` and `contentUrl` use the as-typed input string and the + relative split filename, never the resolved absolute filesystem + path, so cards published to HuggingFace / MLCommons don't leak + the auditor's local layout. + - Croissant `version` (`sc:version`, dataset version) is omitted + deliberately — the audit doesn't have first-class evidence for + it; vocab conformance is declared via `conformsTo`. Operators + that publish hand-edit `version` like they do `license` / + `citeAs`. + - The card is now also surfaced in the `--output-format json` + stdout envelope alongside the on-disk report so CI consumers + don't need a second file slurp. +- **`forgelm audit --pii-ml [--pii-ml-language LANG]`** + new + `[ingestion-pii-ml]` extra (item #1) — opt-in + [Presidio](https://github.com/microsoft/presidio) ML-NER PII detector + layered on top of the existing regex detector. Adds the + unstructured-identifier categories the regex inherently misses + (`person`, `organization`, `location`) into the same `pii_summary` / + `pii_severity` blocks under disjoint category names. Severity tiers + in the new `PII_ML_SEVERITY` table: `person → medium`, + `organization → low`, `location → low` (deliberately below the regex + `critical`/`high` tiers because NER false-positive rates are + materially higher than regex-anchored detection). The pre-flight + check covers BOTH the missing-extra branch AND the missing-spaCy-model + branch — `presidio-analyzer` does *not* transitively ship a spaCy + NER model, so the install recipe is now two lines: + ``` + pip install 'forgelm[ingestion-pii-ml]' + python -m spacy download en_core_web_lg + ``` + Without either, `forgelm audit --pii-ml` raises `ImportError` with + the recipe before any rows are scanned. Per-row Presidio failures + are scoped to `(ValueError, RuntimeError)` so a single malformed row + never blocks the audit, but a deep `OSError` from a missing model + surfaces loudly instead of silently scoring zero ML coverage. + `--pii-ml-language` (default `"en"`) lets non-English corpora point + at the matching spaCy model; Presidio raises a typed exception when + no engine is registered for the requested language. - **Wizard "audit first" entry point** (item #4) — when the wizard resolves a JSONL (either typed directly or produced by the Phase 11.5 `_offer_ingest_for_directory` ingest flow), it now offers diff --git a/README.md b/README.md index d90a461f..320eb915 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ - **PDF Page Header/Footer Dedup (v0.5.1)**: Lines that recur on ≥ 70 % of PDF pages (watermarks, page numbers, copyright lines) are stripped automatically — the audit's near-duplicate counts stop misfiring on long policy / book PDFs. - **Dataset Audit**: `forgelm audit data/sft.jsonl --output ./audit/` — produces `data_audit_report.json` with sample count, length distribution, top-3 language detection, LSH-banded near-duplicate rate (`O(n × k)` typical case, exact recall at the default Hamming threshold; v0.5.2 added optional **MinHash LSH** via `--dedup-method minhash` for >50K-row corpora), cross-split leakage check, null/empty rate, and PII flag counts with **severity tiers** (critical / high / medium / low + worst-tier verdict). v0.5.2 also added an always-on **secrets/credential scan** covering the nine families enumerated in `forgelm.data_audit.SECRET_TYPES` — AWS access keys, GitHub tokens, Slack tokens, OpenAI API keys, Google API keys, JWTs, full OpenSSH private-key blocks, full PGP private-key blocks, and Azure storage connection strings — and an opt-in **heuristic quality filter** (`--quality-filter`). CPU-only; streaming JSONL reader keeps memory bounded on multi-million-row splits; feeds EU AI Act Article 10 governance artifact automatically when present at training time. Legacy `--data-audit` flag still works as a deprecation alias. - **Secrets-Aware Ingest (v0.5.2)**: `forgelm ingest … --secrets-mask` scrubs credentials before chunks land in the JSONL — fine-tuning on text containing real API keys memorises them at training time. Pairs with `--pii-mask`; secrets run first so combined detectors don't double-count overlapping spans. -- **Phase 12.5 polish (v0.5.3)**: `forgelm ingest --all-mask` shorthand for combined PII + secrets scrubbing; `forgelm audit --croissant` emits a [Google Croissant 1.0](http://mlcommons.org/croissant/) dataset card alongside the existing report; `forgelm audit --pii-ml` layers [Presidio](https://github.com/microsoft/presidio) NER on top of the regex detector via the optional `[ingestion-pii-ml]` extra (`person` / `organization` / `location` categories merged into the same `pii_severity` block); the wizard now offers to run `forgelm audit` automatically after a JSONL is selected (BYOD onboarding closes the audit loop). +- **Phase 12.5 polish (v0.5.3)**: `forgelm ingest --all-mask` shorthand for combined PII + secrets scrubbing; `forgelm audit --croissant` emits a [Google Croissant 1.0](http://mlcommons.org/croissant/) dataset card alongside the existing report; `forgelm audit --pii-ml [--pii-ml-language LANG]` layers [Presidio](https://github.com/microsoft/presidio) NER on top of the regex detector via the optional `[ingestion-pii-ml]` extra **plus a separate `python -m spacy download en_core_web_lg`** step (or the matching model for the chosen language); the wizard now offers to run `forgelm audit` automatically after a JSONL is selected (BYOD onboarding closes the audit loop). ### Quickstart Layer (v0.4.5) - **One-Command Templates**: `forgelm quickstart customer-support` — 4 bundled templates (SFT customer-support, code-assistant, medical-qa-tr, GRPO math-reasoning) plus a bring-your-own-data domain-expert scaffold. Auto-downsizes models on small GPUs. @@ -173,7 +173,7 @@ pip install "forgelm[merging]" # mergekit model merging pip install "forgelm[ingestion]" # PDF/DOCX/EPUB/Markdown → JSONL + langdetect + xxhash pip install "forgelm[ingestion-scale]" # MinHash LSH dedup (datasketch) for >50K-row corpora pip install "forgelm[ingestion-secrets]" # detect-secrets scanner for SFT corpora (falls back to regex if absent) -pip install "forgelm[ingestion-pii-ml]" # Presidio ML-NER for person/organization/location PII (Phase 12.5) +pip install "forgelm[ingestion-pii-ml]" # Presidio ML-NER for person/organization/location PII (Phase 12.5; ALSO needs `python -m spacy download en_core_web_lg`) pip install "forgelm[export]" # GGUF export via llama-cpp-python pip install "forgelm[chat]" # Rich terminal rendering for `forgelm chat` ``` diff --git a/docs/guides/data_audit-tr.md b/docs/guides/data_audit-tr.md index 980370b1..54c9e5d3 100644 --- a/docs/guides/data_audit-tr.md +++ b/docs/guides/data_audit-tr.md @@ -367,10 +367,32 @@ yüksektir — bu yüzden ML katmanları regex'in `critical` / `high` katlarının altında bilinçli konumlandırılır. Bir ML bulgusunu sert bir gate olarak değerlendirmeden önce satır bazında span'leri inceleyin. -Presidio'nun ilk analyzer build'i bir spaCy İngilizce NER modeli -indirir (~ 50 MB, tek seferlik). Sonraki çalışmalar cached modeli -yeniden kullanır. Adaptör yalnızca opt-in; `--pii-ml` olmadan audit -zero-extra-deps regex yolunda kalır. +**İki adımlı kurulum.** `presidio-analyzer` paketi spaCy NER modelini +transitively içermez — tek seferlik ayrı bir indirme: + +```bash +pip install 'forgelm[ingestion-pii-ml]' +python -m spacy download en_core_web_lg # ~ 50 MB; gerekli +``` + +Model yoksa `forgelm audit --pii-ml` hiçbir satır taranmadan **önce** +aynı install hint ile `ImportError` raise eder +(`forgelm.data_audit._require_presidio`'daki pre-flight, spaCy'nin +`OSError`'unu yakalayıp typed install error'a dönüştürür). Bu fail-loud +davranış bilinçli — önceki prototiplerde model eksikken sıfır ML +bulgusu sessizce raporlanıyordu, opt-in bir detector için kritik bir +compliance kör noktası. + +İngilizce olmayan korpuslar için eşleşen spaCy modelini kurup dil +kodunu geçirin: + +```bash +python -m spacy download xx_ent_wiki_sm # çok dilli, daha küçük +forgelm audit data/ --pii-ml --pii-ml-language xx +``` + +Adaptör yalnızca opt-in; `--pii-ml` olmadan audit zero-extra-deps +regex yolunda kalır. --- diff --git a/docs/guides/data_audit.md b/docs/guides/data_audit.md index 0052b1b6..b02f9176 100644 --- a/docs/guides/data_audit.md +++ b/docs/guides/data_audit.md @@ -364,9 +364,31 @@ detection, so the ML tiers sit deliberately below the regex `critical`/`high` floors — review the per-row spans before treating an ML finding as a hard gate. -Presidio's first analyzer build downloads a spaCy English NER model -(~ 50 MB, one-time). Subsequent runs reuse the cached model. The -adapter is opt-in only; without `--pii-ml` the audit stays on the +**Two-step install.** `presidio-analyzer` does **not** transitively +ship a spaCy NER model — that's a separate one-time download: + +```bash +pip install 'forgelm[ingestion-pii-ml]' +python -m spacy download en_core_web_lg # ~ 50 MB; required +``` + +Without the model, `forgelm audit --pii-ml` raises an `ImportError` +with the same install hint **before** any rows are scanned (the +pre-flight check in `forgelm.data_audit._require_presidio` catches +spaCy's `OSError` and re-raises it as the typed install error). This +fail-loud behaviour exists because earlier prototypes silently scored +zero ML findings on a missing model — a critical compliance blind +spot for an opt-in detector that operators expect to actually run. + +For non-English corpora, install the matching spaCy model and pass +the language code: + +```bash +python -m spacy download xx_ent_wiki_sm # multilingual, smaller +forgelm audit data/ --pii-ml --pii-ml-language xx +``` + +The adapter is opt-in only; without `--pii-ml` the audit stays on the zero-extra-deps regex path. --- diff --git a/forgelm/cli.py b/forgelm/cli.py index 07d8413a..7ceeba7a 100644 --- a/forgelm/cli.py +++ b/forgelm/cli.py @@ -491,8 +491,23 @@ def _add_audit_subcommand(subparsers) -> None: help=( "Phase 12.5 opt-in: layer Presidio's ML-NER PII detection (person / organization / " "location categories) on top of the regex detector. Requires the optional " - "'forgelm[ingestion-pii-ml]' extra; raises an install-hint ImportError when missing. " - "Findings merge into pii_summary / pii_severity under disjoint category names." + "'forgelm[ingestion-pii-ml]' extra AND a spaCy NER model " + "(`python -m spacy download en_core_web_lg`); raises an install-hint ImportError " + "when either is missing. Findings merge into pii_summary / pii_severity under " + "disjoint category names." + ), + ) + p.add_argument( + "--pii-ml-language", + type=str, + default="en", + metavar="LANG", + help=( + "Phase 12.5: language code passed to Presidio's NLP engine when --pii-ml is on " + "(default: 'en'). Presidio raises a typed exception if no engine is registered for " + "the requested language — surface it to the operator instead of silently " + "running an English NER on a non-English corpus. Set to e.g. 'tr' on a Turkish " + "corpus AND make sure the matching spaCy model is installed." ), ) _add_common_subparser_flags(p, include_output_format=True) @@ -1387,6 +1402,7 @@ def _run_data_audit( minhash_jaccard: Optional[float] = None, enable_quality_filter: bool = False, enable_pii_ml: bool = False, + pii_ml_language: str = "en", emit_croissant: bool = False, invoked_via_legacy_flag: bool = False, ) -> None: @@ -1423,6 +1439,7 @@ def _run_data_audit( minhash_jaccard=jaccard, enable_quality_filter=enable_quality_filter, enable_pii_ml=enable_pii_ml, + pii_ml_language=pii_ml_language, emit_croissant=emit_croissant, ) except OSError as exc: @@ -1469,6 +1486,13 @@ def _run_data_audit( "near_duplicate_pairs_per_split": report.near_duplicate_summary.get("pairs_per_split", {}), "near_duplicate_summary": report.near_duplicate_summary, "cross_split_leakage_pairs": list((report.cross_split_overlap.get("pairs") or {}).keys()), + # Phase 12.5: Croissant 1.0 dataset card. Empty dict when the + # ``--croissant`` flag was not passed — same additive shape as + # ``secrets_summary`` / ``quality_summary``. Surfacing it here + # mirrors the on-disk report so a CI step that reads stdout + # via ``--output-format json`` does not need to slurp the + # file separately. + "croissant": report.croissant, "notes": report.notes, } print(json.dumps(summary, indent=2, ensure_ascii=False)) @@ -1496,6 +1520,7 @@ def _run_audit_cmd(args, output_format: str) -> None: minhash_jaccard=getattr(args, "jaccard_threshold", None), enable_quality_filter=getattr(args, "quality_filter", False), enable_pii_ml=getattr(args, "pii_ml", False), + pii_ml_language=getattr(args, "pii_ml_language", "en"), emit_croissant=getattr(args, "croissant", False), invoked_via_legacy_flag=False, ) diff --git a/forgelm/data_audit.py b/forgelm/data_audit.py index 0e3cb4da..8aa7f00a 100644 --- a/forgelm/data_audit.py +++ b/forgelm/data_audit.py @@ -406,48 +406,78 @@ def _get_presidio_analyzer() -> Any: """Return a cached :class:`presidio_analyzer.AnalyzerEngine` instance. Raises: - ImportError: when the ``[ingestion-pii-ml]`` extra is missing. - Also surfaced through the public :func:`_require_presidio` - helper so callers that gate on Presidio availability up - front can produce a clean message before processing starts. + ImportError: when the ``[ingestion-pii-ml]`` extra is missing OR + when ``presidio-analyzer`` is installed but the spaCy NER + model it depends on is not — both surfaces produce the same + actionable install hint via :func:`_require_presidio` so + the operator never sees a deep ``OSError`` from spaCy + mid-stream. """ - _require_presidio() + if not _HAS_PRESIDIO: + raise ImportError(_PRESIDIO_INSTALL_HINT) return _PresidioAnalyzer() -def _require_presidio() -> None: - """Raise a clear ImportError when the optional ML-NER backend is missing. +# A single, canonical install hint shared by the import-sentinel branch +# and the spaCy-model-missing branch. Both failure modes look identical +# to the operator (``--pii-ml`` doesn't work; here is the recipe), so +# we keep the message in one place to avoid drift. +_PRESIDIO_INSTALL_HINT: str = ( + "Presidio PII ML detection requires the 'ingestion-pii-ml' extra " + "AND a spaCy English NER model. Install with:\n" + " pip install 'forgelm[ingestion-pii-ml]'\n" + " python -m spacy download en_core_web_lg" +) - Mirrors :func:`_require_datasketch` so any audit code path that - needs Presidio fails with an actionable install hint instead of a - deep ``AttributeError`` the operator can't act on. + +def _require_presidio() -> None: + """Raise a clear ImportError when the optional ML-NER backend is unusable. + + Mirrors :func:`_require_datasketch` but extends the check to model + availability: a fresh ``[ingestion-pii-ml]`` install does **not** + transitively ship the spaCy NER model — that's a separate + ``python -m spacy download en_core_web_lg`` step. Without the model + the first per-row analyzer call would raise ``OSError`` deep inside + spaCy and (because :func:`detect_pii_ml`'s last-ditch ``except`` + swallows per-row failures) the audit would silently report zero + ML PII coverage. Pre-flighting the analyzer build here surfaces + the missing-model case before any rows are scanned. """ if not _HAS_PRESIDIO: + raise ImportError(_PRESIDIO_INSTALL_HINT) + try: + _get_presidio_analyzer() + except OSError as exc: + # spaCy raises ``OSError`` ("Can't find model 'en_core_web_lg'") + # when the model package isn't on the import path. Re-raise as + # ImportError so callers that already catch ImportError for the + # extras-missing branch see the same exception class for both + # failure modes. + _get_presidio_analyzer.cache_clear() raise ImportError( - "Presidio PII ML detection requires the 'ingestion-pii-ml' extra. " - "Install with: pip install 'forgelm[ingestion-pii-ml]'" - ) - - -# Map of Presidio's canonical entity-type strings (which mirror the -# spaCy NER label set) to the ForgeLM category names that feed -# :data:`PII_ML_SEVERITY` / :data:`PII_ML_TYPES`. Keys come from -# Presidio's recognizer registry; multiple Presidio labels can collapse -# into a single ForgeLM bucket (``GPE`` / ``LOCATION`` both → ``location``) -# because the audit's downstream consumer (compliance reviewer) doesn't -# need finer granularity than person / organization / location. + f"Presidio analyzer build failed (likely missing spaCy NER model): {exc}\n{_PRESIDIO_INSTALL_HINT}" + ) from exc + + +# Presidio canonicalises spaCy NER labels through its +# ``model_to_presidio_entity_mapping`` (PER/PERSON → PERSON, +# LOC/GPE → LOCATION, ORG → ORGANIZATION, NORP → NRP); what +# ``analyzer.analyze()`` emits as ``entity_type`` is the canonical +# Presidio name, never the raw spaCy tag. Keep this map keyed on +# canonical Presidio names only so a future maintainer doesn't read +# dead spaCy keys as live coverage. ``NRP`` (nationality / religious / +# political group) is deliberately not mapped — it's a distinct privacy +# signal from ``location`` and Presidio's NRP precision is too low to +# grade as ``person`` without further work; revisit if compliance +# reviewers ask for it. _PRESIDIO_ENTITY_MAP: Dict[str, str] = { "PERSON": "person", - "ORG": "organization", "ORGANIZATION": "organization", "LOCATION": "location", - "GPE": "location", - "NORP": "location", - "FAC": "location", } -def detect_pii_ml(text: Any) -> Dict[str, int]: +def detect_pii_ml(text: Any, *, language: str = "en") -> Dict[str, int]: """ML-NER PII detector — counts ``person`` / ``organization`` / ``location`` spans. Layered ON TOP of :func:`detect_pii` (regex). The two detectors @@ -456,15 +486,27 @@ def detect_pii_ml(text: Any) -> Dict[str, int]: unstructured-identifier signal (ML) without double-counting the same span. + Args: + text: Per-row payload; defensive against ``None`` / numbers / dicts. + language: Presidio NLP-engine language code. Default ``"en"``. + Set via :func:`audit_dataset`'s ``pii_ml_language`` parameter. + Presidio raises a typed exception if no NLP engine is + registered for the requested language — surfaced as the + same per-row swallow as any other analyzer failure. + Returns an empty dict when: * ``text`` is not a non-empty string (defensive — callers pass JSONL payloads that may be ``None`` / numbers / dicts); * Presidio is not installed (the caller must opt in explicitly via :func:`_require_presidio` if they want a hard failure); - * the analyzer raises on the input — Presidio can fail on - pathological strings, model-load races, or version skews. We - swallow per-row failures and return zero counts so a single bad - row never blocks the audit. + * the analyzer raises a recoverable error on this row — pathological + strings, transient NLP engine state, language-specific recogniser + misses. We swallow narrowly-typed failures (``ValueError`` / + ``RuntimeError``) and let everything else propagate so genuine + bugs (``OSError`` on missing model, ``MemoryError``, ``KeyboardInterrupt``) + stay visible. The pre-flight in :func:`_require_presidio` covers + the missing-model case so it never reaches this path in the + common run. """ if not isinstance(text, str) or not text.strip(): return {} @@ -472,11 +514,15 @@ def detect_pii_ml(text: Any) -> Dict[str, int]: return {} try: analyzer = _get_presidio_analyzer() - results = analyzer.analyze(text=text, language="en") - except Exception: # pragma: no cover — Presidio raises on edge cases - # Per-row resilience: a single malformed row must not abort the - # audit. The aggregator counts no ML PII for this row, which is - # the correct behaviour for opt-in best-effort detection. + results = analyzer.analyze(text=text, language=language) + except (ValueError, RuntimeError): # pragma: no cover — Presidio edge cases + # Per-row resilience for the narrow class of failures Presidio + # raises on bad input or transient engine state. ``OSError`` is + # deliberately NOT caught here — that's the missing-spaCy-model + # signal and ``_require_presidio``'s pre-flight should have + # converted it to ImportError before any row was scanned. If + # one slips through (e.g. lazy model load triggered later), + # surfacing it loudly is the correct behaviour. return {} counts: Dict[str, int] = {} for finding in results: @@ -1514,6 +1560,7 @@ class _StreamingAggregator: minhash_num_perm: int = DEFAULT_MINHASH_NUM_PERM # Phase 12.5: opt-in ML-NER PII detection (Presidio). Off by default. enable_pii_ml: bool = False + pii_ml_language: str = "en" enable_quality_filter: bool = False @@ -1558,7 +1605,10 @@ def _record_text_metrics(agg: _StreamingAggregator, payload: str) -> None: # bucket. The category names are disjoint from the regex set # (person / organization / location vs. email / phone / *_id), # so the merged counts present both views without double-counting. - for kind, count in detect_pii_ml(payload).items(): + # ``pii_ml_language`` is plumbed through so a Turkish-majority + # corpus can be audited with a Turkish spaCy model rather than + # silently scoring zero NER findings under English. + for kind, count in detect_pii_ml(payload, language=agg.pii_ml_language).items(): agg.pii_counts[kind] = agg.pii_counts.get(kind, 0) + count for kind, count in detect_secrets(payload).items(): agg.secrets_counts[kind] = agg.secrets_counts.get(kind, 0) + count @@ -1730,6 +1780,7 @@ def _audit_split( minhash_num_perm: int = DEFAULT_MINHASH_NUM_PERM, enable_quality_filter: bool = False, enable_pii_ml: bool = False, + pii_ml_language: str = "en", ) -> Tuple[Dict[str, Any], List[Any], Dict[str, int], int, int]: """Stream a JSONL split into a metrics record. @@ -1768,6 +1819,7 @@ def _audit_split( minhash_num_perm=minhash_num_perm, enable_quality_filter=enable_quality_filter, enable_pii_ml=enable_pii_ml, + pii_ml_language=pii_ml_language, ) for row, parse_err, decode_err in _read_jsonl_split(path): _ingest_row(agg, row, parse_err, decode_err) @@ -2086,6 +2138,7 @@ def _process_split( minhash_num_perm: int = DEFAULT_MINHASH_NUM_PERM, enable_quality_filter: bool = False, enable_pii_ml: bool = False, + pii_ml_language: str = "en", ) -> _SplitOutcome: """Stream + audit one split. Tolerates per-split filesystem failures. @@ -2106,6 +2159,7 @@ def _process_split( minhash_num_perm=minhash_num_perm, enable_quality_filter=enable_quality_filter, enable_pii_ml=enable_pii_ml, + pii_ml_language=pii_ml_language, ) except OSError as exc: logger.warning("Could not read split '%s' (%s): %s — skipping.", split_name, path, exc) @@ -2349,7 +2403,6 @@ def _build_quality_summary( # parse the block without modification, while the rest of the audit JSON # stays untouched. The card lives under the new ``croissant`` key on the # ``AuditReport`` dataclass; callers that don't want it never see it. -_CROISSANT_VOCAB_VERSION = "1.0" _CROISSANT_CONTEXT: Dict[str, Any] = { "@language": "en", "@vocab": "https://schema.org/", @@ -2418,15 +2471,21 @@ def _build_croissant_metadata( distribution: List[Dict[str, Any]] = [] record_sets: List[Dict[str, Any]] = [] for split_name, info in splits_info.items(): - split_path = splits_paths.get(split_name) sample_count = int(info.get("sample_count", 0)) file_id = f"{split_name}.jsonl" + # ``contentUrl`` deliberately uses the *file_id* (relative + # filename), not the absolute filesystem path: cards published + # to HuggingFace / MLCommons must not leak the auditor's local + # ``/Users/...`` / ``/home/builder/...`` layout. Operators that + # want a real ``contentUrl`` (HF Hub URL, S3 path, etc.) can + # hand-edit the field at publish time the same way they do + # ``license`` / ``citeAs``. distribution.append( { "@type": "cr:FileObject", "@id": file_id, "name": file_id, - "contentUrl": str(split_path) if split_path is not None else file_id, + "contentUrl": file_id, "encodingFormat": "application/jsonlines", "description": f"Split {split_name!r}: {sample_count} sample(s).", } @@ -2462,6 +2521,14 @@ def _build_croissant_metadata( } ) + # ``url`` carries ``source_input`` (as-typed) rather than the + # resolved absolute path — see ``contentUrl`` rationale above. + # ``version`` (``sc:version``) describes the *dataset* version, + # not the Croissant vocabulary version (vocab conformance is + # declared via ``conformsTo``). The audit doesn't have first-class + # evidence for dataset version, so the field is omitted; operators + # that publish the card hand-edit ``version`` like they do + # ``license`` / ``citeAs``. return { "@context": dict(_CROISSANT_CONTEXT), "@type": "sc:Dataset", @@ -2473,8 +2540,7 @@ def _build_croissant_metadata( "Inline counts/quality/PII summaries live in the parent " "data_audit_report.json under the canonical audit keys." ), - "url": str(source_path), - "version": _CROISSANT_VOCAB_VERSION, + "url": source_input, "datePublished": generated_at, "distribution": distribution, "recordSet": record_sets, @@ -2512,6 +2578,7 @@ def audit_dataset( minhash_num_perm: int = DEFAULT_MINHASH_NUM_PERM, enable_quality_filter: bool = False, enable_pii_ml: bool = False, + pii_ml_language: str = "en", emit_croissant: bool = False, ) -> AuditReport: """Run the audit pipeline over a JSONL file or split-keyed directory. @@ -2537,9 +2604,15 @@ def audit_dataset( enable_pii_ml: Phase 12.5 opt-in — layer Presidio's ML-NER PII detector (``person`` / ``organization`` / ``location``) on top of the regex detector. Requires the optional - ``[ingestion-pii-ml]`` extra; raises ``ImportError`` with an - install hint when missing so the failure surfaces before - any rows are scanned. + ``[ingestion-pii-ml]`` extra AND a spaCy NER model + (``python -m spacy download en_core_web_lg``); raises + ``ImportError`` with the install hint when either is + missing so the failure surfaces before any rows are scanned. + pii_ml_language: Phase 12.5 — language code passed to Presidio's + NLP engine. Default ``"en"``. Set to ``"tr"`` (or whichever + spaCy model the operator has loaded) for non-English + corpora; Presidio raises a typed exception when no engine + is registered for the requested language. emit_croissant: Phase 12.5 opt-in flag — when ``True``, populate the report's ``croissant`` field with a Google Croissant 1.0 dataset card (``@type: sc:Dataset``) so the same JSON file @@ -2595,6 +2668,7 @@ def audit_dataset( minhash_num_perm=minhash_num_perm, enable_quality_filter=enable_quality_filter, enable_pii_ml=enable_pii_ml, + pii_ml_language=pii_ml_language, ) splits_info[split_name] = outcome.info signatures_by_split[split_name] = outcome.signatures diff --git a/forgelm/wizard.py b/forgelm/wizard.py index 66c25da5..43fd0fdc 100644 --- a/forgelm/wizard.py +++ b/forgelm/wizard.py @@ -456,6 +456,15 @@ def _offer_ingest_for_directory(directory: Path) -> Optional[str]: return str(out_path) +_AUDIT_LARGE_FILE_THRESHOLD_BYTES: int = 100 * 1024 * 1024 # 100 MB +"""Threshold above which the wizard's audit offer warns operators that the +scan can take real time. Below 100 MB the runtime is dominated by Python +startup; above 100 MB the dedup signature pass becomes a meaningful +fraction of wall clock and the operator deserves to know before saying +yes. Tuned empirically against the streaming reader's throughput +(simhash + LSH banding ≈ 8-15 MB/s of JSONL on a single CPU).""" + + def _offer_audit_for_jsonl(jsonl_path: Path) -> bool: """Phase 12.5 wizard hook: optionally audit a JSONL the user just selected. @@ -466,12 +475,41 @@ def _offer_audit_for_jsonl(jsonl_path: Path) -> bool: declined or the audit could not start. Audit failures are surfaced as warnings: the audit is informational, not a gate, and a malformed JSONL or missing optional dep must NOT crash the wizard. + + Exception ladder (per :doc:`docs/standards/error-handling.md`'s + "log-and-swallow is permitted only on explicitly-non-fatal paths" + rule): the courtesy audit is one such path — the wizard's contract + is "produce a config", not "vouch for the dataset". The bare + ``except Exception`` at the bottom of the ladder is the exception + the standard mentions, scoped to a single short call (not a + library-wide swallow), and explicitly re-raises ``BaseException`` + subclasses (``KeyboardInterrupt``, ``SystemExit``) by virtue of + matching ``Exception`` only. """ - if not _prompt_yes_no( - f"\n Run a quality + governance audit on '{jsonl_path}' before training? " - "(length stats, language, near-duplicates, PII, secrets — CPU-only, ~seconds)", - default=True, - ): + # File-size-aware copy: small JSONL → "~ seconds" is honest; large + # JSONL → warn the operator the scan takes real time before they + # commit to it. The streaming audit doesn't load the whole file into + # memory, so accepting on a large file is safe — just slow. + try: + size_bytes = jsonl_path.stat().st_size + except OSError: + size_bytes = 0 # Fall through to the small-file copy; the + # subsequent ``audit_dataset`` call surfaces the real OSError. + if size_bytes > _AUDIT_LARGE_FILE_THRESHOLD_BYTES: + size_mb = size_bytes / (1024 * 1024) + prompt = ( + f"\n Run a quality + governance audit on '{jsonl_path}' " + f"({size_mb:.0f} MB) before training? Scan is CPU-only and " + "streams the file; runtime depends on size (≈ 8–15 MB/s of " + "JSONL on a single CPU)." + ) + else: + prompt = ( + f"\n Run a quality + governance audit on '{jsonl_path}' " + "before training? (length stats, language, near-duplicates, " + "PII, secrets — CPU-only, runtime depends on dataset size)" + ) + if not _prompt_yes_no(prompt, default=True): print(" Skipped — audit can be run later via:") print(f" forgelm audit {jsonl_path}") return False @@ -487,6 +525,11 @@ def _offer_audit_for_jsonl(jsonl_path: Path) -> bool: print(f"\n Running audit on '{jsonl_path}'…") try: + # Default flags only — the courtesy audit doesn't reach for + # ``--quality-filter`` / ``--pii-ml`` / ``--croissant`` because + # those need either an optional extra or a configuration choice + # the wizard hasn't asked the operator about. Operators that + # want the richer audit run ``forgelm audit`` directly. report = audit_dataset(str(jsonl_path)) except (FileNotFoundError, ValueError, OSError) as exc: # Filesystem / parse errors — same defensive shape we use for @@ -506,7 +549,9 @@ def _offer_audit_for_jsonl(jsonl_path: Path) -> bool: # version-mismatch surfaced through datasketch's own internals, or # an unanticipated edge case in a downstream audit module must NOT # crash the wizard. The audit is informational; we log loudly and - # let the operator continue. + # let the operator continue. ``KeyboardInterrupt`` / + # ``SystemExit`` propagate by virtue of being ``BaseException`` + # subclasses, so Ctrl-C still aborts the wizard cleanly. print(f" Audit could not run: {exc}") return False diff --git a/notebooks/data_curation.ipynb b/notebooks/data_curation.ipynb index adcba7dc..14a92109 100644 --- a/notebooks/data_curation.ipynb +++ b/notebooks/data_curation.ipynb @@ -399,7 +399,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": "## What's next\n\n- **Train on the curated JSONL**: point any of the training notebooks at\n `./curated/policies.jsonl` via `data.dataset_name_or_path`.\n - [`quickstart_sft.ipynb`](./quickstart_sft.ipynb) — first SFT run\n - [`multi_dataset.ipynb`](./multi_dataset.ipynb) — mix the curated\n set with public datasets\n- **Long-form guides**:\n - [`docs/guides/ingestion.md`](https://github.com/cemililik/ForgeLM/blob/main/docs/guides/ingestion.md)\n — every flag, every chunking strategy, every extra\n - [`docs/guides/data_audit.md`](https://github.com/cemililik/ForgeLM/blob/main/docs/guides/data_audit.md)\n — audit JSON schema, severity tiers, governance integration\n- **Wizard**: `forgelm --wizard` offers to run `forgelm ingest` inline\n when you point it at a directory of raw docs (Phase 11.5+), and as of\n Phase 12.5 it also offers to run `forgelm audit` automatically once a\n JSONL is selected — closes the BYOD audit loop end-to-end.\n- **Phase 12.5 audit add-ons (v0.5.3)**:\n - `--croissant` emits a [Google Croissant 1.0](http://mlcommons.org/croissant/)\n dataset card under the report's `croissant` key so the same JSON\n file doubles as both the EU AI Act Article 10 governance artifact\n and a Croissant-consumer dataset card.\n - `--pii-ml` layers\n [Presidio](https://github.com/microsoft/presidio) NER on top of\n the regex detector — adds `person` / `organization` /\n `location` categories to `pii_summary` / `pii_severity`. Requires\n `pip install 'forgelm[ingestion-pii-ml]'`." + "source": "## What's next\n\n- **Train on the curated JSONL**: point any of the training notebooks at\n `./curated/policies.jsonl` via `data.dataset_name_or_path`.\n - [`quickstart_sft.ipynb`](./quickstart_sft.ipynb) — first SFT run\n - [`multi_dataset.ipynb`](./multi_dataset.ipynb) — mix the curated\n set with public datasets\n- **Long-form guides**:\n - [`docs/guides/ingestion.md`](https://github.com/cemililik/ForgeLM/blob/main/docs/guides/ingestion.md)\n — every flag, every chunking strategy, every extra\n - [`docs/guides/data_audit.md`](https://github.com/cemililik/ForgeLM/blob/main/docs/guides/data_audit.md)\n — audit JSON schema, severity tiers, governance integration\n- **Wizard**: `forgelm --wizard` offers to run `forgelm ingest` inline\n when you point it at a directory of raw docs (Phase 11.5+), and as of\n Phase 12.5 it also offers to run `forgelm audit` automatically once a\n JSONL is selected — closes the BYOD audit loop end-to-end.\n- **Phase 12.5 audit add-ons (v0.5.3)**:\n - `--croissant` emits a [Google Croissant 1.0](http://mlcommons.org/croissant/)\n dataset card under the report's `croissant` key so the same JSON\n file doubles as both the EU AI Act Article 10 governance artifact\n and a Croissant-consumer dataset card.\n - `--pii-ml [--pii-ml-language LANG]` layers\n [Presidio](https://github.com/microsoft/presidio) NER on top of\n the regex detector — adds `person` / `organization` /\n `location` categories to `pii_summary` / `pii_severity`. **Two-step\n install:**\n ```bash\n pip install 'forgelm[ingestion-pii-ml]'\n python -m spacy download en_core_web_lg\n ```\n `presidio-analyzer` does not transitively ship a spaCy NER model;\n without the model `--pii-ml` raises an `ImportError` with the\n install recipe **before** any rows are scanned. Pass\n `--pii-ml-language tr` (and download the matching spaCy model) to\n audit Turkish corpora." } ], "metadata": { diff --git a/tests/test_phase12_5.py b/tests/test_phase12_5.py index 2ca7946c..8191e508 100644 --- a/tests/test_phase12_5.py +++ b/tests/test_phase12_5.py @@ -197,9 +197,14 @@ def test_croissant_section_empty_by_default(self, tmp_path): assert exc_info.value.code == 0 report = json.loads((out_dir / "data_audit_report.json").read_text(encoding="utf-8")) - # Key may or may not be present depending on serializer; the contract - # is "no Croissant content unless --croissant is passed". - assert report.get("croissant", {}) == {} + # Pin the contract: ``croissant`` is *always* present in the + # serialised report (dataclass ``field(default_factory=dict)``) + # and is the empty dict when ``--croissant`` is off. This + # matches the precedent set by ``secrets_summary`` / + # ``quality_summary`` and prevents a future serializer change + # from silently dropping the key for empty dicts. + assert "croissant" in report + assert report["croissant"] == {} def test_croissant_flag_emits_minimal_card(self, tmp_path): ds = tmp_path / "data.jsonl" @@ -290,3 +295,234 @@ def test_presidio_severity_table_exposes_ml_tiers(self): assert PII_ML_SEVERITY["person"] == "medium" assert PII_ML_SEVERITY["organization"] == "low" assert PII_ML_SEVERITY["location"] == "low" + + def test_presidio_entity_map_only_canonical_keys(self): + # Presidio canonicalises spaCy NER labels (PER/PERSON → PERSON, + # LOC/GPE → LOCATION, ORG → ORGANIZATION, NORP → NRP) before + # they reach ``analyzer.analyze().entity_type``. Pin the map to + # the canonical names only so a future maintainer doesn't read + # raw spaCy keys (``ORG`` / ``GPE`` / ``NORP`` / ``FAC``) as + # live coverage. NRP is deliberately excluded — it's a distinct + # privacy signal from ``location``. + from forgelm.data_audit import _PRESIDIO_ENTITY_MAP + + assert set(_PRESIDIO_ENTITY_MAP.keys()) == {"PERSON", "ORGANIZATION", "LOCATION"} + assert _PRESIDIO_ENTITY_MAP["PERSON"] == "person" + assert _PRESIDIO_ENTITY_MAP["ORGANIZATION"] == "organization" + assert _PRESIDIO_ENTITY_MAP["LOCATION"] == "location" + + def test_presidio_missing_spacy_model_surfaces_install_hint(self): + # Even when ``presidio-analyzer`` is importable, the spaCy model + # is a separate ``python -m spacy download …`` step. spaCy + # raises ``OSError("Can't find model …")`` when the model + # package isn't on the import path; ``_require_presidio`` must + # catch that and re-raise as ``ImportError`` with the install + # recipe so the operator never gets a deep spaCy traceback. + from forgelm.data_audit import _get_presidio_analyzer, _require_presidio + + # Reset the cache so our patched class is the one that gets + # constructed; otherwise a previous successful build could + # return a stale instance. + _get_presidio_analyzer.cache_clear() + try: + + class _BoomAnalyzer: + def __init__(self): + raise OSError("Can't find model 'en_core_web_lg'") + + with patch("forgelm.data_audit._HAS_PRESIDIO", True): + with patch("forgelm.data_audit._PresidioAnalyzer", _BoomAnalyzer): + with pytest.raises(ImportError) as exc_info: + _require_presidio() + finally: + # Always restore the cache so subsequent tests in the same + # session don't see ``_BoomAnalyzer``. + _get_presidio_analyzer.cache_clear() + msg = str(exc_info.value) + assert "spacy" in msg.lower() or "spaCy" in msg + assert "ingestion-pii-ml" in msg + assert "en_core_web_lg" in msg + + def test_presidio_entity_map_collapses_findings_to_buckets(self): + # Behavioural test for ``detect_pii_ml`` — a stub Presidio + # analyzer that emits canonical entity_type strings should + # produce the right counts; non-canonical ones (raw spaCy + # labels) should be ignored. This test would have caught the + # original review's M2 immediately. + from forgelm.data_audit import _get_presidio_analyzer, detect_pii_ml + + class _Finding: + def __init__(self, entity_type): + self.entity_type = entity_type + + class _StubAnalyzer: + def analyze(self, text, language): + # Emit the full Presidio canonical set + a couple of raw + # spaCy labels (which Presidio would never actually + # emit) to confirm our map only honours canonicals. + return [ + _Finding("PERSON"), + _Finding("ORGANIZATION"), + _Finding("LOCATION"), + _Finding("LOCATION"), + _Finding("NRP"), # Presidio canonical, not mapped + _Finding("ORG"), # raw spaCy, must NOT be honoured + _Finding("GPE"), # raw spaCy, must NOT be honoured + ] + + _get_presidio_analyzer.cache_clear() + try: + with patch("forgelm.data_audit._HAS_PRESIDIO", True): + with patch("forgelm.data_audit._get_presidio_analyzer", return_value=_StubAnalyzer()): + counts = detect_pii_ml("Alice works at Acme Corp in Berlin.") + finally: + _get_presidio_analyzer.cache_clear() + assert counts == {"person": 1, "organization": 1, "location": 2} + + +# --------------------------------------------------------------------------- +# Cross-cutting tests — coverage gaps surfaced by the Phase 12.5 review +# --------------------------------------------------------------------------- + + +class TestCroissantMultiSplit: + """Multi-split layout produces one cr:FileObject + cr:RecordSet per split.""" + + def _write_jsonl(self, path: Path, rows) -> None: + with open(path, "w", encoding="utf-8") as fh: + for row in rows: + fh.write(json.dumps(row) + "\n") + + def test_multi_split_card_carries_one_record_per_split(self, tmp_path): + # train.jsonl / validation.jsonl / test.jsonl is the canonical + # multi-split layout the audit recognises. Each must produce + # its own ``cr:FileObject`` (in ``distribution``) and its own + # ``cr:RecordSet`` (with the right ``@id``). + data_dir = tmp_path / "splits" + data_dir.mkdir() + for split, rows in [ + ("train", [{"text": "alpha"}, {"text": "beta"}]), + ("validation", [{"text": "gamma"}]), + ("test", [{"text": "delta"}]), + ]: + self._write_jsonl(data_dir / f"{split}.jsonl", rows) + + out_dir = tmp_path / "audit" + with patch( + "sys.argv", + [ + "forgelm", + "audit", + str(data_dir), + "--output", + str(out_dir), + "--croissant", + ], + ): + with pytest.raises(SystemExit) as exc_info: + main() + assert exc_info.value.code == 0 + + report = json.loads((out_dir / "data_audit_report.json").read_text(encoding="utf-8")) + crois = report["croissant"] + ids = {entry["@id"] for entry in crois["distribution"]} + assert ids == {"train.jsonl", "validation.jsonl", "test.jsonl"} + record_ids = {entry["@id"] for entry in crois["recordSet"]} + assert record_ids == {"train", "validation", "test"} + # contentUrl must be the relative file_id, not an absolute + # filesystem path — see the Phase 12.5 review m3 finding. + for entry in crois["distribution"]: + assert entry["contentUrl"] == entry["@id"] + assert "/" not in entry["contentUrl"] + + +class TestAllMaskSymmetric: + """``--all-mask`` set-union covers the secrets-mask-already-true direction too.""" + + def test_all_mask_with_secrets_mask_already_set(self, tmp_path): + # Symmetric counterpart to test_all_mask_combines_with_individual_flags_no_error + # so a future refactor that drops one branch of the boolean + # union trips a test. + aws_key = "AKIA" + "IOSFODNN7" + "EXAMPLE" + src = tmp_path / "input.txt" + src.write_text( + f"alice@example.com mentioned key={aws_key}", + encoding="utf-8", + ) + out = tmp_path / "out.jsonl" + with patch( + "sys.argv", + [ + "forgelm", + "ingest", + str(src), + "--output", + str(out), + "--secrets-mask", + "--all-mask", + ], + ): + with pytest.raises(SystemExit) as exc_info: + main() + assert exc_info.value.code == 0 + written = out.read_text(encoding="utf-8") + assert "[REDACTED]" in written + assert "[REDACTED-SECRET]" in written + assert aws_key not in written + + +class TestPiiMlJsonEnvelope: + """``forgelm audit --output-format json`` surfaces the Croissant card.""" + + def _write_jsonl(self, path: Path, rows) -> None: + with open(path, "w", encoding="utf-8") as fh: + for row in rows: + fh.write(json.dumps(row) + "\n") + + def test_envelope_carries_croissant_when_flag_on(self, tmp_path, capsys): + ds = tmp_path / "data.jsonl" + self._write_jsonl(ds, [{"text": "alpha"}]) + out_dir = tmp_path / "audit" + with patch( + "sys.argv", + [ + "forgelm", + "audit", + str(ds), + "--output", + str(out_dir), + "--croissant", + "--output-format", + "json", + ], + ): + with pytest.raises(SystemExit) as exc_info: + main() + assert exc_info.value.code == 0 + envelope = json.loads(capsys.readouterr().out) + assert "croissant" in envelope + assert envelope["croissant"]["@type"] == "sc:Dataset" + + def test_envelope_croissant_is_empty_when_flag_off(self, tmp_path, capsys): + ds = tmp_path / "data.jsonl" + self._write_jsonl(ds, [{"text": "alpha"}]) + out_dir = tmp_path / "audit" + with patch( + "sys.argv", + [ + "forgelm", + "audit", + str(ds), + "--output", + str(out_dir), + "--output-format", + "json", + ], + ): + with pytest.raises(SystemExit) as exc_info: + main() + assert exc_info.value.code == 0 + envelope = json.loads(capsys.readouterr().out) + # Phase 12 precedent: present-and-empty when off. + assert "croissant" in envelope + assert envelope["croissant"] == {} From 50c370d587f87dbf5161f765bff450e31955bc9a Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 29 Apr 2026 18:46:05 +0300 Subject: [PATCH 03/12] release(0.5.0): consolidate Phases 11 + 11.5 + 12 + 12.5 into one comprehensive v0.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Originally planned as four sequential PyPI tags (v0.5.0 / v0.5.1 / v0.5.2 / v0.5.3). The four phases form one coherent surface (ingest → polish → mature → polish) that's hard to use in parts; consolidating into a single comprehensive v0.5.0 release keeps the user-facing release notes focused while git history retains the four phases as separate commit batches. pyproject.toml - version: 0.5.2rc1 → 0.5.0 CHANGELOG.md - Collapse [Unreleased] + [0.5.2] + [0.5.1] + [0.5.0] (summary stub) into a single [0.5.0] — 2026-04-29 entry. - Theme paragraph + consolidation note explaining the merger. - Internal phase boundaries preserved as section markers (### Phase 12.5 / Phase 12 / Phase 11.5 / Phase 11) so reviewers can map back to PR history (#11, #12, #13, current). - [Unreleased] is now empty (next minor picks up Phase 14). README.md - "Document Ingestion & Data Audit (v0.5.0 — polished in v0.5.1, matured in v0.5.2)" → "(v0.5.0 — Phases 11 + 11.5 + 12 + 12.5 consolidated)". - Drop "v0.5.2 added X" / "v0.5.1 added Y" / "Phase 12.5 polish (v0.5.3)" history phrasing — features now read as part of v0.5.0. docs/roadmap.md + roadmap-tr.md - Status table: four separate "🟡 Merged" rows for Phases 11 / 11.5 / 12 / 12.5 collapsed into one combined row pointing at releases.md#v050. Phase 14 row reslotted from v0.5.3 to v0.5.1. - "Latest PyPI: v0.4.5" wording trimmed (the v0.5.0/.1/.2 sentence about three pending tags is gone). - "Next: v0.5.3" → "Next: v0.5.1" (Phase 14). - Phase count: 16 complete + 3 planned → 17 complete + 2 planned (12.5 moves into the consolidated release; 13 + 14 remain planned). - Mermaid diagram: P11 / P115 / P12 / P125 all point at the same V2[v0.5.0 pending] node; styling bumped to merged-but-not-tagged light-green. docs/roadmap/releases.md - v0.5.0 / v0.5.1 / v0.5.2 sections collapsed into a single comprehensive v0.5.0 entry organised by phase (### Phase 11 / 11.5 / 12 / 12.5). - v0.5.3 (Phase 14) → v0.5.1 (Phase 14). docs/roadmap/phase-{11-5,12-5,12-data-curation,14-pipeline}.md - Consolidation note at the top of each historical phase plan ("originally targeted vX.Y.Z, ships as part of consolidated v0.5.0; see releases.md#v050"). Body kept intact as the historical planning trail. - phase-14: Target release v0.5.3 → v0.5.1; "starts after v0.5.2 PyPI tag" → "starts after v0.5.0 PyPI tag". - phase-12-5-backlog rows: "ships with v0.5.3" → "ships with v0.5.0". docs/guides/{ingestion,data_audit}{,-tr}.md + docs/reference/usage{,-tr}.md - Drop the "v0.5.2 added" / "v0.5.1 promoted" history phrasing inside feature descriptions. Features now read as v0.5.0 capabilities; phase labels (Phase 11 / 11.5 / 12 / 12.5) are kept as breadcrumbs for reviewers. - Section headings like "Document Ingestion (v0.5.0+; token-aware in v0.5.1; markdown + secrets-mask in v0.5.2)" → "(v0.5.0)". - Inline comments in CLI examples drop "v0.5.2:" / "v0.5.3 (Phase 12.5):" prefixes since the features all ship in the same release. notebooks/data_curation.ipynb - Install pin: 0.5.2 → 0.5.0 (cells 1 + 16). - "Phase 12.5 audit add-ons (v0.5.3)" markdown → "Other v0.5.0 audit add-ons". - Section 2 header drops the "Phase 12.5 shorthand (v0.5.3)" framing for --all-mask; the flag is just part of the v0.5.0 release. forgelm/cli.py - One internal comment tweaked from "v0.5.1 consumers" to "any pre-Phase-12 JSON consumer" (the legacy envelope key still has a real reason to exist; the framing just stops referencing a release that won't exist as a separate PyPI tag). - Phase labels in user-visible help text (Phase 11.5 / 12 / 12.5) kept intact — they're descriptive history, not version pins. Verification - ruff clean (78 files) - 840 tests pass (13 skipped) - 7/7 bilingual H2 parity (configuration / usage / distributed / data_preparation / architecture / ingestion / data_audit) Co-Authored-By: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 104 +++++++----------- README.md | 18 +-- docs/guides/data_audit-tr.md | 6 +- docs/guides/data_audit.md | 6 +- docs/guides/ingestion-tr.md | 13 +-- docs/guides/ingestion.md | 22 +++- docs/reference/usage-tr.md | 26 ++--- docs/reference/usage.md | 26 ++--- docs/roadmap-tr.md | 33 +++--- docs/roadmap.md | 35 +++--- docs/roadmap/phase-11-5-backlog.md | 7 +- docs/roadmap/phase-12-5-backlog.md | 16 ++- .../phase-12-data-curation-maturity.md | 7 +- docs/roadmap/phase-14-pipeline-chains.md | 4 +- docs/roadmap/releases.md | 81 ++++++-------- forgelm/cli.py | 8 +- notebooks/data_curation.ipynb | 24 +--- pyproject.toml | 2 +- 18 files changed, 212 insertions(+), 226 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b4f95c50..a7b2776f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,48 @@ All notable changes to ForgeLM are documented here. ## [Unreleased] -### Added — Phase 12.5 (Data Curation Polish backlog) +(no unreleased changes — next minor will pick up Phase 14, tracked in +[docs/roadmap.md](docs/roadmap.md)) + +--- + +## [0.5.0] — 2026-04-29 + +**Theme:** "Document Ingestion + Data Curation Pipeline" — Phases 11, +11.5, 12, and 12.5 ship as one comprehensive release. + +> **Note on consolidation.** Originally planned as four sequential +> PyPI tags (`v0.5.0` / `v0.5.1` / `v0.5.2` / `v0.5.3`) but consolidated +> into a single `v0.5.0` because the four phases form one coherent +> surface (ingest → polish → mature → polish) that's hard to use in +> parts. Git history retains the four phases as separate commit +> batches; this entry collapses them into the user-facing release +> notes. Section markers below preserve the phase boundary so +> reviewers can map back to [docs/roadmap/releases.md](docs/roadmap/releases.md). + +The release adds: + +- **Phase 11** — `forgelm ingest` (PDF / DOCX / EPUB / TXT / Markdown + → SFT-ready JSONL) + `forgelm audit` (length / language / + near-duplicate / cross-split leakage / PII) + EU AI Act Article 10 + governance integration. +- **Phase 11.5** — operational polish on the Phase 11 surface: LSH + banding, streaming reader, token-aware chunking, PDF + header/footer dedup, PII severity tiers, atomic audit writes. +- **Phase 12** — data curation maturity: MinHash LSH dedup option, + markdown-aware splitter, code/secrets leakage tagger, heuristic + quality filter, DOCX table preservation. +- **Phase 12.5** — small additive polish: `--all-mask` shorthand, + Croissant 1.0 dataset card emission, optional Presidio ML-NER PII + adapter, wizard "audit first" entry point. + +CI / docs / standards bookkeeping accompanying every phase is folded +into "Cross-cutting review hardening" at the bottom (rounds 1–12 of +review-cycle fixes applied across the four phases above). + +--- + +### Phase 12.5 — Data Curation Polish (backlog items #1–#4) Four follow-up items from [`docs/roadmap/phase-12-5-backlog.md`](docs/roadmap/phase-12-5-backlog.md) @@ -470,67 +511,6 @@ Operational polish on top of `v0.5.0`'s ingestion + audit surface — no new tra --- -## [0.5.2] — 2026-04-28 - -**Theme:** "Data Curation Maturity" (Phase 12 Tier 1) — direct continuation of the Phase 11 / 11.5 ingestion + audit lineage. Closes the four gaps surfaced by the post-`v0.5.1` competitive review (LLaMA-Factory / Axolotl / Unsloth / NeMo Curator / Dolma / RedPajama / LlamaIndex / LangChain / Marker / Docling). Tier 2/3 items (Presidio adapter, Croissant metadata, `--all-mask`, wizard "audit first") deferred to [Phase 12.5 backlog](docs/roadmap/phase-12-5-backlog.md). - -### Added - -- **MinHash LSH dedup option** — opt-in `--dedup-method minhash --jaccard-threshold 0.85` route via `datasketch` (`[ingestion-scale]` extra) for >50K-row corpora. Default simhash + LSH banding stays untouched. -- **Markdown-aware splitter** — new `--strategy markdown` preserves heading hierarchy (`# H1` / `## H2`), keeps fenced code blocks atomic, and inlines a heading breadcrumb so SFT loss sees document context. -- **Code / secrets leakage tagger** — new `secrets_summary` block in audit JSON (AWS / GitHub / Slack / OpenAI / Google / JWT / OpenSSH / PGP / Azure storage). Ingest gains `--secrets-mask` (mask order: secrets → PII so combined detectors don't double-count). `[ingestion-secrets]` extra is reserved for a follow-up release; the regex-only fallback is what runs today. -- **Heuristic quality filter** — opt-in `--quality-filter` adds a `quality_summary` block with Gopher / C4 / RefinedWeb-style heuristics (mean-word-length, alphabetic ratio, end-of-line punctuation, short-paragraph ratio, repeated lines). ML classifiers stay deferred to Phase 13+. -- **DOCX table preservation** — `_extract_docx` emits Markdown table syntax (header + `---` separator + body rows) at extraction time, **before any chunking strategy runs**, instead of the previous `" | "` flat join. Uneven rows padded; all-blank rows trimmed. -- **`secrets_redaction_counts`** — added to the `--output-format json` envelope of `forgelm ingest` for CI/CD consumers; `{secret_type: count}` map of redacted spans. - -### Changed / Fixed (review-cycle hardening) - -- **Regex hygiene standard** ([`docs/standards/regex.md`](docs/standards/regex.md)) — codified 8 hard rules distilled from Phase 11 / 11.5 / 12 review cycles (Unicode-aware `\w`, no single-char classes, bounded quantifiers, no two competing unbounded greedy quantifiers, no `\s` under MULTILINE, no `.*?` + back-reference + DOTALL, intentional ASCII-only classes permitted with `re.ASCII`). The 8th rule covers `^.*` anti-pattern. -- **Replaced `_MARKDOWN_CODE_FENCE` regex with `_parse_md_fence`** — non-regex per-line parser; provably O(n). The previous regex had two unbounded greedy quantifiers in sequence over overlapping character classes (SonarCloud `python:S5852`). -- **`_strip_code_fences`** in `data_audit.py` rewritten as a per-line state machine; tracks the opening fence's run length per CommonMark §4.5 so a 4-backtick opener isn't prematurely closed by a 3-backtick line. -- **`_markdown_sections` complexity refactor** — extracted `_advance_fence_state`, `_flush_section`, `_render_sections` helpers so the dispatch loop reads top-down (cognitive complexity below SonarCloud's 15-line threshold). -- **`_parse_md_fence`** — now rejects backtick fence openers whose info string contains a backtick (CommonMark §4.5). -- **MinHash knob validation** — `audit_dataset` validates `minhash_jaccard ∈ [0.0, 1.0]` and `minhash_num_perm > 0` at the API boundary. -- **`re.ASCII` flag** on secret regexes (`github_token` / `openai_api_key` / `google_api_key` / `jwt`) so `\w` is restricted to ASCII; full BEGIN…END envelope matching for OpenSSH / RSA / DSA / EC / PGP private-key blocks. -- **Documentation drift sweep** — multiple rounds of EN ↔ TR mirror sync (`data_audit-tr.md`, `ingestion-tr.md`, `regex.md`, `data_audit.md`); legacy JSON-key documentation now references the actual `splits..near_duplicate_pairs` path. -- **ReDoS regression tests** — `tests/test_phase12_review_fixes.py` adds linearity benchmarks at 1K / 5K / 10K (median of 5 runs) for `_MARKDOWN_HEADING_PATTERN`, `_strip_code_fences`, and `_parse_md_fence`. - ---- - -## [0.5.1] — 2026-04-27 - -**Theme:** "Ingestion / Audit Polish" (Phase 11.5) — operational polish on top of `v0.5.0`'s ingestion + audit surface. No new training capabilities, but materially better handling for large corpora and a cleaner CLI shape. - -### Added - -- **`forgelm audit` subcommand** — promoted from the `--data-audit` top-level flag; first-class subcommand with its own `--output` default. The flag is preserved as a deprecation alias. -- **LSH-banded near-duplicate detection** — replaces the `O(n²)` pair scan inside each split (and across splits) with locality-sensitive-hashing bands (4 × 16-bit on the 64-bit fingerprint). Drops average-case to `O(n × k)` and unblocks audits on 100K+ row corpora. -- **Streaming `_read_jsonl_split`** — JSONL reader yields rows lazily; per-split aggregator stays generator-based until simhash collection. Bounds memory on multi-million-row splits. -- **Token-aware `--chunk-tokens`** — optional ingestion flag that sizes chunks against an HF tokenizer instead of raw character counts. -- **PDF page-level header / footer dedup** — repeated page headers (company watermark, page number) used to inflate near-duplicate counts; common-prefix / common-suffix detection across pages strips them automatically. -- **PII severity tiers** — audit output adds a `pii_severity` block grading each PII type as `low / medium / high / critical` (e.g. `credit_card` → critical, `phone` → low) so compliance reviewers get a one-glance verdict. -- **`summarize_report` truncation policy** — multi-split summaries get a `verbose=False` default that suppresses zero-finding splits. -- **Structured ingestion notes** — `IngestionResult.extra_notes` keeps the human-readable list but gains a parallel `notes_structured: {key: value}` map for programmatic consumers. -- **Wizard "ingest first" entry point** — first-class wizard option that routes to `forgelm ingest`, surfaces a JSONL path, and folds it back into the BYOD prompt. -- **xxhash backend for simhash + token-level memo** — drop-in faster non-crypto digest path (BLAKE2b kept as fallback). Token-level `lru_cache` memoizes repeat tokens for a 2–5× speedup on long corpora. -- **Atomic audit report write** — `data_audit_report.json` is written via `tempfile.NamedTemporaryFile` + atomic rename so a crashed audit never leaves a half-written report on disk. - ---- - -## [0.5.0] — 2026-04-27 - -**Theme:** "Document Ingestion & Data Audit" (Phase 11). Bridges raw enterprise corpora to ForgeLM's training data format and surfaces governance signals before training. - -### Added - -- **`forgelm/ingestion.py` + `forgelm ingest` subcommand** — multi-format document → JSONL pipeline with `paragraph` (default) and `sliding` chunking strategies, recursive directory walk, optional `--pii-mask`. Supported extensions: `.pdf` (`pypdf`), `.docx` (`python-docx`), `.epub` (`ebooklib` + `beautifulsoup4`), `.txt`, `.md`. Output is `{"text": ...}` JSONL recognised by ForgeLM's data loader as pre-formatted SFT input. OCR is intentionally out of scope; scanned PDFs warn and produce zero chunks. -- **`forgelm/data_audit.py` + `forgelm --data-audit` flag** — per-split metrics (sample count, column schema, length distribution `min/max/mean/p50/p95`, top-3 language detection, null/empty rate), 64-bit simhash near-duplicate detection within each split, cross-split overlap report (catches train/test leakage), PII regex with Luhn-validated credit cards and TC Kimlik checksum-validated TR IDs. -- **EU AI Act Article 10 integration** — `generate_data_governance_report` inlines `data_audit_report.json` under the `data_audit` key when present in the trainer's `output_dir`. -- **`pyproject.toml` `[ingestion]` extra** — `pypdf`, `python-docx`, `ebooklib`, `beautifulsoup4`, `langdetect`. Cross-platform; no native compilation. Plain TXT / Markdown ingestion + the audit module work without installing the extra (PII regex, simhash, length stats are pure stdlib). -- **Tests + docs** — `tests/test_ingestion.py` and `tests/test_data_audit.py` (54 tests; PDF round-trip skips when `pypdf` missing). New guides: `docs/guides/ingestion.md`, `docs/guides/data_audit.md`. README feature section, install matrix, and roadmap status updated. - ---- - ## [0.4.5] — 2026-04-26 ### Added diff --git a/README.md b/README.md index 320eb915..7323eeb3 100644 --- a/README.md +++ b/README.md @@ -27,14 +27,16 @@ - **LLM-as-Judge**: API-based (OpenAI) or local model scoring for quality assessment - **Auto-Revert**: Opt-in (`evaluation.auto_revert: true`) — automatically discards models that fail loss, benchmark, or safety thresholds before artifacts are written -### Document Ingestion & Data Audit (v0.5.0 — polished in v0.5.1, matured in v0.5.2) -- **Multi-Format Ingestion**: `forgelm ingest ./policies/ --recursive --output data/policies.jsonl` — turns raw PDF / DOCX / EPUB / TXT / Markdown into the SFT-ready JSONL the trainer accepts. Optional dep: `pip install forgelm[ingestion]`. v0.5.2 added a **Markdown-aware splitter** (`--strategy markdown`) and DOCX table preservation in Markdown table syntax. -- **Chunking Strategies**: `paragraph` (default; preserves boundaries), `sliding` (fixed window with overlap), or `markdown` (v0.5.2 — heading-aware splitter that keeps fenced code blocks atomic and inlines a heading breadcrumb at the top of each chunk for SFT context). A `semantic` strategy is reserved for a follow-up phase — the implementation in `forgelm.ingestion` raises `NotImplementedError` today and the CLI hides it from `--strategy` choices. Token-aware mode in v0.5.1: `--chunk-tokens 1024 --tokenizer Qwen/Qwen2.5-7B-Instruct` sizes chunks against your model's actual vocabulary. +### Document Ingestion & Data Audit (v0.5.0 — Phases 11 + 11.5 + 12 + 12.5 consolidated) +- **Multi-Format Ingestion**: `forgelm ingest ./policies/ --recursive --output data/policies.jsonl` — turns raw PDF / DOCX / EPUB / TXT / Markdown into the SFT-ready JSONL the trainer accepts. Optional dep: `pip install forgelm[ingestion]`. Includes a **Markdown-aware splitter** (`--strategy markdown`) and DOCX table preservation in Markdown table syntax. +- **Chunking Strategies**: `paragraph` (default; preserves boundaries), `sliding` (fixed window with overlap), or `markdown` (heading-aware splitter that keeps fenced code blocks atomic and inlines a heading breadcrumb at the top of each chunk for SFT context). A `semantic` strategy is reserved for a follow-up phase — the implementation in `forgelm.ingestion` raises `NotImplementedError` today and the CLI hides it from `--strategy` choices. Token-aware mode: `--chunk-tokens 1024 --tokenizer Qwen/Qwen2.5-7B-Instruct` sizes chunks against your model's actual vocabulary. - **PII Masking on Ingest**: `--pii-mask` redacts emails, phones, credit cards (Luhn-validated), IBAN, and national IDs (TR / DE / FR / US-SSN) before chunks land in the JSONL. -- **PDF Page Header/Footer Dedup (v0.5.1)**: Lines that recur on ≥ 70 % of PDF pages (watermarks, page numbers, copyright lines) are stripped automatically — the audit's near-duplicate counts stop misfiring on long policy / book PDFs. -- **Dataset Audit**: `forgelm audit data/sft.jsonl --output ./audit/` — produces `data_audit_report.json` with sample count, length distribution, top-3 language detection, LSH-banded near-duplicate rate (`O(n × k)` typical case, exact recall at the default Hamming threshold; v0.5.2 added optional **MinHash LSH** via `--dedup-method minhash` for >50K-row corpora), cross-split leakage check, null/empty rate, and PII flag counts with **severity tiers** (critical / high / medium / low + worst-tier verdict). v0.5.2 also added an always-on **secrets/credential scan** covering the nine families enumerated in `forgelm.data_audit.SECRET_TYPES` — AWS access keys, GitHub tokens, Slack tokens, OpenAI API keys, Google API keys, JWTs, full OpenSSH private-key blocks, full PGP private-key blocks, and Azure storage connection strings — and an opt-in **heuristic quality filter** (`--quality-filter`). CPU-only; streaming JSONL reader keeps memory bounded on multi-million-row splits; feeds EU AI Act Article 10 governance artifact automatically when present at training time. Legacy `--data-audit` flag still works as a deprecation alias. -- **Secrets-Aware Ingest (v0.5.2)**: `forgelm ingest … --secrets-mask` scrubs credentials before chunks land in the JSONL — fine-tuning on text containing real API keys memorises them at training time. Pairs with `--pii-mask`; secrets run first so combined detectors don't double-count overlapping spans. -- **Phase 12.5 polish (v0.5.3)**: `forgelm ingest --all-mask` shorthand for combined PII + secrets scrubbing; `forgelm audit --croissant` emits a [Google Croissant 1.0](http://mlcommons.org/croissant/) dataset card alongside the existing report; `forgelm audit --pii-ml [--pii-ml-language LANG]` layers [Presidio](https://github.com/microsoft/presidio) NER on top of the regex detector via the optional `[ingestion-pii-ml]` extra **plus a separate `python -m spacy download en_core_web_lg`** step (or the matching model for the chosen language); the wizard now offers to run `forgelm audit` automatically after a JSONL is selected (BYOD onboarding closes the audit loop). +- **PDF Page Header/Footer Dedup**: Lines that recur on ≥ 70 % of PDF pages (watermarks, page numbers, copyright lines) are stripped automatically — the audit's near-duplicate counts stop misfiring on long policy / book PDFs. +- **Dataset Audit**: `forgelm audit data/sft.jsonl --output ./audit/` — produces `data_audit_report.json` with sample count, length distribution, top-3 language detection, LSH-banded near-duplicate rate (`O(n × k)` typical case, exact recall at the default Hamming threshold; optional **MinHash LSH** via `--dedup-method minhash` for >50K-row corpora), cross-split leakage check, null/empty rate, and PII flag counts with **severity tiers** (critical / high / medium / low + worst-tier verdict). Always-on **secrets/credential scan** covers the nine families enumerated in `forgelm.data_audit.SECRET_TYPES` — AWS access keys, GitHub tokens, Slack tokens, OpenAI API keys, Google API keys, JWTs, full OpenSSH private-key blocks, full PGP private-key blocks, and Azure storage connection strings — and an opt-in **heuristic quality filter** (`--quality-filter`). CPU-only; streaming JSONL reader keeps memory bounded on multi-million-row splits; feeds EU AI Act Article 10 governance artifact automatically when present at training time. Legacy `--data-audit` flag still works as a deprecation alias. +- **Secrets-Aware Ingest**: `forgelm ingest … --secrets-mask` scrubs credentials before chunks land in the JSONL — fine-tuning on text containing real API keys memorises them at training time. Pairs with `--pii-mask`; secrets run first so combined detectors don't double-count overlapping spans. `--all-mask` is a one-flag shorthand for both. +- **Optional ML-NER PII**: `forgelm audit --pii-ml [--pii-ml-language LANG]` layers [Presidio](https://github.com/microsoft/presidio) NER on top of the regex detector via the optional `[ingestion-pii-ml]` extra **plus a separate `python -m spacy download en_core_web_lg`** step (or the matching model for the chosen language). Adds `person` / `organization` / `location` categories into the same `pii_summary` / `pii_severity` blocks under disjoint category names. +- **Croissant 1.0 dataset card**: `forgelm audit --croissant` emits a [Google Croissant 1.0](http://mlcommons.org/croissant/) dataset card under the report's `croissant` key so the same JSON file doubles as both the EU AI Act Article 10 governance artifact and a Croissant-consumer dataset card. +- **Wizard "audit first"**: when the wizard resolves a JSONL (typed or produced by `forgelm ingest`) it offers to run `forgelm audit` inline and prints the verdict before continuing — closes the BYOD audit loop end-to-end. ### Quickstart Layer (v0.4.5) - **One-Command Templates**: `forgelm quickstart customer-support` — 4 bundled templates (SFT customer-support, code-assistant, medical-qa-tr, GRPO math-reasoning) plus a bring-your-own-data domain-expert scaffold. Auto-downsizes models on small GPUs. @@ -73,7 +75,7 @@ forgelm quickstart customer-support # render config + train + chat forgelm quickstart code-assistant --dry-run # render config only forgelm quickstart medical-qa-tr --model your-org/your-model # override -# Have raw docs? Ingest them first (v0.5.0+; v0.5.1 added token-aware sizing) +# Have raw docs? Ingest them first (v0.5.0; supports token-aware sizing) pip install -e ".[ingestion]" forgelm ingest ./policies/ --recursive --output data/policies.jsonl forgelm audit data/policies.jsonl --output ./audit/ # `forgelm --data-audit ...` still works as legacy alias diff --git a/docs/guides/data_audit-tr.md b/docs/guides/data_audit-tr.md index 54c9e5d3..56a4e07d 100644 --- a/docs/guides/data_audit-tr.md +++ b/docs/guides/data_audit-tr.md @@ -2,11 +2,11 @@ `forgelm audit PATH` bir JSONL veri setini analiz eder ve kalite, governance ile PII sinyallerini kapsayan bir `data_audit_report.json` -üretir. Faz 11 (`v0.5.0`'da tanıtıldı) altyapıyı, Faz 11.5 (`v0.5.1`) +üretir. Faz 11 altyapıyı, Faz 11.5 (`v0.5.0`'da birleşti) ise top-level flag'i first-class subcommand'a yükseltti, LSH-banded near-duplicate tespitini, streaming JSONL okuyucusunu, PII şiddet katmanlarını, atomik disk yazımı ve verbose-by-default kısaltma -politikasını ekledi. **Faz 12 (`v0.5.2`)** opt-in MinHash LSH dedup +politikasını ekledi. **Faz 12 (`v0.5.0`'da birleşti)** opt-in MinHash LSH dedup yöntemini (`--dedup-method minhash`), her zaman çalışan code/credential leakage taramasını (`secrets_summary`), ve opt-in heuristic kalite filtresini (`--quality-filter`) ekledi. @@ -268,7 +268,7 @@ forgelm ingest ./policies/ --recursive --output data/policies.jsonl --secrets-ma Opsiyonel / forward-compatibility: `[ingestion-secrets]` extra'sı bir `detect-secrets>=1.5.0` bağımlılığı tanımlar ama bu **bir sonraki sürüm -için ayrılmıştır**. v0.5.2 itibarıyla `forgelm.data_audit.detect_secrets()` +için ayrılmıştır**. v0.5.0 itibarıyla `forgelm.data_audit.detect_secrets()` yalnızca yukarıdaki regex setine dayanır; extra'yı kurmak bugün audit davranışını değiştirmez. Extra, `forgelm[ingestion-secrets]` pin'leyen operatörlerin entegrasyon geldiğinde forward-compatible olabilmesi diff --git a/docs/guides/data_audit.md b/docs/guides/data_audit.md index b02f9176..e6169326 100644 --- a/docs/guides/data_audit.md +++ b/docs/guides/data_audit.md @@ -3,11 +3,11 @@ `forgelm audit PATH` analyzes a JSONL dataset and produces a `data_audit_report.json` covering quality, governance, PII, and (Phase 12 onwards) credential-leakage and heuristic-quality signals. Phase 11 -(`v0.5.0`) shipped the underlying audit; Phase 11.5 (`v0.5.1`) promoted +shipped the underlying audit; Phase 11.5 (consolidated into `v0.5.0`) promoted it from a top-level flag to a first-class subcommand and added LSH-banded near-duplicate detection, streaming JSONL reading, PII severity tiers, atomic on-disk write, and a verbose-by-default -truncation policy. **Phase 12 (`v0.5.2`)** added an opt-in MinHash LSH +truncation policy. **Phase 12 (consolidated into `v0.5.0`)** added an opt-in MinHash LSH dedup method (`--dedup-method minhash`), a code/credential leakage scan that runs always-on (`secrets_summary`), and an opt-in heuristic quality filter (`--quality-filter`). @@ -268,7 +268,7 @@ forgelm ingest ./policies/ --recursive --output data/policies.jsonl --secrets-ma Optional / forward-compatibility: the `[ingestion-secrets]` extra declares a `detect-secrets>=1.5.0` dependency that is **reserved for a -follow-up release**. As of v0.5.2 the audit's +follow-up release**. As of v0.5.0 the audit's `forgelm.data_audit.detect_secrets()` relies solely on the regex set above; installing the extra today does not change audit behaviour. The extra exists so operators who pin `forgelm[ingestion-secrets]` in diff --git a/docs/guides/ingestion-tr.md b/docs/guides/ingestion-tr.md index e1a3fae2..76197911 100644 --- a/docs/guides/ingestion-tr.md +++ b/docs/guides/ingestion-tr.md @@ -1,12 +1,11 @@ # Doküman Yutma Rehberi Ham kurumsal külliyatı (PDF / DOCX / EPUB / TXT / Markdown) ForgeLM'in -eğittiği SFT-uyumlu JSONL'a dönüştürün. Faz 11; `v0.5.0`'da tanıtıldı. -Faz 11.5 (`v0.5.1`) token-aware chunking, PDF sayfa header/footer dedup -ve yapılandırılmış ingestion notları ekledi. **Faz 12 (`v0.5.2`)** -markdown-aware splitter (`--strategy markdown`), code/credential -leakage scrubbing (`--secrets-mask`) ve DOCX tablolarının markdown -syntax'ı ile çıkarılmasını ekledi. +eğittiği SFT-uyumlu JSONL'a dönüştürün. `v0.5.0` ile birlikte gelir +(Faz 11 + 11.5 + 12 + 12.5 birleşmiş): paragraph / sliding / +markdown-aware splitter, token-aware chunking, PDF sayfa header/footer +dedup, code/credential scrubbing (`--secrets-mask`), DOCX tablolarının +markdown syntax'ı ile çıkarılması, `--all-mask` kısayolu. > Sonrasında [`forgelm audit`](data_audit-tr.md) ile uzunluk dağılımı / > dil / near-duplicate / PII metriklerini yüzeye çıkarın; chunk'ları Q&A @@ -296,7 +295,7 @@ sayfadan kısa belgelerde devre dışı kalır. Yapılandırılmış notlar ## Sınırlamalar - **OCR:** Kapsam dışı. Harici araçlar kullanın — aşağıdaki örneklere bakın. -- **Tablolar / şekiller:** Faz 12'den (`v0.5.2`) itibaren `_extract_docx()` +- **Tablolar / şekiller:** Faz 12'den (`v0.5.0`'da birleşti) itibaren `_extract_docx()` DOCX tablolarını **Markdown tablo syntax'ına** (header + `---` ayraç + body satırları) **chunking stratejisi çalışmadan önce, extraction aşamasında** dönüştürür — yani tüm stratejiler render edilmiş diff --git a/docs/guides/ingestion.md b/docs/guides/ingestion.md index 55d0d272..7068011c 100644 --- a/docs/guides/ingestion.md +++ b/docs/guides/ingestion.md @@ -1,10 +1,20 @@ # Document Ingestion Guide Convert raw enterprise corpora (PDF / DOCX / EPUB / TXT / Markdown) into the -SFT-ready JSONL ForgeLM trains on. Phase 11; introduced in `v0.5.0`. Phase -11.5 (`v0.5.1`) added token-aware chunking, PDF page header/footer dedup, -and structured ingestion notes — each documented inline in the relevant -section below. +SFT-ready JSONL ForgeLM trains on. Ships with `v0.5.0` (Phases 11 + 11.5 ++ 12 + 12.5 consolidated): paragraph / sliding / markdown chunking, +token-aware sizing, PDF page header/footer dedup, code/credential +scrubbing (`--secrets-mask`), DOCX tables rendered as markdown, and the +`--all-mask` shorthand for combined PII + secrets masking. Earlier +phase boundaries kept inline as breadcrumbs for reviewers; the +behaviours all ship in the same release. + +> **Phase trail (kept for reviewers):** Phase 11 introduced the +> ingestion pipeline; Phase 11.5 added token-aware chunking + PDF page +> header/footer dedup + structured ingestion notes; Phase 12 added +> markdown-aware splitting + secrets scrubbing + DOCX table preservation; +> Phase 12.5 added `--all-mask`. The four phases were consolidated into +> a single `v0.5.0` release. > Pair with [`forgelm audit`](data_audit.md) afterwards to surface > length-distribution / language / near-duplicate / PII metrics, and with @@ -20,7 +30,7 @@ pip install 'forgelm[ingestion]' ``` The extra brings in `pypdf`, `python-docx`, `ebooklib`, `beautifulsoup4`, -`langdetect`, and (since v0.5.1) the optional non-cryptographic `xxhash` +`langdetect`, and the optional non-cryptographic `xxhash` backend used by the simhash digest. They are optional because plain text + Markdown work without any of them. Importing the module does not pull these in unless the matching extractor is exercised; `xxhash`'s absence @@ -316,7 +326,7 @@ assistants, code-with-data prompts. ## Limitations - **OCR:** out of scope. Use external tooling — see the worked example below. -- **Tables / figures:** Since Phase 12 (`v0.5.2`), `_extract_docx()` +- **Tables / figures:** Since Phase 12 (consolidated into `v0.5.0`), `_extract_docx()` converts DOCX tables to **Markdown table syntax** (header + `---` separator + body rows) at extraction time, **before any chunking strategy runs** — so all strategies see the rendered Markdown, not a diff --git a/docs/reference/usage-tr.md b/docs/reference/usage-tr.md index 5662ac44..275f4745 100644 --- a/docs/reference/usage-tr.md +++ b/docs/reference/usage-tr.md @@ -110,7 +110,7 @@ forgelm --config my_config.yaml --generate-data Bu komut, eğitim başlamadan önce bir öğretmen modelden eğitim verisi üretmek için `synthetic` config bölümünü kullanır. Tüm sentetik veri seçenekleri için [Konfigürasyon Rehberi](configuration-tr.md)'ne bakın. -### Doküman Yutma (v0.5.0+; v0.5.1 token-aware; v0.5.2 markdown + secrets-mask) +### Doküman Yutma (v0.5.0) Ham PDF / DOCX / EPUB / TXT / Markdown'ı SFT'ye uygun JSONL'a dönüştürür. Opsiyonel bağımlılık: `pip install forgelm[ingestion]`. Ayrıntılar için [Doküman Yutma Rehberi](../guides/ingestion-tr.md). @@ -128,31 +128,31 @@ forgelm ingest ./scan.pdf --strategy sliding --chunk-size 1024 --overlap 128 \ # Yazmadan önce PII'yi maskele forgelm ingest ./customer_emails/ --pii-mask --output data/anon.jsonl -# Token-aware chunking (v0.5.1) — chunk'ları modelinizin vocab'ına göre boyutlandırır +# Token-aware chunking — chunk'ları modelinizin vocab'ına göre boyutlandırır forgelm ingest ./policies/ --recursive --output data/policies.jsonl \ --chunk-tokens 1024 --tokenizer "Qwen/Qwen2.5-7B-Instruct" -# v0.5.2: teknik wiki / README'ler için markdown-aware splitter +# Teknik wiki / README'ler için markdown-aware splitter forgelm ingest ./engineering_wiki/ --recursive --strategy markdown \ --output data/wiki.jsonl -# v0.5.2: chunk'lar JSONL'a inmeden önce credential'ları temizle +# Chunk'lar JSONL'a inmeden önce credential'ları temizle forgelm ingest ./mixed_corpus/ --secrets-mask --output data/clean.jsonl -# v0.5.2: secrets + PII maskelemeyi birleştir (önce secrets, sonra PII — +# Secrets + PII maskelemeyi birleştir (önce secrets, sonra PII — # örtüşen span'ler iki kez sayılmasın) forgelm ingest ./mixed_corpus/ --secrets-mask --pii-mask --output data/scrubbed.jsonl -# v0.5.3 (Faz 12.5): yukarıdaki kombinasyonun tek-flag kısayolu +# Yukarıdaki kombinasyonun tek-flag kısayolu forgelm ingest ./mixed_corpus/ --all-mask --output data/scrubbed.jsonl ``` -### Veri Seti Denetimi (v0.5.0+; v0.5.1 subcommand; v0.5.2 MinHash + quality + secrets) +### Veri Seti Denetimi (v0.5.0) CPU-only kalite + governance denetimi. `data_audit_report.json` üretir. Ayrıntılar için [Denetim Rehberi](../guides/data_audit-tr.md). ```bash -# Tek split (v0.5.1 subcommand) +# Tek split (subcommand) forgelm audit data/sft.jsonl --output ./audit/ # Çoklu split (train.jsonl / validation.jsonl / test.jsonl içeren dizin) @@ -164,17 +164,17 @@ forgelm audit data/ --verbose # Özel Hamming eşiği forgelm audit data/ --near-dup-threshold 5 -# v0.5.2: >50K satırlık korpuslar için MinHash LSH dedup +# >50K satırlık korpuslar için MinHash LSH dedup # Önce extra'yı kurun: pip install forgelm[ingestion-scale] forgelm audit data/large_corpus.jsonl --dedup-method minhash --jaccard-threshold 0.85 -# v0.5.2: opt-in heuristik kalite filtresi (Gopher/C4 stili) +# Opt-in heuristik kalite filtresi (Gopher/C4 stili) forgelm audit data/ --quality-filter -# v0.5.3 (Faz 12.5): audit'in yanında bir Google Croissant 1.0 dataset card emit et +# Audit'in yanında bir Google Croissant 1.0 dataset card emit et forgelm audit data/ --output ./audit/ --croissant -# v0.5.3 (Faz 12.5): regex PII detector üzerine Presidio NER katmanı ekle +# Regex PII detector üzerine Presidio NER katmanı ekle # (`[ingestion-pii-ml]` extra'sını gerektirir — person/organization/location kategorileri ekler) forgelm audit data/ --output ./audit/ --pii-ml @@ -185,7 +185,7 @@ forgelm audit data/sft.jsonl --output ./audit/ --output-format json forgelm --data-audit data/sft.jsonl --output ./audit/ ``` -Denetim şunları yakalar: split başına örnek sayısı + uzunluk dağılımı, top-3 dil tespiti, **LSH-banded** simhash near-duplicate oranı (Faz 11.5; uç eşiklerde brute-force fallback; v0.5.2 opsiyonel **MinHash LSH** yolu ekledi), cross-split sızıntı (sessiz train-test örtüşmesi), PII flag sayıları + **şiddet katmanları** (`pii_severity` bloğu her PII tipini critical / high / medium / low olarak puanlar ve bir `worst_tier` verdict yüzdürür), her zaman açık **credentials/secrets taraması** (`secrets_summary` — `forgelm.data_audit.SECRET_TYPES`'ta tanımlı dokuz aile: `aws_access_key`, `github_token`, `slack_token`, `openai_api_key`, `google_api_key`, `jwt`, `openssh_private_key` (tam BEGIN…END bloğu), `pgp_private_key` (tam BEGIN…END bloğu), `azure_storage_key`), ve opt-in **heuristik kalite filtresi** (`--quality-filter`, `quality_summary` bloğu üretir). +Denetim şunları yakalar: split başına örnek sayısı + uzunluk dağılımı, top-3 dil tespiti, **LSH-banded** simhash near-duplicate oranı (uç eşiklerde brute-force fallback; opsiyonel **MinHash LSH** yolu `--dedup-method minhash` ile), cross-split sızıntı (sessiz train-test örtüşmesi), PII flag sayıları + **şiddet katmanları** (`pii_severity` bloğu her PII tipini critical / high / medium / low olarak puanlar ve bir `worst_tier` verdict yüzdürür), her zaman açık **credentials/secrets taraması** (`secrets_summary` — `forgelm.data_audit.SECRET_TYPES`'ta tanımlı dokuz aile: `aws_access_key`, `github_token`, `slack_token`, `openai_api_key`, `google_api_key`, `jwt`, `openssh_private_key` (tam BEGIN…END bloğu), `pgp_private_key` (tam BEGIN…END bloğu), `azure_storage_key`), ve opt-in **heuristik kalite filtresi** (`--quality-filter`, `quality_summary` bloğu üretir). Trainer'ın `output_dir`'ünde `data_audit_report.json` mevcutsa, bulgular EU AI Act Madde 10 governance artifact'ında `data_audit` anahtarı altında otomatik olarak inline edilir. diff --git a/docs/reference/usage.md b/docs/reference/usage.md index eb850c7c..ca254fde 100644 --- a/docs/reference/usage.md +++ b/docs/reference/usage.md @@ -129,7 +129,7 @@ forgelm --config my_config.yaml --generate-data This uses the `synthetic` config section to generate training data from a teacher model before training begins. See the [Configuration Guide](configuration.md) for all synthetic data options. -### Document Ingestion (v0.5.0+; token-aware in v0.5.1; markdown + secrets-mask in v0.5.2) +### Document Ingestion (v0.5.0) Convert raw PDF / DOCX / EPUB / TXT / Markdown into SFT-ready JSONL. Optional dep: `pip install forgelm[ingestion]`. See [Ingestion Guide](../guides/ingestion.md). @@ -147,30 +147,30 @@ forgelm ingest ./scan.pdf --strategy sliding --chunk-size 1024 --overlap 128 \ # Mask PII before writing forgelm ingest ./customer_emails/ --pii-mask --output data/anon.jsonl -# Token-aware chunking (v0.5.1) — sizes chunks against your model's vocab +# Token-aware chunking — sizes chunks against your model's vocab forgelm ingest ./policies/ --recursive --output data/policies.jsonl \ --chunk-tokens 1024 --tokenizer "Qwen/Qwen2.5-7B-Instruct" -# v0.5.2: markdown-aware splitter for technical wikis / READMEs +# Markdown-aware splitter for technical wikis / READMEs forgelm ingest ./engineering_wiki/ --recursive --strategy markdown \ --output data/wiki.jsonl -# v0.5.2: scrub credentials before chunks land in the JSONL +# Scrub credentials before chunks land in the JSONL forgelm ingest ./mixed_corpus/ --secrets-mask --output data/clean.jsonl -# v0.5.2: combine secrets + PII masking (secrets first to avoid double-counting) +# Combine secrets + PII masking (secrets first to avoid double-counting) forgelm ingest ./mixed_corpus/ --secrets-mask --pii-mask --output data/scrubbed.jsonl -# v0.5.3 (Phase 12.5): one-flag shorthand for the same combination above +# One-flag shorthand for the same combination above forgelm ingest ./mixed_corpus/ --all-mask --output data/scrubbed.jsonl ``` -### Dataset Audit (v0.5.0+; subcommand promoted in v0.5.1; MinHash + quality + secrets in v0.5.2) +### Dataset Audit (v0.5.0) CPU-only quality + governance audit. Produces `data_audit_report.json`. See [Audit Guide](../guides/data_audit.md). ```bash -# Single split (v0.5.1 subcommand) +# Single split (subcommand) forgelm audit data/sft.jsonl --output ./audit/ # Multi-split directory (train.jsonl / validation.jsonl / test.jsonl) @@ -182,16 +182,16 @@ forgelm audit data/ --verbose # Custom Hamming threshold for simhash near-duplicate detection forgelm audit data/ --near-dup-threshold 5 -# v0.5.2: MinHash LSH dedup for >50K-row corpora (needs `[ingestion-scale]` extra) +# MinHash LSH dedup for >50K-row corpora (needs `[ingestion-scale]` extra) forgelm audit data/large_corpus.jsonl --dedup-method minhash --jaccard-threshold 0.85 -# v0.5.2: opt-in heuristic quality filter (Gopher/C4 style) +# Opt-in heuristic quality filter (Gopher/C4 style) forgelm audit data/ --quality-filter -# v0.5.3 (Phase 12.5): emit a Google Croissant 1.0 dataset card alongside the audit +# Emit a Google Croissant 1.0 dataset card alongside the audit forgelm audit data/ --output ./audit/ --croissant -# v0.5.3 (Phase 12.5): layer Presidio NER on top of the regex PII detector +# Layer Presidio NER on top of the regex PII detector # (needs `[ingestion-pii-ml]` extra — adds person/organization/location categories) forgelm audit data/ --output ./audit/ --pii-ml @@ -202,7 +202,7 @@ forgelm audit data/sft.jsonl --output ./audit/ --output-format json forgelm --data-audit data/sft.jsonl --output ./audit/ ``` -The audit captures: per-split sample count + length distribution, top-3 language detection, **LSH-banded** simhash near-duplicate rate (Phase 11.5; brute-force fallback at edge thresholds; v0.5.2 added optional **MinHash LSH** path), cross-split leakage (silent train-test overlap), PII flag counts with **severity tiers**, **always-on credentials/secrets scan** (`secrets_summary` — nine families per `forgelm.data_audit.SECRET_TYPES`: `aws_access_key`, `github_token`, `slack_token`, `openai_api_key`, `google_api_key`, `jwt`, `openssh_private_key` (full BEGIN…END block), `pgp_private_key` (full BEGIN…END block), `azure_storage_key`), and an opt-in **heuristic quality filter** (Gopher/C4 style) that adds a `quality_summary` block. +The audit captures: per-split sample count + length distribution, top-3 language detection, **LSH-banded** simhash near-duplicate rate (brute-force fallback at edge thresholds; optional **MinHash LSH** path via `--dedup-method minhash`), cross-split leakage (silent train-test overlap), PII flag counts with **severity tiers**, **always-on credentials/secrets scan** (`secrets_summary` — nine families per `forgelm.data_audit.SECRET_TYPES`: `aws_access_key`, `github_token`, `slack_token`, `openai_api_key`, `google_api_key`, `jwt`, `openssh_private_key` (full BEGIN…END block), `pgp_private_key` (full BEGIN…END block), `azure_storage_key`), and an opt-in **heuristic quality filter** (Gopher/C4 style) that adds a `quality_summary` block. When `data_audit_report.json` is present in the trainer's `output_dir` at training time, its findings are inlined under the `data_audit` key of the EU AI Act Article 10 governance artifact automatically. diff --git a/docs/roadmap-tr.md b/docs/roadmap-tr.md index 55fe8290..dc5b0578 100644 --- a/docs/roadmap-tr.md +++ b/docs/roadmap-tr.md @@ -9,24 +9,26 @@ | ✅ Tamam | [Faz 1-9](roadmap/completed-phases.md) | SOTA iyileştirmeleri, değerlendirme, güvenilirlik, kurumsal entegrasyon, ekosistem, hizalama stack'i, güvenlik, EU AI Act uyumluluğu (Madde 9-17 + Ek IV), gelişmiş güvenlik zekası | | ✅ Tamam | [Faz 10 — Post-Training Tamamlama](roadmap/phase-10-post-training.md) | `inference.py`, `chat`, `export` (GGUF), `--fit-check`, `deploy` — `v0.4.0` | | ✅ Tamam | [Faz 10.5 — Quickstart Katmanı ve Onboarding](roadmap/phase-10-5-quickstart.md) | `forgelm quickstart