release(v0.5.0): Phase 11 + 11.5 + 12 + 12.5 consolidated release

README.md

@@ -27,13 +27,16 @@
 - **LLM-as-Judge**: API-based (OpenAI) or local model scoring for quality assessment
 - **Auto-Revert**: Opt-in (`evaluation.auto_revert: true`) — automatically discards models that fail loss, benchmark, or safety thresholds before artifacts are written
 
-### Document Ingestion & Data Audit (v0.5.0 — polished in v0.5.1, matured in v0.5.2)
-- **Multi-Format Ingestion**: `forgelm ingest ./policies/ --recursive --output data/policies.jsonl` — turns raw PDF / DOCX / EPUB / TXT / Markdown into the SFT-ready JSONL the trainer accepts. Optional dep: `pip install forgelm[ingestion]`. v0.5.2 added a **Markdown-aware splitter** (`--strategy markdown`) and DOCX table preservation in Markdown table syntax.
-- **Chunking Strategies**: `paragraph` (default; preserves boundaries), `sliding` (fixed window with overlap), or `markdown` (v0.5.2 — heading-aware splitter that keeps fenced code blocks atomic and inlines a heading breadcrumb at the top of each chunk for SFT context). A `semantic` strategy is reserved for a follow-up phase — the implementation in `forgelm.ingestion` raises `NotImplementedError` today and the CLI hides it from `--strategy` choices. Token-aware mode in v0.5.1: `--chunk-tokens 1024 --tokenizer Qwen/Qwen2.5-7B-Instruct` sizes chunks against your model's actual vocabulary.
+### Document Ingestion & Data Audit (v0.5.0 — Phases 11 + 11.5 + 12 + 12.5 consolidated)
+- **Multi-Format Ingestion**: `forgelm ingest ./policies/ --recursive --output data/policies.jsonl` — turns raw PDF / DOCX / EPUB / TXT / Markdown into the SFT-ready JSONL the trainer accepts. Optional dep: `pip install forgelm[ingestion]`. Includes a **Markdown-aware splitter** (`--strategy markdown`) and DOCX table preservation in Markdown table syntax.
+- **Chunking Strategies**: `paragraph` (default; preserves boundaries), `sliding` (fixed window with overlap), or `markdown` (heading-aware splitter that keeps fenced code blocks atomic and inlines a heading breadcrumb at the top of each chunk for SFT context). A `semantic` strategy is reserved for a follow-up phase — the implementation in `forgelm.ingestion` raises `NotImplementedError` today and the CLI hides it from `--strategy` choices. Token-aware mode: `--chunk-tokens 1024 --tokenizer Qwen/Qwen2.5-7B-Instruct` sizes chunks against your model's actual vocabulary.
 - **PII Masking on Ingest**: `--pii-mask` redacts emails, phones, credit cards (Luhn-validated), IBAN, and national IDs (TR / DE / FR / US-SSN) before chunks land in the JSONL.
-- **PDF Page Header/Footer Dedup (v0.5.1)**: Lines that recur on ≥ 70 % of PDF pages (watermarks, page numbers, copyright lines) are stripped automatically — the audit's near-duplicate counts stop misfiring on long policy / book PDFs.
-- **Dataset Audit**: `forgelm audit data/sft.jsonl --output ./audit/` — produces `data_audit_report.json` with sample count, length distribution, top-3 language detection, LSH-banded near-duplicate rate (`O(n × k)` typical case, exact recall at the default Hamming threshold; v0.5.2 added optional **MinHash LSH** via `--dedup-method minhash` for >50K-row corpora), cross-split leakage check, null/empty rate, and PII flag counts with **severity tiers** (critical / high / medium / low + worst-tier verdict). v0.5.2 also added an always-on **secrets/credential scan** covering the nine families enumerated in `forgelm.data_audit.SECRET_TYPES` — AWS access keys, GitHub tokens, Slack tokens, OpenAI API keys, Google API keys, JWTs, full OpenSSH private-key blocks, full PGP private-key blocks, and Azure storage connection strings — and an opt-in **heuristic quality filter** (`--quality-filter`). CPU-only; streaming JSONL reader keeps memory bounded on multi-million-row splits; feeds EU AI Act Article 10 governance artifact automatically when present at training time. Legacy `--data-audit` flag still works as a deprecation alias.
-- **Secrets-Aware Ingest (v0.5.2)**: `forgelm ingest … --secrets-mask` scrubs credentials before chunks land in the JSONL — fine-tuning on text containing real API keys memorises them at training time. Pairs with `--pii-mask`; secrets run first so combined detectors don't double-count overlapping spans.
+- **PDF Page Header/Footer Dedup**: Lines that recur on ≥ 70 % of PDF pages (watermarks, page numbers, copyright lines) are stripped automatically — the audit's near-duplicate counts stop misfiring on long policy / book PDFs.
+- **Dataset Audit**: `forgelm audit data/sft.jsonl --output ./audit/` — produces `data_audit_report.json` with sample count, length distribution, top-3 language detection, LSH-banded near-duplicate rate (`O(n × k)` typical case, exact recall at the default Hamming threshold; optional **MinHash LSH** via `--dedup-method minhash` for >50K-row corpora), cross-split leakage check, null/empty rate, and PII flag counts with **severity tiers** (critical / high / medium / low + worst-tier verdict). Always-on **secrets/credential scan** covers the nine families enumerated in `forgelm.data_audit.SECRET_TYPES` — AWS access keys, GitHub tokens, Slack tokens, OpenAI API keys, Google API keys, JWTs, full OpenSSH private-key blocks, full PGP private-key blocks, and Azure storage connection strings — and an opt-in **heuristic quality filter** (`--quality-filter`). CPU-only; streaming JSONL reader keeps memory bounded on multi-million-row splits; feeds EU AI Act Article 10 governance artifact automatically when present at training time. Legacy `--data-audit` flag still works as a deprecation alias.
+- **Secrets-Aware Ingest**: `forgelm ingest … --secrets-mask` scrubs credentials before chunks land in the JSONL — fine-tuning on text containing real API keys memorises them at training time. Pairs with `--pii-mask`; secrets run first so combined detectors don't double-count overlapping spans. `--all-mask` is a one-flag shorthand for both.
+- **Optional ML-NER PII**: `forgelm audit --pii-ml [--pii-ml-language LANG]` layers [Presidio](https://github.com/microsoft/presidio) NER on top of the regex detector via the optional `[ingestion-pii-ml]` extra **plus a separate `python -m spacy download en_core_web_lg`** step (or the matching model for the chosen language). Adds `person` / `organization` / `location` categories into the same `pii_summary` / `pii_severity` blocks under disjoint category names.
+- **Croissant 1.0 dataset card**: `forgelm audit --croissant` emits a [Google Croissant 1.0](http://mlcommons.org/croissant/) dataset card under the report's `croissant` key so the same JSON file doubles as both the EU AI Act Article 10 governance artifact and a Croissant-consumer dataset card.
+- **Wizard "audit first"**: when the wizard resolves a JSONL (typed or produced by `forgelm ingest`) it offers to run `forgelm audit` inline and prints the verdict before continuing — closes the BYOD audit loop end-to-end.
 
 ### Quickstart Layer (v0.4.5)
 - **One-Command Templates**: `forgelm quickstart customer-support` — 4 bundled templates (SFT customer-support, code-assistant, medical-qa-tr, GRPO math-reasoning) plus a bring-your-own-data domain-expert scaffold. Auto-downsizes models on small GPUs.
@@ -72,7 +75,7 @@ forgelm quickstart customer-support           # render config + train + chat
 forgelm quickstart code-assistant --dry-run   # render config only
 forgelm quickstart medical-qa-tr --model your-org/your-model  # override
 
-# Have raw docs? Ingest them first (v0.5.0+; v0.5.1 added token-aware sizing)
+# Have raw docs? Ingest them first (v0.5.0; supports token-aware sizing)
 pip install -e ".[ingestion]"
 forgelm ingest ./policies/ --recursive --output data/policies.jsonl
 forgelm audit data/policies.jsonl --output ./audit/   # `forgelm --data-audit ...` still works as legacy alias
@@ -172,6 +175,7 @@ pip install "forgelm[merging]"           # mergekit model merging
 pip install "forgelm[ingestion]"         # PDF/DOCX/EPUB/Markdown → JSONL + langdetect + xxhash
 pip install "forgelm[ingestion-scale]"   # MinHash LSH dedup (datasketch) for >50K-row corpora
 pip install "forgelm[ingestion-secrets]" # detect-secrets scanner for SFT corpora (falls back to regex if absent)
+pip install "forgelm[ingestion-pii-ml]"  # Presidio ML-NER for person/organization/location PII (Phase 12.5; ALSO needs `python -m spacy download en_core_web_lg`)
 pip install "forgelm[export]"            # GGUF export via llama-cpp-python
 pip install "forgelm[chat]"              # Rich terminal rendering for `forgelm chat`
 ```

docs/guides/data_audit-tr.md

@@ -2,11 +2,11 @@
 
 `forgelm audit PATH` bir JSONL veri setini analiz eder ve kalite,
 governance ile PII sinyallerini kapsayan bir `data_audit_report.json`
-üretir. Faz 11 (`v0.5.0`'da tanıtıldı) altyapıyı, Faz 11.5 (`v0.5.1`)
+üretir. Faz 11 altyapıyı, Faz 11.5 (`v0.5.0`'da birleşti)
 ise top-level flag'i first-class subcommand'a yükseltti, LSH-banded
 near-duplicate tespitini, streaming JSONL okuyucusunu, PII şiddet
 katmanlarını, atomik disk yazımı ve verbose-by-default kısaltma
-politikasını ekledi. **Faz 12 (`v0.5.2`)** opt-in MinHash LSH dedup
+politikasını ekledi. **Faz 12 (`v0.5.0`'da birleşti)** opt-in MinHash LSH dedup
 yöntemini (`--dedup-method minhash`), her zaman çalışan
 code/credential leakage taramasını (`secrets_summary`), ve opt-in
 heuristic kalite filtresini (`--quality-filter`) ekledi.
@@ -268,7 +268,7 @@ forgelm ingest ./policies/ --recursive --output data/policies.jsonl --secrets-ma
 
 Opsiyonel / forward-compatibility: `[ingestion-secrets]` extra'sı bir
 `detect-secrets>=1.5.0` bağımlılığı tanımlar ama bu **bir sonraki sürüm
-için ayrılmıştır**. v0.5.2 itibarıyla `forgelm.data_audit.detect_secrets()`
+için ayrılmıştır**. v0.5.0 itibarıyla `forgelm.data_audit.detect_secrets()`
 yalnızca yukarıdaki regex setine dayanır; extra'yı kurmak bugün audit
 davranışını değiştirmez. Extra, `forgelm[ingestion-secrets]` pin'leyen
 operatörlerin entegrasyon geldiğinde forward-compatible olabilmesi
@@ -321,6 +321,115 @@ install ile uyumlu kalır.
 
 ---
 
+### ML-NER PII adaptörü — `--pii-ml` (Faz 12.5, opt-in)
+
+Varsayılan regex detector, EU AI Act Madde 10'un önemsediği yapılandırılmış
+identifier'ları (email, telefon, IBAN, kredi kartı, ulusal kimlik) kapsar.
+Faz 12.5, regex'in doğal olarak kaçırdığı yapılandırılmamış identifier
+kategorilerini ekleyen opt-in **Presidio** ([microsoft/presidio](https://github.com/microsoft/presidio))
+adaptörünü ML-NER olarak üstüne ekler: `person`, `organization`,
+`location`.
+
+```bash
+pip install 'forgelm[ingestion-pii-ml]'
+forgelm audit data/ --output ./audit/ --pii-ml
+```
+
+Yeni kategoriler mevcut `pii_summary` ve `pii_severity` bloklarına
+disjoint isimlerle birleşir, bu yüzden regex baseline ML sinyaliyle
+yan yana görünür kalır:
+
+```json
+{
+  "pii_summary": {
+    "email": 12,
+    "phone": 3,
+    "person": 47,         // ← Presidio
+    "organization": 18,   // ← Presidio
+    "location": 9         // ← Presidio
+  },
+  "pii_severity": {
+    "by_tier": {"critical": 0, "high": 0, "medium": 59, "low": 30},
+    "by_type": {
+      "email": {"count": 12, "tier": "medium"},
+      "person": {"count": 47, "tier": "medium"},
+      "organization": {"count": 18, "tier": "low"}
+    },
+    "worst_tier": "medium"
+  }
+}
+```
+
+Şiddet ataması `forgelm.data_audit.PII_ML_SEVERITY`'de yaşar:
+`person → medium`, `organization → low`, `location → low`. NER
+false-positive oranları regex-anchored detection'dan materyal olarak
+yüksektir — bu yüzden ML katmanları regex'in `critical` / `high`
+katlarının altında bilinçli konumlandırılır. Bir ML bulgusunu sert
+bir gate olarak değerlendirmeden önce satır bazında span'leri inceleyin.
+
+**İki adımlı kurulum.** `presidio-analyzer` paketi spaCy NER modelini
+transitively içermez — tek seferlik ayrı bir indirme:
+
+```bash
+pip install 'forgelm[ingestion-pii-ml]'
+python -m spacy download en_core_web_lg   # ~ 50 MB; gerekli
+```
+
+Model yoksa `forgelm audit --pii-ml` hiçbir satır taranmadan **önce**
+aynı install hint ile `ImportError` raise eder
+(`forgelm.data_audit._require_presidio`'daki pre-flight, spaCy'nin
+`OSError`'unu yakalayıp typed install error'a dönüştürür). Bu fail-loud
+davranış bilinçli — önceki prototiplerde model eksikken sıfır ML
+bulgusu sessizce raporlanıyordu, opt-in bir detector için kritik bir
+compliance kör noktası.
+
+İngilizce olmayan korpuslar için eşleşen spaCy modelini kurup dil
+kodunu geçirin:
+
+```bash
+python -m spacy download xx_ent_wiki_sm        # çok dilli, daha küçük
+forgelm audit data/ --pii-ml --pii-ml-language xx
+```
+
+Adaptör yalnızca opt-in; `--pii-ml` olmadan audit zero-extra-deps
+regex yolunda kalır.
+
+---
+
+### Croissant 1.0 dataset card — `--croissant` (Faz 12.5, opt-in)
+
+`--croissant`, `data_audit_report.json` içinde yeni bir top-level
+`croissant` bloğunu [Google Croissant 1.0](https://mlcommons.org/croissant/)
+dataset card'ı (`@type: sc:Dataset`) ile doldurur. Card, kanonik
+`mlcommons.org/croissant/1.0` context'iyle uyumludur — bu yüzden
+Croissant farkındalıklı consumer'lar (HuggingFace dataset cards,
+MLCommons referans loader'ları, Croissant validator) bloğu hiçbir
+modifikasyon olmadan parse edebilir.
+
+```bash
+forgelm audit data/ --output ./audit/ --croissant
+```
+
+Card şunları taşır:
+
+* dataset seviyesinde kimlik (`name`, `description`, `datePublished`,
+  `url`) — `version` bilinçli olarak çıkarılmıştır, audit'in dataset
+  sürümü için first-class kanıtı yoktur; card'ı yayınlayan operatörler
+  `version`'ı `license` / `citeAs` gibi elle düzenler,
+* her JSONL split'i için bir `cr:FileObject` (Croissant consumer'ı
+  altta yatan dosyaları bulabilsin diye),
+* her split için audit'in column-detection katmanından türetilen
+  `cr:Field` girdileriyle bir `cr:RecordSet`.
+
+Flag kapalıyken blok boştur — mevcut consumer'lar byte-eşdeğer çıktı
+görür, `secrets_summary` / `quality_summary`'nin koyduğu emsalin
+aynısı. Card'ı HuggingFace / MLCommons'a yayınlamak isteyen operatörler
+audit'in birinci sınıf delili olmayan ek Croissant alanlarını
+(`license`, `citeAs`, `keywords`) audit'i yeniden çalıştırmadan
+elle düzenleyebilir.
+
+---
+
 ## Layout gereksinimleri
 
 | Girdi şekli | Ne elde edersin |
@@ -373,6 +482,9 @@ forgelm audit PATH \
   [--dedup-method {simhash,minhash}] \
   [--jaccard-threshold X] \
   [--quality-filter] \
+  [--pii-ml] \
+  [--pii-ml-language LANG] \
+  [--croissant] \
   [--output-format {text,json}] \
   [--quiet | --log-level {DEBUG,INFO,WARNING,ERROR}]
 ```