diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 057dd869..385949d5 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -71,6 +71,20 @@ jobs: forgelm --config "/tmp/qs-$tpl.yaml" --dry-run done + - name: Ingestion + audit smoke test (Phase 11) + run: | + # Minimal end-to-end: TXT in → JSONL out → audit report out. + # Plain TXT path doesn't need the [ingestion] extra; the audit + # module is pure stdlib. Catches CLI wiring drift without paying + # for pypdf / langdetect installs. + mkdir -p /tmp/p11 + echo "Article 10 governs data quality." > /tmp/p11/sample.txt + echo "Section two: representativeness, traceability, bias review." > /tmp/p11/sample2.txt + forgelm ingest /tmp/p11/ --recursive --output /tmp/p11/out.jsonl + test -s /tmp/p11/out.jsonl + forgelm --data-audit /tmp/p11/out.jsonl --output /tmp/p11/audit/ + test -s /tmp/p11/audit/data_audit_report.json + wheel-install-smoke: # ------------------------------------------------------------------ # This is the only test that catches package_data globs being broken diff --git a/CHANGELOG.md b/CHANGELOG.md index 84b5d5e5..abed448d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,37 @@ All notable changes to ForgeLM are documented here. ## [Unreleased] +### Added + +**Document Ingestion & Data Audit (Phase 11)** — bridges raw enterprise corpora (legal, medical, policy manuals) to ForgeLM's training data format and surfaces governance signals before training starts. + +- **`forgelm/ingestion.py`** + **`forgelm ingest`** subcommand: + - Multi-format extractors: PDF (`pypdf`), DOCX (`python-docx`), EPUB (`ebooklib` + `beautifulsoup4`), TXT, Markdown. + - Two chunking strategies: `paragraph` (default; greedy, never splits a paragraph) and `sliding` (fixed window with `--overlap`). `semantic` raises `NotImplementedError` and is reserved for a follow-up phase. + - Output is `{"text": "..."}` JSONL — recognized as pre-formatted SFT input by `forgelm/data.py` without further preprocessing. + - `--recursive` walks directory trees; unsupported extensions are skipped silently, supported files with no extractable text skip with a warning. + - `--pii-mask` redacts detected PII spans before chunks land in the JSONL (shared regex set with the audit module). + - OCR is intentionally out of scope; scanned PDFs without a text layer warn and produce zero chunks. + +- **`forgelm/data_audit.py`** + **`forgelm --data-audit`** top-level flag: + - Per-split metrics: sample count, column schema, text length distribution (`min/max/mean/p50/p95`), null/empty rate, top-3 language detection (best-effort via `langdetect`). + - 64-bit simhash near-duplicate detection within each split; Hamming-distance threshold 3 ≈ 95% similarity (canonical web-page-dedup setting). + - Cross-split overlap report — guards against silent train-test leakage that destroys benchmark fidelity. + - PII regex set (`email`, `phone`, `credit_card` Luhn-validated, `iban`, `tr_id` checksum-validated, `de_id`, `fr_ssn`, `us_ssn`); per-split + aggregate counts. + - Layout: single `.jsonl` file → treated as `train`; directory containing `train.jsonl` / `validation.jsonl` / `test.jsonl` (any subset) auto-discovered. + - Writes `data_audit_report.json` under `--output` (default `./audit/`); `--output-format json` mirrors the report on stdout for CI/CD pipelines. + - CPU-only; no GPU, no network. + +- **EU AI Act Article 10 integration** — `generate_data_governance_report` now inlines `data_audit_report.json` under the `data_audit` key when present in the trainer's `output_dir`. Compliance bundle becomes a single self-contained document instead of a pointer. + +- **`pyproject.toml` `[ingestion]` extra** — `pypdf`, `python-docx`, `ebooklib`, `beautifulsoup4`, `langdetect`. Cross-platform, no native compilation. + +- **Tests** — `tests/test_ingestion.py` (TXT path + chunking strategies; PDF round-trip skips when `pypdf` missing) and `tests/test_data_audit.py` (PII regex + Luhn / TC Kimlik validators, simhash properties, end-to-end audit on file + split-keyed directory layouts, governance integration). All GPU/network-free. + +- **Documentation** — new guides at `docs/guides/ingestion.md` and `docs/guides/data_audit.md`; README feature section, CLI epilog, install matrix, and roadmap status updated. + +--- + ## [0.4.5] — 2026-04-26 ### Added @@ -71,7 +102,7 @@ All notable changes to ForgeLM are documented here. --- -## [0.4.0] — 2026-04-25 +## [0.4.0] — 2026-04-26 ### Added diff --git a/README.md b/README.md index e4017b02..31ec0828 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,6 @@ # ForgeLM +[![PyPI](https://img.shields.io/pypi/v/forgelm.svg)](https://pypi.org/project/forgelm/) [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/) [![CI](https://github.com/cemililik/ForgeLM/actions/workflows/ci.yml/badge.svg)](https://github.com/cemililik/ForgeLM/actions/workflows/ci.yml) @@ -26,6 +27,12 @@ - **LLM-as-Judge**: API-based (OpenAI) or local model scoring for quality assessment - **Auto-Revert**: Automatically discard models that fail loss, benchmark, or safety thresholds +### Document Ingestion & Data Audit (v0.5.0) +- **Multi-Format Ingestion**: `forgelm ingest ./policies/ --recursive --output data/policies.jsonl` — turns raw PDF / DOCX / EPUB / TXT / Markdown into the SFT-ready JSONL the trainer accepts. Optional dep: `pip install forgelm[ingestion]`. +- **Chunking Strategies**: `paragraph` (default; preserves boundaries) or `sliding` (fixed window with overlap). +- **PII Masking on Ingest**: `--pii-mask` redacts emails, phones, credit cards (Luhn-validated), IBAN, and national IDs (TR / DE / FR / US-SSN) before chunks land in the JSONL. +- **Dataset Audit**: `forgelm --data-audit data/sft.jsonl --output ./audit/` — produces `data_audit_report.json` with sample count, length distribution, top-3 language detection, simhash near-duplicate rate, cross-split leakage check, null/empty rate, and PII flag counts. CPU-only; feeds EU AI Act Article 10 governance artifact automatically when present at training time. + ### Quickstart Layer (v0.4.5) - **One-Command Templates**: `forgelm quickstart customer-support` — bundled templates for SFT, code, BYOD domain expert, Turkish medical Q&A, and GRPO math reasoning. Auto-downsizes models on small GPUs. - **Conservative Defaults**: Every template ships QLoRA 4-bit, rank=8, batch=1, gradient checkpointing on — designed to run on a single 12 GB GPU. @@ -63,6 +70,11 @@ forgelm quickstart customer-support # render config + train + chat forgelm quickstart code-assistant --dry-run # render config only forgelm quickstart medical-qa-tr --model your-org/your-model # override +# Have raw docs? Ingest them first (v0.5.0+) +pip install -e ".[ingestion]" +forgelm ingest ./policies/ --recursive --output data/policies.jsonl +forgelm --data-audit data/policies.jsonl --output ./audit/ + # Or generate config interactively forgelm --wizard @@ -90,6 +102,8 @@ See the [Quick Start Guide](docs/guides/quickstart.md) for a complete walkthroug | Guide | Description | |-------|-------------| | [Quick Start](docs/guides/quickstart.md) | First fine-tuned model in 5 minutes | +| [Document Ingestion](docs/guides/ingestion.md) ([Türkçe](docs/guides/ingestion-tr.md)) | Raw PDF/DOCX/EPUB → SFT-ready JSONL | +| [Dataset Audit](docs/guides/data_audit.md) ([Türkçe](docs/guides/data_audit-tr.md)) | Length, language, dedup, cross-split leakage, PII | | [Alignment (DPO/SimPO/KTO/GRPO)](docs/guides/alignment.md) | Complete post-training stack | | [CI/CD Pipeline Integration](docs/guides/cicd_pipeline.md) | GitHub Actions, GitLab CI, Docker | | [Enterprise Deployment](docs/guides/enterprise_deployment.md) | Docker, air-gapped, multi-GPU | @@ -108,11 +122,24 @@ See the [Quick Start Guide](docs/guides/quickstart.md) for a complete walkthroug ## Notebooks +Each notebook is runnable in Colab with a free T4 GPU. + +**Getting started** - [Quick Start — SFT Fine-Tuning](notebooks/quickstart_sft.ipynb) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cemililik/ForgeLM/blob/main/notebooks/quickstart_sft.ipynb) + +**Alignment methods** (post-SFT preference / RL) - [DPO Preference Alignment](notebooks/dpo_alignment.ipynb) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cemililik/ForgeLM/blob/main/notebooks/dpo_alignment.ipynb) - [KTO Binary Feedback](notebooks/kto_binary_feedback.ipynb) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cemililik/ForgeLM/blob/main/notebooks/kto_binary_feedback.ipynb) - [GRPO Reasoning RL](notebooks/grpo_reasoning.ipynb) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cemililik/ForgeLM/blob/main/notebooks/grpo_reasoning.ipynb) -- [Multi-Dataset Training](notebooks/multi_dataset.ipynb) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cemililik/ForgeLM/blob/main/notebooks/multi_dataset.ipynb) + +**Advanced training** +- [Multi-Dataset Training](notebooks/multi_dataset.ipynb) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cemililik/ForgeLM/blob/main/notebooks/multi_dataset.ipynb) — mix multiple datasets with configurable ratios. +- [GaLore Memory Optimization](notebooks/galore_memory_optimization.ipynb) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cemililik/ForgeLM/blob/main/notebooks/galore_memory_optimization.ipynb) — full-parameter training via gradient low-rank projection (LoRA alternative). +- [Synthetic Data Pipeline](notebooks/synthetic_data_training.ipynb) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cemililik/ForgeLM/blob/main/notebooks/synthetic_data_training.ipynb) — teacher-to-student distillation (API / local / pre-generated backends). + +**Post-training & safety** +- [Post-Training Workflow](notebooks/post_training_workflow.ipynb) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cemililik/ForgeLM/blob/main/notebooks/post_training_workflow.ipynb) — end-to-end Phase 10 toolchain: `--fit-check` → `chat` → `export` (GGUF) → `deploy`. +- [Safety Evaluation & Red-Teaming](notebooks/safety_evaluation.ipynb) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cemililik/ForgeLM/blob/main/notebooks/safety_evaluation.ipynb) — 140 adversarial prompts × 6 categories (Llama Guard). --- @@ -137,6 +164,9 @@ pip install -e ".[eval]" # lm-evaluation-harness benchmarks pip install -e ".[tracking]" # W&B experiment tracking pip install -e ".[distributed]" # DeepSpeed multi-GPU pip install -e ".[merging]" # mergekit model merging +pip install -e ".[ingestion]" # PDF/DOCX/EPUB → JSONL + langdetect +pip install -e ".[export]" # GGUF export via llama-cpp-python +pip install -e ".[chat]" # Rich terminal rendering for `forgelm chat` pip install -e ".[dev]" # pytest, ruff (development) ``` @@ -208,7 +238,7 @@ forgelm/ configs/deepspeed/ # ZeRO-2, ZeRO-3, ZeRO-3+Offload presets notebooks/ # Colab-ready Jupyter notebooks -tests/ # 430 passed (+34 skipped) across 30 test files +tests/ # pytest suite spanning every module (run with `pytest tests/`) docs/guides/ # Quickstart, alignment, CI/CD, enterprise, safety guides ``` diff --git a/docs/guides/data_audit-tr.md b/docs/guides/data_audit-tr.md new file mode 100644 index 00000000..55dfd8a3 --- /dev/null +++ b/docs/guides/data_audit-tr.md @@ -0,0 +1,218 @@ +# Veri Seti Denetimi Rehberi + +`forgelm --data-audit` bir JSONL veri setini analiz eder ve kalite, +governance ile PII sinyallerini kapsayan bir `data_audit_report.json` +üretir. Faz 11; `v0.5.0`'da tanıtıldı. Trainer'ın `output_dir`'ünde +mevcutsa, rapor EU AI Act Madde 10 veri governance artifact'ına +otomatik olarak beslenir. + +--- + +## Çalıştırma + +```bash +# Tek split (`train` olarak değerlendirilir) +forgelm --data-audit data/sft.jsonl --output ./audit/ + +# Çoklu split: train.jsonl / validation.jsonl / test.jsonl içeren dizin +forgelm --data-audit data/ --output ./audit/ +``` + +`--output` varsayılan olarak `./audit/`'tir. Dizin yoksa oluşturulur; +**tam** `data_audit_report.json` her zaman oraya yazılır. Stdout +varsayılan olarak insan-okunabilir özet gösterir; `--output-format json` +geçince stdout'a **özet** JSON zarfı (üst seviye metrikler + rapor yolu ++ notlar) düşer — tam rapor yine `--output` altında diskte kalır. CI/CD +tüketicileri her detayı istediğinde stdout özetini parse etmek yerine +dosyayı `report_path`'tan slurp etmeli. + +GPU gerekmiyor. Ağ çağrısı yok. CPU-only. + +--- + +## Ne elde edersin + +### Split başına metrikler + +```json +{ + "splits": { + "train": { + "sample_count": 1240, + "columns": ["text"], + "text_length": {"min": 32, "max": 4096, "mean": 1834.2, "p50": 1900, "p95": 3580}, + "null_or_empty_count": 3, + "null_or_empty_rate": 0.0024, + "languages_top3": [ + {"code": "tr", "count": 950}, + {"code": "en", "count": 240}, + {"code": "de", "count": 50} + ], + "simhash_distinct": 1180, + "near_duplicate_pairs": 60, + "pii_counts": {"email": 18, "phone": 4} + } + } +} +``` + +### Cross-split örtüşmesi + +```json +{ + "cross_split_overlap": { + "hamming_threshold": 3, + "pairs": { + "train__test": { + "leaked_rows_in_train": 7, + "leak_rate_train": 0.0056, + "leaked_rows_in_test": 7, + "leak_rate_test": 0.7 + } + } + } +} +``` + +Audit leak rate'i **her iki yönde** de raporlar çünkü birbirinden farklı +hikâyeler anlatırlar. 1240 train + 10 test satırında 7'sinin sızdığı bir +durumda `leak_rate_train = 7/1240 = %0.56` önemsiz görünür ama +`leak_rate_test = 7/10 = %70` benchmark güvenirliğini fiilen yok eden +metriktir. Her zaman küçük tarafın oranını okuyun — test bütünlüğünün +sessiz katili odur. + +### PII özeti + +```json +{ + "pii_summary": { + "email": 18, + "phone": 4, + "credit_card": 1, + "tr_id": 2 + } +} +``` + +Her satırın metin payload'ı regex ile taranır; kredi kartları Luhn +doğrulaması, TR Kimlik No'ları TC Kimlik checksum'ından geçirilir. +Diğer kategoriler regex şekli üzerinden yüzeylenir — false positive'ler +kasıtlıdır. Veri setini paylaşmadan önce `forgelm ingest --pii-mask` +ile (veya kendi ön işlemenizde) maskeleyin. + +**Pattern önceliği belgeli.** `_PII_PATTERNS` iter sırası hem tespit +önceliğini hem maskeleme önceliğini yönetir — en spesifik pattern'lar +(`email`, `iban`, `credit_card`, ulusal kimlikler) önce taranır, +ardından gürültülü `phone` pattern'i. Bir span iki kategoriyle +eşleşebileceğinde, ilk / dar olan kazanır ve span sonraki pattern +görmeden değiştirilir. Phone, bare digit run'ları (timestamp, log line +numarası, ISO tarih) flag'lemesin diye kasıtlı olarak `+CC` veya +`(area)` formatına anchored. + +### Near-duplicate tespiti + +Case-fold edilmiş kelime token'ları üzerinde 64-bit simhash, Hamming +mesafe ≤ 3 ile (simhash makalesinin canonical web-page-dedup +kurulumunda kullandığı eşik, bu genişlikte ≈%95 benzerlik). Hem +**split-içi** çiftleri (`near_duplicate_pairs` per split) hem de +yukarıdaki **cross-split** sızıntıyı ortaya çıkarır. + +Satır sayısında karesel; ~50K satıra kadar olan veri setlerinde +audit-zamanı kullanım için uygun. Daha büyük külliyatlar için LSH band +indeksi gerekecek — `v0.5.0` kapsamı dışı. + +--- + +## Layout gereksinimleri + +| Girdi şekli | Ne elde edersin | +|---|---| +| `*.jsonl` dosyası | `train` adlı tek split | +| `train.jsonl`, `validation.jsonl`, `test.jsonl`'in herhangi birini içeren `dir/` | Her mevcut dosya kendi split'i olur | +| Yaygın alias'ları (`dev`, `val`, `valid`, `eval`, `holdout`) içeren `dir/` | Canonical split adlarına katlanır — `dev.jsonl` → `validation`, `eval.jsonl` → `test`, vb. | +| Yalnızca canonical olmayan `*.jsonl` içeren `dir/` | Pseudo-split fallback: her `*.jsonl` kendi split'i olur VE cross-split leakage analizi gerçek bir partition olmadan anlamsızdır uyarısı yayılır | + +Auditor şu öncelik sırasıyla bulduğu ilk metin-taşıyan sütunu okur: +`text` → `content` → `completion` → `prompt`. `messages` formatlı chat +verisinde rol etiketli içerikler birleştirilir. + +**Şema kaymaları yüzeye çıkarılır.** Heterojen JSONL (opsiyonel +alanları olan satırlar) izinlidir — sütun şeması satırlar arası +anahtarların union'ı olarak hesaplanır; satır 0'dan sonra ortaya çıkan +herhangi bir sütun `schema_drift_columns` altında raporlanır, böylece +operatörler kaymanın kasıtlı olup olmadığına karar verebilir. + +--- + +## Madde 10 governance entegrasyonu + +`data_audit_report.json` trainer'ın `training.output_dir`'ünde eğitim +zamanında mevcut olduğunda, +[`generate_data_governance_report`](../../forgelm/compliance.py) +bulguları governance artifact'ının `data_audit` anahtarı altında +otomatik olarak inline eder. Compliance bundle'ınız ayrı dosyaya +işaret eden bir pointer yerine tek başına okunabilir bir doküman olur. + +Önerilen iş akışı: + +```bash +# Önce denetim — uzun bir eğitim koşusuna girmeden sorunları yüzeye çıkar +forgelm --data-audit data/policies.jsonl --output ./checkpoints/policy-run/ + +# Eğit (governance artifact denetimi inline edecek) +forgelm --config configs/policy-run.yaml +``` + +--- + +## CLI referansı + +```text +forgelm --data-audit PATH \ + [--output DIR] \ + [--output-format {text,json}] \ + [--quiet | --log-level {DEBUG,INFO,WARNING,ERROR}] +``` + +`PATH` bir `.jsonl` dosyası veya bir dizin olabilir. `--output` +varsayılan olarak `./audit/`'tir. + +Üst düzey flag'tir (subcommand değil) — trainer'a dokunmadan çıkar. + +> **Not:** Bu davranış kılavuzun başındaki özetle eşleşir: +> `--output-format json` stdout'a küçük bir zarf (success flag, üst +> seviye metrikler, rapor yolu) yazar. Tam `data_audit_report.json` +> her zaman `--output` altına yazılır. Her detayı istiyorsanız +> diskten okuyun. + +--- + +## Sorun giderme + +| Belirti | Sebep | Çözüm | +|---|---|---| +| `Audit failed: ... not found or empty` | Yol mevcut değil veya `.jsonl` yok | Yolu doğrulayın; dosya veya `train.jsonl` dizin layout'u verin | +| Dil istatistiklerinde `"unknown (install forgelm[ingestion])"` | `langdetect` yüklü değil | `pip install 'forgelm[ingestion]'` | +| Cross-split leakage tüm satırların %100'ünü flag'liyor | Tüm split'ler aynı içeriği barındırıyor | Yeniden karıştırın; muhtemelen aynı JSONL'i her split'e kopyaladınız | +| Büyük veri setinde `near_duplicate_pairs` çok büyük | Simhash karesel on-binlerce satırda koştu | Önce örnekleyin; LSH index desteği takip edecek | + +--- + +## Programmatic API + +```python +from dataclasses import asdict +from forgelm.data_audit import audit_dataset + +report = audit_dataset("data/sft.jsonl", output_dir="./audit/") +print(report.total_samples, report.pii_summary) + +# Veya manuel serileştir: +import json +json.dump(asdict(report), open("custom_path.json", "w"), indent=2) +``` + +`AuditReport` düz bir dataclass'tır — `dataclasses.asdict()` size +JSON-hazır bir dict verir. PII regex helper'ları (`detect_pii`, +`mask_pii`) ve simhash fonksiyonu (`compute_simhash`) da public +API'nin parçası. diff --git a/docs/guides/data_audit.md b/docs/guides/data_audit.md new file mode 100644 index 00000000..e6a1a5df --- /dev/null +++ b/docs/guides/data_audit.md @@ -0,0 +1,214 @@ +# Dataset Audit Guide + +`forgelm --data-audit` analyzes a JSONL dataset and produces a +`data_audit_report.json` covering quality, governance, and PII signals. +Phase 11; introduced in `v0.5.0`. The report feeds the EU AI Act Article 10 +data governance artifact automatically when present in the trainer's +`output_dir`. + +--- + +## Run it + +```bash +# Single split (treated as 'train') +forgelm --data-audit data/sft.jsonl --output ./audit/ + +# Multi-split: directory containing train.jsonl / validation.jsonl / test.jsonl +forgelm --data-audit data/ --output ./audit/ +``` + +`--output` defaults to `./audit/`. The directory is created if missing; +the **full** `data_audit_report.json` is always written there. Stdout shows +a human-readable summary by default; pass `--output-format json` to get +a **summary** JSON envelope (top-level metrics + report path + notes) on +stdout — the full report still lives on disk under `--output`. CI/CD +consumers should slurp the file from `report_path` rather than parsing +the stdout summary when they need every detail. + +No GPU required. No network calls. CPU-only. + +--- + +## What you get + +### Per-split metrics + +```json +{ + "splits": { + "train": { + "sample_count": 1240, + "columns": ["text"], + "text_length": {"min": 32, "max": 4096, "mean": 1834.2, "p50": 1900, "p95": 3580}, + "null_or_empty_count": 3, + "null_or_empty_rate": 0.0024, + "languages_top3": [ + {"code": "tr", "count": 950}, + {"code": "en", "count": 240}, + {"code": "de", "count": 50} + ], + "simhash_distinct": 1180, + "near_duplicate_pairs": 60, + "pii_counts": {"email": 18, "phone": 4} + } + } +} +``` + +### Cross-split overlap + +```json +{ + "cross_split_overlap": { + "hamming_threshold": 3, + "pairs": { + "train__test": { + "leaked_rows_in_train": 7, + "leak_rate_train": 0.0056, + "leaked_rows_in_test": 7, + "leak_rate_test": 0.7 + } + } + } +} +``` + +The audit reports leak rate **in both directions** because they tell +different stories. With 1240 train rows and 10 test rows where 7 leak, +`leak_rate_train = 7/1240 = 0.56%` looks negligible but +`leak_rate_test = 7/10 = 70%` is the metric that actually destroys +benchmark fidelity. Always read the smaller-side rate — that is the +silent killer of test integrity. + +### PII summary + +```json +{ + "pii_summary": { + "email": 18, + "phone": 4, + "credit_card": 1, + "tr_id": 2 + } +} +``` + +Each row's text payload is scanned with regex; credit cards run through +Luhn validation, TR national IDs run through the TC Kimlik No checksum. +Other categories surface on regex shape alone — false positives are +intentional. Mask with `forgelm ingest --pii-mask` (or in your own +preprocessing) before publishing the dataset. + +**Pattern precedence is documented.** `_PII_PATTERNS` iteration order +governs both detection priority and mask precedence — most specific +patterns (`email`, `iban`, `credit_card`, national IDs) are scanned +first, then the noisier `phone` pattern. When a span could match two +categories, the first / narrower one wins and the span is replaced +before the next pattern sees it. Phone is intentionally anchored to +`+CC` or `(area)` formats so bare digit runs (timestamps, log line +numbers, ISO dates) do not flag. + +### Near-duplicate detection + +64-bit simhash over case-folded word tokens, paired with Hamming distance +≤ 3 (the cutoff the simhash paper uses for the canonical web-page-dedup +deployment, ≈95% similarity at this width). Exposes both: + +- **Within-split** pairs: `near_duplicate_pairs` per split. +- **Cross-split** leakage: above. + +Quadratic in row count; suitable for datasets up to ~50K rows. Larger +corpora need an LSH band index — out of scope for v0.5.0. + +--- + +## Layout requirements + +| Input shape | What you get | +|---|---| +| `*.jsonl` file | Single split named `train` | +| `dir/` containing any of `train.jsonl`, `validation.jsonl`, `test.jsonl` | Each present file becomes its own split | +| `dir/` containing common aliases (`dev`, `val`, `valid`, `eval`, `holdout`) | Folded onto canonical split names — `dev.jsonl` → `validation`, `eval.jsonl` → `test`, etc. | +| `dir/` containing only non-canonical `*.jsonl` | Pseudo-split fallback: each `*.jsonl` becomes its own split AND a warning is emitted that cross-split leakage analysis is meaningless without a real partition | + +The auditor reads the first text-bearing column it finds, in this priority: +`text` → `content` → `completion` → `prompt`. For `messages`-format chat +data, the role-tagged content is concatenated. + +**Schema drift is surfaced.** Heterogeneous JSONL (rows with optional fields) +is allowed — the column schema is the union of keys across rows; any column +that appears after row 0 is reported under `schema_drift_columns` so +operators can decide whether the drift is intentional. + +--- + +## Article 10 governance integration + +When `data_audit_report.json` exists in the trainer's `training.output_dir` +at training time, [`generate_data_governance_report`](../../forgelm/compliance.py) +inlines its findings under the `data_audit` key of the governance artifact. +Your compliance bundle becomes a single self-contained document rather than +a pointer to a separate file. + +The recommended workflow: + +```bash +# Audit first — surfaces issues before you commit to a long training run +forgelm --data-audit data/policies.jsonl --output ./checkpoints/policy-run/ + +# Train (governance artifact will inline the audit) +forgelm --config configs/policy-run.yaml +``` + +--- + +## CLI reference + +```text +forgelm --data-audit PATH \ + [--output DIR] \ + [--output-format {text,json}] \ + [--quiet | --log-level {DEBUG,INFO,WARNING,ERROR}] +``` + +`PATH` may be a `.jsonl` file or a directory. `--output` defaults to +`./audit/`. + +Top-level flag (not a subcommand) — exits without touching the trainer. + +> **Note:** This matches the behavior summarised at the top of this guide: +> `--output-format json` writes a small envelope (success flag, top-level +> metrics, report path) to stdout. The full `data_audit_report.json` is +> always written to `--output`. Read it from disk if you need every detail. + +--- + +## Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| `Audit failed: ... not found or empty` | Path doesn't exist or has no `.jsonl` | Verify the path; pass a file or a `train.jsonl` directory layout | +| `"unknown (install forgelm[ingestion])"` in language stats | `langdetect` not installed | `pip install 'forgelm[ingestion]'` | +| Cross-split leakage flags 100% of rows | All splits contain identical content | Re-shuffle; you probably copied the same JSONL into every split | +| `near_duplicate_pairs` enormous on a large dataset | Simhash quadratic ran for tens of thousands of rows | Sample first; LSH index support is a follow-up | + +--- + +## Programmatic API + +```python +from dataclasses import asdict +from forgelm.data_audit import audit_dataset + +report = audit_dataset("data/sft.jsonl", output_dir="./audit/") +print(report.total_samples, report.pii_summary) + +# Or serialize manually: +import json +json.dump(asdict(report), open("custom_path.json", "w"), indent=2) +``` + +`AuditReport` is a plain dataclass — `dataclasses.asdict()` gives you a +JSON-ready dict. The PII regex helpers (`detect_pii`, `mask_pii`) and the +simhash function (`compute_simhash`) are also part of the public API. diff --git a/docs/guides/ingestion-tr.md b/docs/guides/ingestion-tr.md new file mode 100644 index 00000000..d9d72eae --- /dev/null +++ b/docs/guides/ingestion-tr.md @@ -0,0 +1,279 @@ +# Doküman Yutma Rehberi + +Ham kurumsal külliyatı (PDF / DOCX / EPUB / TXT / Markdown) ForgeLM'in +eğittiği SFT-uyumlu JSONL'a dönüştürün. Faz 11; `v0.5.0`'da tanıtıldı. + +> Sonrasında [`forgelm --data-audit`](data_audit-tr.md) ile uzunluk +> dağılımı / dil / near-duplicate / PII metriklerini yüzeye çıkarın; +> chunk'ları Q&A `messages` formuna genişletmek için +> [`forgelm --generate-data`](../reference/usage-tr.md) ile zincirleyin. + +--- + +## Kurulum + +```bash +pip install 'forgelm[ingestion]' +``` + +Bu opsiyonel grup `pypdf`, `python-docx`, `ebooklib`, `beautifulsoup4` ve +`langdetect`'i getirir. Düz metin + Markdown için bu paketlerin hiçbiri +gerekmediği için opsiyonel tutuldu. Modülün kendisi import edildiğinde +ilgili extractor çağrılmadığı sürece bu bağımlılıkları yüklemez. + +OCR **kapsam dışındadır.** Metin katmanı olmayan taranmış PDF'ler bir +uyarı yüzdürür ve sıfır chunk üretir; ingest etmeden önce Tesseract veya +AWS Textract ile ön işleyin. + +--- + +## Tek komut — dosya gir, JSONL çık + +```bash +forgelm ingest ./book.epub --output data/sft.jsonl +forgelm ingest ./policies/ --recursive --output data/policies.jsonl +forgelm ingest ./scan.pdf --strategy sliding --chunk-size 1024 --overlap 128 \ + --output data/scan.jsonl +``` + +Çıktı satır başına bir chunk: + +```json +{"text": "EU AI Act Madde 10, yüksek-riskli AI sağlayıcılarından ..."} +{"text": "Veri kalitesi kriterleri arasında uygunluk, temsil edicilik ..."} +``` + +Trainer'ın veri yükleyicisi `{"text": "..."}` formatını önceden +formatlanmış SFT sütunu olarak tanır (bkz. +[`forgelm/data.py`](../../forgelm/data.py)) — başka bir ön işleme +gerekmez. + +--- + +## Chunking stratejileri + +| Strateji | Ne zaman | Davranış | +|---|---|---| +| `paragraph` (varsayılan) | Düz yazı, politika dokümanları, makaleler | Açgözlü paragraf paketleyici; bir paragrafı asla yarıda bölmez. | +| `sliding` | Uzun teknik dokümanlar, prose ile karışık kod | Sabit boyutlu karakter penceresi + bağlam taşıması için `--overlap`. | + +> Embedding tabanlı semantik chunking takip eden bir faza ayrılmıştır — +> bugün `NotImplementedError` raise eder ve runtime crash'i önlemek için +> CLI `--strategy` choice listesinden bilinçli olarak gizlenmiştir. + +`--chunk-size` **karakter olarak** ölçülür, token değil. Kaba bir kural +olarak `--chunk-size 2048` tipik İngilizce / Türkçe metinde ≈500–700 +token'a karşılık gelir — değeri `model.max_length` ile uyumlu seçin +(örneğin `max_length: 2048` token'lı bir model `--chunk-size 6000-8000` +ile rahat bir manevra alanı bulur, çünkü formatter sistem prompt + chat +template overhead'ine yer ayırır). Soft cap'i aşan paragraflar tek +başlarına yayılır — yarıda bölmekten daha iyidir. + +**Sliding overlap sınırlıdır.** `--overlap` hem `< --chunk-size` hem de +`≤ --chunk-size // 2` olmalıdır — bunun üstündeki değerler chunk +sayısını patlatır (`--overlap 199 --chunk-size 200` kombinasyonu +karakter başına ~bir chunk üretir). Patolojik kombinasyonları CLI +önden reddeder. + +**Dosyalar leksikografik sırayla işlenir** (sıralanmış glob sonucu), +böylece aynı girdi + bayraklarla yeniden çalıştırma byte-byte aynı +JSONL'i üretir. + +--- + +## Yazma sırasında PII maskeleme + +Tespit edilen e-posta, telefon, Luhn-doğrulanmış kredi kartı, IBAN ve +ulusal kimlik numaralarını (TR Kimlik No, DE Personalausweis, FR INSEE, +US SSN) chunk'lar JSONL'a inmeden önce redact etmek için `--pii-mask` +ekleyin: + +```bash +forgelm ingest ./customer_emails/ --output data/anon.jsonl --pii-mask +``` + +Tespit edilen span'ler `[REDACTED]` ile değiştirilir. Tespit regex +tabanlıdır — false positive'ler kasıtlıdır. Sonrasında +`forgelm --data-audit` ile çıktıyı doğrulayın. + +--- + +## Recursive dizin yürüyüşü + +```text +./policies/ +├── 2024_q1.pdf +├── 2024_q2.pdf +└── archive/ + ├── 2023.docx + └── 2022.epub +``` + +```bash +forgelm ingest ./policies/ --recursive --output data/all_policies.jsonl +``` + +Desteklenmeyen uzantıya sahip dosyalar (`.png`, `.zip`, vb.) sessizce +atlanır. Desteklenen uzantısı olup ekstrakte edilebilir metin içermeyen +dosyalar (taranmış PDF'ler, boş DOCX) bir uyarıyla atlanır. + +**Şifreli PDF'ler** açıkça yakalanır: önce boş şifreyle decrypt denenir +(owner-encrypted ama hâlâ okunabilen PDF'leri kapsar). Başarısız olursa +dosya başına extractor `ValueError` raise eder — toplu ingestion +döngüsü bunu "extraction failed" olarak yakalar, dosya adını içeren +bir uyarı log'lar, sonuçtaki `files_skipped`'i artırır ve sonraki +dosyaya geçer. Uyarı metni harici decrypt yolunu önerir: + +```bash +qpdf --decrypt --password= input.pdf out.pdf +# veya +pdftk input.pdf input_pw output out.pdf +``` + +CLI'a şifre flag'i bilinçli olarak eklenmedi — şifreleri shell +geçmişinden uzak tutmak daha güvenli. + +**Metin gibi davranan binary içerik** (zip / image yeniden adlandırılmış +`.txt`) dosyanın %1'inden fazlası Unicode replacement karakteri olarak +decode olduğunda bir uyarı yüzdürür. Chunk'lar yine yazılır — +operatörler kalıp kalmayacaklarına karar verir — ama uyarı CI loglarında +görülecek kadar yüksek seslidir. + +--- + +## Uçtan uca örnek + +```bash +# 1. Yutma +forgelm ingest ./policies/ --recursive --output data/policies.jsonl + +# 2. Denetim (near-duplicate'leri, PII'yi, uzunluk aykırılarını yakalar) +forgelm --data-audit data/policies.jsonl --output ./audit/ + +# 3. (opsiyonel) Öğretmen modelle Q&A'ya genişletme +forgelm --config configs/synth.yaml --generate-data + +# 4. Eğitim +forgelm quickstart domain-expert --dataset data/policies.jsonl +``` + +--- + +## CLI referansı + +```text +forgelm ingest INPUT_PATH \ + --output FILE \ + [--chunk-size N] \ + [--overlap N] \ + [--strategy {sliding,paragraph}] \ + [--recursive] \ + [--pii-mask] \ + [--output-format {text,json}] \ + [--quiet | --log-level {DEBUG,INFO,WARNING,ERROR}] +``` + +`--output-format json` standart çıktıya makine-okunabilir bir özet +yazar (dosya yolları, chunk sayısı, format sayıları, notlar). CI/CD +pipeline'larında kullanışlı. + +--- + +## Sorun giderme + +| Belirti | Sebep | Çözüm | +|---|---|---| +| `ImportError: PDF ingestion requires the 'ingestion' extra` | Opsiyonel grup yüklü değil | `pip install 'forgelm[ingestion]'` | +| "No extractable text in '.pdf'" uyarısı + 0 chunk | Taranmış PDF, metin katmanı yok | Önce OCR (Tesseract / AWS Textract) | +| `FileNotFoundError: No supported files found at ''` | Dizinde sadece desteklenmeyen uzantı | Dosya uzantılarının `.pdf / .docx / .epub / .txt / .md` ile eşleştiğini doğrulayın | +| `ValueError: overlap must be in [0, chunk_size)` | `--overlap >= --chunk-size` | `--overlap`'i azaltın | + +--- + +## Sınırlamalar + +- **OCR:** Kapsam dışı. Harici araçlar kullanın — aşağıdaki örneklere bakın. +- **Tablolar / şekiller:** PDF/DOCX tablolarındaki hücreler düz metin olarak + satır-major düzende düzleştirilir. Görsel yapı kaybolur. +- **Metadata:** başlık / yazar / sayfa numarası düşürülür — yalnızca gövde + metni JSONL'a iner. +- **Encoding:** non-UTF-8 girdi `errors="replace"` ile okunur; binary + noise Unicode replacement karakterlerine dönüşür. +- **Semantik chunking:** embedding desteği bir sonraki fazda gelene kadar + `NotImplementedError` raise eder. + +--- + +## Taranmış PDF'lerle çalışma (OCR teslim akışı) + +`forgelm ingest` OCR yapmaz. Metin katmanı olmayan taranmış PDF'ler +uyarı yüzdürür ve sıfır chunk üretir. Doğru yol önce harici OCR yapıp +sonucu ingestion'dan geçirmektir. İki tarif, maliyet sırasıyla: + +### Tarif A — Tesseract (yerel, ücretsiz) + +```bash +# Bir kerelik kurulum. +brew install tesseract # macOS +# veya: sudo apt install tesseract-ocr tesseract-ocr-tur tesseract-ocr-eng + +# 1. Her taranmış PDF sayfasını searchable PDF'e dönüştür (metin katmanı eklenir). +ocrmypdf scan_input.pdf scan_with_text_layer.pdf --language eng+tur + +# 2. Şimdi normal text-bearing PDF olarak ingest et. +forgelm ingest scan_with_text_layer.pdf --output data/scan.jsonl +``` + +`ocrmypdf` önerilen sarmalayıcıdır — deskew, sayfa rotasyonunu yönetir +ve OCR'lı karakterleri görsel katmana yakmak yerine gizli bir metin +katmanı yazar. Yalnızca düz metne ihtiyaç varsa pure `tesseract` da +çalışır: + +```bash +# Sayfa-başına çıkarım, TXT'e birleştir, TXT'i ingest et. +pdftoppm -r 300 scan_input.pdf scan_page -png +for img in scan_page-*.png; do tesseract "$img" - >> scan_pages.txt; done +forgelm ingest scan_pages.txt --output data/scan.jsonl +``` + +### Tarif B — AWS Textract (bulut, ücretli; tablo / formlarda daha iyi) + +```bash +# 1. Taranmış PDF'leri S3'e yükle. +aws s3 cp scan_input.pdf s3://my-ingest-bucket/scan_input.pdf + +# 2. Async Textract işi başlat. (Polling + SNS notification pattern için +# AWS Textract dokümanına bakın; tam shell loop kapsam dışı.) +aws textract start-document-text-detection \ + --document-location "{\"S3Object\":{\"Bucket\":\"my-ingest-bucket\",\"Name\":\"scan_input.pdf\"}}" + +# 3. İş tamamlandığında LINE blok'larını metne dök ve ingest et. +python -c " +import boto3, sys +client = boto3.client('textract') +job = sys.argv[1] +result = client.get_document_text_detection(JobId=job) +for block in result['Blocks']: + if block['BlockType'] == 'LINE': + print(block['Text']) +" $JOB_ID > scan_textract.txt +forgelm ingest scan_textract.txt --output data/scan.jsonl +``` + +Textract çok-sütunlu sayfalarda (akademik makaleler, dergiler), karışık +dilli sayfalarda ve formlar / tablolarda Tesseract'tan kayda değer +şekilde daha iyidir. Karşılığı: sayfa başına maliyet (yazıldığı sırada +text detection için ~$0.0015 / sayfa) ve ağ bağımlılığı. + +### PII içeren formlar? + +OCR'dan sonra, dataset'i yayımlamadan önce `--pii-mask` ile ön işleyin: + +```bash +ocrmypdf medical_scan.pdf medical_with_text.pdf --language tur+eng +forgelm ingest medical_with_text.pdf --output data/medical.jsonl --pii-mask +forgelm --data-audit data/medical.jsonl --output ./audit/ +``` + +Audit adımı redaksiyonun çalıştığını doğrular: `data_audit_report.json`'da +kalan herhangi bir PII flag'i, maskelemeden kaçan bir satırdır. diff --git a/docs/guides/ingestion.md b/docs/guides/ingestion.md new file mode 100644 index 00000000..bfec098b --- /dev/null +++ b/docs/guides/ingestion.md @@ -0,0 +1,278 @@ +# Document Ingestion Guide + +Convert raw enterprise corpora (PDF / DOCX / EPUB / TXT / Markdown) into the +SFT-ready JSONL ForgeLM trains on. Phase 11; introduced in `v0.5.0`. + +> Pair with [`forgelm --data-audit`](data_audit.md) afterwards to surface +> length-distribution / language / near-duplicate / PII metrics, and with +> [`forgelm --generate-data`](../reference/usage.md) if you want the chunks +> expanded into Q&A `messages` form via a teacher model. + +--- + +## Install + +```bash +pip install 'forgelm[ingestion]' +``` + +The extra brings in `pypdf`, `python-docx`, `ebooklib`, `beautifulsoup4`, and +`langdetect`. They are optional because plain text + Markdown work without +any of them. Importing the module does not pull these in unless the matching +extractor is exercised. + +OCR is **out of scope.** Scanned PDFs without a text layer surface a warning +and produce zero chunks; pre-process them with Tesseract or AWS Textract +before ingest. + +--- + +## Single command — file in, JSONL out + +```bash +forgelm ingest ./book.epub --output data/sft.jsonl +forgelm ingest ./policies/ --recursive --output data/policies.jsonl +forgelm ingest ./scan.pdf --strategy sliding --chunk-size 1024 --overlap 128 \ + --output data/scan.jsonl +``` + +Output is one chunk per line: + +```json +{"text": "Article 10 of the EU AI Act requires high-risk AI providers to ..."} +{"text": "Data quality criteria include relevance, representativeness ..."} +``` + +The trainer's data loader treats `{"text": "..."}` as a pre-formatted SFT +column (see [`forgelm/data.py`](../../forgelm/data.py)) — no further +preprocessing required. + +--- + +## Chunking strategies + +| Strategy | When | Behavior | +|---|---|---| +| `paragraph` (default) | Prose, policy docs, articles | Greedy paragraph packer; never splits a paragraph mid-sentence. | +| `sliding` | Long technical documents, code mixed with prose | Fixed-size character window with `--overlap` for context bleed. | + +> Semantic / embedding-based chunking is reserved for a follow-up phase — it +> raises `NotImplementedError` today and is intentionally hidden from the CLI +> `--strategy` choice list to avoid runtime crashes. + +`--chunk-size` is measured in **characters, not tokens**. As a rough rule, +`--chunk-size 2048` corresponds to ≈500–700 tokens for typical English / +Turkish text — set the value with `model.max_length` in mind (e.g. a model +with `max_length: 2048` tokens benefits from `--chunk-size 6000-8000` so the +formatter has headroom for system prompt + chat template overhead). +Paragraphs longer than the soft cap are emitted on their own — better than +mid-sentence splits. + +**Sliding overlap is bounded.** `--overlap` must be both `< --chunk-size` +*and* `≤ --chunk-size // 2` — values above that explode chunk count +(`--overlap 199 --chunk-size 200` would emit ~one chunk per character). +The CLI rejects pathological combinations up front. + +**Files are processed in lexicographic order** (sorted glob result), so a +re-run with the same input + flags produces the same JSONL byte-for-byte. + +--- + +## PII masking on the way in + +Add `--pii-mask` to redact emails, phone numbers, credit cards (Luhn-validated), +IBAN, and national IDs (TR Kimlik No, DE Personalausweis, FR INSEE, US SSN) +before any chunk lands in the JSONL: + +```bash +forgelm ingest ./customer_emails/ --output data/anon.jsonl --pii-mask +``` + +Detected spans are replaced with `[REDACTED]`. Detection is regex-based — +false positives are intentional. Audit your output afterwards with +`forgelm --data-audit` to verify. + +--- + +## Recursive directory walk + +```text +./policies/ +├── 2024_q1.pdf +├── 2024_q2.pdf +└── archive/ + ├── 2023.docx + └── 2022.epub +``` + +```bash +forgelm ingest ./policies/ --recursive --output data/all_policies.jsonl +``` + +Files with unsupported extensions (`.png`, `.zip`, etc.) are skipped silently. +Files with supported extensions but no extractable text (scanned PDFs, +empty DOCX) skip with a warning. + +**Encrypted PDFs** are caught explicitly: an empty-password decrypt is +attempted automatically (covers owner-encrypted PDFs that are still +readable). If that fails, the per-file extractor raises `ValueError` +internally — the batch ingestion loop catches that as "extraction +failed", logs a warning that names the file, increments +`files_skipped` in the result, and moves on to the next file. The +warning text recommends decrypting externally first: + +```bash +qpdf --decrypt --password= input.pdf out.pdf +# or +pdftk input.pdf input_pw output out.pdf +``` + +Wiring a CLI password flag is intentionally avoided — keeping passwords +out of shell history is safer. + +**Binary content masquerading as text** (a `.txt` file that's actually a +zip / image renamed) surfaces a warning when more than 1% of the file +decodes as Unicode replacement characters. The chunks still write — +operators decide whether to keep them — but the warning is loud enough +to catch in CI logs. + +--- + +## End-to-end example + +```bash +# 1. Ingest +forgelm ingest ./policies/ --recursive --output data/policies.jsonl + +# 2. Audit (catches near-duplicates, PII, length outliers) +forgelm --data-audit data/policies.jsonl --output ./audit/ + +# 3. (optional) Expand to Q&A via a teacher model +forgelm --config configs/synth.yaml --generate-data + +# 4. Train +forgelm quickstart domain-expert --dataset data/policies.jsonl +``` + +--- + +## CLI reference + +```text +forgelm ingest INPUT_PATH \ + --output FILE \ + [--chunk-size N] \ + [--overlap N] \ + [--strategy {sliding,paragraph}] \ + [--recursive] \ + [--pii-mask] \ + [--output-format {text,json}] \ + [--quiet | --log-level {DEBUG,INFO,WARNING,ERROR}] +``` + +`--output-format json` emits a machine-readable summary to stdout (file +paths, chunk count, format counts, notes). Useful for CI/CD pipelines. + +--- + +## Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| `ImportError: PDF ingestion requires the 'ingestion' extra` | Extra not installed | `pip install 'forgelm[ingestion]'` | +| "No extractable text in '.pdf'" warning + 0 chunks | Scanned PDF, no text layer | OCR first (Tesseract / AWS Textract) | +| `FileNotFoundError: No supported files found at ''` | Directory has only unsupported extensions | Verify file extensions match `.pdf / .docx / .epub / .txt / .md` | +| `ValueError: overlap must be in [0, chunk_size)` | `--overlap >= --chunk-size` | Reduce `--overlap` | + +--- + +## Limitations + +- **OCR:** out of scope. Use external tooling — see the worked example below. +- **Tables / figures:** PDF/DOCX tables are flattened to plain text. Structure is lost. +- **Metadata:** title / author / page numbers are dropped — only body text reaches the JSONL. +- **Encoding:** non-UTF-8 input is read with `errors="replace"`; binary noise becomes Unicode replacement characters. +- **Semantic chunking:** raises `NotImplementedError` until embedding support lands in a follow-up phase. + +--- + +## Working with scanned PDFs (OCR handoff) + +`forgelm ingest` does not perform OCR. Scanned PDFs without a text layer +surface as a warning and emit zero chunks. The clean way is to OCR +externally first, then pipe the result through ingestion. Two recipes, +in increasing order of cost: + +### Recipe A — Tesseract (local, free) + +```bash +# Install once. +brew install tesseract # macOS +# or: sudo apt install tesseract-ocr tesseract-ocr-tur tesseract-ocr-eng # Debian/Ubuntu + +# 1. Convert each scanned PDF page to a searchable PDF (text layer added). +ocrmypdf scan_input.pdf scan_with_text_layer.pdf --language eng+tur + +# 2. Now ingest as a normal text-bearing PDF. +forgelm ingest scan_with_text_layer.pdf --output data/scan.jsonl +``` + +`ocrmypdf` is the recommended wrapper — it handles deskew, page rotation, +and writes a hidden text layer rather than burning OCR'd characters into +the visual layer. Pure `tesseract` works too if you only need plain text: + +```bash +# Per-page extraction, concatenate to a TXT, ingest the TXT. +pdftoppm -r 300 scan_input.pdf scan_page -png +for img in scan_page-*.png; do tesseract "$img" - >> scan_pages.txt; done +forgelm ingest scan_pages.txt --output data/scan.jsonl +``` + +### Recipe B — AWS Textract (cloud, paid; better on tables / forms) + +```bash +# 1. Upload scanned PDFs to S3. +aws s3 cp scan_input.pdf s3://my-ingest-bucket/scan_input.pdf + +# 2. Start an async Textract job. (See AWS Textract docs for the polling +# + SNS notification pattern; a complete shell loop is out of scope.) +aws textract start-document-text-detection \ + --document-location "{\"S3Object\":{\"Bucket\":\"my-ingest-bucket\",\"Name\":\"scan_input.pdf\"}}" + +# 3. Once the job finishes, dump LINE blocks to text and ingest. +python -c " +import boto3, sys +client = boto3.client('textract') +job = sys.argv[1] +result = client.get_document_text_detection(JobId=job) +for block in result['Blocks']: + if block['BlockType'] == 'LINE': + print(block['Text']) +" $JOB_ID > scan_textract.txt +forgelm ingest scan_textract.txt --output data/scan.jsonl +``` + +Textract is materially better than Tesseract on: +- Multi-column layouts (academic papers, magazines) +- Mixed-language pages +- Forms / tables (separate `start-document-analysis` API) + +The trade-off is per-page cost (~$0.0015 / page for text detection at +the time of writing) and a network dependency. + +### What about forms with PII? + +Pre-process with `--pii-mask` after OCR, before publishing the dataset: + +```bash +ocrmypdf medical_scan.pdf medical_with_text.pdf --language tur+eng +forgelm ingest medical_with_text.pdf --output data/medical.jsonl --pii-mask +forgelm --data-audit data/medical.jsonl --output ./audit/ +``` + +The audit step verifies redaction worked: any remaining PII flag in +`data_audit_report.json` is a row that escaped masking. + +For larger corpora or a domain with lots of jargon, consider running +`forgelm --generate-data` (synthetic data pipeline) on the ingested JSONL +to expand a small seed into many Q&A pairs before fine-tuning. diff --git a/docs/guides/quickstart.md b/docs/guides/quickstart.md index 965dc744..e7daac36 100644 --- a/docs/guides/quickstart.md +++ b/docs/guides/quickstart.md @@ -84,6 +84,22 @@ data: dataset_name_or_path: "timdettmers/openassistant-guanaco" # or your dataset ``` +### Option C: I have raw documents (PDFs / DOCX / EPUBs), not JSONL + +Run the Phase 11 ingestion + audit pipeline first, then point any of the +options above at the resulting JSONL: + +```bash +pip install -e ".[ingestion]" +forgelm ingest ./policies/ --recursive --output data/policies.jsonl +forgelm --data-audit data/policies.jsonl --output ./audit/ +# Now `data/policies.jsonl` is ready to plug into a config. +``` + +See the [Document Ingestion Guide](ingestion.md) and [Dataset Audit +Guide](data_audit.md) for chunking strategies, PII masking, and the +governance signals the audit surfaces. + ## 3. Validate (Dry Run) ```bash @@ -269,3 +285,11 @@ synthetic: - [Enterprise Deployment](enterprise_deployment.md) — Docker, offline, multi-GPU - [Safety & Compliance](safety_compliance.md) — EU AI Act, safety evaluation - [Troubleshooting](troubleshooting.md) — common issues and solutions + +### Runnable notebooks (Colab) + +- [Quick Start — SFT](../../notebooks/quickstart_sft.ipynb) +- [Post-Training Workflow](../../notebooks/post_training_workflow.ipynb) — `--fit-check` → `chat` → `export` → `deploy` +- [Multi-Dataset Training](../../notebooks/multi_dataset.ipynb), [GaLore Memory Optimization](../../notebooks/galore_memory_optimization.ipynb), [Synthetic Data Pipeline](../../notebooks/synthetic_data_training.ipynb) +- [Safety Evaluation & Red-Teaming](../../notebooks/safety_evaluation.ipynb) +- Alignment: [DPO](../../notebooks/dpo_alignment.ipynb), [KTO](../../notebooks/kto_binary_feedback.ipynb), [GRPO](../../notebooks/grpo_reasoning.ipynb) diff --git a/docs/qms/sop_data_management.md b/docs/qms/sop_data_management.md index 7469721b..8cc00ee0 100644 --- a/docs/qms/sop_data_management.md +++ b/docs/qms/sop_data_management.md @@ -71,11 +71,24 @@ ForgeLM automated checks: - Dataset fingerprinting (SHA-256 hash, size, timestamp) - Format validation per trainer type (SFT, DPO, KTO, GRPO) - Text cleaning (`clean_text: true`) +- **Audit pipeline (`forgelm --data-audit `, v0.5.0+)** — produces + `data_audit_report.json` with per-split sample counts, length distribution, + top-3 language detection, simhash near-duplicate rate, cross-split + leakage check, and PII flag counts (email / phone / Luhn-validated credit + card / IBAN / TR–DE–FR–US national IDs). The report is auto-inlined into + the EU AI Act Article 10 governance artifact when present in the trainer's + `output_dir` — operators must run the audit **before** training to keep + the bundle self-contained. +- **Ingestion pipeline (`forgelm ingest`, v0.5.0+)** — turns raw PDF / DOCX / + EPUB / TXT / Markdown into SFT-ready JSONL with optional `--pii-mask` to + redact detected PII spans before chunks reach storage. Manual checks: -- [ ] Sample review: inspect 50+ random examples -- [ ] Check for duplicate entries -- [ ] Verify label correctness (for preference/KTO data) +- [ ] Run `forgelm --data-audit` and review `data_audit_report.json` for: + cross-split leakage > 0%, near-duplicate rate, unexpected language mix, + PII flag counts. +- [ ] Sample review: inspect 50+ random examples. +- [ ] Verify label correctness (for preference/KTO data). ## 6. Data Storage and Retention diff --git a/docs/reference/usage-tr.md b/docs/reference/usage-tr.md index 0a17ee25..d9dc67b9 100644 --- a/docs/reference/usage-tr.md +++ b/docs/reference/usage-tr.md @@ -69,6 +69,44 @@ forgelm --config my_config.yaml --generate-data Bu komut, eğitim başlamadan önce bir öğretmen modelden eğitim verisi üretmek için `synthetic` config bölümünü kullanır. Tüm sentetik veri seçenekleri için [Konfigürasyon Rehberi](configuration-tr.md)'ne bakın. +### Doküman Yutma (v0.5.0+) + +Ham PDF / DOCX / EPUB / TXT / Markdown'ı SFT'ye uygun JSONL'a dönüştürür. Opsiyonel bağımlılık: `pip install forgelm[ingestion]`. Ayrıntılar için [Doküman Yutma Rehberi](../guides/ingestion-tr.md). + +```bash +# Tek dosya +forgelm ingest ./book.epub --output data/sft.jsonl + +# Recursive dizin yürüyüşü + paragraf chunking +forgelm ingest ./policies/ --recursive --output data/policies.jsonl + +# Kayan pencere + örtüşme (uzun teknik dokümanlar) +forgelm ingest ./scan.pdf --strategy sliding --chunk-size 1024 --overlap 128 \ + --output data/scan.jsonl + +# Yazmadan önce PII'yi maskele +forgelm ingest ./customer_emails/ --pii-mask --output data/anon.jsonl +``` + +### Veri Seti Denetimi (v0.5.0+) + +CPU-only kalite + governance denetimi. `data_audit_report.json` üretir. Ayrıntılar için [Denetim Rehberi](../guides/data_audit-tr.md). + +```bash +# Tek split +forgelm --data-audit data/sft.jsonl --output ./audit/ + +# Çoklu split (train.jsonl / validation.jsonl / test.jsonl içeren dizin) +forgelm --data-audit data/ --output ./audit/ + +# stdout'a makine-okunabilir özet +forgelm --data-audit data/sft.jsonl --output ./audit/ --output-format json +``` + +Denetim şunları yakalar: split başına örnek sayısı + uzunluk dağılımı, top-3 dil tespiti, simhash near-duplicate oranı, cross-split sızıntı (sessiz train-test örtüşmesi), PII flag sayıları (e-posta / telefon / Luhn-doğrulanmış kredi kartı / IBAN / TR-DE-FR-US ulusal kimlikleri). + +Trainer'ın `output_dir`'ünde `data_audit_report.json` mevcutsa, bulgular EU AI Act Madde 10 governance artifact'ında `data_audit` anahtarı altında otomatik olarak inline edilir. + ### Değerlendirme, Birleştirme ve Uyumluluk ```bash diff --git a/docs/reference/usage.md b/docs/reference/usage.md index bf6939dd..6aeed9d1 100644 --- a/docs/reference/usage.md +++ b/docs/reference/usage.md @@ -129,6 +129,44 @@ forgelm --config my_config.yaml --generate-data This uses the `synthetic` config section to generate training data from a teacher model before training begins. See the [Configuration Guide](configuration.md) for all synthetic data options. +### Document Ingestion (v0.5.0+) + +Convert raw PDF / DOCX / EPUB / TXT / Markdown into SFT-ready JSONL. Optional dep: `pip install forgelm[ingestion]`. See [Ingestion Guide](../guides/ingestion.md). + +```bash +# Single file +forgelm ingest ./book.epub --output data/sft.jsonl + +# Recursive directory walk + paragraph chunking +forgelm ingest ./policies/ --recursive --output data/policies.jsonl + +# Sliding window with overlap (long technical docs) +forgelm ingest ./scan.pdf --strategy sliding --chunk-size 1024 --overlap 128 \ + --output data/scan.jsonl + +# Mask PII before writing +forgelm ingest ./customer_emails/ --pii-mask --output data/anon.jsonl +``` + +### Dataset Audit (v0.5.0+) + +CPU-only quality + governance audit. Produces `data_audit_report.json`. See [Audit Guide](../guides/data_audit.md). + +```bash +# Single split +forgelm --data-audit data/sft.jsonl --output ./audit/ + +# Multi-split directory (train.jsonl / validation.jsonl / test.jsonl) +forgelm --data-audit data/ --output ./audit/ + +# Machine-readable summary on stdout +forgelm --data-audit data/sft.jsonl --output ./audit/ --output-format json +``` + +The audit captures: per-split sample count + length distribution, top-3 language detection, simhash near-duplicate rate, cross-split leakage (silent train-test overlap), PII flag counts (email / phone / Luhn-validated credit card / IBAN / TR-DE-FR-US national IDs). + +When `data_audit_report.json` is present in the trainer's `output_dir` at training time, its findings are inlined under the `data_audit` key of the EU AI Act Article 10 governance artifact automatically. + ### Evaluation & Merging ```bash diff --git a/docs/roadmap-tr.md b/docs/roadmap-tr.md index 756e926c..b276c450 100644 --- a/docs/roadmap-tr.md +++ b/docs/roadmap-tr.md @@ -9,17 +9,19 @@ | ✅ Tamam | [Faz 1-9](roadmap/completed-phases.md) | SOTA iyileştirmeleri, değerlendirme, güvenilirlik, kurumsal entegrasyon, ekosistem, hizalama stack'i, güvenlik, EU AI Act uyumluluğu (Madde 9-17 + Ek IV), gelişmiş güvenlik zekası | | ✅ Tamam | [Faz 10 — Post-Training Tamamlama](roadmap/phase-10-post-training.md) | `inference.py`, `chat`, `export` (GGUF), `--fit-check`, `deploy` — `v0.4.0` | | ✅ Tamam | [Faz 10.5 — Quickstart Katmanı ve Onboarding](roadmap/phase-12-quickstart.md) | `forgelm quickstart