diff --git a/CHANGELOG.md b/CHANGELOG.md index 1fc4ef21..4f3038ad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,151 @@ All notable changes to ForgeLM are documented here. ## [Unreleased] +_Empty — v0.5.0 is finalized. The next minor (`v0.5.1`) covers Phase 14 +(Multi-Stage Pipeline Chains); see [docs/roadmap.md](docs/roadmap.md) +for the active phase plan._ + +--- + +## [0.5.0] — 2026-04-29 + +**Theme:** "Document Ingestion + Data Curation Pipeline" — Phases 11, +11.5, 12, and 12.5 ship as one comprehensive release. + +> **Note on consolidation.** Originally planned as four sequential +> PyPI tags (`v0.5.0` / `v0.5.1` / `v0.5.2` / `v0.5.3`) but consolidated +> into a single `v0.5.0` because the four phases form one coherent +> surface (ingest → polish → mature → polish) that's hard to use in +> parts. Git history retains the four phases as separate commit +> batches; this entry collapses them into the user-facing release +> notes. Section markers below preserve the phase boundary so +> reviewers can map back to [docs/roadmap/releases.md](docs/roadmap/releases.md). + +The release adds: + +- **Phase 11** — `forgelm ingest` (PDF / DOCX / EPUB / TXT / Markdown + → SFT-ready JSONL) + `forgelm audit` (length / language / + near-duplicate / cross-split leakage / PII) + EU AI Act Article 10 + governance integration. +- **Phase 11.5** — operational polish on the Phase 11 surface: LSH + banding, streaming reader, token-aware chunking, PDF + header/footer dedup, PII severity tiers, atomic audit writes. +- **Phase 12** — data curation maturity: MinHash LSH dedup option, + markdown-aware splitter, code/secrets leakage tagger, heuristic + quality filter, DOCX table preservation. +- **Phase 12.5** — small additive polish: `--all-mask` shorthand, + Croissant 1.0 dataset card emission, optional Presidio ML-NER PII + adapter, wizard "audit first" entry point. + +CI / docs / standards bookkeeping accompanying every phase is folded +into "Cross-cutting review hardening" at the bottom (rounds 1–12 of +review-cycle fixes applied across the four phases above). + +--- + +### Phase 12.5 — Data Curation Polish (backlog items #1–#4) + +Four follow-up items from +[`docs/roadmap/phase-12-5-backlog.md`](docs/roadmap/phase-12-5-backlog.md) +ship together — none require new architecture; each is a small +additive surface on top of the Phase 12 ingestion + audit lineage. + +- **`forgelm ingest --all-mask`** (item #3) — one-flag shorthand for + `--secrets-mask --pii-mask` in the documented mask order (secrets + first so combined detectors don't double-count overlapping spans). + Composes additively with explicit flags (set-union, no error). Pure + UX; no new behaviour. +- **`forgelm audit --croissant`** (item #2) — opt-in + [Google Croissant 1.0](http://mlcommons.org/croissant/) dataset card + emitted under a new `croissant` key in `data_audit_report.json`. The + card carries dataset-level identity, one `cr:FileObject` per JSONL + split, and a `cr:RecordSet` per split with `cr:Field` entries + derived from the audit's column detection. Existing audit JSON keys + are byte-equivalent — the block stays empty when the flag is off + (same precedent as `secrets_summary` / `quality_summary`). Lets the + same JSON file double as both the EU AI Act Article 10 governance + artifact and a Croissant-consumer dataset card. + - `url` and `contentUrl` use the as-typed input string and the + relative split filename, never the resolved absolute filesystem + path, so cards published to HuggingFace / MLCommons don't leak + the auditor's local layout. + - Croissant `version` (`sc:version`, dataset version) is omitted + deliberately — the audit doesn't have first-class evidence for + it; vocab conformance is declared via `conformsTo`. Operators + that publish hand-edit `version` like they do `license` / + `citeAs`. + - The card is now also surfaced in the `--output-format json` + stdout envelope alongside the on-disk report so CI consumers + don't need a second file slurp. +- **`forgelm audit --pii-ml [--pii-ml-language LANG]`** + new + `[ingestion-pii-ml]` extra (item #1) — opt-in + [Presidio](https://github.com/microsoft/presidio) ML-NER PII detector + layered on top of the existing regex detector. Adds the + unstructured-identifier categories the regex inherently misses + (`person`, `organization`, `location`) into the same `pii_summary` / + `pii_severity` blocks under disjoint category names. Severity tiers + in the new `PII_ML_SEVERITY` table: `person → medium`, + `organization → low`, `location → low` (deliberately below the regex + `critical`/`high` tiers because NER false-positive rates are + materially higher than regex-anchored detection). The pre-flight + check covers BOTH the missing-extra branch AND the missing-spaCy-model + branch — `presidio-analyzer` does *not* transitively ship a spaCy + NER model, so the install recipe is now two lines: + ```bash + pip install 'forgelm[ingestion-pii-ml]' + python -m spacy download en_core_web_lg + ``` + Without either, `forgelm audit --pii-ml` raises `ImportError` with + the recipe before any rows are scanned. Per-row Presidio failures + are scoped to `(ValueError, RuntimeError)` so a single malformed row + never blocks the audit, but a deep `OSError` from a missing model + surfaces loudly instead of silently scoring zero ML coverage. + `--pii-ml-language` (default `"en"`) lets non-English corpora point + at the matching spaCy model; Presidio raises a typed exception when + no engine is registered for the requested language. +- **Wizard "audit first" entry point** (item #4) — when the wizard + resolves a JSONL (either typed directly or produced by the + Phase 11.5 `_offer_ingest_for_directory` ingest flow), it now offers + to run `forgelm audit` on it inline and prints `summarize_report`'s + verdict before continuing. Mirrors the + `_offer_ingest_for_directory` shape exactly. Closes the BYOD audit + loop end-to-end. Audit is informational, not a gate — failures fall + through to the "continue without audit" path. + +Touch points (so the next reviewer can audit blast radius quickly): + +- `forgelm/ingestion.py` — no module changes (the flag composes at the + CLI boundary into the existing `pii_mask` / `secrets_mask` booleans). +- `forgelm/cli.py` — three new flags on the existing subparsers + (`--all-mask` on `forgelm ingest`; `--croissant` and `--pii-ml` on + `forgelm audit`); dispatcher signatures threaded through. +- `forgelm/data_audit.py` — `_HAS_PRESIDIO` sentinel, `_require_presidio`, + `_get_presidio_analyzer` (cached), `detect_pii_ml`, + `PII_ML_SEVERITY`, `PII_ML_TYPES`, `_PRESIDIO_ENTITY_MAP`, + `_build_croissant_metadata`, `_CROISSANT_CONTEXT`. New + `enable_pii_ml` / `emit_croissant` parameters on `audit_dataset` / + `_process_split` / `_audit_split`; new `enable_pii_ml` field on + `_StreamingAggregator`; new `croissant` field on `AuditReport`. + `_build_pii_severity` now consults the merged + `PII_SEVERITY ∪ PII_ML_SEVERITY` table. +- `forgelm/wizard.py` — new `_offer_audit_for_jsonl(path)` helper; + invoked from `_offer_ingest_for_directory` (after ingest produces + a JSONL), `_validate_local_jsonl` (after a directly-provided JSONL + passes validation), and `_prompt_dataset_path_with_ingest_offer` + (after a non-directory JSONL is provided to the full wizard). +- `pyproject.toml` — new `[ingestion-pii-ml]` extra + (`presidio-analyzer>=2.2.0,<3.0.0`). +- `tests/test_phase12_5.py` — 11 new tests, four classes (one per + backlog row). +- `tests/test_wizard_byod.py` — three existing tests get an extra + `"n"` answer to decline the new audit-first offer (the offer + behaviour has its own coverage in `test_phase12_5.py`). +- Docs — `README.md` install matrix + Phase 12.5 feature line; + `docs/standards/architecture.md` extras matrix; `docs/guides/ingestion{,-tr}.md` + + `docs/guides/data_audit{,-tr}.md` get dedicated sections per + feature; `notebooks/data_curation.ipynb` mentions `--all-mask` and + the Phase 12.5 audit add-ons inline. + ### Fixed — post-PR-#13 review-cycle batches (rounds 8-12) Inline-comment batches landing on top of PR #13 (now merged to `main`). @@ -367,67 +512,6 @@ Operational polish on top of `v0.5.0`'s ingestion + audit surface — no new tra --- -## [0.5.2] — 2026-04-28 - -**Theme:** "Data Curation Maturity" (Phase 12 Tier 1) — direct continuation of the Phase 11 / 11.5 ingestion + audit lineage. Closes the four gaps surfaced by the post-`v0.5.1` competitive review (LLaMA-Factory / Axolotl / Unsloth / NeMo Curator / Dolma / RedPajama / LlamaIndex / LangChain / Marker / Docling). Tier 2/3 items (Presidio adapter, Croissant metadata, `--all-mask`, wizard "audit first") deferred to [Phase 12.5 backlog](docs/roadmap/phase-12-5-backlog.md). - -### Added - -- **MinHash LSH dedup option** — opt-in `--dedup-method minhash --jaccard-threshold 0.85` route via `datasketch` (`[ingestion-scale]` extra) for >50K-row corpora. Default simhash + LSH banding stays untouched. -- **Markdown-aware splitter** — new `--strategy markdown` preserves heading hierarchy (`# H1` / `## H2`), keeps fenced code blocks atomic, and inlines a heading breadcrumb so SFT loss sees document context. -- **Code / secrets leakage tagger** — new `secrets_summary` block in audit JSON (AWS / GitHub / Slack / OpenAI / Google / JWT / OpenSSH / PGP / Azure storage). Ingest gains `--secrets-mask` (mask order: secrets → PII so combined detectors don't double-count). `[ingestion-secrets]` extra is reserved for a follow-up release; the regex-only fallback is what runs today. -- **Heuristic quality filter** — opt-in `--quality-filter` adds a `quality_summary` block with Gopher / C4 / RefinedWeb-style heuristics (mean-word-length, alphabetic ratio, end-of-line punctuation, short-paragraph ratio, repeated lines). ML classifiers stay deferred to Phase 13+. -- **DOCX table preservation** — `_extract_docx` emits Markdown table syntax (header + `---` separator + body rows) at extraction time, **before any chunking strategy runs**, instead of the previous `" | "` flat join. Uneven rows padded; all-blank rows trimmed. -- **`secrets_redaction_counts`** — added to the `--output-format json` envelope of `forgelm ingest` for CI/CD consumers; `{secret_type: count}` map of redacted spans. - -### Changed / Fixed (review-cycle hardening) - -- **Regex hygiene standard** ([`docs/standards/regex.md`](docs/standards/regex.md)) — codified 8 hard rules distilled from Phase 11 / 11.5 / 12 review cycles (Unicode-aware `\w`, no single-char classes, bounded quantifiers, no two competing unbounded greedy quantifiers, no `\s` under MULTILINE, no `.*?` + back-reference + DOTALL, intentional ASCII-only classes permitted with `re.ASCII`). The 8th rule covers `^.*` anti-pattern. -- **Replaced `_MARKDOWN_CODE_FENCE` regex with `_parse_md_fence`** — non-regex per-line parser; provably O(n). The previous regex had two unbounded greedy quantifiers in sequence over overlapping character classes (SonarCloud `python:S5852`). -- **`_strip_code_fences`** in `data_audit.py` rewritten as a per-line state machine; tracks the opening fence's run length per CommonMark §4.5 so a 4-backtick opener isn't prematurely closed by a 3-backtick line. -- **`_markdown_sections` complexity refactor** — extracted `_advance_fence_state`, `_flush_section`, `_render_sections` helpers so the dispatch loop reads top-down (cognitive complexity below SonarCloud's 15-line threshold). -- **`_parse_md_fence`** — now rejects backtick fence openers whose info string contains a backtick (CommonMark §4.5). -- **MinHash knob validation** — `audit_dataset` validates `minhash_jaccard ∈ [0.0, 1.0]` and `minhash_num_perm > 0` at the API boundary. -- **`re.ASCII` flag** on secret regexes (`github_token` / `openai_api_key` / `google_api_key` / `jwt`) so `\w` is restricted to ASCII; full BEGIN…END envelope matching for OpenSSH / RSA / DSA / EC / PGP private-key blocks. -- **Documentation drift sweep** — multiple rounds of EN ↔ TR mirror sync (`data_audit-tr.md`, `ingestion-tr.md`, `regex.md`, `data_audit.md`); legacy JSON-key documentation now references the actual `splits..near_duplicate_pairs` path. -- **ReDoS regression tests** — `tests/test_phase12_review_fixes.py` adds linearity benchmarks at 1K / 5K / 10K (median of 5 runs) for `_MARKDOWN_HEADING_PATTERN`, `_strip_code_fences`, and `_parse_md_fence`. - ---- - -## [0.5.1] — 2026-04-27 - -**Theme:** "Ingestion / Audit Polish" (Phase 11.5) — operational polish on top of `v0.5.0`'s ingestion + audit surface. No new training capabilities, but materially better handling for large corpora and a cleaner CLI shape. - -### Added - -- **`forgelm audit` subcommand** — promoted from the `--data-audit` top-level flag; first-class subcommand with its own `--output` default. The flag is preserved as a deprecation alias. -- **LSH-banded near-duplicate detection** — replaces the `O(n²)` pair scan inside each split (and across splits) with locality-sensitive-hashing bands (4 × 16-bit on the 64-bit fingerprint). Drops average-case to `O(n × k)` and unblocks audits on 100K+ row corpora. -- **Streaming `_read_jsonl_split`** — JSONL reader yields rows lazily; per-split aggregator stays generator-based until simhash collection. Bounds memory on multi-million-row splits. -- **Token-aware `--chunk-tokens`** — optional ingestion flag that sizes chunks against an HF tokenizer instead of raw character counts. -- **PDF page-level header / footer dedup** — repeated page headers (company watermark, page number) used to inflate near-duplicate counts; common-prefix / common-suffix detection across pages strips them automatically. -- **PII severity tiers** — audit output adds a `pii_severity` block grading each PII type as `low / medium / high / critical` (e.g. `credit_card` → critical, `phone` → low) so compliance reviewers get a one-glance verdict. -- **`summarize_report` truncation policy** — multi-split summaries get a `verbose=False` default that suppresses zero-finding splits. -- **Structured ingestion notes** — `IngestionResult.extra_notes` keeps the human-readable list but gains a parallel `notes_structured: {key: value}` map for programmatic consumers. -- **Wizard "ingest first" entry point** — first-class wizard option that routes to `forgelm ingest`, surfaces a JSONL path, and folds it back into the BYOD prompt. -- **xxhash backend for simhash + token-level memo** — drop-in faster non-crypto digest path (BLAKE2b kept as fallback). Token-level `lru_cache` memoizes repeat tokens for a 2–5× speedup on long corpora. -- **Atomic audit report write** — `data_audit_report.json` is written via `tempfile.NamedTemporaryFile` + atomic rename so a crashed audit never leaves a half-written report on disk. - ---- - -## [0.5.0] — 2026-04-27 - -**Theme:** "Document Ingestion & Data Audit" (Phase 11). Bridges raw enterprise corpora to ForgeLM's training data format and surfaces governance signals before training. - -### Added - -- **`forgelm/ingestion.py` + `forgelm ingest` subcommand** — multi-format document → JSONL pipeline with `paragraph` (default) and `sliding` chunking strategies, recursive directory walk, optional `--pii-mask`. Supported extensions: `.pdf` (`pypdf`), `.docx` (`python-docx`), `.epub` (`ebooklib` + `beautifulsoup4`), `.txt`, `.md`. Output is `{"text": ...}` JSONL recognised by ForgeLM's data loader as pre-formatted SFT input. OCR is intentionally out of scope; scanned PDFs warn and produce zero chunks. -- **`forgelm/data_audit.py` + `forgelm --data-audit` flag** — per-split metrics (sample count, column schema, length distribution `min/max/mean/p50/p95`, top-3 language detection, null/empty rate), 64-bit simhash near-duplicate detection within each split, cross-split overlap report (catches train/test leakage), PII regex with Luhn-validated credit cards and TC Kimlik checksum-validated TR IDs. -- **EU AI Act Article 10 integration** — `generate_data_governance_report` inlines `data_audit_report.json` under the `data_audit` key when present in the trainer's `output_dir`. -- **`pyproject.toml` `[ingestion]` extra** — `pypdf`, `python-docx`, `ebooklib`, `beautifulsoup4`, `langdetect`. Cross-platform; no native compilation. Plain TXT / Markdown ingestion + the audit module work without installing the extra (PII regex, simhash, length stats are pure stdlib). -- **Tests + docs** — `tests/test_ingestion.py` and `tests/test_data_audit.py` (54 tests; PDF round-trip skips when `pypdf` missing). New guides: `docs/guides/ingestion.md`, `docs/guides/data_audit.md`. README feature section, install matrix, and roadmap status updated. - ---- - ## [0.4.5] — 2026-04-26 ### Added diff --git a/README.md b/README.md index f09e8d32..7323eeb3 100644 --- a/README.md +++ b/README.md @@ -27,13 +27,16 @@ - **LLM-as-Judge**: API-based (OpenAI) or local model scoring for quality assessment - **Auto-Revert**: Opt-in (`evaluation.auto_revert: true`) — automatically discards models that fail loss, benchmark, or safety thresholds before artifacts are written -### Document Ingestion & Data Audit (v0.5.0 — polished in v0.5.1, matured in v0.5.2) -- **Multi-Format Ingestion**: `forgelm ingest ./policies/ --recursive --output data/policies.jsonl` — turns raw PDF / DOCX / EPUB / TXT / Markdown into the SFT-ready JSONL the trainer accepts. Optional dep: `pip install forgelm[ingestion]`. v0.5.2 added a **Markdown-aware splitter** (`--strategy markdown`) and DOCX table preservation in Markdown table syntax. -- **Chunking Strategies**: `paragraph` (default; preserves boundaries), `sliding` (fixed window with overlap), or `markdown` (v0.5.2 — heading-aware splitter that keeps fenced code blocks atomic and inlines a heading breadcrumb at the top of each chunk for SFT context). A `semantic` strategy is reserved for a follow-up phase — the implementation in `forgelm.ingestion` raises `NotImplementedError` today and the CLI hides it from `--strategy` choices. Token-aware mode in v0.5.1: `--chunk-tokens 1024 --tokenizer Qwen/Qwen2.5-7B-Instruct` sizes chunks against your model's actual vocabulary. +### Document Ingestion & Data Audit (v0.5.0 — Phases 11 + 11.5 + 12 + 12.5 consolidated) +- **Multi-Format Ingestion**: `forgelm ingest ./policies/ --recursive --output data/policies.jsonl` — turns raw PDF / DOCX / EPUB / TXT / Markdown into the SFT-ready JSONL the trainer accepts. Optional dep: `pip install forgelm[ingestion]`. Includes a **Markdown-aware splitter** (`--strategy markdown`) and DOCX table preservation in Markdown table syntax. +- **Chunking Strategies**: `paragraph` (default; preserves boundaries), `sliding` (fixed window with overlap), or `markdown` (heading-aware splitter that keeps fenced code blocks atomic and inlines a heading breadcrumb at the top of each chunk for SFT context). A `semantic` strategy is reserved for a follow-up phase — the implementation in `forgelm.ingestion` raises `NotImplementedError` today and the CLI hides it from `--strategy` choices. Token-aware mode: `--chunk-tokens 1024 --tokenizer Qwen/Qwen2.5-7B-Instruct` sizes chunks against your model's actual vocabulary. - **PII Masking on Ingest**: `--pii-mask` redacts emails, phones, credit cards (Luhn-validated), IBAN, and national IDs (TR / DE / FR / US-SSN) before chunks land in the JSONL. -- **PDF Page Header/Footer Dedup (v0.5.1)**: Lines that recur on ≥ 70 % of PDF pages (watermarks, page numbers, copyright lines) are stripped automatically — the audit's near-duplicate counts stop misfiring on long policy / book PDFs. -- **Dataset Audit**: `forgelm audit data/sft.jsonl --output ./audit/` — produces `data_audit_report.json` with sample count, length distribution, top-3 language detection, LSH-banded near-duplicate rate (`O(n × k)` typical case, exact recall at the default Hamming threshold; v0.5.2 added optional **MinHash LSH** via `--dedup-method minhash` for >50K-row corpora), cross-split leakage check, null/empty rate, and PII flag counts with **severity tiers** (critical / high / medium / low + worst-tier verdict). v0.5.2 also added an always-on **secrets/credential scan** covering the nine families enumerated in `forgelm.data_audit.SECRET_TYPES` — AWS access keys, GitHub tokens, Slack tokens, OpenAI API keys, Google API keys, JWTs, full OpenSSH private-key blocks, full PGP private-key blocks, and Azure storage connection strings — and an opt-in **heuristic quality filter** (`--quality-filter`). CPU-only; streaming JSONL reader keeps memory bounded on multi-million-row splits; feeds EU AI Act Article 10 governance artifact automatically when present at training time. Legacy `--data-audit` flag still works as a deprecation alias. -- **Secrets-Aware Ingest (v0.5.2)**: `forgelm ingest … --secrets-mask` scrubs credentials before chunks land in the JSONL — fine-tuning on text containing real API keys memorises them at training time. Pairs with `--pii-mask`; secrets run first so combined detectors don't double-count overlapping spans. +- **PDF Page Header/Footer Dedup**: Lines that recur on ≥ 70 % of PDF pages (watermarks, page numbers, copyright lines) are stripped automatically — the audit's near-duplicate counts stop misfiring on long policy / book PDFs. +- **Dataset Audit**: `forgelm audit data/sft.jsonl --output ./audit/` — produces `data_audit_report.json` with sample count, length distribution, top-3 language detection, LSH-banded near-duplicate rate (`O(n × k)` typical case, exact recall at the default Hamming threshold; optional **MinHash LSH** via `--dedup-method minhash` for >50K-row corpora), cross-split leakage check, null/empty rate, and PII flag counts with **severity tiers** (critical / high / medium / low + worst-tier verdict). Always-on **secrets/credential scan** covers the nine families enumerated in `forgelm.data_audit.SECRET_TYPES` — AWS access keys, GitHub tokens, Slack tokens, OpenAI API keys, Google API keys, JWTs, full OpenSSH private-key blocks, full PGP private-key blocks, and Azure storage connection strings — and an opt-in **heuristic quality filter** (`--quality-filter`). CPU-only; streaming JSONL reader keeps memory bounded on multi-million-row splits; feeds EU AI Act Article 10 governance artifact automatically when present at training time. Legacy `--data-audit` flag still works as a deprecation alias. +- **Secrets-Aware Ingest**: `forgelm ingest … --secrets-mask` scrubs credentials before chunks land in the JSONL — fine-tuning on text containing real API keys memorises them at training time. Pairs with `--pii-mask`; secrets run first so combined detectors don't double-count overlapping spans. `--all-mask` is a one-flag shorthand for both. +- **Optional ML-NER PII**: `forgelm audit --pii-ml [--pii-ml-language LANG]` layers [Presidio](https://github.com/microsoft/presidio) NER on top of the regex detector via the optional `[ingestion-pii-ml]` extra **plus a separate `python -m spacy download en_core_web_lg`** step (or the matching model for the chosen language). Adds `person` / `organization` / `location` categories into the same `pii_summary` / `pii_severity` blocks under disjoint category names. +- **Croissant 1.0 dataset card**: `forgelm audit --croissant` emits a [Google Croissant 1.0](http://mlcommons.org/croissant/) dataset card under the report's `croissant` key so the same JSON file doubles as both the EU AI Act Article 10 governance artifact and a Croissant-consumer dataset card. +- **Wizard "audit first"**: when the wizard resolves a JSONL (typed or produced by `forgelm ingest`) it offers to run `forgelm audit` inline and prints the verdict before continuing — closes the BYOD audit loop end-to-end. ### Quickstart Layer (v0.4.5) - **One-Command Templates**: `forgelm quickstart customer-support` — 4 bundled templates (SFT customer-support, code-assistant, medical-qa-tr, GRPO math-reasoning) plus a bring-your-own-data domain-expert scaffold. Auto-downsizes models on small GPUs. @@ -72,7 +75,7 @@ forgelm quickstart customer-support # render config + train + chat forgelm quickstart code-assistant --dry-run # render config only forgelm quickstart medical-qa-tr --model your-org/your-model # override -# Have raw docs? Ingest them first (v0.5.0+; v0.5.1 added token-aware sizing) +# Have raw docs? Ingest them first (v0.5.0; supports token-aware sizing) pip install -e ".[ingestion]" forgelm ingest ./policies/ --recursive --output data/policies.jsonl forgelm audit data/policies.jsonl --output ./audit/ # `forgelm --data-audit ...` still works as legacy alias @@ -172,6 +175,7 @@ pip install "forgelm[merging]" # mergekit model merging pip install "forgelm[ingestion]" # PDF/DOCX/EPUB/Markdown → JSONL + langdetect + xxhash pip install "forgelm[ingestion-scale]" # MinHash LSH dedup (datasketch) for >50K-row corpora pip install "forgelm[ingestion-secrets]" # detect-secrets scanner for SFT corpora (falls back to regex if absent) +pip install "forgelm[ingestion-pii-ml]" # Presidio ML-NER for person/organization/location PII (Phase 12.5; ALSO needs `python -m spacy download en_core_web_lg`) pip install "forgelm[export]" # GGUF export via llama-cpp-python pip install "forgelm[chat]" # Rich terminal rendering for `forgelm chat` ``` diff --git a/docs/guides/data_audit-tr.md b/docs/guides/data_audit-tr.md index 207d2068..be7c8ada 100644 --- a/docs/guides/data_audit-tr.md +++ b/docs/guides/data_audit-tr.md @@ -2,11 +2,11 @@ `forgelm audit PATH` bir JSONL veri setini analiz eder ve kalite, governance ile PII sinyallerini kapsayan bir `data_audit_report.json` -üretir. Faz 11 (`v0.5.0`'da tanıtıldı) altyapıyı, Faz 11.5 (`v0.5.1`) +üretir. Faz 11 altyapıyı, Faz 11.5 (`v0.5.0`'da birleşti) ise top-level flag'i first-class subcommand'a yükseltti, LSH-banded near-duplicate tespitini, streaming JSONL okuyucusunu, PII şiddet katmanlarını, atomik disk yazımı ve verbose-by-default kısaltma -politikasını ekledi. **Faz 12 (`v0.5.2`)** opt-in MinHash LSH dedup +politikasını ekledi. **Faz 12 (`v0.5.0`'da birleşti)** opt-in MinHash LSH dedup yöntemini (`--dedup-method minhash`), her zaman çalışan code/credential leakage taramasını (`secrets_summary`), ve opt-in heuristic kalite filtresini (`--quality-filter`) ekledi. @@ -268,7 +268,7 @@ forgelm ingest ./policies/ --recursive --output data/policies.jsonl --secrets-ma Opsiyonel / forward-compatibility: `[ingestion-secrets]` extra'sı bir `detect-secrets>=1.5.0` bağımlılığı tanımlar ama bu **bir sonraki sürüm -için ayrılmıştır**. v0.5.2 itibarıyla `forgelm.data_audit.detect_secrets()` +için ayrılmıştır**. v0.5.0 itibarıyla `forgelm.data_audit.detect_secrets()` yalnızca yukarıdaki regex setine dayanır; extra'yı kurmak bugün audit davranışını değiştirmez. Extra, `forgelm[ingestion-secrets]` pin'leyen operatörlerin entegrasyon geldiğinde forward-compatible olabilmesi @@ -321,6 +321,115 @@ install ile uyumlu kalır. --- +### ML-NER PII adaptörü — `--pii-ml` (Faz 12.5, opt-in) + +Varsayılan regex detector, EU AI Act Madde 10'un önemsediği yapılandırılmış +identifier'ları (email, telefon, IBAN, kredi kartı, ulusal kimlik) kapsar. +Faz 12.5, regex'in doğal olarak kaçırdığı yapılandırılmamış identifier +kategorilerini ekleyen opt-in **Presidio** ([microsoft/presidio](https://github.com/microsoft/presidio)) +adaptörünü ML-NER olarak üstüne ekler: `person`, `organization`, +`location`. + +```bash +pip install 'forgelm[ingestion-pii-ml]' +forgelm audit data/ --output ./audit/ --pii-ml +``` + +Yeni kategoriler mevcut `pii_summary` ve `pii_severity` bloklarına +disjoint isimlerle birleşir, bu yüzden regex baseline ML sinyaliyle +yan yana görünür kalır: + +```json +{ + "pii_summary": { + "email": 12, + "phone": 3, + "person": 47, // ← Presidio + "organization": 18, // ← Presidio + "location": 9 // ← Presidio + }, + "pii_severity": { + "by_tier": {"critical": 0, "high": 0, "medium": 59, "low": 30}, + "by_type": { + "email": {"count": 12, "tier": "medium"}, + "person": {"count": 47, "tier": "medium"}, + "organization": {"count": 18, "tier": "low"} + }, + "worst_tier": "medium" + } +} +``` + +Şiddet ataması `forgelm.data_audit.PII_ML_SEVERITY`'de yaşar: +`person → medium`, `organization → low`, `location → low`. NER +false-positive oranları regex-anchored detection'dan materyal olarak +yüksektir — bu yüzden ML katmanları regex'in `critical` / `high` +katlarının altında bilinçli konumlandırılır. Bir ML bulgusunu sert +bir gate olarak değerlendirmeden önce satır bazında span'leri inceleyin. + +**İki adımlı kurulum.** `presidio-analyzer` paketi spaCy NER modelini +transitively içermez — tek seferlik ayrı bir indirme: + +```bash +pip install 'forgelm[ingestion-pii-ml]' +python -m spacy download en_core_web_lg # ~ 50 MB; gerekli +``` + +Model yoksa `forgelm audit --pii-ml` hiçbir satır taranmadan **önce** +aynı install hint ile `ImportError` raise eder +(`forgelm.data_audit._require_presidio`'daki pre-flight, spaCy'nin +`OSError`'unu yakalayıp typed install error'a dönüştürür). Bu fail-loud +davranış bilinçli — önceki prototiplerde model eksikken sıfır ML +bulgusu sessizce raporlanıyordu, opt-in bir detector için kritik bir +compliance kör noktası. + +İngilizce olmayan korpuslar için eşleşen spaCy modelini kurup dil +kodunu geçirin: + +```bash +python -m spacy download xx_ent_wiki_sm # çok dilli, daha küçük +forgelm audit data/ --pii-ml --pii-ml-language xx +``` + +Adaptör yalnızca opt-in; `--pii-ml` olmadan audit zero-extra-deps +regex yolunda kalır. + +--- + +### Croissant 1.0 dataset card — `--croissant` (Faz 12.5, opt-in) + +`--croissant`, `data_audit_report.json` içinde yeni bir top-level +`croissant` bloğunu [Google Croissant 1.0](https://mlcommons.org/croissant/) +dataset card'ı (`@type: sc:Dataset`) ile doldurur. Card, kanonik +`mlcommons.org/croissant/1.0` context'iyle uyumludur — bu yüzden +Croissant farkındalıklı consumer'lar (HuggingFace dataset cards, +MLCommons referans loader'ları, Croissant validator) bloğu hiçbir +modifikasyon olmadan parse edebilir. + +```bash +forgelm audit data/ --output ./audit/ --croissant +``` + +Card şunları taşır: + +* dataset seviyesinde kimlik (`name`, `description`, `datePublished`, + `url`) — `version` bilinçli olarak çıkarılmıştır, audit'in dataset + sürümü için first-class kanıtı yoktur; card'ı yayınlayan operatörler + `version`'ı `license` / `citeAs` gibi elle düzenler, +* her JSONL split'i için bir `cr:FileObject` (Croissant consumer'ı + altta yatan dosyaları bulabilsin diye), +* her split için audit'in column-detection katmanından türetilen + `cr:Field` girdileriyle bir `cr:RecordSet`. + +Flag kapalıyken blok boştur — mevcut consumer'lar byte-eşdeğer çıktı +görür, `secrets_summary` / `quality_summary`'nin koyduğu emsalin +aynısı. Card'ı HuggingFace / MLCommons'a yayınlamak isteyen operatörler +audit'in birinci sınıf delili olmayan ek Croissant alanlarını +(`license`, `citeAs`, `keywords`) audit'i yeniden çalıştırmadan +elle düzenleyebilir. + +--- + ## Layout gereksinimleri | Girdi şekli | Ne elde edersin | @@ -373,6 +482,9 @@ forgelm audit PATH \ [--dedup-method {simhash,minhash}] \ [--jaccard-threshold X] \ [--quality-filter] \ + [--pii-ml] \ + [--pii-ml-language LANG] \ + [--croissant] \ [--output-format {text,json}] \ [--quiet | --log-level {DEBUG,INFO,WARNING,ERROR}] ``` diff --git a/docs/guides/data_audit.md b/docs/guides/data_audit.md index 2ae9f760..98591d92 100644 --- a/docs/guides/data_audit.md +++ b/docs/guides/data_audit.md @@ -3,11 +3,11 @@ `forgelm audit PATH` analyzes a JSONL dataset and produces a `data_audit_report.json` covering quality, governance, PII, and (Phase 12 onwards) credential-leakage and heuristic-quality signals. Phase 11 -(`v0.5.0`) shipped the underlying audit; Phase 11.5 (`v0.5.1`) promoted +shipped the underlying audit; Phase 11.5 (consolidated into `v0.5.0`) promoted it from a top-level flag to a first-class subcommand and added LSH-banded near-duplicate detection, streaming JSONL reading, PII severity tiers, atomic on-disk write, and a verbose-by-default -truncation policy. **Phase 12 (`v0.5.2`)** added an opt-in MinHash LSH +truncation policy. **Phase 12 (consolidated into `v0.5.0`)** added an opt-in MinHash LSH dedup method (`--dedup-method minhash`), a code/credential leakage scan that runs always-on (`secrets_summary`), and an opt-in heuristic quality filter (`--quality-filter`). @@ -268,7 +268,7 @@ forgelm ingest ./policies/ --recursive --output data/policies.jsonl --secrets-ma Optional / forward-compatibility: the `[ingestion-secrets]` extra declares a `detect-secrets>=1.5.0` dependency that is **reserved for a -follow-up release**. As of v0.5.2 the audit's +follow-up release**. As of v0.5.0 the audit's `forgelm.data_audit.detect_secrets()` relies solely on the regex set above; installing the extra today does not change audit behaviour. The extra exists so operators who pin `forgelm[ingestion-secrets]` in @@ -319,6 +319,113 @@ the audit reproducible (Annex IV requirement) and bare-install-friendly. --- +### ML-NER PII adapter — `--pii-ml` (Phase 12.5, opt-in) + +The default regex detector covers the structured identifiers EU AI Act +Article 10 cares about (email, phone, IBAN, credit card, national IDs). +Phase 12.5 adds an opt-in **Presidio** ([microsoft/presidio](https://github.com/microsoft/presidio)) +adapter that layers ML-NER on top — adding the unstructured-identifier +categories regex inherently misses: `person`, `organization`, `location`. + +```bash +pip install 'forgelm[ingestion-pii-ml]' +forgelm audit data/ --output ./audit/ --pii-ml +``` + +The new categories merge into the existing `pii_summary` and +`pii_severity` blocks under disjoint names so the regex baseline stays +visible alongside the ML signal: + +```json +{ + "pii_summary": { + "email": 12, + "phone": 3, + "person": 47, // ← Presidio + "organization": 18, // ← Presidio + "location": 9 // ← Presidio + }, + "pii_severity": { + "by_tier": {"critical": 0, "high": 0, "medium": 59, "low": 30}, + "by_type": { + "email": {"count": 12, "tier": "medium"}, + "person": {"count": 47, "tier": "medium"}, + "organization": {"count": 18, "tier": "low"} + }, + "worst_tier": "medium" + } +} +``` + +Severity assignment lives in `forgelm.data_audit.PII_ML_SEVERITY`: +`person → medium`, `organization → low`, `location → low`. NER +false-positive rates are materially higher than regex-anchored +detection, so the ML tiers sit deliberately below the regex +`critical`/`high` floors — review the per-row spans before treating an +ML finding as a hard gate. + +**Two-step install.** `presidio-analyzer` does **not** transitively +ship a spaCy NER model — that's a separate one-time download: + +```bash +pip install 'forgelm[ingestion-pii-ml]' +python -m spacy download en_core_web_lg # ~ 50 MB; required +``` + +Without the model, `forgelm audit --pii-ml` raises an `ImportError` +with the same install hint **before** any rows are scanned (the +pre-flight check in `forgelm.data_audit._require_presidio` catches +spaCy's `OSError` and re-raises it as the typed install error). This +fail-loud behaviour exists because earlier prototypes silently scored +zero ML findings on a missing model — a critical compliance blind +spot for an opt-in detector that operators expect to actually run. + +For non-English corpora, install the matching spaCy model and pass +the language code: + +```bash +python -m spacy download xx_ent_wiki_sm # multilingual, smaller +forgelm audit data/ --pii-ml --pii-ml-language xx +``` + +The adapter is opt-in only; without `--pii-ml` the audit stays on the +zero-extra-deps regex path. + +--- + +### Croissant 1.0 dataset card — `--croissant` (Phase 12.5, opt-in) + +`--croissant` populates a new top-level `croissant` block in +`data_audit_report.json` with a [Google Croissant 1.0](https://mlcommons.org/croissant/) +dataset card (`@type: sc:Dataset`). The card is conformant with the +canonical `mlcommons.org/croissant/1.0` context, so Croissant-aware +consumers (HuggingFace dataset cards, MLCommons reference loaders, the +Croissant validator) can parse the block without modification. + +```bash +forgelm audit data/ --output ./audit/ --croissant +``` + +The card carries: + +* dataset-level identity (`name`, `description`, `datePublished`, + `url`) — `version` is intentionally omitted because the audit has + no first-class evidence for dataset version; operators that publish + the card hand-edit `version` like they do `license` / `citeAs`, +* one `cr:FileObject` per JSONL split (so a Croissant consumer can + locate the underlying files), +* one `cr:RecordSet` per split with `cr:Field` entries derived from + the audit's column-detection layer. + +The block is empty when the flag is off — existing consumers see +byte-equivalent output, the same precedent set by `secrets_summary` / +`quality_summary`. Operators that want to publish the card to +HuggingFace / MLCommons can hand-edit the additional Croissant fields +the audit doesn't have first-class evidence for (`license`, `citeAs`, +`keywords`) without re-running the audit. + +--- + ## Layout requirements | Input shape | What you get | @@ -369,6 +476,9 @@ forgelm audit PATH \ [--dedup-method {simhash,minhash}] \ [--jaccard-threshold X] \ [--quality-filter] \ + [--pii-ml] \ + [--pii-ml-language LANG] \ + [--croissant] \ [--output-format {text,json}] \ [--quiet | --log-level {DEBUG,INFO,WARNING,ERROR}] ``` diff --git a/docs/guides/ingestion-tr.md b/docs/guides/ingestion-tr.md index f470e0cb..16919f22 100644 --- a/docs/guides/ingestion-tr.md +++ b/docs/guides/ingestion-tr.md @@ -1,12 +1,11 @@ # Doküman Yutma Rehberi Ham kurumsal külliyatı (PDF / DOCX / EPUB / TXT / Markdown) ForgeLM'in -eğittiği SFT-uyumlu JSONL'a dönüştürün. Faz 11; `v0.5.0`'da tanıtıldı. -Faz 11.5 (`v0.5.1`) token-aware chunking, PDF sayfa header/footer dedup -ve yapılandırılmış ingestion notları ekledi. **Faz 12 (`v0.5.2`)** -markdown-aware splitter (`--strategy markdown`), code/credential -leakage scrubbing (`--secrets-mask`) ve DOCX tablolarının markdown -syntax'ı ile çıkarılmasını ekledi. +eğittiği SFT-uyumlu JSONL'a dönüştürün. `v0.5.0` ile birlikte gelir +(Faz 11 + 11.5 + 12 + 12.5 birleşmiş): paragraph / sliding / +markdown-aware splitter, token-aware chunking, PDF sayfa header/footer +dedup, code/credential scrubbing (`--secrets-mask`), DOCX tablolarının +markdown syntax'ı ile çıkarılması, `--all-mask` kısayolu. > Sonrasında [`forgelm audit`](data_audit-tr.md) ile uzunluk dağılımı / > dil / near-duplicate / PII metriklerini yüzeye çıkarın; chunk'ları Q&A @@ -141,6 +140,26 @@ yukarıdaki regex seti çalışır. --- +## Hepsi bir arada maskeleme — `--all-mask` (Faz 12.5) + +`--all-mask`, `--secrets-mask --pii-mask` için tek-flag kısayoludur — +"paylaşılan korpus üzerinde eğitime başlamadan tespit edilebilir her +şeyi temizle" iş akışının yaygın hâlidir. İki detector belgelenen +sıraya göre çalışır (önce secrets — birleşik PII detector'lar örtüşen +span'leri çift saymasın diye); ortaya çıkan JSONL eşleşme bulunan +yerlerde hem `[REDACTED-SECRET]` hem de `[REDACTED]` belirteçlerini +taşır. + +```bash +forgelm ingest ./mixed_corpus/ --recursive --all-mask --output data/clean.jsonl +``` + +Açık flag'lerle additive birleşir — `--all-mask --pii-mask` hata +değildir; iki flag'in boolean unionu çalışır. Kısayol yalnızca +ergonomi içindir; yeni bir detector eklemez. + +--- + ## Recursive dizin yürüyüşü ```text @@ -214,6 +233,7 @@ forgelm ingest INPUT_PATH \ [--recursive] \ [--pii-mask] \ [--secrets-mask] \ + [--all-mask] \ [--output-format {text,json}] \ [--quiet | --log-level {DEBUG,INFO,WARNING,ERROR}] ``` @@ -276,7 +296,7 @@ sayfadan kısa belgelerde devre dışı kalır. Yapılandırılmış notlar ## Sınırlamalar - **OCR:** Kapsam dışı. Harici araçlar kullanın — aşağıdaki örneklere bakın. -- **Tablolar / şekiller:** Faz 12'den (`v0.5.2`) itibaren `_extract_docx()` +- **Tablolar / şekiller:** Faz 12'den (`v0.5.0`'da birleşti) itibaren `_extract_docx()` DOCX tablolarını **Markdown tablo syntax'ına** (header + `---` ayraç + body satırları) **chunking stratejisi çalışmadan önce, extraction aşamasında** dönüştürür — yani tüm stratejiler render edilmiş diff --git a/docs/guides/ingestion.md b/docs/guides/ingestion.md index 4ae8da99..769ab1f5 100644 --- a/docs/guides/ingestion.md +++ b/docs/guides/ingestion.md @@ -1,10 +1,20 @@ # Document Ingestion Guide Convert raw enterprise corpora (PDF / DOCX / EPUB / TXT / Markdown) into the -SFT-ready JSONL ForgeLM trains on. Phase 11; introduced in `v0.5.0`. Phase -11.5 (`v0.5.1`) added token-aware chunking, PDF page header/footer dedup, -and structured ingestion notes — each documented inline in the relevant -section below. +SFT-ready JSONL ForgeLM trains on. Ships with `v0.5.0` (Phases 11 + 11.5 ++ 12 + 12.5 consolidated): paragraph / sliding / markdown chunking, +token-aware sizing, PDF page header/footer dedup, code/credential +scrubbing (`--secrets-mask`), DOCX tables rendered as markdown, and the +`--all-mask` shorthand for combined PII + secrets masking. Earlier +phase boundaries kept inline as breadcrumbs for reviewers; the +behaviours all ship in the same release. + +> **Phase trail (kept for reviewers):** Phase 11 introduced the +> ingestion pipeline; Phase 11.5 added token-aware chunking + PDF page +> header/footer dedup + structured ingestion notes; Phase 12 added +> markdown-aware splitting + secrets scrubbing + DOCX table preservation; +> Phase 12.5 added `--all-mask`. The four phases were consolidated into +> a single `v0.5.0` release. > Pair with [`forgelm audit`](data_audit.md) afterwards to surface > length-distribution / language / near-duplicate / PII metrics, and with @@ -20,7 +30,7 @@ pip install 'forgelm[ingestion]' ``` The extra brings in `pypdf`, `python-docx`, `ebooklib`, `beautifulsoup4`, -`langdetect`, and (since v0.5.1) the optional non-cryptographic `xxhash` +`langdetect`, and the optional non-cryptographic `xxhash` backend used by the simhash digest. They are optional because plain text + Markdown work without any of them. Importing the module does not pull these in unless the matching extractor is exercised; `xxhash`'s absence @@ -130,6 +140,26 @@ installing it today does **not** change ingest masking behaviour. --- +## All-in-one masking — `--all-mask` (Phase 12.5) + +`--all-mask` is a one-flag shorthand for `--secrets-mask --pii-mask` — +the common "scrub everything detectable before training on a shared +corpus" workflow. The two underlying detectors run in the documented +order (secrets first so combined PII detectors don't double-count +overlapping spans); the resulting JSONL carries both `[REDACTED-SECRET]` +and `[REDACTED]` tokens where matches were found. + +```bash +forgelm ingest ./mixed_corpus/ --recursive --all-mask --output data/clean.jsonl +``` + +Composes additively with explicit flags — `--all-mask --pii-mask` is +not an error; the boolean union of the two flags is what runs. The +shorthand exists purely as ergonomics; it does not introduce any new +detector. + +--- + ## Recursive directory walk ```text @@ -204,6 +234,7 @@ forgelm ingest INPUT_PATH \ [--recursive] \ [--pii-mask] \ [--secrets-mask] \ + [--all-mask] \ [--output-format {text,json}] \ [--quiet | --log-level {DEBUG,INFO,WARNING,ERROR}] ``` @@ -296,7 +327,7 @@ assistants, code-with-data prompts. ## Limitations - **OCR:** out of scope. Use external tooling — see the worked example below. -- **Tables / figures:** Since Phase 12 (`v0.5.2`), `_extract_docx()` +- **Tables / figures:** Since Phase 12 (consolidated into `v0.5.0`), `_extract_docx()` converts DOCX tables to **Markdown table syntax** (header + `---` separator + body rows) at extraction time, **before any chunking strategy runs** — so all strategies see the rendered Markdown, not a diff --git a/docs/reference/usage-tr.md b/docs/reference/usage-tr.md index 31d7275b..275f4745 100644 --- a/docs/reference/usage-tr.md +++ b/docs/reference/usage-tr.md @@ -110,7 +110,7 @@ forgelm --config my_config.yaml --generate-data Bu komut, eğitim başlamadan önce bir öğretmen modelden eğitim verisi üretmek için `synthetic` config bölümünü kullanır. Tüm sentetik veri seçenekleri için [Konfigürasyon Rehberi](configuration-tr.md)'ne bakın. -### Doküman Yutma (v0.5.0+; v0.5.1 token-aware; v0.5.2 markdown + secrets-mask) +### Doküman Yutma (v0.5.0) Ham PDF / DOCX / EPUB / TXT / Markdown'ı SFT'ye uygun JSONL'a dönüştürür. Opsiyonel bağımlılık: `pip install forgelm[ingestion]`. Ayrıntılar için [Doküman Yutma Rehberi](../guides/ingestion-tr.md). @@ -128,28 +128,31 @@ forgelm ingest ./scan.pdf --strategy sliding --chunk-size 1024 --overlap 128 \ # Yazmadan önce PII'yi maskele forgelm ingest ./customer_emails/ --pii-mask --output data/anon.jsonl -# Token-aware chunking (v0.5.1) — chunk'ları modelinizin vocab'ına göre boyutlandırır +# Token-aware chunking — chunk'ları modelinizin vocab'ına göre boyutlandırır forgelm ingest ./policies/ --recursive --output data/policies.jsonl \ --chunk-tokens 1024 --tokenizer "Qwen/Qwen2.5-7B-Instruct" -# v0.5.2: teknik wiki / README'ler için markdown-aware splitter +# Teknik wiki / README'ler için markdown-aware splitter forgelm ingest ./engineering_wiki/ --recursive --strategy markdown \ --output data/wiki.jsonl -# v0.5.2: chunk'lar JSONL'a inmeden önce credential'ları temizle +# Chunk'lar JSONL'a inmeden önce credential'ları temizle forgelm ingest ./mixed_corpus/ --secrets-mask --output data/clean.jsonl -# v0.5.2: secrets + PII maskelemeyi birleştir (önce secrets, sonra PII — +# Secrets + PII maskelemeyi birleştir (önce secrets, sonra PII — # örtüşen span'ler iki kez sayılmasın) forgelm ingest ./mixed_corpus/ --secrets-mask --pii-mask --output data/scrubbed.jsonl + +# Yukarıdaki kombinasyonun tek-flag kısayolu +forgelm ingest ./mixed_corpus/ --all-mask --output data/scrubbed.jsonl ``` -### Veri Seti Denetimi (v0.5.0+; v0.5.1 subcommand; v0.5.2 MinHash + quality + secrets) +### Veri Seti Denetimi (v0.5.0) CPU-only kalite + governance denetimi. `data_audit_report.json` üretir. Ayrıntılar için [Denetim Rehberi](../guides/data_audit-tr.md). ```bash -# Tek split (v0.5.1 subcommand) +# Tek split (subcommand) forgelm audit data/sft.jsonl --output ./audit/ # Çoklu split (train.jsonl / validation.jsonl / test.jsonl içeren dizin) @@ -161,13 +164,20 @@ forgelm audit data/ --verbose # Özel Hamming eşiği forgelm audit data/ --near-dup-threshold 5 -# v0.5.2: >50K satırlık korpuslar için MinHash LSH dedup +# >50K satırlık korpuslar için MinHash LSH dedup # Önce extra'yı kurun: pip install forgelm[ingestion-scale] forgelm audit data/large_corpus.jsonl --dedup-method minhash --jaccard-threshold 0.85 -# v0.5.2: opt-in heuristik kalite filtresi (Gopher/C4 stili) +# Opt-in heuristik kalite filtresi (Gopher/C4 stili) forgelm audit data/ --quality-filter +# Audit'in yanında bir Google Croissant 1.0 dataset card emit et +forgelm audit data/ --output ./audit/ --croissant + +# Regex PII detector üzerine Presidio NER katmanı ekle +# (`[ingestion-pii-ml]` extra'sını gerektirir — person/organization/location kategorileri ekler) +forgelm audit data/ --output ./audit/ --pii-ml + # stdout'a makine-okunabilir özet forgelm audit data/sft.jsonl --output ./audit/ --output-format json @@ -175,7 +185,7 @@ forgelm audit data/sft.jsonl --output ./audit/ --output-format json forgelm --data-audit data/sft.jsonl --output ./audit/ ``` -Denetim şunları yakalar: split başına örnek sayısı + uzunluk dağılımı, top-3 dil tespiti, **LSH-banded** simhash near-duplicate oranı (Faz 11.5; uç eşiklerde brute-force fallback; v0.5.2 opsiyonel **MinHash LSH** yolu ekledi), cross-split sızıntı (sessiz train-test örtüşmesi), PII flag sayıları + **şiddet katmanları** (`pii_severity` bloğu her PII tipini critical / high / medium / low olarak puanlar ve bir `worst_tier` verdict yüzdürür), her zaman açık **credentials/secrets taraması** (`secrets_summary` — `forgelm.data_audit.SECRET_TYPES`'ta tanımlı dokuz aile: `aws_access_key`, `github_token`, `slack_token`, `openai_api_key`, `google_api_key`, `jwt`, `openssh_private_key` (tam BEGIN…END bloğu), `pgp_private_key` (tam BEGIN…END bloğu), `azure_storage_key`), ve opt-in **heuristik kalite filtresi** (`--quality-filter`, `quality_summary` bloğu üretir). +Denetim şunları yakalar: split başına örnek sayısı + uzunluk dağılımı, top-3 dil tespiti, **LSH-banded** simhash near-duplicate oranı (uç eşiklerde brute-force fallback; opsiyonel **MinHash LSH** yolu `--dedup-method minhash` ile), cross-split sızıntı (sessiz train-test örtüşmesi), PII flag sayıları + **şiddet katmanları** (`pii_severity` bloğu her PII tipini critical / high / medium / low olarak puanlar ve bir `worst_tier` verdict yüzdürür), her zaman açık **credentials/secrets taraması** (`secrets_summary` — `forgelm.data_audit.SECRET_TYPES`'ta tanımlı dokuz aile: `aws_access_key`, `github_token`, `slack_token`, `openai_api_key`, `google_api_key`, `jwt`, `openssh_private_key` (tam BEGIN…END bloğu), `pgp_private_key` (tam BEGIN…END bloğu), `azure_storage_key`), ve opt-in **heuristik kalite filtresi** (`--quality-filter`, `quality_summary` bloğu üretir). Trainer'ın `output_dir`'ünde `data_audit_report.json` mevcutsa, bulgular EU AI Act Madde 10 governance artifact'ında `data_audit` anahtarı altında otomatik olarak inline edilir. diff --git a/docs/reference/usage.md b/docs/reference/usage.md index ca354f5a..ca254fde 100644 --- a/docs/reference/usage.md +++ b/docs/reference/usage.md @@ -129,7 +129,7 @@ forgelm --config my_config.yaml --generate-data This uses the `synthetic` config section to generate training data from a teacher model before training begins. See the [Configuration Guide](configuration.md) for all synthetic data options. -### Document Ingestion (v0.5.0+; token-aware in v0.5.1; markdown + secrets-mask in v0.5.2) +### Document Ingestion (v0.5.0) Convert raw PDF / DOCX / EPUB / TXT / Markdown into SFT-ready JSONL. Optional dep: `pip install forgelm[ingestion]`. See [Ingestion Guide](../guides/ingestion.md). @@ -147,27 +147,30 @@ forgelm ingest ./scan.pdf --strategy sliding --chunk-size 1024 --overlap 128 \ # Mask PII before writing forgelm ingest ./customer_emails/ --pii-mask --output data/anon.jsonl -# Token-aware chunking (v0.5.1) — sizes chunks against your model's vocab +# Token-aware chunking — sizes chunks against your model's vocab forgelm ingest ./policies/ --recursive --output data/policies.jsonl \ --chunk-tokens 1024 --tokenizer "Qwen/Qwen2.5-7B-Instruct" -# v0.5.2: markdown-aware splitter for technical wikis / READMEs +# Markdown-aware splitter for technical wikis / READMEs forgelm ingest ./engineering_wiki/ --recursive --strategy markdown \ --output data/wiki.jsonl -# v0.5.2: scrub credentials before chunks land in the JSONL +# Scrub credentials before chunks land in the JSONL forgelm ingest ./mixed_corpus/ --secrets-mask --output data/clean.jsonl -# v0.5.2: combine secrets + PII masking (secrets first to avoid double-counting) +# Combine secrets + PII masking (secrets first to avoid double-counting) forgelm ingest ./mixed_corpus/ --secrets-mask --pii-mask --output data/scrubbed.jsonl + +# One-flag shorthand for the same combination above +forgelm ingest ./mixed_corpus/ --all-mask --output data/scrubbed.jsonl ``` -### Dataset Audit (v0.5.0+; subcommand promoted in v0.5.1; MinHash + quality + secrets in v0.5.2) +### Dataset Audit (v0.5.0) CPU-only quality + governance audit. Produces `data_audit_report.json`. See [Audit Guide](../guides/data_audit.md). ```bash -# Single split (v0.5.1 subcommand) +# Single split (subcommand) forgelm audit data/sft.jsonl --output ./audit/ # Multi-split directory (train.jsonl / validation.jsonl / test.jsonl) @@ -179,12 +182,19 @@ forgelm audit data/ --verbose # Custom Hamming threshold for simhash near-duplicate detection forgelm audit data/ --near-dup-threshold 5 -# v0.5.2: MinHash LSH dedup for >50K-row corpora (needs `[ingestion-scale]` extra) +# MinHash LSH dedup for >50K-row corpora (needs `[ingestion-scale]` extra) forgelm audit data/large_corpus.jsonl --dedup-method minhash --jaccard-threshold 0.85 -# v0.5.2: opt-in heuristic quality filter (Gopher/C4 style) +# Opt-in heuristic quality filter (Gopher/C4 style) forgelm audit data/ --quality-filter +# Emit a Google Croissant 1.0 dataset card alongside the audit +forgelm audit data/ --output ./audit/ --croissant + +# Layer Presidio NER on top of the regex PII detector +# (needs `[ingestion-pii-ml]` extra — adds person/organization/location categories) +forgelm audit data/ --output ./audit/ --pii-ml + # Machine-readable summary on stdout forgelm audit data/sft.jsonl --output ./audit/ --output-format json @@ -192,7 +202,7 @@ forgelm audit data/sft.jsonl --output ./audit/ --output-format json forgelm --data-audit data/sft.jsonl --output ./audit/ ``` -The audit captures: per-split sample count + length distribution, top-3 language detection, **LSH-banded** simhash near-duplicate rate (Phase 11.5; brute-force fallback at edge thresholds; v0.5.2 added optional **MinHash LSH** path), cross-split leakage (silent train-test overlap), PII flag counts with **severity tiers**, **always-on credentials/secrets scan** (`secrets_summary` — nine families per `forgelm.data_audit.SECRET_TYPES`: `aws_access_key`, `github_token`, `slack_token`, `openai_api_key`, `google_api_key`, `jwt`, `openssh_private_key` (full BEGIN…END block), `pgp_private_key` (full BEGIN…END block), `azure_storage_key`), and an opt-in **heuristic quality filter** (Gopher/C4 style) that adds a `quality_summary` block. +The audit captures: per-split sample count + length distribution, top-3 language detection, **LSH-banded** simhash near-duplicate rate (brute-force fallback at edge thresholds; optional **MinHash LSH** path via `--dedup-method minhash`), cross-split leakage (silent train-test overlap), PII flag counts with **severity tiers**, **always-on credentials/secrets scan** (`secrets_summary` — nine families per `forgelm.data_audit.SECRET_TYPES`: `aws_access_key`, `github_token`, `slack_token`, `openai_api_key`, `google_api_key`, `jwt`, `openssh_private_key` (full BEGIN…END block), `pgp_private_key` (full BEGIN…END block), `azure_storage_key`), and an opt-in **heuristic quality filter** (Gopher/C4 style) that adds a `quality_summary` block. When `data_audit_report.json` is present in the trainer's `output_dir` at training time, its findings are inlined under the `data_audit` key of the EU AI Act Article 10 governance artifact automatically. diff --git a/docs/roadmap-tr.md b/docs/roadmap-tr.md index aa4cef5c..6ecc748c 100644 --- a/docs/roadmap-tr.md +++ b/docs/roadmap-tr.md @@ -9,24 +9,26 @@ | ✅ Tamam | [Faz 1-9](roadmap/completed-phases.md) | SOTA iyileştirmeleri, değerlendirme, güvenilirlik, kurumsal entegrasyon, ekosistem, hizalama stack'i, güvenlik, EU AI Act uyumluluğu (Madde 9-17 + Ek IV), gelişmiş güvenlik zekası | | ✅ Tamam | [Faz 10 — Post-Training Tamamlama](roadmap/phase-10-post-training.md) | `inference.py`, `chat`, `export` (GGUF), `--fit-check`, `deploy` — `v0.4.0` | | ✅ Tamam | [Faz 10.5 — Quickstart Katmanı ve Onboarding](roadmap/phase-10-5-quickstart.md) | `forgelm quickstart