From 2f5722ab9852b64bd0d245c182b4dd674352b4bf Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 28 Apr 2026 02:43:36 +0300 Subject: [PATCH 01/29] feat(phase-12): Data Curation Maturity (v0.5.2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Direct continuation of the Phase 11 / 11.5 ingestion + audit lineage. Closes the four concrete gaps surfaced by the post-`v0.5.1` competitive review (LLaMA-Factory / Axolotl / Unsloth / NeMo Curator / Dolma / RedPajama / LlamaIndex / LangChain / Marker / Docling). All five Tier 1 must-haves landed in this commit; Tier 2/3 (Presidio adapter, Croissant metadata, --all-mask, wizard "audit first") are deferred to a Phase 12.5 backlog file. Tier 1 features --------------- * MinHash LSH dedup option (`forgelm/data_audit.py`): `compute_minhash`, `find_near_duplicates_minhash`, `_count_leaked_rows_minhash`. Opt-in `--dedup-method minhash --jaccard-threshold 0.85` via the optional `[ingestion-scale]` extra (datasketch>=1.6.0). Default simhash + LSH banding from Phase 11.5 stays untouched and remains the only method that runs without an optional dependency. `audit_dataset(...)` API gains `dedup_method`, `minhash_jaccard`, `minhash_num_perm` parameters; `near_duplicate_summary.method` records which path ran. Cross-split overlap and within-split duplicate scan share the same method flag. * Markdown-aware splitter (`forgelm/ingestion.py`): `_chunk_markdown`, `_chunk_markdown_tokens`, `_markdown_sections`, `_heading_breadcrumb`. New `--strategy markdown` parses heading hierarchy (`# H1` … `###### H6`), keeps code-fenced blocks atomic (heading-shaped lines inside ```` ``` ```` are not interpreted as section boundaries), and inlines a heading **breadcrumb** at the top of each chunk so SFT loss sees document context. Composes with the Phase 11.5 token-aware mode. * Code/secret leakage tagger (`forgelm/data_audit.py`): `detect_secrets`, `mask_secrets`, `_SECRET_PATTERNS`. Always-on audit-side scan with prefix-anchored regex set covering AWS access keys (`AKIA…` / `ASIA…`), GitHub PATs, Slack tokens, OpenAI API keys (`sk-…` / `sk-proj-…`), Google API keys, JWTs, OpenSSH/RSA/DSA/EC/PGP private-key block headers, Azure storage connection strings. Adds `secrets_summary` block alongside `pii_summary`. Ingest side: `--secrets-mask` rewrites detected spans with `[REDACTED-SECRET]`; runs **before** PII masking when both flags are set so combined detectors don't double-count overlapping spans (e.g. GitHub tokens that partially match the German Personalausweis regex). Optional `[ingestion-secrets]` extra (detect-secrets>=1.5.0) for fuller plugin coverage; bare install gets the regex fallback. * Heuristic quality filter (`forgelm/data_audit.py`): `_row_quality_flags`, `_QUALITY_CHECKS`. Opt-in `forgelm audit --quality-filter` runs Gopher / C4 / RefinedWeb-style checks per row: `low_alpha_ratio`, `low_punct_endings`, `abnormal_mean_word_length`, `short_paragraphs`. Surfaces `quality_summary` with per-check counts, `samples_flagged`, and `overall_quality_score`. ML-based classifiers deliberately out of scope to keep the audit deterministic for Annex IV reproducibility. * DOCX/Markdown table preservation (`forgelm/ingestion.py`): `_docx_table_to_markdown`. `_extract_docx` now renders tables as markdown table syntax (header row + `---` separator + body rows) instead of the previous `" | "`-joined flat line. Uneven rows are right-padded; all-blank rows dropped; first non-empty row becomes the header. Combined with `--strategy markdown` the table block stays intact across chunks. API additions (additive — Phase 11/11.5 consumers unaffected) ------------------------------------------------------------- * `AuditReport.secrets_summary: Dict[str, int]` and `AuditReport.quality_summary: Dict[str, Any]`. * `IngestionResult.secrets_redaction_counts: dict`. * `audit_dataset(..., dedup_method, minhash_jaccard, minhash_num_perm, enable_quality_filter)` keyword args. * `ingest_path(..., secrets_mask=False)` keyword arg. * New constants: `DEDUP_METHODS`, `DEFAULT_MINHASH_JACCARD`, `DEFAULT_MINHASH_NUM_PERM`, `SECRET_TYPES`. CLI additions ------------- * `forgelm ingest`: `--strategy markdown`, `--secrets-mask`. * `forgelm audit`: `--dedup-method {simhash,minhash}`, `--jaccard-threshold X` (validated to `[0.0, 1.0]` at parse time via new `_non_negative_float` argparse helper), `--quality-filter`. * `_run_data_audit` distinguishes `EXIT_CONFIG_ERROR` (filesystem) from `EXIT_TRAINING_ERROR` (missing `[ingestion-scale]` extra when `--dedup-method=minhash` was requested). pyproject.toml -------------- * New extras: `[ingestion-scale]` (datasketch>=1.6.0,<2.0.0), `[ingestion-secrets]` (detect-secrets>=1.5.0,<2.0.0). * Version bump `0.5.1rc1 → 0.5.2rc1`. Tests ----- * `tests/test_data_audit_phase12.py` — 18 new tests (TestSecretsDetection, TestSecretsMasking, TestAuditPicksUpSecrets, TestQualityFilterPerRow, TestQualityFilterEnabled, TestMinHashLshDedup [skipped without datasketch], TestMinHashMissingExtra). * `tests/test_ingestion_phase12.py` — 13 new tests (TestMarkdownSections, TestChunkMarkdown, TestMarkdownStrategyExposed, TestDocxTableToMarkdown with `_FakeCell`/`_FakeRow`/`_FakeTable` helpers, TestSecretsMaskIngest). * `tests/test_cli_subcommands.py` — `test_audit_quality_filter_flag`, `test_audit_rejects_invalid_jaccard_threshold` added to TestAuditSubcommand. Internal refactors ------------------ * `_StreamingAggregator` gains `minhashes`, `secrets_counts`, `quality_flags_counts`, `quality_samples_flagged`, `dedup_method`, `minhash_num_perm`, `enable_quality_filter` fields. * `_SplitOutcome.fingerprints` → `_SplitOutcome.signatures` (the same field carries simhash ints OR MinHash instances depending on method). * `_audit_split` returns `(info, signatures, pii_split, parse_errors, decode_errors)` where `signatures` is method-dependent. * `_pair_leak_payload` and `_cross_split_overlap` switched to keyword-only `dedup_method` parameter and dispatch on it (simhash → Hamming; minhash → Jaccard). * `describe_strategies()` lists `markdown` alongside the others. Documentation ------------- * `docs/roadmap/phase-12-data-curation-maturity.md`: marked Tier 1 done; Tier 2/3 noted as deferred. * `docs/roadmap/phase-12-5-backlog.md`: new follow-up backlog file (mirrors Phase 11.5's pattern). * `docs/roadmap.md` + `roadmap-tr.md` + `roadmap/releases.md`: Phase 12 → Done, mermaid diagram updated, milestone text refreshed. * `docs/guides/{ingestion,data_audit}{,-tr}.md`: new sections for markdown splitter, secrets scan, quality filter, MinHash, DOCX table preservation; CLI references updated. * `docs/reference/usage{,-tr}.md`: code examples for the four new flags + the markdown strategy. * `docs/standards/architecture.md`: extras matrix gains `ingestion-scale` and `ingestion-secrets` rows. * `docs/qms/sop_data_management.md`: quality-check checklist gains `secrets_summary` (stop-the-line) and `quality_summary` rows; `forgelm --data-audit` legacy spelling updated to `forgelm audit`. * `README.md`: feature blurb refreshed for the v0.5.2 surface; separate "Secrets-Aware Ingest" bullet added. * `CHANGELOG.md`: detailed Unreleased — Phase 12 section. Verification ------------ * `ruff check` clean, `ruff format` clean. * `pytest tests/` — 792 passed, 12 skipped (vs. previous 760 / 8; +32 tests, +4 datasketch-skips). All Phase 11 / 11.5 contracts preserved. * End-to-end smoke (`/tmp/p12_smoke/`): - `forgelm audit data.jsonl` → secrets_summary populates, "Secrets : CRITICAL — 1 flagged" verdict line. - `forgelm audit --quality-filter` → quality_summary populates, "Quality : N sample(s) flagged" verdict. - `forgelm ingest wiki.md --strategy markdown --secrets-mask` → breadcrumb-inlined chunks, AWS key replaced with `[REDACTED-SECRET]`, IngestionResult.secrets_redaction_counts surfaces the count. --- CHANGELOG.md | 43 ++ README.md | 7 +- docs/guides/data_audit-tr.md | 14 +- docs/guides/data_audit.md | 138 +++- docs/guides/ingestion-tr.md | 5 +- docs/guides/ingestion.md | 71 +- docs/qms/sop_data_management.md | 38 +- docs/reference/usage-tr.md | 4 +- docs/reference/usage.md | 24 +- docs/roadmap-tr.md | 13 +- docs/roadmap.md | 13 +- docs/roadmap/phase-12-5-backlog.md | 40 + .../phase-12-data-curation-maturity.md | 14 +- docs/roadmap/releases.md | 26 +- docs/standards/architecture.md | 4 +- forgelm/cli.py | 123 +++- forgelm/data_audit.py | 691 ++++++++++++++++-- forgelm/ingestion.py | 276 ++++++- pyproject.toml | 19 +- tests/test_cli_subcommands.py | 36 + tests/test_data_audit_phase12.py | 234 ++++++ tests/test_ingestion_phase12.py | 216 ++++++ 22 files changed, 1905 insertions(+), 144 deletions(-) create mode 100644 docs/roadmap/phase-12-5-backlog.md create mode 100644 tests/test_data_audit_phase12.py create mode 100644 tests/test_ingestion_phase12.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 13d64ab2..733edf31 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,49 @@ All notable changes to ForgeLM are documented here. ## [Unreleased] +### Added — Phase 12 (Data Curation Maturity, targeting v0.5.2) + +Direct continuation of the Phase 11 / 11.5 ingestion + audit lineage. Closes the four concrete gaps surfaced by the post-`v0.5.1` competitive review (LLaMA-Factory / Axolotl / Unsloth / NeMo Curator / Dolma / RedPajama / LlamaIndex / LangChain / Marker / Docling). Tier 1 (5 must-have tasks) shipped; Tier 2/3 (Presidio adapter, Croissant metadata, `--all-mask`, wizard "audit first") deferred to a [Phase 12.5 backlog](docs/roadmap/phase-12-5-backlog.md). + +- **MinHash LSH dedup option** (`forgelm/data_audit.py`: `compute_minhash`, `find_near_duplicates_minhash`, `_count_leaked_rows_minhash`) — Opt-in `--dedup-method minhash --jaccard-threshold 0.85` route via the optional `[ingestion-scale]` extra (`datasketch>=1.6.0`). Default simhash + LSH banding from Phase 11.5 stays untouched and remains the only method that runs without an optional dependency. `audit_dataset(...)` API gains `dedup_method`, `minhash_jaccard`, `minhash_num_perm` parameters; `near_duplicate_summary.method` records which path ran. Cross-split overlap + within-split duplicate scan share the same method flag. MinHash is approximate (permutation noise; default `num_perm=128`) — pin `num_perm` for cross-run determinism. +- **Markdown-aware splitter** (`forgelm/ingestion.py`: `_chunk_markdown`, `_chunk_markdown_tokens`, `_markdown_sections`, `_heading_breadcrumb`) — New `--strategy markdown` parses heading hierarchy (`# H1` … `###### H6`), keeps code-fenced blocks atomic (heading-shaped lines inside ```` ``` ```` blocks are not interpreted as section boundaries), and inlines a heading **breadcrumb** at the top of each chunk so SFT loss sees the document context. Composes with the Phase 11.5 token-aware mode (`--chunk-tokens` + `--tokenizer`). +- **Code / secret leakage tagger** (`forgelm/data_audit.py`: `detect_secrets`, `mask_secrets`, `_SECRET_PATTERNS`) — Always-on audit-side scan with prefix-anchored regex set covering AWS access keys (`AKIA…` / `ASIA…`), GitHub PATs (`ghp_`, `gho_`, `ghs_`, `ghu_`, `ghr_`, `github_pat_`), Slack tokens, OpenAI API keys (`sk-…` / `sk-proj-…`), Google API keys, JWTs, OpenSSH/RSA/DSA/EC/PGP private-key block headers, Azure storage connection strings. Adds a `secrets_summary` block alongside `pii_summary`. Ingest side: `forgelm ingest --secrets-mask` rewrites detected spans with `[REDACTED-SECRET]`; runs **before** PII masking when both flags are set so combined detectors don't double-count overlapping spans (e.g. GitHub tokens that partially match the German Personalausweis regex). Optional `[ingestion-secrets]` extra (`detect-secrets>=1.5.0`) for fuller plugin coverage; bare install gets the regex fallback with an INFO log pointing to the install hint. +- **Heuristic quality filter** (`forgelm/data_audit.py`: `_row_quality_flags`, `_QUALITY_CHECKS`) — Opt-in `forgelm audit --quality-filter` runs Gopher / C4 / RefinedWeb-style checks per row: `low_alpha_ratio` (< 70 % letters among non-whitespace), `low_punct_endings` (< 50 % of non-empty lines end with punctuation), `abnormal_mean_word_length` (outside 3–12 chars), `short_paragraphs` (> 50 % of `\n\n`-blocks have < 5 words). Surfaces `quality_summary` with per-check counts, `samples_flagged`, and `overall_quality_score`. ML-based classifiers (fastText / DeBERTa) deliberately out of scope — keeps the audit deterministic for Annex IV reproducibility. +- **DOCX / Markdown table preservation** (`forgelm/ingestion.py`: `_docx_table_to_markdown`) — `_extract_docx` now renders tables as markdown table syntax (header row + `---` separator + body rows) instead of the previous `" | "`-joined flat line. Uneven rows are right-padded with empty cells; all-blank rows are dropped; the first non-empty row becomes the header (no heuristic — that's the convention DOCX authors use). Combined with `--strategy markdown` the table block stays intact across chunks. + +### Public API additions + +- `AuditReport` gains `secrets_summary: Dict[str, int]` and `quality_summary: Dict[str, Any]` fields (additive — Phase 11/11.5 consumers reading just `pii_summary` / `near_duplicate_summary` keep working). +- `IngestionResult` gains `secrets_redaction_counts: dict` field. +- `audit_dataset(...)` accepts `dedup_method`, `minhash_jaccard`, `minhash_num_perm`, `enable_quality_filter` keyword arguments. +- `ingest_path(...)` accepts `secrets_mask: bool` keyword argument. +- New constants: `DEDUP_METHODS`, `DEFAULT_MINHASH_JACCARD`, `DEFAULT_MINHASH_NUM_PERM`, `SECRET_TYPES`. + +### CLI additions + +- `forgelm ingest`: `--strategy markdown`, `--secrets-mask`. +- `forgelm audit`: `--dedup-method {simhash,minhash}`, `--jaccard-threshold X` (validated to `[0.0, 1.0]` at parse time), `--quality-filter`. +- New argparse type helper `_non_negative_float` (mirrors `_non_negative_int`'s pattern). +- `_run_data_audit` now distinguishes `EXIT_CONFIG_ERROR` (filesystem/path errors) from `EXIT_TRAINING_ERROR` (missing `[ingestion-scale]` extra when `--dedup-method=minhash` was requested). + +### `pyproject.toml` + +- New extras: `[ingestion-scale]` (`datasketch>=1.6.0,<2.0.0`), `[ingestion-secrets]` (`detect-secrets>=1.5.0,<2.0.0`). +- Version bump `0.5.1rc1 → 0.5.2rc1`. + +### Tests + +- `tests/test_data_audit_phase12.py` — 18 new tests across `TestSecretsDetection`, `TestSecretsMasking`, `TestAuditPicksUpSecrets`, `TestQualityFilterPerRow`, `TestQualityFilterEnabled`, `TestMinHashLshDedup` (skipped without `datasketch`), `TestMinHashMissingExtra`. +- `tests/test_ingestion_phase12.py` — 13 new tests across `TestMarkdownSections`, `TestChunkMarkdown`, `TestMarkdownStrategyExposed`, `TestDocxTableToMarkdown`, `TestSecretsMaskIngest`. +- `tests/test_cli_subcommands.py` — `test_audit_quality_filter_flag`, `test_audit_rejects_invalid_jaccard_threshold` added to `TestAuditSubcommand`. + +### Changed (no behavioural delta unless noted) + +- `_StreamingAggregator` gains `minhashes`, `secrets_counts`, `quality_flags_counts`, `quality_samples_flagged`, `dedup_method`, `minhash_num_perm`, `enable_quality_filter` fields. Field rename: `_SplitOutcome.fingerprints` → `_SplitOutcome.signatures` (the same field carries simhash ints OR MinHash instances, depending on method). +- `_audit_split(...)` now returns `(info, signatures, pii_split, parse_errors, decode_errors)` where `signatures` is method-dependent. `_process_split` and `audit_dataset` were updated in lockstep. +- `_pair_leak_payload` and `_cross_split_overlap` switched to keyword-only `dedup_method` parameter and dispatch on it (simhash → Hamming; minhash → Jaccard). +- `describe_strategies()` now lists `markdown` alongside `sliding` / `paragraph` / `semantic`. + ### Added — Phase 11.5 (Ingestion / Audit Polish, targeting v0.5.1) Operational polish on top of `v0.5.0`'s ingestion + audit surface — no new training capabilities, but materially better handling for large corpora and a cleaner CLI shape. All 12 follow-ups carved out of Phase 11's review backlog. diff --git a/README.md b/README.md index 0e7f0806..ca7dce46 100644 --- a/README.md +++ b/README.md @@ -27,12 +27,13 @@ - **LLM-as-Judge**: API-based (OpenAI) or local model scoring for quality assessment - **Auto-Revert**: Automatically discard models that fail loss, benchmark, or safety thresholds -### Document Ingestion & Data Audit (v0.5.0 — polished in v0.5.1) -- **Multi-Format Ingestion**: `forgelm ingest ./policies/ --recursive --output data/policies.jsonl` — turns raw PDF / DOCX / EPUB / TXT / Markdown into the SFT-ready JSONL the trainer accepts. Optional dep: `pip install forgelm[ingestion]`. +### Document Ingestion & Data Audit (v0.5.0 — polished in v0.5.1, matured in v0.5.2) +- **Multi-Format Ingestion**: `forgelm ingest ./policies/ --recursive --output data/policies.jsonl` — turns raw PDF / DOCX / EPUB / TXT / Markdown into the SFT-ready JSONL the trainer accepts. Optional dep: `pip install forgelm[ingestion]`. v0.5.2 added a **markdown-aware splitter** (`--strategy markdown`) and DOCX table preservation in markdown table syntax. - **Chunking Strategies**: `paragraph` (default; preserves boundaries) or `sliding` (fixed window with overlap). Token-aware mode in v0.5.1: `--chunk-tokens 1024 --tokenizer Qwen/Qwen2.5-7B-Instruct` sizes chunks against your model's actual vocabulary. - **PII Masking on Ingest**: `--pii-mask` redacts emails, phones, credit cards (Luhn-validated), IBAN, and national IDs (TR / DE / FR / US-SSN) before chunks land in the JSONL. - **PDF Page Header/Footer Dedup (v0.5.1)**: Lines that recur on ≥ 70 % of PDF pages (watermarks, page numbers, copyright lines) are stripped automatically — the audit's near-duplicate counts stop misfiring on long policy / book PDFs. -- **Dataset Audit**: `forgelm audit data/sft.jsonl --output ./audit/` — produces `data_audit_report.json` with sample count, length distribution, top-3 language detection, LSH-banded near-duplicate rate (`O(n × k)` typical case, exact recall at the default Hamming threshold), cross-split leakage check, null/empty rate, and PII flag counts with **severity tiers** (critical / high / medium / low + worst-tier verdict). CPU-only; streaming JSONL reader keeps memory bounded on multi-million-row splits; feeds EU AI Act Article 10 governance artifact automatically when present at training time. Legacy `--data-audit` flag still works as a deprecation alias. +- **Dataset Audit**: `forgelm audit data/sft.jsonl --output ./audit/` — produces `data_audit_report.json` with sample count, length distribution, top-3 language detection, LSH-banded near-duplicate rate (`O(n × k)` typical case, exact recall at the default Hamming threshold; v0.5.2 added optional **MinHash LSH** via `--dedup-method minhash` for >50K-row corpora), cross-split leakage check, null/empty rate, and PII flag counts with **severity tiers** (critical / high / medium / low + worst-tier verdict). v0.5.2 also added an always-on **secrets/credential scan** (AWS / GitHub / Slack / OpenAI / Google / JWT / private-key headers) and an opt-in **heuristic quality filter** (`--quality-filter`). CPU-only; streaming JSONL reader keeps memory bounded on multi-million-row splits; feeds EU AI Act Article 10 governance artifact automatically when present at training time. Legacy `--data-audit` flag still works as a deprecation alias. +- **Secrets-Aware Ingest (v0.5.2)**: `forgelm ingest … --secrets-mask` scrubs credentials before chunks land in the JSONL — fine-tuning on text containing real API keys memorises them at training time. Pairs with `--pii-mask`; secrets run first so combined detectors don't double-count overlapping spans. ### Quickstart Layer (v0.4.5) - **One-Command Templates**: `forgelm quickstart customer-support` — bundled templates for SFT, code, BYOD domain expert, Turkish medical Q&A, and GRPO math reasoning. Auto-downsizes models on small GPUs. diff --git a/docs/guides/data_audit-tr.md b/docs/guides/data_audit-tr.md index 71038569..ee1dfd0c 100644 --- a/docs/guides/data_audit-tr.md +++ b/docs/guides/data_audit-tr.md @@ -6,7 +6,10 @@ governance ile PII sinyallerini kapsayan bir `data_audit_report.json` ise top-level flag'i first-class subcommand'a yükseltti, LSH-banded near-duplicate tespitini, streaming JSONL okuyucusunu, PII şiddet katmanlarını, atomik disk yazımı ve verbose-by-default kısaltma -politikasını ekledi. +politikasını ekledi. **Faz 12 (`v0.5.2`)** opt-in MinHash LSH dedup +yöntemini (`--dedup-method minhash`), her zaman çalışan +code/credential leakage taramasını (`secrets_summary`), ve opt-in +heuristic kalite filtresini (`--quality-filter`) ekledi. Trainer'ın `output_dir`'ünde mevcutsa, rapor EU AI Act Madde 10 veri governance artifact'ına otomatik olarak beslenir. @@ -25,8 +28,15 @@ forgelm audit data/ --output ./audit/ # Bulgu olmayan split'leri de göster forgelm audit data/ --verbose -# Daha geniş / dar near-duplicate eşiği +# Daha geniş / dar simhash near-duplicate eşiği forgelm audit data/ --near-dup-threshold 5 + +# Faz 12: 50K+ satır corpus için MinHash LSH (`[ingestion-scale]` extra) +pip install 'forgelm[ingestion-scale]' +forgelm audit data/large_corpus.jsonl --dedup-method minhash --jaccard-threshold 0.85 + +# Faz 12: opt-in heuristic kalite filtresi (Gopher/C4 stili) +forgelm audit data/ --quality-filter ``` > **Eski alias:** `forgelm --data-audit PATH` deprecation alias'ı diff --git a/docs/guides/data_audit.md b/docs/guides/data_audit.md index 54bb0d19..2a808f68 100644 --- a/docs/guides/data_audit.md +++ b/docs/guides/data_audit.md @@ -1,12 +1,16 @@ # Dataset Audit Guide `forgelm audit PATH` analyzes a JSONL dataset and produces a -`data_audit_report.json` covering quality, governance, and PII signals. -Phase 11 (introduced in `v0.5.0`) shipped the underlying audit; -Phase 11.5 (`v0.5.1`) promoted it from a top-level flag to a first-class -subcommand and added LSH-banded near-duplicate detection, streaming -JSONL reading, PII severity tiers, an atomic on-disk write, and a -verbose-by-default truncation policy on the human summary. +`data_audit_report.json` covering quality, governance, PII, and (Phase 12 +onwards) credential-leakage and heuristic-quality signals. Phase 11 +(`v0.5.0`) shipped the underlying audit; Phase 11.5 (`v0.5.1`) promoted +it from a top-level flag to a first-class subcommand and added +LSH-banded near-duplicate detection, streaming JSONL reading, PII +severity tiers, atomic on-disk write, and a verbose-by-default +truncation policy. **Phase 12 (`v0.5.2`)** added an opt-in MinHash LSH +dedup method (`--dedup-method minhash`), a code/credential leakage scan +that runs always-on (`secrets_summary`), and an opt-in heuristic +quality filter (`--quality-filter`). The report feeds the EU AI Act Article 10 data governance artifact automatically when present in the trainer's `output_dir`. @@ -25,8 +29,15 @@ forgelm audit data/ --output ./audit/ # Show every split (including those with no findings) forgelm audit data/ --verbose -# Tighter / wider near-duplicate threshold +# Tighter / wider simhash near-duplicate threshold forgelm audit data/ --near-dup-threshold 5 + +# Phase 12: MinHash LSH dedup for >50K-row corpora (needs `[ingestion-scale]` extra) +pip install 'forgelm[ingestion-scale]' +forgelm audit data/large_corpus.jsonl --dedup-method minhash --jaccard-threshold 0.85 + +# Phase 12: opt-in heuristic quality filter (Gopher/C4 style) +forgelm audit data/ --quality-filter ``` > **Legacy alias:** `forgelm --data-audit PATH` keeps working unchanged @@ -185,6 +196,106 @@ Phase 11.5 also made the simhash backend pluggable: a corpus's traffic with a small footprint, which is where most of the real wall-clock improvement comes from. +### MinHash LSH dedup (Phase 12) + +For corpora above ~50K rows, simhash + LSH banding starts to feel its +edge: the band-bucket fan-out grows and false-positive checks dominate +the wall clock. Phase 12 adds an opt-in **MinHash LSH** path via the +optional `[ingestion-scale]` extra (the `datasketch` package). Surface: + +```bash +pip install 'forgelm[ingestion-scale]' +forgelm audit data/large_corpus.jsonl \ + --dedup-method minhash --jaccard-threshold 0.85 +``` + +```python +from forgelm.data_audit import audit_dataset +audit_dataset("data/large_corpus.jsonl", + dedup_method="minhash", minhash_jaccard=0.85) +``` + +The two methods are **not interchangeable on identical thresholds** — +simhash at Hamming distance ≤ 3 ≈ MinHash at Jaccard ≥ 0.85 in +practice, but the underlying definitions of "similar" differ. MinHash +is approximate (permutation noise; default `num_perm=128`), so the same +pair can be flagged at slightly different similarity scores between +runs of MinHash itself when `num_perm` changes. Pin `num_perm` if you +need cross-run determinism. Audit JSON gains a +`near_duplicate_summary.method` field that records which path ran; +older consumers reading just `pairs_per_split` continue to work +unchanged. + +### Code / secret leakage tagger (Phase 12, always-on) + +Audit now scans every row for credentials and tokens that should never +have entered an SFT corpus — fine-tuning on text that contains a real +API key memorises that key in the model. The detector uses a narrow +prefix-anchored regex set (false-positive rate intentionally low) and +emits a `secrets_summary` block alongside `pii_summary`: + +```json +{ + "secrets_summary": { + "aws_access_key": 1, + "github_token": 2, + "openai_api_key": 1 + } +} +``` + +Coverage: AWS access keys (`AKIA…` / `ASIA…`), GitHub PATs (`ghp_`, +`gho_`, `ghs_`, `ghu_`, `ghr_`, `github_pat_`), Slack tokens (`xox[baprs]-`), +OpenAI API keys (`sk-…` and project-scoped `sk-proj-…`), Google API +keys (`AIza…`), JSON Web Tokens (anchored on the `eyJ`-encoded JSON +header), OpenSSH / RSA / DSA / EC / PGP private-key block headers, and +Azure storage connection strings. + +Operator-side, two paths to scrub these out before training: + +```bash +# Pre-process: rewrite the JSONL after the fact via the helper API +python -c "from forgelm.data_audit import mask_secrets; \ + print(mask_secrets(open('data.jsonl').read()))" > data_clean.jsonl + +# Or scrub during ingest (Phase 12; before chunks ever land in JSONL) +forgelm ingest ./policies/ --recursive --output data/policies.jsonl --secrets-mask +``` + +Optional: install `[ingestion-secrets]` to layer `detect-secrets`'s +plugin set on top of the regex fallback. + +### Heuristic quality filter (Phase 12, opt-in) + +`forgelm audit --quality-filter` runs Gopher / C4 / RefinedWeb-style +heuristics per row and surfaces a `quality_summary` block: + +```json +{ + "quality_summary": { + "samples_flagged": 47, + "by_check": { + "low_alpha_ratio": 12, + "low_punct_endings": 8, + "abnormal_mean_word_length": 3, + "short_paragraphs": 27 + }, + "overall_quality_score": 0.94 + } +} +``` + +Checks (all conservative; no row is silently dropped): + +- `low_alpha_ratio` — < 70 % of non-whitespace chars are letters. +- `low_punct_endings` — < 50 % of non-empty lines end with punctuation. +- `abnormal_mean_word_length` — outside the 3-12 char window. +- `short_paragraphs` — > 50 % of `\n\n`-separated blocks have < 5 words. + +ML-based quality classifiers (fastText / DeBERTa style) are deliberately +**out of scope**; a deterministic regex/length/structure pipeline keeps +the audit reproducible (Annex IV requirement) and bare-install-friendly. + --- ## Layout requirements @@ -234,6 +345,9 @@ forgelm audit PATH \ [--output DIR] \ [--verbose] \ [--near-dup-threshold N] \ + [--dedup-method {simhash,minhash}] \ + [--jaccard-threshold X] \ + [--quality-filter] \ [--output-format {text,json}] \ [--quiet | --log-level {DEBUG,INFO,WARNING,ERROR}] ``` @@ -242,8 +356,14 @@ forgelm audit PATH \ `./audit/`. `--verbose` shows every split in the human summary even when it has zero findings (default folds clean splits into one tail line so multi-split audits stay short — has no effect on the on-disk JSON -report). `--near-dup-threshold N` overrides the default Hamming-distance -cutoff of 3 (≈95 % similarity). +report). `--near-dup-threshold N` overrides the default simhash +Hamming-distance cutoff of 3 (≈95 % similarity); ignored when +`--dedup-method=minhash`. `--dedup-method` (Phase 12) selects the +near-duplicate engine — `simhash` (default) or `minhash` (needs +`[ingestion-scale]` extra; `--jaccard-threshold` controls the cutoff, +default 0.85). `--quality-filter` (Phase 12) opts into the heuristic +quality scoring. The credential/secrets scan is **always on** — there +is no flag to disable it. > **Note:** This matches the behavior summarised at the top of this guide: > `--output-format json` writes a small envelope (success flag, top-level diff --git a/docs/guides/ingestion-tr.md b/docs/guides/ingestion-tr.md index 5290ee13..18c0cba0 100644 --- a/docs/guides/ingestion-tr.md +++ b/docs/guides/ingestion-tr.md @@ -3,7 +3,10 @@ Ham kurumsal külliyatı (PDF / DOCX / EPUB / TXT / Markdown) ForgeLM'in eğittiği SFT-uyumlu JSONL'a dönüştürün. Faz 11; `v0.5.0`'da tanıtıldı. Faz 11.5 (`v0.5.1`) token-aware chunking, PDF sayfa header/footer dedup -ve yapılandırılmış ingestion notları ekledi. +ve yapılandırılmış ingestion notları ekledi. **Faz 12 (`v0.5.2`)** +markdown-aware splitter (`--strategy markdown`), code/credential +leakage scrubbing (`--secrets-mask`) ve DOCX tablolarının markdown +syntax'ı ile çıkarılmasını ekledi. > Sonrasında [`forgelm audit`](data_audit-tr.md) ile uzunluk dağılımı / > dil / near-duplicate / PII metriklerini yüzeye çıkarın; chunk'ları Q&A diff --git a/docs/guides/ingestion.md b/docs/guides/ingestion.md index 4bed59c5..09949553 100644 --- a/docs/guides/ingestion.md +++ b/docs/guides/ingestion.md @@ -60,6 +60,7 @@ preprocessing required. |---|---|---| | `paragraph` (default) | Prose, policy docs, articles | Greedy paragraph packer; never splits a paragraph mid-sentence. | | `sliding` | Long technical documents, code mixed with prose | Fixed-size character window with `--overlap` for context bleed. | +| `markdown` (Phase 12) | Markdown docs, technical wikis, READMEs | Heading-aware: chunks at `# H1` / `## H2` boundaries, keeps code-fenced blocks atomic, inlines a heading breadcrumb so SFT loss sees document context. | > Semantic / embedding-based chunking is reserved for a follow-up phase — it > raises `NotImplementedError` today and is intentionally hidden from the CLI @@ -97,6 +98,30 @@ Detected spans are replaced with `[REDACTED]`. Detection is regex-based — false positives are intentional. Audit your output afterwards with `forgelm audit` to verify. +### Secrets masking (Phase 12) + +Add `--secrets-mask` to scrub credentials and tokens before chunks land +in the JSONL — fine-tuning a model on a corpus that contains a real API +key memorises that key in the model. Detected categories: AWS access +keys (`AKIA…`), GitHub PATs (`ghp_…` and friends), Slack tokens, OpenAI +API keys (`sk-…` / `sk-proj-…`), Google API keys (`AIza…`), JWTs, +OpenSSH/RSA/DSA/EC/PGP private-key block headers, Azure storage +connection strings. + +```bash +# Standalone +forgelm ingest ./engineering_wiki/ --output data/wiki.jsonl --secrets-mask + +# Combined with PII masking — secrets run first, PII second so combined +# detectors can't double-count overlapping spans +forgelm ingest ./mixed_corpus/ --output data/clean.jsonl --secrets-mask --pii-mask +``` + +Detected spans are replaced with `[REDACTED-SECRET]`. Optional extra +`pip install 'forgelm[ingestion-secrets]'` adds `detect-secrets`'s +plugin set on top; without it the regex-only fallback covers ~10 +common patterns and an INFO log says so. + --- ## Recursive directory walk @@ -169,17 +194,18 @@ forgelm ingest INPUT_PATH \ [--chunk-size N | --chunk-tokens N --tokenizer MODEL_NAME] \ [--overlap N] \ [--overlap-tokens N] \ - [--strategy {sliding,paragraph}] \ + [--strategy {sliding,paragraph,markdown}] \ [--recursive] \ [--pii-mask] \ + [--secrets-mask] \ [--output-format {text,json}] \ [--quiet | --log-level {DEBUG,INFO,WARNING,ERROR}] ``` `--output-format json` emits a machine-readable summary to stdout (file -paths, chunk count, format counts, notes, and `notes_structured` — -machine-readable `{key: value}` introduced in Phase 11.5). Useful for -CI/CD pipelines. +paths, chunk count, format counts, notes, `notes_structured` — +machine-readable `{key: value}` from Phase 11.5 — and `secrets_redaction_counts` +from Phase 12). Useful for CI/CD pipelines. ### Token-aware chunking — `--chunk-tokens` (Phase 11.5) @@ -211,6 +237,43 @@ documents shorter than 3 pages. The structured-notes payload reports `pdf_header_footer_lines_stripped` so operators can see post-hoc that dedup actually did work. +### Markdown-aware splitter — `--strategy markdown` (Phase 12) + +When the input has real markdown structure (technical wikis, README +collections, knowledge-base exports), heading-aware chunking beats +paragraph-greedy chunking: each chunk is a coherent section starting +with a heading, and the parent heading path is **inlined as a +breadcrumb** at the top of each chunk so SFT loss sees the document +context. + +```bash +forgelm ingest ./engineering_wiki/ --recursive --output data/wiki.jsonl \ + --strategy markdown --chunk-size 4000 +``` + +Behaviour notes: + +- Boundaries are markdown headings (`# H1` … `###### H6`); the chunker + never breaks mid-section. +- Code-fenced blocks (` ``` `) are kept atomic — never split mid-block. + This means a single section containing a long code listing may exceed + the soft cap (mirrors the paragraph strategy's "long-paragraph on + its own" rule). +- Heading-shaped lines **inside** a fenced block (`# whoami`, + `# noqa: E402`) are not interpreted as section boundaries. +- Composes with token-aware mode: `--strategy markdown --chunk-tokens + 1024 --tokenizer "Qwen/Qwen2.5-7B-Instruct"`. + +### DOCX table preservation (Phase 12) + +DOCX tables now extract as **markdown table syntax** (header row + +`---` separator + body rows) instead of the previous `" | "`-joined +flat line. Combined with `--strategy markdown` this keeps tables +intact across chunks. Uneven rows are right-padded with empty cells; +all-blank rows are dropped; the first non-empty row becomes the +header. SFT use cases where this matters: tabular Q&A, financial +assistants, code-with-data prompts. + --- ## Troubleshooting diff --git a/docs/qms/sop_data_management.md b/docs/qms/sop_data_management.md index 8cc00ee0..f9bc1703 100644 --- a/docs/qms/sop_data_management.md +++ b/docs/qms/sop_data_management.md @@ -71,22 +71,34 @@ ForgeLM automated checks: - Dataset fingerprinting (SHA-256 hash, size, timestamp) - Format validation per trainer type (SFT, DPO, KTO, GRPO) - Text cleaning (`clean_text: true`) -- **Audit pipeline (`forgelm --data-audit `, v0.5.0+)** — produces - `data_audit_report.json` with per-split sample counts, length distribution, - top-3 language detection, simhash near-duplicate rate, cross-split - leakage check, and PII flag counts (email / phone / Luhn-validated credit - card / IBAN / TR–DE–FR–US national IDs). The report is auto-inlined into - the EU AI Act Article 10 governance artifact when present in the trainer's - `output_dir` — operators must run the audit **before** training to keep - the bundle self-contained. -- **Ingestion pipeline (`forgelm ingest`, v0.5.0+)** — turns raw PDF / DOCX / - EPUB / TXT / Markdown into SFT-ready JSONL with optional `--pii-mask` to - redact detected PII spans before chunks reach storage. +- **Audit pipeline (`forgelm audit `, v0.5.1+; legacy + `forgelm --data-audit` still works)** — produces + `data_audit_report.json` with per-split sample counts, length + distribution, top-3 language detection, simhash near-duplicate rate + (or **MinHash LSH** in v0.5.2 via `--dedup-method minhash` for + >50K-row corpora), cross-split leakage check, PII flag counts + (email / phone / Luhn-validated credit card / IBAN / + TR–DE–FR–US national IDs) with **severity tiers** (v0.5.1) and + always-on **secrets/credentials scan** (v0.5.2 — AWS / GitHub / + Slack / OpenAI / Google / JWT / private-key headers). Optional + **heuristic quality filter** (`--quality-filter`, v0.5.2) adds a + `quality_summary` block. The report is auto-inlined into the EU AI + Act Article 10 governance artifact when present in the trainer's + `output_dir` — operators must run the audit **before** training to + keep the bundle self-contained. +- **Ingestion pipeline (`forgelm ingest`, v0.5.0+; v0.5.2 added + markdown-aware splitter + DOCX→markdown table preservation + + `--secrets-mask`)** — turns raw PDF / DOCX / EPUB / TXT / Markdown + into SFT-ready JSONL with optional `--pii-mask` and (v0.5.2) + `--secrets-mask` to redact detected PII / credential spans before + chunks reach storage. Manual checks: -- [ ] Run `forgelm --data-audit` and review `data_audit_report.json` for: +- [ ] Run `forgelm audit` and review `data_audit_report.json` for: cross-split leakage > 0%, near-duplicate rate, unexpected language mix, - PII flag counts. + PII flag counts, **`secrets_summary` (v0.5.2 — any non-zero count is + a stop-the-line event; remediate before training)**, and + `quality_summary` if `--quality-filter` was used. - [ ] Sample review: inspect 50+ random examples. - [ ] Verify label correctness (for preference/KTO data). diff --git a/docs/reference/usage-tr.md b/docs/reference/usage-tr.md index ff549a97..e517d8d5 100644 --- a/docs/reference/usage-tr.md +++ b/docs/reference/usage-tr.md @@ -69,7 +69,7 @@ forgelm --config my_config.yaml --generate-data Bu komut, eğitim başlamadan önce bir öğretmen modelden eğitim verisi üretmek için `synthetic` config bölümünü kullanır. Tüm sentetik veri seçenekleri için [Konfigürasyon Rehberi](configuration-tr.md)'ne bakın. -### Doküman Yutma (v0.5.0+; v0.5.1'de token-aware mod) +### Doküman Yutma (v0.5.0+; v0.5.1 token-aware; v0.5.2 markdown + secrets-mask) Ham PDF / DOCX / EPUB / TXT / Markdown'ı SFT'ye uygun JSONL'a dönüştürür. Opsiyonel bağımlılık: `pip install forgelm[ingestion]`. Ayrıntılar için [Doküman Yutma Rehberi](../guides/ingestion-tr.md). @@ -92,7 +92,7 @@ forgelm ingest ./policies/ --recursive --output data/policies.jsonl \ --chunk-tokens 1024 --tokenizer "Qwen/Qwen2.5-7B-Instruct" ``` -### Veri Seti Denetimi (v0.5.0+; v0.5.1'de subcommand'a yükseltildi) +### Veri Seti Denetimi (v0.5.0+; v0.5.1 subcommand; v0.5.2 MinHash + quality + secrets) CPU-only kalite + governance denetimi. `data_audit_report.json` üretir. Ayrıntılar için [Denetim Rehberi](../guides/data_audit-tr.md). diff --git a/docs/reference/usage.md b/docs/reference/usage.md index 01bc6cd7..f9339de4 100644 --- a/docs/reference/usage.md +++ b/docs/reference/usage.md @@ -129,7 +129,7 @@ forgelm --config my_config.yaml --generate-data This uses the `synthetic` config section to generate training data from a teacher model before training begins. See the [Configuration Guide](configuration.md) for all synthetic data options. -### Document Ingestion (v0.5.0+; token-aware in v0.5.1) +### Document Ingestion (v0.5.0+; token-aware in v0.5.1; markdown + secrets-mask in v0.5.2) Convert raw PDF / DOCX / EPUB / TXT / Markdown into SFT-ready JSONL. Optional dep: `pip install forgelm[ingestion]`. See [Ingestion Guide](../guides/ingestion.md). @@ -150,9 +150,19 @@ forgelm ingest ./customer_emails/ --pii-mask --output data/anon.jsonl # Token-aware chunking (v0.5.1) — sizes chunks against your model's vocab forgelm ingest ./policies/ --recursive --output data/policies.jsonl \ --chunk-tokens 1024 --tokenizer "Qwen/Qwen2.5-7B-Instruct" + +# v0.5.2: markdown-aware splitter for technical wikis / READMEs +forgelm ingest ./engineering_wiki/ --recursive --strategy markdown \ + --output data/wiki.jsonl + +# v0.5.2: scrub credentials before chunks land in the JSONL +forgelm ingest ./mixed_corpus/ --secrets-mask --output data/clean.jsonl + +# v0.5.2: combine secrets + PII masking (secrets first to avoid double-counting) +forgelm ingest ./mixed_corpus/ --secrets-mask --pii-mask --output data/scrubbed.jsonl ``` -### Dataset Audit (v0.5.0+; subcommand promoted in v0.5.1) +### Dataset Audit (v0.5.0+; subcommand promoted in v0.5.1; MinHash + quality + secrets in v0.5.2) CPU-only quality + governance audit. Produces `data_audit_report.json`. See [Audit Guide](../guides/data_audit.md). @@ -166,9 +176,15 @@ forgelm audit data/ --output ./audit/ # Show every split (no zero-finding fold) forgelm audit data/ --verbose -# Custom Hamming threshold for near-duplicate detection +# Custom Hamming threshold for simhash near-duplicate detection forgelm audit data/ --near-dup-threshold 5 +# v0.5.2: MinHash LSH dedup for >50K-row corpora (needs `[ingestion-scale]` extra) +forgelm audit data/large_corpus.jsonl --dedup-method minhash --jaccard-threshold 0.85 + +# v0.5.2: opt-in heuristic quality filter (Gopher/C4 style) +forgelm audit data/ --quality-filter + # Machine-readable summary on stdout forgelm audit data/sft.jsonl --output ./audit/ --output-format json @@ -176,7 +192,7 @@ forgelm audit data/sft.jsonl --output ./audit/ --output-format json forgelm --data-audit data/sft.jsonl --output ./audit/ ``` -The audit captures: per-split sample count + length distribution, top-3 language detection, **LSH-banded** simhash near-duplicate rate (Phase 11.5; brute-force fallback at edge thresholds), cross-split leakage (silent train-test overlap), PII flag counts with **severity tiers** (`pii_severity` block grading each PII type as critical / high / medium / low and surfacing a `worst_tier` verdict). +The audit captures: per-split sample count + length distribution, top-3 language detection, **LSH-banded** simhash near-duplicate rate (Phase 11.5; brute-force fallback at edge thresholds; v0.5.2 added optional **MinHash LSH** path), cross-split leakage (silent train-test overlap), PII flag counts with **severity tiers**, **always-on credentials/secrets scan** (`secrets_summary` — AWS / GitHub / Slack / OpenAI / Google / JWT / private-key headers), and an opt-in **heuristic quality filter** (Gopher/C4 style) that adds a `quality_summary` block. When `data_audit_report.json` is present in the trainer's `output_dir` at training time, its findings are inlined under the `data_audit` key of the EU AI Act Article 10 governance artifact automatically. diff --git a/docs/roadmap-tr.md b/docs/roadmap-tr.md index ce4ca6c3..323027c1 100644 --- a/docs/roadmap-tr.md +++ b/docs/roadmap-tr.md @@ -11,7 +11,7 @@ | ✅ Tamam | [Faz 10.5 — Quickstart Katmanı ve Onboarding](roadmap/phase-10-5-quickstart.md) | `forgelm quickstart