From e5cf9b57ef555c89e51e41c662793c2a8da7ea1f Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 27 Apr 2026 04:44:54 +0300 Subject: [PATCH 1/7] feat(phase-11.5): Ingestion / audit polish (v0.5.1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operational follow-up to v0.5.0's ingestion + audit surface — the 12 items carved out of Phase 11's review backlog, all landed in one focused pass. No new training capabilities; behaviour at default flags is identical, the gains are large-corpus throughput, ergonomics, and a cleaner CLI shape. Highlights: * `forgelm audit PATH` is now a first-class subcommand with `--verbose`, `--near-dup-threshold`, and its own `--output` default. The legacy `forgelm --data-audit FLAG` is preserved as a deprecation alias and logs a one-line notice; existing pipelines need zero changes. * Near-duplicate detection swapped from O(n²) brute force to LSH banding (`bands = threshold + 1` by pigeonhole). Recall stays exact at the default threshold; brute-force fallback retained when bands shrink below 4 bits. Both within-split and cross-split helpers benefit. * `_read_jsonl_split` is now a generator and `_audit_split` consumes it row-by-row via a `_StreamingAggregator` — RAM use no longer scales with split size. * `--chunk-tokens N --tokenizer MODEL` on `forgelm ingest` sizes chunks against an HF AutoTokenizer instead of raw character counts. `trust_remote_code=False` hard-pinned for safety. * PDF page header/footer dedup (`_strip_repeating_page_lines`) strips lines that recur on >=70% of pages — kills audit near-duplicate noise on long policy / book PDFs. * `pii_severity` block alongside `pii_summary`: `total`, `by_tier`, `by_type`, `worst_tier`. Tier table follows PCI-DSS / GDPR Art. 9 weighting. Notes line leads with the worst tier. * `summarize_report(verbose=False)` (default) folds zero-finding splits into a single tail line. `forgelm audit --verbose` reverses it. * `IngestionResult.notes_structured` carries machine-readable `{key: value}` for CI/CD consumers; `extra_notes` keeps the prose. JSON output exposes both. * Wizard's BYOD + full-flow dataset prompts now offer to run `forgelm ingest` inline when the typed path is a directory of raw documents — closes the onboarding loop end-to-end. * Optional `xxhash.xxh3_64` digest + module-scope token-level `lru_cache(maxsize=10_000)` accelerate the simhash hot path; BLAKE2b remains as the bare-install fallback. * `data_audit_report.json` is written via tempfile + os.replace so a crashed audit can never leave a half-written report on disk. newline="\n" pinned for byte-exact reproducibility. Module deltas: * `forgelm/data_audit.py`: PII_SEVERITY table, `_token_digest` (cached + xxhash optional), LSH-banded `find_near_duplicates`, `_count_leaked_rows` rebuilt on the same banded index, `_read_jsonl_split` promoted to a generator, `_audit_split(split_name, path, ...)` rewritten as a streaming aggregator, `_build_pii_severity`, `_atomic_write_json`, `summarize_report(verbose=...)`, `_split_has_findings`, `_render_split_block`, `_render_pii_severity`. * `forgelm/ingestion.py`: `_strip_repeating_page_lines`, `_extract_pdf` threads dedup state, `_chunk_sliding_tokens`, `_chunk_paragraph_tokens`, `_strategy_dispatch_tokens`, `_load_tokenizer`, `IngestionResult.notes_structured` + `pdf_header_footer_lines_stripped`, `_process_one_file` extended for token-aware dispatch. * `forgelm/cli.py`: `_add_audit_subcommand`, `_run_audit_cmd`, dispatcher hooks "audit", `--data-audit` reflows through `_run_data_audit` with `invoked_via_legacy_flag=True`. * `forgelm/wizard.py`: `_offer_ingest_for_directory` + `_prompt_dataset_path_with_ingest_offer`; BYOD path and Step 5 dataset prompt both use them. * `pyproject.toml`: version bumped to 0.5.1rc1; `xxhash` added to the `[ingestion]` extra (optional — fallback path keeps bare installs OK). Tests: * `tests/test_data_audit.py`: TestLshBandedNearDuplicates, TestPiiSeverity, TestSummarizeVerbosePolicy, TestAtomicWrite, TestStreamingReader, TestTokenCachePerformance. * `tests/test_ingestion.py`: TestPdfHeaderFooterDedup, TestStructuredIngestionNotes, TestTokenAwareCli; TestSummary updated for the new "Next: forgelm audit ..." hint. * `tests/test_cli_subcommands.py`: TestAuditSubcommand (subcommand happy path, JSON envelope, legacy `--data-audit` alias parity). * `tests/test_wizard_byod.py`: refreshed for the ingest-first wording (empty-directory rejection + decline-the-offer path). Result: 754 passed, 8 skipped (extras-skip). Lint clean. Docs: * docs/roadmap.md + roadmap-tr.md: Phase 11.5 marked done; v0.5.1 set as the merged-pending-publish milestone; v0.5.2 reserved for Phase 14. * docs/roadmap/phase-11-5-backlog.md: rewritten as a "what landed" record with module pointers per item. * docs/roadmap/releases.md: v0.5.1 ticked across 11 features; v0.5.2 reslotted as Pipeline Chains. * docs/guides/{ingestion,data_audit}{,-tr}.md: token-aware mode, PDF header/footer dedup, audit subcommand, PII severity, verbose policy documented; legacy --data-audit retained as deprecation alias. * docs/reference/usage{,-tr}.md: CLI examples updated for the new subcommand + token-aware ingestion. * README.md: feature blurb refreshed; quick-start hint promoted to `forgelm audit`. * CHANGELOG.md: detailed Unreleased entry covering all 12 items + tests + behavioural deltas. --- CHANGELOG.md | 32 +- README.md | 11 +- docs/guides/data_audit-tr.md | 42 +- docs/guides/data_audit.md | 93 ++- docs/guides/ingestion-tr.md | 56 +- docs/guides/ingestion.md | 44 +- docs/reference/usage-tr.md | 27 +- docs/reference/usage.md | 27 +- docs/roadmap-tr.md | 30 +- docs/roadmap.md | 30 +- docs/roadmap/phase-11-5-backlog.md | 107 +-- docs/roadmap/phase-11-data-ingestion.md | 6 +- docs/roadmap/releases.md | 26 +- forgelm/cli.py | 151 ++++- forgelm/data_audit.py | 836 ++++++++++++++++++------ forgelm/ingestion.py | 269 +++++++- forgelm/wizard.py | 145 +++- pyproject.toml | 6 +- tests/test_cli_subcommands.py | 66 ++ tests/test_data_audit.py | 130 ++++ tests/test_ingestion.py | 81 ++- tests/test_wizard_byod.py | 25 +- 22 files changed, 1879 insertions(+), 361 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index abed448d..27b350f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,37 @@ All notable changes to ForgeLM are documented here. ## [Unreleased] -### Added +### Added — Phase 11.5 (Ingestion / Audit Polish, targeting v0.5.1) + +Operational polish on top of `v0.5.0`'s ingestion + audit surface — no new training capabilities, but materially better handling for large corpora and a cleaner CLI shape. All 12 follow-ups carved out of Phase 11's review backlog. + +- **`forgelm audit PATH` subcommand** — Promotes the `--data-audit` top-level flag to a first-class subcommand with `--verbose`, `--near-dup-threshold`, and its own `--output` default (`./audit/`). The legacy `forgelm --data-audit PATH` flag keeps working as a deprecation alias and logs a one-line notice; existing CI pipelines need no changes. Removal targeted no earlier than `v0.7.0`. +- **LSH-banded near-duplicate detection** (`find_near_duplicates`, `_count_leaked_rows`) — Pigeonhole-banded LSH (default `bands = threshold + 1`) drops within-split + cross-split scans from `O(n²)` to ~`O(n × k)`. Recall stays exact at the default Hamming threshold; brute-force fallback remains for edge thresholds where bands shrink below 4 bits. Unblocks audits on 100K+ row corpora. +- **Streaming `_read_jsonl_split`** — The audit's JSONL reader is now a generator yielding `(row, parse_err, decode_err)`; `_audit_split` consumes it row-by-row via a `_StreamingAggregator` so RAM stays bounded on multi-million-row splits. Per-line tolerance semantics (parse errors, decode errors, non-dict rows) preserved. +- **Token-aware ingestion** (`--chunk-tokens`, `--tokenizer`, `--overlap-tokens`) — Optional flags on `forgelm ingest` size chunks against an HF `AutoTokenizer.encode` instead of raw character counts, so chunks line up with `model.max_length` exactly. `--tokenizer` is required with `--chunk-tokens` (we refuse to default to a hidden vocab because the chunk count would silently differ per-model). `trust_remote_code=False` is hard-pinned for safety. +- **PDF page-level header / footer dedup** (`_strip_repeating_page_lines`) — Lines that recur as the first or last non-empty line on ≥ 70 % of a PDF's pages (company watermarks, page numbers, copyright lines) are stripped automatically before chunking. Reduces audit `near_duplicate_pairs` noise on long policy / book PDFs. Skipped on documents shorter than 3 pages. +- **PII severity tiers** — Audit JSON now carries a `pii_severity` block (`total`, `by_tier`, `by_type`, `worst_tier`) alongside the flat `pii_summary`. Tiers map regulatory weighting: `credit_card` / `iban` → critical (PCI-DSS), national IDs → high (GDPR Art. 9), `email` → medium, `phone` → low. The aggregate notes line leads with the worst tier (`WORST tier: CRITICAL`) so reviewers cannot miss it. +- **`summarize_report` truncation policy** — Default `verbose=False` folds zero-finding splits into a single tail line so multi-split summaries stay short; `--verbose` on the new `audit` subcommand reverses this for full output. Has no effect on the on-disk JSON report. +- **Structured ingestion notes** — `IngestionResult.extra_notes` keeps the human-readable list; new `notes_structured: {key: value}` (and an explicit `pdf_header_footer_lines_stripped` field) carries machine-readable counts for CI/CD consumers. JSON output exposes both. +- **Wizard "ingest first" entry point** — `_offer_ingest_for_directory` + `_prompt_dataset_path_with_ingest_offer`: BYOD quickstart and the full 8-step wizard now offer to run `forgelm ingest` inline when the typed dataset path is a directory of raw documents, then feed the produced JSONL straight back into the BYOD path. Closes the onboarding loop end-to-end. +- **xxhash backend for simhash + token-level memo** — Optional `xxhash.xxh3_64` digest path (added to `forgelm[ingestion]`) is several times faster than BLAKE2b on short keys; BLAKE2b stays as the fallback. A module-scope `lru_cache(maxsize=10_000)` memoises the per-token digest — Zipfian token frequency means the cache covers most corpus traffic. +- **Atomic audit-report write** — `data_audit_report.json` is now written via `tempfile.NamedTemporaryFile` + `os.replace` so a crashed audit can never leave a half-written report on disk. `newline="\n"` pinned for byte-exact reproducibility across Windows / Linux / macOS. + +### Tests + +- `tests/test_data_audit.py` — `TestLshBandedNearDuplicates` (LSH parity vs. brute force + high-threshold fallback), `TestPiiSeverity` (critical-tier verdict + neutral case), `TestSummarizeVerbosePolicy` (clean splits folded vs. expanded), `TestAtomicWrite` (no `.tmp` leftovers), `TestStreamingReader` (per-line tuple yields), `TestTokenCachePerformance` (cross-text cache hits). +- `tests/test_ingestion.py` — `TestPdfHeaderFooterDedup` (multi-page header/footer collapse, short-doc skip, no-repeats pass-through), `TestStructuredIngestionNotes`, `TestTokenAwareCli` (validates the `--chunk-tokens` requires `--tokenizer` rule). +- `tests/test_cli_subcommands.py` — `TestAuditSubcommand` (subcommand happy path, JSON envelope, legacy `--data-audit` alias). +- `tests/test_wizard_byod.py` — refreshed for the new ingest-first wording (empty directory rejection, decline-the-ingest-offer path). + +### Changed — Phase 11 (no behavioural delta unless noted) + +- `AuditReport` gains a `pii_severity: Dict[str, Any]` field. JSON consumers reading only `pii_summary` continue to work; the new field is additive. +- `find_near_duplicates(fingerprints, *, threshold, bits=64)` accepts a `bits` keyword for adaptive banding (default 64 matches `compute_simhash`). +- `_read_jsonl_split` is now a generator. The legacy buffered tuple return is gone — callers that were materialising rows can wrap with `list(...)`. +- `_audit_split(split_name, path, ...)` now takes a path instead of an in-memory list; `_process_split` calls it directly. Returns `(info, fingerprints, pii_split, parse_errors, decode_errors)` so OSError handling stays in the orchestrator. + +### Previously added — Phase 11 **Document Ingestion & Data Audit (Phase 11)** — bridges raw enterprise corpora (legal, medical, policy manuals) to ForgeLM's training data format and surfaces governance signals before training starts. diff --git a/README.md b/README.md index 31ec0828..0e7f0806 100644 --- a/README.md +++ b/README.md @@ -27,11 +27,12 @@ - **LLM-as-Judge**: API-based (OpenAI) or local model scoring for quality assessment - **Auto-Revert**: Automatically discard models that fail loss, benchmark, or safety thresholds -### Document Ingestion & Data Audit (v0.5.0) +### Document Ingestion & Data Audit (v0.5.0 — polished in v0.5.1) - **Multi-Format Ingestion**: `forgelm ingest ./policies/ --recursive --output data/policies.jsonl` — turns raw PDF / DOCX / EPUB / TXT / Markdown into the SFT-ready JSONL the trainer accepts. Optional dep: `pip install forgelm[ingestion]`. -- **Chunking Strategies**: `paragraph` (default; preserves boundaries) or `sliding` (fixed window with overlap). +- **Chunking Strategies**: `paragraph` (default; preserves boundaries) or `sliding` (fixed window with overlap). Token-aware mode in v0.5.1: `--chunk-tokens 1024 --tokenizer Qwen/Qwen2.5-7B-Instruct` sizes chunks against your model's actual vocabulary. - **PII Masking on Ingest**: `--pii-mask` redacts emails, phones, credit cards (Luhn-validated), IBAN, and national IDs (TR / DE / FR / US-SSN) before chunks land in the JSONL. -- **Dataset Audit**: `forgelm --data-audit data/sft.jsonl --output ./audit/` — produces `data_audit_report.json` with sample count, length distribution, top-3 language detection, simhash near-duplicate rate, cross-split leakage check, null/empty rate, and PII flag counts. CPU-only; feeds EU AI Act Article 10 governance artifact automatically when present at training time. +- **PDF Page Header/Footer Dedup (v0.5.1)**: Lines that recur on ≥ 70 % of PDF pages (watermarks, page numbers, copyright lines) are stripped automatically — the audit's near-duplicate counts stop misfiring on long policy / book PDFs. +- **Dataset Audit**: `forgelm audit data/sft.jsonl --output ./audit/` — produces `data_audit_report.json` with sample count, length distribution, top-3 language detection, LSH-banded near-duplicate rate (`O(n × k)` typical case, exact recall at the default Hamming threshold), cross-split leakage check, null/empty rate, and PII flag counts with **severity tiers** (critical / high / medium / low + worst-tier verdict). CPU-only; streaming JSONL reader keeps memory bounded on multi-million-row splits; feeds EU AI Act Article 10 governance artifact automatically when present at training time. Legacy `--data-audit` flag still works as a deprecation alias. ### Quickstart Layer (v0.4.5) - **One-Command Templates**: `forgelm quickstart customer-support` — bundled templates for SFT, code, BYOD domain expert, Turkish medical Q&A, and GRPO math reasoning. Auto-downsizes models on small GPUs. @@ -70,10 +71,10 @@ forgelm quickstart customer-support # render config + train + chat forgelm quickstart code-assistant --dry-run # render config only forgelm quickstart medical-qa-tr --model your-org/your-model # override -# Have raw docs? Ingest them first (v0.5.0+) +# Have raw docs? Ingest them first (v0.5.0+; v0.5.1 added token-aware sizing) pip install -e ".[ingestion]" forgelm ingest ./policies/ --recursive --output data/policies.jsonl -forgelm --data-audit data/policies.jsonl --output ./audit/ +forgelm audit data/policies.jsonl --output ./audit/ # `forgelm --data-audit ...` still works as legacy alias # Or generate config interactively forgelm --wizard diff --git a/docs/guides/data_audit-tr.md b/docs/guides/data_audit-tr.md index 55dfd8a3..32019f3c 100644 --- a/docs/guides/data_audit-tr.md +++ b/docs/guides/data_audit-tr.md @@ -1,10 +1,15 @@ # Veri Seti Denetimi Rehberi -`forgelm --data-audit` bir JSONL veri setini analiz eder ve kalite, +`forgelm audit PATH` bir JSONL veri setini analiz eder ve kalite, governance ile PII sinyallerini kapsayan bir `data_audit_report.json` -üretir. Faz 11; `v0.5.0`'da tanıtıldı. Trainer'ın `output_dir`'ünde -mevcutsa, rapor EU AI Act Madde 10 veri governance artifact'ına -otomatik olarak beslenir. +üretir. Faz 11 (`v0.5.0`'da tanıtıldı) altyapıyı, Faz 11.5 (`v0.5.1`) +ise top-level flag'i first-class subcommand'a yükseltti, LSH-banded +near-duplicate tespitini, streaming JSONL okuyucusunu, PII şiddet +katmanlarını, atomik disk yazımı ve verbose-by-default kısaltma +politikasını ekledi. + +Trainer'ın `output_dir`'ünde mevcutsa, rapor EU AI Act Madde 10 veri +governance artifact'ına otomatik olarak beslenir. --- @@ -12,12 +17,22 @@ otomatik olarak beslenir. ```bash # Tek split (`train` olarak değerlendirilir) -forgelm --data-audit data/sft.jsonl --output ./audit/ +forgelm audit data/sft.jsonl --output ./audit/ # Çoklu split: train.jsonl / validation.jsonl / test.jsonl içeren dizin -forgelm --data-audit data/ --output ./audit/ +forgelm audit data/ --output ./audit/ + +# Bulgu olmayan split'leri de göster +forgelm audit data/ --verbose + +# Daha geniş / dar near-duplicate eşiği +forgelm audit data/ --near-dup-threshold 5 ``` +> **Eski alias:** `forgelm --data-audit PATH` deprecation alias'ı +> olarak korunuyor; çalışmaya devam ediyor ama bir uyarı log'lanıyor. +> Yeni script'lerde `audit` subcommand'ını kullanın. + `--output` varsayılan olarak `./audit/`'tir. Dizin yoksa oluşturulur; **tam** `data_audit_report.json` her zaman oraya yazılır. Stdout varsayılan olarak insan-okunabilir özet gösterir; `--output-format json` @@ -157,7 +172,7 @@ işaret eden bir pointer yerine tek başına okunabilir bir doküman olur. ```bash # Önce denetim — uzun bir eğitim koşusuna girmeden sorunları yüzeye çıkar -forgelm --data-audit data/policies.jsonl --output ./checkpoints/policy-run/ +forgelm audit data/policies.jsonl --output ./checkpoints/policy-run/ # Eğit (governance artifact denetimi inline edecek) forgelm --config configs/policy-run.yaml @@ -168,14 +183,23 @@ forgelm --config configs/policy-run.yaml ## CLI referansı ```text -forgelm --data-audit PATH \ +forgelm audit PATH \ [--output DIR] \ + [--verbose] \ + [--near-dup-threshold N] \ [--output-format {text,json}] \ [--quiet | --log-level {DEBUG,INFO,WARNING,ERROR}] ``` `PATH` bir `.jsonl` dosyası veya bir dizin olabilir. `--output` -varsayılan olarak `./audit/`'tir. +varsayılan olarak `./audit/`'tir. `--verbose`, insan-okunabilir özette +bulgu olmayan split'leri de gösterir (varsayılan, çoklu-split +denetimlerini kısa tutmak için tüm temiz split'leri tek bir kuyruk +satırına katlar — diskteki JSON raporu etkilenmez). `--near-dup-threshold` +varsayılan Hamming eşiğini (3, ≈%95 benzerlik) ezer. + +Eski `forgelm --data-audit PATH` flag'i deprecation alias olarak +korunuyor; davranış aynı, sadece ek bir uyarı log'lanıyor. Üst düzey flag'tir (subcommand değil) — trainer'a dokunmadan çıkar. diff --git a/docs/guides/data_audit.md b/docs/guides/data_audit.md index e6a1a5df..3592cbe2 100644 --- a/docs/guides/data_audit.md +++ b/docs/guides/data_audit.md @@ -1,10 +1,15 @@ # Dataset Audit Guide -`forgelm --data-audit` analyzes a JSONL dataset and produces a +`forgelm audit PATH` analyzes a JSONL dataset and produces a `data_audit_report.json` covering quality, governance, and PII signals. -Phase 11; introduced in `v0.5.0`. The report feeds the EU AI Act Article 10 -data governance artifact automatically when present in the trainer's -`output_dir`. +Phase 11 (introduced in `v0.5.0`) shipped the underlying audit; +Phase 11.5 (`v0.5.1`) promoted it from a top-level flag to a first-class +subcommand and added LSH-banded near-duplicate detection, streaming +JSONL reading, PII severity tiers, an atomic on-disk write, and a +verbose-by-default truncation policy on the human summary. + +The report feeds the EU AI Act Article 10 data governance artifact +automatically when present in the trainer's `output_dir`. --- @@ -12,12 +17,22 @@ data governance artifact automatically when present in the trainer's ```bash # Single split (treated as 'train') -forgelm --data-audit data/sft.jsonl --output ./audit/ +forgelm audit data/sft.jsonl --output ./audit/ # Multi-split: directory containing train.jsonl / validation.jsonl / test.jsonl -forgelm --data-audit data/ --output ./audit/ +forgelm audit data/ --output ./audit/ + +# Show every split (including those with no findings) +forgelm audit data/ --verbose + +# Tighter / wider near-duplicate threshold +forgelm audit data/ --near-dup-threshold 5 ``` +> **Legacy alias:** `forgelm --data-audit PATH` keeps working unchanged +> as a deprecation alias and logs a one-line notice. New scripts should +> use the `audit` subcommand. Removal targeted no earlier than `v0.7.0`. + `--output` defaults to `./audit/`. The directory is created if missing; the **full** `data_audit_report.json` is always written there. Stdout shows a human-readable summary by default; pass `--output-format json` to get @@ -100,6 +115,32 @@ Other categories surface on regex shape alone — false positives are intentional. Mask with `forgelm ingest --pii-mask` (or in your own preprocessing) before publishing the dataset. +### PII severity tiers (Phase 11.5) + +The flat `pii_summary` map gives compliance reviewers no guidance on +*how bad* a finding is. Phase 11.5 adds a `pii_severity` block alongside: + +```json +{ + "pii_severity": { + "total": 25, + "by_tier": {"critical": 1, "high": 2, "medium": 18, "low": 4}, + "by_type": { + "credit_card": {"count": 1, "tier": "critical"}, + "tr_id": {"count": 2, "tier": "high"}, + "email": {"count": 18, "tier": "medium"}, + "phone": {"count": 4, "tier": "low"} + }, + "worst_tier": "critical" + } +} +``` + +The tier table is consensus regulatory weighting (PCI-DSS for financial +identifiers; GDPR Art. 9 + ENISA for government IDs). Pipelines that +gate on PII severity should read `pii_severity.worst_tier` and refuse +to publish on `critical` / `high` without explicit review. + **Pattern precedence is documented.** `_PII_PATTERNS` iteration order governs both detection priority and mask precedence — most specific patterns (`email`, `iban`, `credit_card`, national IDs) are scanned @@ -118,8 +159,24 @@ deployment, ≈95% similarity at this width). Exposes both: - **Within-split** pairs: `near_duplicate_pairs` per split. - **Cross-split** leakage: above. -Quadratic in row count; suitable for datasets up to ~50K rows. Larger -corpora need an LSH band index — out of scope for v0.5.0. +Phase 11.5 swapped the underlying scan to **LSH banding**: pigeonhole +chooses `bands = threshold + 1`, candidate pairs are exactly the rows +that collide in any band-bucket, and the Hamming check only runs on +candidates. Recall stays exact at the default threshold; cost drops from +`O(n²)` to roughly `O(n × k)` (the `_count_leaked_rows` cross-split +helper uses the same banded shape). The brute-force path remains as the +fallback when the threshold is high enough that bands shrink below +4 bits — `find_near_duplicates` returns the same result either way. + +Phase 11.5 also made the simhash backend pluggable: + +- **xxhash.xxh3_64** drives the per-token digest when the optional + `xxhash` dep is installed (now part of `forgelm[ingestion]`); it is + several times faster than BLAKE2b on short keys. +- **BLAKE2b** is the fallback so a bare install still works. +- A module-scope `lru_cache(maxsize=10_000)` memoises the digest at the + token level — Zipfian token frequency means the cache covers most of + a corpus's traffic with a small footprint. --- @@ -166,21 +223,31 @@ forgelm --config configs/policy-run.yaml ## CLI reference ```text -forgelm --data-audit PATH \ +forgelm audit PATH \ [--output DIR] \ + [--verbose] \ + [--near-dup-threshold N] \ [--output-format {text,json}] \ [--quiet | --log-level {DEBUG,INFO,WARNING,ERROR}] ``` `PATH` may be a `.jsonl` file or a directory. `--output` defaults to -`./audit/`. - -Top-level flag (not a subcommand) — exits without touching the trainer. +`./audit/`. `--verbose` shows every split in the human summary even when +it has zero findings (default folds clean splits into one tail line so +multi-split audits stay short — has no effect on the on-disk JSON +report). `--near-dup-threshold N` overrides the default Hamming-distance +cutoff of 3 (≈95 % similarity). > **Note:** This matches the behavior summarised at the top of this guide: > `--output-format json` writes a small envelope (success flag, top-level > metrics, report path) to stdout. The full `data_audit_report.json` is -> always written to `--output`. Read it from disk if you need every detail. +> always written to `--output` via `tempfile.NamedTemporaryFile` + +> `os.replace` — Phase 11.5 hardening so a crashed audit can never leave +> a half-written report on disk. + +The legacy `forgelm --data-audit PATH` flag is preserved as a +deprecation alias and logs a one-line notice. Behaviour is identical; +new scripts should use the subcommand. --- diff --git a/docs/guides/ingestion-tr.md b/docs/guides/ingestion-tr.md index d9d72eae..efd8f981 100644 --- a/docs/guides/ingestion-tr.md +++ b/docs/guides/ingestion-tr.md @@ -2,10 +2,12 @@ Ham kurumsal külliyatı (PDF / DOCX / EPUB / TXT / Markdown) ForgeLM'in eğittiği SFT-uyumlu JSONL'a dönüştürün. Faz 11; `v0.5.0`'da tanıtıldı. +Faz 11.5 (`v0.5.1`) token-aware chunking, PDF sayfa header/footer dedup +ve yapılandırılmış ingestion notları ekledi. -> Sonrasında [`forgelm --data-audit`](data_audit-tr.md) ile uzunluk -> dağılımı / dil / near-duplicate / PII metriklerini yüzeye çıkarın; -> chunk'ları Q&A `messages` formuna genişletmek için +> Sonrasında [`forgelm audit`](data_audit-tr.md) ile uzunluk dağılımı / +> dil / near-duplicate / PII metriklerini yüzeye çıkarın; chunk'ları Q&A +> `messages` formuna genişletmek için > [`forgelm --generate-data`](../reference/usage-tr.md) ile zincirleyin. --- @@ -16,10 +18,12 @@ eğittiği SFT-uyumlu JSONL'a dönüştürün. Faz 11; `v0.5.0`'da tanıtıldı. pip install 'forgelm[ingestion]' ``` -Bu opsiyonel grup `pypdf`, `python-docx`, `ebooklib`, `beautifulsoup4` ve -`langdetect`'i getirir. Düz metin + Markdown için bu paketlerin hiçbiri -gerekmediği için opsiyonel tutuldu. Modülün kendisi import edildiğinde -ilgili extractor çağrılmadığı sürece bu bağımlılıkları yüklemez. +Bu opsiyonel grup `pypdf`, `python-docx`, `ebooklib`, `beautifulsoup4`, +`langdetect` ve (Faz 11.5'ten itibaren) opsiyonel non-cryptographic +hızlı `xxhash` backend'i getirir. Düz metin + Markdown için bu +paketlerin hiçbiri gerekmediği için opsiyonel tutuldu. Modülün kendisi +import edildiğinde ilgili extractor çağrılmadığı sürece bu bağımlılıkları +yüklemez. OCR **kapsam dışındadır.** Metin katmanı olmayan taranmış PDF'ler bir uyarı yüzdürür ve sıfır chunk üretir; ingest etmeden önce Tesseract veya @@ -94,7 +98,7 @@ forgelm ingest ./customer_emails/ --output data/anon.jsonl --pii-mask Tespit edilen span'ler `[REDACTED]` ile değiştirilir. Tespit regex tabanlıdır — false positive'ler kasıtlıdır. Sonrasında -`forgelm --data-audit` ile çıktıyı doğrulayın. +`forgelm audit` ile çıktıyı doğrulayın. --- @@ -164,8 +168,9 @@ forgelm quickstart domain-expert --dataset data/policies.jsonl ```text forgelm ingest INPUT_PATH \ --output FILE \ - [--chunk-size N] \ + [--chunk-size N | --chunk-tokens N --tokenizer MODEL_NAME] \ [--overlap N] \ + [--overlap-tokens N] \ [--strategy {sliding,paragraph}] \ [--recursive] \ [--pii-mask] \ @@ -174,8 +179,37 @@ forgelm ingest INPUT_PATH \ ``` `--output-format json` standart çıktıya makine-okunabilir bir özet -yazar (dosya yolları, chunk sayısı, format sayıları, notlar). CI/CD -pipeline'larında kullanışlı. +yazar (dosya yolları, chunk sayısı, format sayıları, notlar ve Faz +11.5'te eklenen `notes_structured` makine-okunabilir `{key: value}` +yapısı). CI/CD pipeline'larında kullanışlı. + +### Token-aware chunking — `--chunk-tokens` (Faz 11.5) + +Karakter tabanlı chunking pratik ama yoğun metinde modelinizin +`max_length` token bütçesini aşabilir. `--chunk-tokens N` ile birlikte +`--tokenizer MODEL_NAME` geçirerek chunk'ları aynı tokenizer üzerinden +boyutlandırabilirsiniz: + +```bash +forgelm ingest ./policies/ --recursive --output data/policies.jsonl \ + --chunk-tokens 1024 --tokenizer "Qwen/Qwen2.5-7B-Instruct" +``` + +`--chunk-tokens` set edildiğinde `--chunk-size` görmezden gelinir (bir +uyarı log'lanır). `--overlap-tokens N` token cinsinden sliding overlap +boyutudur ve `--overlap` ile aynı yarım-pencere üst sınırına tabidir. +`--chunk-tokens` ile `--tokenizer` zorunludur — varsayılan vocab +seçmiyoruz çünkü chunk sayısı modelden modele sessizce değişirdi. + +### PDF sayfa header/footer dedup (Faz 11.5) + +`forgelm ingest`, bir PDF'in sayfalarının ≥ %70'inde ilk veya son +boş-olmayan satır olarak tekrarlayan satırları (şirket filigranı, +copyright satırı, sayfa numarası vb.) chunk'lamadan önce otomatik +olarak temizler. Açık/kapatma flag'i yok; bayrak gerektirmiyor. +Audit'in `near_duplicate_pairs` metriğindeki gürültüyü azaltır. 3 +sayfadan kısa belgelerde devre dışı kalır. Yapılandırılmış notlar +çıktısı `pdf_header_footer_lines_stripped` alanını taşır. --- diff --git a/docs/guides/ingestion.md b/docs/guides/ingestion.md index bfec098b..d36948a3 100644 --- a/docs/guides/ingestion.md +++ b/docs/guides/ingestion.md @@ -1,9 +1,12 @@ # Document Ingestion Guide Convert raw enterprise corpora (PDF / DOCX / EPUB / TXT / Markdown) into the -SFT-ready JSONL ForgeLM trains on. Phase 11; introduced in `v0.5.0`. +SFT-ready JSONL ForgeLM trains on. Phase 11; introduced in `v0.5.0`. Phase +11.5 (`v0.5.1`) added token-aware chunking, PDF page header/footer dedup, +and structured ingestion notes — see [Phase 11.5 deltas](#phase-115-deltas) +for the short list. -> Pair with [`forgelm --data-audit`](data_audit.md) afterwards to surface +> Pair with [`forgelm audit`](data_audit.md) afterwards to surface > length-distribution / language / near-duplicate / PII metrics, and with > [`forgelm --generate-data`](../reference/usage.md) if you want the chunks > expanded into Q&A `messages` form via a teacher model. @@ -161,8 +164,9 @@ forgelm quickstart domain-expert --dataset data/policies.jsonl ```text forgelm ingest INPUT_PATH \ --output FILE \ - [--chunk-size N] \ + [--chunk-size N | --chunk-tokens N --tokenizer MODEL_NAME] \ [--overlap N] \ + [--overlap-tokens N] \ [--strategy {sliding,paragraph}] \ [--recursive] \ [--pii-mask] \ @@ -171,7 +175,39 @@ forgelm ingest INPUT_PATH \ ``` `--output-format json` emits a machine-readable summary to stdout (file -paths, chunk count, format counts, notes). Useful for CI/CD pipelines. +paths, chunk count, format counts, notes, and `notes_structured` — +machine-readable `{key: value}` introduced in Phase 11.5). Useful for +CI/CD pipelines. + +### Token-aware chunking — `--chunk-tokens` (Phase 11.5) + +Char-based chunking is convenient but can blow past your model's +`max_length` token budget on dense text. Pass `--chunk-tokens N` along +with `--tokenizer MODEL_NAME` to size chunks against the same tokenizer +your trainer will use: + +```bash +forgelm ingest ./policies/ --recursive --output data/policies.jsonl \ + --chunk-tokens 1024 --tokenizer "Qwen/Qwen2.5-7B-Instruct" +``` + +`--chunk-size` is ignored when `--chunk-tokens` is set (a warning logs +the override). `--overlap-tokens N` is the sliding-window equivalent of +`--overlap` measured in tokens, with the same half-window cap. +`--tokenizer` is **required** with `--chunk-tokens` — we refuse to +pick a default vocab because the resulting chunk count would silently +differ per-model. + +### PDF page-level header/footer dedup (Phase 11.5) + +`forgelm ingest` now strips lines that recur as the first or last +non-empty line on ≥ 70 % of a PDF's pages — typical company watermarks, +copyright lines, page numbers — before chunking. This is automatic; no +flag, no opt-out beyond "send a non-PDF". Reduces audit +`near_duplicate_pairs` noise on long policy / book PDFs. Skipped on +documents shorter than 3 pages. The structured-notes payload reports +`pdf_header_footer_lines_stripped` so operators can see post-hoc that +dedup actually did work. --- diff --git a/docs/reference/usage-tr.md b/docs/reference/usage-tr.md index d9dc67b9..ff549a97 100644 --- a/docs/reference/usage-tr.md +++ b/docs/reference/usage-tr.md @@ -69,7 +69,7 @@ forgelm --config my_config.yaml --generate-data Bu komut, eğitim başlamadan önce bir öğretmen modelden eğitim verisi üretmek için `synthetic` config bölümünü kullanır. Tüm sentetik veri seçenekleri için [Konfigürasyon Rehberi](configuration-tr.md)'ne bakın. -### Doküman Yutma (v0.5.0+) +### Doküman Yutma (v0.5.0+; v0.5.1'de token-aware mod) Ham PDF / DOCX / EPUB / TXT / Markdown'ı SFT'ye uygun JSONL'a dönüştürür. Opsiyonel bağımlılık: `pip install forgelm[ingestion]`. Ayrıntılar için [Doküman Yutma Rehberi](../guides/ingestion-tr.md). @@ -86,24 +86,37 @@ forgelm ingest ./scan.pdf --strategy sliding --chunk-size 1024 --overlap 128 \ # Yazmadan önce PII'yi maskele forgelm ingest ./customer_emails/ --pii-mask --output data/anon.jsonl + +# Token-aware chunking (v0.5.1) — chunk'ları modelinizin vocab'ına göre boyutlandırır +forgelm ingest ./policies/ --recursive --output data/policies.jsonl \ + --chunk-tokens 1024 --tokenizer "Qwen/Qwen2.5-7B-Instruct" ``` -### Veri Seti Denetimi (v0.5.0+) +### Veri Seti Denetimi (v0.5.0+; v0.5.1'de subcommand'a yükseltildi) CPU-only kalite + governance denetimi. `data_audit_report.json` üretir. Ayrıntılar için [Denetim Rehberi](../guides/data_audit-tr.md). ```bash -# Tek split -forgelm --data-audit data/sft.jsonl --output ./audit/ +# Tek split (v0.5.1 subcommand) +forgelm audit data/sft.jsonl --output ./audit/ # Çoklu split (train.jsonl / validation.jsonl / test.jsonl içeren dizin) -forgelm --data-audit data/ --output ./audit/ +forgelm audit data/ --output ./audit/ + +# Tüm split'leri göster (bulgu olmasa bile) +forgelm audit data/ --verbose + +# Özel Hamming eşiği +forgelm audit data/ --near-dup-threshold 5 # stdout'a makine-okunabilir özet -forgelm --data-audit data/sft.jsonl --output ./audit/ --output-format json +forgelm audit data/sft.jsonl --output ./audit/ --output-format json + +# Eski alias (çalışmaya devam ediyor; bir uyarı log'lanır) +forgelm --data-audit data/sft.jsonl --output ./audit/ ``` -Denetim şunları yakalar: split başına örnek sayısı + uzunluk dağılımı, top-3 dil tespiti, simhash near-duplicate oranı, cross-split sızıntı (sessiz train-test örtüşmesi), PII flag sayıları (e-posta / telefon / Luhn-doğrulanmış kredi kartı / IBAN / TR-DE-FR-US ulusal kimlikleri). +Denetim şunları yakalar: split başına örnek sayısı + uzunluk dağılımı, top-3 dil tespiti, **LSH-banded** simhash near-duplicate oranı (Faz 11.5; uç eşiklerde brute-force fallback), cross-split sızıntı (sessiz train-test örtüşmesi), PII flag sayıları + **şiddet katmanları** (`pii_severity` bloğu her PII tipini critical / high / medium / low olarak puanlar ve bir `worst_tier` verdict yüzdürür). Trainer'ın `output_dir`'ünde `data_audit_report.json` mevcutsa, bulgular EU AI Act Madde 10 governance artifact'ında `data_audit` anahtarı altında otomatik olarak inline edilir. diff --git a/docs/reference/usage.md b/docs/reference/usage.md index 6aeed9d1..01bc6cd7 100644 --- a/docs/reference/usage.md +++ b/docs/reference/usage.md @@ -129,7 +129,7 @@ forgelm --config my_config.yaml --generate-data This uses the `synthetic` config section to generate training data from a teacher model before training begins. See the [Configuration Guide](configuration.md) for all synthetic data options. -### Document Ingestion (v0.5.0+) +### Document Ingestion (v0.5.0+; token-aware in v0.5.1) Convert raw PDF / DOCX / EPUB / TXT / Markdown into SFT-ready JSONL. Optional dep: `pip install forgelm[ingestion]`. See [Ingestion Guide](../guides/ingestion.md). @@ -146,24 +146,37 @@ forgelm ingest ./scan.pdf --strategy sliding --chunk-size 1024 --overlap 128 \ # Mask PII before writing forgelm ingest ./customer_emails/ --pii-mask --output data/anon.jsonl + +# Token-aware chunking (v0.5.1) — sizes chunks against your model's vocab +forgelm ingest ./policies/ --recursive --output data/policies.jsonl \ + --chunk-tokens 1024 --tokenizer "Qwen/Qwen2.5-7B-Instruct" ``` -### Dataset Audit (v0.5.0+) +### Dataset Audit (v0.5.0+; subcommand promoted in v0.5.1) CPU-only quality + governance audit. Produces `data_audit_report.json`. See [Audit Guide](../guides/data_audit.md). ```bash -# Single split -forgelm --data-audit data/sft.jsonl --output ./audit/ +# Single split (v0.5.1 subcommand) +forgelm audit data/sft.jsonl --output ./audit/ # Multi-split directory (train.jsonl / validation.jsonl / test.jsonl) -forgelm --data-audit data/ --output ./audit/ +forgelm audit data/ --output ./audit/ + +# Show every split (no zero-finding fold) +forgelm audit data/ --verbose + +# Custom Hamming threshold for near-duplicate detection +forgelm audit data/ --near-dup-threshold 5 # Machine-readable summary on stdout -forgelm --data-audit data/sft.jsonl --output ./audit/ --output-format json +forgelm audit data/sft.jsonl --output ./audit/ --output-format json + +# Legacy alias (kept working; logs a one-line deprecation notice) +forgelm --data-audit data/sft.jsonl --output ./audit/ ``` -The audit captures: per-split sample count + length distribution, top-3 language detection, simhash near-duplicate rate, cross-split leakage (silent train-test overlap), PII flag counts (email / phone / Luhn-validated credit card / IBAN / TR-DE-FR-US national IDs). +The audit captures: per-split sample count + length distribution, top-3 language detection, **LSH-banded** simhash near-duplicate rate (Phase 11.5; brute-force fallback at edge thresholds), cross-split leakage (silent train-test overlap), PII flag counts with **severity tiers** (`pii_severity` block grading each PII type as critical / high / medium / low and surfacing a `worst_tier` verdict). When `data_audit_report.json` is present in the trainer's `output_dir` at training time, its findings are inlined under the `data_audit` key of the EU AI Act Article 10 governance artifact automatically. diff --git a/docs/roadmap-tr.md b/docs/roadmap-tr.md index b276c450..5cdadae4 100644 --- a/docs/roadmap-tr.md +++ b/docs/roadmap-tr.md @@ -9,19 +9,20 @@ | ✅ Tamam | [Faz 1-9](roadmap/completed-phases.md) | SOTA iyileştirmeleri, değerlendirme, güvenilirlik, kurumsal entegrasyon, ekosistem, hizalama stack'i, güvenlik, EU AI Act uyumluluğu (Madde 9-17 + Ek IV), gelişmiş güvenlik zekası | | ✅ Tamam | [Faz 10 — Post-Training Tamamlama](roadmap/phase-10-post-training.md) | `inference.py`, `chat`, `export` (GGUF), `--fit-check`, `deploy` — `v0.4.0` | | ✅ Tamam | [Faz 10.5 — Quickstart Katmanı ve Onboarding](roadmap/phase-12-quickstart.md) | `forgelm quickstart