diff --git a/CHANGELOG.md b/CHANGELOG.md index 91386420..83c01e20 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,138 @@ All notable changes to ForgeLM are documented here. ## [Unreleased] +Phase 15 (Ingestion Pipeline Reliability) Wave 1 + Wave 2 landed on the +development branch. Closes the silent-failure gap the 2026-05-11 pilot +exposed across PDF / DOCX / EPUB / TXT / MD ingestion plus the +user-facing playground notebook. Targeted for v0.6.0. + +### Added (Phase 15 Wave 1) + +- **`forgelm/_pypdf_normalise.py`** — Turkish glyph normalisation + profile (`turkish` default, `none` opt-out, future-pluggable + `--normalise-profile`) mapping the audit-measured pypdf font-fallback + artefacts (`ø Õ ú ÷ ࡟` → `İ ı ş ğ •`) back to their correct Turkish + characters at chunk-write time. +- **`forgelm/_script_sanity.py`** — language-aware Unicode-block sanity + check (`tr` / `en` / `de` / `fr` / `es` / `it` / `pt`) that fires a + WARNING + structured `script_sanity_summary` block when the out-of- + script ratio exceeds the calibrated 1.5 % threshold. Catches both + pypdf font corruption and TXT encoding mis-routing. +- **`forgelm ingest --language-hint LANG`** + `--script-sanity-threshold` + + `--normalise-profile {turkish,none}` + `--no-normalise-unicode` CLI + flags wiring Tasks 2 + 3 to the operator surface. +- **`forgelm ingest` end-of-run quality pre-signal (Task 4)** — three + cheap row-level checks (alpha-ratio, weird-char ratio, repeated-line + ratio) emitting `[WARN] N/M chunks below ingestion quality threshold` + + a structured `quality_presignal` block in `notes_structured`. Opt + out via `--no-quality-presignal`. +- **`forgelm doctor` pypdf-normalise diagnostic (`pypdf_normalise.turkish`)** + — confirms the glyph-normalisation table loaded and round-trips + cleanly without running a test ingest. +- **DOCX explicit header / footer extraction (Task 6)** — `_extract_docx` + reads `doc.sections[i].header.paragraphs` + `.footer.paragraphs` and + subtracts those lines from the body before chunking. +- **EPUB spine-order + nav / cover / copyright skip (Task 7)** — + `_extract_epub` iterates `book.spine` (reading order) instead of + file order; default skip-list (`nav`, `cover`, `copyright`, + `colophon`, `titlepage`, `frontmatter`) opt-out via + `--epub-no-skip-frontmatter`. +- **TXT UTF-8 BOM strip + MD YAML frontmatter detection (Task 8)** — + `_extract_text` / new `_extract_markdown` strip the BOM via + `encoding="utf-8-sig"` and detect `---\n...\n---\n` YAML frontmatter; + `--keep-md-frontmatter` opts back in. +- **`forgelm/ingestion.strip_paragraph_packed_headers`** — second-pass + dedup against paragraph-packed text catches header lines that + survived the page-level pass. +- **`tests/test_ingestion_reliability.py`** — 36 regression tests + locking the Wave 1 + Wave 2 behaviour across PDF / DOCX / EPUB / + TXT / MD fixtures. + +### Added (Phase 15 Wave 2) + +- **`forgelm ingest --strip-pattern REGEX` (Task 11)** — operator- + controlled regex stripping with ReDoS guard. Patterns are + structurally validated at CLI-parse time (rejects nested unbounded + quantifiers like `(a+)+b` and `.*?` + back-reference under DOTALL + per the SonarCloud `python:S5852` shape rule), wrapped in a 5-second + per-pattern SIGALRM budget on POSIX. Opt out of the timeout via + `--strip-pattern-no-timeout`. +- **`forgelm ingest --page-range START-END` (Task 12)** — restrict + PDF extraction to a contiguous page slice (1-indexed inclusive). + Validation failures (`start < 1`, `start > end`, + `start > page_count`) abort the run with `EXIT_CONFIG_ERROR` via a + new `IngestParameterError(ValueError)` that bypasses the per-file + soft-fail catch. +- **PDF front-matter / back-matter heuristic (Task 13, default ON)** — + three-signal heuristic (alpha < 0.45 + underscore > 0.10 + ≥ 5 + inline page-number matches) drops up to 12 leading + 12 trailing + pages and emits a WARNING + `frontmatter_pages_dropped` field in + `notes_structured`. Opt out via `--keep-frontmatter`. +- **`forgelm ingest --strip-urls {keep,mask,strip}` (Task 14)** — + detected URLs are masked with `[URL]`, stripped outright, or kept + (default). Independent of `--all-mask` (URL handling is a + content-shape decision, not a GDPR redaction). +- **PDF multi-column layout warning (Task 15)** — samples the first + three pages' Tj-text positions via pypdf's `visitor_text` callback, + fires a WARNING when a > 30 %-of-page-width two-cluster gap is + detected. No fix attempt — operator switches strategy / pre- + processes externally. + +### Changed (Phase 15) + +- **`forgelm audit --quality-filter` is now default-ON in v0.6.0** + (Task 5). Pre-v0.6.0 invocations with explicit `--quality-filter` + keep identical behaviour. Operators wanting the pre-v0.6.0 opt-in + semantics pass the new `--no-quality-filter` companion flag. +- **`_strip_repeating_page_lines` window-based dedup (Task 1)** — + replaces the pre-Phase-15 outermost-row-only iteration. Catches + variable-outer-line + constant-deeper-line corpora (the audit §1.1 + trap) by inspecting the top-3 / bottom-3 rows per page on every + pass. A second pass runs after paragraph packing to mop up survivor + headers. +- **`notebooks/ingestion_playground.ipynb` (Task 9)** — Cell 5 exposes + `CHUNK_TOKENS` + `TOKENIZER` + `LANGUAGE_HINT` knobs; Cell 8 markdown + explains the new quality-filter checks; Cell 9 passes + `--quality-filter` explicitly for forward-compat across v0.5 → v0.6; + Cell 10 pretty-prints the `quality_summary` block alongside the + existing PII / secrets / near-duplicate / language sections. +- **`IngestionResult` schema** — additive Phase 15 fields: + `pdf_paragraph_packed_lines_stripped`, `script_sanity_triggered`, + `strip_pattern_substitutions`, `urls_handled`, + `frontmatter_pages_dropped`. No pre-Phase-15 key was renamed. + +### Fixed (review-absorption rounds 3 + 4 + 5 follow-up to v0.5.7) + +(No public surface change, but training-pipeline UX, doctor robustness, +and test-isolation guarantees improve. Round 5 closes the residual +CodeRabbit findings against the round-3/4 commits.) + +### Fixed (round 5 — CodeRabbit follow-up) + +- **`forgelm/cli/_training.py::_preflight_numpy_torch_abi`** — any + unexpected exception from the underlying probe (corrupted torch + install where `torch.__version__` raises `AttributeError`, etc.) + is now caught and converted into a structured + `abi_preflight_crashed` JSON envelope. Previously the raw Python + traceback would pre-empt the `--output-format json` contract that + every other CLI failure path honours. Exit code stays + `EXIT_TRAINING_ERROR` (= 2), matching the broken-ABI verdict so + CI/CD branching doesn't need to distinguish "ABI bad" from "ABI + probe died". +- **`CLAUDE.md`** — exit-code contract was stated as `0/1/2/3/4`, + but the canonical table in `docs/standards/error-handling.md` + documented `0/1/2/3/4/5` since v0.5.5 (Phase 22 added + `EXIT_WIZARD_CANCELLED = 5`). CLAUDE.md now matches the actual + contract — the standard was right, the agent guidance was stale. +- **`docs/roadmap/completed-phases.md::Phase 12` summary** — Tier 1 + status line claimed a `[ingestion-secrets]` extra "via + `detect-secrets` with regex fallback", but that extra was never + published; only `[ingestion-pii-ml]` exists in `pyproject.toml`'s + extras surface. Reworded to record the historical plan accurately + ("`detect-secrets` integration was originally planned as a + `[ingestion-secrets]` extra but deferred — only + `[ingestion-pii-ml]` ultimately shipped"). Pure docs accuracy fix. + ## [0.5.7] — 2026-05-11 Patch release on top of `v0.5.6`. Two production blockers and one UX @@ -1726,5 +1858,7 @@ Major release: ForgeLM goes from a basic SFT fine-tuning tool to a full-stack LL - Basic evaluation checks (max loss, baseline comparison) - Auto-revert on quality degradation -[Unreleased]: https://github.com/cemililik/ForgeLM/compare/v0.5.5...HEAD +[Unreleased]: https://github.com/cemililik/ForgeLM/compare/v0.5.7...HEAD +[0.5.7]: https://github.com/cemililik/ForgeLM/compare/v0.5.6...v0.5.7 +[0.5.6]: https://github.com/cemililik/ForgeLM/compare/v0.5.5...v0.5.6 [0.5.5]: https://github.com/cemililik/ForgeLM/compare/v0.5.0...v0.5.5 diff --git a/docs/guides/data_audit-tr.md b/docs/guides/data_audit-tr.md index e39e7fe0..9bc74def 100644 --- a/docs/guides/data_audit-tr.md +++ b/docs/guides/data_audit-tr.md @@ -8,8 +8,11 @@ near-duplicate tespitini, streaming JSONL okuyucusunu, PII şiddet katmanlarını, atomik disk yazımı ve verbose-by-default kısaltma politikasını ekledi. **Faz 12 (`v0.5.0`'da birleşti)** opt-in MinHash LSH dedup yöntemini (`--dedup-method minhash`), her zaman çalışan -code/credential leakage taramasını (`secrets_summary`), ve opt-in -heuristic kalite filtresini (`--quality-filter`) ekledi. +code/credential leakage taramasını (`secrets_summary`) ve heuristic +kalite filtresini (`--quality-filter`) ekledi — başlangıçta opt-in, +**Faz 15 Görev 5 ile v0.6.0'dan itibaren default-AÇIK**; v0.6.0 öncesi +opt-in semantiğini isteyen operatörler yeni `--no-quality-filter` +companion bayrağını kullanır. Trainer'ın `output_dir`'ünde mevcutsa, rapor EU AI Act Madde 10 veri governance artifact'ına otomatik olarak beslenir. @@ -35,7 +38,9 @@ forgelm audit data/ --near-dup-threshold 5 pip install 'forgelm[ingestion-scale]' forgelm audit data/large_corpus.jsonl --dedup-method minhash --jaccard-threshold 0.85 -# Faz 12: opt-in heuristic kalite filtresi (Gopher/C4 stili) +# Faz 12 + Faz 15: heuristic kalite filtresi (Gopher/C4 stili). +# v0.6.0'dan itibaren default-AÇIK; explicit flag yedek/zararsızdır. +# v0.6.0 öncesi opt-in davranışına dönmek için --no-quality-filter geçirin. forgelm audit data/ --quality-filter # Faz 17: çoklu-split corpora için split-seviyesi paralelizm @@ -484,7 +489,7 @@ forgelm audit PATH \ [--near-dup-threshold N] \ [--dedup-method {simhash,minhash}] \ [--jaccard-threshold X] \ - [--quality-filter] \ + [--quality-filter | --no-quality-filter] \ [--pii-ml] \ [--pii-ml-language LANG] \ [--croissant] \ @@ -517,9 +522,11 @@ varsayılan simhash Hamming eşiğini (3, ≈%95 benzerlik) ezer; `--dedup-method` (Faz 12) near-duplicate motorunu seçer — `simhash` (varsayılan) veya `minhash` (`[ingestion-scale]` extra'sı şart; `--jaccard-threshold` cutoff'u kontrol eder, varsayılan 0.85). -`--quality-filter` (Faz 12) heuristic kalite skorlamasını opt-in -çalıştırır. Credential/secrets taraması **her zaman açık** — kapatma -flag'i yoktur. +`--quality-filter` (Faz 12, **Faz 15 Görev 5 ile v0.6.0'dan itibaren +default-AÇIK**) heuristic kalite skorlamasını çalıştırır; +`--no-quality-filter` v0.6.0 öncesi opt-in semantiklerini isteyen +operatörler için opt-out sağlar. Credential/secrets taraması **her +zaman açık** — kapatma flag'i yoktur. Eski `forgelm --data-audit PATH` flag'i deprecation alias olarak korunuyor; davranış aynı, sadece ek bir uyarı log'lanıyor. diff --git a/docs/guides/data_audit.md b/docs/guides/data_audit.md index 2d091f21..4a3e7c0e 100644 --- a/docs/guides/data_audit.md +++ b/docs/guides/data_audit.md @@ -9,8 +9,10 @@ LSH-banded near-duplicate detection, streaming JSONL reading, PII severity tiers, atomic on-disk write, and a verbose-by-default truncation policy. **Phase 12 (consolidated into `v0.5.0`)** added an opt-in MinHash LSH dedup method (`--dedup-method minhash`), a code/credential leakage scan -that runs always-on (`secrets_summary`), and an opt-in heuristic -quality filter (`--quality-filter`). +that runs always-on (`secrets_summary`), and a heuristic quality filter +(`--quality-filter`) — initially opt-in, **flipped to default-ON in v0.6.0 +per Phase 15 Task 5**; operators wanting the pre-v0.6.0 opt-in semantics +pass the new `--no-quality-filter` companion. The report feeds the EU AI Act Article 10 data governance artifact automatically when present in the trainer's `output_dir`. @@ -36,7 +38,9 @@ forgelm audit data/ --near-dup-threshold 5 pip install 'forgelm[ingestion-scale]' forgelm audit data/large_corpus.jsonl --dedup-method minhash --jaccard-threshold 0.85 -# Phase 12: opt-in heuristic quality filter (Gopher/C4 style) +# Phase 12 + Phase 15: heuristic quality filter (Gopher/C4 style). +# DEFAULT-ON from v0.6.0; explicit flag is redundant but harmless. +# Pass --no-quality-filter to restore the pre-v0.6.0 opt-in semantics. forgelm audit data/ --quality-filter # Phase 17: split-level parallelism for multi-split corpora @@ -478,7 +482,7 @@ forgelm audit PATH \ [--near-dup-threshold N] \ [--dedup-method {simhash,minhash}] \ [--jaccard-threshold X] \ - [--quality-filter] \ + [--quality-filter | --no-quality-filter] \ [--pii-ml] \ [--pii-ml-language LANG] \ [--croissant] \ @@ -496,8 +500,10 @@ Hamming-distance cutoff of 3 (≈95 % similarity); ignored when `--dedup-method=minhash`. `--dedup-method` (Phase 12) selects the near-duplicate engine — `simhash` (default) or `minhash` (needs `[ingestion-scale]` extra; `--jaccard-threshold` controls the cutoff, -default 0.85). `--quality-filter` (Phase 12) opts into the heuristic -quality scoring. The credential/secrets scan is **always on** — there +default 0.85). `--quality-filter` (Phase 12, **default-ON from v0.6.0 +per Phase 15 Task 5**) runs the heuristic quality scoring; +`--no-quality-filter` opts out for operators wanting the pre-v0.6.0 +opt-in semantics. The credential/secrets scan is **always on** — there is no flag to disable it. > **Note:** This matches the behavior summarised at the top of this guide: diff --git a/docs/guides/ingestion-tr.md b/docs/guides/ingestion-tr.md index 0c1299c9..3183e5bd 100644 --- a/docs/guides/ingestion-tr.md +++ b/docs/guides/ingestion-tr.md @@ -234,6 +234,17 @@ forgelm ingest INPUT_PATH \ [--pii-mask] \ [--secrets-mask] \ [--all-mask] \ + [--language-hint LANG] \ + [--script-sanity-threshold X] \ + [--normalise-profile {turkish,none} | --no-normalise-unicode] \ + [--no-quality-presignal] \ + [--epub-no-skip-frontmatter] \ + [--keep-md-frontmatter] \ + [--strip-pattern REGEX ...] \ + [--strip-pattern-no-timeout] \ + [--page-range START-END] \ + [--keep-frontmatter] \ + [--strip-urls {keep,mask,strip}] \ [--output-format {text,json}] \ [--quiet | --log-level {DEBUG,INFO,WARNING,ERROR}] ``` @@ -245,12 +256,15 @@ PGP / Azure credential span'lerini chunk'lar JSONL'a inmeden önce `[REDACTED-SECRET]` ile değiştirir. `--output-format json` standart çıktıya makine-okunabilir bir özet -yazar: dosya yolları, chunk sayısı, format sayıları, notlar, Faz -11.5'te eklenen `notes_structured` (makine-okunabilir `{key: value}` -yapısı) ve Faz 12'de eklenen `secrets_redaction_counts` (`--secrets-mask` -çalıştığında her secret tipinden kaç span'in `[REDACTED-SECRET]` ile -değiştirildiğini sayan `{secret_type: count}` haritası). CI/CD -pipeline'larında kullanışlı. +yazar. Zarf Faz 11.5 alanlarını (`notes`, `notes_structured`, +`pdf_header_footer_lines_stripped`), Faz 12 alanlarını +(`secrets_redaction_counts`) ve **Faz 15** ek alanlarını taşır: +`pdf_paragraph_packed_lines_stripped` (ikinci-pas dedup'ın sıyırdığı +sağkalan header sayısı), `script_sanity_triggered` (eşiği aşan dosya +sayısı), `strip_pattern_substitutions` (`--strip-pattern` eşleşme +sayısı), `urls_handled` (`--strip-urls` ile mask/strip edilen URL +sayısı) ve `frontmatter_pages_dropped` (front-matter heuristic'inin +düşürdüğü sayfa sayısı). CI/CD pipeline'larında kullanışlı. ### Token-aware chunking — `--chunk-tokens` (Faz 11.5) @@ -270,15 +284,214 @@ boyutudur ve `--overlap` ile aynı yarım-pencere üst sınırına tabidir. `--chunk-tokens` ile `--tokenizer` zorunludur — varsayılan vocab seçmiyoruz çünkü chunk sayısı modelden modele sessizce değişirdi. -### PDF sayfa header/footer dedup (Faz 11.5) +### PDF sayfa header/footer dedup (Faz 11.5 + Faz 15 Görev 1) `forgelm ingest`, bir PDF'in sayfalarının ≥ %70'inde ilk veya son -boş-olmayan satır olarak tekrarlayan satırları (şirket filigranı, -copyright satırı, sayfa numarası vb.) chunk'lamadan önce otomatik -olarak temizler. Açık/kapatma flag'i yok; bayrak gerektirmiyor. -Audit'in `near_duplicate_pairs` metriğindeki gürültüyü azaltır. 3 -sayfadan kısa belgelerde devre dışı kalır. Yapılandırılmış notlar -çıktısı `pdf_header_footer_lines_stripped` alanını taşır. +birkaç boş-olmayan satır olarak tekrarlayan satırları (şirket +filigranı, copyright satırı, sayfa numarası vb.) chunk'lamadan önce +otomatik olarak temizler. + +**Faz 15 Görev 1** inceleme penceresini sadece en dıştaki satırdan +sayfa başına **üst-3 / alt-3 satıra** genişletir. Bu sayede, dış +satırı değişken (bölüm başlığı gibi) ama bir alttaki satırı sabit +(yayıncı kimliği gibi) olan bir corpus tamamen temizlenir. Faz 15 +öncesi implementasyon, en dış satır sınıra ulaşmadığında pas 1'de +döngüden çıkıyordu ve audit'in 2026-05-11 pilot corpus'unda 74 / 82 +chunk'a sızan sabit kimlik satırını gözden kaçırıyordu. + +Paragraph paketlemenin ardından ikinci bir pas, chunker'ın orta-bloğa +yapıştırdığı sağ-kalan header'ları temizler — yapılandırılmış notların +`pdf_paragraph_packed_lines_stripped` alanı bu pas'ı raporlar. + +Her iki pas da otomatiktir; flag yok, "PDF olmayan dosya gönder" +dışında opt-out yok. 3 sayfadan kısa belgelerde devre dışı kalır. +Yapılandırılmış notlar çıktısı `pdf_header_footer_lines_stripped` +alanını taşır. + +### Script-sanity kontrolü + glyph normalizasyonu (Faz 15 Görevler 2 + 3) + +Pypdf, custom glyph adları içeren fontlarla bezenmiş PDF'lerde +zaman zaman font-fallback artefaktları üretir (audit, gerçek bir +Türkçe pilotta `ø Õ ú ÷ ࡟` ölçtü). v0.6.0 iki çözüm sunar: + +- **`--language-hint LANG`** her dosya ekstraksiyonundan sonra bir + Unicode-block sanity kontrolü çalıştırır. Out-of-script karakter + oranı kalibre edilmiş %1.5 eşiğini geçtiğinde (`--script-sanity-threshold` + ile ayarlanabilir), bir WARNING tetiklenir + yapılandırılmış + `script_sanity_summary` bloğu `notes_structured` içine düşer. + Desteklenen diller: `tr`, `en`, `de`, `fr`, `es`, `it`, `pt` + (CJK / Arapça Faz 16+'ya ertelendi). +- **`--normalise-profile {turkish,none}`** çıkarılmış metne + dil-spesifik bir glyph normalizasyon tablosu uygular. Modül-düzeyi + varsayılan **`none`** (Faz 15 round-2 / C-2: dil ipucu olmayan + korpusların sessizce yeniden yazılmasını önlemek için). Operatör + `--language-hint tr` geçirdiğinde dispatcher (CLI ve kütüphane API + her ikisinde de) otomatik olarak `turkish` profiline türetir; her + diğer ipucu (veya ipuçsuz çalışma) `none` kalır. Açık + `--normalise-profile turkish` her zaman kazanır. + `--no-normalise-unicode` veya `--normalise-profile none` + normalizasyonu tamamen devre dışı bırakır. Tablonun yüklendiğini + doğrulamak için `forgelm doctor`'ı çalıştırın — profile sağlıklı + olduğunda `pypdf_normalise.turkish: pass` satırını verir. + +### Ingest sırasında kalite ön-sinyali (Faz 15 Görev 4) + +Her çalışmanın sonunda `forgelm ingest`, üretilen her chunk'a üç ucuz +satır-seviyesi kontrol (alpha oranı, garip karakter oranı, +tekrarlanan satır oranı) uygular ve bir chunk eşiğin altına düştüğünde +tek satırlık bir nudge yazar: + +```text +[WARN] 74/82 chunks below ingestion quality threshold. Run + `forgelm audit ./out.jsonl` for detail. +``` + +Tam tanılama `forgelm audit --quality-filter` (v0.6.0'dan itibaren +default-on) altında yaşamaya devam eder; ön-sinyal yalnızca "hey, buna +bakmak isteyebilirsin" mümkün olan en küçük uyarıdır. `--no-quality-presignal` +ile kapatılır. Yapılandırılmış payload `notes_structured.quality_presignal` +altında `samples_evaluated`, `samples_flagged` ve check-bazında +sayıları taşır. + +### DOCX header / footer çıkarma (Faz 15 Görev 6) + +Word belgeleri her section'ın `` / `` parçaları altında +tekrarlayan başlık ve altbilgileri açık-açık bildirir. v0.6.0 bu +parçaları önceden okur ve satırlarını body extraction'ından çıkartır, +böylece 10 sayfa boyunca tekrarlanan 3-satırlık bir header **sıfır** +header satırı JSONL'a düşürür. + +### EPUB spine sırası + nav / cover / copyright atlama (Faz 15 Görev 7) + +EPUB ekstraksiyonu `book.spine`'ı (okuma sırası) `book.get_items()` +(dosya sırası) yerine iterate eder; bölümler doğru sırayla iner. +Varsayılan skip-list dosya adı veya `epub:type` değeri `nav`, `cover`, +`copyright`, `colophon`, `titlepage` veya `frontmatter` ile eşleşen +item'ları filtreler — TOC boilerplate'inin pure noise olduğu SFT +eğitiminde işe yarar. `--epub-no-skip-frontmatter` ile opt-out. + +### TXT BOM + Markdown YAML frontmatter (Faz 15 Görev 8) + +TXT dosyaları `encoding="utf-8-sig"` ile okunur, böylece dosyanın +başındaki bir UTF-8 BOM chunk'lamadan önce transparent biçimde +strip edilir. Markdown dosyaları ayrıca `^---\n…\n---\n` YAML +frontmatter'ı tespit edip varsayılan olarak strip eder — metadata +bloğunda eğitim *istediğinizde* `--keep-md-frontmatter` ile geri +açabilirsiniz. + +### Operatör strip-pattern'leri — `--strip-pattern REGEX` (Faz 15 Wave 2 Görev 11) + +Dedup heuristiği'nin gözden kaçırdığı bilinen boilerplate (değişken +running header, DOI satırı, watermark) için escape hatch. Pattern +başına bir kez `--strip-pattern REGEX` geçirin; eşleşmeler +chunk'lamadan önce çıkarılmış metinden silinir: + +```bash +forgelm ingest ./corpus/ --output data/clean.jsonl \ + --strip-pattern '^Confidential — internal use only$' \ + --strip-pattern '^https://example\.com/qr\?KOD=\d+$' +``` + +Her pattern **önceden yapısal olarak doğrulanır**: iç içe sınırsız +quantifier'lar (`(a+)+b`) ve DOTALL altında `.*?` + geri-referans +(SonarCloud `python:S5852` polinomial-runtime şekli) `EXIT_CONFIG_ERROR` +ile reddedilir. POSIX'te per-pattern 5-saniyelik SIGALRM bütçesi +en kötü durum eşleşme maliyetini sınırlar (pattern'lerinizin lineer +olduğunu bağımsız doğruladığınızda `--strip-pattern-no-timeout` ile +opt-out edebilirsiniz). + +### `--page-range START-END` (Faz 15 Wave 2 Görev 12) + +PDF ekstraksiyonunu sürekli bir sayfa dilimine sınırlar (1-indeksli, +inclusive). Heuristik front-matter'ı gözden kaçırdığında veya yalnızca +belirli bir bölümü istediğinizde kullanışlıdır: + +```bash +forgelm ingest ./book.pdf --output data/ch3.jsonl --page-range 50-90 +``` + +Doğrulama hataları (`start < 1`, `start > end`, `start > page_count`) +çalışmayı `EXIT_CONFIG_ERROR` (1) ile sonlandırır; CI/CD pipeline'ları +herhangi bir operatör-sağlamış parametre hatası için aynı şekilde +branch eder. + +### PDF front-matter / back-matter heuristiği (Faz 15 Wave 2 Görev 13, varsayılan AÇIK) + +v0.6.0, PDF'in ilk 12 / son 12 sayfasında üç-sinyalli bir heuristic +etkinleştirir: bir sayfanın alpha oranı < 0.30 (round-4'te 0.45'ten +sıkıştırıldı) VE leader oranı > 0.10 (uzunluğu ≥ 3 olan `_` + `.` +çalışmalarının birleşimi) VE ≥ 5 inline `\n<1-3 rakam>\n` sayfa +numarası eşleşmesi varsa, sayfa düşürülür ve indeksleri listeleyen +bir WARNING tetiklenir. ToC / masthead / index / glossary +boilerplate'ini yakalar. + +`--keep-frontmatter` ile Faz 15 öncesi "her şeyi tut" davranışına +geri dönülür. Yapılandırılmış notlar `frontmatter_pages_dropped` +raporlar; downstream audit operasyonu spot-check edebilir. + +> **Kalibrasyon uyarısı (round-5 bağımsız review).** Heuristic, +> audit'in pilot Türkçe-textbook ToC şekli için kalibre edilmiştir: +> tek-kelimelik bölüm başlıkları + ağır nokta leader'lar + inline +> sayfa numaraları (pilot sayfalarda alpha ≈ 0.15). Tam-cümle bölüm +> başlıkları olan gerçekçi İngilizce ToC'lar (`Chapter 1: Introduction +> to the Subject ……… 14`) alpha ≈ 0.47 ölçer, 0.30 eşiğin üzerinde, +> ve **değişmeden geçer**. Bu tür corpus'lar üzerinde çalışan +> operatörler ya (a) front-matter heuristic'i atlayacağını kabul edip +> `--strip-pattern` ile downstream'de budamalı, ya da (b) front-matter'ı +> manuel olarak atlamak için `--page-range 12-N` geçirmelidir. +> 3-sinyalli AND filtresi (alpha + leader + inline-sayfa-numarası) +> aynı zamanda gerçekçi form template / egzersiz sayfalarını da +> korur — bunlar inline-sayfa-numarası sinyalinden yoksundur ve +> drop'u asla tetiklemez. + +### `--strip-urls {keep,mask,strip}` (Faz 15 Wave 2 Görev 14) + +QR-kod referansları, DOI altbilgileri veya modelin ezberleyebileceği +sosyal-medya linkleri gömen corpus'lar için URL davranışı: + +- `keep` (varsayılan) — URL'leri olduğu gibi geçirir. +- `mask` — her URL'yi `[URL]` placeholder'ı ile değiştirir. +- `strip` — URL'leri tamamen siler. + +URL yönetimi bilinçli olarak `--all-mask` (Faz 12.5 PII + secrets +kısayolu) ile **bağımsızdır**. URL stripping bir content-shape +kararıdır, GDPR redaksiyonu değildir, ve iki flag ailesi ortogonal +kalır. + +### Çok-kolonlu PDF uyarısı (Faz 15 Wave 2 Görev 15) + +İki-kolonlu akademik makaleler, hükümet regülasyon yayınları ve +çok-kolonlu hukuki layoutlar pypdf'in metin-ekstraksiyon okuma +sırasını şaşırtır. v0.6.0 ilk üç sayfanın metin pozisyonlarını +pypdf'in `visitor_text` callback'i üzerinden örnekler ve sayfa +genişliğinin %30'undan büyük iki-küme boşluğu tespit ettiğinde bir +WARNING tetikler: + +```text +[WARN] Detected 2-column layout in 'paper.pdf' — reading order may be + scrambled. Consider --strategy sliding with a larger --chunk-size, + or pre-process the PDF with a layout-aware tool. +``` + +Otomatik fix yok — bu, operatörün strateji değiştirmesi gerektiğine +dair "hiç-yoktan-iyi" bir sinyaldir. Camelot-py / pdfplumber entegrasyonu +Wave 3 backlog'undadır. + +> **Bilinen sınırlama (Faz 15 round-3 review).** Mevcut dedektör bir +> sayfadaki tüm çıkarılmış Tj glyph'lerinin min / max x-pozisyonlarını +> karşılaştırır, satır-başlangıç pozisyonlarını değil. pypdf'in +> `visitor_text` callback'i glyph başına ateşlendiği için tek-kolonlu +> bir sayfanın glyph'leri doğal olarak geniş bir x-aralığına yayılır +> (tipik gövde satırında ≈ %64 sayfa genişliği), bu nedenle algoritma +> tek-kolonlu corpus'larda false-positive vermemek için göreceli olarak +> yüksek %30 sayfa-genişlik boşluk eşiği kullanır. Yan etki: tipik +> 5–10 mm oluğa sahip (≈ sayfa-genişliğinin %5–8'i) gerçek dünya +> iki-kolonlu sayfası bugün uyarıyı tetiklemez. Dedektör aşırı layout'larda +> (akademik posterler, geniş oluklu regülasyon yayınları) güvenilirdir +> ama yayın-kalite iki-kolonlu makaleleri kaçırır. Histogram-tabanlı +> bimodal-mode refactor Wave 3 takipi olarak izlenmektedir (bkz. +> [`docs/roadmap/phase-15-ingestion-reliability.md`](../roadmap/phase-15-ingestion-reliability.md) +> Wave 3 — multi-column layout extraction). ### Markdown-bilen splitter — `--strategy markdown` (Faz 12) diff --git a/docs/guides/ingestion.md b/docs/guides/ingestion.md index 5bec0a6b..f138da29 100644 --- a/docs/guides/ingestion.md +++ b/docs/guides/ingestion.md @@ -235,14 +235,31 @@ forgelm ingest INPUT_PATH \ [--pii-mask] \ [--secrets-mask] \ [--all-mask] \ + [--language-hint LANG] \ + [--script-sanity-threshold X] \ + [--normalise-profile {turkish,none} | --no-normalise-unicode] \ + [--no-quality-presignal] \ + [--epub-no-skip-frontmatter] \ + [--keep-md-frontmatter] \ + [--strip-pattern REGEX ...] \ + [--strip-pattern-no-timeout] \ + [--page-range START-END] \ + [--keep-frontmatter] \ + [--strip-urls {keep,mask,strip}] \ [--output-format {text,json}] \ [--quiet | --log-level {DEBUG,INFO,WARNING,ERROR}] ``` -`--output-format json` emits a machine-readable summary to stdout (file -paths, chunk count, format counts, notes, `notes_structured` — -machine-readable `{key: value}` from Phase 11.5 — and `secrets_redaction_counts` -from Phase 12). Useful for CI/CD pipelines. +`--output-format json` emits a machine-readable summary to stdout. The +envelope carries Phase 11.5 keys (`notes`, `notes_structured`, +`pdf_header_footer_lines_stripped`), Phase 12 keys +(`secrets_redaction_counts`), and the **Phase 15** additive fields: +`pdf_paragraph_packed_lines_stripped` (survivor-header second-pass +count), `script_sanity_triggered` (files with out-of-script char ratio +above threshold), `strip_pattern_substitutions` (`--strip-pattern` +match count), `urls_handled` (URLs masked / stripped by +`--strip-urls`), and `frontmatter_pages_dropped` (front-matter +heuristic drop count). Useful for CI/CD pipelines. ### Token-aware chunking — `--chunk-tokens` (Phase 11.5) @@ -263,16 +280,209 @@ the override). `--overlap-tokens N` is the sliding-window equivalent of pick a default vocab because the resulting chunk count would silently differ per-model. -### PDF page-level header/footer dedup (Phase 11.5) +### PDF page-level header/footer dedup (Phase 11.5 + Phase 15 Task 1) + +`forgelm ingest` strips lines that recur as the first or last few +non-empty lines on ≥ 70 % of a PDF's pages — typical company +watermarks, copyright lines, page numbers — before chunking. + +**Phase 15 Task 1** widens the inspection window from the single +outermost row to the **top-3 / bottom-3 rows** per page so a corpus +with a variable outer line (per-chapter section title) plus a constant +deeper line (publication identifier) gets fully dedup'd. The pre-Phase-15 +implementation exited the dedup loop on pass 1 when the outermost line +didn't recur, leaving the deeper constant line stranded in 74 / 82 +chunks on the audit's pilot corpus. + +A second pass runs after paragraph packing to mop up survivor headers +the chunker re-glued mid-block — surfaced as the structured-notes +field `pdf_paragraph_packed_lines_stripped`. + +Both passes are automatic; no flag, no opt-out beyond "send a non-PDF". +Skipped on documents shorter than 3 pages. The structured-notes payload +reports `pdf_header_footer_lines_stripped` so operators can see +post-hoc that dedup actually did work. + +### Script-sanity check + glyph normalisation (Phase 15 Tasks 2 + 3) + +Pypdf occasionally produces font-fallback artefacts on PDFs whose +fonts declare custom glyph names (the audit measured `ø Õ ú ÷ ࡟` for +Turkish characters on a real-world pilot). Two mitigations ship in +v0.6.0: + +- **`--language-hint LANG`** runs a Unicode-block sanity check after + each per-file extract. When the out-of-script char ratio exceeds the + calibrated 1.5 % threshold (tunable via `--script-sanity-threshold`), + a WARNING fires + a structured `script_sanity_summary` block lands + in `notes_structured`. Supported languages: `tr`, `en`, `de`, `fr`, + `es`, `it`, `pt` (CJK / Arabic deferred to Phase 16+). +- **`--normalise-profile {turkish,none}`** applies a language-specific + glyph normalisation table to extracted text. The module-level + default is **`none`** (Phase 15 round-2 / C-2: a hint-less corpus + must not be silently rewritten). When the operator passes + `--language-hint tr` the dispatcher (CLI and library API alike) + auto-derives the `turkish` profile; every other hint (and the + unset case) stays on `none`. An explicit `--normalise-profile turkish` + always wins. `--no-normalise-unicode` or `--normalise-profile none` + disables normalisation entirely. Verify the table loaded via + `forgelm doctor` — it surfaces a `pypdf_normalise.turkish: pass` + row when the profile is healthy. + +### Ingest-time quality pre-signal (Phase 15 Task 4) + +At the end of every run `forgelm ingest` applies three cheap row-level +checks (alpha-ratio, weird-char ratio, repeated-line ratio) to every +emitted chunk and surfaces a single nudge line when any chunk falls +below threshold: -`forgelm ingest` now strips lines that recur as the first or last -non-empty line on ≥ 70 % of a PDF's pages — typical company watermarks, -copyright lines, page numbers — before chunking. This is automatic; no -flag, no opt-out beyond "send a non-PDF". Reduces audit -`near_duplicate_pairs` noise on long policy / book PDFs. Skipped on -documents shorter than 3 pages. The structured-notes payload reports -`pdf_header_footer_lines_stripped` so operators can see post-hoc that -dedup actually did work. +```text +[WARN] 74/82 chunks below ingestion quality threshold. Run + `forgelm audit ./out.jsonl` for detail. +``` + +Full diagnostics still live in `forgelm audit --quality-filter` +(default-on from v0.6.0); the pre-signal is just the smallest possible +"hey, you might want to look at this" prompt. Opt out with +`--no-quality-presignal`. The structured payload lands under +`notes_structured.quality_presignal` with `samples_evaluated`, +`samples_flagged`, and per-check counts. + +### DOCX header / footer subtraction (Phase 15 Task 6) + +Word documents declare repeating headers and footers explicitly under +each section's `` / `` parts. v0.6.0 reads those parts +up-front and subtracts their lines from the body extraction, so a +3-line repeating header across 10 pages emits **zero** header lines +into the JSONL. + +### EPUB spine order + nav / cover / copyright skip (Phase 15 Task 7) + +EPUB extraction iterates `book.spine` (reading order) instead of +`book.get_items()` (file order), so chapters land in the correct +sequence. The default skip-list filters items whose file name or +`epub:type` matches `nav`, `cover`, `copyright`, `colophon`, +`titlepage`, or `frontmatter` — useful for SFT training where TOC +boilerplate is pure noise. Opt out with `--epub-no-skip-frontmatter`. + +### TXT BOM + Markdown YAML frontmatter (Phase 15 Task 8) + +TXT files are read via `encoding="utf-8-sig"` so a UTF-8 BOM at file +start is stripped transparently before chunking. Markdown files +additionally detect `^---\n…\n---\n` YAML frontmatter and strip it +by default — opt-in retention via `--keep-md-frontmatter` when you +*want* to train on the metadata block. + +### Operator strip-patterns — `--strip-pattern REGEX` (Phase 15 Wave 2 Task 11) + +Escape hatch for known boilerplate the dedup heuristic misses (variable +running headers, DOI lines, watermarks). Pass `--strip-pattern REGEX` +once per pattern; matches are deleted from the extracted text before +chunking: + +```bash +forgelm ingest ./corpus/ --output data/clean.jsonl \ + --strip-pattern '^Confidential — internal use only$' \ + --strip-pattern '^https://example\.com/qr\?KOD=\d+$' +``` + +Each pattern is **structurally validated up-front**: nested unbounded +quantifiers (`(a+)+b`) and `.*?` + back-reference under DOTALL (the +SonarCloud `python:S5852` polynomial-runtime shape) are rejected with +`EXIT_CONFIG_ERROR`. A 5-second per-pattern SIGALRM budget bounds the +worst-case match cost on POSIX (opt out with +`--strip-pattern-no-timeout` when you've independently verified your +patterns are linear). + +### `--page-range START-END` (Phase 15 Wave 2 Task 12) + +Restrict PDF extraction to a contiguous page slice (1-indexed, +inclusive). Useful when the heuristic misses front-matter or you only +want a specific chapter: + +```bash +forgelm ingest ./book.pdf --output data/ch3.jsonl --page-range 50-90 +``` + +Validation failures (`start < 1`, `start > end`, `start > page_count`) +abort the run with `EXIT_CONFIG_ERROR` (1) so CI/CD pipelines branch +the same way for any operator-supplied parameter mistake. + +### PDF front-matter / back-matter heuristic (Phase 15 Wave 2 Task 13, default ON) + +v0.6.0 enables a three-signal heuristic on the first 12 / last 12 PDF +pages: when a page's alpha ratio < 0.30 (tightened from 0.45 in +round-4) AND leader ratio > 0.10 (combined `_` + `.` runs of length +≥ 3) AND it carries ≥ 5 inline `\n<1-3 digits>\n` page-number matches, +it is dropped with a WARNING listing the page indices. Catches ToC / +masthead / index / glossary boilerplate. + +Opt out with `--keep-frontmatter` to restore the pre-Phase-15 "keep +everything" behaviour. The structured-notes payload reports +`frontmatter_pages_dropped` so an audit downstream can spot-check the +operation. + +> **Calibration caveat (round-5 independent review).** The heuristic +> is calibrated for the audit's pilot Turkish-textbook ToC shape: +> single-word chapter titles + heavy dotted leaders + inline page +> numbers (alpha ≈ 0.15 on the pilot pages). Realistic +> English-language ToCs with full-sentence chapter titles +> (`Chapter 1: Introduction to the Subject ……… 14`) measure +> alpha ≈ 0.47, well above the 0.30 threshold, and **will pass +> through** unchanged. Operators on such corpora should either +> (a) accept that front-matter survives the heuristic and prune +> downstream via `--strip-pattern`, or (b) pass `--page-range +> 12-N` to skip the front-matter manually. The 3-signal AND filter +> (alpha + leader + inline-page-number) also protects realistic form +> templates / exercise pages, which lack the inline-page-number +> signal and so never trigger the drop. + +### `--strip-urls {keep,mask,strip}` (Phase 15 Wave 2 Task 14) + +URL handling for corpora that embed QR-code references, DOI footers, +or social-media links the model would memorise: + +- `keep` (default) — pass URLs through unchanged. +- `mask` — replace each URL with the literal `[URL]` placeholder. +- `strip` — delete URLs outright. + +URL handling is intentionally **independent of `--all-mask`** (the +Phase 12.5 PII + secrets shorthand). URL stripping is a content-shape +decision, not a GDPR redaction, and the two flag families stay +orthogonal. + +### Multi-column PDF warning (Phase 15 Wave 2 Task 15) + +Two-column academic papers, government regulatory publications, and +multi-column legal layouts confuse pypdf's text-extraction reading +order. v0.6.0 samples the first three pages' text positions via +pypdf's `visitor_text` callback and fires a WARNING when a > 30 %-of- +page-width two-cluster gap is detected: + +```text +[WARN] Detected 2-column layout in 'paper.pdf' — reading order may be + scrambled. Consider --strategy sliding with a larger --chunk-size, + or pre-process the PDF with a layout-aware tool. +``` + +No automatic fix — this is a better-than-nothing signal that the +operator should change strategies. Camelot-py / pdfplumber integration +is on the Wave 3 backlog. + +> **Known limitation (Phase 15 round-3 review).** The current detector +> compares the min / max x-positions across all extracted Tj glyphs on a +> page, not per-line start positions. Because pypdf's `visitor_text` +> callback fires per glyph, a single-column page's glyphs naturally span +> a wide x-range (≈ 64 % of page width on a typical body line), which +> means the algorithm has to use a relatively high 30 %-of-page-width +> gap threshold to avoid false-positiving on single-column corpora. +> Side-effect: a real-world two-column page with a typical 5–10 mm +> gutter (≈ 5–8 %-of-page-width) **does not** trip the warning today. +> The detector is reliable on extreme layouts (academic posters, +> wide-gutter regulatory publications) but misses publication-grade +> two-column papers. A histogram-based bimodal-mode refactor is tracked +> as a Wave 3 follow-up (see +> [`docs/roadmap/phase-15-ingestion-reliability.md`](../roadmap/phase-15-ingestion-reliability.md) +> Wave 3 — multi-column layout extraction). ### Markdown-aware splitter — `--strategy markdown` (Phase 12) diff --git a/docs/reference/data_ingestion_architecture-tr.md b/docs/reference/data_ingestion_architecture-tr.md index b0a8bd74..29d71f04 100644 --- a/docs/reference/data_ingestion_architecture-tr.md +++ b/docs/reference/data_ingestion_architecture-tr.md @@ -207,14 +207,18 @@ aşağı akış audit'inin near-duplicate sayımını şişirir. ```mermaid graph TD PAGES[sayfa başına metin] --> SPLIT[Satırlara böl] - SPLIT --> COUNT[Tüm sayfalarda ilk / son satır
frekansını say] + SPLIT --> WINDOW[Sayfa başına üst-3 / alt-3 satırı incele
_PDF_EDGE_WINDOW = 3] + WINDOW --> COUNT[Tüm sayfalarda satır
frekansını say] COUNT --> CHECK{>= yüzde 70 sayfa?} - CHECK -- evet --> POP[Eşleşen kenar satırlarını çıkar] - POP --> COUNT + CHECK -- evet --> POP[Eşleşen window satırlarını çıkar
window içinde herhangi bir offset] + POP --> WINDOW CHECK -- hayır --> JOIN[Sayfaları çift yeni satırla birleştir] - JOIN --> OUT[Temizlenmiş PDF metni] + JOIN --> PACK[strip_paragraph_packed_headers
chunk blokları üzerinde ikinci pas dedup] + PACK --> OUT[Temizlenmiş PDF metni] style CHECK fill:#002244,stroke:#00aaff + style WINDOW fill:#002244,stroke:#00aaff + style PACK fill:#002244,stroke:#00aaff ``` [`_strip_repeating_page_lines`](../../forgelm/ingestion.py) uygulama @@ -224,13 +228,38 @@ notları: sinyal çok zayıf. - Eşik `max(2, math.ceil(_PDF_REPEAT_THRESHOLD * page_count))`; böylece %70 kuralı tam %70'te tetiklenir, integer kesmesi altında %60'ta değil. -- İteratiftir: her geçiş bir kenar satırını sıyırır, sonra yeniden sayar; - böylece çok satırlı başlıklar ("şirket adı\nCONFIDENTIAL") tamamen - sıyrılır, ikinci satır artık eşleşmeyen yeni bir "ilk satır" olarak - kalmaz. +- **Faz 15 Görev 1** incelemeyi katı en-dış satırdan sayfa başına + **üst-3 / alt-3 satıra** genişletir (`_PDF_EDGE_WINDOW = 3`); böylece + değişken-dış-satırlı (her bölümde farklı başlık) ama bir alttaki + satırı sabit (yayıncı kimliği) bir corpus, dedup'ı bir-satır-daha-derin + sabiti soymadan dışarı kilitlenemez. Bug, döngünün *exit condition*'ıydı, + en-dış-satır kontrolü değildi — Faz 15 öncesi iterator, katı en-dış + pozisyonda recurrence bulunmadığında hemen kırılıyordu. Çözüm bir + window kontrolüdür, ek bir pas değildir. +- Paragraph paketlemenin ardından **ikinci bir pas** + (`strip_paragraph_packed_headers`) chunker'ın orta-bloğa yapıştırdığı + sağ-kalan header'ları temizler. `notes_structured`'da + `pdf_paragraph_packed_lines_stripped` olarak görünür. - Toplam sıyrılan satır sayısı `IngestionResult.notes_structured` altında `pdf_header_footer_lines_stripped` anahtarıyla görünür. +### Faz 15 sınırlamaları özeti + +Çok-kolonlu PDF'ler, OCR (yalnızca taranmış) PDF'leri ve RTL +script'leri v0.6.0'da desteklenmemeye devam eder: + +- **Çok-kolon** — `_maybe_warn_multi_column` yalnızca WARNING tetikler; + okuma sırası pypdf üzerinden hâlâ sol-üst-dan-sağ-alta serileştirilir. + Faz 16+ yeni bir `[ingestion-tables]` extra altında camelot-py / + pdfplumber fallback'i ekleyebilir. +- **OCR** — text-layer-tespit retry'ı yok. Audit'in mevcut "Taranmış + PDF'lerle çalışma (OCR teslim akışı)" tarifi + [`docs/guides/ingestion-tr.md`](../guides/ingestion-tr.md) içindedir; + otomatik `ocrmypdf` önerisi Wave 3'e ertelendi (audit §6). +- **RTL** — Arapça / İbranice için ekstraksiyon-sıralama normalizasyonu + Wave 3'e ertelendi. RTL corpus'lar üzerinde çalışan operatörler bugün + ters-glyph sırası beklemeli ve layout-bilen bir araçla ön-işlemeli. + ## Chunk stratejileri `CHUNK_STRATEGIES` içinde dört kayıtlı strateji; üçü CLI `--strategy` diff --git a/docs/reference/data_ingestion_architecture.md b/docs/reference/data_ingestion_architecture.md index 1fdc00bf..fbc94c20 100644 --- a/docs/reference/data_ingestion_architecture.md +++ b/docs/reference/data_ingestion_architecture.md @@ -205,14 +205,18 @@ inflate near-duplicate counts during the downstream audit. ```mermaid graph TD PAGES[per-page text] --> SPLIT[Split into lines] - SPLIT --> COUNT[Count first / last line
frequency across pages] + SPLIT --> WINDOW[Inspect top-3 / bottom-3 rows
per page _PDF_EDGE_WINDOW = 3] + WINDOW --> COUNT[Count line frequency
across pages] COUNT --> CHECK{>= 70 percent of pages?} - CHECK -- yes --> POP[Pop matching edge lines] - POP --> COUNT + CHECK -- yes --> POP[Pop matching window lines
any offset within the window] + POP --> WINDOW CHECK -- no --> JOIN[Re-join pages with double newline] - JOIN --> OUT[Cleaned PDF text] + JOIN --> PACK[strip_paragraph_packed_headers
second-pass dedup over chunk blocks] + PACK --> OUT[Cleaned PDF text] style CHECK fill:#002244,stroke:#00aaff + style WINDOW fill:#002244,stroke:#00aaff + style PACK fill:#002244,stroke:#00aaff ``` Implementation notes from @@ -222,12 +226,39 @@ Implementation notes from signal too weak. - Cutoff is `max(2, math.ceil(_PDF_REPEAT_THRESHOLD * page_count))` so the 70 % rule fires at exactly 70 %, not 60 % under integer truncation. -- Iterates: each pass strips one edge line, then re-counts so multi-line - headers ("company name\nCONFIDENTIAL") are fully stripped rather than - leaving the second line stranded. +- **Phase 15 Task 1** widens the inspection from the strict outermost + row to the top-3 / bottom-3 rows per page (`_PDF_EDGE_WINDOW = 3`), + so a variable-outer-line corpus (per-chapter title on top, page number + on bottom) no longer locks the dedup out from peeling the + constant-deeper-line one row deeper. The bug was the loop's + *exit condition*, not the outermost-line check — the pre-Phase-15 + iterator broke as soon as no recurrence was found at the strict + outermost position. The fix is a window check, not an extra pass. +- A **second pass** runs after paragraph packing + (`strip_paragraph_packed_headers`) to mop up survivor headers the + chunker re-glued mid-block. Surfaces as + `pdf_paragraph_packed_lines_stripped` in `notes_structured`. - Total stripped line count rolls into `IngestionResult.notes_structured` under `pdf_header_footer_lines_stripped` for downstream visibility. +### Phase 15 limitations recap + +Multi-column PDFs, OCR (scanned-only) PDFs, and RTL scripts remain +unsupported in v0.6.0: + +- **Multi-column** — only a WARNING is emitted via + `_maybe_warn_multi_column`; reading order is still serialised + left-to-right per pypdf. Phase 16+ may add camelot-py / pdfplumber + fallback under a new `[ingestion-tables]` extra. +- **OCR** — no text-layer-detection retry. The audit's existing + "Working with scanned PDFs (OCR handoff)" recipe in + [`docs/guides/ingestion.md`](../guides/ingestion.md) is the + operator-facing path; an automatic `ocrmypdf` recommendation is + Wave 3 deferred (audit §6). +- **RTL** — extraction-order normalisation for Arabic / Hebrew is + Wave 3 deferred. Operators on RTL corpora today should expect + reverse-glyph order and pre-process with a layout-aware tool. + ## Chunking strategies Four registered strategies in `CHUNK_STRATEGIES`; three exposed via the diff --git a/docs/roadmap.md b/docs/roadmap.md index 73fec437..a573b455 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -14,10 +14,10 @@ | ✅ Done | Phase 22 — CLI wizard parity with the in-browser surface | `forgelm --wizard` runs the same 9-step flow as the web wizard (welcome → use-case → model → strategy → trainer → dataset → training-params → compliance → evaluation), with idempotent re-run via `--wizard-start-from `, schema-driven defaults SOT, distinct `EXIT_WIZARD_CANCELLED = 5` exit code, state persistence under `$XDG_CACHE_HOME`, and validate-on-exit — bundled into `v0.5.5` (PyPI 2026-05-10) | | ✅ Done | Site documentation correction sweep | All visible YAML / artefact-path / CLI / schema claims on `site/*.html` now validate against the live `forgelm/` surface. Hero YAML demo rewritten with real Pydantic field names, compliance artefact tree redrawn against the on-disk layout, ghost YAML keys + CLI flags removed, wording aligned with live behaviour. Six-language i18n (en / tr / de / fr / es / zh) now at full parity (731 keys each) — bundled into `v0.5.5` (PyPI 2026-05-10) | | 📋 Planned | [Phase 14 — Multi-Stage Pipeline Chains](roadmap/phase-14-pipeline-chains.md) | SFT → DPO → GRPO chained config, pipeline provenance artifacts → `v0.6.0` | -| 📋 Planned | [Phase 15 — Ingestion Pipeline Reliability](roadmap/phase-15-ingestion-reliability.md) | Close silent-failure gap on `forgelm ingest` across PDF / DOCX / EPUB / TXT / Markdown + playground notebook UX alignment; surface the quality signal at ingest time + default-on `--quality-filter` audit; multi-line PDF dedup, Turkish glyph normalisation, DOCX header / footer extraction, EPUB spine + skip, TXT / MD BOM + frontmatter; dual-golden regression fixtures → `v0.6.0` | +| 🚧 In review | [Phase 15 — Ingestion Pipeline Reliability](roadmap/phase-15-ingestion-reliability.md) | Wave 1 + Wave 2 landed on `development`: window-based multi-line PDF dedup, Turkish glyph normalisation profile, language-aware Unicode-block sanity check, ingest-time quality pre-signal, default-on audit `--quality-filter`, DOCX explicit header/footer subtraction, EPUB spine + nav/cover skip, TXT UTF-8 BOM + MD YAML frontmatter strip, notebook playground alignment, plus Wave 2 `--strip-pattern` (ReDoS-guarded), `--page-range`, front-matter heuristic, `--strip-urls`, multi-column warning → `v0.6.0` | | 📋 Planned | [Phase 13 — Pro CLI & Observability Dashboard](roadmap/phase-13-pro-cli.md) | License-gated dashboard, HPO, scheduled jobs, team config store → `v0.6.0-pro` (gated on adoption + ISO/SOC 2 baseline shipped in v0.5.5) | -> **Status legend:** ✅ Released (PyPI) · 🟡 Merged on main, publish pending · ⏳ Planned +> **Status legend:** ✅ Released (PyPI) · 🟡 Merged on main, publish pending · 🚧 In review (PR open) · 📋 / ⏳ Planned **Released:** `v0.5.0` — "Document Ingestion + Data Curation Pipeline" — PyPI 2026-04-30 (Phases 11 + 11.5 + 12 + 12.5 consolidated). diff --git a/docs/roadmap/phase-15-ingestion-reliability.md b/docs/roadmap/phase-15-ingestion-reliability.md index acc8beac..1fc3016b 100644 --- a/docs/roadmap/phase-15-ingestion-reliability.md +++ b/docs/roadmap/phase-15-ingestion-reliability.md @@ -1,6 +1,6 @@ # Phase 15: Ingestion Pipeline Reliability -> **Not:** Bu dosya tek bir planlanan fazı detaylandırır. Tüm fazların özeti için [../roadmap.md](../roadmap.md). +> **Note:** This file details a single planned phase. See [../roadmap.md](../roadmap.md) for the cross-phase summary. **Goal:** Close the silent-failure gap in `forgelm ingest` exposed by an end-to-end pilot test against a real-world Turkish-language formal-publication PDF on 2026-05-11. The current pipeline produces output that *looks* successful (no errors, no warnings) but is polluted with multi-line running headers, font-fallback glyphs, bullet-character corruption, page-number leakage, and front-matter noise — degrading downstream SFT quality without any operator-facing signal. Phase 15 turns these silent failures into either auto-corrections or loud warnings, **across all five supported formats** (PDF, DOCX, EPUB, TXT, Markdown), and bakes the regression coverage into a fixture matrix. @@ -21,16 +21,16 @@ These splits are estimates, not commitments — task 1's multi-line dedup may surprise on degenerate corpora, and task 9's notebook update only matters once tasks 4+5 land, so Wave 1 can finish faster if extraction tasks complete clean. -1. [ ] **Multi-line PDF edge dedup — fix exit-condition trap, not "add an extra check".** +1. [x] **Multi-line PDF edge dedup — fix exit-condition trap, not "add an extra check".** The current `_strip_repeating_page_lines` in [`forgelm/ingestion.py`](../../forgelm/ingestion.py) is already iterative — a `while True:` loop is supposed to peel one edge line at a time and re-check on each pass. The audit (§1.1) traced the observed failure to the loop's **exit condition**: pass 1 only inspects `lines[0]` and `lines[-1]`, so when the outermost line varies per page (a per-section title on top, a page number on the bottom) the loop strips zero lines on pass 1 and breaks *before* ever reaching the constant line that sits one row deeper. Future implementers reading just the docstring will assume iterative-peel is enough — the comment block above the loop must spell out *why iterative-peel alone fails on variable-outer-line corpora*. Concrete change: replace the pass-1-only `_repeating_edge_lines` with a top-N / bottom-N window check (N = 3 default, exposed as `_PDF_EDGE_WINDOW`) so a line recurring at **any** position within the top-3 / bottom-3 rows across ≥ 70 % of pages gets stripped in a single pass. Inline comment (1–2 sentences) explains the iteration-trap rationale so this regression cannot return silently. Run a second dedup pass after paragraph packing to catch headers that the chunker re-glued mid-block. Acceptance: on the synthetic Turkish-formal-publication regression fixture, the `""` pollution drops from ≥ 70 % of chunks to ≤ 5 %. -2. [ ] **Unicode / script sanity layer.** +2. [x] **Unicode / script sanity layer.** New helper `_check_script_sanity()` runs after every per-file extraction. For each format-detected language (uses existing `langdetect` budget — already an `[ingestion]` extra dependency through `langdetect`), computes the ratio of code-points outside the language's expected Unicode blocks. Above a configurable threshold, logs a `WARNING` naming the file and a guess at the cause (font fallback, mojibake, mixed-script noise). Threshold and language hint exposed as `--script-sanity-threshold` and `--language-hint` CLI flags. **Default-threshold calibration is fixture-driven** — the audit measured ≈ 7 % corrupt-char ratio on the pilot's front matter and 0 % on clean body chunks, so the placeholder `0.5 %` value in this draft is *intentionally aggressive* and will be re-calibrated against the regression fixture corpus (Task 10) before merge. Final default is set by a precision / recall sweep on the fixture matrix (target: ≥ 95 % recall on font-corrupt fixtures, ≤ 5 % false-positive rate on clean mixed-script fixtures); likely range `1.0 – 2.0 %`. Acceptance: the front-matter chunks and the custom-glyph-bullet section of the synthetic Turkish-formal-publication fixture each emit one targeted warning *with the calibrated default*, and at least one legitimate mixed-script fixture (e.g. a Turkish corpus with embedded English code samples) emits zero warnings. -3. [ ] **Turkish pypdf glyph normalisation table.** +3. [x] **Turkish pypdf glyph normalisation table.** New module `forgelm/_pypdf_normalise.py` ships a static `dict` mapping known-pypdf-fallback glyphs to their correct Turkish characters (`ø → İ`, `Õ → ı`, `ú → ş`, `÷ → ğ`, `ö → Ğ` *(with trailing space)*, `࡟ → •` etc., derived from the audit's measured artefacts). Applied at chunk-write time by default, opt-out via `--no-normalise-unicode`. Module is **language-specific** but pluggable: a `--normalise-profile turkish|none` flag selects the table, leaving room for `arabic` / `cjk` profiles in later phases. Acceptance: (a) re-ingesting the synthetic Turkish-formal-publication fixture produces zero `ø Õ ú ÷ ࡟` artefacts in the JSONL; (b) `forgelm doctor` reports a `pypdf_normalise.turkish: pass` diagnostic row so an operator can verify the table loaded correctly without having to run a test ingest first. -4. [ ] **Ingest-time quality pre-signal.** +4. [x] **Ingest-time quality pre-signal.** At the end of `ingest_path()`, run a lightweight 3-check sanity sweep (alpha-ratio per chunk, weird-char-ratio per chunk, repeated-line-ratio per chunk — *cheap* versions of the audit's full quality filter, not the expensive ones). If any chunk falls below the warn threshold, print a single end-of-summary line: ``` @@ -40,19 +40,19 @@ These splits are estimates, not commitments — task 1's multi-line dedup may su The threshold and disable-flag (`--no-quality-presignal`) live next to the existing `--output-format` flag. The summary line is the smallest possible nudge — full diagnostics stay in the audit step. -5. [ ] **Audit `--quality-filter` default-on.** +5. [x] **Audit `--quality-filter` default-on.** In [`forgelm/cli/_parser.py`](../../forgelm/cli/_parser.py) the audit sub-parser flips `--quality-filter` from opt-in to default-on; add `--no-quality-filter` for parity. Update [`docs/guides/data_audit.md`](../guides/data_audit.md) + TR mirror. Backward-compat: existing scripted invocations continue to work (no behaviour change when the flag is explicitly passed). -6. [ ] **DOCX explicit header / footer extraction.** +6. [x] **DOCX explicit header / footer extraction.** Extend `_extract_docx` in [`forgelm/ingestion.py`](../../forgelm/ingestion.py) to read `doc.sections[i].header` and `.footer` once per document, collect into a set, and subtract those exact lines from the body extraction. Word DOCX declares these explicitly — no statistical guess needed. Acceptance: a fixture DOCX with a 3-line header repeating across 10 pages produces JSONL with zero header lines. -7. [ ] **EPUB spine order + nav / cover / copyright skip.** +7. [x] **EPUB spine order + nav / cover / copyright skip.** Modify `_extract_epub` in [`forgelm/ingestion.py`](../../forgelm/ingestion.py) to iterate `book.spine` (reading order) instead of `book.get_items()` (file order). Maintain a configurable skip-list of item names / `epub:type` values (`nav`, `cover`, `copyright`, `colophon`, `titlepage`, `frontmatter`) — defaulted to skip, opt-out via `--epub-no-skip-frontmatter`. Acceptance: a fixture EPUB with explicit cover / TOC / copyright pages emits JSONL containing only the body chapters in reading order. -8. [ ] **TXT / MD BOM strip + frontmatter detection.** +8. [x] **TXT / MD BOM strip + frontmatter detection.** Extend `_extract_text` in [`forgelm/ingestion.py`](../../forgelm/ingestion.py) to strip the UTF-8 BOM (``) if present at position 0. For `.md` files only, detect and strip YAML frontmatter (`^---\n...\n---\n` at file start) by default; opt-out via `--keep-md-frontmatter`. CRLF → LF normalisation already happens on output via `newline="\n"`; verify no regression. Acceptance: a fixture TXT with a UTF-8 BOM and a fixture MD with YAML frontmatter produce clean chunks. -9. [ ] **Notebook UX alignment — surface the quality signal in the playground.** +9. [x] **Notebook UX alignment — surface the quality signal in the playground.** `notebooks/ingestion_playground.ipynb` is the canonical "first 5 minutes with ForgeLM ingestion" surface and currently terminates in a misleading "looks clean" UX even when 161 / 188 chunks are quality-flagged (the 2026-05-11 second-pass evidence). After Task 5 ships `--quality-filter` as default-on, the notebook needs four small updates so the new signal reaches Colab users without the operator having to know to look for it: - **Cell 5 config block** — expose `CHUNK_TOKENS = None` and `TOKENIZER = None` (token-aware mode) alongside the existing `CHUNK_SIZE` knob. Comment block explains: "set both to switch from char-based to token-based chunking; `TOKENIZER` is required when `CHUNK_TOKENS` is set." Addresses the audit §3.6 finding that token-aware mode forced operators to drop to a custom command. - **Cell 8 markdown** — extend the "what audit gives you" list with the quality-filter checks (alpha-ratio, mean word length, end-of-line punctuation, repeated-line ratio, short-paragraph ratio). Honest framing: heuristic, conservative, surfaces extraction-layer noise. @@ -60,34 +60,34 @@ These splits are estimates, not commitments — task 1's multi-line dedup may su - **Cell 10 pretty-print** — surface `quality_summary` alongside the existing PII / secrets / near-duplicate / language blocks. Show `samples_flagged / samples_evaluated`, `overall_quality_score`, and the `by_check` breakdown when any check fires. Acceptance: re-running the playground notebook against the synthetic Turkish-formal-publication regression fixture surfaces a non-silent quality signal before the operator clicks "download ZIP", AND operators can flip the notebook into token-aware mode without leaving the markdown surface. -10. [ ] **Wave 1 regression fixture matrix.** +10. [x] **Wave 1 regression fixture matrix.** Commit the 7 fixtures listed in the audit's section 7 under `tests/fixtures/ingestion/`. **All fixtures are synthesised inside the repository under the project's permissive licence**; no external copyright is incurred. The `tr_textbook_4pages.pdf` fixture is a four-page Turkish-formal-publication PDF **constructed from scratch inside the repository** with a deterministic regeneration script alongside it. By deliberate construction it exhibits each of the six failure modes documented in audit §1: a multi-line running header (variable section title + constant publication-identifier line), Type-1-font Turkish glyph fallback on its first two pages, custom-bullet `U+085F` in one chapter, a recurring institutional-style URL pattern with varying numeric query strings, ToC-style underscore leaders on its third / fourth pages, and inline page numbers. Each fixture has a golden JSONL output alongside; new test module `tests/test_ingestion_reliability.py` runs `ingest_path()` against every fixture and byte-compares to the golden file. The `tr_textbook_4pages.pdf` fixture ships with **two** golden JSONLs — `--chunk-size 4000` and `--chunk-size 1024` — to lock in both the running-header-heavy and the front-matter-isolated failure modes per the audit's section 1.7 cross-config replay finding. The `mixed_directory/` fixture's acceptance is explicit: `forgelm ingest mixed_directory/ --recursive --output golden.jsonl` produces a JSONL whose chunks appear in **lexicographic file-name order** across all 5 formats (PDF, DOCX, EPUB, TXT, MD), with format-tagged provenance derivable from `notes_structured.format_counts`. Determinism guarantees from the audit's section 5 make byte-compare viable; if pypdf's per-page extraction shifts under a version bump, the test fails loudly and the fixture is regenerated explicitly via the committed regeneration script. #### Wave 2 — strong nice-to-have (v0.6.x, ~2 weeks) -11. [ ] **`--strip-pattern REGEX` flag — operator-controlled regex with ReDoS guard.** +1. [x] **`--strip-pattern REGEX` flag — operator-controlled regex with ReDoS guard.** Operator escape hatch for known-boilerplate that the dedup heuristic misses. Multiple patterns allowed (`--strip-pattern A --strip-pattern B`). Applied after extraction, before chunking. **Operator-supplied regexes must pass the regex-hardening guardrails defined in [`docs/standards/regex.md`](../standards/regex.md)** — the same 8-rule discipline (no unbounded greedy quantifiers in sequence, no nested groups with overlapping ranges, bounded repetition) that ForgeLM's own internal regexes obey. Implementation choice: validate the user-supplied pattern at CLI-parse time via the same SonarCloud python:S5852-style structural check ForgeLM applies internally, **and** wrap the match call in a per-pattern timeout (`re.match` with `signal.alarm` or `regex.match(... timeout=N)` if the `regex` library is added as a Wave 2 dependency — operator can opt out via `--strip-pattern-no-timeout` if they're certain). Documented in the operator guide with the canonical examples (running headers with variable content, journal DOI lines, watermarks). Acceptance: a pathological `(a+)+b`-class pattern aborts with a clear `EXIT_CONFIG_ERROR (1)` and a message naming the offending pattern, instead of hanging the ingestion process. -12. [ ] **`--page-range` PDF flag.** +2. [x] **`--page-range` PDF flag.** Simple `--page-range 12-193` filtering for PDF files. Skips front-matter / back-matter manually when the heuristic in task 13 mis-fires. Validated as `min ≤ max`, `min ≥ 1`, `max ≤ pdf_page_count`. Validation failure **exits with `EXIT_CONFIG_ERROR (1)`** via the CLI dispatcher (matching the existing `_run_ingest_cmd` pattern in [`forgelm/cli/subcommands/_ingest.py`](../../forgelm/cli/subcommands/_ingest.py)) and prints a clear message naming the offending range and the actual page count. Library-level `ingest_path()` raises `ValueError` as before — the exit-code translation lives at the CLI boundary per the [error-handling standard](../standards/error-handling.md). -13. [ ] **Front-matter heuristic detection.** +3. [x] **Front-matter heuristic detection.** Heuristic that looks at the first N pages of a PDF (N=12 default): if a page has alpha-ratio < 0.45 *and* underscore-ratio > 0.10 *and* contains ≥ 5 page-number-pattern matches, the page is tagged as "front-matter candidate" and dropped, with a `WARNING` listing the dropped page indices and the per-page metrics. Opt-out via `--keep-frontmatter`. Symmetric pass on the last N pages catches back-matter (bibliographies, indexes); same threshold. -14. [ ] **URL / QR pattern auto-strip option.** +4. [x] **URL / QR pattern auto-strip option.** `--strip-urls` flag: detects URL patterns (http(s)://, www.) per chunk and either replaces with `[URL]` or strips entirely (`--strip-urls=mask|strip|keep`, default `keep`). Useful for exercise-reference QR-noise and academic-paper DOI-noise. Existing PII / secrets mask pipeline integration: applies *after* secrets mask, *before* PII mask, preserving the mask-order discipline. **Interaction with `--all-mask`:** the `--all-mask` shorthand (Phase 12.5) is explicitly *not* extended to include `--strip-urls`. Rationale: PII / secrets are GDPR-grade compliance redactions whose accidental over-application is recoverable (the original text is still in the source PDF); URL stripping is a content-shape decision that operators may want to keep off while still demanding the privacy redactions. The two flag families stay independent and the operator opts into URL handling explicitly. Documented at the CLI help text for both `--all-mask` and `--strip-urls`. -15. [ ] **Multi-column PDF detection.** +5. [x] **Multi-column PDF detection.** Heuristic that estimates column count from pypdf's text-position metadata: per page, cluster x-positions of extracted spans; if two clusters dominate (separation > 30 % of page width), tag the document as 2-column. No fix attempt — just a `WARNING` at run start: `[WARN] Detected 2-column layout — reading order may be scrambled. Consider --strategy sliding with larger chunk-size, or pre-process with a layout-aware tool.` Better-than-nothing signal; cheaper than a layout-aware extractor. #### Wave 3 — deferred (post-Phase 15) -16. [ ] OCR detection + automated `ocrmypdf` recommendation -17. [ ] Document profile / preset system (`--profile academic`, `--profile legal`, `--profile textbook`) -18. [ ] PDF table extractor (camelot-py / pdfplumber integration as optional `[ingestion-tables]` extra) -19. [ ] Footnote / endnote separation (DOCX + PDF) -20. [ ] Language-aware paragraph segmentation (CJK / Arabic / RTL scripts) -21. [ ] RTL script extraction-order normalisation +1. [ ] OCR detection + automated `ocrmypdf` recommendation +2. [ ] Document profile / preset system (`--profile academic`, `--profile legal`, `--profile textbook`) +3. [ ] PDF table extractor (camelot-py / pdfplumber integration as optional `[ingestion-tables]` extra) +4. [ ] Footnote / endnote separation (DOCX + PDF) +5. [ ] Language-aware paragraph segmentation (CJK / Arabic / RTL scripts) +6. [ ] RTL script extraction-order normalisation These items go on the [risks-and-decisions](risks-and-decisions.md) backlog with the Phase 15 audit as their provenance reference, to be re-prioritised after Wave 1 lands and operator feedback arrives. diff --git a/docs/usermanuals/en/data/audit.md b/docs/usermanuals/en/data/audit.md index e955289d..ddc01495 100644 --- a/docs/usermanuals/en/data/audit.md +++ b/docs/usermanuals/en/data/audit.md @@ -100,7 +100,7 @@ Authoritative source: `forgelm/cli/_parser.py::_add_audit_subcommand`. | `--near-dup-threshold N` | Hamming-distance cutoff for simhash near-duplicate detection (default 3 ≈ 95% similarity). Ignored under `--dedup-method=minhash`. | | `--dedup-method {simhash,minhash}` | Near-duplicate algorithm. `simhash` is the default (Phase 11.5 path); `minhash` opts into LSH-banded MinHash via the optional `forgelm[ingestion-scale]` extra (datasketch). | | `--jaccard-threshold X` | Jaccard threshold for `--dedup-method=minhash` (default 0.85). Ignored when `--dedup-method=simhash`. | -| `--quality-filter` | Run heuristic quality checks (mean word length, alphabetic-character ratio, end-of-line punctuation ratio, repeated-line ratio, short-paragraph ratio). Adds `quality_summary` to the report. See [Quality Filter](#/data/quality-filter). | +| `--quality-filter` / `--no-quality-filter` | Heuristic quality checks (mean word length, alphabetic-character ratio, end-of-line punctuation ratio, repeated-line ratio, short-paragraph ratio). **Default ON from v0.6.0** (Phase 15 Task 5); pre-v0.6.0 the flag was opt-in. Pass `--no-quality-filter` to skip. Populates `quality_summary` in the report. See [Quality Filter](#/data/quality-filter). | | `--croissant` | Emit a [Google Croissant 1.0](https://mlcommons.org/croissant/) dataset card under the report's `croissant` key. See [Croissant 1.0 Dataset Card](#/data/croissant-card). | | `--pii-ml` | Layer Presidio's ML-NER PII detection on top of the regex detector. Requires the optional `forgelm[ingestion-pii-ml]` extra **and** a spaCy NER model (e.g. `python -m spacy download en_core_web_lg`). See [ML-NER PII (Presidio)](#/data/pii-ml). | | `--pii-ml-language LANG` | spaCy NLP language code for `--pii-ml` (default `en`). Set to e.g. `tr` on a Turkish corpus AND make sure the matching spaCy model is installed. | diff --git a/docs/usermanuals/en/data/ingestion.md b/docs/usermanuals/en/data/ingestion.md index d0a69437..527673f2 100644 --- a/docs/usermanuals/en/data/ingestion.md +++ b/docs/usermanuals/en/data/ingestion.md @@ -94,6 +94,23 @@ that appear most often: | `--secrets-mask` | Redact AWS keys, GitHub PATs, JWTs, etc. See [Secrets Scrubbing](#/data/secrets). | | `--all-mask` | Shorthand for `--secrets-mask --pii-mask` together. | +### Phase 15 (v0.6.0) additions + +| Flag | Description | +|---|---| +| `--language-hint LANG` | Enable per-file Unicode-block sanity check (`tr` / `en` / `de` / `fr` / `es` / `it` / `pt`). Fires a WARNING when the out-of-script char ratio exceeds the calibrated 1.5 % threshold. No-op when unset. | +| `--script-sanity-threshold X` | Override the 1.5 % default sanity-threshold (range `[0.0, 1.0]`). | +| `--normalise-profile {turkish,none}` | Apply a language-specific glyph normalisation table to extracted text. Auto-derived to `turkish` when `--language-hint tr` is set; `none` otherwise. Explicit value wins. | +| `--no-normalise-unicode` | Shortcut for `--normalise-profile none`. | +| `--no-quality-presignal` | Skip the end-of-run quality pre-signal (alpha / weird-char / repeated-line cheap checks). Default ON. | +| `--epub-no-skip-frontmatter` | Keep EPUB nav / cover / copyright / colophon / titlepage / frontmatter items in the JSONL. Default skips them. | +| `--keep-md-frontmatter` | Retain `---\n…\n---\n` YAML frontmatter at the start of Markdown files. Default strips. | +| `--strip-pattern REGEX` | Operator-controlled regex stripping (repeatable). Patterns are ReDoS-validated up-front (nested unbounded quantifiers + `.*?` + DOTALL back-ref shapes rejected) and run under a 5-second per-pattern SIGALRM budget on POSIX. | +| `--strip-pattern-no-timeout` | Disable the per-pattern SIGALRM timeout. | +| `--page-range START-END` | Restrict PDF extraction to a contiguous 1-indexed page range. Validation failures abort with `EXIT_CONFIG_ERROR (1)`. | +| `--keep-frontmatter` | Opt out of the default-ON PDF front-matter / back-matter heuristic (alpha < 0.45 + underscore > 0.10 + ≥ 5 page-number matches → drop up to 12 leading + trailing pages). | +| `--strip-urls {keep,mask,strip}` | URL handling for inline URLs: `keep` (default), `mask` (`[URL]` placeholder), `strip` (delete). Independent of `--all-mask`. | + ## Common pitfalls :::warn diff --git a/docs/usermanuals/en/data/quality-filter.md b/docs/usermanuals/en/data/quality-filter.md index 78c57b26..d3855e89 100644 --- a/docs/usermanuals/en/data/quality-filter.md +++ b/docs/usermanuals/en/data/quality-filter.md @@ -42,8 +42,19 @@ audit: ``` ```shell +# v0.6.0+: quality-filter is DEFAULT-ON; the explicit flag is harmless. +# Heuristics populate quality_summary in data_audit_report.json but do +# NOT drop rows or write a cleaned JSONL — that only happens when the +# `audit.quality_filter.drop_flagged: true` + `write_clean_output: PATH` +# YAML keys above are set in a config-driven run. +$ forgelm audit data/ingested.jsonl +✓ wrote audit/data_audit_report.json (quality_summary: 45 / 12,400 flagged) + +# Pre-v0.6.0 (or to be explicit), pass the flag: $ forgelm audit data/ingested.jsonl --quality-filter -✓ dropped 45 rows; wrote data/clean.jsonl (12,355 rows) + +# Opt out of the new default if your CI gates depend on opt-in semantics: +$ forgelm audit data/ingested.jsonl --no-quality-filter ``` ## Tuning thresholds diff --git a/docs/usermanuals/en/reference/cli.md b/docs/usermanuals/en/reference/cli.md index 4bf33a31..1080daff 100644 --- a/docs/usermanuals/en/reference/cli.md +++ b/docs/usermanuals/en/reference/cli.md @@ -108,10 +108,25 @@ $ forgelm ingest INPUT_PATH \ [--chunk-size N] [--overlap N] \ [--chunk-tokens N] [--overlap-tokens N] [--tokenizer MODEL_NAME] \ [--pii-mask] [--secrets-mask] [--all-mask] \ - [--pii-ml-language LANG] + [--language-hint LANG] [--script-sanity-threshold X] \ + [--normalise-profile {turkish,none} | --no-normalise-unicode] \ + [--no-quality-presignal] \ + [--epub-no-skip-frontmatter] [--keep-md-frontmatter] \ + [--strip-pattern REGEX ...] [--strip-pattern-no-timeout] \ + [--page-range START-END] [--keep-frontmatter] \ + [--strip-urls {keep,mask,strip}] \ + [--output-format {text,json}] ``` -See [Document Ingestion](#/data/ingestion). +Pass `--output-format json` to get the machine-readable envelope +described in [JSON Output Contract](#/reference/json-output) — useful +for CI gates that branch on chunk count / files-processed without +parsing the text summary. Phase 15 (v0.6.0) added the +`--language-hint`, `--script-sanity-threshold`, `--normalise-profile`, +`--no-normalise-unicode`, `--no-quality-presignal`, +`--epub-no-skip-frontmatter`, `--keep-md-frontmatter`, `--strip-pattern`, +`--strip-pattern-no-timeout`, `--page-range`, `--keep-frontmatter`, and +`--strip-urls` flags. See [Document Ingestion](#/data/ingestion). ## Chat: `forgelm chat` diff --git a/docs/usermanuals/en/reference/json-output.md b/docs/usermanuals/en/reference/json-output.md index eb29f5e3..98c19a21 100644 --- a/docs/usermanuals/en/reference/json-output.md +++ b/docs/usermanuals/en/reference/json-output.md @@ -147,7 +147,7 @@ Pre-train data audit. Full report; key fields shown. "secrets_summary": {"total_findings": 0, "by_kind": {}}, "cross_split_overlap": {"pairs": {}}, "leakage": {"...": "..."}, - "quality_filter": null, + "quality_summary": {"samples_evaluated": 100, "samples_flagged": 3, "by_check": {"...": "..."}}, "near_duplicates": {"...": "..."}, "languages_top3": [{"code": "en", "count": 87}], "generated_at": "2026-05-04T12:34:56+00:00", @@ -155,8 +155,55 @@ Pre-train data audit. Full report; key fields shown. } ``` +> **Phase 15 (v0.6.0) note:** `quality_summary` is populated by default from v0.6.0+ because the audit's `--quality-filter` flag was flipped to default-on. Pass `--no-quality-filter` to skip; the field is `{}` in that case. + The full schema is in [`docs/guides/data_audit.md`](#/data/audit). Pin against `report_path` (where the on-disk JSON lives) + `success` for CI gates. +## `forgelm ingest` + +Document ingestion stdout envelope. Phase 15 added additive fields under +`notes_structured`; the pre-Phase-15 keys are unchanged. + +```json +{ + "success": true, + "output_path": "data/curated.jsonl", + "chunk_count": 245, + "files_processed": 12, + "files_skipped": 0, + "total_chars": 824513, + "format_counts": {".pdf": 4, ".docx": 6, ".md": 2}, + "pii_redaction_counts": {"email": 3, "phone": 1}, + "secrets_redaction_counts": {}, + "pdf_header_footer_lines_stripped": 18, + "pdf_paragraph_packed_lines_stripped": 4, + "script_sanity_triggered": 1, + "strip_pattern_substitutions": 12, + "urls_handled": 7, + "frontmatter_pages_dropped": 3, + "notes": ["..."], + "notes_structured": { + "files_processed": 12, + "files_skipped": 0, + "chunk_count": 245, + "total_chars": 824513, + "strategy": "paragraph", + "format_counts": {".pdf": 4, ".docx": 6, ".md": 2}, + "pdf_header_footer_lines_stripped": 18, + "pdf_paragraph_packed_lines_stripped": 4, + "script_sanity_summary": {"files_checked": 12, "files_triggered": 1, "...": "..."}, + "frontmatter_pages_dropped": [0, 1, 11], + "strip_pattern_substitutions": 12, + "urls_handled": 7, + "quality_presignal": {"samples_evaluated": 245, "samples_flagged": 17, "by_check": {"alpha_ratio": 5, "weird_chars": 1, "repeated_lines": 11}} + } +} +``` + +The Phase 15 additive fields are documented at +[Document Ingestion](#/data/ingestion). Pre-Phase-15 consumers reading +`output_path` / `chunk_count` / `notes` keep working unchanged. + ## `forgelm verify-audit` Audit-log chain integrity check. diff --git a/docs/usermanuals/tr/data/audit.md b/docs/usermanuals/tr/data/audit.md index 44992595..d3cd9501 100644 --- a/docs/usermanuals/tr/data/audit.md +++ b/docs/usermanuals/tr/data/audit.md @@ -100,7 +100,7 @@ Yetkili kaynak: `forgelm/cli/_parser.py::_add_audit_subcommand`. | `--near-dup-threshold N` | simhash near-duplicate detection için Hamming-mesafesi eşiği (varsayılan 3 ≈ %95 benzerlik). `--dedup-method=minhash` altında yok sayılır. | | `--dedup-method {simhash,minhash}` | Near-duplicate algoritması. Varsayılan `simhash` (Phase 11.5 yolu); `minhash` opsiyonel `forgelm[ingestion-scale]` extra'sı (datasketch) üzerinden LSH-banded MinHash'e geçer. | | `--jaccard-threshold X` | `--dedup-method=minhash` için Jaccard eşiği (varsayılan 0.85). `--dedup-method=simhash` ile yok sayılır. | -| `--quality-filter` | Heuristik kalite kontrolleri (mean word length, alphabetic-character ratio, end-of-line punctuation ratio, repeated-line ratio, short-paragraph ratio). Rapora `quality_summary` ekler. Bkz. [Kalite Filtresi](#/data/quality-filter). | +| `--quality-filter` / `--no-quality-filter` | Heuristik kalite kontrolleri (mean word length, alphabetic-character ratio, end-of-line punctuation ratio, repeated-line ratio, short-paragraph ratio). **v0.6.0'dan itibaren default-AÇIK** (Faz 15 Görev 5); öncesinde opt-in idi. Atlamak için `--no-quality-filter` geçirin. Rapora `quality_summary` ekler. Bkz. [Kalite Filtresi](#/data/quality-filter). | | `--croissant` | Raporun `croissant` anahtarı altına bir [Google Croissant 1.0](https://mlcommons.org/croissant/) dataset card emit eder. Bkz. [Croissant 1.0 Dataset Kartı](#/data/croissant-card). | | `--pii-ml` | Regex detector'ın üzerine Presidio'nun ML-NER PII tespitini katmanlar. Opsiyonel `forgelm[ingestion-pii-ml]` extra'sını **ve** bir spaCy NER modelini (örn. `python -m spacy download en_core_web_lg`) gerektirir. Bkz. [ML-NER PII (Presidio)](#/data/pii-ml). | | `--pii-ml-language LANG` | `--pii-ml` için spaCy NLP dili kodu (varsayılan `en`). Türkçe corpus için örn. `tr` set edin VE eşleşen spaCy modelinin kurulu olduğundan emin olun. | diff --git a/docs/usermanuals/tr/data/ingestion.md b/docs/usermanuals/tr/data/ingestion.md index 092be686..f7a15d1b 100644 --- a/docs/usermanuals/tr/data/ingestion.md +++ b/docs/usermanuals/tr/data/ingestion.md @@ -93,6 +93,23 @@ Yetkili liste için `forgelm ingest --help`. En sık görülenler: | `--secrets-mask` | AWS anahtarları, GitHub PAT'ler, JWT'leri vb. redakte et. Bkz. [Sırlar](#/data/secrets). | | `--all-mask` | `--secrets-mask --pii-mask`'in birleşik kısayolu. | +### Faz 15 (v0.6.0) eklemeleri + +| Bayrak | Açıklama | +|---|---| +| `--language-hint LANG` | Her dosya ekstraksiyonundan sonra Unicode-block sanity kontrolü çalıştırır (`tr` / `en` / `de` / `fr` / `es` / `it` / `pt`). Out-of-script karakter oranı kalibre edilmiş %1.5 eşiğini aştığında WARNING tetikler. Kapalıyken no-op. | +| `--script-sanity-threshold X` | Varsayılan %1.5 sanity eşiğini geçersiz kıl (aralık `[0.0, 1.0]`). | +| `--normalise-profile {turkish,none}` | Çıkarılan metne dil-spesifik glyph normalizasyon tablosu uygular. `--language-hint tr` set edilince otomatik `turkish` türetilir; aksi halde `none`. Açık değer kazanır. | +| `--no-normalise-unicode` | `--normalise-profile none` için kısayol. | +| `--no-quality-presignal` | Çalışma sonu kalite ön-sinyalini (alpha / weird-char / repeated-line ucuz kontrolleri) atla. Default AÇIK. | +| `--epub-no-skip-frontmatter` | EPUB nav / cover / copyright / colophon / titlepage / frontmatter öğelerini JSONL'da tut. Default bunları atlar. | +| `--keep-md-frontmatter` | Markdown dosyalarının başındaki `---\n…\n---\n` YAML frontmatter'ı koru. Default strip eder. | +| `--strip-pattern REGEX` | Operatör kontrolünde regex stripping (tekrarlanabilir). Pattern'lar önceden ReDoS-doğrulamadan geçer (iç içe sınırsız quantifier + DOTALL altında `.*?` + back-ref şekilleri reddedilir) ve POSIX'te pattern başına 5-saniyelik SIGALRM bütçesinde çalışır. | +| `--strip-pattern-no-timeout` | Pattern başına SIGALRM timeout'u devre dışı bırak. | +| `--page-range START-END` | PDF ekstraksiyonunu sürekli 1-indeksli sayfa aralığına sınırla. Doğrulama hatası `EXIT_CONFIG_ERROR (1)` ile durur. | +| `--keep-frontmatter` | Default-AÇIK PDF front-matter / back-matter heuristic'inden opt-out (alpha < 0.45 + underscore > 0.10 + ≥ 5 sayfa numarası → ilk + son 12 sayfaya kadar düşür). | +| `--strip-urls {keep,mask,strip}` | Inline URL davranışı: `keep` (varsayılan), `mask` (`[URL]` placeholder), `strip` (sil). `--all-mask`'ten bağımsız. | + ## Sık hatalar :::warn diff --git a/docs/usermanuals/tr/data/quality-filter.md b/docs/usermanuals/tr/data/quality-filter.md index 14a3f71a..75f2406b 100644 --- a/docs/usermanuals/tr/data/quality-filter.md +++ b/docs/usermanuals/tr/data/quality-filter.md @@ -42,8 +42,19 @@ audit: ``` ```shell +# v0.6.0+: quality-filter is DEFAULT-ON; the explicit flag is harmless. +# Heuristics populate quality_summary in data_audit_report.json but do +# NOT drop rows or write a cleaned JSONL — that only happens when the +# `audit.quality_filter.drop_flagged: true` + `write_clean_output: PATH` +# YAML keys above are set in a config-driven run. +$ forgelm audit data/ingested.jsonl +✓ wrote audit/data_audit_report.json (quality_summary: 45 / 12,400 flagged) + +# Pre-v0.6.0 (or to be explicit), pass the flag: $ forgelm audit data/ingested.jsonl --quality-filter -✓ 45 satır düşürüldü; data/clean.jsonl yazıldı (12,355 satır) + +# Opt out of the new default if your CI gates depend on opt-in semantics: +$ forgelm audit data/ingested.jsonl --no-quality-filter ``` ## Eşik ayarlama diff --git a/docs/usermanuals/tr/reference/cli.md b/docs/usermanuals/tr/reference/cli.md index d485030d..01ba0423 100644 --- a/docs/usermanuals/tr/reference/cli.md +++ b/docs/usermanuals/tr/reference/cli.md @@ -108,10 +108,25 @@ $ forgelm ingest INPUT_PATH \ [--chunk-size N] [--overlap N] \ [--chunk-tokens N] [--overlap-tokens N] [--tokenizer MODEL_NAME] \ [--pii-mask] [--secrets-mask] [--all-mask] \ - [--pii-ml-language LANG] + [--language-hint LANG] [--script-sanity-threshold X] \ + [--normalise-profile {turkish,none} | --no-normalise-unicode] \ + [--no-quality-presignal] \ + [--epub-no-skip-frontmatter] [--keep-md-frontmatter] \ + [--strip-pattern REGEX ...] [--strip-pattern-no-timeout] \ + [--page-range START-END] [--keep-frontmatter] \ + [--strip-urls {keep,mask,strip}] \ + [--output-format {text,json}] ``` -Bkz. [Doküman Ingestion](#/data/ingestion). +`--output-format json` ile [JSON Output Contract](#/reference/json-output) +sayfasındaki makine-okunabilir envelope alınır — chunk count / files- +processed üzerinden branch eden CI gate'ler için kullanışlı, metin +özetini parse etmeye gerek yok. Faz 15 (v0.6.0), `--language-hint`, +`--script-sanity-threshold`, `--normalise-profile`, `--no-normalise-unicode`, +`--no-quality-presignal`, `--epub-no-skip-frontmatter`, `--keep-md-frontmatter`, +`--strip-pattern`, `--strip-pattern-no-timeout`, `--page-range`, +`--keep-frontmatter` ve `--strip-urls` bayraklarını ekledi. Bkz. +[Doküman Ingestion](#/data/ingestion). ## Chat: `forgelm chat` diff --git a/docs/usermanuals/tr/reference/json-output.md b/docs/usermanuals/tr/reference/json-output.md index b952b37c..26b6bae8 100644 --- a/docs/usermanuals/tr/reference/json-output.md +++ b/docs/usermanuals/tr/reference/json-output.md @@ -147,7 +147,7 @@ Eğitim öncesi veri denetimi. Tam rapor; ana alanlar gösterildi. "secrets_summary": {"total_findings": 0, "by_kind": {}}, "cross_split_overlap": {"pairs": {}}, "leakage": {"...": "..."}, - "quality_filter": null, + "quality_summary": {"samples_evaluated": 100, "samples_flagged": 3, "by_check": {"...": "..."}}, "near_duplicates": {"...": "..."}, "languages_top3": [{"code": "en", "count": 87}], "generated_at": "2026-05-04T12:34:56+00:00", @@ -155,8 +155,58 @@ Eğitim öncesi veri denetimi. Tam rapor; ana alanlar gösterildi. } ``` +> **Faz 15 (v0.6.0) notu:** `quality_summary`, v0.6.0'dan itibaren default +> olarak doldurulur çünkü audit'in `--quality-filter` bayrağı default-AÇIK'a +> çevrildi. Atlamak için `--no-quality-filter` geçirin; o durumda alan `{}` +> olur. + Tam şema [`docs/guides/data_audit.md`](#/data/audit)'dedir. CI gate'leri için `report_path` (on-disk JSON'un yeri) + `success`'e karşı pin'leyin. +## `forgelm ingest` + +Doküman ingestion stdout envelope'ı. Faz 15, `notes_structured` altına +additive alanlar ekledi; Faz 15 öncesi anahtarlar değişmedi. + +```json +{ + "success": true, + "output_path": "data/curated.jsonl", + "chunk_count": 245, + "files_processed": 12, + "files_skipped": 0, + "total_chars": 824513, + "format_counts": {".pdf": 4, ".docx": 6, ".md": 2}, + "pii_redaction_counts": {"email": 3, "phone": 1}, + "secrets_redaction_counts": {}, + "pdf_header_footer_lines_stripped": 18, + "pdf_paragraph_packed_lines_stripped": 4, + "script_sanity_triggered": 1, + "strip_pattern_substitutions": 12, + "urls_handled": 7, + "frontmatter_pages_dropped": 3, + "notes": ["..."], + "notes_structured": { + "files_processed": 12, + "files_skipped": 0, + "chunk_count": 245, + "total_chars": 824513, + "strategy": "paragraph", + "format_counts": {".pdf": 4, ".docx": 6, ".md": 2}, + "pdf_header_footer_lines_stripped": 18, + "pdf_paragraph_packed_lines_stripped": 4, + "script_sanity_summary": {"files_checked": 12, "files_triggered": 1, "...": "..."}, + "frontmatter_pages_dropped": [0, 1, 11], + "strip_pattern_substitutions": 12, + "urls_handled": 7, + "quality_presignal": {"samples_evaluated": 245, "samples_flagged": 17, "by_check": {"alpha_ratio": 5, "weird_chars": 1, "repeated_lines": 11}} + } +} +``` + +Faz 15 additive alanları [Doküman Ingestion](#/data/ingestion)'da +belgelenmiştir. Faz 15 öncesi `output_path` / `chunk_count` / `notes` +okuyan tüketiciler değişmeden çalışmaya devam eder. + ## `forgelm verify-audit` Audit log chain bütünlüğü kontrolü. diff --git a/forgelm/_pypdf_normalise.py b/forgelm/_pypdf_normalise.py new file mode 100644 index 00000000..bfe5b2be --- /dev/null +++ b/forgelm/_pypdf_normalise.py @@ -0,0 +1,210 @@ +"""Phase 15 Task 3 — language-specific glyph-fallback normalisation for pypdf. + +Some PDFs use Type-1 PostScript fonts whose ``Encoding`` dictionary maps +non-ASCII characters via custom glyph names that pypdf's CMap fallback +cannot resolve. pypdf produces what it can — typically Latin-1 placeholder +glyphs that look nothing like the source character. The 2026-05-11 +ingestion-reliability audit measured this on a real-world Turkish-language +formal-publication PDF: Turkish characters surfaced as ``ø Õ ú ÷ ࡟`` etc., +with ~7 % corrupt-char ratio on the front-matter pages and a chapter-level +section using ``U+085F`` (Mandaic punctuation) for bullet points. + +This module ships a small, **language-specific** lookup table that maps +the measured corrupt glyphs back to their correct Turkish characters, +plus a profile dispatcher so future phases can drop in additional +profiles (Arabic, CJK) without re-shaping the API. + +Design rules: + +* **Pure data, no I/O.** The table is a module-level constant so it is + trivially testable and adds no import-time cost. +* **Single-pass replacement.** :func:`apply_profile` uses one ``str`` + scan per profile so even a megabyte-sized chunk normalises in linear + time. The previous draft used a regex-alternation; the dict approach + is simpler and avoids the cost of building a giant pattern. +* **Order matters.** A handful of the measured artefacts are + *multi-character* sequences (e.g. ``ö`` followed by a trailing space + for ``Ğ``). The applier handles those before the single-char passes + so a single-char substitution does not break the multi-char hit. +* **Opt-out per profile.** Callers pass ``profile="none"`` to disable + normalisation entirely; the function returns the input unchanged. + +The public surface is intentionally minimal — one function and one +``PROFILES`` map. The doctor diagnostic (Phase 15 task 3 acceptance) +loads :data:`PROFILES` and reports ``pypdf_normalise.: pass`` +so an operator can verify the table is intact without running an ingest. +""" + +from __future__ import annotations + +from typing import Dict, FrozenSet, Tuple + +# --------------------------------------------------------------------------- +# Turkish profile — derived from the 2026-05-11 audit measurements +# --------------------------------------------------------------------------- + +# Multi-character sequences are applied FIRST so a single-char substitution +# below cannot prematurely consume one of their leading characters. Each +# entry is ``(source_sequence, target_codepoint)``. The trailing space in +# ``"ö "`` (the audit-measured ``Ğ`` artefact) is intentional — pypdf emits +# the corrupt glyph with a stray space because the source font's advance +# width is over-allocated. Stripping the space would re-glue ``Ğ`` to the +# following character and silently break tokenisation. +_TURKISH_MULTI: Tuple[Tuple[str, str], ...] = ( + ("ö ", "Ğ"), # U+011E LATIN CAPITAL LETTER G WITH BREVE +) + +# Single-character substitutions. Kept as a plain ``dict`` so callers can +# inspect the size / membership in a single keyword (``in``). The values +# are bare Python str (not literal escapes) so a future reader can read +# what the table actually contains in their editor. +_TURKISH_SINGLE: Dict[str, str] = { + "ø": "İ", # U+0130 LATIN CAPITAL LETTER I WITH DOT ABOVE + "Õ": "ı", # U+0131 LATIN SMALL LETTER DOTLESS I + "ú": "ş", # U+015F LATIN SMALL LETTER S WITH CEDILLA + "÷": "ğ", # U+011F LATIN SMALL LETTER G WITH BREVE + "࡟": "•", # U+085F MANDAIC PUNCTUATION → U+2022 BULLET (custom bullet glyph) +} + +# Code points the profile is allowed to *emit*. Surface-exposed so the +# script-sanity check (Phase 15 task 2) can mask its output against the +# normaliser's expected vocabulary — i.e. a chunk that still carries a +# normalised Turkish char counts as "in-script" even though that char +# came from a glyph-fallback corruption. +_TURKISH_EMITTED: FrozenSet[str] = frozenset(list(_TURKISH_SINGLE.values()) + [target for _, target in _TURKISH_MULTI]) + + +# --------------------------------------------------------------------------- +# Public surface +# --------------------------------------------------------------------------- + + +PROFILES: Tuple[str, ...] = ("turkish", "none") +"""Supported normalisation profiles. + +``turkish`` is the only language-specific profile shipped today; future +work may add ``arabic`` / ``cjk`` once the failure-mode evidence is +collected. ``none`` is an explicit no-op so a CLI flag can disable the +behaviour without special-casing ``profile is None`` at every call site. +""" + +DEFAULT_PROFILE: str = "none" +"""Default profile applied at chunk-write time. + +Phase 15 round-1 review (C-2): the original default of ``turkish`` +silently rewrote legitimate non-Turkish letters (``Bjørk`` → ``BjİrkĞ``, +``Õrö`` collapse) because the normaliser fires regardless of the +operator's language hint. The fix is to default to ``none`` and let +the CLI dispatcher derive the right profile from ``--language-hint``: +``--language-hint tr`` auto-selects ``turkish`` unless +``--normalise-profile`` is set explicitly; every other hint stays on +``none``. Operators on a Turkish corpus continue to get the benefit +without having to know the flag's name; everyone else is safe by +default. +""" + + +def apply_profile(text: str, profile: str = DEFAULT_PROFILE) -> str: + """Apply the normalisation profile to ``text``. + + Returns the input unchanged when ``profile == "none"`` or when the + profile name is not recognised (the latter is **not** an error — a + forward-compatible caller might pass a profile name that older + ForgeLM releases do not know yet, and we silently fall through + rather than crash a long ingestion run). Operators who want strict + profile validation should check :data:`PROFILES` membership at the + CLI boundary first. + + The 1:1 substitutions preserve every other code point byte-for-byte — + the function does not strip whitespace, normalise line endings, or + apply any other transformation. This matters for the determinism + contract documented in + ``docs/reference/data_ingestion_architecture.md``: re-running + ingestion against the same source with the same profile must + produce a byte-identical JSONL. + """ + if profile == "none" or not text: + return text + if profile == "turkish": + return _apply_turkish(text) + # Unknown profile — silently no-op for forward compatibility. + return text + + +def _apply_turkish(text: str) -> str: + """Turkish-specific applier. Multi-char passes precede single-char ones.""" + out = text + for src, dst in _TURKISH_MULTI: + out = out.replace(src, dst) + # ``str.translate`` is the fastest way to apply a bag of single-character + # substitutions in CPython — it visits each source char exactly once and + # the lookup is constant-time. + out = out.translate(str.maketrans(_TURKISH_SINGLE)) + return out + + +def count_substitutions(text: str, profile: str = DEFAULT_PROFILE) -> int: + """Return the **exact** number of substitutions ``apply_profile`` would + perform on ``text``. + + Phase 15 round-3 review: the previous heuristic in ``ingestion.py`` + zipped the pre- and post-normalisation strings character-by-character + and added an absolute length delta. That over-counted when a + multi-char rule (``"ö " → "Ğ"``) shortened the text by one character + AND the surrounding chars happened to differ, AND it under-counted + when consecutive substitutions cancelled in the zip view. The exact + count is cheap to compute via ``text.count(src)`` for each rule + (multi-char rules counted first because their substitution consumes + one of the single-char triggers as a prefix — same precedence as + :func:`apply_profile`). + """ + if profile == "none" or not text: + return 0 + if profile != "turkish": + return 0 + total = 0 + remaining = text + for src, _dst in _TURKISH_MULTI: + hits = remaining.count(src) + if hits: + total += hits + remaining = remaining.replace(src, "") + for src in _TURKISH_SINGLE: + total += remaining.count(src) + return total + + +def profile_summary(profile: str) -> Dict[str, int]: + """Return ``{"multi": N, "single": M}`` for the named profile. + + Doctor uses this to render the diagnostic row without exposing the + raw tables (which would clutter the JSON envelope on every probe). + Returns ``{"multi": 0, "single": 0}`` for ``"none"`` or unknown + profile names so callers can rely on the schema being identical. + """ + if profile == "turkish": + return {"multi": len(_TURKISH_MULTI), "single": len(_TURKISH_SINGLE)} + return {"multi": 0, "single": 0} + + +def emitted_codepoints(profile: str) -> FrozenSet[str]: + """Return the set of code points the profile may emit. + + Used by the script-sanity check (Phase 15 task 2) so a chunk whose + only "unexpected" chars are the targets the normaliser produced is + not flagged as font-corrupt. Empty frozenset for ``"none"`` / + unknown profiles. + """ + if profile == "turkish": + return _TURKISH_EMITTED + return frozenset() + + +__all__ = [ + "PROFILES", + "DEFAULT_PROFILE", + "apply_profile", + "count_substitutions", + "profile_summary", + "emitted_codepoints", +] diff --git a/forgelm/_script_sanity.py b/forgelm/_script_sanity.py new file mode 100644 index 00000000..18db02b2 --- /dev/null +++ b/forgelm/_script_sanity.py @@ -0,0 +1,484 @@ +"""Phase 15 Task 2 — Unicode / script sanity check for extracted text. + +Runs **after** every per-file extraction. For a language hint (e.g. +``"tr"`` for Turkish), counts the ratio of code points that fall outside +the language's expected Unicode blocks (plus a small allow-list of +universal punctuation / digits / ASCII whitespace) and emits a single +operator-facing ``WARNING`` when the ratio crosses the configured +threshold. + +Why this exists: pypdf's font-fallback failures (audit §1.2 / §1.3) and +mis-decoded TXT files produce text that *parses* but is full of +out-of-script garbage — the 2026-05-11 pilot measured ≈ 7 % corrupt-char +ratio on the front-matter pages of a Turkish PDF and 100 % U+085F bullet +glyphs on a chapter-level section. Both classes of failure would be +caught by a single Unicode-block sanity check. + +Design rules: + +* **No new heavy deps.** Block detection uses :func:`unicodedata.name` + (stdlib) plus a small per-language allow-list. ``langdetect`` (already + shipped under ``[ingestion]``) is not invoked here — the check runs + whether or not a language was auto-detected, because the operator + supplies the hint explicitly via ``--language-hint``. +* **Fail-soft.** The check never raises and never blocks ingestion; + it logs a warning and surfaces structured metrics in ``notes_structured`` + so machine-driven pipelines can react without grepping log lines. +* **Calibrated default threshold.** The 2026-05-11 audit measured ~7 % + corruption on the worst-case pages and 0 % on clean body. The + documented placeholder of 0.5 % in the audit was intentionally + aggressive; we ship a calibrated **1.5 %** default that catches the + pilot's two-chunk front-matter corruption (~7 % ratio) and the + bullet-glyph chapter (~3 % ratio over 17 chunks) while leaving room + for legitimate mixed-script content (Turkish corpora with English + code samples sit comfortably under 1 %). +* **Profile-aware.** Code points the normaliser is allowed to produce + (per ``forgelm._pypdf_normalise.emitted_codepoints``) are considered + in-script even when they fall outside the strict per-language block + list — otherwise the warning would fire on every successful + normalisation pass. + +Public surface: one function (:func:`check_script_sanity`) and one +constant (:data:`DEFAULT_THRESHOLD`). +""" + +from __future__ import annotations + +import logging +from dataclasses import dataclass, field +from typing import Dict, FrozenSet, List, Optional, Tuple + +from ._pypdf_normalise import emitted_codepoints + +logger = logging.getLogger("forgelm.ingestion.script_sanity") + +DEFAULT_THRESHOLD: float = 0.015 +"""Calibrated 1.5 % ratio — fires on the audit's two failure modes +(front-matter font corruption ≈ 7 %, bullet-glyph chapter ≈ 3 %) while +leaving room for legitimate mixed-script content (1 % envelope). +""" + + +# --------------------------------------------------------------------------- +# Per-language allow-lists +# --------------------------------------------------------------------------- + +# Universal allow-list: ASCII printables (incl. punctuation and digits), +# whitespace, and a few common typographic glyphs that turn up in any +# corpus regardless of source language. Members are bare strings so the +# membership test is a hash lookup, not a range comparison. +_UNIVERSAL_ALLOW: FrozenSet[str] = frozenset( + # ASCII printables 0x20–0x7E + tab / newline / CR. + [chr(c) for c in range(0x20, 0x7F)] + + ["\t", "\n", "\r", "\f", "\v"] + # Typographic punctuation that survives any language. + + [ + "‘", # ‘ LEFT SINGLE QUOTE + "’", # ’ RIGHT SINGLE QUOTE + "“", # “ LEFT DOUBLE QUOTE + "”", # ” RIGHT DOUBLE QUOTE + "–", # – EN DASH + "—", # — EM DASH + "…", # … HORIZONTAL ELLIPSIS + " ", # NO-BREAK SPACE + "•", # • BULLET + "«", # « LEFT GUILLEMET + "»", # » RIGHT GUILLEMET + ] +) + + +# Per-language allow-lists. Each entry is ``(ranges, extras)`` where: +# +# * ``ranges`` is a tuple of ``(low_inclusive, high_inclusive)`` +# contiguous Unicode block boundaries. +# * ``extras`` is a frozenset of discrete characters in the +# **scattered** region — Latin-1 Supplement carries one Turkish +# character (``Ç``) at U+00C7 and another (``ç``) at U+00E7 +# while ``ø``, ``Õ``, ``ú``, ``÷`` live in the same block but are +# not Turkish. A wide range like ``(0x00A0, 0x024F)`` would silently +# mark every Western European letter as in-script and let the +# audit's font-fallback corruption pass unnoticed. +# +# A character is in-script iff its code point falls in any range OR +# appears in the language's ``extras`` set OR appears in +# ``_UNIVERSAL_ALLOW``. +_LANGUAGE_BLOCKS: Dict[str, Tuple[Tuple[Tuple[int, int], ...], FrozenSet[str]]] = { + "tr": ( + ( + (0x0020, 0x007E), # Basic Latin + (0x2000, 0x206F), # General Punctuation + ), + frozenset( + # Turkish-specific letters in Latin-1 Supplement + Latin Extended-A. + # Discrete listing prevents the audit's øÕú÷ artefacts (also in + # Latin-1 Supplement) from masquerading as in-script. + [ + "Ç", + "ç", # U+00C7 / U+00E7 + "Ö", + "ö", # U+00D6 / U+00F6 + "Ü", + "ü", # U+00DC / U+00FC + "Ğ", + "ğ", # U+011E / U+011F + "İ", + "ı", # U+0130 / U+0131 + "Ş", + "ş", # U+015E / U+015F + "Â", + "â", # U+00C2 / U+00E2 — Ottoman/loanwords + "Î", + "î", # U+00CE / U+00EE + "Û", + "û", # U+00DB / U+00FB + ] + ), + ), + "en": ( + ( + (0x0020, 0x007E), + (0x2000, 0x206F), + ), + frozenset( + # English keeps a few diacritic loan-words (café, naïve, résumé) + # without opening the door to every Latin-1 character. + ["é", "É", "è", "ë", "ï", "ô", "ç", "à", "â", "ä", "ö", "ü", "ñ"] + ), + ), + "de": ( + ( + (0x0020, 0x007E), + (0x2000, 0x206F), + ), + frozenset(["Ä", "ä", "Ö", "ö", "Ü", "ü", "ß"]), + ), + "fr": ( + ( + (0x0020, 0x007E), + (0x2000, 0x206F), + ), + frozenset( + [ + "à", + "â", + "ä", + "ç", + "é", + "è", + "ê", + "ë", + "î", + "ï", + "ô", + "ö", + "ù", + "û", + "ü", + "ÿ", + "À", + "Â", + "Ä", + "Ç", + "É", + "È", + "Ê", + "Ë", + "Î", + "Ï", + "Ô", + "Ö", + "Ù", + "Û", + "Ü", + "Ÿ", + ] + ), + ), + "es": ( + ( + (0x0020, 0x007E), + (0x2000, 0x206F), + ), + frozenset(["á", "é", "í", "ó", "ú", "ñ", "ü", "Á", "É", "Í", "Ó", "Ú", "Ñ", "Ü", "¡", "¿"]), + ), + "it": ( + ( + (0x0020, 0x007E), + (0x2000, 0x206F), + ), + frozenset(["à", "è", "é", "ì", "í", "ò", "ó", "ù", "À", "È", "É", "Ì", "Í", "Ò", "Ó", "Ù"]), + ), + "pt": ( + ( + (0x0020, 0x007E), + (0x2000, 0x206F), + ), + frozenset( + [ + "á", + "à", + "â", + "ã", + "ç", + "é", + "ê", + "í", + "ó", + "ô", + "õ", + "ú", + "Á", + "À", + "Â", + "Ã", + "Ç", + "É", + "Ê", + "Í", + "Ó", + "Ô", + "Õ", + "Ú", + ] + ), + ), +} +"""Per-language allow-lists. + +Only a handful of European languages are seeded here because they were +the most concrete real-world need at Phase-15 close. Adding ``ar`` / +``zh`` / ``ja`` / ``ko`` is a one-line edit but is intentionally deferred +to Phase 16+ once the wave-1 regression fixtures are in place to anchor +the calibration — claims about CJK / Arabic ratios without measured +fixtures would be marketing-grade, not engineering-grade. +""" + +SUPPORTED_LANGUAGES: Tuple[str, ...] = tuple(sorted(_LANGUAGE_BLOCKS.keys())) + + +@dataclass(frozen=True) +class ScriptSanityReport: + """Outcome of one ``check_script_sanity`` invocation. + + All fields are part of the public ``notes_structured`` schema; once + surfaced under ``script_sanity_summary`` they are stable across + releases. Adding fields is allowed; renaming or removing fields is + a breaking change. + """ + + file_path: str + language_hint: str + total_chars: int + out_of_script_chars: int + ratio: float + threshold: float + triggered: bool + # Per-codepoint counts for the worst offenders (top 8 by count). Useful + # both for the warning text and for downstream regression tests that + # need to assert the exact glyph caught the sanity check. + top_offenders: Tuple[Tuple[str, int], ...] = field(default_factory=tuple) + cause_hint: str = "" + + +def _is_in_script( + char: str, + ranges: Tuple[Tuple[int, int], ...], + language_extras: FrozenSet[str], + profile_extras: FrozenSet[str], +) -> bool: + """Return ``True`` iff ``char`` is in the language ranges or any allow-list.""" + if char in _UNIVERSAL_ALLOW or char in language_extras or char in profile_extras: + return True + cp = ord(char) + for low, high in ranges: + if low <= cp <= high: + return True + return False + + +def _classify_cause(top_offenders: Tuple[Tuple[str, int], ...]) -> str: + """Heuristic cause-of-corruption guess for the warning message. + + Looks at the most-frequent offending code points and matches against + three well-known patterns: + + * **Font fallback** — Latin-1 region (U+0080..U+00FF) but matching + the Turkish-glyph-fallback profile from + :mod:`forgelm._pypdf_normalise`. Operators see "looks like + pypdf font fallback — run ``forgelm doctor`` to confirm the + normaliser table loaded, then re-ingest". + * **Mojibake** — replacement characters (U+FFFD) dominating. Often + means the source file was read with the wrong encoding. + * **Custom bullet / private use** — code points in PUA blocks + (U+E000..U+F8FF) or in Mandaic / non-Latin scripts not + registered for this language. Operators see "custom-glyph + bullet detected — consider a normalisation profile or + pre-process the source". + + Returns an empty string when no heuristic fires; the warning still + emits but without a cause guess. + """ + if not top_offenders: + return "" + # Look at the top three only — anything below that is unlikely to be + # a single systematic failure mode. + head = [glyph for glyph, _ in top_offenders[:3]] + replacement_count = sum(1 for g in head if g == "�") + if replacement_count: + return "mojibake (replacement chars U+FFFD dominate)" + latin1_extras = sum(1 for g in head if 0x80 <= ord(g) <= 0xFF) + if latin1_extras >= 2: + return "pypdf font fallback (Latin-1 substitutes)" + pua_or_extra = sum(1 for g in head if 0xE000 <= ord(g) <= 0xF8FF or ord(g) >= 0x0800) + if pua_or_extra: + return "custom glyph / private-use block" + return "" + + +def check_script_sanity( + text: str, + *, + language_hint: str, + file_path: str, + threshold: float = DEFAULT_THRESHOLD, + profile: str = "turkish", + logger_override: Optional[logging.Logger] = None, +) -> ScriptSanityReport: + """Compute the out-of-script ratio for ``text`` and warn when over threshold. + + The check is **fail-soft** — empty / whitespace-only input, an + unknown ``language_hint``, or a zero-character corpus all return an + untriggered :class:`ScriptSanityReport` without logging. Callers can + therefore call this on every file without guards. + + Args: + text: Extracted text from one file. + language_hint: BCP-47-ish language code (``"tr"``, ``"en"``, ...). + Unknown values disable the check; the report is returned with + ``triggered=False`` so callers can detect the no-op case. + file_path: Display path for the warning message (the report + carries this verbatim so the operator can act on a specific + file without re-deriving it from log lines). + threshold: Out-of-script ratio above which the warning fires. + Default :data:`DEFAULT_THRESHOLD`. + profile: Glyph-normalisation profile active for this ingestion + run. Code points the profile is allowed to emit are treated + as in-script even if they fall outside the language ranges — + this prevents the warning from firing on every successful + normalisation pass. + logger_override: Test hook so callers can pin the warning + destination without monkeypatching the module logger. + + Returns: + A :class:`ScriptSanityReport`. ``triggered`` is true iff a + warning was emitted; ``ratio`` is always populated so callers + can roll into structured notes regardless of trigger state. + """ + use_logger = logger_override or logger + entry = _LANGUAGE_BLOCKS.get(language_hint) + if entry is None or not text: + # Unknown language hint or empty text: produce an untriggered report + # so the caller can still record the fact that the check ran. + return ScriptSanityReport( + file_path=file_path, + language_hint=language_hint, + total_chars=len(text), + out_of_script_chars=0, + ratio=0.0, + threshold=threshold, + triggered=False, + ) + + ranges, language_extras = entry + profile_extras = emitted_codepoints(profile) + out_of_script: Dict[str, int] = {} + total = 0 + for char in text: + if char.isspace(): + continue # whitespace is neutral; don't count it toward the denominator + total += 1 + if not _is_in_script(char, ranges, language_extras, profile_extras): + out_of_script[char] = out_of_script.get(char, 0) + 1 + + if total == 0: + return ScriptSanityReport( + file_path=file_path, + language_hint=language_hint, + total_chars=len(text), + out_of_script_chars=0, + ratio=0.0, + threshold=threshold, + triggered=False, + ) + + offending = sum(out_of_script.values()) + ratio = offending / total + top_offenders = tuple(sorted(out_of_script.items(), key=lambda kv: kv[1], reverse=True)[:8]) + cause_hint = _classify_cause(top_offenders) + + triggered = ratio > threshold + if triggered: + sample = ", ".join(f"{glyph!r}={count}" for glyph, count in top_offenders[:5]) + cause_suffix = f" — likely cause: {cause_hint}" if cause_hint else "" + use_logger.warning( + "Script-sanity warning for '%s' (language_hint=%s): " + "%.2f%% of non-whitespace chars fall outside the expected blocks " + "(threshold=%.2f%%). Top offenders: %s%s", + file_path, + language_hint, + ratio * 100, + threshold * 100, + sample, + cause_suffix, + ) + + return ScriptSanityReport( + file_path=file_path, + language_hint=language_hint, + total_chars=len(text), + out_of_script_chars=offending, + ratio=ratio, + threshold=threshold, + triggered=triggered, + top_offenders=top_offenders, + cause_hint=cause_hint, + ) + + +def report_to_structured(reports: List[ScriptSanityReport]) -> Dict[str, object]: + """Aggregate per-file reports into the ``notes_structured`` block. + + Keeps the on-disk schema tight: only triggered reports show up under + ``per_file`` so a clean 5000-file corpus does not produce a 5000-row + JSON payload. Aggregate totals always emit so consumers can rely on + the keys being present. + """ + triggered = [r for r in reports if r.triggered] + aggregate = { + "files_checked": len(reports), + "files_triggered": len(triggered), + "max_ratio": max((r.ratio for r in reports), default=0.0), + "threshold": reports[0].threshold if reports else DEFAULT_THRESHOLD, + } + if triggered: + aggregate["per_file"] = [ + { + "file_path": r.file_path, + "ratio": r.ratio, + "language_hint": r.language_hint, + "cause_hint": r.cause_hint, + "top_offenders": [{"glyph": glyph, "count": count} for glyph, count in r.top_offenders], + } + for r in triggered + ] + return aggregate + + +__all__ = [ + "DEFAULT_THRESHOLD", + "SUPPORTED_LANGUAGES", + "ScriptSanityReport", + "check_script_sanity", + "report_to_structured", +] diff --git a/forgelm/_strip_pattern.py b/forgelm/_strip_pattern.py new file mode 100644 index 00000000..753df875 --- /dev/null +++ b/forgelm/_strip_pattern.py @@ -0,0 +1,434 @@ +"""Phase 15 Wave 2 Task 11 — operator-supplied strip-pattern with ReDoS guard. + +The ``--strip-pattern REGEX`` flag lets operators delete known-boilerplate +lines that ForgeLM's automatic dedup heuristic misses (running headers +with varying content, watermark lines, DOI / journal headers, etc.). +Because the pattern is **operator-controlled**, two failure modes have +to be ruled out before the ingestion run starts: + +1. **Pathological back-tracking** — e.g. ``(a+)+b`` against a long + string of ``a`` characters is the textbook ReDoS shape. Validating + the pattern up-front at CLI parse time catches the most common + anti-patterns described in ``docs/standards/regex.md`` (rules 4 + 6). +2. **Run-time hang** — even a pattern that *looks* safe can hit a + degenerate input. A SIGALRM-based timeout per regex bounds the + worst-case match cost; on Windows + non-main threads the timeout is + a no-op and the operator is expected to vet the pattern themselves + (same trade-off ``forgelm reverse-pii --type custom`` makes). + +The public surface is intentionally small: + +* :func:`validate_strip_pattern` — CLI-time structural validation. +* :func:`compile_strip_patterns` — turn a list of validated patterns + into compiled :class:`re.Pattern` objects. +* :func:`apply_strip_patterns` — strip every matching span (or whole + line, in MULTILINE mode) with a per-pattern SIGALRM guard. +* :class:`StripPatternError` — single exception type the CLI catches. + +This module imports only stdlib so it can run before any ingestion +extras are installed. +""" + +from __future__ import annotations + +import logging +import os +import re +import signal +import threading +import time +from typing import List, Optional, Tuple + +logger = logging.getLogger("forgelm.ingestion.strip_pattern") + + +# --------------------------------------------------------------------------- +# Constants +# --------------------------------------------------------------------------- + +DEFAULT_TIMEOUT_S: int = 5 +"""Per-pattern match budget in seconds. + +Five seconds is generous for legitimate boilerplate-stripping on a +typical 200-page corpus (each page's extracted text is < 4 KB and a +sane regex finishes in microseconds), but small enough that a +pathological pattern aborts the run before it eats the user's session. +Operator can opt out via ``--strip-pattern-no-timeout`` for +known-safe patterns on POSIX main thread; the guard is always a no-op +elsewhere. +""" + +_MAX_PATTERN_LENGTH: int = 2048 +"""Generous upper bound on the operator's pattern string. + +ReDoS is largely a function of pattern complexity, not length, but a +genuinely useful boilerplate-stripping pattern fits inside this +envelope easily. Anything over 2 KB is overwhelmingly more likely to +be a paste-bin accident than a legitimate use case; rejecting it at +CLI-parse time saves a confusing run-time hang. +""" + + +class StripPatternError(ValueError): + """Raised when a strip-pattern is malformed or fails the ReDoS guard. + + Inherits from :class:`ValueError` so existing CLI dispatchers that + catch ``ValueError`` (per ``forgelm/cli/subcommands/_ingest.py``) + keep working without an explicit ``except`` for this subclass. + """ + + +# --------------------------------------------------------------------------- +# Structural validation (CLI-parse time) +# --------------------------------------------------------------------------- + + +def validate_strip_pattern(pattern: str) -> str: + """Validate ``pattern`` up-front; return it verbatim on success. + + Mirrors the SonarCloud ``python:S5852`` / regex.md rule 4 + 6 + structural checks ForgeLM applies to its own internal regexes: + + * Reject empty / overlong patterns. + * Reject patterns with **two unbounded greedy / lazy quantifiers + in sequence** that share a character class (rule 4) — the + ``(a+)+b`` / ``(.+?)(.+)`` shapes. + * Reject patterns combining ``.*?`` with a back-reference under + DOTALL (rule 6) — even when CPython handles it, SonarCloud + flags it and the operator can rewrite as multiple narrower + patterns. + * Reject patterns that fail :func:`re.compile` (syntax error). + + Raises :class:`StripPatternError` with a message that names the + offending construct so the operator can rewrite the pattern. + """ + if not pattern: + raise StripPatternError("--strip-pattern must be a non-empty regex.") + if len(pattern) > _MAX_PATTERN_LENGTH: + raise StripPatternError( + f"--strip-pattern is too long ({len(pattern)} chars; max {_MAX_PATTERN_LENGTH}). " + "If your boilerplate is more than 2 KB, split it across multiple --strip-pattern flags." + ) + try: + compiled = re.compile(pattern) + except re.error as exc: + raise StripPatternError(f"--strip-pattern {pattern!r} is not a valid regular expression: {exc}") from exc + + _check_unbounded_quantifier_sequence(pattern) + _check_dotall_backreference(pattern, compiled.flags) + + return pattern + + +_NESTED_UNBOUNDED_REJECTION_MSG = ( + "contains a nested unbounded quantifier (e.g. `(a+)+`, `(\\w+)+`, " + "`(\\w+\\s+)+`), which is a textbook ReDoS shape (regex.md rule 4). " + "Rewrite with bounded repetition (e.g. `a{1,100}`) or split across " + "multiple narrower --strip-pattern flags." +) + + +def _scan_atom_end(pattern: str, i: int, n: int) -> int: + """Return the index just past the atom that starts at ``pattern[i]``. + + An *atom* is one of: an escape pair (``\\X``), a character class + (``[…]`` honouring escapes), or a single literal character. Group + boundaries are handled by the caller, not here. Extracted from + :func:`_check_unbounded_quantifier_sequence` purely to keep the + caller below the Sonar S3776 cognitive-complexity ceiling. + """ + char = pattern[i] + if char == "\\": + return i + 2 + if char == "[": + j = i + 1 + while j < n: + if pattern[j] == "\\": + j += 2 + continue + if pattern[j] == "]": + break + j += 1 + return j + 1 + return i + 1 + + +def _advance_past_brace_quantifier(pattern: str, atom_end: int, n: int) -> int: + """Skip a ``{n,m}`` quantifier span so the outer loop won't re-parse digits.""" + j = atom_end + 1 + while j < n and pattern[j] != "}": + j += 1 + return j + 1 + + +def _close_group_or_raise( + pattern: str, + i: int, + n: int, + stack: List[Tuple[int, bool]], +) -> int: + """Handle ``)`` at ``pattern[i]``; raise on the nested-unbounded shape. + + Pops the current group, checks the immediately-following character + for ``+`` / ``*``, and propagates the "this atom was unbounded" flag + to the parent group. Returns the post-quantifier index so the + caller's main loop can advance. + """ + _open_idx, inner_unbounded = stack.pop() if len(stack) > 1 else (-1, False) + outer_quantifier = pattern[i + 1] if i + 1 < n else "" + if inner_unbounded and outer_quantifier in ("+", "*"): + raise StripPatternError(f"--strip-pattern {pattern!r} {_NESTED_UNBOUNDED_REJECTION_MSG}") + if stack: + top_open, _ = stack[-1] + stack[-1] = (top_open, outer_quantifier in ("+", "*")) + next_i = i + 1 + if outer_quantifier in ("+", "*", "?"): + next_i += 1 + return next_i + + +def _check_unbounded_quantifier_sequence(pattern: str) -> None: + """Reject patterns with two unbounded quantifiers in sequence. + + The classic ``(a+)+`` ReDoS shape — and its character-class + cousins ``(\\w+)+``, ``(\\d+)+``, ``(\\s+)+``, ``([abc]+)+``, + ``(\\w+\\s+)+`` — explodes on adversarial input even when CPython's + engine is "well-behaved" in practice. Phase 15 round-1 review (S-1) + found the original backward-walk validator skipped the escape-shape + variants because stepping past a ``\\`` consumed the matching ``(``; + the rewrite below walks **forward atom-by-atom**, tracking per + group whether the most recently parsed atom was unbounded + (quantified with ``+`` / ``*``). When a group's closing ``)`` is + immediately followed by another ``+`` / ``*``, that is the + nested-unbounded shape and we reject. + + The check is structural, not a full-AST parse — regex.md rule 4 + is about pattern *shape*, so an atom-by-atom forward scan is + sufficient. The SIGALRM timeout remains the safety net for the + long tail of exotic variants we still miss. Atom parsing and the + close-paren handler are extracted into helpers (``_scan_atom_end``, + ``_close_group_or_raise``, ``_advance_past_brace_quantifier``) to + keep this function below Sonar S3776's cognitive-complexity ceiling. + + Scope (post-round-3 review S-A): + This validator covers the **nested** unbounded shape + documented in ``docs/standards/regex.md`` rule 4 — i.e. a + group whose last atom carries ``+`` / ``*`` AND whose + closing ``)`` is itself followed by another ``+`` / ``*``. + Adjacent / sequential unbounded shapes (``(.+?)(.+)``, + ``(?:a|b)+(c|d)+``, ``(.+?)[ \\t]*$``) are also mentioned + by regex.md rule 4 but are **not** caught here. They + empirically do not backtrack catastrophically under + CPython 3.10–3.13 on adversarial 100-char input (verified + in the round-3 review), so they fall back on the per-pattern + SIGALRM timeout as the runtime safety net. Operators on + non-CPython runtimes (PyPy 7.x, Pyston) should treat the + SIGALRM net as the primary defence and consider replacing + adjacent-quantified patterns with bounded alternatives. + """ + # Each stack entry is ``(open_index, last_atom_unbounded)``. The + # bottom entry represents the implicit "outer" pattern; pushed + # entries represent groups. ``last_atom_unbounded`` is updated to + # match the most recently parsed atom — bounded atoms reset it, + # unbounded ones set it. + stack: List[Tuple[int, bool]] = [(-1, False)] + i = 0 + n = len(pattern) + while i < n: + char = pattern[i] + if char == "(": + stack.append((i, False)) + i += 1 + continue + if char == ")": + i = _close_group_or_raise(pattern, i, n, stack) + continue + + atom_end = _scan_atom_end(pattern, i, n) + quantifier = pattern[atom_end] if atom_end < n else "" + if stack: + top_open, _ = stack[-1] + stack[-1] = (top_open, quantifier in ("+", "*")) + + i = atom_end + if quantifier in ("+", "*", "?"): + i += 1 + elif quantifier == "{": + i = _advance_past_brace_quantifier(pattern, atom_end, n) + + +def _check_dotall_backreference(pattern: str, flags: int) -> None: + """Reject ``.*?`` + back-reference under DOTALL. + + Mirrors regex.md rule 6 / SonarCloud python:S5852. Cheap to detect + structurally: presence of ``.*?`` (or ``.+?``) **and** a back-reference + (``\\1`` … or ``(?P=name)``) under :data:`re.DOTALL`. The pattern + flags are read from the compiled object so operator-supplied inline + ``(?s)`` flags are honoured. + """ + if not (flags & re.DOTALL): + return + if ".*?" not in pattern and ".+?" not in pattern: + return + # Look for any back-reference: ``\`` or ``(?P=name)``. + if re.search(r"\\[1-9]|\(\?P=", pattern): + raise StripPatternError( + f"--strip-pattern {pattern!r} combines a lazy `.*?` / `.+?` with a " + "back-reference under DOTALL — this is the SonarCloud python:S5852 " + "polynomial-runtime shape. Rewrite as a line-walker or split the " + "pattern across multiple --strip-pattern flags." + ) + + +# --------------------------------------------------------------------------- +# Compilation +# --------------------------------------------------------------------------- + + +def compile_strip_patterns(patterns: List[str]) -> List[Tuple[str, re.Pattern[str]]]: + """Compile a list of validated strip-patterns to ``(raw, compiled)`` pairs. + + The raw string is preserved alongside the compiled pattern so the + timeout warning can quote the offending pattern verbatim instead of + re-deriving it from :attr:`re.Pattern.pattern` (which collapses some + operator-supplied character classes). + """ + out: List[Tuple[str, re.Pattern[str]]] = [] + for raw in patterns: + validate_strip_pattern(raw) + # MULTILINE so ``^`` / ``$`` work line-by-line, which is the dominant + # use case (strip an entire boilerplate line, not a substring). + compiled = re.compile(raw, flags=re.MULTILINE) + out.append((raw, compiled)) + return out + + +# --------------------------------------------------------------------------- +# Application +# --------------------------------------------------------------------------- + + +def apply_strip_patterns( + text: str, + patterns: List[Tuple[str, re.Pattern[str]]], + *, + timeout_s: Optional[int] = DEFAULT_TIMEOUT_S, + logger_override: Optional[logging.Logger] = None, +) -> Tuple[str, int]: + """Strip every match from ``text``; return ``(new_text, total_matches)``. + + Patterns are applied in order. Each match is replaced with an empty + string. On POSIX main-thread runs a per-pattern SIGALRM budget of + ``timeout_s`` seconds bounds the worst-case backtracking cost; on + Windows or worker threads the guard is a no-op (same trade-off + ``forgelm reverse-pii --type custom`` makes). + + When a timeout fires, the offending pattern is **skipped** with a + loud ``WARNING`` and the remaining patterns continue to apply. + Skipping is the right trade-off: aborting the ingestion run would + drop user data on the floor, while silently swallowing the timeout + would leave the operator wondering why their pattern did nothing. + + Args: + text: Text to scan. + patterns: Output of :func:`compile_strip_patterns`. + timeout_s: Per-pattern budget. ``None`` disables the guard + entirely (used by ``--strip-pattern-no-timeout``). A + non-positive integer is **rejected** with :class:`ValueError` + — review round-2 (S-1) found that silently treating + ``0`` / negatives as "no timeout" hid operator misconfiguration + of the documented 5-second default. + logger_override: Test seam; defaults to the module logger. + + Returns: + The stripped text and the total number of substitutions made. + """ + if timeout_s is not None and timeout_s <= 0: + raise ValueError( + f"timeout_s must be a positive int or None (got {timeout_s!r}); " + "use None to disable the SIGALRM guard explicitly." + ) + + use_logger = logger_override or logger + if not patterns or not text: + return text, 0 + + use_alarm = ( + timeout_s is not None + and os.name == "posix" + and hasattr(signal, "SIGALRM") + and threading.current_thread() is threading.main_thread() + ) + + total_subs = 0 + out = text + for raw, compiled in patterns: + # Round-5 review (S-E) perf nit: if an earlier pattern consumed + # the entire text, skip the remaining ones — each would otherwise + # set up + tear down a SIGALRM around a no-op subn call. + if not out: + break + new_out, subs = _apply_one(out, raw, compiled, timeout_s, use_alarm, use_logger) + out = new_out + total_subs += subs + return out, total_subs + + +def _apply_one( + text: str, + raw: str, + compiled: re.Pattern[str], + timeout_s: Optional[int], + use_alarm: bool, + use_logger: logging.Logger, +) -> Tuple[str, int]: + """Apply one pattern under the ReDoS guard; warn + skip on timeout.""" + if not use_alarm or timeout_s is None: + new_out, subs = compiled.subn("", text) + return new_out, subs + + def _alarm(_sig, _frame): # pragma: no cover — exercised via integration tests + raise OSError(f"strip-pattern {raw!r} exceeded {timeout_s}s (ReDoS guard)") + + previous_handler = signal.signal(signal.SIGALRM, _alarm) + # Round-3 review (Gemini): the inner alarm must NOT extend an outer + # caller's shorter remaining budget. ``signal.alarm(N)`` returns the + # seconds remaining on any prior alarm; clamp our budget to the + # smaller of ``timeout_s`` and that outer remainder so a nested + # caller can never exceed the surrounding deadline. + previous_remaining = signal.alarm(0) + if previous_remaining > 0: + chosen = min(timeout_s, previous_remaining) + else: + chosen = timeout_s + signal.alarm(chosen) + started = time.monotonic() + try: + new_out, subs = compiled.subn("", text) + return new_out, subs + except OSError as exc: + use_logger.warning( + "Skipping --strip-pattern (timeout): %s. Remaining patterns will still apply.", + exc, + ) + return text, 0 + finally: + signal.alarm(0) + signal.signal(signal.SIGALRM, previous_handler) + if previous_remaining > 0: + # Honour the outer caller's ORIGINAL remaining budget (not + # ``chosen``) minus the wall-clock seconds we burned. Round + # up partial seconds and clamp to ≥1 so we never silently + # cancel the outer alarm. + elapsed = time.monotonic() - started + remaining = max(int(previous_remaining - elapsed) + 1, 1) + signal.alarm(remaining) + + +__all__ = [ + "DEFAULT_TIMEOUT_S", + "StripPatternError", + "validate_strip_pattern", + "compile_strip_patterns", + "apply_strip_patterns", +] diff --git a/forgelm/cli/_parser.py b/forgelm/cli/_parser.py index 99b8a558..2e916d56 100644 --- a/forgelm/cli/_parser.py +++ b/forgelm/cli/_parser.py @@ -294,6 +294,138 @@ def _add_ingest_subcommand(subparsers) -> None: metavar="MODEL_NAME", help="HuggingFace model name passed to AutoTokenizer.from_pretrained when --chunk-tokens is set.", ) + # ---- Phase 15 flags (Wave 1) ----------------------------------------- + p.add_argument( + "--language-hint", + type=str, + default=None, + metavar="LANG", + help=( + "Phase 15 Wave 1: BCP-47-style language code (e.g. 'tr', 'en'). When set, the " + "extraction layer runs a Unicode-block sanity check after each per-file extract " + "and emits a WARNING (with structured notes) when the out-of-script char ratio " + "exceeds --script-sanity-threshold. Catches pypdf font-fallback corruption + " + "TXT encoding mis-routing without blocking the run. No-op when unset." + ), + ) + p.add_argument( + "--script-sanity-threshold", + type=_non_negative_float, + default=None, + metavar="X", + help=( + "Phase 15 Wave 1: ratio (0.0-1.0) above which the script-sanity check warns " + "(default: library calibrated 0.015 = 1.5%%). Operators on noisy mixed-script " + "corpora can relax this; values < 1%% start false-positiving on legitimate " + "English-with-code-samples Turkish corpora." + ), + ) + p.add_argument( + "--normalise-profile", + type=str, + default=None, + choices=["turkish", "none"], + help=( + "Phase 15 Wave 1: glyph-normalisation profile applied to extracted text. " + "'turkish' (default) maps known pypdf-font-fallback artefacts back to the " + "correct Turkish characters; 'none' disables normalisation entirely " + "(equivalent to --no-normalise-unicode)." + ), + ) + p.add_argument( + "--no-normalise-unicode", + action="store_true", + help=( + "Phase 15 Wave 1: opt-out shortcut equivalent to --normalise-profile none. " + "Useful when the operator has already pre-normalised the corpus." + ), + ) + p.add_argument( + "--no-quality-presignal", + action="store_true", + help=( + "Phase 15 Wave 1: skip the end-of-run quality pre-signal (default ON). The " + "pre-signal scans every emitted chunk for low alpha-ratio, weird-char-ratio, " + "and repeated-line-ratio and prints '[WARN] N/M chunks below ingestion quality " + "threshold' when any chunk fails. Full diagnostics still live in `forgelm audit`." + ), + ) + p.add_argument( + "--epub-no-skip-frontmatter", + action="store_true", + help=( + "Phase 15 Wave 1: keep EPUB nav / cover / copyright / colophon / titlepage / " + "frontmatter items. Default is to skip them so SFT training does not learn " + "from boilerplate. Operators wanting raw extraction pass this flag." + ), + ) + p.add_argument( + "--keep-md-frontmatter", + action="store_true", + help=( + "Phase 15 Wave 1: retain YAML front-matter (`---\\n...\\n---\\n`) at the start " + "of Markdown files. Default is to strip it so chunk 0 does not lead with " + "Jekyll / Hugo / MkDocs metadata." + ), + ) + # ---- Phase 15 flags (Wave 2) ----------------------------------------- + p.add_argument( + "--strip-pattern", + action="append", + default=None, + metavar="REGEX", + help=( + "Phase 15 Wave 2: regex applied to extracted text BEFORE chunking; every match " + "is deleted. Repeat for multiple patterns. Each pattern is validated up-front " + "for ReDoS-prone shapes (nested unbounded quantifiers, `.*?` + back-reference " + "under DOTALL) and aborts the run with EXIT_CONFIG_ERROR on the offender. A " + "5-second per-pattern SIGALRM budget bounds the worst-case match cost on POSIX." + ), + ) + p.add_argument( + "--strip-pattern-no-timeout", + action="store_true", + help=( + "Phase 15 Wave 2: disable the per-pattern SIGALRM timeout. Only safe when the " + "operator has independently verified every --strip-pattern is linear under " + "their input distribution; the documented default of 5s is a safety net." + ), + ) + p.add_argument( + "--page-range", + type=str, + default=None, + metavar="START-END", + help=( + "Phase 15 Wave 2: restrict PDF extraction to pages START-END (inclusive, " + "1-indexed). Validation failures (start < 1, start > end, start > page_count) " + "exit with EXIT_CONFIG_ERROR. Ignored for non-PDF inputs." + ), + ) + p.add_argument( + "--keep-frontmatter", + action="store_true", + help=( + "Phase 15 Wave 2: opt OUT of the PDF front-matter / back-matter heuristic. " + "From v0.6.0 the heuristic is DEFAULT ON: the first / last 12 PDF pages are " + "checked against three signals (alpha < 0.45, underscore > 0.10, ≥ 5 inline " + "page-number matches) and dropped when all three fire. Pass --keep-frontmatter " + "to preserve the pre-Phase-15 behaviour (everything kept)." + ), + ) + p.add_argument( + "--strip-urls", + type=str, + default="keep", + choices=["keep", "mask", "strip"], + help=( + "Phase 15 Wave 2: how to handle inline URLs detected in extracted text. 'keep' " + "(default) leaves them untouched. 'mask' replaces each URL with '[URL]'. " + "'strip' deletes them outright. Useful for QR-code reference noise and DOI " + "headers. Independent of --all-mask (Phase 12.5) — URL handling is a content-" + "shape decision, not a GDPR redaction." + ), + ) _add_common_subparser_flags(p, include_output_format=True) @@ -406,14 +538,25 @@ def _add_audit_subcommand(subparsers) -> None: "Must lie in [0.0, 1.0]; ignored when --dedup-method=simhash." ), ) + # Phase 15 Task 5: --quality-filter is DEFAULT ON from v0.6.0. The audit + # JSON's quality_summary block always populates; operators who explicitly + # want the old "opt-in" behaviour pass --no-quality-filter. The argparse + # ``BooleanOptionalAction`` form gives both --quality-filter (no-op, kept + # for forward compat of explicit-on invocations) and --no-quality-filter + # in one flag definition. Using a plain pair of store_true / store_false + # would surface as two separate help-text lines; the BooleanOptionalAction + # form keeps the operator-facing help compact and matches the precedent + # set by `python -m argparse --help`. p.add_argument( "--quality-filter", - action="store_true", + action=argparse.BooleanOptionalAction, + default=True, help=( - "Phase 12 opt-in: run heuristic quality checks (mean word length, alphabetic-character " - "ratio, end-of-line punctuation ratio, repeated-line ratio, short-paragraph ratio). " - "Findings appear under quality_summary in the audit JSON. ML-based classifiers are " - "deferred to Phase 13+." + "Phase 15 Wave 1 (Task 5): heuristic quality checks (mean word length, alphabetic-" + "character ratio, end-of-line punctuation ratio, repeated-line ratio, short-paragraph " + "ratio). DEFAULT ON in v0.6.0+ — operators who want the pre-v0.6.0 opt-in semantics " + "pass --no-quality-filter. Findings land under quality_summary in the audit JSON. " + "ML-based classifiers are deferred to Phase 13+." ), ) p.add_argument( diff --git a/forgelm/cli/subcommands/_doctor.py b/forgelm/cli/subcommands/_doctor.py index 299e1289..461bfa9c 100644 --- a/forgelm/cli/subcommands/_doctor.py +++ b/forgelm/cli/subcommands/_doctor.py @@ -77,6 +77,8 @@ _PROBE_HF_HUB_OFFLINE_CACHE = "hf_hub.offline_cache" _PROBE_DISK_WORKSPACE = "disk.workspace" _PROBE_OPERATOR_IDENTITY = "operator.identity" +# Phase 15 Task 3 — pypdf font-fallback normalisation profile loaded? +_PROBE_PYPDF_NORMALISE_TURKISH = "pypdf_normalise.turkish" # Optional extras advertised in pyproject.toml [project.optional-dependencies]. # Each entry is ``(extra_name, importable_module, human_purpose_blurb)``. @@ -845,6 +847,67 @@ def _check_operator_identity() -> _CheckResult: ) +# --------------------------------------------------------------------------- +# Phase 15 Task 3 — pypdf font-fallback normalisation probe +# --------------------------------------------------------------------------- + + +def _check_pypdf_normalise_turkish() -> _CheckResult: + """Verify the Turkish glyph-normalisation profile loads + applies cleanly. + + Phase 15 Task 3 acceptance criterion: ``forgelm doctor`` must report + a ``pypdf_normalise.turkish: pass`` row so an operator can confirm + the table loaded correctly without running a test ingest. The probe + runs through one round-trip sample (corrupt glyphs → normalised + output) and fails loudly if the table is missing or the dispatcher + no-ops on the active profile. + """ + try: + from ..._pypdf_normalise import ( + DEFAULT_PROFILE, + apply_profile, + profile_summary, + ) + except ImportError as exc: + return _CheckResult( + name=_PROBE_PYPDF_NORMALISE_TURKISH, + status=_STATUS_FAIL, + detail=( + "forgelm._pypdf_normalise import failed — Phase 15 normalisation " + f"profile is unreachable: {exc.__class__.__name__}: {exc}." + ), + extras={"profile": "turkish", "loaded": False}, + ) + summary = profile_summary("turkish") + sample_input = "ø Õ ú ÷ ࡟" + sample_output = apply_profile(sample_input, "turkish") + if sample_output == sample_input: + return _CheckResult( + name=_PROBE_PYPDF_NORMALISE_TURKISH, + status=_STATUS_FAIL, + detail=( + "Turkish glyph-normalisation profile loaded but produced no substitutions " + "on the canonical fixture (`ø Õ ú ÷ ࡟`). The table may have been emptied " + "or the dispatcher is no-opping; re-install forgelm or file a bug." + ), + extras={"profile": "turkish", "expected_substitutions": True, "actual_output": sample_output}, + ) + return _CheckResult( + name=_PROBE_PYPDF_NORMALISE_TURKISH, + status=_STATUS_PASS, + detail=( + f"Turkish glyph profile loaded: {summary['single']} single-char + " + f"{summary['multi']} multi-char substitutions. Active profile: {DEFAULT_PROFILE!r}." + ), + extras={ + "profile": "turkish", + "single_substitutions": summary["single"], + "multi_substitutions": summary["multi"], + "default_profile": DEFAULT_PROFILE, + }, + ) + + # --------------------------------------------------------------------------- # Orchestration # --------------------------------------------------------------------------- @@ -866,6 +929,10 @@ def _build_check_plan(*, offline: bool) -> List[Tuple[str, Callable[[], _CheckRe ] for extra_name, module, purpose in _OPTIONAL_EXTRAS: plan.append((f"extras.{extra_name}", _make_extra_probe(extra_name, module, purpose))) + # Phase 15 Task 3: pypdf font-fallback normalisation probe sits next to + # the [ingestion] extras probe so operators can correlate "the extra is + # missing" with "...so the profile cannot apply" in a single glance. + plan.append((_PROBE_PYPDF_NORMALISE_TURKISH, _check_pypdf_normalise_turkish)) if offline: plan.append((_PROBE_HF_HUB_OFFLINE_CACHE, _check_hf_cache_offline)) else: diff --git a/forgelm/cli/subcommands/_ingest.py b/forgelm/cli/subcommands/_ingest.py index 73a5b442..652a078b 100644 --- a/forgelm/cli/subcommands/_ingest.py +++ b/forgelm/cli/subcommands/_ingest.py @@ -1,16 +1,100 @@ -"""``forgelm ingest`` dispatcher (Phase 11 raw-document → JSONL flow).""" +"""``forgelm ingest`` dispatcher (Phase 11 raw-document → JSONL flow). + +Phase 15 (v0.6.0) added the script-sanity / glyph-normalisation / +strip-pattern / page-range / front-matter / strip-urls / quality +pre-signal knobs; they all funnel through the same dispatch path so +the JSON envelope on the way out stays a single-source contract. +""" from __future__ import annotations import json import sys +from typing import NoReturn, Optional, Tuple +from ..._strip_pattern import DEFAULT_TIMEOUT_S as _STRIP_PATTERN_DEFAULT_TIMEOUT_S from .._exit_codes import EXIT_CONFIG_ERROR, EXIT_TRAINING_ERROR from .._logging import logger +def _emit_error_and_exit( + exc: Exception, + *, + output_format: str, + exit_code: int, + log_prefix: str = "%s", +) -> NoReturn: + """Centralised JSON-vs-text error envelope + ``sys.exit`` helper. + + Round-2 review (nit on _ingest.py:82-168): the dispatcher used to + repeat the same ``if json: print(...) else: logger.error(...)`` + pattern in every ``except`` block. Centralising the envelope keeps + the four code paths byte-identical in behaviour while removing + duplication, and lowers the cognitive-complexity score Sonar + python:S3776 flagged on ``_run_ingest_cmd``. + """ + if output_format == "json": + print(json.dumps({"success": False, "error": str(exc)})) + else: + logger.error(log_prefix, exc) + sys.exit(exit_code) + + +def _parse_page_range(value: Optional[str]) -> Optional[Tuple[int, int]]: + """Parse ``--page-range START-END`` into a ``(start, end)`` tuple. + + Raises ``ValueError`` with an operator-facing message on malformed + input so the surrounding ``except ValueError`` block converts it to + EXIT_CONFIG_ERROR. + """ + if value is None: + return None + raw = value.strip() + if "-" not in raw: + raise ValueError(f"--page-range must be 'START-END' (e.g. '12-193'); got {raw!r}.") + parts = raw.split("-") + if len(parts) != 2: + raise ValueError(f"--page-range must be exactly two integers joined by '-'; got {raw!r}.") + try: + start = int(parts[0]) + end = int(parts[1]) + except ValueError as exc: + raise ValueError(f"--page-range parts must be integers (e.g. '12-193'); got {raw!r}.") from exc + return (start, end) + + +def _resolve_normalise_profile(args) -> str: + """Resolve --normalise-profile vs --no-normalise-unicode vs --language-hint. + + Phase 15 round-1 review (C-2): the original implementation + defaulted to the ``turkish`` profile unconditionally, which + silently rewrote legitimate non-Turkish letters (Norwegian + ``ø``, Estonian ``Õ``, math ``÷``). The fix couples the profile + to the operator's language hint: + + * Explicit ``--no-normalise-unicode`` always wins (``"none"``). + * Explicit ``--normalise-profile X`` wins next (operator + override). + * Otherwise, derive from ``--language-hint``: ``tr`` → ``turkish``; + every other hint (and the unset case) → ``"none"``. + + So a Turkish operator who already passes ``--language-hint tr`` + keeps getting normalisation "for free"; everyone else is safe by + default. + """ + if getattr(args, "no_normalise_unicode", False): + return "none" + explicit_profile = getattr(args, "normalise_profile", None) + if explicit_profile is not None: + return explicit_profile + language_hint = (getattr(args, "language_hint", None) or "").lower() + if language_hint == "tr": + return "turkish" + return "none" + + def _run_ingest_cmd(args, output_format: str) -> None: - """Phase 11 dispatch: raw documents → SFT-ready JSONL.""" + """Phase 11 dispatch (with Phase 15 knobs): raw documents → SFT-ready JSONL.""" # Phase 12.5: --all-mask is a shorthand that ORs into the two individual # masking flags. Resolve it here at the CLI boundary so ``ingest_path`` # keeps its narrow API (only ``pii_mask`` / ``secrets_mask`` booleans). @@ -26,8 +110,46 @@ def _run_ingest_cmd(args, output_format: str) -> None: # ``OptionalDependencyError`` is guaranteed to be bound when the # ``except`` clause runs; if the import itself fails it propagates as a # plain ``ImportError`` (a real bug, not an operator-facing condition). + from ..._strip_pattern import StripPatternError from ...ingestion import OptionalDependencyError, ingest_path, summarize_result + # Phase 15 args (Wave 1 + Wave 2). Resolved up here so the + # ``ingest_path`` call below stays a single straight-line. + try: + page_range = _parse_page_range(getattr(args, "page_range", None)) + except ValueError as exc: + _emit_error_and_exit(exc, output_format=output_format, exit_code=EXIT_CONFIG_ERROR) + strip_patterns = getattr(args, "strip_pattern", None) or None + strip_pattern_timeout = ( + None if getattr(args, "strip_pattern_no_timeout", False) else _STRIP_PATTERN_DEFAULT_TIMEOUT_S + ) + quality_presignal = not getattr(args, "no_quality_presignal", False) + + # Round-5 independent review (C-B): if the operator passes a + # ``--language-hint`` that the script-sanity layer doesn't have an + # allow-list for, the check would silently no-op and produce a + # ``script_sanity_summary`` block with ``max_ratio=0.0`` — the + # textbook silent-failure UX Phase 15 was launched to fix, just + # transplanted to a new surface. Warn loudly so the operator either + # picks a supported hint or accepts the no-op. + language_hint_raw = getattr(args, "language_hint", None) + if language_hint_raw: + from ..._script_sanity import SUPPORTED_LANGUAGES + + if language_hint_raw.lower() not in SUPPORTED_LANGUAGES: + logger.warning( + "--language-hint %r is not in the script-sanity allow-list (%s). " + "Script-sanity check will silently no-op for this corpus — the " + "`script_sanity_summary` block in notes_structured will report " + "`max_ratio: 0.0` even on heavily corrupted text. Use one of the " + "supported codes or omit the flag to disable the check explicitly.", + language_hint_raw, + ", ".join(SUPPORTED_LANGUAGES), + ) + + sanity_threshold_raw = getattr(args, "script_sanity_threshold", None) + sanity_kwargs = {} if sanity_threshold_raw is None else {"script_sanity_threshold": sanity_threshold_raw} + try: result = ingest_path( args.input_path, @@ -41,6 +163,29 @@ def _run_ingest_cmd(args, output_format: str) -> None: chunk_tokens=getattr(args, "chunk_tokens", None), overlap_tokens=getattr(args, "overlap_tokens", 0), tokenizer=getattr(args, "tokenizer", None), + language_hint=getattr(args, "language_hint", None), + normalise_profile=_resolve_normalise_profile(args), + keep_md_frontmatter=getattr(args, "keep_md_frontmatter", False), + epub_skip_frontmatter=not getattr(args, "epub_no_skip_frontmatter", False), + keep_frontmatter=getattr(args, "keep_frontmatter", False), + page_range=page_range, + strip_patterns=strip_patterns, + strip_pattern_timeout=strip_pattern_timeout, + strip_urls=getattr(args, "strip_urls", "keep"), + quality_presignal=quality_presignal, + **sanity_kwargs, + ) + except StripPatternError as exc: + # Phase 15 Wave 2: a ReDoS-prone --strip-pattern aborts the run + # before any I/O. StripPatternError is a ValueError subclass so it + # would be caught by the generic block below; routed here first so + # the error message says "strip-pattern" specifically, not "ingest + # failed". Exit code stays EXIT_CONFIG_ERROR per the contract. + _emit_error_and_exit( + exc, + output_format=output_format, + exit_code=EXIT_CONFIG_ERROR, + log_prefix="Strip-pattern rejected: %s", ) except ( FileNotFoundError, # NOSONAR — OSError subclass; listed explicitly so the error type is visible to readers @@ -56,11 +201,12 @@ def _run_ingest_cmd(args, output_format: str) -> None: # through with a confusing traceback. ValueError stays first because # ingest_path raises it for invalid chunking parameters before any # filesystem access. - if output_format == "json": - print(json.dumps({"success": False, "error": str(exc)})) - else: - logger.error("Ingest failed: %s", exc) - sys.exit(EXIT_CONFIG_ERROR) + _emit_error_and_exit( + exc, + output_format=output_format, + exit_code=EXIT_CONFIG_ERROR, + log_prefix="Ingest failed: %s", + ) except OptionalDependencyError as exc: # Catch the narrow optional-extras subclass only. Plain ``ImportError`` # would mask genuine bugs (e.g. an internal forgelm import error) under @@ -69,11 +215,7 @@ def _run_ingest_cmd(args, output_format: str) -> None: # a missing optional extra is a *runtime* failure of the dispatched # feature, not a config validation failure — exit with # EXIT_TRAINING_ERROR so CI/CD retry logic treats it the same way. - if output_format == "json": - print(json.dumps({"success": False, "error": str(exc)})) - else: - logger.error("%s", exc) - sys.exit(EXIT_TRAINING_ERROR) + _emit_error_and_exit(exc, output_format=output_format, exit_code=EXIT_TRAINING_ERROR) if output_format == "json": print( @@ -89,6 +231,13 @@ def _run_ingest_cmd(args, output_format: str) -> None: "pii_redaction_counts": result.pii_redaction_counts, "secrets_redaction_counts": result.secrets_redaction_counts, "pdf_header_footer_lines_stripped": result.pdf_header_footer_lines_stripped, + # Phase 15 additions — additive fields, no rename of any + # pre-Phase-15 key so v0.5 consumers keep parsing. + "pdf_paragraph_packed_lines_stripped": result.pdf_paragraph_packed_lines_stripped, + "script_sanity_triggered": result.script_sanity_triggered, + "strip_pattern_substitutions": result.strip_pattern_substitutions, + "urls_handled": result.urls_handled, + "frontmatter_pages_dropped": result.frontmatter_pages_dropped, "notes": result.extra_notes, "notes_structured": result.notes_structured, }, diff --git a/forgelm/ingestion.py b/forgelm/ingestion.py index d5520481..8087a898 100644 --- a/forgelm/ingestion.py +++ b/forgelm/ingestion.py @@ -36,6 +36,18 @@ from pathlib import Path from typing import Any, Callable, Dict, Iterable, List, Optional, Tuple +from ._pypdf_normalise import DEFAULT_PROFILE as _DEFAULT_NORMALISE_PROFILE +from ._pypdf_normalise import apply_profile as _apply_normalise_profile +from ._pypdf_normalise import count_substitutions as _count_normalise_substitutions +from ._script_sanity import ( + DEFAULT_THRESHOLD as _DEFAULT_SCRIPT_SANITY_THRESHOLD, +) +from ._script_sanity import ( + ScriptSanityReport, + check_script_sanity, +) +from ._strip_pattern import apply_strip_patterns as _apply_strip_patterns + logger = logging.getLogger("forgelm.ingestion") @@ -59,6 +71,23 @@ class OptionalDependencyError(ImportError): """ +class IngestParameterError(ValueError): + """Raised when an operator-supplied parameter is invalid against a real file. + + Phase 15 introduced page-range / strip-pattern / page-range-vs-page-count + validators that run *inside* the per-file extractor — the existing + soft-fail catch in :func:`_extract_text_for_ingest` would otherwise log + the failure and continue with the rest of the corpus, hiding the + operator's mistake. ``IngestParameterError`` propagates through the + soft-fail catch so the CLI dispatcher can translate it to + ``EXIT_CONFIG_ERROR`` per the contract. + + Subclasses :class:`ValueError` so existing call sites (and the CLI's + documented exception-handling shape) keep working without an explicit + ``except`` for this subclass. + """ + + CHUNK_STRATEGIES: Tuple[str, ...] = ("sliding", "paragraph", "markdown", "semantic") # Validation messages — pinned as module constants so ruff / SonarCloud @@ -82,6 +111,15 @@ class IngestionResult: so machine-driven pipelines do not need to regex-match the prose. The structured payload is stable across releases; the free-text list is for human consumption and may rephrase the same facts. + + Phase 15 added: + + * ``pdf_paragraph_packed_lines_stripped`` — second-pass header dedup + count surfaced for parity with the page-level + :attr:`pdf_header_footer_lines_stripped`. + * ``script_sanity_triggered`` / ``strip_pattern_substitutions`` / + ``urls_handled`` / ``frontmatter_pages_dropped`` — + operator-facing counts for the new Wave 1 / Wave 2 behaviours. """ output_path: Path @@ -95,6 +133,104 @@ class IngestionResult: extra_notes: List[str] = field(default_factory=list) notes_structured: Dict[str, Any] = field(default_factory=dict) pdf_header_footer_lines_stripped: int = 0 + pdf_paragraph_packed_lines_stripped: int = 0 + script_sanity_triggered: int = 0 + strip_pattern_substitutions: int = 0 + urls_handled: int = 0 + frontmatter_pages_dropped: int = 0 + + +# --------------------------------------------------------------------------- +# Phase 15 — extraction context + post-extract normalisations +# --------------------------------------------------------------------------- + + +# Default skip-list for EPUB items the operator almost never wants to +# train on (nav, cover, copyright, colophon, title-page, front-matter). +# Implemented as a tuple of normalised lower-case tokens so a file named +# ``cover.xhtml`` or an ``epub:type="nav cover-image"`` declaration both +# match via the whole-token splitter (``_EPUB_NAME_TOKEN_SPLIT``) without +# per-source spelling drift. +# +# Round-5 review (S-D): the standalone token ``"toc"`` was removed +# because it is short and common enough that a legitimate chapter +# filename can contain it as a sub-word (``historical_toc.xhtml``, +# ``the_toc_of_war.xhtml``). The canonical EPUB-3 navigation document +# always carries the ``"nav"`` property AND is named ``nav.xhtml`` / +# ``navigation.xhtml`` in practice; the ``"nav"`` token catches both +# without the false-positive surface ``"toc"`` introduced. Operators +# on EPUBs whose navigation file is literally named ``toc.xhtml`` (and +# does not carry the ``nav`` property) can add the token back via the +# library ``epub_skip_items`` ctx kwarg. +_DEFAULT_EPUB_SKIP_ITEMS: Tuple[str, ...] = ( + "nav", + "cover", + "copyright", + "colophon", + "titlepage", + "frontmatter", +) + + +# ``[ \t]*`` not ``\s*``: callers strip linewise so the ``\s`` form +# would over-match newlines and tab into the YAML body. Anchored on +# both sides so it stays a state-machine-friendly probe rather than +# the regex we apply to the body itself. +_YAML_FRONTMATTER_PATTERN = re.compile(r"\A---[ \t]*\n.*?\n---[ \t]*\n", re.DOTALL) + + +# Operator-facing URL pattern for ``--strip-urls`` (Phase 15 Task 14). +# The character class is bounded and non-overlapping, so an unbounded +# ``+`` does not backtrack catastrophically — there is no greedy/lazy +# interaction across a shared character set. Dropping the original +# ``{1,2048}`` upper bound (review round-2 S-5) keeps a 3-KB URL from +# being truncated mid-string and left as a partial residue in strip +# mode; URL handling now consumes the URL to its natural whitespace / +# punctuation boundary in one match. ``A-Z`` only (no ``a-z``) under +# IGNORECASE — Sonar python:S5869 false-positive otherwise. +_URL_PATTERN = re.compile(r"\b(?:https?|ftp)://[A-Z0-9._~:/?#\[\]@!$&'()*+,;=%-]+", re.IGNORECASE) + + +@dataclass +class _ExtractContext: + """Mutable context threaded through the per-file extraction path. + + Bundles every Phase 15 knob so individual extractors do not balloon + in argument count. Mutable in three places only: + + * ``dedup_state`` — counters accumulated from PDF header / footer + dedup (existing pre-Phase-15 behaviour). + * ``script_sanity_reports`` — list of :class:`ScriptSanityReport` + objects produced by every per-file sanity check (one entry per + file, triggered or not). + * ``frontmatter_dropped_pages`` — indices of PDF front-matter pages + the heuristic dropped, for inclusion in the structured notes. + + Everything else is read-only after construction. + """ + + dedup_state: Optional[Dict[str, int]] = None + page_range: Optional[Tuple[int, int]] = None # 1-indexed inclusive + normalise_profile: str = _DEFAULT_NORMALISE_PROFILE + language_hint: Optional[str] = None + script_sanity_threshold: float = _DEFAULT_SCRIPT_SANITY_THRESHOLD + script_sanity_reports: List[ScriptSanityReport] = field(default_factory=list) + keep_md_frontmatter: bool = False + epub_skip_frontmatter: bool = True + epub_skip_items: Tuple[str, ...] = _DEFAULT_EPUB_SKIP_ITEMS + # Phase 15 Wave 2 Task 13: front-matter / back-matter heuristic is DEFAULT ON + # in v0.6.0 (audit recommended; opt-out via --keep-frontmatter to preserve + # the pre-Phase-15 "keep every page" behaviour). The library-level default + # also moves from True to False so a library caller without the kwarg sees + # the new behaviour — operators who freeze a script + .yaml + .py can + # restore the pre-15 behaviour with one explicit kwarg. + keep_frontmatter: bool = False + strip_patterns: List[Tuple[str, "re.Pattern[str]"]] = field(default_factory=list) + strip_pattern_timeout: Optional[int] = 5 + strip_urls_mode: str = "keep" # keep | mask | strip + frontmatter_dropped_pages: List[int] = field(default_factory=list) + urls_handled_total: int = 0 + strip_pattern_substitutions_total: int = 0 # --------------------------------------------------------------------------- @@ -104,53 +240,165 @@ class IngestionResult: _PDF_REPEAT_MIN_PAGES: int = 3 _PDF_REPEAT_THRESHOLD: float = 0.7 +_PDF_EDGE_WINDOW: int = 3 +"""Phase 15 Task 1 — number of rows from each page edge inspected per pass. + +The pre-Phase-15 implementation only considered the **topmost** / **bottommost** +line per page on each pass. That worked for the homogeneous single-line +header case but silently exited the dedup loop the moment the outermost +row varied per page (e.g. a per-section title on top, a page number on +the bottom): no top/bottom recurrence at the cutoff → exit pass 1 → +never reach the constant publication-identifier line sitting one row +deeper. The audit (§1.1, 2026-05-11) traced 74/82 of a Turkish PDF's +chunks back to exactly this trap. + +Widening the inspection to the top-3 / bottom-3 rows per page lets a +line recurring at **any** position within that window get stripped in +the same pass, so a variable-outer-line corpus no longer locks the +deeper-row constant out. Three is conservative: real-world page edges +rarely carry more than a 2-line running header + a 1-line page number +or copyright row, so 3 catches the modal case while leaving paragraph +text alone (a paragraph that legitimately re-appears at row 4+ across +≥ 70 % of pages is itself almost-certainly boilerplate worth flagging, +but we do not strip it because that is outside the documented header +/ footer contract). +""" -def _repeating_edge_lines(page_lines: List[List[str]], cutoff: int) -> Tuple[set, set]: - """Return (repeating_first_lines, repeating_last_lines) at this pass.""" - first_counts: Counter = Counter(lines[0] for lines in page_lines if lines) - last_counts: Counter = Counter(lines[-1] for lines in page_lines if lines) +def _windowed_repeating_edge_lines( + page_lines: List[List[str]], + cutoff: int, + window: int = _PDF_EDGE_WINDOW, +) -> Tuple[set, set]: + """Return the set of lines recurring near each page's top / bottom edge. + + For each page, inspects up to ``window`` lines from the start and + from the end (de-duplicated per page so a 1-line page does not + inflate the count). A line counted at any offset within the window + is treated as "edge-class" — when it recurs in ≥ ``cutoff`` pages it + is eligible to be stripped, no matter whether it lived at the + absolute outermost slot on every page or shifted by one row when + the outer slot was occupied by a variable-content line. + """ + first_counts: Counter = Counter() + last_counts: Counter = Counter() + for lines in page_lines: + if not lines: + continue + # ``set`` de-dupes inside a single page so a short page that + # happens to repeat a line wouldn't drive up the per-page count + # past the integer 1. + first_counts.update(set(lines[:window])) + last_counts.update(set(lines[-window:])) return ( {ln for ln, n in first_counts.items() if n >= cutoff}, {ln for ln, n in last_counts.items() if n >= cutoff}, ) -def _pop_repeating_edges( +def _pop_one_page_window( + lines: List[str], + repeating_firsts: set, + repeating_lasts: set, + window: int, +) -> Tuple[List[str], int]: + """Per-page worker for :func:`_pop_windowed_edges`. + + Round-3 review (CodeRabbit) — the head walk inspects every + position in the top-N window and drops any line whose stripped + form is in ``repeating_firsts``, but the pre-round-3 tail walk + peeled consecutively from the very end and stopped at the first + non-match. That asymmetry let a *constant deeper bottom-edge* + line survive when the outermost last line varied — the audit + §1.1 trap re-occurring at the bottom of the page (variable page + number on the last line, constant footer one row deeper): + + .. code-block:: text + + page N: [..., "Body N.", "FOOTER", "N"] + + With the outermost line varying ("1" / "2" / ...) the asymmetric + tail walk would never reach the constant ``"FOOTER"`` one row + deeper. The symmetric walk below mirrors the head walk so a + bottom-edge constant line at any offset within the window is + stripped in a single pass. + """ + kept: List[str] = [] + dropped = 0 + for idx, ln in enumerate(lines): + if idx < window and ln in repeating_firsts: + dropped += 1 + continue + kept.append(ln) + # Symmetric tail walk: drop any line within the last ``window`` + # positions whose stripped form is in ``repeating_lasts``. Walking + # in reverse so ``rev_idx`` mirrors the head walk's ``idx``. + surviving: List[str] = [] + for rev_idx, ln in enumerate(reversed(kept)): + # rev_idx == 0 is the last line; rev_idx < window covers the + # bottom-N window inclusive. + if rev_idx < window and ln in repeating_lasts: + dropped += 1 + continue + surviving.append(ln) + # ``surviving`` is in reversed order; flip to restore page order. + kept = list(reversed(surviving)) + return kept, dropped + + +def _pop_windowed_edges( page_lines: List[List[str]], repeating_firsts: set, repeating_lasts: set, + window: int = _PDF_EDGE_WINDOW, ) -> int: - """Pop the leading / trailing repeating line from each page; return total stripped.""" + """Drop matching lines anywhere in the top-N / bottom-N window of each page. + + Returns the total number of lines removed across all pages so the + caller can roll the count into structured ingestion notes for + downstream visibility. Pages that end up empty after the pop are + left as empty lists (the caller filters them out at the join step). + Per-page work is delegated to :func:`_pop_one_page_window`. + """ stripped = 0 for lines in page_lines: - if lines and lines[0] in repeating_firsts: - lines.pop(0) - stripped += 1 - if lines and lines[-1] in repeating_lasts: - lines.pop() - stripped += 1 + if not lines: + continue + kept, dropped = _pop_one_page_window(lines, repeating_firsts, repeating_lasts, window) + stripped += dropped + lines[:] = kept return stripped def _strip_repeating_page_lines(pages: List[str]) -> Tuple[List[str], int]: - """Strip leading / trailing lines that repeat across pages. - - Page-level headers (company watermark, document title) and footers - (page number text, copyright line) end up as the first / last line - of every page after :func:`_extract_pdf` and inflate near-duplicate - counts during the audit. We iterate: at each pass we collect the - first and last non-empty line of every page, find lines that recur - in ≥ 70 % of pages (default), and pop those from the start / end of - every page. The pass repeats until no more lines meet the threshold, - so multi-line headers (e.g. ``Line 1: company name`` followed by - ``Line 2: CONFIDENTIAL``) are stripped fully rather than leaving the - second line stranded as a new "first line" that no longer matches - the original count. + """Strip page-edge lines that repeat across the document. + + Phase 15 Task 1 (audit §1.1): + The pre-Phase-15 implementation collected only ``lines[0]`` and + ``lines[-1]`` per pass and looped until no more recurrence met + the 70 % cutoff. The mechanism was correct in principle (it did + iterate) but the loop's exit condition killed it on a corpus + with a *variable* outer line — a per-chapter section title on + top, an incrementing page number on the bottom — because pass 1 + found no recurrence at the outermost slot and broke before + reaching the deeper-row constant line that ought to have been + stripped on pass 2. Widening the inspection to the top-N / + bottom-N rows per page (default :data:`_PDF_EDGE_WINDOW` = 3) + catches a line that recurs at any position within the window + in a single pass. + + Why ``iterative-peel alone is not enough``: + The same comment is duplicated inline above the loop body so a + future implementer reading just the docstring cannot reintroduce + the regression by collapsing the window back to 1. The bug is + the loop's *exit condition*, not the outer-most-line check — a + casual reviewer can easily mis-read "we already iterate" as + "the algorithm is correct" and patch only the symptom. Returns: - ``(cleaned_pages, lines_stripped)``. Caller can roll the count - into structured ingestion notes for downstream visibility. + ``(cleaned_pages, lines_stripped)``. Empty pages (after stripping) + are filtered out at the join step so they do not pollute the + downstream paragraph chunker with phantom blank pages. The dedup is a no-op on documents with fewer than 3 pages — the statistical signal is too weak to distinguish "header" from @@ -166,15 +414,21 @@ def _strip_repeating_page_lines(pages: List[str]) -> Tuple[List[str], int]: cutoff = max(2, math.ceil(_PDF_REPEAT_THRESHOLD * len(pages))) total_stripped = 0 + # Pass 1 / pass 2 / … : window-based detection peels one layer per + # pass. Iterating is still needed because once the outermost lines + # are stripped the next-deepest row of the next pass might now sit + # at offset 0 and meet the cutoff at the same window, but the + # iteration is no longer *gated* on the strict-outermost recurrence — + # so a variable-outer-line corpus is no longer locked out (audit §1.1). while True: - repeating_firsts, repeating_lasts = _repeating_edge_lines(page_lines, cutoff) + repeating_firsts, repeating_lasts = _windowed_repeating_edge_lines(page_lines, cutoff) if not repeating_firsts and not repeating_lasts: break - stripped = _pop_repeating_edges(page_lines, repeating_firsts, repeating_lasts) + stripped = _pop_windowed_edges(page_lines, repeating_firsts, repeating_lasts) if stripped == 0: - # Edge case: the only repeating line was already alone on a - # short page, so popping it left fewer pages than ``cutoff`` - # — break instead of looping with no work. + # Defensive break: no edge in the current window matched + # despite the set being non-empty (e.g. every match was + # already at depth > window). Avoid an infinite loop. break total_stripped += stripped @@ -182,6 +436,139 @@ def _strip_repeating_page_lines(pages: List[str]) -> Tuple[List[str], int]: return cleaned, total_stripped +def strip_paragraph_packed_headers(text: str, *, threshold: float = _PDF_REPEAT_THRESHOLD) -> Tuple[str, int]: + """Second-pass dedup against text already glued into paragraph blocks. + + Phase 15 Task 1, second part: after the page-level dedup runs and + the paragraph chunker greedy-packs the result, a header line that + survived (e.g. because it sat at row 4 on some pages and was + therefore outside the window) can still re-appear at the start of + several chunks. This helper looks at the first line of each + ``\\n\\n``-separated block and strips any line that recurs in + ``threshold`` fraction of blocks. Returns ``(cleaned_text, n_stripped)``. + + Operates on already-extracted text, so it never blocks on I/O and + has no PDF-specific dependency. Skipping it would leave the + Task 1 acceptance criterion fragile: the page-level dedup catches + *most* runs but the audit recommends a second pass after chunker + packing to mop up the residue. + """ + blocks = text.split("\n\n") + if len(blocks) < _PDF_REPEAT_MIN_PAGES: + return text, 0 + first_lines: List[str] = [] + for block in blocks: + block = block.strip() + if not block: + first_lines.append("") + continue + first_lines.append(block.splitlines()[0].strip()) + counts = Counter(ln for ln in first_lines if ln) + cutoff = max(2, math.ceil(threshold * len([ln for ln in first_lines if ln]))) + repeating = {ln for ln, n in counts.items() if n >= cutoff} + if not repeating: + return text, 0 + + stripped = 0 + rebuilt: List[str] = [] + for block in blocks: + if not block.strip(): + rebuilt.append(block) + continue + block_lines = block.splitlines() + # Drop any leading lines that match the repeating set; this + # mops up the survivor headers that the page-level pass missed. + idx = 0 + while idx < len(block_lines) and block_lines[idx].strip() in repeating: + stripped += 1 + idx += 1 + rebuilt.append("\n".join(block_lines[idx:])) + return "\n\n".join(rebuilt), stripped + + +def _post_extract_normalise(text: str, ctx: _ExtractContext, file_path: Path) -> str: + """Apply Phase 15 post-extract transforms in a deterministic order. + + Order matters and is documented as a Phase 15 contract: + + 1. **Glyph normalisation** (Task 3) — fix pypdf font-fallback + artefacts before any other pass sees them. If the script-sanity + check sees the corrupt glyphs first the warning would fire + loudly even when the normaliser would have repaired them in the + next step. + 2. **Script-sanity check** (Task 2) — run after normalisation so + the warning fires only on residual corruption. The report is + appended to ``ctx.script_sanity_reports`` whether or not it + triggered, so the structured notes can record "we checked". + 3. **Operator strip-patterns** (Wave 2 Task 11) — applied before + URL handling because a pattern might legitimately delete a + URL-bearing line wholesale. + 4. **URL handling** (Wave 2 Task 14) — mask / strip per + ``ctx.strip_urls_mode``. + + Front-matter heuristic (Task 13) and PDF-specific multi-column + detection (Task 15) live inside :func:`_extract_pdf` itself + because they need pypdf state and run before this post-extract + pass. + """ + if not text: + return text + + # 1. Glyph normalisation. The function no-ops when profile == "none". + # Phase 15 round-1 review (C-2): when the profile is active and + # actually rewrites characters, surface the count at INFO so an + # operator who didn't intend to apply a Turkish profile to a + # Norwegian / Estonian corpus has a single log line to spot the + # mismatch. The check compares character identity rather than full + # text equality so a long-document round-trip stays cheap. + if ctx.normalise_profile and ctx.normalise_profile != "none": + # Round-3 review: use the exact-count helper from + # ``_pypdf_normalise`` instead of the previous zip-diff heuristic, + # which over- and under-counted on multi-char rule boundaries. + substitutions = _count_normalise_substitutions(text, ctx.normalise_profile) + normalised = _apply_normalise_profile(text, ctx.normalise_profile) + if substitutions: + logger.info( + "Normalisation profile %r applied %d substitution(s) on '%s'. " + "Pass --no-normalise-unicode or --normalise-profile none to disable.", + ctx.normalise_profile, + substitutions, + file_path, + ) + text = normalised + + # 2. Script-sanity check. Always invoked when a language hint is + # configured; the helper itself silently no-ops on unknown hints so + # the operator can pass an unsupported code without crashing. + if ctx.language_hint: + report = check_script_sanity( + text, + language_hint=ctx.language_hint, + file_path=str(file_path), + threshold=ctx.script_sanity_threshold, + profile=ctx.normalise_profile, + ) + ctx.script_sanity_reports.append(report) + + # 3. Operator strip-patterns. + if ctx.strip_patterns: + text, subs = _apply_strip_patterns( + text, + ctx.strip_patterns, + timeout_s=ctx.strip_pattern_timeout, + ) + ctx.strip_pattern_substitutions_total += subs + + # 4. URL handling. + if ctx.strip_urls_mode in ("mask", "strip"): + replacement = "[URL]" if ctx.strip_urls_mode == "mask" else "" + new_text, n = _URL_PATTERN.subn(replacement, text) + text = new_text + ctx.urls_handled_total += n + + return text + + def _try_pdf_decrypt(reader: Any, path: Path) -> None: """Attempt empty-password decrypt for owner-encrypted PDFs; raise ValueError otherwise. @@ -205,14 +592,19 @@ def _try_pdf_decrypt(reader: Any, path: Path) -> None: result = reader.decrypt("") except (FileNotDecryptedError, NotImplementedError, DependencyError) as exc: raise ValueError( - f"PDF '{path}' is encrypted. Decrypt it first (qpdf --decrypt / pdftk input_pw …) and re-run ingest." + f"PDF {path!r} is encrypted. Decrypt it first (qpdf --decrypt / pdftk input_pw …) and re-run ingest." ) from exc if not result: # PasswordType.NOT_DECRYPTED == 0 → empty password didn't unlock it. - raise ValueError(f"PDF '{path}' is encrypted with a non-empty password. Decrypt externally before ingest.") + raise ValueError(f"PDF {path!r} is encrypted with a non-empty password. Decrypt externally before ingest.") -def _read_pdf_pages(reader: Any, path: Path) -> List[str]: +def _read_pdf_pages( + reader: Any, + path: Path, + *, + page_range: Optional[Tuple[int, int]] = None, +) -> List[Tuple[int, str]]: """Extract per-page text; tolerate page-level failures with a loud warning. Page-level extraction failure is **not** propagated as a file-level @@ -222,9 +614,29 @@ def _read_pdf_pages(reader: Any, path: Path) -> List[str]: exactly which page failed and can spot-check it. The post-loop "no extractable text" warning still catches the all-pages-failed case downstream in :func:`_extract_pdf`. + + ``page_range`` (1-indexed inclusive) lets the caller restrict + extraction to a subset of pages. The validator in + :func:`_validate_page_range` enforces the bounds before we reach + this helper; here we only slice. Returns ``(page_index, text)`` pairs + so downstream callers (front-matter heuristic, multi-column warning) + can refer to the original page numbers in their reporting even after + the slice. """ - pages: List[str] = [] - for idx, page in enumerate(reader.pages): + total = len(reader.pages) + if page_range is not None: + start_1idx, end_1idx = page_range + # Clamp end to total so a range of 5-9999 on a 200-page doc + # silently shrinks to 5-200; the explicit out-of-range error + # comes from _validate_page_range, which runs at CLI dispatch. + end_1idx = min(end_1idx, total) + indices = range(start_1idx - 1, end_1idx) + else: + indices = range(total) + + pages: List[Tuple[int, str]] = [] + for idx in indices: + page = reader.pages[idx] try: text = page.extract_text() or "" except Exception as exc: # noqa: BLE001 — best-effort: pypdf's per-page extraction surface is wide (KeyError on malformed object refs, AssertionError on broken cross-ref tables, UnicodeDecodeError on font encodings, plus its own internal errors); per-page soft-fail keeps the run going so a single bad page cannot abort a multi-thousand-document corpus ingest. # NOSONAR @@ -236,11 +648,298 @@ def _read_pdf_pages(reader: Any, path: Path) -> List[str]: ) text = "" if text.strip(): - pages.append(text) + pages.append((idx, text)) return pages -def _extract_pdf(path: Path, *, dedup_state: Optional[Dict[str, int]] = None) -> str: +def _validate_page_range(page_range: Tuple[int, int], total_pages: int, path: Path) -> Tuple[int, int]: + """Validate ``page_range`` against ``total_pages``. + + Raises :class:`IngestParameterError` (a :class:`ValueError` subclass) + with an operator-facing message on failure. The CLI dispatcher + translates this to ``EXIT_CONFIG_ERROR`` per the existing per- + subcommand pattern. Library callers see the raw exception, in line + with the documented contract. + + Why :class:`IngestParameterError` and not plain :class:`ValueError`: + the per-file extractor's soft-fail catch in + :func:`_extract_text_for_ingest` swallows generic exceptions to keep + a multi-thousand-file corpus running when one bad page or one + locked file is in the way. An operator-supplied parameter mistake + is a different class of failure — it should abort the run loudly + rather than fold into the log. The custom subclass lets the catch + re-raise this one specifically. + """ + start, end = page_range + if start < 1: + raise IngestParameterError(f"--page-range start ({start}) must be >= 1 for {path!r} (1-indexed).") + if start > end: + raise IngestParameterError(f"--page-range start ({start}) must be <= end ({end}) for {path!r}.") + if start > total_pages: + raise IngestParameterError(f"--page-range start ({start}) exceeds PDF page count ({total_pages}) for {path!r}.") + return (start, min(end, total_pages)) + + +# --------------------------------------------------------------------------- +# Phase 15 Task 13 — front-matter heuristic for PDFs +# --------------------------------------------------------------------------- + +_FRONTMATTER_PROBE_PAGES: int = 12 +"""Number of leading / trailing pages probed by the front-matter heuristic. + +The 12-page envelope matches the audit's empirical observation: ToC / +front-matter material in a 200-page corpus rarely exceeds the first +dozen pages, and the symmetric trailing pass catches index / glossary +back-matter under the same envelope. Configurable in future phases if +operator feedback shifts the modal value. +""" + +_FRONTMATTER_ALPHA_RATIO_MAX: float = 0.30 +"""Alpha threshold for the front-matter heuristic. + +Round-3-follow-up review (S-E) tightened this from 0.45 → 0.30. The +**primary** protection against form-template / exercise-page +false-positives is **not** the alpha threshold (a constructed short- +label form template + inline page numbers measures alpha ≈ 0.18, +BELOW 0.30) but the **3-signal AND filter**: realistic form templates +have NO inline ``\\n\\n`` page-number matches, so +``page_num_hits >= 5`` fails first and the page is kept. The +0.45 → 0.30 tightening narrows the band of false-positive pages that +ALSO carry synthetic-looking inline number sequences (a thin slice of +edge cases) but is not the dominant defence. + +Side-effect of the 0.30 tightening: realistic English-language ToCs +with full-sentence chapter titles (``Chapter 1: Introduction to the +Subject ……… 14``) measure alpha ≈ 0.47 and now PASS THROUGH the +heuristic. The heuristic remains calibrated for the audit's pilot +shape (Turkish single-word chapter titles + heavy dotted leaders + +inline page numbers, alpha ≈ 0.15). Operators on long-title ToC +corpora should use ``--strip-pattern`` / ``--page-range`` as a manual +fallback. Documented in ``docs/guides/ingestion.md``. +""" +_FRONTMATTER_LEADER_RATIO_MIN: float = 0.10 +_FRONTMATTER_PAGE_NUM_PATTERN = re.compile(r"\n\d{1,3}\n") +# Round-3 review: ToC pages commonly use **dotted** leaders (`.....`) +# OR underscore leaders (`____`). A bare-underscore-only count missed +# the dot-leader case on real-world publications, leaving genuine ToC +# pages slipping past the heuristic. Combine both leader styles into a +# single regex; runs of length ≥ 3 reliably distinguish leaders from +# legitimate punctuation. +_FRONTMATTER_LEADER_RUN_PATTERN = re.compile(r"[._]{3,}") + + +def _is_frontmatter_page(text: str) -> bool: + """Apply the audit's three-signal heuristic: low alpha + high leader ratio + many page-numbers. + + A page is treated as "front-matter candidate" iff **all three** + conditions hold: + + * Alphabetic-character ratio < 0.45. + * Leader-character ratio > 0.10. Round-3 fix: the leader count + now covers BOTH underscore runs (``_____``) AND dot runs + (``.....``) of length ≥ 3, denominated over the non-whitespace + character count (parity with ``alpha_ratio``). Pre-round-3 the + check only counted single ``_`` characters against the raw text + length, which missed the dotted-leader ToC pages that the + docstring already promised to handle. + * At least 5 inline-page-number matches (``\\n<1-3 digits>\\n``). + + Three independent signals keep the false-positive rate low — body + text rarely meets even two of them simultaneously. + """ + if not text: + return False + non_ws = [c for c in text if not c.isspace()] + if not non_ws: + return False + alpha_ratio = sum(1 for c in non_ws if c.isalpha()) / len(non_ws) + leader_chars = sum(len(m) for m in _FRONTMATTER_LEADER_RUN_PATTERN.findall(text)) + leader_ratio = leader_chars / len(non_ws) + page_num_hits = len(_FRONTMATTER_PAGE_NUM_PATTERN.findall(text)) + return ( + alpha_ratio < _FRONTMATTER_ALPHA_RATIO_MAX + and leader_ratio > _FRONTMATTER_LEADER_RATIO_MIN + and page_num_hits >= 5 + ) + + +def _drop_frontmatter_pages( + pages: List[Tuple[int, str]], + probe: int = _FRONTMATTER_PROBE_PAGES, +) -> Tuple[List[Tuple[int, str]], List[int]]: + """Drop leading + trailing front-matter pages; return the remainder + dropped indices. + + Symmetric pass: the first ``probe`` pages are dropped from the + head while they keep matching the heuristic; the last ``probe`` + pages similarly from the tail. Stops at the first non-matching + page in each direction so a real body page can never be dropped + mid-document. + """ + if not pages: + return pages, [] + dropped: List[int] = [] + head_keep_idx = 0 + for pg_idx, (orig_idx, text) in enumerate(pages[:probe]): + if _is_frontmatter_page(text): + dropped.append(orig_idx) + head_keep_idx = pg_idx + 1 + else: + break + tail_keep_idx = len(pages) + for offset in range(min(probe, len(pages) - head_keep_idx)): + pg = pages[-(offset + 1)] + orig_idx, text = pg + if _is_frontmatter_page(text): + dropped.append(orig_idx) + tail_keep_idx = len(pages) - (offset + 1) + else: + break + remaining = pages[head_keep_idx:tail_keep_idx] + return remaining, sorted(set(dropped)) + + +# --------------------------------------------------------------------------- +# Phase 15 Task 15 — multi-column PDF detection (warning only) +# --------------------------------------------------------------------------- + +_MULTI_COLUMN_SEPARATION_PCT: float = 0.30 +"""Fraction-of-page-width gap between text clusters that flags 2-column. + +Round-2 review (nit on ingestion.py:700-758) proposed lowering this from +0.30 → 0.08 to match real-world 5-8 %-of-page-width gutters. After +investigation we **kept the 0.30 threshold** because the current +detector's cluster split is computed against the min / max of all +extracted text-matrix x-offsets, NOT against per-line start positions. +pypdf's ``visitor_text`` callback fires per-glyph with the running text +matrix, so a single-line "Body content here." emits x-positions spanning +the line's horizontal extent (e.g. 72 → 250 on a US-letter page) — the +min / max gap on a single-column page can already exceed 25 %, well +above the proposed 8 % threshold. A genuine 2-column page has its +right-column glyphs starting near the page midline (≈ 300 / 612 ≈ 50 %) +so 0.30 catches it without false-positiving on single-column body text. + +A histogram-based bimodal-mode detector would let us safely drop the +threshold below 0.10; that refactor is out of scope for the Phase 15 +round-2 fix and is tracked as a Wave 3 follow-up (see +`docs/roadmap/phase-15-ingestion-reliability.md` Wave 3 — multi-column +layout extraction). +""" + + +def _select_sample_pages(reader: Any, page_range: Optional[Tuple[int, int]]) -> List[Any]: + """Slice the first three text-bearing pages for the multi-column probe. + + Honours ``--page-range`` when set so the warning reflects the real + extraction surface (round-2 review nit). Returns an empty list on + any slicing failure — the probe degrades to silence rather than + blocking the run. + """ + try: + if page_range is not None: + start_1idx, end_1idx = page_range + return reader.pages[start_1idx - 1 : end_1idx][:3] + return reader.pages[:3] + except Exception: # noqa: BLE001 — pypdf wrapper objects may raise on slice on torn PDFs. + return [] + + +def _collect_multi_column_samples(sample_pages: List[Any]) -> Tuple[List[float], Optional[float]]: + """Drive pypdf's ``visitor_text`` callback across ``sample_pages``. + + Returns ``(x_positions, page_width)``. The page width comes from the + first page that exposes a ``mediabox``. Sampling stops once 40 + positions have been collected — enough to call multi-column with + confidence without walking a 500-page document. + """ + x_positions: List[float] = [] + page_width: Optional[float] = None + + def visitor(text: str, _cm: Any, tm: Any, _fontdict: Any, _fontsize: Any) -> None: + # ``tm`` is a 6-tuple ``(a, b, c, d, e, f)`` representing the + # text-matrix; the horizontal offset is element 4. Pypdf calls + # this with the documented kwargs (``font_dict`` / ``font_size`` + # under the historical CamelCase names) — the snake_case + # underscored names here keep Sonar python:S117 happy without + # changing the call contract. + if text and tm is not None: + try: + x_positions.append(float(tm[4])) + except (TypeError, IndexError, ValueError): + return + + for page in sample_pages: + try: + page.extract_text(visitor_text=visitor) + if page_width is None and getattr(page, "mediabox", None) is not None: + page_width = float(page.mediabox.width) + except Exception: # noqa: BLE001 — sampling is best-effort; per-page failures from torn PDFs / unusable font tables should not block the multi-column probe. # nosec B112 + continue + if len(x_positions) >= 40: + break + return x_positions, page_width + + +def _is_two_cluster_distribution(x_positions: List[float], page_width: float) -> bool: + """Return ``True`` iff the x-positions split into two clusters past the threshold.""" + if not x_positions or page_width <= 0: + return False + sorted_xs = sorted(x_positions) + midpoint = (sorted_xs[0] + sorted_xs[-1]) / 2.0 + left_cluster_max = max((x for x in sorted_xs if x <= midpoint), default=midpoint) + right_cluster_min = min((x for x in sorted_xs if x > midpoint), default=midpoint) + gap = right_cluster_min - left_cluster_max + return gap > _MULTI_COLUMN_SEPARATION_PCT * page_width + + +def _maybe_warn_multi_column(reader: Any, path: Path, *, page_range: Optional[Tuple[int, int]] = None) -> bool: + """Sample the first text-bearing pages for a two-cluster x-distribution. + + Walks pypdf's text fragments via the ``visitor_text`` callback, + accumulates per-fragment x-coordinates, and reports a two-cluster + distribution as a single ``WARNING``. No fix attempt — the + operator is expected to switch to ``--strategy sliding`` with a + larger chunk-size or to pre-process the PDF with a layout-aware + tool. Returns ``True`` iff a warning was emitted (test seam). + + Round-2 review (nit on ingestion.py:803): when ``--page-range`` is + set the warning previously sampled pages 0-2 even if those pages + were going to be dropped from the actual extraction; we now respect + the same slice the extractor will use so the warning reflects the + real run, not the file's first three pages. + """ + sample_pages = _select_sample_pages(reader, page_range) + if not sample_pages: + return False + x_positions, page_width = _collect_multi_column_samples(sample_pages) + if page_width is None: + return False + if not _is_two_cluster_distribution(x_positions, page_width): + return False + logger.warning( + "Detected 2-column layout in '%s' — reading order may be scrambled. " + "Consider --strategy sliding with a larger --chunk-size, or pre-process " + "the PDF with a layout-aware tool (camelot-py / pdfplumber).", + path, + ) + return True + + +def _extract_pdf(path: Path, *, ctx: Optional[_ExtractContext] = None) -> str: + """Extract PDF text with Phase 15's hardened pipeline. + + Phase 15 additions vs pre-15: + + * ``page_range`` filtering (Wave 2 Task 12) before per-page extraction. + * Multi-column detection warning (Wave 2 Task 15) sampled on the + first three pages of the slice. + * Front-matter heuristic drop (Wave 2 Task 13) on the first 12 / last + 12 pages of the slice, opt-out via ``ctx.keep_frontmatter``. + * Glyph-normalisation + script-sanity + strip-pattern + strip-URLs + are applied by :func:`_post_extract_normalise` after this function + returns, so this body deliberately stays PDF-specific. + """ + if ctx is None: + ctx = _ExtractContext() try: from pypdf import PdfReader except ModuleNotFoundError as exc: # pragma: no cover — covered by extras @@ -258,13 +957,19 @@ def _extract_pdf(path: Path, *, dedup_state: Optional[Dict[str, int]] = None) -> try: reader = PdfReader(str(path)) except Exception as exc: # noqa: BLE001 — best-effort: PdfReader open surfaces OSError (file/permission), pypdf-internal errors (bad header, unsupported version, malformed xref), and on rare adversarial inputs InfiniteLoopError; converting all to a typed ValueError keeps the per-file error path uniform with DOCX/EPUB extractors. # NOSONAR - raise ValueError(f"Could not open PDF '{path}': {exc}") from exc + raise ValueError(f"Could not open PDF {path!r}: {exc}") from exc if getattr(reader, "is_encrypted", False): _try_pdf_decrypt(reader, path) - pages = _read_pdf_pages(reader, path) - if not pages: + page_range = ctx.page_range + if page_range is not None: + page_range = _validate_page_range(page_range, len(reader.pages), path) + + _maybe_warn_multi_column(reader, path, page_range=page_range) + + indexed_pages = _read_pdf_pages(reader, path, page_range=page_range) + if not indexed_pages: logger.warning( "No extractable text in '%s'. Likely a scanned PDF without a text layer; " "run OCR (Tesseract / AWS Textract) before ingest.", @@ -272,12 +977,32 @@ def _extract_pdf(path: Path, *, dedup_state: Optional[Dict[str, int]] = None) -> ) return "" - cleaned_pages, stripped = _strip_repeating_page_lines(pages) - if dedup_state is not None and stripped: + if not ctx.keep_frontmatter: + indexed_pages, dropped = _drop_frontmatter_pages(indexed_pages) + if dropped: + ctx.frontmatter_dropped_pages.extend(dropped) + logger.warning( + "Dropped %d front-matter / back-matter page(s) from '%s' " + "(indices=%s). Use --keep-frontmatter to retain them.", + len(dropped), + path, + dropped, + ) + + page_texts = [text for _, text in indexed_pages] + cleaned_pages, stripped = _strip_repeating_page_lines(page_texts) + if ctx.dedup_state is not None and stripped: # Roll into the run-level structured notes so the operator can # tell post-hoc that header/footer dedup was actually doing work. - dedup_state["lines_stripped"] = dedup_state.get("lines_stripped", 0) + stripped - return "\n\n".join(cleaned_pages) + ctx.dedup_state["lines_stripped"] = ctx.dedup_state.get("lines_stripped", 0) + stripped + + joined = "\n\n".join(cleaned_pages) + joined, secondary_stripped = strip_paragraph_packed_headers(joined) + if ctx.dedup_state is not None and secondary_stripped: + ctx.dedup_state["paragraph_packed_stripped"] = ( + ctx.dedup_state.get("paragraph_packed_stripped", 0) + secondary_stripped + ) + return joined def _escape_md_cell(text: Optional[str]) -> str: @@ -355,7 +1080,102 @@ def _iter_docx_blocks(doc: Any) -> Iterable[Any]: yield Table(child, doc) -def _extract_docx(path: Path) -> str: +def _add_part_lines_to_boilerplate(part: Any, boilerplate: set) -> None: + """Drain non-empty stripped lines from a header/footer part into ``boilerplate``.""" + for paragraph in getattr(part, "paragraphs", []): + text = getattr(paragraph, "text", "") or "" + for ln in text.splitlines(): + cleaned = ln.strip() + if cleaned: + boilerplate.add(cleaned) + + +def _collect_docx_header_footer_lines(doc: Any) -> set: + """Phase 15 Task 6 — return every line declared in a section's header / footer. + + Word documents declare headers / footers explicitly under each + section's ```` / ```` parts. python-docx exposes them + as ``doc.sections[i].header.paragraphs`` and ``.footer.paragraphs``. + We collect their non-empty stripped lines into a single ``set`` so a + multi-section document with the same boilerplate on each section + only contributes one entry per unique line. Body extraction + afterwards subtracts these lines, eliminating the + "header bleeds into body chunk" failure mode the audit (§3.2) + documented. Per-section drain logic is delegated to + :func:`_add_part_lines_to_boilerplate` to keep this function below + Sonar S3776's cognitive-complexity ceiling. + + Defensive: a malformed section (no header element) raises + ``AttributeError`` from python-docx; we swallow it per-section so a + single bad section cannot kill extraction of the rest. + """ + boilerplate: set = set() + for section in getattr(doc, "sections", []): + for source_name in ("header", "footer"): + try: + part = getattr(section, source_name, None) + except AttributeError: + continue + if part is None: + continue + _add_part_lines_to_boilerplate(part, boilerplate) + return boilerplate + + +_DOCX_BOILERPLATE_MAX_LEN: int = 80 +"""Max stripped-line length eligible for boilerplate subtraction (round-2 S-4). + +A body paragraph beginning with ``Chapter 1`` is overwhelmingly longer +than its standalone-title-as-header counterpart, so the length floor +protects body text from being mistakenly dropped along with the +running-header line of the same text. +""" + + +def _strip_docx_boilerplate(text: str, boilerplate: set) -> Optional[str]: + """Return ``text`` minus header/footer boilerplate lines, or ``None`` if empty. + + Splits the paragraph into lines, drops any line that is **short + enough** to plausibly be a header (≤ 80 chars) AND exactly matches + a header / footer entry. Returns the cleaned text on success, or + ``None`` when nothing survives (caller skips the paragraph). + """ + kept = [ + ln + for ln in text.splitlines() + if not (len(ln.strip()) <= _DOCX_BOILERPLATE_MAX_LEN and ln.strip() in boilerplate) + ] + cleaned = "\n".join(kept).strip() + return cleaned or None + + +def _render_docx_block(element: Any, boilerplate: set, table_cls: Any) -> Optional[str]: + """Render one ``_iter_docx_blocks`` element as text, or ``None`` to skip.""" + if isinstance(element, table_cls): + rendered = _docx_table_to_markdown(element) + return rendered or None + text = getattr(element, "text", "") + if not text or not text.strip(): + return None + if not boilerplate: + return text + return _strip_docx_boilerplate(text, boilerplate) + + +def _extract_docx(path: Path, *, ctx: Optional[_ExtractContext] = None) -> str: + """Extract DOCX text with explicit header / footer subtraction (Phase 15 Task 6). + + Round-2 review (S-4 + Sonar S3776): per-block rendering is delegated + to :func:`_render_docx_block` so this function stays a thin + open-doc → iterate-blocks → join loop below the cognitive-complexity + ceiling. The pre-review code subtracted EVERY paragraph line whose + stripped form appeared in the header / footer set, which dropped + legitimate body paragraphs matching a header verbatim + (``Chapter 1`` body vs ``Chapter 1`` header). The new helper now + short-circuits on lines longer than :data:`_DOCX_BOILERPLATE_MAX_LEN`. + """ + if ctx is None: + ctx = _ExtractContext() try: from docx import Document from docx.table import Table @@ -370,26 +1190,106 @@ def _extract_docx(path: Path) -> str: ) from exc doc = Document(str(path)) - # Phase 12: render tables as markdown so SFT models see the header / - # separator / body structure rather than a single ``|``-joined line. - # Walking the body in document order keeps tables next to the - # paragraphs that introduce them — appending all paragraphs first - # and all tables last (the previous behaviour) reordered content. + boilerplate = _collect_docx_header_footer_lines(doc) blocks: List[str] = [] for element in _iter_docx_blocks(doc): - if isinstance(element, Table): - rendered = _docx_table_to_markdown(element) - if rendered: - blocks.append(rendered) - else: - text = element.text - if text and text.strip(): - blocks.append(text) - + rendered = _render_docx_block(element, boilerplate, Table) + if rendered: + blocks.append(rendered) return "\n\n".join(blocks) -def _extract_epub(path: Path) -> str: +_EPUB_NAME_TOKEN_SPLIT = re.compile(r"[/\\.\-_ ]+") + + +def _epub_item_matches_skip(item_name: str, item_type: str, skip_list: Tuple[str, ...]) -> bool: + """Return ``True`` iff the item's file name or epub:type matches the skip-list. + + Phase 15 round-1 review (C-1): the pre-review implementation used a + plain substring match, which silently skipped legitimate chapters + whose filenames *contained* a skip token — ``recovery.xhtml`` got + dropped because ``cover`` is a substring of ``recovery``; + ``navy.xhtml`` and ``discovery_chapter.xhtml`` had the same fate. + A novel with chapters named "Recovery from War" or "The Royal Navy" + would silently lose entire bodies of text with no operator-facing + warning — the same silent-failure shape Phase 15 was launched to + fix. + + The fix is a **whole-token** match. We split the file name on + common path / extension / separator characters and compare each + resulting token against the skip-list. This catches the canonical + cases (``cover.xhtml``, ``nav.xhtml``, ``copyright.xhtml``, + ``cover-page.xhtml``, ``oebps/cover.xhtml``) while keeping + ``recovery.xhtml`` / ``navy.xhtml`` / ``discovery_chapter.xhtml`` + safe. + + Exact-token match on the ``epub:type`` value is unchanged. + """ + name_lc = item_name.lower() + skip_set = set(skip_list) + name_tokens = {token for token in _EPUB_NAME_TOKEN_SPLIT.split(name_lc) if token} + if name_tokens & skip_set: + return True + # Round-3-follow-up review (S-C): EPUB-3 manifest properties are + # SPACE-SEPARATED tokens (e.g. ``properties="nav cover-image"``). + # The pre-fix code did an exact-string match on the joined form, + # which silently failed for any manifest item carrying more than + # one property. Apply the same path/extension/separator splitter + # the filename probe uses so any property token can match. + type_lc = (item_type or "").lower() + if not type_lc: + return False + type_tokens = {token for token in _EPUB_NAME_TOKEN_SPLIT.split(type_lc) if token} + return bool(type_tokens & skip_set) + + +def _resolve_spine_item(book: Any, spine_entry: Any, item_document_type: int) -> Optional[Tuple[Any, str, str]]: + """Look up the spine entry's manifest item; return ``(item, name, type_str)`` or ``None``. + + ``None`` is returned for missing items or non-document types so the + caller can ``continue``. The ``type_str`` is the space-joined EPUB-3 + manifest properties (e.g. ``"nav cover-image"``) — empty for items + without explicit properties. + """ + item_id = spine_entry[0] if isinstance(spine_entry, (tuple, list)) else spine_entry + item = book.get_item_with_id(item_id) + if item is None or item.get_type() != item_document_type: + return None + item_name = getattr(item, "file_name", "") or getattr(item, "name", "") or "" + # Round-2 nit: ebooklib exposes EPUB-3 manifest properties (e.g. + # ``properties=["nav", "cover-image"]``) on ``item.properties``; + # the pre-round-2 code probed ``media_overlay`` which is always + # empty for nav / cover items. Join with spaces so any property + # token can match the skip-list via the whole-token splitter + # introduced for C-1. + item_properties = getattr(item, "properties", None) or [] + item_type = " ".join(str(p) for p in item_properties) if item_properties else "" + return item, item_name, item_type + + +def _emit_epub_skip_warning(path: Path, skipped_items: List[str]) -> None: + """Emit the Phase 15 round-1 C-1 WARNING naming skipped frontmatter items.""" + if not skipped_items: + return + preview = ", ".join(skipped_items[:6]) + (", …" if len(skipped_items) > 6 else "") + logger.warning( + "EPUB '%s': skipped %d frontmatter / navigation item(s): %s. " + "Pass --epub-no-skip-frontmatter to keep these in the JSONL.", + path, + len(skipped_items), + preview, + ) + + +def _extract_epub(path: Path, *, ctx: Optional[_ExtractContext] = None) -> str: + """Extract EPUB text in reading order; skip nav / cover / copyright (Phase 15 Task 7). + + Spine resolution and the skip-WARNING surface are extracted into + ``_resolve_spine_item`` and ``_emit_epub_skip_warning`` to keep + this function below the Sonar S3776 cognitive-complexity ceiling. + """ + if ctx is None: + ctx = _ExtractContext() try: from bs4 import BeautifulSoup from ebooklib import ITEM_DOCUMENT, epub @@ -412,22 +1312,54 @@ def _extract_epub(path: Path) -> str: str(path), options={"ignore_ncx": True, "ignore_missing_css": True}, ) + chunks: List[str] = [] - for item in book.get_items(): - if item.get_type() != ITEM_DOCUMENT: + skipped_items: List[str] = [] + for spine_entry in getattr(book, "spine", []): + resolved = _resolve_spine_item(book, spine_entry, ITEM_DOCUMENT) + if resolved is None: + continue + item, item_name, item_type = resolved + if ctx.epub_skip_frontmatter and _epub_item_matches_skip(item_name, item_type, ctx.epub_skip_items): + skipped_items.append(item_name) continue soup = BeautifulSoup(item.get_content(), "html.parser") text = soup.get_text(separator="\n").strip() if text: chunks.append(text) + _emit_epub_skip_warning(path, skipped_items) return "\n\n".join(chunks) -def _extract_text(path: Path) -> str: - raw = path.read_text(encoding="utf-8", errors="replace") - # Detect binary contamination — files mis-routed through the .txt extension - # (e.g. unzipped binaries renamed) come back as a sea of U+FFFD replacement - # characters. Below 1% is normal for legacy encodings; above is suspicious. +def _read_text_with_bom_strip(path: Path) -> str: + """Phase 15 Task 8 — read a TXT / MD file and strip a leading UTF-8 BOM. + + Why a dedicated helper: the previous behaviour passed the BOM through + as a literal character into chunk 0; downstream tokenisers rarely + handle U+FEFF correctly. We use ``encoding="utf-8-sig"`` for the + initial decode attempt — it strips the BOM transparently — and fall + back to ``encoding="utf-8"`` with ``errors="replace"`` when the + file is genuinely not UTF-8 (so the existing binary-contamination + warning still fires). + + Round-2 review (S-2): the fallback path used to read with + ``encoding="utf-8"`` rather than ``"utf-8-sig"``, leaking a literal + ``\\ufeff`` into chunk 0 whenever a file mixed a BOM with downstream + non-UTF-8 bytes. We now use ``"utf-8-sig"`` on both paths and + additionally strip an explicit leading ``\\ufeff`` so any encoding- + detection edge case is caught belt-and-braces. + """ + try: + raw = path.read_text(encoding="utf-8-sig") + except UnicodeDecodeError: + raw = path.read_text(encoding="utf-8-sig", errors="replace") + if raw.startswith(""): + raw = raw[1:] + return raw + + +def _warn_if_binary_contamination(raw: str, path: Path) -> None: + """Existing binary-content warning, factored out for reuse.""" if raw: replacement_count = raw.count("�") if replacement_count / max(len(raw), 1) > 0.01: @@ -438,6 +1370,40 @@ def _extract_text(path: Path) -> str: replacement_count, replacement_count * 100 / len(raw), ) + + +def _extract_text(path: Path, *, ctx: Optional[_ExtractContext] = None) -> str: + """Extract plain TXT with UTF-8 BOM stripping (Phase 15 Task 8). + + The ``ctx`` parameter is accepted for signature parity with the + other format extractors (the dispatcher calls every extractor with + ``ctx=ctx``) and intentionally unused inside this body — TXT has + no PDF / DOCX / EPUB-style options that need threading through. + """ + del ctx # signature parity only; Sonar S1854. + raw = _read_text_with_bom_strip(path) + _warn_if_binary_contamination(raw, path) + return raw + + +def _extract_markdown(path: Path, *, ctx: Optional[_ExtractContext] = None) -> str: + """Extract Markdown with UTF-8 BOM + YAML-frontmatter stripping (Phase 15 Task 8). + + YAML frontmatter is the ``---\\n…\\n---\\n`` block at the start of + the file used by Jekyll / Hugo / MkDocs / Pelican. Without + stripping it, chunk 0 of every imported MD file leads with + ``title:`` / ``date:`` / ``author:`` rows that are pure metadata + noise for SFT. Opt-out via ``ctx.keep_md_frontmatter`` for + workflows that genuinely want to train on the metadata. + """ + if ctx is None: + ctx = _ExtractContext() + raw = _read_text_with_bom_strip(path) + _warn_if_binary_contamination(raw, path) + if not ctx.keep_md_frontmatter: + match = _YAML_FRONTMATTER_PATTERN.match(raw) + if match is not None: + raw = raw[match.end() :] return raw @@ -446,7 +1412,7 @@ def _extract_text(path: Path) -> str: ".docx": _extract_docx, ".epub": _extract_epub, ".txt": _extract_text, - ".md": _extract_text, + ".md": _extract_markdown, } @@ -1053,24 +2019,36 @@ class _FileOutcome: def _extract_text_for_ingest( fpath: Path, extractor: Callable[..., str], - pdf_dedup_state: Optional[Dict[str, int]], + ctx: _ExtractContext, ) -> Optional[str]: """Run ``extractor`` against ``fpath``; return ``None`` to mean "skip this file". + Phase 15: threads the shared :class:`_ExtractContext` into every + extractor (uniform signature ``(path, *, ctx)``) and applies the + post-extract normalisation pipeline (glyph normalise → script sanity + → strip-patterns → strip-URLs) before returning. + ``ImportError`` re-propagates (missing optional extra is a *runtime* failure of the dispatched feature, not a per-file skip). All other exceptions log a warning and signal a skip — consistent with the silently-tolerant per-file model the CLI relies on. """ try: - if extractor is _extract_pdf: - return _extract_pdf(fpath, dedup_state=pdf_dedup_state) - return extractor(fpath) - except ImportError: + text = extractor(fpath, ctx=ctx) + except (ImportError, IngestParameterError): + # ImportError: missing optional extra is a *runtime* failure of + # the dispatched feature — propagate so the CLI dispatcher can + # translate to EXIT_TRAINING_ERROR with a clear install-hint. + # IngestParameterError (Phase 15): operator-supplied parameter + # mistake — propagate so the run aborts with EXIT_CONFIG_ERROR + # instead of silently soft-failing per-file. raise except Exception as exc: # noqa: BLE001 — best-effort: per-file dispatcher catch — each format-specific extractor (PDF/DOCX/EPUB/TXT/MD) raises its own typed ValueError above + a wide tail of corpus-data-driven failures; per-file soft-fail with skip-and-continue keeps a multi-thousand-file corpus ingest running. ImportError stays narrow above so missing-extra failures still propagate. # NOSONAR logger.warning("Skipping '%s' (extraction failed): %s", fpath, exc) return None + if not text: + return text + return _post_extract_normalise(text, ctx, fpath) def _select_chunks_iter( @@ -1101,6 +2079,7 @@ def _emit_chunk( outcome: _FileOutcome, mask_pii: Optional[Callable[..., Any]], mask_secrets: Optional[Callable[..., Any]] = None, + sampler: Optional[Callable[[str], None]] = None, ) -> None: """Mask (optional), serialise, and write one chunk; update outcome counters. @@ -1113,6 +2092,12 @@ def _emit_chunk( pass. Future PII / secret regex additions could legitimately overlap (e.g. an Azure connection-string substring resembling an IBAN); this ordering future-proofs against that case without operator action. + + Round-2 review (S-3): the quality pre-signal previously read each + written line back via ``json.loads`` to recover the chunk text, an + avoidable per-chunk round-trip on large corpora. The ``sampler`` + kwarg now hands the post-mask chunk string straight to the + pre-signal collector, bypassing the JSON parse entirely. """ if mask_secrets is not None: payload, secret_counts = mask_secrets(payload, return_counts=True) @@ -1126,6 +2111,8 @@ def _emit_chunk( payload, redaction_counts = mask_pii(payload, return_counts=True) for kind, count in redaction_counts.items(): outcome.pii_counts[kind] = outcome.pii_counts.get(kind, 0) + count + if sampler is not None: + sampler(payload) out_fh.write(json.dumps({"text": payload}, ensure_ascii=False) + "\n") outcome.chunks_written += 1 outcome.chars_written += len(payload) @@ -1143,24 +2130,28 @@ def _process_one_file( chunk_tokens: Optional[int] = None, overlap_tokens: int = 0, tokenizer: Any = None, - pdf_dedup_state: Optional[Dict[str, int]] = None, + extract_ctx: Optional[_ExtractContext] = None, + sampler: Optional[Callable[[str], None]] = None, ) -> _FileOutcome: """Extract → chunk → optionally mask → emit JSONL for a single file. ``chunk_tokens`` is honoured when set: token-aware chunking takes over via :func:`_strategy_dispatch_tokens`, and ``chunk_size`` becomes a - fallback only when the tokenizer is missing. ``pdf_dedup_state`` is - threaded into :func:`_extract_pdf` so cross-page header/footer dedup - counts roll up into the run-level structured notes. + fallback only when the tokenizer is missing. ``extract_ctx`` carries + the Phase 15 extraction options + side-output state (header/footer + dedup counters, script-sanity reports, front-matter dropped pages, + etc.) so they roll up into the run-level structured notes. ImportError propagates (missing optional extra is not a per-file skip). Any other extraction failure is logged + counted as a skip. """ + if extract_ctx is None: + extract_ctx = _ExtractContext() extractor = _select_extractor(fpath) if extractor is None: return _FileOutcome(file_skipped=True) - text = _extract_text_for_ingest(fpath, extractor, pdf_dedup_state) + text = _extract_text_for_ingest(fpath, extractor, extract_ctx) if text is None: return _FileOutcome(file_skipped=True) if not text or not text.strip(): @@ -1181,7 +2172,7 @@ def _process_one_file( payload = chunk.strip() if not payload: continue - _emit_chunk(payload, out_fh, outcome, mask_pii, mask_secrets=mask_secrets) + _emit_chunk(payload, out_fh, outcome, mask_pii, mask_secrets=mask_secrets, sampler=sampler) return outcome @@ -1197,7 +2188,7 @@ def _process_one_file( DEFAULT_SLIDING_OVERLAP: int = 200 -def ingest_path( +def ingest_path( # NOSONAR python:S107 — every kwarg is a documented operator-facing knob; collapsing them into a config dataclass would be a breaking API change for v0.5 callers that already pass these by name (per `forgelm.__all__` / `__api_version__` contract). input_path: str, *, output_path: str, @@ -1211,6 +2202,20 @@ def ingest_path( chunk_tokens: Optional[int] = None, overlap_tokens: int = 0, tokenizer: Optional[str] = None, + # ---- Phase 15 knobs ------------------------------------------------- + language_hint: Optional[str] = None, + script_sanity_threshold: float = _DEFAULT_SCRIPT_SANITY_THRESHOLD, + # ``None`` triggers the C-2 auto-derive: when ``language_hint=="tr"`` + # the profile is "turkish"; otherwise "none". An explicit value wins. + normalise_profile: Optional[str] = None, + keep_md_frontmatter: bool = False, + epub_skip_frontmatter: bool = True, + keep_frontmatter: bool = False, + page_range: Optional[Tuple[int, int]] = None, + strip_patterns: Optional[List[str]] = None, + strip_pattern_timeout: Optional[int] = 5, + strip_urls: str = "keep", + quality_presignal: bool = True, ) -> IngestionResult: """Ingest a single file or a directory tree into a SFT-compatible JSONL. @@ -1265,6 +2270,30 @@ def ingest_path( dst = Path(output_path).expanduser().resolve() dst.parent.mkdir(parents=True, exist_ok=True) + if strip_urls not in ("keep", "mask", "strip"): + raise ValueError(f"strip_urls must be one of 'keep' / 'mask' / 'strip', got {strip_urls!r}.") + if page_range is not None and (not isinstance(page_range, tuple) or len(page_range) != 2): + raise ValueError("page_range must be a (start, end) tuple of 1-indexed inclusive page numbers.") + + # Phase 15 Wave 2 Task 11: compile + validate operator-supplied + # --strip-pattern values up-front so a malformed regex aborts the + # run BEFORE any I/O. The CLI dispatcher catches StripPatternError + # (subclass of ValueError) and exits with EXIT_CONFIG_ERROR. + compiled_strip_patterns: List[Tuple[str, "re.Pattern[str]"]] = [] + if strip_patterns: + from ._strip_pattern import compile_strip_patterns + + compiled_strip_patterns = compile_strip_patterns(strip_patterns) + + # Phase 15 round-2 (C-2) library-level profile derivation. When the + # caller passes ``normalise_profile=None`` we derive "turkish" from + # ``language_hint == "tr"`` and "none" otherwise. Explicit kwarg + # values (including the literal ``"none"``) bypass derivation so + # operator intent is honoured. The CLI dispatcher mirrors this + # logic in ``_resolve_normalise_profile`` for parity. + if normalise_profile is None: + normalise_profile = "turkish" if (language_hint or "").lower() == "tr" else "none" + chunk_size_explicit = chunk_size is not None effective_chunk_size = chunk_size if chunk_size_explicit else DEFAULT_CHUNK_SIZE @@ -1339,6 +2368,40 @@ def ingest_path( notes: List[str] = [] pdf_dedup_state: Dict[str, int] = {} + extract_ctx = _ExtractContext( + dedup_state=pdf_dedup_state, + page_range=page_range, + normalise_profile=normalise_profile, + language_hint=language_hint, + script_sanity_threshold=script_sanity_threshold, + keep_md_frontmatter=keep_md_frontmatter, + epub_skip_frontmatter=epub_skip_frontmatter, + keep_frontmatter=keep_frontmatter, + strip_patterns=compiled_strip_patterns, + strip_pattern_timeout=strip_pattern_timeout, + strip_urls_mode=strip_urls, + ) + + # Phase 15 Task 4: collect each chunk's text on the fly so the + # end-of-run quality pre-signal can compute its three cheap checks + # without re-reading the JSONL. Round-2 review (S-3): we now pass + # the sampler straight into ``_emit_chunk`` instead of wrapping the + # file handle and re-parsing the JSON payload back out of every + # written line. + sampled_chunks: List[str] = [] + quality_sample_limit = 5000 # keep memory bounded on huge corpora + + sampler: Optional[Callable[[str], None]] + if quality_presignal: + + def _maybe_sample(payload: str) -> None: + if len(sampled_chunks) < quality_sample_limit: + sampled_chunks.append(payload) + + sampler = _maybe_sample + else: + sampler = None + # newline="\n" pins LF on Windows. JSONL Files spec requires LF, and # piping through tooling (jq -c, wc -l, downstream HF dataset loaders) # avoids CRLF surprises. Linux/macOS default is already LF. @@ -1355,7 +2418,8 @@ def ingest_path( chunk_tokens=chunk_tokens, overlap_tokens=overlap_tokens, tokenizer=tokenizer_obj, - pdf_dedup_state=pdf_dedup_state, + extract_ctx=extract_ctx, + sampler=sampler, ) chunk_count += outcome.chunks_written total_chars += outcome.chars_written @@ -1387,11 +2451,46 @@ def ingest_path( else: notes.append("Secrets masking enabled — no credentials detected in this corpus") pdf_lines_stripped = pdf_dedup_state.get("lines_stripped", 0) + pdf_paragraph_stripped = pdf_dedup_state.get("paragraph_packed_stripped", 0) if pdf_lines_stripped: notes.append( f"PDF header/footer dedup stripped {pdf_lines_stripped} repeated line(s) " "(reduces audit near-duplicate noise)." ) + if pdf_paragraph_stripped: + notes.append(f"PDF post-pack dedup stripped {pdf_paragraph_stripped} survivor header line(s).") + + # Phase 15 Task 4: quality pre-signal at end of run. + quality_presignal_payload: Optional[Dict[str, Any]] = None + if quality_presignal and sampled_chunks: + quality_presignal_payload = _compute_quality_presignal(sampled_chunks) + flagged = quality_presignal_payload["samples_flagged"] + evaluated = quality_presignal_payload["samples_evaluated"] + if flagged: + notes.append( + f"{flagged}/{evaluated} chunks below ingestion quality threshold. " + "Run `forgelm audit ` for detail." + ) + # The audit guide names the same call shape, so the operator + # can grep this string to find the doc page. Emitting via + # logger.warning makes it visible in CI logs regardless of + # --output-format json/text. + logger.warning( + "[WARN] %d/%d chunks below ingestion quality threshold. Run `forgelm audit %s` for detail.", + flagged, + evaluated, + dst, + ) + + # Phase 15 Task 2: aggregate script-sanity reports. + from ._script_sanity import report_to_structured as _script_sanity_to_structured + + script_sanity_summary: Optional[Dict[str, Any]] = None + if extract_ctx.script_sanity_reports: + script_sanity_summary = _script_sanity_to_structured(extract_ctx.script_sanity_reports) + triggered_count = script_sanity_summary.get("files_triggered", 0) + if triggered_count: + notes.append(f"Script-sanity check fired on {triggered_count} file(s) — see warnings.") structured_notes: Dict[str, Any] = { "files_processed": files_processed, @@ -1405,9 +2504,24 @@ def ingest_path( } if pdf_lines_stripped: structured_notes["pdf_header_footer_lines_stripped"] = pdf_lines_stripped + if pdf_paragraph_stripped: + structured_notes["pdf_paragraph_packed_lines_stripped"] = pdf_paragraph_stripped if chunk_tokens is not None: structured_notes["chunk_tokens"] = chunk_tokens structured_notes["tokenizer"] = tokenizer + if script_sanity_summary is not None: + structured_notes["script_sanity_summary"] = script_sanity_summary + if extract_ctx.frontmatter_dropped_pages: + structured_notes["frontmatter_pages_dropped"] = sorted(set(extract_ctx.frontmatter_dropped_pages)) + if extract_ctx.strip_pattern_substitutions_total: + structured_notes["strip_pattern_substitutions"] = extract_ctx.strip_pattern_substitutions_total + notes.append(f"--strip-pattern removed {extract_ctx.strip_pattern_substitutions_total} span(s).") + if extract_ctx.urls_handled_total: + structured_notes["urls_handled"] = extract_ctx.urls_handled_total + action = {"mask": "masked", "strip": "stripped"}.get(strip_urls, strip_urls) + notes.append(f"--strip-urls {action} {extract_ctx.urls_handled_total} URL(s).") + if quality_presignal_payload is not None: + structured_notes["quality_presignal"] = quality_presignal_payload logger.info( "ingest: source=%s output=%s files=%d chunks=%d chars=%d strategy=%s", @@ -1431,7 +2545,133 @@ def ingest_path( extra_notes=notes, notes_structured=structured_notes, pdf_header_footer_lines_stripped=pdf_lines_stripped, + pdf_paragraph_packed_lines_stripped=pdf_paragraph_stripped, + script_sanity_triggered=(script_sanity_summary or {}).get("files_triggered", 0) if script_sanity_summary else 0, + strip_pattern_substitutions=extract_ctx.strip_pattern_substitutions_total, + urls_handled=extract_ctx.urls_handled_total, + frontmatter_pages_dropped=len(set(extract_ctx.frontmatter_dropped_pages)), + ) + + +# --------------------------------------------------------------------------- +# Phase 15 Task 4 — end-of-run quality pre-signal helpers +# --------------------------------------------------------------------------- + + +_PRESIGNAL_MIN_NON_WS_CHARS: int = 80 +"""Minimum non-whitespace char count before the alpha-ratio check applies. + +Round-3-follow-up review (C-B): a perfectly clean 41-char corpus +(``"Para 1.\\n\\nPara 2.\\n\\n…"``) was tripping the alpha-ratio +threshold because dots + whitespace dragged the ratio below 0.70 on a +chunk too small to be statistically meaningful. The pre-signal is +intended to catch corpus-scale corruption (74 / 82 chunks on the audit +PDF), not micro-corpora where the alpha denominator is dominated by +sentence-end punctuation. An 80-char floor (≈ one short sentence) +suppresses the regression on clean v0.5 callers without weakening the +signal on the audit's pilot shape. +""" + + +def _check_alpha_ratio(text: str) -> bool: + """Phase 15 Task 4 — alpha-ratio cheap-check. + + Returns ``True`` when the chunk fails the check (alpha ratio < 0.70). + Same threshold as the audit's full quality filter so the pre-signal + aligns with the deeper diagnostic. + + Round-3-follow-up (C-B): chunks below + :data:`_PRESIGNAL_MIN_NON_WS_CHARS` non-whitespace characters skip + the check entirely. Below that floor the alpha-ratio is dominated + by punctuation/whitespace and false-flags clean small corpora that + were silent on v0.5 — the change must not regress operator UX on + well-formed minimal-size inputs. + """ + non_ws = [c for c in text if not c.isspace()] + if not non_ws: + return True + if len(non_ws) < _PRESIGNAL_MIN_NON_WS_CHARS: + return False + alpha = sum(1 for c in non_ws if c.isalpha()) / len(non_ws) + return alpha < 0.70 + + +def _check_weird_char_ratio(text: str) -> bool: + """Phase 15 Task 4 — weird-character cheap-check. + + Returns ``True`` when the chunk has > 5 % "weird" characters + (control chars + U+FFFD replacement char + isolated PUA glyphs + + Mandaic punctuation). Catches the audit's font-corruption mode + without running the full Unicode-block sanity pass. + + Round-5 review (S-B): the pre-round-5 range ``0x0800-0x097F`` + covered Samaritan / Mandaic / Syriac Supplement / Arabic + Extended-A / Devanagari — 384 code points spanning 5 scripts. + The ``and not c.isalpha()`` filter saved Devanagari / Syriac + alphabetics, but the range was much wider than the audit's + actual target (U+085F Mandaic punctuation as a custom bullet + glyph). Narrowed to Mandaic only (0x0840-0x085F) so a Sanskrit / + Hindi / Arabic-script corpus does not trip on legitimate + non-alpha punctuation in those scripts. + """ + if not text: + return False + weird = sum( + 1 + for c in text + if c == "�" + or (ord(c) < 0x20 and c not in ("\n", "\r", "\t")) + or (0xE000 <= ord(c) <= 0xF8FF) + or (0x0840 <= ord(c) <= 0x085F and not c.isalpha()) ) + return weird / len(text) > 0.05 + + +def _check_repeated_line_ratio(text: str) -> bool: + """Phase 15 Task 4 — repeated-line cheap-check. + + Returns ``True`` when ≥ 30 % of a chunk's lines are duplicates. + Same threshold as the audit's full ``repeated_lines`` check, but + cheap because we work on already-emitted chunk text instead of + re-reading the JSONL. + """ + lines = [ln for ln in text.splitlines() if ln.strip()] + if len(lines) < 3: + return False + counts = Counter(lines) + repeated = sum(n for ln, n in counts.items() if n >= 2) + return repeated / len(lines) > 0.30 + + +def _compute_quality_presignal(chunks: List[str]) -> Dict[str, Any]: + """Apply the three Phase 15 Task 4 cheap checks to a sample of chunks. + + Returns a structured payload that always carries the schema + callers depend on, even when nothing was flagged. The threshold, + sample size, and per-check counts are surfaced so downstream + tooling (audit, CI gates) can decide what to do — we do not block + the run. + """ + flagged = 0 + by_check = {"alpha_ratio": 0, "weird_chars": 0, "repeated_lines": 0} + for chunk in chunks: + chunk_flagged = False + if _check_alpha_ratio(chunk): + by_check["alpha_ratio"] += 1 + chunk_flagged = True + if _check_weird_char_ratio(chunk): + by_check["weird_chars"] += 1 + chunk_flagged = True + if _check_repeated_line_ratio(chunk): + by_check["repeated_lines"] += 1 + chunk_flagged = True + if chunk_flagged: + flagged += 1 + return { + "samples_evaluated": len(chunks), + "samples_flagged": flagged, + "by_check": by_check, + } def summarize_result(result: IngestionResult) -> str: diff --git a/notebooks/ingestion_playground.ipynb b/notebooks/ingestion_playground.ipynb index b861a39a..64c81ce5 100644 --- a/notebooks/ingestion_playground.ipynb +++ b/notebooks/ingestion_playground.ipynb @@ -123,7 +123,20 @@ "source": [ "# Tweak these and re-run the cell. The NEXT cell executes them.\n", "STRATEGY = \"paragraph\" # \"paragraph\" | \"sliding\" | \"markdown\"\n", - "CHUNK_SIZE = 1024 # soft character cap per chunk\n", + "CHUNK_SIZE = 1024 # soft character cap per chunk (ignored when CHUNK_TOKENS is set)\n", + "\n", + "# Phase 15 Wave 1 addition — token-aware chunking knobs. BOTH must be set\n", + "# together to switch from character-based to tokenizer-based chunking;\n", + "# the cell raises ValueError if only one of CHUNK_TOKENS / TOKENIZER is\n", + "# set so a partial configuration cannot silently fall back to --chunk-size.\n", + "CHUNK_TOKENS = None # e.g. 512 to cap chunks at 512 tokens\n", + "TOKENIZER = None # e.g. \"meta-llama/Llama-3.2-3B\" (HF Hub id)\n", + "\n", + "# Phase 15 Wave 1 addition — language hint enables the Unicode-block\n", + "# sanity check after each per-file extract. Set to \"tr\" for Turkish\n", + "# corpora, \"en\" for English, etc. Leave as None to disable the check.\n", + "LANGUAGE_HINT = None # e.g. \"tr\" for a Turkish corpus\n", + "\n", "MASK_PII = True # redact PII (names, emails, phones, IDs)\n", "MASK_SECRETS = True # redact API keys, tokens, SSH/PGP keys, JWTs\n", "RECURSIVE = True # descend into subdirectories\n", @@ -134,6 +147,18 @@ "\n", "Path(OUTPUT_PATH).parent.mkdir(parents=True, exist_ok=True)\n", "\n", + "# Fail fast on partial token-mode configuration so the operator does\n", + "# not silently get character-based chunking when they asked for tokens\n", + "# (or get a CLI error from --chunk-tokens without --tokenizer).\n", + "_token_mode_kwargs_set = sum(1 for v in (CHUNK_TOKENS, TOKENIZER) if v)\n", + "if _token_mode_kwargs_set == 1:\n", + " raise ValueError(\n", + " \"Token-aware chunking requires BOTH CHUNK_TOKENS and TOKENIZER to be set. \"\n", + " f\"Got CHUNK_TOKENS={CHUNK_TOKENS!r} TOKENIZER={TOKENIZER!r} — \"\n", + " \"set both to enable token-aware mode, or leave both as None for character-based chunking.\"\n", + " )\n", + "_token_mode = _token_mode_kwargs_set == 2\n", + "\n", "# Build a SINGLE-LINE command string with shell-safe quoting so paths\n", "# containing spaces or other shell metachars don't get split or\n", "# re-interpreted by bash. ``shlex.quote`` wraps any non-safe value in\n", @@ -142,7 +167,15 @@ "if RECURSIVE:\n", " parts.append(\"--recursive\")\n", "parts += [\"--strategy\", shlex.quote(STRATEGY)]\n", - "parts += [\"--chunk-size\", str(CHUNK_SIZE)]\n", + "if _token_mode:\n", + " # Token-aware mode (Phase 11.5 + Phase 15): the --chunk-size flag is\n", + " # logged as ignored on the ingest side, so we omit it from the command.\n", + " parts += [\"--chunk-tokens\", str(CHUNK_TOKENS)]\n", + " parts += [\"--tokenizer\", shlex.quote(str(TOKENIZER))]\n", + "else:\n", + " parts += [\"--chunk-size\", str(CHUNK_SIZE)]\n", + "if LANGUAGE_HINT:\n", + " parts += [\"--language-hint\", shlex.quote(str(LANGUAGE_HINT))]\n", "if MASK_PII:\n", " parts.append(\"--pii-mask\")\n", "if MASK_SECRETS:\n", @@ -206,6 +239,12 @@ "- near-duplicate scan (simhash by default, LSH-banded)\n", "- PII severity tiers (`critical` / `high` / `medium` / `low`)\n", "- always-on secrets pass — flags credentials that survived ingestion\n", + "- **Phase 15 Wave 1: heuristic quality checks (default ON in v0.6.0+)** —\n", + " surfaces extraction-layer noise via cheap, conservative heuristics:\n", + " alpha-ratio per row, mean word length, end-of-line punctuation,\n", + " repeated-line ratio, short-paragraph ratio. Heuristic by design, not\n", + " a model gate — operator decides whether to filter. Findings land\n", + " under `quality_summary.by_check` in the audit JSON.\n", "\n", "If you ran ingestion with `--secrets-mask`, the secrets count here\n", "should be **zero** — that's the audit's job, to confirm the masker did\n", @@ -219,8 +258,13 @@ "outputs": [], "source": [ "# Step 5: Audit the curated JSONL.\n", + "#\n", + "# Phase 15 Wave 1 addition: --quality-filter explicitly passed for\n", + "# forward-compat with operators on pre-v0.6.0 releases. From v0.6.0+\n", + "# the flag is default-on; passing it explicitly here is redundant but\n", + "# harmless, and keeps the notebook robust across the v0.5 → v0.6 transition.\n", "AUDIT_DIR = \"./audit\"\n", - "!forgelm audit {OUTPUT_PATH} --output {AUDIT_DIR} --output-format json" + "!forgelm audit {OUTPUT_PATH} --output {AUDIT_DIR} --quality-filter --output-format json" ] }, { @@ -270,7 +314,28 @@ " print(\"\\nSecrets summary (POST-MASK — should be empty if --secrets-mask was on):\")\n", " print(f\" {secrets}\")\n", " else:\n", - " print(\"\\nSecrets: clean (no credentials reached the JSONL).\")" + " print(\"\\nSecrets: clean (no credentials reached the JSONL).\")\n", + "\n", + " # Phase 15 Wave 1 (Task 9): surface quality_summary alongside the existing\n", + " # PII / secrets / near-duplicate / language blocks. The audit emits this\n", + " # block by default in v0.6.0+; an empty `{}` means the operator passed\n", + " # --no-quality-filter so the check was deliberately skipped.\n", + " quality = report.get(\"quality_summary\", {})\n", + " if quality:\n", + " evaluated = quality.get(\"samples_evaluated\", 0)\n", + " flagged = quality.get(\"samples_flagged\", 0)\n", + " score = quality.get(\"overall_quality_score\")\n", + " print(\"\\nQuality summary (Phase 15 heuristic — operator-decides, not a gate):\")\n", + " print(f\" samples_flagged / samples_evaluated: {flagged} / {evaluated}\")\n", + " if score is not None:\n", + " print(f\" overall_quality_score: {score:.4f}\")\n", + " by_check = quality.get(\"by_check\")\n", + " if by_check:\n", + " print(\" by_check:\")\n", + " for check_name, count in sorted(by_check.items()):\n", + " print(f\" {check_name:>26s}: {count}\")\n", + " else:\n", + " print(\"\\nQuality summary: skipped (operator passed --no-quality-filter).\")" ] }, { @@ -359,4 +424,4 @@ }, "nbformat": 4, "nbformat_minor": 4 -} \ No newline at end of file +} diff --git a/tests/fixtures/ingestion/README.md b/tests/fixtures/ingestion/README.md new file mode 100644 index 00000000..43b20904 --- /dev/null +++ b/tests/fixtures/ingestion/README.md @@ -0,0 +1,49 @@ +# Phase 15 regression fixtures + +> **Audience:** ForgeLM contributors maintaining the Phase 15 +> ingestion-reliability regression suite. + +This directory carries the **deterministic** Phase 15 fixtures that +do not depend on a specific `pypdf` / `python-docx` / `ebooklib` +version — small, plain-text-or-XML inputs whose extraction behaviour +is contractually stable across library upgrades. Binary fixtures that +*do* depend on extractor version (PDF / DOCX / EPUB) are **synthesised +at test time** by `tests/test_ingestion_reliability.py` so a future +`pypdf` upgrade does not silently break the regression suite. + +All fixtures are synthesised inside the repository under the +project's permissive licence. No third-party copyright is incurred. + +## On-disk fixtures (here) + +| File | Purpose | +|---|---| +| `txt_with_bom_crlf.txt` | UTF-8 BOM strip + CRLF passthrough (Task 8) | +| `md_with_frontmatter_and_html.md` | YAML frontmatter strip + embedded HTML (Task 8) | +| `mixed_directory/sample.txt` | Recursive ingest cross-format sample (Task 10) | +| `mixed_directory/sample.md` | Recursive ingest cross-format sample (Task 10) | + +## In-test synthesised fixtures + +The PDF / DOCX / EPUB fixtures are generated on-the-fly inside +`tests/test_ingestion_reliability.py` via helper factories. Synthesis +uses `pypdf.PdfWriter` / `python-docx` / `ebooklib` directly — the +same libraries the extractor calls — so the binary path exercises +the same code that real-world corpora hit. Goldens are computed in +the same run and asserted against expected behaviours (header line +not in any chunk, alpha-ratio > 0.65, etc.) rather than byte-compared, +which keeps the suite resilient against pypdf minor-version drift. + +This trades the audit's "commit the binary + commit the golden" +pattern for "commit the synthesis helper + assert on properties" — +both are valid regression-locking strategies, and the second is more +sustainable for a test-driven repo on a moving dependency surface. + +## Why "synthesised, not real-world" + +Real-world corpora carry institutional copyright + privacy concerns +that block inclusion under the project's permissive licence. The +synthetic fixtures exhibit the same **technical phenomena** the audit +documented (corrupt glyphs, multi-line headers, ToC underscore +leaders, URL noise, BOM + CRLF, YAML frontmatter, embedded HTML) +without redistributing any third-party content. diff --git a/tests/fixtures/ingestion/md_with_frontmatter_and_html.md b/tests/fixtures/ingestion/md_with_frontmatter_and_html.md new file mode 100644 index 00000000..6d0bc1be --- /dev/null +++ b/tests/fixtures/ingestion/md_with_frontmatter_and_html.md @@ -0,0 +1,26 @@ +--- +title: Phase 15 fixture +author: ForgeLM tests +date: 2026-05-11 +tags: + - phase-15 + - regression +--- + + + +# Body heading + +This is the body content that should survive YAML-frontmatter stripping. +It uses a fully-formed sentence so the alpha-ratio quality check stays +comfortably above the 70 % threshold. + +
+HTML blocks pass through verbatim today; Wave 3 may revisit them. +
+ +Another paragraph closes the sample so the chunker has at least two +paragraph boundaries to work with under the paragraph strategy. diff --git a/tests/fixtures/ingestion/mixed_directory/sample.md b/tests/fixtures/ingestion/mixed_directory/sample.md new file mode 100644 index 00000000..a0d00bc7 --- /dev/null +++ b/tests/fixtures/ingestion/mixed_directory/sample.md @@ -0,0 +1,6 @@ +# Mixed-directory markdown sample + +Body paragraph one of the markdown sample. The chunker emits this as a +single-paragraph chunk under the paragraph strategy. + +Body paragraph two closes the sample. diff --git a/tests/fixtures/ingestion/mixed_directory/sample.txt b/tests/fixtures/ingestion/mixed_directory/sample.txt new file mode 100644 index 00000000..301ee85c --- /dev/null +++ b/tests/fixtures/ingestion/mixed_directory/sample.txt @@ -0,0 +1,3 @@ +Mixed-directory recursive sample (TXT). Body paragraph one. + +Body paragraph two ensures the paragraph chunker sees a real boundary. diff --git a/tests/fixtures/ingestion/txt_with_bom_crlf.txt b/tests/fixtures/ingestion/txt_with_bom_crlf.txt new file mode 100644 index 00000000..887eecef --- /dev/null +++ b/tests/fixtures/ingestion/txt_with_bom_crlf.txt @@ -0,0 +1,4 @@ +Header Line One +A second line of body content. +A third line follows the CRLF. +Final line ends the file. diff --git a/tests/test_doctor.py b/tests/test_doctor.py index f0410b16..d9882cb1 100644 --- a/tests/test_doctor.py +++ b/tests/test_doctor.py @@ -1051,3 +1051,34 @@ def test_doctor_helpers_reachable_via_facade(self) -> None: "_check_operator_identity", ): assert hasattr(_cli_facade, name), f"forgelm.cli must re-export {name!r}" + + +class TestPypdfNormaliseDoctorProbe: + """Phase 15 Task 3 / round-2 N-3 — dedicated coverage for the new probe.""" + + def test_pass_when_table_round_trips_canonical_fixture(self): + from forgelm.cli.subcommands._doctor import _check_pypdf_normalise_turkish + + result = _check_pypdf_normalise_turkish() + assert result.status == "pass" + assert result.name == "pypdf_normalise.turkish" + assert result.extras.get("single_substitutions", 0) >= 5 + assert result.extras.get("default_profile") in ("none", "turkish") + + def test_fail_when_profile_silently_no_ops(self, monkeypatch): + """If a future refactor breaks the dispatcher, the probe fails loudly.""" + import forgelm.cli.subcommands._doctor as doctor_mod + + # Patch the dispatcher used by the probe to no-op on the fixture so + # the probe's "did the table actually rewrite?" guard fires. + def fake_apply_profile(text, _profile): + return text + + # The probe re-imports apply_profile inside its body — patch the + # underlying module so the late import resolves to our shim. + import forgelm._pypdf_normalise as norm_mod + + monkeypatch.setattr(norm_mod, "apply_profile", fake_apply_profile) + result = doctor_mod._check_pypdf_normalise_turkish() + assert result.status == "fail" + assert "no substitutions" in result.detail.lower() diff --git a/tests/test_ingestion_reliability.py b/tests/test_ingestion_reliability.py new file mode 100644 index 00000000..3e6ea104 --- /dev/null +++ b/tests/test_ingestion_reliability.py @@ -0,0 +1,1045 @@ +"""Phase 15 ingestion-reliability regression suite. + +Mirrors the Phase 15 roadmap (`docs/roadmap/phase-15-ingestion-reliability.md`). +The roadmap captures the audit findings that drove every task; the +audit notes themselves live outside the public tree per the +`tools/check_no_analysis_refs.py` policy. Each test class targets one +Wave 1 / Wave 2 task and asserts the documented acceptance criterion. + +Test strategy: + +* **In-test synthesis** for PDF / DOCX / EPUB. Binary fixtures + committed against pypdf-extracted goldens drift on every minor + version bump; synthesising the inputs inside the test keeps the + suite resilient. The synthesis helpers use only ``pypdf.PdfWriter``, + ``python-docx``, and ``ebooklib`` — the same libraries the + extractors call. +* **On-disk fixtures** for TXT / MD where the inputs are deterministic + plain text and the extractor output is contractually stable. +* **Behaviour assertions**, not byte-compare goldens. Each test + asserts the documented contract (e.g. "no chunk contains the + repeating header line", "alpha-ratio recovers above 0.65 after + normalisation"). The audit's "fixture + golden" pattern is the + spirit; in-test behaviour assertions are the letter. +""" + +from __future__ import annotations + +import importlib +import json +from pathlib import Path +from typing import List + +import pytest + +from forgelm.ingestion import ( + ingest_path, + strip_paragraph_packed_headers, +) + +# --------------------------------------------------------------------------- +# Capability gates +# --------------------------------------------------------------------------- + + +def _has(module_name: str) -> bool: + try: + importlib.import_module(module_name) + return True + except ImportError: + return False + + +HAS_PYPDF = _has("pypdf") +HAS_DOCX = _has("docx") +HAS_EPUB = _has("ebooklib") + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +FIXTURE_DIR = Path(__file__).parent / "fixtures" / "ingestion" + + +def _read_jsonl(path: Path) -> List[str]: + return [json.loads(line)["text"] for line in path.read_text(encoding="utf-8").splitlines() if line.strip()] + + +def _synth_multipage_pdf(pages: List[List[str]]) -> bytes: + """Build an N-page PDF whose pages carry the supplied line lists. + + Extends the ``_hand_built_pdf`` helper pattern from + ``tests/test_ingestion.py`` to N pages so Phase 15's multi-line + header dedup (Task 1) can be exercised against a realistic shape. + Each page is laid out with vertically-stacked Tj-encoded lines so + pypdf's ``extract_text()`` sees them in order. + """ + pdf_objs: List[bytes] = [] + + # Object 1: Catalog. Object 2: Pages tree (filled after we know N). + # Object 3+: Pages + content streams. + # Page indices in the Pages tree start at 3 and grow by 2 per page + # (one page object + one content stream object per page). With N + # pages we end up at 2N + 2 objects total. + page_object_ids: List[int] = [] + content_streams: List[bytes] = [] + + for page_lines in pages: + # Each line drops 14 points; start at the top of the page. + # ``Td`` carries an (x, y) translation per line. + lines: List[str] = [] + y = 740 + for ln in page_lines: + escaped = ln.replace("\\", "\\\\").replace("(", "\\(").replace(")", "\\)") + lines.append(f"BT /F1 11 Tf 72 {y} Td ({escaped}) Tj ET") + y -= 14 + content_streams.append("\n".join(lines).encode("latin-1", errors="replace")) + + next_id = 3 + for _ in content_streams: + page_obj_id = next_id + page_object_ids.append(page_obj_id) + next_id += 2 + + pages_obj_kids = " ".join(f"{pid} 0 R" for pid in page_object_ids) + catalog = b"1 0 obj\n<< /Type /Catalog /Pages 2 0 R >>\nendobj\n" + pages_obj = ( + b"2 0 obj\n<< /Type /Pages /Kids [" + + pages_obj_kids.encode("latin-1") + + b"] /Count " + + str(len(page_object_ids)).encode("latin-1") + + b" >>\nendobj\n" + ) + + pdf_objs.append(catalog) + pdf_objs.append(pages_obj) + + for idx, content in enumerate(content_streams): + page_obj_id = page_object_ids[idx] + content_obj_id = page_obj_id + 1 + # Share a single Type1 font (Helvetica) across every page; resolved + # via the Pages tree's resource dictionary on each Page object. + # Each page declares /F1 separately so the test stays close to + # the pre-Phase-15 ``_hand_built_pdf`` shape. + page = ( + f"{page_obj_id} 0 obj\n" + f"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] " + f"/Contents {content_obj_id} 0 R " + f"/Resources << /Font << /F1 {next_id} 0 R >> >> >>\n" + f"endobj\n" + ).encode("latin-1") + content_obj = ( + f"{content_obj_id} 0 obj\n<< /Length ".encode("latin-1") + + str(len(content)).encode("latin-1") + + b" >>\nstream\n" + + content + + b"\nendstream\nendobj\n" + ) + pdf_objs.append(page) + pdf_objs.append(content_obj) + + font_obj = (f"{next_id} 0 obj\n<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>\nendobj\n").encode("latin-1") + pdf_objs.append(font_obj) + + pdf = b"%PDF-1.4\n" + offsets = [0] + for obj in pdf_objs: + offsets.append(len(pdf)) + pdf += obj + + xref_offset = len(pdf) + pdf += f"xref\n0 {len(pdf_objs) + 1}\n".encode("latin-1") + pdf += b"0000000000 65535 f \n" + for offset in offsets[1:]: + pdf += f"{offset:010d} 00000 n \n".encode("latin-1") + pdf += ( + b"trailer\n<< /Size " + + str(len(pdf_objs) + 1).encode("latin-1") + + b" /Root 1 0 R >>\nstartxref\n" + + str(xref_offset).encode("latin-1") + + b"\n%%EOF" + ) + return pdf + + +# --------------------------------------------------------------------------- +# Task 1 — multi-line PDF edge dedup (window-based fix) +# --------------------------------------------------------------------------- + + +class TestTask1MultiLinePdfEdgeDedup: + """Phase 15 Task 1: variable-outer-line corpora no longer lock the dedup out.""" + + def test_strip_paragraph_packed_headers_drops_repeating_block_starts(self): + # Each block leads with the same header line; the second-pass + # dedup should strip them after the page-level pass. + text = "\n\n".join(f"REPEATING HEADER\nBody for block {i}." for i in range(5)) + cleaned, stripped = strip_paragraph_packed_headers(text) + assert stripped == 5 + assert "REPEATING HEADER" not in cleaned + + def test_window_based_dedup_strips_deeper_constant_line(self): + """Variable outer line + constant deeper line — the pre-Phase-15 bug.""" + from forgelm.ingestion import _strip_repeating_page_lines + + pages = [f"Section {i} Title\nPUBLICATION IDENTIFIER\n\nBody of page {i}.\n\n{i + 1}" for i in range(1, 5)] + cleaned, stripped = _strip_repeating_page_lines(pages) + # The constant publication identifier must be stripped despite + # the variable outer line (the audit §1.1 trap). + for page in cleaned: + assert "PUBLICATION IDENTIFIER" not in page + assert stripped >= 4 # at least one strip per page + + def test_window_based_dedup_strips_deeper_constant_bottom_line(self): + """Round-3 regression: the same audit §1.1 trap at the BOTTOM edge. + + CodeRabbit caught that the pre-round-3 tail walk was asymmetric: + it stopped at the first non-match instead of inspecting every + position in the window. A page shaped like + ``[..., "FOOTER", ""]`` would leave the constant + ``FOOTER`` stranded because the varying page number on the + last line broke the peel loop on pass 1. The symmetric tail + walk below covers that case. + """ + from forgelm.ingestion import _strip_repeating_page_lines + + # FOOTER is constant one row above the varying page number. + pages = [f"Body of page {i}.\n\nFOOTER\n\n{i}" for i in range(1, 5)] + cleaned, stripped = _strip_repeating_page_lines(pages) + for page in cleaned: + assert "FOOTER" not in page + assert stripped >= 4 + + def test_short_doc_skips_dedup(self): + """Documents with fewer than 3 pages keep the pre-15 no-op behaviour.""" + from forgelm.ingestion import _strip_repeating_page_lines + + pages = ["HEADER\nbody1", "HEADER\nbody2"] + cleaned, stripped = _strip_repeating_page_lines(pages) + assert stripped == 0 + assert cleaned == pages + + @pytest.mark.skipif(not HAS_PYPDF, reason="pypdf extra not installed") + def test_synthesised_pdf_with_variable_outer_constant_inner_dedups(self, tmp_path): + """End-to-end: variable outer + constant inner header gets dedup'd.""" + pages = [ + ["Section 1 Title", "PUBLICATION ID 12345", "Body of page one is here."], + ["Section 2 Title", "PUBLICATION ID 12345", "Body of page two is here."], + ["Section 3 Title", "PUBLICATION ID 12345", "Body of page three is here."], + ["Section 4 Title", "PUBLICATION ID 12345", "Body of page four is here."], + ] + pdf_path = tmp_path / "multipage.pdf" + pdf_path.write_bytes(_synth_multipage_pdf(pages)) + out = tmp_path / "out.jsonl" + result = ingest_path( + str(pdf_path), + output_path=str(out), + strategy="paragraph", + chunk_size=4000, + keep_frontmatter=True, # opt out of the heuristic for this test + ) + all_text = "\n".join(_read_jsonl(out)) + assert "PUBLICATION ID 12345" not in all_text + assert result.pdf_header_footer_lines_stripped > 0 + + +# --------------------------------------------------------------------------- +# Task 2 — Unicode / script sanity layer +# --------------------------------------------------------------------------- + + +class TestTask2ScriptSanity: + """Phase 15 Task 2: out-of-script char ratio drives a WARNING.""" + + def test_check_script_sanity_fires_on_corrupt_glyphs(self): + from forgelm._script_sanity import check_script_sanity + + # ø Õ ú ÷ ࡟ are all out-of-script for Turkish (Latin Extended-A range). + # Mix in plenty of regular ASCII to keep the ratio measurable. + text = "Body text only contains regular ASCII " * 5 + "øÕú÷࡟øÕú÷࡟øÕú÷࡟" + report = check_script_sanity( + text, + language_hint="tr", + file_path="test.txt", + threshold=0.01, + profile="none", # disable the emit-list allowance + ) + assert report.triggered + assert report.ratio > 0.01 + assert any(glyph == "ø" for glyph, _ in report.top_offenders) + + def test_check_script_sanity_no_op_on_unknown_hint(self): + from forgelm._script_sanity import check_script_sanity + + report = check_script_sanity( + "any text at all", + language_hint="zz", # unknown + file_path="test.txt", + ) + assert not report.triggered + assert report.ratio == pytest.approx(0.0) + + def test_check_script_sanity_silent_on_clean_text(self): + from forgelm._script_sanity import check_script_sanity + + report = check_script_sanity( + "Sadece düzgün Türkçe metin içeren bir cümle. " * 20, + language_hint="tr", + file_path="clean.txt", + ) + assert not report.triggered + assert report.ratio < 0.01 + + def test_profile_allowlist_protects_normaliser_output(self): + """Characters the normaliser is allowed to emit are in-script.""" + from forgelm._script_sanity import check_script_sanity + + # • U+2022 is a normaliser-emitted bullet; with profile=turkish it + # should not count as out-of-script even though it's outside the + # strict Latin Extended-A range. + text = "Body text with bullets • • • appearing repeatedly across the chunk. " * 10 + report = check_script_sanity(text, language_hint="tr", file_path="bullets.txt", profile="turkish") + assert not report.triggered + + +# --------------------------------------------------------------------------- +# Task 3 — Turkish pypdf glyph normalisation table +# --------------------------------------------------------------------------- + + +class TestTask3TurkishGlyphNormalisation: + """Phase 15 Task 3: corrupt Turkish glyphs map back to the correct chars.""" + + def test_apply_profile_substitutes_all_documented_glyphs(self): + from forgelm._pypdf_normalise import apply_profile + + corrupt = "ø Õ ú ÷ ࡟" + cleaned = apply_profile(corrupt, "turkish") + assert "ø" not in cleaned + assert "Õ" not in cleaned + assert "ú" not in cleaned + assert "÷" not in cleaned + assert "࡟" not in cleaned + # All five canonical Turkish targets show up. + for target in ("İ", "ı", "ş", "ğ", "•"): + assert target in cleaned + + def test_apply_profile_none_is_identity(self): + from forgelm._pypdf_normalise import apply_profile + + assert apply_profile("ø Õ ú", "none") == "ø Õ ú" + + def test_unknown_profile_silently_no_ops(self): + from forgelm._pypdf_normalise import apply_profile + + assert apply_profile("ø Õ ú", "klingon") == "ø Õ ú" + + def test_multichar_substitution_applied_before_single_char(self): + from forgelm._pypdf_normalise import apply_profile + + # ``ö `` (with trailing space) maps to ``Ğ`` — the multi-char rule + # must fire before a single-char ``ö`` substitution could land + # (none today, but the ordering protects future additions). + assert apply_profile("ö Body", "turkish") == "ĞBody" + + +# --------------------------------------------------------------------------- +# Task 4 — Ingest-time quality pre-signal +# --------------------------------------------------------------------------- + + +class TestTask4QualityPresignal: + """Phase 15 Task 4: ingest_path emits a quality_presignal block.""" + + def test_quality_presignal_populates_on_clean_text(self, tmp_path): + src = tmp_path / "doc.txt" + src.write_text("A clean paragraph of normal English text.\n\nAnother clean paragraph here.\n") + out = tmp_path / "out.jsonl" + result = ingest_path(str(src), output_path=str(out), strategy="paragraph") + ns = result.notes_structured + assert "quality_presignal" in ns + assert ns["quality_presignal"]["samples_evaluated"] >= 1 + + def test_quality_presignal_flags_short_repetitive_text(self, tmp_path): + src = tmp_path / "noisy.txt" + # Lots of repeated single-line chunks — should trip repeated_lines + # and/or low_alpha checks (depending on chunker boundaries). + src.write_text("RT\n\nRT\n\nRT\n\nRT\n\n") + out = tmp_path / "out.jsonl" + result = ingest_path(str(src), output_path=str(out), strategy="paragraph") + # Some chunk should have been flagged. + ns = result.notes_structured + # Either flagged or no chunks (depends on chunker filtering); both + # are correct contract — we only assert the block exists. + assert "quality_presignal" in ns + + def test_no_quality_presignal_when_disabled(self, tmp_path): + src = tmp_path / "doc.txt" + src.write_text("Body text.") + out = tmp_path / "out.jsonl" + result = ingest_path( + str(src), + output_path=str(out), + strategy="paragraph", + quality_presignal=False, + ) + assert "quality_presignal" not in result.notes_structured + + +# --------------------------------------------------------------------------- +# Task 6 — DOCX explicit header / footer extraction +# --------------------------------------------------------------------------- + + +@pytest.mark.skipif(not HAS_DOCX, reason="python-docx extra not installed") +class TestTask6DocxHeaderFooter: + """Phase 15 Task 6: header/footer lines are subtracted from the body.""" + + def test_docx_header_subtracted_from_body(self, tmp_path): + from docx import Document + + doc = Document() + section = doc.sections[0] + # python-docx exposes header via .header; first paragraph is the + # editable one. + section.header.paragraphs[0].text = "ACME CONFIDENTIAL" + section.footer.paragraphs[0].text = "Page footer text" + + # Body paragraphs include the same header line we just declared. + for i in range(3): + doc.add_paragraph("ACME CONFIDENTIAL") + doc.add_paragraph(f"Body paragraph {i} that the chunker keeps.") + + docx_path = tmp_path / "doc.docx" + doc.save(str(docx_path)) + + out = tmp_path / "out.jsonl" + ingest_path(str(docx_path), output_path=str(out), strategy="paragraph") + all_text = "\n".join(_read_jsonl(out)) + assert "ACME CONFIDENTIAL" not in all_text + assert "Body paragraph" in all_text + + +# --------------------------------------------------------------------------- +# Task 7 — EPUB spine + nav / cover / copyright skip +# --------------------------------------------------------------------------- + + +@pytest.mark.skipif(not HAS_EPUB, reason="ebooklib extra not installed") +class TestTask7EpubSpineSkip: + """Phase 15 Task 7: nav / cover / copyright items are skipped by default.""" + + def test_epub_skips_nav_and_cover(self, tmp_path): + from ebooklib import epub + + book = epub.EpubBook() + book.set_identifier("phase-15-fixture") + book.set_title("Phase 15 fixture") + book.set_language("en") + + cover = epub.EpubHtml(title="Cover", file_name="cover.xhtml", content="

COVER PAGE

") + nav_html = epub.EpubHtml(title="Nav", file_name="nav.xhtml", content="

NAV TOC

") + chap = epub.EpubHtml( + title="Chapter 1", + file_name="chap1.xhtml", + content="

Body of chapter one.

", + ) + for item in (cover, nav_html, chap): + book.add_item(item) + book.spine = [cover, nav_html, chap] + book.toc = [chap] + book.add_item(epub.EpubNcx()) + book.add_item(epub.EpubNav()) + + epub_path = tmp_path / "book.epub" + epub.write_epub(str(epub_path), book) + + out = tmp_path / "out.jsonl" + ingest_path(str(epub_path), output_path=str(out), strategy="paragraph") + all_text = "\n".join(_read_jsonl(out)) + assert "COVER PAGE" not in all_text + assert "NAV TOC" not in all_text + assert "Body of chapter one." in all_text + + +# --------------------------------------------------------------------------- +# Task 8 — TXT BOM strip + MD frontmatter detection +# --------------------------------------------------------------------------- + + +class TestTask8TxtBomMdFrontmatter: + """Phase 15 Task 8: UTF-8 BOM + YAML frontmatter are stripped by default.""" + + def test_txt_with_bom_strips_bom(self, tmp_path): + out = tmp_path / "out.jsonl" + ingest_path( + str(FIXTURE_DIR / "txt_with_bom_crlf.txt"), + output_path=str(out), + strategy="paragraph", + ) + all_text = "\n".join(_read_jsonl(out)) + # The BOM character U+FEFF must not survive into the JSONL. + assert "" not in all_text + assert "Header Line One" in all_text + + def test_md_frontmatter_stripped_by_default(self, tmp_path): + out = tmp_path / "out.jsonl" + ingest_path( + str(FIXTURE_DIR / "md_with_frontmatter_and_html.md"), + output_path=str(out), + strategy="paragraph", + ) + all_text = "\n".join(_read_jsonl(out)) + assert "title:" not in all_text + assert "author:" not in all_text + assert "Body heading" in all_text + + def test_md_keep_frontmatter_opt_in(self, tmp_path): + out = tmp_path / "out.jsonl" + ingest_path( + str(FIXTURE_DIR / "md_with_frontmatter_and_html.md"), + output_path=str(out), + strategy="paragraph", + keep_md_frontmatter=True, + ) + all_text = "\n".join(_read_jsonl(out)) + assert "title:" in all_text + + +# --------------------------------------------------------------------------- +# Task 10 — mixed_directory recursive ingest +# --------------------------------------------------------------------------- + + +class TestTask10MixedDirectory: + """Phase 15 Task 10: cross-format ingest produces lex-sorted, deterministic output.""" + + def test_mixed_directory_recursive_produces_both_formats(self, tmp_path): + out = tmp_path / "out.jsonl" + result = ingest_path( + str(FIXTURE_DIR / "mixed_directory"), + output_path=str(out), + strategy="paragraph", + recursive=True, + ) + assert result.files_processed >= 2 + # format_counts should record both .txt and .md. + formats = set(result.format_counts.keys()) + assert ".txt" in formats + assert ".md" in formats + + def test_recursive_walk_order_is_lex_sorted(self, tmp_path): + # Synthesize a fresh dir with files whose lex order differs from + # filesystem insertion order to confirm the walker sorts. + d = tmp_path / "ordered" + d.mkdir() + (d / "zebra.txt").write_text("Zebra body text.\n\nMore zebra content.\n") + (d / "alpha.txt").write_text("Alpha body text.\n\nMore alpha content.\n") + out = tmp_path / "out.jsonl" + ingest_path( + str(d), + output_path=str(out), + strategy="paragraph", + recursive=True, + ) + chunks = _read_jsonl(out) + # The first chunk must come from alpha.txt, the second batch from zebra.txt. + assert "Alpha" in chunks[0] + # The last chunk should originate from zebra.txt. + assert any("Zebra" in chunk for chunk in chunks) + + +# --------------------------------------------------------------------------- +# Task 11 — --strip-pattern with ReDoS guard +# --------------------------------------------------------------------------- + + +class TestTask11StripPattern: + """Phase 15 Wave 2 Task 11: operator regex applied + ReDoS-validated.""" + + def test_strip_pattern_removes_matching_lines(self, tmp_path): + src = tmp_path / "doc.txt" + src.write_text("Normal body text.\nWATERMARK ID 12345\nMore body text.\n\nWATERMARK ID 99999\nFinal.") + out = tmp_path / "out.jsonl" + ingest_path( + str(src), + output_path=str(out), + strategy="paragraph", + strip_patterns=[r"^WATERMARK ID \d+$"], + ) + all_text = "\n".join(_read_jsonl(out)) + assert "WATERMARK" not in all_text + assert "Normal body text" in all_text + + def test_strip_pattern_redos_shape_rejected(self, tmp_path): + src = tmp_path / "doc.txt" + src.write_text("body") + out = tmp_path / "out.jsonl" + with pytest.raises(ValueError, match="ReDoS"): + ingest_path( + str(src), + output_path=str(out), + strip_patterns=[r"(a+)+b"], + ) + + def test_strip_pattern_invalid_regex_rejected(self, tmp_path): + src = tmp_path / "doc.txt" + src.write_text("body") + out = tmp_path / "out.jsonl" + with pytest.raises(ValueError, match="not a valid regular expression"): + ingest_path(str(src), output_path=str(out), strip_patterns=["[unclosed"]) + + +# --------------------------------------------------------------------------- +# Task 12 — --page-range PDF flag +# --------------------------------------------------------------------------- + + +@pytest.mark.skipif(not HAS_PYPDF, reason="pypdf extra not installed") +class TestTask12PageRange: + """Phase 15 Wave 2 Task 12: page-range slices PDF extraction.""" + + def test_page_range_extracts_only_requested_pages(self, tmp_path): + pages = [ + ["First page content here."], + ["Second page content here."], + ["Third page content here."], + ["Fourth page content here."], + ] + pdf = tmp_path / "doc.pdf" + pdf.write_bytes(_synth_multipage_pdf(pages)) + out = tmp_path / "out.jsonl" + ingest_path( + str(pdf), + output_path=str(out), + page_range=(2, 3), + keep_frontmatter=True, + quality_presignal=False, + ) + all_text = "\n".join(_read_jsonl(out)) + assert "First page" not in all_text + assert "Fourth page" not in all_text + assert "Second page" in all_text or "Third page" in all_text + + def test_page_range_invalid_start_raises(self, tmp_path): + pages = [["A"], ["B"]] + pdf = tmp_path / "doc.pdf" + pdf.write_bytes(_synth_multipage_pdf(pages)) + out = tmp_path / "out.jsonl" + with pytest.raises(ValueError, match="page-range"): + ingest_path(str(pdf), output_path=str(out), page_range=(0, 1)) + + def test_page_range_start_after_end_raises(self, tmp_path): + pages = [["A"], ["B"]] + pdf = tmp_path / "doc.pdf" + pdf.write_bytes(_synth_multipage_pdf(pages)) + out = tmp_path / "out.jsonl" + with pytest.raises(ValueError, match="page-range"): + ingest_path(str(pdf), output_path=str(out), page_range=(5, 1)) + + +# --------------------------------------------------------------------------- +# Task 13 — Front-matter heuristic (default ON in v0.6.0) +# --------------------------------------------------------------------------- + + +class TestTask13FrontmatterHeuristic: + """Phase 15 Wave 2 Task 13: heuristic drops alpha < 0.45 + underscore > 0.10 + ≥ 5 page numbers.""" + + def test_is_frontmatter_page_fires_on_toc_shape(self): + from forgelm.ingestion import _is_frontmatter_page + + toc_page = ( + "Chapter 1______________________________________ 10\n" + "Chapter 2______________________________________ 25\n" + "Chapter 3______________________________________ 40\n" + "\n10\n\n11\n\n12\n\n13\n\n14\n" + ) + assert _is_frontmatter_page(toc_page) + + def test_is_frontmatter_page_skips_clean_body(self): + from forgelm.ingestion import _is_frontmatter_page + + body = ( + "This is a perfectly normal page of body content covering one or two " + "topics with full sentences and no ToC dotted leaders or repeating page " + "numbers. It should not trip the heuristic." * 3 + ) + assert not _is_frontmatter_page(body) + + +# --------------------------------------------------------------------------- +# Task 14 — URL strip option +# --------------------------------------------------------------------------- + + +class TestTask14StripUrls: + """Phase 15 Wave 2 Task 14: --strip-urls mask|strip|keep applies post-extract.""" + + def test_strip_urls_mask_replaces_url_with_placeholder(self, tmp_path): + src = tmp_path / "doc.txt" + src.write_text("Visit https://example.com/path?q=1 for more.\n\nAlso http://foo.bar/baz.\n") + out = tmp_path / "out.jsonl" + ingest_path( + str(src), + output_path=str(out), + strategy="paragraph", + strip_urls="mask", + ) + all_text = "\n".join(_read_jsonl(out)) + assert "https://example.com" not in all_text + assert "[URL]" in all_text + + def test_strip_urls_strip_removes_url(self, tmp_path): + src = tmp_path / "doc.txt" + src.write_text("Visit https://example.com/path for more.\n") + out = tmp_path / "out.jsonl" + ingest_path( + str(src), + output_path=str(out), + strategy="paragraph", + strip_urls="strip", + ) + all_text = "\n".join(_read_jsonl(out)) + assert "https://" not in all_text + assert "Visit for more." in all_text + + def test_strip_urls_keep_default(self, tmp_path): + src = tmp_path / "doc.txt" + src.write_text("Visit https://example.com/path for more.\n") + out = tmp_path / "out.jsonl" + ingest_path(str(src), output_path=str(out), strategy="paragraph") + all_text = "\n".join(_read_jsonl(out)) + assert "https://example.com" in all_text + + def test_strip_urls_invalid_mode_rejected(self, tmp_path): + src = tmp_path / "doc.txt" + src.write_text("body") + out = tmp_path / "out.jsonl" + with pytest.raises(ValueError, match="strip_urls"): + ingest_path(str(src), output_path=str(out), strip_urls="bogus") + + +# --------------------------------------------------------------------------- +# Task 15 — Multi-column PDF detection (warning only) +# --------------------------------------------------------------------------- + + +@pytest.mark.skipif(not HAS_PYPDF, reason="pypdf extra not installed") +class TestTask15MultiColumnWarning: + """Phase 15 Wave 2 Task 15: warns on two-cluster x-distribution. No fix.""" + + def test_single_column_no_warning(self, tmp_path, caplog): + # Single-column PDF: every Tj sits at x=72. + pages = [["Body content line one.", "Body content line two."]] + pdf = tmp_path / "single.pdf" + pdf.write_bytes(_synth_multipage_pdf(pages)) + out = tmp_path / "out.jsonl" + with caplog.at_level("WARNING"): + ingest_path( + str(pdf), + output_path=str(out), + strategy="paragraph", + keep_frontmatter=True, + quality_presignal=False, + ) + assert not any("2-column" in r.message for r in caplog.records) + + def test_positive_case_fires_warning_via_mocked_visitor(self, caplog): + """Round-2 positive-case coverage for ``_maybe_warn_multi_column``. + + Hand-rolling a two-column PDF byte stream (via _synth_multipage_pdf) + does not let us control the Tj x-coordinates pypdf surfaces to the + visitor callback (they all end up at x=72). We bypass the byte + layer and call the probe with a fake reader whose pages emit two + clearly-separated x-clusters; the warning must fire. + """ + from unittest.mock import MagicMock + + from forgelm.ingestion import _maybe_warn_multi_column + + fake_page = MagicMock() + fake_page.mediabox.width = 612.0 + + def fake_extract_text(visitor_text=None, **_kwargs): + if visitor_text is None: + return "" + # Two clusters: left column around x=72, right column around x=380. + for x in (72, 73, 74, 75, 380, 381, 382, 383) * 6: + visitor_text("body", None, (1, 0, 0, 1, x, 700), None, 11) + return "body" + + fake_page.extract_text = fake_extract_text + fake_reader = MagicMock() + fake_reader.pages = [fake_page] + + with caplog.at_level("WARNING"): + fired = _maybe_warn_multi_column(fake_reader, "fake.pdf") + assert fired + assert any("2-column" in r.message for r in caplog.records) + + def test_partial_token_mode_rejected_at_notebook_layer(self): + """Notebook Cell 5 fail-fast: half-set CHUNK_TOKENS/TOKENIZER must abort. + + Round-3 review: the pre-round-3 version of this test recomputed + the predicate from hardcoded pairs and asserted a tautology, so + a regression deleting the notebook's fail-fast check would still + pass the test. We now parse the actual notebook JSON and assert + that Cell 5's source carries both (a) the partial-config + ``ValueError`` raise and (b) the predicate string the cell uses. + Catches accidental cell-source deletion or rewording that drops + the safety net. + """ + import json + from pathlib import Path + + nb_path = Path(__file__).resolve().parent.parent / "notebooks" / "ingestion_playground.ipynb" + assert nb_path.is_file(), f"notebook fixture missing at {nb_path}" + + nb = json.loads(nb_path.read_text(encoding="utf-8")) + cell5_source = nb["cells"][5]["source"] + if isinstance(cell5_source, list): + cell5_text = "".join(cell5_source) + else: + cell5_text = cell5_source + + # Cell 5 must compute the count via the canonical predicate AND + # raise ValueError when exactly one of CHUNK_TOKENS / TOKENIZER + # is truthy. Both substrings are pinned so a partial-deletion + # regression trips this test. + assert "_token_mode_kwargs_set" in cell5_text, ( + "Cell 5 must compute _token_mode_kwargs_set to detect partial config" + ) + assert "sum(1 for v in (CHUNK_TOKENS, TOKENIZER) if v)" in cell5_text, ( + "Cell 5 predicate must count truthy CHUNK_TOKENS / TOKENIZER" + ) + assert "raise ValueError" in cell5_text, "Cell 5 must raise ValueError on partial token-mode config" + assert "Token-aware chunking requires BOTH" in cell5_text, "Cell 5 error message must name both knobs" + + +# --------------------------------------------------------------------------- +# Cross-cutting: structured notes additive shape (back-compat) +# --------------------------------------------------------------------------- + + +class TestRoundTwoFixes: + """Phase 15 round-2 review absorption — regressions for the new behaviours.""" + + def test_epub_skip_does_not_substring_match(self): + """C-1 regression: ``recovery.xhtml`` must NOT trip the ``cover`` skip token.""" + from forgelm.ingestion import _epub_item_matches_skip + + skip = ("nav", "cover", "copyright", "colophon", "titlepage", "frontmatter") + # Pre-round-2 substring match would have skipped all of these. + assert not _epub_item_matches_skip("recovery.xhtml", "", skip) + assert not _epub_item_matches_skip("discovery_chapter.xhtml", "", skip) + assert not _epub_item_matches_skip("undercover.xhtml", "", skip) + assert not _epub_item_matches_skip("navy.xhtml", "", skip) + assert not _epub_item_matches_skip("navigation_chapter.xhtml", "", skip) + # Canonical cases still get caught. + assert _epub_item_matches_skip("cover.xhtml", "", skip) + assert _epub_item_matches_skip("oebps/nav.xhtml", "", skip) + assert _epub_item_matches_skip("cover-page.xhtml", "", skip) + # ``epub:type`` exact match still works. + assert _epub_item_matches_skip("chapter1.xhtml", "cover", skip) + + def test_normalise_profile_couples_to_language_hint(self, tmp_path): + """C-2 regression: default profile is no-op unless --language-hint=tr. + + Without a language hint, ``--normalise-profile`` must NOT silently + rewrite Nordic / mathematical characters. + """ + src = tmp_path / "doc.txt" + # ``ø``, ``Õ``, ``÷`` are legitimate non-Turkish chars (Nordic / + # Estonian / math). Without language_hint, the normaliser must + # leave them alone. + src.write_text("Visit Bjørk Õrö ÷ ten.\n\nAnother paragraph.\n") + out = tmp_path / "out.jsonl" + ingest_path(str(src), output_path=str(out), strategy="paragraph") + all_text = "\n".join(_read_jsonl(out)) + assert "Bjørk" in all_text + assert "Õrö" in all_text + assert "÷" in all_text + + def test_normalise_profile_active_when_language_hint_is_tr(self, tmp_path): + """C-2 confirm: --language-hint tr auto-enables the turkish profile.""" + src = tmp_path / "doc.txt" + src.write_text("Body text with corruption: ø Õ ú ÷ ࡟\n\nMore body.\n") + out = tmp_path / "out.jsonl" + ingest_path( + str(src), + output_path=str(out), + strategy="paragraph", + language_hint="tr", + ) + all_text = "\n".join(_read_jsonl(out)) + # Audit-measured artefacts are mapped to the correct Turkish chars. + assert "ø" not in all_text + assert "Õ" not in all_text + assert "İ" in all_text + assert "ş" in all_text + + def test_bom_strip_works_on_latin1_fallback(self, tmp_path): + """S-2 regression: BOM is stripped even when the fallback path fires. + + A file with a UTF-8 BOM plus downstream non-UTF-8 bytes used to + leak ``\\ufeff`` because the fallback opened with ``utf-8``, not + ``utf-8-sig``. The fix uses ``utf-8-sig`` on both paths plus an + explicit leading-codepoint strip. + """ + src = tmp_path / "broken.txt" + # BOM + valid text + a stray Latin-1 byte that breaks strict UTF-8. + src.write_bytes(b"\xef\xbb\xbfHello\n\xc3\x28body line\n") + out = tmp_path / "out.jsonl" + ingest_path(str(src), output_path=str(out), strategy="paragraph") + all_text = "\n".join(_read_jsonl(out)) + assert "" not in all_text + assert "Hello" in all_text + + def test_strip_pattern_catches_escape_shape_redos(self, tmp_path): + """S-1 regression: ``(\\w+)+x`` must be rejected up-front.""" + from forgelm._strip_pattern import StripPatternError + + src = tmp_path / "doc.txt" + src.write_text("body") + out = tmp_path / "out.jsonl" + with pytest.raises((ValueError, StripPatternError), match="ReDoS"): + ingest_path( + str(src), + output_path=str(out), + strip_patterns=[r"(\w+)+x"], + ) + + +class TestRoundFourFixes: + """Round-4 regression coverage for the path-repr / presignal-floor / + EPUB-property-split / frontmatter-alpha-threshold changes that + landed in commit ``dc671f5`` without dedicated tests.""" + + def test_page_range_error_repr_escapes_ansi_in_path(self, tmp_path): + """C-A round-4: path with ANSI escapes must NOT render raw in the error message. + + Pre-round-4 the error used bare ``'{path}'`` interpolation, so a + path containing ``\\x1b[31m...`` rendered the literal escape + sequence into the error string (terminal-injection vector). + The fix switched to ``{path!r}`` which repr-quotes the value. + """ + if not HAS_PYPDF: + pytest.skip("pypdf extra not installed") + # Synthesise a minimal PDF whose filename contains an ANSI + # escape sequence; pass an out-of-range page-range to trigger + # the IngestParameterError. + evil_name = "evil\x1b[31mRED\x1b[0m.pdf" + evil_path = tmp_path / evil_name + evil_path.write_bytes(_synth_multipage_pdf([["body"]])) + out = tmp_path / "out.jsonl" + with pytest.raises(ValueError) as excinfo: + ingest_path(str(evil_path), output_path=str(out), page_range=(99, 999)) + # The error message must repr-escape the path; the literal ESC + # byte (\x1b == \x1b) must NOT appear unescaped in the rendered + # message. ``repr()`` of an ESC-bearing string renders ``\x1b``. + msg = str(excinfo.value) + assert "\x1b" not in msg, "ANSI escape must be repr-escaped in error message" + + def test_presignal_floor_silent_on_short_clean_corpus(self, tmp_path): + """C-B round-4: clean 5-paragraph 41-char corpus must NOT trip the alpha-ratio check. + + Pre-round-4 the alpha-ratio check ran on every chunk regardless + of size. A perfectly clean 5-para 41-char TXT flagged 1/1 + chunks because dots + whitespace dragged alpha < 0.70. + ``_PRESIGNAL_MIN_NON_WS_CHARS = 80`` floor suppresses the + false-positive. + """ + src = tmp_path / "doc.txt" + src.write_text("Para 1.\n\nPara 2.\n\nPara 3.\n\nPara 4.\n\nPara 5.") + out = tmp_path / "out.jsonl" + result = ingest_path(str(src), output_path=str(out), strategy="paragraph") + qp = result.notes_structured.get("quality_presignal", {}) + assert qp.get("samples_flagged", -1) == 0, "clean small corpus must not flag" + assert qp.get("by_check", {}).get("alpha_ratio", -1) == 0 + + @pytest.mark.skipif(not HAS_EPUB, reason="ebooklib extra not installed") + def test_epub_item_type_split_on_multi_property_string(self): + """S-C round-4: ``properties="nav cover-image"`` must match via tokens, not exact-string. + + EPUB-3 manifest properties are space-separated tokens. The + pre-round-4 code exact-matched the joined form against the + skip-list, silently passing through any item with > 1 property. + """ + from forgelm.ingestion import _epub_item_matches_skip + + skip = ("nav", "toc", "cover", "copyright", "colophon", "titlepage", "frontmatter") + # Multi-token property strings (real EPUB-3 manifest shape). + assert _epub_item_matches_skip("ch01.xhtml", "nav cover-image", skip) + assert _epub_item_matches_skip("ch01.xhtml", "scripted mathml nav", skip) + # Single token that matches — back-compat. + assert _epub_item_matches_skip("any.xhtml", "cover", skip) + # Empty property string falls through to filename check. + assert not _epub_item_matches_skip("body.xhtml", "", skip) + # Property string with NO skip token must NOT match. + assert not _epub_item_matches_skip("body.xhtml", "scripted mathml", skip) + + def test_frontmatter_threshold_audit_pilot_still_caught(self): + """S-E round-4 / S-C round-4: audit's Turkish-pilot ToC shape (≈ 0.15 alpha) + must still trip the heuristic after the 0.45 → 0.30 tightening.""" + from forgelm.ingestion import _is_frontmatter_page + + # Audit pilot shape: short Turkish chapter titles + heavy + # dotted leaders + inline page numbers. + toc = ( + "Bölüm 1 ........................ 14\n" + "Bölüm 2 ........................ 28\n" + "Bölüm 3 ........................ 42\n" + "Bölüm 4 ........................ 67\n" + "Bölüm 5 ........................ 95\n" + "\n14\n\n28\n\n42\n\n67\n\n95\n" + ) + assert _is_frontmatter_page(toc) + + def test_frontmatter_protects_realistic_form_template_via_3signal_AND(self): + """S-E round-4: real form templates have NO inline page numbers, so the + 3-signal AND filter (alpha + leader + page-num-count) protects them + even when alpha + leader trip individually.""" + from forgelm.ingestion import _is_frontmatter_page + + # Realistic form template — labels + underscore leaders + NO page numbers. + form = ( + "Name: ___________________\n" + "Date: ___________________\n" + "Phone: __________________\n" + "Address: ________________\n" + "City: ___________________\n" + ) + # Without the inline-page-number signal, the 3-signal AND filter + # keeps the page even though alpha + leader ratios are low. + assert not _is_frontmatter_page(form) + + +class TestStructuredNotesAdditive: + """Phase 15 contract: notes_structured additions never rename pre-15 keys.""" + + def test_pre15_keys_still_present_on_clean_run(self, tmp_path): + src = tmp_path / "doc.txt" + src.write_text("Body.\n\nMore body.\n") + out = tmp_path / "out.jsonl" + result = ingest_path(str(src), output_path=str(out), strategy="paragraph") + ns = result.notes_structured + for key in ( + "files_processed", + "files_skipped", + "chunk_count", + "total_chars", + "strategy", + "format_counts", + "pii_redaction_counts", + "secrets_redaction_counts", + ): + assert key in ns, f"Pre-Phase-15 key {key!r} disappeared" + + +if __name__ == "__main__": + import sys + + sys.exit(pytest.main([__file__, "-v"])) diff --git a/tests/test_pypdf_normalise.py b/tests/test_pypdf_normalise.py new file mode 100644 index 00000000..bdaaf3e3 --- /dev/null +++ b/tests/test_pypdf_normalise.py @@ -0,0 +1,129 @@ +"""Phase 15 Task 3 unit tests — pypdf glyph-normalisation profile.""" + +from __future__ import annotations + +import pytest + +from forgelm._pypdf_normalise import ( + DEFAULT_PROFILE, + PROFILES, + apply_profile, + count_substitutions, + emitted_codepoints, + profile_summary, +) + + +class TestProfileVocabulary: + def test_default_profile_is_none_after_round2_fix(self): + # Round-2 review (C-2): the module-level default flipped from + # "turkish" to "none" so a library caller without an explicit + # language hint does not silently rewrite non-Turkish text. + # CLI dispatcher (`_resolve_normalise_profile`) + ingest_path + # auto-derive "turkish" when ``language_hint="tr"`` is set. + assert DEFAULT_PROFILE == "none" + + def test_profiles_includes_none_for_explicit_opt_out(self): + assert "none" in PROFILES + assert "turkish" in PROFILES + + +class TestApplyProfile: + def test_turkish_profile_round_trip_canonical_glyphs(self): + # Audit-measured artefacts: ø Õ ú ÷ ࡟ → İ ı ş ğ • + assert apply_profile("ø Õ ú ÷ ࡟", "turkish") == "İ ı ş ğ •" + + def test_multichar_substitution_fires_before_single_char(self): + # ``ö `` (with trailing space) maps to ``Ğ`` — the multi-char + # pass must run before the single-char str.translate() lookup. + assert apply_profile("ö abc", "turkish") == "Ğabc" + + def test_none_profile_returns_input_unchanged(self): + assert apply_profile("ø Õ ú", "none") == "ø Õ ú" + + def test_unknown_profile_silently_no_ops(self): + # Forward compat: don't crash an ingest run on an unknown profile. + assert apply_profile("ø Õ ú", "klingon") == "ø Õ ú" + + def test_empty_text_returns_empty(self): + assert apply_profile("", "turkish") == "" + + def test_preserves_clean_turkish_text(self): + # The normaliser must never rewrite correct Turkish characters. + clean = "Türkçe metin doğru karakterler içerir." + assert apply_profile(clean, "turkish") == clean + + +class TestProfileSummary: + def test_turkish_summary_returns_table_sizes(self): + summary = profile_summary("turkish") + assert summary["single"] >= 5 + assert summary["multi"] >= 1 + + def test_none_profile_summary_is_zero(self): + assert profile_summary("none") == {"multi": 0, "single": 0} + + def test_unknown_profile_summary_is_zero(self): + assert profile_summary("klingon") == {"multi": 0, "single": 0} + + +class TestEmittedCodepoints: + def test_turkish_emit_set_contains_targets(self): + emit = emitted_codepoints("turkish") + for target in ("İ", "ı", "ş", "ğ", "•", "Ğ"): + assert target in emit + + def test_none_profile_emits_nothing(self): + assert emitted_codepoints("none") == frozenset() + + +class TestCountSubstitutions: + """Round-3 review N-1: pin the exact-count contract. + + ``count_substitutions`` replaced the zip-diff heuristic ingestion.py + used to log substitution counts. The function must return EXACTLY + the number of rules that would fire under :func:`apply_profile`. + """ + + def test_empty_text_returns_zero(self): + assert count_substitutions("", "turkish") == 0 + + def test_none_profile_returns_zero(self): + assert count_substitutions("ø Õ ú ÷", "none") == 0 + + def test_unknown_profile_returns_zero(self): + assert count_substitutions("ø Õ ú ÷", "klingon") == 0 + + def test_single_char_substitutions_counted_per_occurrence(self): + # Five distinct single-char artefacts, one occurrence each → 5. + assert count_substitutions("ø Õ ú ÷ ࡟", "turkish") == 5 + + def test_multi_char_rule_counted_once_per_match(self): + # ``"ö "`` (with trailing space) → ``"Ğ"``; single occurrence. + assert count_substitutions("ö abc", "turkish") == 1 + + def test_multi_char_rule_consumes_trigger_before_single_pass(self): + # ``"ö ø"`` → multi-char rule fires on ``"ö "`` (count 1), then + # ``ø`` single-char rule fires on the trailing ``ø`` (count 1). + # Total = 2; matches apply_profile result ``"Ğİ"``. + assert count_substitutions("ö ø", "turkish") == 2 + + def test_repeated_single_char_counted_per_occurrence(self): + # ``"øøøø"`` → four substitutions even though only one rule key. + assert count_substitutions("øøøø", "turkish") == 4 + + def test_clean_turkish_text_returns_zero(self): + # No artefact characters → no substitutions. + assert count_substitutions("Türkçe metin doğru karakterler içerir.", "turkish") == 0 + + def test_partial_multi_char_match_does_not_count_single_consumed(self): + # ``"öö abc"`` → only ONE ``"ö "`` match (the SECOND ``ö`` + # followed by space); the first ``ö`` has no trailing space. + # apply_profile produces ``"öĞabc"`` → 1 substitution. + assert count_substitutions("öö abc", "turkish") == 1 + + +if __name__ == "__main__": + import sys + + sys.exit(pytest.main([__file__, "-v"])) diff --git a/tests/test_script_sanity.py b/tests/test_script_sanity.py new file mode 100644 index 00000000..fdcf3e11 --- /dev/null +++ b/tests/test_script_sanity.py @@ -0,0 +1,172 @@ +"""Phase 15 Task 2 unit tests — Unicode / script-sanity checker.""" + +from __future__ import annotations + +import logging + +import pytest + +from forgelm._script_sanity import ( + DEFAULT_THRESHOLD, + SUPPORTED_LANGUAGES, + ScriptSanityReport, + check_script_sanity, + report_to_structured, +) + + +class TestDefaults: + def test_threshold_is_calibrated_for_audit_findings(self): + # Phase 15 calibration: catches 7 % corruption (front matter) and + # 3 % corruption (bullet chapter) while keeping 1 % mixed-script + # corpora silent. 1.5 % sits in the middle. + assert DEFAULT_THRESHOLD == pytest.approx(0.015) + + def test_supported_languages_includes_turkish_and_english(self): + assert "tr" in SUPPORTED_LANGUAGES + assert "en" in SUPPORTED_LANGUAGES + + +class TestCheckScriptSanity: + def test_fires_on_audit_canonical_glyphs(self, caplog): + # The audit's two-chunk front-matter corruption shape: ~7 % weird chars. + # Mix a smaller clean prefix with enough corrupt glyphs to clear + # the 1.5 % threshold by a clean margin. + text = "Sadece düzgün bir Türkçe paragraf. " * 10 + "øø÷÷ÕÕúú" * 8 + with caplog.at_level(logging.WARNING): + report = check_script_sanity( + text, + language_hint="tr", + file_path="probe.pdf", + threshold=0.02, + profile="none", + ) + assert report.triggered + assert any(g == "ø" for g, _ in report.top_offenders) + # Warning fired loudly enough that a CI log scrape catches it. + assert any("Script-sanity warning" in r.message for r in caplog.records) + + def test_silent_on_clean_turkish(self): + clean = "Türkçe metin doğru karakterler içerir. " * 50 + report = check_script_sanity(clean, language_hint="tr", file_path="clean.txt") + assert not report.triggered + assert report.ratio == pytest.approx(0.0) + + def test_silent_on_unknown_language(self): + # Unknown language hint — check is a no-op (not an error). + report = check_script_sanity("any text", language_hint="zz", file_path="x.txt") + assert not report.triggered + + def test_silent_on_empty_text(self): + report = check_script_sanity("", language_hint="tr", file_path="empty.txt") + assert not report.triggered + + def test_profile_allowlist_protects_bullets(self): + # Bullets are normaliser-emitted under the turkish profile, so + # they should not count as out-of-script. + text = "Bir satır • Başka satır • Yine başka • Daha fazla. " * 20 + report = check_script_sanity(text, language_hint="tr", file_path="b.txt", profile="turkish") + assert not report.triggered + + def test_cause_hint_identifies_font_fallback(self): + text = "Body text " * 100 + "ø" * 50 + "÷" * 50 + report = check_script_sanity( + text, + language_hint="tr", + file_path="font.pdf", + threshold=0.01, + profile="none", + ) + assert report.triggered + assert "font fallback" in report.cause_hint.lower() + + def test_cause_hint_identifies_mojibake(self): + text = "Body text " * 100 + "�" * 100 + report = check_script_sanity( + text, + language_hint="tr", + file_path="moji.txt", + threshold=0.01, + profile="none", + ) + assert report.triggered + assert "mojibake" in report.cause_hint.lower() + + def test_logger_override_for_test_isolation(self): + custom_logger = logging.getLogger("test.script_sanity") + captured: list = [] + handler = logging.Handler() + handler.emit = lambda record: captured.append(record.getMessage()) + custom_logger.addHandler(handler) + custom_logger.setLevel(logging.WARNING) + + try: + check_script_sanity( + "Body. " * 50 + "øø" * 30, + language_hint="tr", + file_path="t.txt", + threshold=0.01, + profile="none", + logger_override=custom_logger, + ) + # Captured by the test-local logger, not the module logger. + assert any("Script-sanity" in m for m in captured) + finally: + # Round-2 nit: avoid leaking the test handler into other tests + # that share the ``test.script_sanity`` logger name. + custom_logger.removeHandler(handler) + + +class TestReportToStructured: + def test_aggregate_with_triggered_reports(self): + reports = [ + ScriptSanityReport( + file_path="a.txt", + language_hint="tr", + total_chars=100, + out_of_script_chars=20, + ratio=0.20, + threshold=0.015, + triggered=True, + top_offenders=(("ø", 10), ("÷", 10)), + cause_hint="pypdf font fallback (Latin-1 substitutes)", + ), + ScriptSanityReport( + file_path="b.txt", + language_hint="tr", + total_chars=100, + out_of_script_chars=0, + ratio=0.0, + threshold=0.015, + triggered=False, + ), + ] + agg = report_to_structured(reports) + assert agg["files_checked"] == 2 + assert agg["files_triggered"] == 1 + assert agg["max_ratio"] == pytest.approx(0.20) + assert "per_file" in agg + assert agg["per_file"][0]["file_path"] == "a.txt" + + def test_aggregate_with_no_triggered_reports(self): + reports = [ + ScriptSanityReport( + file_path="clean.txt", + language_hint="tr", + total_chars=100, + out_of_script_chars=0, + ratio=0.0, + threshold=0.015, + triggered=False, + ) + ] + agg = report_to_structured(reports) + assert agg["files_triggered"] == 0 + # No per_file when nothing triggered — keeps the JSON tight. + assert "per_file" not in agg + + +if __name__ == "__main__": + import sys + + sys.exit(pytest.main([__file__, "-v"])) diff --git a/tests/test_strip_pattern.py b/tests/test_strip_pattern.py new file mode 100644 index 00000000..00c91ac1 --- /dev/null +++ b/tests/test_strip_pattern.py @@ -0,0 +1,147 @@ +"""Phase 15 Wave 2 Task 11 unit tests — operator strip-pattern + ReDoS guard.""" + +from __future__ import annotations + +import re + +import pytest + +from forgelm._strip_pattern import ( + DEFAULT_TIMEOUT_S, + StripPatternError, + apply_strip_patterns, + compile_strip_patterns, + validate_strip_pattern, +) + + +class TestValidateStripPattern: + def test_accepts_simple_pattern(self): + assert validate_strip_pattern(r"^WATERMARK \d+$") == r"^WATERMARK \d+$" + + def test_rejects_empty(self): + with pytest.raises(StripPatternError, match="non-empty"): + validate_strip_pattern("") + + def test_rejects_invalid_regex_syntax(self): + with pytest.raises(StripPatternError, match="not a valid"): + validate_strip_pattern("[unclosed") + + def test_rejects_nested_unbounded_quantifier(self): + # Classic ReDoS shape: (a+)+ — nested unbounded quantifier. + with pytest.raises(StripPatternError, match="ReDoS"): + validate_strip_pattern("(a+)+b") + + @pytest.mark.parametrize( + "pattern", + [ + r"(\w+)+x", # round-2 (S-1): escape-shape nested unbounded. + r"(\d+)+x", + r"(\s+)+x", + r"^(\w+\s+)+x", # classic textbook ReDoS. + r"([abc]+)+d", # character-class nested unbounded. + ], + ) + def test_rejects_escape_and_class_nested_unbounded(self, pattern): + with pytest.raises(StripPatternError, match="ReDoS"): + validate_strip_pattern(pattern) + + @pytest.mark.parametrize( + "pattern", + [ + r"^WATERMARK \d+$", + r"a{1,100}b", + r"\d{1,5}", + r"(\d+)b", # single unbounded inside group, no outer quantifier. + r"[A-Z]+", # standalone unbounded, no nesting. + r"(a+b)+c", # group ends with bounded literal, outer quantifier OK. + r"\b(?:https?|ftp)://[A-Z0-9]+", + ], + ) + def test_accepts_safe_patterns(self, pattern): + # Sanity check: the new forward-walking validator must NOT + # false-positive on patterns whose last group atom is bounded + # or which carry only a single unbounded quantifier overall. + assert validate_strip_pattern(pattern) == pattern + + def test_rejects_dotall_lazy_with_backref(self): + # rule 6: .*? + back-reference under DOTALL is the S5852 shape. + with pytest.raises(StripPatternError, match="DOTALL"): + validate_strip_pattern(r"(?s)(a).*?\1") + + def test_rejects_overlong_pattern(self): + # 3 KB pattern exceeds the 2 KB safety bound. + with pytest.raises(StripPatternError, match="too long"): + validate_strip_pattern("a" * 3000) + + def test_accepts_anchored_negated_class(self): + # The canonical "strip header line" pattern shape. + assert validate_strip_pattern(r"^[A-Z ]+CONFIDENTIAL[A-Z ]*$") + + +class TestCompileStripPatterns: + def test_compile_returns_raw_plus_compiled_pairs(self): + pairs = compile_strip_patterns([r"^WATERMARK$", r"^FOOTER$"]) + assert len(pairs) == 2 + assert all(isinstance(p[1], re.Pattern) for p in pairs) + assert pairs[0][0] == r"^WATERMARK$" + + def test_compile_propagates_validation_error(self): + with pytest.raises(StripPatternError): + compile_strip_patterns([r"(a+)+b"]) + + +class TestApplyStripPatterns: + def test_apply_strips_matching_lines(self): + patterns = compile_strip_patterns([r"^WATERMARK \d+$"]) + text = "Body\nWATERMARK 1\nMore body.\nWATERMARK 2\nFinal." + out, subs = apply_strip_patterns(text, patterns) + assert subs == 2 + assert "WATERMARK" not in out + + def test_apply_no_patterns_returns_input_unchanged(self): + text = "Body text" + assert apply_strip_patterns(text, [])[0] == "Body text" + + def test_apply_empty_text_returns_zero(self): + patterns = compile_strip_patterns([r"^X$"]) + out, subs = apply_strip_patterns("", patterns) + assert out == "" + assert subs == 0 + + def test_apply_chains_patterns_in_order(self): + patterns = compile_strip_patterns([r"^A$", r"^B$"]) + text = "A\nB\nKeep\nA\nB" + out, subs = apply_strip_patterns(text, patterns) + assert subs == 4 + assert "Keep" in out + + def test_default_timeout_constant_documented(self): + # Operator-facing constant — 5s is the documented default in + # `forgelm/_strip_pattern.py` + CLI help text. + assert DEFAULT_TIMEOUT_S == 5 + + def test_apply_rejects_zero_timeout(self): + # Round-2 (S-1): zero / negative timeout_s used to silently + # disable the alarm; now rejected with ValueError so operator + # misconfiguration is loud. + patterns = compile_strip_patterns([r"^X$"]) + with pytest.raises(ValueError, match="positive int"): + apply_strip_patterns("X", patterns, timeout_s=0) + + def test_apply_rejects_negative_timeout(self): + patterns = compile_strip_patterns([r"^X$"]) + with pytest.raises(ValueError, match="positive int"): + apply_strip_patterns("X", patterns, timeout_s=-5) + + def test_apply_accepts_none_timeout(self): + patterns = compile_strip_patterns([r"^X$"]) + # ``None`` is the explicit "no guard" signal; must keep working. + out, _ = apply_strip_patterns("X\nbody", patterns, timeout_s=None) + assert "X" not in out + + +if __name__ == "__main__": + import sys + + sys.exit(pytest.main([__file__, "-v"]))