feat(ingestion): Phase 15 Wave 1+2 — ingestion-pipeline reliability

CHANGELOG.md

@@ -4,6 +4,138 @@ All notable changes to ForgeLM are documented here.
 
 ## [Unreleased]
 
+Phase 15 (Ingestion Pipeline Reliability) Wave 1 + Wave 2 landed on the
+development branch. Closes the silent-failure gap the 2026-05-11 pilot
+exposed across PDF / DOCX / EPUB / TXT / MD ingestion plus the
+user-facing playground notebook. Targeted for v0.6.0.
+
+### Added (Phase 15 Wave 1)
+
+- **`forgelm/_pypdf_normalise.py`** — Turkish glyph normalisation
+  profile (`turkish` default, `none` opt-out, future-pluggable
+  `--normalise-profile`) mapping the audit-measured pypdf font-fallback
+  artefacts (`ø Õ ú ÷ ࡟` → `İ ı ş ğ •`) back to their correct Turkish
+  characters at chunk-write time.
+- **`forgelm/_script_sanity.py`** — language-aware Unicode-block sanity
+  check (`tr` / `en` / `de` / `fr` / `es` / `it` / `pt`) that fires a
+  WARNING + structured `script_sanity_summary` block when the out-of-
+  script ratio exceeds the calibrated 1.5 % threshold. Catches both
+  pypdf font corruption and TXT encoding mis-routing.
+- **`forgelm ingest --language-hint LANG`** + `--script-sanity-threshold`
+  + `--normalise-profile {turkish,none}` + `--no-normalise-unicode` CLI
+  flags wiring Tasks 2 + 3 to the operator surface.
+- **`forgelm ingest` end-of-run quality pre-signal (Task 4)** — three
+  cheap row-level checks (alpha-ratio, weird-char ratio, repeated-line
+  ratio) emitting `[WARN] N/M chunks below ingestion quality threshold`
+  + a structured `quality_presignal` block in `notes_structured`. Opt
+  out via `--no-quality-presignal`.
+- **`forgelm doctor` pypdf-normalise diagnostic (`pypdf_normalise.turkish`)**
+  — confirms the glyph-normalisation table loaded and round-trips
+  cleanly without running a test ingest.
+- **DOCX explicit header / footer extraction (Task 6)** — `_extract_docx`
+  reads `doc.sections[i].header.paragraphs` + `.footer.paragraphs` and
+  subtracts those lines from the body before chunking.
+- **EPUB spine-order + nav / cover / copyright skip (Task 7)** —
+  `_extract_epub` iterates `book.spine` (reading order) instead of
+  file order; default skip-list (`nav`, `cover`, `copyright`,
+  `colophon`, `titlepage`, `frontmatter`) opt-out via
+  `--epub-no-skip-frontmatter`.
+- **TXT UTF-8 BOM strip + MD YAML frontmatter detection (Task 8)** —
+  `_extract_text` / new `_extract_markdown` strip the BOM via
+  `encoding="utf-8-sig"` and detect `---\n...\n---\n` YAML frontmatter;
+  `--keep-md-frontmatter` opts back in.
+- **`forgelm/ingestion.strip_paragraph_packed_headers`** — second-pass
+  dedup against paragraph-packed text catches header lines that
+  survived the page-level pass.
+- **`tests/test_ingestion_reliability.py`** — 36 regression tests
+  locking the Wave 1 + Wave 2 behaviour across PDF / DOCX / EPUB /
+  TXT / MD fixtures.
+
+### Added (Phase 15 Wave 2)
+
+- **`forgelm ingest --strip-pattern REGEX` (Task 11)** — operator-
+  controlled regex stripping with ReDoS guard. Patterns are
+  structurally validated at CLI-parse time (rejects nested unbounded
+  quantifiers like `(a+)+b` and `.*?` + back-reference under DOTALL
+  per the SonarCloud `python:S5852` shape rule), wrapped in a 5-second
+  per-pattern SIGALRM budget on POSIX. Opt out of the timeout via
+  `--strip-pattern-no-timeout`.
+- **`forgelm ingest --page-range START-END` (Task 12)** — restrict
+  PDF extraction to a contiguous page slice (1-indexed inclusive).
+  Validation failures (`start < 1`, `start > end`,
+  `start > page_count`) abort the run with `EXIT_CONFIG_ERROR` via a
+  new `IngestParameterError(ValueError)` that bypasses the per-file
+  soft-fail catch.
+- **PDF front-matter / back-matter heuristic (Task 13, default ON)** —
+  three-signal heuristic (alpha < 0.45 + underscore > 0.10 + ≥ 5
+  inline page-number matches) drops up to 12 leading + 12 trailing
+  pages and emits a WARNING + `frontmatter_pages_dropped` field in
+  `notes_structured`. Opt out via `--keep-frontmatter`.
+- **`forgelm ingest --strip-urls {keep,mask,strip}` (Task 14)** —
+  detected URLs are masked with `[URL]`, stripped outright, or kept
+  (default). Independent of `--all-mask` (URL handling is a
+  content-shape decision, not a GDPR redaction).
+- **PDF multi-column layout warning (Task 15)** — samples the first
+  three pages' Tj-text positions via pypdf's `visitor_text` callback,
+  fires a WARNING when a > 30 %-of-page-width two-cluster gap is
+  detected. No fix attempt — operator switches strategy / pre-
+  processes externally.
+
+### Changed (Phase 15)
+
+- **`forgelm audit --quality-filter` is now default-ON in v0.6.0**
+  (Task 5). Pre-v0.6.0 invocations with explicit `--quality-filter`
+  keep identical behaviour. Operators wanting the pre-v0.6.0 opt-in
+  semantics pass the new `--no-quality-filter` companion flag.
+- **`_strip_repeating_page_lines` window-based dedup (Task 1)** —
+  replaces the pre-Phase-15 outermost-row-only iteration. Catches
+  variable-outer-line + constant-deeper-line corpora (the audit §1.1
+  trap) by inspecting the top-3 / bottom-3 rows per page on every
+  pass. A second pass runs after paragraph packing to mop up survivor
+  headers.
+- **`notebooks/ingestion_playground.ipynb` (Task 9)** — Cell 5 exposes
+  `CHUNK_TOKENS` + `TOKENIZER` + `LANGUAGE_HINT` knobs; Cell 8 markdown
+  explains the new quality-filter checks; Cell 9 passes
+  `--quality-filter` explicitly for forward-compat across v0.5 → v0.6;
+  Cell 10 pretty-prints the `quality_summary` block alongside the
+  existing PII / secrets / near-duplicate / language sections.
+- **`IngestionResult` schema** — additive Phase 15 fields:
+  `pdf_paragraph_packed_lines_stripped`, `script_sanity_triggered`,
+  `strip_pattern_substitutions`, `urls_handled`,
+  `frontmatter_pages_dropped`. No pre-Phase-15 key was renamed.
+
+### Fixed (review-absorption rounds 3 + 4 + 5 follow-up to v0.5.7)
+
+(No public surface change, but training-pipeline UX, doctor robustness,
+and test-isolation guarantees improve. Round 5 closes the residual
+CodeRabbit findings against the round-3/4 commits.)
+
+### Fixed (round 5 — CodeRabbit follow-up)
+
+- **`forgelm/cli/_training.py::_preflight_numpy_torch_abi`** — any
+  unexpected exception from the underlying probe (corrupted torch
+  install where `torch.__version__` raises `AttributeError`, etc.)
+  is now caught and converted into a structured
+  `abi_preflight_crashed` JSON envelope. Previously the raw Python
+  traceback would pre-empt the `--output-format json` contract that
+  every other CLI failure path honours. Exit code stays
+  `EXIT_TRAINING_ERROR` (= 2), matching the broken-ABI verdict so
+  CI/CD branching doesn't need to distinguish "ABI bad" from "ABI
+  probe died".
+- **`CLAUDE.md`** — exit-code contract was stated as `0/1/2/3/4`,
+  but the canonical table in `docs/standards/error-handling.md`
+  documented `0/1/2/3/4/5` since v0.5.5 (Phase 22 added
+  `EXIT_WIZARD_CANCELLED = 5`). CLAUDE.md now matches the actual
+  contract — the standard was right, the agent guidance was stale.
+- **`docs/roadmap/completed-phases.md::Phase 12` summary** — Tier 1
+  status line claimed a `[ingestion-secrets]` extra "via
+  `detect-secrets` with regex fallback", but that extra was never
+  published; only `[ingestion-pii-ml]` exists in `pyproject.toml`'s
+  extras surface. Reworded to record the historical plan accurately
+  ("`detect-secrets` integration was originally planned as a
+  `[ingestion-secrets]` extra but deferred — only
+  `[ingestion-pii-ml]` ultimately shipped"). Pure docs accuracy fix.
+
 ## [0.5.7] — 2026-05-11
 
 Patch release on top of `v0.5.6`. Two production blockers and one UX
@@ -1726,5 +1858,7 @@ Major release: ForgeLM goes from a basic SFT fine-tuning tool to a full-stack LL
 - Basic evaluation checks (max loss, baseline comparison)
 - Auto-revert on quality degradation
 
-[Unreleased]: https://github.com/cemililik/ForgeLM/compare/v0.5.5...HEAD
+[Unreleased]: https://github.com/cemililik/ForgeLM/compare/v0.5.7...HEAD
+[0.5.7]: https://github.com/cemililik/ForgeLM/compare/v0.5.6...v0.5.7
+[0.5.6]: https://github.com/cemililik/ForgeLM/compare/v0.5.5...v0.5.6
 [0.5.5]: https://github.com/cemililik/ForgeLM/compare/v0.5.0...v0.5.5

docs/guides/data_audit-tr.md

@@ -8,8 +8,11 @@ near-duplicate tespitini, streaming JSONL okuyucusunu, PII şiddet
 katmanlarını, atomik disk yazımı ve verbose-by-default kısaltma
 politikasını ekledi. **Faz 12 (`v0.5.0`'da birleşti)** opt-in MinHash LSH dedup
 yöntemini (`--dedup-method minhash`), her zaman çalışan
-code/credential leakage taramasını (`secrets_summary`), ve opt-in
-heuristic kalite filtresini (`--quality-filter`) ekledi.
+code/credential leakage taramasını (`secrets_summary`) ve heuristic
+kalite filtresini (`--quality-filter`) ekledi — başlangıçta opt-in,
+**Faz 15 Görev 5 ile v0.6.0'dan itibaren default-AÇIK**; v0.6.0 öncesi
+opt-in semantiğini isteyen operatörler yeni `--no-quality-filter`
+companion bayrağını kullanır.
 
 Trainer'ın `output_dir`'ünde mevcutsa, rapor EU AI Act Madde 10 veri
 governance artifact'ına otomatik olarak beslenir.
@@ -35,7 +38,9 @@ forgelm audit data/ --near-dup-threshold 5
 pip install 'forgelm[ingestion-scale]'
 forgelm audit data/large_corpus.jsonl --dedup-method minhash --jaccard-threshold 0.85
 
-# Faz 12: opt-in heuristic kalite filtresi (Gopher/C4 stili)
+# Faz 12 + Faz 15: heuristic kalite filtresi (Gopher/C4 stili).
+# v0.6.0'dan itibaren default-AÇIK; explicit flag yedek/zararsızdır.
+# v0.6.0 öncesi opt-in davranışına dönmek için --no-quality-filter geçirin.
 forgelm audit data/ --quality-filter
 
 # Faz 17: çoklu-split corpora için split-seviyesi paralelizm
@@ -484,7 +489,7 @@ forgelm audit PATH \
   [--near-dup-threshold N] \
   [--dedup-method {simhash,minhash}] \
   [--jaccard-threshold X] \
-  [--quality-filter] \
+  [--quality-filter | --no-quality-filter] \
   [--pii-ml] \
   [--pii-ml-language LANG] \
   [--croissant] \
@@ -517,9 +522,11 @@ varsayılan simhash Hamming eşiğini (3, ≈%95 benzerlik) ezer;
 `--dedup-method` (Faz 12) near-duplicate motorunu seçer — `simhash`
 (varsayılan) veya `minhash` (`[ingestion-scale]` extra'sı şart;
 `--jaccard-threshold` cutoff'u kontrol eder, varsayılan 0.85).
-`--quality-filter` (Faz 12) heuristic kalite skorlamasını opt-in
-çalıştırır. Credential/secrets taraması **her zaman açık** — kapatma
-flag'i yoktur.
+`--quality-filter` (Faz 12, **Faz 15 Görev 5 ile v0.6.0'dan itibaren
+default-AÇIK**) heuristic kalite skorlamasını çalıştırır;
+`--no-quality-filter` v0.6.0 öncesi opt-in semantiklerini isteyen
+operatörler için opt-out sağlar. Credential/secrets taraması **her
+zaman açık** — kapatma flag'i yoktur.
 
 Eski `forgelm --data-audit PATH` flag'i deprecation alias olarak
 korunuyor; davranış aynı, sadece ek bir uyarı log'lanıyor.

docs/guides/data_audit.md

@@ -9,8 +9,10 @@ LSH-banded near-duplicate detection, streaming JSONL reading, PII
 severity tiers, atomic on-disk write, and a verbose-by-default
 truncation policy. **Phase 12 (consolidated into `v0.5.0`)** added an opt-in MinHash LSH
 dedup method (`--dedup-method minhash`), a code/credential leakage scan
-that runs always-on (`secrets_summary`), and an opt-in heuristic
-quality filter (`--quality-filter`).
+that runs always-on (`secrets_summary`), and a heuristic quality filter
+(`--quality-filter`) — initially opt-in, **flipped to default-ON in v0.6.0
+per Phase 15 Task 5**; operators wanting the pre-v0.6.0 opt-in semantics
+pass the new `--no-quality-filter` companion.
 
 The report feeds the EU AI Act Article 10 data governance artifact
 automatically when present in the trainer's `output_dir`.
@@ -36,7 +38,9 @@ forgelm audit data/ --near-dup-threshold 5
 pip install 'forgelm[ingestion-scale]'
 forgelm audit data/large_corpus.jsonl --dedup-method minhash --jaccard-threshold 0.85
 
-# Phase 12: opt-in heuristic quality filter (Gopher/C4 style)
+# Phase 12 + Phase 15: heuristic quality filter (Gopher/C4 style).
+# DEFAULT-ON from v0.6.0; explicit flag is redundant but harmless.
+# Pass --no-quality-filter to restore the pre-v0.6.0 opt-in semantics.
 forgelm audit data/ --quality-filter
 
 # Phase 17: split-level parallelism for multi-split corpora
@@ -478,7 +482,7 @@ forgelm audit PATH \
   [--near-dup-threshold N] \
   [--dedup-method {simhash,minhash}] \
   [--jaccard-threshold X] \
-  [--quality-filter] \
+  [--quality-filter | --no-quality-filter] \
   [--pii-ml] \
   [--pii-ml-language LANG] \
   [--croissant] \
@@ -496,8 +500,10 @@ Hamming-distance cutoff of 3 (≈95 % similarity); ignored when
 `--dedup-method=minhash`. `--dedup-method` (Phase 12) selects the
 near-duplicate engine — `simhash` (default) or `minhash` (needs
 `[ingestion-scale]` extra; `--jaccard-threshold` controls the cutoff,
-default 0.85). `--quality-filter` (Phase 12) opts into the heuristic
-quality scoring. The credential/secrets scan is **always on** — there
+default 0.85). `--quality-filter` (Phase 12, **default-ON from v0.6.0
+per Phase 15 Task 5**) runs the heuristic quality scoring;
+`--no-quality-filter` opts out for operators wanting the pre-v0.6.0
+opt-in semantics. The credential/secrets scan is **always on** — there
 is no flag to disable it.
 
 > **Note:** This matches the behavior summarised at the top of this guide: