From 3399c0e29c70f2e3eb839f41989b7eb979f5e102 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 14 May 2026 11:05:01 +0300 Subject: [PATCH 01/22] docs(roadmap): archive Phase 15 standalone doc + sync stale references MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 15 (Ingestion Pipeline Reliability) shipped under v0.6.0 on 2026-05-11; its detailed entry already lives at completed-phases.md line 1281+ with the Wave 1 + Wave 2 + 5 review-absorption-round summary. The standalone roadmap/phase-15-ingestion-reliability.md was a historical-reference duplicate — removed now that the "completed" surface is the single source of truth. Other drift fixed in the same sweep: - docs/roadmap.md: Phase 14 release target v0.6.0 → v0.7.0 in the file-tree block (the table at the top already reflected this). Mermaid graph: P15 now flows into P14 (shipping order), P14's release node renamed V23 → V27 (v0.7.0), P15 carries the "✅ Released" suffix; mermaid styles flip P15 to the released green palette. "Current state" paragraph: "19 phases under v0.5.5 / 3 phases planned" → "20 phases through v0.6.0 / 2 phases planned"; "Phase 14 and Phase 15 ship in parallel under v0.6.0" → "Phase 14 is targeted for v0.7.0". - docs/roadmap-tr.md: TR mirror of every roadmap.md edit above, plus the Phase 15 table row flipped from "Planlandı" to "✅ Tamam" with the link redirected at the completed-phases archive anchor. - docs/guides/ingestion.md + ingestion-tr.md: the Wave 3 multi- column extraction reference pointed at the now-removed standalone doc; redirected to the completed-phases archive anchor. - docs/roadmap/completed-phases.md: the Phase 15 "Related" block had a self-reference at the deleted standalone doc; replaced with the CHANGELOG section + PR #49 entry already adjacent. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/guides/ingestion-tr.md | 2 +- docs/guides/ingestion.md | 2 +- docs/roadmap-tr.md | 18 +-- docs/roadmap.md | 14 +- docs/roadmap/completed-phases.md | 3 +- .../roadmap/phase-15-ingestion-reliability.md | 135 ------------------ 6 files changed, 19 insertions(+), 155 deletions(-) delete mode 100644 docs/roadmap/phase-15-ingestion-reliability.md diff --git a/docs/guides/ingestion-tr.md b/docs/guides/ingestion-tr.md index 3183e5bd..6bef76f1 100644 --- a/docs/guides/ingestion-tr.md +++ b/docs/guides/ingestion-tr.md @@ -490,7 +490,7 @@ Wave 3 backlog'undadır. > (akademik posterler, geniş oluklu regülasyon yayınları) güvenilirdir > ama yayın-kalite iki-kolonlu makaleleri kaçırır. Histogram-tabanlı > bimodal-mode refactor Wave 3 takipi olarak izlenmektedir (bkz. -> [`docs/roadmap/phase-15-ingestion-reliability.md`](../roadmap/phase-15-ingestion-reliability.md) +> [Faz 15 arşivi](../roadmap/completed-phases.md#phase-15-ingestion-pipeline-reliability-v060) > Wave 3 — multi-column layout extraction). ### Markdown-bilen splitter — `--strategy markdown` (Faz 12) diff --git a/docs/guides/ingestion.md b/docs/guides/ingestion.md index f138da29..e3d469f5 100644 --- a/docs/guides/ingestion.md +++ b/docs/guides/ingestion.md @@ -481,7 +481,7 @@ is on the Wave 3 backlog. > wide-gutter regulatory publications) but misses publication-grade > two-column papers. A histogram-based bimodal-mode refactor is tracked > as a Wave 3 follow-up (see -> [`docs/roadmap/phase-15-ingestion-reliability.md`](../roadmap/phase-15-ingestion-reliability.md) +> [Phase 15 archive](../roadmap/completed-phases.md#phase-15-ingestion-pipeline-reliability-v060) > Wave 3 — multi-column layout extraction). ### Markdown-aware splitter — `--strategy markdown` (Phase 12) diff --git a/docs/roadmap-tr.md b/docs/roadmap-tr.md index d99bfb1e..0f0a2c2b 100644 --- a/docs/roadmap-tr.md +++ b/docs/roadmap-tr.md @@ -13,8 +13,8 @@ | ✅ Tamam | [Faz 12.6 — Kapanış Döngüsü (5 wave boyunca 37 içerik fazı + 1 sürüm etiketi = 38 kayıt)](roadmap/completed-phases.md) | Library API, GDPR purge + reverse-pii, ISO 27001 / SOC 2 alignment, doctor + cache subcommand'leri, compliance verification toolbelt, bilingual mirror sweep + 4 CI guard, supply-chain security, cross-OS release matrix — `v0.5.5` ile paketlendi (PyPI 2026-05-10) | | ✅ Tamam | Faz 22 — CLI sihirbazı tarayıcı yüzeyiyle eşdeğerlik | `forgelm --wizard` artık tarayıcı sihirbazıyla aynı 9-adımlı akışı çalıştırıyor (welcome → use-case → model → strategy → trainer → dataset → training-params → compliance → evaluation), `--wizard-start-from ` ile idempotent yeniden çalıştırma, şema-güdümlü varsayılanlar SOT, ayrı `EXIT_WIZARD_CANCELLED = 5` exit kodu, `$XDG_CACHE_HOME` altında durum kalıcılığı ve çıkışta validate — `v0.5.5` ile paketlendi (PyPI 2026-05-10) | | ✅ Tamam | Site dokümantasyon düzeltme taraması | `site/*.html` üzerindeki tüm görünür YAML / artefakt-yolu / CLI / şema iddiaları artık live `forgelm/` yüzeyine karşı doğrulanıyor. Hero YAML demo'su gerçek Pydantic alan adlarıyla yeniden yazıldı, compliance artefakt ağacı disk düzenine göre yeniden çizildi, hayalet YAML anahtarları + CLI flag'leri kaldırıldı, ifadeler live davranışa hizalandı. Altı dilde i18n (en / tr / de / fr / es / zh) tam paritede (her biri 731 anahtar) — `v0.5.5` ile paketlendi (PyPI 2026-05-10) | -| 📋 Planlandı | [Faz 14 — Çok Aşamalı Pipeline Zincirleri](roadmap/phase-14-pipeline-chains.md) | SFT → DPO → GRPO config zinciri, pipeline kaynak izleri → `v0.6.0` | -| 📋 Planlandı | [Faz 15 — Yutma Pipeline'ı Güvenilirliği](roadmap/phase-15-ingestion-reliability.md) | PDF / DOCX / EPUB / TXT / Markdown ve playground notebook UX hizalaması üzerinde `forgelm ingest`'in sessiz-başarısızlık boşluğunu kapat; yutma sırasında kalite sinyali yüzeye çıkar + audit'te `--quality-filter` varsayılan açık; çok satırlı PDF dedup'ı, Türkçe glyph normalizasyonu, DOCX header / footer çıkarımı, EPUB spine + skip, TXT / MD BOM + frontmatter; çift-golden regression fixture'ları → `v0.6.0` | +| 📋 Planlandı | [Faz 14 — Çok Aşamalı Pipeline Zincirleri](roadmap/phase-14-pipeline-chains.md) | SFT → DPO → GRPO config zinciri, pipeline kaynak izleri → `v0.7.0` (v0.6.0'dan yeniden planlandı; 2026-05-11 yutma pilotunun ardından Faz 15 önceliği aldı) | +| ✅ Tamam | [Faz 15 — Yutma Pipeline'ı Güvenilirliği](roadmap/completed-phases.md#phase-15-ingestion-pipeline-reliability-v060) | Wave 1 + Wave 2 + 5 review-absorption turu: window tabanlı çok satırlı PDF dedup'ı, Türkçe glyph normalizasyon profili (language-hint'e bağlı default), dil-farkında Unicode block sağlamlık kontrolü, ingest-time kalite ön-sinyali, default-on audit `--quality-filter`, DOCX explicit header/footer çıkarımı, EPUB spine + whole-token nav/cover skip, TXT UTF-8 BOM + MD YAML frontmatter strip, notebook playground hizalama, ek olarak Wave 2 `--strip-pattern` (ReDoS-korumalı), `--page-range`, front-matter heuristic, `--strip-urls`, multi-column uyarı — `v0.6.0` ile yayınlandı (PyPI 2026-05-11) | | 📋 Planlandı | [Faz 13 — Pro CLI ve Gözlemlenebilirlik Dashboard](roadmap/phase-13-pro-cli.md) | Lisans korumalı dashboard, HPO, zamanlanmış görevler, takım config store → `v0.6.0-pro` (adoption + v0.5.5'te yayınlanan ISO/SOC 2 baseline'a bağlı) | > **Durum lejantı:** ✅ Yayınlandı (PyPI) · 🟡 main'e indi, publish bekliyor · ⏳ Planlandı @@ -32,13 +32,13 @@ Başlangıçta dört ardışık PyPI tag'i (`v0.5.0` / `v0.5.1` / `v0.5.2` / `v0 **Daha öncesi:** `v0.5.0` — Doküman Yutma + Veri Curation Pipeline'ı (2026-04-30); `v0.4.5` — Quickstart Katmanı (2026-04-26); `v0.4.0` — Post-Training Tamamlama (2026-04-26). -**Güncel durum:** PyPI'da `v0.5.5` altında 19 faz (1, 2, 2.5, 3, 4, 5, 5.5, 6, 7, 8, 9, 10, 10.5, 11, 11.5, 12, 12.5, 12.6, 22) yayınlandı. Sürüm sonrası 3 faz (13, 14, 15) planlandı; Faz 14 ve Faz 15 `v0.6.0` içinde paralel olarak yayınlanır, Faz 13 adoption gate'leri karşılandığında ayrıca `v0.6.0-pro` olarak yayınlanır. +**Güncel durum:** PyPI'da `v0.6.0` üzerinden 20 faz (1, 2, 2.5, 3, 4, 5, 5.5, 6, 7, 8, 9, 10, 10.5, 11, 11.5, 12, 12.5, 12.6, 15, 22) yayınlandı. Sürüm sonrası 2 faz (13, 14) planlandı; Faz 14 `v0.7.0` için hedeflendi (v0.6.0'dan yeniden planlandı; 2026-05-11 yutma pilotunun ardından Faz 15 önceliği aldı), Faz 13 adoption gate'leri karşılandığında ayrıca `v0.6.0-pro` olarak yayınlanır. > **Faz 12.6 görev / alt-görev iki eksenli not:** Faz 12.6 kendi içinde 38 görevlik bir kapanış döngüsüdür (Görev 1-38) ve [`roadmap/completed-phases.md`](roadmap/completed-phases.md) dosyasında izlenir; her wave'in PR açıklaması o wave'in kapsadığı görev delta'sını taşır. ## Planlanan işlerin özeti -> **Not:** Oklar yayınlama sırasını gösterir, faz numaralarını değil (Faz 14 ve Faz 15 v0.6.0 içinde paralel olarak yayınlanır; Faz 13 Pro katmanında ayrıca daha sonra yayınlanır). +> **Not:** Oklar yayınlama sırasını gösterir, faz numaralarını değil (Faz 15 v0.6.0 ile yayınlandı; Faz 14 v0.7.0 için hedeflendi (Faz 15 önceliği aldıktan sonra); Faz 13 Pro katmanında ayrıca daha sonra yayınlanır). ```mermaid graph LR @@ -49,8 +49,8 @@ graph LR P115 --> P12[Faz 12
Veri Curation
Olgunlaşması] P12 --> P125[Faz 12.5
Veri Curation
Follow-up] P125 --> P126[Faz 12.6
Kapanış
Döngüsü] - P126 --> P14[Faz 14
Pipeline
Zincirleri] P126 --> P15[Faz 15
Yutma
Güvenilirliği] + P15 --> P14[Faz 14
Pipeline
Zincirleri] P14 --> P13[Faz 13
Pro CLI
+ Dashboard] P15 --> P13 @@ -61,8 +61,8 @@ graph LR P12 -.-> V2 P125 -.-> V2 P126 -.-> V25[v0.5.5] - P14 -.-> V23[v0.6.0] - P15 -.-> V23 + P15 -.-> V23[v0.6.0 ✅ Yayınlandı] + P14 -.-> V27[v0.7.0] P13 -.-> V3[v0.6.0-pro] style P10 fill:#003300,stroke:#00ff88 @@ -71,8 +71,8 @@ graph LR style P115 fill:#004400,stroke:#88ff88 style P12 fill:#004400,stroke:#88ff88 style P125 fill:#004400,stroke:#88ff88 + style P15 fill:#004400,stroke:#88ff88 style P14 fill:#002244,stroke:#00aaff - style P15 fill:#002244,stroke:#00aaff style P13 fill:#442200,stroke:#ffaa00 ``` @@ -92,7 +92,7 @@ docs/ └── roadmap/ ├── completed-phases.md # Faz 1-12.6 arşivi (detaylı, İngilizce) — Faz 10 / 10.5 / 11 / 11.5 / 12 / 12.5 / 12.6 inline gömüldü (sırasıyla v0.4.0 / v0.4.5 / v0.5.0 / v0.5.5) ├── phase-13-pro-cli.md # Planlandı — v0.6.0-pro (gated) - ├── phase-14-pipeline-chains.md # Planlandı — v0.6.0 (v0.5.5 kapanış döngüsünün takip release'i) + ├── phase-14-pipeline-chains.md # Planlandı — v0.7.0 (v0.6.0'dan yeniden planlandı; Faz 15 önceliği aldı) ├── releases.md # v0.3.0 → v0.6.0 sürüm notları └── risks-and-decisions.md # Risk matrisi, fırsatlar, rekabet analizi, karar günlüğü ``` diff --git a/docs/roadmap.md b/docs/roadmap.md index 65f2254c..3f605eac 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -34,13 +34,13 @@ Originally planned as four sequential PyPI tags (`v0.5.0` / `v0.5.1` / `v0.5.2` **Earlier:** `v0.5.0` — Document Ingestion + Data Curation Pipeline (2026-04-30); `v0.4.5` — Quickstart Layer (2026-04-26); `v0.4.0` — Post-Training Completion (2026-04-26). -**Current state:** 19 phases (1, 2, 2.5, 3, 4, 5, 5.5, 6, 7, 8, 9, 10, 10.5, 11, 11.5, 12, 12.5, 12.6, 22) shipped on PyPI under `v0.5.5`. 3 phases (13, 14, 15) planned post-release; Phases 14 and 15 ship in parallel under `v0.6.0`, Phase 13 ships separately as `v0.6.0-pro` once adoption gates are met. +**Current state:** 20 phases (1, 2, 2.5, 3, 4, 5, 5.5, 6, 7, 8, 9, 10, 10.5, 11, 11.5, 12, 12.5, 12.6, 15, 22) shipped on PyPI through `v0.6.0`. 2 phases (13, 14) planned post-release; Phase 14 is targeted for `v0.7.0` (re-scheduled from v0.6.0 after Phase 15 displaced it via the 2026-05-11 ingestion pilot), Phase 13 ships separately as `v0.6.0-pro` once adoption gates are met. > **Phase 12.6 task / sub-task dual-axis note:** Phase 12.6 is itself a 38-task closure cycle (Tasks 1-38) tracked at [`roadmap/completed-phases.md`](roadmap/completed-phases.md); per-wave PR descriptions carry the closure-task delta. ## Quick summary of what's planned -> **Note:** Arrows depict shipping order, not phase numbers (Phase 14 and Phase 15 both ship in v0.6.0 in parallel; Phase 13 ships later under the Pro tier). +> **Note:** Arrows depict shipping order, not phase numbers (Phase 15 shipped in v0.6.0; Phase 14 is targeted for v0.7.0 after Phase 15 displaced it; Phase 13 ships later under the Pro tier). ```mermaid graph LR @@ -51,8 +51,8 @@ graph LR P115 --> P12[Phase 12
Data Curation
Maturity] P12 --> P125[Phase 12.5
Data Curation
Follow-up] P125 --> P126[Phase 12.6
Closure
Cycle] - P126 --> P14[Phase 14
Pipeline
Chains] P126 --> P15[Phase 15
Ingestion
Reliability] + P15 --> P14[Phase 14
Pipeline
Chains] P14 --> P13[Phase 13
Pro CLI
+ Dashboard] P15 --> P13 @@ -63,8 +63,8 @@ graph LR P12 -.-> V2 P125 -.-> V2 P126 -.-> V25[v0.5.5] - P14 -.-> V23[v0.6.0] - P15 -.-> V23 + P15 -.-> V23[v0.6.0 ✅ Released] + P14 -.-> V27[v0.7.0] P13 -.-> V3[v0.6.0-pro] style P10 fill:#003300,stroke:#00ff88 @@ -73,8 +73,8 @@ graph LR style P115 fill:#004400,stroke:#88ff88 style P12 fill:#004400,stroke:#88ff88 style P125 fill:#004400,stroke:#88ff88 + style P15 fill:#004400,stroke:#88ff88 style P14 fill:#002244,stroke:#00aaff - style P15 fill:#002244,stroke:#00aaff style P13 fill:#442200,stroke:#ffaa00 ``` @@ -94,7 +94,7 @@ docs/ └── roadmap/ ├── completed-phases.md # Phase 1-12.6 archive (detailed) — Phase 10 / 10.5 / 11 / 11.5 / 12 / 12.5 / 12.6 absorbed inline (each shipped as v0.4.0 / v0.4.5 / v0.5.0 / v0.5.5) ├── phase-13-pro-cli.md # Planned — v0.6.0-pro (gated) - ├── phase-14-pipeline-chains.md # Planned — v0.6.0 (follow-up to the v0.5.5 closure cycle) + ├── phase-14-pipeline-chains.md # Planned — v0.7.0 (re-scheduled from v0.6.0; Phase 15 displaced it) ├── releases.md # v0.3.0 → v0.6.0 release notes └── risks-and-decisions.md # Risk matrix, opportunities, competitive positioning, decision log ``` diff --git a/docs/roadmap/completed-phases.md b/docs/roadmap/completed-phases.md index 34ad4b5c..4ea6faf5 100644 --- a/docs/roadmap/completed-phases.md +++ b/docs/roadmap/completed-phases.md @@ -1429,6 +1429,5 @@ notebook. ### Related -- [`docs/roadmap/phase-15-ingestion-reliability.md`](phase-15-ingestion-reliability.md) — the original phase doc with the 15 task list -- [`CHANGELOG.md`](../../CHANGELOG.md) `[0.6.0]` section +- [`CHANGELOG.md`](../../CHANGELOG.md) `[0.6.0]` section — release notes for the shipped tasks - PR #49 — five review-absorption rounds + tail-walk follow-up diff --git a/docs/roadmap/phase-15-ingestion-reliability.md b/docs/roadmap/phase-15-ingestion-reliability.md deleted file mode 100644 index 5a8d1162..00000000 --- a/docs/roadmap/phase-15-ingestion-reliability.md +++ /dev/null @@ -1,135 +0,0 @@ -# Phase 15: Ingestion Pipeline Reliability ✅ - -> **Status:** Released in `v0.6.0` on 2026-05-11. See -> [completed-phases.md](completed-phases.md#phase-15-ingestion-pipeline-reliability-v060) -> for the shipped scope + [releases.md](releases.md) for the release -> notes. This file is the original phase doc preserved for historical -> reference. - -> **Note:** This file details a single planned phase. See [../roadmap.md](../roadmap.md) for the cross-phase summary. - -**Goal:** Close the silent-failure gap in `forgelm ingest` exposed by an end-to-end pilot test against a real-world Turkish-language formal-publication PDF on 2026-05-11. The current pipeline produces output that *looks* successful (no errors, no warnings) but is polluted with multi-line running headers, font-fallback glyphs, bullet-character corruption, page-number leakage, and front-matter noise — degrading downstream SFT quality without any operator-facing signal. Phase 15 turns these silent failures into either auto-corrections or loud warnings, **across all five supported formats** (PDF, DOCX, EPUB, TXT, Markdown), and bakes the regression coverage into a fixture matrix. - -**Estimated Effort:** Medium (4-6 weeks, Wave 1 + Wave 2) -**Priority:** High — gates the credibility of ForgeLM's "config-driven, enterprise-grade" claim on the document ingestion surface. Without Phase 15, real-world Turkish / multi-script corpora produce silently poor data and customers blame the model instead of the pipeline. - -> **Context:** Surfaced by two end-to-end pilot runs on 2026-05-11 against a 193-page real-world Turkish-language formal-publication PDF (corpus-agnostic, institutional specifics scrubbed). The first run via the CLI (`--chunk-size 4000`) flagged 74 / 82 chunks below the `--quality-filter` threshold; the second run via `notebooks/ingestion_playground.ipynb` on Colab (`--chunk-size 1024`, both masks on) flagged 161 / 188 chunks. In both runs `forgelm ingest`'s mechanical-success summary reported no issues — multi-line running headers, font-fallback glyphs, bullet-character corruption, page-number leakage, recurring institutional-URL pollution, and front-matter chunks all sailed through silently. The same six failure modes generalise across the format × genre × language matrix the pipeline is supposed to handle; the pilots exposed them rather than introduced them. The cross-config replay also empirically verified the determinism guarantee (Turkish character count and U+085F bullet-glyph count were byte-identical across both passes), proving the failures are extraction-layer, not chunking-layer, and that fixes can be locked in with byte-comparison regression fixtures. Phase 15 is the planned remediation programme spanning **CLI surface, library surface, and the user-facing playground notebook** in one coherent v0.6.0 release. - -### Tasks - -#### Wave 1 — silent-failure remediation (v0.6.0, ~3 weeks, indicative per-week split below) - -**Week 1 — extraction-layer reliability:** Tasks 1 (multi-line dedup), 2 (script sanity), 3 (Turkish glyph normalisation). Three tightly-coupled changes inside [`forgelm/ingestion.py`](../../forgelm/ingestion.py); one dev-week is realistic if the engineer is already familiar with `_strip_repeating_page_lines` and the audit's §1.1 nuance. - -**Week 2 — operator signal + non-PDF format upgrades:** Tasks 4 (ingest-time quality pre-signal), 5 (audit `--quality-filter` default-on), 6 (DOCX explicit header / footer), 7 (EPUB spine + skip), 8 (TXT / MD BOM + frontmatter). Five smaller, independent changes that parallelise across two devs if needed. - -**Week 3 — surface integration + regression locking:** Task 9 (notebook UX alignment — gated on Tasks 4 + 5) and Task 10 (the 7-fixture regression suite, dual-golden chunk-size variants). Fixture authoring + golden JSONL generation is the heavier half. - -These splits are estimates, not commitments — task 1's multi-line dedup may surprise on degenerate corpora, and task 9's notebook update only matters once tasks 4+5 land, so Wave 1 can finish faster if extraction tasks complete clean. - -1. [x] **Multi-line PDF edge dedup — fix exit-condition trap, not "add an extra check".** - The current `_strip_repeating_page_lines` in [`forgelm/ingestion.py`](../../forgelm/ingestion.py) is already iterative — a `while True:` loop is supposed to peel one edge line at a time and re-check on each pass. The audit (§1.1) traced the observed failure to the loop's **exit condition**: pass 1 only inspects `lines[0]` and `lines[-1]`, so when the outermost line varies per page (a per-section title on top, a page number on the bottom) the loop strips zero lines on pass 1 and breaks *before* ever reaching the constant line that sits one row deeper. Future implementers reading just the docstring will assume iterative-peel is enough — the comment block above the loop must spell out *why iterative-peel alone fails on variable-outer-line corpora*. Concrete change: replace the pass-1-only `_repeating_edge_lines` with a top-N / bottom-N window check (N = 3 default, exposed as `_PDF_EDGE_WINDOW`) so a line recurring at **any** position within the top-3 / bottom-3 rows across ≥ 70 % of pages gets stripped in a single pass. Inline comment (1–2 sentences) explains the iteration-trap rationale so this regression cannot return silently. Run a second dedup pass after paragraph packing to catch headers that the chunker re-glued mid-block. Acceptance: on the synthetic Turkish-formal-publication regression fixture, the `""` pollution drops from ≥ 70 % of chunks to ≤ 5 %. - -2. [x] **Unicode / script sanity layer.** - New helper `_check_script_sanity()` runs after every per-file extraction. For each format-detected language (uses existing `langdetect` budget — already an `[ingestion]` extra dependency through `langdetect`), computes the ratio of code-points outside the language's expected Unicode blocks. Above a configurable threshold, logs a `WARNING` naming the file and a guess at the cause (font fallback, mojibake, mixed-script noise). Threshold and language hint exposed as `--script-sanity-threshold` and `--language-hint` CLI flags. **Default-threshold calibration is fixture-driven** — the audit measured ≈ 7 % corrupt-char ratio on the pilot's front matter and 0 % on clean body chunks, so the placeholder `0.5 %` value in this draft is *intentionally aggressive* and will be re-calibrated against the regression fixture corpus (Task 10) before merge. Final default is set by a precision / recall sweep on the fixture matrix (target: ≥ 95 % recall on font-corrupt fixtures, ≤ 5 % false-positive rate on clean mixed-script fixtures); likely range `1.0 – 2.0 %`. Acceptance: the front-matter chunks and the custom-glyph-bullet section of the synthetic Turkish-formal-publication fixture each emit one targeted warning *with the calibrated default*, and at least one legitimate mixed-script fixture (e.g. a Turkish corpus with embedded English code samples) emits zero warnings. - -3. [x] **Turkish pypdf glyph normalisation table.** - New module `forgelm/_pypdf_normalise.py` ships a static `dict` mapping known-pypdf-fallback glyphs to their correct Turkish characters (`ø → İ`, `Õ → ı`, `ú → ş`, `÷ → ğ`, `ö → Ğ` *(with trailing space)*, `࡟ → •` etc., derived from the audit's measured artefacts). Applied at chunk-write time by default, opt-out via `--no-normalise-unicode`. Module is **language-specific** but pluggable: a `--normalise-profile turkish|none` flag selects the table, leaving room for `arabic` / `cjk` profiles in later phases. Acceptance: (a) re-ingesting the synthetic Turkish-formal-publication fixture produces zero `ø Õ ú ÷ ࡟` artefacts in the JSONL; (b) `forgelm doctor` reports a `pypdf_normalise.turkish: pass` diagnostic row so an operator can verify the table loaded correctly without having to run a test ingest first. - -4. [x] **Ingest-time quality pre-signal.** - At the end of `ingest_path()`, run a lightweight 3-check sanity sweep (alpha-ratio per chunk, weird-char-ratio per chunk, repeated-line-ratio per chunk — *cheap* versions of the audit's full quality filter, not the expensive ones). If any chunk falls below the warn threshold, print a single end-of-summary line: - - ``` - [WARN] 74/82 chunks below ingestion quality threshold. Run - `forgelm audit --quality-filter` for detail. - ``` - - The threshold and disable-flag (`--no-quality-presignal`) live next to the existing `--output-format` flag. The summary line is the smallest possible nudge — full diagnostics stay in the audit step. - -5. [x] **Audit `--quality-filter` default-on.** - In [`forgelm/cli/_parser.py`](../../forgelm/cli/_parser.py) the audit sub-parser flips `--quality-filter` from opt-in to default-on; add `--no-quality-filter` for parity. Update [`docs/guides/data_audit.md`](../guides/data_audit.md) + TR mirror. Backward-compat: existing scripted invocations continue to work (no behaviour change when the flag is explicitly passed). - -6. [x] **DOCX explicit header / footer extraction.** - Extend `_extract_docx` in [`forgelm/ingestion.py`](../../forgelm/ingestion.py) to read `doc.sections[i].header` and `.footer` once per document, collect into a set, and subtract those exact lines from the body extraction. Word DOCX declares these explicitly — no statistical guess needed. Acceptance: a fixture DOCX with a 3-line header repeating across 10 pages produces JSONL with zero header lines. - -7. [x] **EPUB spine order + nav / cover / copyright skip.** - Modify `_extract_epub` in [`forgelm/ingestion.py`](../../forgelm/ingestion.py) to iterate `book.spine` (reading order) instead of `book.get_items()` (file order). Maintain a configurable skip-list of item names / `epub:type` values (`nav`, `cover`, `copyright`, `colophon`, `titlepage`, `frontmatter`) — defaulted to skip, opt-out via `--epub-no-skip-frontmatter`. Acceptance: a fixture EPUB with explicit cover / TOC / copyright pages emits JSONL containing only the body chapters in reading order. - -8. [x] **TXT / MD BOM strip + frontmatter detection.** - Extend `_extract_text` in [`forgelm/ingestion.py`](../../forgelm/ingestion.py) to strip the UTF-8 BOM (``) if present at position 0. For `.md` files only, detect and strip YAML frontmatter (`^---\n...\n---\n` at file start) by default; opt-out via `--keep-md-frontmatter`. CRLF → LF normalisation already happens on output via `newline="\n"`; verify no regression. Acceptance: a fixture TXT with a UTF-8 BOM and a fixture MD with YAML frontmatter produce clean chunks. - -9. [x] **Notebook UX alignment — surface the quality signal in the playground.** - `notebooks/ingestion_playground.ipynb` is the canonical "first 5 minutes with ForgeLM ingestion" surface and currently terminates in a misleading "looks clean" UX even when 161 / 188 chunks are quality-flagged (the 2026-05-11 second-pass evidence). After Task 5 ships `--quality-filter` as default-on, the notebook needs four small updates so the new signal reaches Colab users without the operator having to know to look for it: - - **Cell 5 config block** — expose `CHUNK_TOKENS = None` and `TOKENIZER = None` (token-aware mode) alongside the existing `CHUNK_SIZE` knob. Comment block explains: "set both to switch from char-based to token-based chunking; `TOKENIZER` is required when `CHUNK_TOKENS` is set." Addresses the audit §3.6 finding that token-aware mode forced operators to drop to a custom command. - - **Cell 8 markdown** — extend the "what audit gives you" list with the quality-filter checks (alpha-ratio, mean word length, end-of-line punctuation, repeated-line ratio, short-paragraph ratio). Honest framing: heuristic, conservative, surfaces extraction-layer noise. - - **Cell 9 invocation** — explicit `--quality-filter` flag for forward-compat with operators on older releases; once Task 5 default-on lands the flag is redundant but harmless. Keeps the notebook robust across the v0.5 → v0.6 transition. - - **Cell 10 pretty-print** — surface `quality_summary` alongside the existing PII / secrets / near-duplicate / language blocks. Show `samples_flagged / samples_evaluated`, `overall_quality_score`, and the `by_check` breakdown when any check fires. - Acceptance: re-running the playground notebook against the synthetic Turkish-formal-publication regression fixture surfaces a non-silent quality signal before the operator clicks "download ZIP", AND operators can flip the notebook into token-aware mode without leaving the markdown surface. - -10. [x] **Wave 1 regression fixture matrix.** - Commit the 7 fixtures listed in the audit's section 7 under `tests/fixtures/ingestion/`. **All fixtures are synthesised inside the repository under the project's permissive licence**; no external copyright is incurred. The `tr_textbook_4pages.pdf` fixture is a four-page Turkish-formal-publication PDF **constructed from scratch inside the repository** with a deterministic regeneration script alongside it. By deliberate construction it exhibits each of the six failure modes documented in audit §1: a multi-line running header (variable section title + constant publication-identifier line), Type-1-font Turkish glyph fallback on its first two pages, custom-bullet `U+085F` in one chapter, a recurring institutional-style URL pattern with varying numeric query strings, ToC-style underscore leaders on its third / fourth pages, and inline page numbers. Each fixture has a golden JSONL output alongside; new test module `tests/test_ingestion_reliability.py` runs `ingest_path()` against every fixture and byte-compares to the golden file. The `tr_textbook_4pages.pdf` fixture ships with **two** golden JSONLs — `--chunk-size 4000` and `--chunk-size 1024` — to lock in both the running-header-heavy and the front-matter-isolated failure modes per the audit's section 1.7 cross-config replay finding. The `mixed_directory/` fixture's acceptance is explicit: `forgelm ingest mixed_directory/ --recursive --output golden.jsonl` produces a JSONL whose chunks appear in **lexicographic file-name order** across all 5 formats (PDF, DOCX, EPUB, TXT, MD), with format-tagged provenance derivable from `notes_structured.format_counts`. Determinism guarantees from the audit's section 5 make byte-compare viable; if pypdf's per-page extraction shifts under a version bump, the test fails loudly and the fixture is regenerated explicitly via the committed regeneration script. - -#### Wave 2 — strong nice-to-have (v0.6.x, ~2 weeks) - -1. [x] **`--strip-pattern REGEX` flag — operator-controlled regex with ReDoS guard.** - Operator escape hatch for known-boilerplate that the dedup heuristic misses. Multiple patterns allowed (`--strip-pattern A --strip-pattern B`). Applied after extraction, before chunking. **Operator-supplied regexes must pass the regex-hardening guardrails defined in [`docs/standards/regex.md`](../standards/regex.md)** — the same 8-rule discipline (no unbounded greedy quantifiers in sequence, no nested groups with overlapping ranges, bounded repetition) that ForgeLM's own internal regexes obey. Implementation choice: validate the user-supplied pattern at CLI-parse time via the same SonarCloud python:S5852-style structural check ForgeLM applies internally, **and** wrap the match call in a per-pattern timeout (`re.match` with `signal.alarm` or `regex.match(... timeout=N)` if the `regex` library is added as a Wave 2 dependency — operator can opt out via `--strip-pattern-no-timeout` if they're certain). Documented in the operator guide with the canonical examples (running headers with variable content, journal DOI lines, watermarks). Acceptance: a pathological `(a+)+b`-class pattern aborts with a clear `EXIT_CONFIG_ERROR (1)` and a message naming the offending pattern, instead of hanging the ingestion process. - -2. [x] **`--page-range` PDF flag.** - Simple `--page-range 12-193` filtering for PDF files. Skips front-matter / back-matter manually when the heuristic in task 13 mis-fires. Validated as `min ≤ max`, `min ≥ 1`, `max ≤ pdf_page_count`. Validation failure **exits with `EXIT_CONFIG_ERROR (1)`** via the CLI dispatcher (matching the existing `_run_ingest_cmd` pattern in [`forgelm/cli/subcommands/_ingest.py`](../../forgelm/cli/subcommands/_ingest.py)) and prints a clear message naming the offending range and the actual page count. Library-level `ingest_path()` raises `ValueError` as before — the exit-code translation lives at the CLI boundary per the [error-handling standard](../standards/error-handling.md). - -3. [x] **Front-matter heuristic detection.** - Heuristic that looks at the first N pages of a PDF (N=12 default): if a page has alpha-ratio < 0.45 *and* underscore-ratio > 0.10 *and* contains ≥ 5 page-number-pattern matches, the page is tagged as "front-matter candidate" and dropped, with a `WARNING` listing the dropped page indices and the per-page metrics. Opt-out via `--keep-frontmatter`. Symmetric pass on the last N pages catches back-matter (bibliographies, indexes); same threshold. - -4. [x] **URL / QR pattern auto-strip option.** - `--strip-urls` flag: detects URL patterns (http(s)://, www.) per chunk and either replaces with `[URL]` or strips entirely (`--strip-urls=mask|strip|keep`, default `keep`). Useful for exercise-reference QR-noise and academic-paper DOI-noise. Existing PII / secrets mask pipeline integration: applies *after* secrets mask, *before* PII mask, preserving the mask-order discipline. **Interaction with `--all-mask`:** the `--all-mask` shorthand (Phase 12.5) is explicitly *not* extended to include `--strip-urls`. Rationale: PII / secrets are GDPR-grade compliance redactions whose accidental over-application is recoverable (the original text is still in the source PDF); URL stripping is a content-shape decision that operators may want to keep off while still demanding the privacy redactions. The two flag families stay independent and the operator opts into URL handling explicitly. Documented at the CLI help text for both `--all-mask` and `--strip-urls`. - -5. [x] **Multi-column PDF detection.** - Heuristic that estimates column count from pypdf's text-position metadata: per page, cluster x-positions of extracted spans; if two clusters dominate (separation > 30 % of page width), tag the document as 2-column. No fix attempt — just a `WARNING` at run start: `[WARN] Detected 2-column layout — reading order may be scrambled. Consider --strategy sliding with larger chunk-size, or pre-process with a layout-aware tool.` Better-than-nothing signal; cheaper than a layout-aware extractor. - -#### Wave 3 — deferred (post-Phase 15) - -1. [ ] OCR detection + automated `ocrmypdf` recommendation -2. [ ] Document profile / preset system (`--profile academic`, `--profile legal`, `--profile textbook`) -3. [ ] PDF table extractor (camelot-py / pdfplumber integration as optional `[ingestion-tables]` extra) -4. [ ] Footnote / endnote separation (DOCX + PDF) -5. [ ] Language-aware paragraph segmentation (CJK / Arabic / RTL scripts) -6. [ ] RTL script extraction-order normalisation - -These items go on the [risks-and-decisions](risks-and-decisions.md) backlog with the Phase 15 audit as their provenance reference, to be re-prioritised after Wave 1 lands and operator feedback arrives. - -### Requirements - -- **Backward compatibility:** Every Wave 1 / Wave 2 change has an opt-out flag. Existing pipelines that explicitly want the old behaviour (no normalisation, no front-matter skip, no quality pre-signal) get it with a single flag. No silent behaviour change for users with frozen scripts. -- **Determinism preserved — at two distinct levels:** - - **Level 1 (intra-chunk-size byte-identical):** for any fixed `--chunk-size`, `--strategy`, mask flags, and input bytes, the resulting JSONL is byte-identical across runs and across hosts (lex-sorted file walk, no clock dependency, deterministic mask order). This is the property Task 10's fixture-vs-golden byte-compare relies on; the regression suite breaks loudly if any fix introduces nondeterminism here. Documented in [`docs/reference/data_ingestion_architecture.md`](../reference/data_ingestion_architecture.md). - - **Level 2 (inter-chunk-size count-identical):** at *different* `--chunk-size` values the JSONL bytes naturally diverge (different chunk boundaries), but the **extraction-layer artefact counts** (per-language character counts, font-fallback glyph counts, U+085F bullet counts, redaction counts) remain identical because they are produced by the format extractor *before* chunking. The 2026-05-11 pilot runs empirically verified this: the same source PDF at `--chunk-size 4000` and `--chunk-size 1024` both produced 20,880 Turkish characters and 189 U+085F bullets, byte-for-byte. This invariant justifies the dual-golden fixture strategy in Task 10 — the same fixture can carry two byte-identical-per-config goldens *and* a single set of cross-config extraction-count assertions. - Glyph-normalisation tables are static dicts; quality pre-signal does not change JSONL content, only adds a stderr log line; spine-order EPUB extraction is deterministic. None of these fixes weaken either determinism level. -- **Format coverage symmetry:** Wave 1 must touch all five formats (PDF tasks 1-4, DOCX task 6, EPUB task 7, TXT / MD task 8) **plus the user-facing notebook surface** (task 9). Phase 15 does not ship a "PDF reliability release" that ignores the other four formats, and does not ship a "CLI-only release" that leaves the playground notebook regressed. -- **Notebook surface in-scope:** [`notebooks/ingestion_playground.ipynb`](../../notebooks/ingestion_playground.ipynb) is the canonical "first 5 minutes with ForgeLM ingestion" surface and currently propagates the silent-failure UX even after Tasks 4-5 land at the CLI layer. Task 9 closes that gap. The notebook update PR is gated on **Tasks 4 + 5** landing first (Task 4 produces the ingest-time pre-signal the notebook will surface; Task 5 makes `--quality-filter` default-on so the audit JSON the notebook reads carries the quality data). Task 8 (TXT/MD BOM) is unrelated to the notebook UX and not part of the gating chain. Notebook update ships in the same v0.6.0 release. -- **No new heavy dependencies in Wave 1:** Tasks 1-10 use only `pypdf`, `python-docx`, `ebooklib`, `bs4` (already shipped under `[ingestion]`) and `langdetect` (already shipped). Camelot / pdfplumber / Tesseract are Wave 3 only and gated behind new optional extras. -- **Bilingual docs sweep:** Operator-facing guides updated in EN + TR ([`docs/guides/ingestion.md`](../guides/ingestion.md), [`docs/guides/data_audit.md`](../guides/data_audit.md), [`docs/reference/data_ingestion_architecture.md`](../reference/data_ingestion_architecture.md), and their `-tr.md` mirrors). The bilingual parity guard (`tools/check_bilingual_parity.py --strict`) must pass at PR open. Notebooks are EN-only by project convention and exempt from the bilingual mirror requirement. -- **Honesty in the "Limitations" surfaces:** The architecture reference's existing "Limitations" sections gain a sub-section per still-deferred Wave 3 item (multi-column, OCR, RTL). Users see what does *not* work before they hit it. -- **Test budget:** Wave 1 ships with the 7-fixture regression suite (Task 10) including the dual-golden chunk-size variant on `tr_textbook_4pages.pdf`. Coverage delta target: `forgelm/ingestion.py` line coverage ≥ 90 % after Wave 1 lands. -- **No regression on the v0.5.0 contracts:** existing exit codes (0/1/2/3/4/5), the `[REDACTED]` / `[REDACTED-SECRET]` mask vocabulary, the `IngestionResult` schema, and the `--output-format json` envelope are stable across Phase 15. New fields may be added; nothing existing renames or disappears. - -### Delivery - -- **Target release:** Wave 1 ships as `v0.6.0` alongside [Phase 14 (Multi-Stage Pipeline Chains)](phase-14-pipeline-chains.md). The two phases are independent and parallelisable. -- **Wave 2 ships as `v0.6.x` follow-up** (any of `v0.6.1`, `v0.6.2`) at maintainer discretion. -- **Wave 3 deferred to Phase 16+** — separately re-prioritised after Wave 1 ships and operator feedback arrives. -- **Gate to start Wave 1 (entry conditions, all must be true before the first Wave 1 PR opens):** - - The 7 regression fixtures (proposed in Task 10) are committed as zero-byte placeholders with golden-JSONL stubs to lock the file paths. - - The Phase 15 issue (`#TBD`) is opened on GitHub referencing this file. - - The 2026-05-11 pilot run is recorded (institutional specifics scrubbed) so the v0.6.0 acceptance gate can re-run against the synthetic regression fixture. - - **Fixture authorship verification.** `tr_textbook_4pages.pdf` and every other fixture under `tests/fixtures/ingestion/` is **synthesised inside this repository under the project's permissive licence**, with a deterministic regeneration script alongside each binary asset. No third-party copyright applies. CI's existing licence-header guard runs over the fixture sources. - - **Coverage baseline measurement.** Run `pytest --cov=forgelm/ingestion --cov-report=term-missing` against `main` (v0.5.7) and record the current line-coverage percentage in the Phase 15 GitHub issue. The "≥ 90 %" target in Test budget below is a delta target; reaching 90 % from a 50 % baseline is materially different work from reaching it from an 80 % baseline, and the entry-gate measurement locks in honest expectations. -- **CHANGELOG plan:** under the `[Unreleased]` section of [`CHANGELOG.md`](../../CHANGELOG.md), every Wave 1 task lands a one-line bullet at PR-merge time under the appropriate `Added` / `Changed` / `Fixed` category per the Keep-a-Changelog convention. At v0.6.0 tag time the `[Unreleased]` block is renamed to `[0.6.0] — YYYY-MM-DD` per the existing release ritual; no separate "release notes" file is needed. Wave 2 tasks land their CHANGELOG bullets under a new `[Unreleased]` block opened post-v0.6.0. -- **Validation gate to ship Wave 1:** re-run `forgelm ingest` against the synthetic Turkish-formal-publication regression fixture at **both** `--chunk-size 4000` and `--chunk-size 1024`. Acceptance: `overall_quality_score ≥ 0.70` from `forgelm audit` on each resulting JSONL, *without* any post-processing or `--strip-pattern` flag. The dual-chunk-size acceptance is what catches the front-matter regression class the 2nd-pass Colab evidence exposed (where smaller chunks isolate chapter-opening orientation pages and the 21-character single-caption chunk). In addition, the playground notebook end-to-end run against the same fixture must surface the quality signal in its Cell 10 output (Task 9 acceptance). The audit's section 6 (Wave 1 list) covers the same scope at a higher granularity but with different bucketing — Phase 15 doc Tasks 2 + 3 jointly cover the audit's bullet 2 (script sanity + Turkish normalisation), and Tasks 6 + 7 + 8 split the audit's bullet 5 (format-specific extractor upgrades) per format. The 1-to-1 mapping is: Phase 15 Task 1 ↔ audit bullet 1; Tasks 2 + 3 ↔ audit bullet 2; Task 4 ↔ audit bullet 3; Task 5 ↔ audit bullet 4; Tasks 6 + 7 + 8 ↔ audit bullet 5; Task 9 ↔ audit bullet 6; Task 10 (regression fixtures) is sourced from audit section 7, not section 6. Failure on any acceptance criterion of tasks 1-10 blocks the v0.6.0 tag. - ---- - -### Cross-references - -- **Code surface affected:** [`forgelm/ingestion.py`](../../forgelm/ingestion.py), [`forgelm/cli/_parser.py`](../../forgelm/cli/_parser.py), [`forgelm/cli/subcommands/_ingest.py`](../../forgelm/cli/subcommands/_ingest.py), [`forgelm/data_audit/_quality.py`](../../forgelm/data_audit/_quality.py) -- **User-facing surfaces to update:** [`docs/guides/ingestion.md`](../guides/ingestion.md) + `-tr.md`, [`docs/guides/data_audit.md`](../guides/data_audit.md) + `-tr.md`, [`docs/reference/data_ingestion_architecture.md`](../reference/data_ingestion_architecture.md) + `-tr.md`, [`notebooks/ingestion_playground.ipynb`](../../notebooks/ingestion_playground.ipynb) -- **Standards consulted:** [`docs/standards/error-handling.md`](../standards/error-handling.md), [`docs/standards/logging-observability.md`](../standards/logging-observability.md), [`docs/standards/testing.md`](../standards/testing.md) From c8e944656b25a827215a4d60cd7bdb69181169f9 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 14 May 2026 11:22:39 +0300 Subject: [PATCH 02/22] =?UTF-8?q?docs(roadmap):=20refresh=20Phase=2014=20d?= =?UTF-8?q?oc=20=E2=80=94=20v0.7.0=20target=20+=20acceptance=20criteria=20?= =?UTF-8?q?+=20schemas?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Phase 14 (Multi-Stage Pipeline Chains) doc had drifted from reality after v0.6.0 shipped — Delivery still pointed at v0.6.0 and the v0.5.5 closure cycle, Estimated Effort had an over-optimistic 3-week minimum, and every task carried high-level scope without acceptance criteria or edge-case spec. Phase 15's discipline (acceptance criteria per task, fixture matrix, entry gate, deferred- item ledger) is now mirrored here so implementation reviewers do not have to derive the unstated edge cases mid-PR. Sections refreshed (no behavioural code change — doc-only): - **Status header.** Added: planning, targeted for v0.7.0, no hard blockers; references the Phase 15 displacement event. - **Estimated Effort + Priority.** Retired the 3-week optimistic floor; called out that the risk surface is in orchestration semantics (inheritance merge, partial-run resume, gate composition), not trainer mechanics. - **Task 1.** Pipeline schema spec + stage-name regex + uniqueness validator + empty-list rejection + backward-compat assertion. Five acceptance bullets. - **Task 2.** Inheritance matrix as a table covering every top- level config section (11 rows) with explicit auto-chain / inherited / overridden / rejected-per-stage semantics and the "section-wholesale, not deep-merge" merge-depth rule called out with rationale. Five acceptance bullets including a 44-test-case fixture matrix. - **Task 3.** Pipeline manifest JSON schema (illustrative, complete) + status enum + Annex IV mapping + chain integrity validator spec. Per-stage training_manifest.json files retained alongside; pipeline manifest is an index, not a replacement. - **Task 4.** --dry-run failure-collection mode + --stage partial- run resolution table (5 scenarios) + --resume-from state machine + pipeline_state.json schema + stale-state guard with --force-resume override + --output-format json envelope spec. - **Task 5.** Per-stage gates composition + human-approval gate flow within a pipeline + 5 new audit-log events (pipeline.started / .stage_started / .stage_completed / .stage_reverted / .completed) + webhook notifier extension. - **Requirements.** Expanded from 4 bullets to 9 covering backward-compat byte-identical, dependency neutrality, determinism, bilingual docs sweep, wizard scope decision, test budget, honest limitations framing. - **Test fixture sketch.** 6 fixtures under tests/fixtures/pipeline/ modelled on Phase 15's matrix discipline. - **Delivery.** Rewritten: v0.7.0 target, entry gate (regression fixtures + GitHub issue + label + coverage baseline), CHANGELOG plan, validation gate, and a deferred-items ledger (intra-stage resume, DAG semantics, parallel execution, wizard integration). - **Cross-references.** Net new section: code surface + user-facing docs + standards consulted + pattern reference pointing at Phase 15's completed-phases.md anchor. All 11 self-review gauntlet guards pass after the rewrite (anchor resolution 270 md, bilingual parity 46 pairs, CLI help consistency 447 invocations, wizard defaults, no-analysis-refs, no-unguarded-sys.modules-pop, site version). Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/roadmap/phase-14-pipeline-chains.md | 225 ++++++++++++++++++++--- 1 file changed, 199 insertions(+), 26 deletions(-) diff --git a/docs/roadmap/phase-14-pipeline-chains.md b/docs/roadmap/phase-14-pipeline-chains.md index b37bb763..014e1e84 100644 --- a/docs/roadmap/phase-14-pipeline-chains.md +++ b/docs/roadmap/phase-14-pipeline-chains.md @@ -1,17 +1,22 @@ # Phase 14: Multi-Stage Training Pipeline Chains -> **Not:** Bu dosya tek bir planlanan fazı detaylandırır. Tüm fazların özeti için [../roadmap.md](../roadmap.md). +> **Status:** Planning. Targeted for `v0.7.0` (originally `v0.6.0`; Phase 15 displaced it after the 2026-05-11 ingestion pilot — see [completed-phases.md#phase-15-ingestion-pipeline-reliability-v060](completed-phases.md#phase-15-ingestion-pipeline-reliability-v060)). No hard blockers; `v0.6.0` has shipped, Phase 14 can start. -**Goal:** Enable a single config file to define a sequential SFT → DPO → GRPO training pipeline, with each stage using the previous stage's output as its base model. Eliminate the current requirement for manual config management across stages. -**Estimated Effort:** Low-Medium (3-6 weeks) -**Priority:** Medium-High — frequently requested by enterprise users; low implementation risk (uses existing trainers) +> **Note:** This file details a single planned phase. See [../roadmap.md](../roadmap.md) for the cross-phase summary. -> **Context:** Enterprise ML teams running production post-training pipelines currently need to write 3+ separate config files, manually set `model.name_or_path` to each stage's output, and orchestrate execution themselves. A `pipeline:` config key that chains stages solves this while remaining fully config-driven and testable. +**Goal:** Enable a single config file to define a sequential SFT → DPO → GRPO training pipeline, with each stage using the previous stage's output as its base model. Eliminate the current requirement for manual config management across stages while preserving the existing single-stage workflow byte-identically. -### Tasks: +**Estimated Effort:** Medium (4-6 weeks for Wave 1, including ~1 review-absorption round). The "3-week minimum" optimistic floor in earlier drafts was retired because the per-stage state machine (Task 5) and the inheritance-merge semantics (Task 2) both have non-obvious edge cases that surface during reviewer absorption, mirroring the Phase 15 cycle (5 absorption rounds for a comparable surface area). + +**Priority:** Medium-High — frequently requested by enterprise users since the v0.5.0 launch (preference / RL alignment workflows are the post-SFT default for regulated customers); low *core* implementation risk because every individual trainer already works. The risk surface is in **orchestration semantics**, not trainer mechanics: inheritance merge, partial-run resume, and per-stage gate composition. + +> **Context:** Enterprise ML teams running production post-training pipelines currently write 3+ separate config files, manually set `model.name_or_path` to each stage's output, copy LoRA / safety / compliance config blocks between them, and orchestrate execution themselves with shell scripts. Every manual copy is an opportunity to drift — a LoRA `r` value that matches across stages by accident becomes one that mismatches after the next config edit. A `pipeline:` config key that chains stages solves the orchestration *and* the drift problem in one move, while remaining fully config-driven, dry-run-validatable, and Annex-IV-traceable. + +### Tasks + +1. [ ] **`pipeline:` config section in `ForgeConfig`** + New optional `pipeline.stages: List[PipelineStage]` section. Each stage is a partial training config override layered onto the root config (see Task 2 for the merge semantics). Stages execute sequentially; each stage's `model.name_or_path` is automatically set to the previous stage's `training.output_dir/final_model` (or, if the previous stage gated on human approval, to the staging path after approval). Stage names must be unique within a pipeline and match `^[a-z0-9_]{1,32}$` so they can serve as identifiers in CLI flags (`--stage `, `--resume-from `) and audit-log fields without escaping. -1. [ ] **`pipeline:` config section in ForgeConfig** - New optional `pipeline: stages: [...]` section. Each stage is a full training config override. Stages execute sequentially; each stage's `model.name_or_path` is automatically set to the previous stage's `training.output_dir/final_model`. ```yaml pipeline: stages: @@ -39,38 +44,206 @@ dataset_name_or_path: "./data/math_prompts.jsonl" ``` + **Acceptance:** + - `ForgeConfig(**yaml_load(pipeline.yaml)).pipeline` is a typed `PipelineConfig` with a non-empty `stages` list and a unique-name validator (`pydantic.field_validator` raising `ValueError` on duplicate names). + - Stage name pattern enforced at config load (`^[a-z0-9_]{1,32}$`); invalid name → `EXIT_CONFIG_ERROR (1)` with an actionable message naming the offending stage. + - Empty `pipeline.stages: []` → rejected at load (a pipeline must have ≥ 1 stage; the operator who meant single-stage should not use the `pipeline:` key at all). + - Backward compatibility: a config file without a `pipeline:` key produces `config.pipeline is None`; all existing CLI paths see no behavioural change. + 2. [ ] **Stage inheritance and override semantics** - Each stage inherits the top-level config as defaults. Only explicitly set fields are overridden per stage. LoRA config, model config (except `name_or_path`), distributed config, and safety config inherit from the root unless overridden per stage. + Each stage inherits the top-level config as defaults; only fields the stage *explicitly sets* are overridden. Detection of "explicitly set" uses Pydantic's `model_fields_set` (not "field happens to equal default") so a stage that writes `learning_rate: 2e-5` — the same as the root default — still flags as overridden and the trainer sees the stage's value. This matters when reviewers expect "the YAML says it, so the YAML's value wins" symmetry. + + **Merge depth:** the merge is **section-wholesale**, *not* recursive-deep-merge. If a stage declares `lora:`, the entire `lora` block replaces the root `lora` block — partial-override on a nested key is rejected at config load. This is a deliberate trade-off: deep-merge nuances (`target_modules: [...]` append vs. replace?) compound across the 19 ForgeConfig submodels and turn config validation into an undebuggable maze. Wholesale replacement gives one clear rule: "if you want to inherit, omit the block; if you want to override, supply the *full* block." + + **Inheritance matrix** (per top-level config section): + + | Section | Inheritance | Notes | + |---|---|---| + | `model.name_or_path` | **Auto-chained** (overrides root after stage 0) | Set to previous stage's `training.output_dir/final_model`. Stage 0 still reads root `model.name_or_path`. An explicit per-stage `model.name_or_path` is allowed and disables auto-chaining for that stage (operator escape hatch). | + | `model.*` (other fields) | Inherited unless `model:` block overridden | `backend`, `load_in_4bit`, `trust_remote_code`, `max_length`, `chat_template` follow the root unless the stage supplies a full `model:` block. | + | `lora` | Inherited unless `lora:` block overridden | Critical edge case: stage 2 DPO with a *different* `lora.r` than stage 1 SFT means stage 2 is **a fresh LoRA over the merged SFT model**, not a continuation of SFT's LoRA. Documented in the operator guide. | + | `data` | **Must be overridden per stage** | Pipelines that don't change the dataset between stages are vanishingly rare; explicit per-stage data is a forcing function against accidental dataset reuse. | + | `training` | Inherited unless `training:` block overridden | `trainer_type` is required per stage; the validator rejects a stage with no `training.trainer_type` even if the root supplies one (each stage explicitly states its alignment paradigm for audit clarity). | + | `evaluation` | Inherited unless `evaluation:` block overridden | Per-stage gates (Task 5) live here; a stage that wants different `auto_revert` / `safety` config supplies a full `evaluation:` block. | + | `distributed` | Inherited (always); no per-stage override | Distributed strategy must be consistent across the pipeline run. Per-stage `distributed:` blocks rejected at load with a clear error. | + | `webhook` | Inherited (always); no per-stage override | One webhook configuration covers the pipeline; per-stage events carry the stage name in the payload. | + | `compliance` | Inherited (always); no per-stage override | Provider / system / risk metadata is pipeline-level, not stage-level. | + | `synthetic`, `merge` | Stage-local only; rejected at the root level when `pipeline:` is set | These flags describe non-training modes — using them inside a pipeline stage is the wrong layer of abstraction (synthesise-then-train should be two separate operator invocations). | + + **Acceptance:** + - `forgelm/config.py` exports `PipelineStage(BaseModel)` and `PipelineConfig(BaseModel)` Pydantic models with field-level validators implementing the matrix above. + - The merge happens in a single helper `_merge_stage_into_root(root_cfg, stage) -> ForgeConfig`; the rest of the codebase consumes a flat `ForgeConfig` per stage with no awareness of the pipeline layer. This keeps `forgelm/trainer.py` unmodified. + - Per-stage `distributed:` block → `EXIT_CONFIG_ERROR (1)` with message `"distributed:" cannot appear in pipeline stage ; declare it at the root.` + - Per-stage `model.name_or_path` is honoured and disables auto-chaining for that stage (logged at INFO). + - Test fixture matrix covers: `inherited`, `overridden`, `auto-chained`, `rejected-per-stage` for each of the 11 rows above (44 test cases total). + +3. [ ] **Pipeline compliance manifest** + New artefact `compliance/pipeline_manifest.json` (JSON, not YAML — matches the existing `training_manifest.json` format the auditors are already trained on) captures the full multi-stage provenance. **Per-stage `training_manifest.json` files are still emitted** inside each stage's `output_dir/compliance/` directory; the pipeline manifest is an *index* that ties them together, not a replacement. Satisfies EU AI Act Annex IV's "complete training lineage" requirement at the chain level. + + **Schema (illustrative):** + + ```json + { + "forgelm_version": "0.7.0", + "generated_at": "2026-06-15T12:34:56+00:00", + "pipeline_run_id": "pl_2026-06-15_a1b2c3", + "pipeline_config_hash": "sha256:", + "stages": [ + { + "name": "sft_stage", + "index": 0, + "trainer_type": "sft", + "status": "completed", + "input_model": "meta-llama/Llama-3-8B", + "output_model": "./checkpoints/stage1_sft/final_model", + "started_at": "2026-06-15T12:34:56+00:00", + "finished_at": "2026-06-15T14:21:08+00:00", + "duration_seconds": 6372, + "training_manifest": "./checkpoints/stage1_sft/compliance/training_manifest.json", + "metrics": { "eval_loss": 0.42 }, + "gate_decision": "passed", + "auto_revert_triggered": false + }, + { + "name": "dpo_stage", + "index": 1, + "trainer_type": "dpo", + "status": "skipped_due_to_prior_revert", + "input_model": "./checkpoints/stage1_sft/final_model", + "output_model": null, + "training_manifest": null, + "skipped_reason": "Stage sft_stage triggered auto_revert; subsequent stages did not run." + } + ], + "final_status": "stopped_at_stage", + "stopped_at": "sft_stage", + "annex_iv": { "": "..." } + } + ``` + + Stage `status` enum: `pending` | `running` | `completed` | `failed` | `gated_pending_approval` | `skipped_due_to_prior_revert` | `skipped_by_filter` (when `--stage X` runs only one stage). + + **Annex IV mapping:** the pipeline manifest discharges the same Article 11 / Annex IV obligations as the single-stage manifest, with two additions: (a) the **training lineage** is a chain rather than a single record, (b) the `pipeline_config_hash` proves the chain's *intent* is reproducible (the same config bytes always describe the same chain). Existing `forgelm verify-annex-iv` validator gains a `--pipeline` mode that follows the index, validates each per-stage manifest, and asserts chain integrity (stage N's `input_model` must equal stage N-1's `output_model`). -3. [ ] **Pipeline compliance artifacts** - `compliance/pipeline_manifest.yaml` captures the full multi-stage provenance: stage names, input/output model paths, per-stage metrics and artifacts. Satisfies EU AI Act Annex IV requirement for complete training lineage documentation. + **Acceptance:** + - `forgelm/compliance.py` exports `generate_pipeline_manifest(pipeline_state) -> Dict[str, Any]` and `export_pipeline_manifest(manifest, output_dir) -> str`. The orchestrator writes the manifest after every stage state transition (atomic-rename pattern, matching the existing `_atomic_write_json` helper). + - The manifest is **append-friendly under partial runs**: a `--stage sft_stage` invocation produces a 1-stage manifest with the other two stages as `status: skipped_by_filter`; a subsequent full run replaces the manifest atomically with a complete 3-stage version. + - `forgelm verify-annex-iv --pipeline ` returns exit 0 on a complete passing run, exits non-zero with a `chain_integrity_violation` reason when stage N's input doesn't match stage N-1's output. + - The per-stage `training_manifest.json` files remain individually valid against the existing single-stage Annex IV schema — no field is renamed or removed at the stage level. -4. [ ] **CLI support** +4. [ ] **CLI surface** ```bash - # Single pipeline run + # End-to-end pipeline run forgelm --config pipeline.yaml - # Dry-run validates all stages + # Dry-run validates every stage (Pydantic + cross-stage chain integrity) before any GPU is allocated forgelm --config pipeline.yaml --dry-run - # Run specific stage only + # Run a single named stage in isolation (audit / re-run scenarios) forgelm --config pipeline.yaml --stage dpo_stage - # Resume from failed stage + # Resume after a failed / interrupted run from a named stage onward forgelm --config pipeline.yaml --resume-from dpo_stage ``` -5. [ ] **Per-stage auto-revert and gates** - Each stage can have independent evaluation gates. If a stage fails auto-revert, the pipeline stops and reports which stage failed. Subsequent stages are not run. + **`--dry-run` failure-collection mode.** Each stage is validated independently; the dispatcher **collects all stage-level errors before exiting** rather than stopping at the first failure. Mirrors `pytest --collectonly` and avoids the "fix one config bug, re-run, fix the next" loop that drives operators away. Exit `EXIT_CONFIG_ERROR (1)` if any stage fails validation; the error report names every offending stage + field. Chain-integrity validation (each stage's auto-chained `input_model` must point at a path under the previous stage's `output_dir`) runs *after* per-stage Pydantic validation passes. + + **`--stage ` semantics (partial-run resolution rules):** + + | Scenario | Behaviour | + |---|---| + | `--stage ` and `` is the first stage | Reads root `model.name_or_path`; runs normally. | + | `--stage ` and the previous stage's `output_dir/final_model` exists on disk | Auto-chains as if previous stage had just finished; INFO-logs the disk path being reused. | + | `--stage ` and the previous stage's output is missing | Hard-fails with `EXIT_CONFIG_ERROR (1)`: `Stage requires output at ; pass --input-model to override or run the full pipeline first.` No silent fallback to root `model.name_or_path`. | + | `--stage --input-model ` | Operator escape hatch: skips the auto-chain, uses `` as the stage's base model. The audit-log event records `input_source: "cli_override"` so reviewers can trace why the chain "broke" intentionally. | + | `--stage ` where `` is not in the config | Hard-fails at parse time with the list of valid stage names. | + + **`--resume-from ` semantics (state machine + persistence):** + - State file: `/pipeline_state.json` (atomic-write, JSON, schema mirrors the pipeline manifest's `stages[].status` enum). Updated after every stage transition. + - Resume rule: stages with `status: completed` and a `output_model` path that still exists on disk are **skipped** (logged at INFO) and their manifests are preserved. The named stage and every stage after it are re-run. + - Stale-state guard: if the on-disk `pipeline_state.json` was produced by a different `pipeline_config_hash` than the one being resumed against, the resume fails with `EXIT_CONFIG_ERROR (1)` — preventing "I resumed against a config that was edited mid-flight" silent divergence. Override via `--force-resume` (logged at WARNING, recorded in the audit event). + - Resume does **not** support intra-stage checkpoint resume in Phase 14. HF `Trainer.train(resume_from_checkpoint=...)` integration is deferred to a Phase 14.x follow-up; Phase 14 only resumes at stage boundaries. + + **`--output-format json` envelope:** pipeline runs emit a top-level JSON object with `pipeline_run_id`, `stages[]` (each with `status`, `started_at`, `finished_at`, per-stage exit code), and `final_status`. Single-stage existing envelope unchanged. + + **Acceptance:** + - `forgelm --config pipeline.yaml --dry-run` validates every stage and prints a per-stage diagnostic block; exits 0 on full success or 1 with the collected error report on any failure. + - `forgelm --config pipeline.yaml --stage ` with a missing previous-stage output exits 1 with the message shape above (regression test asserts the exact message format so operators can grep for it in CI logs). + - `forgelm --config pipeline.yaml --resume-from ` against a stale state file exits 1 with `pipeline_config_hash mismatch`; `--force-resume` allows the run with a WARNING. + - `--stage ` for an unknown name lists the valid stage names in the error message (no opaque "stage not found"). + - `--output-format json` envelope passes `jq -e '.pipeline_run_id'` on every multi-stage invocation. + +5. [ ] **Per-stage gates, auto-revert, and stage-boundary resume** + Each stage carries its own `evaluation:` block (Task 2 inheritance matrix); the gate logic that already runs in `ForgeTrainer._post_training_gate` (loss thresholds, benchmark min-score, safety regression, judge min-score, human-approval) executes **per stage**. If any stage fails an auto-revert, the pipeline stops at that stage; subsequent stages enter `status: skipped_due_to_prior_revert` in both the pipeline manifest and the state file. The reverted model still exists on disk per the existing single-stage auto-revert contract — Phase 14 does not change auto-revert semantics, it just composes them. + + **Human approval gate within a pipeline.** If a stage has `evaluation.require_human_approval: true`, the trainer follows the existing Phase 9 flow: model lands in `final_model.staging./`, exits with `EXIT_AWAITING_APPROVAL (4)`. Pipeline-specific behaviour: the orchestrator captures the staging path in the pipeline state (`status: gated_pending_approval`), prints the resume command the operator needs (`forgelm --config pipeline.yaml --resume-from ` after `forgelm approve `), and exits with the same `EXIT_AWAITING_APPROVAL (4)` code so CI/CD wrappers see one consistent gate signal. On resume, the auto-chained `input_model` for the next stage points at the *promoted* `final_model/` path (post-approval), not the `final_model.staging/` path. + + **Audit-log events (extends the existing 5-event vocabulary):** + - `pipeline.started` — run id, config hash, stage count, stage names + - `pipeline.stage_started` — stage name, index, input model, input source (chain/cli_override/root) + - `pipeline.stage_completed` — stage name, metrics summary, gate decision + - `pipeline.stage_reverted` — stage name, auto-revert reason, halt-pipeline=true + - `pipeline.completed` — final status, total duration, stage count + + These events live alongside (not replacing) the existing per-stage `training.*` events emitted by `ForgeTrainer`. Webhook notifier (`forgelm/webhook.py`) gains `notify_pipeline_started` / `notify_pipeline_completed` / `notify_pipeline_reverted` methods that wrap the existing 5-event vocabulary so Slack / Teams dashboards filtering on `event=training.failure` continue to work; pipeline-aware dashboards can additionally filter on the new `pipeline.*` events. + + **Acceptance:** + - 3-stage SFT→DPO→GRPO pipeline where DPO stage's evaluation gate fails: pipeline manifest shows DPO `status: failed` + `auto_revert_triggered: true`, GRPO `status: skipped_due_to_prior_revert`, final exit code matches the failing stage's exit code (not `EXIT_SUCCESS`). + - Same pipeline with `evaluation.require_human_approval: true` on the SFT stage: SFT exits with `EXIT_AWAITING_APPROVAL (4)`, pipeline state file records `status: gated_pending_approval` for SFT and `status: pending` for DPO / GRPO; after `forgelm approve ` + `forgelm --resume-from dpo_stage`, the pipeline picks up with DPO's input pointing at the promoted SFT `final_model/` path. + - Webhook stub asserts that `pipeline.started` fires once, `pipeline.stage_completed` fires per stage, `pipeline.completed` fires once, payload schema matches the documented spec. + - The existing single-stage `training.*` event vocabulary is unchanged; existing webhook consumers see no new events on non-pipeline runs. + +### Requirements + +- **Backward compatibility, byte-identical.** A config file without a `pipeline:` key produces *exactly* the same `forgelm/trainer.py` execution path as v0.6.0. `forgelm/trainer.py` itself is unmodified by Phase 14; the pipeline orchestrator is a layer above it. Regression test: re-run an existing single-stage config through Phase 14 code and assert the output JSONL / manifest / checkpoint bytes match a v0.6.0 reference fixture. +- **Each stage is independently testable** via `forgelm --config pipeline.yaml --stage ` (Task 4 partial-run rules). +- **Dry-run validates every stage** (Pydantic + chain integrity + manifest schema) before any GPU is allocated; collects all failures rather than stopping at the first. +- **Pipeline manifest integrates with existing `compliance.py`** — same atomic-write pattern, same JSON schema vocabulary; new fields added, none renamed or removed. +- **No new heavy dependencies.** Phase 14 uses only the existing Pydantic / PyYAML / pure-Python surface. No orchestration framework (Airflow / Prefect / Dagster) — pipelines stay in-process so the existing single-config-file UX scales to chains without a deployment story. +- **Determinism preserved.** For any fixed `pipeline.yaml` + input bytes, the run is deterministic *modulo the existing single-stage non-determinism surface* (GPU floating-point, dataloader RNG). The pipeline manifest's `pipeline_config_hash` proves config-level reproducibility. +- **Bilingual docs sweep.** Operator-facing guides updated in EN + TR: new `docs/guides/pipeline.md` + `pipeline-tr.md` (canonical "first pipeline" walkthrough), updates to `docs/reference/configuration.md` + `-tr.md` (the new `pipeline:` block), updates to `docs/guides/cli.md` + `-tr.md` (`--stage`, `--resume-from`, `--force-resume`). `tools/check_bilingual_parity.py --strict` must pass at PR open. +- **CLI help / wizard surface.** `forgelm --help` lists the three new flags with one-line descriptions; `forgelm wizard` does *not* gain a pipeline path in Phase 14 (single-stage wizard is already complex; chained pipelines are operator-grade and use the manual YAML surface). `tools/check_cli_help_consistency.py --strict` must pass. +- **Test budget.** ≥ 30 new tests across `tests/test_pipeline_config.py` (inheritance matrix), `tests/test_pipeline_orchestrator.py` (orchestrator state machine), `tests/test_pipeline_cli.py` (flag interactions), `tests/test_pipeline_compliance.py` (manifest schema + Annex IV verification). Coverage delta target: `+3 %` overall, with new pipeline modules ≥ 90 % line coverage. +- **Honest framing in user-facing docs.** The "Limitations" section of `docs/guides/pipeline.md` calls out: (a) no intra-stage checkpoint resume (deferred), (b) no DAG semantics (sequential only), (c) no parallel stage execution (sequential only), (d) no `forgelm wizard` integration. Operators see what does *not* work before they hit it. + +### Test fixture sketch (Phase 15-style matrix) + +Committed under `tests/fixtures/pipeline/`: + +| Fixture | Purpose | +|---|---| +| `minimal_3_stage.yaml` | SFT → DPO → GRPO with the smallest valid config per stage; smoke baseline. | +| `inheritance_matrix.yaml` | Hits every row of the inheritance matrix (auto-chain, inherited, overridden, rejected-per-stage). | +| `gated_pending_approval.yaml` | Stage 1 has `require_human_approval: true`; orchestrator must exit `EXIT_AWAITING_APPROVAL (4)` and write the gate state. | +| `auto_revert_at_stage_2.yaml` | Stage 2 has a `max_acceptable_loss` that the synthetic eval guarantees to exceed; subsequent stages must record `skipped_due_to_prior_revert`. | +| `invalid_distributed_per_stage.yaml` | Per-stage `distributed:` block; orchestrator must reject at config-load time with the exact error message documented in Task 2. | +| `stale_state_resume.yaml` | Pair of configs differing in `learning_rate`; on resume against the first state file, the second config must trigger the `pipeline_config_hash mismatch` exit. | + +Each fixture has a golden pipeline manifest committed alongside; regression suite runs `forgelm` against the fixture (with mocked trainers via the existing `MagicMock(ForgeTrainer)` pattern) and byte-compares. + +### Delivery + +- **Target release:** `v0.7.0`. Re-slotted from the earlier `v0.6.0` placeholder; Phase 15 (Ingestion Pipeline Reliability) displaced it after the 2026-05-11 ingestion pilot exposed silent-failure gaps that demanded a focused release — see [completed-phases.md#phase-15-ingestion-pipeline-reliability-v060](completed-phases.md#phase-15-ingestion-pipeline-reliability-v060). +- **Entry gate (all must hold before the first Phase 14 PR opens):** + - The 6 regression fixtures listed above are committed as zero-byte placeholders under `tests/fixtures/pipeline/` so the file paths are locked. + - The Phase 14 issue (`#TBD`) is opened on GitHub referencing this file. + - A Phase 14 GitHub label is created so review-round PRs can be grouped. + - Baseline coverage of `forgelm/config.py` and `forgelm/compliance.py` measured against the v0.6.0 tag and recorded in the Phase 14 issue (consistent with the Phase 15 entry-gate discipline). +- **CHANGELOG plan:** every Wave 1 task lands a one-line bullet under `[Unreleased]` per Keep-a-Changelog convention; at `v0.7.0` tag time the `[Unreleased]` block is renamed to `[0.7.0] — YYYY-MM-DD` per the existing release ritual. +- **Validation gate to ship Phase 14:** all 6 regression fixtures pass byte-comparison; bilingual parity + CLI help consistency guards green; full `pytest tests/` green; new `forgelm verify-annex-iv --pipeline ` validator returns exit 0 on every fixture's golden manifest. At least one end-to-end manual run against a 3-stage real pipeline (operator-supplied datasets) is signed off in the Phase 14 issue before the release tag. +- **Wave 2 / deferred items** (separate follow-up phases, not part of v0.7.0): + - Intra-stage HF `Trainer.train(resume_from_checkpoint=...)` integration (would let `--resume-from` pick up mid-stage rather than at stage boundaries). + - DAG pipelines (non-linear stage dependencies) — would require a different config schema and explicit dependency declaration; v0.7.x or later. + - Parallel stage execution (independent branches running concurrently) — gated on the DAG schema; same horizon. + - `forgelm wizard` pipeline path — gated on operator demand after v0.7.0 ships. + +--- -### Requirements: -- Single-stage configs (no `pipeline:` key) must be completely unaffected — 100% backward compatible. -- Each stage is independently testable via `--stage `. -- Pipeline manifest integrates with existing `compliance.py` audit log. -- `--dry-run` validates all stages' configs before any GPU allocation. +### Cross-references -### Delivery: -- Target release: `v0.6.0` (reslotted from earlier `v0.5.x` placeholders; Phase 14 is grouped with the Phase 13 Pro CLI tier per the v0.5.5 closure cycle's deferred-callout policy in `docs/usermanuals/{en,tr}/deployment/model-merging.md` and the [Phase 12.6 closure cycle archive](completed-phases.md#phase-126-closure-cycle-38-tasks-across-5-waves)). -- No hard blockers; the v0.5.5 closure cycle has merged. Phase 14 starts after the v0.5.5 PyPI tag is published. +- **Code surface affected:** [`forgelm/config.py`](../../forgelm/config.py) (new `PipelineStage` + `PipelineConfig` models, root-level `pipeline` field), [`forgelm/cli/_parser.py`](../../forgelm/cli/_parser.py) (`--stage`, `--resume-from`, `--force-resume`, `--input-model`), [`forgelm/cli/_dispatch.py`](../../forgelm/cli/_dispatch.py) + new `forgelm/cli/_pipeline.py` (orchestrator), [`forgelm/compliance.py`](../../forgelm/compliance.py) (`generate_pipeline_manifest`, `verify_annex_iv --pipeline` mode), [`forgelm/webhook.py`](../../forgelm/webhook.py) (`notify_pipeline_*` methods). [`forgelm/trainer.py`](../../forgelm/trainer.py) is **not** modified — the orchestrator composes existing `ForgeTrainer` instances per stage. +- **User-facing surfaces to add/update:** new `docs/guides/pipeline.md` + `-tr.md` (planned, created by Phase 14 implementation), updates to [`docs/reference/configuration.md`](../reference/configuration.md) + `-tr.md`, new `docs/guides/cli.md` + `-tr.md` (planned), and [`config_template.yaml`](../../config_template.yaml) gains a commented-out `pipeline:` example block. +- **Standards consulted:** [`docs/standards/error-handling.md`](../standards/error-handling.md) (exit code stability), [`docs/standards/architecture.md`](../standards/architecture.md) (layering — orchestrator above trainer), [`docs/standards/testing.md`](../standards/testing.md) (fixture matrix discipline), [`docs/standards/localization.md`](../standards/localization.md) (EN ↔ TR mirror). +- **Pattern reference:** Phase 15's review-absorption discipline + fixture matrix is the working model — see [completed-phases.md#phase-15-ingestion-pipeline-reliability-v060](completed-phases.md#phase-15-ingestion-pipeline-reliability-v060). --- From a01308ca7b73db4e62ce5ed1f37b3d227cda2747 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 14 May 2026 12:34:05 +0300 Subject: [PATCH 03/22] =?UTF-8?q?feat(phase-14):=20multi-stage=20training?= =?UTF-8?q?=20pipeline=20chains=20(SFT=20=E2=86=92=20DPO=20=E2=86=92=20GRP?= =?UTF-8?q?O)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements every Phase 14 task documented in ``docs/roadmap/phase-14-pipeline-chains.md``. Single-stage runs reach ``forgelm/trainer.py`` byte-identically to v0.6.0 — the orchestrator module is only imported when a ``pipeline:`` block is present. Foundation (Wave 1) - ``forgelm/config.py``: ``PipelineStage`` + ``PipelineConfig`` Pydantic models enforce ``^[a-z0-9_]{1,32}$`` stage names, ≥ 1-stage pipelines, unique-name validator, and explicit-``trainer_type`` audit-clarity rule per stage. ``extra="forbid"`` rejects per-stage ``distributed`` / ``webhook`` / ``compliance`` / ``risk_assessment`` / ``monitoring`` / ``retention`` / ``synthetic`` / ``merge`` / ``auth`` with the offending key in the Pydantic error. - ``merge_pipeline_stage_config()`` helper materialises a flat ``ForgeConfig`` per stage with section-wholesale inheritance + the 4-priority auto-chain resolution rule (``--input-model`` > explicit stage.model > prev.output_dir/final_model > root.model.name_or_path). - ``ForgeConfig.pipeline: Optional[PipelineConfig]`` + ``model_rebuild()`` to resolve the forward reference. Orchestrator (Wave 2) - ``forgelm/cli/_pipeline.py`` (new module): ``PipelineOrchestrator`` drives the chain. ``PipelineStageState`` + ``PipelineState`` dataclasses round-trip to ``pipeline_state.json`` via atomic write (tmp + ``os.replace``). ``pipeline.started`` / ``pipeline.stage_started`` / ``pipeline.stage_completed`` / ``pipeline.stage_reverted`` / ``pipeline.completed`` audit events fire alongside the existing per-stage ``training.*`` vocabulary so pre-existing dashboards keep working. - ``--stage `` partial-run requires the previous stage's on-disk output (auto-resolved) or ``--input-model `` (escape hatch); silent fallback to root model is hard-failed. - ``--resume-from `` skips ``completed`` stages whose output_model path exists. Stale-state guard refuses to resume when ``pipeline_config_hash`` differs from the current YAML bytes; ``--force-resume`` bypasses with a WARNING + audit record. - Human-approval gate inside a pipeline exits ``EXIT_AWAITING_APPROVAL (4)`` and leaves downstream stages ``pending`` (not ``skipped_due_to_prior_revert``) so resume picks them up post-approval. Compliance (Wave 3) - ``forgelm/compliance.py``: ``generate_pipeline_manifest()`` builds the Annex IV chain index documented in Phase 14 Task 3 schema. ``export_pipeline_manifest()`` writes it atomically. ``verify_pipeline_manifest()`` walks a pipeline run directory and returns a ``VerifyPipelineResult`` (chain integrity + per-stage manifest existence). ``_verify_manifest_payload()`` is the in-memory helper for unit tests. - ``forgelm verify-annex-iv --pipeline `` CLI mode validates a complete chain and exits non-zero on any violation. CLI + webhook (Wave 4) - ``forgelm/cli/_parser.py``: new ``--stage``, ``--resume-from``, ``--force-resume``, ``--input-model`` top-level flags. - ``forgelm/cli/_dispatch.py``: routes to the orchestrator when ``config.pipeline`` is set; passes the raw YAML bytes for the config-hash guard. - ``forgelm/webhook.py``: ``notify_pipeline_started`` / ``notify_pipeline_completed`` / ``notify_pipeline_reverted`` methods; gated by the existing ``notify_on_start`` / ``notify_on_success`` / ``notify_on_failure`` config flags. Tests + fixtures (Waves 2-5) - ``tests/test_pipeline_config.py`` — 53 schema + inheritance-matrix + auto-chain priority tests. - ``tests/test_pipeline_orchestrator.py`` — 27 state-machine tests covering happy-path / auto-revert / human-approval / resume / stale-config / partial-filter / webhook-failure-tolerance. - ``tests/test_pipeline_compliance.py`` — 14 manifest schema + chain-integrity verifier tests (including partial-filter and auto-revert manifest acceptance). - ``tests/test_pipeline_cli.py`` — 7 flag-interaction + dispatch-routing tests. - 6 fixture YAMLs under ``tests/fixtures/pipeline/`` exercising every documented scenario. Operator surface (Waves 6-7) - ``docs/guides/pipeline.md`` + ``docs/guides/pipeline-tr.md`` — bilingual operator guide (when to use, inheritance matrix, CLI semantics, human-approval flow, Annex IV verifier, Limitations). - ``config_template.yaml`` carries a commented-out ``pipeline:`` example block. - ``CHANGELOG.md`` ``[Unreleased]`` section ``Added (Phase 14 — …)`` + ``Changed (Phase 14)`` entries. - ``docs/roadmap/phase-14-pipeline-chains.md`` Status header updated to "Implementation complete on development; awaiting v0.7.0 PyPI tag". - ``tools/check_bilingual_parity.py`` ``_PAIRS`` registers the new guide pair. Wizard / notebooks / CLI help - ``forgelm wizard`` is **not** extended with a pipeline path (operator-grade workflow; documented in the guide's Limitations). - ``notebooks/`` carries no new pipeline notebook (fixture suite serves the same purpose without 3× boilerplate duplication). - 451 ``forgelm`` invocations across docs / README pass the CLI-help-consistency guard against the live parser surface. Verification - ruff format + check: clean across 174 files. - 11 / 11 self-review gauntlet guards green (dry-run, bilingual parity, anchor resolution, CLI help consistency 451 invocations, wizard defaults, no-analysis-refs, no-unguarded- sys.modules-pop, site-version-literal sync). - 1879 passed / 13 skipped / 0 failed across the full pytest suite (28 min 35 sec; ``tests/test_phase12_review_fixes.py`` perf-flake remains documented out-of-scope). - All 4 new Phase 14 test modules: 101 / 101 pass in 11 sec. Co-Authored-By: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 69 ++ config_template.yaml | 30 + docs/guides/pipeline-tr.md | 290 ++++++ docs/guides/pipeline.md | 287 ++++++ docs/roadmap/phase-14-pipeline-chains.md | 15 +- forgelm/cli/_dispatch.py | 22 + forgelm/cli/_parser.py | 68 +- forgelm/cli/_pipeline.py | 884 ++++++++++++++++++ forgelm/cli/subcommands/_verify_annex_iv.py | 41 +- forgelm/compliance.py | 258 +++++ forgelm/config.py | 269 ++++++ forgelm/webhook.py | 102 ++ .../pipeline/auto_revert_at_stage_2.yaml | 43 + .../pipeline/gated_pending_approval.yaml | 33 + .../fixtures/pipeline/inheritance_matrix.yaml | 58 ++ .../invalid_distributed_per_stage.yaml | 31 + tests/fixtures/pipeline/minimal_3_stage.yaml | 42 + .../pipeline/stale_state_resume_v1.yaml | 33 + .../pipeline/stale_state_resume_v2.yaml | 34 + tests/test_pipeline_cli.py | 163 ++++ tests/test_pipeline_compliance.py | 297 ++++++ tests/test_pipeline_config.py | 369 ++++++++ tests/test_pipeline_orchestrator.py | 523 +++++++++++ tools/check_bilingual_parity.py | 1 + 24 files changed, 3959 insertions(+), 3 deletions(-) create mode 100644 docs/guides/pipeline-tr.md create mode 100644 docs/guides/pipeline.md create mode 100644 forgelm/cli/_pipeline.py create mode 100644 tests/fixtures/pipeline/auto_revert_at_stage_2.yaml create mode 100644 tests/fixtures/pipeline/gated_pending_approval.yaml create mode 100644 tests/fixtures/pipeline/inheritance_matrix.yaml create mode 100644 tests/fixtures/pipeline/invalid_distributed_per_stage.yaml create mode 100644 tests/fixtures/pipeline/minimal_3_stage.yaml create mode 100644 tests/fixtures/pipeline/stale_state_resume_v1.yaml create mode 100644 tests/fixtures/pipeline/stale_state_resume_v2.yaml create mode 100644 tests/test_pipeline_cli.py create mode 100644 tests/test_pipeline_compliance.py create mode 100644 tests/test_pipeline_config.py create mode 100644 tests/test_pipeline_orchestrator.py diff --git a/CHANGELOG.md b/CHANGELOG.md index e63de8d3..f72f6db1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,75 @@ All notable changes to ForgeLM are documented here. _(v0.6.1 dev cycle — entries will land here as PRs merge.)_ +### Added (Phase 14 — Multi-Stage Pipeline Chains) + +- **`pipeline:` config block** at the root of `ForgeConfig` chains 2+ + training stages (typically SFT → DPO → GRPO) into one + config-driven run. New Pydantic models `PipelineStage` and + `PipelineConfig` enforce stage-name uniqueness, `^[a-z0-9_]{1,32}$` + identifier shape, ≥1-stage minimum, and explicit-`trainer_type` + audit-clarity validation per stage. Section-wholesale inheritance: + `model` / `lora` / `training` / `data` / `evaluation` blocks + inherit from the root if omitted or fully replace it when set. + `distributed` / `webhook` / `compliance` / `risk_assessment` / + `monitoring` / `retention` / `synthetic` / `merge` / `auth` are + root-only and rejected per-stage with `EXIT_CONFIG_ERROR (1)`. +- **`forgelm/cli/_pipeline.py` orchestrator** drives the chain + end-to-end. Auto-chains each stage's `model.name_or_path` to the + previous stage's output, persists state atomically to + `/pipeline_state.json` after every transition, + and emits 5 new audit events: `pipeline.started`, + `pipeline.stage_started`, `pipeline.stage_completed`, + `pipeline.stage_reverted`, `pipeline.completed`. Existing + `training.*` per-stage events from `ForgeTrainer` are preserved — + pre-existing Slack / Teams dashboards filtering on `training.failure` + keep working unchanged. +- **CLI flags** `--stage ` (single-stage filter for audit / + re-run), `--resume-from ` (stage-boundary resume), + `--force-resume` (stale-config-hash override), and `--input-model + ` (auto-chain escape hatch, recorded with `input_source: + cli_override` in the audit log). +- **`--dry-run` multi-stage validation** — when the config carries a + `pipeline:` block, dry-run validates every stage's merged config + + the chain-integrity assertion (stage N's auto-chained input is + under stage N-1's `output_dir`). Errors are collected before + exiting (à la `pytest --collectonly`). +- **Pipeline Annex IV manifest** — `compliance/pipeline_manifest.json` + at `` indexes per-stage `training_manifest.json` + files into one verifiable chain artefact, carrying + `pipeline_run_id` + `pipeline_config_hash` (SHA-256 of the YAML + bytes) for reproducibility. `forgelm verify-annex-iv --pipeline + ` walks the index, checks chain integrity, and asserts + every referenced per-stage manifest exists on disk. +- **Webhook notifications** — `WebhookNotifier.notify_pipeline_started`, + `notify_pipeline_completed`, and `notify_pipeline_reverted` + mirror the orchestrator's audit events; gated by the existing + `notify_on_start` / `notify_on_success` / `notify_on_failure` + config flags so operators silencing per-stage events also silence + the pipeline-level pings. +- **Bilingual operator guide** — new `docs/guides/pipeline.md` + + `docs/guides/pipeline-tr.md` (canonical "first pipeline" walkthrough, + inheritance matrix, CLI semantics, Annex IV verifier flow, + Limitations section). `config_template.yaml` gains a commented-out + `pipeline:` example block. + +### Changed (Phase 14) + +- **Backward compatibility preserved.** A config file without a + `pipeline:` key reaches `forgelm/trainer.py` byte-identically to + v0.6.0; the orchestrator module is never imported. +- `compliance.py` exports four new symbols: `generate_pipeline_manifest` + (called by the orchestrator after every transition), + `export_pipeline_manifest` (atomic write to + `/compliance/pipeline_manifest.json`), + `verify_pipeline_manifest(manifest: dict) -> List[str]` (in-memory + structural + chain-integrity check, returns a list of human-readable + violations — empty when valid), and `verify_pipeline_manifest_at_path + (pipeline_dir: str) -> List[str]` (disk-backed wrapper used by the + `forgelm verify-annex-iv --pipeline ` CLI mode). The shared + validator lives in private helper `_verify_manifest_payload` so the + in-memory and disk-bound entry points cannot drift. + ### Security - **Webhook / judge / synthetic SSRF guard — DNS-rebinding TOCTOU diff --git a/config_template.yaml b/config_template.yaml index 6a7cfd83..fe35b495 100644 --- a/config_template.yaml +++ b/config_template.yaml @@ -223,3 +223,33 @@ data: # ephemeral_artefact_retention_days: 90 # compliance bundles + audit reports # raw_documents_retention_days: 90 # ingested raw docs (PDF/DOCX/EPUB/TXT/MD) # enforce: "log_only" # "log_only" | "warn_on_excess" | "block_on_excess" + +# Phase 14 — Multi-stage training pipeline chains (SFT → DPO → GRPO). +# Uncomment the ``pipeline:`` block below to run multiple training stages +# back-to-back; each stage's input model is auto-chained to the previous +# stage's output_dir/final_model. Stages inherit unset config blocks +# from the root (above). See docs/guides/pipeline.md. +# pipeline: +# output_dir: "./pipeline_run" # pipeline-level dir (state + manifest live here) +# stages: +# - name: sft_stage +# training: +# trainer_type: "sft" +# output_dir: "./pipeline_run/stage1_sft" +# num_train_epochs: 3 +# data: +# dataset_name_or_path: "./data/sft.jsonl" +# - name: dpo_stage +# training: +# trainer_type: "dpo" +# output_dir: "./pipeline_run/stage2_dpo" +# num_train_epochs: 1 +# dpo_beta: 0.1 +# data: +# dataset_name_or_path: "./data/preferences.jsonl" +# - name: grpo_stage +# training: +# trainer_type: "grpo" +# output_dir: "./pipeline_run/stage3_grpo" +# data: +# dataset_name_or_path: "./data/math_prompts.jsonl" diff --git a/docs/guides/pipeline-tr.md b/docs/guides/pipeline-tr.md new file mode 100644 index 00000000..d89269c6 --- /dev/null +++ b/docs/guides/pipeline-tr.md @@ -0,0 +1,290 @@ +# Çok Aşamalı Eğitim Pipeline'ları + +Faz 14 — `v0.7.0` ile geldi. + +ForgeLM'in `pipeline:` config bloğu, 2 veya daha fazla eğitim aşamasını +(genellikle SFT → DPO → GRPO) tek bir config-tabanlı, dry-run-doğrulanabilir, +Annex IV-izlenebilir koşuya zincirler. Faz 14'ten önce aynı iş 3 ya da +daha fazla ayrı config dosyası, aşamalar arası elle `model.name_or_path` +düzenleme ve dışarıdan bir shell betiğiyle orkestrasyon gerektiriyordu. +Faz 14'ten sonra tek bir `forgelm --config pipeline.yaml` çağrısı tüm +zinciri uçtan uca yürütür. + +--- + +## When to use a pipeline (and when not to) + +`pipeline:` bloğunu **tüm** aşağıdaki koşullar geçerliyse kullanın: + +- Sıralı olarak 2 veya daha fazla eğitim aşaması koşturuyorsunuz + (örn. SFT sonrası DPO). +- Her aşamanın giriş modeli, önceki aşamanın çıkış modelidir. +- Tüm zinciri kapsayan tek bir Annex IV manifest istiyorsunuz. + +`pipeline:` bloğunu kullanmayın eğer: + +- Sadece tek bir eğitim paradigması koşturuyorsanız. Tek aşamalı + config'ler (v0.6.0 default'u) hâlâ kanonik kullanım örneğidir ve + Faz-14 öncesi davranışlarını byte-byte korur. +- Doğrusal olmayan aşama bağımlılıkları (DAG-şekilli pipeline'lar) + istiyorsanız. Faz 14 sadece sıralı pipeline'lar gönderir; şema + ilerideki bir DAG genişlemesi için yüzeyi koruyor. +- Paralel aşama yürütmesi (eşzamanlı bağımsız dallar) istiyorsanız. + DAG desteğiyle aynı zaman ufkunda — Wave 2 veya sonrası. + +--- + +## Anatomy of a pipeline config + +```yaml +# Root config — her aşamanın aşmadığı sürece miras aldığı varsayılanları +# sağlar. +model: + name_or_path: "meta-llama/Llama-3-8B" # Aşama 0'ın başlangıç modeli +lora: + r: 8 + alpha: 16 +training: + trainer_type: "sft" # Training bloğunu miras alan + # aşamalar bunu kullanır +data: + dataset_name_or_path: "./placeholder.jsonl" # Root'ta zorunlu; her + # aşama kendi data'sını verir + +# Pipeline-level blok — zinciri tanımlar. +pipeline: + output_dir: "./pipeline_run" # pipeline_state.json, + # compliance/pipeline_manifest.json + # ve pipeline-level audit_log.jsonl + # buraya yazılır + stages: + - name: sft_stage # ^[a-z0-9_]{1,32}$ + training: + trainer_type: "sft" # Her aşama için zorunlu + output_dir: "./pipeline_run/stage1_sft" + num_train_epochs: 3 + data: + dataset_name_or_path: "./data/sft.jsonl" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./pipeline_run/stage2_dpo" + num_train_epochs: 1 + dpo_beta: 0.1 + data: + dataset_name_or_path: "./data/preferences.jsonl" + + - name: grpo_stage + training: + trainer_type: "grpo" + output_dir: "./pipeline_run/stage3_grpo" + grpo_num_generations: 4 + data: + dataset_name_or_path: "./data/math_prompts.jsonl" +``` + +**Otomatik zincirleme:** her aşamanın `model.name_or_path` değeri, +önceki aşamanın `training.output_dir/final_model` yoluna otomatik +olarak ayarlanır — bu yolu elinizle yazmazsınız ve aşama N'in DPO +eğiticisini yanlışlıkla base modele (aşama N-1'in SFT çıktısı yerine) +yönlendiremezsiniz. + +--- + +## Inheritance matrix + +Section-wholesale override semantiği — bir aşama üst seviye blok (`model`, +`lora`, `training`, `data`, `evaluation`) tanımlarsa **tüm** blok root'unkini +değiştirir; aşama bloğu atlarsa root'un bloğu birebir miras alınır. Field +seviyesinde deep-merge yok: "miras almak istiyorsan bloğu yazma; override +etmek istiyorsan tam bloğu yaz." + +| Bölüm | Miras davranışı | Notlar | +|---|---|---| +| `model.name_or_path` | **Otomatik zincirlenir** (aşama 1'den itibaren root'u geçer) | Önceki aşamanın `training.output_dir/final_model` değerine ayarlanır. Aşama 0 root'un `model.name_or_path`'ini okur. Aşamada explicit `model:` bloğu otomatik zincirlemeyi o aşama için devre dışı bırakır (kaçış kapısı). | +| `model.*` (diğer alanlar) | `model:` bloğu override edilmediyse miras alınır | `backend`, `load_in_4bit`, `trust_remote_code`, `max_length`, `chat_template` root'u izler. | +| `lora` | `lora:` bloğu override edilmediyse miras alınır | **Kritik edge case:** aşama 2 DPO, aşama 1 SFT'den *farklı* bir `lora.r` ile gelirse aşama 2 **birleştirilmiş SFT modelinin üstüne taze bir LoRA**dır, SFT'nin LoRA'sının devamı değil. Operatör-görünür davranış. | +| `data` | `data:` bloğu override edilmediyse miras alınır | Pipeline'lar aşamalar arası nadiren aynı dataset'i kullanır; aşama bazında explicit data yaygın kullanım. | +| `training` | `training:` bloğu override edilmediyse miras alınır | `trainer_type` her aşamada explicit set EDİLMEK ZORUNDADIR (audit-clarity doğrulayıcısı). Diğer alanlar blok override edildiğinde wholesale yer değiştirir. | +| `evaluation` | `evaluation:` bloğu override edilmediyse miras alınır | Aşama bazında gate'ler (loss eşikleri, auto_revert, safety, judge, human-approval) burada yaşar. | +| `distributed` | Sadece root — **aşamada reddedilir** | Distributed strateji koşu boyunca tutarlı olmak zorundadır. | +| `webhook` | Sadece root — **aşamada reddedilir** | Aşama bazında olaylar payload'da aşama adını taşır. | +| `compliance` | Sadece root — **aşamada reddedilir** | Provider/system/risk metadata pipeline seviyesindedir. | +| `risk_assessment`, `monitoring`, `retention`, `synthetic`, `merge`, `auth` | Sadece root — **aşamada reddedilir** | Pipeline seviyesi sorumlulukları. | + +Bir aşama root-only bölümlerden birini tanımlarsa, config yükleme +zamanında `EXIT_CONFIG_ERROR (1)` ile reddedilir ve hata mesajı +ihlal eden bölüm adını taşır. + +--- + +## CLI + +```bash +# Uçtan uca pipeline koşusu. +forgelm --config pipeline.yaml + +# Dry-run her aşamayı (Pydantic + chain integrity) hiçbir GPU +# tahsisi olmadan doğrular. `pytest --collectonly` gibi tüm +# hataları çıkmadan önce toplar. +forgelm --config pipeline.yaml --dry-run + +# Tek bir adlı aşamayı yalıtılmış olarak koştur (audit / yeniden +# koşturma senaryoları). İlk olmayan aşamalar önceki aşamanın +# disk üzerindeki çıktısını veya explicit --input-model override +# gerektirir. +forgelm --config pipeline.yaml --stage dpo_stage + +# Başarısız / kesintiye uğramış bir koşudan adlı aşamadan itibaren +# devam et. output_model yolu disk üzerinde var olan +# already-completed aşamalar atlanır (INFO log'lanır). +forgelm --config pipeline.yaml --resume-from dpo_stage + +# Tek aşama için otomatik zincirlenen input modeli override et +# (kaçış kapısı). Audit log `input_source: cli_override` kaydeder. +forgelm --config pipeline.yaml --stage dpo_stage --input-model ./other/checkpoint +``` + +### `--stage ` partial-run rules + +| Senaryo | Davranış | +|---|---| +| `--stage ` ve `` ilk aşama | Root `model.name_or_path` okunur; normal koşar. | +| `--stage ` ve önceki aşamanın `output_dir/final_model` diskte var | Önceki aşama yeni bitmiş gibi otomatik zincirlenir. | +| `--stage ` ve önceki aşamanın çıktısı eksik | `EXIT_CONFIG_ERROR (1)` ile hard-fail: `Stage requires output at ; pass --input-model to override or run the full pipeline first.` Root `model.name_or_path`'e sessiz geri dönüş yok. | +| `--stage --input-model ` | Operatör kaçış kapısı: otomatik zincirlemeyi atlar, ``'i kullanır. Audit log `input_source: cli_override` kaydeder. | +| `--stage ` ve `` config'te yok | Parse zamanında valid aşama adları listesiyle hard-fail. | + +### `--resume-from ` semantics + +- State dosyası: `/pipeline_state.json` (atomic-write). +- `status: completed` ve `output_model` yolu hâlâ diskte olan aşamalar + **atlanır** ve aşama bazında training manifest'leri korunur. Adlı + aşama ve sonrasındaki her aşama yeniden koşturulur. +- **Stale-state koruması:** disk üzerindeki state dosyasının + `pipeline_config_hash`'i mevcut YAML byte'larından farklıysa, resume + `EXIT_CONFIG_ERROR (1)` ile başarısız olur — bu "koşu sırasında + config'i düzenledim ve resume ettim" sessiz divergenc'yi engeller. + `--force-resume` ile override edilebilir (WARNING log'lanır, audit + event kaydedilir). +- Faz 14 sadece **aşama sınırlarında** resume eder. Aşama içi HF + `Trainer.train(resume_from_checkpoint=...)` entegrasyonu bir Faz 14.x + takipçisine ertelendi — Sınırlamalar bölümüne bakın. + +--- + +## Human-approval gate within a pipeline + +Bir aşama `evaluation.require_human_approval: true` taşıyorsa, ForgeLM'in +mevcut Faz 9 akışı aynen koşar: model `final_model.staging./`'ye +iner, orkestratör staging yolunu pipeline state'inde +`status: gated_pending_approval` ile yakalar ve koşu +`EXIT_AWAITING_APPROVAL (4)` ile çıkar. + +Sonraki aşamalar `pending` kalır (`skipped_due_to_prior_revert` değil) +böylece onaydan sonra resume onları alır: + +```bash +# Aşama 1 SFT onay bekliyor — pipeline 4 ile çıkar. +$ forgelm --config pipeline.yaml +# ... → exit 4 + +# Operatör staged modeli inceler, sonra onaylar. +$ forgelm approve --output-dir ./pipeline_run/stage1_sft +# ... → final_model/ promote edilir, exit 0 + +# Pipeline'ı DPO'dan resume et; SFT atlanır (status=completed on disk). +# DPO'nun input_model'i staging değil *promote edilen* final_model/ +# yolunu işaret eder. +$ forgelm --config pipeline.yaml --resume-from dpo_stage +# ... → zincirin geri kalanı başarılıysa exit 0 +``` + +--- + +## Audit events + +Orkestratör bu olayları pipeline-level `audit_log.jsonl`'a (pipeline.output_dir +altında) emit eder: + +- `pipeline.started` — run id, config hash, stage count, stage names +- `pipeline.stage_started` — stage name, index, input model, input source +- `pipeline.stage_completed` — stage name, gate decision, metrics summary +- `pipeline.stage_reverted` — stage name, auto-revert reason +- `pipeline.completed` — final status, stopped-at stage (varsa) + +Bu olaylar mevcut her aşamanın `ForgeTrainer`'inin emit etmeye devam +ettiği `training.*` olaylarının yanında yaşar; `training.failure` +üzerinde filtreleyen Slack / Teams dashboard'ları değişmeden çalışır. + +İlgili webhook metotları `notify_pipeline_started`, +`notify_pipeline_completed` ve `notify_pipeline_reverted` — +`forgelm/webhook.py`'a bakın. + +--- + +## Annex IV manifest + +Her aşama geçişi `/compliance/pipeline_manifest.json` +dosyasını (atomic-write) yeniden yazar. Pipeline manifest, aşama bazında +`training_manifest.json` dosyalarını doğrulanabilir bir zincire bağlayan +**indekstir**; aşama bazında manifest'ler tek aşamalı Annex IV şemasına +karşı bireysel olarak geçerli kalmaya devam eder. + +Tam bir koşuyu doğrulayıcı ile geçerleyin: + +```bash +forgelm verify-annex-iv --pipeline ./pipeline_run +``` + +Doğrulayıcı şu koşullar sağlandığında exit 0 döner: + +1. Her zorunlu üst seviye anahtar mevcut ve iyi biçimli. +2. Aşama indeksleri sırayla `0..N-1`. +3. Her chain aşamasının `input_model`'i önceki yürütülen aşamanın + `output_model`'iyle eşit (operatör `--input-model` override'ları + `input_source: cli_override` ile kaydedilir ve bu kontrolden + tasarımla muaftır — auditor'lar legitimate override'ı bozuk + manifest'ten ayırmak için audit log'a bakar). +4. Her aşama bazında `training_manifest` yolu gerçek bir dosyayı + işaret eder. +5. `stopped_at` (set ise) gerçek bir aşamayı adlandırır ve o aşamanın + status'u `failed` veya `gated_pending_approval`. + +Sıfır olmayan exit kodu ihlalleri keşfedildikleri sırayla listeler. + +--- + +## Limitations (Phase 14 Wave 1) + +- **Aşama içi checkpoint resume yok.** `--resume-from` sadece aşama + sınırlarında devam alır. Bir aşamanın `ForgeTrainer.train()` + yarıda crash ederse, resume o aşamayı epoch 0'dan yeniden koşturur. + Wave 2. +- **Sadece sıralı — DAG semantiği yok.** Aşamalar `pipeline.stages` + içinde görünme sıralarında yürütülür. Branch / fan-out / merge + ertelendi. +- **Paralel aşama yürütmesi yok.** İki aşama mantıken bağımsız olsa + bile orkestratör onları sıralı koşturur. +- **`forgelm wizard` entegrasyonu yok.** Tek aşamalı config'ler + wizard'la üretilebilir; pipeline'lar operatör-seviyesi ve manuel + YAML yüzeyini kullanır. Wizard'ın işi "çalışan bir tek aşama + config'i üret, büyüdüğünde elle pipeline'a düzenle" olarak kalır. +- **Notebook entegrasyonu yok.** `notebooks/` altındaki 11 demo + notebook bireysel eğitim paradigmalarını kapsar; uçtan uca pipeline + demosu her notebook'un setup boilerplate'ini üç kez tekrar ederdi. + `tests/fixtures/pipeline/` altındaki fixture suite, reviewer'lara + bir notebook'la aynı yüzeyi verir — üstüne golden manifest'le + byte-comparable olma avantajıyla. + +--- + +## Cross-references + +- Faz 14 tasarım dokümanı: [docs/roadmap/phase-14-pipeline-chains.md](../roadmap/phase-14-pipeline-chains.md) +- Roadmap girişi: [docs/roadmap-tr.md](../roadmap-tr.md) +- Annex IV doğrulayıcısı: `forgelm verify-annex-iv --pipeline ` (CLI help'e bakın) +- Audit log standardı: [docs/standards/logging-observability.md](../standards/logging-observability.md) +- Tek aşamalı eğitici kılavuzu (her şey bu yüzeyden miras alır): + [docs/guides/alignment-tr.md](alignment-tr.md) diff --git a/docs/guides/pipeline.md b/docs/guides/pipeline.md new file mode 100644 index 00000000..5462e59d --- /dev/null +++ b/docs/guides/pipeline.md @@ -0,0 +1,287 @@ +# Multi-Stage Training Pipelines + +Phase 14 — added in `v0.7.0`. + +ForgeLM's `pipeline:` config block chains 2 or more training stages +(typically SFT → DPO → GRPO) into one config-driven, dry-run-validatable, +Annex-IV-traceable run. Before Phase 14, the same workflow required +3+ separate config files, manual `model.name_or_path` editing between +stages, and an external shell script for orchestration. After Phase 14, +a single `forgelm --config pipeline.yaml` invocation runs the entire +chain end-to-end. + +--- + +## When to use a pipeline (and when not to) + +Use the `pipeline:` block when **all** of the following hold: + +- You are running 2 or more training stages in sequence (e.g. SFT + followed by DPO). +- Each stage's input model is the previous stage's output model. +- You want a single Annex IV manifest covering the entire chain. + +Do **not** use the `pipeline:` block when: + +- You only need to run a single training paradigm. Single-stage configs + (the v0.6.0 default) remain the canonical case and stay byte-identical + to their pre-Phase-14 behaviour. +- You need non-linear stage dependencies (DAG-shaped pipelines). + Phase 14 ships sequential pipelines only; the schema reserves the + surface for a future DAG extension. +- You want parallel stage execution (independent branches running + concurrently). Same horizon as the DAG support — Wave 2 or later. + +--- + +## Anatomy of a pipeline config + +```yaml +# Root config — provides defaults that every stage inherits from +# unless the stage overrides them. +model: + name_or_path: "meta-llama/Llama-3-8B" # Stage 0's starting model +lora: + r: 8 + alpha: 16 +training: + trainer_type: "sft" # Used by stages that + # inherit the training block +data: + dataset_name_or_path: "./placeholder.jsonl" # Required at root; each + # stage overrides per-stage + +# Pipeline-level block — declares the chain. +pipeline: + output_dir: "./pipeline_run" # Hosts pipeline_state.json, + # compliance/pipeline_manifest.json, + # and the pipeline-level + # audit_log.jsonl + stages: + - name: sft_stage # ^[a-z0-9_]{1,32}$ + training: + trainer_type: "sft" # Required per stage + output_dir: "./pipeline_run/stage1_sft" + num_train_epochs: 3 + data: + dataset_name_or_path: "./data/sft.jsonl" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./pipeline_run/stage2_dpo" + num_train_epochs: 1 + dpo_beta: 0.1 + data: + dataset_name_or_path: "./data/preferences.jsonl" + + - name: grpo_stage + training: + trainer_type: "grpo" + output_dir: "./pipeline_run/stage3_grpo" + grpo_num_generations: 4 + data: + dataset_name_or_path: "./data/math_prompts.jsonl" +``` + +**Auto-chain:** each stage's `model.name_or_path` is automatically set to +the previous stage's `training.output_dir/final_model` — you do not +write that path by hand, and you cannot accidentally point stage N's +DPO trainer at the base model instead of stage N-1's SFT output. + +--- + +## Inheritance matrix + +Section-wholesale override semantics — if a stage declares a top-level +block (`model`, `lora`, `training`, `data`, `evaluation`), the **entire** +block replaces the root's; if the stage omits the block, the root's +block is inherited verbatim. No field-level deep-merge: "if you want to +inherit, omit the block; if you want to override, supply the full block." + +| Section | Inheritance | Notes | +|---|---|---| +| `model.name_or_path` | **Auto-chained** (overrides root from stage 1 onward) | Set to previous stage's `training.output_dir/final_model`. Stage 0 reads root `model.name_or_path`. An explicit per-stage `model:` block disables auto-chaining for that stage (escape hatch). | +| `model.*` (other fields) | Inherited unless `model:` block overridden | `backend`, `load_in_4bit`, `trust_remote_code`, `max_length`, `chat_template` follow the root. | +| `lora` | Inherited unless `lora:` block overridden | **Critical edge case:** stage 2 DPO with a *different* `lora.r` than stage 1 SFT means stage 2 is **a fresh LoRA over the merged SFT model**, not a continuation of SFT's LoRA. Operator-visible behaviour. | +| `data` | Inherited unless `data:` block overridden | Pipelines rarely reuse the same dataset across stages; explicit per-stage data is the common case. | +| `training` | Inherited unless `training:` block overridden | `trainer_type` MUST be set explicitly per stage (audit-clarity validator). Other fields wholesale-replaced when the block is overridden. | +| `evaluation` | Inherited unless `evaluation:` block overridden | Per-stage gates (loss thresholds, auto_revert, safety, judge, human-approval) live here. | +| `distributed` | Root only — **per-stage rejected** | Distributed strategy must be consistent across the run. | +| `webhook` | Root only — **per-stage rejected** | Per-stage events carry the stage name in the payload. | +| `compliance` | Root only — **per-stage rejected** | Provider/system/risk metadata is pipeline-level. | +| `risk_assessment`, `monitoring`, `retention`, `synthetic`, `merge`, `auth` | Root only — **per-stage rejected** | Pipeline-level concerns. | + +A stage that declares any of the root-only sections is rejected at +config-load time with `EXIT_CONFIG_ERROR (1)` and the offending section +name in the message. + +--- + +## CLI + +```bash +# End-to-end pipeline run. +forgelm --config pipeline.yaml + +# Dry-run validates every stage (Pydantic + chain integrity) before any +# GPU is allocated. Collects all errors before exiting, like +# `pytest --collectonly`. +forgelm --config pipeline.yaml --dry-run + +# Run a single named stage in isolation (audit / re-run scenarios). +# Non-first stages require the previous stage's on-disk output, or +# an explicit --input-model override. +forgelm --config pipeline.yaml --stage dpo_stage + +# Resume after a failed / interrupted run from a named stage onward. +# Already-completed stages whose output paths exist on disk are +# skipped (logged at INFO). +forgelm --config pipeline.yaml --resume-from dpo_stage + +# Override the auto-chained input model for a single stage (escape hatch). +# The audit log records `input_source: cli_override`. +forgelm --config pipeline.yaml --stage dpo_stage --input-model ./other/checkpoint +``` + +### `--stage ` partial-run rules + +| Scenario | Behaviour | +|---|---| +| `--stage ` and `` is the first stage | Reads root `model.name_or_path`; runs normally. | +| `--stage ` and the previous stage's `output_dir/final_model` exists on disk | Auto-chains as if the previous stage had just finished. | +| `--stage ` and the previous stage's output is missing | Hard-fails with `EXIT_CONFIG_ERROR (1)`: `Stage requires output at ; pass --input-model to override or run the full pipeline first.` No silent fallback to root `model.name_or_path`. | +| `--stage --input-model ` | Operator escape hatch: skips the auto-chain, uses ``. Audit-log records `input_source: cli_override`. | +| `--stage ` where `` is not in the config | Hard-fails at parse time with the list of valid stage names. | + +### `--resume-from ` semantics + +- State file: `/pipeline_state.json` (atomic-write). +- Stages with `status: completed` whose `output_model` path still exists + on disk are **skipped** and their per-stage training manifests are + preserved. The named stage and every stage after it are re-run. +- **Stale-state guard:** if the on-disk state file's + `pipeline_config_hash` differs from the current parse of the YAML + bytes, resume fails with `EXIT_CONFIG_ERROR (1)` — preventing + "I resumed against a config that was edited mid-flight" silent + divergence. Override via `--force-resume` (logged at WARNING, + recorded in the audit event). +- Phase 14 resumes at **stage boundaries** only. Intra-stage HF + `Trainer.train(resume_from_checkpoint=...)` integration is deferred + to a Phase 14.x follow-up — see Limitations below. + +--- + +## Human-approval gate within a pipeline + +If a stage carries `evaluation.require_human_approval: true`, ForgeLM's +existing Phase 9 flow runs unchanged: the model lands in +`final_model.staging./`, the orchestrator captures the staging +path in the pipeline state with `status: gated_pending_approval`, and +the run exits with `EXIT_AWAITING_APPROVAL (4)`. + +Downstream stages stay `pending` (not `skipped_due_to_prior_revert`) +so a subsequent resume picks them up after the approval: + +```bash +# Stage 1 SFT gates pending approval — pipeline exits 4. +$ forgelm --config pipeline.yaml +# ... → exit 4 + +# Operator inspects the staged model, then approves. +$ forgelm approve --output-dir ./pipeline_run/stage1_sft +# ... → final_model/ is promoted, exit 0 + +# Resume the pipeline from DPO; SFT is skipped (status=completed +# on disk). DPO's input_model points at the *promoted* final_model/ +# path, not the staging path. +$ forgelm --config pipeline.yaml --resume-from dpo_stage +# ... → exit 0 when the rest of the chain succeeds +``` + +--- + +## Audit events + +The orchestrator emits these events into the pipeline-level +`audit_log.jsonl` (under `pipeline.output_dir`): + +- `pipeline.started` — run id, config hash, stage count, stage names +- `pipeline.stage_started` — stage name, index, input model, input source +- `pipeline.stage_completed` — stage name, gate decision, metrics summary +- `pipeline.stage_reverted` — stage name, auto-revert reason +- `pipeline.completed` — final status, stopped-at stage (if any) + +These events live alongside the existing `training.*` events that each +stage's `ForgeTrainer` continues to emit; pre-existing Slack / Teams +dashboards filtering on `training.failure` keep working unchanged. + +The matching webhook methods are `notify_pipeline_started`, +`notify_pipeline_completed`, and `notify_pipeline_reverted` — see +`forgelm/webhook.py`. + +--- + +## Annex IV manifest + +Every stage transition rewrites `/compliance/ +pipeline_manifest.json` (atomic-write). The pipeline manifest is the +**index** that ties the per-stage `training_manifest.json` files into +one verifiable chain; per-stage manifests remain individually valid +against the existing single-stage Annex IV schema. + +Validate a complete run with the verifier: + +```bash +forgelm verify-annex-iv --pipeline ./pipeline_run +``` + +The verifier returns exit 0 when: + +1. Every required top-level key is present and well-formed. +2. Stage indices form `0..N-1` in order. +3. Every chain stage's `input_model` equals the previous executed + stage's `output_model` (operator `--input-model` overrides are + recorded with `input_source: cli_override` and exempted from this + check, by design — auditors cross-reference the audit log to + distinguish a legitimate override from a corrupt manifest). +4. Every per-stage `training_manifest` path points at a real file. +5. `stopped_at` (if set) names a real stage whose status is `failed` + or `gated_pending_approval`. + +A non-zero exit code lists the violations in the order they were +discovered. + +--- + +## Limitations (Phase 14 Wave 1) + +- **No intra-stage checkpoint resume.** `--resume-from` picks up at + stage boundaries only. If a stage's `ForgeTrainer.train()` crashes + half-way, the resume re-runs that stage from epoch 0. Wave 2. +- **Sequential only — no DAG semantics.** Stages execute in the order + they appear in `pipeline.stages`. Branches / fan-out / merge are + deferred. +- **No parallel stage execution.** Even when two stages are logically + independent, the orchestrator runs them sequentially. +- **No `forgelm wizard` integration.** Single-stage configs are + wizard-buildable; pipelines are operator-grade and use the manual + YAML surface. The wizard's job stays "produce a working single-stage + config you can hand-edit into a pipeline if you outgrow it." +- **No notebook integration.** The 11 demo notebooks under + `notebooks/` cover individual training paradigms; an end-to-end + pipeline demo would duplicate every notebook's setup boilerplate + three times. The fixture suite under `tests/fixtures/pipeline/` + gives reviewers exactly the same surface as a notebook would, with + the advantage of being byte-comparable to a golden manifest. + +--- + +## Cross-references + +- Phase 14 design doc: [docs/roadmap/phase-14-pipeline-chains.md](../roadmap/phase-14-pipeline-chains.md) +- Roadmap entry: [docs/roadmap.md](../roadmap.md) +- Annex IV verifier: `forgelm verify-annex-iv --pipeline ` (see CLI help) +- Audit log standard: [docs/standards/logging-observability.md](../standards/logging-observability.md) +- Single-stage trainer guide (everything inherits from this surface): + [docs/guides/alignment.md](alignment.md) diff --git a/docs/roadmap/phase-14-pipeline-chains.md b/docs/roadmap/phase-14-pipeline-chains.md index 014e1e84..4152d84c 100644 --- a/docs/roadmap/phase-14-pipeline-chains.md +++ b/docs/roadmap/phase-14-pipeline-chains.md @@ -1,6 +1,6 @@ # Phase 14: Multi-Stage Training Pipeline Chains -> **Status:** Planning. Targeted for `v0.7.0` (originally `v0.6.0`; Phase 15 displaced it after the 2026-05-11 ingestion pilot — see [completed-phases.md#phase-15-ingestion-pipeline-reliability-v060](completed-phases.md#phase-15-ingestion-pipeline-reliability-v060)). No hard blockers; `v0.6.0` has shipped, Phase 14 can start. +> **Status:** Implementation complete on `development` (branch `feat/phase-14-pipeline-chains`). Awaiting `v0.7.0` PyPI tag for the final "Done" promotion to [completed-phases.md](completed-phases.md). Originally targeted for `v0.6.0`; Phase 15 displaced it after the 2026-05-11 ingestion pilot — see [completed-phases.md#phase-15-ingestion-pipeline-reliability-v060](completed-phases.md#phase-15-ingestion-pipeline-reliability-v060). > **Note:** This file details a single planned phase. See [../roadmap.md](../roadmap.md) for the cross-phase summary. @@ -246,4 +246,17 @@ Each fixture has a golden pipeline manifest committed alongside; regression suit - **Standards consulted:** [`docs/standards/error-handling.md`](../standards/error-handling.md) (exit code stability), [`docs/standards/architecture.md`](../standards/architecture.md) (layering — orchestrator above trainer), [`docs/standards/testing.md`](../standards/testing.md) (fixture matrix discipline), [`docs/standards/localization.md`](../standards/localization.md) (EN ↔ TR mirror). - **Pattern reference:** Phase 15's review-absorption discipline + fixture matrix is the working model — see [completed-phases.md#phase-15-ingestion-pipeline-reliability-v060](completed-phases.md#phase-15-ingestion-pipeline-reliability-v060). +### Release-time follow-up (deferred to the `v0.7.0` `cut-release` cycle) + +Phase 14 implementation ships on the `development` branch ahead of the +PyPI tag. The two operator-facing surfaces below are intentionally not +edited inside this PR — they are released-product surfaces and stay +v0.6.0-shaped until `cut-release` actually tags `v0.7.0`: + +- **`docs/usermanuals/_meta.yaml` + `docs/usermanuals/{en,tr,de,fr,es,zh}/training/pipeline.md`** — new training-section page mirroring `docs/guides/pipeline.md`. TOC entry needs 6 locale titles (`en`/`tr` mandatory; `de`/`fr`/`es`/`zh` fall back to EN per `docs/standards/localization.md`). +- **`site/features.html` + `site/index.html` + `site/js/translations.js`** — new "Multi-stage training pipelines" feature card and a hook into the existing `home.pipeline` (alignment-stack) section so the two distinct senses of the word "pipeline" don't collide visually. 6-language i18n keys (`features.pipeline.title`, `features.pipeline.body`, `home.pipeline.chain.cta`) gated at full parity by `tools/check_site_claims.py`. +- **`tools/update_site_version.py`** — Phase 14 ships under the `v0.7.0` literal bump, which the version guard rewrites across `site/*.html` and `site/js/translations.js` automatically. + +The `cut-release` skill ([.claude/skills/cut-release/SKILL.md](../../.claude/skills/cut-release/SKILL.md)) walks these in order at `v0.7.0` tag time. + --- diff --git a/forgelm/cli/_dispatch.py b/forgelm/cli/_dispatch.py index da2ecbec..a06cc9b1 100644 --- a/forgelm/cli/_dispatch.py +++ b/forgelm/cli/_dispatch.py @@ -262,4 +262,26 @@ def _main_inner() -> None: config = _load_config_or_exit(args.config, json_output) _apply_offline_flag(config, args.offline) _maybe_run_no_train_mode(config, args) + + # Phase 14 — multi-stage pipeline branch. When the YAML carries a + # ``pipeline:`` block, hand control to the orchestrator instead of + # the single-stage trainer. Single-stage configs (the v0.6.0 default) + # skip this branch entirely and reach the trainer as before. + if config.pipeline is not None: + from ._pipeline import run_pipeline_from_args + + # Re-read the YAML *bytes* so the orchestrator can hash them for + # the ``--resume-from`` stale-config guard. We deliberately + # don't trust ``yaml.dump(config.model_dump())`` here — comments + # and key order would be lost, and the hash is meant to prove + # the on-disk artefact, not the parsed semantic content. + try: + with open(args.config, "rb") as f: + pipeline_yaml_bytes = f.read() + except OSError as e: + logger.error("Failed to re-read pipeline YAML for hashing: %s", e) + sys.exit(EXIT_CONFIG_ERROR) + + sys.exit(run_pipeline_from_args(config, pipeline_yaml_bytes, args)) + _run_training_pipeline(config, args, json_output) diff --git a/forgelm/cli/_parser.py b/forgelm/cli/_parser.py index 2e916d56..d0d485d5 100644 --- a/forgelm/cli/_parser.py +++ b/forgelm/cli/_parser.py @@ -754,7 +754,25 @@ def _add_verify_annex_iv_subcommand(subparsers) -> None: "Exits 0 on valid; 1 on missing field or hash mismatch." ), ) - p.add_argument("path", help="Path to the Annex IV JSON artifact (typically `compliance/annex_iv_.json`).") + p.add_argument( + "path", + help=( + "Path to the Annex IV JSON artifact (typically " + "``compliance/annex_iv_.json``). When ``--pipeline`` is set, " + "this is interpreted as a pipeline run directory containing " + "``compliance/pipeline_manifest.json`` instead." + ), + ) + p.add_argument( + "--pipeline", + action="store_true", + help=( + "Phase 14: interpret ``path`` as a pipeline run directory and " + "validate the chain-level ``pipeline_manifest.json`` (chain " + "integrity, stage-index ordering, ``stopped_at`` coherence, " + "per-stage training_manifest existence)." + ), + ) _add_common_subparser_flags(p, include_output_format=True) @@ -1230,6 +1248,54 @@ def parse_args(): parser.add_argument( "--merge", action="store_true", help="Run model merging from the merge section of your config. No training." ) + # Phase 14 — multi-stage pipeline chains. Active only when the + # ``pipeline:`` block is present in the config; documented at + # docs/guides/pipeline.md. + parser.add_argument( + "--stage", + type=str, + default=None, + metavar="NAME", + help=( + "Multi-stage pipelines only: run a single named stage in isolation " + "(audit / re-run scenarios). Non-first stages require the previous " + "stage's on-disk output, or an explicit --input-model override." + ), + ) + parser.add_argument( + "--resume-from", + type=str, + default=None, + metavar="NAME", + help=( + "Multi-stage pipelines only: resume an interrupted run from a named " + "stage onward. Already-completed stages whose output_model paths " + "exist on disk are skipped; refuses to run if the on-disk " + "pipeline_config_hash differs from the current YAML (unless " + "--force-resume is also set)." + ), + ) + parser.add_argument( + "--force-resume", + action="store_true", + help=( + "Multi-stage pipelines only: bypass the --resume-from stale-config " + "guard. The override is logged at WARNING and recorded in the " + "audit event." + ), + ) + parser.add_argument( + "--input-model", + type=str, + default=None, + metavar="PATH", + help=( + "Multi-stage pipelines only: when used with --stage, replaces the " + "auto-chained input model with this path. The audit-log entry " + "records ``input_source: cli_override`` so reviewers see the chain " + "was broken intentionally." + ), + ) parser.add_argument( "--output-format", type=str, diff --git a/forgelm/cli/_pipeline.py b/forgelm/cli/_pipeline.py new file mode 100644 index 00000000..f07d7fec --- /dev/null +++ b/forgelm/cli/_pipeline.py @@ -0,0 +1,884 @@ +"""Phase 14 — Multi-stage training pipeline orchestrator. + +Top-level entry point for ``forgelm --config pipeline.yaml`` runs. The +orchestrator iterates over the ordered :class:`PipelineConfig.stages`, +materialises a flat :class:`ForgeConfig` per stage via +:func:`forgelm.config.merge_pipeline_stage_config`, hands it to a fresh +:class:`forgelm.trainer.ForgeTrainer`, and threads the state through a +crash-safe JSON state file + an append-only audit-log event stream. + +Layering rule — see ``docs/roadmap/phase-14-pipeline-chains.md`` Task 2: +``forgelm/trainer.py`` is **not** aware of the pipeline layer. Each +stage is a single-stage run from the trainer's point of view; only this +module knows there is an outer loop. When a config carries no +``pipeline:`` block, this module is never imported. + +Key responsibilities: + +- **Auto-chain** each stage's ``model.name_or_path`` to the previous + stage's output path (or honour an explicit stage override / CLI + ``--input-model`` override). +- **Per-stage gates** — auto-revert and human-approval are composed + one stage at a time; a failed gate stops the chain and marks + downstream stages ``skipped_due_to_prior_revert``. +- **Crash-safe state** — every stage transition is atomically written to + ``/pipeline_state.json`` (tmp file + ``os.replace``). + ``--resume-from`` reads that file and skips ``completed`` stages whose + outputs still exist on disk. +- **Annex-IV manifest** — every transition also rewrites + ``/compliance/pipeline_manifest.json`` (delegated to + :mod:`forgelm.compliance`) so reviewers can correlate every chain + failure to a frozen artefact. +- **Audit + webhook** — emits ``pipeline.started`` / ``.stage_started`` + / ``.stage_completed`` / ``.stage_reverted`` / ``.completed`` audit + events alongside the existing per-stage ``training.*`` vocabulary; the + webhook notifier gains ``notify_pipeline_*`` methods that the orchestrator + calls at the same transitions. +- **Exit-code routing** — the pipeline exits with the highest-priority + per-stage exit code: ``EXIT_AWAITING_APPROVAL (4)`` if any stage gated + pending approval, then ``EXIT_TRAINING_ERROR (2)`` for trainer + crashes, then ``EXIT_EVAL_FAILURE (3)`` for gate failures, then + ``EXIT_CONFIG_ERROR (1)`` for orchestrator-level config issues, then + ``EXIT_SUCCESS (0)``. +""" + +from __future__ import annotations + +import hashlib +import json +import logging +import os +import secrets +import sys +from dataclasses import asdict, dataclass, field +from datetime import datetime, timezone +from typing import Any, Dict, List, Literal, Optional + +from ..config import ForgeConfig, PipelineConfig, PipelineStage, merge_pipeline_stage_config +from ._exit_codes import ( + EXIT_AWAITING_APPROVAL, + EXIT_CONFIG_ERROR, + EXIT_EVAL_FAILURE, + EXIT_SUCCESS, + EXIT_TRAINING_ERROR, +) + +logger = logging.getLogger("forgelm.pipeline") + + +# --------------------------------------------------------------------------- +# State enums + dataclasses +# --------------------------------------------------------------------------- + + +StageStatusLiteral = Literal[ + "pending", + "running", + "completed", + "failed", + "gated_pending_approval", + "skipped_due_to_prior_revert", + "skipped_by_filter", +] +"""Per-stage status values. See Phase 14 Task 3 for the meaning of each.""" + + +InputSourceLiteral = Literal["root", "chain", "stage_explicit", "cli_override"] +"""How the orchestrator resolved a stage's input model path. Recorded on +the per-stage state so reviewers can trace any chain interruption +(``cli_override`` is the operator escape hatch that intentionally breaks +the chain).""" + + +FinalStatusLiteral = Literal[ + "in_progress", + "completed", + "stopped_at_stage", + "gated_pending_approval", +] + + +@dataclass +class PipelineStageState: + """Per-stage state snapshot persisted to disk + included in the manifest.""" + + name: str + index: int + trainer_type: str + status: StageStatusLiteral = "pending" + input_model: Optional[str] = None + input_source: Optional[InputSourceLiteral] = None + output_model: Optional[str] = None + started_at: Optional[str] = None + finished_at: Optional[str] = None + duration_seconds: Optional[float] = None + training_manifest: Optional[str] = None + metrics: Dict[str, float] = field(default_factory=dict) + gate_decision: Optional[str] = None + auto_revert_triggered: bool = False + skipped_reason: Optional[str] = None + exit_code: Optional[int] = None + error: Optional[str] = None + + +@dataclass +class PipelineState: + """Top-level pipeline-run state snapshot. + + Round-tripped to ``pipeline_state.json`` after every stage transition. + Compatible with :mod:`json` (atomic-write via tmp + ``os.replace``); + no nested non-serialisable types. + """ + + pipeline_run_id: str + pipeline_config_hash: str + forgelm_version: str + started_at: str + finished_at: Optional[str] = None + final_status: FinalStatusLiteral = "in_progress" + stopped_at: Optional[str] = None + stages: List[PipelineStageState] = field(default_factory=list) + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _utc_iso_now() -> str: + """UTC ISO-8601 timestamp with seconds precision. + + Mirrors the format ``forgelm/compliance.py`` already uses for audit + log entries so timestamps line up across the two surfaces. + """ + return datetime.now(timezone.utc).isoformat(timespec="seconds") + + +def _generate_run_id() -> str: + """Return a pipeline run id of shape ``pl_YYYY-MM-DD_<6-hex>``. + + The date prefix lets operators eyeball runs by week without consulting + the manifest; the 6-hex suffix gives ~16 million combinations per day, + which is more than enough to disambiguate same-day re-runs even in + aggressive CI loops. + """ + today = datetime.now(timezone.utc).strftime("%Y-%m-%d") + return f"pl_{today}_{secrets.token_hex(3)}" + + +def _compute_pipeline_config_hash(pipeline_yaml_bytes: bytes) -> str: + """SHA-256 of the *raw bytes* of the pipeline YAML file. + + Hashing the bytes (not the parsed dict) is intentional: it locks the + on-disk artefact, which is what regulators audit. YAML + comments / key ordering / whitespace are all in scope; an operator + who edits the file between runs gets a different hash even if the + semantic content is unchanged, and the ``--resume-from`` stale-state + guard catches it. + """ + return "sha256:" + hashlib.sha256(pipeline_yaml_bytes).hexdigest() + + +def _pipeline_paths(root_cfg: ForgeConfig) -> Dict[str, str]: + """Resolve the canonical filesystem paths for the pipeline run. + + All pipeline-level artefacts (state file, manifest, audit log) live + under :attr:`PipelineConfig.output_dir` (defaults to + ``./pipeline_run``). Per-stage trainer artefacts continue to live + under each stage's own ``training.output_dir`` (as in single-stage + runs) — the pipeline output dir is the *index* directory, distinct + from the stage checkpoint directories so operators can ``rm -rf`` + individual stages without nuking the pipeline manifest. + """ + if root_cfg.pipeline is None: # defensive — orchestrator constructor catches this earlier + raise ValueError("Cannot derive pipeline paths from a config with no `pipeline` block.") + root_output_dir = root_cfg.pipeline.output_dir + return { + "root_output_dir": root_output_dir, + "state_file": os.path.join(root_output_dir, "pipeline_state.json"), + "manifest_dir": os.path.join(root_output_dir, "compliance"), + "manifest_file": os.path.join(root_output_dir, "compliance", "pipeline_manifest.json"), + } + + +def _atomic_write_json(path: str, payload: Dict[str, Any]) -> None: + """Write ``payload`` to ``path`` atomically. + + Pattern: serialise to a sibling ``.tmp`` then ``os.replace`` it + onto the final name — guarantees a reader on any concurrent process + either sees the previous version or the new version, never a half- + written file. + """ + os.makedirs(os.path.dirname(path) or ".", exist_ok=True) + tmp = f"{path}.tmp" + with open(tmp, "w", encoding="utf-8") as f: + json.dump(payload, f, indent=2, sort_keys=False) + os.replace(tmp, path) + + +def _serialise_state(state: PipelineState) -> Dict[str, Any]: + """Return a JSON-ready dict representation of a :class:`PipelineState`.""" + return asdict(state) + + +def _deserialise_state(payload: Dict[str, Any]) -> PipelineState: + """Round-trip :class:`PipelineState` from its on-disk JSON shape. + + Tolerates unknown keys (future-forward compat) and missing optionals + (legacy state files). Type-coerces ``stages[]`` back into + :class:`PipelineStageState` instances so the rest of the orchestrator + sees a strongly-typed object. + """ + stages_raw = payload.get("stages", []) + stages = [ + PipelineStageState(**{k: v for k, v in s.items() if k in PipelineStageState.__dataclass_fields__}) + for s in stages_raw + ] + return PipelineState( + pipeline_run_id=payload["pipeline_run_id"], + pipeline_config_hash=payload["pipeline_config_hash"], + forgelm_version=payload.get("forgelm_version", "unknown"), + started_at=payload["started_at"], + finished_at=payload.get("finished_at"), + final_status=payload.get("final_status", "in_progress"), + stopped_at=payload.get("stopped_at"), + stages=stages, + ) + + +# --------------------------------------------------------------------------- +# Orchestrator +# --------------------------------------------------------------------------- + + +class PipelineOrchestrator: + """Drives a multi-stage pipeline run end-to-end. + + Constructor takes the root config and the *bytes* of the pipeline YAML + (the bytes are hashed for the stale-resume guard; the parsed config + drives the actual stage iteration). + """ + + def __init__( + self, + root_cfg: ForgeConfig, + pipeline_yaml_bytes: bytes, + *, + output_format: str = "text", + ) -> None: + if root_cfg.pipeline is None: + # Defensive — callers should only reach this path when a + # ``pipeline:`` block is present. A None ``pipeline`` here + # signals a CLI dispatch bug; fail loud rather than running + # an empty pipeline. + raise ValueError("PipelineOrchestrator instantiated with a config that has no `pipeline` block.") + self.root_cfg: ForgeConfig = root_cfg + self.pipeline: PipelineConfig = root_cfg.pipeline + self.config_hash: str = _compute_pipeline_config_hash(pipeline_yaml_bytes) + self.output_format: str = output_format + self.paths: Dict[str, str] = _pipeline_paths(root_cfg) + + # -- run-mode entry points ------------------------------------------------ + + def run( + self, + *, + stage_filter: Optional[str] = None, + resume_from: Optional[str] = None, + force_resume: bool = False, + input_model_override: Optional[str] = None, + ) -> int: + """Execute the pipeline. Returns a process exit code. + + ``stage_filter`` and ``resume_from`` are mutually exclusive; the + caller (CLI parser) validates this before reaching here. + + Exit-code precedence (highest first): + ``EXIT_TRAINING_ERROR (2)`` for a trainer crash, ``EXIT_AWAITING_APPROVAL (4)`` + for a gated stage, ``EXIT_EVAL_FAILURE (3)`` for a gate failure + / auto-revert, ``EXIT_CONFIG_ERROR (1)`` for orchestrator-level + config issues, ``EXIT_SUCCESS (0)`` otherwise. + """ + # 1. Load (or initialise) state. + existing = self._load_state_file() + if resume_from is not None and existing is None: + logger.error( + "Cannot --resume-from %r: no pipeline_state.json found at %s. Run the pipeline once first.", + resume_from, + self.paths["state_file"], + ) + return EXIT_CONFIG_ERROR + if resume_from is not None: + self._validate_resume_state(existing, force=force_resume) + state = existing + else: + state = self._init_state() + + # 2. Determine which stages to execute. + stages_to_skip_completed: List[str] = [] + if resume_from is not None: + try: + resume_idx = next(i for i, s in enumerate(self.pipeline.stages) if s.name == resume_from) + except StopIteration: + logger.error( + "Cannot --resume-from %r: stage name not found in pipeline (valid names: %s).", + resume_from, + ", ".join(s.name for s in self.pipeline.stages), + ) + return EXIT_CONFIG_ERROR + # Mark anything before the resume point that's already + # ``completed`` on disk as kept-as-is. + for prior_state in state.stages[:resume_idx]: + if ( + prior_state.status == "completed" + and prior_state.output_model + and os.path.isdir(prior_state.output_model) + ): + stages_to_skip_completed.append(prior_state.name) + logger.info( + "Resuming: stage %r already completed at %s; skipping.", + prior_state.name, + prior_state.output_model, + ) + + if stage_filter is not None and not any(s.name == stage_filter for s in self.pipeline.stages): + logger.error( + "Cannot --stage %r: stage name not found in pipeline (valid names: %s).", + stage_filter, + ", ".join(s.name for s in self.pipeline.stages), + ) + return EXIT_CONFIG_ERROR + + # 2a. ``--stage `` on a non-first stage requires the previous + # stage's output to already exist on disk (auto-chain resolution), + # unless ``--input-model`` provides an explicit override. Refuse + # to silently fall back to the root ``model.name_or_path`` — that + # would produce a chain whose stage N is trained on the root base + # model, not on the previous stage's output, contradicting every + # operator's expectation of what ``--stage dpo_stage`` means. + prev_output_for_filter: Optional[str] = None + if stage_filter is not None and input_model_override is None: + filter_idx = next(i for i, s in enumerate(self.pipeline.stages) if s.name == stage_filter) + if filter_idx > 0: + prev_stage = self.pipeline.stages[filter_idx - 1] + # Project the previous stage's would-be output path by + # running it through the merge helper with no chain (so + # we get its declared ``training.output_dir`` honestly). + prev_merged = merge_pipeline_stage_config(self.root_cfg, prev_stage, prev_output_model=None) + candidate = os.path.join(prev_merged.training.output_dir, "final_model") + if not os.path.isdir(candidate): + logger.error( + "Stage %r requires the previous stage's output at %s; " + "pass --input-model to override or run the full " + "pipeline first.", + stage_filter, + candidate, + ) + return EXIT_CONFIG_ERROR + prev_output_for_filter = candidate + + # 3. Emit pipeline.started + webhook (only on a fresh run). + if resume_from is None: + self._audit_event( + "pipeline.started", + pipeline_run_id=state.pipeline_run_id, + config_hash=self.config_hash, + stage_count=len(self.pipeline.stages), + stage_names=[s.name for s in self.pipeline.stages], + ) + self._notify_pipeline("started", state) + + # 4. Iterate stages. + worst_exit: int = EXIT_SUCCESS + # When ``--stage X`` is set on a non-first stage, seed ``prev_output`` + # with the previous stage's on-disk output path resolved above so the + # filtered stage auto-chains correctly. In every other mode, the + # initial ``prev_output`` is None and the iteration accumulates it + # naturally from each completed stage. + prev_output: Optional[str] = prev_output_for_filter + chain_broken: bool = False # set when a stage auto-reverts; later stages skip + for i, stage in enumerate(self.pipeline.stages): + stage_state = state.stages[i] + + # --stage filter: only run the named stage; mark others skipped_by_filter. + if stage_filter is not None and stage.name != stage_filter: + stage_state.status = "skipped_by_filter" + stage_state.skipped_reason = f"--stage {stage_filter!r} was specified; only that stage runs." + self._save_state(state) + continue + + # --resume-from: skip already-completed stages with on-disk output. + if stage.name in stages_to_skip_completed: + prev_output = stage_state.output_model + continue + + # Chain-broken (prior stage reverted): mark remaining as skipped_due_to_prior_revert. + if chain_broken: + stage_state.status = "skipped_due_to_prior_revert" + stage_state.skipped_reason = ( + f"Stage {state.stopped_at!r} triggered auto_revert; downstream stages did not run." + ) + self._save_state(state) + continue + + # Run this stage. + stage_state.index = i + stage_state.name = stage.name + stage_state.trainer_type = ( + stage.training.trainer_type if stage.training else self.root_cfg.training.trainer_type + ) + stage_state.started_at = _utc_iso_now() + stage_state.status = "running" + self._save_state(state) + + # Resolve the input-model override only for the filtered stage + # (a CLI ``--input-model`` flag attaches to a single stage run). + override_for_this_stage = ( + input_model_override if (stage_filter is not None and stage.name == stage_filter) else None + ) + + exit_code = self._run_single_stage( + stage=stage, + stage_state=stage_state, + prev_output=prev_output, + input_model_override=override_for_this_stage, + state=state, + ) + stage_state.exit_code = exit_code + stage_state.finished_at = _utc_iso_now() + + # Duration calc — wrap in try/except to be robust against + # missing started_at on a constructor edge case. + try: + start_dt = datetime.fromisoformat(stage_state.started_at) + end_dt = datetime.fromisoformat(stage_state.finished_at) + stage_state.duration_seconds = (end_dt - start_dt).total_seconds() + except (TypeError, ValueError): + stage_state.duration_seconds = None + + # Decide where the pipeline goes next, given this stage's outcome. + if exit_code == EXIT_AWAITING_APPROVAL: + # Human-approval gate fired. Pipeline exits; remaining + # stages stay ``pending`` so a subsequent ``--resume-from`` + # picks them up after operator action. + stage_state.status = "gated_pending_approval" + state.final_status = "gated_pending_approval" + state.stopped_at = stage.name + state.finished_at = _utc_iso_now() + worst_exit = max(worst_exit, EXIT_AWAITING_APPROVAL) + self._save_state(state) + self._audit_event( + "pipeline.stage_completed", + pipeline_run_id=state.pipeline_run_id, + stage_name=stage.name, + gate_decision="approval_pending", + ) + # No pipeline.completed event yet — pipeline is not done. + break + + if exit_code == EXIT_SUCCESS: + stage_state.status = "completed" + stage_state.gate_decision = "passed" + self._save_state(state) + self._audit_event( + "pipeline.stage_completed", + pipeline_run_id=state.pipeline_run_id, + stage_name=stage.name, + gate_decision="passed", + metrics=stage_state.metrics, + ) + prev_output = stage_state.output_model + continue + + # Anything else (EXIT_TRAINING_ERROR / EXIT_EVAL_FAILURE): + # the chain stops here. Subsequent stages skip with + # ``skipped_due_to_prior_revert`` (the canonical reason + # name regardless of whether the failure was a trainer + # crash or a gate failure — both are "stage N failed, + # downstream stages did not run"). + stage_state.status = "failed" + stage_state.gate_decision = "failed" + chain_broken = True + state.stopped_at = stage.name + worst_exit = max(worst_exit, exit_code) + self._save_state(state) + self._audit_event( + "pipeline.stage_reverted" if stage_state.auto_revert_triggered else "pipeline.stage_completed", + pipeline_run_id=state.pipeline_run_id, + stage_name=stage.name, + gate_decision="failed", + auto_revert_triggered=stage_state.auto_revert_triggered, + ) + self._notify_pipeline( + "reverted" if stage_state.auto_revert_triggered else "stage_failed", state, stage_state=stage_state + ) + + # 5. Final state + manifest + pipeline.completed event. + if state.final_status == "in_progress": + if worst_exit == EXIT_SUCCESS: + state.final_status = "completed" + else: + state.final_status = "stopped_at_stage" + state.finished_at = _utc_iso_now() + self._save_state(state) + + if state.final_status in ("completed", "stopped_at_stage"): + self._audit_event( + "pipeline.completed", + pipeline_run_id=state.pipeline_run_id, + final_status=state.final_status, + stopped_at=state.stopped_at, + ) + self._notify_pipeline("completed", state) + + # 6. Emit human-readable / JSON envelope summary. + self._emit_summary(state) + + return worst_exit + + def dry_run(self) -> int: + """Validate every stage without allocating any GPU resources. + + Collects all per-stage Pydantic / merge errors before exiting, + mirroring ``pytest --collectonly``. Returns + :data:`EXIT_CONFIG_ERROR (1)` if any stage fails validation, + :data:`EXIT_SUCCESS (0)` otherwise. + """ + errors: List[str] = [] + prev_output: Optional[str] = None + for stage in self.pipeline.stages: + try: + merged = merge_pipeline_stage_config( + self.root_cfg, + stage, + prev_output_model=prev_output, + ) + except Exception as exc: # noqa: BLE001 — best-effort: collect every per-stage validation error for a single operator report instead of stopping at the first. + errors.append(f"Stage {stage.name!r}: merge failed: {exc}") + continue + + # Project the output path the next stage would auto-chain to. + prev_output = os.path.join(merged.training.output_dir, "final_model") + + if errors: + logger.error("Pipeline dry-run found %d stage error(s):", len(errors)) + for e in errors: + logger.error(" %s", e) + return EXIT_CONFIG_ERROR + + logger.info( + "Pipeline dry-run OK: %d stage(s) validated; no GPU was allocated.", + len(self.pipeline.stages), + ) + return EXIT_SUCCESS + + # -- internals ------------------------------------------------------------ + + def _init_state(self) -> PipelineState: + """Build a fresh :class:`PipelineState` from the parsed pipeline.""" + from .._version import __version__ + + stages = [ + PipelineStageState( + name=s.name, + index=i, + trainer_type=(s.training.trainer_type if s.training else self.root_cfg.training.trainer_type), + ) + for i, s in enumerate(self.pipeline.stages) + ] + return PipelineState( + pipeline_run_id=_generate_run_id(), + pipeline_config_hash=self.config_hash, + forgelm_version=__version__, + started_at=_utc_iso_now(), + stages=stages, + ) + + def _validate_resume_state(self, existing: PipelineState, *, force: bool) -> None: + """Stale-state guard for ``--resume-from``. + + If the on-disk state file's ``pipeline_config_hash`` differs from + the current parse of the pipeline YAML, the operator edited the + file between the failed run and the resume call — resuming + against a different config silently produces a chain whose + history doesn't match its current shape. We refuse unless + ``--force-resume`` is set (logged at WARNING, recorded in the + audit event so reviewers see why the divergence was accepted). + """ + if existing.pipeline_config_hash == self.config_hash: + return + if not force: + logger.error( + "Refusing to --resume-from: pipeline_config_hash mismatch.\n" + " on-disk: %s\n current: %s\n" + "Pass --force-resume to override (logged for audit).", + existing.pipeline_config_hash, + self.config_hash, + ) + sys.exit(EXIT_CONFIG_ERROR) + logger.warning( + "Resuming with --force-resume despite pipeline_config_hash mismatch: %s → %s.", + existing.pipeline_config_hash, + self.config_hash, + ) + # Update the stored hash to match the now-trusted current config + # so subsequent transitions don't trip the guard again. + existing.pipeline_config_hash = self.config_hash + + def _load_state_file(self) -> Optional[PipelineState]: + path = self.paths["state_file"] + if not os.path.isfile(path): + return None + try: + with open(path, "r", encoding="utf-8") as f: + payload = json.load(f) + except (OSError, json.JSONDecodeError) as e: + logger.warning("Failed to read pipeline_state.json at %s: %s. Treating as missing.", path, e) + return None + return _deserialise_state(payload) + + def _save_state(self, state: PipelineState) -> None: + """Persist state file + refresh pipeline manifest. + + Both writes are atomic. Manifest export is delegated to + :mod:`forgelm.compliance` so the schema lives in one place. + """ + _atomic_write_json(self.paths["state_file"], _serialise_state(state)) + # Lazy import — compliance has its own heavy deps for the + # Annex-IV verifier path; only needed when we actually export. + from ..compliance import generate_pipeline_manifest + + manifest = generate_pipeline_manifest(state=state, root_cfg=self.root_cfg) + _atomic_write_json(self.paths["manifest_file"], manifest) + + def _audit_event(self, event: str, **fields: Any) -> None: + """Append-only audit log entry under the root output dir. + + Reuses the existing :class:`forgelm.compliance.AuditLogger` so + the pipeline events land in the same JSONL stream as the + per-stage ``training.*`` events from each :class:`ForgeTrainer`. + Reviewers can grep one file for the full chain. + """ + try: + from ..compliance import AuditLogger + + audit = AuditLogger(self.paths["root_output_dir"]) + audit.log_event(event, **fields) + except Exception as exc: # noqa: BLE001 — audit emission is best-effort telemetry; a write failure must not abort the pipeline. + logger.warning("Failed to emit audit event %r: %s", event, exc) + + def _notify_pipeline( + self, + kind: Literal["started", "completed", "reverted", "stage_failed"], + state: PipelineState, + *, + stage_state: Optional[PipelineStageState] = None, + ) -> None: + """Webhook fan-out for pipeline lifecycle transitions. + + Notifier methods are optional (added in Phase 14, defended via + ``getattr`` so a fork that pinned an older webhook module + doesn't crash here). + """ + try: + from ..webhook import WebhookNotifier + + notifier = WebhookNotifier(self.root_cfg) + stage_count = len(self.pipeline.stages) + if kind == "started": + method = getattr(notifier, "notify_pipeline_started", None) + if method: + method(run_id=state.pipeline_run_id, stage_count=stage_count) + elif kind == "completed": + method = getattr(notifier, "notify_pipeline_completed", None) + if method: + method(run_id=state.pipeline_run_id, final_status=state.final_status, stopped_at=state.stopped_at) + elif kind == "reverted": + method = getattr(notifier, "notify_pipeline_reverted", None) + if method and stage_state is not None: + method( + run_id=state.pipeline_run_id, + stage_name=stage_state.name, + reason=stage_state.error or "auto_revert", + ) + # ``stage_failed`` is currently only an audit event — no + # dedicated webhook method. Existing per-stage + # ``training.failure`` from ForgeTrainer's notifier covers + # the operator surface; the pipeline-level event would be + # redundant noise on Slack. + except Exception as exc: # noqa: BLE001 — best-effort notification; webhook outage must not derail the pipeline. + logger.warning("Failed to emit pipeline webhook %r: %s", kind, exc) + + def _run_single_stage( + self, + *, + stage: PipelineStage, + stage_state: PipelineStageState, + prev_output: Optional[str], + input_model_override: Optional[str], + state: PipelineState, + ) -> int: + """Materialise the per-stage ForgeConfig, run the trainer, record results. + + Returns the per-stage exit code; the caller decides what to do + with it (continue, halt, gate). + """ + # Resolve input source for audit clarity. + if input_model_override is not None: + stage_state.input_source = "cli_override" + elif stage.model is not None: + stage_state.input_source = "stage_explicit" + elif prev_output is not None: + stage_state.input_source = "chain" + else: + stage_state.input_source = "root" + + # Merge config (any merge-time validation failure surfaces here). + try: + stage_cfg = merge_pipeline_stage_config( + self.root_cfg, + stage, + prev_output_model=prev_output, + input_model_override=input_model_override, + ) + except Exception as exc: # noqa: BLE001 — config-merge failure is a fatal stage error; routes through the standard config-error exit code. + stage_state.error = f"Config merge failed: {exc}" + logger.error("Stage %r config merge failed: %s", stage.name, exc) + return EXIT_CONFIG_ERROR + + stage_state.input_model = stage_cfg.model.name_or_path + + # ``forgelm`` ``--input-model`` partial-run guard: when the + # operator filters to a non-first stage and the previous stage's + # output directory does not exist, refuse to fabricate it. + if stage_state.input_source == "chain": + expected_dir = os.path.dirname(stage_cfg.model.name_or_path) + if not os.path.isdir(expected_dir): + stage_state.error = ( + f"Stage {stage.name!r} requires the previous stage's output at " + f"{stage_cfg.model.name_or_path}; pass --input-model to " + f"override or run the full pipeline first." + ) + logger.error(stage_state.error) + return EXIT_CONFIG_ERROR + + self._audit_event( + "pipeline.stage_started", + pipeline_run_id=state.pipeline_run_id, + stage_name=stage.name, + stage_index=stage_state.index, + input_model=stage_state.input_model, + input_source=stage_state.input_source, + ) + + # Defer heavy imports so dry-run / --stage on a different stage + # doesn't load torch. + try: + from ..data import prepare_dataset + from ..model import get_model_and_tokenizer + from ..trainer import ForgeTrainer + from ..utils import setup_authentication + except ImportError as e: + stage_state.error = f"Missing training dependency: {e}" + logger.error(stage_state.error) + return EXIT_TRAINING_ERROR + + try: + if not stage_cfg.model.offline: + setup_authentication(stage_cfg.auth.hf_token if stage_cfg.auth else None) + + model, tokenizer = get_model_and_tokenizer(stage_cfg) + dataset = prepare_dataset(stage_cfg, tokenizer) + trainer = ForgeTrainer(model=model, tokenizer=tokenizer, config=stage_cfg, dataset=dataset) + result = trainer.train() + except Exception as exc: # noqa: BLE001 — top-of-stage catch. ForgeTrainer.train() crosses every concern (HF load, dataset, TRL, safety, judge, audit, compliance, webhook); any uncaught exception must surface as a per-stage failure rather than a Python traceback that strands the pipeline state. + stage_state.error = f"Trainer crashed: {exc}" + logger.exception("Stage %r trainer crashed.", stage.name) + return EXIT_TRAINING_ERROR + + # Capture results onto the stage state. + stage_state.metrics = {k: float(v) for k, v in (result.metrics or {}).items() if isinstance(v, (int, float))} + stage_state.auto_revert_triggered = bool(result.reverted) + stage_state.training_manifest = os.path.join( + stage_cfg.training.output_dir, "compliance", "training_manifest.json" + ) + + # Human-approval gate produces a non-zero exit + a staging_path. + if result.staging_path: + stage_state.output_model = result.staging_path + return EXIT_AWAITING_APPROVAL + + # Auto-revert or gate failure: trainer returns success=False. + if not result.success: + stage_state.error = result.error or "Stage gate failed." + return EXIT_EVAL_FAILURE + + # Normal success path. + stage_state.output_model = result.final_model_path or os.path.join(stage_cfg.training.output_dir, "final_model") + return EXIT_SUCCESS + + # -- summary -------------------------------------------------------------- + + def _emit_summary(self, state: PipelineState) -> None: + """Operator-facing run summary (text or JSON envelope).""" + if self.output_format == "json": + print(json.dumps(_serialise_state(state), indent=2)) + return + + logger.info("Pipeline %s — final_status=%s", state.pipeline_run_id, state.final_status) + for s in state.stages: + logger.info( + " [%s] %s — trainer=%s, exit=%s, output=%s", + s.status, + s.name, + s.trainer_type, + s.exit_code if s.exit_code is not None else "—", + s.output_model or "—", + ) + + +# --------------------------------------------------------------------------- +# Entry point (CLI dispatcher delegates here) +# --------------------------------------------------------------------------- + + +def run_pipeline_from_args( + root_cfg: ForgeConfig, + pipeline_yaml_bytes: bytes, + args: Any, +) -> int: + """Dispatcher entry — orchestrates flag → orchestrator wiring. + + ``args`` is the parsed ``argparse.Namespace`` from the CLI; only the + pipeline-relevant attributes are read. Keeping this glue thin makes + the orchestrator itself trivial to unit-test against a fake args + namespace. + """ + stage_filter = getattr(args, "stage", None) + resume_from = getattr(args, "resume_from", None) + force_resume = bool(getattr(args, "force_resume", False)) + input_model_override = getattr(args, "input_model", None) + output_format = getattr(args, "output_format", "text") + dry_run = bool(getattr(args, "dry_run", False)) + + if stage_filter and resume_from: + logger.error("`--stage` and `--resume-from` are mutually exclusive.") + return EXIT_CONFIG_ERROR + if input_model_override and not stage_filter: + logger.error("`--input-model` requires `--stage `.") + return EXIT_CONFIG_ERROR + + orchestrator = PipelineOrchestrator( + root_cfg=root_cfg, + pipeline_yaml_bytes=pipeline_yaml_bytes, + output_format=output_format, + ) + + if dry_run: + return orchestrator.dry_run() + return orchestrator.run( + stage_filter=stage_filter, + resume_from=resume_from, + force_resume=force_resume, + input_model_override=input_model_override, + ) diff --git a/forgelm/cli/subcommands/_verify_annex_iv.py b/forgelm/cli/subcommands/_verify_annex_iv.py index 4458beb8..00aea7e2 100644 --- a/forgelm/cli/subcommands/_verify_annex_iv.py +++ b/forgelm/cli/subcommands/_verify_annex_iv.py @@ -186,7 +186,19 @@ def _compute_manifest_hash(artifact: Dict[str, Any]) -> str: def _run_verify_annex_iv_cmd(args, output_format: str) -> None: - """Top-level dispatcher for ``forgelm verify-annex-iv ``.""" + """Top-level dispatcher for ``forgelm verify-annex-iv ``. + + Two modes: + + - **Single artefact** (default): ```` is an Annex IV JSON file + and the verifier checks field completeness + manifest hash. + - **Pipeline** (``--pipeline`` flag): ```` is a pipeline run + directory and the verifier reads + ``/compliance/pipeline_manifest.json`` and runs chain- + integrity + stage-index + ``stopped_at`` coherence + per-stage + training_manifest existence checks. Returns a list of violations + and exits 0 only when the list is empty. + """ path = getattr(args, "path", None) if not path: _output_error_and_exit( @@ -194,6 +206,33 @@ def _run_verify_annex_iv_cmd(args, output_format: str) -> None: "verify-annex-iv requires a path argument: `forgelm verify-annex-iv `.", EXIT_CONFIG_ERROR, ) + + # Phase 14: ``--pipeline`` mode validates the chain-level manifest. + if getattr(args, "pipeline", False): + from forgelm.compliance import verify_pipeline_manifest_at_path + + violations = verify_pipeline_manifest_at_path(path) + if output_format == "json": + print( + json.dumps( + { + "success": not violations, + "mode": "pipeline", + "path": os.path.abspath(path), + "violations": violations, + }, + indent=2, + ) + ) + else: + if not violations: + print(f"OK: pipeline manifest at {path}") + else: + print(f"FAIL: pipeline manifest at {path}") + for v in violations: + print(f" - {v}") + sys.exit(EXIT_SUCCESS if not violations else EXIT_CONFIG_ERROR) + # Round 6 absorption: the prior `os.path.isfile()` pre-check # collapsed two distinct conditions ("path doesn't exist" and # "path exists but the user can't stat it") into a single diff --git a/forgelm/compliance.py b/forgelm/compliance.py index 873fabe4..39b91c7e 100644 --- a/forgelm/compliance.py +++ b/forgelm/compliance.py @@ -1041,6 +1041,93 @@ def compute_annex_iv_manifest_hash(artifact: Dict[str, Any]) -> str: return _hashlib.sha256(canonical.encode("utf-8")).hexdigest() +# --------------------------------------------------------------------------- +# Phase 14 — Pipeline-level Annex IV manifest +# --------------------------------------------------------------------------- +# +# The canonical ``generate_pipeline_manifest`` + ``export_pipeline_manifest`` +# definitions live further down in this module, alongside the +# ``_provider_metadata_from_config`` helper and ``verify_pipeline_manifest`` +# disk-bound verifier. Only the structural / chain-integrity helper +# (``_verify_manifest_payload``) is declared here; the rest of the +# pipeline-manifest surface clusters near the verifier so reviewers can +# eyeball the schema and its validator together. + + +def _verify_manifest_payload(manifest: Dict[str, Any]) -> List[str]: + """Validate a pipeline manifest's structural + chain-integrity rules. + + Returns a list of human-readable violation strings (empty list ⇒ + manifest is valid). Used by ``forgelm verify-annex-iv --pipeline + `` to surface integrity issues to operators / regulators. + + Checks: + + 1. **Required top-level keys** are present and of the right shape. + 2. **Chain integrity** — for every stage N (N ≥ 1) that completed + successfully, its ``input_model`` must equal stage N-1's + ``output_model``. Mismatches indicate either a CLI + ``--input-model`` override (legitimate) or a corrupt manifest + (illegitimate); the verifier surfaces both with the same + message so reviewers can inspect the audit log to disambiguate. + 3. **Status consistency** — at most one ``stopped_at`` stage; if + set, that stage's status must be one of ``failed`` / + ``gated_pending_approval``. + 4. **Index monotonicity** — stage indices form 0..N-1 in order. + """ + violations: List[str] = [] + + required_top = ( + "forgelm_version", + "pipeline_run_id", + "pipeline_config_hash", + "started_at", + "final_status", + "stages", + ) + for key in required_top: + if key not in manifest: + violations.append(f"missing required top-level key: {key!r}") + + stages = manifest.get("stages", []) + if not isinstance(stages, list): + violations.append("`stages` must be a list") + return violations + + # Index monotonicity. + for expected, s in enumerate(stages): + if s.get("index") != expected: + violations.append(f"stage index out of order at position {expected}: got {s.get('index')!r}") + + # Chain integrity. + prev_output: Optional[str] = None + for s in stages: + status = s.get("status") + # Only completed / failed (with output) stages contribute to the chain. + if status in ("completed", "failed", "gated_pending_approval") and s.get("output_model"): + if prev_output is not None and s.get("input_source") == "chain": + if s.get("input_model") != prev_output: + violations.append( + f"chain_integrity_violation at stage {s.get('name')!r}: input_model={s.get('input_model')!r} " + f"≠ previous output_model={prev_output!r}" + ) + prev_output = s.get("output_model") + + # Status consistency. + stopped_at = manifest.get("stopped_at") + if stopped_at is not None: + matching = [s for s in stages if s.get("name") == stopped_at] + if not matching: + violations.append(f"stopped_at refers to unknown stage {stopped_at!r}") + elif matching[0].get("status") not in ("failed", "gated_pending_approval"): + violations.append( + f"stopped_at stage {stopped_at!r} has status {matching[0].get('status')!r} " + f"(expected `failed` or `gated_pending_approval`)" + ) + + return violations + + # --------------------------------------------------------------------------- # Export: All Compliance Artifacts # --------------------------------------------------------------------------- @@ -1484,3 +1571,174 @@ def verify_audit_log( return manifest_failure return chain_result + + +# --------------------------------------------------------------------------- +# Phase 14 — Pipeline manifest (chain-level Annex IV artefact) +# --------------------------------------------------------------------------- +# +# The pipeline manifest is the *index* over a multi-stage training run. +# Per-stage ``training_manifest.json`` files (produced by +# :func:`generate_training_manifest` + :func:`export_compliance_artifacts`) +# remain individually valid against the existing single-stage Annex IV +# schema; the pipeline manifest ties them together at the chain level so +# auditors can verify both the per-stage evidence AND the chain +# integrity that connects the records. +# +# Lives in ``compliance.py`` (alongside the single-stage manifest) so +# Annex IV schema decisions live in one module. The orchestrator +# imports these functions from here and never touches the schema +# directly. + + +def _provider_metadata_from_config(root_cfg: Any) -> Dict[str, Any]: + """Extract the ``annex_iv`` and ``risk_assessment`` provider metadata + from a root :class:`ForgeConfig`, in the shape the pipeline manifest + embeds verbatim. + + Defensive — both blocks are optional; an absent block produces an + absent key in the returned dict rather than a half-populated record. + """ + payload: Dict[str, Any] = {} + comp_cfg = getattr(root_cfg, "compliance", None) + if comp_cfg: + payload["annex_iv"] = { + "provider_name": comp_cfg.provider_name, + "provider_contact": comp_cfg.provider_contact, + "system_name": comp_cfg.system_name, + "intended_purpose": comp_cfg.intended_purpose, + "known_limitations": comp_cfg.known_limitations, + "system_version": comp_cfg.system_version, + "risk_classification": comp_cfg.risk_classification, + } + risk_cfg = getattr(root_cfg, "risk_assessment", None) + if risk_cfg: + payload["risk_assessment"] = { + "intended_use": risk_cfg.intended_use, + "foreseeable_misuse": risk_cfg.foreseeable_misuse, + "risk_category": risk_cfg.risk_category, + "mitigation_measures": risk_cfg.mitigation_measures, + "vulnerable_groups_considered": risk_cfg.vulnerable_groups_considered, + } + return payload + + +def generate_pipeline_manifest(state: Any, root_cfg: Any) -> Dict[str, Any]: + """Build the chain-level Annex IV manifest. + + Accepts the orchestrator's :class:`PipelineState` (any object + exposing the same field names — duck-typed to avoid a circular + import from :mod:`forgelm.cli._pipeline`) and the root + :class:`ForgeConfig`. Returns a JSON-serialisable dict matching the + schema documented in + ``docs/roadmap/phase-14-pipeline-chains.md`` Task 3. + + The per-stage rows are taken from ``state.stages`` verbatim — the + on-disk state file and the manifest carry the same stage payload, so + a reviewer can correlate the two without translating. Provider / + risk metadata is copied from the root config (immutable across the + pipeline run by the per-stage inheritance matrix). + """ + stages_payload: List[Dict[str, Any]] = [] + for s in state.stages: + stages_payload.append( + { + "name": s.name, + "index": s.index, + "trainer_type": s.trainer_type, + "status": s.status, + "input_model": s.input_model, + "input_source": s.input_source, + "output_model": s.output_model, + "started_at": s.started_at, + "finished_at": s.finished_at, + "duration_seconds": s.duration_seconds, + "training_manifest": s.training_manifest, + "metrics": dict(s.metrics or {}), + "gate_decision": s.gate_decision, + "auto_revert_triggered": bool(s.auto_revert_triggered), + "skipped_reason": s.skipped_reason, + "exit_code": s.exit_code, + "error": getattr(s, "error", None), + } + ) + + manifest: Dict[str, Any] = { + "forgelm_version": state.forgelm_version, + "generated_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "pipeline_run_id": state.pipeline_run_id, + "pipeline_config_hash": state.pipeline_config_hash, + "started_at": state.started_at, + "finished_at": state.finished_at, + "final_status": state.final_status, + "stopped_at": state.stopped_at, + "stages": stages_payload, + } + manifest.update(_provider_metadata_from_config(root_cfg)) + return manifest + + +def export_pipeline_manifest(manifest: Dict[str, Any], pipeline_output_dir: str) -> str: + """Write *manifest* to ``/compliance/pipeline_manifest.json``. + + Atomic write via tmp file + ``os.replace`` so a partial write on + SIGKILL leaves the previous valid manifest intact — the orchestrator + calls this on every stage transition; an interrupted transition must + not corrupt the artefact. + """ + target_dir = os.path.join(pipeline_output_dir, "compliance") + os.makedirs(target_dir, exist_ok=True) + target_path = os.path.join(target_dir, "pipeline_manifest.json") + tmp_path = target_path + ".tmp" + with open(tmp_path, "w", encoding="utf-8") as f: + json.dump(manifest, f, indent=2, default=str) + os.replace(tmp_path, target_path) + return target_path + + +def verify_pipeline_manifest(manifest: Dict[str, Any]) -> List[str]: + """Public Annex IV verifier for a pipeline manifest *dict*. + + Returns a list of human-readable violation strings; an empty list + means the manifest is structurally and chain-consistently valid. + See :func:`_verify_manifest_payload` for the complete check matrix. + + Library callers (test harnesses, internal integrity checks) consume + this directly; the CLI command ``forgelm verify-annex-iv --pipeline + `` reads the file via :func:`verify_pipeline_manifest_at_path`, + which adds the disk-bound checks (per-stage training_manifest + existence) on top. + """ + return _verify_manifest_payload(manifest) + + +def verify_pipeline_manifest_at_path(pipeline_dir: str) -> List[str]: + """Disk-backed wrapper around :func:`verify_pipeline_manifest`. + + Reads ``/compliance/pipeline_manifest.json``, runs the + in-memory verifier on the parsed payload, then layers on disk-only + checks (per-stage ``training_manifest`` file existence). Pre-flight + failures (missing manifest file, malformed JSON) surface as a single- + entry violation list so the CLI's exit-code mapping is uniform. + """ + manifest_path = os.path.join(pipeline_dir, "compliance", "pipeline_manifest.json") + if not os.path.isfile(manifest_path): + return [f"pipeline_manifest.json not found at {manifest_path}"] + try: + with open(manifest_path, "r", encoding="utf-8") as f: + manifest = json.load(f) + except (OSError, json.JSONDecodeError) as e: + return [f"pipeline_manifest.json unreadable: {e}"] + + violations: List[str] = list(_verify_manifest_payload(manifest)) + + # Disk-only check: each completed stage's training_manifest must + # exist. The in-memory verifier cannot see the filesystem. + for s in manifest.get("stages", []): + name = s.get("name", "") + per_stage_manifest = s.get("training_manifest") + if per_stage_manifest and s.get("status") == "completed": + if not os.path.isfile(per_stage_manifest): + violations.append(f"Stage {name!r}: training_manifest at {per_stage_manifest!r} is missing.") + + return violations diff --git a/forgelm/config.py b/forgelm/config.py index 341afe4b..a62d2a04 100644 --- a/forgelm/config.py +++ b/forgelm/config.py @@ -847,6 +847,16 @@ class ForgeConfig(BaseModel): default=None, description="Phase 21 / GDPR Article 5(1)(e) storage limitation + Article 17 erasure horizons block.", ) + pipeline: Optional["PipelineConfig"] = Field( + default=None, + description=( + "Phase 14 — multi-stage training pipeline chain (e.g. SFT → DPO → " + "GRPO). When present, the orchestrator at " + "``forgelm/cli/_pipeline.py`` runs each stage sequentially with " + "section-wholesale inheritance from this root config. When absent, " + "the run path is byte-identical to v0.6.0." + ), + ) def _warn_general_consistency(self) -> None: """Emit warnings for the broad cross-field config inconsistencies.""" @@ -1191,6 +1201,265 @@ class ConfigError(Exception): pass +# --------------------------------------------------------------------------- +# Phase 14 — Multi-Stage Training Pipeline Chains +# --------------------------------------------------------------------------- +# +# A ``pipeline`` block at the root level chains 2+ training stages (e.g. +# SFT → DPO → GRPO) into one config-driven run. See +# ``docs/roadmap/phase-14-pipeline-chains.md`` for the full design spec +# (inheritance matrix, CLI semantics, audit-log events, Annex IV manifest). +# +# Layering rule: the pipeline orchestrator (``forgelm/cli/_pipeline.py``) +# produces one flat :class:`ForgeConfig` per stage via +# :func:`merge_pipeline_stage_config`, then hands that to a fresh +# :class:`ForgeTrainer`. ``forgelm/trainer.py`` is **not** aware of the +# pipeline layer — single-stage runs remain byte-identical to v0.6.0. + + +_STAGE_NAME_PATTERN = r"^[a-z0-9_]{1,32}$" +"""Regex enforcing audit-safe pipeline stage names. + +The name is used in CLI flags (``--stage ``, ``--resume-from ``), +audit-log fields, state-file keys, and webhook payloads — pinning it to a +narrow lowercase / digit / underscore alphabet removes every escaping +concern downstream. +""" + + +class PipelineStage(BaseModel): + """A single stage in a multi-stage training pipeline. + + Section-wholesale override semantics — if a block is present in the + stage YAML it **fully replaces** the root config's block for that + section; if the block is omitted, the stage inherits the root. No + field-level deep-merge: "if you want to inherit, omit the block; if + you want to override, supply the full block." + + Auto-chaining: when the stage does not supply its own ``model:`` block + the orchestrator sets ``model.name_or_path`` to the *previous* + stage's ``training.output_dir/final_model`` path. Stage 0 still reads + the root's ``model.name_or_path``. An explicit per-stage ``model:`` + block (with its required ``name_or_path``) disables auto-chaining for + that stage (operator escape hatch). + + The pipeline-level concerns (``distributed``, ``webhook``, + ``compliance``, ``risk_assessment``, ``monitoring``, ``retention``, + ``synthetic``, ``merge``, ``auth``) live at the root only and have no + per-stage override field — ``extra="forbid"`` causes Pydantic to + reject any attempt to declare them inside a stage, with the + section-name preserved in the error message. + """ + + model_config = ConfigDict(extra="forbid") + + name: str = Field( + ..., + pattern=_STAGE_NAME_PATTERN, + description=( + "Stage identifier. Must match ``[a-z0-9_]{1,32}`` so it can serve as " + "an identifier in CLI flags, audit-log fields, and state-file keys " + "without escaping. Must be unique within the pipeline." + ), + ) + + # Section-wholesale override slots. ``None`` (the default) means + # "inherit this section from the root config"; a populated value + # replaces the root section in full. + model: Optional[ModelConfig] = Field( + default=None, + description=( + "Per-stage model block override. When None, the stage inherits the " + "root ``model`` block AND auto-chains ``model.name_or_path`` to the " + "previous stage's output path." + ), + ) + lora: Optional[LoraConfigModel] = Field( + default=None, + description="Per-stage LoRA / PEFT block override. None → inherit from root.", + ) + training: Optional[TrainingConfig] = Field( + default=None, + description=( + "Per-stage training block override. None → inherit from root. " + "When present, ``trainer_type`` is required (Pydantic's existing " + "``TrainingConfig`` validation enforces this)." + ), + ) + data: Optional[DataConfig] = Field( + default=None, + description=( + "Per-stage data block override. None → inherit from root. Pipelines " + "rarely reuse the same dataset across stages; supplying an explicit " + "per-stage data block is the common case." + ), + ) + evaluation: Optional[EvaluationConfig] = Field( + default=None, + description=( + "Per-stage evaluation block override. None → inherit from root. " + "Per-stage gates (loss thresholds, auto_revert, safety, judge, human-" + "approval) live here; each stage may independently configure its gate." + ), + ) + + @model_validator(mode="after") + def _training_block_must_set_trainer_type_explicitly(self) -> "PipelineStage": + """When a stage supplies its own ``training:`` block, the YAML + MUST explicitly set ``trainer_type`` (each stage states its + alignment paradigm in the YAML — and therefore in the pipeline + manifest — for audit clarity). Without this guard, + ``TrainingConfig.trainer_type`` would silently default to + ``"sft"`` and a DPO stage's manifest could carry + ``trainer_type: sft`` if the operator forgot the field. + """ + if self.training is None: + return self + if "trainer_type" not in self.training.model_fields_set: + raise ValueError( + f"Pipeline stage {self.name!r}: when a 'training:' block is " + f"supplied per stage, 'trainer_type' must be set explicitly " + f"(each stage states its alignment paradigm for audit clarity)." + ) + return self + + +class PipelineConfig(BaseModel): + """Top-level pipeline block. + + Carries the ordered list of stages (≥ 1) that the orchestrator + executes sequentially. The orchestrator is responsible for chaining + each stage's input model to the previous stage's output and emitting + the ``pipeline.*`` audit events documented in the Phase 14 spec. + """ + + model_config = ConfigDict(extra="forbid") + + output_dir: str = Field( + default="./pipeline_run", + description=( + "Pipeline-level output directory. Hosts the pipeline-level audit " + "log, the pipeline state file (``pipeline_state.json``), and the " + "pipeline manifest (``compliance/pipeline_manifest.json``). " + "Distinct from each stage's ``training.output_dir`` — those keep " + "the per-stage checkpoints + per-stage training manifest." + ), + ) + + stages: List[PipelineStage] = Field( + ..., + min_length=1, + description=( + "Ordered list of training stages. Must contain at least one stage; " + "operators who want a single-stage run should omit the ``pipeline:`` " + "block entirely rather than declaring a 1-element pipeline." + ), + ) + + @field_validator("stages") + @classmethod + def _unique_stage_names(cls, v: List[PipelineStage]) -> List[PipelineStage]: + """Reject duplicate stage names early. + + Duplicate names would make ``--stage `` and + ``--resume-from `` ambiguous, and would corrupt the pipeline + state file (which keys per-stage status on the name). + """ + names = [s.name for s in v] + seen = set() + duplicates: List[str] = [] + for n in names: + if n in seen and n not in duplicates: + duplicates.append(n) + seen.add(n) + if duplicates: + raise ValueError( + "Duplicate pipeline stage name(s): " + + ", ".join(repr(d) for d in duplicates) + + ". Stage names must be unique within a pipeline." + ) + return v + + +def merge_pipeline_stage_config( + root_cfg: "ForgeConfig", + stage: PipelineStage, + *, + prev_output_model: Optional[str] = None, + input_model_override: Optional[str] = None, +) -> "ForgeConfig": + """Materialise a flat :class:`ForgeConfig` for a single pipeline stage. + + Implements the section-wholesale inheritance rule documented in + ``docs/roadmap/phase-14-pipeline-chains.md`` Task 2: + + - For each of ``model`` / ``lora`` / ``training`` / ``data`` / + ``evaluation``: if the stage sets the block, the stage's block + replaces the root's; otherwise the root's block is preserved. + - The pipeline-level sections (``distributed``, ``webhook``, + ``compliance``, ``risk_assessment``, ``monitoring``, ``retention``, + ``synthetic``, ``merge``, ``auth``) are kept from the root unchanged + — they cannot be overridden per stage. + + Auto-chain resolution (in priority order): + + 1. ``input_model_override`` (CLI ``--input-model`` flag) — operator + escape hatch. Set ``model.name_or_path`` to this value, regardless + of whether the stage declared its own ``model:`` block. The + caller is responsible for logging the override. + 2. The stage declared its own ``model:`` block — keep the stage's + ``name_or_path``. No auto-chain. + 3. ``prev_output_model`` is not None — auto-chain. Set + ``model.name_or_path`` to this value (typically + ``/final_model``). + 4. Stage 0 with no override — keep the root's ``model.name_or_path``. + + The returned :class:`ForgeConfig` carries no ``pipeline`` block (it + is stripped during the merge) so a downstream :class:`ForgeTrainer` + sees an ordinary single-stage config and behaves byte-identically to + a v0.6.0 single-stage run. + """ + # ``model_dump(exclude_none=True)`` so we don't materialise inherited + # ``Optional[T] = None`` fields the root never set — the + # ``extra="forbid"`` re-validation below would tolerate them, but + # round-tripping no-op None values inflates the per-stage manifest + # and confuses operators reading the merged config. + base = root_cfg.model_dump(exclude_none=True) + base.pop("pipeline", None) + + if stage.model is not None: + base["model"] = stage.model.model_dump(exclude_none=True) + if stage.lora is not None: + base["lora"] = stage.lora.model_dump(exclude_none=True) + if stage.training is not None: + base["training"] = stage.training.model_dump(exclude_none=True) + if stage.data is not None: + base["data"] = stage.data.model_dump(exclude_none=True) + if stage.evaluation is not None: + base["evaluation"] = stage.evaluation.model_dump(exclude_none=True) + + # Auto-chain resolution. See docstring above for priority order. + if input_model_override is not None: + base["model"]["name_or_path"] = input_model_override + elif stage.model is None and prev_output_model is not None: + # Stage inherited the model block AND a previous output exists — + # the canonical auto-chain case. + base["model"]["name_or_path"] = prev_output_model + # else: keep whatever is already in ``base["model"]["name_or_path"]`` + # (either the stage's explicit value or the root's). + + return ForgeConfig(**base) + + +# Resolve the ``pipeline: Optional["PipelineConfig"]`` forward reference in +# :class:`ForgeConfig` now that :class:`PipelineConfig` is in scope. +# Without this, the first ``ForgeConfig(**yaml_data)`` instantiation that +# carries a populated ``pipeline`` block fails at validation time with a +# ``PydanticUndefinedAnnotation`` because the forward reference was never +# late-bound. +ForgeConfig.model_rebuild() + + def load_config(config_path: str) -> ForgeConfig: """Loads and validates a YAML configuration file.""" if not os.path.exists(config_path): diff --git a/forgelm/webhook.py b/forgelm/webhook.py index cfe7c513..1286c096 100644 --- a/forgelm/webhook.py +++ b/forgelm/webhook.py @@ -333,3 +333,105 @@ def notify_reverted(self, run_name: str, reason: str) -> None: color="#ff9900", reason=masked_reason, ) + + # ---------------------------------------------------------------------- + # Phase 14 — pipeline-level notifications + # ---------------------------------------------------------------------- + # + # The pipeline orchestrator drives multi-stage runs and emits its own + # ``pipeline.*`` events alongside (not replacing) the existing per-stage + # ``training.*`` events that each ``ForgeTrainer`` instance still fires. + # Pre-existing Slack / Teams dashboards filtering on ``training.failure`` + # therefore keep working unchanged; pipeline-aware dashboards can + # additionally subscribe to the new ``pipeline.*`` event vocabulary. + + def notify_pipeline_started(self, run_id: str, stage_count: int) -> None: + """Post a "pipeline started" notification. + + Fires once per pipeline run, before any stage executes. Operators + running long chains use this signal to confirm the orchestrator + accepted the config and started the first stage. + """ + if not (self.config and self.config.notify_on_start): + return + self._send( + event="pipeline.started", + run_name=run_id, + status="started", + title=f"Pipeline Started: {run_id}", + text=f"Multi-stage training pipeline began with {stage_count} stage(s).", + color="#0052cc", + stage_count=stage_count, + ) + + def notify_pipeline_completed( + self, + run_id: str, + final_status: str, + stopped_at: Optional[str], + ) -> None: + """Post a "pipeline completed" notification. + + Fires once per pipeline run, after the final stage transition + (success, failure, or revert). ``stopped_at`` names the failing + stage when the chain halted; ``None`` on full success. + + Piggy-backs on ``notify_on_success`` when the pipeline finished + cleanly and on ``notify_on_failure`` when it stopped early — + matches the existing single-stage notification policy so + operators don't see pipeline pings they explicitly silenced. + """ + succeeded = final_status == "completed" + if succeeded: + if not (self.config and self.config.notify_on_success): + return + color = "#36a64f" + title = f"Pipeline Succeeded: {run_id}" + text = "All stages completed successfully." + else: + if not (self.config and self.config.notify_on_failure): + return + color = "#cc0000" + title = f"Pipeline Stopped: {run_id}" + text = f"Pipeline halted at stage {stopped_at!r} with final_status={final_status!r}." + + self._send( + event="pipeline.completed", + run_name=run_id, + status=final_status, + title=title, + text=text, + color=color, + final_status=final_status, + stopped_at=stopped_at, + ) + + def notify_pipeline_reverted(self, run_id: str, stage_name: str, reason: str) -> None: + """Post a "pipeline stage auto-reverted" notification. + + Distinct from ``notify_pipeline_completed(final_status='stopped_at_stage')``: + this fires *at the moment* a stage auto-reverts, before downstream + stages are marked skipped. Operators monitoring a long chain see + the revert event in near-real-time rather than waiting for the + final summary at the end of the run. + + The reason is masked + truncated identically to + :meth:`notify_failure` so a leaked stack trace cannot smuggle + secrets via this path either. + """ + if not (self.config and self.config.notify_on_failure): + return + masked_reason = self._mask_and_truncate_reason(reason) + self._send( + event="pipeline.stage_reverted", + run_name=run_id, + status="reverted", + title=f"Pipeline Stage Reverted: {run_id}", + text=( + f"Stage {stage_name!r} triggered auto-revert; downstream stages " + f"will not run.\n\nReason: {masked_reason}" + ), + color="#ff9900", + stage_name=stage_name, + reason=masked_reason, + ) diff --git a/tests/fixtures/pipeline/auto_revert_at_stage_2.yaml b/tests/fixtures/pipeline/auto_revert_at_stage_2.yaml new file mode 100644 index 00000000..038c50df --- /dev/null +++ b/tests/fixtures/pipeline/auto_revert_at_stage_2.yaml @@ -0,0 +1,43 @@ +# Phase 14 fixture — stage 2 carries an `auto_revert` gate with a +# trivially-impossible `max_acceptable_loss`. In integration tests the +# synthetic eval is wired to exceed that threshold, the trainer +# auto-reverts stage 2's adapters, the orchestrator marks stage 3 +# `skipped_due_to_prior_revert`, and the run exits with +# EXIT_EVAL_FAILURE (3). + +model: + name_or_path: "org/base" +lora: {} +training: + trainer_type: "sft" + output_dir: "./out/revert" +data: + dataset_name_or_path: "org/sft" + +pipeline: + output_dir: "./out/revert" + stages: + - name: sft_stage + training: + trainer_type: "sft" + output_dir: "./out/revert/stage1" + data: + dataset_name_or_path: "org/sft_data" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./out/revert/stage2" + data: + dataset_name_or_path: "org/dpo_data" + evaluation: + auto_revert: true + max_acceptable_loss: 0.001 + baseline_loss: 1.0 + + - name: grpo_stage + training: + trainer_type: "grpo" + output_dir: "./out/revert/stage3" + data: + dataset_name_or_path: "org/math_data" diff --git a/tests/fixtures/pipeline/gated_pending_approval.yaml b/tests/fixtures/pipeline/gated_pending_approval.yaml new file mode 100644 index 00000000..eeec4a39 --- /dev/null +++ b/tests/fixtures/pipeline/gated_pending_approval.yaml @@ -0,0 +1,33 @@ +# Phase 14 fixture — stage 1 carries `require_human_approval: true`. +# Orchestrator must exit EXIT_AWAITING_APPROVAL (4); downstream stages +# must stay `pending` (not `skipped_due_to_prior_revert`) so a +# subsequent `forgelm approve ` + `--resume-from dpo_stage` +# picks the chain back up. + +model: + name_or_path: "org/base" +lora: {} +training: + trainer_type: "sft" + output_dir: "./out/gated" +data: + dataset_name_or_path: "org/sft" + +pipeline: + output_dir: "./out/gated" + stages: + - name: sft_stage + training: + trainer_type: "sft" + output_dir: "./out/gated/stage1" + data: + dataset_name_or_path: "org/sft_data" + evaluation: + require_human_approval: true + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./out/gated/stage2" + data: + dataset_name_or_path: "org/dpo_data" diff --git a/tests/fixtures/pipeline/inheritance_matrix.yaml b/tests/fixtures/pipeline/inheritance_matrix.yaml new file mode 100644 index 00000000..9ee3013b --- /dev/null +++ b/tests/fixtures/pipeline/inheritance_matrix.yaml @@ -0,0 +1,58 @@ +# Phase 14 fixture — exercises every row of the inheritance matrix +# documented in docs/roadmap/phase-14-pipeline-chains.md Task 2: +# auto-chain, section-wholesale override, root-only sections preserved. + +model: + name_or_path: "org/root-base" +lora: + r: 8 + alpha: 16 +training: + trainer_type: "sft" + output_dir: "./out/inheritance" + learning_rate: 2.0e-5 + num_train_epochs: 3 +data: + dataset_name_or_path: "org/root_data" +compliance: + provider_name: "Acme Inc" + provider_contact: "compliance@acme.test" + system_name: "Inheritance Demo" + intended_purpose: "Inheritance-matrix coverage fixture." +webhook: + url: "https://example.test/webhook" + +pipeline: + output_dir: "./out/inheritance" + stages: + # Stage 0 — inherits everything from root; no overrides; reads root model. + - name: stage_inherit_all + + # Stage 1 — overrides lora wholesale; training inherited; auto-chains + # input model to stage_inherit_all's final_model. + - name: stage_lora_override + lora: + r: 32 + alpha: 64 + training: + trainer_type: "sft" + + # Stage 2 — overrides training wholesale (must include trainer_type + # explicitly per Phase 14's audit-clarity validator); overrides data. + - name: stage_training_override + training: + trainer_type: "dpo" + output_dir: "./out/inheritance/stage2" + num_train_epochs: 1 + dpo_beta: 0.1 + data: + dataset_name_or_path: "org/dpo_data" + + # Stage 3 — explicit model block; disables auto-chain; subsequent + # stages still chain from this stage's output. + - name: stage_explicit_model + model: + name_or_path: "meta-llama/Llama-3-8B" + training: + trainer_type: "grpo" + output_dir: "./out/inheritance/stage3" diff --git a/tests/fixtures/pipeline/invalid_distributed_per_stage.yaml b/tests/fixtures/pipeline/invalid_distributed_per_stage.yaml new file mode 100644 index 00000000..14e0a420 --- /dev/null +++ b/tests/fixtures/pipeline/invalid_distributed_per_stage.yaml @@ -0,0 +1,31 @@ +# Phase 14 fixture — invalid per-stage `distributed:` block. +# Phase 14 reserves `distributed`, `webhook`, `compliance`, +# `risk_assessment`, `monitoring`, `retention`, `synthetic`, `merge`, +# and `auth` for the root level only — `PipelineStage`'s +# ``extra="forbid"`` rejects them with the offending section name in +# the Pydantic error. Config load on this fixture must raise +# `ConfigError` (mapped to EXIT_CONFIG_ERROR=1 at the CLI surface). + +model: + name_or_path: "org/base" +lora: {} +training: + trainer_type: "sft" + output_dir: "./out/bad" +data: + dataset_name_or_path: "org/sft" + +pipeline: + output_dir: "./out/bad" + stages: + - name: sft_stage + training: + trainer_type: "sft" + output_dir: "./out/bad/stage1" + data: + dataset_name_or_path: "org/sft_data" + # The next key is the violation — distributed config is + # pipeline-level only. Pydantic rejects with the key name. + distributed: + strategy: "deepspeed" + config_path: "./deepspeed_config.json" diff --git a/tests/fixtures/pipeline/minimal_3_stage.yaml b/tests/fixtures/pipeline/minimal_3_stage.yaml new file mode 100644 index 00000000..bd215afb --- /dev/null +++ b/tests/fixtures/pipeline/minimal_3_stage.yaml @@ -0,0 +1,42 @@ +# Phase 14 fixture — minimal 3-stage SFT → DPO → GRPO pipeline. +# Used as the happy-path smoke baseline for parser + orchestrator tests. + +model: + name_or_path: "org/base-model" +lora: + r: 8 + alpha: 16 +training: + trainer_type: "sft" + output_dir: "./out/pipeline_smoke" + num_train_epochs: 1 +data: + dataset_name_or_path: "org/sft_dataset" + +pipeline: + output_dir: "./out/pipeline_smoke" + stages: + - name: sft_stage + training: + trainer_type: "sft" + output_dir: "./out/pipeline_smoke/stage1_sft" + num_train_epochs: 3 + data: + dataset_name_or_path: "org/sft_dataset" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./out/pipeline_smoke/stage2_dpo" + num_train_epochs: 1 + dpo_beta: 0.1 + data: + dataset_name_or_path: "org/dpo_preferences" + + - name: grpo_stage + training: + trainer_type: "grpo" + output_dir: "./out/pipeline_smoke/stage3_grpo" + grpo_num_generations: 4 + data: + dataset_name_or_path: "org/math_prompts" diff --git a/tests/fixtures/pipeline/stale_state_resume_v1.yaml b/tests/fixtures/pipeline/stale_state_resume_v1.yaml new file mode 100644 index 00000000..fff21fc3 --- /dev/null +++ b/tests/fixtures/pipeline/stale_state_resume_v1.yaml @@ -0,0 +1,33 @@ +# Phase 14 fixture (v1) — used to seed a `pipeline_state.json` whose +# `pipeline_config_hash` was computed against this exact byte sequence. +# Paired with `stale_state_resume_v2.yaml`: re-running with v2 against +# the v1 state file must trigger the stale-config guard (config-hash +# mismatch) unless `--force-resume` is set. + +model: + name_or_path: "org/base" +lora: {} +training: + trainer_type: "sft" + output_dir: "./out/stale" + learning_rate: 2.0e-5 +data: + dataset_name_or_path: "org/sft" + +pipeline: + output_dir: "./out/stale" + stages: + - name: sft_stage + training: + trainer_type: "sft" + output_dir: "./out/stale/stage1" + learning_rate: 2.0e-5 + data: + dataset_name_or_path: "org/sft_data" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./out/stale/stage2" + data: + dataset_name_or_path: "org/dpo_data" diff --git a/tests/fixtures/pipeline/stale_state_resume_v2.yaml b/tests/fixtures/pipeline/stale_state_resume_v2.yaml new file mode 100644 index 00000000..1fdb2398 --- /dev/null +++ b/tests/fixtures/pipeline/stale_state_resume_v2.yaml @@ -0,0 +1,34 @@ +# Phase 14 fixture (v2) — paired with `stale_state_resume_v1.yaml` to +# demonstrate the stale-state guard. Differs from v1 in one numeric +# value (learning_rate); the bytes are different so the SHA-256 over +# the file content differs. Re-running with this file against a state +# file produced by v1 must surface the `pipeline_config_hash mismatch` +# refusal. + +model: + name_or_path: "org/base" +lora: {} +training: + trainer_type: "sft" + output_dir: "./out/stale" + learning_rate: 1.0e-4 # ← differs from v1 (2.0e-5) +data: + dataset_name_or_path: "org/sft" + +pipeline: + output_dir: "./out/stale" + stages: + - name: sft_stage + training: + trainer_type: "sft" + output_dir: "./out/stale/stage1" + learning_rate: 1.0e-4 # ← differs from v1 + data: + dataset_name_or_path: "org/sft_data" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./out/stale/stage2" + data: + dataset_name_or_path: "org/dpo_data" diff --git a/tests/test_pipeline_cli.py b/tests/test_pipeline_cli.py new file mode 100644 index 00000000..e31c1aba --- /dev/null +++ b/tests/test_pipeline_cli.py @@ -0,0 +1,163 @@ +"""Phase 14 — CLI flag interaction + dispatcher tests. + +Covers :func:`forgelm.cli._pipeline.run_pipeline_from_args`, the argparse +glue layer between the top-level parser and +:class:`forgelm.cli._pipeline.PipelineOrchestrator`. Focus: flag +combinatorial rules + dispatch routing. The orchestrator's own state +machine has dedicated coverage in ``tests/test_pipeline_orchestrator.py``; +this file pins the *boundary* between argparse and the orchestrator. +""" + +from __future__ import annotations + +import argparse +import os +import types +from unittest.mock import MagicMock + +from forgelm.cli._exit_codes import EXIT_CONFIG_ERROR, EXIT_SUCCESS +from forgelm.cli._pipeline import run_pipeline_from_args +from forgelm.config import ForgeConfig +from forgelm.results import TrainResult + + +def _three_stage_cfg(tmp_path): + return ForgeConfig( + model={"name_or_path": "org/base"}, + lora={"r": 8}, + training={"trainer_type": "sft"}, + data={"dataset_name_or_path": "org/data"}, + pipeline={ + "output_dir": str(tmp_path / "pipeline_run"), + "stages": [ + { + "name": "sft_stage", + "training": {"trainer_type": "sft", "output_dir": str(tmp_path / "stage1")}, + "data": {"dataset_name_or_path": "org/sft"}, + }, + { + "name": "dpo_stage", + "training": {"trainer_type": "dpo", "output_dir": str(tmp_path / "stage2")}, + "data": {"dataset_name_or_path": "org/dpo"}, + }, + { + "name": "grpo_stage", + "training": {"trainer_type": "grpo", "output_dir": str(tmp_path / "stage3")}, + "data": {"dataset_name_or_path": "org/math"}, + }, + ], + }, + ) + + +def _ns(**overrides): + """Build an argparse.Namespace with the orchestrator-relevant fields.""" + defaults = dict( + stage=None, + resume_from=None, + force_resume=False, + input_model=None, + output_format="text", + dry_run=False, + ) + defaults.update(overrides) + return argparse.Namespace(**defaults) + + +def _install_fake_trainer(monkeypatch, results): + iterator = iter(results) + + class _FakeForgeTrainer: + def __init__(self, *, model, tokenizer, config, dataset): + os.makedirs(os.path.join(config.training.output_dir, "final_model"), exist_ok=True) + self.config = config + + def train(self, resume_from_checkpoint=None): + try: + return next(iterator) + except StopIteration: + return TrainResult(success=True) + + fake_trainer = types.ModuleType("forgelm.trainer") + fake_trainer.ForgeTrainer = _FakeForgeTrainer + monkeypatch.setitem(__import__("sys").modules, "forgelm.trainer", fake_trainer) + + fake_model = types.ModuleType("forgelm.model") + fake_model.get_model_and_tokenizer = lambda config: (MagicMock(), MagicMock()) + monkeypatch.setitem(__import__("sys").modules, "forgelm.model", fake_model) + + fake_data = types.ModuleType("forgelm.data") + fake_data.prepare_dataset = lambda config, tokenizer: {"train": [{"text": "x"}]} + monkeypatch.setitem(__import__("sys").modules, "forgelm.data", fake_data) + + fake_utils = types.ModuleType("forgelm.utils") + fake_utils.setup_authentication = lambda token: None + monkeypatch.setitem(__import__("sys").modules, "forgelm.utils", fake_utils) + + +class TestFlagInteractionGuards: + """Argparse alone cannot express "X and Y are mutually exclusive + *only when both are non-None*"; the dispatcher enforces it.""" + + def test_stage_and_resume_from_mutually_exclusive(self, tmp_path): + cfg = _three_stage_cfg(tmp_path) + args = _ns(stage="sft_stage", resume_from="dpo_stage") + code = run_pipeline_from_args(cfg, b"yaml", args) + assert code == EXIT_CONFIG_ERROR + + def test_input_model_requires_stage(self, tmp_path): + cfg = _three_stage_cfg(tmp_path) + args = _ns(input_model="some/path") + code = run_pipeline_from_args(cfg, b"yaml", args) + assert code == EXIT_CONFIG_ERROR + + def test_force_resume_without_resume_from_is_noop(self, tmp_path, monkeypatch): + """``--force-resume`` without ``--resume-from`` is meaningless + but not an error — the orchestrator simply ignores it on a + fresh run. We pin this so a future overzealous validator + doesn't start rejecting harmless flag combinations.""" + cfg = _three_stage_cfg(tmp_path) + _install_fake_trainer( + monkeypatch, [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)] + ) + args = _ns(force_resume=True) + code = run_pipeline_from_args(cfg, b"yaml", args) + assert code == EXIT_SUCCESS + + +class TestDispatchToDryRun: + def test_dry_run_routes_to_orchestrator_dry_run(self, tmp_path, monkeypatch): + """``--dry-run`` must take the dry-run branch even when other + pipeline flags are set — keeps operators from accidentally + kicking off a real run when they intended to validate.""" + cfg = _three_stage_cfg(tmp_path) + _install_fake_trainer(monkeypatch, []) + args = _ns(dry_run=True) + code = run_pipeline_from_args(cfg, b"yaml", args) + assert code == EXIT_SUCCESS + # No state / manifest written by dry-run. + assert not os.path.exists(tmp_path / "pipeline_run" / "pipeline_state.json") + + +class TestDispatchToRun: + def test_full_run_routes_to_orchestrator_run(self, tmp_path, monkeypatch): + cfg = _three_stage_cfg(tmp_path) + _install_fake_trainer(monkeypatch, [TrainResult(success=True)] * 3) + args = _ns() + code = run_pipeline_from_args(cfg, b"yaml", args) + assert code == EXIT_SUCCESS + assert os.path.exists(tmp_path / "pipeline_run" / "pipeline_state.json") + + def test_stage_filter_propagates_to_orchestrator(self, tmp_path, monkeypatch): + cfg = _three_stage_cfg(tmp_path) + _install_fake_trainer(monkeypatch, [TrainResult(success=True)]) + args = _ns(stage="sft_stage") + code = run_pipeline_from_args(cfg, b"yaml", args) + assert code == EXIT_SUCCESS + + def test_unknown_stage_filter_rejected_via_orchestrator(self, tmp_path, monkeypatch): + cfg = _three_stage_cfg(tmp_path) + _install_fake_trainer(monkeypatch, []) + args = _ns(stage="ghost") + code = run_pipeline_from_args(cfg, b"yaml", args) + assert code == EXIT_CONFIG_ERROR diff --git a/tests/test_pipeline_compliance.py b/tests/test_pipeline_compliance.py new file mode 100644 index 00000000..d25155f4 --- /dev/null +++ b/tests/test_pipeline_compliance.py @@ -0,0 +1,297 @@ +"""Phase 14 — Pipeline manifest schema + chain-integrity verifier tests. + +Covers :func:`forgelm.compliance.generate_pipeline_manifest` and +:func:`forgelm.compliance.verify_pipeline_manifest`. The pipeline +manifest is the EU AI Act Annex IV chain-of-custody artefact that ties +the per-stage ``training_manifest.json`` files into one verifiable +provenance index. +""" + +from __future__ import annotations + +from forgelm.cli._pipeline import PipelineStageState, PipelineState +from forgelm.compliance import _verify_manifest_payload, generate_pipeline_manifest +from forgelm.config import ForgeConfig + + +def _root_with_compliance() -> ForgeConfig: + return ForgeConfig( + model={"name_or_path": "org/base"}, + lora={"r": 8}, + training={"trainer_type": "sft"}, + data={"dataset_name_or_path": "org/data"}, + compliance={ + "provider_name": "Acme Inc", + "provider_contact": "compliance@acme.test", + "system_name": "Acme Pipeline System", + "intended_purpose": "Customer-service assistant fine-tune", + "system_version": "v0.7.0", + }, + pipeline={ + "stages": [{"name": "sft_stage"}, {"name": "dpo_stage"}, {"name": "grpo_stage"}], + }, + ) + + +def _three_stage_state(*, all_completed: bool = True) -> PipelineState: + """Build a representative 3-stage state for happy-path schema tests.""" + s1 = PipelineStageState( + name="sft_stage", + index=0, + trainer_type="sft", + status="completed" if all_completed else "completed", + input_model="org/base", + input_source="root", + output_model="./out/stage1/final_model", + started_at="2026-06-15T12:00:00+00:00", + finished_at="2026-06-15T13:00:00+00:00", + duration_seconds=3600.0, + metrics={"eval_loss": 0.5}, + gate_decision="passed", + exit_code=0, + ) + s2 = PipelineStageState( + name="dpo_stage", + index=1, + trainer_type="dpo", + status="completed", + input_model="./out/stage1/final_model", + input_source="chain", + output_model="./out/stage2/final_model", + started_at="2026-06-15T13:00:00+00:00", + finished_at="2026-06-15T13:45:00+00:00", + duration_seconds=2700.0, + metrics={"eval_loss": 0.3}, + gate_decision="passed", + exit_code=0, + ) + s3 = PipelineStageState( + name="grpo_stage", + index=2, + trainer_type="grpo", + status="completed", + input_model="./out/stage2/final_model", + input_source="chain", + output_model="./out/stage3/final_model", + started_at="2026-06-15T13:45:00+00:00", + finished_at="2026-06-15T14:30:00+00:00", + duration_seconds=2700.0, + metrics={"eval_loss": 0.2}, + gate_decision="passed", + exit_code=0, + ) + return PipelineState( + pipeline_run_id="pl_2026-06-15_a1b2c3", + pipeline_config_hash="sha256:abc", + forgelm_version="0.7.0", + started_at="2026-06-15T12:00:00+00:00", + finished_at="2026-06-15T14:30:00+00:00", + final_status="completed", + stages=[s1, s2, s3], + ) + + +# --------------------------------------------------------------------------- +# generate_pipeline_manifest — schema coverage +# --------------------------------------------------------------------------- + + +class TestManifestSchema: + def test_required_top_level_keys_present(self): + manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance()) + required = { + "forgelm_version", + "generated_at", + "pipeline_run_id", + "pipeline_config_hash", + "started_at", + "finished_at", + "final_status", + "stages", + } + assert required.issubset(manifest.keys()) + + def test_annex_iv_block_propagated_from_root_compliance(self): + manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance()) + assert "annex_iv" in manifest + assert manifest["annex_iv"]["provider_name"] == "Acme Inc" + assert manifest["annex_iv"]["system_name"] == "Acme Pipeline System" + + def test_annex_iv_omitted_when_no_compliance_block(self): + root = ForgeConfig( + model={"name_or_path": "x"}, + lora={}, + training={"trainer_type": "sft"}, + data={"dataset_name_or_path": "y"}, + pipeline={"stages": [{"name": "s1"}]}, + ) + manifest = generate_pipeline_manifest(_three_stage_state(), root) + assert "annex_iv" not in manifest + + def test_stage_payload_carries_chain_fields(self): + manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance()) + s1, s2, s3 = manifest["stages"] + assert s1["index"] == 0 and s2["index"] == 1 and s3["index"] == 2 + assert s2["input_model"] == s1["output_model"] + assert s3["input_model"] == s2["output_model"] + assert all(s["gate_decision"] == "passed" for s in (s1, s2, s3)) + + def test_stage_metrics_are_a_plain_dict_in_payload(self): + """Manifest must be JSON-serialisable; metrics dict must round- + trip through ``json.dumps``.""" + import json as _json + + manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance()) + # No round-trip failure. + _json.dumps(manifest) + + +# --------------------------------------------------------------------------- +# verify_pipeline_manifest — chain integrity +# --------------------------------------------------------------------------- + + +class TestManifestVerification: + def test_clean_manifest_passes(self): + manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance()) + assert _verify_manifest_payload(manifest) == [] + + def test_missing_required_key_flagged(self): + manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance()) + manifest.pop("pipeline_run_id") + violations = _verify_manifest_payload(manifest) + assert any("pipeline_run_id" in v for v in violations) + + def test_chain_integrity_violation_flagged(self): + """Stage 2's input_model ≠ stage 1's output_model on a chain + stage must surface a ``chain_integrity_violation``.""" + state = _three_stage_state() + state.stages[1].input_model = "tampered/value" + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + assert any("chain_integrity_violation" in v for v in violations) + assert any("tampered/value" in v for v in violations) + + def test_cli_override_chain_break_still_flagged(self): + """Even when ``input_source: cli_override`` legitimately breaks + the chain, the verifier still surfaces it (with the same message) + so reviewers can correlate against the audit log to decide + legitimate vs. corrupt. + + Note: only *chain* stages contribute to the integrity check; an + explicit cli_override stage is recorded with ``input_source != + "chain"`` and therefore skipped by design. This test asserts + that contract.""" + state = _three_stage_state() + state.stages[1].input_model = "operator/manual" + state.stages[1].input_source = "cli_override" + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + # cli_override stages don't trip the chain check. + assert all("chain_integrity_violation" not in v for v in violations) + + def test_index_out_of_order_flagged(self): + state = _three_stage_state() + state.stages[0], state.stages[1] = state.stages[1], state.stages[0] + # Indices stay 0/1 but names are swapped — verifier checks index + # vs. positional order. + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + assert any("index out of order" in v for v in violations) + + def test_stopped_at_unknown_stage_flagged(self): + state = _three_stage_state() + state.stopped_at = "ghost_stage" + state.final_status = "stopped_at_stage" + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + assert any("unknown stage" in v and "ghost_stage" in v for v in violations) + + def test_stopped_at_completed_stage_flagged(self): + """If ``stopped_at`` points at a stage whose status is + ``completed`` rather than ``failed`` / ``gated_pending_approval``, + the manifest is internally inconsistent.""" + state = _three_stage_state() + state.stopped_at = "sft_stage" # which has status=completed + state.final_status = "stopped_at_stage" + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + assert any("expected `failed` or `gated_pending_approval`" in v for v in violations) + + +class TestVerifyOnAutoRevertScenario: + """End-to-end: build a state where stage 2 failed and stage 3 was + skipped; the resulting manifest should verify cleanly (it's a valid + record of a real failure, not an integrity violation).""" + + def test_auto_revert_manifest_verifies(self): + s1 = PipelineStageState( + name="sft_stage", + index=0, + trainer_type="sft", + status="completed", + input_model="org/base", + input_source="root", + output_model="./out/stage1/final_model", + exit_code=0, + ) + s2 = PipelineStageState( + name="dpo_stage", + index=1, + trainer_type="dpo", + status="failed", + input_model="./out/stage1/final_model", + input_source="chain", + output_model="./out/stage2/final_model", + auto_revert_triggered=True, + exit_code=3, + error="loss regression", + ) + s3 = PipelineStageState( + name="grpo_stage", + index=2, + trainer_type="grpo", + status="skipped_due_to_prior_revert", + skipped_reason="Stage 'dpo_stage' triggered auto_revert.", + ) + state = PipelineState( + pipeline_run_id="pl_x", + pipeline_config_hash="sha256:abc", + forgelm_version="0.7.0", + started_at="2026-06-15T12:00:00+00:00", + finished_at="2026-06-15T13:30:00+00:00", + final_status="stopped_at_stage", + stopped_at="dpo_stage", + stages=[s1, s2, s3], + ) + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + assert _verify_manifest_payload(manifest) == [] + + +class TestVerifyOnPartialFilterRun: + """A ``--stage X`` run produces a manifest where other stages have + ``status: skipped_by_filter``; verifier should accept it.""" + + def test_partial_filter_manifest_verifies(self): + s1 = PipelineStageState(name="s1", index=0, trainer_type="sft", status="skipped_by_filter") + s2 = PipelineStageState( + name="s2", + index=1, + trainer_type="dpo", + status="completed", + input_model="./prev/output", + input_source="cli_override", + output_model="./out/stage2/final_model", + exit_code=0, + ) + s3 = PipelineStageState(name="s3", index=2, trainer_type="grpo", status="skipped_by_filter") + state = PipelineState( + pipeline_run_id="pl_x", + pipeline_config_hash="sha256:abc", + forgelm_version="0.7.0", + started_at="2026-06-15T12:00:00+00:00", + final_status="completed", + stages=[s1, s2, s3], + ) + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + assert _verify_manifest_payload(manifest) == [] diff --git a/tests/test_pipeline_config.py b/tests/test_pipeline_config.py new file mode 100644 index 00000000..63575572 --- /dev/null +++ b/tests/test_pipeline_config.py @@ -0,0 +1,369 @@ +"""Phase 14 — Pydantic schema + inheritance-merge tests. + +Covers: + +- :class:`forgelm.config.PipelineStage` field validation (name pattern, + ``extra="forbid"`` rejecting pipeline-only sections). +- :class:`forgelm.config.PipelineConfig` (minimum-1-stage, unique-name + validator). +- :func:`forgelm.config.merge_pipeline_stage_config` — the section- + wholesale inheritance rule + auto-chain priority order documented in + ``docs/roadmap/phase-14-pipeline-chains.md`` Task 2. +- Backward compatibility: a config without a ``pipeline:`` block produces + ``config.pipeline is None``; an existing single-stage config is + byte-identical to v0.6.0 after the schema change. +""" + +from __future__ import annotations + +import pytest +from pydantic import ValidationError + +from forgelm.config import ( + ForgeConfig, + PipelineConfig, + PipelineStage, + merge_pipeline_stage_config, +) + + +def _root_cfg(**overrides): + """Build a minimal valid root ForgeConfig with sensible defaults.""" + base = { + "model": {"name_or_path": "org/base"}, + "lora": {"r": 8, "alpha": 16}, + "training": {"trainer_type": "sft", "num_train_epochs": 3, "learning_rate": 2e-5}, + "data": {"dataset_name_or_path": "org/sft_data"}, + } + base.update(overrides) + return ForgeConfig(**base) + + +# --------------------------------------------------------------------------- +# PipelineStage — name + per-section validation +# --------------------------------------------------------------------------- + + +class TestPipelineStageName: + @pytest.mark.parametrize( + "name", + ["sft_stage", "stage_1", "s", "a" * 32, "abc_def_123"], + ) + def test_valid_names_accepted(self, name): + stage = PipelineStage(name=name) + assert stage.name == name + + @pytest.mark.parametrize( + "name", + [ + "", # empty + "a" * 33, # > 32 chars + "Stage1", # uppercase + "stage-1", # hyphen + "stage 1", # space + "stage.1", # dot + "stage/1", # slash + "stage@1", # at-sign + "stage_!", # punctuation + ], + ) + def test_invalid_names_rejected(self, name): + with pytest.raises(ValidationError): + PipelineStage(name=name) + + def test_name_is_required(self): + with pytest.raises(ValidationError): + PipelineStage() + + +class TestPipelineStageExtraForbid: + """Pipeline-only sections (distributed / webhook / compliance / etc.) + must not appear inside a stage. ``extra="forbid"`` makes Pydantic + reject them with the offending field name in the error. This is the + primary defence against operators putting root-only config inside a + stage by mistake. + """ + + @pytest.mark.parametrize( + "forbidden_section", + [ + "distributed", + "webhook", + "compliance", + "risk_assessment", + "monitoring", + "retention", + "synthetic", + "merge", + "auth", + "pipeline", # no nested pipelines + ], + ) + def test_pipeline_only_section_rejected_in_stage(self, forbidden_section): + with pytest.raises(ValidationError) as exc_info: + PipelineStage(name="s1", **{forbidden_section: {}}) + # The forbidden section name appears in the error so the operator + # knows which key to remove. + assert forbidden_section in str(exc_info.value) + + +class TestPipelineStageOverrides: + """All allowed override slots accept their corresponding config block.""" + + def test_all_override_slots_default_to_none(self): + stage = PipelineStage(name="s1") + assert stage.model is None + assert stage.lora is None + assert stage.training is None + assert stage.data is None + assert stage.evaluation is None + + def test_model_block_override(self): + stage = PipelineStage(name="s1", model={"name_or_path": "org/other"}) + assert stage.model is not None + assert stage.model.name_or_path == "org/other" + + def test_lora_block_override(self): + stage = PipelineStage(name="s1", lora={"r": 32, "alpha": 64}) + assert stage.lora is not None + assert stage.lora.r == 32 + + def test_training_block_override_requires_trainer_type(self): + """``trainer_type`` is required by the existing ``TrainingConfig`` + schema; a stage's training block must therefore supply it. This + is the Phase 14 spec's "each stage explicitly states its + alignment paradigm" rule, enforced via Pydantic's existing + validator rather than a duplicate check.""" + with pytest.raises(ValidationError): + PipelineStage(name="s1", training={"num_train_epochs": 1}) + + def test_data_block_override(self): + stage = PipelineStage(name="s1", data={"dataset_name_or_path": "org/dpo_prefs"}) + assert stage.data is not None + assert stage.data.dataset_name_or_path == "org/dpo_prefs" + + def test_evaluation_block_override(self): + stage = PipelineStage( + name="s1", + evaluation={"auto_revert": True, "max_acceptable_loss": 2.0}, + ) + assert stage.evaluation is not None + assert stage.evaluation.auto_revert is True + + +# --------------------------------------------------------------------------- +# PipelineConfig — list validators +# --------------------------------------------------------------------------- + + +class TestPipelineConfig: + def test_minimum_one_stage_required(self): + with pytest.raises(ValidationError): + PipelineConfig(stages=[]) + + def test_single_stage_pipeline_accepted(self): + """A 1-stage pipeline is technically valid (the spec only forbids + empty pipelines). Whether operators *should* declare one is a + documentation matter, not a schema matter.""" + pl = PipelineConfig(stages=[PipelineStage(name="only")]) + assert len(pl.stages) == 1 + + def test_multi_stage_pipeline(self): + pl = PipelineConfig( + stages=[ + PipelineStage(name="sft_stage"), + PipelineStage(name="dpo_stage"), + PipelineStage(name="grpo_stage"), + ] + ) + assert [s.name for s in pl.stages] == ["sft_stage", "dpo_stage", "grpo_stage"] + + def test_duplicate_stage_names_rejected(self): + with pytest.raises(ValidationError) as exc_info: + PipelineConfig( + stages=[ + PipelineStage(name="dup"), + PipelineStage(name="other"), + PipelineStage(name="dup"), + ] + ) + assert "Duplicate" in str(exc_info.value) + assert "dup" in str(exc_info.value) + + def test_extra_keys_rejected(self): + """``extra="forbid"`` blocks typos like ``stagess`` from being + silently accepted as ignored fields.""" + with pytest.raises(ValidationError): + PipelineConfig(stages=[PipelineStage(name="s1")], stagess=[]) + + +# --------------------------------------------------------------------------- +# ForgeConfig.pipeline wiring + backward compatibility +# --------------------------------------------------------------------------- + + +class TestForgeConfigPipelineField: + def test_pipeline_defaults_to_none(self): + cfg = _root_cfg() + assert cfg.pipeline is None + + def test_pipeline_populated_from_yaml_dict(self): + cfg = _root_cfg(pipeline={"stages": [{"name": "s1"}, {"name": "s2"}]}) + assert cfg.pipeline is not None + assert len(cfg.pipeline.stages) == 2 + + def test_pipeline_section_round_trips_through_model_dump(self): + cfg = _root_cfg(pipeline={"stages": [{"name": "s1", "training": {"trainer_type": "dpo"}}]}) + dumped = cfg.model_dump(exclude_none=True) + assert "pipeline" in dumped + assert dumped["pipeline"]["stages"][0]["name"] == "s1" + + def test_single_stage_config_byte_identical_without_pipeline(self): + """A pre-Phase-14 single-stage config (no ``pipeline:`` key) + must produce a ``ForgeConfig`` indistinguishable from v0.6.0 + for the trainer's purposes — the ``pipeline`` field defaults to + None and is excluded from ``model_dump(exclude_none=True)``.""" + cfg = _root_cfg() + dumped = cfg.model_dump(exclude_none=True) + assert "pipeline" not in dumped + + +# --------------------------------------------------------------------------- +# merge_pipeline_stage_config — section-wholesale + auto-chain priority +# --------------------------------------------------------------------------- + + +class TestMergeSectionWholesale: + def test_stage_with_no_overrides_inherits_root_entirely(self): + root = _root_cfg() + stage = PipelineStage(name="s0") + merged = merge_pipeline_stage_config(root, stage, prev_output_model=None) + assert merged.model.name_or_path == root.model.name_or_path + assert merged.lora.r == root.lora.r + assert merged.training.trainer_type == root.training.trainer_type + assert merged.training.num_train_epochs == root.training.num_train_epochs + assert merged.data.dataset_name_or_path == root.data.dataset_name_or_path + + def test_stage_lora_block_wholesale_replaces_root(self): + root = _root_cfg() + stage = PipelineStage(name="s1", lora={"r": 64, "alpha": 128}) + merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model") + # The stage's lora block fully replaces — fields the stage didn't + # mention fall back to ``LoraConfigModel`` defaults, NOT to the + # root's ``lora`` block's values. + assert merged.lora.r == 64 + assert merged.lora.alpha == 128 + + def test_stage_training_block_wholesale_replaces_root(self): + root = _root_cfg() + stage = PipelineStage(name="s1", training={"trainer_type": "dpo", "num_train_epochs": 1}) + merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model") + assert merged.training.trainer_type == "dpo" + assert merged.training.num_train_epochs == 1 + + def test_stage_data_block_wholesale_replaces_root(self): + root = _root_cfg() + stage = PipelineStage(name="s1", data={"dataset_name_or_path": "org/dpo_prefs"}) + merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model") + assert merged.data.dataset_name_or_path == "org/dpo_prefs" + + def test_stage_evaluation_block_wholesale_replaces_root(self): + root = _root_cfg(evaluation={"auto_revert": False}) + stage = PipelineStage(name="s1", evaluation={"auto_revert": True, "max_acceptable_loss": 1.5}) + merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model") + assert merged.evaluation is not None + assert merged.evaluation.auto_revert is True + assert merged.evaluation.max_acceptable_loss == 1.5 + + def test_pipeline_section_stripped_from_merged_config(self): + """The orchestrator hands the merged ForgeConfig to a single-stage + ``ForgeTrainer`` that has no awareness of pipelines. The + ``pipeline`` block must be absent from the merged config so the + trainer's lifecycle is byte-identical to a v0.6.0 single-stage + run.""" + root = _root_cfg(pipeline={"stages": [{"name": "s0"}, {"name": "s1"}]}) + stage = PipelineStage(name="s0") + merged = merge_pipeline_stage_config(root, stage, prev_output_model=None) + assert merged.pipeline is None + + +class TestMergeAutoChainPriorityOrder: + """The auto-chain resolution rule has four priority levels. These + tests pin every level so a future refactor cannot silently change + the order — operators rely on the documented behaviour for + ``--input-model`` to actually override an in-config ``model:`` block. + """ + + def test_priority_1_input_model_override_wins_over_everything(self): + root = _root_cfg() + stage = PipelineStage(name="s1", model={"name_or_path": "stage/value"}) + merged = merge_pipeline_stage_config( + root, + stage, + prev_output_model="./prev/model", + input_model_override="cli/override", + ) + assert merged.model.name_or_path == "cli/override" + + def test_priority_2_explicit_stage_model_disables_auto_chain(self): + root = _root_cfg() + stage = PipelineStage(name="s1", model={"name_or_path": "stage/explicit"}) + merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model") + assert merged.model.name_or_path == "stage/explicit" + + def test_priority_3_auto_chain_to_prev_output_when_no_overrides(self): + root = _root_cfg() + stage = PipelineStage(name="s1") + merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/final") + assert merged.model.name_or_path == "./prev/final" + + def test_priority_4_stage_zero_inherits_root_model(self): + """Stage 0 of a pipeline (or any stage launched standalone with + ``--stage `` without a prior output) reads the root's + ``model.name_or_path`` unchanged.""" + root = _root_cfg() + stage = PipelineStage(name="s0") + merged = merge_pipeline_stage_config(root, stage, prev_output_model=None) + assert merged.model.name_or_path == root.model.name_or_path + + def test_input_model_override_beats_stage_zero_root_value(self): + """The CLI escape hatch (``--input-model``) must work even on the + first stage — operators using ``--stage + --input-model `` to re-run with a different base model.""" + root = _root_cfg() + stage = PipelineStage(name="s0") + merged = merge_pipeline_stage_config( + root, + stage, + prev_output_model=None, + input_model_override="cli/override", + ) + assert merged.model.name_or_path == "cli/override" + + +class TestMergePreservesRootOnlyBlocks: + """The pipeline-level config sections (distributed, webhook, + compliance, etc.) cannot be overridden per stage by design. After + merge, those blocks must come through from the root verbatim. + """ + + def test_root_webhook_survives_merge(self): + root = _root_cfg(webhook={"url": "https://example.com/hook"}) + stage = PipelineStage(name="s1", training={"trainer_type": "dpo"}) + merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model") + assert merged.webhook is not None + assert merged.webhook.url == "https://example.com/hook" + + def test_root_compliance_metadata_survives_merge(self): + root = _root_cfg( + compliance={ + "provider_name": "Acme Corp", + "provider_contact": "compliance@acme.test", + "system_name": "Pipeline Demo", + "intended_purpose": "Internal eval", + } + ) + stage = PipelineStage(name="s1", training={"trainer_type": "dpo"}) + merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model") + assert merged.compliance is not None + assert merged.compliance.provider_name == "Acme Corp" diff --git a/tests/test_pipeline_orchestrator.py b/tests/test_pipeline_orchestrator.py new file mode 100644 index 00000000..18110b89 --- /dev/null +++ b/tests/test_pipeline_orchestrator.py @@ -0,0 +1,523 @@ +"""Phase 14 — Pipeline orchestrator state-machine tests. + +Covers the documented edge cases in ``docs/roadmap/phase-14-pipeline-chains.md`` +Tasks 4 + 5: + +- Happy-path 3-stage chain ⇒ all three trainers run, state file + + pipeline manifest written, exit 0. +- Auto-revert at stage 2 ⇒ stage 3 enters ``skipped_due_to_prior_revert``, + ``pipeline.stage_reverted`` audit event fires, exit 3. +- Human-approval gate at stage 1 ⇒ exit 4, downstream stages still + ``pending``, state preserved for ``--resume-from``. +- ``--resume-from `` skips already-completed stages whose output + directory still exists; stale-state guard rejects a config-hash + mismatch unless ``--force-resume`` is set. +- ``--stage `` partial-run filters; missing-prev-output produces + a clear config error. +- Dry-run validates every stage without touching a trainer. + +The :class:`forgelm.trainer.ForgeTrainer` is replaced via lazy-import +monkeypatch — the orchestrator only reaches the heavy modules through +``from ..trainer import ForgeTrainer`` inside ``_run_single_stage``, so +patching ``sys.modules`` and the dataset / model helpers is enough. +""" + +from __future__ import annotations + +import json +import os +import types +from unittest.mock import MagicMock, patch + +import pytest + +from forgelm.cli._exit_codes import ( + EXIT_AWAITING_APPROVAL, + EXIT_CONFIG_ERROR, + EXIT_EVAL_FAILURE, + EXIT_SUCCESS, +) +from forgelm.cli._pipeline import ( + PipelineOrchestrator, + PipelineStageState, + PipelineState, + _compute_pipeline_config_hash, + _deserialise_state, + _generate_run_id, + _serialise_state, +) +from forgelm.config import ForgeConfig +from forgelm.results import TrainResult + +# --------------------------------------------------------------------------- +# Shared fixtures +# --------------------------------------------------------------------------- + + +def _three_stage_config(tmp_path) -> ForgeConfig: + """Build a minimal valid 3-stage pipeline ForgeConfig.""" + return ForgeConfig( + model={"name_or_path": "org/base"}, + lora={"r": 8}, + training={"trainer_type": "sft", "output_dir": str(tmp_path)}, + data={"dataset_name_or_path": "org/sft"}, + pipeline={ + "output_dir": str(tmp_path / "pipeline_run"), + "stages": [ + { + "name": "sft_stage", + "training": {"trainer_type": "sft", "output_dir": str(tmp_path / "stage1")}, + "data": {"dataset_name_or_path": "org/sft_data"}, + }, + { + "name": "dpo_stage", + "training": {"trainer_type": "dpo", "output_dir": str(tmp_path / "stage2")}, + "data": {"dataset_name_or_path": "org/dpo_data"}, + }, + { + "name": "grpo_stage", + "training": {"trainer_type": "grpo", "output_dir": str(tmp_path / "stage3")}, + "data": {"dataset_name_or_path": "org/math_data"}, + }, + ], + }, + ) + + +def _install_trainer_mocks(monkeypatch, train_results): + """Inject mock trainer/model/data modules. + + ``train_results`` is a list of :class:`TrainResult` returned in order + by successive ``ForgeTrainer.train()`` calls. The mock also + materialises the per-stage output dir on disk so the chain-integrity + guard in the orchestrator does not trip when subsequent stages + auto-chain from a "fresh" directory. + """ + iterator = iter(train_results) + instantiated_configs = [] + + class _FakeForgeTrainer: + def __init__(self, *, model, tokenizer, config, dataset): + instantiated_configs.append(config) + # Materialise final_model dir on disk so the next stage's + # chain-integrity guard sees the path it auto-chains to. + final_dir = os.path.join(config.training.output_dir, "final_model") + os.makedirs(final_dir, exist_ok=True) + self.config = config + + def train(self, resume_from_checkpoint=None): + try: + return next(iterator) + except StopIteration: + return TrainResult(success=True) + + fake_trainer_mod = types.ModuleType("forgelm.trainer") + fake_trainer_mod.ForgeTrainer = _FakeForgeTrainer + monkeypatch.setitem(__import__("sys").modules, "forgelm.trainer", fake_trainer_mod) + + fake_model_mod = types.ModuleType("forgelm.model") + fake_model_mod.get_model_and_tokenizer = lambda config: (MagicMock(), MagicMock()) + monkeypatch.setitem(__import__("sys").modules, "forgelm.model", fake_model_mod) + + fake_data_mod = types.ModuleType("forgelm.data") + fake_data_mod.prepare_dataset = lambda config, tokenizer: {"train": [{"text": "x"}]} + monkeypatch.setitem(__import__("sys").modules, "forgelm.data", fake_data_mod) + + fake_utils_mod = types.ModuleType("forgelm.utils") + fake_utils_mod.setup_authentication = lambda token: None + monkeypatch.setitem(__import__("sys").modules, "forgelm.utils", fake_utils_mod) + + return instantiated_configs + + +# --------------------------------------------------------------------------- +# Pure helpers +# --------------------------------------------------------------------------- + + +class TestPipelineConfigHash: + def test_deterministic(self): + h1 = _compute_pipeline_config_hash(b"some yaml bytes") + h2 = _compute_pipeline_config_hash(b"some yaml bytes") + assert h1 == h2 + assert h1.startswith("sha256:") + + def test_different_inputs_different_hashes(self): + assert _compute_pipeline_config_hash(b"v1") != _compute_pipeline_config_hash(b"v2") + + def test_whitespace_sensitive(self): + """Hashing raw bytes — whitespace / key order / comments all matter. + + This is intentional: regulators audit the on-disk artefact, not + the parsed semantic content. An operator who edits the file + between runs gets a new hash even if the YAML is semantically + equivalent.""" + assert _compute_pipeline_config_hash(b"a: 1\nb: 2") != _compute_pipeline_config_hash(b"b: 2\na: 1") + + +class TestRunIdShape: + def test_run_id_format(self): + run_id = _generate_run_id() + # ``pl_YYYY-MM-DD_<6-hex>`` + assert run_id.startswith("pl_") + parts = run_id.split("_") + assert len(parts) == 3 + assert len(parts[1]) == 10 # YYYY-MM-DD + assert len(parts[2]) == 6 # 6-hex + + def test_run_id_unique(self): + ids = {_generate_run_id() for _ in range(50)} + assert len(ids) == 50, "run_id collisions in 50 samples — too narrow randomness?" + + +class TestStateRoundTrip: + def test_serialise_deserialise(self): + state = PipelineState( + pipeline_run_id="pl_test", + pipeline_config_hash="sha256:abc", + forgelm_version="0.7.0-dev", + started_at="2026-06-15T12:00:00+00:00", + stages=[ + PipelineStageState(name="s1", index=0, trainer_type="sft", status="completed"), + PipelineStageState(name="s2", index=1, trainer_type="dpo", status="pending"), + ], + ) + payload = _serialise_state(state) + round = _deserialise_state(payload) + assert round.pipeline_run_id == state.pipeline_run_id + assert len(round.stages) == 2 + assert round.stages[0].name == "s1" + assert round.stages[0].status == "completed" + + def test_deserialise_tolerates_unknown_keys(self): + """Future-forward compatibility: a newer manifest schema with + extra fields must not crash an older reader.""" + payload = { + "pipeline_run_id": "pl_test", + "pipeline_config_hash": "sha256:abc", + "forgelm_version": "0.7.0", + "started_at": "2026-06-15T12:00:00+00:00", + "final_status": "completed", + "stages": [ + {"name": "s1", "index": 0, "trainer_type": "sft", "status": "completed", "future_field": "ignore me"} + ], + "future_top_level": "also ignore", + } + state = _deserialise_state(payload) + assert state.stages[0].name == "s1" + + +# --------------------------------------------------------------------------- +# Orchestrator behaviour — dry-run +# --------------------------------------------------------------------------- + + +class TestDryRun: + def test_dry_run_validates_every_stage_without_running_trainer(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + configs_seen = _install_trainer_mocks(monkeypatch, []) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch.dry_run() + assert code == EXIT_SUCCESS + # Trainer must not have been instantiated. + assert configs_seen == [] + + def test_dry_run_does_not_write_state_or_manifest(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, []) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + orch.dry_run() + assert not os.path.exists(orch.paths["state_file"]) + assert not os.path.exists(orch.paths["manifest_file"]) + + +# --------------------------------------------------------------------------- +# Orchestrator behaviour — happy path +# --------------------------------------------------------------------------- + + +class TestHappyPath: + def test_three_stage_chain_runs_all_stages(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + results = [ + TrainResult( + success=True, metrics={"eval_loss": 0.5}, final_model_path=str(tmp_path / "stage1" / "final_model") + ), + TrainResult( + success=True, metrics={"eval_loss": 0.3}, final_model_path=str(tmp_path / "stage2" / "final_model") + ), + TrainResult( + success=True, metrics={"eval_loss": 0.2}, final_model_path=str(tmp_path / "stage3" / "final_model") + ), + ] + configs_seen = _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch.run() + assert code == EXIT_SUCCESS + assert len(configs_seen) == 3 + # Stage 2's input model must be stage 1's output (auto-chain). + assert configs_seen[1].model.name_or_path == str(tmp_path / "stage1" / "final_model") + assert configs_seen[2].model.name_or_path == str(tmp_path / "stage2" / "final_model") + + def test_state_file_written_with_final_status_completed(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + results = [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)] + _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + orch.run() + assert os.path.exists(orch.paths["state_file"]) + with open(orch.paths["state_file"]) as f: + payload = json.load(f) + assert payload["final_status"] == "completed" + assert payload["stopped_at"] is None + assert all(s["status"] == "completed" for s in payload["stages"]) + + def test_manifest_written_with_chain_intact(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + results = [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)] + _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + orch.run() + assert os.path.exists(orch.paths["manifest_file"]) + with open(orch.paths["manifest_file"]) as f: + manifest = json.load(f) + assert manifest["pipeline_run_id"].startswith("pl_") + assert manifest["final_status"] == "completed" + # Chain integrity in the manifest payload. + s1, s2, s3 = manifest["stages"] + assert s2["input_model"] == s1["output_model"] + assert s3["input_model"] == s2["output_model"] + + +# --------------------------------------------------------------------------- +# Orchestrator behaviour — auto-revert + gate failures +# --------------------------------------------------------------------------- + + +class TestAutoRevert: + def test_stage_2_auto_revert_stops_pipeline(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + results = [ + TrainResult(success=True), + TrainResult(success=False, reverted=True, error="loss regression"), + ] + configs_seen = _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch.run() + assert code == EXIT_EVAL_FAILURE + # Stage 3 must NOT have been instantiated. + assert len(configs_seen) == 2 + # State file records the stop-at + skipped status. + with open(orch.paths["state_file"]) as f: + payload = json.load(f) + assert payload["final_status"] == "stopped_at_stage" + assert payload["stopped_at"] == "dpo_stage" + statuses = [s["status"] for s in payload["stages"]] + assert statuses == ["completed", "failed", "skipped_due_to_prior_revert"] + + def test_gate_failure_without_revert_also_stops_pipeline(self, tmp_path, monkeypatch): + """A stage that returns ``success=False`` but ``reverted=False`` + (e.g., a benchmark min-score failure that didn't auto-revert) + still halts the chain — downstream stages would be operating + on an explicitly-failed checkpoint.""" + cfg = _three_stage_config(tmp_path) + results = [TrainResult(success=False, reverted=False, error="benchmark below min")] + _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch.run() + assert code == EXIT_EVAL_FAILURE + with open(orch.paths["state_file"]) as f: + payload = json.load(f) + assert payload["stages"][0]["status"] == "failed" + + +# --------------------------------------------------------------------------- +# Orchestrator behaviour — human-approval gate +# --------------------------------------------------------------------------- + + +class TestHumanApprovalGate: + def test_gated_stage_exits_with_awaiting_approval(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + staging = str(tmp_path / "stage1" / "final_model.staging") + os.makedirs(staging, exist_ok=True) + results = [TrainResult(success=True, staging_path=staging)] + _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch.run() + assert code == EXIT_AWAITING_APPROVAL + + def test_gated_pending_approval_preserves_downstream_pending(self, tmp_path, monkeypatch): + """After a gated stage, downstream stages must stay ``pending`` + (not ``skipped_due_to_prior_revert``) so a subsequent + ``--resume-from`` picks them up post-approval.""" + cfg = _three_stage_config(tmp_path) + staging = str(tmp_path / "stage1" / "final_model.staging") + os.makedirs(staging, exist_ok=True) + results = [TrainResult(success=True, staging_path=staging)] + _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + orch.run() + with open(orch.paths["state_file"]) as f: + payload = json.load(f) + assert payload["final_status"] == "gated_pending_approval" + assert payload["stages"][0]["status"] == "gated_pending_approval" + assert payload["stages"][1]["status"] == "pending" + assert payload["stages"][2]["status"] == "pending" + + +# --------------------------------------------------------------------------- +# Orchestrator behaviour — --stage filter +# --------------------------------------------------------------------------- + + +class TestStageFilter: + def test_unknown_stage_name_rejected(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, []) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch.run(stage_filter="nonexistent_stage") + assert code == EXIT_CONFIG_ERROR + + def test_filter_to_first_stage_runs_only_it(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + results = [TrainResult(success=True)] + configs_seen = _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch.run(stage_filter="sft_stage") + assert code == EXIT_SUCCESS + assert len(configs_seen) == 1 + with open(orch.paths["state_file"]) as f: + payload = json.load(f) + statuses = [s["status"] for s in payload["stages"]] + assert statuses == ["completed", "skipped_by_filter", "skipped_by_filter"] + + def test_filter_to_middle_stage_without_prev_output_fails_cleanly(self, tmp_path, monkeypatch): + """``--stage dpo_stage`` without sft_stage's output on disk must + fail with a clear EXIT_CONFIG_ERROR — not silently fall back to + the root ``model.name_or_path``.""" + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=True)]) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + # Make sure the prev output dir does NOT exist. + assert not os.path.exists(tmp_path / "stage1" / "final_model") + code = orch.run(stage_filter="dpo_stage") + assert code == EXIT_CONFIG_ERROR + + def test_filter_with_input_model_override_succeeds(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + configs_seen = _install_trainer_mocks(monkeypatch, [TrainResult(success=True)]) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + override_path = str(tmp_path / "manual_input") + os.makedirs(override_path, exist_ok=True) + code = orch.run(stage_filter="dpo_stage", input_model_override=override_path) + assert code == EXIT_SUCCESS + assert len(configs_seen) == 1 + assert configs_seen[0].model.name_or_path == override_path + + +# --------------------------------------------------------------------------- +# Orchestrator behaviour — --resume-from +# --------------------------------------------------------------------------- + + +class TestResumeFrom: + def test_resume_without_state_file_fails(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, []) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch.run(resume_from="dpo_stage") + assert code == EXIT_CONFIG_ERROR + + def test_resume_skips_completed_stages(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + # First run: stage 1 completes, stage 2 fails (training crash). + results_run1 = [ + TrainResult(success=True), + TrainResult(success=False, error="oom"), + ] + _install_trainer_mocks(monkeypatch, results_run1) + orch1 = PipelineOrchestrator(cfg, b"yaml bytes") + orch1.run() + # Second run: resume from dpo_stage. Stage 1 already completed + # AND its output_model dir exists (mock created it) — must skip. + results_run2 = [TrainResult(success=True), TrainResult(success=True)] + configs_seen2 = _install_trainer_mocks(monkeypatch, results_run2) + orch2 = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch2.run(resume_from="dpo_stage") + assert code == EXIT_SUCCESS + # Only stage 2 + stage 3 ran — stage 1 was skipped. + assert len(configs_seen2) == 2 + + def test_stale_config_hash_rejects_resume(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=False, error="x")]) + orch1 = PipelineOrchestrator(cfg, b"original yaml") + orch1.run() + # Operator edited the YAML; resume against a different hash. + orch2 = PipelineOrchestrator(cfg, b"edited yaml") + with pytest.raises(SystemExit) as exc_info: + orch2.run(resume_from="dpo_stage") + assert exc_info.value.code == EXIT_CONFIG_ERROR + + def test_force_resume_accepts_stale_hash_with_warning(self, tmp_path, monkeypatch, caplog): + import logging + + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=False, error="x")]) + orch1 = PipelineOrchestrator(cfg, b"original yaml") + orch1.run() + _install_trainer_mocks(monkeypatch, [TrainResult(success=True), TrainResult(success=True)]) + orch2 = PipelineOrchestrator(cfg, b"edited yaml") + with caplog.at_level(logging.WARNING, logger="forgelm.pipeline"): + code = orch2.run(resume_from="dpo_stage", force_resume=True) + assert code == EXIT_SUCCESS + warning_emitted = any("pipeline_config_hash mismatch" in r.message for r in caplog.records) + assert warning_emitted + + def test_resume_with_unknown_stage_name_fails(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=False, error="x")]) + orch1 = PipelineOrchestrator(cfg, b"yaml") + orch1.run() + orch2 = PipelineOrchestrator(cfg, b"yaml") + code = orch2.run(resume_from="nonexistent") + assert code == EXIT_CONFIG_ERROR + + +# --------------------------------------------------------------------------- +# Orchestrator behaviour — audit + webhook (best-effort emission, no crash) +# --------------------------------------------------------------------------- + + +class TestAuditAndWebhook: + def test_audit_events_emitted_on_happy_path(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + results = [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)] + _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml") + orch.run() + # Audit log file lives under the root output dir. + audit_path = os.path.join(orch.paths["root_output_dir"], "audit_log.jsonl") + assert os.path.exists(audit_path) + with open(audit_path) as f: + events = [json.loads(line) for line in f if line.strip()] + event_names = [e["event"] for e in events] + assert "pipeline.started" in event_names + assert event_names.count("pipeline.stage_started") == 3 + assert event_names.count("pipeline.stage_completed") == 3 + assert "pipeline.completed" in event_names + + def test_webhook_failures_do_not_abort_pipeline(self, tmp_path, monkeypatch): + """A crashing webhook notifier must not derail the pipeline. + + Mirrors the existing per-stage webhook discipline — best-effort + telemetry never blocks the operator's actual work.""" + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks( + monkeypatch, [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)] + ) + with patch("forgelm.webhook.WebhookNotifier", side_effect=RuntimeError("notifier down")): + orch = PipelineOrchestrator(cfg, b"yaml") + code = orch.run() + assert code == EXIT_SUCCESS diff --git a/tools/check_bilingual_parity.py b/tools/check_bilingual_parity.py index 28743bad..06553b9e 100644 --- a/tools/check_bilingual_parity.py +++ b/tools/check_bilingual_parity.py @@ -74,6 +74,7 @@ ("docs/guides/iso_soc2_deployer_guide.md", "docs/guides/iso_soc2_deployer_guide-tr.md"), ("docs/guides/library_api.md", "docs/guides/library_api-tr.md"), ("docs/guides/performance.md", "docs/guides/performance-tr.md"), + ("docs/guides/pipeline.md", "docs/guides/pipeline-tr.md"), ("docs/guides/quickstart.md", "docs/guides/quickstart-tr.md"), ("docs/guides/safety_compliance.md", "docs/guides/safety_compliance-tr.md"), ("docs/guides/troubleshooting.md", "docs/guides/troubleshooting-tr.md"), From 0d58f1bca877b3122356f884380891482503a72f Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 14 May 2026 13:22:50 +0300 Subject: [PATCH 04/22] =?UTF-8?q?fix:=20PR=20#53=20review-response=20?= =?UTF-8?q?=E2=80=94=20Phase=2014=20blocking=20+=20significant=20+=20nitpi?= =?UTF-8?q?ck?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses two review cycles on the Phase 14 pipeline-chains branch. Blocking (Review 1): - B-1 dispatch order: pipeline branch now runs before _maybe_run_no_train_mode so --dry-run reaches the orchestrator instead of the single-stage validator. - B-2 strong-reference revert: stale state-file paths from prior runs no longer leak into the resume flow; hash mismatch now hard-aborts with a precise error unless --force-resume is given. - B-3 chain integrity: _verify_manifest_payload now compares each stage against its immediate predecessor rather than the most-recent-with-output, so a hash mismatch on stage N cannot be masked by stage N-2's output. Significant: - F-1 leaf-vs-parent dir: replaced os.path.dirname(...) with os.path.isdir(stage_cfg.model.name_or_path) — the parent check was always true for any non-root path. - F-2 empty --input-model: '' is normalised to None at dispatch time so the orchestrator never sees a falsy override that the truthy guard would silently treat as 'no override'. - F-3 --force-resume audit trail: force path now emits a dedicated pipeline.force_resume event with both old + new pipeline_config_hash before mutating state. Mismatch path without --force returns a precise diagnostic string rather than raising. - S-2 finalised-manifest verifier: flags a manifest that still has running stages as a structural integrity failure. - S-4 atomic state writes: pipeline_state.json writes go through a tmp + os.replace round-trip to survive crashes mid-write. - R1-S4 tamper guard: _load_state_file catches KeyError/TypeError/ ValueError from _deserialise_state and treats a structurally invalid state file as 'no resume state' instead of crashing. Nitpicks: - N-1 docstring drift on PipelineStage.training (clarified that trainer_type is required when training: is supplied). - N-2 verifier flags running stages on finalised manifest. - N-3 audit-write failures are non-fatal — pipeline continues, warning is logged, regression test added. - N-5 _emit_summary JSON is round-trippable (regression test added). - N-6 gated stages emit a dedicated pipeline.stage_gated audit event (separate from .stage_completed). - N-7 inheritance-matrix 'data' row marked 'strongly discouraged' in both EN + TR guides and the roadmap doc. - N-9 release-time follow-up callout added to roadmap doc. - N-10 force-resume test now asserts both old + new hashes in the audit payload, not just one. N-8 (test helper deduplication into tests/_helpers/) deferred to a follow-up PR — review explicitly labeled it 'Skip — follow-up PR'. Verification: - pytest tests/test_pipeline_*.py → 115 passed - ruff format + ruff check → clean - 11-guard gauntlet → all green (dry-run, bilingual parity, anchors, wizard defaults, no-analysis-refs, sys.modules guard, site version, CLI-help consistency, field descriptions, pip-audit, bandit) Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/guides/pipeline-tr.md | 6 +- docs/guides/pipeline.md | 6 +- docs/roadmap/phase-14-pipeline-chains.md | 6 +- forgelm/cli/_dispatch.py | 16 +- forgelm/cli/_pipeline.py | 169 ++++++++++++++++----- forgelm/compliance.py | 82 ++++++++--- tests/test_pipeline_cli.py | 156 ++++++++++++++++++++ tests/test_pipeline_compliance.py | 180 +++++++++++++++++++++++ tests/test_pipeline_orchestrator.py | 121 ++++++++++++++- 9 files changed, 664 insertions(+), 78 deletions(-) diff --git a/docs/guides/pipeline-tr.md b/docs/guides/pipeline-tr.md index d89269c6..b5e8bc62 100644 --- a/docs/guides/pipeline-tr.md +++ b/docs/guides/pipeline-tr.md @@ -105,7 +105,7 @@ etmek istiyorsan tam bloğu yaz." | `model.name_or_path` | **Otomatik zincirlenir** (aşama 1'den itibaren root'u geçer) | Önceki aşamanın `training.output_dir/final_model` değerine ayarlanır. Aşama 0 root'un `model.name_or_path`'ini okur. Aşamada explicit `model:` bloğu otomatik zincirlemeyi o aşama için devre dışı bırakır (kaçış kapısı). | | `model.*` (diğer alanlar) | `model:` bloğu override edilmediyse miras alınır | `backend`, `load_in_4bit`, `trust_remote_code`, `max_length`, `chat_template` root'u izler. | | `lora` | `lora:` bloğu override edilmediyse miras alınır | **Kritik edge case:** aşama 2 DPO, aşama 1 SFT'den *farklı* bir `lora.r` ile gelirse aşama 2 **birleştirilmiş SFT modelinin üstüne taze bir LoRA**dır, SFT'nin LoRA'sının devamı değil. Operatör-görünür davranış. | -| `data` | `data:` bloğu override edilmediyse miras alınır | Pipeline'lar aşamalar arası nadiren aynı dataset'i kullanır; aşama bazında explicit data yaygın kullanım. | +| `data` | `data:` bloğu override edilmediyse miras alınır (önerilmez) | Şema aşama bazında override'ı **zorunlu kılmıyor** — operatörler farklı bir trainer'a karşı aynı veri setini ablation için yeniden koşturabilir — ama production'da aşamalar arası aynı dataset'i kullanan pipeline'lar son derece nadirdir. Operator kılavuzu inheritance'ı bir kötü-koku olarak işaretler; gerçek pipeline'ların neredeyse tümünde SFT küratör SFT dataset'i kullanır, DPO preference-pairs dataset'i, GRPO matematik/ödül dataset'i. | | `training` | `training:` bloğu override edilmediyse miras alınır | `trainer_type` her aşamada explicit set EDİLMEK ZORUNDADIR (audit-clarity doğrulayıcısı). Diğer alanlar blok override edildiğinde wholesale yer değiştirir. | | `evaluation` | `evaluation:` bloğu override edilmediyse miras alınır | Aşama bazında gate'ler (loss eşikleri, auto_revert, safety, judge, human-approval) burada yaşar. | | `distributed` | Sadece root — **aşamada reddedilir** | Distributed strateji koşu boyunca tutarlı olmak zorundadır. | @@ -210,8 +210,10 @@ altında) emit eder: - `pipeline.started` — run id, config hash, stage count, stage names - `pipeline.stage_started` — stage name, index, input model, input source -- `pipeline.stage_completed` — stage name, gate decision, metrics summary +- `pipeline.stage_completed` — stage name, gate decision (`passed` / `failed`), metrics summary +- `pipeline.stage_gated` — stage name, gate decision `approval_pending`, staging path (bir aşama `EXIT_AWAITING_APPROVAL` ile çıkış yaptığında emit edilir) - `pipeline.stage_reverted` — stage name, auto-revert reason +- `pipeline.force_resume` — operatör onaylı stale-hash override, `old_config_hash` + `new_config_hash` taşır - `pipeline.completed` — final status, stopped-at stage (varsa) Bu olaylar mevcut her aşamanın `ForgeTrainer`'inin emit etmeye devam diff --git a/docs/guides/pipeline.md b/docs/guides/pipeline.md index 5462e59d..404db8d5 100644 --- a/docs/guides/pipeline.md +++ b/docs/guides/pipeline.md @@ -104,7 +104,7 @@ inherit, omit the block; if you want to override, supply the full block." | `model.name_or_path` | **Auto-chained** (overrides root from stage 1 onward) | Set to previous stage's `training.output_dir/final_model`. Stage 0 reads root `model.name_or_path`. An explicit per-stage `model:` block disables auto-chaining for that stage (escape hatch). | | `model.*` (other fields) | Inherited unless `model:` block overridden | `backend`, `load_in_4bit`, `trust_remote_code`, `max_length`, `chat_template` follow the root. | | `lora` | Inherited unless `lora:` block overridden | **Critical edge case:** stage 2 DPO with a *different* `lora.r` than stage 1 SFT means stage 2 is **a fresh LoRA over the merged SFT model**, not a continuation of SFT's LoRA. Operator-visible behaviour. | -| `data` | Inherited unless `data:` block overridden | Pipelines rarely reuse the same dataset across stages; explicit per-stage data is the common case. | +| `data` | Inherited unless `data:` block overridden (strongly discouraged) | The schema does **not** force a per-stage override — operators may rerun the same dataset against a different trainer for ablation studies — but pipelines that legitimately reuse the same dataset across stages are vanishingly rare in production. The operator guide flags inheritance as a smell; almost every real chain has SFT use a curated SFT dataset, DPO use a preference-pairs dataset, and GRPO use a math/reward dataset. | | `training` | Inherited unless `training:` block overridden | `trainer_type` MUST be set explicitly per stage (audit-clarity validator). Other fields wholesale-replaced when the block is overridden. | | `evaluation` | Inherited unless `evaluation:` block overridden | Per-stage gates (loss thresholds, auto_revert, safety, judge, human-approval) live here. | | `distributed` | Root only — **per-stage rejected** | Distributed strategy must be consistent across the run. | @@ -208,8 +208,10 @@ The orchestrator emits these events into the pipeline-level - `pipeline.started` — run id, config hash, stage count, stage names - `pipeline.stage_started` — stage name, index, input model, input source -- `pipeline.stage_completed` — stage name, gate decision, metrics summary +- `pipeline.stage_completed` — stage name, gate decision (`passed` / `failed`), metrics summary +- `pipeline.stage_gated` — stage name, gate decision `approval_pending`, staging path (emitted when a stage exits `EXIT_AWAITING_APPROVAL`) - `pipeline.stage_reverted` — stage name, auto-revert reason +- `pipeline.force_resume` — operator-approved stale-hash override with `old_config_hash` + `new_config_hash` - `pipeline.completed` — final status, stopped-at stage (if any) These events live alongside the existing `training.*` events that each diff --git a/docs/roadmap/phase-14-pipeline-chains.md b/docs/roadmap/phase-14-pipeline-chains.md index 4152d84c..501ff6cf 100644 --- a/docs/roadmap/phase-14-pipeline-chains.md +++ b/docs/roadmap/phase-14-pipeline-chains.md @@ -62,7 +62,7 @@ | `model.name_or_path` | **Auto-chained** (overrides root after stage 0) | Set to previous stage's `training.output_dir/final_model`. Stage 0 still reads root `model.name_or_path`. An explicit per-stage `model.name_or_path` is allowed and disables auto-chaining for that stage (operator escape hatch). | | `model.*` (other fields) | Inherited unless `model:` block overridden | `backend`, `load_in_4bit`, `trust_remote_code`, `max_length`, `chat_template` follow the root unless the stage supplies a full `model:` block. | | `lora` | Inherited unless `lora:` block overridden | Critical edge case: stage 2 DPO with a *different* `lora.r` than stage 1 SFT means stage 2 is **a fresh LoRA over the merged SFT model**, not a continuation of SFT's LoRA. Documented in the operator guide. | - | `data` | **Must be overridden per stage** | Pipelines that don't change the dataset between stages are vanishingly rare; explicit per-stage data is a forcing function against accidental dataset reuse. | + | `data` | Inherited unless `data:` block overridden (strongly discouraged) | Pipelines that don't change the dataset between stages are vanishingly rare — almost every real chain has SFT use a curated SFT dataset, DPO use a preference pairs dataset, and GRPO use a math/reward dataset. The schema does not *force* a per-stage override (operators may legitimately rerun the same dataset against a different trainer for ablation studies), but the operator guide flags the inheritance case as a smell and the dry-run summary prints a warning when two consecutive stages share the same `dataset_name_or_path`. | | `training` | Inherited unless `training:` block overridden | `trainer_type` is required per stage; the validator rejects a stage with no `training.trainer_type` even if the root supplies one (each stage explicitly states its alignment paradigm for audit clarity). | | `evaluation` | Inherited unless `evaluation:` block overridden | Per-stage gates (Task 5) live here; a stage that wants different `auto_revert` / `safety` config supplies a full `evaluation:` block. | | `distributed` | Inherited (always); no per-stage override | Distributed strategy must be consistent across the pipeline run. Per-stage `distributed:` blocks rejected at load with a clear error. | @@ -181,8 +181,10 @@ **Audit-log events (extends the existing 5-event vocabulary):** - `pipeline.started` — run id, config hash, stage count, stage names - `pipeline.stage_started` — stage name, index, input model, input source (chain/cli_override/root) - - `pipeline.stage_completed` — stage name, metrics summary, gate decision + - `pipeline.stage_completed` — stage name, metrics summary, gate decision (`passed` / `failed`) + - `pipeline.stage_gated` — stage name, gate decision `approval_pending`, staging path (emitted instead of `stage_completed` when a stage exits `EXIT_AWAITING_APPROVAL`; lets dashboard / SIEM rules filter on the event name alone — Phase 14 review F-N-1) - `pipeline.stage_reverted` — stage name, auto-revert reason, halt-pipeline=true + - `pipeline.force_resume` — operator-approved stale-hash override; carries `old_config_hash` + `new_config_hash` so reviewers can correlate to the audit trail (Phase 14 review F-B-2) - `pipeline.completed` — final status, total duration, stage count These events live alongside (not replacing) the existing per-stage `training.*` events emitted by `ForgeTrainer`. Webhook notifier (`forgelm/webhook.py`) gains `notify_pipeline_started` / `notify_pipeline_completed` / `notify_pipeline_reverted` methods that wrap the existing 5-event vocabulary so Slack / Teams dashboards filtering on `event=training.failure` continue to work; pipeline-aware dashboards can additionally filter on the new `pipeline.*` events. diff --git a/forgelm/cli/_dispatch.py b/forgelm/cli/_dispatch.py index a06cc9b1..94b88e6a 100644 --- a/forgelm/cli/_dispatch.py +++ b/forgelm/cli/_dispatch.py @@ -261,12 +261,16 @@ def _main_inner() -> None: config = _load_config_or_exit(args.config, json_output) _apply_offline_flag(config, args.offline) - _maybe_run_no_train_mode(config, args) - # Phase 14 — multi-stage pipeline branch. When the YAML carries a - # ``pipeline:`` block, hand control to the orchestrator instead of - # the single-stage trainer. Single-stage configs (the v0.6.0 default) - # skip this branch entirely and reach the trainer as before. + # Phase 14 — multi-stage pipeline branch MUST run BEFORE + # ``_maybe_run_no_train_mode``. The single-stage no-train modes + # (``--dry-run``, ``--fit-check``, ``--benchmark-only``, ``--merge``, + # ``--generate-data``, ``--compliance-export``) call ``sys.exit`` + # internally; if they ran first on a pipeline config, the orchestrator + # would never reach the multi-stage validation path and the documented + # ``--dry-run`` per-stage error collection contract (Phase 14 Task 4) + # would be silently violated — the operator would see the legacy + # single-stage dry-run summary instead. if config.pipeline is not None: from ._pipeline import run_pipeline_from_args @@ -284,4 +288,6 @@ def _main_inner() -> None: sys.exit(run_pipeline_from_args(config, pipeline_yaml_bytes, args)) + # Single-stage path — unchanged from v0.6.0. + _maybe_run_no_train_mode(config, args) _run_training_pipeline(config, args, json_output) diff --git a/forgelm/cli/_pipeline.py b/forgelm/cli/_pipeline.py index f07d7fec..bb003b91 100644 --- a/forgelm/cli/_pipeline.py +++ b/forgelm/cli/_pipeline.py @@ -34,12 +34,19 @@ events alongside the existing per-stage ``training.*`` vocabulary; the webhook notifier gains ``notify_pipeline_*`` methods that the orchestrator calls at the same transitions. -- **Exit-code routing** — the pipeline exits with the highest-priority - per-stage exit code: ``EXIT_AWAITING_APPROVAL (4)`` if any stage gated - pending approval, then ``EXIT_TRAINING_ERROR (2)`` for trainer - crashes, then ``EXIT_EVAL_FAILURE (3)`` for gate failures, then - ``EXIT_CONFIG_ERROR (1)`` for orchestrator-level config issues, then - ``EXIT_SUCCESS (0)``. +- **Exit-code routing** — the pipeline aggregates per-stage outcomes + via ``worst_exit = max(worst_exit, code)``, which gives the + precedence order + ``EXIT_AWAITING_APPROVAL (4)`` > ``EXIT_EVAL_FAILURE (3)`` > + ``EXIT_TRAINING_ERROR (2)`` > ``EXIT_CONFIG_ERROR (1)`` > + ``EXIT_SUCCESS (0)``. In practice an approval gate causes an + immediate ``break`` (no later stages run) and any other failure + sets ``chain_broken = True`` (downstream stages skip), so the + precedence rule is reached only when a single stage produces a + single non-zero code. Numeric ordering of the exit-code constants + matches the documented severity ramp — operators reading the exit + code in a shell only see the "worst" outcome from a single stage + that ran. """ from __future__ import annotations @@ -49,7 +56,6 @@ import logging import os import secrets -import sys from dataclasses import asdict, dataclass, field from datetime import datetime, timezone from typing import Any, Dict, List, Literal, Optional @@ -293,11 +299,16 @@ def run( ``stage_filter`` and ``resume_from`` are mutually exclusive; the caller (CLI parser) validates this before reaching here. - Exit-code precedence (highest first): - ``EXIT_TRAINING_ERROR (2)`` for a trainer crash, ``EXIT_AWAITING_APPROVAL (4)`` - for a gated stage, ``EXIT_EVAL_FAILURE (3)`` for a gate failure - / auto-revert, ``EXIT_CONFIG_ERROR (1)`` for orchestrator-level - config issues, ``EXIT_SUCCESS (0)`` otherwise. + Exit-code aggregation uses ``worst_exit = max(worst_exit, code)`` + across stages, giving the numeric precedence: + ``EXIT_AWAITING_APPROVAL (4)`` > ``EXIT_EVAL_FAILURE (3)`` > + ``EXIT_TRAINING_ERROR (2)`` > ``EXIT_CONFIG_ERROR (1)`` > + ``EXIT_SUCCESS (0)``. An approval gate triggers an immediate + ``break`` (later stages stay ``pending``) and any other non-zero + outcome sets ``chain_broken = True`` (later stages skip with + ``skipped_due_to_prior_revert``), so multi-stage compounding is + rare — in practice the return value reflects the single failing + stage's exit code. """ # 1. Load (or initialise) state. existing = self._load_state_file() @@ -309,7 +320,14 @@ def run( ) return EXIT_CONFIG_ERROR if resume_from is not None: - self._validate_resume_state(existing, force=force_resume) + refusal = self._validate_resume_state(existing, force=force_resume) + if refusal is not None: + # Library-style refusal — return the exit code through + # the normal ``run()`` flow rather than ``sys.exit``-ing + # mid-method. Matches the surrounding error paths + # (missing state file, unknown stage name) and keeps the + # test surface uniform on ``assert code == EXIT_CONFIG_ERROR``. + return EXIT_CONFIG_ERROR state = existing else: state = self._init_state() @@ -467,11 +485,19 @@ def run( state.finished_at = _utc_iso_now() worst_exit = max(worst_exit, EXIT_AWAITING_APPROVAL) self._save_state(state) + # Phase 14 review F-N-1: dedicated ``pipeline.stage_gated`` + # event (instead of ``pipeline.stage_completed`` with a + # ``gate_decision=approval_pending`` sub-field). Lets + # dashboard / SIEM rules filter on the event name alone + # without parsing payload sub-fields — "stage_completed" + # reads as "stage finished cleanly" to a regex-based + # alert rule. self._audit_event( - "pipeline.stage_completed", + "pipeline.stage_gated", pipeline_run_id=state.pipeline_run_id, stage_name=stage.name, gate_decision="approval_pending", + staging_path=stage_state.output_model, ) # No pipeline.completed event yet — pipeline is not done. break @@ -594,7 +620,7 @@ def _init_state(self) -> PipelineState: stages=stages, ) - def _validate_resume_state(self, existing: PipelineState, *, force: bool) -> None: + def _validate_resume_state(self, existing: PipelineState, *, force: bool) -> Optional[str]: """Stale-state guard for ``--resume-from``. If the on-disk state file's ``pipeline_config_hash`` differs from @@ -602,28 +628,54 @@ def _validate_resume_state(self, existing: PipelineState, *, force: bool) -> Non file between the failed run and the resume call — resuming against a different config silently produces a chain whose history doesn't match its current shape. We refuse unless - ``--force-resume`` is set (logged at WARNING, recorded in the - audit event so reviewers see why the divergence was accepted). + ``--force-resume`` is set (logged at WARNING **and** emitted as a + ``pipeline.force_resume`` audit event so reviewers see why the + divergence was accepted). + + Return semantics: + + - ``None`` — the resume is OK to proceed (no hash divergence, or + divergence accepted via ``--force-resume``). + - error-string — the resume must abort; the caller maps this to + ``EXIT_CONFIG_ERROR``. Using a return value (rather than + ``sys.exit`` mid-method) keeps the orchestrator's exit policy + consistent with the rest of ``run()`` — every refusal flows + back through the same ``return`` branch, the same audit-log + finalisation runs, and the test surface uses a uniform + ``assert code == EXIT_CONFIG_ERROR`` shape. """ if existing.pipeline_config_hash == self.config_hash: - return + return None if not force: - logger.error( - "Refusing to --resume-from: pipeline_config_hash mismatch.\n" - " on-disk: %s\n current: %s\n" - "Pass --force-resume to override (logged for audit).", - existing.pipeline_config_hash, - self.config_hash, + msg = ( + f"Refusing to --resume-from: pipeline_config_hash mismatch.\n" + f" on-disk: {existing.pipeline_config_hash}\n" + f" current: {self.config_hash}\n" + "Pass --force-resume to override (logged for audit)." ) - sys.exit(EXIT_CONFIG_ERROR) + logger.error(msg) + return msg + old_hash = existing.pipeline_config_hash logger.warning( "Resuming with --force-resume despite pipeline_config_hash mismatch: %s → %s.", - existing.pipeline_config_hash, + old_hash, self.config_hash, ) + # Append-only audit event so a reviewer reading the JSONL stream + # later can distinguish an operator-approved stale-config resume + # from a normal resume. Emitted BEFORE the stored hash is + # updated so the event payload still records both sides of the + # divergence. + self._audit_event( + "pipeline.force_resume", + pipeline_run_id=existing.pipeline_run_id, + old_config_hash=old_hash, + new_config_hash=self.config_hash, + ) # Update the stored hash to match the now-trusted current config # so subsequent transitions don't trip the guard again. existing.pipeline_config_hash = self.config_hash + return None def _load_state_file(self) -> Optional[PipelineState]: path = self.paths["state_file"] @@ -635,7 +687,23 @@ def _load_state_file(self) -> Optional[PipelineState]: except (OSError, json.JSONDecodeError) as e: logger.warning("Failed to read pipeline_state.json at %s: %s. Treating as missing.", path, e) return None - return _deserialise_state(payload) + # Phase 14 review F-R1-S4: a tampered state file (wrong shape, + # missing required keys, ``stages`` not a list of dicts, etc.) + # could crash ``_deserialise_state`` with an opaque + # ``TypeError`` / ``KeyError`` traceback instead of producing + # an actionable config-error path. Catch the common shape + # failures here and treat the file as missing — the caller's + # ``--resume-from`` branch then surfaces ``EXIT_CONFIG_ERROR`` + # with a clear message. + try: + return _deserialise_state(payload) + except (KeyError, TypeError, ValueError) as e: + logger.warning( + "pipeline_state.json at %s is structurally invalid (%s); treating as missing.", + path, + e, + ) + return None def _save_state(self, state: PipelineState) -> None: """Persist state file + refresh pipeline manifest. @@ -652,12 +720,24 @@ def _save_state(self, state: PipelineState) -> None: _atomic_write_json(self.paths["manifest_file"], manifest) def _audit_event(self, event: str, **fields: Any) -> None: - """Append-only audit log entry under the root output dir. - - Reuses the existing :class:`forgelm.compliance.AuditLogger` so - the pipeline events land in the same JSONL stream as the - per-stage ``training.*`` events from each :class:`ForgeTrainer`. - Reviewers can grep one file for the full chain. + """Append-only audit log entry under the **pipeline** root output dir. + + Two separate audit-log streams exist on a multi-stage run: + + - **Pipeline-level** (this method): ``/ + audit_log.jsonl`` — carries ``pipeline.*`` lifecycle events + (started / stage_started / stage_completed / stage_gated / + stage_reverted / force_resume / completed). + - **Stage-level** (emitted by each ``ForgeTrainer``): + ``/audit_log.jsonl`` — carries the + existing ``training.*`` vocabulary unchanged from a single- + stage v0.6.0 run. + + Both files are append-only JSONL. Reviewers can correlate by + joining on ``pipeline_run_id`` (present in every pipeline event) + with the per-stage ``training_manifest.json`` ``training_run_id``. + See ``docs/guides/pipeline.md`` "Audit events" for the full + join key matrix. """ try: from ..compliance import AuditLogger @@ -748,12 +828,16 @@ def _run_single_stage( stage_state.input_model = stage_cfg.model.name_or_path - # ``forgelm`` ``--input-model`` partial-run guard: when the - # operator filters to a non-first stage and the previous stage's - # output directory does not exist, refuse to fabricate it. + # ``forgelm`` chain-presence guard: when the orchestrator + # auto-chained the input model from the previous stage's output, + # refuse to proceed unless that exact directory exists. Phase + # 14 review F-S-1: pre-fix this checked ``os.path.dirname(...)`` + # (the parent), accepting ``./stage1/`` even when ``./stage1/ + # final_model`` was missing and letting a bad chain state fall + # through into ``get_model_and_tokenizer`` for a non-actionable + # downstream crash. if stage_state.input_source == "chain": - expected_dir = os.path.dirname(stage_cfg.model.name_or_path) - if not os.path.isdir(expected_dir): + if not os.path.isdir(stage_cfg.model.name_or_path): stage_state.error = ( f"Stage {stage.name!r} requires the previous stage's output at " f"{stage_cfg.model.name_or_path}; pass --input-model to " @@ -857,7 +941,14 @@ def run_pipeline_from_args( stage_filter = getattr(args, "stage", None) resume_from = getattr(args, "resume_from", None) force_resume = bool(getattr(args, "force_resume", False)) - input_model_override = getattr(args, "input_model", None) + # Phase 14 review F-F-2: argparse cannot reject an empty string + # value for ``--input-model "" --stage X`` at parse time, and the + # downstream merge helper uses ``is not None`` rather than a truthy + # check, so the empty string would silently overwrite the auto- + # chained model path. Normalise to ``None`` here so the rest of the + # orchestrator sees "no override" rather than "override with nothing". + raw_input_model = getattr(args, "input_model", None) + input_model_override = raw_input_model if raw_input_model else None output_format = getattr(args, "output_format", "text") dry_run = bool(getattr(args, "dry_run", False)) diff --git a/forgelm/compliance.py b/forgelm/compliance.py index 39b91c7e..92136813 100644 --- a/forgelm/compliance.py +++ b/forgelm/compliance.py @@ -1064,15 +1064,24 @@ def _verify_manifest_payload(manifest: Dict[str, Any]) -> List[str]: Checks: 1. **Required top-level keys** are present and of the right shape. - 2. **Chain integrity** — for every stage N (N ≥ 1) that completed - successfully, its ``input_model`` must equal stage N-1's - ``output_model``. Mismatches indicate either a CLI - ``--input-model`` override (legitimate) or a corrupt manifest - (illegitimate); the verifier surfaces both with the same - message so reviewers can inspect the audit log to disambiguate. + 2. **Chain integrity** — for every stage N with + ``input_source == "chain"``, the *immediate* previous stage's + ``output_model`` must match its ``input_model``. If the previous + stage has no ``output_model`` (e.g. failed before saving) the + chain link is unreconstructible and the verifier flags it as a + ``chain_integrity_violation`` (Phase 14 review F-B-3 hardening: + pre-fix the verifier walked across stages that *had* an + output_model, silently accepting a manifest whose chain could + not actually be reconstructed). Stages with + ``input_source != "chain"`` (``root`` / ``stage_explicit`` / + ``cli_override``) intentionally break the chain — by design, + reviewers inspect the audit log to validate them. 3. **Status consistency** — at most one ``stopped_at`` stage; if set, that stage's status must be one of ``failed`` / - ``gated_pending_approval``. + ``gated_pending_approval``. Additionally, a finalised manifest + (``final_status != "in_progress"``) must not carry any stage + still in ``running`` status — that indicates a process crash + mid-stage that the archive must surface (Phase 14 review F-N-2). 4. **Index monotonicity** — stage indices form 0..N-1 in order. """ violations: List[str] = [] @@ -1099,19 +1108,35 @@ def _verify_manifest_payload(manifest: Dict[str, Any]) -> List[str]: if s.get("index") != expected: violations.append(f"stage index out of order at position {expected}: got {s.get('index')!r}") - # Chain integrity. - prev_output: Optional[str] = None - for s in stages: - status = s.get("status") - # Only completed / failed (with output) stages contribute to the chain. - if status in ("completed", "failed", "gated_pending_approval") and s.get("output_model"): - if prev_output is not None and s.get("input_source") == "chain": - if s.get("input_model") != prev_output: - violations.append( - f"chain_integrity_violation at stage {s.get('name')!r}: input_model={s.get('input_model')!r} " - f"≠ previous output_model={prev_output!r}" - ) - prev_output = s.get("output_model") + # Chain integrity — Phase 14 review F-B-3 + F-N-3 hardening: every + # chain stage is compared against the **immediate** previous stage, + # not the most-recent stage with an ``output_model``. Pre-fix, if + # stage N-1 failed without saving an ``output_model`` (or any + # hand-crafted manifest with a None output midway), the chain check + # silently skipped that gap and compared stage N against stage N-2, + # masking a real broken link. + for idx, s in enumerate(stages): + if s.get("input_source") != "chain": + continue + if idx == 0: + violations.append( + f"Stage {s.get('name')!r}: input_source='chain' but no previous stage exists (stage 0 cannot chain)." + ) + continue + prev = stages[idx - 1] + prev_output_model = prev.get("output_model") + if not prev_output_model: + violations.append( + f"chain_integrity_violation at stage {s.get('name')!r}: claims " + f"input_source='chain' but previous stage {prev.get('name')!r} has " + f"no output_model (status={prev.get('status')!r})." + ) + continue + if s.get("input_model") != prev_output_model: + violations.append( + f"chain_integrity_violation at stage {s.get('name')!r}: input_model={s.get('input_model')!r} " + f"≠ previous output_model={prev_output_model!r}" + ) # Status consistency. stopped_at = manifest.get("stopped_at") @@ -1125,6 +1150,21 @@ def _verify_manifest_payload(manifest: Dict[str, Any]) -> List[str]: f"(expected `failed` or `gated_pending_approval`)" ) + # Running-stage consistency (N-2): a finalised manifest + # (final_status != "in_progress") with a stage in ``running`` status + # is a tell that the orchestrator process died mid-stage and never + # updated the state file. Archival audit must surface this so the + # operator knows the chain wasn't actually completed. + final_status = manifest.get("final_status") + if final_status and final_status != "in_progress": + running_stages = [s.get("name") for s in stages if s.get("status") == "running"] + if running_stages: + violations.append( + f"stage(s) {running_stages!r} still in `running` status on a " + f"finalised manifest (final_status={final_status!r}); the " + f"orchestrator likely crashed mid-stage without updating state." + ) + return violations @@ -1659,7 +1699,7 @@ def generate_pipeline_manifest(state: Any, root_cfg: Any) -> Dict[str, Any]: "auto_revert_triggered": bool(s.auto_revert_triggered), "skipped_reason": s.skipped_reason, "exit_code": s.exit_code, - "error": getattr(s, "error", None), + "error": s.error, } ) diff --git a/tests/test_pipeline_cli.py b/tests/test_pipeline_cli.py index e31c1aba..5a296eae 100644 --- a/tests/test_pipeline_cli.py +++ b/tests/test_pipeline_cli.py @@ -15,6 +15,8 @@ import types from unittest.mock import MagicMock +import pytest + from forgelm.cli._exit_codes import EXIT_CONFIG_ERROR, EXIT_SUCCESS from forgelm.cli._pipeline import run_pipeline_from_args from forgelm.config import ForgeConfig @@ -66,9 +68,11 @@ def _ns(**overrides): def _install_fake_trainer(monkeypatch, results): iterator = iter(results) + instantiated_configs: list = [] class _FakeForgeTrainer: def __init__(self, *, model, tokenizer, config, dataset): + instantiated_configs.append(config) os.makedirs(os.path.join(config.training.output_dir, "final_model"), exist_ok=True) self.config = config @@ -94,6 +98,8 @@ def train(self, resume_from_checkpoint=None): fake_utils.setup_authentication = lambda token: None monkeypatch.setitem(__import__("sys").modules, "forgelm.utils", fake_utils) + return instantiated_configs + class TestFlagInteractionGuards: """Argparse alone cannot express "X and Y are mutually exclusive @@ -111,6 +117,28 @@ def test_input_model_requires_stage(self, tmp_path): code = run_pipeline_from_args(cfg, b"yaml", args) assert code == EXIT_CONFIG_ERROR + def test_empty_input_model_normalised_to_none(self, tmp_path, monkeypatch): + """Phase 14 review F-F-2 regression: ``--input-model ""`` + (empty string) used to slip past the dispatcher's truthy + check and propagate through ``merge_pipeline_stage_config`` + (``is not None`` branch), silently overwriting the auto-chained + model path with the empty string. The orchestrator now + normalises a falsy ``--input-model`` value to ``None`` before + dispatch so it never reaches the merge helper. + """ + cfg = _three_stage_cfg(tmp_path) + # Pre-create stage 1's output so the chain check on stage 2 is + # not the failure point — the test asserts the empty override + # is normalised, not that the chain is broken. + (tmp_path / "stage1" / "final_model").mkdir(parents=True, exist_ok=True) + configs_seen = _install_fake_trainer(monkeypatch, [TrainResult(success=True)]) + args = _ns(stage="dpo_stage", input_model="") + code = run_pipeline_from_args(cfg, b"yaml", args) + assert code == EXIT_SUCCESS + # The stage's input_model must come from the auto-chain (stage + # 1's on-disk final_model), NOT from the empty string override. + assert configs_seen[0].model.name_or_path == str(tmp_path / "stage1" / "final_model") + def test_force_resume_without_resume_from_is_noop(self, tmp_path, monkeypatch): """``--force-resume`` without ``--resume-from`` is meaningless but not an error — the orchestrator simply ignores it on a @@ -161,3 +189,131 @@ def test_unknown_stage_filter_rejected_via_orchestrator(self, tmp_path, monkeypa args = _ns(stage="ghost") code = run_pipeline_from_args(cfg, b"yaml", args) assert code == EXIT_CONFIG_ERROR + + +class TestTopLevelDispatchOrdering: + """Regression for the Phase 14 review F-B-1: the single-stage + ``--dry-run`` path inside ``_maybe_run_no_train_mode`` MUST NOT run + before the orchestrator dispatch when the YAML carries a + ``pipeline:`` block. Otherwise ``forgelm --config pipeline.yaml + --dry-run`` falls through to the legacy single-stage dry-run summary + and the documented per-stage chain-integrity validation never fires. + + Exercises the actual top-level ``main()`` (not just + ``run_pipeline_from_args``) by writing a YAML to disk and invoking + the CLI in-process via the installed entry point. + """ + + def test_pipeline_dry_run_routes_to_orchestrator_not_legacy(self, tmp_path, monkeypatch, caplog): + import logging + import sys as _sys + + import yaml + + yaml_path = tmp_path / "pipeline.yaml" + yaml_path.write_text( + yaml.safe_dump( + { + "model": {"name_or_path": "org/base"}, + "lora": {}, + "training": {"trainer_type": "sft"}, + "data": {"dataset_name_or_path": "org/data"}, + "pipeline": { + "output_dir": str(tmp_path / "pipeline_run"), + "stages": [ + { + "name": "sft_stage", + "training": {"trainer_type": "sft", "output_dir": str(tmp_path / "s1")}, + "data": {"dataset_name_or_path": "org/sft"}, + }, + { + "name": "dpo_stage", + "training": {"trainer_type": "dpo", "output_dir": str(tmp_path / "s2")}, + "data": {"dataset_name_or_path": "org/dpo"}, + }, + ], + }, + } + ) + ) + + from forgelm.cli._dispatch import main as cli_main + + monkeypatch.setattr(_sys, "argv", ["forgelm", "--config", str(yaml_path), "--dry-run"]) + with caplog.at_level(logging.INFO, logger="forgelm.pipeline"): + with pytest.raises(SystemExit) as exc_info: + cli_main() + # The orchestrator's dry-run exits 0 on a clean validation; the + # legacy single-stage dry-run would also exit 0 here, so the + # discriminator below is the orchestrator's INFO log line — + # ``Pipeline dry-run OK: stage(s) validated`` is emitted + # only by ``PipelineOrchestrator.dry_run``. The legacy + # single-stage dry-run path lives at ``forgelm.cli._dry_run`` and + # emits a different log surface. + assert exc_info.value.code == 0 + orchestrator_log_emitted = any("pipeline dry-run ok" in r.message.lower() for r in caplog.records) + assert orchestrator_log_emitted, ( + "Dispatcher routed --dry-run through the legacy single-stage path " + "instead of the pipeline orchestrator (Phase 14 F-B-1 regression). " + f"Captured records: {[r.message for r in caplog.records]!r}" + ) + + def test_pipeline_branch_runs_before_no_train_modes_on_pipeline_config(self, tmp_path, monkeypatch): + """Structural assertion: importing ``_dispatch`` and following + the code path on a ``config.pipeline is not None`` config must + invoke ``run_pipeline_from_args`` strictly before + ``_maybe_run_no_train_mode``. Sentinels track the call order.""" + import yaml + + from forgelm.cli import _dispatch + + yaml_path = tmp_path / "pipeline.yaml" + yaml_path.write_text( + yaml.safe_dump( + { + "model": {"name_or_path": "org/base"}, + "lora": {}, + "training": {"trainer_type": "sft"}, + "data": {"dataset_name_or_path": "org/data"}, + "pipeline": { + "output_dir": str(tmp_path / "pipeline_run"), + "stages": [ + { + "name": "sft_stage", + "training": {"trainer_type": "sft", "output_dir": str(tmp_path / "s1")}, + "data": {"dataset_name_or_path": "org/sft"}, + }, + ], + }, + } + ) + ) + + call_order: list[str] = [] + + def _fake_maybe_no_train(config, args): + call_order.append("no_train_mode") + + def _fake_run_pipeline(config, yaml_bytes, args): + call_order.append("pipeline_dispatch") + return 0 + + monkeypatch.setattr(_dispatch, "_maybe_run_no_train_mode", _fake_maybe_no_train) + import forgelm.cli._pipeline as _pipeline_mod + + monkeypatch.setattr(_pipeline_mod, "run_pipeline_from_args", _fake_run_pipeline) + # The dispatcher imports run_pipeline_from_args lazily inside + # main() — patch the function on the source module so the lazy + # import resolves to our fake. + import sys + + monkeypatch.setattr(sys, "argv", ["forgelm", "--config", str(yaml_path)]) + with pytest.raises(SystemExit): + _dispatch.main() + # Pipeline dispatch must fire; no_train_mode must NOT have run + # before it. + assert call_order == ["pipeline_dispatch"], ( + f"Expected pipeline_dispatch first (and only), got {call_order!r}. " + f"Phase 14 F-B-1 regression: legacy no-train path executed before " + f"the pipeline branch on a pipeline config." + ) diff --git a/tests/test_pipeline_compliance.py b/tests/test_pipeline_compliance.py index d25155f4..4be11758 100644 --- a/tests/test_pipeline_compliance.py +++ b/tests/test_pipeline_compliance.py @@ -268,6 +268,186 @@ def test_auto_revert_manifest_verifies(self): assert _verify_manifest_payload(manifest) == [] +class TestStrictChainIntegrity: + """Phase 14 review F-B-3 + F-N-3 regression: the verifier must + compare every chain stage against its **immediate** predecessor, + not the most-recent stage that happens to carry an + ``output_model``. Without this, a broken/missing prev output is + silently bridged. + """ + + def test_chain_stage_with_prev_missing_output_flagged(self): + """Stage 0 completed normally, stage 1 failed without saving an + output, stage 2 claims input_source='chain'. Pre-fix the + verifier compared stage 2 against stage 0's output, masking the + gap; the strict check now flags it.""" + s0 = PipelineStageState( + name="s0", + index=0, + trainer_type="sft", + status="completed", + input_source="root", + output_model="./out/s0/final_model", + ) + s1 = PipelineStageState( + name="s1", + index=1, + trainer_type="dpo", + status="failed", + input_model="./out/s0/final_model", + input_source="chain", + output_model=None, # crashed before save + ) + s2 = PipelineStageState( + name="s2", + index=2, + trainer_type="grpo", + status="completed", + input_model="./out/s0/final_model", # plausibly stale + input_source="chain", + output_model="./out/s2/final_model", + ) + state = PipelineState( + pipeline_run_id="pl_x", + pipeline_config_hash="sha256:abc", + forgelm_version="0.7.0", + started_at="2026-06-15T12:00:00+00:00", + final_status="stopped_at_stage", + stopped_at="s1", + stages=[s0, s1, s2], + ) + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + assert any("chain_integrity_violation" in v and "'s2'" in v for v in violations), ( + f"Expected stage 's2' to fail chain integrity due to gap; got: {violations!r}" + ) + + def test_chain_stage_at_index_zero_flagged(self): + """Stage 0 cannot have input_source='chain' (there is no + previous stage).""" + s0 = PipelineStageState( + name="s0", + index=0, + trainer_type="sft", + status="completed", + input_source="chain", # wrong — no prev exists + input_model="./somewhere", + output_model="./out/s0/final_model", + ) + state = PipelineState( + pipeline_run_id="pl_x", + pipeline_config_hash="sha256:abc", + forgelm_version="0.7.0", + started_at="2026-06-15T12:00:00+00:00", + final_status="completed", + stages=[s0], + ) + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + assert any("'s0'" in v and "stage 0 cannot chain" in v for v in violations) + + +class TestVerifierFlagsRunningOnFinalisedManifest: + """Phase 14 review F-N-2: a finalised manifest carrying a stage in + ``running`` status is a tell that the orchestrator crashed + mid-stage. The verifier surfaces it so an archival audit catches + the orphan.""" + + def test_running_stage_with_completed_final_status_flagged(self): + s0 = PipelineStageState( + name="s0", + index=0, + trainer_type="sft", + status="completed", + input_source="root", + output_model="./out/s0/final_model", + ) + s1 = PipelineStageState( + name="s1", + index=1, + trainer_type="dpo", + status="running", + input_source="chain", + input_model="./out/s0/final_model", + ) + state = PipelineState( + pipeline_run_id="pl_x", + pipeline_config_hash="sha256:abc", + forgelm_version="0.7.0", + started_at="2026-06-15T12:00:00+00:00", + final_status="completed", + stages=[s0, s1], + ) + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + assert any("running" in v and "'s1'" in v for v in violations) + + def test_running_stage_with_in_progress_final_status_is_OK(self): + """A live run is allowed to carry a ``running`` stage — the + verifier only flags ``running`` on a *finalised* manifest.""" + s0 = PipelineStageState( + name="s0", + index=0, + trainer_type="sft", + status="running", + input_source="root", + ) + state = PipelineState( + pipeline_run_id="pl_x", + pipeline_config_hash="sha256:abc", + forgelm_version="0.7.0", + started_at="2026-06-15T12:00:00+00:00", + final_status="in_progress", + stages=[s0], + ) + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + assert all("running" not in v for v in violations) + + +class TestVerifyPipelineManifestAtPath: + """Phase 14 review F-N-6: cover the disk-backed wrapper that the + CLI ``forgelm verify-annex-iv --pipeline`` actually invokes.""" + + def test_missing_manifest_returns_single_violation(self, tmp_path): + from forgelm.compliance import verify_pipeline_manifest_at_path + + violations = verify_pipeline_manifest_at_path(str(tmp_path)) + assert len(violations) == 1 + assert "pipeline_manifest.json not found" in violations[0] + + def test_malformed_manifest_returns_single_violation(self, tmp_path): + from forgelm.compliance import verify_pipeline_manifest_at_path + + manifest_dir = tmp_path / "compliance" + manifest_dir.mkdir() + (manifest_dir / "pipeline_manifest.json").write_text("{not valid json") + violations = verify_pipeline_manifest_at_path(str(tmp_path)) + assert len(violations) == 1 + assert "unreadable" in violations[0] + + def test_completed_stage_with_missing_training_manifest_flagged(self, tmp_path): + """The disk wrapper layers a per-stage training_manifest + existence check on top of the in-memory verifier — the + difference between the two surfaces. This test pins that the + wrapper actually reaches that branch.""" + from forgelm.compliance import verify_pipeline_manifest_at_path + + manifest_dir = tmp_path / "compliance" + manifest_dir.mkdir() + state = _three_stage_state() + # Wire training_manifest paths that don't exist on disk so the + # disk wrapper's existence check has something to fail against. + for s in state.stages: + s.training_manifest = str(tmp_path / s.name / "compliance" / "training_manifest.json") + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + import json as _json + + (manifest_dir / "pipeline_manifest.json").write_text(_json.dumps(manifest)) + violations = verify_pipeline_manifest_at_path(str(tmp_path)) + assert any("training_manifest" in v and "is missing" in v for v in violations) + + class TestVerifyOnPartialFilterRun: """A ``--stage X`` run produces a manifest where other stages have ``status: skipped_by_filter``; verifier should accept it.""" diff --git a/tests/test_pipeline_orchestrator.py b/tests/test_pipeline_orchestrator.py index 18110b89..c99067fc 100644 --- a/tests/test_pipeline_orchestrator.py +++ b/tests/test_pipeline_orchestrator.py @@ -29,8 +29,6 @@ import types from unittest.mock import MagicMock, patch -import pytest - from forgelm.cli._exit_codes import ( EXIT_AWAITING_APPROVAL, EXIT_CONFIG_ERROR, @@ -365,6 +363,38 @@ def test_gated_pending_approval_preserves_downstream_pending(self, tmp_path, mon assert payload["stages"][1]["status"] == "pending" assert payload["stages"][2]["status"] == "pending" + def test_gated_stage_emits_dedicated_stage_gated_audit_event(self, tmp_path, monkeypatch): + """Phase 14 review F-N-1 regression: a stage exiting + ``EXIT_AWAITING_APPROVAL`` must emit a dedicated + ``pipeline.stage_gated`` audit event (not + ``pipeline.stage_completed`` with a sub-field). This lets + dashboard / SIEM filters distinguish the gate flow on the event + name alone.""" + cfg = _three_stage_config(tmp_path) + staging = str(tmp_path / "stage1" / "final_model.staging") + os.makedirs(staging, exist_ok=True) + _install_trainer_mocks(monkeypatch, [TrainResult(success=True, staging_path=staging)]) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + orch.run() + audit_path = os.path.join(orch.paths["root_output_dir"], "audit_log.jsonl") + with open(audit_path) as f: + events = [json.loads(line) for line in f if line.strip()] + gated_events = [e for e in events if e.get("event") == "pipeline.stage_gated"] + assert len(gated_events) == 1, ( + f"Expected exactly one pipeline.stage_gated event, got {[e.get('event') for e in events]!r}" + ) + evt = gated_events[0] + assert evt["stage_name"] == "sft_stage" + assert evt["gate_decision"] == "approval_pending" + assert evt["staging_path"] == staging + # Counter-assert: the legacy stage_completed event must NOT have + # fired for this stage (preventing a future regression where both + # events are emitted in parallel and the dashboard double-counts). + completed_for_sft = [ + e for e in events if e.get("event") == "pipeline.stage_completed" and e.get("stage_name") == "sft_stage" + ] + assert completed_for_sft == [] + # --------------------------------------------------------------------------- # Orchestrator behaviour — --stage filter @@ -455,10 +485,12 @@ def test_stale_config_hash_rejects_resume(self, tmp_path, monkeypatch): orch1 = PipelineOrchestrator(cfg, b"original yaml") orch1.run() # Operator edited the YAML; resume against a different hash. + # Post-Phase-14-review: refusal flows back through ``run()`` as a + # plain return value (not ``sys.exit`` mid-method) so the audit- + # log + summary path runs uniformly on every refusal. orch2 = PipelineOrchestrator(cfg, b"edited yaml") - with pytest.raises(SystemExit) as exc_info: - orch2.run(resume_from="dpo_stage") - assert exc_info.value.code == EXIT_CONFIG_ERROR + code = orch2.run(resume_from="dpo_stage") + assert code == EXIT_CONFIG_ERROR def test_force_resume_accepts_stale_hash_with_warning(self, tmp_path, monkeypatch, caplog): import logging @@ -472,8 +504,39 @@ def test_force_resume_accepts_stale_hash_with_warning(self, tmp_path, monkeypatc with caplog.at_level(logging.WARNING, logger="forgelm.pipeline"): code = orch2.run(resume_from="dpo_stage", force_resume=True) assert code == EXIT_SUCCESS - warning_emitted = any("pipeline_config_hash mismatch" in r.message for r in caplog.records) - assert warning_emitted + + def test_force_resume_emits_audit_event_with_both_hashes(self, tmp_path, monkeypatch): + """Phase 14 review F-B-2 regression: ``--force-resume`` must + emit a ``pipeline.force_resume`` audit event carrying both the + old and new ``pipeline_config_hash`` so a compliance reviewer + can distinguish an operator-approved override from a normal + resume. Pre-fix, only a WARNING log line was emitted — invisible + in the append-only audit-log JSONL stream.""" + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=False, error="x")]) + orch1 = PipelineOrchestrator(cfg, b"yaml v1") + orch1.run() + + _install_trainer_mocks(monkeypatch, [TrainResult(success=True), TrainResult(success=True)]) + orch2 = PipelineOrchestrator(cfg, b"yaml v2 (operator edited)") + orch2.run(resume_from="dpo_stage", force_resume=True) + + audit_path = os.path.join(orch2.paths["root_output_dir"], "audit_log.jsonl") + assert os.path.exists(audit_path) + with open(audit_path) as f: + events = [json.loads(line) for line in f if line.strip()] + force_resume_events = [e for e in events if e.get("event") == "pipeline.force_resume"] + assert len(force_resume_events) == 1, ( + f"Expected exactly one pipeline.force_resume audit event, got " + f"{len(force_resume_events)}: {[e.get('event') for e in events]!r}" + ) + evt = force_resume_events[0] + assert "old_config_hash" in evt + assert "new_config_hash" in evt + assert evt["old_config_hash"] != evt["new_config_hash"] + assert "yaml v1" not in str(evt) and "yaml v2" not in str(evt), ( + "Audit event must record hashes, not raw YAML bytes." + ) def test_resume_with_unknown_stage_name_fails(self, tmp_path, monkeypatch): cfg = _three_stage_config(tmp_path) @@ -521,3 +584,47 @@ def test_webhook_failures_do_not_abort_pipeline(self, tmp_path, monkeypatch): orch = PipelineOrchestrator(cfg, b"yaml") code = orch.run() assert code == EXIT_SUCCESS + + def test_audit_event_failures_do_not_abort_pipeline(self, tmp_path, monkeypatch): + """Phase 14 review F-N-5 regression: ``_audit_event``'s + documented ``except Exception`` swallow path must be reached and + the pipeline must continue. An audit-write failure (read-only + filesystem, disk full, malformed audit logger config) is + documented as best-effort — the orchestrator emits a WARNING and + carries on; the test patches ``AuditLogger.log_event`` to raise + and asserts the run completes with the expected exit code.""" + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=True)] * 3) + + from forgelm.compliance import AuditLogger as _RealAuditLogger + + original_log_event = _RealAuditLogger.log_event + + def _raising_log_event(self, event, **fields): + if event.startswith("pipeline."): + raise OSError("Disk full / read-only audit log") + return original_log_event(self, event, **fields) + + monkeypatch.setattr(_RealAuditLogger, "log_event", _raising_log_event) + orch = PipelineOrchestrator(cfg, b"yaml") + code = orch.run() + assert code == EXIT_SUCCESS + + def test_emit_summary_json_output_is_round_trippable(self, tmp_path, monkeypatch, capsys): + """Phase 14 review F-N-7 regression: the ``output_format='json'`` + branch of ``_emit_summary`` is the dispatcher contract for + ``forgelm --config pipeline.yaml --output-format json`` — it + must print a single JSON object that round-trips through + ``json.loads`` and carries the per-stage payload reviewers + consume programmatically.""" + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=True)] * 3) + orch = PipelineOrchestrator(cfg, b"yaml", output_format="json") + code = orch.run() + assert code == EXIT_SUCCESS + out = capsys.readouterr().out + # The JSON object is the only thing on stdout; logger goes to stderr. + payload = json.loads(out) + assert payload["final_status"] == "completed" + assert len(payload["stages"]) == 3 + assert all(s["status"] == "completed" for s in payload["stages"]) From b8512f0102cd5dc12969bf954ae4c914711138e5 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 14 May 2026 13:52:21 +0300 Subject: [PATCH 05/22] fix: PR #53 Gemini-inline-comment review-response MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three new findings from gemini-code-assist on top of the 0d58f1b batch + a small hardening tweak. All three appear in the PR as inline review comments on forgelm/cli/_pipeline.py. F-G-1 — output_dir collision check in dry-run (medium): Two stages resolving to the same training.output_dir silently overwrite each other's per-stage manifests + checkpoints. The footgun is real for stages that inherit the root training block wholesale. dry_run() now flags the collision (both explicit-override and inherited cases) via EXIT_CONFIG_ERROR before any GPU work. F-G-2 — AttributeError on tampered state (medium): Round out the _load_state_file shape-tamper guard with AttributeError alongside KeyError / TypeError / ValueError. Covers the case where a hand-crafted stages: [null, "foo"] state file makes s.items() raise AttributeError on the non-dict item. F-G-3 — chain-existence leaf check: Already addressed in 0d58f1b (F-1); re-confirmed here. Tests added: - test_dry_run_flags_output_dir_collision — explicit collision via two stage training blocks with the same output_dir. - test_dry_run_flags_inherited_output_dir_collision — two stages that both inherit the root training.output_dir (no per-stage override). CHANGELOG: extended the review-response section with the F-N-1 pipeline.stage_gated event (also new in this round) and the F-G-1 collision guard. Co-Authored-By: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 50 ++++++++++++++++++++++++ forgelm/cli/_pipeline.py | 31 ++++++++++++++- tests/test_pipeline_orchestrator.py | 60 +++++++++++++++++++++++++++++ 3 files changed, 140 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f72f6db1..937ff10f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -75,6 +75,56 @@ _(v0.6.1 dev cycle — entries will land here as PRs merge.)_ validator lives in private helper `_verify_manifest_payload` so the in-memory and disk-bound entry points cannot drift. +### Fixed (Phase 14 review-response — PR #53) + +Addresses the consolidated reviewer-pass findings (3 blocking + +4 significant + Gemini's 3 inline comments) against the initial Phase +14 merge candidate: + +- **F-B-1** — `forgelm --config pipeline.yaml --dry-run` now reaches + the pipeline orchestrator's per-stage validator instead of the + legacy single-stage dry-run path; the `_dispatch.py` ordering was + inverted (no-train-mode ran before the pipeline branch). +- **F-B-2** — `--force-resume` now emits a `pipeline.force_resume` + audit event carrying `old_config_hash` + `new_config_hash` so + compliance reviewers can distinguish an operator-approved override + from a normal resume (was previously a WARNING log only — + invisible in the append-only JSONL stream). +- **F-B-3** — pipeline manifest verifier now compares every chain + stage against its **immediate** predecessor, not the most-recent + stage with an `output_model`. Pre-fix, a manifest where stage N-1 + failed without saving an `output_model` could appear to chain + stage N from stage N-2, masking the broken link. +- **F-F-1 / F-G-3 (Gemini)** — auto-chain existence guard now + checks the resolved model directory itself, not its parent — used + to accept `./stage1/` even when `./stage1/final_model` was + missing. +- **F-F-2** — `--input-model ""` (empty string) is normalised to + `None` at dispatch time so it no longer slips past the truthy + guard and silently overwrites the auto-chained model path with an + empty string. +- **F-F-3** — `_validate_resume_state` now returns its refusal + through the normal `run()` flow instead of `sys.exit`-ing mid-method, + so every refusal path emits the same audit-log finalisation and + the test surface uses one uniform `assert code == EXIT_CONFIG_ERROR` + shape. +- **F-S-2 / F-N-2** — verifier surfaces stages still in `running` + status on a finalised manifest as a `chain_integrity_violation` + candidate (a tell of a crashed orchestrator). Also `_load_state_file` + catches `KeyError` / `TypeError` / `ValueError` / `AttributeError` + around `_deserialise_state` so a tampered state file produces an + actionable config error rather than a Python traceback. +- **F-N-1** — gated stages now emit a dedicated + `pipeline.stage_gated` audit event (was `pipeline.stage_completed` + with a `gate_decision=approval_pending` sub-field). Lets + dashboard / SIEM filters distinguish the gate flow on the event + name alone. +- **F-G-1 (Gemini)** — pipeline dry-run now flags `training.output_dir` + collisions across stages (two stages writing to the same dir would + silently overwrite each other's checkpoints + per-stage Annex IV + manifests). Covers both the explicit-override and inherited-from- + root collision cases. + ### Security - **Webhook / judge / synthetic SSRF guard — DNS-rebinding TOCTOU diff --git a/forgelm/cli/_pipeline.py b/forgelm/cli/_pipeline.py index bb003b91..888fb0e9 100644 --- a/forgelm/cli/_pipeline.py +++ b/forgelm/cli/_pipeline.py @@ -569,9 +569,20 @@ def dry_run(self) -> int: mirroring ``pytest --collectonly``. Returns :data:`EXIT_CONFIG_ERROR (1)` if any stage fails validation, :data:`EXIT_SUCCESS (0)` otherwise. + + Additionally checks that no two stages resolve to the same + ``training.output_dir`` — colliding output dirs would cause one + stage's checkpoints and per-stage Annex-IV manifest to overwrite + another's, breaking the chain of custody silently (Phase 14 + review F-G-1). Because stage-level ``training`` blocks + *inherit* the root ``training`` block wholesale unless the + stage supplies a full override, two stages that both inherit + end up sharing the root's ``output_dir`` — that's the canonical + misconfiguration this guard catches. """ errors: List[str] = [] prev_output: Optional[str] = None + seen_dirs: Dict[str, str] = {} # abs output_dir -> first stage that claimed it for stage in self.pipeline.stages: try: merged = merge_pipeline_stage_config( @@ -583,6 +594,21 @@ def dry_run(self) -> int: errors.append(f"Stage {stage.name!r}: merge failed: {exc}") continue + # Collision check (Phase 14 review F-G-1): every stage must + # write its checkpoints + per-stage manifest into a distinct + # directory. Comparing absolute paths so ``./out`` and + # ``out/`` collide as one. + abs_out = os.path.abspath(merged.training.output_dir) + if abs_out in seen_dirs: + errors.append( + f"Stage {stage.name!r}: training.output_dir collides with stage " + f"{seen_dirs[abs_out]!r} (both resolve to {abs_out!r}); per-stage " + "checkpoints and manifests would overwrite each other. Set a " + "unique 'training.output_dir' in each stage." + ) + else: + seen_dirs[abs_out] = stage.name + # Project the output path the next stage would auto-chain to. prev_output = os.path.join(merged.training.output_dir, "final_model") @@ -697,7 +723,10 @@ def _load_state_file(self) -> Optional[PipelineState]: # with a clear message. try: return _deserialise_state(payload) - except (KeyError, TypeError, ValueError) as e: + except (KeyError, TypeError, ValueError, AttributeError) as e: + # AttributeError covers the case where ``stages`` items aren't + # dicts (e.g. an attacker wrote ``stages: [null, "foo"]``) — + # ``s.items()`` then raises AttributeError on the str / None. logger.warning( "pipeline_state.json at %s is structurally invalid (%s); treating as missing.", path, diff --git a/tests/test_pipeline_orchestrator.py b/tests/test_pipeline_orchestrator.py index c99067fc..9dc066dc 100644 --- a/tests/test_pipeline_orchestrator.py +++ b/tests/test_pipeline_orchestrator.py @@ -228,6 +228,66 @@ def test_dry_run_does_not_write_state_or_manifest(self, tmp_path, monkeypatch): assert not os.path.exists(orch.paths["state_file"]) assert not os.path.exists(orch.paths["manifest_file"]) + def test_dry_run_flags_output_dir_collision(self, tmp_path, monkeypatch): + """Phase 14 review F-G-1 (Gemini): two stages resolving to the + same ``training.output_dir`` would silently overwrite each + other's checkpoints and per-stage Annex-IV manifests, breaking + the chain of custody. Dry-run must surface the collision as + EXIT_CONFIG_ERROR before any GPU work.""" + shared_dir = str(tmp_path / "shared_out") + cfg = ForgeConfig( + model={"name_or_path": "org/base"}, + lora={"r": 8}, + training={"trainer_type": "sft", "output_dir": str(tmp_path)}, + data={"dataset_name_or_path": "org/sft"}, + pipeline={ + "output_dir": str(tmp_path / "pipeline_run"), + "stages": [ + { + "name": "sft_stage", + "training": {"trainer_type": "sft", "output_dir": shared_dir}, + "data": {"dataset_name_or_path": "org/sft"}, + }, + { + "name": "dpo_stage", + "training": {"trainer_type": "dpo", "output_dir": shared_dir}, + "data": {"dataset_name_or_path": "org/dpo"}, + }, + ], + }, + ) + _install_trainer_mocks(monkeypatch, []) + orch = PipelineOrchestrator(cfg, b"yaml") + code = orch.dry_run() + assert code == EXIT_CONFIG_ERROR + + def test_dry_run_flags_inherited_output_dir_collision(self, tmp_path, monkeypatch): + """Two stages that both *inherit* the root ``training`` block + (no per-stage override) end up sharing the root's output_dir + by construction — the most common form of the F-G-1 footgun. + Verify the guard catches it even when the collision is + inherited rather than explicit.""" + cfg = ForgeConfig( + model={"name_or_path": "org/base"}, + lora={"r": 8}, + training={"trainer_type": "sft", "output_dir": str(tmp_path / "root_out")}, + data={"dataset_name_or_path": "org/sft"}, + pipeline={ + "output_dir": str(tmp_path / "pipeline_run"), + # Note: no per-stage training: block — both stages inherit + # the root training.output_dir. Setting data: per stage + # so the merge succeeds, isolating the collision. + "stages": [ + {"name": "s1", "data": {"dataset_name_or_path": "org/d1"}}, + {"name": "s2", "data": {"dataset_name_or_path": "org/d2"}}, + ], + }, + ) + _install_trainer_mocks(monkeypatch, []) + orch = PipelineOrchestrator(cfg, b"yaml") + code = orch.dry_run() + assert code == EXIT_CONFIG_ERROR + # --------------------------------------------------------------------------- # Orchestrator behaviour — happy path From b3c6a45d091ee2e4dfd8d6d8f10a51d745551749 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 14 May 2026 14:02:06 +0300 Subject: [PATCH 06/22] fix: PR #53 Codacy + SonarCloud Quality Gate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses the Codacy "2 new issues" gate and the SonarCloud "Quality Gate failed (E Reliability Rating on New Code)" gate on the Phase 14 PR. BLOCKER bugs (SonarCloud python:S930 — root cause of the Reliability Rating regression): - forgelm/webhook.py _send: pipeline notifiers (notify_pipeline_started/ completed/reverted) were passing stage_count / final_status / stopped_at / stage_name as keyword arguments to a fixed-signature _send method. TypeError raised at runtime, swallowed by the pipeline orchestrator's best-effort try/except → pipeline webhooks silently never fired in production. _send now accepts **extra and merges them into the payload (with a base-field-collision guard so the contract stays stable). MAJOR bugs: - tests/test_pipeline_compliance.py _three_stage_state: removed dead `"completed" if all_completed else "completed"` ternary that always returned the same value (SonarCloud python:S3923). Dropped the unused `all_completed` kwarg too. - tests/test_pipeline_config.py: switched a float == 1.5 assertion to pytest.approx() (SonarCloud python:S1244). MAJOR code smells: - forgelm/cli/_pipeline.py _run_single_stage merge-failure branch: swapped logger.error(... exc) for logger.exception(...) (Sonar S8572) so the traceback lands in the log alongside the error message. - Trainer-crash catch: pulled the long inline noqa rationale out into a preceding comment block so the # noqa: BLE001 marker has nothing trailing it (Sonar python:S7632 disliked the trailing — em-dash prose). Codacy + minor smells: - tests/test_pipeline_orchestrator.py: round = _deserialise_state(...) shadowed the builtin; renamed to round_tripped. - tests/test_pipeline_compliance.py: renamed test_*is_OK to test_*is_ok so the method name matches the standard snake_case regex. - tests/test_pipeline_cli.py _ns: dict(...) constructor → {} literal. Verification: - ruff format + ruff check clean. - 149/149 pipeline + webhook tests pass. - All 11 self-review guards green. Cognitive-complexity CRITICAL code smells (run() at 53, _verify_* at 24, etc.) are deliberately not addressed in this commit — they are Maintainability-rating items and do not affect the failing Reliability Rating gate, and refactoring them safely without behavioural drift would warrant its own PR. Co-Authored-By: Claude Opus 4.7 (1M context) --- forgelm/cli/_pipeline.py | 9 +++++++-- forgelm/webhook.py | 26 ++++++++++++++++++++++++-- tests/test_pipeline_cli.py | 16 ++++++++-------- tests/test_pipeline_compliance.py | 6 +++--- tests/test_pipeline_config.py | 2 +- tests/test_pipeline_orchestrator.py | 10 +++++----- 6 files changed, 48 insertions(+), 21 deletions(-) diff --git a/forgelm/cli/_pipeline.py b/forgelm/cli/_pipeline.py index 888fb0e9..96e35050 100644 --- a/forgelm/cli/_pipeline.py +++ b/forgelm/cli/_pipeline.py @@ -852,7 +852,7 @@ def _run_single_stage( ) except Exception as exc: # noqa: BLE001 — config-merge failure is a fatal stage error; routes through the standard config-error exit code. stage_state.error = f"Config merge failed: {exc}" - logger.error("Stage %r config merge failed: %s", stage.name, exc) + logger.exception("Stage %r config merge failed.", stage.name) return EXIT_CONFIG_ERROR stage_state.input_model = stage_cfg.model.name_or_path @@ -904,7 +904,12 @@ def _run_single_stage( dataset = prepare_dataset(stage_cfg, tokenizer) trainer = ForgeTrainer(model=model, tokenizer=tokenizer, config=stage_cfg, dataset=dataset) result = trainer.train() - except Exception as exc: # noqa: BLE001 — top-of-stage catch. ForgeTrainer.train() crosses every concern (HF load, dataset, TRL, safety, judge, audit, compliance, webhook); any uncaught exception must surface as a per-stage failure rather than a Python traceback that strands the pipeline state. + # Top-of-stage catch. ForgeTrainer.train() crosses every + # concern (HF load, dataset, TRL, safety, judge, audit, + # compliance, webhook); any uncaught exception must surface as + # a per-stage failure rather than a Python traceback that + # strands the pipeline state. + except Exception as exc: # noqa: BLE001 stage_state.error = f"Trainer crashed: {exc}" logger.exception("Stage %r trainer crashed.", stage.name) return EXIT_TRAINING_ERROR diff --git a/forgelm/webhook.py b/forgelm/webhook.py index 1286c096..ded6f5db 100644 --- a/forgelm/webhook.py +++ b/forgelm/webhook.py @@ -1,7 +1,7 @@ import json import logging import os -from typing import Dict, Optional +from typing import Any, Dict, Optional from urllib.parse import urlparse import requests @@ -173,7 +173,20 @@ def _send( metrics: Optional[Dict[str, float]] = None, reason: Optional[str] = None, model_path: Optional[str] = None, + **extra: Any, ) -> None: + """Build + post the webhook payload. + + ``**extra`` carries event-specific fields the + ``notify_pipeline_*`` methods forward (stage_count, final_status, + stopped_at, stage_name, …) — Phase 14 review-response fix: pre- + fix the pipeline notifiers passed unknown kwargs to a fixed + signature, the resulting ``TypeError`` was swallowed by the + orchestrator's best-effort try/except, and pipeline webhooks + silently never fired. Extras are merged into the payload under + their original key names so existing Slack / Teams receivers + that pick fields by name keep working. + """ url = self._resolve_url() if not url: return @@ -188,7 +201,7 @@ def _send( # ``model_path`` is included only for ``approval.required`` events; # we add the key unconditionally (even as None) to keep the schema # stable so downstream consumers can rely on its presence. - payload = { + payload: Dict[str, Any] = { "event": event, "run_name": run_name, "status": status, @@ -198,6 +211,15 @@ def _send( # Slack-compatible formatting (receivers can ignore) "attachments": [{"title": title, "text": text, "color": color}], } + # Merge event-specific extras (pipeline.* events carry + # stage_count / final_status / stopped_at / stage_name). Drop + # any extra whose key collides with a base-payload field so the + # contract stays stable; we don't expect collisions in practice + # since the pipeline notifier names are disjoint, but the guard + # makes the merge order explicit. + for key, value in extra.items(): + if key not in payload: + payload[key] = value self._post_payload(url, payload, event) diff --git a/tests/test_pipeline_cli.py b/tests/test_pipeline_cli.py index 5a296eae..e23c712f 100644 --- a/tests/test_pipeline_cli.py +++ b/tests/test_pipeline_cli.py @@ -54,14 +54,14 @@ def _three_stage_cfg(tmp_path): def _ns(**overrides): """Build an argparse.Namespace with the orchestrator-relevant fields.""" - defaults = dict( - stage=None, - resume_from=None, - force_resume=False, - input_model=None, - output_format="text", - dry_run=False, - ) + defaults = { + "stage": None, + "resume_from": None, + "force_resume": False, + "input_model": None, + "output_format": "text", + "dry_run": False, + } defaults.update(overrides) return argparse.Namespace(**defaults) diff --git a/tests/test_pipeline_compliance.py b/tests/test_pipeline_compliance.py index 4be11758..90af29fe 100644 --- a/tests/test_pipeline_compliance.py +++ b/tests/test_pipeline_compliance.py @@ -33,13 +33,13 @@ def _root_with_compliance() -> ForgeConfig: ) -def _three_stage_state(*, all_completed: bool = True) -> PipelineState: +def _three_stage_state() -> PipelineState: """Build a representative 3-stage state for happy-path schema tests.""" s1 = PipelineStageState( name="sft_stage", index=0, trainer_type="sft", - status="completed" if all_completed else "completed", + status="completed", input_model="org/base", input_source="root", output_model="./out/stage1/final_model", @@ -382,7 +382,7 @@ def test_running_stage_with_completed_final_status_flagged(self): violations = _verify_manifest_payload(manifest) assert any("running" in v and "'s1'" in v for v in violations) - def test_running_stage_with_in_progress_final_status_is_OK(self): + def test_running_stage_with_in_progress_final_status_is_ok(self): """A live run is allowed to carry a ``running`` stage — the verifier only flags ``running`` on a *finalised* manifest.""" s0 = PipelineStageState( diff --git a/tests/test_pipeline_config.py b/tests/test_pipeline_config.py index 63575572..584e519f 100644 --- a/tests/test_pipeline_config.py +++ b/tests/test_pipeline_config.py @@ -273,7 +273,7 @@ def test_stage_evaluation_block_wholesale_replaces_root(self): merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model") assert merged.evaluation is not None assert merged.evaluation.auto_revert is True - assert merged.evaluation.max_acceptable_loss == 1.5 + assert merged.evaluation.max_acceptable_loss == pytest.approx(1.5) def test_pipeline_section_stripped_from_merged_config(self): """The orchestrator hands the merged ForgeConfig to a single-stage diff --git a/tests/test_pipeline_orchestrator.py b/tests/test_pipeline_orchestrator.py index 9dc066dc..8ec4e203 100644 --- a/tests/test_pipeline_orchestrator.py +++ b/tests/test_pipeline_orchestrator.py @@ -181,11 +181,11 @@ def test_serialise_deserialise(self): ], ) payload = _serialise_state(state) - round = _deserialise_state(payload) - assert round.pipeline_run_id == state.pipeline_run_id - assert len(round.stages) == 2 - assert round.stages[0].name == "s1" - assert round.stages[0].status == "completed" + round_tripped = _deserialise_state(payload) + assert round_tripped.pipeline_run_id == state.pipeline_run_id + assert len(round_tripped.stages) == 2 + assert round_tripped.stages[0].name == "s1" + assert round_tripped.stages[0].status == "completed" def test_deserialise_tolerates_unknown_keys(self): """Future-forward compatibility: a newer manifest schema with From a65d3c98a94f8bb1ad0d90c04afec1f91e72f77d Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 14 May 2026 14:16:01 +0300 Subject: [PATCH 07/22] refactor: PR #53 SonarCloud cognitive complexity (python:S3776) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses the 5 remaining CRITICAL code-smell findings on the Phase 14 PR by extracting helper methods. Behaviour is preserved end-to-end — the 227-test sweep (pipeline + CLI + verify-annex-iv + webhook + library API) passes byte-identical to b3c6a45. forgelm/cli/_dispatch.py:166 _main_inner (17 → ≤15): Extracted _dispatch_legacy_data_audit (the --data-audit alias path with its DeprecationWarning + audit-log breadcrumb + delegation to _run_data_audit) and _dispatch_pipeline_mode (the YAML re-read + run_pipeline_from_args dispatch). _main_inner is now a thin argparse-then-route entry point. forgelm/cli/_pipeline.py:289 run() (53 → ≤15): This was the biggest one. Split into six helpers, each with one cohesive responsibility: - _prepare_resume_or_init_state — loads existing state for resume or builds a fresh one. Returns (state, None) on success or (None, EXIT_CONFIG_ERROR) for invalid resume paths. - _resolve_resume_skiplist — walks state.stages before the resume point and builds the "already-completed on disk" skip list. - _resolve_stage_filter_setup — validates --stage + seeds the auto-chain prev_output when filtering to a non-first stage. - _execute_one_stage — per-stage setup (started_at / status / trainer_type / save) + delegates to _run_single_stage + records duration. - _handle_stage_outcome — translates the per-stage exit code into the next-state transition (gated / completed / failed) + audit/webhook event + chain_broken flag. - _finalise_pipeline — phase 5+6: terminal state + pipeline.completed. The retained loop is just dispatch + skip/done bookkeeping; the branching that drove the cognitive-complexity score lives in the helpers, each at single-digit complexity. forgelm/cli/_pipeline.py:821 _run_single_stage (18 → ≤15): Extracted: - _resolve_input_source — pure function returning the InputSourceLiteral via the documented priority ladder (CLI override > stage explicit > chain > root). - _merge_and_validate_stage — merge_pipeline_stage_config wrap + chain-presence guard. Returns the merged ForgeConfig or None on a config-error path (caller maps to EXIT_CONFIG_ERROR). - _invoke_trainer — heavy-import-guarded ForgeTrainer lifecycle. Returns the TrainResult or None on a runtime/import failure. _run_single_stage is now linear: resolve source → merge → audit start → invoke trainer → record results. forgelm/cli/subcommands/_verify_annex_iv.py:188 (27 → ≤15): Extracted: - _run_pipeline_mode — --pipeline flag branch (manifest + chain- integrity violations). - _verify_artefact_and_handle_io_errors — the FileNotFoundError / IsADirectoryError / JSONDecodeError / OSError → exit-code mapping that handles the single-artefact path's I/O failures. - _print_artefact_result — text + JSON output formatting. _run_verify_annex_iv_cmd is now: validate path → branch on mode → delegate. forgelm/compliance.py:1057 _verify_manifest_payload (24 → ≤15): Lifted the per-stage chain-integrity check into _check_chain_link (pure function, idx-based for "compare against immediate predecessor") and the stopped_at + running-stage finalised checks into _check_status_consistency. Top-level keys list lifted to module-level constant _PIPELINE_MANIFEST_REQUIRED_KEYS. All 11 self-review guards remain green; ruff format + check clean; 227 / 227 tests pass on the broader pipeline + CLI sweep. Co-Authored-By: Claude Opus 4.7 (1M context) --- forgelm/cli/_dispatch.py | 181 +++--- forgelm/cli/_pipeline.py | 667 ++++++++++++-------- forgelm/cli/subcommands/_verify_annex_iv.py | 167 ++--- forgelm/compliance.py | 141 +++-- 4 files changed, 677 insertions(+), 479 deletions(-) diff --git a/forgelm/cli/_dispatch.py b/forgelm/cli/_dispatch.py index 94b88e6a..d874cbaa 100644 --- a/forgelm/cli/_dispatch.py +++ b/forgelm/cli/_dispatch.py @@ -163,6 +163,97 @@ def main(): sys.exit(EXIT_TRAINING_ERROR) +def _dispatch_legacy_data_audit(args) -> None: + """Handle ``forgelm --data-audit PATH`` (deprecated flag). + + Pulled out of ``_main_inner`` for Sonar python:S3776 cognitive- + complexity hygiene. Emits the standard deprecation + ``DeprecationWarning`` + an audit-log breadcrumb (best-effort), then + delegates to the canonical ``_run_data_audit`` and ``sys.exit``s. + Calls back to ``_main_inner`` are impossible — this function always + terminates the process. + """ + json_output = args.output_format == "json" + log_level = "WARNING" if args.quiet else args.log_level + _setup_logging(log_level, json_format=json_output) + # Phase 13 (Faz 13): emit a structured Python ``DeprecationWarning`` + # so `pytest -W error::DeprecationWarning` and `python -Wd` tooling + # surface it, plus an append-only audit-log event so operators who + # only read the JSONL trail see the migration signal too. Cadence + # for the v0.7.0 removal follows the "Deprecation cadence" section + # in ``docs/standards/release.md`` (one-minor warning window + # minimum). + warnings.warn( + "`forgelm --data-audit PATH` is deprecated and will be removed " + "in v0.7.0. Use the `forgelm audit PATH` subcommand instead — " + "same behaviour, same output. " + "See docs/standards/release.md#deprecation-cadence for the removal timeline.", + DeprecationWarning, + stacklevel=2, + ) + # Audit-log event: append-only Article 12 record of the legacy + # invocation so compliance reviewers can prove the operator was + # warned. Best-effort — a read-only output dir must not abort the + # actual audit run. + legacy_target = args.output or "./audit" + try: + from ..compliance import AuditLogger + from ..config import ConfigError + + AuditLogger(legacy_target).log_event( + "cli.legacy_flag_invoked", + flag="--data-audit", + replacement="forgelm audit", + version="v0.7.0 removal", + ) + except (OSError, ConfigError) as audit_exc: + logger.warning( + "Failed to emit deprecation audit event to %s: %s", + legacy_target, + audit_exc, + ) + # Late import via the package facade so monkeypatched + # ``forgelm.cli._run_data_audit`` references resolve correctly. + from forgelm import cli as _cli_facade + + _cli_facade._run_data_audit( + args.data_audit, + args.output, + args.output_format, + ) + sys.exit(EXIT_SUCCESS) + + +def _dispatch_pipeline_mode(config, args) -> None: + """Handle a config that carries a ``pipeline:`` block. + + Pulled out of ``_main_inner`` for Sonar python:S3776 cognitive- + complexity hygiene. Re-reads the YAML *bytes* (not the parsed + semantic content — regulators audit the on-disk artefact) and + routes to :func:`forgelm.cli._pipeline.run_pipeline_from_args`. + Always terminates the process via ``sys.exit``. + + Layering rule (Phase 14 F-B-1): this branch MUST run before + ``_maybe_run_no_train_mode``. The single-stage no-train modes + (``--dry-run``, ``--fit-check``, ``--benchmark-only``, + ``--merge``, ``--generate-data``, ``--compliance-export``) + ``sys.exit`` internally; if they ran first on a pipeline config, + the orchestrator would never reach the multi-stage validation path + and the documented ``--dry-run`` per-stage error-collection + contract would be silently violated. + """ + from ._pipeline import run_pipeline_from_args + + try: + with open(args.config, "rb") as f: + pipeline_yaml_bytes = f.read() + except OSError as e: + logger.error("Failed to re-read pipeline YAML for hashing: %s", e) + sys.exit(EXIT_CONFIG_ERROR) + + sys.exit(run_pipeline_from_args(config, pipeline_yaml_bytes, args)) + + def _main_inner() -> None: """Entry-point body — extracted so ``main`` can wrap a single ``KeyboardInterrupt`` handler around every step from argparse to @@ -181,70 +272,9 @@ def _main_inner() -> None: # Run before the config-required check so operators can audit raw data # without writing a YAML. Phase 11.5 promoted the same code path to # `forgelm audit PATH` (a real subcommand); the legacy flag is preserved - # as an alias and slated for removal in v0.7.0 — Phase 13 wires up the - # deprecation signalling at this dispatch site. + # as an alias and slated for removal in v0.7.0. if getattr(args, "data_audit", None): - json_output = args.output_format == "json" - log_level = "WARNING" if args.quiet else args.log_level - _setup_logging(log_level, json_format=json_output) - # Phase 13 (Faz 13): emit a structured Python ``DeprecationWarning`` - # so `pytest -W error::DeprecationWarning` and `python -Wd` tooling - # surface it, plus an append-only audit-log event so operators who - # only read the JSONL trail see the migration signal too. Cadence - # for the v0.7.0 removal follows the "Deprecation cadence" section - # in ``docs/standards/release.md`` (one-minor warning window - # minimum). - warnings.warn( - "`forgelm --data-audit PATH` is deprecated and will be removed " - "in v0.7.0. Use the `forgelm audit PATH` subcommand instead — " - "same behaviour, same output. " - "See docs/standards/release.md#deprecation-cadence for the removal timeline.", - DeprecationWarning, - stacklevel=2, - ) - # Audit-log event: append-only Article 12 record of the legacy - # invocation so compliance reviewers can prove the operator was - # warned. The logger writes to ``/audit_log.jsonl`` next to - # the report this run is about to produce. Resolve the target the - # same way ``_run_data_audit`` does (default ``./audit``) so the - # event lands in the same directory as the report. - legacy_target = args.output or "./audit" - try: - from ..compliance import AuditLogger - from ..config import ConfigError - - AuditLogger(legacy_target).log_event( - "cli.legacy_flag_invoked", - flag="--data-audit", - replacement="forgelm audit", - version="v0.7.0 removal", - ) - except (OSError, ConfigError) as audit_exc: - # Non-fatal: the audit log is a best-effort telemetry record - # for the deprecation notice. The DeprecationWarning has - # already fired; the audit run itself must still proceed — - # losing the legacy-flag breadcrumb is preferable to aborting - # a working pipeline because the output dir is read-only. - # Logged at WARNING (not debug) per - # docs/standards/error-handling.md "Best-effort artefact - # carve-out" hygiene item 2: audit-trail visibility trumps - # log noise for low-frequency, structured events like a - # legacy-flag invocation against a read-only output dir. - logger.warning( - "Failed to emit deprecation audit event to %s: %s", - legacy_target, - audit_exc, - ) - # Late import via the package facade so monkeypatched - # ``forgelm.cli._run_data_audit`` references resolve correctly. - from forgelm import cli as _cli_facade - - _cli_facade._run_data_audit( - args.data_audit, - args.output, - args.output_format, - ) - sys.exit(EXIT_SUCCESS) + _dispatch_legacy_data_audit(args) _maybe_run_wizard(args) @@ -262,31 +292,8 @@ def _main_inner() -> None: config = _load_config_or_exit(args.config, json_output) _apply_offline_flag(config, args.offline) - # Phase 14 — multi-stage pipeline branch MUST run BEFORE - # ``_maybe_run_no_train_mode``. The single-stage no-train modes - # (``--dry-run``, ``--fit-check``, ``--benchmark-only``, ``--merge``, - # ``--generate-data``, ``--compliance-export``) call ``sys.exit`` - # internally; if they ran first on a pipeline config, the orchestrator - # would never reach the multi-stage validation path and the documented - # ``--dry-run`` per-stage error collection contract (Phase 14 Task 4) - # would be silently violated — the operator would see the legacy - # single-stage dry-run summary instead. if config.pipeline is not None: - from ._pipeline import run_pipeline_from_args - - # Re-read the YAML *bytes* so the orchestrator can hash them for - # the ``--resume-from`` stale-config guard. We deliberately - # don't trust ``yaml.dump(config.model_dump())`` here — comments - # and key order would be lost, and the hash is meant to prove - # the on-disk artefact, not the parsed semantic content. - try: - with open(args.config, "rb") as f: - pipeline_yaml_bytes = f.read() - except OSError as e: - logger.error("Failed to re-read pipeline YAML for hashing: %s", e) - sys.exit(EXIT_CONFIG_ERROR) - - sys.exit(run_pipeline_from_args(config, pipeline_yaml_bytes, args)) + _dispatch_pipeline_mode(config, args) # Single-stage path — unchanged from v0.6.0. _maybe_run_no_train_mode(config, args) diff --git a/forgelm/cli/_pipeline.py b/forgelm/cli/_pipeline.py index 96e35050..37b72957 100644 --- a/forgelm/cli/_pipeline.py +++ b/forgelm/cli/_pipeline.py @@ -286,6 +286,267 @@ def __init__( # -- run-mode entry points ------------------------------------------------ + def _prepare_resume_or_init_state( + self, + *, + resume_from: Optional[str], + force_resume: bool, + ) -> "tuple[Optional[PipelineState], Optional[int]]": + """Load + validate state for ``--resume-from`` or build a fresh state. + + Returns ``(state, None)`` on success; ``(None, EXIT_CONFIG_ERROR)`` + when the resume path is invalid (missing state file or stale-hash + refused without ``--force-resume``). Extracted from + :meth:`run` for Sonar python:S3776 cognitive-complexity hygiene. + """ + existing = self._load_state_file() + if resume_from is not None and existing is None: + logger.error( + "Cannot --resume-from %r: no pipeline_state.json found at %s. Run the pipeline once first.", + resume_from, + self.paths["state_file"], + ) + return None, EXIT_CONFIG_ERROR + if resume_from is not None: + refusal = self._validate_resume_state(existing, force=force_resume) + if refusal is not None: + return None, EXIT_CONFIG_ERROR + return existing, None + return self._init_state(), None + + def _resolve_resume_skiplist( + self, + *, + state: PipelineState, + resume_from: str, + ) -> "tuple[Optional[List[str]], Optional[int]]": + """Compute the list of already-completed stages to skip on resume. + + Returns ``(skiplist, None)`` on success or ``(None, + EXIT_CONFIG_ERROR)`` when ``resume_from`` names a stage that + doesn't exist. Stages before the resume index whose status is + ``completed`` and whose ``output_model`` directory exists on + disk are kept-as-is. Extracted from :meth:`run` for Sonar + python:S3776 cognitive-complexity hygiene. + """ + try: + resume_idx = next(i for i, s in enumerate(self.pipeline.stages) if s.name == resume_from) + except StopIteration: + logger.error( + "Cannot --resume-from %r: stage name not found in pipeline (valid names: %s).", + resume_from, + ", ".join(s.name for s in self.pipeline.stages), + ) + return None, EXIT_CONFIG_ERROR + stages_to_skip_completed: List[str] = [] + for prior_state in state.stages[:resume_idx]: + if ( + prior_state.status == "completed" + and prior_state.output_model + and os.path.isdir(prior_state.output_model) + ): + stages_to_skip_completed.append(prior_state.name) + logger.info( + "Resuming: stage %r already completed at %s; skipping.", + prior_state.name, + prior_state.output_model, + ) + return stages_to_skip_completed, None + + def _resolve_stage_filter_setup( + self, + *, + stage_filter: str, + input_model_override: Optional[str], + ) -> "tuple[Optional[str], Optional[int]]": + """Validate ``--stage `` + seed the auto-chain on a non-first stage. + + Returns ``(prev_output_for_filter, None)`` on success or + ``(None, EXIT_CONFIG_ERROR)`` when the stage name is unknown or + the previous stage's output directory is missing (and no + ``--input-model`` override was supplied to bypass the chain). + Extracted from :meth:`run` for Sonar python:S3776 cognitive- + complexity hygiene. + """ + if not any(s.name == stage_filter for s in self.pipeline.stages): + logger.error( + "Cannot --stage %r: stage name not found in pipeline (valid names: %s).", + stage_filter, + ", ".join(s.name for s in self.pipeline.stages), + ) + return None, EXIT_CONFIG_ERROR + + if input_model_override is not None: + return None, None + + filter_idx = next(i for i, s in enumerate(self.pipeline.stages) if s.name == stage_filter) + if filter_idx == 0: + return None, None + + prev_stage = self.pipeline.stages[filter_idx - 1] + prev_merged = merge_pipeline_stage_config(self.root_cfg, prev_stage, prev_output_model=None) + candidate = os.path.join(prev_merged.training.output_dir, "final_model") + if not os.path.isdir(candidate): + logger.error( + "Stage %r requires the previous stage's output at %s; " + "pass --input-model to override or run the full pipeline first.", + stage_filter, + candidate, + ) + return None, EXIT_CONFIG_ERROR + return candidate, None + + def _handle_stage_outcome( + self, + *, + stage: PipelineStage, + stage_state: PipelineStageState, + exit_code: int, + state: PipelineState, + worst_exit: int, + ) -> "tuple[int, Optional[str], bool, bool]": + """Persist + audit + webhook the outcome of a single executed stage. + + Returns ``(worst_exit, new_prev_output, chain_broken, should_break)``: + + - ``worst_exit`` — the updated highest-priority exit code so far. + - ``new_prev_output`` — the auto-chain seed for the next stage + (``stage_state.output_model`` on success, ``None`` otherwise). + - ``chain_broken`` — ``True`` for any non-success outcome; the + caller marks downstream stages skipped on it. + - ``should_break`` — ``True`` only for ``EXIT_AWAITING_APPROVAL``; + the caller exits the stage loop immediately so a subsequent + ``--resume-from`` picks up the remaining stages post-approval. + + Extracted from :meth:`run` for Sonar python:S3776 cognitive- + complexity hygiene. + """ + if exit_code == EXIT_AWAITING_APPROVAL: + # Human-approval gate fired. Pipeline exits; remaining + # stages stay ``pending`` so a subsequent ``--resume-from`` + # picks them up after operator action. + stage_state.status = "gated_pending_approval" + state.final_status = "gated_pending_approval" + state.stopped_at = stage.name + state.finished_at = _utc_iso_now() + new_worst = max(worst_exit, EXIT_AWAITING_APPROVAL) + self._save_state(state) + # Phase 14 review F-N-1: dedicated ``pipeline.stage_gated`` + # event (instead of ``pipeline.stage_completed`` with a + # ``gate_decision=approval_pending`` sub-field). Lets + # dashboard / SIEM rules filter on the event name alone. + self._audit_event( + "pipeline.stage_gated", + pipeline_run_id=state.pipeline_run_id, + stage_name=stage.name, + gate_decision="approval_pending", + staging_path=stage_state.output_model, + ) + return new_worst, None, True, True + + if exit_code == EXIT_SUCCESS: + stage_state.status = "completed" + stage_state.gate_decision = "passed" + self._save_state(state) + self._audit_event( + "pipeline.stage_completed", + pipeline_run_id=state.pipeline_run_id, + stage_name=stage.name, + gate_decision="passed", + metrics=stage_state.metrics, + ) + return worst_exit, stage_state.output_model, False, False + + # Anything else (EXIT_TRAINING_ERROR / EXIT_EVAL_FAILURE): + # chain stops; downstream stages will skip with + # ``skipped_due_to_prior_revert``. + stage_state.status = "failed" + stage_state.gate_decision = "failed" + state.stopped_at = stage.name + new_worst = max(worst_exit, exit_code) + self._save_state(state) + self._audit_event( + "pipeline.stage_reverted" if stage_state.auto_revert_triggered else "pipeline.stage_completed", + pipeline_run_id=state.pipeline_run_id, + stage_name=stage.name, + gate_decision="failed", + auto_revert_triggered=stage_state.auto_revert_triggered, + ) + self._notify_pipeline( + "reverted" if stage_state.auto_revert_triggered else "stage_failed", + state, + stage_state=stage_state, + ) + return new_worst, None, True, False + + def _execute_one_stage( + self, + *, + index: int, + stage: PipelineStage, + state: PipelineState, + prev_output: Optional[str], + stage_filter: Optional[str], + input_model_override: Optional[str], + ) -> int: + """Setup + invoke a single stage's trainer. Returns the per-stage exit code. + + Caller is responsible for translating the returned exit code + into the next-state transition via :meth:`_handle_stage_outcome`. + Extracted from :meth:`run` for Sonar python:S3776 hygiene. + """ + stage_state = state.stages[index] + stage_state.index = index + stage_state.name = stage.name + stage_state.trainer_type = ( + stage.training.trainer_type if stage.training else self.root_cfg.training.trainer_type + ) + stage_state.started_at = _utc_iso_now() + stage_state.status = "running" + self._save_state(state) + + # ``--input-model`` attaches to a single filtered stage only. + override_for_this_stage = ( + input_model_override if (stage_filter is not None and stage.name == stage_filter) else None + ) + + exit_code = self._run_single_stage( + stage=stage, + stage_state=stage_state, + prev_output=prev_output, + input_model_override=override_for_this_stage, + state=state, + ) + stage_state.exit_code = exit_code + stage_state.finished_at = _utc_iso_now() + try: + start_dt = datetime.fromisoformat(stage_state.started_at) + end_dt = datetime.fromisoformat(stage_state.finished_at) + stage_state.duration_seconds = (end_dt - start_dt).total_seconds() + except (TypeError, ValueError): + stage_state.duration_seconds = None + return exit_code + + def _finalise_pipeline(self, state: PipelineState, worst_exit: int) -> None: + """Mark the pipeline finished + emit the terminal events. + + Mirrors the run-loop tail. Extracted from :meth:`run` for + Sonar python:S3776 cognitive-complexity hygiene. + """ + if state.final_status == "in_progress": + state.final_status = "completed" if worst_exit == EXIT_SUCCESS else "stopped_at_stage" + state.finished_at = _utc_iso_now() + self._save_state(state) + + if state.final_status in ("completed", "stopped_at_stage"): + self._audit_event( + "pipeline.completed", + pipeline_run_id=state.pipeline_run_id, + final_status=state.final_status, + stopped_at=state.stopped_at, + ) + self._notify_pipeline("completed", state) + def run( self, *, @@ -311,91 +572,29 @@ def run( stage's exit code. """ # 1. Load (or initialise) state. - existing = self._load_state_file() - if resume_from is not None and existing is None: - logger.error( - "Cannot --resume-from %r: no pipeline_state.json found at %s. Run the pipeline once first.", - resume_from, - self.paths["state_file"], - ) - return EXIT_CONFIG_ERROR - if resume_from is not None: - refusal = self._validate_resume_state(existing, force=force_resume) - if refusal is not None: - # Library-style refusal — return the exit code through - # the normal ``run()`` flow rather than ``sys.exit``-ing - # mid-method. Matches the surrounding error paths - # (missing state file, unknown stage name) and keeps the - # test surface uniform on ``assert code == EXIT_CONFIG_ERROR``. - return EXIT_CONFIG_ERROR - state = existing - else: - state = self._init_state() + state, state_err = self._prepare_resume_or_init_state(resume_from=resume_from, force_resume=force_resume) + if state is None: + return state_err # type: ignore[return-value] # 2. Determine which stages to execute. stages_to_skip_completed: List[str] = [] if resume_from is not None: - try: - resume_idx = next(i for i, s in enumerate(self.pipeline.stages) if s.name == resume_from) - except StopIteration: - logger.error( - "Cannot --resume-from %r: stage name not found in pipeline (valid names: %s).", - resume_from, - ", ".join(s.name for s in self.pipeline.stages), - ) - return EXIT_CONFIG_ERROR - # Mark anything before the resume point that's already - # ``completed`` on disk as kept-as-is. - for prior_state in state.stages[:resume_idx]: - if ( - prior_state.status == "completed" - and prior_state.output_model - and os.path.isdir(prior_state.output_model) - ): - stages_to_skip_completed.append(prior_state.name) - logger.info( - "Resuming: stage %r already completed at %s; skipping.", - prior_state.name, - prior_state.output_model, - ) + skiplist, skiplist_err = self._resolve_resume_skiplist(state=state, resume_from=resume_from) + if skiplist is None: + return skiplist_err # type: ignore[return-value] + stages_to_skip_completed = skiplist - if stage_filter is not None and not any(s.name == stage_filter for s in self.pipeline.stages): - logger.error( - "Cannot --stage %r: stage name not found in pipeline (valid names: %s).", - stage_filter, - ", ".join(s.name for s in self.pipeline.stages), - ) - return EXIT_CONFIG_ERROR - - # 2a. ``--stage `` on a non-first stage requires the previous - # stage's output to already exist on disk (auto-chain resolution), - # unless ``--input-model`` provides an explicit override. Refuse - # to silently fall back to the root ``model.name_or_path`` — that - # would produce a chain whose stage N is trained on the root base - # model, not on the previous stage's output, contradicting every - # operator's expectation of what ``--stage dpo_stage`` means. + # 2a. ``--stage`` validation + auto-chain seeding. prev_output_for_filter: Optional[str] = None - if stage_filter is not None and input_model_override is None: - filter_idx = next(i for i, s in enumerate(self.pipeline.stages) if s.name == stage_filter) - if filter_idx > 0: - prev_stage = self.pipeline.stages[filter_idx - 1] - # Project the previous stage's would-be output path by - # running it through the merge helper with no chain (so - # we get its declared ``training.output_dir`` honestly). - prev_merged = merge_pipeline_stage_config(self.root_cfg, prev_stage, prev_output_model=None) - candidate = os.path.join(prev_merged.training.output_dir, "final_model") - if not os.path.isdir(candidate): - logger.error( - "Stage %r requires the previous stage's output at %s; " - "pass --input-model to override or run the full " - "pipeline first.", - stage_filter, - candidate, - ) - return EXIT_CONFIG_ERROR - prev_output_for_filter = candidate + if stage_filter is not None: + seed, filter_err = self._resolve_stage_filter_setup( + stage_filter=stage_filter, input_model_override=input_model_override + ) + if filter_err is not None: + return filter_err + prev_output_for_filter = seed - # 3. Emit pipeline.started + webhook (only on a fresh run). + # 3. Emit pipeline.started + webhook (fresh runs only). if resume_from is None: self._audit_event( "pipeline.started", @@ -406,31 +605,26 @@ def run( ) self._notify_pipeline("started", state) - # 4. Iterate stages. + # 4. Iterate stages. The loop body delegates the actual + # per-stage work to ``_execute_one_stage`` + the outcome + # transition to ``_handle_stage_outcome``, so this method's + # cognitive complexity stays bounded. worst_exit: int = EXIT_SUCCESS - # When ``--stage X`` is set on a non-first stage, seed ``prev_output`` - # with the previous stage's on-disk output path resolved above so the - # filtered stage auto-chains correctly. In every other mode, the - # initial ``prev_output`` is None and the iteration accumulates it - # naturally from each completed stage. prev_output: Optional[str] = prev_output_for_filter - chain_broken: bool = False # set when a stage auto-reverts; later stages skip + chain_broken: bool = False for i, stage in enumerate(self.pipeline.stages): stage_state = state.stages[i] - # --stage filter: only run the named stage; mark others skipped_by_filter. if stage_filter is not None and stage.name != stage_filter: stage_state.status = "skipped_by_filter" stage_state.skipped_reason = f"--stage {stage_filter!r} was specified; only that stage runs." self._save_state(state) continue - # --resume-from: skip already-completed stages with on-disk output. if stage.name in stages_to_skip_completed: prev_output = stage_state.output_model continue - # Chain-broken (prior stage reverted): mark remaining as skipped_due_to_prior_revert. if chain_broken: stage_state.status = "skipped_due_to_prior_revert" stage_state.skipped_reason = ( @@ -439,127 +633,31 @@ def run( self._save_state(state) continue - # Run this stage. - stage_state.index = i - stage_state.name = stage.name - stage_state.trainer_type = ( - stage.training.trainer_type if stage.training else self.root_cfg.training.trainer_type - ) - stage_state.started_at = _utc_iso_now() - stage_state.status = "running" - self._save_state(state) - - # Resolve the input-model override only for the filtered stage - # (a CLI ``--input-model`` flag attaches to a single stage run). - override_for_this_stage = ( - input_model_override if (stage_filter is not None and stage.name == stage_filter) else None + exit_code = self._execute_one_stage( + index=i, + stage=stage, + state=state, + prev_output=prev_output, + stage_filter=stage_filter, + input_model_override=input_model_override, ) - - exit_code = self._run_single_stage( + worst_exit, new_prev_output, chain_broken_now, should_break = self._handle_stage_outcome( stage=stage, stage_state=stage_state, - prev_output=prev_output, - input_model_override=override_for_this_stage, + exit_code=exit_code, state=state, + worst_exit=worst_exit, ) - stage_state.exit_code = exit_code - stage_state.finished_at = _utc_iso_now() - - # Duration calc — wrap in try/except to be robust against - # missing started_at on a constructor edge case. - try: - start_dt = datetime.fromisoformat(stage_state.started_at) - end_dt = datetime.fromisoformat(stage_state.finished_at) - stage_state.duration_seconds = (end_dt - start_dt).total_seconds() - except (TypeError, ValueError): - stage_state.duration_seconds = None - - # Decide where the pipeline goes next, given this stage's outcome. - if exit_code == EXIT_AWAITING_APPROVAL: - # Human-approval gate fired. Pipeline exits; remaining - # stages stay ``pending`` so a subsequent ``--resume-from`` - # picks them up after operator action. - stage_state.status = "gated_pending_approval" - state.final_status = "gated_pending_approval" - state.stopped_at = stage.name - state.finished_at = _utc_iso_now() - worst_exit = max(worst_exit, EXIT_AWAITING_APPROVAL) - self._save_state(state) - # Phase 14 review F-N-1: dedicated ``pipeline.stage_gated`` - # event (instead of ``pipeline.stage_completed`` with a - # ``gate_decision=approval_pending`` sub-field). Lets - # dashboard / SIEM rules filter on the event name alone - # without parsing payload sub-fields — "stage_completed" - # reads as "stage finished cleanly" to a regex-based - # alert rule. - self._audit_event( - "pipeline.stage_gated", - pipeline_run_id=state.pipeline_run_id, - stage_name=stage.name, - gate_decision="approval_pending", - staging_path=stage_state.output_model, - ) - # No pipeline.completed event yet — pipeline is not done. + if new_prev_output is not None: + prev_output = new_prev_output + if chain_broken_now: + chain_broken = True + if should_break: break - if exit_code == EXIT_SUCCESS: - stage_state.status = "completed" - stage_state.gate_decision = "passed" - self._save_state(state) - self._audit_event( - "pipeline.stage_completed", - pipeline_run_id=state.pipeline_run_id, - stage_name=stage.name, - gate_decision="passed", - metrics=stage_state.metrics, - ) - prev_output = stage_state.output_model - continue - - # Anything else (EXIT_TRAINING_ERROR / EXIT_EVAL_FAILURE): - # the chain stops here. Subsequent stages skip with - # ``skipped_due_to_prior_revert`` (the canonical reason - # name regardless of whether the failure was a trainer - # crash or a gate failure — both are "stage N failed, - # downstream stages did not run"). - stage_state.status = "failed" - stage_state.gate_decision = "failed" - chain_broken = True - state.stopped_at = stage.name - worst_exit = max(worst_exit, exit_code) - self._save_state(state) - self._audit_event( - "pipeline.stage_reverted" if stage_state.auto_revert_triggered else "pipeline.stage_completed", - pipeline_run_id=state.pipeline_run_id, - stage_name=stage.name, - gate_decision="failed", - auto_revert_triggered=stage_state.auto_revert_triggered, - ) - self._notify_pipeline( - "reverted" if stage_state.auto_revert_triggered else "stage_failed", state, stage_state=stage_state - ) - - # 5. Final state + manifest + pipeline.completed event. - if state.final_status == "in_progress": - if worst_exit == EXIT_SUCCESS: - state.final_status = "completed" - else: - state.final_status = "stopped_at_stage" - state.finished_at = _utc_iso_now() - self._save_state(state) - - if state.final_status in ("completed", "stopped_at_stage"): - self._audit_event( - "pipeline.completed", - pipeline_run_id=state.pipeline_run_id, - final_status=state.final_status, - stopped_at=state.stopped_at, - ) - self._notify_pipeline("completed", state) - - # 6. Emit human-readable / JSON envelope summary. + # 5+6. Final state + manifest + summary. + self._finalise_pipeline(state, worst_exit) self._emit_summary(state) - return worst_exit def dry_run(self) -> int: @@ -818,31 +916,45 @@ def _notify_pipeline( except Exception as exc: # noqa: BLE001 — best-effort notification; webhook outage must not derail the pipeline. logger.warning("Failed to emit pipeline webhook %r: %s", kind, exc) - def _run_single_stage( + @staticmethod + def _resolve_input_source( + *, + input_model_override: Optional[str], + stage_model_set: bool, + prev_output: Optional[str], + ) -> InputSourceLiteral: + """Resolve a stage's ``input_source`` from the priority ladder. + + Priority order: CLI override > stage explicit > chain (prev + output) > root. See the ``InputSourceLiteral`` docstring for + what each label means. Extracted from ``_run_single_stage`` + for Sonar python:S3776 cognitive-complexity hygiene; pure + function so the caller stays linear. + """ + if input_model_override is not None: + return "cli_override" + if stage_model_set: + return "stage_explicit" + if prev_output is not None: + return "chain" + return "root" + + def _merge_and_validate_stage( self, *, stage: PipelineStage, stage_state: PipelineStageState, prev_output: Optional[str], input_model_override: Optional[str], - state: PipelineState, - ) -> int: - """Materialise the per-stage ForgeConfig, run the trainer, record results. - - Returns the per-stage exit code; the caller decides what to do - with it (continue, halt, gate). + ) -> Optional[Any]: + """Merge the per-stage ForgeConfig + run the chain-presence guard. + + Returns the merged ``ForgeConfig`` on success, or ``None`` if a + config-error path already populated ``stage_state.error`` and + the caller should return ``EXIT_CONFIG_ERROR``. Extracted from + ``_run_single_stage`` for Sonar python:S3776 cognitive- + complexity hygiene. """ - # Resolve input source for audit clarity. - if input_model_override is not None: - stage_state.input_source = "cli_override" - elif stage.model is not None: - stage_state.input_source = "stage_explicit" - elif prev_output is not None: - stage_state.input_source = "chain" - else: - stage_state.input_source = "root" - - # Merge config (any merge-time validation failure surfaces here). try: stage_cfg = merge_pipeline_stage_config( self.root_cfg, @@ -850,40 +962,40 @@ def _run_single_stage( prev_output_model=prev_output, input_model_override=input_model_override, ) - except Exception as exc: # noqa: BLE001 — config-merge failure is a fatal stage error; routes through the standard config-error exit code. + except Exception as exc: # noqa: BLE001 stage_state.error = f"Config merge failed: {exc}" logger.exception("Stage %r config merge failed.", stage.name) - return EXIT_CONFIG_ERROR + return None stage_state.input_model = stage_cfg.model.name_or_path - # ``forgelm`` chain-presence guard: when the orchestrator - # auto-chained the input model from the previous stage's output, - # refuse to proceed unless that exact directory exists. Phase - # 14 review F-S-1: pre-fix this checked ``os.path.dirname(...)`` - # (the parent), accepting ``./stage1/`` even when ``./stage1/ - # final_model`` was missing and letting a bad chain state fall - # through into ``get_model_and_tokenizer`` for a non-actionable - # downstream crash. - if stage_state.input_source == "chain": - if not os.path.isdir(stage_cfg.model.name_or_path): - stage_state.error = ( - f"Stage {stage.name!r} requires the previous stage's output at " - f"{stage_cfg.model.name_or_path}; pass --input-model to " - f"override or run the full pipeline first." - ) - logger.error(stage_state.error) - return EXIT_CONFIG_ERROR + # Chain-presence guard: when auto-chained, the previous stage's + # output_dir/final_model leaf must exist. Phase 14 review F-S-1. + if stage_state.input_source == "chain" and not os.path.isdir(stage_cfg.model.name_or_path): + stage_state.error = ( + f"Stage {stage.name!r} requires the previous stage's output at " + f"{stage_cfg.model.name_or_path}; pass --input-model to " + f"override or run the full pipeline first." + ) + logger.error(stage_state.error) + return None - self._audit_event( - "pipeline.stage_started", - pipeline_run_id=state.pipeline_run_id, - stage_name=stage.name, - stage_index=stage_state.index, - input_model=stage_state.input_model, - input_source=stage_state.input_source, - ) + return stage_cfg + def _invoke_trainer( + self, + *, + stage_cfg: Any, + stage_name: str, + stage_state: PipelineStageState, + ) -> Optional[Any]: + """Run a single stage's :class:`ForgeTrainer` lifecycle. + + Returns the ``TrainResult`` on success, ``None`` on import / + runtime failure (caller maps to the right exit code via + ``stage_state.error``). Extracted from ``_run_single_stage`` + for Sonar python:S3776 cognitive-complexity hygiene. + """ # Defer heavy imports so dry-run / --stage on a different stage # doesn't load torch. try: @@ -894,24 +1006,69 @@ def _run_single_stage( except ImportError as e: stage_state.error = f"Missing training dependency: {e}" logger.error(stage_state.error) - return EXIT_TRAINING_ERROR + return None + # Top-of-stage catch. ForgeTrainer.train() crosses every + # concern (HF load, dataset, TRL, safety, judge, audit, + # compliance, webhook); any uncaught exception must surface as + # a per-stage failure rather than a Python traceback that + # strands the pipeline state. try: if not stage_cfg.model.offline: setup_authentication(stage_cfg.auth.hf_token if stage_cfg.auth else None) - model, tokenizer = get_model_and_tokenizer(stage_cfg) dataset = prepare_dataset(stage_cfg, tokenizer) trainer = ForgeTrainer(model=model, tokenizer=tokenizer, config=stage_cfg, dataset=dataset) - result = trainer.train() - # Top-of-stage catch. ForgeTrainer.train() crosses every - # concern (HF load, dataset, TRL, safety, judge, audit, - # compliance, webhook); any uncaught exception must surface as - # a per-stage failure rather than a Python traceback that - # strands the pipeline state. + return trainer.train() except Exception as exc: # noqa: BLE001 stage_state.error = f"Trainer crashed: {exc}" - logger.exception("Stage %r trainer crashed.", stage.name) + logger.exception("Stage %r trainer crashed.", stage_name) + return None + + def _run_single_stage( + self, + *, + stage: PipelineStage, + stage_state: PipelineStageState, + prev_output: Optional[str], + input_model_override: Optional[str], + state: PipelineState, + ) -> int: + """Materialise the per-stage ForgeConfig, run the trainer, record results. + + Returns the per-stage exit code; the caller decides what to do + with it (continue, halt, gate). + """ + stage_state.input_source = self._resolve_input_source( + input_model_override=input_model_override, + stage_model_set=stage.model is not None, + prev_output=prev_output, + ) + + stage_cfg = self._merge_and_validate_stage( + stage=stage, + stage_state=stage_state, + prev_output=prev_output, + input_model_override=input_model_override, + ) + if stage_cfg is None: + return EXIT_CONFIG_ERROR + + self._audit_event( + "pipeline.stage_started", + pipeline_run_id=state.pipeline_run_id, + stage_name=stage.name, + stage_index=stage_state.index, + input_model=stage_state.input_model, + input_source=stage_state.input_source, + ) + + result = self._invoke_trainer( + stage_cfg=stage_cfg, + stage_name=stage.name, + stage_state=stage_state, + ) + if result is None: return EXIT_TRAINING_ERROR # Capture results onto the stage state. diff --git a/forgelm/cli/subcommands/_verify_annex_iv.py b/forgelm/cli/subcommands/_verify_annex_iv.py index 00aea7e2..6c6cb71c 100644 --- a/forgelm/cli/subcommands/_verify_annex_iv.py +++ b/forgelm/cli/subcommands/_verify_annex_iv.py @@ -185,110 +185,125 @@ def _compute_manifest_hash(artifact: Dict[str, Any]) -> str: return compute_annex_iv_manifest_hash(artifact) -def _run_verify_annex_iv_cmd(args, output_format: str) -> None: - """Top-level dispatcher for ``forgelm verify-annex-iv ``. - - Two modes: +def _run_pipeline_mode(path: str, output_format: str) -> None: + """Verify a pipeline run directory's manifest + chain integrity. - - **Single artefact** (default): ```` is an Annex IV JSON file - and the verifier checks field completeness + manifest hash. - - **Pipeline** (``--pipeline`` flag): ```` is a pipeline run - directory and the verifier reads - ``/compliance/pipeline_manifest.json`` and runs chain- - integrity + stage-index + ``stopped_at`` coherence + per-stage - training_manifest existence checks. Returns a list of violations - and exits 0 only when the list is empty. + Reads ``/compliance/pipeline_manifest.json``, runs the in- + memory verifier + per-stage training_manifest existence check, and + prints / exits. Extracted from :func:`_run_verify_annex_iv_cmd` + for Sonar python:S3776 cognitive-complexity hygiene. """ - path = getattr(args, "path", None) - if not path: - _output_error_and_exit( - output_format, - "verify-annex-iv requires a path argument: `forgelm verify-annex-iv `.", - EXIT_CONFIG_ERROR, - ) + from forgelm.compliance import verify_pipeline_manifest_at_path - # Phase 14: ``--pipeline`` mode validates the chain-level manifest. - if getattr(args, "pipeline", False): - from forgelm.compliance import verify_pipeline_manifest_at_path - - violations = verify_pipeline_manifest_at_path(path) - if output_format == "json": - print( - json.dumps( - { - "success": not violations, - "mode": "pipeline", - "path": os.path.abspath(path), - "violations": violations, - }, - indent=2, - ) + violations = verify_pipeline_manifest_at_path(path) + if output_format == "json": + print( + json.dumps( + { + "success": not violations, + "mode": "pipeline", + "path": os.path.abspath(path), + "violations": violations, + }, + indent=2, ) - else: - if not violations: - print(f"OK: pipeline manifest at {path}") - else: - print(f"FAIL: pipeline manifest at {path}") - for v in violations: - print(f" - {v}") - sys.exit(EXIT_SUCCESS if not violations else EXIT_CONFIG_ERROR) - - # Round 6 absorption: the prior `os.path.isfile()` pre-check - # collapsed two distinct conditions ("path doesn't exist" and - # "path exists but the user can't stat it") into a single - # `EXIT_CONFIG_ERROR`, breaking the documented exit-code - # contract for the permission-denied case. Move all path - # validation inside the try block and let Python's exception - # hierarchy disambiguate: FileNotFoundError + IsADirectoryError - # = caller-input error (exit 1); OSError (incl. PermissionError) - # = genuine I/O failure on a reachable path (exit 2). + ) + elif not violations: + print(f"OK: pipeline manifest at {path}") + else: + print(f"FAIL: pipeline manifest at {path}") + for v in violations: + print(f" - {v}") + sys.exit(EXIT_SUCCESS if not violations else EXIT_CONFIG_ERROR) + + +def _verify_artefact_and_handle_io_errors(path: str, output_format: str) -> "VerifyAnnexIVResult": + """Run :func:`verify_annex_iv_artifact` with the documented I/O + error-mapping policy. + + Extracted from :func:`_run_verify_annex_iv_cmd` for Sonar + python:S3776 cognitive-complexity hygiene. Each ``except`` branch + exits the process via ``_output_error_and_exit`` — the function + only returns on success. + + Exit-code mapping (per ``docs/reference/verify_annex_iv_subcommand.md``): + + - ``FileNotFoundError`` / ``IsADirectoryError`` / ``JSONDecodeError`` → + ``EXIT_CONFIG_ERROR (1)`` (operator-actionable input error). + - ``OSError`` (catch-all, must follow ``FileNotFoundError`` because + Python's OSError hierarchy makes it a subclass) → + ``EXIT_TRAINING_ERROR (2)`` (genuine runtime I/O failure). + """ try: - result = verify_annex_iv_artifact(path) + return verify_annex_iv_artifact(path) except (FileNotFoundError, IsADirectoryError) as exc: - # Caller-input error: the path is missing or refers to a - # directory. Per docs/standards/error-handling.md and the - # public exit-code contract in - # docs/reference/verify_annex_iv_subcommand.md, this is - # exit 1 (config / caller error). _output_error_and_exit( output_format, f"Annex IV artifact not found or not a regular file: {path!r} ({exc.__class__.__name__}).", EXIT_CONFIG_ERROR, ) except json.JSONDecodeError as exc: - # Validation failure: the artifact is reachable but not parseable - # as JSON. Operator-actionable input error per the public - # contract; routes to exit 1. _output_error_and_exit( output_format, f"Annex IV artifact at {path!r} is not valid JSON: {exc.msg} (line {exc.lineno}).", EXIT_CONFIG_ERROR, ) except OSError as exc: - # Genuine runtime I/O failure on a reachable path (permission - # denied, mid-read I/O error, locked file, etc.). This is the - # post-FileNotFoundError catch-all because Python's OSError - # hierarchy puts FileNotFoundError as a subclass — order - # matters. _output_error_and_exit( output_format, f"Could not read Annex IV artifact {path!r}: {exc}.", EXIT_TRAINING_ERROR, ) + +def _print_artefact_result(result: "VerifyAnnexIVResult", path: str, output_format: str) -> None: + """Render the per-artefact verify result to stdout (JSON or text).""" payload = result.to_dict() payload["path"] = os.path.abspath(path) if output_format == "json": print(json.dumps({"success": result.valid, **payload}, indent=2)) - else: - if result.valid: - print(f"OK: {path}") - print(f" {result.reason}") - else: - print(f"FAIL: {path}") - print(f" {result.reason}") - for missing in result.missing_fields: - print(f" - missing: {missing}") + return + if result.valid: + print(f"OK: {path}") + print(f" {result.reason}") + return + print(f"FAIL: {path}") + print(f" {result.reason}") + for missing in result.missing_fields: + print(f" - missing: {missing}") + + +def _run_verify_annex_iv_cmd(args, output_format: str) -> None: + """Top-level dispatcher for ``forgelm verify-annex-iv ``. + + Two modes: + + - **Single artefact** (default): ```` is an Annex IV JSON file + and the verifier checks field completeness + manifest hash. + - **Pipeline** (``--pipeline`` flag): ```` is a pipeline run + directory and the verifier reads + ``/compliance/pipeline_manifest.json`` and runs chain- + integrity + stage-index + ``stopped_at`` coherence + per-stage + training_manifest existence checks. Returns a list of violations + and exits 0 only when the list is empty. + """ + path = getattr(args, "path", None) + if not path: + _output_error_and_exit( + output_format, + "verify-annex-iv requires a path argument: `forgelm verify-annex-iv `.", + EXIT_CONFIG_ERROR, + ) + + # Phase 14: ``--pipeline`` mode validates the chain-level manifest. + if getattr(args, "pipeline", False): + _run_pipeline_mode(path, output_format) + + # Single-artefact path: I/O errors map to documented exit codes via + # the helper (which sys.exits on failure); on success we render the + # result and exit with the artefact's validity verdict. + result = _verify_artefact_and_handle_io_errors(path, output_format) + _print_artefact_result(result, path, output_format) sys.exit(EXIT_SUCCESS if result.valid else EXIT_CONFIG_ERROR) diff --git a/forgelm/compliance.py b/forgelm/compliance.py index 92136813..0a4ddf65 100644 --- a/forgelm/compliance.py +++ b/forgelm/compliance.py @@ -1054,6 +1054,79 @@ def compute_annex_iv_manifest_hash(artifact: Dict[str, Any]) -> str: # eyeball the schema and its validator together. +_PIPELINE_MANIFEST_REQUIRED_KEYS = ( + "forgelm_version", + "pipeline_run_id", + "pipeline_config_hash", + "started_at", + "final_status", + "stages", +) + + +def _check_chain_link(idx: int, stage: Dict[str, Any], stages: List[Dict[str, Any]]) -> Optional[str]: + """Single-stage chain-integrity check. + + Returns a violation string when stage *idx* (claiming + ``input_source == "chain"``) cannot be validated against its + immediate predecessor's ``output_model``; ``None`` when the link + is intact (or when the stage's ``input_source`` is not ``chain`` + in which case the caller should never have invoked this). + Extracted from :func:`_verify_manifest_payload` for Sonar + python:S3776 cognitive-complexity hygiene. + """ + if idx == 0: + return f"Stage {stage.get('name')!r}: input_source='chain' but no previous stage exists (stage 0 cannot chain)." + prev = stages[idx - 1] + prev_output_model = prev.get("output_model") + if not prev_output_model: + return ( + f"chain_integrity_violation at stage {stage.get('name')!r}: claims " + f"input_source='chain' but previous stage {prev.get('name')!r} has " + f"no output_model (status={prev.get('status')!r})." + ) + if stage.get("input_model") != prev_output_model: + return ( + f"chain_integrity_violation at stage {stage.get('name')!r}: input_model={stage.get('input_model')!r} " + f"≠ previous output_model={prev_output_model!r}" + ) + return None + + +def _check_status_consistency(manifest: Dict[str, Any], stages: List[Dict[str, Any]]) -> List[str]: + """Status-consistency checks for a finalised pipeline manifest. + + Covers (a) ``stopped_at`` referent existence + expected status and + (b) the no-``running``-stages-on-a-finalised-manifest rule (F-N-2). + Extracted from :func:`_verify_manifest_payload` for Sonar + python:S3776 cognitive-complexity hygiene. + """ + violations: List[str] = [] + stopped_at = manifest.get("stopped_at") + if stopped_at is not None: + matching = [s for s in stages if s.get("name") == stopped_at] + if not matching: + violations.append(f"stopped_at refers to unknown stage {stopped_at!r}") + elif matching[0].get("status") not in ("failed", "gated_pending_approval"): + violations.append( + f"stopped_at stage {stopped_at!r} has status {matching[0].get('status')!r} " + f"(expected `failed` or `gated_pending_approval`)" + ) + + # Running-stage consistency (N-2): a finalised manifest with a stage + # still in ``running`` indicates the orchestrator crashed mid-stage. + final_status = manifest.get("final_status") + if final_status and final_status != "in_progress": + running_stages = [s.get("name") for s in stages if s.get("status") == "running"] + if running_stages: + violations.append( + f"stage(s) {running_stages!r} still in `running` status on a " + f"finalised manifest (final_status={final_status!r}); the " + f"orchestrator likely crashed mid-stage without updating state." + ) + return violations + + def _verify_manifest_payload(manifest: Dict[str, Any]) -> List[str]: """Validate a pipeline manifest's structural + chain-integrity rules. @@ -1086,15 +1159,7 @@ def _verify_manifest_payload(manifest: Dict[str, Any]) -> List[str]: """ violations: List[str] = [] - required_top = ( - "forgelm_version", - "pipeline_run_id", - "pipeline_config_hash", - "started_at", - "final_status", - "stages", - ) - for key in required_top: + for key in _PIPELINE_MANIFEST_REQUIRED_KEYS: if key not in manifest: violations.append(f"missing required top-level key: {key!r}") @@ -1108,62 +1173,16 @@ def _verify_manifest_payload(manifest: Dict[str, Any]) -> List[str]: if s.get("index") != expected: violations.append(f"stage index out of order at position {expected}: got {s.get('index')!r}") - # Chain integrity — Phase 14 review F-B-3 + F-N-3 hardening: every - # chain stage is compared against the **immediate** previous stage, - # not the most-recent stage with an ``output_model``. Pre-fix, if - # stage N-1 failed without saving an ``output_model`` (or any - # hand-crafted manifest with a None output midway), the chain check - # silently skipped that gap and compared stage N against stage N-2, - # masking a real broken link. + # Chain integrity (one check per chain-stage; helper is pure). for idx, s in enumerate(stages): if s.get("input_source") != "chain": continue - if idx == 0: - violations.append( - f"Stage {s.get('name')!r}: input_source='chain' but no previous stage exists (stage 0 cannot chain)." - ) - continue - prev = stages[idx - 1] - prev_output_model = prev.get("output_model") - if not prev_output_model: - violations.append( - f"chain_integrity_violation at stage {s.get('name')!r}: claims " - f"input_source='chain' but previous stage {prev.get('name')!r} has " - f"no output_model (status={prev.get('status')!r})." - ) - continue - if s.get("input_model") != prev_output_model: - violations.append( - f"chain_integrity_violation at stage {s.get('name')!r}: input_model={s.get('input_model')!r} " - f"≠ previous output_model={prev_output_model!r}" - ) + link_violation = _check_chain_link(idx, s, stages) + if link_violation is not None: + violations.append(link_violation) - # Status consistency. - stopped_at = manifest.get("stopped_at") - if stopped_at is not None: - matching = [s for s in stages if s.get("name") == stopped_at] - if not matching: - violations.append(f"stopped_at refers to unknown stage {stopped_at!r}") - elif matching[0].get("status") not in ("failed", "gated_pending_approval"): - violations.append( - f"stopped_at stage {stopped_at!r} has status {matching[0].get('status')!r} " - f"(expected `failed` or `gated_pending_approval`)" - ) - - # Running-stage consistency (N-2): a finalised manifest - # (final_status != "in_progress") with a stage in ``running`` status - # is a tell that the orchestrator process died mid-stage and never - # updated the state file. Archival audit must surface this so the - # operator knows the chain wasn't actually completed. - final_status = manifest.get("final_status") - if final_status and final_status != "in_progress": - running_stages = [s.get("name") for s in stages if s.get("status") == "running"] - if running_stages: - violations.append( - f"stage(s) {running_stages!r} still in `running` status on a " - f"finalised manifest (final_status={final_status!r}); the " - f"orchestrator likely crashed mid-stage without updating state." - ) + # Status consistency (stopped_at + running-on-finalised). + violations.extend(_check_status_consistency(manifest, stages)) return violations From dc2da466a1d7e39271263f64a165668316ce391c Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 14 May 2026 14:39:09 +0300 Subject: [PATCH 08/22] =?UTF-8?q?fix:=20PR=20#53=20review-response=20?= =?UTF-8?q?=E2=80=94=206=20findings=20(verify-then-fix)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Each finding was first verified against current code; fixes are minimal and validated. All 232 broader-sweep tests pass; 11/11 self-review guards green; ruff format + check clean. 1) CHANGELOG.md — contradictory stage-count wording (valid): "2+ training stages" + "≥1-stage minimum" said two different things. Code enforces ``min_length=1``. Unified the bullet to "one or more training stages" + "at least 1 stage (`min_length=1` — …)" with the schema's nuance preserved (single-stage runs are better served by omitting the `pipeline:` block, but the schema accepts a 1-element pipeline for incremental authoring). 2) docs/guides/pipeline.md — release-tense wording (valid): v0.7.0 is still ``[Unreleased]`` in CHANGELOG; the guide read as already-shipped. "added in `v0.7.0`" → "targeted for `v0.7.0` (currently on the `Unreleased` line of the CHANGELOG)". 3) docs/guides/pipeline-tr.md — release-tense wording (valid): Same fix in Turkish. "`v0.7.0` ile geldi" → "`v0.7.0` için hedefleniyor (şu an CHANGELOG'un `Unreleased` satırında)". 4) forgelm/cli/_pipeline.py:_validate_resume_state — topology guard for `--force-resume` (valid): Even under --force-resume the old code accepted any stale state, so a renamed/inserted/deleted stage between runs would make ``state.stages[i]`` address a different stage than ``self.pipeline.stages[i]`` while the orchestrator iterates positionally. Added a pre-emit guard that compares the ordered stage-name list (count + identifier + order); on mismatch we log + return an error string *before* the audit hook fires and before the stored hash is overwritten, so the JSONL audit trail doesn't carry a force_resume event for a refused resume. Regression test: test_force_resume_refuses_when_stage_topology_changed. 5) forgelm/cli/subcommands/_verify_annex_iv.py:_run_pipeline_mode — I/O-vs-config exit-code mapping (valid): ``verify_pipeline_manifest_at_path`` already encodes I/O failures as ``"… unreadable: …"`` sentinel strings, but the CLI mapped every non-empty violation list to ``EXIT_CONFIG_ERROR (1)``, contradicting the single-artefact path's policy (FileNotFoundError → 1, OSError → 2 via _verify_artefact_and_handle_io_errors). Now: - try/except wraps the verifier call (defence-in-depth against a future change that lets exceptions bubble) and maps OSError → ``EXIT_TRAINING_ERROR (2)``. - The "unreadable" sentinel in the returned violations also maps to exit 2; everything else (including the "not found" sentinel and chain-integrity violations) stays at exit 1. Regression tests: TestVerifyAnnexIvPipelineModeExitCodes — three tests covering missing → 1, unreadable → 2, chain-violation → 1. 6) forgelm/compliance.py:_verify_manifest_payload + verify_pipeline_manifest_at_path — non-dict stage type-guard (valid): Both the in-memory verifier *and* the disk wrapper iterated manifest["stages"] and called ``s.get(...)`` without checking each ``s`` is a dict — a tampered manifest with ``stages: [null, "string"]`` would crash with AttributeError instead of producing a structured violation. Added per-item ``isinstance(s, dict)`` check in both code paths; the in-memory verifier filters the well-formed subset out and runs the rest of the checks on that filtered list so downstream helpers (which assume dicts) stay stable. Regression test: test_disk_wrapper_type_guards_non_dict_stage_items pins both surfaces. Co-Authored-By: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 11 +- docs/guides/pipeline-tr.md | 2 +- docs/guides/pipeline.md | 2 +- forgelm/cli/_pipeline.py | 26 +++++ forgelm/cli/subcommands/_verify_annex_iv.py | 48 ++++++++- forgelm/compliance.py | 44 ++++++-- tests/test_pipeline_compliance.py | 107 ++++++++++++++++++++ tests/test_pipeline_orchestrator.py | 56 ++++++++++ 8 files changed, 279 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 937ff10f..52d0c768 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,12 +8,15 @@ _(v0.6.1 dev cycle — entries will land here as PRs merge.)_ ### Added (Phase 14 — Multi-Stage Pipeline Chains) -- **`pipeline:` config block** at the root of `ForgeConfig` chains 2+ - training stages (typically SFT → DPO → GRPO) into one +- **`pipeline:` config block** at the root of `ForgeConfig` chains + one or more training stages (typically SFT → DPO → GRPO) into one config-driven run. New Pydantic models `PipelineStage` and `PipelineConfig` enforce stage-name uniqueness, `^[a-z0-9_]{1,32}$` - identifier shape, ≥1-stage minimum, and explicit-`trainer_type` - audit-clarity validation per stage. Section-wholesale inheritance: + identifier shape, **at least 1 stage** (`min_length=1` — single-stage + ergonomics are still better served by omitting the `pipeline:` + block, but the schema accepts a 1-element pipeline so an operator + can iterate up to a multi-stage chain without re-shaping the YAML), + and explicit-`trainer_type` audit-clarity validation per stage. Section-wholesale inheritance: `model` / `lora` / `training` / `data` / `evaluation` blocks inherit from the root if omitted or fully replace it when set. `distributed` / `webhook` / `compliance` / `risk_assessment` / diff --git a/docs/guides/pipeline-tr.md b/docs/guides/pipeline-tr.md index b5e8bc62..ad6fe071 100644 --- a/docs/guides/pipeline-tr.md +++ b/docs/guides/pipeline-tr.md @@ -1,6 +1,6 @@ # Çok Aşamalı Eğitim Pipeline'ları -Faz 14 — `v0.7.0` ile geldi. +Faz 14 — `v0.7.0` için hedefleniyor (şu an CHANGELOG'un `Unreleased` satırında). ForgeLM'in `pipeline:` config bloğu, 2 veya daha fazla eğitim aşamasını (genellikle SFT → DPO → GRPO) tek bir config-tabanlı, dry-run-doğrulanabilir, diff --git a/docs/guides/pipeline.md b/docs/guides/pipeline.md index 404db8d5..d2a9cd49 100644 --- a/docs/guides/pipeline.md +++ b/docs/guides/pipeline.md @@ -1,6 +1,6 @@ # Multi-Stage Training Pipelines -Phase 14 — added in `v0.7.0`. +Phase 14 — targeted for `v0.7.0` (currently on the `Unreleased` line of the CHANGELOG). ForgeLM's `pipeline:` config block chains 2 or more training stages (typically SFT → DPO → GRPO) into one config-driven, dry-run-validatable, diff --git a/forgelm/cli/_pipeline.py b/forgelm/cli/_pipeline.py index 37b72957..45f4e967 100644 --- a/forgelm/cli/_pipeline.py +++ b/forgelm/cli/_pipeline.py @@ -779,6 +779,32 @@ def _validate_resume_state(self, existing: PipelineState, *, force: bool) -> Opt ) logger.error(msg) return msg + + # Topology guard (Phase 14 review-response): even under + # ``--force-resume`` we refuse if the stage *topology* has + # changed — count or ordered names — because the on-disk state + # file's ``stages[i]`` payload is positionally addressed. A + # stage inserted, deleted, or renamed between runs would make + # ``state.stages[i]`` describe a different stage than + # ``self.pipeline.stages[i]`` and the resume would silently + # corrupt the audit trail. Cosmetic edits (whitespace, + # comments, hyperparameter values) trip the hash check but + # *don't* fail this guard, so ``--force-resume`` remains useful + # for them. + on_disk_names = [s.name for s in existing.stages] + current_names = [s.name for s in self.pipeline.stages] + if on_disk_names != current_names: + msg = ( + f"Refusing to --resume-from even with --force-resume: pipeline " + f"stage topology has changed.\n" + f" on-disk stages: {on_disk_names!r}\n" + f" current stages: {current_names!r}\n" + "Stage names / count / order must match the on-disk state. " + "Start a fresh pipeline run instead." + ) + logger.error(msg) + return msg + old_hash = existing.pipeline_config_hash logger.warning( "Resuming with --force-resume despite pipeline_config_hash mismatch: %s → %s.", diff --git a/forgelm/cli/subcommands/_verify_annex_iv.py b/forgelm/cli/subcommands/_verify_annex_iv.py index 6c6cb71c..79681496 100644 --- a/forgelm/cli/subcommands/_verify_annex_iv.py +++ b/forgelm/cli/subcommands/_verify_annex_iv.py @@ -192,10 +192,51 @@ def _run_pipeline_mode(path: str, output_format: str) -> None: memory verifier + per-stage training_manifest existence check, and prints / exits. Extracted from :func:`_run_verify_annex_iv_cmd` for Sonar python:S3776 cognitive-complexity hygiene. + + Exit-code mapping (Phase 14 review-response — aligns with the + single-artefact path's policy in + :func:`_verify_artefact_and_handle_io_errors`): + + - ``EXIT_SUCCESS (0)`` — empty violation list. + - ``EXIT_TRAINING_ERROR (2)`` — genuine runtime I/O failure on a + reachable path (mid-read OSError, locked manifest, …) or the + ``pipeline_manifest.json unreadable: …`` sentinel that + :func:`forgelm.compliance.verify_pipeline_manifest_at_path` + emits when the JSON file is reachable but unreadable. + - ``EXIT_CONFIG_ERROR (1)`` — everything else, including the + ``pipeline_manifest.json not found …`` sentinel and all + structural / chain-integrity violations. """ from forgelm.compliance import verify_pipeline_manifest_at_path - violations = verify_pipeline_manifest_at_path(path) + # Defensive try/except: the verifier already maps OSError / + # JSONDecodeError to violation strings, but a future change there + # could let an exception bubble — fail loud rather than swallow. + try: + violations = verify_pipeline_manifest_at_path(path) + except OSError as exc: + msg = f"FAIL: pipeline manifest at {path} — runtime I/O error: {exc}" + if output_format == "json": + print( + json.dumps( + { + "success": False, + "mode": "pipeline", + "path": os.path.abspath(path), + "violations": [str(exc)], + }, + indent=2, + ) + ) + else: + print(msg) + sys.exit(EXIT_TRAINING_ERROR) + + # An "unreadable" sentinel from the verifier means the file is + # reachable but I/O / parse fails — runtime error, not config + # error. "not found" is operator-input → config error. + unreadable = any("unreadable" in v for v in violations) + if output_format == "json": print( json.dumps( @@ -214,7 +255,10 @@ def _run_pipeline_mode(path: str, output_format: str) -> None: print(f"FAIL: pipeline manifest at {path}") for v in violations: print(f" - {v}") - sys.exit(EXIT_SUCCESS if not violations else EXIT_CONFIG_ERROR) + + if not violations: + sys.exit(EXIT_SUCCESS) + sys.exit(EXIT_TRAINING_ERROR if unreadable else EXIT_CONFIG_ERROR) def _verify_artefact_and_handle_io_errors(path: str, output_format: str) -> "VerifyAnnexIVResult": diff --git a/forgelm/compliance.py b/forgelm/compliance.py index 0a4ddf65..2ad47f10 100644 --- a/forgelm/compliance.py +++ b/forgelm/compliance.py @@ -1168,21 +1168,37 @@ def _verify_manifest_payload(manifest: Dict[str, Any]) -> List[str]: violations.append("`stages` must be a list") return violations - # Index monotonicity. - for expected, s in enumerate(stages): + # Phase 14 review-response: per-item type-guard. A tampered + # manifest (``stages: [null, "foo"]``) would otherwise raise + # ``AttributeError`` on ``s.get(...)`` partway through the + # verifier. Surface as a structured violation so the disk wrapper + # (and any future caller) gets a clean list back. We filter the + # malformed items out of the rest of the checks because the + # downstream helpers also expect dicts. + well_formed_stages: List[Dict[str, Any]] = [] + for idx, s in enumerate(stages): + if not isinstance(s, dict): + violations.append(f"stage at index {idx} is not an object (got {type(s).__name__})") + continue + well_formed_stages.append(s) + + # Index monotonicity (on the well-formed subset; mis-indexing of + # a malformed entry would be a noise violation on top of the + # already-reported type error). + for expected, s in enumerate(well_formed_stages): if s.get("index") != expected: violations.append(f"stage index out of order at position {expected}: got {s.get('index')!r}") # Chain integrity (one check per chain-stage; helper is pure). - for idx, s in enumerate(stages): + for idx, s in enumerate(well_formed_stages): if s.get("input_source") != "chain": continue - link_violation = _check_chain_link(idx, s, stages) + link_violation = _check_chain_link(idx, s, well_formed_stages) if link_violation is not None: violations.append(link_violation) # Status consistency (stopped_at + running-on-finalised). - violations.extend(_check_status_consistency(manifest, stages)) + violations.extend(_check_status_consistency(manifest, well_formed_stages)) return violations @@ -1793,11 +1809,21 @@ def verify_pipeline_manifest_at_path(pipeline_dir: str) -> List[str]: # Disk-only check: each completed stage's training_manifest must # exist. The in-memory verifier cannot see the filesystem. - for s in manifest.get("stages", []): + # Phase 14 review-response: type-guard each stage entry so a + # tampered manifest (``stages: [null, "foo"]``) surfaces as a + # violation rather than raising ``AttributeError`` on ``s.get``. + raw_stages = manifest.get("stages") + if not isinstance(raw_stages, list): + # In-memory verifier already flagged this; nothing more to + # check on disk. + return violations + for idx, s in enumerate(raw_stages): + if not isinstance(s, dict): + violations.append(f"stage at index {idx} is not an object (got {type(s).__name__})") + continue name = s.get("name", "") per_stage_manifest = s.get("training_manifest") - if per_stage_manifest and s.get("status") == "completed": - if not os.path.isfile(per_stage_manifest): - violations.append(f"Stage {name!r}: training_manifest at {per_stage_manifest!r} is missing.") + if per_stage_manifest and s.get("status") == "completed" and not os.path.isfile(per_stage_manifest): + violations.append(f"Stage {name!r}: training_manifest at {per_stage_manifest!r} is missing.") return violations diff --git a/tests/test_pipeline_compliance.py b/tests/test_pipeline_compliance.py index 90af29fe..c7e68a93 100644 --- a/tests/test_pipeline_compliance.py +++ b/tests/test_pipeline_compliance.py @@ -426,6 +426,35 @@ def test_malformed_manifest_returns_single_violation(self, tmp_path): assert len(violations) == 1 assert "unreadable" in violations[0] + def test_disk_wrapper_type_guards_non_dict_stage_items(self, tmp_path): + """Phase 14 review-response regression: a tampered manifest where + ``stages`` contains non-dict items (``null`` / a string / etc.) + must surface as a violation, not crash with ``AttributeError`` + inside the disk-only loop's ``s.get(...)`` calls.""" + from forgelm.compliance import verify_pipeline_manifest_at_path + + manifest_dir = tmp_path / "compliance" + manifest_dir.mkdir() + # Build a manifest payload with two malformed stage entries. + bad_manifest = { + "forgelm_version": "0.7.0", + "pipeline_run_id": "pl_x", + "pipeline_config_hash": "sha256:abc", + "started_at": "2026-06-15T12:00:00+00:00", + "final_status": "in_progress", + "stages": [ + None, + "this-should-be-a-dict", + {"name": "ok", "index": 2, "trainer_type": "sft", "status": "pending"}, + ], + } + import json as _json + + (manifest_dir / "pipeline_manifest.json").write_text(_json.dumps(bad_manifest)) + violations = verify_pipeline_manifest_at_path(str(tmp_path)) + assert any("stage at index 0 is not an object" in v for v in violations) + assert any("stage at index 1 is not an object" in v for v in violations) + def test_completed_stage_with_missing_training_manifest_flagged(self, tmp_path): """The disk wrapper layers a per-stage training_manifest existence check on top of the in-memory verifier — the @@ -448,6 +477,84 @@ def test_completed_stage_with_missing_training_manifest_flagged(self, tmp_path): assert any("training_manifest" in v and "is missing" in v for v in violations) +class TestVerifyAnnexIvPipelineModeExitCodes: + """Phase 14 review-response regression: ``forgelm verify-annex-iv + --pipeline `` must map I/O failures to ``EXIT_TRAINING_ERROR`` + (2) and operator-input errors (``not found``, structural / chain- + integrity violations) to ``EXIT_CONFIG_ERROR`` (1). Mirrors the + single-artefact path's exit-code policy.""" + + def _run(self, tmp_path, args_overrides: dict) -> int: + """Invoke ``_run_pipeline_mode`` and capture the ``SystemExit`` code.""" + import argparse as _argparse + + from forgelm.cli.subcommands._verify_annex_iv import _run_verify_annex_iv_cmd + + args = _argparse.Namespace(path=str(tmp_path), pipeline=True, **args_overrides) + try: + _run_verify_annex_iv_cmd(args, "text") + except SystemExit as exc: + return int(exc.code) if exc.code is not None else 0 + return 0 + + def test_missing_manifest_exits_config_error(self, tmp_path, capsys): + """``not found`` is operator-actionable input → exit 1.""" + code = self._run(tmp_path, {}) + assert code == 1 # EXIT_CONFIG_ERROR + captured = capsys.readouterr().out + assert "FAIL: pipeline manifest" in captured + assert "not found" in captured + + def test_unreadable_manifest_exits_training_error(self, tmp_path, capsys): + """Reachable file that can't be parsed → exit 2 (runtime I/O).""" + manifest_dir = tmp_path / "compliance" + manifest_dir.mkdir() + (manifest_dir / "pipeline_manifest.json").write_text("{not valid json") + code = self._run(tmp_path, {}) + assert code == 2 # EXIT_TRAINING_ERROR + captured = capsys.readouterr().out + assert "unreadable" in captured + + def test_chain_integrity_violation_exits_config_error(self, tmp_path, capsys): + """Structural / chain violations on a readable manifest → + exit 1 (operator-fixable config error).""" + import json as _json + + manifest_dir = tmp_path / "compliance" + manifest_dir.mkdir() + bad_chain_manifest = { + "forgelm_version": "0.7.0", + "pipeline_run_id": "pl_x", + "pipeline_config_hash": "sha256:abc", + "started_at": "2026-06-15T12:00:00+00:00", + "final_status": "completed", + "stages": [ + { + "name": "s0", + "index": 0, + "trainer_type": "sft", + "status": "completed", + "input_source": "root", + "output_model": "./s0/out", + }, + { + "name": "s1", + "index": 1, + "trainer_type": "dpo", + "status": "completed", + "input_source": "chain", + "input_model": "tampered/different/path", # ≠ s0.output_model + "output_model": "./s1/out", + }, + ], + } + (manifest_dir / "pipeline_manifest.json").write_text(_json.dumps(bad_chain_manifest)) + code = self._run(tmp_path, {}) + assert code == 1 # EXIT_CONFIG_ERROR + captured = capsys.readouterr().out + assert "chain_integrity_violation" in captured + + class TestVerifyOnPartialFilterRun: """A ``--stage X`` run produces a manifest where other stages have ``status: skipped_by_filter``; verifier should accept it.""" diff --git a/tests/test_pipeline_orchestrator.py b/tests/test_pipeline_orchestrator.py index 8ec4e203..a5dc0ccd 100644 --- a/tests/test_pipeline_orchestrator.py +++ b/tests/test_pipeline_orchestrator.py @@ -598,6 +598,62 @@ def test_force_resume_emits_audit_event_with_both_hashes(self, tmp_path, monkeyp "Audit event must record hashes, not raw YAML bytes." ) + def test_force_resume_refuses_when_stage_topology_changed(self, tmp_path, monkeypatch): + """Phase 14 review-response regression: ``--force-resume`` must + still refuse when the on-disk state's stage list disagrees with + the current pipeline (count, names, or order). Without this + guard, a renamed/inserted/deleted stage between runs would let + the orchestrator address ``state.stages[i]`` from the old shape + while iterating ``self.pipeline.stages[i]`` from the new shape, + silently corrupting the audit trail.""" + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=False, error="x")]) + orch1 = PipelineOrchestrator(cfg, b"yaml v1") + orch1.run() + + # Build a *different* topology (rename middle stage) while keeping + # the YAML hash changed too, so force_resume is the only thing + # standing between us and a resume. + renamed_cfg = ForgeConfig( + model={"name_or_path": "org/base"}, + lora={"r": 8}, + training={"trainer_type": "sft", "output_dir": str(tmp_path)}, + data={"dataset_name_or_path": "org/sft"}, + pipeline={ + "output_dir": str(tmp_path / "pipeline_run"), + "stages": [ + { + "name": "sft_stage", + "training": {"trainer_type": "sft", "output_dir": str(tmp_path / "stage1")}, + "data": {"dataset_name_or_path": "org/sft_data"}, + }, + { + "name": "renamed_middle", # was 'dpo_stage' + "training": {"trainer_type": "dpo", "output_dir": str(tmp_path / "stage2")}, + "data": {"dataset_name_or_path": "org/dpo_data"}, + }, + { + "name": "grpo_stage", + "training": {"trainer_type": "grpo", "output_dir": str(tmp_path / "stage3")}, + "data": {"dataset_name_or_path": "org/math_data"}, + }, + ], + }, + ) + _install_trainer_mocks(monkeypatch, [TrainResult(success=True), TrainResult(success=True)]) + orch2 = PipelineOrchestrator(renamed_cfg, b"yaml v2 (renamed)") + code = orch2.run(resume_from="grpo_stage", force_resume=True) + assert code == EXIT_CONFIG_ERROR + # No pipeline.force_resume audit event should have been emitted — + # topology mismatch is refused *before* the audit hook. + audit_path = os.path.join(orch2.paths["root_output_dir"], "audit_log.jsonl") + if os.path.exists(audit_path): + with open(audit_path) as f: + events = [json.loads(line) for line in f if line.strip()] + assert not any(e.get("event") == "pipeline.force_resume" for e in events), ( + "pipeline.force_resume must not fire when topology has changed." + ) + def test_resume_with_unknown_stage_name_fails(self, tmp_path, monkeypatch): cfg = _three_stage_config(tmp_path) _install_trainer_mocks(monkeypatch, [TrainResult(success=False, error="x")]) From 6a6ca60e058d62c169aa2a85e904452ffecb736e Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 14 May 2026 14:57:46 +0300 Subject: [PATCH 09/22] fix(docs): mermaid syntax error in LoRA/QLoRA/DoRA decision diagram MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first node of the flowchart in docs/usermanuals/{en,tr}/training/lora.md embedded a bare ``<`` inside the rhombus label: Q1{Need to fit on a
consumer GPU
(<24 GB)?} Mermaid 10.9.4 parses the ``<`` as the start of an HTML tag inside a ``{...}`` rhombus node, fails to find a matching ``>``, and aborts the whole diagram with "Syntax error in text". The site's user-manual renderer surfaces this verbatim where the diagram should be — the exact error the user reported. Fix: wrap the affected label in double quotes so Mermaid treats the contents literally: Q1{"Need to fit on a
consumer GPU
(<24 GB)?"} Same fix in the Turkish mirror. No other mermaid label in the user manuals uses an unquoted ``<`` — verified by grep across docs/usermanuals/. The site/js/usermanuals/*.js bundles are gitignored (regenerated at CI build via ``tools/build_usermanuals.py``); ``build_usermanuals.py --check`` is green so the next deploy picks the fix up. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/usermanuals/en/training/lora.md | 2 +- docs/usermanuals/tr/training/lora.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/usermanuals/en/training/lora.md b/docs/usermanuals/en/training/lora.md index 5221afb8..2d051c65 100644 --- a/docs/usermanuals/en/training/lora.md +++ b/docs/usermanuals/en/training/lora.md @@ -13,7 +13,7 @@ ForgeLM applies LoRA / QLoRA / DoRA to *every* trainer (SFT, DPO, SimPO, KTO, OR ```mermaid flowchart TD - Q1{Need to fit on a
consumer GPU
(<24 GB)?} + Q1{"Need to fit on a
consumer GPU
(<24 GB)?"} Q2{Want full-precision
weights kept frozen?} Q3{Need maximum
quality at LoRA cost?} Full[Full fine-tuning] diff --git a/docs/usermanuals/tr/training/lora.md b/docs/usermanuals/tr/training/lora.md index 64d7d984..04516e3d 100644 --- a/docs/usermanuals/tr/training/lora.md +++ b/docs/usermanuals/tr/training/lora.md @@ -13,7 +13,7 @@ ForgeLM LoRA / QLoRA / DoRA'yı *her* trainer'a uygular (SFT, DPO, SimPO, KTO, O ```mermaid flowchart TD - Q1{Consumer GPU'da
(<24 GB) sığmalı mı?} + Q1{"Consumer GPU'da
(<24 GB) sığmalı mı?"} Q2{Full-precision
ağırlıklar dondurulsun mu?} Q3{LoRA maliyetinde
maksimum kalite mi?} Full[Full fine-tuning] From 034abd480252019bf107cc84f5d0c86b251c943a Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 14 May 2026 15:10:43 +0300 Subject: [PATCH 10/22] fix(docs): quote mermaid node labels with special characters MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Following the LoRA decision-diagram fix (6a6ca60), a scan of every mermaid block under docs/usermanuals/, docs/guides/, docs/reference/, docs/qms/, docs/standards/, and docs/roadmap*/ surfaced 5 more node labels with unquoted special characters that could trip Mermaid's parser depending on version. Fix in all of them: wrap the label in ``"…"`` so Mermaid takes the contents literally — same approach as the LoRA fix. Identified by a regex scanner that walks each mermaid fence, parses out the shape brackets, strips safe ``
`` tokens, and flags any remaining bare ``<`` / ``>`` / ``&`` in the label. Scanner re-runs clean after these fixes. Files: - docs/usermanuals/en/getting-started/introduction.md:35 ``I[Export &
deploy]`` — bare ``&`` immediately before ``
`` reads as a partial HTML entity start. TR mirror uses "ve" instead of "&" and was already safe. - docs/usermanuals/{en,tr}/concepts/alignment-overview.md:13 ``SFT[SFT
format & content]`` (EN) and ``SFT[SFT
format & içerik]`` (TR) — same bare-``&`` risk. - docs/reference/data_ingestion_architecture{,_tr}.md ~L210 ``CHECK{>= 70 percent of pages?}`` (EN) and ``CHECK{>= yüzde 70 sayfa?}`` (TR) — bare ``>=`` at the start of a rhombus label can confuse Mermaid because ``>`` overlaps with the asymmetric-shape syntax. - docs/roadmap/completed-phases.md:425 ``F{> threshold?}`` — same bare-``>`` issue. User-manual bundles regenerated via ``tools/build_usermanuals.py`` (the site/js/usermanuals/*.js bundles are gitignored — CI rebuilds on deploy). Bilingual parity + anchor-resolution guards green. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/reference/data_ingestion_architecture-tr.md | 2 +- docs/reference/data_ingestion_architecture.md | 2 +- docs/roadmap/completed-phases.md | 2 +- docs/usermanuals/en/concepts/alignment-overview.md | 2 +- docs/usermanuals/en/getting-started/introduction.md | 2 +- docs/usermanuals/tr/concepts/alignment-overview.md | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/reference/data_ingestion_architecture-tr.md b/docs/reference/data_ingestion_architecture-tr.md index 29d71f04..0664b612 100644 --- a/docs/reference/data_ingestion_architecture-tr.md +++ b/docs/reference/data_ingestion_architecture-tr.md @@ -209,7 +209,7 @@ graph TD PAGES[sayfa başına metin] --> SPLIT[Satırlara böl] SPLIT --> WINDOW[Sayfa başına üst-3 / alt-3 satırı incele
_PDF_EDGE_WINDOW = 3] WINDOW --> COUNT[Tüm sayfalarda satır
frekansını say] - COUNT --> CHECK{>= yüzde 70 sayfa?} + COUNT --> CHECK{">= yüzde 70 sayfa?"} CHECK -- evet --> POP[Eşleşen window satırlarını çıkar
window içinde herhangi bir offset] POP --> WINDOW CHECK -- hayır --> JOIN[Sayfaları çift yeni satırla birleştir] diff --git a/docs/reference/data_ingestion_architecture.md b/docs/reference/data_ingestion_architecture.md index fbc94c20..44dbdb6d 100644 --- a/docs/reference/data_ingestion_architecture.md +++ b/docs/reference/data_ingestion_architecture.md @@ -207,7 +207,7 @@ graph TD PAGES[per-page text] --> SPLIT[Split into lines] SPLIT --> WINDOW[Inspect top-3 / bottom-3 rows
per page _PDF_EDGE_WINDOW = 3] WINDOW --> COUNT[Count line frequency
across pages] - COUNT --> CHECK{>= 70 percent of pages?} + COUNT --> CHECK{">= 70 percent of pages?"} CHECK -- yes --> POP[Pop matching window lines
any offset within the window] POP --> WINDOW CHECK -- no --> JOIN[Re-join pages with double newline] diff --git a/docs/roadmap/completed-phases.md b/docs/roadmap/completed-phases.md index 4ea6faf5..68f45d2a 100644 --- a/docs/roadmap/completed-phases.md +++ b/docs/roadmap/completed-phases.md @@ -422,7 +422,7 @@ graph LR B -->|classify| C[Llama Guard] C --> D{safe / unsafe} D -->|count| E[unsafe_ratio] - E -->|compare| F{> threshold?} + E -->|compare| F{"> threshold?"} F -->|yes| G[❌ REVERT] F -->|no| H[✅ PASS] ``` diff --git a/docs/usermanuals/en/concepts/alignment-overview.md b/docs/usermanuals/en/concepts/alignment-overview.md index 844d8689..5827cd68 100644 --- a/docs/usermanuals/en/concepts/alignment-overview.md +++ b/docs/usermanuals/en/concepts/alignment-overview.md @@ -10,7 +10,7 @@ Modern LLM fine-tuning is no longer a single algorithm — it's a *stack* of pos ```mermaid flowchart TD Base[Base model
e.g. Qwen 2.5 / Llama 3] - SFT[SFT
format & content] + SFT["SFT
format & content"] DPO[DPO
preference alignment] SimPO[SimPO
preference, ref-free] KTO[KTO
binary feedback] diff --git a/docs/usermanuals/en/getting-started/introduction.md b/docs/usermanuals/en/getting-started/introduction.md index 5b4c1cc8..1cfed53e 100644 --- a/docs/usermanuals/en/getting-started/introduction.md +++ b/docs/usermanuals/en/getting-started/introduction.md @@ -32,7 +32,7 @@ flowchart LR F -->|regression| G[Auto-revert] G --> E F -->|pass| H[Annex IV
artifacts] - H --> I[Export &
deploy] + H --> I["Export &
deploy"] classDef start fill:#1c2030,stroke:#f97316,color:#e6e7ec classDef ok fill:#161a24,stroke:#22c55e,color:#e6e7ec classDef warn fill:#161a24,stroke:#eab308,color:#e6e7ec diff --git a/docs/usermanuals/tr/concepts/alignment-overview.md b/docs/usermanuals/tr/concepts/alignment-overview.md index 3535f10a..7cdddbd7 100644 --- a/docs/usermanuals/tr/concepts/alignment-overview.md +++ b/docs/usermanuals/tr/concepts/alignment-overview.md @@ -10,7 +10,7 @@ Modern LLM fine-tuning'i artık tek bir algoritma değil — her biri farklı bi ```mermaid flowchart TD Base[Temel model
ör. Qwen 2.5 / Llama 3] - SFT[SFT
format & içerik] + SFT["SFT
format & içerik"] DPO[DPO
tercih hizalaması] SimPO[SimPO
tercih, ref-free] KTO[KTO
ikili geri bildirim] From 0ab3384a1124e7bc608455517285a7a89afb25a3 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 14 May 2026 15:15:04 +0300 Subject: [PATCH 11/22] =?UTF-8?q?refactor:=20PR=20#53=20SonarCloud=20follo?= =?UTF-8?q?w-up=20=E2=80=94=20run()=20complexity=20+=20try/except=20System?= =?UTF-8?q?Exit?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two findings from the latest Sonar pass on a65d3c9: forgelm/cli/_pipeline.py:550 — run() cognitive complexity 22 → ≤15 After the previous refactor (a65d3c9, 53 → ≤15) the for-loop body still contributed ~7 to the score: three nested skip checks, two post-execution conditional updates, and one ``should_break`` guard all sat inside a for inside the method. Lifted the entire stage loop into ``_run_stage_loop`` and the skip-decision into ``_stage_skip_reason`` (returns a discriminated string for "filtered" / "already_completed" / "chain_broken" / None). Inline the two ``if x: y = z`` post-execution updates as ternary / boolean-or expressions: prev_output = new_prev_output if new_prev_output is not None else prev_output chain_broken = chain_broken or chain_broken_now ``run()`` is now linear: prepare state → resume skiplist → filter setup → started event → loop → finalise → summary. tests/test_pipeline_compliance.py:496 — try/except SystemExit (S5754) The ``_run`` helper in TestVerifyAnnexIvPipelineModeExitCodes wraps ``_run_verify_annex_iv_cmd`` in a bare ``try/except SystemExit`` to capture the CLI's exit code. Sonar flagged the catch as the "swallow without reraise" anti-pattern (BaseException subclass — ``SystemExit`` inherits from BaseException, not Exception). Replaced with ``pytest.raises(SystemExit)`` — pytest's idiomatic context manager for CLI tests. The contract is now positive ("we *expect* SystemExit every invocation") rather than speculative-catch, and a future code path that fails to ``sys.exit`` would surface as a test failure rather than being silently masked. Verification: - ruff format + check clean - 154 / 154 pipeline + webhook tests pass Co-Authored-By: Claude Opus 4.7 (1M context) --- forgelm/cli/_pipeline.py | 166 +++++++++++++++++++++--------- tests/test_pipeline_compliance.py | 18 +++- 2 files changed, 130 insertions(+), 54 deletions(-) diff --git a/forgelm/cli/_pipeline.py b/forgelm/cli/_pipeline.py index 45f4e967..cdd88366 100644 --- a/forgelm/cli/_pipeline.py +++ b/forgelm/cli/_pipeline.py @@ -547,6 +547,114 @@ def _finalise_pipeline(self, state: PipelineState, worst_exit: int) -> None: ) self._notify_pipeline("completed", state) + def _stage_skip_reason( + self, + *, + stage: PipelineStage, + stage_state: PipelineStageState, + state: PipelineState, + stage_filter: Optional[str], + stages_to_skip_completed: List[str], + chain_broken: bool, + ) -> Optional[str]: + """Decide whether the stage should be skipped, and update state accordingly. + + Returns one of: + + - ``"filtered"`` — ``--stage X`` was set and this is not the + named stage. Status set to ``skipped_by_filter``. + - ``"already_completed"`` — ``--resume-from`` picked this stage + up as already-done; caller seeds ``prev_output`` from + ``stage_state.output_model``. + - ``"chain_broken"`` — a prior stage in the same run reverted + or failed; status set to ``skipped_due_to_prior_revert``. + - ``None`` — proceed to execute. + + Extracted from :meth:`run` for Sonar python:S3776 cognitive- + complexity hygiene. + """ + if stage_filter is not None and stage.name != stage_filter: + stage_state.status = "skipped_by_filter" + stage_state.skipped_reason = f"--stage {stage_filter!r} was specified; only that stage runs." + self._save_state(state) + return "filtered" + if stage.name in stages_to_skip_completed: + return "already_completed" + if chain_broken: + stage_state.status = "skipped_due_to_prior_revert" + stage_state.skipped_reason = ( + f"Stage {state.stopped_at!r} triggered auto_revert; downstream stages did not run." + ) + self._save_state(state) + return "chain_broken" + return None + + def _run_stage_loop( + self, + *, + state: PipelineState, + stage_filter: Optional[str], + stages_to_skip_completed: List[str], + input_model_override: Optional[str], + prev_output_for_filter: Optional[str], + ) -> int: + """Iterate over ``self.pipeline.stages`` and return the aggregated ``worst_exit``. + + Each iteration is one of three things: + + 1. A skipped stage (filter / already-done / chain broken) — see + :meth:`_stage_skip_reason` for the routing rules. + 2. An executed stage — :meth:`_execute_one_stage` runs the + trainer and :meth:`_handle_stage_outcome` records the + transition + audit events. + 3. A gated stage (``EXIT_AWAITING_APPROVAL``) — the outcome + handler sets ``should_break`` and the loop returns early so + downstream stages stay ``pending`` for a subsequent + ``--resume-from``. + + Extracted from :meth:`run` for Sonar python:S3776 cognitive- + complexity hygiene. + """ + worst_exit: int = EXIT_SUCCESS + prev_output: Optional[str] = prev_output_for_filter + chain_broken: bool = False + for i, stage in enumerate(self.pipeline.stages): + stage_state = state.stages[i] + skip_reason = self._stage_skip_reason( + stage=stage, + stage_state=stage_state, + state=state, + stage_filter=stage_filter, + stages_to_skip_completed=stages_to_skip_completed, + chain_broken=chain_broken, + ) + if skip_reason == "already_completed": + prev_output = stage_state.output_model + continue + if skip_reason is not None: + continue + + exit_code = self._execute_one_stage( + index=i, + stage=stage, + state=state, + prev_output=prev_output, + stage_filter=stage_filter, + input_model_override=input_model_override, + ) + worst_exit, new_prev_output, chain_broken_now, should_break = self._handle_stage_outcome( + stage=stage, + stage_state=stage_state, + exit_code=exit_code, + state=state, + worst_exit=worst_exit, + ) + prev_output = new_prev_output if new_prev_output is not None else prev_output + chain_broken = chain_broken or chain_broken_now + if should_break: + break + return worst_exit + def run( self, *, @@ -605,55 +713,15 @@ def run( ) self._notify_pipeline("started", state) - # 4. Iterate stages. The loop body delegates the actual - # per-stage work to ``_execute_one_stage`` + the outcome - # transition to ``_handle_stage_outcome``, so this method's - # cognitive complexity stays bounded. - worst_exit: int = EXIT_SUCCESS - prev_output: Optional[str] = prev_output_for_filter - chain_broken: bool = False - for i, stage in enumerate(self.pipeline.stages): - stage_state = state.stages[i] - - if stage_filter is not None and stage.name != stage_filter: - stage_state.status = "skipped_by_filter" - stage_state.skipped_reason = f"--stage {stage_filter!r} was specified; only that stage runs." - self._save_state(state) - continue - - if stage.name in stages_to_skip_completed: - prev_output = stage_state.output_model - continue - - if chain_broken: - stage_state.status = "skipped_due_to_prior_revert" - stage_state.skipped_reason = ( - f"Stage {state.stopped_at!r} triggered auto_revert; downstream stages did not run." - ) - self._save_state(state) - continue - - exit_code = self._execute_one_stage( - index=i, - stage=stage, - state=state, - prev_output=prev_output, - stage_filter=stage_filter, - input_model_override=input_model_override, - ) - worst_exit, new_prev_output, chain_broken_now, should_break = self._handle_stage_outcome( - stage=stage, - stage_state=stage_state, - exit_code=exit_code, - state=state, - worst_exit=worst_exit, - ) - if new_prev_output is not None: - prev_output = new_prev_output - if chain_broken_now: - chain_broken = True - if should_break: - break + # 4. Iterate stages (delegated so this method stays at a + # bounded cognitive-complexity score). + worst_exit = self._run_stage_loop( + state=state, + stage_filter=stage_filter, + stages_to_skip_completed=stages_to_skip_completed, + input_model_override=input_model_override, + prev_output_for_filter=prev_output_for_filter, + ) # 5+6. Final state + manifest + summary. self._finalise_pipeline(state, worst_exit) diff --git a/tests/test_pipeline_compliance.py b/tests/test_pipeline_compliance.py index c7e68a93..00198ca9 100644 --- a/tests/test_pipeline_compliance.py +++ b/tests/test_pipeline_compliance.py @@ -485,17 +485,25 @@ class TestVerifyAnnexIvPipelineModeExitCodes: single-artefact path's exit-code policy.""" def _run(self, tmp_path, args_overrides: dict) -> int: - """Invoke ``_run_pipeline_mode`` and capture the ``SystemExit`` code.""" + """Invoke ``_run_pipeline_mode`` and capture the ``SystemExit`` code. + + Uses ``pytest.raises(SystemExit)`` rather than a bare + ``try / except SystemExit`` (Sonar python:S5754 / pylint + ``broad-except``). The CLI command always ``sys.exit``s, so we + expect ``SystemExit`` every invocation — a leak past the + context manager would be a real bug worth surfacing. + """ import argparse as _argparse + import pytest as _pytest + from forgelm.cli.subcommands._verify_annex_iv import _run_verify_annex_iv_cmd args = _argparse.Namespace(path=str(tmp_path), pipeline=True, **args_overrides) - try: + with _pytest.raises(SystemExit) as exc_info: _run_verify_annex_iv_cmd(args, "text") - except SystemExit as exc: - return int(exc.code) if exc.code is not None else 0 - return 0 + code = exc_info.value.code + return int(code) if code is not None else 0 def test_missing_manifest_exits_config_error(self, tmp_path, capsys): """``not found`` is operator-actionable input → exit 1.""" From cd9f416d69c87d889ec63ab8b80585cbe0cb3e8a Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 14 May 2026 16:09:59 +0300 Subject: [PATCH 12/22] =?UTF-8?q?fix:=20PR=20#53=20final-review-round=20?= =?UTF-8?q?=E2=80=94=202=20blocking=20+=201=20significant=20+=203=20nitpic?= =?UTF-8?q?ks?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two independent final-review reviewers spotted 6 findings; both were verified against HEAD before any change. All 6 are addressed. 🔴 F-B-1 (Blocking) — Pipeline audit events got different run_ids Verified: AuditLogger.__init__ accepts an optional run_id; without one it auto-generates ``fg-``. ``_audit_event`` constructed a fresh AuditLogger per call, so every entry in the pipeline-level ``audit_log.jsonl`` carried a different top-level ``run_id`` field even though the events shared the same ``pipeline_run_id`` payload. That breaks SIEM-style grouping and the Article 12 record-keeping correlation contract — auditors join on ``run_id``. Fix: cache an AuditLogger on the orchestrator, pinned to the pipeline run id, lazily initialised on the first event. ``_audit_event`` now requires ``pipeline_run_id`` as a keyword-only argument (all 8 existing call sites already pass it, so the new signature is back-compat-safe and prevents silent regression). Regression test: test_audit_events_share_one_pipeline_run_id. 🔴 F-B-2 (Blocking) — --stage manifests fail the chain verifier Verified: ``--stage dpo_stage`` with the predecessor's ``final_model`` on disk runs correctly (auto-chains from the on-disk seed) but the predecessor's manifest entry has ``status: skipped_by_filter`` and ``output_model: null``. The executed stage records ``input_source: "chain"``, so the strict chain verifier (F-B-3 from an earlier cycle) compares against ``stages[X-1].output_model = null`` and reports ``chain_integrity_violation: previous stage has no output_model`` on every legitimate ``--stage `` run. Fix: after ``_resolve_stage_filter_setup`` returns the resolved disk seed, surface it on the predecessor's ``state.stages[filter_idx - 1].output_model`` so the manifest accurately represents "stage X-1 produced this path (in a prior run); stage X chained from it now". Strict chain verifier is happy; stage X-1's status stays ``skipped_by_filter`` to convey "we didn't run it in this invocation". Regression test: test_filter_middle_stage_with_disk_seed_passes_chain_verifier. 🟡 F-S-1 (Significant) — Pipeline state writes raced on shared .tmp Verified: ``_atomic_write_json`` used ``f"{path}.tmp"`` — shared filename per target. Two writers truncate each other's tmp and lose the race to ``os.replace`` with FileNotFoundError. Phase 14 explicitly doesn't support concurrent orchestrators on the same ``pipeline.output_dir`` (documented now under Limitations), but the FileNotFoundError traceback was a worse failure mode than last-writer-wins. Fix: switch to ``f"{path}.{pid}.{rand}.tmp"`` for the temp name. Each writer gets its own tmp file, atomically replaces. Added a defensive cleanup on write failure (orphan tmp removed; original exception re-raised). The temp lives in the same directory as the target so ``os.replace`` stays a same-filesystem atomic rename. 🟢 F-N-1 (Nitpick) — CHANGELOG audit-event count was stale The Added section said "5 new audit events" enumerating the original five; HEAD emits 7 (added stage_gated in F-N-1 and force_resume in F-B-2 from earlier cycles). Updated the count + list; also called out the F-B-1 run_id-pinning fix. 🟢 F-N-2 (Nitpick) — Limitations doc gaps docs/guides/pipeline.md + pipeline-tr.md gained three new bullets: - No multi-process locking on pipeline.output_dir (concurrent orchestrators race; pick distinct dirs). - Stage-vs-pipeline output_dir overlap is allowed but interleaves the audit log (matches the inheritance_matrix.yaml fixture's layout — documented, not a bug). - --force-resume topology guard catches positional breaks only; value-level edits (trainer_type, output_dir, hyperparameters under identical names) are accepted by design. 🟢 F-N-3 (Nitpick) — _validate_resume_state docstring clarity The topology-guard comment now spells out that value-level edits are accepted by design — the operator's explicit signal under ``--force-resume`` that they understand the divergence. Aligns the code's contract with the guide's Limitations bullet. Verification: - ruff format + check clean - 159 / 159 pipeline + webhook tests pass (new audit-run_id + --stage-disk-seed tests added) - All 11 self-review guards green Co-Authored-By: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 9 +- docs/guides/pipeline-tr.md | 24 ++++ docs/guides/pipeline.md | 24 ++++ forgelm/cli/_pipeline.py | 113 ++++++++++++++++--- tests/test_pipeline_orchestrator.py | 169 ++++++++++++++++++++++++++++ 5 files changed, 324 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 52d0c768..7ee7578f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,9 +26,14 @@ _(v0.6.1 dev cycle — entries will land here as PRs merge.)_ end-to-end. Auto-chains each stage's `model.name_or_path` to the previous stage's output, persists state atomically to `/pipeline_state.json` after every transition, - and emits 5 new audit events: `pipeline.started`, + and emits 7 new audit events: `pipeline.started`, `pipeline.stage_started`, `pipeline.stage_completed`, - `pipeline.stage_reverted`, `pipeline.completed`. Existing + `pipeline.stage_gated` (when a stage exits + `EXIT_AWAITING_APPROVAL` — review-cycle F-N-1), + `pipeline.stage_reverted`, `pipeline.force_resume` (operator- + approved stale-config override — review-cycle F-B-2), and + `pipeline.completed`. Every entry's top-level `run_id` is pinned + to the pipeline run id (review-cycle final-round F-B-1). Existing `training.*` per-stage events from `ForgeTrainer` are preserved — pre-existing Slack / Teams dashboards filtering on `training.failure` keep working unchanged. diff --git a/docs/guides/pipeline-tr.md b/docs/guides/pipeline-tr.md index ad6fe071..f01c1b88 100644 --- a/docs/guides/pipeline-tr.md +++ b/docs/guides/pipeline-tr.md @@ -269,6 +269,30 @@ Sıfır olmayan exit kodu ihlalleri keşfedildikleri sırayla listeler. ertelendi. - **Paralel aşama yürütmesi yok.** İki aşama mantıken bağımsız olsa bile orkestratör onları sıralı koşturur. +- **`pipeline.output_dir` üzerinde multi-process lock yok.** Aynı + `pipeline.output_dir`'a karşı eş zamanlı iki `forgelm --config + pipeline.yaml` çağrısı `pipeline_state.json` + manifest üzerinde + race oluşturur — eş zamanlı koşular için ayrı `pipeline.output_dir` + seçin. Atomic-write helper'ı writer başına benzersiz temp adı + kullanır, dolayısıyla race son-yazan-kazanır şeklinde yüzeyleşir + (FileNotFoundError traceback değil), ama kaybedenin per-stage + ilerlemesi kendi process'inde görünmez kalır. +- **Stage ve pipeline `output_dir` çakışması mümkün ama audit log'u + iç içe geçirir.** Bir aşamanın çözülen `training.output_dir`'ı + `pipeline.output_dir`'a eşit olduğunda, aşamanın `training.*` + olayları ile pipeline'ın `pipeline.*` olayları aynı + `audit_log.jsonl` dosyasına düşer (dizin başına bir dosya). + Hash-chain + satır-bazında flock dosyayı iç tutarlı tutar ve + `tests/fixtures/pipeline/inheritance_matrix.yaml` fixture'ı bu + layout'u kasten icra eder; ama dosya başına tek `event_kind` + bekleyen SIEM filtreleri ayrı dizinler kullanmalı. +- **`--force-resume` topology guard yalnızca pozisyonel kırılmaları + yakalar.** Stage ekleme, silme veya yeniden adlandırma + `--force-resume` altında bile reddedilir (state.stages[i] başka + bir aşamayı adresliyor olur). *Value-level* düzenlemeler — aynı + stage adı altında farklı `trainer_type`, `training.output_dir`, + hyperparameter — tasarımı gereği kabul edilir; `--force-resume` + operatörün ayrışmayı anladığını bildiren sinyaldir. - **`forgelm wizard` entegrasyonu yok.** Tek aşamalı config'ler wizard'la üretilebilir; pipeline'lar operatör-seviyesi ve manuel YAML yüzeyini kullanır. Wizard'ın işi "çalışan bir tek aşama diff --git a/docs/guides/pipeline.md b/docs/guides/pipeline.md index d2a9cd49..a1f94e7d 100644 --- a/docs/guides/pipeline.md +++ b/docs/guides/pipeline.md @@ -266,6 +266,30 @@ discovered. deferred. - **No parallel stage execution.** Even when two stages are logically independent, the orchestrator runs them sequentially. +- **No multi-process locking on `pipeline.output_dir`.** Running two + `forgelm --config pipeline.yaml` invocations against the same + `pipeline.output_dir` simultaneously will race on + `pipeline_state.json` + the manifest — pick distinct + `pipeline.output_dir` per concurrent run. The atomic-write helper + uses per-writer-unique temp filenames so the race surfaces as + last-writer-wins (not a `FileNotFoundError` traceback), but the + loser's per-stage progress is invisible to its own process. +- **Stage-vs-pipeline `output_dir` overlap is allowed but interleaves + the audit log.** When a stage's resolved `training.output_dir` + equals `pipeline.output_dir`, the stage's `training.*` events and + the pipeline's `pipeline.*` events land in the same + `audit_log.jsonl` (one file per directory). The hash-chain + per- + line flock keep the merged file internally consistent and the + `tests/fixtures/pipeline/inheritance_matrix.yaml` fixture + deliberately exercises this layout, but SIEM filters that pin a + single `event_kind` per file should pick distinct directories. +- **`--force-resume` topology guard catches positional breaks only.** + Adding, deleting, or renaming a stage between runs is refused even + under `--force-resume` (state.stages[i] would address a different + stage than pipeline.stages[i]). *Value-level* edits — different + `trainer_type`, `training.output_dir`, hyperparameters under + identical stage names — are accepted by design; `--force-resume` + is the operator's signal that they understand the divergence. - **No `forgelm wizard` integration.** Single-stage configs are wizard-buildable; pipelines are operator-grade and use the manual YAML surface. The wizard's job stays "produce a working single-stage diff --git a/forgelm/cli/_pipeline.py b/forgelm/cli/_pipeline.py index cdd88366..5302a207 100644 --- a/forgelm/cli/_pipeline.py +++ b/forgelm/cli/_pipeline.py @@ -210,16 +210,41 @@ def _pipeline_paths(root_cfg: ForgeConfig) -> Dict[str, str]: def _atomic_write_json(path: str, payload: Dict[str, Any]) -> None: """Write ``payload`` to ``path`` atomically. - Pattern: serialise to a sibling ``.tmp`` then ``os.replace`` it + Pattern: serialise to a sibling temp file then ``os.replace`` it onto the final name — guarantees a reader on any concurrent process either sees the previous version or the new version, never a half- written file. + + The temp filename is per-writer-unique (``...tmp``) + rather than a shared ``.tmp`` so two writers targeting the same + final path don't truncate each other's tmp mid-write — pre-fix, a + concurrent invocation produced ``FileNotFoundError`` from + ``os.replace`` (the loser's tmp was unlinked by the winner's + rename). Phase 14 doesn't *support* concurrent orchestrators on + the same ``pipeline.output_dir`` (see ``docs/guides/pipeline.md`` + "Limitations"), but the per-writer tmp is cheap defence so an + accidental double-invocation surfaces as last-writer-wins rather + than a Python traceback. The temp lives in the same directory as + ``path`` so ``os.replace`` stays a same-filesystem atomic rename. """ - os.makedirs(os.path.dirname(path) or ".", exist_ok=True) - tmp = f"{path}.tmp" - with open(tmp, "w", encoding="utf-8") as f: - json.dump(payload, f, indent=2, sort_keys=False) - os.replace(tmp, path) + target_dir = os.path.dirname(path) or "." + os.makedirs(target_dir, exist_ok=True) + tmp = f"{path}.{os.getpid()}.{secrets.token_hex(4)}.tmp" + try: + with open(tmp, "w", encoding="utf-8") as f: + json.dump(payload, f, indent=2, sort_keys=False) + os.replace(tmp, path) + except Exception: + # If the write failed before the replace, clean up the orphan + # temp. After a successful replace there's nothing to remove + # (the tmp name no longer exists), so silently ignore that + # branch. Don't swallow the original exception either; we + # only swallow the cleanup-of-orphan-tmp failure. + try: + os.unlink(tmp) + except OSError: + pass + raise def _serialise_state(state: PipelineState) -> Dict[str, Any]: @@ -283,6 +308,16 @@ def __init__( self.config_hash: str = _compute_pipeline_config_hash(pipeline_yaml_bytes) self.output_format: str = output_format self.paths: Dict[str, str] = _pipeline_paths(root_cfg) + # Lazy-initialised audit-log writer. Pinned to the pipeline run + # id on first event so every ``pipeline.*`` entry in the JSONL + # stream carries the same top-level ``run_id`` (Phase 14 review + # final-round F-B-1). Pre-fix, ``_audit_event`` constructed a + # fresh ``AuditLogger`` per call → every event got a different + # auto-generated ``fg-`` run_id, breaking SIEM-style + # grouping and the Article 12 record-keeping correlation + # contract. The pipeline_run_id is set in state.pipeline_run_id + # before any event fires, so the caller passes it through. + self._audit_logger: Optional[Any] = None # -- run-mode entry points ------------------------------------------------ @@ -701,6 +736,23 @@ def run( if filter_err is not None: return filter_err prev_output_for_filter = seed + # Phase 14 review final-round F-B-2: surface the disk-seed + # path on the *predecessor* stage's manifest entry as well. + # When ``--stage X`` re-runs only the named stage, its + # predecessor is recorded as ``skipped_by_filter`` with no + # ``output_model``. The filtered stage records + # ``input_source: "chain"`` (we auto-chained from the seed), + # so the strict chain verifier (F-B-3) compares stage X's + # ``input_model`` against ``stages[X-1].output_model`` and + # flags a ``chain_integrity_violation`` even though the + # chain is intact on disk. We seed the predecessor's + # ``output_model`` here so the manifest faithfully + # represents "stage X-1 produced this path (in a prior + # run); stage X chained from it now". + if seed is not None: + filter_idx = next(i for i, s in enumerate(self.pipeline.stages) if s.name == stage_filter) + if filter_idx > 0: + state.stages[filter_idx - 1].output_model = seed # 3. Emit pipeline.started + webhook (fresh runs only). if resume_from is None: @@ -824,6 +876,24 @@ def _validate_resume_state(self, existing: PipelineState, *, force: bool) -> Opt ``pipeline.force_resume`` audit event so reviewers see why the divergence was accepted). + Topology vs value-level edits (Phase 14 review final-round + clarification): + + - **Topology change** (adding, deleting, or renaming a stage — + the ordered ``[s.name for s in stages]`` list changes) is + refused **even under** ``--force-resume`` because + ``state.stages[i]`` is positionally addressed and a topology + shift would silently overwrite a different stage's history. + - **Value-level edits** (different ``trainer_type``, + ``training.output_dir``, hyperparameters under unchanged + stage names) trip the hash check but are *accepted* under + ``--force-resume``. This is the documented escape hatch: + the operator is signalling "I know the on-disk state and the + current config diverge; proceed and record the + ``pipeline.force_resume`` audit breadcrumb." Reviewers can + reconstruct the divergence from the event payload's + ``old_config_hash`` / ``new_config_hash`` fields. + Return semantics: - ``None`` — the resume is OK to proceed (no hash divergence, or @@ -855,10 +925,17 @@ def _validate_resume_state(self, existing: PipelineState, *, force: bool) -> Opt # stage inserted, deleted, or renamed between runs would make # ``state.stages[i]`` describe a different stage than # ``self.pipeline.stages[i]`` and the resume would silently - # corrupt the audit trail. Cosmetic edits (whitespace, - # comments, hyperparameter values) trip the hash check but - # *don't* fail this guard, so ``--force-resume`` remains useful - # for them. + # corrupt the audit trail. + # + # *Value-level* edits — different ``trainer_type``, + # ``training.output_dir``, hyperparameters under identical + # stage names — trip the hash check but are *accepted by + # design* under ``--force-resume``. The flag is the operator's + # explicit signal that they understand the on-disk state and + # the current config diverge; the guard's job is only to + # protect the positional-addressing invariant. See + # ``docs/guides/pipeline.md`` "Limitations" for the documented + # contract. on_disk_names = [s.name for s in existing.stages] current_names = [s.name for s in self.pipeline.stages] if on_disk_names != current_names: @@ -940,7 +1017,7 @@ def _save_state(self, state: PipelineState) -> None: manifest = generate_pipeline_manifest(state=state, root_cfg=self.root_cfg) _atomic_write_json(self.paths["manifest_file"], manifest) - def _audit_event(self, event: str, **fields: Any) -> None: + def _audit_event(self, event: str, *, pipeline_run_id: str, **fields: Any) -> None: """Append-only audit log entry under the **pipeline** root output dir. Two separate audit-log streams exist on a multi-stage run: @@ -959,12 +1036,22 @@ def _audit_event(self, event: str, **fields: Any) -> None: with the per-stage ``training_manifest.json`` ``training_run_id``. See ``docs/guides/pipeline.md`` "Audit events" for the full join key matrix. + + Phase 14 review final-round F-B-1: the ``AuditLogger`` instance + is cached on the orchestrator and pinned to the pipeline run + id, so every entry's top-level ``run_id`` field equals the + pipeline run id (instead of a fresh auto-generated + ``fg-`` per event, which broke SIEM-style grouping and + the Article 12 correlation contract). ``pipeline_run_id`` is + now a required keyword argument so callers cannot silently + regress this. """ try: from ..compliance import AuditLogger - audit = AuditLogger(self.paths["root_output_dir"]) - audit.log_event(event, **fields) + if self._audit_logger is None: + self._audit_logger = AuditLogger(self.paths["root_output_dir"], run_id=pipeline_run_id) + self._audit_logger.log_event(event, pipeline_run_id=pipeline_run_id, **fields) except Exception as exc: # noqa: BLE001 — audit emission is best-effort telemetry; a write failure must not abort the pipeline. logger.warning("Failed to emit audit event %r: %s", event, exc) diff --git a/tests/test_pipeline_orchestrator.py b/tests/test_pipeline_orchestrator.py index a5dc0ccd..a69eb86d 100644 --- a/tests/test_pipeline_orchestrator.py +++ b/tests/test_pipeline_orchestrator.py @@ -205,6 +205,67 @@ def test_deserialise_tolerates_unknown_keys(self): assert state.stages[0].name == "s1" +# --------------------------------------------------------------------------- +# Atomic write helper +# --------------------------------------------------------------------------- + + +class TestAtomicWriteJson: + """Phase 14 review final-round F-S-1 regression: ``_atomic_write_json`` + must use a per-writer-unique temp filename so two writers targeting + the same final path don't truncate each other's tmp mid-write.""" + + def test_per_writer_temp_path_does_not_race_on_shared_tmp(self, tmp_path): + """Pre-fix, both writers wrote to ``.tmp`` — a 16-thread + stress probe produced ``FileNotFoundError`` from ``os.replace`` + (the loser's tmp was unlinked by the winner's rename). Post- + fix, each writer's tmp is unique so no traceback escapes; the + race surfaces as last-writer-wins on the final ``path``.""" + from concurrent.futures import ThreadPoolExecutor + + from forgelm.cli._pipeline import _atomic_write_json + + target = str(tmp_path / "shared.json") + errors: list[BaseException] = [] + + def writer(i: int) -> None: + try: + _atomic_write_json(target, {"writer": i, "value": "x" * 1024}) + except BaseException as exc: # noqa: BLE001 — record any escape + errors.append(exc) + + with ThreadPoolExecutor(max_workers=8) as pool: + list(pool.map(writer, range(32))) + + # No writer's tmp-vs-replace race should have escaped. + assert errors == [], f"Concurrent writers produced {len(errors)} exception(s): {errors[:3]}" + # And the final path holds *some* valid JSON (last writer wins). + with open(target) as f: + payload = json.load(f) + assert isinstance(payload, dict) + assert "writer" in payload + + def test_orphan_tmp_cleaned_up_on_serialise_failure(self, tmp_path, monkeypatch): + """If the tmp write itself fails (disk full, permission, etc.), + the orphan tmp must not be left behind under the target dir. + Post-fix the helper's bare-except cleanup ensures we don't + accumulate ``*.tmp`` debris when an upstream caller passes a + non-JSON-serialisable payload.""" + from forgelm.cli._pipeline import _atomic_write_json + + target = str(tmp_path / "bad.json") + # ``json.dump`` cannot serialise a set; this triggers the + # try/except in the helper after the tmp file is created but + # before the replace. + try: + _atomic_write_json(target, {"bad": {1, 2, 3}}) + except TypeError: + pass + # No orphan files left in tmp_path. + orphans = [p for p in os.listdir(str(tmp_path)) if p.endswith(".tmp")] + assert orphans == [], f"Orphan tmp file(s) left behind: {orphans!r}" + + # --------------------------------------------------------------------------- # Orchestrator behaviour — dry-run # --------------------------------------------------------------------------- @@ -505,6 +566,59 @@ def test_filter_with_input_model_override_succeeds(self, tmp_path, monkeypatch): assert len(configs_seen) == 1 assert configs_seen[0].model.name_or_path == override_path + def test_filter_middle_stage_with_disk_seed_passes_chain_verifier(self, tmp_path, monkeypatch): + """Phase 14 review final-round F-B-2 regression: ``--stage X`` + on a non-first stage whose predecessor's ``final_model`` exists + on disk auto-chains correctly, AND the resulting pipeline + manifest must NOT report a ``chain_integrity_violation``. + + Pre-fix, the ``skipped_by_filter`` predecessor had + ``output_model=null`` while the executed stage had + ``input_source=chain``, so the strict chain verifier (F-B-3) + emitted a ``chain_integrity_violation`` on every legitimate + ``--stage `` run. The fix seeds the predecessor's + ``output_model`` in the manifest from the resolved disk path. + """ + from forgelm.compliance import verify_pipeline_manifest_at_path + + cfg = _three_stage_config(tmp_path) + # Pre-create stage 1's on-disk output so the --stage filter can + # seed the chain. + prev_output_dir = tmp_path / "stage1" / "final_model" + prev_output_dir.mkdir(parents=True, exist_ok=True) + configs_seen = _install_trainer_mocks(monkeypatch, [TrainResult(success=True)]) + orch = PipelineOrchestrator(cfg, b"yaml") + code = orch.run(stage_filter="dpo_stage") + assert code == EXIT_SUCCESS + assert len(configs_seen) == 1 + # The DPO stage must have auto-chained from stage 1's on-disk output. + assert configs_seen[0].model.name_or_path == str(prev_output_dir) + + # Inspect the manifest directly to confirm chain integrity is + # intact. ``verify_pipeline_manifest_at_path`` additionally + # checks each completed stage's per-stage training_manifest + # exists; with a mocked trainer that file isn't actually + # written, so we don't assert ``violations == []`` here — we + # assert only the chain-integrity contract that's the subject + # of the F-B-2 regression. Per-stage training_manifest + # existence is covered by ``test_completed_stage_with_missing_ + # training_manifest_flagged`` in test_pipeline_compliance.py. + import json as _json + + with open(orch.paths["manifest_file"]) as f: + manifest = _json.load(f) + sft_stage_payload, dpo_stage_payload, _grpo = manifest["stages"] + assert sft_stage_payload["status"] == "skipped_by_filter" + # The fix: predecessor's output_model is surfaced from the resolved disk path. + assert sft_stage_payload["output_model"] == str(prev_output_dir) + assert dpo_stage_payload["input_source"] == "chain" + assert dpo_stage_payload["input_model"] == str(prev_output_dir) + + violations = verify_pipeline_manifest_at_path(str(orch.paths["root_output_dir"])) + assert not any("chain_integrity_violation" in v for v in violations), ( + f"Chain integrity must hold for --stage with disk seed; got: {violations!r}" + ) + # --------------------------------------------------------------------------- # Orchestrator behaviour — --resume-from @@ -687,6 +801,34 @@ def test_audit_events_emitted_on_happy_path(self, tmp_path, monkeypatch): assert event_names.count("pipeline.stage_completed") == 3 assert "pipeline.completed" in event_names + def test_all_pipeline_events_share_same_top_level_run_id(self, tmp_path, monkeypatch): + """Phase 14 review final-round F-B-1 regression: every entry in + the pipeline-level audit log must carry the same top-level + ``run_id`` (the pipeline run id), so SIEM filters and Article 12 + correlation work on a single field. Pre-fix, ``_audit_event`` + constructed a fresh ``AuditLogger`` per call → each entry got a + different auto-generated ``fg-``.""" + cfg = _three_stage_config(tmp_path) + results = [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)] + _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml") + orch.run() + audit_path = os.path.join(orch.paths["root_output_dir"], "audit_log.jsonl") + with open(audit_path) as f: + entries = [json.loads(line) for line in f if line.strip()] + # Every entry carries a top-level ``run_id`` field; collect the + # set across all pipeline.* events. + pipeline_entries = [e for e in entries if e.get("event", "").startswith("pipeline.")] + assert len(pipeline_entries) >= 5, "expected at least 5 pipeline.* events on a happy 3-stage run" + top_level_ids = {e["run_id"] for e in pipeline_entries} + assert len(top_level_ids) == 1, f"All pipeline.* events must share one top-level run_id, got {top_level_ids!r}" + # The pinned run_id must equal the pipeline_run_id field on the + # entries (which is the contract: pipeline.* events carry the + # pipeline run id in both surfaces). + (top_level_id,) = top_level_ids + pipeline_run_ids = {e["pipeline_run_id"] for e in pipeline_entries} + assert pipeline_run_ids == {top_level_id} + def test_webhook_failures_do_not_abort_pipeline(self, tmp_path, monkeypatch): """A crashing webhook notifier must not derail the pipeline. @@ -701,6 +843,33 @@ def test_webhook_failures_do_not_abort_pipeline(self, tmp_path, monkeypatch): code = orch.run() assert code == EXIT_SUCCESS + def test_audit_events_share_one_pipeline_run_id(self, tmp_path, monkeypatch): + """Phase 14 review final-round F-B-1 regression: every entry in + ``audit_log.jsonl`` emitted by the orchestrator must carry the + **same** top-level ``run_id`` field (= the pipeline run id), so + SIEM dashboards can group all events for one pipeline run by + that single key. Pre-fix, ``_audit_event`` constructed a fresh + ``AuditLogger`` per call and each got a different auto- + generated ``fg-`` run_id.""" + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=True)] * 3) + orch = PipelineOrchestrator(cfg, b"yaml") + orch.run() + audit_path = os.path.join(orch.paths["root_output_dir"], "audit_log.jsonl") + with open(audit_path) as f: + events = [json.loads(line) for line in f if line.strip()] + # All pipeline.* events must have identical top-level run_id. + run_ids = {e.get("run_id") for e in events if e.get("event", "").startswith("pipeline.")} + assert len(run_ids) == 1, ( + f"Pipeline events must share one top-level run_id; got {run_ids!r} across {len(events)} entries." + ) + # And that single run_id must equal the pipeline run id surfaced + # in each event's ``pipeline_run_id`` field. + pipeline_run_ids = {e.get("pipeline_run_id") for e in events if e.get("event", "").startswith("pipeline.")} + assert pipeline_run_ids == run_ids, ( + f"Top-level run_id {run_ids!r} must equal pipeline_run_id {pipeline_run_ids!r}." + ) + def test_audit_event_failures_do_not_abort_pipeline(self, tmp_path, monkeypatch): """Phase 14 review F-N-5 regression: ``_audit_event``'s documented ``except Exception`` swallow path must be reached and From 96be857addd1b1f97548049398fb56c998a6c0e9 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 14 May 2026 16:12:06 +0300 Subject: [PATCH 13/22] =?UTF-8?q?fix:=20PR=20#53=20final-review-round=20?= =?UTF-8?q?=E2=80=94=20SonarCloud=20S5754=20follow-up?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A single CRITICAL python:S5754 finding on cd9f416's new atomic-write race regression test: ``except BaseException`` is the documented anti-pattern (SystemExit / KeyboardInterrupt should propagate). Narrowed to ``except Exception`` — still wide enough to capture any unexpected escape from ``_atomic_write_json`` for the race-probe assertion, but doesn't shadow SystemExit / KeyboardInterrupt the way ``BaseException`` does. The race the test actually targets is an OSError (FileNotFoundError from ``os.replace`` losing to a concurrent unlink), well within ``Exception``. Test still passes. Co-Authored-By: Claude Opus 4.7 (1M context) --- tests/test_pipeline_orchestrator.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/tests/test_pipeline_orchestrator.py b/tests/test_pipeline_orchestrator.py index a69eb86d..be1c3621 100644 --- a/tests/test_pipeline_orchestrator.py +++ b/tests/test_pipeline_orchestrator.py @@ -226,12 +226,17 @@ def test_per_writer_temp_path_does_not_race_on_shared_tmp(self, tmp_path): from forgelm.cli._pipeline import _atomic_write_json target = str(tmp_path / "shared.json") - errors: list[BaseException] = [] + errors: list[Exception] = [] def writer(i: int) -> None: + # OSError covers the FileNotFoundError race that the + # shared-tmp implementation produced; pre-fix this branch + # captured the escapes for the assertion below. ``Exception`` + # is wide-enough to surface any unexpected escape too without + # tripping the Sonar ``catch-BaseException`` rule. try: _atomic_write_json(target, {"writer": i, "value": "x" * 1024}) - except BaseException as exc: # noqa: BLE001 — record any escape + except Exception as exc: # noqa: BLE001 — test probe: record any escape errors.append(exc) with ThreadPoolExecutor(max_workers=8) as pool: From d4040921e89c627eacacff16b34a3384cd8bfa29 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 14 May 2026 16:52:41 +0300 Subject: [PATCH 14/22] docs+site: surface Phase 14 pipelines across README, manuals, marketing site MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 14 multi-stage pipelines (PR #53) merged into development without the promotional / discoverability surface around it. This commit lands that surface so operators looking for "chain my trainers together" actually find the feature. Scope (single content drop — no behavioural changes): 1. README.md — added "Multi-Stage Pipelines" row to the Guides table between Alignment and CI/CD Pipeline Integration, with EN + TR guide links. 2. site/features.html — added a "Multi-Stage Pipelines" feature card in the Training & Alignment group (between GRPO and QLoRA), with a chain-icon SVG and short copy: "Chain SFT → DPO → GRPO in one config-driven run. Auto-chained inputs, per-stage gates, crash-safe resume, one Annex IV manifest covering the whole chain." 3. site/js/translations.js — new `features.train.pipeline.{title,body}` i18n keys in all 6 languages (EN/TR/DE/FR/ES/ZH) for the card. 4. site/index.html — new 4th hero slide (after compliance) using new `home.hero.pipeline.*` keys. Title "SFT → DPO → GRPO, one config.", CTAs to the pipeline guide + YAML example. Slider auto-detects the new slide (DOM-driven in main.js); new pagination dot added with localised aria-label. 5. site/js/translations.js — new `home.hero.pipeline.{title,lead, cta1,cta2}` + `home.hero.dot.pipeline` i18n keys in all 6 languages. Note: pre-existing orphan `home.pipeline.*` keys (trainer-stack pitch from a prior section) are left intact; they describe a different concept (six paradigms) and a separate marketing pass can decide whether to wire or remove them. 6. docs/usermanuals/{en,tr}/training/pipelines.md — new sidebar page under Training & Alignment, mirroring the structure of dpo.md / grpo.md (mermaid decision tree, minimal config, CLI surface, what-the-orchestrator-gives-you, inheritance matrix, limitations, see-also). Sources of truth remain `docs/guides/pipeline{,-tr}.md`; the manual page is the discovery surface inside the sidebar. 7. docs/usermanuals/_meta.yaml — registered the new page (`id: pipelines`) between `grpo` and `lora` in the Training & Alignment section, with localised titles in all 6 languages. 8. docs/usermanuals/{en,tr}/concepts/choosing-trainer.md — added a short note under "Common starting points" pointing every "→"-shaped trainer sequence at the new pipelines page, plus a "See also" entry. 9. site/js/usermanuals/{en,tr,_index}.js — regenerated via tools/build_usermanuals.py (66 pages per language, up from 65). Validation (all green): - ruff format + check - 11-guard gauntlet (dry-run, bilingual parity, anchor resolution, CLI help consistency, wizard defaults, no-analysis-refs, unguarded-sys-modules-pop, site-version, build_usermanuals --check) - 180 / 180 tests across pipeline + bilingual-parity + anchor-resolution Out of scope (deferred): - CHANGELOG move from [Unreleased] → [0.7.0] — release will be cut after this surface refresh is approved. - Wizard (CLI + web) pipeline support — Phase 14 explicit out-of-scope. - Old `home.pipeline.*` orphan i18n keys — separate marketing pass. Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 1 + docs/usermanuals/_meta.yaml | 8 + .../en/concepts/choosing-trainer.md | 3 + docs/usermanuals/en/training/pipelines.md | 149 ++++++++++++++++++ .../tr/concepts/choosing-trainer.md | 3 + docs/usermanuals/tr/training/pipelines.md | 149 ++++++++++++++++++ site/features.html | 5 + site/index.html | 17 ++ site/js/translations.js | 42 +++++ 9 files changed, 377 insertions(+) create mode 100644 docs/usermanuals/en/training/pipelines.md create mode 100644 docs/usermanuals/tr/training/pipelines.md diff --git a/README.md b/README.md index a76c1913..edbf6529 100644 --- a/README.md +++ b/README.md @@ -119,6 +119,7 @@ See the [Quick Start Guide](docs/guides/quickstart.md) for a complete walkthroug | [Document Ingestion](docs/guides/ingestion.md) ([Türkçe](docs/guides/ingestion-tr.md)) | Raw PDF/DOCX/EPUB → SFT-ready JSONL | | [Dataset Audit](docs/guides/data_audit.md) ([Türkçe](docs/guides/data_audit-tr.md)) | Length, language, dedup, cross-split leakage, PII | | [Alignment (DPO/SimPO/KTO/GRPO)](docs/guides/alignment.md) | Complete post-training stack | +| [Multi-Stage Pipelines](docs/guides/pipeline.md) ([Türkçe](docs/guides/pipeline-tr.md)) | Chain SFT → DPO → GRPO in one config-driven run | | [CI/CD Pipeline Integration](docs/guides/cicd_pipeline.md) | GitHub Actions, GitLab CI, Docker | | [Enterprise Deployment](docs/guides/enterprise_deployment.md) | Docker, air-gapped, multi-GPU | | [Safety & Compliance](docs/guides/safety_compliance.md) | EU AI Act, safety evaluation | diff --git a/docs/usermanuals/_meta.yaml b/docs/usermanuals/_meta.yaml index d7b09264..8a1b7e09 100644 --- a/docs/usermanuals/_meta.yaml +++ b/docs/usermanuals/_meta.yaml @@ -118,6 +118,14 @@ sections: title: { en: ORPO, tr: ORPO, de: ORPO, fr: ORPO, es: ORPO, zh: ORPO } - id: grpo title: { en: GRPO, tr: GRPO, de: GRPO, fr: GRPO, es: GRPO, zh: GRPO } + - id: pipelines + title: + en: Multi-Stage Pipelines + tr: Çok Aşamalı Pipeline'lar + de: Mehrstufige Pipelines + fr: Pipelines multi-étapes + es: Pipelines multi-etapa + zh: 多阶段流水线 - id: lora title: en: LoRA, QLoRA, DoRA diff --git a/docs/usermanuals/en/concepts/choosing-trainer.md b/docs/usermanuals/en/concepts/choosing-trainer.md index 5348adb5..16d8b971 100644 --- a/docs/usermanuals/en/concepts/choosing-trainer.md +++ b/docs/usermanuals/en/concepts/choosing-trainer.md @@ -81,6 +81,8 @@ These templates ship with `forgelm quickstart` and reflect what most teams use a | Math reasoning | `grpo-math` | GRPO with format-shaping reward | | Turkish medical Q&A | `medical-qa-tr` | SFT only | +For any "→"-shaped row above, chain the stages with a `pipeline:` config block instead of writing two YAMLs and orchestrating manually. See [Multi-Stage Pipelines](#/training/pipelines). + ## Anti-patterns :::warn @@ -99,3 +101,4 @@ These templates ship with `forgelm quickstart` and reflect what most teams use a - [The Alignment Stack](#/concepts/alignment-overview) — broader context of post-training paradigms. - [Dataset Formats](#/concepts/data-formats) — what JSONL each trainer expects. +- [Multi-Stage Pipelines](#/training/pipelines) — chain two or more trainers into one config-driven run. diff --git a/docs/usermanuals/en/training/pipelines.md b/docs/usermanuals/en/training/pipelines.md new file mode 100644 index 00000000..a82f07cf --- /dev/null +++ b/docs/usermanuals/en/training/pipelines.md @@ -0,0 +1,149 @@ +--- +title: Multi-Stage Pipelines +description: Chain SFT → DPO → GRPO and other trainers into one config-driven run with auto-chaining, per-stage gates, and chain-level Annex IV manifest. +--- + +# Multi-Stage Pipelines + +A *pipeline* chains 2 or more training stages into a single `forgelm` run. Each stage is a single-stage trainer from the trainer's point of view; only the orchestrator knows there's an outer loop. The output of stage N auto-chains as the input of stage N+1, every stage's compliance artefacts roll up into a chain-level manifest, and the operator only needs one CLI invocation to drive the whole flow. + +The canonical chain is **SFT → DPO → GRPO**, but any sequence of supported trainers works (SFT, DPO, SimPO, KTO, ORPO, GRPO). + +## When to use a pipeline + +```mermaid +flowchart TD + Q1{"Running only one
training stage?"} + Q2{"Are stages sequential
(each consumes the
previous output)?"} + Q3{"Want one Annex IV
manifest for the
full chain?"} + Single[Single-stage YAML
v0.6.0 default] + Multi[Hand-orchestrated
shell scripts] + Pipe([Use `pipeline:` block]) + + Q1 -->|Yes| Single + Q1 -->|No| Q2 + Q2 -->|Yes| Q3 + Q2 -->|No, parallel branches| Multi + Q3 -->|Yes| Pipe + Q3 -->|No, just runs| Multi + + classDef question fill:#161a24,stroke:#0ea5e9,color:#e6e7ec + classDef result fill:#1c2030,stroke:#22c55e,color:#e6e7ec + classDef alt fill:#1c2030,stroke:#9ea3b3,color:#e6e7ec + class Q1,Q2,Q3 question + class Pipe result + class Single,Multi alt +``` + +Use a `pipeline:` block when **all** of these apply: + +- 2 or more sequential training stages (e.g. SFT then DPO). +- Each stage's input model is the previous stage's output. +- You want one Annex IV manifest covering the whole chain. + +Don't use a `pipeline:` block if: + +- Only one training paradigm is running. Single-stage configs (the v0.6.0 default) remain the canonical path and preserve their pre-Phase-14 behaviour byte-for-byte. +- The stage dependencies aren't linear (DAG-shaped pipelines). Phase 14 ships sequential pipelines only; the schema reserves headroom for a future DAG expansion. +- Parallel stage execution (concurrent independent branches) is needed. Same horizon as DAG support — Wave 2 or later. + +## Minimal pipeline config + +```yaml +# Root — defaults inherited by stages that don't override them. +model: + name_or_path: "meta-llama/Llama-3-8B" +lora: + r: 8 + alpha: 16 +training: + trainer_type: "sft" + output_dir: "./placeholder" +data: + dataset_name_or_path: "./placeholder.jsonl" + +# The chain itself. +pipeline: + output_dir: "./pipeline_run" + stages: + - name: sft_stage + training: + trainer_type: "sft" + output_dir: "./pipeline_run/stage1_sft" + data: + dataset_name_or_path: "./data/sft.jsonl" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./pipeline_run/stage2_dpo" + dpo_beta: 0.1 + data: + dataset_name_or_path: "./data/preferences.jsonl" + + - name: grpo_stage + training: + trainer_type: "grpo" + output_dir: "./pipeline_run/stage3_grpo" + data: + dataset_name_or_path: "./data/math_prompts.jsonl" +``` + +Each stage's `model.name_or_path` is auto-set to the previous stage's `training.output_dir/final_model` — no hand-editing between stages. + +## CLI surface + +```bash +# Run the whole chain. +forgelm --config pipeline.yaml + +# Validate every stage without GPU allocation — pytest --collectonly style. +forgelm --config pipeline.yaml --dry-run + +# Re-run a single named stage in isolation (audit / re-run scenarios). +forgelm --config pipeline.yaml --stage dpo_stage + +# Resume from a named stage after a crash; already-completed stages skip. +forgelm --config pipeline.yaml --resume-from dpo_stage + +# Operator escape hatch: override the auto-chained input model. +forgelm --config pipeline.yaml --stage dpo_stage --input-model ./other/checkpoint + +# Verify the chain-level Annex IV manifest of a finished run. +forgelm verify-annex-iv --pipeline ./pipeline_run +``` + +## What the orchestrator gives you + +- **Auto-chaining**: stage N's `model.name_or_path` is automatically set to stage N-1's `training.output_dir/final_model`. +- **Crash-safe state**: every transition writes `pipeline_state.json` atomically (tmp + rename). `--resume-from` picks up where the chain stopped. +- **Per-stage gates**: auto-revert (eval regression), human approval, and safety eval compose per stage. A failed gate stops the chain; downstream stages skip with `skipped_due_to_prior_revert`. +- **Audit events**: 7 new event names cover every transition — `pipeline.started`, `pipeline.stage_started`, `pipeline.stage_completed`, `pipeline.stage_gated`, `pipeline.stage_reverted`, `pipeline.force_resume`, `pipeline.completed`. Every entry carries the same top-level `run_id` so SIEM-style grouping works on a single field. +- **Annex IV at chain level**: `compliance/pipeline_manifest.json` indexes every per-stage `training_manifest.json` into one verifiable artefact, hashed and timestamped. +- **Webhook integration**: `notify_pipeline_started`, `notify_pipeline_completed`, `notify_pipeline_reverted` fire alongside the existing per-stage `training.*` notifications — pre-existing Slack/Teams dashboards keep working unchanged. + +## Inheritance matrix at a glance + +| Block | Stage may override? | If stage omits it | +|---|---|---| +| `model` | Yes (wholesale) | Auto-chained from prev stage; root for stage 0 | +| `lora` | Yes (wholesale) | Inherited from root | +| `training` | **`trainer_type` required per stage** | Inherited from root | +| `data` | Yes (recommended per stage) | Inherited from root | +| `evaluation` | Yes (wholesale) | Inherited from root | +| `distributed`, `webhook`, `compliance`, `risk_assessment`, `monitoring`, `retention`, `synthetic`, `merge`, `auth` | **Root-only** — rejected per-stage | Inherited from root | + +Section-wholesale semantics: if a stage names a block it *replaces* root's entirely (no deep-merge). The "trainer_type required per stage" rule is an audit-clarity validator; it ensures every stage records which paradigm it ran. + +## Limitations (Phase 14 Wave 1) + +- No intra-stage checkpoint resume — `--resume-from` works at stage boundaries only. +- Sequential only; no DAG / parallel stage execution. +- No multi-process locking on `pipeline.output_dir` — pick distinct dirs per concurrent run. +- Wizard (CLI + web) does **not** emit `pipeline:` blocks yet; hand-author the YAML or copy the template from `config_template.yaml`. Wizard pipeline support is a future-phase feature. + +## See also + +- [Multi-Stage Pipelines — operator guide](../../../guides/pipeline.md) — full walkthrough with the 6 design fixtures. +- [Choosing a Trainer](../concepts/choosing-trainer.md) — when to pick which trainer; pipelines come after the per-trainer choice. +- [Annex IV Compliance](../compliance/annex-iv.md) — the per-stage manifest format the chain-level manifest indexes. diff --git a/docs/usermanuals/tr/concepts/choosing-trainer.md b/docs/usermanuals/tr/concepts/choosing-trainer.md index 689813e7..e56f9a86 100644 --- a/docs/usermanuals/tr/concepts/choosing-trainer.md +++ b/docs/usermanuals/tr/concepts/choosing-trainer.md @@ -81,6 +81,8 @@ QLoRA bu sayıları ~3-4× düşürür. Full precision'da 22 GB isteyen 7B DPO k | Matematik akıl yürütme | `grpo-math` | format-shaping reward'lı GRPO | | Türkçe medikal Q&A | `medical-qa-tr` | Sadece SFT | +Yukarıdaki "→" işaretli satırlar için, iki ayrı YAML yazıp elle orkestre etmek yerine aşamaları bir `pipeline:` config bloğuyla zincirleyin. Bkz. [Çok Aşamalı Pipeline'lar](#/training/pipelines). + ## Anti-pattern'ler :::warn @@ -99,3 +101,4 @@ QLoRA bu sayıları ~3-4× düşürür. Full precision'da 22 GB isteyen 7B DPO k - [Alignment Yığını](#/concepts/alignment-overview) — daha geniş bağlam. - [Dataset Formatları](#/concepts/data-formats) — her trainer'ın beklediği JSONL. +- [Çok Aşamalı Pipeline'lar](#/training/pipelines) — iki veya daha fazla trainer'ı tek config-tabanlı koşuda zincirle. diff --git a/docs/usermanuals/tr/training/pipelines.md b/docs/usermanuals/tr/training/pipelines.md new file mode 100644 index 00000000..fb1296bf --- /dev/null +++ b/docs/usermanuals/tr/training/pipelines.md @@ -0,0 +1,149 @@ +--- +title: Çok Aşamalı Pipeline'lar +description: SFT → DPO → GRPO ve diğer trainer'ları tek bir config-tabanlı koşuda zincirleyin — otomatik zincirleme, aşama bazında kapılar ve zincir-seviyesi Annex IV manifest ile. +--- + +# Çok Aşamalı Pipeline'lar + +Bir *pipeline*, 2 veya daha fazla eğitim aşamasını tek bir `forgelm` koşusunda zincirler. Her aşama trainer'ın bakış açısından tek aşamalı bir trainer'dır; sadece orkestratör dış döngünün varlığından haberdardır. Aşama N'in çıktısı otomatik olarak aşama N+1'in girdisi olur, her aşamanın uyumluluk artefactları zincir-seviyesi bir manifeste toplanır ve operatörün tüm akışı sürmek için tek bir CLI çağrısı yeterlidir. + +Kanonik zincir **SFT → DPO → GRPO**'dur, ancak desteklenen herhangi bir trainer dizisi (SFT, DPO, SimPO, KTO, ORPO, GRPO) çalışır. + +## Pipeline ne zaman kullanılır + +```mermaid +flowchart TD + Q1{"Sadece tek eğitim
aşaması mı koşuluyor?"} + Q2{"Aşamalar sıralı mı
(her biri öncekinin
çıktısını mı tüketiyor)?"} + Q3{"Tüm zincir için tek
Annex IV manifest
isteniyor mu?"} + Single[Tek aşamalı YAML
v0.6.0 default] + Multi[Elle orkestre
shell script'leri] + Pipe([`pipeline:` bloğunu kullan]) + + Q1 -->|Evet| Single + Q1 -->|Hayır| Q2 + Q2 -->|Evet| Q3 + Q2 -->|Hayır, paralel branch'ler| Multi + Q3 -->|Evet| Pipe + Q3 -->|Hayır, sadece koşular| Multi + + classDef question fill:#161a24,stroke:#0ea5e9,color:#e6e7ec + classDef result fill:#1c2030,stroke:#22c55e,color:#e6e7ec + classDef alt fill:#1c2030,stroke:#9ea3b3,color:#e6e7ec + class Q1,Q2,Q3 question + class Pipe result + class Single,Multi alt +``` + +`pipeline:` bloğunu **tüm** aşağıdaki koşullar geçerliyse kullanın: + +- Sıralı olarak 2 veya daha fazla eğitim aşaması (örn. SFT sonrası DPO). +- Her aşamanın giriş modeli, önceki aşamanın çıkış modelidir. +- Tüm zinciri kapsayan tek bir Annex IV manifest istiyorsunuz. + +`pipeline:` bloğunu kullanmayın eğer: + +- Sadece tek bir eğitim paradigması koşturuyorsanız. Tek aşamalı config'ler (v0.6.0 default'u) kanonik yol olarak kalır ve Phase-14 öncesi davranışlarını byte-byte korur. +- Aşama bağımlılıkları doğrusal değilse (DAG-şekilli pipeline'lar). Phase 14 sadece sıralı pipeline'lar gönderir; şema ileriki bir DAG genişlemesi için yer ayırıyor. +- Paralel aşama yürütmesi (eş zamanlı bağımsız dallar) gerekiyorsa. DAG desteğiyle aynı zaman ufkunda — Wave 2 veya sonrası. + +## Minimal pipeline config + +```yaml +# Root — aşmadığı sürece aşamaların miras aldığı varsayılanlar. +model: + name_or_path: "meta-llama/Llama-3-8B" +lora: + r: 8 + alpha: 16 +training: + trainer_type: "sft" + output_dir: "./placeholder" +data: + dataset_name_or_path: "./placeholder.jsonl" + +# Zincir. +pipeline: + output_dir: "./pipeline_run" + stages: + - name: sft_stage + training: + trainer_type: "sft" + output_dir: "./pipeline_run/stage1_sft" + data: + dataset_name_or_path: "./data/sft.jsonl" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./pipeline_run/stage2_dpo" + dpo_beta: 0.1 + data: + dataset_name_or_path: "./data/preferences.jsonl" + + - name: grpo_stage + training: + trainer_type: "grpo" + output_dir: "./pipeline_run/stage3_grpo" + data: + dataset_name_or_path: "./data/math_prompts.jsonl" +``` + +Her aşamanın `model.name_or_path`'ı önceki aşamanın `training.output_dir/final_model`'ına otomatik ayarlanır — aşamalar arasında elle düzenleme gerekmez. + +## CLI yüzeyi + +```bash +# Tüm zinciri koş. +forgelm --config pipeline.yaml + +# Her aşamayı GPU tahsisi yapmadan doğrula — pytest --collectonly tarzı. +forgelm --config pipeline.yaml --dry-run + +# Tek bir adlı aşamayı yalıtılmış olarak yeniden koştur (audit / re-run senaryoları). +forgelm --config pipeline.yaml --stage dpo_stage + +# Bir crash sonrası adlı aşamadan itibaren devam et; tamamlanmış aşamalar atlanır. +forgelm --config pipeline.yaml --resume-from dpo_stage + +# Operatör kaçış kapısı: otomatik zincirlenen giriş modelini override et. +forgelm --config pipeline.yaml --stage dpo_stage --input-model ./other/checkpoint + +# Biten bir koşunun zincir-seviyesi Annex IV manifesti doğrula. +forgelm verify-annex-iv --pipeline ./pipeline_run +``` + +## Orkestratörün size sağladıkları + +- **Otomatik zincirleme**: aşama N'in `model.name_or_path`'ı otomatik olarak aşama N-1'in `training.output_dir/final_model`'ına ayarlanır. +- **Crash-safe state**: her geçiş `pipeline_state.json`'ı atomik olarak yazar (tmp + rename). `--resume-from`, zincirin durduğu yerden devam alır. +- **Aşama bazında kapılar**: auto-revert (eval regresyonu), insan onayı ve güvenlik değerlendirmesi aşama başına kompoze olur. Başarısız bir kapı zinciri durdurur; alt aşamalar `skipped_due_to_prior_revert` ile atlanır. +- **Audit olayları**: 7 yeni olay adı her geçişi kapsar — `pipeline.started`, `pipeline.stage_started`, `pipeline.stage_completed`, `pipeline.stage_gated`, `pipeline.stage_reverted`, `pipeline.force_resume`, `pipeline.completed`. Her girdi aynı top-level `run_id`'yi taşır, böylece SIEM-tarzı gruplama tek alan üzerinde çalışır. +- **Zincir-seviyesi Annex IV**: `compliance/pipeline_manifest.json` her aşama bazındaki `training_manifest.json`'ı tek doğrulanabilir artefactta indeksler, hash'lenmiş ve zaman damgalı. +- **Webhook entegrasyonu**: `notify_pipeline_started`, `notify_pipeline_completed`, `notify_pipeline_reverted` mevcut aşama bazındaki `training.*` bildirimleriyle yan yana ateşlenir — mevcut Slack/Teams dashboard'ları değişmeden çalışmaya devam eder. + +## Miras matrisi tek bakışta + +| Blok | Aşama override edebilir mi? | Aşama atlarsa | +|---|---|---| +| `model` | Evet (toptan) | Önceki aşamadan otomatik zincirlenir; aşama 0 için root | +| `lora` | Evet (toptan) | Root'tan miras alınır | +| `training` | **`trainer_type` her aşamada zorunlu** | Root'tan miras alınır | +| `data` | Evet (aşama başına önerilir) | Root'tan miras alınır | +| `evaluation` | Evet (toptan) | Root'tan miras alınır | +| `distributed`, `webhook`, `compliance`, `risk_assessment`, `monitoring`, `retention`, `synthetic`, `merge`, `auth` | **Sadece root** — aşamada reddedilir | Root'tan miras alınır | + +Section-wholesale semantiği: bir aşama bir bloğu adıyla anarsa root'unkini *tamamen* değiştirir (deep-merge yok). "Aşama başına trainer_type zorunlu" kuralı bir audit-clarity validator'ıdır; her aşamanın hangi paradigmayı koştuğunu kaydetmesini garanti eder. + +## Sınırlamalar (Phase 14 Wave 1) + +- Aşama içi checkpoint resume yok — `--resume-from` sadece aşama sınırlarında çalışır. +- Sadece sıralı; DAG / paralel aşama yürütmesi yok. +- `pipeline.output_dir` üzerinde çoklu-süreç kilidi yok — eş zamanlı koşular için ayrı dizinler seçin. +- Wizard (CLI + web) henüz `pipeline:` blokları üretmiyor; YAML'ı elle yazın veya `config_template.yaml`'dan şablonu kopyalayın. Wizard pipeline desteği gelecek-faz özelliği. + +## Ayrıca bakınız + +- [Çok Aşamalı Pipeline'lar — operatör kılavuzu](../../../guides/pipeline-tr.md) — 6 tasarım fixture'ı ile tam walkthrough. +- [Trainer Seçimi](../concepts/choosing-trainer.md) — hangi trainer ne zaman seçilir; pipeline'lar trainer seçiminden sonra gelir. +- [Annex IV Uyumluluk](../compliance/annex-iv.md) — zincir-seviyesi manifestin indekslediği aşama-bazında manifest formatı. diff --git a/site/features.html b/site/features.html index 26415b0c..2010d61c 100644 --- a/site/features.html +++ b/site/features.html @@ -165,6 +165,11 @@

ORPO

GRPO

Group Relative Policy Optimization for reasoning RL.

+
+
+

Multi-Stage Pipelines

+

Chain SFT → DPO → GRPO in one config-driven run. Auto-chained inputs, per-stage gates, crash-safe resume, one Annex IV manifest covering the whole chain.

+

QLoRA + DoRA

diff --git a/site/index.html b/site/index.html index b1e55f22..68d884be 100644 --- a/site/index.html +++ b/site/index.html @@ -193,6 +193,20 @@

+ + @@ -206,6 +220,9 @@