diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index dac71126..d1c687da 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -34,9 +34,23 @@ jobs: # breakage early. The notify-failure job dedups against existing # 'nightly-failure' issues to bound noise (see also: testing.md # CI gates discussion). Do NOT add `|| true` here. + # + # This list MUST mirror ``pyproject.toml`` ``[project].dependencies`` + # because the next step (``pip install -e . --no-deps``) does not + # install transitive deps. When a new runtime dependency lands in + # pyproject, add it here too or the matrix job will crash in + # ``import forgelm`` before any test runs. Each entry below maps + # 1:1 to a pyproject line; the comment after each call out the + # source-of-truth. run: | pip install --upgrade pip - pip install torch transformers peft datasets accelerate trl pydantic pyyaml tensorboard huggingface_hub requests + # Core: torch / transformers / peft / datasets / accelerate / + # trl / pydantic / pyyaml / tensorboard / huggingface_hub / + # requests / requests-toolbelt (the last needed by + # forgelm._http for HostHeaderSSLAdapter; SSRF DNS-rebinding + # hardening, issue #14, v0.6.0). + pip install torch transformers peft datasets accelerate trl \ + pydantic pyyaml tensorboard huggingface_hub requests requests-toolbelt pip install pytest pytest-cov ruff - name: Install ForgeLM (editable, no deps — already installed above) @@ -201,6 +215,12 @@ jobs: python-version: "3.10" - name: Install minimum dependency versions + # This list MUST mirror the documented minimums in + # ``pyproject.toml`` ``[project].dependencies`` because the + # next step (``pip install -e . --no-deps``) does not install + # transitive deps. When a new runtime dependency lands in + # pyproject, add it here too or the job will crash in ``import + # forgelm`` before any test runs. run: | pip install --upgrade pip # Install TRL first (most restrictive), let pip resolve transitive deps @@ -210,7 +230,8 @@ jobs: "peft==0.11.0" \ "pydantic==2.0.0" \ "pyyaml==6.0.1" \ - "tensorboard==2.15.0" + "tensorboard==2.15.0" \ + "requests-toolbelt==1.0.0" pip install pytest pytest-cov - name: Install ForgeLM diff --git a/.markdownlint.json b/.markdownlint.json index b410e052..1e1e8696 100644 --- a/.markdownlint.json +++ b/.markdownlint.json @@ -3,8 +3,10 @@ "//1": "MD051 (link-fragments-valid): docs use SPA hash-router routes like `#/data/audit` for in-app navigation; markdownlint's anchor-resolution doesn't understand SPA conventions and would flag every cross-reference.", "//2": "MD014 (commands-show-output): shell examples in operator docs deliberately show only the command (`$ forgelm audit ...`) without sample output — output snippets rot quickly across versions and would force a doc update on every CLI banner change.", "//3": "MD013 (line-length): bilingual (EN+TR) reference + design docs intentionally carry long lines for paragraph-style explanations, table cells, and prose with long inline links. The 80-col limit is a code-style holdover that hurts readability of documentation; the project's bilingual_parity guard already enforces structural alignment, and the rest of the prose stays human-readable at any reasonable width.", + "//4": "MD025 (single-h1): docs/usermanuals//
/.md pages use a YAML frontmatter `title:` AND a body `# Heading` by design — the frontmatter title feeds the sidebar nav and SPA page title, the body H1 is the visible page heading. Every page in docs/usermanuals/ follows this pattern; disabling MD025 keeps the established docs convention intact.", "default": true, "MD051": false, "MD014": false, - "MD013": false + "MD013": false, + "MD025": false } diff --git a/CHANGELOG.md b/CHANGELOG.md index e63de8d3..161950cd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,148 @@ All notable changes to ForgeLM are documented here. ## [Unreleased] -_(v0.6.1 dev cycle — entries will land here as PRs merge.)_ +_(v0.7.1 dev cycle — entries will land here as PRs merge.)_ + +## [0.7.0] — 2026-05-14 + +Phase 14 (Multi-Stage Pipeline Chains) closes the "operators have to write +shell wrappers to chain SFT → DPO → GRPO" gap that's lived in the issue +queue since v0.4. One YAML, one CLI invocation, one Annex IV manifest for +the whole chain — including auto-chained inputs, per-stage gates, +crash-safe resume, and 7 new pipeline-scoped audit events that join on a +single top-level `run_id`. The release also lands a critical SSRF +hardening for outbound webhook / judge / synthetic destinations +(issue [#14](https://github.com/cemililik/ForgeLM/issues/14)). +Single-stage configs reach `forgelm/trainer.py` byte-identical to +v0.6.0; the orchestrator module is never imported when no `pipeline:` +block is present. + +### Added (Phase 14 — Multi-Stage Pipeline Chains) + +- **`pipeline:` config block** at the root of `ForgeConfig` chains + one or more training stages (typically SFT → DPO → GRPO) into one + config-driven run. New Pydantic models `PipelineStage` and + `PipelineConfig` enforce stage-name uniqueness, `^[a-z0-9_]{1,32}$` + identifier shape, **at least 1 stage** (`min_length=1` — single-stage + ergonomics are still better served by omitting the `pipeline:` + block, but the schema accepts a 1-element pipeline so an operator + can iterate up to a multi-stage chain without re-shaping the YAML), + and explicit-`trainer_type` audit-clarity validation per stage. Section-wholesale inheritance: + `model` / `lora` / `training` / `data` / `evaluation` blocks + inherit from the root if omitted or fully replace it when set. + `distributed` / `webhook` / `compliance` / `risk_assessment` / + `monitoring` / `retention` / `synthetic` / `merge` / `auth` are + root-only and rejected per-stage with `EXIT_CONFIG_ERROR (1)`. +- **`forgelm/cli/_pipeline.py` orchestrator** drives the chain + end-to-end. Auto-chains each stage's `model.name_or_path` to the + previous stage's output, persists state atomically to + `/pipeline_state.json` after every transition, + and emits 7 new audit events: `pipeline.started`, + `pipeline.stage_started`, `pipeline.stage_completed`, + `pipeline.stage_gated` (when a stage exits + `EXIT_AWAITING_APPROVAL` — review-cycle F-N-1), + `pipeline.stage_reverted`, `pipeline.force_resume` (operator- + approved stale-config override — review-cycle F-B-2), and + `pipeline.completed`. Every entry's top-level `run_id` is pinned + to the pipeline run id (review-cycle final-round F-B-1). Existing + `training.*` per-stage events from `ForgeTrainer` are preserved — + pre-existing Slack / Teams dashboards filtering on `training.failure` + keep working unchanged. +- **CLI flags** `--stage ` (single-stage filter for audit / + re-run), `--resume-from ` (stage-boundary resume), + `--force-resume` (stale-config-hash override), and `--input-model + ` (auto-chain escape hatch, recorded with `input_source: + cli_override` in the audit log). +- **`--dry-run` multi-stage validation** — when the config carries a + `pipeline:` block, dry-run validates every stage's merged config + + the chain-integrity assertion (stage N's auto-chained input is + under stage N-1's `output_dir`). Errors are collected before + exiting (à la `pytest --collectonly`). +- **Pipeline Annex IV manifest** — `compliance/pipeline_manifest.json` + at `` indexes per-stage `training_manifest.json` + files into one verifiable chain artefact, carrying + `pipeline_run_id` + `pipeline_config_hash` (SHA-256 of the YAML + bytes) for reproducibility. `forgelm verify-annex-iv --pipeline + ` walks the index, checks chain integrity, and asserts + every referenced per-stage manifest exists on disk. +- **Webhook notifications** — `WebhookNotifier.notify_pipeline_started`, + `notify_pipeline_completed`, and `notify_pipeline_reverted` + mirror the orchestrator's audit events; gated by the existing + `notify_on_start` / `notify_on_success` / `notify_on_failure` + config flags so operators silencing per-stage events also silence + the pipeline-level pings. +- **Bilingual operator guide** — new `docs/guides/pipeline.md` + + `docs/guides/pipeline-tr.md` (canonical "first pipeline" walkthrough, + inheritance matrix, CLI semantics, Annex IV verifier flow, + Limitations section). `config_template.yaml` gains a commented-out + `pipeline:` example block. + +### Changed (Phase 14) + +- **Backward compatibility preserved.** A config file without a + `pipeline:` key reaches `forgelm/trainer.py` byte-identically to + v0.6.0; the orchestrator module is never imported. +- `compliance.py` exports four new symbols: `generate_pipeline_manifest` + (called by the orchestrator after every transition), + `export_pipeline_manifest` (atomic write to + `/compliance/pipeline_manifest.json`), + `verify_pipeline_manifest(manifest: dict) -> List[str]` (in-memory + structural + chain-integrity check, returns a list of human-readable + violations — empty when valid), and `verify_pipeline_manifest_at_path + (pipeline_dir: str) -> List[str]` (disk-backed wrapper used by the + `forgelm verify-annex-iv --pipeline ` CLI mode). The shared + validator lives in private helper `_verify_manifest_payload` so the + in-memory and disk-bound entry points cannot drift. + +### Fixed (Phase 14 review-response — PR #53) + +Addresses the consolidated reviewer-pass findings (3 blocking + +4 significant + Gemini's 3 inline comments) against the initial Phase +14 merge candidate: + +- **F-B-1** — `forgelm --config pipeline.yaml --dry-run` now reaches + the pipeline orchestrator's per-stage validator instead of the + legacy single-stage dry-run path; the `_dispatch.py` ordering was + inverted (no-train-mode ran before the pipeline branch). +- **F-B-2** — `--force-resume` now emits a `pipeline.force_resume` + audit event carrying `old_config_hash` + `new_config_hash` so + compliance reviewers can distinguish an operator-approved override + from a normal resume (was previously a WARNING log only — + invisible in the append-only JSONL stream). +- **F-B-3** — pipeline manifest verifier now compares every chain + stage against its **immediate** predecessor, not the most-recent + stage with an `output_model`. Pre-fix, a manifest where stage N-1 + failed without saving an `output_model` could appear to chain + stage N from stage N-2, masking the broken link. +- **F-F-1 / F-G-3 (Gemini)** — auto-chain existence guard now + checks the resolved model directory itself, not its parent — used + to accept `./stage1/` even when `./stage1/final_model` was + missing. +- **F-F-2** — `--input-model ""` (empty string) is normalised to + `None` at dispatch time so it no longer slips past the truthy + guard and silently overwrites the auto-chained model path with an + empty string. +- **F-F-3** — `_validate_resume_state` now returns its refusal + through the normal `run()` flow instead of `sys.exit`-ing mid-method, + so every refusal path emits the same audit-log finalisation and + the test surface uses one uniform `assert code == EXIT_CONFIG_ERROR` + shape. +- **F-S-2 / F-N-2** — verifier surfaces stages still in `running` + status on a finalised manifest as a `chain_integrity_violation` + candidate (a tell of a crashed orchestrator). Also `_load_state_file` + catches `KeyError` / `TypeError` / `ValueError` / `AttributeError` + around `_deserialise_state` so a tampered state file produces an + actionable config error rather than a Python traceback. +- **F-N-1** — gated stages now emit a dedicated + `pipeline.stage_gated` audit event (was `pipeline.stage_completed` + with a `gate_decision=approval_pending` sub-field). Lets + dashboard / SIEM filters distinguish the gate flow on the event + name alone. +- **F-G-1 (Gemini)** — pipeline dry-run now flags `training.output_dir` + collisions across stages (two stages writing to the same dir would + silently overwrite each other's checkpoints + per-stage Annex IV + manifests). Covers both the explicit-override and inherited-from- + root collision cases. ### Security @@ -1932,7 +2073,8 @@ Major release: ForgeLM goes from a basic SFT fine-tuning tool to a full-stack LL - Basic evaluation checks (max loss, baseline comparison) - Auto-revert on quality degradation -[Unreleased]: https://github.com/cemililik/ForgeLM/compare/v0.6.0...HEAD +[Unreleased]: https://github.com/cemililik/ForgeLM/compare/v0.7.0...HEAD +[0.7.0]: https://github.com/cemililik/ForgeLM/compare/v0.6.0...v0.7.0 [0.6.0]: https://github.com/cemililik/ForgeLM/compare/v0.5.7...v0.6.0 [0.5.7]: https://github.com/cemililik/ForgeLM/compare/v0.5.6...v0.5.7 [0.5.6]: https://github.com/cemililik/ForgeLM/compare/v0.5.5...v0.5.6 diff --git a/README.md b/README.md index a76c1913..edbf6529 100644 --- a/README.md +++ b/README.md @@ -119,6 +119,7 @@ See the [Quick Start Guide](docs/guides/quickstart.md) for a complete walkthroug | [Document Ingestion](docs/guides/ingestion.md) ([Türkçe](docs/guides/ingestion-tr.md)) | Raw PDF/DOCX/EPUB → SFT-ready JSONL | | [Dataset Audit](docs/guides/data_audit.md) ([Türkçe](docs/guides/data_audit-tr.md)) | Length, language, dedup, cross-split leakage, PII | | [Alignment (DPO/SimPO/KTO/GRPO)](docs/guides/alignment.md) | Complete post-training stack | +| [Multi-Stage Pipelines](docs/guides/pipeline.md) ([Türkçe](docs/guides/pipeline-tr.md)) | Chain SFT → DPO → GRPO in one config-driven run | | [CI/CD Pipeline Integration](docs/guides/cicd_pipeline.md) | GitHub Actions, GitLab CI, Docker | | [Enterprise Deployment](docs/guides/enterprise_deployment.md) | Docker, air-gapped, multi-GPU | | [Safety & Compliance](docs/guides/safety_compliance.md) | EU AI Act, safety evaluation | diff --git a/config_template.yaml b/config_template.yaml index 6a7cfd83..fe35b495 100644 --- a/config_template.yaml +++ b/config_template.yaml @@ -223,3 +223,33 @@ data: # ephemeral_artefact_retention_days: 90 # compliance bundles + audit reports # raw_documents_retention_days: 90 # ingested raw docs (PDF/DOCX/EPUB/TXT/MD) # enforce: "log_only" # "log_only" | "warn_on_excess" | "block_on_excess" + +# Phase 14 — Multi-stage training pipeline chains (SFT → DPO → GRPO). +# Uncomment the ``pipeline:`` block below to run multiple training stages +# back-to-back; each stage's input model is auto-chained to the previous +# stage's output_dir/final_model. Stages inherit unset config blocks +# from the root (above). See docs/guides/pipeline.md. +# pipeline: +# output_dir: "./pipeline_run" # pipeline-level dir (state + manifest live here) +# stages: +# - name: sft_stage +# training: +# trainer_type: "sft" +# output_dir: "./pipeline_run/stage1_sft" +# num_train_epochs: 3 +# data: +# dataset_name_or_path: "./data/sft.jsonl" +# - name: dpo_stage +# training: +# trainer_type: "dpo" +# output_dir: "./pipeline_run/stage2_dpo" +# num_train_epochs: 1 +# dpo_beta: 0.1 +# data: +# dataset_name_or_path: "./data/preferences.jsonl" +# - name: grpo_stage +# training: +# trainer_type: "grpo" +# output_dir: "./pipeline_run/stage3_grpo" +# data: +# dataset_name_or_path: "./data/math_prompts.jsonl" diff --git a/docs/guides/ingestion-tr.md b/docs/guides/ingestion-tr.md index 3183e5bd..6bef76f1 100644 --- a/docs/guides/ingestion-tr.md +++ b/docs/guides/ingestion-tr.md @@ -490,7 +490,7 @@ Wave 3 backlog'undadır. > (akademik posterler, geniş oluklu regülasyon yayınları) güvenilirdir > ama yayın-kalite iki-kolonlu makaleleri kaçırır. Histogram-tabanlı > bimodal-mode refactor Wave 3 takipi olarak izlenmektedir (bkz. -> [`docs/roadmap/phase-15-ingestion-reliability.md`](../roadmap/phase-15-ingestion-reliability.md) +> [Faz 15 arşivi](../roadmap/completed-phases.md#phase-15-ingestion-pipeline-reliability-v060) > Wave 3 — multi-column layout extraction). ### Markdown-bilen splitter — `--strategy markdown` (Faz 12) diff --git a/docs/guides/ingestion.md b/docs/guides/ingestion.md index f138da29..e3d469f5 100644 --- a/docs/guides/ingestion.md +++ b/docs/guides/ingestion.md @@ -481,7 +481,7 @@ is on the Wave 3 backlog. > wide-gutter regulatory publications) but misses publication-grade > two-column papers. A histogram-based bimodal-mode refactor is tracked > as a Wave 3 follow-up (see -> [`docs/roadmap/phase-15-ingestion-reliability.md`](../roadmap/phase-15-ingestion-reliability.md) +> [Phase 15 archive](../roadmap/completed-phases.md#phase-15-ingestion-pipeline-reliability-v060) > Wave 3 — multi-column layout extraction). ### Markdown-aware splitter — `--strategy markdown` (Phase 12) diff --git a/docs/guides/pipeline-tr.md b/docs/guides/pipeline-tr.md new file mode 100644 index 00000000..84f520da --- /dev/null +++ b/docs/guides/pipeline-tr.md @@ -0,0 +1,316 @@ +# Çok Aşamalı Eğitim Pipeline'ları + +Faz 14 — `v0.7.0` ile geliyor (CHANGELOG kaydı 2026-05-14; PyPI tag bekleniyor). + +ForgeLM'in `pipeline:` config bloğu, 2 veya daha fazla eğitim aşamasını +(genellikle SFT → DPO → GRPO) tek bir config-tabanlı, dry-run-doğrulanabilir, +Annex IV-izlenebilir koşuya zincirler. Faz 14'ten önce aynı iş 3 ya da +daha fazla ayrı config dosyası, aşamalar arası elle `model.name_or_path` +düzenleme ve dışarıdan bir shell betiğiyle orkestrasyon gerektiriyordu. +Faz 14'ten sonra tek bir `forgelm --config pipeline.yaml` çağrısı tüm +zinciri uçtan uca yürütür. + +--- + +## When to use a pipeline (and when not to) + +`pipeline:` bloğunu **tüm** aşağıdaki koşullar geçerliyse kullanın: + +- Sıralı olarak 2 veya daha fazla eğitim aşaması koşturuyorsunuz + (örn. SFT sonrası DPO). +- Her aşamanın giriş modeli, önceki aşamanın çıkış modelidir. +- Tüm zinciri kapsayan tek bir Annex IV manifest istiyorsunuz. + +`pipeline:` bloğunu kullanmayın eğer: + +- Sadece tek bir eğitim paradigması koşturuyorsanız. Tek aşamalı + config'ler (v0.6.0 default'u) hâlâ kanonik kullanım örneğidir ve + Faz-14 öncesi davranışlarını byte-byte korur. +- Doğrusal olmayan aşama bağımlılıkları (DAG-şekilli pipeline'lar) + istiyorsanız. Faz 14 sadece sıralı pipeline'lar gönderir; şema + ilerideki bir DAG genişlemesi için yüzeyi koruyor. +- Paralel aşama yürütmesi (eşzamanlı bağımsız dallar) istiyorsanız. + DAG desteğiyle aynı zaman ufkunda — Wave 2 veya sonrası. + +--- + +## Anatomy of a pipeline config + +```yaml +# Root config — her aşamanın aşmadığı sürece miras aldığı varsayılanları +# sağlar. +model: + name_or_path: "meta-llama/Llama-3-8B" # Aşama 0'ın başlangıç modeli +lora: + r: 8 + alpha: 16 +training: + trainer_type: "sft" # Training bloğunu miras alan + # aşamalar bunu kullanır +data: + dataset_name_or_path: "./placeholder.jsonl" # Root'ta zorunlu; her + # aşama kendi data'sını verir + +# Pipeline-level blok — zinciri tanımlar. +pipeline: + output_dir: "./pipeline_run" # pipeline_state.json, + # compliance/pipeline_manifest.json + # ve pipeline-level audit_log.jsonl + # buraya yazılır + stages: + - name: sft_stage # ^[a-z0-9_]{1,32}$ + training: + trainer_type: "sft" # Her aşama için zorunlu + output_dir: "./pipeline_run/stage1_sft" + num_train_epochs: 3 + data: + dataset_name_or_path: "./data/sft.jsonl" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./pipeline_run/stage2_dpo" + num_train_epochs: 1 + dpo_beta: 0.1 + data: + dataset_name_or_path: "./data/preferences.jsonl" + + - name: grpo_stage + training: + trainer_type: "grpo" + output_dir: "./pipeline_run/stage3_grpo" + grpo_num_generations: 4 + data: + dataset_name_or_path: "./data/math_prompts.jsonl" +``` + +**Otomatik zincirleme:** her aşamanın `model.name_or_path` değeri, +önceki aşamanın `training.output_dir/final_model` yoluna otomatik +olarak ayarlanır — bu yolu elinizle yazmazsınız ve aşama N'in DPO +eğiticisini yanlışlıkla base modele (aşama N-1'in SFT çıktısı yerine) +yönlendiremezsiniz. + +--- + +## Inheritance matrix + +Section-wholesale override semantiği — bir aşama üst seviye blok (`model`, +`lora`, `training`, `data`, `evaluation`) tanımlarsa **tüm** blok root'unkini +değiştirir; aşama bloğu atlarsa root'un bloğu birebir miras alınır. Field +seviyesinde deep-merge yok: "miras almak istiyorsan bloğu yazma; override +etmek istiyorsan tam bloğu yaz." + +| Bölüm | Miras davranışı | Notlar | +|---|---|---| +| `model.name_or_path` | **Otomatik zincirlenir** (aşama 1'den itibaren root'u geçer) | Önceki aşamanın `training.output_dir/final_model` değerine ayarlanır. Aşama 0 root'un `model.name_or_path`'ini okur. Aşamada explicit `model:` bloğu otomatik zincirlemeyi o aşama için devre dışı bırakır (kaçış kapısı). | +| `model.*` (diğer alanlar) | `model:` bloğu override edilmediyse miras alınır | `backend`, `load_in_4bit`, `trust_remote_code`, `max_length`, `chat_template` root'u izler. | +| `lora` | `lora:` bloğu override edilmediyse miras alınır | **Kritik edge case:** aşama 2 DPO, aşama 1 SFT'den *farklı* bir `lora.r` ile gelirse aşama 2 **birleştirilmiş SFT modelinin üstüne taze bir LoRA**dır, SFT'nin LoRA'sının devamı değil. Operatör-görünür davranış. | +| `data` | `data:` bloğu override edilmediyse miras alınır (önerilmez) | Şema aşama bazında override'ı **zorunlu kılmıyor** — operatörler farklı bir trainer'a karşı aynı veri setini ablation için yeniden koşturabilir — ama production'da aşamalar arası aynı dataset'i kullanan pipeline'lar son derece nadirdir. Operator kılavuzu inheritance'ı bir kötü-koku olarak işaretler; gerçek pipeline'ların neredeyse tümünde SFT küratör SFT dataset'i kullanır, DPO preference-pairs dataset'i, GRPO matematik/ödül dataset'i. | +| `training` | `training:` bloğu override edilmediyse miras alınır | `trainer_type` her aşamada explicit set EDİLMEK ZORUNDADIR (audit-clarity doğrulayıcısı). Diğer alanlar blok override edildiğinde wholesale yer değiştirir. | +| `evaluation` | `evaluation:` bloğu override edilmediyse miras alınır | Aşama bazında gate'ler (loss eşikleri, auto_revert, safety, judge, human-approval) burada yaşar. | +| `distributed` | Sadece root — **aşamada reddedilir** | Distributed strateji koşu boyunca tutarlı olmak zorundadır. | +| `webhook` | Sadece root — **aşamada reddedilir** | Aşama bazında olaylar payload'da aşama adını taşır. | +| `compliance` | Sadece root — **aşamada reddedilir** | Provider/system/risk metadata pipeline seviyesindedir. | +| `risk_assessment`, `monitoring`, `retention`, `synthetic`, `merge`, `auth` | Sadece root — **aşamada reddedilir** | Pipeline seviyesi sorumlulukları. | + +Bir aşama root-only bölümlerden birini tanımlarsa, config yükleme +zamanında `EXIT_CONFIG_ERROR (1)` ile reddedilir ve hata mesajı +ihlal eden bölüm adını taşır. + +--- + +## CLI + +```bash +# Uçtan uca pipeline koşusu. +forgelm --config pipeline.yaml + +# Dry-run her aşamayı (Pydantic + chain integrity) hiçbir GPU +# tahsisi olmadan doğrular. `pytest --collectonly` gibi tüm +# hataları çıkmadan önce toplar. +forgelm --config pipeline.yaml --dry-run + +# Tek bir adlı aşamayı yalıtılmış olarak koştur (audit / yeniden +# koşturma senaryoları). İlk olmayan aşamalar önceki aşamanın +# disk üzerindeki çıktısını veya explicit --input-model override +# gerektirir. +forgelm --config pipeline.yaml --stage dpo_stage + +# Başarısız / kesintiye uğramış bir koşudan adlı aşamadan itibaren +# devam et. output_model yolu disk üzerinde var olan +# already-completed aşamalar atlanır (INFO log'lanır). +forgelm --config pipeline.yaml --resume-from dpo_stage + +# Tek aşama için otomatik zincirlenen input modeli override et +# (kaçış kapısı). Audit log `input_source: cli_override` kaydeder. +forgelm --config pipeline.yaml --stage dpo_stage --input-model ./other/checkpoint +``` + +### `--stage ` partial-run rules + +| Senaryo | Davranış | +|---|---| +| `--stage ` ve `` ilk aşama | Root `model.name_or_path` okunur; normal koşar. | +| `--stage ` ve önceki aşamanın `output_dir/final_model` diskte var | Önceki aşama yeni bitmiş gibi otomatik zincirlenir. | +| `--stage ` ve önceki aşamanın çıktısı eksik | `EXIT_CONFIG_ERROR (1)` ile hard-fail: `Stage requires output at ; pass --input-model to override or run the full pipeline first.` Root `model.name_or_path`'e sessiz geri dönüş yok. | +| `--stage --input-model ` | Operatör kaçış kapısı: otomatik zincirlemeyi atlar, ``'i kullanır. Audit log `input_source: cli_override` kaydeder. | +| `--stage ` ve `` config'te yok | Parse zamanında valid aşama adları listesiyle hard-fail. | + +### `--resume-from ` semantics + +- State dosyası: `/pipeline_state.json` (atomic-write). +- `status: completed` ve `output_model` yolu hâlâ diskte olan aşamalar + **atlanır** ve aşama bazında training manifest'leri korunur. Adlı + aşama ve sonrasındaki her aşama yeniden koşturulur. +- **Stale-state koruması:** disk üzerindeki state dosyasının + `pipeline_config_hash`'i mevcut YAML byte'larından farklıysa, resume + `EXIT_CONFIG_ERROR (1)` ile başarısız olur — bu "koşu sırasında + config'i düzenledim ve resume ettim" sessiz divergenc'yi engeller. + `--force-resume` ile override edilebilir (WARNING log'lanır, audit + event kaydedilir). +- Faz 14 sadece **aşama sınırlarında** resume eder. Aşama içi HF + `Trainer.train(resume_from_checkpoint=...)` entegrasyonu bir Faz 14.x + takipçisine ertelendi — Sınırlamalar bölümüne bakın. + +--- + +## Human-approval gate within a pipeline + +Bir aşama `evaluation.require_human_approval: true` taşıyorsa, ForgeLM'in +mevcut Faz 9 akışı aynen koşar: model `final_model.staging./`'ye +iner, orkestratör staging yolunu pipeline state'inde +`status: gated_pending_approval` ile yakalar ve koşu +`EXIT_AWAITING_APPROVAL (4)` ile çıkar. + +Sonraki aşamalar `pending` kalır (`skipped_due_to_prior_revert` değil) +böylece onaydan sonra resume onları alır: + +```bash +# Aşama 1 SFT onay bekliyor — pipeline 4 ile çıkar. +$ forgelm --config pipeline.yaml +# ... → exit 4 + +# Operatör staged modeli inceler, sonra onaylar. +$ forgelm approve --output-dir ./pipeline_run/stage1_sft +# ... → final_model/ promote edilir, exit 0 + +# Pipeline'ı DPO'dan resume et; SFT atlanır (status=completed on disk). +# DPO'nun input_model'i staging değil *promote edilen* final_model/ +# yolunu işaret eder. +$ forgelm --config pipeline.yaml --resume-from dpo_stage +# ... → zincirin geri kalanı başarılıysa exit 0 +``` + +--- + +## Audit events + +Orkestratör bu olayları pipeline-level `audit_log.jsonl`'a (pipeline.output_dir +altında) emit eder: + +- `pipeline.started` — run id, config hash, stage count, stage names +- `pipeline.stage_started` — stage name, index, input model, input source +- `pipeline.stage_completed` — stage name, gate decision (`passed` / `failed`), metrics summary +- `pipeline.stage_gated` — stage name, gate decision `approval_pending`, staging path (bir aşama `EXIT_AWAITING_APPROVAL` ile çıkış yaptığında emit edilir) +- `pipeline.stage_reverted` — stage name, auto-revert reason +- `pipeline.force_resume` — operatör onaylı stale-hash override, `old_config_hash` + `new_config_hash` taşır +- `pipeline.completed` — final status, stopped-at stage (varsa) + +Bu olaylar mevcut her aşamanın `ForgeTrainer`'inin emit etmeye devam +ettiği `training.*` olaylarının yanında yaşar; `training.failure` +üzerinde filtreleyen Slack / Teams dashboard'ları değişmeden çalışır. + +İlgili webhook metotları `notify_pipeline_started`, +`notify_pipeline_completed` ve `notify_pipeline_reverted` — +`forgelm/webhook.py`'a bakın. + +--- + +## Annex IV manifest + +Her aşama geçişi `/compliance/pipeline_manifest.json` +dosyasını (atomic-write) yeniden yazar. Pipeline manifest, aşama bazında +`training_manifest.json` dosyalarını doğrulanabilir bir zincire bağlayan +**indekstir**; aşama bazında manifest'ler tek aşamalı Annex IV şemasına +karşı bireysel olarak geçerli kalmaya devam eder. + +Tam bir koşuyu doğrulayıcı ile geçerleyin: + +```bash +forgelm verify-annex-iv --pipeline ./pipeline_run +``` + +Doğrulayıcı şu koşullar sağlandığında exit 0 döner: + +1. Her zorunlu üst seviye anahtar mevcut ve iyi biçimli. +2. Aşama indeksleri sırayla `0..N-1`. +3. Her chain aşamasının `input_model`'i önceki yürütülen aşamanın + `output_model`'iyle eşit (operatör `--input-model` override'ları + `input_source: cli_override` ile kaydedilir ve bu kontrolden + tasarımla muaftır — auditor'lar legitimate override'ı bozuk + manifest'ten ayırmak için audit log'a bakar). +4. Her aşama bazında `training_manifest` yolu gerçek bir dosyayı + işaret eder. +5. `stopped_at` (set ise) gerçek bir aşamayı adlandırır ve o aşamanın + status'u `failed` veya `gated_pending_approval`. + +Sıfır olmayan exit kodu ihlalleri keşfedildikleri sırayla listeler. + +--- + +## Limitations (Phase 14 Wave 1) + +- **Aşama içi checkpoint resume yok.** `--resume-from` sadece aşama + sınırlarında devam alır. Bir aşamanın `ForgeTrainer.train()` + yarıda crash ederse, resume o aşamayı epoch 0'dan yeniden koşturur. + Wave 2. +- **Sadece sıralı — DAG semantiği yok.** Aşamalar `pipeline.stages` + içinde görünme sıralarında yürütülür. Branch / fan-out / merge + ertelendi. +- **Paralel aşama yürütmesi yok.** İki aşama mantıken bağımsız olsa + bile orkestratör onları sıralı koşturur. +- **`pipeline.output_dir` üzerinde multi-process lock yok.** Aynı + `pipeline.output_dir`'a karşı eş zamanlı iki `forgelm --config + pipeline.yaml` çağrısı `pipeline_state.json` + manifest üzerinde + race oluşturur — eş zamanlı koşular için ayrı `pipeline.output_dir` + seçin. Atomic-write helper'ı writer başına benzersiz temp adı + kullanır, dolayısıyla race son-yazan-kazanır şeklinde yüzeyleşir + (FileNotFoundError traceback değil), ama kaybedenin per-stage + ilerlemesi kendi process'inde görünmez kalır. +- **Stage ve pipeline `output_dir` çakışması mümkün ama audit log'u + iç içe geçirir.** Bir aşamanın çözülen `training.output_dir`'ı + `pipeline.output_dir`'a eşit olduğunda, aşamanın `training.*` + olayları ile pipeline'ın `pipeline.*` olayları aynı + `audit_log.jsonl` dosyasına düşer (dizin başına bir dosya). + Hash-chain + satır-bazında flock dosyayı iç tutarlı tutar ve + `tests/fixtures/pipeline/inheritance_matrix.yaml` fixture'ı bu + layout'u kasten icra eder; ama dosya başına tek `event_kind` + bekleyen SIEM filtreleri ayrı dizinler kullanmalı. +- **`--force-resume` topology guard yalnızca pozisyonel kırılmaları + yakalar.** Stage ekleme, silme veya yeniden adlandırma + `--force-resume` altında bile reddedilir (state.stages[i] başka + bir aşamayı adresliyor olur). *Value-level* düzenlemeler — aynı + stage adı altında farklı `trainer_type`, `training.output_dir`, + hyperparameter — tasarımı gereği kabul edilir; `--force-resume` + operatörün ayrışmayı anladığını bildiren sinyaldir. +- **`forgelm wizard` entegrasyonu yok.** Tek aşamalı config'ler + wizard'la üretilebilir; pipeline'lar operatör-seviyesi ve manuel + YAML yüzeyini kullanır. Wizard'ın işi "çalışan bir tek aşama + config'i üret, büyüdüğünde elle pipeline'a düzenle" olarak kalır. +- **Notebook entegrasyonu yok.** `notebooks/` altındaki 11 demo + notebook bireysel eğitim paradigmalarını kapsar; uçtan uca pipeline + demosu her notebook'un setup boilerplate'ini üç kez tekrar ederdi. + `tests/fixtures/pipeline/` altındaki fixture suite, reviewer'lara + bir notebook'la aynı yüzeyi verir — üstüne golden manifest'le + byte-comparable olma avantajıyla. + +--- + +## Cross-references + +- Faz 14 tasarım dokümanı: [docs/roadmap/phase-14-pipeline-chains.md](../roadmap/phase-14-pipeline-chains.md) +- Roadmap girişi: [docs/roadmap-tr.md](../roadmap-tr.md) +- Annex IV doğrulayıcısı: `forgelm verify-annex-iv --pipeline ` (CLI help'e bakın) +- Audit log standardı: [docs/standards/logging-observability.md](../standards/logging-observability.md) +- Tek aşamalı eğitici kılavuzu (her şey bu yüzeyden miras alır): + [docs/guides/alignment-tr.md](alignment-tr.md) diff --git a/docs/guides/pipeline.md b/docs/guides/pipeline.md new file mode 100644 index 00000000..3f3253b1 --- /dev/null +++ b/docs/guides/pipeline.md @@ -0,0 +1,313 @@ +# Multi-Stage Training Pipelines + +Phase 14 — landing in `v0.7.0` (CHANGELOG entry dated 2026-05-14; PyPI tag pending). + +ForgeLM's `pipeline:` config block chains 2 or more training stages +(typically SFT → DPO → GRPO) into one config-driven, dry-run-validatable, +Annex-IV-traceable run. Before Phase 14, the same workflow required +3+ separate config files, manual `model.name_or_path` editing between +stages, and an external shell script for orchestration. After Phase 14, +a single `forgelm --config pipeline.yaml` invocation runs the entire +chain end-to-end. + +--- + +## When to use a pipeline (and when not to) + +Use the `pipeline:` block when **all** of the following hold: + +- You are running 2 or more training stages in sequence (e.g. SFT + followed by DPO). +- Each stage's input model is the previous stage's output model. +- You want a single Annex IV manifest covering the entire chain. + +Do **not** use the `pipeline:` block when: + +- You only need to run a single training paradigm. Single-stage configs + (the v0.6.0 default) remain the canonical case and stay byte-identical + to their pre-Phase-14 behaviour. +- You need non-linear stage dependencies (DAG-shaped pipelines). + Phase 14 ships sequential pipelines only; the schema reserves the + surface for a future DAG extension. +- You want parallel stage execution (independent branches running + concurrently). Same horizon as the DAG support — Wave 2 or later. + +--- + +## Anatomy of a pipeline config + +```yaml +# Root config — provides defaults that every stage inherits from +# unless the stage overrides them. +model: + name_or_path: "meta-llama/Llama-3-8B" # Stage 0's starting model +lora: + r: 8 + alpha: 16 +training: + trainer_type: "sft" # Used by stages that + # inherit the training block +data: + dataset_name_or_path: "./placeholder.jsonl" # Required at root; each + # stage overrides per-stage + +# Pipeline-level block — declares the chain. +pipeline: + output_dir: "./pipeline_run" # Hosts pipeline_state.json, + # compliance/pipeline_manifest.json, + # and the pipeline-level + # audit_log.jsonl + stages: + - name: sft_stage # ^[a-z0-9_]{1,32}$ + training: + trainer_type: "sft" # Required per stage + output_dir: "./pipeline_run/stage1_sft" + num_train_epochs: 3 + data: + dataset_name_or_path: "./data/sft.jsonl" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./pipeline_run/stage2_dpo" + num_train_epochs: 1 + dpo_beta: 0.1 + data: + dataset_name_or_path: "./data/preferences.jsonl" + + - name: grpo_stage + training: + trainer_type: "grpo" + output_dir: "./pipeline_run/stage3_grpo" + grpo_num_generations: 4 + data: + dataset_name_or_path: "./data/math_prompts.jsonl" +``` + +**Auto-chain:** each stage's `model.name_or_path` is automatically set to +the previous stage's `training.output_dir/final_model` — you do not +write that path by hand, and you cannot accidentally point stage N's +DPO trainer at the base model instead of stage N-1's SFT output. + +--- + +## Inheritance matrix + +Section-wholesale override semantics — if a stage declares a top-level +block (`model`, `lora`, `training`, `data`, `evaluation`), the **entire** +block replaces the root's; if the stage omits the block, the root's +block is inherited verbatim. No field-level deep-merge: "if you want to +inherit, omit the block; if you want to override, supply the full block." + +| Section | Inheritance | Notes | +|---|---|---| +| `model.name_or_path` | **Auto-chained** (overrides root from stage 1 onward) | Set to previous stage's `training.output_dir/final_model`. Stage 0 reads root `model.name_or_path`. An explicit per-stage `model:` block disables auto-chaining for that stage (escape hatch). | +| `model.*` (other fields) | Inherited unless `model:` block overridden | `backend`, `load_in_4bit`, `trust_remote_code`, `max_length`, `chat_template` follow the root. | +| `lora` | Inherited unless `lora:` block overridden | **Critical edge case:** stage 2 DPO with a *different* `lora.r` than stage 1 SFT means stage 2 is **a fresh LoRA over the merged SFT model**, not a continuation of SFT's LoRA. Operator-visible behaviour. | +| `data` | Inherited unless `data:` block overridden (strongly discouraged) | The schema does **not** force a per-stage override — operators may rerun the same dataset against a different trainer for ablation studies — but pipelines that legitimately reuse the same dataset across stages are vanishingly rare in production. The operator guide flags inheritance as a smell; almost every real chain has SFT use a curated SFT dataset, DPO use a preference-pairs dataset, and GRPO use a math/reward dataset. | +| `training` | Inherited unless `training:` block overridden | `trainer_type` MUST be set explicitly per stage (audit-clarity validator). Other fields wholesale-replaced when the block is overridden. | +| `evaluation` | Inherited unless `evaluation:` block overridden | Per-stage gates (loss thresholds, auto_revert, safety, judge, human-approval) live here. | +| `distributed` | Root only — **per-stage rejected** | Distributed strategy must be consistent across the run. | +| `webhook` | Root only — **per-stage rejected** | Per-stage events carry the stage name in the payload. | +| `compliance` | Root only — **per-stage rejected** | Provider/system/risk metadata is pipeline-level. | +| `risk_assessment`, `monitoring`, `retention`, `synthetic`, `merge`, `auth` | Root only — **per-stage rejected** | Pipeline-level concerns. | + +A stage that declares any of the root-only sections is rejected at +config-load time with `EXIT_CONFIG_ERROR (1)` and the offending section +name in the message. + +--- + +## CLI + +```bash +# End-to-end pipeline run. +forgelm --config pipeline.yaml + +# Dry-run validates every stage (Pydantic + chain integrity) before any +# GPU is allocated. Collects all errors before exiting, like +# `pytest --collectonly`. +forgelm --config pipeline.yaml --dry-run + +# Run a single named stage in isolation (audit / re-run scenarios). +# Non-first stages require the previous stage's on-disk output, or +# an explicit --input-model override. +forgelm --config pipeline.yaml --stage dpo_stage + +# Resume after a failed / interrupted run from a named stage onward. +# Already-completed stages whose output paths exist on disk are +# skipped (logged at INFO). +forgelm --config pipeline.yaml --resume-from dpo_stage + +# Override the auto-chained input model for a single stage (escape hatch). +# The audit log records `input_source: cli_override`. +forgelm --config pipeline.yaml --stage dpo_stage --input-model ./other/checkpoint +``` + +### `--stage ` partial-run rules + +| Scenario | Behaviour | +|---|---| +| `--stage ` and `` is the first stage | Reads root `model.name_or_path`; runs normally. | +| `--stage ` and the previous stage's `output_dir/final_model` exists on disk | Auto-chains as if the previous stage had just finished. | +| `--stage ` and the previous stage's output is missing | Hard-fails with `EXIT_CONFIG_ERROR (1)`: `Stage requires output at ; pass --input-model to override or run the full pipeline first.` No silent fallback to root `model.name_or_path`. | +| `--stage --input-model ` | Operator escape hatch: skips the auto-chain, uses ``. Audit-log records `input_source: cli_override`. | +| `--stage ` where `` is not in the config | Hard-fails at parse time with the list of valid stage names. | + +### `--resume-from ` semantics + +- State file: `/pipeline_state.json` (atomic-write). +- Stages with `status: completed` whose `output_model` path still exists + on disk are **skipped** and their per-stage training manifests are + preserved. The named stage and every stage after it are re-run. +- **Stale-state guard:** if the on-disk state file's + `pipeline_config_hash` differs from the current parse of the YAML + bytes, resume fails with `EXIT_CONFIG_ERROR (1)` — preventing + "I resumed against a config that was edited mid-flight" silent + divergence. Override via `--force-resume` (logged at WARNING, + recorded in the audit event). +- Phase 14 resumes at **stage boundaries** only. Intra-stage HF + `Trainer.train(resume_from_checkpoint=...)` integration is deferred + to a Phase 14.x follow-up — see Limitations below. + +--- + +## Human-approval gate within a pipeline + +If a stage carries `evaluation.require_human_approval: true`, ForgeLM's +existing Phase 9 flow runs unchanged: the model lands in +`final_model.staging./`, the orchestrator captures the staging +path in the pipeline state with `status: gated_pending_approval`, and +the run exits with `EXIT_AWAITING_APPROVAL (4)`. + +Downstream stages stay `pending` (not `skipped_due_to_prior_revert`) +so a subsequent resume picks them up after the approval: + +```bash +# Stage 1 SFT gates pending approval — pipeline exits 4. +$ forgelm --config pipeline.yaml +# ... → exit 4 + +# Operator inspects the staged model, then approves. +$ forgelm approve --output-dir ./pipeline_run/stage1_sft +# ... → final_model/ is promoted, exit 0 + +# Resume the pipeline from DPO; SFT is skipped (status=completed +# on disk). DPO's input_model points at the *promoted* final_model/ +# path, not the staging path. +$ forgelm --config pipeline.yaml --resume-from dpo_stage +# ... → exit 0 when the rest of the chain succeeds +``` + +--- + +## Audit events + +The orchestrator emits these events into the pipeline-level +`audit_log.jsonl` (under `pipeline.output_dir`): + +- `pipeline.started` — run id, config hash, stage count, stage names +- `pipeline.stage_started` — stage name, index, input model, input source +- `pipeline.stage_completed` — stage name, gate decision (`passed` / `failed`), metrics summary +- `pipeline.stage_gated` — stage name, gate decision `approval_pending`, staging path (emitted when a stage exits `EXIT_AWAITING_APPROVAL`) +- `pipeline.stage_reverted` — stage name, auto-revert reason +- `pipeline.force_resume` — operator-approved stale-hash override with `old_config_hash` + `new_config_hash` +- `pipeline.completed` — final status, stopped-at stage (if any) + +These events live alongside the existing `training.*` events that each +stage's `ForgeTrainer` continues to emit; pre-existing Slack / Teams +dashboards filtering on `training.failure` keep working unchanged. + +The matching webhook methods are `notify_pipeline_started`, +`notify_pipeline_completed`, and `notify_pipeline_reverted` — see +`forgelm/webhook.py`. + +--- + +## Annex IV manifest + +Every stage transition rewrites `/compliance/ +pipeline_manifest.json` (atomic-write). The pipeline manifest is the +**index** that ties the per-stage `training_manifest.json` files into +one verifiable chain; per-stage manifests remain individually valid +against the existing single-stage Annex IV schema. + +Validate a complete run with the verifier: + +```bash +forgelm verify-annex-iv --pipeline ./pipeline_run +``` + +The verifier returns exit 0 when: + +1. Every required top-level key is present and well-formed. +2. Stage indices form `0..N-1` in order. +3. Every chain stage's `input_model` equals the previous executed + stage's `output_model` (operator `--input-model` overrides are + recorded with `input_source: cli_override` and exempted from this + check, by design — auditors cross-reference the audit log to + distinguish a legitimate override from a corrupt manifest). +4. Every per-stage `training_manifest` path points at a real file. +5. `stopped_at` (if set) names a real stage whose status is `failed` + or `gated_pending_approval`. + +A non-zero exit code lists the violations in the order they were +discovered. + +--- + +## Limitations (Phase 14 Wave 1) + +- **No intra-stage checkpoint resume.** `--resume-from` picks up at + stage boundaries only. If a stage's `ForgeTrainer.train()` crashes + half-way, the resume re-runs that stage from epoch 0. Wave 2. +- **Sequential only — no DAG semantics.** Stages execute in the order + they appear in `pipeline.stages`. Branches / fan-out / merge are + deferred. +- **No parallel stage execution.** Even when two stages are logically + independent, the orchestrator runs them sequentially. +- **No multi-process locking on `pipeline.output_dir`.** Running two + `forgelm --config pipeline.yaml` invocations against the same + `pipeline.output_dir` simultaneously will race on + `pipeline_state.json` + the manifest — pick distinct + `pipeline.output_dir` per concurrent run. The atomic-write helper + uses per-writer-unique temp filenames so the race surfaces as + last-writer-wins (not a `FileNotFoundError` traceback), but the + loser's per-stage progress is invisible to its own process. +- **Stage-vs-pipeline `output_dir` overlap is allowed but interleaves + the audit log.** When a stage's resolved `training.output_dir` + equals `pipeline.output_dir`, the stage's `training.*` events and + the pipeline's `pipeline.*` events land in the same + `audit_log.jsonl` (one file per directory). The hash-chain + per- + line flock keep the merged file internally consistent and the + `tests/fixtures/pipeline/inheritance_matrix.yaml` fixture + deliberately exercises this layout, but SIEM filters that pin a + single `event_kind` per file should pick distinct directories. +- **`--force-resume` topology guard catches positional breaks only.** + Adding, deleting, or renaming a stage between runs is refused even + under `--force-resume` (state.stages[i] would address a different + stage than pipeline.stages[i]). *Value-level* edits — different + `trainer_type`, `training.output_dir`, hyperparameters under + identical stage names — are accepted by design; `--force-resume` + is the operator's signal that they understand the divergence. +- **No `forgelm wizard` integration.** Single-stage configs are + wizard-buildable; pipelines are operator-grade and use the manual + YAML surface. The wizard's job stays "produce a working single-stage + config you can hand-edit into a pipeline if you outgrow it." +- **No notebook integration.** The 11 demo notebooks under + `notebooks/` cover individual training paradigms; an end-to-end + pipeline demo would duplicate every notebook's setup boilerplate + three times. The fixture suite under `tests/fixtures/pipeline/` + gives reviewers exactly the same surface as a notebook would, with + the advantage of being byte-comparable to a golden manifest. + +--- + +## Cross-references + +- Phase 14 design doc: [docs/roadmap/phase-14-pipeline-chains.md](../roadmap/phase-14-pipeline-chains.md) +- Roadmap entry: [docs/roadmap.md](../roadmap.md) +- Annex IV verifier: `forgelm verify-annex-iv --pipeline ` (see CLI help) +- Audit log standard: [docs/standards/logging-observability.md](../standards/logging-observability.md) +- Single-stage trainer guide (everything inherits from this surface): + [docs/guides/alignment.md](alignment.md) diff --git a/docs/reference/configuration-tr.md b/docs/reference/configuration-tr.md index 4db2dbc4..a47b3a33 100644 --- a/docs/reference/configuration-tr.md +++ b/docs/reference/configuration-tr.md @@ -300,3 +300,72 @@ uzatmasını engeller. | Alan | Tip | Varsayılan | Açıklama | |------|-----|-----------|----------| | `hf_token` | string | `null` | HuggingFace tokeni (tercih: `HUGGINGFACE_TOKEN` ortam değişkeni) | + +--- + +## `pipeline` (İsteğe bağlı — Çok Aşamalı Eğitim Zincirleri, Faz 14) + +2+ eğitim aşamasını (tipik olarak SFT → DPO → GRPO) tek bir config-tabanlı koşuda zincirler: otomatik zincirleme, aşama bazında kapılar, crash-safe resume ve zincir seviyesi Annex IV manifesti. Atlandığında ForgeLM v0.6.0 tek-aşamalı koşusu ile byte-byte aynı davranır; orkestratör modülü import edilmez. Operatör adım adım: [Çok Aşamalı Pipeline kılavuzu](../guides/pipeline-tr.md). + +| Alan | Tip | Varsayılan | Açıklama | +|------|-----|-----------|----------| +| `output_dir` | string | `"./pipeline_run"` | Zincir seviyesi artefakların kök dizini: `pipeline_state.json`, `compliance/pipeline_manifest.json` ve pipeline-kapsamlı `audit_log.jsonl`. Aşama bazında trainer artefaktları her aşamanın kendi `training.output_dir`'ı altında kalır. | +| `stages` | `List[PipelineStage]` | `[]` (en az 1 zorunlu) | Sıralı aşama listesi. Her aşamanın `model.name_or_path`'ı, aşama explicit `model:` bloğu vermediği sürece, önceki aşamanın `training.output_dir/final_model`'ına otomatik ayarlanır. | + +### `pipeline.stages[].*` — PipelineStage alanları + +`PipelineStage`, root config üzerine bindirilen aşama bazında bir override'dır. Bölüm-toptan miras: bir blok atlanırsa root'un bloğu birebir miras alınır; blok verilirse root'unkini TAMAMEN değiştirir (deep-merge yok). + +| Alan | Tip | Varsayılan | Açıklama | +|------|-----|-----------|----------| +| `name` | string | — (zorunlu) | `^[a-z0-9_]{1,32}$` deseniyle eşleşen aşama tanımlayıcısı. Pipeline içinde benzersiz. `--stage `, `--resume-from `, audit-log payload'larında ve aşama bazında manifest girdilerinde kullanılır. | +| `model` | `Optional[ModelConfig]` | `null` | Root `model:` bloğunun aşama bazında override'ı. `null` iken önceki aşamanın `final_model`'ından otomatik zincirlenir (aşama 0 için root). Set edildiğinde o aşama için otomatik zincirleme devre dışı (operatör kaçış kapısı). | +| `lora` | `Optional[LoraConfig]` | `null` | Aşama bazında LoRA config. `null` ise root'tan toptan miras alınır. | +| `training` | `Optional[TrainingConfig]` | `null` | Aşama bazında training config. `null` ise root'tan toptan miras alınır. **Verildiğinde `trainer_type` AÇIKÇA SET EDİLMEK ZORUNDA** — her aşama hangi hizalama paradigmasını koştuğunu manifestte audit-clarity için kaydeder. | +| `data` | `Optional[DataConfig]` | `null` | Aşama bazında data config. `null` ise root'tan toptan miras alınır; aşama bazında override norm — her aşama tipik olarak farklı bir dataset tüketir (SFT/DPO/preference/vb.). | +| `evaluation` | `Optional[EvaluationConfig]` | `null` | Aşama bazında kapılar (loss eşikleri, `auto_revert`, safety, judge, human-approval). Her aşama kendi kapısını bağımsız konfigüre edebilir. | + +Sadece-root bölümleri — **aşama seviyesinde reddedilir**, `EXIT_CONFIG_ERROR (1)`: `distributed`, `webhook`, `compliance`, `risk_assessment`, `monitoring`, `retention`, `synthetic`, `merge`, `auth`. Bunlar pipeline seviyesi konulardır (distributed stratejisi koşu boyunca tutarlı kalır; compliance metadata tüm zinciri kapsar; vb.). + +### Örnek + +```yaml +# Root varsayılanları — blok atlayan aşamalarca miras alınır. +model: { name_or_path: "meta-llama/Llama-3-8B" } +lora: { r: 8, alpha: 16 } +training: { trainer_type: "sft", output_dir: "./placeholder" } +data: { dataset_name_or_path: "./placeholder.jsonl" } + +pipeline: + output_dir: "./pipeline_run" + stages: + - name: sft_stage + training: { trainer_type: "sft", output_dir: "./pipeline_run/stage1_sft" } + data: { dataset_name_or_path: "./data/sft.jsonl" } + - name: dpo_stage + training: { trainer_type: "dpo", output_dir: "./pipeline_run/stage2_dpo", dpo_beta: 0.1 } + data: { dataset_name_or_path: "./data/preferences.jsonl" } + - name: grpo_stage + training: { trainer_type: "grpo", output_dir: "./pipeline_run/stage3_grpo" } + data: { dataset_name_or_path: "./data/math_prompts.jsonl" } +``` + +### CLI yüzeyi + +| Flag | Etki | +|------|------| +| `--stage ` | Sadece adı verilen aşamayı yalıtılmış olarak koşar (audit / re-run senaryoları). Önceki aşamanın disk üzerindeki çıktısından otomatik zincirler. | +| `--resume-from ` | Adı verilen aşamadan itibaren devam eder; tamamlanmış (veya operatör tarafından onaylanmış gated) aşamalar disk üzerinde çıktıları varsa atlanır. | +| `--force-resume` | Resume sırasındaki `pipeline_config_hash` uyuşmazlığını kabul eder (log'lanır + `pipeline.force_resume` ile audit'lenir). Aşama topoloji uyuşmazlığı (sayı / isim / sıra) bu flag'le bile reddedilir. | +| `--input-model ` | Operatör kaçış kapısı — `--stage` hedefi için otomatik zincirlenen modeli override eder. Audit log `input_source: cli_override` ile kaydedilir. | +| `--dry-run` | Her aşamanın merge edilmiş config'ini + cross-stage zincir bütünlüğünü + `training.output_dir` çakışma kontrolünü herhangi bir GPU tahsisi olmadan doğrular; tüm hataları çıkmadan önce toplar. | + +`--fit-check`, `--merge`, `--generate-data`, `--compliance-export`, `--benchmark-only` flag'leri tek-aşama operasyonlarıdır ve `pipeline:` bloğu mevcut olduğunda dispatch zamanında reddedilir — ya `pipeline:` bloğunu kaldırın ya da flag'i kaldırın. + +### Doğrulayıcı + +```bash +forgelm verify-annex-iv --pipeline +``` + +Zincir seviyesi manifestin yapısal alanlarını, zincir bütünlüğünü (her `input_source: chain` aşaması kendi önceki aşamasının `output_model`'ına eşleşir), aşama bazında `training_manifest.json` varlığını ve `stopped_at` / running-status tutarlılığını doğrular. Temiz manifest için `0`, config / zincir ihlali için `1`, runtime I/O hatası için `2` ile çıkar. diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 9891c87b..19a1bb07 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -348,3 +348,72 @@ silently extend the retention horizon by re-using a stale workspace. | Field | Type | Default | Description | |-------|------|---------|-------------| | `hf_token` | string | `null` | HuggingFace token (prefer `HUGGINGFACE_TOKEN` env var) | + +--- + +## `pipeline` (Optional — Multi-Stage Training Chains, Phase 14) + +Chains 2+ training stages (typically SFT → DPO → GRPO) into one config-driven run with auto-chaining, per-stage gates, crash-safe resume, and a chain-level Annex IV manifest. When omitted, ForgeLM behaves byte-identically to a v0.6.0 single-stage run; the orchestrator module is not imported. Full operator walkthrough: [Multi-Stage Pipelines guide](../guides/pipeline.md). + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `output_dir` | string | `"./pipeline_run"` | Root directory for chain-level artefacts: `pipeline_state.json`, `compliance/pipeline_manifest.json`, and the pipeline-scoped `audit_log.jsonl`. Per-stage trainer artefacts continue to live under each stage's own `training.output_dir`. | +| `stages` | `List[PipelineStage]` | `[]` (required: ≥ 1) | Ordered list of stages. Each stage's `model.name_or_path` is auto-set to the previous stage's `training.output_dir/final_model` unless the stage supplies an explicit `model:` block. | + +### `pipeline.stages[].*` — PipelineStage fields + +A `PipelineStage` is a per-stage override layered onto the root config. Section-wholesale inheritance: omitting a block inherits root's wholesale; supplying a block REPLACES root's wholesale (no deep-merge). + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `name` | string | — (required) | Stage identifier matching `^[a-z0-9_]{1,32}$`. Unique within the pipeline. Used as the identifier in `--stage `, `--resume-from `, audit-log payloads, and per-stage manifest entries. | +| `model` | `Optional[ModelConfig]` | `null` | Per-stage override of the root `model:` block. When `null`, auto-chains from the previous stage's `final_model` (or root for stage 0). When set, disables the auto-chain for that stage (operator escape hatch). | +| `lora` | `Optional[LoraConfig]` | `null` | Per-stage LoRA config. Inherits root wholesale when `null`. | +| `training` | `Optional[TrainingConfig]` | `null` | Per-stage training config. Inherits root wholesale when `null`. **When supplied, `trainer_type` MUST be set explicitly** — every stage records its alignment paradigm in the manifest for audit clarity. | +| `data` | `Optional[DataConfig]` | `null` | Per-stage data config. Inherits root wholesale when `null`; per-stage override is the norm because each stage typically consumes a different dataset (SFT/DPO/preference/etc.). | +| `evaluation` | `Optional[EvaluationConfig]` | `null` | Per-stage gates (loss thresholds, `auto_revert`, safety, judge, human-approval). Each stage may independently configure its gate. | + +Root-only sections — **rejected at the stage level** with `EXIT_CONFIG_ERROR (1)`: `distributed`, `webhook`, `compliance`, `risk_assessment`, `monitoring`, `retention`, `synthetic`, `merge`, `auth`. These are pipeline-level concerns (distributed strategy stays consistent across the run; compliance metadata covers the whole chain; etc.). + +### Example + +```yaml +# Root defaults — inherited by stages that omit a block. +model: { name_or_path: "meta-llama/Llama-3-8B" } +lora: { r: 8, alpha: 16 } +training: { trainer_type: "sft", output_dir: "./placeholder" } +data: { dataset_name_or_path: "./placeholder.jsonl" } + +pipeline: + output_dir: "./pipeline_run" + stages: + - name: sft_stage + training: { trainer_type: "sft", output_dir: "./pipeline_run/stage1_sft" } + data: { dataset_name_or_path: "./data/sft.jsonl" } + - name: dpo_stage + training: { trainer_type: "dpo", output_dir: "./pipeline_run/stage2_dpo", dpo_beta: 0.1 } + data: { dataset_name_or_path: "./data/preferences.jsonl" } + - name: grpo_stage + training: { trainer_type: "grpo", output_dir: "./pipeline_run/stage3_grpo" } + data: { dataset_name_or_path: "./data/math_prompts.jsonl" } +``` + +### CLI surface + +| Flag | Effect | +|------|--------| +| `--stage ` | Run only the named stage in isolation (audit / re-run scenarios). Auto-chains from the previous stage's on-disk output. | +| `--resume-from ` | Resume from the named stage onward; already-completed (or human-approved gated) stages with on-disk output are skipped. | +| `--force-resume` | Accept a `pipeline_config_hash` mismatch on resume (logged + audited via `pipeline.force_resume`). Stage topology mismatch (count / names / order) is refused even with this flag. | +| `--input-model ` | Operator escape hatch — overrides the auto-chained model for the `--stage` target. Audit-logged with `input_source: cli_override`. | +| `--dry-run` | Validates every stage's merged config + cross-stage chain integrity + `training.output_dir` collision check before any GPU is allocated; collects all errors before exiting. | + +The `--fit-check`, `--merge`, `--generate-data`, `--compliance-export`, `--benchmark-only` flags are single-stage operations and are rejected at dispatch time when a `pipeline:` block is present — drop the `pipeline:` block or remove the flag. + +### Verifier + +```bash +forgelm verify-annex-iv --pipeline +``` + +Validates the chain-level manifest's structural fields, chain-integrity (every stage with `input_source: chain` matches its immediate predecessor's `output_model`), per-stage `training_manifest.json` existence, and `stopped_at` / running-status consistency. Exit `0` on clean manifest, `1` on config / chain violation, `2` on runtime I/O failure. diff --git a/docs/reference/data_ingestion_architecture-tr.md b/docs/reference/data_ingestion_architecture-tr.md index 29d71f04..0664b612 100644 --- a/docs/reference/data_ingestion_architecture-tr.md +++ b/docs/reference/data_ingestion_architecture-tr.md @@ -209,7 +209,7 @@ graph TD PAGES[sayfa başına metin] --> SPLIT[Satırlara böl] SPLIT --> WINDOW[Sayfa başına üst-3 / alt-3 satırı incele
_PDF_EDGE_WINDOW = 3] WINDOW --> COUNT[Tüm sayfalarda satır
frekansını say] - COUNT --> CHECK{>= yüzde 70 sayfa?} + COUNT --> CHECK{">= yüzde 70 sayfa?"} CHECK -- evet --> POP[Eşleşen window satırlarını çıkar
window içinde herhangi bir offset] POP --> WINDOW CHECK -- hayır --> JOIN[Sayfaları çift yeni satırla birleştir] diff --git a/docs/reference/data_ingestion_architecture.md b/docs/reference/data_ingestion_architecture.md index fbc94c20..44dbdb6d 100644 --- a/docs/reference/data_ingestion_architecture.md +++ b/docs/reference/data_ingestion_architecture.md @@ -207,7 +207,7 @@ graph TD PAGES[per-page text] --> SPLIT[Split into lines] SPLIT --> WINDOW[Inspect top-3 / bottom-3 rows
per page _PDF_EDGE_WINDOW = 3] WINDOW --> COUNT[Count line frequency
across pages] - COUNT --> CHECK{>= 70 percent of pages?} + COUNT --> CHECK{">= 70 percent of pages?"} CHECK -- yes --> POP[Pop matching window lines
any offset within the window] POP --> WINDOW CHECK -- no --> JOIN[Re-join pages with double newline] diff --git a/docs/reference/usage-tr.md b/docs/reference/usage-tr.md index a46794ba..0333b1b1 100644 --- a/docs/reference/usage-tr.md +++ b/docs/reference/usage-tr.md @@ -61,6 +61,46 @@ forgelm --config my_config.yaml --resume ./checkpoints/checkpoint-500 forgelm --config my_config.yaml --offline # İzole mod (HF Hub yok) ``` +### Çok Aşamalı Pipeline'lar (Faz 14, v0.7.0) + +Config bir `pipeline:` bloğu taşıdığında aynı `forgelm --config ...` çağrısı tüm zinciri uçtan uca koşar (tipik olarak SFT → DPO → GRPO): otomatik zincirlenen model yolları ve tüm aşamaları kapsayan tek bir Annex IV manifesti ile. + +```bash +# Tüm zinciri uçtan uca koş +forgelm --config pipeline.yaml + +# Dry-run her aşamanın merge edilmiş config'ini + aşamalar arası zincir +# bütünlüğünü + aşama bazında training.output_dir çakışma kontrolünü GPU +# tahsisi yapmadan doğrular; tüm hataları çıkmadan önce toplar +forgelm --config pipeline.yaml --dry-run + +# Tek bir adlı aşamayı yalıtılmış olarak yeniden koştur (audit / re-run +# senaryoları). Önceki aşamanın disk üzerindeki output_dir/final_model'ından +# otomatik zincirler. +forgelm --config pipeline.yaml --stage dpo_stage + +# Filtrelenen aşamanın otomatik zincirlenen giriş modelini override et +# (operatör kaçış kapısı — input_source: cli_override ile audit'lenir) +forgelm --config pipeline.yaml --stage dpo_stage --input-model ./other/checkpoint + +# Adı verilen aşamadan itibaren devam et; tamamlanmış (veya operatör +# tarafından onaylanmış gated) aşamalar disk üzerinde çıktıları varsa +# INFO log seviyesinde atlanır +forgelm --config pipeline.yaml --resume-from dpo_stage + +# Resume sırasındaki stale pipeline_config_hash'i kabul et (log'lanır + +# pipeline.force_resume eventi ile audit'lenir). Aşama topoloji +# uyuşmazlığı (sayı / isim / sıra) bu flag'le bile reddedilir. +forgelm --config pipeline.yaml --resume-from dpo_stage --force-resume + +# Biten bir pipeline'ın zincir seviyesi Annex IV manifestini doğrula +forgelm verify-annex-iv --pipeline ./pipeline_run +``` + +**Tek-aşama flag reddi:** Config bir `pipeline:` bloğu taşıdığında `--fit-check`, `--merge`, `--generate-data`, `--compliance-export`, `--benchmark-only` desteklenmez — ya bloğu kaldırın ya da flag'i kaldırın. Tersine, `--stage`, `--resume-from`, `--force-resume`, `--input-model` `pipeline:` bloğu gerektirir — tek-aşama config'e karşı çalıştırıldıklarında flag'i sessizce yok saymak yerine `EXIT_CONFIG_ERROR (1)` ile çıkar. + +Operatör adım adım: [Çok Aşamalı Pipeline kılavuzu](../guides/pipeline-tr.md). Şema detayları: [`pipeline` config bloğu](configuration-tr.md#pipeline-isteğe-bağlı-çok-aşamalı-eğitim-zincirleri-faz-14). + ### VRAM Fit Check Eğitim öncesi config'inizin GPU belleğine sığıp sığmayacağını tahmin edin: diff --git a/docs/reference/usage.md b/docs/reference/usage.md index 5caa1944..18e17c2d 100644 --- a/docs/reference/usage.md +++ b/docs/reference/usage.md @@ -82,6 +82,44 @@ forgelm --config my_config.yaml --resume ./checkpoints/checkpoint-500 forgelm --config my_config.yaml --offline ``` +### Multi-Stage Pipelines (Phase 14, v0.7.0) + +When the config carries a `pipeline:` block, the same `forgelm --config ...` invocation runs the full chain (typically SFT → DPO → GRPO) with auto-chained model paths and a single Annex IV manifest covering every stage. + +```bash +# Run the entire chain end-to-end +forgelm --config pipeline.yaml + +# Dry-run validates every stage's merged config + the cross-stage chain +# integrity + per-stage training.output_dir collision check, with no GPU +# allocation; collects all errors before exiting +forgelm --config pipeline.yaml --dry-run + +# Re-run a single named stage in isolation (audit / re-run scenarios). +# Auto-chains from the previous stage's on-disk output_dir/final_model. +forgelm --config pipeline.yaml --stage dpo_stage + +# Override the auto-chained input model for the filtered stage (operator +# escape hatch — audit-logged with input_source: cli_override) +forgelm --config pipeline.yaml --stage dpo_stage --input-model ./other/checkpoint + +# Resume from a named stage onward; already-completed (or human-approved +# gated) stages with on-disk output are skipped at INFO log level +forgelm --config pipeline.yaml --resume-from dpo_stage + +# Accept a stale pipeline_config_hash on resume (logged + audited via the +# pipeline.force_resume event). Stage topology mismatch (count / names / +# order) is refused even with this flag. +forgelm --config pipeline.yaml --resume-from dpo_stage --force-resume + +# Verify a finished pipeline's chain-level Annex IV manifest +forgelm verify-annex-iv --pipeline ./pipeline_run +``` + +**Single-stage flag rejection:** `--fit-check`, `--merge`, `--generate-data`, `--compliance-export`, `--benchmark-only` are not supported when the config carries a `pipeline:` block — drop the block or remove the flag. Conversely, `--stage`, `--resume-from`, `--force-resume`, `--input-model` require a `pipeline:` block — running them against a single-stage config exits with `EXIT_CONFIG_ERROR (1)` rather than silently ignoring the flag. + +Full operator walkthrough: [Multi-Stage Pipelines guide](../guides/pipeline.md). Schema details: [`pipeline` config block](configuration.md#pipeline-optional-multi-stage-training-chains-phase-14). + ### VRAM Fit Check Before training, estimate whether your config fits in GPU memory: diff --git a/docs/roadmap-tr.md b/docs/roadmap-tr.md index d99bfb1e..038d432f 100644 --- a/docs/roadmap-tr.md +++ b/docs/roadmap-tr.md @@ -13,8 +13,8 @@ | ✅ Tamam | [Faz 12.6 — Kapanış Döngüsü (5 wave boyunca 37 içerik fazı + 1 sürüm etiketi = 38 kayıt)](roadmap/completed-phases.md) | Library API, GDPR purge + reverse-pii, ISO 27001 / SOC 2 alignment, doctor + cache subcommand'leri, compliance verification toolbelt, bilingual mirror sweep + 4 CI guard, supply-chain security, cross-OS release matrix — `v0.5.5` ile paketlendi (PyPI 2026-05-10) | | ✅ Tamam | Faz 22 — CLI sihirbazı tarayıcı yüzeyiyle eşdeğerlik | `forgelm --wizard` artık tarayıcı sihirbazıyla aynı 9-adımlı akışı çalıştırıyor (welcome → use-case → model → strategy → trainer → dataset → training-params → compliance → evaluation), `--wizard-start-from ` ile idempotent yeniden çalıştırma, şema-güdümlü varsayılanlar SOT, ayrı `EXIT_WIZARD_CANCELLED = 5` exit kodu, `$XDG_CACHE_HOME` altında durum kalıcılığı ve çıkışta validate — `v0.5.5` ile paketlendi (PyPI 2026-05-10) | | ✅ Tamam | Site dokümantasyon düzeltme taraması | `site/*.html` üzerindeki tüm görünür YAML / artefakt-yolu / CLI / şema iddiaları artık live `forgelm/` yüzeyine karşı doğrulanıyor. Hero YAML demo'su gerçek Pydantic alan adlarıyla yeniden yazıldı, compliance artefakt ağacı disk düzenine göre yeniden çizildi, hayalet YAML anahtarları + CLI flag'leri kaldırıldı, ifadeler live davranışa hizalandı. Altı dilde i18n (en / tr / de / fr / es / zh) tam paritede (her biri 731 anahtar) — `v0.5.5` ile paketlendi (PyPI 2026-05-10) | -| 📋 Planlandı | [Faz 14 — Çok Aşamalı Pipeline Zincirleri](roadmap/phase-14-pipeline-chains.md) | SFT → DPO → GRPO config zinciri, pipeline kaynak izleri → `v0.6.0` | -| 📋 Planlandı | [Faz 15 — Yutma Pipeline'ı Güvenilirliği](roadmap/phase-15-ingestion-reliability.md) | PDF / DOCX / EPUB / TXT / Markdown ve playground notebook UX hizalaması üzerinde `forgelm ingest`'in sessiz-başarısızlık boşluğunu kapat; yutma sırasında kalite sinyali yüzeye çıkar + audit'te `--quality-filter` varsayılan açık; çok satırlı PDF dedup'ı, Türkçe glyph normalizasyonu, DOCX header / footer çıkarımı, EPUB spine + skip, TXT / MD BOM + frontmatter; çift-golden regression fixture'ları → `v0.6.0` | +| 🟡 Merge edildi | [Faz 14 — Çok Aşamalı Pipeline Zincirleri](roadmap/phase-14-pipeline-chains.md) | SFT → DPO → GRPO config zinciri, pipeline kaynak izleri, 7 yeni pipeline-kapsamlı audit olayı, `forgelm verify-annex-iv --pipeline` modu — `development` üzerinde tamamlandı; `v0.7.0` PyPI tag'i release-bekleyen adım (v0.6.0'dan yeniden planlandı; 2026-05-11 yutma pilotunun ardından Faz 15 önceliği aldı) | +| ✅ Tamam | [Faz 15 — Yutma Pipeline'ı Güvenilirliği](roadmap/completed-phases.md#phase-15-ingestion-pipeline-reliability-v060) | Wave 1 + Wave 2 + 5 review-absorption turu: window tabanlı çok satırlı PDF dedup'ı, Türkçe glyph normalizasyon profili (language-hint'e bağlı default), dil-farkında Unicode block sağlamlık kontrolü, ingest-time kalite ön-sinyali, default-on audit `--quality-filter`, DOCX explicit header/footer çıkarımı, EPUB spine + whole-token nav/cover skip, TXT UTF-8 BOM + MD YAML frontmatter strip, notebook playground hizalama, ek olarak Wave 2 `--strip-pattern` (ReDoS-korumalı), `--page-range`, front-matter heuristic, `--strip-urls`, multi-column uyarı — `v0.6.0` ile yayınlandı (PyPI 2026-05-11) | | 📋 Planlandı | [Faz 13 — Pro CLI ve Gözlemlenebilirlik Dashboard](roadmap/phase-13-pro-cli.md) | Lisans korumalı dashboard, HPO, zamanlanmış görevler, takım config store → `v0.6.0-pro` (adoption + v0.5.5'te yayınlanan ISO/SOC 2 baseline'a bağlı) | > **Durum lejantı:** ✅ Yayınlandı (PyPI) · 🟡 main'e indi, publish bekliyor · ⏳ Planlandı @@ -28,17 +28,17 @@ Başlangıçta dört ardışık PyPI tag'i (`v0.5.0` / `v0.5.1` / `v0.5.2` / `v0.5.3`) olarak planlandı, dört faz tek tutarlı yüzey (yut → cila → olgunlaş → cila) oluşturduğu için tek kapsamlı `v0.5.0` release'inde birleştirildi. -**PyPI'deki son sürüm:** `v0.5.7` — "SFT trainer trl-modernizasyon düzeltmesi" (2026-05-10), `v0.5.6` (Intel Mac kurulum düzeltmesi) ve `v0.5.5` "Kapanış Döngüsü Paketi + Faz 22 Sihirbaz + Site Dokümantasyon Taraması" (2026-05-10) üzerine gelen bir takip patch'i. v0.5.5 şunları paketler: **Faz 12.6** (Library API + GDPR purge / reverse-pii + ISO 27001 / SOC 2 alignment + doctor / cache / safety-eval / verify-* subcommand'leri + cross-OS release matrix + supply-chain security baseline; [#14 webhook SSRF hardening](https://github.com/cemililik/ForgeLM/issues/14) follow-up'ını kapsar), **Faz 22** (CLI sihirbazı tarayıcı yüzeyiyle eşdeğerlik + `--wizard-start-from` ile idempotent yeniden çalıştırma + şema-güdümlü varsayılanlar SOT + ayrı `EXIT_WIZARD_CANCELLED = 5` exit kodu) ve `site/*.html` üzerindeki tüm görünür YAML / artefakt-yolu / CLI / şema iddialarını live kod yüzeyine hizalayan ve altı-dil i18n paritesini eski haline getiren bir **site dokümantasyon düzeltme taraması**. v0.5.6, v0.5.5'teki `torch>=2.3` tabanını `torch>=2.2`'ye düşürerek Intel Mac (x86_64) kurulabilirliğini geri kazandırır. v0.5.7, trl 0.13'ün `SFTConfig.max_seq_length` → `SFTConfig.max_length` yeniden adlandırmasından kaynaklanan SFT trainer runtime `TypeError` hatasını düzeltir. +**PyPI'deki son sürüm:** `v0.6.0` — "Faz 15 Yutma Pipeline'ı Güvenilirliği" (2026-05-11). 2026-05-11 pilotunun PDF / DOCX / EPUB / TXT / Markdown yutma ile playground notebook'unda açığa çıkardığı sessiz-başarısızlık boşluklarını kapatır — Wave 1 + Wave 2 + 5 review-absorption turu tek sürümde. Tüm ingest-time koruma listesi için [completed-phases.md](roadmap/completed-phases.md#phase-15-ingestion-pipeline-reliability-v060)'e bakın (window tabanlı PDF dedup'ı, Türkçe glyph normalizasyon profili, dil-farkında Unicode block sağlamlık kontrolü, ingest-time kalite ön-sinyali, default-on audit `--quality-filter`, DOCX explicit header/footer çıkarımı, EPUB spine + whole-token nav/cover skip, TXT UTF-8 BOM + MD YAML frontmatter strip, notebook playground hizalama, Wave 2 `--strip-pattern` (ReDoS-korumalı) + `--page-range` + front-matter heuristic + `--strip-urls` + multi-column uyarı). **Daha öncesi:** `v0.5.0` — Doküman Yutma + Veri Curation Pipeline'ı (2026-04-30); `v0.4.5` — Quickstart Katmanı (2026-04-26); `v0.4.0` — Post-Training Tamamlama (2026-04-26). -**Güncel durum:** PyPI'da `v0.5.5` altında 19 faz (1, 2, 2.5, 3, 4, 5, 5.5, 6, 7, 8, 9, 10, 10.5, 11, 11.5, 12, 12.5, 12.6, 22) yayınlandı. Sürüm sonrası 3 faz (13, 14, 15) planlandı; Faz 14 ve Faz 15 `v0.6.0` içinde paralel olarak yayınlanır, Faz 13 adoption gate'leri karşılandığında ayrıca `v0.6.0-pro` olarak yayınlanır. +**Güncel durum:** PyPI'da `v0.6.0` üzerinden 20 faz (1, 2, 2.5, 3, 4, 5, 5.5, 6, 7, 8, 9, 10, 10.5, 11, 11.5, 12, 12.5, 12.6, 15, 22) yayınlandı. Faz 14 (Çok Aşamalı Pipeline Zincirleri) `development` üzerinde merge edildi; `v0.7.0` PyPI tag'i bekleniyor. Faz 13 adoption gate'leri karşılandığında ayrıca `v0.6.0-pro` olarak yayınlanır. > **Faz 12.6 görev / alt-görev iki eksenli not:** Faz 12.6 kendi içinde 38 görevlik bir kapanış döngüsüdür (Görev 1-38) ve [`roadmap/completed-phases.md`](roadmap/completed-phases.md) dosyasında izlenir; her wave'in PR açıklaması o wave'in kapsadığı görev delta'sını taşır. ## Planlanan işlerin özeti -> **Not:** Oklar yayınlama sırasını gösterir, faz numaralarını değil (Faz 14 ve Faz 15 v0.6.0 içinde paralel olarak yayınlanır; Faz 13 Pro katmanında ayrıca daha sonra yayınlanır). +> **Not:** Oklar yayınlama sırasını gösterir, faz numaralarını değil (Faz 15 v0.6.0 ile yayınlandı; Faz 14 v0.7.0 için hedeflendi (Faz 15 önceliği aldıktan sonra); Faz 13 Pro katmanında ayrıca daha sonra yayınlanır). ```mermaid graph LR @@ -49,8 +49,8 @@ graph LR P115 --> P12[Faz 12
Veri Curation
Olgunlaşması] P12 --> P125[Faz 12.5
Veri Curation
Follow-up] P125 --> P126[Faz 12.6
Kapanış
Döngüsü] - P126 --> P14[Faz 14
Pipeline
Zincirleri] P126 --> P15[Faz 15
Yutma
Güvenilirliği] + P15 --> P14[Faz 14
Pipeline
Zincirleri] P14 --> P13[Faz 13
Pro CLI
+ Dashboard] P15 --> P13 @@ -61,8 +61,8 @@ graph LR P12 -.-> V2 P125 -.-> V2 P126 -.-> V25[v0.5.5] - P14 -.-> V23[v0.6.0] - P15 -.-> V23 + P15 -.-> V23[v0.6.0 ✅ Yayınlandı] + P14 -.-> V27[v0.7.0] P13 -.-> V3[v0.6.0-pro] style P10 fill:#003300,stroke:#00ff88 @@ -71,8 +71,8 @@ graph LR style P115 fill:#004400,stroke:#88ff88 style P12 fill:#004400,stroke:#88ff88 style P125 fill:#004400,stroke:#88ff88 + style P15 fill:#004400,stroke:#88ff88 style P14 fill:#002244,stroke:#00aaff - style P15 fill:#002244,stroke:#00aaff style P13 fill:#442200,stroke:#ffaa00 ``` @@ -92,7 +92,7 @@ docs/ └── roadmap/ ├── completed-phases.md # Faz 1-12.6 arşivi (detaylı, İngilizce) — Faz 10 / 10.5 / 11 / 11.5 / 12 / 12.5 / 12.6 inline gömüldü (sırasıyla v0.4.0 / v0.4.5 / v0.5.0 / v0.5.5) ├── phase-13-pro-cli.md # Planlandı — v0.6.0-pro (gated) - ├── phase-14-pipeline-chains.md # Planlandı — v0.6.0 (v0.5.5 kapanış döngüsünün takip release'i) + ├── phase-14-pipeline-chains.md # development'ta merge edildi — v0.7.0 tag bekleniyor (v0.6.0'dan yeniden planlandı; Faz 15 önceliği aldı) ├── releases.md # v0.3.0 → v0.6.0 sürüm notları └── risks-and-decisions.md # Risk matrisi, fırsatlar, rekabet analizi, karar günlüğü ``` diff --git a/docs/roadmap.md b/docs/roadmap.md index 65f2254c..60d714b3 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -13,7 +13,7 @@ | ✅ Done | [Phase 12.6 — Closure Cycle (37 content phases + 1 release tag = 38 entries across 5 waves)](roadmap/completed-phases.md) | Library API, GDPR purge + reverse-pii, ISO 27001 / SOC 2 alignment, doctor + cache subcommands, compliance verification toolbelt, bilingual mirror sweep + 4 CI guards, supply-chain security, cross-OS release matrix — bundled into `v0.5.5` (PyPI 2026-05-10) | | ✅ Done | Phase 22 — CLI wizard parity with the in-browser surface | `forgelm --wizard` runs the same 9-step flow as the web wizard (welcome → use-case → model → strategy → trainer → dataset → training-params → compliance → evaluation), with idempotent re-run via `--wizard-start-from `, schema-driven defaults SOT, distinct `EXIT_WIZARD_CANCELLED = 5` exit code, state persistence under `$XDG_CACHE_HOME`, and validate-on-exit — bundled into `v0.5.5` (PyPI 2026-05-10) | | ✅ Done | Site documentation correction sweep | All visible YAML / artefact-path / CLI / schema claims on `site/*.html` now validate against the live `forgelm/` surface. Hero YAML demo rewritten with real Pydantic field names, compliance artefact tree redrawn against the on-disk layout, ghost YAML keys + CLI flags removed, wording aligned with live behaviour. Six-language i18n (en / tr / de / fr / es / zh) now at full parity (731 keys each) — bundled into `v0.5.5` (PyPI 2026-05-10) | -| 📋 Planned | [Phase 14 — Multi-Stage Pipeline Chains](roadmap/phase-14-pipeline-chains.md) | SFT → DPO → GRPO chained config, pipeline provenance artifacts → `v0.7.0` (re-scheduled from v0.6.0; Phase 15 displaced it after the 2026-05-11 pilot) | +| 🟡 Merged | [Phase 14 — Multi-Stage Pipeline Chains](roadmap/phase-14-pipeline-chains.md) | SFT → DPO → GRPO chained config, pipeline provenance artifacts, 7 new pipeline-scoped audit events, `forgelm verify-annex-iv --pipeline` mode — implementation complete on `development`; `v0.7.0` PyPI tag is the release-pending step (re-scheduled from v0.6.0; Phase 15 displaced it after the 2026-05-11 pilot) | | ✅ Done | [Phase 15 — Ingestion Pipeline Reliability](roadmap/completed-phases.md#phase-15-ingestion-pipeline-reliability-v060) | Wave 1 + Wave 2 + 5 review-absorption rounds: window-based multi-line PDF dedup, Turkish glyph normalisation profile (language-hint-coupled default), language-aware Unicode-block sanity check, ingest-time quality pre-signal, default-on audit `--quality-filter`, DOCX explicit header/footer subtraction, EPUB spine + whole-token nav/cover skip, TXT UTF-8 BOM + MD YAML frontmatter strip, notebook playground alignment, plus Wave 2 `--strip-pattern` (ReDoS-guarded), `--page-range`, front-matter heuristic, `--strip-urls`, multi-column warning — shipped `v0.6.0` (PyPI 2026-05-11) | | 📋 Planned | [Phase 13 — Pro CLI & Observability Dashboard](roadmap/phase-13-pro-cli.md) | License-gated dashboard, HPO, scheduled jobs, team config store → `v0.6.0-pro` (gated on adoption + ISO/SOC 2 baseline shipped in v0.5.5) | @@ -30,17 +30,17 @@ Originally planned as four sequential PyPI tags (`v0.5.0` / `v0.5.1` / `v0.5.2` / `v0.5.3`), consolidated into one comprehensive `v0.5.0` release because the four phases form one coherent surface (ingest → polish → mature → polish) hard to use in parts. -**Latest release on PyPI:** `v0.5.7` — "SFT trainer trl-modernisation fix" (2026-05-10), a follow-up patch on top of `v0.5.6` (Intel Mac install fix) and the headline `v0.5.5` "Closure Cycle Bundle + Phase 22 Wizard + Site Documentation Sweep" (2026-05-10). v0.5.5 bundled **Phase 12.6** (Library API + GDPR purge / reverse-pii + ISO 27001 / SOC 2 alignment + doctor / cache / safety-eval / verify-* subcommands + cross-OS release matrix + supply-chain security baseline; folds in [#14 webhook SSRF hardening](https://github.com/cemililik/ForgeLM/issues/14)), **Phase 22** (CLI wizard parity with the in-browser surface + idempotent `--wizard-start-from` re-run + schema-driven defaults SOT + distinct `EXIT_WIZARD_CANCELLED = 5` exit code), and a **site documentation correction sweep** that brings every visible YAML / artefact-path / CLI / schema claim on `site/*.html` in line with the live code surface and restores six-language i18n parity. v0.5.6 reverted the v0.5.5 `torch>=2.3` floor back to `torch>=2.2` to restore Intel Mac (x86_64) installability. v0.5.7 fixes a runtime `TypeError` in the SFT trainer caused by trl 0.13 renaming `SFTConfig.max_seq_length` → `SFTConfig.max_length`. +**Latest release on PyPI:** `v0.6.0` — "Phase 15 Ingestion Pipeline Reliability" (2026-05-11). Closes the silent-failure gap exposed by the 2026-05-11 pilot across PDF / DOCX / EPUB / TXT / Markdown ingestion plus the playground notebook — Wave 1 + Wave 2 + 5 review-absorption rounds in one release. See [completed-phases.md](roadmap/completed-phases.md#phase-15-ingestion-pipeline-reliability-v060) for the full list of ingest-time guards (window-based PDF dedup, Turkish glyph normalisation profile, language-aware Unicode-block sanity check, ingest-time quality pre-signal, default-on audit `--quality-filter`, DOCX explicit header/footer subtraction, EPUB spine + whole-token nav/cover skip, TXT UTF-8 BOM + MD YAML frontmatter strip, notebook playground alignment, Wave 2 `--strip-pattern` (ReDoS-guarded) + `--page-range` + front-matter heuristic + `--strip-urls` + multi-column warning). **Earlier:** `v0.5.0` — Document Ingestion + Data Curation Pipeline (2026-04-30); `v0.4.5` — Quickstart Layer (2026-04-26); `v0.4.0` — Post-Training Completion (2026-04-26). -**Current state:** 19 phases (1, 2, 2.5, 3, 4, 5, 5.5, 6, 7, 8, 9, 10, 10.5, 11, 11.5, 12, 12.5, 12.6, 22) shipped on PyPI under `v0.5.5`. 3 phases (13, 14, 15) planned post-release; Phases 14 and 15 ship in parallel under `v0.6.0`, Phase 13 ships separately as `v0.6.0-pro` once adoption gates are met. +**Current state:** 20 phases (1, 2, 2.5, 3, 4, 5, 5.5, 6, 7, 8, 9, 10, 10.5, 11, 11.5, 12, 12.5, 12.6, 15, 22) shipped on PyPI through `v0.6.0`. Phase 14 (Multi-Stage Pipeline Chains) is merged on `development` and pending the `v0.7.0` PyPI tag. Phase 13 ships separately as `v0.6.0-pro` once adoption gates are met. > **Phase 12.6 task / sub-task dual-axis note:** Phase 12.6 is itself a 38-task closure cycle (Tasks 1-38) tracked at [`roadmap/completed-phases.md`](roadmap/completed-phases.md); per-wave PR descriptions carry the closure-task delta. ## Quick summary of what's planned -> **Note:** Arrows depict shipping order, not phase numbers (Phase 14 and Phase 15 both ship in v0.6.0 in parallel; Phase 13 ships later under the Pro tier). +> **Note:** Arrows depict shipping order, not phase numbers (Phase 15 shipped in v0.6.0; Phase 14 is targeted for v0.7.0 after Phase 15 displaced it; Phase 13 ships later under the Pro tier). ```mermaid graph LR @@ -51,8 +51,8 @@ graph LR P115 --> P12[Phase 12
Data Curation
Maturity] P12 --> P125[Phase 12.5
Data Curation
Follow-up] P125 --> P126[Phase 12.6
Closure
Cycle] - P126 --> P14[Phase 14
Pipeline
Chains] P126 --> P15[Phase 15
Ingestion
Reliability] + P15 --> P14[Phase 14
Pipeline
Chains] P14 --> P13[Phase 13
Pro CLI
+ Dashboard] P15 --> P13 @@ -63,8 +63,8 @@ graph LR P12 -.-> V2 P125 -.-> V2 P126 -.-> V25[v0.5.5] - P14 -.-> V23[v0.6.0] - P15 -.-> V23 + P15 -.-> V23[v0.6.0 ✅ Released] + P14 -.-> V27[v0.7.0] P13 -.-> V3[v0.6.0-pro] style P10 fill:#003300,stroke:#00ff88 @@ -73,8 +73,8 @@ graph LR style P115 fill:#004400,stroke:#88ff88 style P12 fill:#004400,stroke:#88ff88 style P125 fill:#004400,stroke:#88ff88 + style P15 fill:#004400,stroke:#88ff88 style P14 fill:#002244,stroke:#00aaff - style P15 fill:#002244,stroke:#00aaff style P13 fill:#442200,stroke:#ffaa00 ``` @@ -94,7 +94,7 @@ docs/ └── roadmap/ ├── completed-phases.md # Phase 1-12.6 archive (detailed) — Phase 10 / 10.5 / 11 / 11.5 / 12 / 12.5 / 12.6 absorbed inline (each shipped as v0.4.0 / v0.4.5 / v0.5.0 / v0.5.5) ├── phase-13-pro-cli.md # Planned — v0.6.0-pro (gated) - ├── phase-14-pipeline-chains.md # Planned — v0.6.0 (follow-up to the v0.5.5 closure cycle) + ├── phase-14-pipeline-chains.md # Merged on development — v0.7.0 tag pending (re-scheduled from v0.6.0; Phase 15 displaced it) ├── releases.md # v0.3.0 → v0.6.0 release notes └── risks-and-decisions.md # Risk matrix, opportunities, competitive positioning, decision log ``` diff --git a/docs/roadmap/completed-phases.md b/docs/roadmap/completed-phases.md index 34ad4b5c..68f45d2a 100644 --- a/docs/roadmap/completed-phases.md +++ b/docs/roadmap/completed-phases.md @@ -422,7 +422,7 @@ graph LR B -->|classify| C[Llama Guard] C --> D{safe / unsafe} D -->|count| E[unsafe_ratio] - E -->|compare| F{> threshold?} + E -->|compare| F{"> threshold?"} F -->|yes| G[❌ REVERT] F -->|no| H[✅ PASS] ``` @@ -1429,6 +1429,5 @@ notebook. ### Related -- [`docs/roadmap/phase-15-ingestion-reliability.md`](phase-15-ingestion-reliability.md) — the original phase doc with the 15 task list -- [`CHANGELOG.md`](../../CHANGELOG.md) `[0.6.0]` section +- [`CHANGELOG.md`](../../CHANGELOG.md) `[0.6.0]` section — release notes for the shipped tasks - PR #49 — five review-absorption rounds + tail-walk follow-up diff --git a/docs/roadmap/phase-14-pipeline-chains.md b/docs/roadmap/phase-14-pipeline-chains.md index b37bb763..c32cdc28 100644 --- a/docs/roadmap/phase-14-pipeline-chains.md +++ b/docs/roadmap/phase-14-pipeline-chains.md @@ -1,17 +1,22 @@ # Phase 14: Multi-Stage Training Pipeline Chains -> **Not:** Bu dosya tek bir planlanan fazı detaylandırır. Tüm fazların özeti için [../roadmap.md](../roadmap.md). +> **Status:** Implementation complete on `development` (branch `feat/phase-14-pipeline-chains`). Awaiting `v0.7.0` PyPI tag for the final "Done" promotion to [completed-phases.md](completed-phases.md). Originally targeted for `v0.6.0`; Phase 15 displaced it after the 2026-05-11 ingestion pilot — see [completed-phases.md#phase-15-ingestion-pipeline-reliability-v060](completed-phases.md#phase-15-ingestion-pipeline-reliability-v060). +> +> **Note:** This file details a single planned phase. See [../roadmap.md](../roadmap.md) for the cross-phase summary. -**Goal:** Enable a single config file to define a sequential SFT → DPO → GRPO training pipeline, with each stage using the previous stage's output as its base model. Eliminate the current requirement for manual config management across stages. -**Estimated Effort:** Low-Medium (3-6 weeks) -**Priority:** Medium-High — frequently requested by enterprise users; low implementation risk (uses existing trainers) +**Goal:** Enable a single config file to define a sequential SFT → DPO → GRPO training pipeline, with each stage using the previous stage's output as its base model. Eliminate the current requirement for manual config management across stages while preserving the existing single-stage workflow byte-identically. -> **Context:** Enterprise ML teams running production post-training pipelines currently need to write 3+ separate config files, manually set `model.name_or_path` to each stage's output, and orchestrate execution themselves. A `pipeline:` config key that chains stages solves this while remaining fully config-driven and testable. +**Estimated Effort:** Medium (4-6 weeks for Wave 1, including ~1 review-absorption round). The "3-week minimum" optimistic floor in earlier drafts was retired because the per-stage state machine (Task 5) and the inheritance-merge semantics (Task 2) both have non-obvious edge cases that surface during reviewer absorption, mirroring the Phase 15 cycle (5 absorption rounds for a comparable surface area). -### Tasks: +**Priority:** Medium-High — frequently requested by enterprise users since the v0.5.0 launch (preference / RL alignment workflows are the post-SFT default for regulated customers); low *core* implementation risk because every individual trainer already works. The risk surface is in **orchestration semantics**, not trainer mechanics: inheritance merge, partial-run resume, and per-stage gate composition. + +> **Context:** Enterprise ML teams running production post-training pipelines currently write 3+ separate config files, manually set `model.name_or_path` to each stage's output, copy LoRA / safety / compliance config blocks between them, and orchestrate execution themselves with shell scripts. Every manual copy is an opportunity to drift — a LoRA `r` value that matches across stages by accident becomes one that mismatches after the next config edit. A `pipeline:` config key that chains stages solves the orchestration *and* the drift problem in one move, while remaining fully config-driven, dry-run-validatable, and Annex-IV-traceable. + +## Tasks + +1. [ ] **`pipeline:` config section in `ForgeConfig`** + New optional `pipeline.stages: List[PipelineStage]` section. Each stage is a partial training config override layered onto the root config (see Task 2 for the merge semantics). Stages execute sequentially; each stage's `model.name_or_path` is automatically set to the previous stage's `training.output_dir/final_model` (or, if the previous stage gated on human approval, to the staging path after approval). Stage names must be unique within a pipeline and match `^[a-z0-9_]{1,32}$` so they can serve as identifiers in CLI flags (`--stage `, `--resume-from `) and audit-log fields without escaping. -1. [ ] **`pipeline:` config section in ForgeConfig** - New optional `pipeline: stages: [...]` section. Each stage is a full training config override. Stages execute sequentially; each stage's `model.name_or_path` is automatically set to the previous stage's `training.output_dir/final_model`. ```yaml pipeline: stages: @@ -39,38 +44,222 @@ dataset_name_or_path: "./data/math_prompts.jsonl" ``` + **Acceptance:** + - `ForgeConfig(**yaml_load(pipeline.yaml)).pipeline` is a typed `PipelineConfig` with a non-empty `stages` list and a unique-name validator (`pydantic.field_validator` raising `ValueError` on duplicate names). + - Stage name pattern enforced at config load (`^[a-z0-9_]{1,32}$`); invalid name → `EXIT_CONFIG_ERROR (1)` with an actionable message naming the offending stage. + - Empty `pipeline.stages: []` → rejected at load (a pipeline must have ≥ 1 stage; the operator who meant single-stage should not use the `pipeline:` key at all). + - Backward compatibility: a config file without a `pipeline:` key produces `config.pipeline is None`; all existing CLI paths see no behavioural change. + 2. [ ] **Stage inheritance and override semantics** - Each stage inherits the top-level config as defaults. Only explicitly set fields are overridden per stage. LoRA config, model config (except `name_or_path`), distributed config, and safety config inherit from the root unless overridden per stage. + Each stage inherits the top-level config as defaults; only fields the stage *explicitly sets* are overridden. Detection of "explicitly set" uses Pydantic's `model_fields_set` (not "field happens to equal default") so a stage that writes `learning_rate: 2e-5` — the same as the root default — still flags as overridden and the trainer sees the stage's value. This matters when reviewers expect "the YAML says it, so the YAML's value wins" symmetry. + + **Merge depth:** the merge is **section-wholesale**, *not* recursive-deep-merge. If a stage declares `lora:`, the entire `lora` block replaces the root `lora` block — partial-override on a nested key is rejected at config load. This is a deliberate trade-off: deep-merge nuances (`target_modules: [...]` append vs. replace?) compound across the 19 ForgeConfig submodels and turn config validation into an undebuggable maze. Wholesale replacement gives one clear rule: "if you want to inherit, omit the block; if you want to override, supply the *full* block." + + **Inheritance matrix** (per top-level config section): + + | Section | Inheritance | Notes | + |---|---|---| + | `model.name_or_path` | **Auto-chained** (overrides root after stage 0) | Set to previous stage's `training.output_dir/final_model`. Stage 0 still reads root `model.name_or_path`. An explicit per-stage `model.name_or_path` is allowed and disables auto-chaining for that stage (operator escape hatch). | + | `model.*` (other fields) | Inherited unless `model:` block overridden | `backend`, `load_in_4bit`, `trust_remote_code`, `max_length`, `chat_template` follow the root unless the stage supplies a full `model:` block. | + | `lora` | Inherited unless `lora:` block overridden | Critical edge case: stage 2 DPO with a *different* `lora.r` than stage 1 SFT means stage 2 is **a fresh LoRA over the merged SFT model**, not a continuation of SFT's LoRA. Documented in the operator guide. | + | `data` | Inherited unless `data:` block overridden (strongly discouraged) | Pipelines that don't change the dataset between stages are vanishingly rare — almost every real chain has SFT use a curated SFT dataset, DPO use a preference pairs dataset, and GRPO use a math/reward dataset. The schema does not *force* a per-stage override (operators may legitimately rerun the same dataset against a different trainer for ablation studies), but the operator guide flags the inheritance case as a smell and the dry-run summary prints a warning when two consecutive stages share the same `dataset_name_or_path`. | + | `training` | Inherited unless `training:` block overridden | `trainer_type` is required per stage; the validator rejects a stage with no `training.trainer_type` even if the root supplies one (each stage explicitly states its alignment paradigm for audit clarity). | + | `evaluation` | Inherited unless `evaluation:` block overridden | Per-stage gates (Task 5) live here; a stage that wants different `auto_revert` / `safety` config supplies a full `evaluation:` block. | + | `distributed` | Inherited (always); no per-stage override | Distributed strategy must be consistent across the pipeline run. Per-stage `distributed:` blocks rejected at load with a clear error. | + | `webhook` | Inherited (always); no per-stage override | One webhook configuration covers the pipeline; per-stage events carry the stage name in the payload. | + | `compliance` | Inherited (always); no per-stage override | Provider / system / risk metadata is pipeline-level, not stage-level. | + | `synthetic`, `merge` | Stage-local only; rejected at the root level when `pipeline:` is set | These flags describe non-training modes — using them inside a pipeline stage is the wrong layer of abstraction (synthesise-then-train should be two separate operator invocations). | + + **Acceptance:** + - `forgelm/config.py` exports `PipelineStage(BaseModel)` and `PipelineConfig(BaseModel)` Pydantic models with field-level validators implementing the matrix above. + - The merge happens in a single helper `_merge_stage_into_root(root_cfg, stage) -> ForgeConfig`; the rest of the codebase consumes a flat `ForgeConfig` per stage with no awareness of the pipeline layer. This keeps `forgelm/trainer.py` unmodified. + - Per-stage `distributed:` block → `EXIT_CONFIG_ERROR (1)` with message `"distributed:" cannot appear in pipeline stage ; declare it at the root.` + - Per-stage `model.name_or_path` is honoured and disables auto-chaining for that stage (logged at INFO). + - Test fixture matrix covers: `inherited`, `overridden`, `auto-chained`, `rejected-per-stage` for each of the 11 rows above (44 test cases total). + +3. [ ] **Pipeline compliance manifest** + New artefact `compliance/pipeline_manifest.json` (JSON, not YAML — matches the existing `training_manifest.json` format the auditors are already trained on) captures the full multi-stage provenance. **Per-stage `training_manifest.json` files are still emitted** inside each stage's `output_dir/compliance/` directory; the pipeline manifest is an *index* that ties them together, not a replacement. Satisfies EU AI Act Annex IV's "complete training lineage" requirement at the chain level. + + **Schema (illustrative):** -3. [ ] **Pipeline compliance artifacts** - `compliance/pipeline_manifest.yaml` captures the full multi-stage provenance: stage names, input/output model paths, per-stage metrics and artifacts. Satisfies EU AI Act Annex IV requirement for complete training lineage documentation. + ```json + { + "forgelm_version": "0.7.0", + "generated_at": "2026-06-15T12:34:56+00:00", + "pipeline_run_id": "pl_2026-06-15_a1b2c3", + "pipeline_config_hash": "sha256:", + "stages": [ + { + "name": "sft_stage", + "index": 0, + "trainer_type": "sft", + "status": "completed", + "input_model": "meta-llama/Llama-3-8B", + "output_model": "./checkpoints/stage1_sft/final_model", + "started_at": "2026-06-15T12:34:56+00:00", + "finished_at": "2026-06-15T14:21:08+00:00", + "duration_seconds": 6372, + "training_manifest": "./checkpoints/stage1_sft/compliance/training_manifest.json", + "metrics": { "eval_loss": 0.42 }, + "gate_decision": "passed", + "auto_revert_triggered": false + }, + { + "name": "dpo_stage", + "index": 1, + "trainer_type": "dpo", + "status": "skipped_due_to_prior_revert", + "input_model": "./checkpoints/stage1_sft/final_model", + "output_model": null, + "training_manifest": null, + "skipped_reason": "Stage sft_stage triggered auto_revert; subsequent stages did not run." + } + ], + "final_status": "stopped_at_stage", + "stopped_at": "sft_stage", + "annex_iv": { "": "..." } + } + ``` + + Stage `status` enum: `pending` | `running` | `completed` | `failed` | `gated_pending_approval` | `skipped_due_to_prior_revert` | `skipped_by_filter` (when `--stage X` runs only one stage). + + **Annex IV mapping:** the pipeline manifest discharges the same Article 11 / Annex IV obligations as the single-stage manifest, with two additions: (a) the **training lineage** is a chain rather than a single record, (b) the `pipeline_config_hash` proves the chain's *intent* is reproducible (the same config bytes always describe the same chain). Existing `forgelm verify-annex-iv` validator gains a `--pipeline` mode that follows the index, validates each per-stage manifest, and asserts chain integrity (stage N's `input_model` must equal stage N-1's `output_model`). + + **Acceptance:** + - `forgelm/compliance.py` exports `generate_pipeline_manifest(pipeline_state) -> Dict[str, Any]` and `export_pipeline_manifest(manifest, output_dir) -> str`. The orchestrator writes the manifest after every stage state transition (atomic-rename pattern, matching the existing `_atomic_write_json` helper). + - The manifest is **append-friendly under partial runs**: a `--stage sft_stage` invocation produces a 1-stage manifest with the other two stages as `status: skipped_by_filter`; a subsequent full run replaces the manifest atomically with a complete 3-stage version. + - `forgelm verify-annex-iv --pipeline ` returns exit 0 on a complete passing run, exits non-zero with a `chain_integrity_violation` reason when stage N's input doesn't match stage N-1's output. + - The per-stage `training_manifest.json` files remain individually valid against the existing single-stage Annex IV schema — no field is renamed or removed at the stage level. + +4. [ ] **CLI surface** -4. [ ] **CLI support** ```bash - # Single pipeline run + # End-to-end pipeline run forgelm --config pipeline.yaml - # Dry-run validates all stages + # Dry-run validates every stage (Pydantic + cross-stage chain integrity) before any GPU is allocated forgelm --config pipeline.yaml --dry-run - # Run specific stage only + # Run a single named stage in isolation (audit / re-run scenarios) forgelm --config pipeline.yaml --stage dpo_stage - # Resume from failed stage + # Resume after a failed / interrupted run from a named stage onward forgelm --config pipeline.yaml --resume-from dpo_stage ``` -5. [ ] **Per-stage auto-revert and gates** - Each stage can have independent evaluation gates. If a stage fails auto-revert, the pipeline stops and reports which stage failed. Subsequent stages are not run. + **`--dry-run` failure-collection mode.** Each stage is validated independently; the dispatcher **collects all stage-level errors before exiting** rather than stopping at the first failure. Mirrors `pytest --collectonly` and avoids the "fix one config bug, re-run, fix the next" loop that drives operators away. Exit `EXIT_CONFIG_ERROR (1)` if any stage fails validation; the error report names every offending stage + field. Chain-integrity validation (each stage's auto-chained `input_model` must point at a path under the previous stage's `output_dir`) runs *after* per-stage Pydantic validation passes. + + **`--stage ` semantics (partial-run resolution rules):** + + | Scenario | Behaviour | + |---|---| + | `--stage ` and `` is the first stage | Reads root `model.name_or_path`; runs normally. | + | `--stage ` and the previous stage's `output_dir/final_model` exists on disk | Auto-chains as if previous stage had just finished; INFO-logs the disk path being reused. | + | `--stage ` and the previous stage's output is missing | Hard-fails with `EXIT_CONFIG_ERROR (1)`: `Stage requires output at ; pass --input-model to override or run the full pipeline first.` No silent fallback to root `model.name_or_path`. | + | `--stage --input-model ` | Operator escape hatch: skips the auto-chain, uses `` as the stage's base model. The audit-log event records `input_source: "cli_override"` so reviewers can trace why the chain "broke" intentionally. | + | `--stage ` where `` is not in the config | Hard-fails at parse time with the list of valid stage names. | + + **`--resume-from ` semantics (state machine + persistence):** + - State file: `/pipeline_state.json` (atomic-write, JSON, schema mirrors the pipeline manifest's `stages[].status` enum). Updated after every stage transition. + - Resume rule: stages with `status: completed` and a `output_model` path that still exists on disk are **skipped** (logged at INFO) and their manifests are preserved. The named stage and every stage after it are re-run. + - Stale-state guard: if the on-disk `pipeline_state.json` was produced by a different `pipeline_config_hash` than the one being resumed against, the resume fails with `EXIT_CONFIG_ERROR (1)` — preventing "I resumed against a config that was edited mid-flight" silent divergence. Override via `--force-resume` (logged at WARNING, recorded in the audit event). + - Resume does **not** support intra-stage checkpoint resume in Phase 14. HF `Trainer.train(resume_from_checkpoint=...)` integration is deferred to a Phase 14.x follow-up; Phase 14 only resumes at stage boundaries. + + **`--output-format json` envelope:** pipeline runs emit a top-level JSON object with `pipeline_run_id`, `stages[]` (each with `status`, `started_at`, `finished_at`, per-stage exit code), and `final_status`. Single-stage existing envelope unchanged. + + **Acceptance:** + - `forgelm --config pipeline.yaml --dry-run` validates every stage and prints a per-stage diagnostic block; exits 0 on full success or 1 with the collected error report on any failure. + - `forgelm --config pipeline.yaml --stage ` with a missing previous-stage output exits 1 with the message shape above (regression test asserts the exact message format so operators can grep for it in CI logs). + - `forgelm --config pipeline.yaml --resume-from ` against a stale state file exits 1 with `pipeline_config_hash mismatch`; `--force-resume` allows the run with a WARNING. + - `--stage ` for an unknown name lists the valid stage names in the error message (no opaque "stage not found"). + - `--output-format json` envelope passes `jq -e '.pipeline_run_id'` on every multi-stage invocation. + +5. [ ] **Per-stage gates, auto-revert, and stage-boundary resume** + Each stage carries its own `evaluation:` block (Task 2 inheritance matrix); the gate logic that already runs in `ForgeTrainer._post_training_gate` (loss thresholds, benchmark min-score, safety regression, judge min-score, human-approval) executes **per stage**. If any stage fails an auto-revert, the pipeline stops at that stage; subsequent stages enter `status: skipped_due_to_prior_revert` in both the pipeline manifest and the state file. The reverted model still exists on disk per the existing single-stage auto-revert contract — Phase 14 does not change auto-revert semantics, it just composes them. + + **Human approval gate within a pipeline.** If a stage has `evaluation.require_human_approval: true`, the trainer follows the existing Phase 9 flow: model lands in `final_model.staging./`, exits with `EXIT_AWAITING_APPROVAL (4)`. Pipeline-specific behaviour: the orchestrator captures the staging path in the pipeline state (`status: gated_pending_approval`), prints the resume command the operator needs (`forgelm --config pipeline.yaml --resume-from ` after `forgelm approve `), and exits with the same `EXIT_AWAITING_APPROVAL (4)` code so CI/CD wrappers see one consistent gate signal. On resume, the auto-chained `input_model` for the next stage points at the *promoted* `final_model/` path (post-approval), not the `final_model.staging/` path. + + **Audit-log events (extends the existing 5-event vocabulary):** + - `pipeline.started` — run id, config hash, stage count, stage names + - `pipeline.stage_started` — stage name, index, input model, input source (chain/cli_override/root) + - `pipeline.stage_completed` — stage name, metrics summary, gate decision (`passed` / `failed`) + - `pipeline.stage_gated` — stage name, gate decision `approval_pending`, staging path (emitted instead of `stage_completed` when a stage exits `EXIT_AWAITING_APPROVAL`; lets dashboard / SIEM rules filter on the event name alone — Phase 14 review F-N-1) + - `pipeline.stage_reverted` — stage name, auto-revert reason, halt-pipeline=true + - `pipeline.force_resume` — operator-approved stale-hash override; carries `old_config_hash` + `new_config_hash` so reviewers can correlate to the audit trail (Phase 14 review F-B-2) + - `pipeline.completed` — final status, total duration, stage count + + These events live alongside (not replacing) the existing per-stage `training.*` events emitted by `ForgeTrainer`. Webhook notifier (`forgelm/webhook.py`) gains `notify_pipeline_started` / `notify_pipeline_completed` / `notify_pipeline_reverted` methods that wrap the existing 5-event vocabulary so Slack / Teams dashboards filtering on `event=training.failure` continue to work; pipeline-aware dashboards can additionally filter on the new `pipeline.*` events. + + **Acceptance:** + - 3-stage SFT→DPO→GRPO pipeline where DPO stage's evaluation gate fails: pipeline manifest shows DPO `status: failed` + `auto_revert_triggered: true`, GRPO `status: skipped_due_to_prior_revert`, final exit code matches the failing stage's exit code (not `EXIT_SUCCESS`). + - Same pipeline with `evaluation.require_human_approval: true` on the SFT stage: SFT exits with `EXIT_AWAITING_APPROVAL (4)`, pipeline state file records `status: gated_pending_approval` for SFT and `status: pending` for DPO / GRPO; after `forgelm approve ` + `forgelm --resume-from dpo_stage`, the pipeline picks up with DPO's input pointing at the promoted SFT `final_model/` path. + - Webhook stub asserts that `pipeline.started` fires once, `pipeline.stage_completed` fires per stage, `pipeline.completed` fires once, payload schema matches the documented spec. + - The existing single-stage `training.*` event vocabulary is unchanged; existing webhook consumers see no new events on non-pipeline runs. + +## Requirements + +- **Backward compatibility, byte-identical.** A config file without a `pipeline:` key produces *exactly* the same `forgelm/trainer.py` execution path as v0.6.0. `forgelm/trainer.py` itself is unmodified by Phase 14; the pipeline orchestrator is a layer above it. Regression test: re-run an existing single-stage config through Phase 14 code and assert the output JSONL / manifest / checkpoint bytes match a v0.6.0 reference fixture. +- **Each stage is independently testable** via `forgelm --config pipeline.yaml --stage ` (Task 4 partial-run rules). +- **Dry-run validates every stage** (Pydantic + chain integrity + manifest schema) before any GPU is allocated; collects all failures rather than stopping at the first. +- **Pipeline manifest integrates with existing `compliance.py`** — same atomic-write pattern, same JSON schema vocabulary; new fields added, none renamed or removed. +- **No new heavy dependencies.** Phase 14 uses only the existing Pydantic / PyYAML / pure-Python surface. No orchestration framework (Airflow / Prefect / Dagster) — pipelines stay in-process so the existing single-config-file UX scales to chains without a deployment story. +- **Determinism preserved.** For any fixed `pipeline.yaml` + input bytes, the run is deterministic *modulo the existing single-stage non-determinism surface* (GPU floating-point, dataloader RNG). The pipeline manifest's `pipeline_config_hash` proves config-level reproducibility. +- **Bilingual docs sweep.** Operator-facing guides updated in EN + TR: new `docs/guides/pipeline.md` + `pipeline-tr.md` (canonical "first pipeline" walkthrough), updates to `docs/reference/configuration.md` + `-tr.md` (the new `pipeline:` block), updates to `docs/guides/cli.md` + `-tr.md` (`--stage`, `--resume-from`, `--force-resume`). `tools/check_bilingual_parity.py --strict` must pass at PR open. +- **CLI help / wizard surface.** `forgelm --help` lists the three new flags with one-line descriptions; `forgelm wizard` does *not* gain a pipeline path in Phase 14 (single-stage wizard is already complex; chained pipelines are operator-grade and use the manual YAML surface). `tools/check_cli_help_consistency.py --strict` must pass. +- **Test budget.** ≥ 30 new tests across `tests/test_pipeline_config.py` (inheritance matrix), `tests/test_pipeline_orchestrator.py` (orchestrator state machine), `tests/test_pipeline_cli.py` (flag interactions), `tests/test_pipeline_compliance.py` (manifest schema + Annex IV verification). Coverage delta target: `+3 %` overall, with new pipeline modules ≥ 90 % line coverage. +- **Honest framing in user-facing docs.** The "Limitations" section of `docs/guides/pipeline.md` calls out: (a) no intra-stage checkpoint resume (deferred), (b) no DAG semantics (sequential only), (c) no parallel stage execution (sequential only), (d) no `forgelm wizard` integration. Operators see what does *not* work before they hit it. + +## Test fixture sketch (Phase 15-style matrix) + +Committed under `tests/fixtures/pipeline/`: + +| Fixture | Purpose | +|---|---| +| `minimal_3_stage.yaml` | SFT → DPO → GRPO with the smallest valid config per stage; smoke baseline. | +| `inheritance_matrix.yaml` | Hits every row of the inheritance matrix (auto-chain, inherited, overridden, rejected-per-stage). | +| `gated_pending_approval.yaml` | Stage 1 has `require_human_approval: true`; orchestrator must exit `EXIT_AWAITING_APPROVAL (4)` and write the gate state. | +| `auto_revert_at_stage_2.yaml` | Stage 2 has a `max_acceptable_loss` that the synthetic eval guarantees to exceed; subsequent stages must record `skipped_due_to_prior_revert`. | +| `invalid_distributed_per_stage.yaml` | Per-stage `distributed:` block; orchestrator must reject at config-load time with the exact error message documented in Task 2. | +| `stale_state_resume.yaml` | Pair of configs differing in `learning_rate`; on resume against the first state file, the second config must trigger the `pipeline_config_hash mismatch` exit. | + +Each fixture has a golden pipeline manifest committed alongside; regression suite runs `forgelm` against the fixture (with mocked trainers via the existing `MagicMock(ForgeTrainer)` pattern) and byte-compares. + +## Delivery + +- **Target release:** `v0.7.0`. Re-slotted from the earlier `v0.6.0` placeholder; Phase 15 (Ingestion Pipeline Reliability) displaced it after the 2026-05-11 ingestion pilot exposed silent-failure gaps that demanded a focused release — see [completed-phases.md#phase-15-ingestion-pipeline-reliability-v060](completed-phases.md#phase-15-ingestion-pipeline-reliability-v060). +- **Entry gate (all must have held prior to the first Phase 14 PR):** + - The 6 regression fixtures listed above were committed as zero-byte placeholders under `tests/fixtures/pipeline/` so the file paths were locked. + - The Phase 14 issue (`#TBD`) was opened on GitHub referencing this file. + - A Phase 14 GitHub label was created so review-round PRs could be grouped. + - Baseline coverage of `forgelm/config.py` and `forgelm/compliance.py` was measured against the v0.6.0 tag and recorded in the Phase 14 issue (consistent with the Phase 15 entry-gate discipline). +- **CHANGELOG plan:** every Wave 1 task lands a one-line bullet under `[Unreleased]` per Keep-a-Changelog convention; at `v0.7.0` tag time the `[Unreleased]` block is renamed to `[0.7.0] — YYYY-MM-DD` per the existing release ritual. +- **Validation gate to ship Phase 14:** all 6 regression fixtures pass byte-comparison; bilingual parity + CLI help consistency guards green; full `pytest tests/` green; new `forgelm verify-annex-iv --pipeline ` validator returns exit 0 on every fixture's golden manifest. At least one end-to-end manual run against a 3-stage real pipeline (operator-supplied datasets) is signed off in the Phase 14 issue before the release tag. +- **Wave 2 / deferred items** (separate follow-up phases, not part of v0.7.0): + - Intra-stage HF `Trainer.train(resume_from_checkpoint=...)` integration (would let `--resume-from` pick up mid-stage rather than at stage boundaries). + - DAG pipelines (non-linear stage dependencies) — would require a different config schema and explicit dependency declaration; v0.7.x or later. + - Parallel stage execution (independent branches running concurrently) — gated on the DAG schema; same horizon. + - `forgelm wizard` pipeline path — gated on operator demand after v0.7.0 ships. + +--- + +## Cross-references + +- **Code surface affected:** [`forgelm/config.py`](../../forgelm/config.py) (new `PipelineStage` + `PipelineConfig` models, root-level `pipeline` field), [`forgelm/cli/_parser.py`](../../forgelm/cli/_parser.py) (`--stage`, `--resume-from`, `--force-resume`, `--input-model`), [`forgelm/cli/_dispatch.py`](../../forgelm/cli/_dispatch.py) + new `forgelm/cli/_pipeline.py` (orchestrator), [`forgelm/compliance.py`](../../forgelm/compliance.py) (`generate_pipeline_manifest`, `verify_annex_iv --pipeline` mode), [`forgelm/webhook.py`](../../forgelm/webhook.py) (`notify_pipeline_*` methods). [`forgelm/trainer.py`](../../forgelm/trainer.py) is **not** modified — the orchestrator composes existing `ForgeTrainer` instances per stage. +- **User-facing surfaces to add/update:** new `docs/guides/pipeline.md` + `-tr.md` (planned, created by Phase 14 implementation), updates to [`docs/reference/configuration.md`](../reference/configuration.md) + `-tr.md`, new `docs/guides/cli.md` + `-tr.md` (planned), and [`config_template.yaml`](../../config_template.yaml) gains a commented-out `pipeline:` example block. +- **Standards consulted:** [`docs/standards/error-handling.md`](../standards/error-handling.md) (exit code stability), [`docs/standards/architecture.md`](../standards/architecture.md) (layering — orchestrator above trainer), [`docs/standards/testing.md`](../standards/testing.md) (fixture matrix discipline), [`docs/standards/localization.md`](../standards/localization.md) (EN ↔ TR mirror). +- **Pattern reference:** Phase 15's review-absorption discipline + fixture matrix is the working model — see [completed-phases.md#phase-15-ingestion-pipeline-reliability-v060](completed-phases.md#phase-15-ingestion-pipeline-reliability-v060). + +## Release-time follow-up (deferred to the `v0.7.0` `cut-release` cycle) + +Phase 14 implementation ships on the `development` branch ahead of the +PyPI tag. The two operator-facing surfaces below are intentionally not +edited inside this PR — they are released-product surfaces and stay +v0.6.0-shaped until `cut-release` actually tags `v0.7.0`: -### Requirements: -- Single-stage configs (no `pipeline:` key) must be completely unaffected — 100% backward compatible. -- Each stage is independently testable via `--stage `. -- Pipeline manifest integrates with existing `compliance.py` audit log. -- `--dry-run` validates all stages' configs before any GPU allocation. +- **`docs/usermanuals/_meta.yaml` + `docs/usermanuals/{en,tr,de,fr,es,zh}/training/pipeline.md`** — new training-section page mirroring `docs/guides/pipeline.md`. TOC entry needs 6 locale titles (`en`/`tr` mandatory; `de`/`fr`/`es`/`zh` fall back to EN per `docs/standards/localization.md`). +- **`site/features.html` + `site/index.html` + `site/js/translations.js`** — new "Multi-stage training pipelines" feature card and a hook into the existing `home.pipeline` (alignment-stack) section so the two distinct senses of the word "pipeline" don't collide visually. 6-language i18n keys (`features.pipeline.title`, `features.pipeline.body`, `home.pipeline.chain.cta`) gated at full parity by `tools/check_site_claims.py`. +- **`tools/update_site_version.py`** — Phase 14 ships under the `v0.7.0` literal bump, which the version guard rewrites across `site/*.html` and `site/js/translations.js` automatically. -### Delivery: -- Target release: `v0.6.0` (reslotted from earlier `v0.5.x` placeholders; Phase 14 is grouped with the Phase 13 Pro CLI tier per the v0.5.5 closure cycle's deferred-callout policy in `docs/usermanuals/{en,tr}/deployment/model-merging.md` and the [Phase 12.6 closure cycle archive](completed-phases.md#phase-126-closure-cycle-38-tasks-across-5-waves)). -- No hard blockers; the v0.5.5 closure cycle has merged. Phase 14 starts after the v0.5.5 PyPI tag is published. +The `cut-release` skill ([.claude/skills/cut-release/SKILL.md](../../.claude/skills/cut-release/SKILL.md)) walks these in order at `v0.7.0` tag time. --- diff --git a/docs/roadmap/phase-15-ingestion-reliability.md b/docs/roadmap/phase-15-ingestion-reliability.md deleted file mode 100644 index 5a8d1162..00000000 --- a/docs/roadmap/phase-15-ingestion-reliability.md +++ /dev/null @@ -1,135 +0,0 @@ -# Phase 15: Ingestion Pipeline Reliability ✅ - -> **Status:** Released in `v0.6.0` on 2026-05-11. See -> [completed-phases.md](completed-phases.md#phase-15-ingestion-pipeline-reliability-v060) -> for the shipped scope + [releases.md](releases.md) for the release -> notes. This file is the original phase doc preserved for historical -> reference. - -> **Note:** This file details a single planned phase. See [../roadmap.md](../roadmap.md) for the cross-phase summary. - -**Goal:** Close the silent-failure gap in `forgelm ingest` exposed by an end-to-end pilot test against a real-world Turkish-language formal-publication PDF on 2026-05-11. The current pipeline produces output that *looks* successful (no errors, no warnings) but is polluted with multi-line running headers, font-fallback glyphs, bullet-character corruption, page-number leakage, and front-matter noise — degrading downstream SFT quality without any operator-facing signal. Phase 15 turns these silent failures into either auto-corrections or loud warnings, **across all five supported formats** (PDF, DOCX, EPUB, TXT, Markdown), and bakes the regression coverage into a fixture matrix. - -**Estimated Effort:** Medium (4-6 weeks, Wave 1 + Wave 2) -**Priority:** High — gates the credibility of ForgeLM's "config-driven, enterprise-grade" claim on the document ingestion surface. Without Phase 15, real-world Turkish / multi-script corpora produce silently poor data and customers blame the model instead of the pipeline. - -> **Context:** Surfaced by two end-to-end pilot runs on 2026-05-11 against a 193-page real-world Turkish-language formal-publication PDF (corpus-agnostic, institutional specifics scrubbed). The first run via the CLI (`--chunk-size 4000`) flagged 74 / 82 chunks below the `--quality-filter` threshold; the second run via `notebooks/ingestion_playground.ipynb` on Colab (`--chunk-size 1024`, both masks on) flagged 161 / 188 chunks. In both runs `forgelm ingest`'s mechanical-success summary reported no issues — multi-line running headers, font-fallback glyphs, bullet-character corruption, page-number leakage, recurring institutional-URL pollution, and front-matter chunks all sailed through silently. The same six failure modes generalise across the format × genre × language matrix the pipeline is supposed to handle; the pilots exposed them rather than introduced them. The cross-config replay also empirically verified the determinism guarantee (Turkish character count and U+085F bullet-glyph count were byte-identical across both passes), proving the failures are extraction-layer, not chunking-layer, and that fixes can be locked in with byte-comparison regression fixtures. Phase 15 is the planned remediation programme spanning **CLI surface, library surface, and the user-facing playground notebook** in one coherent v0.6.0 release. - -### Tasks - -#### Wave 1 — silent-failure remediation (v0.6.0, ~3 weeks, indicative per-week split below) - -**Week 1 — extraction-layer reliability:** Tasks 1 (multi-line dedup), 2 (script sanity), 3 (Turkish glyph normalisation). Three tightly-coupled changes inside [`forgelm/ingestion.py`](../../forgelm/ingestion.py); one dev-week is realistic if the engineer is already familiar with `_strip_repeating_page_lines` and the audit's §1.1 nuance. - -**Week 2 — operator signal + non-PDF format upgrades:** Tasks 4 (ingest-time quality pre-signal), 5 (audit `--quality-filter` default-on), 6 (DOCX explicit header / footer), 7 (EPUB spine + skip), 8 (TXT / MD BOM + frontmatter). Five smaller, independent changes that parallelise across two devs if needed. - -**Week 3 — surface integration + regression locking:** Task 9 (notebook UX alignment — gated on Tasks 4 + 5) and Task 10 (the 7-fixture regression suite, dual-golden chunk-size variants). Fixture authoring + golden JSONL generation is the heavier half. - -These splits are estimates, not commitments — task 1's multi-line dedup may surprise on degenerate corpora, and task 9's notebook update only matters once tasks 4+5 land, so Wave 1 can finish faster if extraction tasks complete clean. - -1. [x] **Multi-line PDF edge dedup — fix exit-condition trap, not "add an extra check".** - The current `_strip_repeating_page_lines` in [`forgelm/ingestion.py`](../../forgelm/ingestion.py) is already iterative — a `while True:` loop is supposed to peel one edge line at a time and re-check on each pass. The audit (§1.1) traced the observed failure to the loop's **exit condition**: pass 1 only inspects `lines[0]` and `lines[-1]`, so when the outermost line varies per page (a per-section title on top, a page number on the bottom) the loop strips zero lines on pass 1 and breaks *before* ever reaching the constant line that sits one row deeper. Future implementers reading just the docstring will assume iterative-peel is enough — the comment block above the loop must spell out *why iterative-peel alone fails on variable-outer-line corpora*. Concrete change: replace the pass-1-only `_repeating_edge_lines` with a top-N / bottom-N window check (N = 3 default, exposed as `_PDF_EDGE_WINDOW`) so a line recurring at **any** position within the top-3 / bottom-3 rows across ≥ 70 % of pages gets stripped in a single pass. Inline comment (1–2 sentences) explains the iteration-trap rationale so this regression cannot return silently. Run a second dedup pass after paragraph packing to catch headers that the chunker re-glued mid-block. Acceptance: on the synthetic Turkish-formal-publication regression fixture, the `""` pollution drops from ≥ 70 % of chunks to ≤ 5 %. - -2. [x] **Unicode / script sanity layer.** - New helper `_check_script_sanity()` runs after every per-file extraction. For each format-detected language (uses existing `langdetect` budget — already an `[ingestion]` extra dependency through `langdetect`), computes the ratio of code-points outside the language's expected Unicode blocks. Above a configurable threshold, logs a `WARNING` naming the file and a guess at the cause (font fallback, mojibake, mixed-script noise). Threshold and language hint exposed as `--script-sanity-threshold` and `--language-hint` CLI flags. **Default-threshold calibration is fixture-driven** — the audit measured ≈ 7 % corrupt-char ratio on the pilot's front matter and 0 % on clean body chunks, so the placeholder `0.5 %` value in this draft is *intentionally aggressive* and will be re-calibrated against the regression fixture corpus (Task 10) before merge. Final default is set by a precision / recall sweep on the fixture matrix (target: ≥ 95 % recall on font-corrupt fixtures, ≤ 5 % false-positive rate on clean mixed-script fixtures); likely range `1.0 – 2.0 %`. Acceptance: the front-matter chunks and the custom-glyph-bullet section of the synthetic Turkish-formal-publication fixture each emit one targeted warning *with the calibrated default*, and at least one legitimate mixed-script fixture (e.g. a Turkish corpus with embedded English code samples) emits zero warnings. - -3. [x] **Turkish pypdf glyph normalisation table.** - New module `forgelm/_pypdf_normalise.py` ships a static `dict` mapping known-pypdf-fallback glyphs to their correct Turkish characters (`ø → İ`, `Õ → ı`, `ú → ş`, `÷ → ğ`, `ö → Ğ` *(with trailing space)*, `࡟ → •` etc., derived from the audit's measured artefacts). Applied at chunk-write time by default, opt-out via `--no-normalise-unicode`. Module is **language-specific** but pluggable: a `--normalise-profile turkish|none` flag selects the table, leaving room for `arabic` / `cjk` profiles in later phases. Acceptance: (a) re-ingesting the synthetic Turkish-formal-publication fixture produces zero `ø Õ ú ÷ ࡟` artefacts in the JSONL; (b) `forgelm doctor` reports a `pypdf_normalise.turkish: pass` diagnostic row so an operator can verify the table loaded correctly without having to run a test ingest first. - -4. [x] **Ingest-time quality pre-signal.** - At the end of `ingest_path()`, run a lightweight 3-check sanity sweep (alpha-ratio per chunk, weird-char-ratio per chunk, repeated-line-ratio per chunk — *cheap* versions of the audit's full quality filter, not the expensive ones). If any chunk falls below the warn threshold, print a single end-of-summary line: - - ``` - [WARN] 74/82 chunks below ingestion quality threshold. Run - `forgelm audit --quality-filter` for detail. - ``` - - The threshold and disable-flag (`--no-quality-presignal`) live next to the existing `--output-format` flag. The summary line is the smallest possible nudge — full diagnostics stay in the audit step. - -5. [x] **Audit `--quality-filter` default-on.** - In [`forgelm/cli/_parser.py`](../../forgelm/cli/_parser.py) the audit sub-parser flips `--quality-filter` from opt-in to default-on; add `--no-quality-filter` for parity. Update [`docs/guides/data_audit.md`](../guides/data_audit.md) + TR mirror. Backward-compat: existing scripted invocations continue to work (no behaviour change when the flag is explicitly passed). - -6. [x] **DOCX explicit header / footer extraction.** - Extend `_extract_docx` in [`forgelm/ingestion.py`](../../forgelm/ingestion.py) to read `doc.sections[i].header` and `.footer` once per document, collect into a set, and subtract those exact lines from the body extraction. Word DOCX declares these explicitly — no statistical guess needed. Acceptance: a fixture DOCX with a 3-line header repeating across 10 pages produces JSONL with zero header lines. - -7. [x] **EPUB spine order + nav / cover / copyright skip.** - Modify `_extract_epub` in [`forgelm/ingestion.py`](../../forgelm/ingestion.py) to iterate `book.spine` (reading order) instead of `book.get_items()` (file order). Maintain a configurable skip-list of item names / `epub:type` values (`nav`, `cover`, `copyright`, `colophon`, `titlepage`, `frontmatter`) — defaulted to skip, opt-out via `--epub-no-skip-frontmatter`. Acceptance: a fixture EPUB with explicit cover / TOC / copyright pages emits JSONL containing only the body chapters in reading order. - -8. [x] **TXT / MD BOM strip + frontmatter detection.** - Extend `_extract_text` in [`forgelm/ingestion.py`](../../forgelm/ingestion.py) to strip the UTF-8 BOM (``) if present at position 0. For `.md` files only, detect and strip YAML frontmatter (`^---\n...\n---\n` at file start) by default; opt-out via `--keep-md-frontmatter`. CRLF → LF normalisation already happens on output via `newline="\n"`; verify no regression. Acceptance: a fixture TXT with a UTF-8 BOM and a fixture MD with YAML frontmatter produce clean chunks. - -9. [x] **Notebook UX alignment — surface the quality signal in the playground.** - `notebooks/ingestion_playground.ipynb` is the canonical "first 5 minutes with ForgeLM ingestion" surface and currently terminates in a misleading "looks clean" UX even when 161 / 188 chunks are quality-flagged (the 2026-05-11 second-pass evidence). After Task 5 ships `--quality-filter` as default-on, the notebook needs four small updates so the new signal reaches Colab users without the operator having to know to look for it: - - **Cell 5 config block** — expose `CHUNK_TOKENS = None` and `TOKENIZER = None` (token-aware mode) alongside the existing `CHUNK_SIZE` knob. Comment block explains: "set both to switch from char-based to token-based chunking; `TOKENIZER` is required when `CHUNK_TOKENS` is set." Addresses the audit §3.6 finding that token-aware mode forced operators to drop to a custom command. - - **Cell 8 markdown** — extend the "what audit gives you" list with the quality-filter checks (alpha-ratio, mean word length, end-of-line punctuation, repeated-line ratio, short-paragraph ratio). Honest framing: heuristic, conservative, surfaces extraction-layer noise. - - **Cell 9 invocation** — explicit `--quality-filter` flag for forward-compat with operators on older releases; once Task 5 default-on lands the flag is redundant but harmless. Keeps the notebook robust across the v0.5 → v0.6 transition. - - **Cell 10 pretty-print** — surface `quality_summary` alongside the existing PII / secrets / near-duplicate / language blocks. Show `samples_flagged / samples_evaluated`, `overall_quality_score`, and the `by_check` breakdown when any check fires. - Acceptance: re-running the playground notebook against the synthetic Turkish-formal-publication regression fixture surfaces a non-silent quality signal before the operator clicks "download ZIP", AND operators can flip the notebook into token-aware mode without leaving the markdown surface. - -10. [x] **Wave 1 regression fixture matrix.** - Commit the 7 fixtures listed in the audit's section 7 under `tests/fixtures/ingestion/`. **All fixtures are synthesised inside the repository under the project's permissive licence**; no external copyright is incurred. The `tr_textbook_4pages.pdf` fixture is a four-page Turkish-formal-publication PDF **constructed from scratch inside the repository** with a deterministic regeneration script alongside it. By deliberate construction it exhibits each of the six failure modes documented in audit §1: a multi-line running header (variable section title + constant publication-identifier line), Type-1-font Turkish glyph fallback on its first two pages, custom-bullet `U+085F` in one chapter, a recurring institutional-style URL pattern with varying numeric query strings, ToC-style underscore leaders on its third / fourth pages, and inline page numbers. Each fixture has a golden JSONL output alongside; new test module `tests/test_ingestion_reliability.py` runs `ingest_path()` against every fixture and byte-compares to the golden file. The `tr_textbook_4pages.pdf` fixture ships with **two** golden JSONLs — `--chunk-size 4000` and `--chunk-size 1024` — to lock in both the running-header-heavy and the front-matter-isolated failure modes per the audit's section 1.7 cross-config replay finding. The `mixed_directory/` fixture's acceptance is explicit: `forgelm ingest mixed_directory/ --recursive --output golden.jsonl` produces a JSONL whose chunks appear in **lexicographic file-name order** across all 5 formats (PDF, DOCX, EPUB, TXT, MD), with format-tagged provenance derivable from `notes_structured.format_counts`. Determinism guarantees from the audit's section 5 make byte-compare viable; if pypdf's per-page extraction shifts under a version bump, the test fails loudly and the fixture is regenerated explicitly via the committed regeneration script. - -#### Wave 2 — strong nice-to-have (v0.6.x, ~2 weeks) - -1. [x] **`--strip-pattern REGEX` flag — operator-controlled regex with ReDoS guard.** - Operator escape hatch for known-boilerplate that the dedup heuristic misses. Multiple patterns allowed (`--strip-pattern A --strip-pattern B`). Applied after extraction, before chunking. **Operator-supplied regexes must pass the regex-hardening guardrails defined in [`docs/standards/regex.md`](../standards/regex.md)** — the same 8-rule discipline (no unbounded greedy quantifiers in sequence, no nested groups with overlapping ranges, bounded repetition) that ForgeLM's own internal regexes obey. Implementation choice: validate the user-supplied pattern at CLI-parse time via the same SonarCloud python:S5852-style structural check ForgeLM applies internally, **and** wrap the match call in a per-pattern timeout (`re.match` with `signal.alarm` or `regex.match(... timeout=N)` if the `regex` library is added as a Wave 2 dependency — operator can opt out via `--strip-pattern-no-timeout` if they're certain). Documented in the operator guide with the canonical examples (running headers with variable content, journal DOI lines, watermarks). Acceptance: a pathological `(a+)+b`-class pattern aborts with a clear `EXIT_CONFIG_ERROR (1)` and a message naming the offending pattern, instead of hanging the ingestion process. - -2. [x] **`--page-range` PDF flag.** - Simple `--page-range 12-193` filtering for PDF files. Skips front-matter / back-matter manually when the heuristic in task 13 mis-fires. Validated as `min ≤ max`, `min ≥ 1`, `max ≤ pdf_page_count`. Validation failure **exits with `EXIT_CONFIG_ERROR (1)`** via the CLI dispatcher (matching the existing `_run_ingest_cmd` pattern in [`forgelm/cli/subcommands/_ingest.py`](../../forgelm/cli/subcommands/_ingest.py)) and prints a clear message naming the offending range and the actual page count. Library-level `ingest_path()` raises `ValueError` as before — the exit-code translation lives at the CLI boundary per the [error-handling standard](../standards/error-handling.md). - -3. [x] **Front-matter heuristic detection.** - Heuristic that looks at the first N pages of a PDF (N=12 default): if a page has alpha-ratio < 0.45 *and* underscore-ratio > 0.10 *and* contains ≥ 5 page-number-pattern matches, the page is tagged as "front-matter candidate" and dropped, with a `WARNING` listing the dropped page indices and the per-page metrics. Opt-out via `--keep-frontmatter`. Symmetric pass on the last N pages catches back-matter (bibliographies, indexes); same threshold. - -4. [x] **URL / QR pattern auto-strip option.** - `--strip-urls` flag: detects URL patterns (http(s)://, www.) per chunk and either replaces with `[URL]` or strips entirely (`--strip-urls=mask|strip|keep`, default `keep`). Useful for exercise-reference QR-noise and academic-paper DOI-noise. Existing PII / secrets mask pipeline integration: applies *after* secrets mask, *before* PII mask, preserving the mask-order discipline. **Interaction with `--all-mask`:** the `--all-mask` shorthand (Phase 12.5) is explicitly *not* extended to include `--strip-urls`. Rationale: PII / secrets are GDPR-grade compliance redactions whose accidental over-application is recoverable (the original text is still in the source PDF); URL stripping is a content-shape decision that operators may want to keep off while still demanding the privacy redactions. The two flag families stay independent and the operator opts into URL handling explicitly. Documented at the CLI help text for both `--all-mask` and `--strip-urls`. - -5. [x] **Multi-column PDF detection.** - Heuristic that estimates column count from pypdf's text-position metadata: per page, cluster x-positions of extracted spans; if two clusters dominate (separation > 30 % of page width), tag the document as 2-column. No fix attempt — just a `WARNING` at run start: `[WARN] Detected 2-column layout — reading order may be scrambled. Consider --strategy sliding with larger chunk-size, or pre-process with a layout-aware tool.` Better-than-nothing signal; cheaper than a layout-aware extractor. - -#### Wave 3 — deferred (post-Phase 15) - -1. [ ] OCR detection + automated `ocrmypdf` recommendation -2. [ ] Document profile / preset system (`--profile academic`, `--profile legal`, `--profile textbook`) -3. [ ] PDF table extractor (camelot-py / pdfplumber integration as optional `[ingestion-tables]` extra) -4. [ ] Footnote / endnote separation (DOCX + PDF) -5. [ ] Language-aware paragraph segmentation (CJK / Arabic / RTL scripts) -6. [ ] RTL script extraction-order normalisation - -These items go on the [risks-and-decisions](risks-and-decisions.md) backlog with the Phase 15 audit as their provenance reference, to be re-prioritised after Wave 1 lands and operator feedback arrives. - -### Requirements - -- **Backward compatibility:** Every Wave 1 / Wave 2 change has an opt-out flag. Existing pipelines that explicitly want the old behaviour (no normalisation, no front-matter skip, no quality pre-signal) get it with a single flag. No silent behaviour change for users with frozen scripts. -- **Determinism preserved — at two distinct levels:** - - **Level 1 (intra-chunk-size byte-identical):** for any fixed `--chunk-size`, `--strategy`, mask flags, and input bytes, the resulting JSONL is byte-identical across runs and across hosts (lex-sorted file walk, no clock dependency, deterministic mask order). This is the property Task 10's fixture-vs-golden byte-compare relies on; the regression suite breaks loudly if any fix introduces nondeterminism here. Documented in [`docs/reference/data_ingestion_architecture.md`](../reference/data_ingestion_architecture.md). - - **Level 2 (inter-chunk-size count-identical):** at *different* `--chunk-size` values the JSONL bytes naturally diverge (different chunk boundaries), but the **extraction-layer artefact counts** (per-language character counts, font-fallback glyph counts, U+085F bullet counts, redaction counts) remain identical because they are produced by the format extractor *before* chunking. The 2026-05-11 pilot runs empirically verified this: the same source PDF at `--chunk-size 4000` and `--chunk-size 1024` both produced 20,880 Turkish characters and 189 U+085F bullets, byte-for-byte. This invariant justifies the dual-golden fixture strategy in Task 10 — the same fixture can carry two byte-identical-per-config goldens *and* a single set of cross-config extraction-count assertions. - Glyph-normalisation tables are static dicts; quality pre-signal does not change JSONL content, only adds a stderr log line; spine-order EPUB extraction is deterministic. None of these fixes weaken either determinism level. -- **Format coverage symmetry:** Wave 1 must touch all five formats (PDF tasks 1-4, DOCX task 6, EPUB task 7, TXT / MD task 8) **plus the user-facing notebook surface** (task 9). Phase 15 does not ship a "PDF reliability release" that ignores the other four formats, and does not ship a "CLI-only release" that leaves the playground notebook regressed. -- **Notebook surface in-scope:** [`notebooks/ingestion_playground.ipynb`](../../notebooks/ingestion_playground.ipynb) is the canonical "first 5 minutes with ForgeLM ingestion" surface and currently propagates the silent-failure UX even after Tasks 4-5 land at the CLI layer. Task 9 closes that gap. The notebook update PR is gated on **Tasks 4 + 5** landing first (Task 4 produces the ingest-time pre-signal the notebook will surface; Task 5 makes `--quality-filter` default-on so the audit JSON the notebook reads carries the quality data). Task 8 (TXT/MD BOM) is unrelated to the notebook UX and not part of the gating chain. Notebook update ships in the same v0.6.0 release. -- **No new heavy dependencies in Wave 1:** Tasks 1-10 use only `pypdf`, `python-docx`, `ebooklib`, `bs4` (already shipped under `[ingestion]`) and `langdetect` (already shipped). Camelot / pdfplumber / Tesseract are Wave 3 only and gated behind new optional extras. -- **Bilingual docs sweep:** Operator-facing guides updated in EN + TR ([`docs/guides/ingestion.md`](../guides/ingestion.md), [`docs/guides/data_audit.md`](../guides/data_audit.md), [`docs/reference/data_ingestion_architecture.md`](../reference/data_ingestion_architecture.md), and their `-tr.md` mirrors). The bilingual parity guard (`tools/check_bilingual_parity.py --strict`) must pass at PR open. Notebooks are EN-only by project convention and exempt from the bilingual mirror requirement. -- **Honesty in the "Limitations" surfaces:** The architecture reference's existing "Limitations" sections gain a sub-section per still-deferred Wave 3 item (multi-column, OCR, RTL). Users see what does *not* work before they hit it. -- **Test budget:** Wave 1 ships with the 7-fixture regression suite (Task 10) including the dual-golden chunk-size variant on `tr_textbook_4pages.pdf`. Coverage delta target: `forgelm/ingestion.py` line coverage ≥ 90 % after Wave 1 lands. -- **No regression on the v0.5.0 contracts:** existing exit codes (0/1/2/3/4/5), the `[REDACTED]` / `[REDACTED-SECRET]` mask vocabulary, the `IngestionResult` schema, and the `--output-format json` envelope are stable across Phase 15. New fields may be added; nothing existing renames or disappears. - -### Delivery - -- **Target release:** Wave 1 ships as `v0.6.0` alongside [Phase 14 (Multi-Stage Pipeline Chains)](phase-14-pipeline-chains.md). The two phases are independent and parallelisable. -- **Wave 2 ships as `v0.6.x` follow-up** (any of `v0.6.1`, `v0.6.2`) at maintainer discretion. -- **Wave 3 deferred to Phase 16+** — separately re-prioritised after Wave 1 ships and operator feedback arrives. -- **Gate to start Wave 1 (entry conditions, all must be true before the first Wave 1 PR opens):** - - The 7 regression fixtures (proposed in Task 10) are committed as zero-byte placeholders with golden-JSONL stubs to lock the file paths. - - The Phase 15 issue (`#TBD`) is opened on GitHub referencing this file. - - The 2026-05-11 pilot run is recorded (institutional specifics scrubbed) so the v0.6.0 acceptance gate can re-run against the synthetic regression fixture. - - **Fixture authorship verification.** `tr_textbook_4pages.pdf` and every other fixture under `tests/fixtures/ingestion/` is **synthesised inside this repository under the project's permissive licence**, with a deterministic regeneration script alongside each binary asset. No third-party copyright applies. CI's existing licence-header guard runs over the fixture sources. - - **Coverage baseline measurement.** Run `pytest --cov=forgelm/ingestion --cov-report=term-missing` against `main` (v0.5.7) and record the current line-coverage percentage in the Phase 15 GitHub issue. The "≥ 90 %" target in Test budget below is a delta target; reaching 90 % from a 50 % baseline is materially different work from reaching it from an 80 % baseline, and the entry-gate measurement locks in honest expectations. -- **CHANGELOG plan:** under the `[Unreleased]` section of [`CHANGELOG.md`](../../CHANGELOG.md), every Wave 1 task lands a one-line bullet at PR-merge time under the appropriate `Added` / `Changed` / `Fixed` category per the Keep-a-Changelog convention. At v0.6.0 tag time the `[Unreleased]` block is renamed to `[0.6.0] — YYYY-MM-DD` per the existing release ritual; no separate "release notes" file is needed. Wave 2 tasks land their CHANGELOG bullets under a new `[Unreleased]` block opened post-v0.6.0. -- **Validation gate to ship Wave 1:** re-run `forgelm ingest` against the synthetic Turkish-formal-publication regression fixture at **both** `--chunk-size 4000` and `--chunk-size 1024`. Acceptance: `overall_quality_score ≥ 0.70` from `forgelm audit` on each resulting JSONL, *without* any post-processing or `--strip-pattern` flag. The dual-chunk-size acceptance is what catches the front-matter regression class the 2nd-pass Colab evidence exposed (where smaller chunks isolate chapter-opening orientation pages and the 21-character single-caption chunk). In addition, the playground notebook end-to-end run against the same fixture must surface the quality signal in its Cell 10 output (Task 9 acceptance). The audit's section 6 (Wave 1 list) covers the same scope at a higher granularity but with different bucketing — Phase 15 doc Tasks 2 + 3 jointly cover the audit's bullet 2 (script sanity + Turkish normalisation), and Tasks 6 + 7 + 8 split the audit's bullet 5 (format-specific extractor upgrades) per format. The 1-to-1 mapping is: Phase 15 Task 1 ↔ audit bullet 1; Tasks 2 + 3 ↔ audit bullet 2; Task 4 ↔ audit bullet 3; Task 5 ↔ audit bullet 4; Tasks 6 + 7 + 8 ↔ audit bullet 5; Task 9 ↔ audit bullet 6; Task 10 (regression fixtures) is sourced from audit section 7, not section 6. Failure on any acceptance criterion of tasks 1-10 blocks the v0.6.0 tag. - ---- - -### Cross-references - -- **Code surface affected:** [`forgelm/ingestion.py`](../../forgelm/ingestion.py), [`forgelm/cli/_parser.py`](../../forgelm/cli/_parser.py), [`forgelm/cli/subcommands/_ingest.py`](../../forgelm/cli/subcommands/_ingest.py), [`forgelm/data_audit/_quality.py`](../../forgelm/data_audit/_quality.py) -- **User-facing surfaces to update:** [`docs/guides/ingestion.md`](../guides/ingestion.md) + `-tr.md`, [`docs/guides/data_audit.md`](../guides/data_audit.md) + `-tr.md`, [`docs/reference/data_ingestion_architecture.md`](../reference/data_ingestion_architecture.md) + `-tr.md`, [`notebooks/ingestion_playground.ipynb`](../../notebooks/ingestion_playground.ipynb) -- **Standards consulted:** [`docs/standards/error-handling.md`](../standards/error-handling.md), [`docs/standards/logging-observability.md`](../standards/logging-observability.md), [`docs/standards/testing.md`](../standards/testing.md) diff --git a/docs/usermanuals/_meta.yaml b/docs/usermanuals/_meta.yaml index d7b09264..8a1b7e09 100644 --- a/docs/usermanuals/_meta.yaml +++ b/docs/usermanuals/_meta.yaml @@ -118,6 +118,14 @@ sections: title: { en: ORPO, tr: ORPO, de: ORPO, fr: ORPO, es: ORPO, zh: ORPO } - id: grpo title: { en: GRPO, tr: GRPO, de: GRPO, fr: GRPO, es: GRPO, zh: GRPO } + - id: pipelines + title: + en: Multi-Stage Pipelines + tr: Çok Aşamalı Pipeline'lar + de: Mehrstufige Pipelines + fr: Pipelines multi-étapes + es: Pipelines multi-etapa + zh: 多阶段流水线 - id: lora title: en: LoRA, QLoRA, DoRA diff --git a/docs/usermanuals/en/concepts/alignment-overview.md b/docs/usermanuals/en/concepts/alignment-overview.md index 844d8689..5827cd68 100644 --- a/docs/usermanuals/en/concepts/alignment-overview.md +++ b/docs/usermanuals/en/concepts/alignment-overview.md @@ -10,7 +10,7 @@ Modern LLM fine-tuning is no longer a single algorithm — it's a *stack* of pos ```mermaid flowchart TD Base[Base model
e.g. Qwen 2.5 / Llama 3] - SFT[SFT
format & content] + SFT["SFT
format & content"] DPO[DPO
preference alignment] SimPO[SimPO
preference, ref-free] KTO[KTO
binary feedback] diff --git a/docs/usermanuals/en/concepts/choosing-trainer.md b/docs/usermanuals/en/concepts/choosing-trainer.md index 5348adb5..16d8b971 100644 --- a/docs/usermanuals/en/concepts/choosing-trainer.md +++ b/docs/usermanuals/en/concepts/choosing-trainer.md @@ -81,6 +81,8 @@ These templates ship with `forgelm quickstart` and reflect what most teams use a | Math reasoning | `grpo-math` | GRPO with format-shaping reward | | Turkish medical Q&A | `medical-qa-tr` | SFT only | +For any "→"-shaped row above, chain the stages with a `pipeline:` config block instead of writing two YAMLs and orchestrating manually. See [Multi-Stage Pipelines](#/training/pipelines). + ## Anti-patterns :::warn @@ -99,3 +101,4 @@ These templates ship with `forgelm quickstart` and reflect what most teams use a - [The Alignment Stack](#/concepts/alignment-overview) — broader context of post-training paradigms. - [Dataset Formats](#/concepts/data-formats) — what JSONL each trainer expects. +- [Multi-Stage Pipelines](#/training/pipelines) — chain two or more trainers into one config-driven run. diff --git a/docs/usermanuals/en/getting-started/introduction.md b/docs/usermanuals/en/getting-started/introduction.md index 5b4c1cc8..1cfed53e 100644 --- a/docs/usermanuals/en/getting-started/introduction.md +++ b/docs/usermanuals/en/getting-started/introduction.md @@ -32,7 +32,7 @@ flowchart LR F -->|regression| G[Auto-revert] G --> E F -->|pass| H[Annex IV
artifacts] - H --> I[Export &
deploy] + H --> I["Export &
deploy"] classDef start fill:#1c2030,stroke:#f97316,color:#e6e7ec classDef ok fill:#161a24,stroke:#22c55e,color:#e6e7ec classDef warn fill:#161a24,stroke:#eab308,color:#e6e7ec diff --git a/docs/usermanuals/en/training/lora.md b/docs/usermanuals/en/training/lora.md index 5221afb8..50947c1d 100644 --- a/docs/usermanuals/en/training/lora.md +++ b/docs/usermanuals/en/training/lora.md @@ -13,9 +13,9 @@ ForgeLM applies LoRA / QLoRA / DoRA to *every* trainer (SFT, DPO, SimPO, KTO, OR ```mermaid flowchart TD - Q1{Need to fit on a
consumer GPU
(<24 GB)?} - Q2{Want full-precision
weights kept frozen?} - Q3{Need maximum
quality at LoRA cost?} + Q1{"Need to fit on a
consumer GPU
(<24 GB)?"} + Q2{"Want full-precision
weights kept frozen?"} + Q3{"Need maximum
quality at LoRA cost?"} Full[Full fine-tuning] LoRA([LoRA]) QLoRA([QLoRA]) diff --git a/docs/usermanuals/en/training/pipelines.md b/docs/usermanuals/en/training/pipelines.md new file mode 100644 index 00000000..a82f07cf --- /dev/null +++ b/docs/usermanuals/en/training/pipelines.md @@ -0,0 +1,149 @@ +--- +title: Multi-Stage Pipelines +description: Chain SFT → DPO → GRPO and other trainers into one config-driven run with auto-chaining, per-stage gates, and chain-level Annex IV manifest. +--- + +# Multi-Stage Pipelines + +A *pipeline* chains 2 or more training stages into a single `forgelm` run. Each stage is a single-stage trainer from the trainer's point of view; only the orchestrator knows there's an outer loop. The output of stage N auto-chains as the input of stage N+1, every stage's compliance artefacts roll up into a chain-level manifest, and the operator only needs one CLI invocation to drive the whole flow. + +The canonical chain is **SFT → DPO → GRPO**, but any sequence of supported trainers works (SFT, DPO, SimPO, KTO, ORPO, GRPO). + +## When to use a pipeline + +```mermaid +flowchart TD + Q1{"Running only one
training stage?"} + Q2{"Are stages sequential
(each consumes the
previous output)?"} + Q3{"Want one Annex IV
manifest for the
full chain?"} + Single[Single-stage YAML
v0.6.0 default] + Multi[Hand-orchestrated
shell scripts] + Pipe([Use `pipeline:` block]) + + Q1 -->|Yes| Single + Q1 -->|No| Q2 + Q2 -->|Yes| Q3 + Q2 -->|No, parallel branches| Multi + Q3 -->|Yes| Pipe + Q3 -->|No, just runs| Multi + + classDef question fill:#161a24,stroke:#0ea5e9,color:#e6e7ec + classDef result fill:#1c2030,stroke:#22c55e,color:#e6e7ec + classDef alt fill:#1c2030,stroke:#9ea3b3,color:#e6e7ec + class Q1,Q2,Q3 question + class Pipe result + class Single,Multi alt +``` + +Use a `pipeline:` block when **all** of these apply: + +- 2 or more sequential training stages (e.g. SFT then DPO). +- Each stage's input model is the previous stage's output. +- You want one Annex IV manifest covering the whole chain. + +Don't use a `pipeline:` block if: + +- Only one training paradigm is running. Single-stage configs (the v0.6.0 default) remain the canonical path and preserve their pre-Phase-14 behaviour byte-for-byte. +- The stage dependencies aren't linear (DAG-shaped pipelines). Phase 14 ships sequential pipelines only; the schema reserves headroom for a future DAG expansion. +- Parallel stage execution (concurrent independent branches) is needed. Same horizon as DAG support — Wave 2 or later. + +## Minimal pipeline config + +```yaml +# Root — defaults inherited by stages that don't override them. +model: + name_or_path: "meta-llama/Llama-3-8B" +lora: + r: 8 + alpha: 16 +training: + trainer_type: "sft" + output_dir: "./placeholder" +data: + dataset_name_or_path: "./placeholder.jsonl" + +# The chain itself. +pipeline: + output_dir: "./pipeline_run" + stages: + - name: sft_stage + training: + trainer_type: "sft" + output_dir: "./pipeline_run/stage1_sft" + data: + dataset_name_or_path: "./data/sft.jsonl" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./pipeline_run/stage2_dpo" + dpo_beta: 0.1 + data: + dataset_name_or_path: "./data/preferences.jsonl" + + - name: grpo_stage + training: + trainer_type: "grpo" + output_dir: "./pipeline_run/stage3_grpo" + data: + dataset_name_or_path: "./data/math_prompts.jsonl" +``` + +Each stage's `model.name_or_path` is auto-set to the previous stage's `training.output_dir/final_model` — no hand-editing between stages. + +## CLI surface + +```bash +# Run the whole chain. +forgelm --config pipeline.yaml + +# Validate every stage without GPU allocation — pytest --collectonly style. +forgelm --config pipeline.yaml --dry-run + +# Re-run a single named stage in isolation (audit / re-run scenarios). +forgelm --config pipeline.yaml --stage dpo_stage + +# Resume from a named stage after a crash; already-completed stages skip. +forgelm --config pipeline.yaml --resume-from dpo_stage + +# Operator escape hatch: override the auto-chained input model. +forgelm --config pipeline.yaml --stage dpo_stage --input-model ./other/checkpoint + +# Verify the chain-level Annex IV manifest of a finished run. +forgelm verify-annex-iv --pipeline ./pipeline_run +``` + +## What the orchestrator gives you + +- **Auto-chaining**: stage N's `model.name_or_path` is automatically set to stage N-1's `training.output_dir/final_model`. +- **Crash-safe state**: every transition writes `pipeline_state.json` atomically (tmp + rename). `--resume-from` picks up where the chain stopped. +- **Per-stage gates**: auto-revert (eval regression), human approval, and safety eval compose per stage. A failed gate stops the chain; downstream stages skip with `skipped_due_to_prior_revert`. +- **Audit events**: 7 new event names cover every transition — `pipeline.started`, `pipeline.stage_started`, `pipeline.stage_completed`, `pipeline.stage_gated`, `pipeline.stage_reverted`, `pipeline.force_resume`, `pipeline.completed`. Every entry carries the same top-level `run_id` so SIEM-style grouping works on a single field. +- **Annex IV at chain level**: `compliance/pipeline_manifest.json` indexes every per-stage `training_manifest.json` into one verifiable artefact, hashed and timestamped. +- **Webhook integration**: `notify_pipeline_started`, `notify_pipeline_completed`, `notify_pipeline_reverted` fire alongside the existing per-stage `training.*` notifications — pre-existing Slack/Teams dashboards keep working unchanged. + +## Inheritance matrix at a glance + +| Block | Stage may override? | If stage omits it | +|---|---|---| +| `model` | Yes (wholesale) | Auto-chained from prev stage; root for stage 0 | +| `lora` | Yes (wholesale) | Inherited from root | +| `training` | **`trainer_type` required per stage** | Inherited from root | +| `data` | Yes (recommended per stage) | Inherited from root | +| `evaluation` | Yes (wholesale) | Inherited from root | +| `distributed`, `webhook`, `compliance`, `risk_assessment`, `monitoring`, `retention`, `synthetic`, `merge`, `auth` | **Root-only** — rejected per-stage | Inherited from root | + +Section-wholesale semantics: if a stage names a block it *replaces* root's entirely (no deep-merge). The "trainer_type required per stage" rule is an audit-clarity validator; it ensures every stage records which paradigm it ran. + +## Limitations (Phase 14 Wave 1) + +- No intra-stage checkpoint resume — `--resume-from` works at stage boundaries only. +- Sequential only; no DAG / parallel stage execution. +- No multi-process locking on `pipeline.output_dir` — pick distinct dirs per concurrent run. +- Wizard (CLI + web) does **not** emit `pipeline:` blocks yet; hand-author the YAML or copy the template from `config_template.yaml`. Wizard pipeline support is a future-phase feature. + +## See also + +- [Multi-Stage Pipelines — operator guide](../../../guides/pipeline.md) — full walkthrough with the 6 design fixtures. +- [Choosing a Trainer](../concepts/choosing-trainer.md) — when to pick which trainer; pipelines come after the per-trainer choice. +- [Annex IV Compliance](../compliance/annex-iv.md) — the per-stage manifest format the chain-level manifest indexes. diff --git a/docs/usermanuals/tr/concepts/alignment-overview.md b/docs/usermanuals/tr/concepts/alignment-overview.md index 3535f10a..7cdddbd7 100644 --- a/docs/usermanuals/tr/concepts/alignment-overview.md +++ b/docs/usermanuals/tr/concepts/alignment-overview.md @@ -10,7 +10,7 @@ Modern LLM fine-tuning'i artık tek bir algoritma değil — her biri farklı bi ```mermaid flowchart TD Base[Temel model
ör. Qwen 2.5 / Llama 3] - SFT[SFT
format & içerik] + SFT["SFT
format & içerik"] DPO[DPO
tercih hizalaması] SimPO[SimPO
tercih, ref-free] KTO[KTO
ikili geri bildirim] diff --git a/docs/usermanuals/tr/concepts/choosing-trainer.md b/docs/usermanuals/tr/concepts/choosing-trainer.md index 689813e7..e56f9a86 100644 --- a/docs/usermanuals/tr/concepts/choosing-trainer.md +++ b/docs/usermanuals/tr/concepts/choosing-trainer.md @@ -81,6 +81,8 @@ QLoRA bu sayıları ~3-4× düşürür. Full precision'da 22 GB isteyen 7B DPO k | Matematik akıl yürütme | `grpo-math` | format-shaping reward'lı GRPO | | Türkçe medikal Q&A | `medical-qa-tr` | Sadece SFT | +Yukarıdaki "→" işaretli satırlar için, iki ayrı YAML yazıp elle orkestre etmek yerine aşamaları bir `pipeline:` config bloğuyla zincirleyin. Bkz. [Çok Aşamalı Pipeline'lar](#/training/pipelines). + ## Anti-pattern'ler :::warn @@ -99,3 +101,4 @@ QLoRA bu sayıları ~3-4× düşürür. Full precision'da 22 GB isteyen 7B DPO k - [Alignment Yığını](#/concepts/alignment-overview) — daha geniş bağlam. - [Dataset Formatları](#/concepts/data-formats) — her trainer'ın beklediği JSONL. +- [Çok Aşamalı Pipeline'lar](#/training/pipelines) — iki veya daha fazla trainer'ı tek config-tabanlı koşuda zincirle. diff --git a/docs/usermanuals/tr/training/lora.md b/docs/usermanuals/tr/training/lora.md index 64d7d984..3c818153 100644 --- a/docs/usermanuals/tr/training/lora.md +++ b/docs/usermanuals/tr/training/lora.md @@ -13,9 +13,9 @@ ForgeLM LoRA / QLoRA / DoRA'yı *her* trainer'a uygular (SFT, DPO, SimPO, KTO, O ```mermaid flowchart TD - Q1{Consumer GPU'da
(<24 GB) sığmalı mı?} - Q2{Full-precision
ağırlıklar dondurulsun mu?} - Q3{LoRA maliyetinde
maksimum kalite mi?} + Q1{"Consumer GPU'da
(<24 GB) sığmalı mı?"} + Q2{"Full-precision
ağırlıklar dondurulsun mu?"} + Q3{"LoRA maliyetinde
maksimum kalite mi?"} Full[Full fine-tuning] LoRA([LoRA]) QLoRA([QLoRA]) diff --git a/docs/usermanuals/tr/training/pipelines.md b/docs/usermanuals/tr/training/pipelines.md new file mode 100644 index 00000000..fb1296bf --- /dev/null +++ b/docs/usermanuals/tr/training/pipelines.md @@ -0,0 +1,149 @@ +--- +title: Çok Aşamalı Pipeline'lar +description: SFT → DPO → GRPO ve diğer trainer'ları tek bir config-tabanlı koşuda zincirleyin — otomatik zincirleme, aşama bazında kapılar ve zincir-seviyesi Annex IV manifest ile. +--- + +# Çok Aşamalı Pipeline'lar + +Bir *pipeline*, 2 veya daha fazla eğitim aşamasını tek bir `forgelm` koşusunda zincirler. Her aşama trainer'ın bakış açısından tek aşamalı bir trainer'dır; sadece orkestratör dış döngünün varlığından haberdardır. Aşama N'in çıktısı otomatik olarak aşama N+1'in girdisi olur, her aşamanın uyumluluk artefactları zincir-seviyesi bir manifeste toplanır ve operatörün tüm akışı sürmek için tek bir CLI çağrısı yeterlidir. + +Kanonik zincir **SFT → DPO → GRPO**'dur, ancak desteklenen herhangi bir trainer dizisi (SFT, DPO, SimPO, KTO, ORPO, GRPO) çalışır. + +## Pipeline ne zaman kullanılır + +```mermaid +flowchart TD + Q1{"Sadece tek eğitim
aşaması mı koşuluyor?"} + Q2{"Aşamalar sıralı mı
(her biri öncekinin
çıktısını mı tüketiyor)?"} + Q3{"Tüm zincir için tek
Annex IV manifest
isteniyor mu?"} + Single[Tek aşamalı YAML
v0.6.0 default] + Multi[Elle orkestre
shell script'leri] + Pipe([`pipeline:` bloğunu kullan]) + + Q1 -->|Evet| Single + Q1 -->|Hayır| Q2 + Q2 -->|Evet| Q3 + Q2 -->|Hayır, paralel branch'ler| Multi + Q3 -->|Evet| Pipe + Q3 -->|Hayır, sadece koşular| Multi + + classDef question fill:#161a24,stroke:#0ea5e9,color:#e6e7ec + classDef result fill:#1c2030,stroke:#22c55e,color:#e6e7ec + classDef alt fill:#1c2030,stroke:#9ea3b3,color:#e6e7ec + class Q1,Q2,Q3 question + class Pipe result + class Single,Multi alt +``` + +`pipeline:` bloğunu **tüm** aşağıdaki koşullar geçerliyse kullanın: + +- Sıralı olarak 2 veya daha fazla eğitim aşaması (örn. SFT sonrası DPO). +- Her aşamanın giriş modeli, önceki aşamanın çıkış modelidir. +- Tüm zinciri kapsayan tek bir Annex IV manifest istiyorsunuz. + +`pipeline:` bloğunu kullanmayın eğer: + +- Sadece tek bir eğitim paradigması koşturuyorsanız. Tek aşamalı config'ler (v0.6.0 default'u) kanonik yol olarak kalır ve Phase-14 öncesi davranışlarını byte-byte korur. +- Aşama bağımlılıkları doğrusal değilse (DAG-şekilli pipeline'lar). Phase 14 sadece sıralı pipeline'lar gönderir; şema ileriki bir DAG genişlemesi için yer ayırıyor. +- Paralel aşama yürütmesi (eş zamanlı bağımsız dallar) gerekiyorsa. DAG desteğiyle aynı zaman ufkunda — Wave 2 veya sonrası. + +## Minimal pipeline config + +```yaml +# Root — aşmadığı sürece aşamaların miras aldığı varsayılanlar. +model: + name_or_path: "meta-llama/Llama-3-8B" +lora: + r: 8 + alpha: 16 +training: + trainer_type: "sft" + output_dir: "./placeholder" +data: + dataset_name_or_path: "./placeholder.jsonl" + +# Zincir. +pipeline: + output_dir: "./pipeline_run" + stages: + - name: sft_stage + training: + trainer_type: "sft" + output_dir: "./pipeline_run/stage1_sft" + data: + dataset_name_or_path: "./data/sft.jsonl" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./pipeline_run/stage2_dpo" + dpo_beta: 0.1 + data: + dataset_name_or_path: "./data/preferences.jsonl" + + - name: grpo_stage + training: + trainer_type: "grpo" + output_dir: "./pipeline_run/stage3_grpo" + data: + dataset_name_or_path: "./data/math_prompts.jsonl" +``` + +Her aşamanın `model.name_or_path`'ı önceki aşamanın `training.output_dir/final_model`'ına otomatik ayarlanır — aşamalar arasında elle düzenleme gerekmez. + +## CLI yüzeyi + +```bash +# Tüm zinciri koş. +forgelm --config pipeline.yaml + +# Her aşamayı GPU tahsisi yapmadan doğrula — pytest --collectonly tarzı. +forgelm --config pipeline.yaml --dry-run + +# Tek bir adlı aşamayı yalıtılmış olarak yeniden koştur (audit / re-run senaryoları). +forgelm --config pipeline.yaml --stage dpo_stage + +# Bir crash sonrası adlı aşamadan itibaren devam et; tamamlanmış aşamalar atlanır. +forgelm --config pipeline.yaml --resume-from dpo_stage + +# Operatör kaçış kapısı: otomatik zincirlenen giriş modelini override et. +forgelm --config pipeline.yaml --stage dpo_stage --input-model ./other/checkpoint + +# Biten bir koşunun zincir-seviyesi Annex IV manifesti doğrula. +forgelm verify-annex-iv --pipeline ./pipeline_run +``` + +## Orkestratörün size sağladıkları + +- **Otomatik zincirleme**: aşama N'in `model.name_or_path`'ı otomatik olarak aşama N-1'in `training.output_dir/final_model`'ına ayarlanır. +- **Crash-safe state**: her geçiş `pipeline_state.json`'ı atomik olarak yazar (tmp + rename). `--resume-from`, zincirin durduğu yerden devam alır. +- **Aşama bazında kapılar**: auto-revert (eval regresyonu), insan onayı ve güvenlik değerlendirmesi aşama başına kompoze olur. Başarısız bir kapı zinciri durdurur; alt aşamalar `skipped_due_to_prior_revert` ile atlanır. +- **Audit olayları**: 7 yeni olay adı her geçişi kapsar — `pipeline.started`, `pipeline.stage_started`, `pipeline.stage_completed`, `pipeline.stage_gated`, `pipeline.stage_reverted`, `pipeline.force_resume`, `pipeline.completed`. Her girdi aynı top-level `run_id`'yi taşır, böylece SIEM-tarzı gruplama tek alan üzerinde çalışır. +- **Zincir-seviyesi Annex IV**: `compliance/pipeline_manifest.json` her aşama bazındaki `training_manifest.json`'ı tek doğrulanabilir artefactta indeksler, hash'lenmiş ve zaman damgalı. +- **Webhook entegrasyonu**: `notify_pipeline_started`, `notify_pipeline_completed`, `notify_pipeline_reverted` mevcut aşama bazındaki `training.*` bildirimleriyle yan yana ateşlenir — mevcut Slack/Teams dashboard'ları değişmeden çalışmaya devam eder. + +## Miras matrisi tek bakışta + +| Blok | Aşama override edebilir mi? | Aşama atlarsa | +|---|---|---| +| `model` | Evet (toptan) | Önceki aşamadan otomatik zincirlenir; aşama 0 için root | +| `lora` | Evet (toptan) | Root'tan miras alınır | +| `training` | **`trainer_type` her aşamada zorunlu** | Root'tan miras alınır | +| `data` | Evet (aşama başına önerilir) | Root'tan miras alınır | +| `evaluation` | Evet (toptan) | Root'tan miras alınır | +| `distributed`, `webhook`, `compliance`, `risk_assessment`, `monitoring`, `retention`, `synthetic`, `merge`, `auth` | **Sadece root** — aşamada reddedilir | Root'tan miras alınır | + +Section-wholesale semantiği: bir aşama bir bloğu adıyla anarsa root'unkini *tamamen* değiştirir (deep-merge yok). "Aşama başına trainer_type zorunlu" kuralı bir audit-clarity validator'ıdır; her aşamanın hangi paradigmayı koştuğunu kaydetmesini garanti eder. + +## Sınırlamalar (Phase 14 Wave 1) + +- Aşama içi checkpoint resume yok — `--resume-from` sadece aşama sınırlarında çalışır. +- Sadece sıralı; DAG / paralel aşama yürütmesi yok. +- `pipeline.output_dir` üzerinde çoklu-süreç kilidi yok — eş zamanlı koşular için ayrı dizinler seçin. +- Wizard (CLI + web) henüz `pipeline:` blokları üretmiyor; YAML'ı elle yazın veya `config_template.yaml`'dan şablonu kopyalayın. Wizard pipeline desteği gelecek-faz özelliği. + +## Ayrıca bakınız + +- [Çok Aşamalı Pipeline'lar — operatör kılavuzu](../../../guides/pipeline-tr.md) — 6 tasarım fixture'ı ile tam walkthrough. +- [Trainer Seçimi](../concepts/choosing-trainer.md) — hangi trainer ne zaman seçilir; pipeline'lar trainer seçiminden sonra gelir. +- [Annex IV Uyumluluk](../compliance/annex-iv.md) — zincir-seviyesi manifestin indekslediği aşama-bazında manifest formatı. diff --git a/forgelm/cli/_dispatch.py b/forgelm/cli/_dispatch.py index da2ecbec..a10e0a22 100644 --- a/forgelm/cli/_dispatch.py +++ b/forgelm/cli/_dispatch.py @@ -163,6 +163,99 @@ def main(): sys.exit(EXIT_TRAINING_ERROR) +def _dispatch_legacy_data_audit(args) -> None: + """Handle ``forgelm --data-audit PATH`` (deprecated flag). + + Pulled out of ``_main_inner`` for Sonar python:S3776 cognitive- + complexity hygiene. Emits the standard deprecation + ``DeprecationWarning`` + an audit-log breadcrumb (best-effort), then + delegates to the canonical ``_run_data_audit`` and ``sys.exit``s. + Calls back to ``_main_inner`` are impossible — this function always + terminates the process. + """ + json_output = args.output_format == "json" + log_level = "WARNING" if args.quiet else args.log_level + _setup_logging(log_level, json_format=json_output) + # Phase 13 (Faz 13): emit a structured Python ``DeprecationWarning`` + # so `pytest -W error::DeprecationWarning` and `python -Wd` tooling + # surface it, plus an append-only audit-log event so operators who + # only read the JSONL trail see the migration signal too. Cadence + # for the v0.8.0 removal follows the "Deprecation cadence" section + # in ``docs/standards/release.md`` (one-minor warning window + # minimum). Originally targeted v0.7.0; pushed one minor out to + # v0.8.0 at the v0.7.0 cut to preserve the one-minor warning + # window for operators who only upgrade once per minor release. + warnings.warn( + "`forgelm --data-audit PATH` is deprecated and will be removed " + "in v0.8.0. Use the `forgelm audit PATH` subcommand instead — " + "same behaviour, same output. " + "See docs/standards/release.md#deprecation-cadence for the removal timeline.", + DeprecationWarning, + stacklevel=2, + ) + # Audit-log event: append-only Article 12 record of the legacy + # invocation so compliance reviewers can prove the operator was + # warned. Best-effort — a read-only output dir must not abort the + # actual audit run. + legacy_target = args.output or "./audit" + try: + from ..compliance import AuditLogger + from ..config import ConfigError + + AuditLogger(legacy_target).log_event( + "cli.legacy_flag_invoked", + flag="--data-audit", + replacement="forgelm audit", + version="v0.8.0 removal", + ) + except (OSError, ConfigError) as audit_exc: + logger.warning( + "Failed to emit deprecation audit event to %s: %s", + legacy_target, + audit_exc, + ) + # Late import via the package facade so monkeypatched + # ``forgelm.cli._run_data_audit`` references resolve correctly. + from forgelm import cli as _cli_facade + + _cli_facade._run_data_audit( + args.data_audit, + args.output, + args.output_format, + ) + sys.exit(EXIT_SUCCESS) + + +def _dispatch_pipeline_mode(config, args) -> None: + """Handle a config that carries a ``pipeline:`` block. + + Pulled out of ``_main_inner`` for Sonar python:S3776 cognitive- + complexity hygiene. Re-reads the YAML *bytes* (not the parsed + semantic content — regulators audit the on-disk artefact) and + routes to :func:`forgelm.cli._pipeline.run_pipeline_from_args`. + Always terminates the process via ``sys.exit``. + + Layering rule (Phase 14 F-B-1): this branch MUST run before + ``_maybe_run_no_train_mode``. The single-stage no-train modes + (``--dry-run``, ``--fit-check``, ``--benchmark-only``, + ``--merge``, ``--generate-data``, ``--compliance-export``) + ``sys.exit`` internally; if they ran first on a pipeline config, + the orchestrator would never reach the multi-stage validation path + and the documented ``--dry-run`` per-stage error-collection + contract would be silently violated. + """ + from ._pipeline import run_pipeline_from_args + + try: + with open(args.config, "rb") as f: + pipeline_yaml_bytes = f.read() + except OSError as e: + logger.error("Failed to re-read pipeline YAML for hashing: %s", e) + sys.exit(EXIT_CONFIG_ERROR) + + sys.exit(run_pipeline_from_args(config, pipeline_yaml_bytes, args)) + + def _main_inner() -> None: """Entry-point body — extracted so ``main`` can wrap a single ``KeyboardInterrupt`` handler around every step from argparse to @@ -181,70 +274,13 @@ def _main_inner() -> None: # Run before the config-required check so operators can audit raw data # without writing a YAML. Phase 11.5 promoted the same code path to # `forgelm audit PATH` (a real subcommand); the legacy flag is preserved - # as an alias and slated for removal in v0.7.0 — Phase 13 wires up the - # deprecation signalling at this dispatch site. + # as an alias and slated for removal in v0.8.0. (Removal originally + # targeted v0.7.0; pushed one minor out at the v0.7.0 cut to preserve + # the documented one-minor warning window — see the matching + # DeprecationWarning message + audit-event ``version`` field below + # and ``docs/standards/release.md#deprecation-cadence``.) if getattr(args, "data_audit", None): - json_output = args.output_format == "json" - log_level = "WARNING" if args.quiet else args.log_level - _setup_logging(log_level, json_format=json_output) - # Phase 13 (Faz 13): emit a structured Python ``DeprecationWarning`` - # so `pytest -W error::DeprecationWarning` and `python -Wd` tooling - # surface it, plus an append-only audit-log event so operators who - # only read the JSONL trail see the migration signal too. Cadence - # for the v0.7.0 removal follows the "Deprecation cadence" section - # in ``docs/standards/release.md`` (one-minor warning window - # minimum). - warnings.warn( - "`forgelm --data-audit PATH` is deprecated and will be removed " - "in v0.7.0. Use the `forgelm audit PATH` subcommand instead — " - "same behaviour, same output. " - "See docs/standards/release.md#deprecation-cadence for the removal timeline.", - DeprecationWarning, - stacklevel=2, - ) - # Audit-log event: append-only Article 12 record of the legacy - # invocation so compliance reviewers can prove the operator was - # warned. The logger writes to ``/audit_log.jsonl`` next to - # the report this run is about to produce. Resolve the target the - # same way ``_run_data_audit`` does (default ``./audit``) so the - # event lands in the same directory as the report. - legacy_target = args.output or "./audit" - try: - from ..compliance import AuditLogger - from ..config import ConfigError - - AuditLogger(legacy_target).log_event( - "cli.legacy_flag_invoked", - flag="--data-audit", - replacement="forgelm audit", - version="v0.7.0 removal", - ) - except (OSError, ConfigError) as audit_exc: - # Non-fatal: the audit log is a best-effort telemetry record - # for the deprecation notice. The DeprecationWarning has - # already fired; the audit run itself must still proceed — - # losing the legacy-flag breadcrumb is preferable to aborting - # a working pipeline because the output dir is read-only. - # Logged at WARNING (not debug) per - # docs/standards/error-handling.md "Best-effort artefact - # carve-out" hygiene item 2: audit-trail visibility trumps - # log noise for low-frequency, structured events like a - # legacy-flag invocation against a read-only output dir. - logger.warning( - "Failed to emit deprecation audit event to %s: %s", - legacy_target, - audit_exc, - ) - # Late import via the package facade so monkeypatched - # ``forgelm.cli._run_data_audit`` references resolve correctly. - from forgelm import cli as _cli_facade - - _cli_facade._run_data_audit( - args.data_audit, - args.output, - args.output_format, - ) - sys.exit(EXIT_SUCCESS) + _dispatch_legacy_data_audit(args) _maybe_run_wizard(args) @@ -261,5 +297,31 @@ def _main_inner() -> None: config = _load_config_or_exit(args.config, json_output) _apply_offline_flag(config, args.offline) + + if config.pipeline is not None: + _dispatch_pipeline_mode(config, args) + + # Phase 14 post-release review: pipeline-only flags must not + # silently route through the single-stage training path. Pre-fix, + # ``forgelm --config single_stage.yaml --stage dpo_stage`` would + # ignore ``--stage`` and run the root YAML's trainer — surprising + # the operator who expected the flag to be load-bearing. + _PIPELINE_ONLY_FLAGS = ( + ("stage", "--stage"), + ("resume_from", "--resume-from"), + ("force_resume", "--force-resume"), + ("input_model", "--input-model"), + ) + for attr, flag_name in _PIPELINE_ONLY_FLAGS: + if getattr(args, attr, None): + logger.error( + "`%s` requires a config with a `pipeline:` block — this is a " + "multi-stage orchestrator flag. Either add `pipeline:` to " + "the YAML or remove the flag.", + flag_name, + ) + sys.exit(EXIT_CONFIG_ERROR) + + # Single-stage path — unchanged from v0.6.0. _maybe_run_no_train_mode(config, args) _run_training_pipeline(config, args, json_output) diff --git a/forgelm/cli/_parser.py b/forgelm/cli/_parser.py index 2e916d56..d0d485d5 100644 --- a/forgelm/cli/_parser.py +++ b/forgelm/cli/_parser.py @@ -754,7 +754,25 @@ def _add_verify_annex_iv_subcommand(subparsers) -> None: "Exits 0 on valid; 1 on missing field or hash mismatch." ), ) - p.add_argument("path", help="Path to the Annex IV JSON artifact (typically `compliance/annex_iv_.json`).") + p.add_argument( + "path", + help=( + "Path to the Annex IV JSON artifact (typically " + "``compliance/annex_iv_.json``). When ``--pipeline`` is set, " + "this is interpreted as a pipeline run directory containing " + "``compliance/pipeline_manifest.json`` instead." + ), + ) + p.add_argument( + "--pipeline", + action="store_true", + help=( + "Phase 14: interpret ``path`` as a pipeline run directory and " + "validate the chain-level ``pipeline_manifest.json`` (chain " + "integrity, stage-index ordering, ``stopped_at`` coherence, " + "per-stage training_manifest existence)." + ), + ) _add_common_subparser_flags(p, include_output_format=True) @@ -1230,6 +1248,54 @@ def parse_args(): parser.add_argument( "--merge", action="store_true", help="Run model merging from the merge section of your config. No training." ) + # Phase 14 — multi-stage pipeline chains. Active only when the + # ``pipeline:`` block is present in the config; documented at + # docs/guides/pipeline.md. + parser.add_argument( + "--stage", + type=str, + default=None, + metavar="NAME", + help=( + "Multi-stage pipelines only: run a single named stage in isolation " + "(audit / re-run scenarios). Non-first stages require the previous " + "stage's on-disk output, or an explicit --input-model override." + ), + ) + parser.add_argument( + "--resume-from", + type=str, + default=None, + metavar="NAME", + help=( + "Multi-stage pipelines only: resume an interrupted run from a named " + "stage onward. Already-completed stages whose output_model paths " + "exist on disk are skipped; refuses to run if the on-disk " + "pipeline_config_hash differs from the current YAML (unless " + "--force-resume is also set)." + ), + ) + parser.add_argument( + "--force-resume", + action="store_true", + help=( + "Multi-stage pipelines only: bypass the --resume-from stale-config " + "guard. The override is logged at WARNING and recorded in the " + "audit event." + ), + ) + parser.add_argument( + "--input-model", + type=str, + default=None, + metavar="PATH", + help=( + "Multi-stage pipelines only: when used with --stage, replaces the " + "auto-chained input model with this path. The audit-log entry " + "records ``input_source: cli_override`` so reviewers see the chain " + "was broken intentionally." + ), + ) parser.add_argument( "--output-format", type=str, diff --git a/forgelm/cli/_pipeline.py b/forgelm/cli/_pipeline.py new file mode 100644 index 00000000..a94ceda9 --- /dev/null +++ b/forgelm/cli/_pipeline.py @@ -0,0 +1,1462 @@ +"""Phase 14 — Multi-stage training pipeline orchestrator. + +Top-level entry point for ``forgelm --config pipeline.yaml`` runs. The +orchestrator iterates over the ordered :class:`PipelineConfig.stages`, +materialises a flat :class:`ForgeConfig` per stage via +:func:`forgelm.config.merge_pipeline_stage_config`, hands it to a fresh +:class:`forgelm.trainer.ForgeTrainer`, and threads the state through a +crash-safe JSON state file + an append-only audit-log event stream. + +Layering rule — see ``docs/roadmap/phase-14-pipeline-chains.md`` Task 2: +``forgelm/trainer.py`` is **not** aware of the pipeline layer. Each +stage is a single-stage run from the trainer's point of view; only this +module knows there is an outer loop. When a config carries no +``pipeline:`` block, this module is never imported. + +Key responsibilities: + +- **Auto-chain** each stage's ``model.name_or_path`` to the previous + stage's output path (or honour an explicit stage override / CLI + ``--input-model`` override). +- **Per-stage gates** — auto-revert and human-approval are composed + one stage at a time; a failed gate stops the chain and marks + downstream stages ``skipped_due_to_prior_revert``. +- **Crash-safe state** — every stage transition is atomically written to + ``/pipeline_state.json`` (tmp file + ``os.replace``). + ``--resume-from`` reads that file and skips ``completed`` stages whose + outputs still exist on disk. +- **Annex-IV manifest** — every transition also rewrites + ``/compliance/pipeline_manifest.json`` (delegated to + :mod:`forgelm.compliance`) so reviewers can correlate every chain + failure to a frozen artefact. +- **Audit + webhook** — emits ``pipeline.started`` / ``.stage_started`` + / ``.stage_completed`` / ``.stage_reverted`` / ``.completed`` audit + events alongside the existing per-stage ``training.*`` vocabulary; the + webhook notifier gains ``notify_pipeline_*`` methods that the orchestrator + calls at the same transitions. +- **Exit-code routing** — the pipeline aggregates per-stage outcomes + via ``worst_exit = max(worst_exit, code)``, which gives the + precedence order + ``EXIT_AWAITING_APPROVAL (4)`` > ``EXIT_EVAL_FAILURE (3)`` > + ``EXIT_TRAINING_ERROR (2)`` > ``EXIT_CONFIG_ERROR (1)`` > + ``EXIT_SUCCESS (0)``. In practice an approval gate causes an + immediate ``break`` (no later stages run) and any other failure + sets ``chain_broken = True`` (downstream stages skip), so the + precedence rule is reached only when a single stage produces a + single non-zero code. Numeric ordering of the exit-code constants + matches the documented severity ramp — operators reading the exit + code in a shell only see the "worst" outcome from a single stage + that ran. +""" + +from __future__ import annotations + +import hashlib +import json +import logging +import os +import secrets +from dataclasses import asdict, dataclass, field +from datetime import datetime, timezone +from typing import Any, Dict, List, Literal, Optional + +from ..config import ForgeConfig, PipelineConfig, PipelineStage, merge_pipeline_stage_config +from ._exit_codes import ( + EXIT_AWAITING_APPROVAL, + EXIT_CONFIG_ERROR, + EXIT_EVAL_FAILURE, + EXIT_SUCCESS, + EXIT_TRAINING_ERROR, +) + +logger = logging.getLogger("forgelm.pipeline") + + +# --------------------------------------------------------------------------- +# State enums + dataclasses +# --------------------------------------------------------------------------- + + +StageStatusLiteral = Literal[ + "pending", + "running", + "completed", + "failed", + "gated_pending_approval", + "skipped_due_to_prior_revert", + "skipped_by_filter", +] +"""Per-stage status values. See Phase 14 Task 3 for the meaning of each.""" + + +InputSourceLiteral = Literal["root", "chain", "stage_explicit", "cli_override"] +"""How the orchestrator resolved a stage's input model path. Recorded on +the per-stage state so reviewers can trace any chain interruption +(``cli_override`` is the operator escape hatch that intentionally breaks +the chain).""" + + +FinalStatusLiteral = Literal[ + "in_progress", + "completed", + "stopped_at_stage", + "gated_pending_approval", +] + + +@dataclass +class PipelineStageState: + """Per-stage state snapshot persisted to disk + included in the manifest.""" + + name: str + index: int + trainer_type: str + status: StageStatusLiteral = "pending" + input_model: Optional[str] = None + input_source: Optional[InputSourceLiteral] = None + output_model: Optional[str] = None + started_at: Optional[str] = None + finished_at: Optional[str] = None + duration_seconds: Optional[float] = None + training_manifest: Optional[str] = None + metrics: Dict[str, float] = field(default_factory=dict) + gate_decision: Optional[str] = None + auto_revert_triggered: bool = False + skipped_reason: Optional[str] = None + exit_code: Optional[int] = None + error: Optional[str] = None + + +@dataclass +class PipelineState: + """Top-level pipeline-run state snapshot. + + Round-tripped to ``pipeline_state.json`` after every stage transition. + Compatible with :mod:`json` (atomic-write via tmp + ``os.replace``); + no nested non-serialisable types. + """ + + pipeline_run_id: str + pipeline_config_hash: str + forgelm_version: str + started_at: str + finished_at: Optional[str] = None + final_status: FinalStatusLiteral = "in_progress" + stopped_at: Optional[str] = None + stages: List[PipelineStageState] = field(default_factory=list) + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _utc_iso_now() -> str: + """UTC ISO-8601 timestamp with seconds precision. + + Mirrors the format ``forgelm/compliance.py`` already uses for audit + log entries so timestamps line up across the two surfaces. + """ + return datetime.now(timezone.utc).isoformat(timespec="seconds") + + +def _generate_run_id() -> str: + """Return a pipeline run id of shape ``pl_YYYY-MM-DD_<6-hex>``. + + The date prefix lets operators eyeball runs by week without consulting + the manifest; the 6-hex suffix gives ~16 million combinations per day, + which is more than enough to disambiguate same-day re-runs even in + aggressive CI loops. + """ + today = datetime.now(timezone.utc).strftime("%Y-%m-%d") + return f"pl_{today}_{secrets.token_hex(3)}" + + +def _compute_pipeline_config_hash(pipeline_yaml_bytes: bytes) -> str: + """SHA-256 of the *raw bytes* of the pipeline YAML file. + + Hashing the bytes (not the parsed dict) is intentional: it locks the + on-disk artefact, which is what regulators audit. YAML + comments / key ordering / whitespace are all in scope; an operator + who edits the file between runs gets a different hash even if the + semantic content is unchanged, and the ``--resume-from`` stale-state + guard catches it. + """ + return "sha256:" + hashlib.sha256(pipeline_yaml_bytes).hexdigest() + + +def _pipeline_paths(root_cfg: ForgeConfig) -> Dict[str, str]: + """Resolve the canonical filesystem paths for the pipeline run. + + All pipeline-level artefacts (state file, manifest, audit log) live + under :attr:`PipelineConfig.output_dir` (defaults to + ``./pipeline_run``). Per-stage trainer artefacts continue to live + under each stage's own ``training.output_dir`` (as in single-stage + runs) — the pipeline output dir is the *index* directory, distinct + from the stage checkpoint directories so operators can ``rm -rf`` + individual stages without nuking the pipeline manifest. + """ + if root_cfg.pipeline is None: # defensive — orchestrator constructor catches this earlier + raise ValueError("Cannot derive pipeline paths from a config with no `pipeline` block.") + root_output_dir = root_cfg.pipeline.output_dir + return { + "root_output_dir": root_output_dir, + "state_file": os.path.join(root_output_dir, "pipeline_state.json"), + "manifest_dir": os.path.join(root_output_dir, "compliance"), + "manifest_file": os.path.join(root_output_dir, "compliance", "pipeline_manifest.json"), + } + + +def _atomic_write_json(path: str, payload: Dict[str, Any]) -> None: + """Write ``payload`` to ``path`` atomically. + + Pattern: serialise to a sibling temp file then ``os.replace`` it + onto the final name — guarantees a reader on any concurrent process + either sees the previous version or the new version, never a half- + written file. + + The temp filename is per-writer-unique (``...tmp``) + rather than a shared ``.tmp`` so two writers targeting the same + final path don't truncate each other's tmp mid-write — pre-fix, a + concurrent invocation produced ``FileNotFoundError`` from + ``os.replace`` (the loser's tmp was unlinked by the winner's + rename). Phase 14 doesn't *support* concurrent orchestrators on + the same ``pipeline.output_dir`` (see ``docs/guides/pipeline.md`` + "Limitations"), but the per-writer tmp is cheap defence so an + accidental double-invocation surfaces as last-writer-wins rather + than a Python traceback. The temp lives in the same directory as + ``path`` so ``os.replace`` stays a same-filesystem atomic rename. + """ + target_dir = os.path.dirname(path) or "." + os.makedirs(target_dir, exist_ok=True) + tmp = f"{path}.{os.getpid()}.{secrets.token_hex(4)}.tmp" + try: + with open(tmp, "w", encoding="utf-8") as f: + json.dump(payload, f, indent=2, sort_keys=False) + os.replace(tmp, path) + except Exception: + # If the write failed before the replace, clean up the orphan + # temp. After a successful replace there's nothing to remove + # (the tmp name no longer exists), so silently ignore that + # branch. Don't swallow the original exception either; we + # only swallow the cleanup-of-orphan-tmp failure. + try: + os.unlink(tmp) + except OSError: + pass + raise + + +def _serialise_state(state: PipelineState) -> Dict[str, Any]: + """Return a JSON-ready dict representation of a :class:`PipelineState`.""" + return asdict(state) + + +def _deserialise_state(payload: Dict[str, Any]) -> PipelineState: + """Round-trip :class:`PipelineState` from its on-disk JSON shape. + + Tolerates unknown keys (future-forward compat) and missing optionals + (legacy state files). Type-coerces ``stages[]`` back into + :class:`PipelineStageState` instances so the rest of the orchestrator + sees a strongly-typed object. + """ + stages_raw = payload.get("stages", []) + stages = [ + PipelineStageState(**{k: v for k, v in s.items() if k in PipelineStageState.__dataclass_fields__}) + for s in stages_raw + ] + return PipelineState( + pipeline_run_id=payload["pipeline_run_id"], + pipeline_config_hash=payload["pipeline_config_hash"], + forgelm_version=payload.get("forgelm_version", "unknown"), + started_at=payload["started_at"], + finished_at=payload.get("finished_at"), + final_status=payload.get("final_status", "in_progress"), + stopped_at=payload.get("stopped_at"), + stages=stages, + ) + + +# --------------------------------------------------------------------------- +# Orchestrator +# --------------------------------------------------------------------------- + + +class PipelineOrchestrator: + """Drives a multi-stage pipeline run end-to-end. + + Constructor takes the root config and the *bytes* of the pipeline YAML + (the bytes are hashed for the stale-resume guard; the parsed config + drives the actual stage iteration). + """ + + def __init__( + self, + root_cfg: ForgeConfig, + pipeline_yaml_bytes: bytes, + *, + output_format: str = "text", + ) -> None: + if root_cfg.pipeline is None: + # Defensive — callers should only reach this path when a + # ``pipeline:`` block is present. A None ``pipeline`` here + # signals a CLI dispatch bug; fail loud rather than running + # an empty pipeline. + raise ValueError("PipelineOrchestrator instantiated with a config that has no `pipeline` block.") + self.root_cfg: ForgeConfig = root_cfg + self.pipeline: PipelineConfig = root_cfg.pipeline + self.config_hash: str = _compute_pipeline_config_hash(pipeline_yaml_bytes) + self.output_format: str = output_format + self.paths: Dict[str, str] = _pipeline_paths(root_cfg) + # Lazy-initialised audit-log writer. Pinned to the pipeline run + # id on first event so every ``pipeline.*`` entry in the JSONL + # stream carries the same top-level ``run_id`` (Phase 14 review + # final-round F-B-1). Pre-fix, ``_audit_event`` constructed a + # fresh ``AuditLogger`` per call → every event got a different + # auto-generated ``fg-`` run_id, breaking SIEM-style + # grouping and the Article 12 record-keeping correlation + # contract. The pipeline_run_id is set in state.pipeline_run_id + # before any event fires, so the caller passes it through. + self._audit_logger: Optional[Any] = None + + # -- run-mode entry points ------------------------------------------------ + + def _prepare_resume_or_init_state( + self, + *, + resume_from: Optional[str], + force_resume: bool, + ) -> "tuple[Optional[PipelineState], Optional[int]]": + """Load + validate state for ``--resume-from`` or build a fresh state. + + Returns ``(state, None)`` on success; ``(None, EXIT_CONFIG_ERROR)`` + when the resume path is invalid (missing state file or stale-hash + refused without ``--force-resume``). Extracted from + :meth:`run` for Sonar python:S3776 cognitive-complexity hygiene. + """ + existing = self._load_state_file() + if resume_from is not None and existing is None: + logger.error( + "Cannot --resume-from %r: no pipeline_state.json found at %s. Run the pipeline once first.", + resume_from, + self.paths["state_file"], + ) + return None, EXIT_CONFIG_ERROR + if resume_from is not None: + refusal = self._validate_resume_state(existing, force=force_resume) + if refusal is not None: + return None, EXIT_CONFIG_ERROR + return existing, None + return self._init_state(), None + + def _resolve_resume_skiplist( + self, + *, + state: PipelineState, + resume_from: str, + ) -> "tuple[Optional[List[str]], Optional[int]]": + """Compute the list of already-completed stages to skip on resume. + + Returns ``(skiplist, None)`` on success or ``(None, + EXIT_CONFIG_ERROR)`` when ``resume_from`` names a stage that + doesn't exist. Stages before the resume index whose status is + ``completed`` and whose ``output_model`` directory exists on + disk are kept-as-is. Extracted from :meth:`run` for Sonar + python:S3776 cognitive-complexity hygiene. + """ + try: + resume_idx = next(i for i, s in enumerate(self.pipeline.stages) if s.name == resume_from) + except StopIteration: + logger.error( + "Cannot --resume-from %r: stage name not found in pipeline (valid names: %s).", + resume_from, + ", ".join(s.name for s in self.pipeline.stages), + ) + return None, EXIT_CONFIG_ERROR + # Phase 14 post-release review: a stage that exited with + # ``EXIT_AWAITING_APPROVAL`` is recorded with + # ``status == "gated_pending_approval"`` rather than + # ``"completed"``. After an operator runs ``forgelm approve`` + # and the promoted-model directory exists on disk, a + # subsequent ``--resume-from `` MUST skip the + # gated stage too — otherwise the orchestrator would re-train + # it, the next stage would auto-chain from the freshly-trained + # output instead of the operator-approved one, and the audit + # log would carry two ``pipeline.stage_started`` events for the + # same stage_index. Treat ``gated_pending_approval`` the same + # as ``completed`` for skip-eligibility: presence of an + # ``output_model`` directory on disk is what proves the stage + # produced a real artefact, regardless of which status label + # it carries. + _SKIPPABLE_STATUSES = {"completed", "gated_pending_approval"} + stages_to_skip_completed: List[str] = [] + for prior_state in state.stages[:resume_idx]: + if ( + prior_state.status in _SKIPPABLE_STATUSES + and prior_state.output_model + and os.path.isdir(prior_state.output_model) + ): + stages_to_skip_completed.append(prior_state.name) + logger.info( + "Resuming: stage %r (%s) already produced output at %s; skipping.", + prior_state.name, + prior_state.status, + prior_state.output_model, + ) + return stages_to_skip_completed, None + + def _resolve_stage_filter_setup( + self, + *, + stage_filter: str, + input_model_override: Optional[str], + ) -> "tuple[Optional[str], Optional[int]]": + """Validate ``--stage `` + seed the auto-chain on a non-first stage. + + Returns ``(prev_output_for_filter, None)`` on success or + ``(None, EXIT_CONFIG_ERROR)`` when the stage name is unknown or + the previous stage's output directory is missing (and no + ``--input-model`` override was supplied to bypass the chain). + Extracted from :meth:`run` for Sonar python:S3776 cognitive- + complexity hygiene. + """ + if not any(s.name == stage_filter for s in self.pipeline.stages): + logger.error( + "Cannot --stage %r: stage name not found in pipeline (valid names: %s).", + stage_filter, + ", ".join(s.name for s in self.pipeline.stages), + ) + return None, EXIT_CONFIG_ERROR + + if input_model_override is not None: + return None, None + + filter_idx = next(i for i, s in enumerate(self.pipeline.stages) if s.name == stage_filter) + if filter_idx == 0: + return None, None + + prev_stage = self.pipeline.stages[filter_idx - 1] + prev_merged = merge_pipeline_stage_config(self.root_cfg, prev_stage, prev_output_model=None) + candidate = os.path.join(prev_merged.training.output_dir, "final_model") + if not os.path.isdir(candidate): + logger.error( + "Stage %r requires the previous stage's output at %s; " + "pass --input-model to override or run the full pipeline first.", + stage_filter, + candidate, + ) + return None, EXIT_CONFIG_ERROR + return candidate, None + + def _handle_stage_outcome( + self, + *, + stage: PipelineStage, + stage_state: PipelineStageState, + exit_code: int, + state: PipelineState, + worst_exit: int, + ) -> "tuple[int, Optional[str], bool, bool]": + """Persist + audit + webhook the outcome of a single executed stage. + + Returns ``(worst_exit, new_prev_output, chain_broken, should_break)``: + + - ``worst_exit`` — the updated highest-priority exit code so far. + - ``new_prev_output`` — the auto-chain seed for the next stage + (``stage_state.output_model`` on success, ``None`` otherwise). + - ``chain_broken`` — ``True`` for any non-success outcome; the + caller marks downstream stages skipped on it. + - ``should_break`` — ``True`` only for ``EXIT_AWAITING_APPROVAL``; + the caller exits the stage loop immediately so a subsequent + ``--resume-from`` picks up the remaining stages post-approval. + + Extracted from :meth:`run` for Sonar python:S3776 cognitive- + complexity hygiene. + """ + if exit_code == EXIT_AWAITING_APPROVAL: + # Human-approval gate fired. Pipeline exits; remaining + # stages stay ``pending`` so a subsequent ``--resume-from`` + # picks them up after operator action. + stage_state.status = "gated_pending_approval" + state.final_status = "gated_pending_approval" + state.stopped_at = stage.name + state.finished_at = _utc_iso_now() + new_worst = max(worst_exit, EXIT_AWAITING_APPROVAL) + self._save_state(state) + # Phase 14 review F-N-1: dedicated ``pipeline.stage_gated`` + # event (instead of ``pipeline.stage_completed`` with a + # ``gate_decision=approval_pending`` sub-field). Lets + # dashboard / SIEM rules filter on the event name alone. + self._audit_event( + "pipeline.stage_gated", + pipeline_run_id=state.pipeline_run_id, + stage_name=stage.name, + gate_decision="approval_pending", + staging_path=stage_state.output_model, + ) + return new_worst, None, True, True + + if exit_code == EXIT_SUCCESS: + stage_state.status = "completed" + stage_state.gate_decision = "passed" + self._save_state(state) + self._audit_event( + "pipeline.stage_completed", + pipeline_run_id=state.pipeline_run_id, + stage_name=stage.name, + gate_decision="passed", + metrics=stage_state.metrics, + ) + return worst_exit, stage_state.output_model, False, False + + # Anything else (EXIT_TRAINING_ERROR / EXIT_EVAL_FAILURE): + # chain stops; downstream stages will skip with + # ``skipped_due_to_prior_revert``. + stage_state.status = "failed" + stage_state.gate_decision = "failed" + state.stopped_at = stage.name + new_worst = max(worst_exit, exit_code) + self._save_state(state) + self._audit_event( + "pipeline.stage_reverted" if stage_state.auto_revert_triggered else "pipeline.stage_completed", + pipeline_run_id=state.pipeline_run_id, + stage_name=stage.name, + gate_decision="failed", + auto_revert_triggered=stage_state.auto_revert_triggered, + ) + self._notify_pipeline( + "reverted" if stage_state.auto_revert_triggered else "stage_failed", + state, + stage_state=stage_state, + ) + return new_worst, None, True, False + + def _execute_one_stage( + self, + *, + index: int, + stage: PipelineStage, + state: PipelineState, + prev_output: Optional[str], + stage_filter: Optional[str], + input_model_override: Optional[str], + ) -> int: + """Setup + invoke a single stage's trainer. Returns the per-stage exit code. + + Caller is responsible for translating the returned exit code + into the next-state transition via :meth:`_handle_stage_outcome`. + Extracted from :meth:`run` for Sonar python:S3776 hygiene. + """ + stage_state = state.stages[index] + stage_state.index = index + stage_state.name = stage.name + stage_state.trainer_type = ( + stage.training.trainer_type if stage.training else self.root_cfg.training.trainer_type + ) + stage_state.started_at = _utc_iso_now() + stage_state.status = "running" + self._save_state(state) + + # ``--input-model`` attaches to a single filtered stage only. + override_for_this_stage = ( + input_model_override if (stage_filter is not None and stage.name == stage_filter) else None + ) + + exit_code = self._run_single_stage( + stage=stage, + stage_state=stage_state, + prev_output=prev_output, + input_model_override=override_for_this_stage, + state=state, + ) + stage_state.exit_code = exit_code + stage_state.finished_at = _utc_iso_now() + try: + start_dt = datetime.fromisoformat(stage_state.started_at) + end_dt = datetime.fromisoformat(stage_state.finished_at) + stage_state.duration_seconds = (end_dt - start_dt).total_seconds() + except (TypeError, ValueError): + stage_state.duration_seconds = None + return exit_code + + def _finalise_pipeline(self, state: PipelineState, worst_exit: int) -> None: + """Mark the pipeline finished + emit the terminal events. + + Mirrors the run-loop tail. Extracted from :meth:`run` for + Sonar python:S3776 cognitive-complexity hygiene. + """ + if state.final_status == "in_progress": + state.final_status = "completed" if worst_exit == EXIT_SUCCESS else "stopped_at_stage" + state.finished_at = _utc_iso_now() + self._save_state(state) + + if state.final_status in ("completed", "stopped_at_stage"): + self._audit_event( + "pipeline.completed", + pipeline_run_id=state.pipeline_run_id, + final_status=state.final_status, + stopped_at=state.stopped_at, + ) + self._notify_pipeline("completed", state) + + def _stage_skip_reason( + self, + *, + stage: PipelineStage, + stage_state: PipelineStageState, + state: PipelineState, + stage_filter: Optional[str], + stages_to_skip_completed: List[str], + chain_broken: bool, + ) -> Optional[str]: + """Decide whether the stage should be skipped, and update state accordingly. + + Returns one of: + + - ``"filtered"`` — ``--stage X`` was set and this is not the + named stage. Status set to ``skipped_by_filter``. + - ``"already_completed"`` — ``--resume-from`` picked this stage + up as already-done; caller seeds ``prev_output`` from + ``stage_state.output_model``. + - ``"chain_broken"`` — a prior stage in the same run reverted + or failed; status set to ``skipped_due_to_prior_revert``. + - ``None`` — proceed to execute. + + Extracted from :meth:`run` for Sonar python:S3776 cognitive- + complexity hygiene. + """ + if stage_filter is not None and stage.name != stage_filter: + stage_state.status = "skipped_by_filter" + stage_state.skipped_reason = f"--stage {stage_filter!r} was specified; only that stage runs." + self._save_state(state) + return "filtered" + if stage.name in stages_to_skip_completed: + return "already_completed" + if chain_broken: + stage_state.status = "skipped_due_to_prior_revert" + stage_state.skipped_reason = ( + f"Stage {state.stopped_at!r} triggered auto_revert; downstream stages did not run." + ) + self._save_state(state) + return "chain_broken" + return None + + def _run_stage_loop( + self, + *, + state: PipelineState, + stage_filter: Optional[str], + stages_to_skip_completed: List[str], + input_model_override: Optional[str], + prev_output_for_filter: Optional[str], + ) -> int: + """Iterate over ``self.pipeline.stages`` and return the aggregated ``worst_exit``. + + Each iteration is one of three things: + + 1. A skipped stage (filter / already-done / chain broken) — see + :meth:`_stage_skip_reason` for the routing rules. + 2. An executed stage — :meth:`_execute_one_stage` runs the + trainer and :meth:`_handle_stage_outcome` records the + transition + audit events. + 3. A gated stage (``EXIT_AWAITING_APPROVAL``) — the outcome + handler sets ``should_break`` and the loop returns early so + downstream stages stay ``pending`` for a subsequent + ``--resume-from``. + + Extracted from :meth:`run` for Sonar python:S3776 cognitive- + complexity hygiene. + """ + worst_exit: int = EXIT_SUCCESS + prev_output: Optional[str] = prev_output_for_filter + chain_broken: bool = False + for i, stage in enumerate(self.pipeline.stages): + stage_state = state.stages[i] + skip_reason = self._stage_skip_reason( + stage=stage, + stage_state=stage_state, + state=state, + stage_filter=stage_filter, + stages_to_skip_completed=stages_to_skip_completed, + chain_broken=chain_broken, + ) + if skip_reason == "already_completed": + prev_output = stage_state.output_model + continue + if skip_reason is not None: + continue + + exit_code = self._execute_one_stage( + index=i, + stage=stage, + state=state, + prev_output=prev_output, + stage_filter=stage_filter, + input_model_override=input_model_override, + ) + worst_exit, new_prev_output, chain_broken_now, should_break = self._handle_stage_outcome( + stage=stage, + stage_state=stage_state, + exit_code=exit_code, + state=state, + worst_exit=worst_exit, + ) + prev_output = new_prev_output if new_prev_output is not None else prev_output + chain_broken = chain_broken or chain_broken_now + if should_break: + break + return worst_exit + + def _check_output_dir_collisions(self) -> Optional[str]: + """Pre-flight: detect two stages writing to the same ``training.output_dir``. + + Returns ``None`` on clean, or a single human-readable error + string listing every collision. Phase 14 post-release review: + previously this check lived only in :meth:`dry_run`, so an + operator running ``forgelm --config pipeline.yaml`` without + first running ``--dry-run`` could overwrite per-stage manifests + and checkpoints across stages silently. Now called from both + :meth:`dry_run` and :meth:`run` so the guard is unmissable. + """ + prev_output: Optional[str] = None + seen_dirs: Dict[str, str] = {} + collisions: List[str] = [] + for stage in self.pipeline.stages: + try: + merged = merge_pipeline_stage_config( + self.root_cfg, + stage, + prev_output_model=prev_output, + ) + except Exception as exc: # noqa: BLE001 — see DEBUG log below + # Merge failures are intentionally skipped here because + # they re-surface through the per-stage pipeline-error + # path with full diagnostics — the pre-flight's job is + # scoped to ``output_dir`` layout only. Codacy / bandit + # B112 flagged the bare ``continue`` as a silent-skip + # anti-pattern; logging the exception at DEBUG keeps the + # skip *traceable* without aborting the pre-flight on + # what is by-design a per-stage local failure. + logger.debug( + "Pre-flight skipped collision check for stage %r: merge raised %s (%s); " + "the per-stage run path will surface the same error.", + stage.name, + type(exc).__name__, + exc, + ) + continue + abs_out = os.path.abspath(merged.training.output_dir) + if abs_out in seen_dirs: + collisions.append( + f"Stage {stage.name!r}: training.output_dir collides with stage " + f"{seen_dirs[abs_out]!r} (both resolve to {abs_out!r})." + ) + else: + seen_dirs[abs_out] = stage.name + prev_output = os.path.join(merged.training.output_dir, "final_model") + if collisions: + return ( + "Pre-flight: per-stage `training.output_dir` collision(s) detected; " + "per-stage checkpoints and manifests would overwrite each other. " + "Set a unique `training.output_dir` in each stage.\n " + "\n ".join(collisions) + ) + return None + + def run( + self, + *, + stage_filter: Optional[str] = None, + resume_from: Optional[str] = None, + force_resume: bool = False, + input_model_override: Optional[str] = None, + ) -> int: + """Execute the pipeline. Returns a process exit code. + + ``stage_filter`` and ``resume_from`` are mutually exclusive; the + caller (CLI parser) validates this before reaching here. + + Exit-code aggregation uses ``worst_exit = max(worst_exit, code)`` + across stages, giving the numeric precedence: + ``EXIT_AWAITING_APPROVAL (4)`` > ``EXIT_EVAL_FAILURE (3)`` > + ``EXIT_TRAINING_ERROR (2)`` > ``EXIT_CONFIG_ERROR (1)`` > + ``EXIT_SUCCESS (0)``. An approval gate triggers an immediate + ``break`` (later stages stay ``pending``) and any other non-zero + outcome sets ``chain_broken = True`` (later stages skip with + ``skipped_due_to_prior_revert``), so multi-stage compounding is + rare — in practice the return value reflects the single failing + stage's exit code. + """ + # 0. Pre-flight: output_dir collisions across stages. Same + # guard that ``dry_run`` runs; mirrored here so an operator + # who skipped ``--dry-run`` doesn't silently overwrite + # per-stage checkpoints + manifests. + collision_msg = self._check_output_dir_collisions() + if collision_msg is not None: + logger.error(collision_msg) + return EXIT_CONFIG_ERROR + + # 1. Load (or initialise) state. + state, state_err = self._prepare_resume_or_init_state(resume_from=resume_from, force_resume=force_resume) + if state is None: + return state_err # type: ignore[return-value] + + # 2. Determine which stages to execute. + stages_to_skip_completed: List[str] = [] + if resume_from is not None: + skiplist, skiplist_err = self._resolve_resume_skiplist(state=state, resume_from=resume_from) + if skiplist is None: + return skiplist_err # type: ignore[return-value] + stages_to_skip_completed = skiplist + + # 2a. ``--stage`` validation + auto-chain seeding. + prev_output_for_filter: Optional[str] = None + if stage_filter is not None: + seed, filter_err = self._resolve_stage_filter_setup( + stage_filter=stage_filter, input_model_override=input_model_override + ) + if filter_err is not None: + return filter_err + prev_output_for_filter = seed + # Phase 14 review final-round F-B-2: surface the disk-seed + # path on the *predecessor* stage's manifest entry as well. + # When ``--stage X`` re-runs only the named stage, its + # predecessor is recorded as ``skipped_by_filter`` with no + # ``output_model``. The filtered stage records + # ``input_source: "chain"`` (we auto-chained from the seed), + # so the strict chain verifier (F-B-3) compares stage X's + # ``input_model`` against ``stages[X-1].output_model`` and + # flags a ``chain_integrity_violation`` even though the + # chain is intact on disk. We seed the predecessor's + # ``output_model`` here so the manifest faithfully + # represents "stage X-1 produced this path (in a prior + # run); stage X chained from it now". + if seed is not None: + filter_idx = next(i for i, s in enumerate(self.pipeline.stages) if s.name == stage_filter) + if filter_idx > 0: + state.stages[filter_idx - 1].output_model = seed + + # 3. Emit pipeline.started + webhook (fresh runs only). + if resume_from is None: + self._audit_event( + "pipeline.started", + pipeline_run_id=state.pipeline_run_id, + config_hash=self.config_hash, + stage_count=len(self.pipeline.stages), + stage_names=[s.name for s in self.pipeline.stages], + ) + self._notify_pipeline("started", state) + + # 4. Iterate stages (delegated so this method stays at a + # bounded cognitive-complexity score). + worst_exit = self._run_stage_loop( + state=state, + stage_filter=stage_filter, + stages_to_skip_completed=stages_to_skip_completed, + input_model_override=input_model_override, + prev_output_for_filter=prev_output_for_filter, + ) + + # 5+6. Final state + manifest + summary. + self._finalise_pipeline(state, worst_exit) + self._emit_summary(state) + return worst_exit + + def dry_run(self) -> int: + """Validate every stage without allocating any GPU resources. + + Collects all per-stage Pydantic / merge errors before exiting, + mirroring ``pytest --collectonly``. Returns + :data:`EXIT_CONFIG_ERROR (1)` if any stage fails validation, + :data:`EXIT_SUCCESS (0)` otherwise. + + Additionally checks that no two stages resolve to the same + ``training.output_dir`` — colliding output dirs would cause one + stage's checkpoints and per-stage Annex-IV manifest to overwrite + another's, breaking the chain of custody silently (Phase 14 + review F-G-1). Because stage-level ``training`` blocks + *inherit* the root ``training`` block wholesale unless the + stage supplies a full override, two stages that both inherit + end up sharing the root's ``output_dir`` — that's the canonical + misconfiguration this guard catches. + """ + errors: List[str] = [] + prev_output: Optional[str] = None + seen_dirs: Dict[str, str] = {} # abs output_dir -> first stage that claimed it + for stage in self.pipeline.stages: + try: + merged = merge_pipeline_stage_config( + self.root_cfg, + stage, + prev_output_model=prev_output, + ) + except Exception as exc: # noqa: BLE001 — best-effort: collect every per-stage validation error for a single operator report instead of stopping at the first. + errors.append(f"Stage {stage.name!r}: merge failed: {exc}") + continue + + # Collision check (Phase 14 review F-G-1): every stage must + # write its checkpoints + per-stage manifest into a distinct + # directory. Comparing absolute paths so ``./out`` and + # ``out/`` collide as one. + abs_out = os.path.abspath(merged.training.output_dir) + if abs_out in seen_dirs: + errors.append( + f"Stage {stage.name!r}: training.output_dir collides with stage " + f"{seen_dirs[abs_out]!r} (both resolve to {abs_out!r}); per-stage " + "checkpoints and manifests would overwrite each other. Set a " + "unique 'training.output_dir' in each stage." + ) + else: + seen_dirs[abs_out] = stage.name + + # Project the output path the next stage would auto-chain to. + prev_output = os.path.join(merged.training.output_dir, "final_model") + + if errors: + logger.error("Pipeline dry-run found %d stage error(s):", len(errors)) + for e in errors: + logger.error(" %s", e) + return EXIT_CONFIG_ERROR + + logger.info( + "Pipeline dry-run OK: %d stage(s) validated; no GPU was allocated.", + len(self.pipeline.stages), + ) + return EXIT_SUCCESS + + # -- internals ------------------------------------------------------------ + + def _init_state(self) -> PipelineState: + """Build a fresh :class:`PipelineState` from the parsed pipeline.""" + from .._version import __version__ + + stages = [ + PipelineStageState( + name=s.name, + index=i, + trainer_type=(s.training.trainer_type if s.training else self.root_cfg.training.trainer_type), + ) + for i, s in enumerate(self.pipeline.stages) + ] + return PipelineState( + pipeline_run_id=_generate_run_id(), + pipeline_config_hash=self.config_hash, + forgelm_version=__version__, + started_at=_utc_iso_now(), + stages=stages, + ) + + def _validate_resume_state(self, existing: PipelineState, *, force: bool) -> Optional[str]: + """Stale-state guard for ``--resume-from``. + + If the on-disk state file's ``pipeline_config_hash`` differs from + the current parse of the pipeline YAML, the operator edited the + file between the failed run and the resume call — resuming + against a different config silently produces a chain whose + history doesn't match its current shape. We refuse unless + ``--force-resume`` is set (logged at WARNING **and** emitted as a + ``pipeline.force_resume`` audit event so reviewers see why the + divergence was accepted). + + Topology vs value-level edits (Phase 14 review final-round + clarification): + + - **Topology change** (adding, deleting, or renaming a stage — + the ordered ``[s.name for s in stages]`` list changes) is + refused **even under** ``--force-resume`` because + ``state.stages[i]`` is positionally addressed and a topology + shift would silently overwrite a different stage's history. + - **Value-level edits** (different ``trainer_type``, + ``training.output_dir``, hyperparameters under unchanged + stage names) trip the hash check but are *accepted* under + ``--force-resume``. This is the documented escape hatch: + the operator is signalling "I know the on-disk state and the + current config diverge; proceed and record the + ``pipeline.force_resume`` audit breadcrumb." Reviewers can + reconstruct the divergence from the event payload's + ``old_config_hash`` / ``new_config_hash`` fields. + + Return semantics: + + - ``None`` — the resume is OK to proceed (no hash divergence, or + divergence accepted via ``--force-resume``). + - error-string — the resume must abort; the caller maps this to + ``EXIT_CONFIG_ERROR``. Using a return value (rather than + ``sys.exit`` mid-method) keeps the orchestrator's exit policy + consistent with the rest of ``run()`` — every refusal flows + back through the same ``return`` branch, the same audit-log + finalisation runs, and the test surface uses a uniform + ``assert code == EXIT_CONFIG_ERROR`` shape. + """ + # Topology guard runs UNCONDITIONALLY — even on hash match — + # because the YAML's bytes hash does not constrain the + # on-disk state file's contents. A direct edit / corruption / + # truncation of ``pipeline_state.json`` could leave the same + # ``pipeline_config_hash`` recorded inside while the + # ``stages[]`` array drifts (missing entries, wrong order, + # tampered ``output_model`` paths). Without this check, a + # ``state.stages[i]`` whose name doesn't match + # ``self.pipeline.stages[i]`` would silently overwrite the + # wrong stage's history. + # + # *Value-level* edits — different ``trainer_type``, + # ``training.output_dir``, hyperparameters under identical + # stage names — trip the hash check but are *accepted by + # design* under ``--force-resume``. The flag is the operator's + # explicit signal that they understand the on-disk state and + # the current config diverge; the guard's job is only to + # protect the positional-addressing invariant. See + # ``docs/guides/pipeline.md`` "Limitations" for the documented + # contract. + on_disk_names = [s.name for s in existing.stages] + current_names = [s.name for s in self.pipeline.stages] + if on_disk_names != current_names: + msg = ( + f"Refusing to --resume-from: pipeline stage topology disagrees " + f"with the on-disk state file (this is refused even with " + f"--force-resume because state.stages[i] is positionally " + f"addressed).\n" + f" on-disk stages: {on_disk_names!r}\n" + f" current stages: {current_names!r}\n" + "Inspect / regenerate the state file before resuming." + ) + logger.error(msg) + return msg + + if existing.pipeline_config_hash == self.config_hash: + return None + if not force: + msg = ( + f"Refusing to --resume-from: pipeline_config_hash mismatch.\n" + f" on-disk: {existing.pipeline_config_hash}\n" + f" current: {self.config_hash}\n" + "Pass --force-resume to override (logged for audit)." + ) + logger.error(msg) + return msg + + old_hash = existing.pipeline_config_hash + logger.warning( + "Resuming with --force-resume despite pipeline_config_hash mismatch: %s → %s.", + old_hash, + self.config_hash, + ) + # Append-only audit event so a reviewer reading the JSONL stream + # later can distinguish an operator-approved stale-config resume + # from a normal resume. Emitted BEFORE the stored hash is + # updated so the event payload still records both sides of the + # divergence. + self._audit_event( + "pipeline.force_resume", + pipeline_run_id=existing.pipeline_run_id, + old_config_hash=old_hash, + new_config_hash=self.config_hash, + ) + # Update the stored hash to match the now-trusted current config + # so subsequent transitions don't trip the guard again. + existing.pipeline_config_hash = self.config_hash + return None + + def _load_state_file(self) -> Optional[PipelineState]: + path = self.paths["state_file"] + if not os.path.isfile(path): + return None + try: + with open(path, "r", encoding="utf-8") as f: + payload = json.load(f) + except (OSError, json.JSONDecodeError) as e: + logger.warning("Failed to read pipeline_state.json at %s: %s. Treating as missing.", path, e) + return None + # Phase 14 review F-R1-S4: a tampered state file (wrong shape, + # missing required keys, ``stages`` not a list of dicts, etc.) + # could crash ``_deserialise_state`` with an opaque + # ``TypeError`` / ``KeyError`` traceback instead of producing + # an actionable config-error path. Catch the common shape + # failures here and treat the file as missing — the caller's + # ``--resume-from`` branch then surfaces ``EXIT_CONFIG_ERROR`` + # with a clear message. + try: + return _deserialise_state(payload) + except (KeyError, TypeError, ValueError, AttributeError) as e: + # AttributeError covers the case where ``stages`` items aren't + # dicts (e.g. an attacker wrote ``stages: [null, "foo"]``) — + # ``s.items()`` then raises AttributeError on the str / None. + logger.warning( + "pipeline_state.json at %s is structurally invalid (%s); treating as missing.", + path, + e, + ) + return None + + def _save_state(self, state: PipelineState) -> None: + """Persist state file + refresh pipeline manifest. + + Both writes are atomic. Manifest export is delegated to + :mod:`forgelm.compliance` so the schema lives in one place. + """ + _atomic_write_json(self.paths["state_file"], _serialise_state(state)) + # Lazy import — compliance has its own heavy deps for the + # Annex-IV verifier path; only needed when we actually export. + from ..compliance import generate_pipeline_manifest + + manifest = generate_pipeline_manifest(state=state, root_cfg=self.root_cfg) + _atomic_write_json(self.paths["manifest_file"], manifest) + + def _audit_event(self, event: str, *, pipeline_run_id: str, **fields: Any) -> None: + """Append-only audit log entry under the **pipeline** root output dir. + + Two separate audit-log streams exist on a multi-stage run: + + - **Pipeline-level** (this method): ``/ + audit_log.jsonl`` — carries ``pipeline.*`` lifecycle events + (started / stage_started / stage_completed / stage_gated / + stage_reverted / force_resume / completed). + - **Stage-level** (emitted by each ``ForgeTrainer``): + ``/audit_log.jsonl`` — carries the + existing ``training.*`` vocabulary unchanged from a single- + stage v0.6.0 run. + + Both files are append-only JSONL. Reviewers can correlate by + joining on ``pipeline_run_id`` (present in every pipeline event) + with the per-stage ``training_manifest.json`` ``training_run_id``. + See ``docs/guides/pipeline.md`` "Audit events" for the full + join key matrix. + + Phase 14 review final-round F-B-1: the ``AuditLogger`` instance + is cached on the orchestrator and pinned to the pipeline run + id, so every entry's top-level ``run_id`` field equals the + pipeline run id (instead of a fresh auto-generated + ``fg-`` per event, which broke SIEM-style grouping and + the Article 12 correlation contract). ``pipeline_run_id`` is + now a required keyword argument so callers cannot silently + regress this. + """ + try: + from ..compliance import AuditLogger + + if self._audit_logger is None: + self._audit_logger = AuditLogger(self.paths["root_output_dir"], run_id=pipeline_run_id) + self._audit_logger.log_event(event, pipeline_run_id=pipeline_run_id, **fields) + except Exception as exc: # noqa: BLE001 — audit emission is best-effort telemetry; a write failure must not abort the pipeline. + logger.warning("Failed to emit audit event %r: %s", event, exc) + + def _notify_pipeline( + self, + kind: Literal["started", "completed", "reverted", "stage_failed"], + state: PipelineState, + *, + stage_state: Optional[PipelineStageState] = None, + ) -> None: + """Webhook fan-out for pipeline lifecycle transitions. + + Notifier methods are optional (added in Phase 14, defended via + ``getattr`` so a fork that pinned an older webhook module + doesn't crash here). + """ + try: + from ..webhook import WebhookNotifier + + notifier = WebhookNotifier(self.root_cfg) + stage_count = len(self.pipeline.stages) + if kind == "started": + method = getattr(notifier, "notify_pipeline_started", None) + if method: + method(run_id=state.pipeline_run_id, stage_count=stage_count) + elif kind == "completed": + method = getattr(notifier, "notify_pipeline_completed", None) + if method: + method(run_id=state.pipeline_run_id, final_status=state.final_status, stopped_at=state.stopped_at) + elif kind == "reverted": + method = getattr(notifier, "notify_pipeline_reverted", None) + if method and stage_state is not None: + method( + run_id=state.pipeline_run_id, + stage_name=stage_state.name, + reason=stage_state.error or "auto_revert", + ) + # ``stage_failed`` is currently only an audit event — no + # dedicated webhook method. Existing per-stage + # ``training.failure`` from ForgeTrainer's notifier covers + # the operator surface; the pipeline-level event would be + # redundant noise on Slack. + except Exception as exc: # noqa: BLE001 — best-effort notification; webhook outage must not derail the pipeline. + logger.warning("Failed to emit pipeline webhook %r: %s", kind, exc) + + @staticmethod + def _resolve_input_source( + *, + input_model_override: Optional[str], + stage_model_set: bool, + prev_output: Optional[str], + ) -> InputSourceLiteral: + """Resolve a stage's ``input_source`` from the priority ladder. + + Priority order: CLI override > stage explicit > chain (prev + output) > root. See the ``InputSourceLiteral`` docstring for + what each label means. Extracted from ``_run_single_stage`` + for Sonar python:S3776 cognitive-complexity hygiene; pure + function so the caller stays linear. + """ + if input_model_override is not None: + return "cli_override" + if stage_model_set: + return "stage_explicit" + if prev_output is not None: + return "chain" + return "root" + + def _merge_and_validate_stage( + self, + *, + stage: PipelineStage, + stage_state: PipelineStageState, + prev_output: Optional[str], + input_model_override: Optional[str], + ) -> Optional[Any]: + """Merge the per-stage ForgeConfig + run the chain-presence guard. + + Returns the merged ``ForgeConfig`` on success, or ``None`` if a + config-error path already populated ``stage_state.error`` and + the caller should return ``EXIT_CONFIG_ERROR``. Extracted from + ``_run_single_stage`` for Sonar python:S3776 cognitive- + complexity hygiene. + """ + try: + stage_cfg = merge_pipeline_stage_config( + self.root_cfg, + stage, + prev_output_model=prev_output, + input_model_override=input_model_override, + ) + except Exception as exc: # noqa: BLE001 + stage_state.error = f"Config merge failed: {exc}" + logger.exception("Stage %r config merge failed.", stage.name) + return None + + stage_state.input_model = stage_cfg.model.name_or_path + + # Chain-presence guard: when auto-chained, the previous stage's + # output_dir/final_model leaf must exist. Phase 14 review F-S-1. + if stage_state.input_source == "chain" and not os.path.isdir(stage_cfg.model.name_or_path): + stage_state.error = ( + f"Stage {stage.name!r} requires the previous stage's output at " + f"{stage_cfg.model.name_or_path}; pass --input-model to " + f"override or run the full pipeline first." + ) + logger.error(stage_state.error) + return None + + return stage_cfg + + def _invoke_trainer( + self, + *, + stage_cfg: Any, + stage_name: str, + stage_state: PipelineStageState, + ) -> Optional[Any]: + """Run a single stage's :class:`ForgeTrainer` lifecycle. + + Returns the ``TrainResult`` on success, ``None`` on import / + runtime failure (caller maps to the right exit code via + ``stage_state.error``). Extracted from ``_run_single_stage`` + for Sonar python:S3776 cognitive-complexity hygiene. + """ + # Defer heavy imports so dry-run / --stage on a different stage + # doesn't load torch. + try: + from ..data import prepare_dataset + from ..model import get_model_and_tokenizer + from ..trainer import ForgeTrainer + from ..utils import setup_authentication + except ImportError as e: + stage_state.error = f"Missing training dependency: {e}" + logger.error(stage_state.error) + return None + + # Top-of-stage catch. ForgeTrainer.train() crosses every + # concern (HF load, dataset, TRL, safety, judge, audit, + # compliance, webhook); any uncaught exception must surface as + # a per-stage failure rather than a Python traceback that + # strands the pipeline state. + try: + if not stage_cfg.model.offline: + setup_authentication(stage_cfg.auth.hf_token if stage_cfg.auth else None) + model, tokenizer = get_model_and_tokenizer(stage_cfg) + dataset = prepare_dataset(stage_cfg, tokenizer) + trainer = ForgeTrainer(model=model, tokenizer=tokenizer, config=stage_cfg, dataset=dataset) + return trainer.train() + except Exception as exc: # noqa: BLE001 + stage_state.error = f"Trainer crashed: {exc}" + logger.exception("Stage %r trainer crashed.", stage_name) + return None + + def _run_single_stage( + self, + *, + stage: PipelineStage, + stage_state: PipelineStageState, + prev_output: Optional[str], + input_model_override: Optional[str], + state: PipelineState, + ) -> int: + """Materialise the per-stage ForgeConfig, run the trainer, record results. + + Returns the per-stage exit code; the caller decides what to do + with it (continue, halt, gate). + """ + stage_state.input_source = self._resolve_input_source( + input_model_override=input_model_override, + stage_model_set=stage.model is not None, + prev_output=prev_output, + ) + + stage_cfg = self._merge_and_validate_stage( + stage=stage, + stage_state=stage_state, + prev_output=prev_output, + input_model_override=input_model_override, + ) + if stage_cfg is None: + return EXIT_CONFIG_ERROR + + self._audit_event( + "pipeline.stage_started", + pipeline_run_id=state.pipeline_run_id, + stage_name=stage.name, + stage_index=stage_state.index, + input_model=stage_state.input_model, + input_source=stage_state.input_source, + ) + + result = self._invoke_trainer( + stage_cfg=stage_cfg, + stage_name=stage.name, + stage_state=stage_state, + ) + if result is None: + return EXIT_TRAINING_ERROR + + # Capture results onto the stage state. + stage_state.metrics = {k: float(v) for k, v in (result.metrics or {}).items() if isinstance(v, (int, float))} + stage_state.auto_revert_triggered = bool(result.reverted) + stage_state.training_manifest = os.path.join( + stage_cfg.training.output_dir, "compliance", "training_manifest.json" + ) + + # Human-approval gate produces a non-zero exit + a staging_path. + if result.staging_path: + stage_state.output_model = result.staging_path + return EXIT_AWAITING_APPROVAL + + # Auto-revert or gate failure: trainer returns success=False. + if not result.success: + stage_state.error = result.error or "Stage gate failed." + return EXIT_EVAL_FAILURE + + # Normal success path. + stage_state.output_model = result.final_model_path or os.path.join(stage_cfg.training.output_dir, "final_model") + return EXIT_SUCCESS + + # -- summary -------------------------------------------------------------- + + def _emit_summary(self, state: PipelineState) -> None: + """Operator-facing run summary (text or JSON envelope).""" + if self.output_format == "json": + print(json.dumps(_serialise_state(state), indent=2)) + return + + logger.info("Pipeline %s — final_status=%s", state.pipeline_run_id, state.final_status) + for s in state.stages: + logger.info( + " [%s] %s — trainer=%s, exit=%s, output=%s", + s.status, + s.name, + s.trainer_type, + s.exit_code if s.exit_code is not None else "—", + s.output_model or "—", + ) + + +# --------------------------------------------------------------------------- +# Entry point (CLI dispatcher delegates here) +# --------------------------------------------------------------------------- + + +def run_pipeline_from_args( + root_cfg: ForgeConfig, + pipeline_yaml_bytes: bytes, + args: Any, +) -> int: + """Dispatcher entry — orchestrates flag → orchestrator wiring. + + ``args`` is the parsed ``argparse.Namespace`` from the CLI; only the + pipeline-relevant attributes are read. Keeping this glue thin makes + the orchestrator itself trivial to unit-test against a fake args + namespace. + """ + stage_filter = getattr(args, "stage", None) + resume_from = getattr(args, "resume_from", None) + force_resume = bool(getattr(args, "force_resume", False)) + # Phase 14 review F-F-2: argparse cannot reject an empty string + # value for ``--input-model "" --stage X`` at parse time, and the + # downstream merge helper uses ``is not None`` rather than a truthy + # check, so the empty string would silently overwrite the auto- + # chained model path. Normalise to ``None`` here so the rest of the + # orchestrator sees "no override" rather than "override with nothing". + raw_input_model = getattr(args, "input_model", None) + input_model_override = raw_input_model if raw_input_model else None + output_format = getattr(args, "output_format", "text") + dry_run = bool(getattr(args, "dry_run", False)) + + if stage_filter and resume_from: + logger.error("`--stage` and `--resume-from` are mutually exclusive.") + return EXIT_CONFIG_ERROR + if input_model_override and not stage_filter: + logger.error("`--input-model` requires `--stage `.") + return EXIT_CONFIG_ERROR + + # Phase 14 post-release review: reject single-stage no-train flags + # on a ``pipeline:`` config. ``_dispatch.py`` routes here BEFORE + # ``_maybe_run_no_train_mode`` (the F-B-1 fix on PR #53 was + # explicit: otherwise the legacy no-train modes would ``sys.exit`` + # before the orchestrator's per-stage validation could run), but + # that means ``--fit-check`` / ``--merge`` / ``--generate-data`` / + # ``--compliance-export`` / ``--benchmark-only`` would silently + # trigger a full pipeline training run instead of the single-stage + # operation the operator asked for. Reject them explicitly with + # an actionable error so the operator either drops the flag or + # switches to a single-stage config. ``--dry-run`` IS supported + # on pipelines (collects every per-stage validation error before + # exiting) and is handled below. + _UNSUPPORTED_ON_PIPELINE = ( + ("fit_check", "--fit-check"), + ("merge", "--merge"), + ("generate_data", "--generate-data"), + ("compliance_export", "--compliance-export"), + ("benchmark_only", "--benchmark-only"), + ) + for attr, flag_name in _UNSUPPORTED_ON_PIPELINE: + if getattr(args, attr, None): + logger.error( + "`%s` is not supported when the config carries a `pipeline:` " + "block. These flags operate on a single-stage config; either " + "drop the `pipeline:` block or remove the flag. Use `--dry-run` " + "for multi-stage validation without GPU allocation.", + flag_name, + ) + return EXIT_CONFIG_ERROR + + orchestrator = PipelineOrchestrator( + root_cfg=root_cfg, + pipeline_yaml_bytes=pipeline_yaml_bytes, + output_format=output_format, + ) + + if dry_run: + return orchestrator.dry_run() + return orchestrator.run( + stage_filter=stage_filter, + resume_from=resume_from, + force_resume=force_resume, + input_model_override=input_model_override, + ) diff --git a/forgelm/cli/subcommands/_verify_annex_iv.py b/forgelm/cli/subcommands/_verify_annex_iv.py index 4458beb8..438ecbf4 100644 --- a/forgelm/cli/subcommands/_verify_annex_iv.py +++ b/forgelm/cli/subcommands/_verify_annex_iv.py @@ -185,71 +185,177 @@ def _compute_manifest_hash(artifact: Dict[str, Any]) -> str: return compute_annex_iv_manifest_hash(artifact) -def _run_verify_annex_iv_cmd(args, output_format: str) -> None: - """Top-level dispatcher for ``forgelm verify-annex-iv ``.""" - path = getattr(args, "path", None) - if not path: - _output_error_and_exit( - output_format, - "verify-annex-iv requires a path argument: `forgelm verify-annex-iv `.", - EXIT_CONFIG_ERROR, +def _run_pipeline_mode(path: str, output_format: str) -> None: + """Verify a pipeline run directory's manifest + chain integrity. + + Reads ``/compliance/pipeline_manifest.json``, runs the in- + memory verifier + per-stage training_manifest existence check, and + prints / exits. Extracted from :func:`_run_verify_annex_iv_cmd` + for Sonar python:S3776 cognitive-complexity hygiene. + + Exit-code mapping (Phase 14 post-release review — now mirrors the + single-artefact path's policy in + :func:`_verify_artefact_and_handle_io_errors`): + + - ``EXIT_SUCCESS (0)`` — empty violation list. + - ``EXIT_CONFIG_ERROR (1)`` — operator-actionable input error: + ``pipeline_manifest.json not found …`` (wrong directory), + ``pipeline_manifest.json invalid JSON: …`` (reachable but + corrupted on disk — operator can regenerate or fix), or any + structural / chain-integrity violation. + - ``EXIT_TRAINING_ERROR (2)`` — genuine runtime I/O failure on a + reachable path: ``pipeline_manifest.json unreadable: …`` + sentinel from a non-parse OSError (locked file, mid-read I/O + failure) or an uncaught OSError that bubbles past the verifier. + + Note: the previous release mapped ``invalid JSON`` to + ``EXIT_TRAINING_ERROR`` because the verifier emitted a shared + "unreadable" sentinel for both ``OSError`` and ``JSONDecodeError``. + The verifier now emits distinct prefixes so the CLI can + differentiate the two on the sentinel alone. + """ + from forgelm.compliance import verify_pipeline_manifest_at_path + + # Defensive try/except: the verifier already maps OSError / + # JSONDecodeError to violation strings, but a future change there + # could let an exception bubble — fail loud rather than swallow. + try: + violations = verify_pipeline_manifest_at_path(path) + except OSError as exc: + msg = f"FAIL: pipeline manifest at {path} — runtime I/O error: {exc}" + if output_format == "json": + print( + json.dumps( + { + "success": False, + "mode": "pipeline", + "path": os.path.abspath(path), + "violations": [str(exc)], + }, + indent=2, + ) + ) + else: + print(msg) + sys.exit(EXIT_TRAINING_ERROR) + + # Sentinel-based exit-code routing. Only the OSError-shaped + # ``unreadable`` sentinel maps to EXIT_TRAINING_ERROR; everything + # else (``invalid JSON``, ``not found``, structural / chain + # violations) is operator-actionable → EXIT_CONFIG_ERROR. + runtime_io_error = any("unreadable" in v for v in violations) + + if output_format == "json": + print( + json.dumps( + { + "success": not violations, + "mode": "pipeline", + "path": os.path.abspath(path), + "violations": violations, + }, + indent=2, + ) ) - # Round 6 absorption: the prior `os.path.isfile()` pre-check - # collapsed two distinct conditions ("path doesn't exist" and - # "path exists but the user can't stat it") into a single - # `EXIT_CONFIG_ERROR`, breaking the documented exit-code - # contract for the permission-denied case. Move all path - # validation inside the try block and let Python's exception - # hierarchy disambiguate: FileNotFoundError + IsADirectoryError - # = caller-input error (exit 1); OSError (incl. PermissionError) - # = genuine I/O failure on a reachable path (exit 2). + elif not violations: + print(f"OK: pipeline manifest at {path}") + else: + print(f"FAIL: pipeline manifest at {path}") + for v in violations: + print(f" - {v}") + + if not violations: + sys.exit(EXIT_SUCCESS) + sys.exit(EXIT_TRAINING_ERROR if runtime_io_error else EXIT_CONFIG_ERROR) + + +def _verify_artefact_and_handle_io_errors(path: str, output_format: str) -> "VerifyAnnexIVResult": + """Run :func:`verify_annex_iv_artifact` with the documented I/O + error-mapping policy. + + Extracted from :func:`_run_verify_annex_iv_cmd` for Sonar + python:S3776 cognitive-complexity hygiene. Each ``except`` branch + exits the process via ``_output_error_and_exit`` — the function + only returns on success. + + Exit-code mapping (per ``docs/reference/verify_annex_iv_subcommand.md``): + + - ``FileNotFoundError`` / ``IsADirectoryError`` / ``JSONDecodeError`` → + ``EXIT_CONFIG_ERROR (1)`` (operator-actionable input error). + - ``OSError`` (catch-all, must follow ``FileNotFoundError`` because + Python's OSError hierarchy makes it a subclass) → + ``EXIT_TRAINING_ERROR (2)`` (genuine runtime I/O failure). + """ try: - result = verify_annex_iv_artifact(path) + return verify_annex_iv_artifact(path) except (FileNotFoundError, IsADirectoryError) as exc: - # Caller-input error: the path is missing or refers to a - # directory. Per docs/standards/error-handling.md and the - # public exit-code contract in - # docs/reference/verify_annex_iv_subcommand.md, this is - # exit 1 (config / caller error). _output_error_and_exit( output_format, f"Annex IV artifact not found or not a regular file: {path!r} ({exc.__class__.__name__}).", EXIT_CONFIG_ERROR, ) except json.JSONDecodeError as exc: - # Validation failure: the artifact is reachable but not parseable - # as JSON. Operator-actionable input error per the public - # contract; routes to exit 1. _output_error_and_exit( output_format, f"Annex IV artifact at {path!r} is not valid JSON: {exc.msg} (line {exc.lineno}).", EXIT_CONFIG_ERROR, ) except OSError as exc: - # Genuine runtime I/O failure on a reachable path (permission - # denied, mid-read I/O error, locked file, etc.). This is the - # post-FileNotFoundError catch-all because Python's OSError - # hierarchy puts FileNotFoundError as a subclass — order - # matters. _output_error_and_exit( output_format, f"Could not read Annex IV artifact {path!r}: {exc}.", EXIT_TRAINING_ERROR, ) + +def _print_artefact_result(result: "VerifyAnnexIVResult", path: str, output_format: str) -> None: + """Render the per-artefact verify result to stdout (JSON or text).""" payload = result.to_dict() payload["path"] = os.path.abspath(path) if output_format == "json": print(json.dumps({"success": result.valid, **payload}, indent=2)) - else: - if result.valid: - print(f"OK: {path}") - print(f" {result.reason}") - else: - print(f"FAIL: {path}") - print(f" {result.reason}") - for missing in result.missing_fields: - print(f" - missing: {missing}") + return + if result.valid: + print(f"OK: {path}") + print(f" {result.reason}") + return + print(f"FAIL: {path}") + print(f" {result.reason}") + for missing in result.missing_fields: + print(f" - missing: {missing}") + + +def _run_verify_annex_iv_cmd(args, output_format: str) -> None: + """Top-level dispatcher for ``forgelm verify-annex-iv ``. + + Two modes: + + - **Single artefact** (default): ```` is an Annex IV JSON file + and the verifier checks field completeness + manifest hash. + - **Pipeline** (``--pipeline`` flag): ```` is a pipeline run + directory and the verifier reads + ``/compliance/pipeline_manifest.json`` and runs chain- + integrity + stage-index + ``stopped_at`` coherence + per-stage + training_manifest existence checks. Returns a list of violations + and exits 0 only when the list is empty. + """ + path = getattr(args, "path", None) + if not path: + _output_error_and_exit( + output_format, + "verify-annex-iv requires a path argument: `forgelm verify-annex-iv `.", + EXIT_CONFIG_ERROR, + ) + + # Phase 14: ``--pipeline`` mode validates the chain-level manifest. + if getattr(args, "pipeline", False): + _run_pipeline_mode(path, output_format) + + # Single-artefact path: I/O errors map to documented exit codes via + # the helper (which sys.exits on failure); on success we render the + # result and exit with the artefact's validity verdict. + result = _verify_artefact_and_handle_io_errors(path, output_format) + _print_artefact_result(result, path, output_format) sys.exit(EXIT_SUCCESS if result.valid else EXIT_CONFIG_ERROR) diff --git a/forgelm/compliance.py b/forgelm/compliance.py index 873fabe4..8e4d3e5d 100644 --- a/forgelm/compliance.py +++ b/forgelm/compliance.py @@ -1041,6 +1041,168 @@ def compute_annex_iv_manifest_hash(artifact: Dict[str, Any]) -> str: return _hashlib.sha256(canonical.encode("utf-8")).hexdigest() +# --------------------------------------------------------------------------- +# Phase 14 — Pipeline-level Annex IV manifest +# --------------------------------------------------------------------------- +# +# The canonical ``generate_pipeline_manifest`` + ``export_pipeline_manifest`` +# definitions live further down in this module, alongside the +# ``_provider_metadata_from_config`` helper and ``verify_pipeline_manifest`` +# disk-bound verifier. Only the structural / chain-integrity helper +# (``_verify_manifest_payload``) is declared here; the rest of the +# pipeline-manifest surface clusters near the verifier so reviewers can +# eyeball the schema and its validator together. + + +_PIPELINE_MANIFEST_REQUIRED_KEYS = ( + "forgelm_version", + "pipeline_run_id", + "pipeline_config_hash", + "started_at", + "final_status", + "stages", +) + + +def _check_chain_link(idx: int, stage: Dict[str, Any], stages: List[Dict[str, Any]]) -> Optional[str]: + """Single-stage chain-integrity check. + + Returns a violation string when stage *idx* (claiming + ``input_source == "chain"``) cannot be validated against its + immediate predecessor's ``output_model``; ``None`` when the link + is intact (or when the stage's ``input_source`` is not ``chain`` + in which case the caller should never have invoked this). + Extracted from :func:`_verify_manifest_payload` for Sonar + python:S3776 cognitive-complexity hygiene. + """ + if idx == 0: + return f"Stage {stage.get('name')!r}: input_source='chain' but no previous stage exists (stage 0 cannot chain)." + prev = stages[idx - 1] + prev_output_model = prev.get("output_model") + if not prev_output_model: + return ( + f"chain_integrity_violation at stage {stage.get('name')!r}: claims " + f"input_source='chain' but previous stage {prev.get('name')!r} has " + f"no output_model (status={prev.get('status')!r})." + ) + if stage.get("input_model") != prev_output_model: + return ( + f"chain_integrity_violation at stage {stage.get('name')!r}: input_model={stage.get('input_model')!r} " + f"≠ previous output_model={prev_output_model!r}" + ) + return None + + +def _check_status_consistency(manifest: Dict[str, Any], stages: List[Dict[str, Any]]) -> List[str]: + """Status-consistency checks for a finalised pipeline manifest. + + Covers (a) ``stopped_at`` referent existence + expected status and + (b) the no-``running``-stages-on-a-finalised-manifest rule (F-N-2). + Extracted from :func:`_verify_manifest_payload` for Sonar + python:S3776 cognitive-complexity hygiene. + """ + violations: List[str] = [] + stopped_at = manifest.get("stopped_at") + if stopped_at is not None: + matching = [s for s in stages if s.get("name") == stopped_at] + if not matching: + violations.append(f"stopped_at refers to unknown stage {stopped_at!r}") + elif matching[0].get("status") not in ("failed", "gated_pending_approval"): + violations.append( + f"stopped_at stage {stopped_at!r} has status {matching[0].get('status')!r} " + f"(expected `failed` or `gated_pending_approval`)" + ) + + # Running-stage consistency (N-2): a finalised manifest with a stage + # still in ``running`` indicates the orchestrator crashed mid-stage. + final_status = manifest.get("final_status") + if final_status and final_status != "in_progress": + running_stages = [s.get("name") for s in stages if s.get("status") == "running"] + if running_stages: + violations.append( + f"stage(s) {running_stages!r} still in `running` status on a " + f"finalised manifest (final_status={final_status!r}); the " + f"orchestrator likely crashed mid-stage without updating state." + ) + return violations + + +def _verify_manifest_payload(manifest: Dict[str, Any]) -> List[str]: + """Validate a pipeline manifest's structural + chain-integrity rules. + + Returns a list of human-readable violation strings (empty list ⇒ + manifest is valid). Used by ``forgelm verify-annex-iv --pipeline + `` to surface integrity issues to operators / regulators. + + Checks: + + 1. **Required top-level keys** are present and of the right shape. + 2. **Chain integrity** — for every stage N with + ``input_source == "chain"``, the *immediate* previous stage's + ``output_model`` must match its ``input_model``. If the previous + stage has no ``output_model`` (e.g. failed before saving) the + chain link is unreconstructible and the verifier flags it as a + ``chain_integrity_violation`` (Phase 14 review F-B-3 hardening: + pre-fix the verifier walked across stages that *had* an + output_model, silently accepting a manifest whose chain could + not actually be reconstructed). Stages with + ``input_source != "chain"`` (``root`` / ``stage_explicit`` / + ``cli_override``) intentionally break the chain — by design, + reviewers inspect the audit log to validate them. + 3. **Status consistency** — at most one ``stopped_at`` stage; if + set, that stage's status must be one of ``failed`` / + ``gated_pending_approval``. Additionally, a finalised manifest + (``final_status != "in_progress"``) must not carry any stage + still in ``running`` status — that indicates a process crash + mid-stage that the archive must surface (Phase 14 review F-N-2). + 4. **Index monotonicity** — stage indices form 0..N-1 in order. + """ + violations: List[str] = [] + + for key in _PIPELINE_MANIFEST_REQUIRED_KEYS: + if key not in manifest: + violations.append(f"missing required top-level key: {key!r}") + + stages = manifest.get("stages", []) + if not isinstance(stages, list): + violations.append("`stages` must be a list") + return violations + + # Phase 14 review-response: per-item type-guard. A tampered + # manifest (``stages: [null, "foo"]``) would otherwise raise + # ``AttributeError`` on ``s.get(...)`` partway through the + # verifier. Surface as a structured violation so the disk wrapper + # (and any future caller) gets a clean list back. We filter the + # malformed items out of the rest of the checks because the + # downstream helpers also expect dicts. + well_formed_stages: List[Dict[str, Any]] = [] + for idx, s in enumerate(stages): + if not isinstance(s, dict): + violations.append(f"stage at index {idx} is not an object (got {type(s).__name__})") + continue + well_formed_stages.append(s) + + # Index monotonicity (on the well-formed subset; mis-indexing of + # a malformed entry would be a noise violation on top of the + # already-reported type error). + for expected, s in enumerate(well_formed_stages): + if s.get("index") != expected: + violations.append(f"stage index out of order at position {expected}: got {s.get('index')!r}") + + # Chain integrity (one check per chain-stage; helper is pure). + for idx, s in enumerate(well_formed_stages): + if s.get("input_source") != "chain": + continue + link_violation = _check_chain_link(idx, s, well_formed_stages) + if link_violation is not None: + violations.append(link_violation) + + # Status consistency (stopped_at + running-on-finalised). + violations.extend(_check_status_consistency(manifest, well_formed_stages)) + + return violations + + # --------------------------------------------------------------------------- # Export: All Compliance Artifacts # --------------------------------------------------------------------------- @@ -1484,3 +1646,193 @@ def verify_audit_log( return manifest_failure return chain_result + + +# --------------------------------------------------------------------------- +# Phase 14 — Pipeline manifest (chain-level Annex IV artefact) +# --------------------------------------------------------------------------- +# +# The pipeline manifest is the *index* over a multi-stage training run. +# Per-stage ``training_manifest.json`` files (produced by +# :func:`generate_training_manifest` + :func:`export_compliance_artifacts`) +# remain individually valid against the existing single-stage Annex IV +# schema; the pipeline manifest ties them together at the chain level so +# auditors can verify both the per-stage evidence AND the chain +# integrity that connects the records. +# +# Lives in ``compliance.py`` (alongside the single-stage manifest) so +# Annex IV schema decisions live in one module. The orchestrator +# imports these functions from here and never touches the schema +# directly. + + +def _provider_metadata_from_config(root_cfg: Any) -> Dict[str, Any]: + """Extract the ``annex_iv`` and ``risk_assessment`` provider metadata + from a root :class:`ForgeConfig`, in the shape the pipeline manifest + embeds verbatim. + + Defensive — both blocks are optional; an absent block produces an + absent key in the returned dict rather than a half-populated record. + """ + payload: Dict[str, Any] = {} + comp_cfg = getattr(root_cfg, "compliance", None) + if comp_cfg: + payload["annex_iv"] = { + "provider_name": comp_cfg.provider_name, + "provider_contact": comp_cfg.provider_contact, + "system_name": comp_cfg.system_name, + "intended_purpose": comp_cfg.intended_purpose, + "known_limitations": comp_cfg.known_limitations, + "system_version": comp_cfg.system_version, + "risk_classification": comp_cfg.risk_classification, + } + risk_cfg = getattr(root_cfg, "risk_assessment", None) + if risk_cfg: + payload["risk_assessment"] = { + "intended_use": risk_cfg.intended_use, + "foreseeable_misuse": risk_cfg.foreseeable_misuse, + "risk_category": risk_cfg.risk_category, + "mitigation_measures": risk_cfg.mitigation_measures, + "vulnerable_groups_considered": risk_cfg.vulnerable_groups_considered, + } + return payload + + +def generate_pipeline_manifest(state: Any, root_cfg: Any) -> Dict[str, Any]: + """Build the chain-level Annex IV manifest. + + Accepts the orchestrator's :class:`PipelineState` (any object + exposing the same field names — duck-typed to avoid a circular + import from :mod:`forgelm.cli._pipeline`) and the root + :class:`ForgeConfig`. Returns a JSON-serialisable dict matching the + schema documented in + ``docs/roadmap/phase-14-pipeline-chains.md`` Task 3. + + The per-stage rows are taken from ``state.stages`` verbatim — the + on-disk state file and the manifest carry the same stage payload, so + a reviewer can correlate the two without translating. Provider / + risk metadata is copied from the root config (immutable across the + pipeline run by the per-stage inheritance matrix). + """ + stages_payload: List[Dict[str, Any]] = [] + for s in state.stages: + stages_payload.append( + { + "name": s.name, + "index": s.index, + "trainer_type": s.trainer_type, + "status": s.status, + "input_model": s.input_model, + "input_source": s.input_source, + "output_model": s.output_model, + "started_at": s.started_at, + "finished_at": s.finished_at, + "duration_seconds": s.duration_seconds, + "training_manifest": s.training_manifest, + "metrics": dict(s.metrics or {}), + "gate_decision": s.gate_decision, + "auto_revert_triggered": bool(s.auto_revert_triggered), + "skipped_reason": s.skipped_reason, + "exit_code": s.exit_code, + "error": s.error, + } + ) + + manifest: Dict[str, Any] = { + "forgelm_version": state.forgelm_version, + "generated_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "pipeline_run_id": state.pipeline_run_id, + "pipeline_config_hash": state.pipeline_config_hash, + "started_at": state.started_at, + "finished_at": state.finished_at, + "final_status": state.final_status, + "stopped_at": state.stopped_at, + "stages": stages_payload, + } + manifest.update(_provider_metadata_from_config(root_cfg)) + return manifest + + +def export_pipeline_manifest(manifest: Dict[str, Any], pipeline_output_dir: str) -> str: + """Write *manifest* to ``/compliance/pipeline_manifest.json``. + + Atomic write via tmp file + ``os.replace`` so a partial write on + SIGKILL leaves the previous valid manifest intact — the orchestrator + calls this on every stage transition; an interrupted transition must + not corrupt the artefact. + """ + target_dir = os.path.join(pipeline_output_dir, "compliance") + os.makedirs(target_dir, exist_ok=True) + target_path = os.path.join(target_dir, "pipeline_manifest.json") + tmp_path = target_path + ".tmp" + with open(tmp_path, "w", encoding="utf-8") as f: + json.dump(manifest, f, indent=2, default=str) + os.replace(tmp_path, target_path) + return target_path + + +def verify_pipeline_manifest(manifest: Dict[str, Any]) -> List[str]: + """Public Annex IV verifier for a pipeline manifest *dict*. + + Returns a list of human-readable violation strings; an empty list + means the manifest is structurally and chain-consistently valid. + See :func:`_verify_manifest_payload` for the complete check matrix. + + Library callers (test harnesses, internal integrity checks) consume + this directly; the CLI command ``forgelm verify-annex-iv --pipeline + `` reads the file via :func:`verify_pipeline_manifest_at_path`, + which adds the disk-bound checks (per-stage training_manifest + existence) on top. + """ + return _verify_manifest_payload(manifest) + + +def verify_pipeline_manifest_at_path(pipeline_dir: str) -> List[str]: + """Disk-backed wrapper around :func:`verify_pipeline_manifest`. + + Reads ``/compliance/pipeline_manifest.json``, runs the + in-memory verifier on the parsed payload, then layers on disk-only + checks (per-stage ``training_manifest`` file existence). Pre-flight + failures (missing manifest file, malformed JSON) surface as a single- + entry violation list so the CLI's exit-code mapping is uniform. + """ + manifest_path = os.path.join(pipeline_dir, "compliance", "pipeline_manifest.json") + if not os.path.isfile(manifest_path): + return [f"pipeline_manifest.json not found at {manifest_path}"] + # Phase 14 post-release review: separate the two failure modes via + # distinct sentinel prefixes so the CLI can map them to the right + # exit code (mirrors the single-artifact verifier in + # _verify_annex_iv.py — FileNotFoundError / JSONDecodeError → + # EXIT_CONFIG_ERROR (1) (operator-actionable input), OSError → + # EXIT_TRAINING_ERROR (2) (genuine runtime I/O failure on a + # reachable path)). + try: + with open(manifest_path, "r", encoding="utf-8") as f: + manifest = json.load(f) + except json.JSONDecodeError as e: + return [f"pipeline_manifest.json invalid JSON: {e}"] + except OSError as e: + return [f"pipeline_manifest.json unreadable: {e}"] + + violations: List[str] = list(_verify_manifest_payload(manifest)) + + # Disk-only check: each completed stage's training_manifest must + # exist. The in-memory verifier cannot see the filesystem. + # Phase 14 review-response: type-guard each stage entry so a + # tampered manifest (``stages: [null, "foo"]``) surfaces as a + # violation rather than raising ``AttributeError`` on ``s.get``. + raw_stages = manifest.get("stages") + if not isinstance(raw_stages, list): + # In-memory verifier already flagged this; nothing more to + # check on disk. + return violations + for idx, s in enumerate(raw_stages): + if not isinstance(s, dict): + violations.append(f"stage at index {idx} is not an object (got {type(s).__name__})") + continue + name = s.get("name", "") + per_stage_manifest = s.get("training_manifest") + if per_stage_manifest and s.get("status") == "completed" and not os.path.isfile(per_stage_manifest): + violations.append(f"Stage {name!r}: training_manifest at {per_stage_manifest!r} is missing.") + + return violations diff --git a/forgelm/config.py b/forgelm/config.py index 341afe4b..a62d2a04 100644 --- a/forgelm/config.py +++ b/forgelm/config.py @@ -847,6 +847,16 @@ class ForgeConfig(BaseModel): default=None, description="Phase 21 / GDPR Article 5(1)(e) storage limitation + Article 17 erasure horizons block.", ) + pipeline: Optional["PipelineConfig"] = Field( + default=None, + description=( + "Phase 14 — multi-stage training pipeline chain (e.g. SFT → DPO → " + "GRPO). When present, the orchestrator at " + "``forgelm/cli/_pipeline.py`` runs each stage sequentially with " + "section-wholesale inheritance from this root config. When absent, " + "the run path is byte-identical to v0.6.0." + ), + ) def _warn_general_consistency(self) -> None: """Emit warnings for the broad cross-field config inconsistencies.""" @@ -1191,6 +1201,265 @@ class ConfigError(Exception): pass +# --------------------------------------------------------------------------- +# Phase 14 — Multi-Stage Training Pipeline Chains +# --------------------------------------------------------------------------- +# +# A ``pipeline`` block at the root level chains 2+ training stages (e.g. +# SFT → DPO → GRPO) into one config-driven run. See +# ``docs/roadmap/phase-14-pipeline-chains.md`` for the full design spec +# (inheritance matrix, CLI semantics, audit-log events, Annex IV manifest). +# +# Layering rule: the pipeline orchestrator (``forgelm/cli/_pipeline.py``) +# produces one flat :class:`ForgeConfig` per stage via +# :func:`merge_pipeline_stage_config`, then hands that to a fresh +# :class:`ForgeTrainer`. ``forgelm/trainer.py`` is **not** aware of the +# pipeline layer — single-stage runs remain byte-identical to v0.6.0. + + +_STAGE_NAME_PATTERN = r"^[a-z0-9_]{1,32}$" +"""Regex enforcing audit-safe pipeline stage names. + +The name is used in CLI flags (``--stage ``, ``--resume-from ``), +audit-log fields, state-file keys, and webhook payloads — pinning it to a +narrow lowercase / digit / underscore alphabet removes every escaping +concern downstream. +""" + + +class PipelineStage(BaseModel): + """A single stage in a multi-stage training pipeline. + + Section-wholesale override semantics — if a block is present in the + stage YAML it **fully replaces** the root config's block for that + section; if the block is omitted, the stage inherits the root. No + field-level deep-merge: "if you want to inherit, omit the block; if + you want to override, supply the full block." + + Auto-chaining: when the stage does not supply its own ``model:`` block + the orchestrator sets ``model.name_or_path`` to the *previous* + stage's ``training.output_dir/final_model`` path. Stage 0 still reads + the root's ``model.name_or_path``. An explicit per-stage ``model:`` + block (with its required ``name_or_path``) disables auto-chaining for + that stage (operator escape hatch). + + The pipeline-level concerns (``distributed``, ``webhook``, + ``compliance``, ``risk_assessment``, ``monitoring``, ``retention``, + ``synthetic``, ``merge``, ``auth``) live at the root only and have no + per-stage override field — ``extra="forbid"`` causes Pydantic to + reject any attempt to declare them inside a stage, with the + section-name preserved in the error message. + """ + + model_config = ConfigDict(extra="forbid") + + name: str = Field( + ..., + pattern=_STAGE_NAME_PATTERN, + description=( + "Stage identifier. Must match ``[a-z0-9_]{1,32}`` so it can serve as " + "an identifier in CLI flags, audit-log fields, and state-file keys " + "without escaping. Must be unique within the pipeline." + ), + ) + + # Section-wholesale override slots. ``None`` (the default) means + # "inherit this section from the root config"; a populated value + # replaces the root section in full. + model: Optional[ModelConfig] = Field( + default=None, + description=( + "Per-stage model block override. When None, the stage inherits the " + "root ``model`` block AND auto-chains ``model.name_or_path`` to the " + "previous stage's output path." + ), + ) + lora: Optional[LoraConfigModel] = Field( + default=None, + description="Per-stage LoRA / PEFT block override. None → inherit from root.", + ) + training: Optional[TrainingConfig] = Field( + default=None, + description=( + "Per-stage training block override. None → inherit from root. " + "When present, ``trainer_type`` is required (Pydantic's existing " + "``TrainingConfig`` validation enforces this)." + ), + ) + data: Optional[DataConfig] = Field( + default=None, + description=( + "Per-stage data block override. None → inherit from root. Pipelines " + "rarely reuse the same dataset across stages; supplying an explicit " + "per-stage data block is the common case." + ), + ) + evaluation: Optional[EvaluationConfig] = Field( + default=None, + description=( + "Per-stage evaluation block override. None → inherit from root. " + "Per-stage gates (loss thresholds, auto_revert, safety, judge, human-" + "approval) live here; each stage may independently configure its gate." + ), + ) + + @model_validator(mode="after") + def _training_block_must_set_trainer_type_explicitly(self) -> "PipelineStage": + """When a stage supplies its own ``training:`` block, the YAML + MUST explicitly set ``trainer_type`` (each stage states its + alignment paradigm in the YAML — and therefore in the pipeline + manifest — for audit clarity). Without this guard, + ``TrainingConfig.trainer_type`` would silently default to + ``"sft"`` and a DPO stage's manifest could carry + ``trainer_type: sft`` if the operator forgot the field. + """ + if self.training is None: + return self + if "trainer_type" not in self.training.model_fields_set: + raise ValueError( + f"Pipeline stage {self.name!r}: when a 'training:' block is " + f"supplied per stage, 'trainer_type' must be set explicitly " + f"(each stage states its alignment paradigm for audit clarity)." + ) + return self + + +class PipelineConfig(BaseModel): + """Top-level pipeline block. + + Carries the ordered list of stages (≥ 1) that the orchestrator + executes sequentially. The orchestrator is responsible for chaining + each stage's input model to the previous stage's output and emitting + the ``pipeline.*`` audit events documented in the Phase 14 spec. + """ + + model_config = ConfigDict(extra="forbid") + + output_dir: str = Field( + default="./pipeline_run", + description=( + "Pipeline-level output directory. Hosts the pipeline-level audit " + "log, the pipeline state file (``pipeline_state.json``), and the " + "pipeline manifest (``compliance/pipeline_manifest.json``). " + "Distinct from each stage's ``training.output_dir`` — those keep " + "the per-stage checkpoints + per-stage training manifest." + ), + ) + + stages: List[PipelineStage] = Field( + ..., + min_length=1, + description=( + "Ordered list of training stages. Must contain at least one stage; " + "operators who want a single-stage run should omit the ``pipeline:`` " + "block entirely rather than declaring a 1-element pipeline." + ), + ) + + @field_validator("stages") + @classmethod + def _unique_stage_names(cls, v: List[PipelineStage]) -> List[PipelineStage]: + """Reject duplicate stage names early. + + Duplicate names would make ``--stage `` and + ``--resume-from `` ambiguous, and would corrupt the pipeline + state file (which keys per-stage status on the name). + """ + names = [s.name for s in v] + seen = set() + duplicates: List[str] = [] + for n in names: + if n in seen and n not in duplicates: + duplicates.append(n) + seen.add(n) + if duplicates: + raise ValueError( + "Duplicate pipeline stage name(s): " + + ", ".join(repr(d) for d in duplicates) + + ". Stage names must be unique within a pipeline." + ) + return v + + +def merge_pipeline_stage_config( + root_cfg: "ForgeConfig", + stage: PipelineStage, + *, + prev_output_model: Optional[str] = None, + input_model_override: Optional[str] = None, +) -> "ForgeConfig": + """Materialise a flat :class:`ForgeConfig` for a single pipeline stage. + + Implements the section-wholesale inheritance rule documented in + ``docs/roadmap/phase-14-pipeline-chains.md`` Task 2: + + - For each of ``model`` / ``lora`` / ``training`` / ``data`` / + ``evaluation``: if the stage sets the block, the stage's block + replaces the root's; otherwise the root's block is preserved. + - The pipeline-level sections (``distributed``, ``webhook``, + ``compliance``, ``risk_assessment``, ``monitoring``, ``retention``, + ``synthetic``, ``merge``, ``auth``) are kept from the root unchanged + — they cannot be overridden per stage. + + Auto-chain resolution (in priority order): + + 1. ``input_model_override`` (CLI ``--input-model`` flag) — operator + escape hatch. Set ``model.name_or_path`` to this value, regardless + of whether the stage declared its own ``model:`` block. The + caller is responsible for logging the override. + 2. The stage declared its own ``model:`` block — keep the stage's + ``name_or_path``. No auto-chain. + 3. ``prev_output_model`` is not None — auto-chain. Set + ``model.name_or_path`` to this value (typically + ``/final_model``). + 4. Stage 0 with no override — keep the root's ``model.name_or_path``. + + The returned :class:`ForgeConfig` carries no ``pipeline`` block (it + is stripped during the merge) so a downstream :class:`ForgeTrainer` + sees an ordinary single-stage config and behaves byte-identically to + a v0.6.0 single-stage run. + """ + # ``model_dump(exclude_none=True)`` so we don't materialise inherited + # ``Optional[T] = None`` fields the root never set — the + # ``extra="forbid"`` re-validation below would tolerate them, but + # round-tripping no-op None values inflates the per-stage manifest + # and confuses operators reading the merged config. + base = root_cfg.model_dump(exclude_none=True) + base.pop("pipeline", None) + + if stage.model is not None: + base["model"] = stage.model.model_dump(exclude_none=True) + if stage.lora is not None: + base["lora"] = stage.lora.model_dump(exclude_none=True) + if stage.training is not None: + base["training"] = stage.training.model_dump(exclude_none=True) + if stage.data is not None: + base["data"] = stage.data.model_dump(exclude_none=True) + if stage.evaluation is not None: + base["evaluation"] = stage.evaluation.model_dump(exclude_none=True) + + # Auto-chain resolution. See docstring above for priority order. + if input_model_override is not None: + base["model"]["name_or_path"] = input_model_override + elif stage.model is None and prev_output_model is not None: + # Stage inherited the model block AND a previous output exists — + # the canonical auto-chain case. + base["model"]["name_or_path"] = prev_output_model + # else: keep whatever is already in ``base["model"]["name_or_path"]`` + # (either the stage's explicit value or the root's). + + return ForgeConfig(**base) + + +# Resolve the ``pipeline: Optional["PipelineConfig"]`` forward reference in +# :class:`ForgeConfig` now that :class:`PipelineConfig` is in scope. +# Without this, the first ``ForgeConfig(**yaml_data)`` instantiation that +# carries a populated ``pipeline`` block fails at validation time with a +# ``PydanticUndefinedAnnotation`` because the forward reference was never +# late-bound. +ForgeConfig.model_rebuild() + + def load_config(config_path: str) -> ForgeConfig: """Loads and validates a YAML configuration file.""" if not os.path.exists(config_path): diff --git a/forgelm/webhook.py b/forgelm/webhook.py index cfe7c513..ded6f5db 100644 --- a/forgelm/webhook.py +++ b/forgelm/webhook.py @@ -1,7 +1,7 @@ import json import logging import os -from typing import Dict, Optional +from typing import Any, Dict, Optional from urllib.parse import urlparse import requests @@ -173,7 +173,20 @@ def _send( metrics: Optional[Dict[str, float]] = None, reason: Optional[str] = None, model_path: Optional[str] = None, + **extra: Any, ) -> None: + """Build + post the webhook payload. + + ``**extra`` carries event-specific fields the + ``notify_pipeline_*`` methods forward (stage_count, final_status, + stopped_at, stage_name, …) — Phase 14 review-response fix: pre- + fix the pipeline notifiers passed unknown kwargs to a fixed + signature, the resulting ``TypeError`` was swallowed by the + orchestrator's best-effort try/except, and pipeline webhooks + silently never fired. Extras are merged into the payload under + their original key names so existing Slack / Teams receivers + that pick fields by name keep working. + """ url = self._resolve_url() if not url: return @@ -188,7 +201,7 @@ def _send( # ``model_path`` is included only for ``approval.required`` events; # we add the key unconditionally (even as None) to keep the schema # stable so downstream consumers can rely on its presence. - payload = { + payload: Dict[str, Any] = { "event": event, "run_name": run_name, "status": status, @@ -198,6 +211,15 @@ def _send( # Slack-compatible formatting (receivers can ignore) "attachments": [{"title": title, "text": text, "color": color}], } + # Merge event-specific extras (pipeline.* events carry + # stage_count / final_status / stopped_at / stage_name). Drop + # any extra whose key collides with a base-payload field so the + # contract stays stable; we don't expect collisions in practice + # since the pipeline notifier names are disjoint, but the guard + # makes the merge order explicit. + for key, value in extra.items(): + if key not in payload: + payload[key] = value self._post_payload(url, payload, event) @@ -333,3 +355,105 @@ def notify_reverted(self, run_name: str, reason: str) -> None: color="#ff9900", reason=masked_reason, ) + + # ---------------------------------------------------------------------- + # Phase 14 — pipeline-level notifications + # ---------------------------------------------------------------------- + # + # The pipeline orchestrator drives multi-stage runs and emits its own + # ``pipeline.*`` events alongside (not replacing) the existing per-stage + # ``training.*`` events that each ``ForgeTrainer`` instance still fires. + # Pre-existing Slack / Teams dashboards filtering on ``training.failure`` + # therefore keep working unchanged; pipeline-aware dashboards can + # additionally subscribe to the new ``pipeline.*`` event vocabulary. + + def notify_pipeline_started(self, run_id: str, stage_count: int) -> None: + """Post a "pipeline started" notification. + + Fires once per pipeline run, before any stage executes. Operators + running long chains use this signal to confirm the orchestrator + accepted the config and started the first stage. + """ + if not (self.config and self.config.notify_on_start): + return + self._send( + event="pipeline.started", + run_name=run_id, + status="started", + title=f"Pipeline Started: {run_id}", + text=f"Multi-stage training pipeline began with {stage_count} stage(s).", + color="#0052cc", + stage_count=stage_count, + ) + + def notify_pipeline_completed( + self, + run_id: str, + final_status: str, + stopped_at: Optional[str], + ) -> None: + """Post a "pipeline completed" notification. + + Fires once per pipeline run, after the final stage transition + (success, failure, or revert). ``stopped_at`` names the failing + stage when the chain halted; ``None`` on full success. + + Piggy-backs on ``notify_on_success`` when the pipeline finished + cleanly and on ``notify_on_failure`` when it stopped early — + matches the existing single-stage notification policy so + operators don't see pipeline pings they explicitly silenced. + """ + succeeded = final_status == "completed" + if succeeded: + if not (self.config and self.config.notify_on_success): + return + color = "#36a64f" + title = f"Pipeline Succeeded: {run_id}" + text = "All stages completed successfully." + else: + if not (self.config and self.config.notify_on_failure): + return + color = "#cc0000" + title = f"Pipeline Stopped: {run_id}" + text = f"Pipeline halted at stage {stopped_at!r} with final_status={final_status!r}." + + self._send( + event="pipeline.completed", + run_name=run_id, + status=final_status, + title=title, + text=text, + color=color, + final_status=final_status, + stopped_at=stopped_at, + ) + + def notify_pipeline_reverted(self, run_id: str, stage_name: str, reason: str) -> None: + """Post a "pipeline stage auto-reverted" notification. + + Distinct from ``notify_pipeline_completed(final_status='stopped_at_stage')``: + this fires *at the moment* a stage auto-reverts, before downstream + stages are marked skipped. Operators monitoring a long chain see + the revert event in near-real-time rather than waiting for the + final summary at the end of the run. + + The reason is masked + truncated identically to + :meth:`notify_failure` so a leaked stack trace cannot smuggle + secrets via this path either. + """ + if not (self.config and self.config.notify_on_failure): + return + masked_reason = self._mask_and_truncate_reason(reason) + self._send( + event="pipeline.stage_reverted", + run_name=run_id, + status="reverted", + title=f"Pipeline Stage Reverted: {run_id}", + text=( + f"Stage {stage_name!r} triggered auto-revert; downstream stages " + f"will not run.\n\nReason: {masked_reason}" + ), + color="#ff9900", + stage_name=stage_name, + reason=masked_reason, + ) diff --git a/pyproject.toml b/pyproject.toml index c81d2bc5..07515d55 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -12,7 +12,7 @@ build-backend = "setuptools.build_meta" [project] name = "forgelm" -version = "0.6.1rc1" +version = "0.7.0" description = "Config-driven LLM fine-tuning with safety evaluation, EU AI Act compliance, 6 alignment methods, and one-command bundled quickstart templates." readme = "README.md" requires-python = ">=3.10" diff --git a/site/compliance.html b/site/compliance.html index 1446a016..46d21160 100644 --- a/site/compliance.html +++ b/site/compliance.html @@ -48,7 +48,7 @@ "downloadUrl": "https://pypi.org/project/forgelm/", "operatingSystem": "Linux, macOS, Windows", "license": "https://www.apache.org/licenses/LICENSE-2.0", - "softwareVersion": "0.6.0", + "softwareVersion": "0.7.0", "offers": { "@type": "Offer", "price": "0", diff --git a/site/contact.html b/site/contact.html index 6feb520b..728fc04e 100644 --- a/site/contact.html +++ b/site/contact.html @@ -48,7 +48,7 @@ "downloadUrl": "https://pypi.org/project/forgelm/", "operatingSystem": "Linux, macOS, Windows", "license": "https://www.apache.org/licenses/LICENSE-2.0", - "softwareVersion": "0.6.0", + "softwareVersion": "0.7.0", "offers": { "@type": "Offer", "price": "0", diff --git a/site/features.html b/site/features.html index 26415b0c..0d365f71 100644 --- a/site/features.html +++ b/site/features.html @@ -48,7 +48,7 @@ "downloadUrl": "https://pypi.org/project/forgelm/", "operatingSystem": "Linux, macOS, Windows", "license": "https://www.apache.org/licenses/LICENSE-2.0", - "softwareVersion": "0.6.0", + "softwareVersion": "0.7.0", "offers": { "@type": "Offer", "price": "0", @@ -127,7 +127,7 @@

Features grouped by lifecycle

-
+
@@ -165,6 +165,11 @@

ORPO

GRPO

Group Relative Policy Optimization for reasoning RL.

+
+
+

Multi-Stage Pipelines

+

Chain SFT → DPO → GRPO in one config-driven run. Auto-chained inputs, per-stage gates, crash-safe resume, one Annex IV manifest covering the whole chain.

+

QLoRA + DoRA

diff --git a/site/guide.html b/site/guide.html index 1810e2fe..2cd5723d 100644 --- a/site/guide.html +++ b/site/guide.html @@ -54,7 +54,7 @@ "downloadUrl": "https://pypi.org/project/forgelm/", "operatingSystem": "Linux, macOS, Windows", "license": "https://www.apache.org/licenses/LICENSE-2.0", - "softwareVersion": "0.6.0", + "softwareVersion": "0.7.0", "offers": { "@type": "Offer", "price": "0", diff --git a/site/index.html b/site/index.html index b1e55f22..b667b020 100644 --- a/site/index.html +++ b/site/index.html @@ -48,7 +48,7 @@ "downloadUrl": "https://pypi.org/project/forgelm/", "operatingSystem": "Linux, macOS, Windows", "license": "https://www.apache.org/licenses/LICENSE-2.0", - "softwareVersion": "0.6.0", + "softwareVersion": "0.7.0", "offers": { "@type": "Offer", "price": "0", @@ -144,7 +144,7 @@

ForgeLM — Config-driven LLM fi
- Open source · Apache 2.0 · v0.6.0 + Open source · Apache 2.0 · v0.7.0
@@ -193,6 +193,20 @@

+ +
@@ -206,6 +220,9 @@

-
$ pip install forgelm==0.6.0
+            
$ pip install forgelm==0.7.0
 
 # or with extras (pick what you need)
-$ pip install 'forgelm[ingestion]==0.6.0'          # PDF/DOCX/EPUB ingest
-$ pip install 'forgelm[ingestion-scale]==0.6.0'    # MinHash LSH for >50K rows
-$ pip install 'forgelm[export]==0.6.0'             # GGUF export
+$ pip install 'forgelm[ingestion]==0.7.0'          # PDF/DOCX/EPUB ingest
+$ pip install 'forgelm[ingestion-scale]==0.7.0'    # MinHash LSH for >50K rows
+$ pip install 'forgelm[export]==0.7.0'             # GGUF export
 
 # sanity-check your environment before the first run
 $ forgelm doctor
diff --git a/site/terms.html b/site/terms.html
index ecc59e86..32404e76 100644
--- a/site/terms.html
+++ b/site/terms.html
@@ -48,7 +48,7 @@
   "downloadUrl": "https://pypi.org/project/forgelm/",
   "operatingSystem": "Linux, macOS, Windows",
   "license": "https://www.apache.org/licenses/LICENSE-2.0",
-  "softwareVersion": "0.6.0",
+  "softwareVersion": "0.7.0",
   "offers": {
     "@type": "Offer",
     "price": "0",
diff --git a/tests/fixtures/pipeline/auto_revert_at_stage_2.yaml b/tests/fixtures/pipeline/auto_revert_at_stage_2.yaml
new file mode 100644
index 00000000..038c50df
--- /dev/null
+++ b/tests/fixtures/pipeline/auto_revert_at_stage_2.yaml
@@ -0,0 +1,43 @@
+# Phase 14 fixture — stage 2 carries an `auto_revert` gate with a
+# trivially-impossible `max_acceptable_loss`.  In integration tests the
+# synthetic eval is wired to exceed that threshold, the trainer
+# auto-reverts stage 2's adapters, the orchestrator marks stage 3
+# `skipped_due_to_prior_revert`, and the run exits with
+# EXIT_EVAL_FAILURE (3).
+
+model:
+  name_or_path: "org/base"
+lora: {}
+training:
+  trainer_type: "sft"
+  output_dir: "./out/revert"
+data:
+  dataset_name_or_path: "org/sft"
+
+pipeline:
+  output_dir: "./out/revert"
+  stages:
+    - name: sft_stage
+      training:
+        trainer_type: "sft"
+        output_dir: "./out/revert/stage1"
+      data:
+        dataset_name_or_path: "org/sft_data"
+
+    - name: dpo_stage
+      training:
+        trainer_type: "dpo"
+        output_dir: "./out/revert/stage2"
+      data:
+        dataset_name_or_path: "org/dpo_data"
+      evaluation:
+        auto_revert: true
+        max_acceptable_loss: 0.001
+        baseline_loss: 1.0
+
+    - name: grpo_stage
+      training:
+        trainer_type: "grpo"
+        output_dir: "./out/revert/stage3"
+      data:
+        dataset_name_or_path: "org/math_data"
diff --git a/tests/fixtures/pipeline/gated_pending_approval.yaml b/tests/fixtures/pipeline/gated_pending_approval.yaml
new file mode 100644
index 00000000..eeec4a39
--- /dev/null
+++ b/tests/fixtures/pipeline/gated_pending_approval.yaml
@@ -0,0 +1,33 @@
+# Phase 14 fixture — stage 1 carries `require_human_approval: true`.
+# Orchestrator must exit EXIT_AWAITING_APPROVAL (4); downstream stages
+# must stay `pending` (not `skipped_due_to_prior_revert`) so a
+# subsequent `forgelm approve ` + `--resume-from dpo_stage`
+# picks the chain back up.
+
+model:
+  name_or_path: "org/base"
+lora: {}
+training:
+  trainer_type: "sft"
+  output_dir: "./out/gated"
+data:
+  dataset_name_or_path: "org/sft"
+
+pipeline:
+  output_dir: "./out/gated"
+  stages:
+    - name: sft_stage
+      training:
+        trainer_type: "sft"
+        output_dir: "./out/gated/stage1"
+      data:
+        dataset_name_or_path: "org/sft_data"
+      evaluation:
+        require_human_approval: true
+
+    - name: dpo_stage
+      training:
+        trainer_type: "dpo"
+        output_dir: "./out/gated/stage2"
+      data:
+        dataset_name_or_path: "org/dpo_data"
diff --git a/tests/fixtures/pipeline/inheritance_matrix.yaml b/tests/fixtures/pipeline/inheritance_matrix.yaml
new file mode 100644
index 00000000..c5fb7208
--- /dev/null
+++ b/tests/fixtures/pipeline/inheritance_matrix.yaml
@@ -0,0 +1,60 @@
+# Phase 14 fixture — exercises every row of the inheritance matrix
+# documented in docs/roadmap/phase-14-pipeline-chains.md Task 2:
+# auto-chain, section-wholesale override, root-only sections preserved.
+
+model:
+  name_or_path: "org/root-base"
+lora:
+  r: 8
+  alpha: 16
+training:
+  trainer_type: "sft"
+  output_dir: "./out/inheritance"
+  learning_rate: 2.0e-5
+  num_train_epochs: 3
+data:
+  dataset_name_or_path: "org/root_data"
+compliance:
+  provider_name: "Acme Inc"
+  provider_contact: "compliance@acme.test"
+  system_name: "Inheritance Demo"
+  intended_purpose: "Inheritance-matrix coverage fixture."
+webhook:
+  url: "https://example.test/webhook"
+
+pipeline:
+  output_dir: "./out/inheritance"
+  stages:
+    # Stage 0 — inherits everything from root; no overrides; reads root model.
+    - name: stage_inherit_all
+
+    # Stage 1 — overrides lora wholesale; training inherited from
+    # root (no ``training:`` block here, so the wholesale-inheritance
+    # path applies and the stage gets root's full training block,
+    # including ``trainer_type: sft``, ``output_dir``, ``learning_rate``,
+    # and ``num_train_epochs``).  Auto-chains input model to
+    # stage_inherit_all's final_model.
+    - name: stage_lora_override
+      lora:
+        r: 32
+        alpha: 64
+
+    # Stage 2 — overrides training wholesale (must include trainer_type
+    # explicitly per Phase 14's audit-clarity validator); overrides data.
+    - name: stage_training_override
+      training:
+        trainer_type: "dpo"
+        output_dir: "./out/inheritance/stage2"
+        num_train_epochs: 1
+        dpo_beta: 0.1
+      data:
+        dataset_name_or_path: "org/dpo_data"
+
+    # Stage 3 — explicit model block; disables auto-chain; subsequent
+    # stages still chain from this stage's output.
+    - name: stage_explicit_model
+      model:
+        name_or_path: "meta-llama/Llama-3-8B"
+      training:
+        trainer_type: "grpo"
+        output_dir: "./out/inheritance/stage3"
diff --git a/tests/fixtures/pipeline/invalid_distributed_per_stage.yaml b/tests/fixtures/pipeline/invalid_distributed_per_stage.yaml
new file mode 100644
index 00000000..14e0a420
--- /dev/null
+++ b/tests/fixtures/pipeline/invalid_distributed_per_stage.yaml
@@ -0,0 +1,31 @@
+# Phase 14 fixture — invalid per-stage `distributed:` block.
+# Phase 14 reserves `distributed`, `webhook`, `compliance`,
+# `risk_assessment`, `monitoring`, `retention`, `synthetic`, `merge`,
+# and `auth` for the root level only — `PipelineStage`'s
+# ``extra="forbid"`` rejects them with the offending section name in
+# the Pydantic error.  Config load on this fixture must raise
+# `ConfigError` (mapped to EXIT_CONFIG_ERROR=1 at the CLI surface).
+
+model:
+  name_or_path: "org/base"
+lora: {}
+training:
+  trainer_type: "sft"
+  output_dir: "./out/bad"
+data:
+  dataset_name_or_path: "org/sft"
+
+pipeline:
+  output_dir: "./out/bad"
+  stages:
+    - name: sft_stage
+      training:
+        trainer_type: "sft"
+        output_dir: "./out/bad/stage1"
+      data:
+        dataset_name_or_path: "org/sft_data"
+      # The next key is the violation — distributed config is
+      # pipeline-level only.  Pydantic rejects with the key name.
+      distributed:
+        strategy: "deepspeed"
+        config_path: "./deepspeed_config.json"
diff --git a/tests/fixtures/pipeline/minimal_3_stage.yaml b/tests/fixtures/pipeline/minimal_3_stage.yaml
new file mode 100644
index 00000000..bd215afb
--- /dev/null
+++ b/tests/fixtures/pipeline/minimal_3_stage.yaml
@@ -0,0 +1,42 @@
+# Phase 14 fixture — minimal 3-stage SFT → DPO → GRPO pipeline.
+# Used as the happy-path smoke baseline for parser + orchestrator tests.
+
+model:
+  name_or_path: "org/base-model"
+lora:
+  r: 8
+  alpha: 16
+training:
+  trainer_type: "sft"
+  output_dir: "./out/pipeline_smoke"
+  num_train_epochs: 1
+data:
+  dataset_name_or_path: "org/sft_dataset"
+
+pipeline:
+  output_dir: "./out/pipeline_smoke"
+  stages:
+    - name: sft_stage
+      training:
+        trainer_type: "sft"
+        output_dir: "./out/pipeline_smoke/stage1_sft"
+        num_train_epochs: 3
+      data:
+        dataset_name_or_path: "org/sft_dataset"
+
+    - name: dpo_stage
+      training:
+        trainer_type: "dpo"
+        output_dir: "./out/pipeline_smoke/stage2_dpo"
+        num_train_epochs: 1
+        dpo_beta: 0.1
+      data:
+        dataset_name_or_path: "org/dpo_preferences"
+
+    - name: grpo_stage
+      training:
+        trainer_type: "grpo"
+        output_dir: "./out/pipeline_smoke/stage3_grpo"
+        grpo_num_generations: 4
+      data:
+        dataset_name_or_path: "org/math_prompts"
diff --git a/tests/fixtures/pipeline/stale_state_resume_v1.yaml b/tests/fixtures/pipeline/stale_state_resume_v1.yaml
new file mode 100644
index 00000000..fff21fc3
--- /dev/null
+++ b/tests/fixtures/pipeline/stale_state_resume_v1.yaml
@@ -0,0 +1,33 @@
+# Phase 14 fixture (v1) — used to seed a `pipeline_state.json` whose
+# `pipeline_config_hash` was computed against this exact byte sequence.
+# Paired with `stale_state_resume_v2.yaml`: re-running with v2 against
+# the v1 state file must trigger the stale-config guard (config-hash
+# mismatch) unless `--force-resume` is set.
+
+model:
+  name_or_path: "org/base"
+lora: {}
+training:
+  trainer_type: "sft"
+  output_dir: "./out/stale"
+  learning_rate: 2.0e-5
+data:
+  dataset_name_or_path: "org/sft"
+
+pipeline:
+  output_dir: "./out/stale"
+  stages:
+    - name: sft_stage
+      training:
+        trainer_type: "sft"
+        output_dir: "./out/stale/stage1"
+        learning_rate: 2.0e-5
+      data:
+        dataset_name_or_path: "org/sft_data"
+
+    - name: dpo_stage
+      training:
+        trainer_type: "dpo"
+        output_dir: "./out/stale/stage2"
+      data:
+        dataset_name_or_path: "org/dpo_data"
diff --git a/tests/fixtures/pipeline/stale_state_resume_v2.yaml b/tests/fixtures/pipeline/stale_state_resume_v2.yaml
new file mode 100644
index 00000000..1fdb2398
--- /dev/null
+++ b/tests/fixtures/pipeline/stale_state_resume_v2.yaml
@@ -0,0 +1,34 @@
+# Phase 14 fixture (v2) — paired with `stale_state_resume_v1.yaml` to
+# demonstrate the stale-state guard.  Differs from v1 in one numeric
+# value (learning_rate); the bytes are different so the SHA-256 over
+# the file content differs.  Re-running with this file against a state
+# file produced by v1 must surface the `pipeline_config_hash mismatch`
+# refusal.
+
+model:
+  name_or_path: "org/base"
+lora: {}
+training:
+  trainer_type: "sft"
+  output_dir: "./out/stale"
+  learning_rate: 1.0e-4  # ← differs from v1 (2.0e-5)
+data:
+  dataset_name_or_path: "org/sft"
+
+pipeline:
+  output_dir: "./out/stale"
+  stages:
+    - name: sft_stage
+      training:
+        trainer_type: "sft"
+        output_dir: "./out/stale/stage1"
+        learning_rate: 1.0e-4  # ← differs from v1
+      data:
+        dataset_name_or_path: "org/sft_data"
+
+    - name: dpo_stage
+      training:
+        trainer_type: "dpo"
+        output_dir: "./out/stale/stage2"
+      data:
+        dataset_name_or_path: "org/dpo_data"
diff --git a/tests/test_check_module_size.py b/tests/test_check_module_size.py
index e40de431..f68c5f02 100644
--- a/tests/test_check_module_size.py
+++ b/tests/test_check_module_size.py
@@ -119,9 +119,9 @@ def test_only_comments_is_zero(self, tmp_path: Path):
 
 
 class TestGrandfatheredSet:
-    def test_contains_seven_modules(self):
+    def test_contains_expected_modules(self):
         tool = _load_tool()
-        assert len(tool._GRANDFATHERED_OVER_CEILING) == 7
+        assert len(tool._GRANDFATHERED_OVER_CEILING) == 8
 
     def test_contains_expected_paths(self):
         tool = _load_tool()
@@ -133,6 +133,10 @@ def test_contains_expected_paths(self):
             "forgelm/config.py",
             "forgelm/cli/_parser.py",
             "forgelm/cli/subcommands/_doctor.py",
+            # Phase 14 (v0.7.0): pipeline orchestrator — sub-package
+            # split tracked for v0.7.x alongside the Phase 15 audit
+            # split pattern.
+            "forgelm/cli/_pipeline.py",
         }
         assert set(tool._GRANDFATHERED_OVER_CEILING) == expected
 
@@ -280,6 +284,10 @@ def test_returns_sorted_paths(self, tmp_path: Path):
         "forgelm/config.py",
         "forgelm/cli/_parser.py",
         "forgelm/cli/subcommands/_doctor.py",
+        # Phase 14 (v0.7.0): pipeline orchestrator — must stay in
+        # lockstep with ``_GRANDFATHERED_OVER_CEILING`` in
+        # ``tools/check_module_size.py``.
+        "forgelm/cli/_pipeline.py",
     ],
 )
 def test_grandfathered_module_exists_in_tree(rel_path: str):
diff --git a/tests/test_cli_subcommands.py b/tests/test_cli_subcommands.py
index dd697555..5832da4a 100644
--- a/tests/test_cli_subcommands.py
+++ b/tests/test_cli_subcommands.py
@@ -172,7 +172,10 @@ def test_legacy_data_audit_flag_emits_deprecation_warning_and_audit_event(self,
         evt = legacy_events[0]
         assert evt["flag"] == "--data-audit"
         assert evt["replacement"] == "forgelm audit"
-        assert evt["version"] == "v0.7.0 removal"
+        # v0.7.0 cut-release moved the removal target out one minor to
+        # v0.8.0 (preserves the one-minor warning window per
+        # docs/standards/release.md#deprecation-cadence).
+        assert evt["version"] == "v0.8.0 removal"
 
         # Sanity: the deprecated path still produced the report — the
         # warning is purely informational.
diff --git a/tests/test_pipeline_cli.py b/tests/test_pipeline_cli.py
new file mode 100644
index 00000000..2b63f727
--- /dev/null
+++ b/tests/test_pipeline_cli.py
@@ -0,0 +1,390 @@
+"""Phase 14 — CLI flag interaction + dispatcher tests.
+
+Covers :func:`forgelm.cli._pipeline.run_pipeline_from_args`, the argparse
+glue layer between the top-level parser and
+:class:`forgelm.cli._pipeline.PipelineOrchestrator`.  Focus: flag
+combinatorial rules + dispatch routing.  The orchestrator's own state
+machine has dedicated coverage in ``tests/test_pipeline_orchestrator.py``;
+this file pins the *boundary* between argparse and the orchestrator.
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import types
+from unittest.mock import MagicMock
+
+import pytest
+
+from forgelm.cli._exit_codes import EXIT_CONFIG_ERROR, EXIT_SUCCESS
+from forgelm.cli._pipeline import run_pipeline_from_args
+from forgelm.config import ForgeConfig
+from forgelm.results import TrainResult
+
+
+def _three_stage_cfg(tmp_path):
+    return ForgeConfig(
+        model={"name_or_path": "org/base"},
+        lora={"r": 8},
+        training={"trainer_type": "sft"},
+        data={"dataset_name_or_path": "org/data"},
+        pipeline={
+            "output_dir": str(tmp_path / "pipeline_run"),
+            "stages": [
+                {
+                    "name": "sft_stage",
+                    "training": {"trainer_type": "sft", "output_dir": str(tmp_path / "stage1")},
+                    "data": {"dataset_name_or_path": "org/sft"},
+                },
+                {
+                    "name": "dpo_stage",
+                    "training": {"trainer_type": "dpo", "output_dir": str(tmp_path / "stage2")},
+                    "data": {"dataset_name_or_path": "org/dpo"},
+                },
+                {
+                    "name": "grpo_stage",
+                    "training": {"trainer_type": "grpo", "output_dir": str(tmp_path / "stage3")},
+                    "data": {"dataset_name_or_path": "org/math"},
+                },
+            ],
+        },
+    )
+
+
+def _ns(**overrides):
+    """Build an argparse.Namespace with the orchestrator-relevant fields."""
+    defaults = {
+        "stage": None,
+        "resume_from": None,
+        "force_resume": False,
+        "input_model": None,
+        "output_format": "text",
+        "dry_run": False,
+    }
+    defaults.update(overrides)
+    return argparse.Namespace(**defaults)
+
+
+def _install_fake_trainer(monkeypatch, results):
+    iterator = iter(results)
+    instantiated_configs: list = []
+
+    class _FakeForgeTrainer:
+        def __init__(self, *, model, tokenizer, config, dataset):
+            instantiated_configs.append(config)
+            os.makedirs(os.path.join(config.training.output_dir, "final_model"), exist_ok=True)
+            self.config = config
+
+        def train(self, resume_from_checkpoint=None):
+            try:
+                return next(iterator)
+            except StopIteration:
+                return TrainResult(success=True)
+
+    fake_trainer = types.ModuleType("forgelm.trainer")
+    fake_trainer.ForgeTrainer = _FakeForgeTrainer
+    monkeypatch.setitem(__import__("sys").modules, "forgelm.trainer", fake_trainer)
+
+    fake_model = types.ModuleType("forgelm.model")
+    fake_model.get_model_and_tokenizer = lambda config: (MagicMock(), MagicMock())
+    monkeypatch.setitem(__import__("sys").modules, "forgelm.model", fake_model)
+
+    fake_data = types.ModuleType("forgelm.data")
+    fake_data.prepare_dataset = lambda config, tokenizer: {"train": [{"text": "x"}]}
+    monkeypatch.setitem(__import__("sys").modules, "forgelm.data", fake_data)
+
+    fake_utils = types.ModuleType("forgelm.utils")
+    fake_utils.setup_authentication = lambda token: None
+    monkeypatch.setitem(__import__("sys").modules, "forgelm.utils", fake_utils)
+
+    return instantiated_configs
+
+
+class TestFlagInteractionGuards:
+    """Argparse alone cannot express "X and Y are mutually exclusive
+    *only when both are non-None*"; the dispatcher enforces it."""
+
+    def test_stage_and_resume_from_mutually_exclusive(self, tmp_path):
+        cfg = _three_stage_cfg(tmp_path)
+        args = _ns(stage="sft_stage", resume_from="dpo_stage")
+        code = run_pipeline_from_args(cfg, b"yaml", args)
+        assert code == EXIT_CONFIG_ERROR
+
+    def test_input_model_requires_stage(self, tmp_path):
+        cfg = _three_stage_cfg(tmp_path)
+        args = _ns(input_model="some/path")
+        code = run_pipeline_from_args(cfg, b"yaml", args)
+        assert code == EXIT_CONFIG_ERROR
+
+    def test_empty_input_model_normalised_to_none(self, tmp_path, monkeypatch):
+        """Phase 14 review F-F-2 regression: ``--input-model ""``
+        (empty string) used to slip past the dispatcher's truthy
+        check and propagate through ``merge_pipeline_stage_config``
+        (``is not None`` branch), silently overwriting the auto-chained
+        model path with the empty string.  The orchestrator now
+        normalises a falsy ``--input-model`` value to ``None`` before
+        dispatch so it never reaches the merge helper.
+        """
+        cfg = _three_stage_cfg(tmp_path)
+        # Pre-create stage 1's output so the chain check on stage 2 is
+        # not the failure point — the test asserts the empty override
+        # is normalised, not that the chain is broken.
+        (tmp_path / "stage1" / "final_model").mkdir(parents=True, exist_ok=True)
+        configs_seen = _install_fake_trainer(monkeypatch, [TrainResult(success=True)])
+        args = _ns(stage="dpo_stage", input_model="")
+        code = run_pipeline_from_args(cfg, b"yaml", args)
+        assert code == EXIT_SUCCESS
+        # Defensive guard (Sonar python:S1244 / S6395 Critical,
+        # post-release review): assert the trainer was instantiated
+        # before reaching for ``configs_seen[0]`` below.  If the stage
+        # silently exits ``EXIT_SUCCESS`` *without* invoking
+        # ``_FakeForgeTrainer`` (e.g. a future dispatch-order change
+        # short-circuits the run), the failure should surface as a
+        # clear "no stage ran" message rather than as an opaque
+        # IndexError on the next line.
+        assert configs_seen, (
+            "stage 'dpo_stage' should have instantiated the fake trainer "
+            "exactly once; got an empty configs_seen list — the orchestrator "
+            "exited EXIT_SUCCESS without invoking _FakeForgeTrainer."
+        )
+        # The stage's input_model must come from the auto-chain (stage
+        # 1's on-disk final_model), NOT from the empty string override.
+        assert configs_seen[0].model.name_or_path == str(tmp_path / "stage1" / "final_model")
+
+    def test_no_train_single_stage_flags_rejected_on_pipeline_config(self, tmp_path):
+        """Phase 14 post-release review BLOCKER 1: pipeline dispatch
+        routes BEFORE ``_maybe_run_no_train_mode`` (F-B-1 fix on PR
+        #53), so without an explicit reject ``--fit-check`` /
+        ``--merge`` / ``--generate-data`` / ``--compliance-export`` /
+        ``--benchmark-only`` would silently trigger a full pipeline
+        training run instead of the single-stage operation the
+        operator asked for.  Each unsupported flag must EXIT_CONFIG_ERROR."""
+        cfg = _three_stage_cfg(tmp_path)
+        for flag, value in [
+            ("fit_check", True),
+            ("merge", "./some/checkpoint"),
+            ("generate_data", "./data.jsonl"),
+            ("compliance_export", "./out"),
+            ("benchmark_only", "mmlu"),
+        ]:
+            args = _ns(**{flag: value})
+            code = run_pipeline_from_args(cfg, b"yaml", args)
+            assert code == EXIT_CONFIG_ERROR, f"flag {flag!r} should be rejected on pipeline config"
+
+    def test_force_resume_without_resume_from_is_noop(self, tmp_path, monkeypatch):
+        """``--force-resume`` without ``--resume-from`` is meaningless
+        but not an error — the orchestrator simply ignores it on a
+        fresh run.  We pin this so a future overzealous validator
+        doesn't start rejecting harmless flag combinations."""
+        cfg = _three_stage_cfg(tmp_path)
+        _install_fake_trainer(
+            monkeypatch, [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)]
+        )
+        args = _ns(force_resume=True)
+        code = run_pipeline_from_args(cfg, b"yaml", args)
+        assert code == EXIT_SUCCESS
+
+
+class TestDispatchToDryRun:
+    def test_dry_run_routes_to_orchestrator_dry_run(self, tmp_path, monkeypatch):
+        """``--dry-run`` must take the dry-run branch even when other
+        pipeline flags are set — keeps operators from accidentally
+        kicking off a real run when they intended to validate."""
+        cfg = _three_stage_cfg(tmp_path)
+        _install_fake_trainer(monkeypatch, [])
+        args = _ns(dry_run=True)
+        code = run_pipeline_from_args(cfg, b"yaml", args)
+        assert code == EXIT_SUCCESS
+        # No state / manifest written by dry-run.
+        assert not os.path.exists(tmp_path / "pipeline_run" / "pipeline_state.json")
+
+
+class TestDispatchToRun:
+    def test_full_run_routes_to_orchestrator_run(self, tmp_path, monkeypatch):
+        cfg = _three_stage_cfg(tmp_path)
+        _install_fake_trainer(monkeypatch, [TrainResult(success=True)] * 3)
+        args = _ns()
+        code = run_pipeline_from_args(cfg, b"yaml", args)
+        assert code == EXIT_SUCCESS
+        assert os.path.exists(tmp_path / "pipeline_run" / "pipeline_state.json")
+
+    def test_stage_filter_propagates_to_orchestrator(self, tmp_path, monkeypatch):
+        cfg = _three_stage_cfg(tmp_path)
+        _install_fake_trainer(monkeypatch, [TrainResult(success=True)])
+        args = _ns(stage="sft_stage")
+        code = run_pipeline_from_args(cfg, b"yaml", args)
+        assert code == EXIT_SUCCESS
+
+    def test_unknown_stage_filter_rejected_via_orchestrator(self, tmp_path, monkeypatch):
+        cfg = _three_stage_cfg(tmp_path)
+        _install_fake_trainer(monkeypatch, [])
+        args = _ns(stage="ghost")
+        code = run_pipeline_from_args(cfg, b"yaml", args)
+        assert code == EXIT_CONFIG_ERROR
+
+
+class TestTopLevelDispatchOrdering:
+    """Regression for the Phase 14 review F-B-1: the single-stage
+    ``--dry-run`` path inside ``_maybe_run_no_train_mode`` MUST NOT run
+    before the orchestrator dispatch when the YAML carries a
+    ``pipeline:`` block.  Otherwise ``forgelm --config pipeline.yaml
+    --dry-run`` falls through to the legacy single-stage dry-run summary
+    and the documented per-stage chain-integrity validation never fires.
+
+    Exercises the actual top-level ``main()`` (not just
+    ``run_pipeline_from_args``) by writing a YAML to disk and invoking
+    the CLI in-process via the installed entry point.
+    """
+
+    def test_pipeline_dry_run_routes_to_orchestrator_not_legacy(self, tmp_path, monkeypatch, caplog):
+        import logging
+        import sys as _sys
+
+        import yaml
+
+        yaml_path = tmp_path / "pipeline.yaml"
+        yaml_path.write_text(
+            yaml.safe_dump(
+                {
+                    "model": {"name_or_path": "org/base"},
+                    "lora": {},
+                    "training": {"trainer_type": "sft"},
+                    "data": {"dataset_name_or_path": "org/data"},
+                    "pipeline": {
+                        "output_dir": str(tmp_path / "pipeline_run"),
+                        "stages": [
+                            {
+                                "name": "sft_stage",
+                                "training": {"trainer_type": "sft", "output_dir": str(tmp_path / "s1")},
+                                "data": {"dataset_name_or_path": "org/sft"},
+                            },
+                            {
+                                "name": "dpo_stage",
+                                "training": {"trainer_type": "dpo", "output_dir": str(tmp_path / "s2")},
+                                "data": {"dataset_name_or_path": "org/dpo"},
+                            },
+                        ],
+                    },
+                }
+            )
+        )
+
+        from forgelm.cli._dispatch import main as cli_main
+
+        monkeypatch.setattr(_sys, "argv", ["forgelm", "--config", str(yaml_path), "--dry-run"])
+        with caplog.at_level(logging.INFO, logger="forgelm.pipeline"):
+            with pytest.raises(SystemExit) as exc_info:
+                cli_main()
+        # The orchestrator's dry-run exits 0 on a clean validation; the
+        # legacy single-stage dry-run would also exit 0 here, so the
+        # discriminator below is the orchestrator's INFO log line —
+        # ``Pipeline dry-run OK:  stage(s) validated`` is emitted
+        # only by ``PipelineOrchestrator.dry_run``.  The legacy
+        # single-stage dry-run path lives at ``forgelm.cli._dry_run`` and
+        # emits a different log surface.
+        assert exc_info.value.code == 0
+        orchestrator_log_emitted = any("pipeline dry-run ok" in r.message.lower() for r in caplog.records)
+        assert orchestrator_log_emitted, (
+            "Dispatcher routed --dry-run through the legacy single-stage path "
+            "instead of the pipeline orchestrator (Phase 14 F-B-1 regression). "
+            f"Captured records: {[r.message for r in caplog.records]!r}"
+        )
+
+    def test_pipeline_branch_runs_before_no_train_modes_on_pipeline_config(self, tmp_path, monkeypatch):
+        """Structural assertion: importing ``_dispatch`` and following
+        the code path on a ``config.pipeline is not None`` config must
+        invoke ``run_pipeline_from_args`` strictly before
+        ``_maybe_run_no_train_mode``.  Sentinels track the call order."""
+        import yaml
+
+        from forgelm.cli import _dispatch
+
+        yaml_path = tmp_path / "pipeline.yaml"
+        yaml_path.write_text(
+            yaml.safe_dump(
+                {
+                    "model": {"name_or_path": "org/base"},
+                    "lora": {},
+                    "training": {"trainer_type": "sft"},
+                    "data": {"dataset_name_or_path": "org/data"},
+                    "pipeline": {
+                        "output_dir": str(tmp_path / "pipeline_run"),
+                        "stages": [
+                            {
+                                "name": "sft_stage",
+                                "training": {"trainer_type": "sft", "output_dir": str(tmp_path / "s1")},
+                                "data": {"dataset_name_or_path": "org/sft"},
+                            },
+                        ],
+                    },
+                }
+            )
+        )
+
+        call_order: list[str] = []
+
+        def _fake_maybe_no_train(config, args):
+            call_order.append("no_train_mode")
+
+        def _fake_run_pipeline(config, yaml_bytes, args):
+            call_order.append("pipeline_dispatch")
+            return 0
+
+        monkeypatch.setattr(_dispatch, "_maybe_run_no_train_mode", _fake_maybe_no_train)
+        import forgelm.cli._pipeline as _pipeline_mod
+
+        monkeypatch.setattr(_pipeline_mod, "run_pipeline_from_args", _fake_run_pipeline)
+        # The dispatcher imports run_pipeline_from_args lazily inside
+        # main() — patch the function on the source module so the lazy
+        # import resolves to our fake.
+        import sys
+
+        monkeypatch.setattr(sys, "argv", ["forgelm", "--config", str(yaml_path)])
+        with pytest.raises(SystemExit):
+            _dispatch.main()
+        # Pipeline dispatch must fire; no_train_mode must NOT have run
+        # before it.
+        assert call_order == ["pipeline_dispatch"], (
+            f"Expected pipeline_dispatch first (and only), got {call_order!r}.  "
+            f"Phase 14 F-B-1 regression: legacy no-train path executed before "
+            f"the pipeline branch on a pipeline config."
+        )
+
+    def test_pipeline_only_flags_rejected_on_non_pipeline_config(self, tmp_path, monkeypatch):
+        """Phase 14 post-release review HIGH 5: pipeline-only flags
+        (``--stage`` / ``--resume-from`` / ``--force-resume`` /
+        ``--input-model``) must be rejected with EXIT_CONFIG_ERROR
+        when the config has no ``pipeline:`` block.  Pre-fix they were
+        silently ignored, surprising operators who expected the flag
+        to be load-bearing."""
+        import sys
+
+        import yaml
+
+        from forgelm.cli import _dispatch
+
+        # Single-stage config — no pipeline: block.
+        yaml_path = tmp_path / "single.yaml"
+        yaml_path.write_text(
+            yaml.safe_dump(
+                {
+                    "model": {"name_or_path": "org/base"},
+                    "lora": {"r": 8},
+                    "training": {"trainer_type": "sft", "output_dir": str(tmp_path / "out")},
+                    "data": {"dataset_name_or_path": "org/data"},
+                }
+            )
+        )
+
+        for flag in ("--stage", "--resume-from", "--input-model"):
+            monkeypatch.setattr(
+                sys,
+                "argv",
+                ["forgelm", "--config", str(yaml_path), flag, "dpo_stage"],
+            )
+            with pytest.raises(SystemExit) as exc_info:
+                _dispatch.main()
+            assert exc_info.value.code == EXIT_CONFIG_ERROR, (
+                f"flag {flag!r} should exit EXIT_CONFIG_ERROR on a non-pipeline config; got exit={exc_info.value.code}"
+            )
diff --git a/tests/test_pipeline_compliance.py b/tests/test_pipeline_compliance.py
new file mode 100644
index 00000000..36e8c18d
--- /dev/null
+++ b/tests/test_pipeline_compliance.py
@@ -0,0 +1,604 @@
+"""Phase 14 — Pipeline manifest schema + chain-integrity verifier tests.
+
+Covers :func:`forgelm.compliance.generate_pipeline_manifest` and
+:func:`forgelm.compliance.verify_pipeline_manifest`.  The pipeline
+manifest is the EU AI Act Annex IV chain-of-custody artefact that ties
+the per-stage ``training_manifest.json`` files into one verifiable
+provenance index.
+"""
+
+from __future__ import annotations
+
+from forgelm.cli._pipeline import PipelineStageState, PipelineState
+from forgelm.compliance import _verify_manifest_payload, generate_pipeline_manifest
+from forgelm.config import ForgeConfig
+
+
+def _root_with_compliance() -> ForgeConfig:
+    return ForgeConfig(
+        model={"name_or_path": "org/base"},
+        lora={"r": 8},
+        training={"trainer_type": "sft"},
+        data={"dataset_name_or_path": "org/data"},
+        compliance={
+            "provider_name": "Acme Inc",
+            "provider_contact": "compliance@acme.test",
+            "system_name": "Acme Pipeline System",
+            "intended_purpose": "Customer-service assistant fine-tune",
+            "system_version": "v0.7.0",
+        },
+        pipeline={
+            "stages": [{"name": "sft_stage"}, {"name": "dpo_stage"}, {"name": "grpo_stage"}],
+        },
+    )
+
+
+def _three_stage_state() -> PipelineState:
+    """Build a representative 3-stage state for happy-path schema tests."""
+    s1 = PipelineStageState(
+        name="sft_stage",
+        index=0,
+        trainer_type="sft",
+        status="completed",
+        input_model="org/base",
+        input_source="root",
+        output_model="./out/stage1/final_model",
+        started_at="2026-06-15T12:00:00+00:00",
+        finished_at="2026-06-15T13:00:00+00:00",
+        duration_seconds=3600.0,
+        metrics={"eval_loss": 0.5},
+        gate_decision="passed",
+        exit_code=0,
+    )
+    s2 = PipelineStageState(
+        name="dpo_stage",
+        index=1,
+        trainer_type="dpo",
+        status="completed",
+        input_model="./out/stage1/final_model",
+        input_source="chain",
+        output_model="./out/stage2/final_model",
+        started_at="2026-06-15T13:00:00+00:00",
+        finished_at="2026-06-15T13:45:00+00:00",
+        duration_seconds=2700.0,
+        metrics={"eval_loss": 0.3},
+        gate_decision="passed",
+        exit_code=0,
+    )
+    s3 = PipelineStageState(
+        name="grpo_stage",
+        index=2,
+        trainer_type="grpo",
+        status="completed",
+        input_model="./out/stage2/final_model",
+        input_source="chain",
+        output_model="./out/stage3/final_model",
+        started_at="2026-06-15T13:45:00+00:00",
+        finished_at="2026-06-15T14:30:00+00:00",
+        duration_seconds=2700.0,
+        metrics={"eval_loss": 0.2},
+        gate_decision="passed",
+        exit_code=0,
+    )
+    return PipelineState(
+        pipeline_run_id="pl_2026-06-15_a1b2c3",
+        pipeline_config_hash="sha256:abc",
+        forgelm_version="0.7.0",
+        started_at="2026-06-15T12:00:00+00:00",
+        finished_at="2026-06-15T14:30:00+00:00",
+        final_status="completed",
+        stages=[s1, s2, s3],
+    )
+
+
+# ---------------------------------------------------------------------------
+# generate_pipeline_manifest — schema coverage
+# ---------------------------------------------------------------------------
+
+
+class TestManifestSchema:
+    def test_required_top_level_keys_present(self):
+        manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance())
+        required = {
+            "forgelm_version",
+            "generated_at",
+            "pipeline_run_id",
+            "pipeline_config_hash",
+            "started_at",
+            "finished_at",
+            "final_status",
+            "stages",
+        }
+        assert required.issubset(manifest.keys())
+
+    def test_annex_iv_block_propagated_from_root_compliance(self):
+        manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance())
+        assert "annex_iv" in manifest
+        assert manifest["annex_iv"]["provider_name"] == "Acme Inc"
+        assert manifest["annex_iv"]["system_name"] == "Acme Pipeline System"
+
+    def test_annex_iv_omitted_when_no_compliance_block(self):
+        root = ForgeConfig(
+            model={"name_or_path": "x"},
+            lora={},
+            training={"trainer_type": "sft"},
+            data={"dataset_name_or_path": "y"},
+            pipeline={"stages": [{"name": "s1"}]},
+        )
+        manifest = generate_pipeline_manifest(_three_stage_state(), root)
+        assert "annex_iv" not in manifest
+
+    def test_stage_payload_carries_chain_fields(self):
+        manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance())
+        s1, s2, s3 = manifest["stages"]
+        assert s1["index"] == 0 and s2["index"] == 1 and s3["index"] == 2
+        assert s2["input_model"] == s1["output_model"]
+        assert s3["input_model"] == s2["output_model"]
+        assert all(s["gate_decision"] == "passed" for s in (s1, s2, s3))
+
+    def test_stage_metrics_are_a_plain_dict_in_payload(self):
+        """Manifest must be JSON-serialisable; metrics dict must round-
+        trip through ``json.dumps``."""
+        import json as _json
+
+        manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance())
+        # No round-trip failure.
+        _json.dumps(manifest)
+
+
+# ---------------------------------------------------------------------------
+# verify_pipeline_manifest — chain integrity
+# ---------------------------------------------------------------------------
+
+
+class TestManifestVerification:
+    def test_clean_manifest_passes(self):
+        manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance())
+        assert _verify_manifest_payload(manifest) == []
+
+    def test_missing_required_key_flagged(self):
+        manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance())
+        manifest.pop("pipeline_run_id")
+        violations = _verify_manifest_payload(manifest)
+        assert any("pipeline_run_id" in v for v in violations)
+
+    def test_chain_integrity_violation_flagged(self):
+        """Stage 2's input_model ≠ stage 1's output_model on a chain
+        stage must surface a ``chain_integrity_violation``."""
+        state = _three_stage_state()
+        state.stages[1].input_model = "tampered/value"
+        manifest = generate_pipeline_manifest(state, _root_with_compliance())
+        violations = _verify_manifest_payload(manifest)
+        assert any("chain_integrity_violation" in v for v in violations)
+        assert any("tampered/value" in v for v in violations)
+
+    def test_cli_override_chain_break_still_flagged(self):
+        """Even when ``input_source: cli_override`` legitimately breaks
+        the chain, the verifier still surfaces it (with the same message)
+        so reviewers can correlate against the audit log to decide
+        legitimate vs. corrupt.
+
+        Note: only *chain* stages contribute to the integrity check; an
+        explicit cli_override stage is recorded with ``input_source !=
+        "chain"`` and therefore skipped by design.  This test asserts
+        that contract."""
+        state = _three_stage_state()
+        state.stages[1].input_model = "operator/manual"
+        state.stages[1].input_source = "cli_override"
+        manifest = generate_pipeline_manifest(state, _root_with_compliance())
+        violations = _verify_manifest_payload(manifest)
+        # cli_override stages don't trip the chain check.
+        assert all("chain_integrity_violation" not in v for v in violations)
+
+    def test_index_out_of_order_flagged(self):
+        state = _three_stage_state()
+        state.stages[0], state.stages[1] = state.stages[1], state.stages[0]
+        # Indices stay 0/1 but names are swapped — verifier checks index
+        # vs. positional order.
+        manifest = generate_pipeline_manifest(state, _root_with_compliance())
+        violations = _verify_manifest_payload(manifest)
+        assert any("index out of order" in v for v in violations)
+
+    def test_stopped_at_unknown_stage_flagged(self):
+        state = _three_stage_state()
+        state.stopped_at = "ghost_stage"
+        state.final_status = "stopped_at_stage"
+        manifest = generate_pipeline_manifest(state, _root_with_compliance())
+        violations = _verify_manifest_payload(manifest)
+        assert any("unknown stage" in v and "ghost_stage" in v for v in violations)
+
+    def test_stopped_at_completed_stage_flagged(self):
+        """If ``stopped_at`` points at a stage whose status is
+        ``completed`` rather than ``failed`` / ``gated_pending_approval``,
+        the manifest is internally inconsistent."""
+        state = _three_stage_state()
+        state.stopped_at = "sft_stage"  # which has status=completed
+        state.final_status = "stopped_at_stage"
+        manifest = generate_pipeline_manifest(state, _root_with_compliance())
+        violations = _verify_manifest_payload(manifest)
+        assert any("expected `failed` or `gated_pending_approval`" in v for v in violations)
+
+
+class TestVerifyOnAutoRevertScenario:
+    """End-to-end: build a state where stage 2 failed and stage 3 was
+    skipped; the resulting manifest should verify cleanly (it's a valid
+    record of a real failure, not an integrity violation)."""
+
+    def test_auto_revert_manifest_verifies(self):
+        s1 = PipelineStageState(
+            name="sft_stage",
+            index=0,
+            trainer_type="sft",
+            status="completed",
+            input_model="org/base",
+            input_source="root",
+            output_model="./out/stage1/final_model",
+            exit_code=0,
+        )
+        s2 = PipelineStageState(
+            name="dpo_stage",
+            index=1,
+            trainer_type="dpo",
+            status="failed",
+            input_model="./out/stage1/final_model",
+            input_source="chain",
+            output_model="./out/stage2/final_model",
+            auto_revert_triggered=True,
+            exit_code=3,
+            error="loss regression",
+        )
+        s3 = PipelineStageState(
+            name="grpo_stage",
+            index=2,
+            trainer_type="grpo",
+            status="skipped_due_to_prior_revert",
+            skipped_reason="Stage 'dpo_stage' triggered auto_revert.",
+        )
+        state = PipelineState(
+            pipeline_run_id="pl_x",
+            pipeline_config_hash="sha256:abc",
+            forgelm_version="0.7.0",
+            started_at="2026-06-15T12:00:00+00:00",
+            finished_at="2026-06-15T13:30:00+00:00",
+            final_status="stopped_at_stage",
+            stopped_at="dpo_stage",
+            stages=[s1, s2, s3],
+        )
+        manifest = generate_pipeline_manifest(state, _root_with_compliance())
+        assert _verify_manifest_payload(manifest) == []
+
+
+class TestStrictChainIntegrity:
+    """Phase 14 review F-B-3 + F-N-3 regression: the verifier must
+    compare every chain stage against its **immediate** predecessor,
+    not the most-recent stage that happens to carry an
+    ``output_model``.  Without this, a broken/missing prev output is
+    silently bridged.
+    """
+
+    def test_chain_stage_with_prev_missing_output_flagged(self):
+        """Stage 0 completed normally, stage 1 failed without saving an
+        output, stage 2 claims input_source='chain'.  Pre-fix the
+        verifier compared stage 2 against stage 0's output, masking the
+        gap; the strict check now flags it."""
+        s0 = PipelineStageState(
+            name="s0",
+            index=0,
+            trainer_type="sft",
+            status="completed",
+            input_source="root",
+            output_model="./out/s0/final_model",
+        )
+        s1 = PipelineStageState(
+            name="s1",
+            index=1,
+            trainer_type="dpo",
+            status="failed",
+            input_model="./out/s0/final_model",
+            input_source="chain",
+            output_model=None,  # crashed before save
+        )
+        s2 = PipelineStageState(
+            name="s2",
+            index=2,
+            trainer_type="grpo",
+            status="completed",
+            input_model="./out/s0/final_model",  # plausibly stale
+            input_source="chain",
+            output_model="./out/s2/final_model",
+        )
+        state = PipelineState(
+            pipeline_run_id="pl_x",
+            pipeline_config_hash="sha256:abc",
+            forgelm_version="0.7.0",
+            started_at="2026-06-15T12:00:00+00:00",
+            final_status="stopped_at_stage",
+            stopped_at="s1",
+            stages=[s0, s1, s2],
+        )
+        manifest = generate_pipeline_manifest(state, _root_with_compliance())
+        violations = _verify_manifest_payload(manifest)
+        assert any("chain_integrity_violation" in v and "'s2'" in v for v in violations), (
+            f"Expected stage 's2' to fail chain integrity due to gap; got: {violations!r}"
+        )
+
+    def test_chain_stage_at_index_zero_flagged(self):
+        """Stage 0 cannot have input_source='chain' (there is no
+        previous stage)."""
+        s0 = PipelineStageState(
+            name="s0",
+            index=0,
+            trainer_type="sft",
+            status="completed",
+            input_source="chain",  # wrong — no prev exists
+            input_model="./somewhere",
+            output_model="./out/s0/final_model",
+        )
+        state = PipelineState(
+            pipeline_run_id="pl_x",
+            pipeline_config_hash="sha256:abc",
+            forgelm_version="0.7.0",
+            started_at="2026-06-15T12:00:00+00:00",
+            final_status="completed",
+            stages=[s0],
+        )
+        manifest = generate_pipeline_manifest(state, _root_with_compliance())
+        violations = _verify_manifest_payload(manifest)
+        assert any("'s0'" in v and "stage 0 cannot chain" in v for v in violations)
+
+
+class TestVerifierFlagsRunningOnFinalisedManifest:
+    """Phase 14 review F-N-2: a finalised manifest carrying a stage in
+    ``running`` status is a tell that the orchestrator crashed
+    mid-stage.  The verifier surfaces it so an archival audit catches
+    the orphan."""
+
+    def test_running_stage_with_completed_final_status_flagged(self):
+        s0 = PipelineStageState(
+            name="s0",
+            index=0,
+            trainer_type="sft",
+            status="completed",
+            input_source="root",
+            output_model="./out/s0/final_model",
+        )
+        s1 = PipelineStageState(
+            name="s1",
+            index=1,
+            trainer_type="dpo",
+            status="running",
+            input_source="chain",
+            input_model="./out/s0/final_model",
+        )
+        state = PipelineState(
+            pipeline_run_id="pl_x",
+            pipeline_config_hash="sha256:abc",
+            forgelm_version="0.7.0",
+            started_at="2026-06-15T12:00:00+00:00",
+            final_status="completed",
+            stages=[s0, s1],
+        )
+        manifest = generate_pipeline_manifest(state, _root_with_compliance())
+        violations = _verify_manifest_payload(manifest)
+        assert any("running" in v and "'s1'" in v for v in violations)
+
+    def test_running_stage_with_in_progress_final_status_is_ok(self):
+        """A live run is allowed to carry a ``running`` stage — the
+        verifier only flags ``running`` on a *finalised* manifest."""
+        s0 = PipelineStageState(
+            name="s0",
+            index=0,
+            trainer_type="sft",
+            status="running",
+            input_source="root",
+        )
+        state = PipelineState(
+            pipeline_run_id="pl_x",
+            pipeline_config_hash="sha256:abc",
+            forgelm_version="0.7.0",
+            started_at="2026-06-15T12:00:00+00:00",
+            final_status="in_progress",
+            stages=[s0],
+        )
+        manifest = generate_pipeline_manifest(state, _root_with_compliance())
+        violations = _verify_manifest_payload(manifest)
+        assert all("running" not in v for v in violations)
+
+
+class TestVerifyPipelineManifestAtPath:
+    """Phase 14 review F-N-6: cover the disk-backed wrapper that the
+    CLI ``forgelm verify-annex-iv --pipeline`` actually invokes."""
+
+    def test_missing_manifest_returns_single_violation(self, tmp_path):
+        from forgelm.compliance import verify_pipeline_manifest_at_path
+
+        violations = verify_pipeline_manifest_at_path(str(tmp_path))
+        assert len(violations) == 1
+        assert "pipeline_manifest.json not found" in violations[0]
+
+    def test_malformed_manifest_returns_single_violation(self, tmp_path):
+        from forgelm.compliance import verify_pipeline_manifest_at_path
+
+        manifest_dir = tmp_path / "compliance"
+        manifest_dir.mkdir()
+        (manifest_dir / "pipeline_manifest.json").write_text("{not valid json")
+        violations = verify_pipeline_manifest_at_path(str(tmp_path))
+        assert len(violations) == 1
+        # Phase 14 post-release review: parse failures now use the
+        # distinct ``invalid JSON`` sentinel so the CLI can route them
+        # to EXIT_CONFIG_ERROR (operator-actionable) rather than the
+        # OSError-shaped ``unreadable`` sentinel which routes to
+        # EXIT_TRAINING_ERROR.
+        assert "invalid JSON" in violations[0]
+
+    def test_disk_wrapper_type_guards_non_dict_stage_items(self, tmp_path):
+        """Phase 14 review-response regression: a tampered manifest where
+        ``stages`` contains non-dict items (``null`` / a string / etc.)
+        must surface as a violation, not crash with ``AttributeError``
+        inside the disk-only loop's ``s.get(...)`` calls."""
+        from forgelm.compliance import verify_pipeline_manifest_at_path
+
+        manifest_dir = tmp_path / "compliance"
+        manifest_dir.mkdir()
+        # Build a manifest payload with two malformed stage entries.
+        bad_manifest = {
+            "forgelm_version": "0.7.0",
+            "pipeline_run_id": "pl_x",
+            "pipeline_config_hash": "sha256:abc",
+            "started_at": "2026-06-15T12:00:00+00:00",
+            "final_status": "in_progress",
+            "stages": [
+                None,
+                "this-should-be-a-dict",
+                {"name": "ok", "index": 2, "trainer_type": "sft", "status": "pending"},
+            ],
+        }
+        import json as _json
+
+        (manifest_dir / "pipeline_manifest.json").write_text(_json.dumps(bad_manifest))
+        violations = verify_pipeline_manifest_at_path(str(tmp_path))
+        assert any("stage at index 0 is not an object" in v for v in violations)
+        assert any("stage at index 1 is not an object" in v for v in violations)
+
+    def test_completed_stage_with_missing_training_manifest_flagged(self, tmp_path):
+        """The disk wrapper layers a per-stage training_manifest
+        existence check on top of the in-memory verifier — the
+        difference between the two surfaces.  This test pins that the
+        wrapper actually reaches that branch."""
+        from forgelm.compliance import verify_pipeline_manifest_at_path
+
+        manifest_dir = tmp_path / "compliance"
+        manifest_dir.mkdir()
+        state = _three_stage_state()
+        # Wire training_manifest paths that don't exist on disk so the
+        # disk wrapper's existence check has something to fail against.
+        for s in state.stages:
+            s.training_manifest = str(tmp_path / s.name / "compliance" / "training_manifest.json")
+        manifest = generate_pipeline_manifest(state, _root_with_compliance())
+        import json as _json
+
+        (manifest_dir / "pipeline_manifest.json").write_text(_json.dumps(manifest))
+        violations = verify_pipeline_manifest_at_path(str(tmp_path))
+        assert any("training_manifest" in v and "is missing" in v for v in violations)
+
+
+class TestVerifyAnnexIvPipelineModeExitCodes:
+    """Phase 14 review-response regression: ``forgelm verify-annex-iv
+    --pipeline `` must map I/O failures to ``EXIT_TRAINING_ERROR``
+    (2) and operator-input errors (``not found``, structural / chain-
+    integrity violations) to ``EXIT_CONFIG_ERROR`` (1).  Mirrors the
+    single-artefact path's exit-code policy."""
+
+    def _run(self, tmp_path, args_overrides: dict) -> int:
+        """Invoke ``_run_pipeline_mode`` and capture the ``SystemExit`` code.
+
+        Uses ``pytest.raises(SystemExit)`` rather than a bare
+        ``try / except SystemExit`` (Sonar python:S5754 / pylint
+        ``broad-except``).  The CLI command always ``sys.exit``s, so we
+        expect ``SystemExit`` every invocation — a leak past the
+        context manager would be a real bug worth surfacing.
+        """
+        import argparse as _argparse
+
+        import pytest as _pytest
+
+        from forgelm.cli.subcommands._verify_annex_iv import _run_verify_annex_iv_cmd
+
+        args = _argparse.Namespace(path=str(tmp_path), pipeline=True, **args_overrides)
+        with _pytest.raises(SystemExit) as exc_info:
+            _run_verify_annex_iv_cmd(args, "text")
+        code = exc_info.value.code
+        return int(code) if code is not None else 0
+
+    def test_missing_manifest_exits_config_error(self, tmp_path, capsys):
+        """``not found`` is operator-actionable input → exit 1."""
+        code = self._run(tmp_path, {})
+        assert code == 1  # EXIT_CONFIG_ERROR
+        captured = capsys.readouterr().out
+        assert "FAIL: pipeline manifest" in captured
+        assert "not found" in captured
+
+    def test_invalid_json_manifest_exits_config_error(self, tmp_path, capsys):
+        """Reachable file that fails to parse as JSON → exit 1
+        (operator-actionable input error).  Phase 14 post-release
+        review: previously this conflated JSONDecodeError with
+        runtime I/O failure and routed to EXIT_TRAINING_ERROR (exit 2);
+        now the verifier emits a distinct ``invalid JSON`` sentinel so
+        the CLI maps parse failures to EXIT_CONFIG_ERROR (1) — the
+        operator can regenerate or fix the manifest, this isn't a
+        production-time I/O failure."""
+        manifest_dir = tmp_path / "compliance"
+        manifest_dir.mkdir()
+        (manifest_dir / "pipeline_manifest.json").write_text("{not valid json")
+        code = self._run(tmp_path, {})
+        assert code == 1  # EXIT_CONFIG_ERROR
+        captured = capsys.readouterr().out
+        assert "invalid JSON" in captured
+
+    def test_chain_integrity_violation_exits_config_error(self, tmp_path, capsys):
+        """Structural / chain violations on a readable manifest →
+        exit 1 (operator-fixable config error)."""
+        import json as _json
+
+        manifest_dir = tmp_path / "compliance"
+        manifest_dir.mkdir()
+        bad_chain_manifest = {
+            "forgelm_version": "0.7.0",
+            "pipeline_run_id": "pl_x",
+            "pipeline_config_hash": "sha256:abc",
+            "started_at": "2026-06-15T12:00:00+00:00",
+            "final_status": "completed",
+            "stages": [
+                {
+                    "name": "s0",
+                    "index": 0,
+                    "trainer_type": "sft",
+                    "status": "completed",
+                    "input_source": "root",
+                    "output_model": "./s0/out",
+                },
+                {
+                    "name": "s1",
+                    "index": 1,
+                    "trainer_type": "dpo",
+                    "status": "completed",
+                    "input_source": "chain",
+                    "input_model": "tampered/different/path",  # ≠ s0.output_model
+                    "output_model": "./s1/out",
+                },
+            ],
+        }
+        (manifest_dir / "pipeline_manifest.json").write_text(_json.dumps(bad_chain_manifest))
+        code = self._run(tmp_path, {})
+        assert code == 1  # EXIT_CONFIG_ERROR
+        captured = capsys.readouterr().out
+        assert "chain_integrity_violation" in captured
+
+
+class TestVerifyOnPartialFilterRun:
+    """A ``--stage X`` run produces a manifest where other stages have
+    ``status: skipped_by_filter``; verifier should accept it."""
+
+    def test_partial_filter_manifest_verifies(self):
+        s1 = PipelineStageState(name="s1", index=0, trainer_type="sft", status="skipped_by_filter")
+        s2 = PipelineStageState(
+            name="s2",
+            index=1,
+            trainer_type="dpo",
+            status="completed",
+            input_model="./prev/output",
+            input_source="cli_override",
+            output_model="./out/stage2/final_model",
+            exit_code=0,
+        )
+        s3 = PipelineStageState(name="s3", index=2, trainer_type="grpo", status="skipped_by_filter")
+        state = PipelineState(
+            pipeline_run_id="pl_x",
+            pipeline_config_hash="sha256:abc",
+            forgelm_version="0.7.0",
+            started_at="2026-06-15T12:00:00+00:00",
+            final_status="completed",
+            stages=[s1, s2, s3],
+        )
+        manifest = generate_pipeline_manifest(state, _root_with_compliance())
+        assert _verify_manifest_payload(manifest) == []
diff --git a/tests/test_pipeline_config.py b/tests/test_pipeline_config.py
new file mode 100644
index 00000000..584e519f
--- /dev/null
+++ b/tests/test_pipeline_config.py
@@ -0,0 +1,369 @@
+"""Phase 14 — Pydantic schema + inheritance-merge tests.
+
+Covers:
+
+- :class:`forgelm.config.PipelineStage` field validation (name pattern,
+  ``extra="forbid"`` rejecting pipeline-only sections).
+- :class:`forgelm.config.PipelineConfig` (minimum-1-stage, unique-name
+  validator).
+- :func:`forgelm.config.merge_pipeline_stage_config` — the section-
+  wholesale inheritance rule + auto-chain priority order documented in
+  ``docs/roadmap/phase-14-pipeline-chains.md`` Task 2.
+- Backward compatibility: a config without a ``pipeline:`` block produces
+  ``config.pipeline is None``; an existing single-stage config is
+  byte-identical to v0.6.0 after the schema change.
+"""
+
+from __future__ import annotations
+
+import pytest
+from pydantic import ValidationError
+
+from forgelm.config import (
+    ForgeConfig,
+    PipelineConfig,
+    PipelineStage,
+    merge_pipeline_stage_config,
+)
+
+
+def _root_cfg(**overrides):
+    """Build a minimal valid root ForgeConfig with sensible defaults."""
+    base = {
+        "model": {"name_or_path": "org/base"},
+        "lora": {"r": 8, "alpha": 16},
+        "training": {"trainer_type": "sft", "num_train_epochs": 3, "learning_rate": 2e-5},
+        "data": {"dataset_name_or_path": "org/sft_data"},
+    }
+    base.update(overrides)
+    return ForgeConfig(**base)
+
+
+# ---------------------------------------------------------------------------
+# PipelineStage — name + per-section validation
+# ---------------------------------------------------------------------------
+
+
+class TestPipelineStageName:
+    @pytest.mark.parametrize(
+        "name",
+        ["sft_stage", "stage_1", "s", "a" * 32, "abc_def_123"],
+    )
+    def test_valid_names_accepted(self, name):
+        stage = PipelineStage(name=name)
+        assert stage.name == name
+
+    @pytest.mark.parametrize(
+        "name",
+        [
+            "",  # empty
+            "a" * 33,  # > 32 chars
+            "Stage1",  # uppercase
+            "stage-1",  # hyphen
+            "stage 1",  # space
+            "stage.1",  # dot
+            "stage/1",  # slash
+            "stage@1",  # at-sign
+            "stage_!",  # punctuation
+        ],
+    )
+    def test_invalid_names_rejected(self, name):
+        with pytest.raises(ValidationError):
+            PipelineStage(name=name)
+
+    def test_name_is_required(self):
+        with pytest.raises(ValidationError):
+            PipelineStage()
+
+
+class TestPipelineStageExtraForbid:
+    """Pipeline-only sections (distributed / webhook / compliance / etc.)
+    must not appear inside a stage.  ``extra="forbid"`` makes Pydantic
+    reject them with the offending field name in the error.  This is the
+    primary defence against operators putting root-only config inside a
+    stage by mistake.
+    """
+
+    @pytest.mark.parametrize(
+        "forbidden_section",
+        [
+            "distributed",
+            "webhook",
+            "compliance",
+            "risk_assessment",
+            "monitoring",
+            "retention",
+            "synthetic",
+            "merge",
+            "auth",
+            "pipeline",  # no nested pipelines
+        ],
+    )
+    def test_pipeline_only_section_rejected_in_stage(self, forbidden_section):
+        with pytest.raises(ValidationError) as exc_info:
+            PipelineStage(name="s1", **{forbidden_section: {}})
+        # The forbidden section name appears in the error so the operator
+        # knows which key to remove.
+        assert forbidden_section in str(exc_info.value)
+
+
+class TestPipelineStageOverrides:
+    """All allowed override slots accept their corresponding config block."""
+
+    def test_all_override_slots_default_to_none(self):
+        stage = PipelineStage(name="s1")
+        assert stage.model is None
+        assert stage.lora is None
+        assert stage.training is None
+        assert stage.data is None
+        assert stage.evaluation is None
+
+    def test_model_block_override(self):
+        stage = PipelineStage(name="s1", model={"name_or_path": "org/other"})
+        assert stage.model is not None
+        assert stage.model.name_or_path == "org/other"
+
+    def test_lora_block_override(self):
+        stage = PipelineStage(name="s1", lora={"r": 32, "alpha": 64})
+        assert stage.lora is not None
+        assert stage.lora.r == 32
+
+    def test_training_block_override_requires_trainer_type(self):
+        """``trainer_type`` is required by the existing ``TrainingConfig``
+        schema; a stage's training block must therefore supply it.  This
+        is the Phase 14 spec's "each stage explicitly states its
+        alignment paradigm" rule, enforced via Pydantic's existing
+        validator rather than a duplicate check."""
+        with pytest.raises(ValidationError):
+            PipelineStage(name="s1", training={"num_train_epochs": 1})
+
+    def test_data_block_override(self):
+        stage = PipelineStage(name="s1", data={"dataset_name_or_path": "org/dpo_prefs"})
+        assert stage.data is not None
+        assert stage.data.dataset_name_or_path == "org/dpo_prefs"
+
+    def test_evaluation_block_override(self):
+        stage = PipelineStage(
+            name="s1",
+            evaluation={"auto_revert": True, "max_acceptable_loss": 2.0},
+        )
+        assert stage.evaluation is not None
+        assert stage.evaluation.auto_revert is True
+
+
+# ---------------------------------------------------------------------------
+# PipelineConfig — list validators
+# ---------------------------------------------------------------------------
+
+
+class TestPipelineConfig:
+    def test_minimum_one_stage_required(self):
+        with pytest.raises(ValidationError):
+            PipelineConfig(stages=[])
+
+    def test_single_stage_pipeline_accepted(self):
+        """A 1-stage pipeline is technically valid (the spec only forbids
+        empty pipelines).  Whether operators *should* declare one is a
+        documentation matter, not a schema matter."""
+        pl = PipelineConfig(stages=[PipelineStage(name="only")])
+        assert len(pl.stages) == 1
+
+    def test_multi_stage_pipeline(self):
+        pl = PipelineConfig(
+            stages=[
+                PipelineStage(name="sft_stage"),
+                PipelineStage(name="dpo_stage"),
+                PipelineStage(name="grpo_stage"),
+            ]
+        )
+        assert [s.name for s in pl.stages] == ["sft_stage", "dpo_stage", "grpo_stage"]
+
+    def test_duplicate_stage_names_rejected(self):
+        with pytest.raises(ValidationError) as exc_info:
+            PipelineConfig(
+                stages=[
+                    PipelineStage(name="dup"),
+                    PipelineStage(name="other"),
+                    PipelineStage(name="dup"),
+                ]
+            )
+        assert "Duplicate" in str(exc_info.value)
+        assert "dup" in str(exc_info.value)
+
+    def test_extra_keys_rejected(self):
+        """``extra="forbid"`` blocks typos like ``stagess`` from being
+        silently accepted as ignored fields."""
+        with pytest.raises(ValidationError):
+            PipelineConfig(stages=[PipelineStage(name="s1")], stagess=[])
+
+
+# ---------------------------------------------------------------------------
+# ForgeConfig.pipeline wiring + backward compatibility
+# ---------------------------------------------------------------------------
+
+
+class TestForgeConfigPipelineField:
+    def test_pipeline_defaults_to_none(self):
+        cfg = _root_cfg()
+        assert cfg.pipeline is None
+
+    def test_pipeline_populated_from_yaml_dict(self):
+        cfg = _root_cfg(pipeline={"stages": [{"name": "s1"}, {"name": "s2"}]})
+        assert cfg.pipeline is not None
+        assert len(cfg.pipeline.stages) == 2
+
+    def test_pipeline_section_round_trips_through_model_dump(self):
+        cfg = _root_cfg(pipeline={"stages": [{"name": "s1", "training": {"trainer_type": "dpo"}}]})
+        dumped = cfg.model_dump(exclude_none=True)
+        assert "pipeline" in dumped
+        assert dumped["pipeline"]["stages"][0]["name"] == "s1"
+
+    def test_single_stage_config_byte_identical_without_pipeline(self):
+        """A pre-Phase-14 single-stage config (no ``pipeline:`` key)
+        must produce a ``ForgeConfig`` indistinguishable from v0.6.0
+        for the trainer's purposes — the ``pipeline`` field defaults to
+        None and is excluded from ``model_dump(exclude_none=True)``."""
+        cfg = _root_cfg()
+        dumped = cfg.model_dump(exclude_none=True)
+        assert "pipeline" not in dumped
+
+
+# ---------------------------------------------------------------------------
+# merge_pipeline_stage_config — section-wholesale + auto-chain priority
+# ---------------------------------------------------------------------------
+
+
+class TestMergeSectionWholesale:
+    def test_stage_with_no_overrides_inherits_root_entirely(self):
+        root = _root_cfg()
+        stage = PipelineStage(name="s0")
+        merged = merge_pipeline_stage_config(root, stage, prev_output_model=None)
+        assert merged.model.name_or_path == root.model.name_or_path
+        assert merged.lora.r == root.lora.r
+        assert merged.training.trainer_type == root.training.trainer_type
+        assert merged.training.num_train_epochs == root.training.num_train_epochs
+        assert merged.data.dataset_name_or_path == root.data.dataset_name_or_path
+
+    def test_stage_lora_block_wholesale_replaces_root(self):
+        root = _root_cfg()
+        stage = PipelineStage(name="s1", lora={"r": 64, "alpha": 128})
+        merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model")
+        # The stage's lora block fully replaces — fields the stage didn't
+        # mention fall back to ``LoraConfigModel`` defaults, NOT to the
+        # root's ``lora`` block's values.
+        assert merged.lora.r == 64
+        assert merged.lora.alpha == 128
+
+    def test_stage_training_block_wholesale_replaces_root(self):
+        root = _root_cfg()
+        stage = PipelineStage(name="s1", training={"trainer_type": "dpo", "num_train_epochs": 1})
+        merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model")
+        assert merged.training.trainer_type == "dpo"
+        assert merged.training.num_train_epochs == 1
+
+    def test_stage_data_block_wholesale_replaces_root(self):
+        root = _root_cfg()
+        stage = PipelineStage(name="s1", data={"dataset_name_or_path": "org/dpo_prefs"})
+        merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model")
+        assert merged.data.dataset_name_or_path == "org/dpo_prefs"
+
+    def test_stage_evaluation_block_wholesale_replaces_root(self):
+        root = _root_cfg(evaluation={"auto_revert": False})
+        stage = PipelineStage(name="s1", evaluation={"auto_revert": True, "max_acceptable_loss": 1.5})
+        merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model")
+        assert merged.evaluation is not None
+        assert merged.evaluation.auto_revert is True
+        assert merged.evaluation.max_acceptable_loss == pytest.approx(1.5)
+
+    def test_pipeline_section_stripped_from_merged_config(self):
+        """The orchestrator hands the merged ForgeConfig to a single-stage
+        ``ForgeTrainer`` that has no awareness of pipelines.  The
+        ``pipeline`` block must be absent from the merged config so the
+        trainer's lifecycle is byte-identical to a v0.6.0 single-stage
+        run."""
+        root = _root_cfg(pipeline={"stages": [{"name": "s0"}, {"name": "s1"}]})
+        stage = PipelineStage(name="s0")
+        merged = merge_pipeline_stage_config(root, stage, prev_output_model=None)
+        assert merged.pipeline is None
+
+
+class TestMergeAutoChainPriorityOrder:
+    """The auto-chain resolution rule has four priority levels.  These
+    tests pin every level so a future refactor cannot silently change
+    the order — operators rely on the documented behaviour for
+    ``--input-model`` to actually override an in-config ``model:`` block.
+    """
+
+    def test_priority_1_input_model_override_wins_over_everything(self):
+        root = _root_cfg()
+        stage = PipelineStage(name="s1", model={"name_or_path": "stage/value"})
+        merged = merge_pipeline_stage_config(
+            root,
+            stage,
+            prev_output_model="./prev/model",
+            input_model_override="cli/override",
+        )
+        assert merged.model.name_or_path == "cli/override"
+
+    def test_priority_2_explicit_stage_model_disables_auto_chain(self):
+        root = _root_cfg()
+        stage = PipelineStage(name="s1", model={"name_or_path": "stage/explicit"})
+        merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model")
+        assert merged.model.name_or_path == "stage/explicit"
+
+    def test_priority_3_auto_chain_to_prev_output_when_no_overrides(self):
+        root = _root_cfg()
+        stage = PipelineStage(name="s1")
+        merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/final")
+        assert merged.model.name_or_path == "./prev/final"
+
+    def test_priority_4_stage_zero_inherits_root_model(self):
+        """Stage 0 of a pipeline (or any stage launched standalone with
+        ``--stage `` without a prior output) reads the root's
+        ``model.name_or_path`` unchanged."""
+        root = _root_cfg()
+        stage = PipelineStage(name="s0")
+        merged = merge_pipeline_stage_config(root, stage, prev_output_model=None)
+        assert merged.model.name_or_path == root.model.name_or_path
+
+    def test_input_model_override_beats_stage_zero_root_value(self):
+        """The CLI escape hatch (``--input-model``) must work even on the
+        first stage — operators using ``--stage 
+        --input-model `` to re-run with a different base model."""
+        root = _root_cfg()
+        stage = PipelineStage(name="s0")
+        merged = merge_pipeline_stage_config(
+            root,
+            stage,
+            prev_output_model=None,
+            input_model_override="cli/override",
+        )
+        assert merged.model.name_or_path == "cli/override"
+
+
+class TestMergePreservesRootOnlyBlocks:
+    """The pipeline-level config sections (distributed, webhook,
+    compliance, etc.) cannot be overridden per stage by design.  After
+    merge, those blocks must come through from the root verbatim.
+    """
+
+    def test_root_webhook_survives_merge(self):
+        root = _root_cfg(webhook={"url": "https://example.com/hook"})
+        stage = PipelineStage(name="s1", training={"trainer_type": "dpo"})
+        merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model")
+        assert merged.webhook is not None
+        assert merged.webhook.url == "https://example.com/hook"
+
+    def test_root_compliance_metadata_survives_merge(self):
+        root = _root_cfg(
+            compliance={
+                "provider_name": "Acme Corp",
+                "provider_contact": "compliance@acme.test",
+                "system_name": "Pipeline Demo",
+                "intended_purpose": "Internal eval",
+            }
+        )
+        stage = PipelineStage(name="s1", training={"trainer_type": "dpo"})
+        merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model")
+        assert merged.compliance is not None
+        assert merged.compliance.provider_name == "Acme Corp"
diff --git a/tests/test_pipeline_orchestrator.py b/tests/test_pipeline_orchestrator.py
new file mode 100644
index 00000000..e4298456
--- /dev/null
+++ b/tests/test_pipeline_orchestrator.py
@@ -0,0 +1,1024 @@
+"""Phase 14 — Pipeline orchestrator state-machine tests.
+
+Covers the documented edge cases in ``docs/roadmap/phase-14-pipeline-chains.md``
+Tasks 4 + 5:
+
+- Happy-path 3-stage chain ⇒ all three trainers run, state file +
+  pipeline manifest written, exit 0.
+- Auto-revert at stage 2 ⇒ stage 3 enters ``skipped_due_to_prior_revert``,
+  ``pipeline.stage_reverted`` audit event fires, exit 3.
+- Human-approval gate at stage 1 ⇒ exit 4, downstream stages still
+  ``pending``, state preserved for ``--resume-from``.
+- ``--resume-from `` skips already-completed stages whose output
+  directory still exists; stale-state guard rejects a config-hash
+  mismatch unless ``--force-resume`` is set.
+- ``--stage `` partial-run filters; missing-prev-output produces
+  a clear config error.
+- Dry-run validates every stage without touching a trainer.
+
+The :class:`forgelm.trainer.ForgeTrainer` is replaced via lazy-import
+monkeypatch — the orchestrator only reaches the heavy modules through
+``from ..trainer import ForgeTrainer`` inside ``_run_single_stage``, so
+patching ``sys.modules`` and the dataset / model helpers is enough.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import types
+from unittest.mock import MagicMock, patch
+
+from forgelm.cli._exit_codes import (
+    EXIT_AWAITING_APPROVAL,
+    EXIT_CONFIG_ERROR,
+    EXIT_EVAL_FAILURE,
+    EXIT_SUCCESS,
+)
+from forgelm.cli._pipeline import (
+    PipelineOrchestrator,
+    PipelineStageState,
+    PipelineState,
+    _compute_pipeline_config_hash,
+    _deserialise_state,
+    _generate_run_id,
+    _serialise_state,
+)
+from forgelm.config import ForgeConfig
+from forgelm.results import TrainResult
+
+# ---------------------------------------------------------------------------
+# Shared fixtures
+# ---------------------------------------------------------------------------
+
+
+def _three_stage_config(tmp_path) -> ForgeConfig:
+    """Build a minimal valid 3-stage pipeline ForgeConfig."""
+    return ForgeConfig(
+        model={"name_or_path": "org/base"},
+        lora={"r": 8},
+        training={"trainer_type": "sft", "output_dir": str(tmp_path)},
+        data={"dataset_name_or_path": "org/sft"},
+        pipeline={
+            "output_dir": str(tmp_path / "pipeline_run"),
+            "stages": [
+                {
+                    "name": "sft_stage",
+                    "training": {"trainer_type": "sft", "output_dir": str(tmp_path / "stage1")},
+                    "data": {"dataset_name_or_path": "org/sft_data"},
+                },
+                {
+                    "name": "dpo_stage",
+                    "training": {"trainer_type": "dpo", "output_dir": str(tmp_path / "stage2")},
+                    "data": {"dataset_name_or_path": "org/dpo_data"},
+                },
+                {
+                    "name": "grpo_stage",
+                    "training": {"trainer_type": "grpo", "output_dir": str(tmp_path / "stage3")},
+                    "data": {"dataset_name_or_path": "org/math_data"},
+                },
+            ],
+        },
+    )
+
+
+def _install_trainer_mocks(monkeypatch, train_results):
+    """Inject mock trainer/model/data modules.
+
+    ``train_results`` is a list of :class:`TrainResult` returned in order
+    by successive ``ForgeTrainer.train()`` calls.  The mock also
+    materialises the per-stage output dir on disk so the chain-integrity
+    guard in the orchestrator does not trip when subsequent stages
+    auto-chain from a "fresh" directory.
+    """
+    iterator = iter(train_results)
+    instantiated_configs = []
+
+    class _FakeForgeTrainer:
+        def __init__(self, *, model, tokenizer, config, dataset):
+            instantiated_configs.append(config)
+            # Materialise final_model dir on disk so the next stage's
+            # chain-integrity guard sees the path it auto-chains to.
+            final_dir = os.path.join(config.training.output_dir, "final_model")
+            os.makedirs(final_dir, exist_ok=True)
+            self.config = config
+
+        def train(self, resume_from_checkpoint=None):
+            try:
+                return next(iterator)
+            except StopIteration:
+                return TrainResult(success=True)
+
+    fake_trainer_mod = types.ModuleType("forgelm.trainer")
+    fake_trainer_mod.ForgeTrainer = _FakeForgeTrainer
+    monkeypatch.setitem(__import__("sys").modules, "forgelm.trainer", fake_trainer_mod)
+
+    fake_model_mod = types.ModuleType("forgelm.model")
+    fake_model_mod.get_model_and_tokenizer = lambda config: (MagicMock(), MagicMock())
+    monkeypatch.setitem(__import__("sys").modules, "forgelm.model", fake_model_mod)
+
+    fake_data_mod = types.ModuleType("forgelm.data")
+    fake_data_mod.prepare_dataset = lambda config, tokenizer: {"train": [{"text": "x"}]}
+    monkeypatch.setitem(__import__("sys").modules, "forgelm.data", fake_data_mod)
+
+    fake_utils_mod = types.ModuleType("forgelm.utils")
+    fake_utils_mod.setup_authentication = lambda token: None
+    monkeypatch.setitem(__import__("sys").modules, "forgelm.utils", fake_utils_mod)
+
+    return instantiated_configs
+
+
+# ---------------------------------------------------------------------------
+# Pure helpers
+# ---------------------------------------------------------------------------
+
+
+class TestPipelineConfigHash:
+    def test_deterministic(self):
+        h1 = _compute_pipeline_config_hash(b"some yaml bytes")
+        h2 = _compute_pipeline_config_hash(b"some yaml bytes")
+        assert h1 == h2
+        assert h1.startswith("sha256:")
+
+    def test_different_inputs_different_hashes(self):
+        assert _compute_pipeline_config_hash(b"v1") != _compute_pipeline_config_hash(b"v2")
+
+    def test_whitespace_sensitive(self):
+        """Hashing raw bytes — whitespace / key order / comments all matter.
+
+        This is intentional: regulators audit the on-disk artefact, not
+        the parsed semantic content.  An operator who edits the file
+        between runs gets a new hash even if the YAML is semantically
+        equivalent."""
+        assert _compute_pipeline_config_hash(b"a: 1\nb: 2") != _compute_pipeline_config_hash(b"b: 2\na: 1")
+
+
+class TestRunIdShape:
+    def test_run_id_format(self):
+        run_id = _generate_run_id()
+        # ``pl_YYYY-MM-DD_<6-hex>``
+        assert run_id.startswith("pl_")
+        parts = run_id.split("_")
+        assert len(parts) == 3
+        assert len(parts[1]) == 10  # YYYY-MM-DD
+        assert len(parts[2]) == 6  # 6-hex
+
+    def test_run_id_unique(self):
+        ids = {_generate_run_id() for _ in range(50)}
+        assert len(ids) == 50, "run_id collisions in 50 samples — too narrow randomness?"
+
+
+class TestStateRoundTrip:
+    def test_serialise_deserialise(self):
+        state = PipelineState(
+            pipeline_run_id="pl_test",
+            pipeline_config_hash="sha256:abc",
+            forgelm_version="0.7.0-dev",
+            started_at="2026-06-15T12:00:00+00:00",
+            stages=[
+                PipelineStageState(name="s1", index=0, trainer_type="sft", status="completed"),
+                PipelineStageState(name="s2", index=1, trainer_type="dpo", status="pending"),
+            ],
+        )
+        payload = _serialise_state(state)
+        round_tripped = _deserialise_state(payload)
+        assert round_tripped.pipeline_run_id == state.pipeline_run_id
+        assert len(round_tripped.stages) == 2
+        assert round_tripped.stages[0].name == "s1"
+        assert round_tripped.stages[0].status == "completed"
+
+    def test_deserialise_tolerates_unknown_keys(self):
+        """Future-forward compatibility: a newer manifest schema with
+        extra fields must not crash an older reader."""
+        payload = {
+            "pipeline_run_id": "pl_test",
+            "pipeline_config_hash": "sha256:abc",
+            "forgelm_version": "0.7.0",
+            "started_at": "2026-06-15T12:00:00+00:00",
+            "final_status": "completed",
+            "stages": [
+                {"name": "s1", "index": 0, "trainer_type": "sft", "status": "completed", "future_field": "ignore me"}
+            ],
+            "future_top_level": "also ignore",
+        }
+        state = _deserialise_state(payload)
+        assert state.stages[0].name == "s1"
+
+
+# ---------------------------------------------------------------------------
+# Atomic write helper
+# ---------------------------------------------------------------------------
+
+
+class TestAtomicWriteJson:
+    """Phase 14 review final-round F-S-1 regression: ``_atomic_write_json``
+    must use a per-writer-unique temp filename so two writers targeting
+    the same final path don't truncate each other's tmp mid-write."""
+
+    def test_per_writer_temp_path_does_not_race_on_shared_tmp(self, tmp_path):
+        """Pre-fix, both writers wrote to ``.tmp`` — a 16-thread
+        stress probe produced ``FileNotFoundError`` from ``os.replace``
+        (the loser's tmp was unlinked by the winner's rename).  Post-
+        fix, each writer's tmp is unique so no traceback escapes; the
+        race surfaces as last-writer-wins on the final ``path``."""
+        from concurrent.futures import ThreadPoolExecutor
+
+        from forgelm.cli._pipeline import _atomic_write_json
+
+        target = str(tmp_path / "shared.json")
+        errors: list[Exception] = []
+
+        def writer(i: int) -> None:
+            # OSError covers the FileNotFoundError race that the
+            # shared-tmp implementation produced; pre-fix this branch
+            # captured the escapes for the assertion below.  ``Exception``
+            # is wide-enough to surface any unexpected escape too without
+            # tripping the Sonar ``catch-BaseException`` rule.
+            try:
+                _atomic_write_json(target, {"writer": i, "value": "x" * 1024})
+            except Exception as exc:  # noqa: BLE001 — test probe: record any escape
+                errors.append(exc)
+
+        with ThreadPoolExecutor(max_workers=8) as pool:
+            list(pool.map(writer, range(32)))
+
+        # No writer's tmp-vs-replace race should have escaped.
+        assert errors == [], f"Concurrent writers produced {len(errors)} exception(s): {errors[:3]}"
+        # And the final path holds *some* valid JSON (last writer wins).
+        with open(target) as f:
+            payload = json.load(f)
+        assert isinstance(payload, dict)
+        assert "writer" in payload
+
+    def test_orphan_tmp_cleaned_up_on_serialise_failure(self, tmp_path, monkeypatch):
+        """If the tmp write itself fails (disk full, permission, etc.),
+        the orphan tmp must not be left behind under the target dir.
+        Post-fix the helper's bare-except cleanup ensures we don't
+        accumulate ``*.tmp`` debris when an upstream caller passes a
+        non-JSON-serialisable payload."""
+        from forgelm.cli._pipeline import _atomic_write_json
+
+        target = str(tmp_path / "bad.json")
+        # ``json.dump`` cannot serialise a set; this triggers the
+        # try/except in the helper after the tmp file is created but
+        # before the replace.
+        try:
+            _atomic_write_json(target, {"bad": {1, 2, 3}})
+        except TypeError:
+            pass
+        # No orphan files left in tmp_path.
+        orphans = [p for p in os.listdir(str(tmp_path)) if p.endswith(".tmp")]
+        assert orphans == [], f"Orphan tmp file(s) left behind: {orphans!r}"
+
+
+# ---------------------------------------------------------------------------
+# Orchestrator behaviour — dry-run
+# ---------------------------------------------------------------------------
+
+
+class TestDryRun:
+    def test_dry_run_validates_every_stage_without_running_trainer(self, tmp_path, monkeypatch):
+        cfg = _three_stage_config(tmp_path)
+        configs_seen = _install_trainer_mocks(monkeypatch, [])
+        orch = PipelineOrchestrator(cfg, b"yaml bytes")
+        code = orch.dry_run()
+        assert code == EXIT_SUCCESS
+        # Trainer must not have been instantiated.
+        assert configs_seen == []
+
+    def test_dry_run_does_not_write_state_or_manifest(self, tmp_path, monkeypatch):
+        cfg = _three_stage_config(tmp_path)
+        _install_trainer_mocks(monkeypatch, [])
+        orch = PipelineOrchestrator(cfg, b"yaml bytes")
+        orch.dry_run()
+        assert not os.path.exists(orch.paths["state_file"])
+        assert not os.path.exists(orch.paths["manifest_file"])
+
+    def test_dry_run_flags_output_dir_collision(self, tmp_path, monkeypatch):
+        """Phase 14 review F-G-1 (Gemini): two stages resolving to the
+        same ``training.output_dir`` would silently overwrite each
+        other's checkpoints and per-stage Annex-IV manifests, breaking
+        the chain of custody.  Dry-run must surface the collision as
+        EXIT_CONFIG_ERROR before any GPU work."""
+        shared_dir = str(tmp_path / "shared_out")
+        cfg = ForgeConfig(
+            model={"name_or_path": "org/base"},
+            lora={"r": 8},
+            training={"trainer_type": "sft", "output_dir": str(tmp_path)},
+            data={"dataset_name_or_path": "org/sft"},
+            pipeline={
+                "output_dir": str(tmp_path / "pipeline_run"),
+                "stages": [
+                    {
+                        "name": "sft_stage",
+                        "training": {"trainer_type": "sft", "output_dir": shared_dir},
+                        "data": {"dataset_name_or_path": "org/sft"},
+                    },
+                    {
+                        "name": "dpo_stage",
+                        "training": {"trainer_type": "dpo", "output_dir": shared_dir},
+                        "data": {"dataset_name_or_path": "org/dpo"},
+                    },
+                ],
+            },
+        )
+        _install_trainer_mocks(monkeypatch, [])
+        orch = PipelineOrchestrator(cfg, b"yaml")
+        code = orch.dry_run()
+        assert code == EXIT_CONFIG_ERROR
+
+    def test_dry_run_flags_inherited_output_dir_collision(self, tmp_path, monkeypatch):
+        """Two stages that both *inherit* the root ``training`` block
+        (no per-stage override) end up sharing the root's output_dir
+        by construction — the most common form of the F-G-1 footgun.
+        Verify the guard catches it even when the collision is
+        inherited rather than explicit."""
+        cfg = ForgeConfig(
+            model={"name_or_path": "org/base"},
+            lora={"r": 8},
+            training={"trainer_type": "sft", "output_dir": str(tmp_path / "root_out")},
+            data={"dataset_name_or_path": "org/sft"},
+            pipeline={
+                "output_dir": str(tmp_path / "pipeline_run"),
+                # Note: no per-stage training: block — both stages inherit
+                # the root training.output_dir.  Setting data: per stage
+                # so the merge succeeds, isolating the collision.
+                "stages": [
+                    {"name": "s1", "data": {"dataset_name_or_path": "org/d1"}},
+                    {"name": "s2", "data": {"dataset_name_or_path": "org/d2"}},
+                ],
+            },
+        )
+        _install_trainer_mocks(monkeypatch, [])
+        orch = PipelineOrchestrator(cfg, b"yaml")
+        code = orch.dry_run()
+        assert code == EXIT_CONFIG_ERROR
+
+
+# ---------------------------------------------------------------------------
+# Orchestrator behaviour — happy path
+# ---------------------------------------------------------------------------
+
+
+class TestHappyPath:
+    def test_three_stage_chain_runs_all_stages(self, tmp_path, monkeypatch):
+        cfg = _three_stage_config(tmp_path)
+        results = [
+            TrainResult(
+                success=True, metrics={"eval_loss": 0.5}, final_model_path=str(tmp_path / "stage1" / "final_model")
+            ),
+            TrainResult(
+                success=True, metrics={"eval_loss": 0.3}, final_model_path=str(tmp_path / "stage2" / "final_model")
+            ),
+            TrainResult(
+                success=True, metrics={"eval_loss": 0.2}, final_model_path=str(tmp_path / "stage3" / "final_model")
+            ),
+        ]
+        configs_seen = _install_trainer_mocks(monkeypatch, results)
+        orch = PipelineOrchestrator(cfg, b"yaml bytes")
+        code = orch.run()
+        assert code == EXIT_SUCCESS
+        assert len(configs_seen) == 3
+        # Stage 2's input model must be stage 1's output (auto-chain).
+        assert configs_seen[1].model.name_or_path == str(tmp_path / "stage1" / "final_model")
+        assert configs_seen[2].model.name_or_path == str(tmp_path / "stage2" / "final_model")
+
+    def test_state_file_written_with_final_status_completed(self, tmp_path, monkeypatch):
+        cfg = _three_stage_config(tmp_path)
+        results = [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)]
+        _install_trainer_mocks(monkeypatch, results)
+        orch = PipelineOrchestrator(cfg, b"yaml bytes")
+        orch.run()
+        assert os.path.exists(orch.paths["state_file"])
+        with open(orch.paths["state_file"]) as f:
+            payload = json.load(f)
+        assert payload["final_status"] == "completed"
+        assert payload["stopped_at"] is None
+        assert all(s["status"] == "completed" for s in payload["stages"])
+
+    def test_manifest_written_with_chain_intact(self, tmp_path, monkeypatch):
+        cfg = _three_stage_config(tmp_path)
+        results = [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)]
+        _install_trainer_mocks(monkeypatch, results)
+        orch = PipelineOrchestrator(cfg, b"yaml bytes")
+        orch.run()
+        assert os.path.exists(orch.paths["manifest_file"])
+        with open(orch.paths["manifest_file"]) as f:
+            manifest = json.load(f)
+        assert manifest["pipeline_run_id"].startswith("pl_")
+        assert manifest["final_status"] == "completed"
+        # Chain integrity in the manifest payload.
+        s1, s2, s3 = manifest["stages"]
+        assert s2["input_model"] == s1["output_model"]
+        assert s3["input_model"] == s2["output_model"]
+
+
+# ---------------------------------------------------------------------------
+# Orchestrator behaviour — auto-revert + gate failures
+# ---------------------------------------------------------------------------
+
+
+class TestAutoRevert:
+    def test_stage_2_auto_revert_stops_pipeline(self, tmp_path, monkeypatch):
+        cfg = _three_stage_config(tmp_path)
+        results = [
+            TrainResult(success=True),
+            TrainResult(success=False, reverted=True, error="loss regression"),
+        ]
+        configs_seen = _install_trainer_mocks(monkeypatch, results)
+        orch = PipelineOrchestrator(cfg, b"yaml bytes")
+        code = orch.run()
+        assert code == EXIT_EVAL_FAILURE
+        # Stage 3 must NOT have been instantiated.
+        assert len(configs_seen) == 2
+        # State file records the stop-at + skipped status.
+        with open(orch.paths["state_file"]) as f:
+            payload = json.load(f)
+        assert payload["final_status"] == "stopped_at_stage"
+        assert payload["stopped_at"] == "dpo_stage"
+        statuses = [s["status"] for s in payload["stages"]]
+        assert statuses == ["completed", "failed", "skipped_due_to_prior_revert"]
+
+    def test_gate_failure_without_revert_also_stops_pipeline(self, tmp_path, monkeypatch):
+        """A stage that returns ``success=False`` but ``reverted=False``
+        (e.g., a benchmark min-score failure that didn't auto-revert)
+        still halts the chain — downstream stages would be operating
+        on an explicitly-failed checkpoint."""
+        cfg = _three_stage_config(tmp_path)
+        results = [TrainResult(success=False, reverted=False, error="benchmark below min")]
+        _install_trainer_mocks(monkeypatch, results)
+        orch = PipelineOrchestrator(cfg, b"yaml bytes")
+        code = orch.run()
+        assert code == EXIT_EVAL_FAILURE
+        with open(orch.paths["state_file"]) as f:
+            payload = json.load(f)
+        assert payload["stages"][0]["status"] == "failed"
+
+
+# ---------------------------------------------------------------------------
+# Orchestrator behaviour — human-approval gate
+# ---------------------------------------------------------------------------
+
+
+class TestHumanApprovalGate:
+    def test_gated_stage_exits_with_awaiting_approval(self, tmp_path, monkeypatch):
+        cfg = _three_stage_config(tmp_path)
+        staging = str(tmp_path / "stage1" / "final_model.staging")
+        os.makedirs(staging, exist_ok=True)
+        results = [TrainResult(success=True, staging_path=staging)]
+        _install_trainer_mocks(monkeypatch, results)
+        orch = PipelineOrchestrator(cfg, b"yaml bytes")
+        code = orch.run()
+        assert code == EXIT_AWAITING_APPROVAL
+
+    def test_gated_pending_approval_preserves_downstream_pending(self, tmp_path, monkeypatch):
+        """After a gated stage, downstream stages must stay ``pending``
+        (not ``skipped_due_to_prior_revert``) so a subsequent
+        ``--resume-from`` picks them up post-approval."""
+        cfg = _three_stage_config(tmp_path)
+        staging = str(tmp_path / "stage1" / "final_model.staging")
+        os.makedirs(staging, exist_ok=True)
+        results = [TrainResult(success=True, staging_path=staging)]
+        _install_trainer_mocks(monkeypatch, results)
+        orch = PipelineOrchestrator(cfg, b"yaml bytes")
+        orch.run()
+        with open(orch.paths["state_file"]) as f:
+            payload = json.load(f)
+        assert payload["final_status"] == "gated_pending_approval"
+        assert payload["stages"][0]["status"] == "gated_pending_approval"
+        assert payload["stages"][1]["status"] == "pending"
+        assert payload["stages"][2]["status"] == "pending"
+
+    def test_gated_stage_emits_dedicated_stage_gated_audit_event(self, tmp_path, monkeypatch):
+        """Phase 14 review F-N-1 regression: a stage exiting
+        ``EXIT_AWAITING_APPROVAL`` must emit a dedicated
+        ``pipeline.stage_gated`` audit event (not
+        ``pipeline.stage_completed`` with a sub-field).  This lets
+        dashboard / SIEM filters distinguish the gate flow on the event
+        name alone."""
+        cfg = _three_stage_config(tmp_path)
+        staging = str(tmp_path / "stage1" / "final_model.staging")
+        os.makedirs(staging, exist_ok=True)
+        _install_trainer_mocks(monkeypatch, [TrainResult(success=True, staging_path=staging)])
+        orch = PipelineOrchestrator(cfg, b"yaml bytes")
+        orch.run()
+        audit_path = os.path.join(orch.paths["root_output_dir"], "audit_log.jsonl")
+        with open(audit_path) as f:
+            events = [json.loads(line) for line in f if line.strip()]
+        gated_events = [e for e in events if e.get("event") == "pipeline.stage_gated"]
+        assert len(gated_events) == 1, (
+            f"Expected exactly one pipeline.stage_gated event, got {[e.get('event') for e in events]!r}"
+        )
+        evt = gated_events[0]
+        assert evt["stage_name"] == "sft_stage"
+        assert evt["gate_decision"] == "approval_pending"
+        assert evt["staging_path"] == staging
+        # Counter-assert: the legacy stage_completed event must NOT have
+        # fired for this stage (preventing a future regression where both
+        # events are emitted in parallel and the dashboard double-counts).
+        completed_for_sft = [
+            e for e in events if e.get("event") == "pipeline.stage_completed" and e.get("stage_name") == "sft_stage"
+        ]
+        assert completed_for_sft == []
+
+
+# ---------------------------------------------------------------------------
+# Orchestrator behaviour — --stage filter
+# ---------------------------------------------------------------------------
+
+
+class TestStageFilter:
+    def test_unknown_stage_name_rejected(self, tmp_path, monkeypatch):
+        cfg = _three_stage_config(tmp_path)
+        _install_trainer_mocks(monkeypatch, [])
+        orch = PipelineOrchestrator(cfg, b"yaml bytes")
+        code = orch.run(stage_filter="nonexistent_stage")
+        assert code == EXIT_CONFIG_ERROR
+
+    def test_filter_to_first_stage_runs_only_it(self, tmp_path, monkeypatch):
+        cfg = _three_stage_config(tmp_path)
+        results = [TrainResult(success=True)]
+        configs_seen = _install_trainer_mocks(monkeypatch, results)
+        orch = PipelineOrchestrator(cfg, b"yaml bytes")
+        code = orch.run(stage_filter="sft_stage")
+        assert code == EXIT_SUCCESS
+        assert len(configs_seen) == 1
+        with open(orch.paths["state_file"]) as f:
+            payload = json.load(f)
+        statuses = [s["status"] for s in payload["stages"]]
+        assert statuses == ["completed", "skipped_by_filter", "skipped_by_filter"]
+
+    def test_filter_to_middle_stage_without_prev_output_fails_cleanly(self, tmp_path, monkeypatch):
+        """``--stage dpo_stage`` without sft_stage's output on disk must
+        fail with a clear EXIT_CONFIG_ERROR — not silently fall back to
+        the root ``model.name_or_path``."""
+        cfg = _three_stage_config(tmp_path)
+        _install_trainer_mocks(monkeypatch, [TrainResult(success=True)])
+        orch = PipelineOrchestrator(cfg, b"yaml bytes")
+        # Make sure the prev output dir does NOT exist.
+        assert not os.path.exists(tmp_path / "stage1" / "final_model")
+        code = orch.run(stage_filter="dpo_stage")
+        assert code == EXIT_CONFIG_ERROR
+
+    def test_filter_with_input_model_override_succeeds(self, tmp_path, monkeypatch):
+        cfg = _three_stage_config(tmp_path)
+        configs_seen = _install_trainer_mocks(monkeypatch, [TrainResult(success=True)])
+        orch = PipelineOrchestrator(cfg, b"yaml bytes")
+        override_path = str(tmp_path / "manual_input")
+        os.makedirs(override_path, exist_ok=True)
+        code = orch.run(stage_filter="dpo_stage", input_model_override=override_path)
+        assert code == EXIT_SUCCESS
+        assert len(configs_seen) == 1
+        assert configs_seen[0].model.name_or_path == override_path
+
+    def test_filter_middle_stage_with_disk_seed_passes_chain_verifier(self, tmp_path, monkeypatch):
+        """Phase 14 review final-round F-B-2 regression: ``--stage X``
+        on a non-first stage whose predecessor's ``final_model`` exists
+        on disk auto-chains correctly, AND the resulting pipeline
+        manifest must NOT report a ``chain_integrity_violation``.
+
+        Pre-fix, the ``skipped_by_filter`` predecessor had
+        ``output_model=null`` while the executed stage had
+        ``input_source=chain``, so the strict chain verifier (F-B-3)
+        emitted a ``chain_integrity_violation`` on every legitimate
+        ``--stage `` run.  The fix seeds the predecessor's
+        ``output_model`` in the manifest from the resolved disk path.
+        """
+        from forgelm.compliance import verify_pipeline_manifest_at_path
+
+        cfg = _three_stage_config(tmp_path)
+        # Pre-create stage 1's on-disk output so the --stage filter can
+        # seed the chain.
+        prev_output_dir = tmp_path / "stage1" / "final_model"
+        prev_output_dir.mkdir(parents=True, exist_ok=True)
+        configs_seen = _install_trainer_mocks(monkeypatch, [TrainResult(success=True)])
+        orch = PipelineOrchestrator(cfg, b"yaml")
+        code = orch.run(stage_filter="dpo_stage")
+        assert code == EXIT_SUCCESS
+        assert len(configs_seen) == 1
+        # The DPO stage must have auto-chained from stage 1's on-disk output.
+        assert configs_seen[0].model.name_or_path == str(prev_output_dir)
+
+        # Inspect the manifest directly to confirm chain integrity is
+        # intact.  ``verify_pipeline_manifest_at_path`` additionally
+        # checks each completed stage's per-stage training_manifest
+        # exists; with a mocked trainer that file isn't actually
+        # written, so we don't assert ``violations == []`` here — we
+        # assert only the chain-integrity contract that's the subject
+        # of the F-B-2 regression.  Per-stage training_manifest
+        # existence is covered by ``test_completed_stage_with_missing_
+        # training_manifest_flagged`` in test_pipeline_compliance.py.
+        import json as _json
+
+        with open(orch.paths["manifest_file"]) as f:
+            manifest = _json.load(f)
+        sft_stage_payload, dpo_stage_payload, _grpo = manifest["stages"]
+        assert sft_stage_payload["status"] == "skipped_by_filter"
+        # The fix: predecessor's output_model is surfaced from the resolved disk path.
+        assert sft_stage_payload["output_model"] == str(prev_output_dir)
+        assert dpo_stage_payload["input_source"] == "chain"
+        assert dpo_stage_payload["input_model"] == str(prev_output_dir)
+
+        violations = verify_pipeline_manifest_at_path(str(orch.paths["root_output_dir"]))
+        assert not any("chain_integrity_violation" in v for v in violations), (
+            f"Chain integrity must hold for --stage  with disk seed; got: {violations!r}"
+        )
+
+
+# ---------------------------------------------------------------------------
+# Orchestrator behaviour — --resume-from
+# ---------------------------------------------------------------------------
+
+
+class TestResumeFrom:
+    def test_resume_without_state_file_fails(self, tmp_path, monkeypatch):
+        cfg = _three_stage_config(tmp_path)
+        _install_trainer_mocks(monkeypatch, [])
+        orch = PipelineOrchestrator(cfg, b"yaml bytes")
+        code = orch.run(resume_from="dpo_stage")
+        assert code == EXIT_CONFIG_ERROR
+
+    def test_output_dir_collision_rejected_in_run_not_only_dry_run(self, tmp_path, monkeypatch):
+        """Phase 14 post-release review HIGH 4: pre-fix the
+        ``training.output_dir`` collision guard only fired during
+        ``--dry-run``.  An operator who skipped dry-run could
+        silently overwrite per-stage manifests + checkpoints across
+        stages.  ``run()`` must execute the same pre-flight."""
+        # Two stages writing to the same directory.
+        cfg = ForgeConfig(
+            model={"name_or_path": "org/base"},
+            lora={"r": 8},
+            training={"trainer_type": "sft", "output_dir": str(tmp_path)},
+            data={"dataset_name_or_path": "org/sft"},
+            pipeline={
+                "output_dir": str(tmp_path / "pipeline_run"),
+                "stages": [
+                    {
+                        "name": "sft_stage",
+                        "training": {"trainer_type": "sft", "output_dir": str(tmp_path / "collide")},
+                        "data": {"dataset_name_or_path": "org/sft_data"},
+                    },
+                    {
+                        "name": "dpo_stage",
+                        "training": {"trainer_type": "dpo", "output_dir": str(tmp_path / "collide")},
+                        "data": {"dataset_name_or_path": "org/dpo_data"},
+                    },
+                ],
+            },
+        )
+        _install_trainer_mocks(monkeypatch, [TrainResult(success=True), TrainResult(success=True)])
+        orch = PipelineOrchestrator(cfg, b"yaml")
+        # Note: NOT calling dry_run() — invoking run() directly.
+        code = orch.run()
+        assert code == EXIT_CONFIG_ERROR
+
+    def test_resume_skips_gated_pending_approval_with_on_disk_output(self, tmp_path, monkeypatch):
+        """Phase 14 post-release review BLOCKER 2: a stage that exited
+        with ``EXIT_AWAITING_APPROVAL`` is recorded as
+        ``status="gated_pending_approval"``.  After ``forgelm approve``
+        promotes the staging path and the operator runs
+        ``--resume-from ``, the gated stage MUST be
+        treated like a completed stage (skip + reuse the on-disk
+        ``output_model``).  Pre-fix, only ``status=="completed"``
+        stages were skipped — so the gated SFT would re-train, the
+        DPO stage would chain from the freshly-trained output instead
+        of the operator-approved one, and the audit log would carry
+        two ``pipeline.stage_started`` events for the same stage_index."""
+        cfg = _three_stage_config(tmp_path)
+        # First run: SFT stage hits a human-approval gate and exits 4.
+        _install_trainer_mocks(
+            monkeypatch, [TrainResult(success=False, staging_path=str(tmp_path / "stage1" / "final_model"))]
+        )
+        orch1 = PipelineOrchestrator(cfg, b"yaml")
+        code1 = orch1.run()
+        assert code1 == EXIT_AWAITING_APPROVAL
+        # Sanity: state recorded SFT as gated, with staging_path as output_model.
+        with open(orch1.paths["state_file"]) as f:
+            payload = json.load(f)
+        assert payload["stages"][0]["status"] == "gated_pending_approval"
+        assert payload["stages"][0]["output_model"] is not None
+
+        # Simulate operator approval — output_model directory already
+        # exists on disk from the mock setup.  Now resume from DPO.
+        results_run2 = [TrainResult(success=True), TrainResult(success=True)]
+        configs_seen2 = _install_trainer_mocks(monkeypatch, results_run2)
+        orch2 = PipelineOrchestrator(cfg, b"yaml")
+        code2 = orch2.run(resume_from="dpo_stage")
+        assert code2 == EXIT_SUCCESS
+        # Only DPO + GRPO ran; SFT was skipped because its gated output exists on disk.
+        assert len(configs_seen2) == 2
+        # DPO must have auto-chained from the gated SFT's output_model,
+        # not been re-trained.
+        assert configs_seen2[0].training.trainer_type == "dpo"
+
+    def test_resume_skips_completed_stages(self, tmp_path, monkeypatch):
+        cfg = _three_stage_config(tmp_path)
+        # First run: stage 1 completes, stage 2 fails (training crash).
+        results_run1 = [
+            TrainResult(success=True),
+            TrainResult(success=False, error="oom"),
+        ]
+        _install_trainer_mocks(monkeypatch, results_run1)
+        orch1 = PipelineOrchestrator(cfg, b"yaml bytes")
+        orch1.run()
+        # Second run: resume from dpo_stage.  Stage 1 already completed
+        # AND its output_model dir exists (mock created it) — must skip.
+        results_run2 = [TrainResult(success=True), TrainResult(success=True)]
+        configs_seen2 = _install_trainer_mocks(monkeypatch, results_run2)
+        orch2 = PipelineOrchestrator(cfg, b"yaml bytes")
+        code = orch2.run(resume_from="dpo_stage")
+        assert code == EXIT_SUCCESS
+        # Only stage 2 + stage 3 ran — stage 1 was skipped.
+        assert len(configs_seen2) == 2
+
+    def test_stale_config_hash_rejects_resume(self, tmp_path, monkeypatch):
+        cfg = _three_stage_config(tmp_path)
+        _install_trainer_mocks(monkeypatch, [TrainResult(success=False, error="x")])
+        orch1 = PipelineOrchestrator(cfg, b"original yaml")
+        orch1.run()
+        # Operator edited the YAML; resume against a different hash.
+        # Post-Phase-14-review: refusal flows back through ``run()`` as a
+        # plain return value (not ``sys.exit`` mid-method) so the audit-
+        # log + summary path runs uniformly on every refusal.
+        orch2 = PipelineOrchestrator(cfg, b"edited yaml")
+        code = orch2.run(resume_from="dpo_stage")
+        assert code == EXIT_CONFIG_ERROR
+
+    def test_force_resume_accepts_stale_hash_with_warning(self, tmp_path, monkeypatch, caplog):
+        import logging
+
+        cfg = _three_stage_config(tmp_path)
+        _install_trainer_mocks(monkeypatch, [TrainResult(success=False, error="x")])
+        orch1 = PipelineOrchestrator(cfg, b"original yaml")
+        orch1.run()
+        _install_trainer_mocks(monkeypatch, [TrainResult(success=True), TrainResult(success=True)])
+        orch2 = PipelineOrchestrator(cfg, b"edited yaml")
+        with caplog.at_level(logging.WARNING, logger="forgelm.pipeline"):
+            code = orch2.run(resume_from="dpo_stage", force_resume=True)
+        assert code == EXIT_SUCCESS
+
+    def test_force_resume_emits_audit_event_with_both_hashes(self, tmp_path, monkeypatch):
+        """Phase 14 review F-B-2 regression: ``--force-resume`` must
+        emit a ``pipeline.force_resume`` audit event carrying both the
+        old and new ``pipeline_config_hash`` so a compliance reviewer
+        can distinguish an operator-approved override from a normal
+        resume.  Pre-fix, only a WARNING log line was emitted — invisible
+        in the append-only audit-log JSONL stream."""
+        cfg = _three_stage_config(tmp_path)
+        _install_trainer_mocks(monkeypatch, [TrainResult(success=False, error="x")])
+        orch1 = PipelineOrchestrator(cfg, b"yaml v1")
+        orch1.run()
+
+        _install_trainer_mocks(monkeypatch, [TrainResult(success=True), TrainResult(success=True)])
+        orch2 = PipelineOrchestrator(cfg, b"yaml v2 (operator edited)")
+        orch2.run(resume_from="dpo_stage", force_resume=True)
+
+        audit_path = os.path.join(orch2.paths["root_output_dir"], "audit_log.jsonl")
+        assert os.path.exists(audit_path)
+        with open(audit_path) as f:
+            events = [json.loads(line) for line in f if line.strip()]
+        force_resume_events = [e for e in events if e.get("event") == "pipeline.force_resume"]
+        assert len(force_resume_events) == 1, (
+            f"Expected exactly one pipeline.force_resume audit event, got "
+            f"{len(force_resume_events)}: {[e.get('event') for e in events]!r}"
+        )
+        evt = force_resume_events[0]
+        assert "old_config_hash" in evt
+        assert "new_config_hash" in evt
+        assert evt["old_config_hash"] != evt["new_config_hash"]
+        assert "yaml v1" not in str(evt) and "yaml v2" not in str(evt), (
+            "Audit event must record hashes, not raw YAML bytes."
+        )
+
+    def test_topology_guard_runs_even_on_hash_match(self, tmp_path, monkeypatch):
+        """Phase 14 post-release review BLOCKER 3: a tampered or
+        truncated ``pipeline_state.json`` could carry the same
+        ``pipeline_config_hash`` as the current YAML (operator never
+        edited the YAML) but a damaged ``stages[]`` array — fewer
+        entries, wrong order, or hand-edited names.  Pre-fix,
+        ``_validate_resume_state`` returned ``None`` immediately on
+        hash match without running the topology check, so the resume
+        would proceed against a corrupted state file and
+        ``state.stages[i]`` would silently overwrite the wrong stage's
+        history.  Topology guard must now run UNCONDITIONALLY."""
+        cfg = _three_stage_config(tmp_path)
+        _install_trainer_mocks(monkeypatch, [TrainResult(success=False, error="x")])
+        orch1 = PipelineOrchestrator(cfg, b"yaml")
+        orch1.run()
+
+        # Surgically truncate the on-disk state to TWO stages while
+        # keeping the YAML (hence the hash) unchanged.  This simulates
+        # disk corruption / hand-edit / partial-write tamper.
+        with open(orch1.paths["state_file"]) as f:
+            payload = json.load(f)
+        payload["stages"] = payload["stages"][:2]
+        with open(orch1.paths["state_file"], "w") as f:
+            json.dump(payload, f)
+
+        _install_trainer_mocks(monkeypatch, [TrainResult(success=True), TrainResult(success=True)])
+        orch2 = PipelineOrchestrator(cfg, b"yaml")
+        # Hash MATCHES (same YAML bytes) but topology drifted — must refuse.
+        code = orch2.run(resume_from="dpo_stage")
+        assert code == EXIT_CONFIG_ERROR
+
+    def test_force_resume_refuses_when_stage_topology_changed(self, tmp_path, monkeypatch):
+        """Phase 14 review-response regression: ``--force-resume`` must
+        still refuse when the on-disk state's stage list disagrees with
+        the current pipeline (count, names, or order).  Without this
+        guard, a renamed/inserted/deleted stage between runs would let
+        the orchestrator address ``state.stages[i]`` from the old shape
+        while iterating ``self.pipeline.stages[i]`` from the new shape,
+        silently corrupting the audit trail."""
+        cfg = _three_stage_config(tmp_path)
+        _install_trainer_mocks(monkeypatch, [TrainResult(success=False, error="x")])
+        orch1 = PipelineOrchestrator(cfg, b"yaml v1")
+        orch1.run()
+
+        # Build a *different* topology (rename middle stage) while keeping
+        # the YAML hash changed too, so force_resume is the only thing
+        # standing between us and a resume.
+        renamed_cfg = ForgeConfig(
+            model={"name_or_path": "org/base"},
+            lora={"r": 8},
+            training={"trainer_type": "sft", "output_dir": str(tmp_path)},
+            data={"dataset_name_or_path": "org/sft"},
+            pipeline={
+                "output_dir": str(tmp_path / "pipeline_run"),
+                "stages": [
+                    {
+                        "name": "sft_stage",
+                        "training": {"trainer_type": "sft", "output_dir": str(tmp_path / "stage1")},
+                        "data": {"dataset_name_or_path": "org/sft_data"},
+                    },
+                    {
+                        "name": "renamed_middle",  # was 'dpo_stage'
+                        "training": {"trainer_type": "dpo", "output_dir": str(tmp_path / "stage2")},
+                        "data": {"dataset_name_or_path": "org/dpo_data"},
+                    },
+                    {
+                        "name": "grpo_stage",
+                        "training": {"trainer_type": "grpo", "output_dir": str(tmp_path / "stage3")},
+                        "data": {"dataset_name_or_path": "org/math_data"},
+                    },
+                ],
+            },
+        )
+        _install_trainer_mocks(monkeypatch, [TrainResult(success=True), TrainResult(success=True)])
+        orch2 = PipelineOrchestrator(renamed_cfg, b"yaml v2 (renamed)")
+        code = orch2.run(resume_from="grpo_stage", force_resume=True)
+        assert code == EXIT_CONFIG_ERROR
+        # No pipeline.force_resume audit event should have been emitted —
+        # topology mismatch is refused *before* the audit hook.
+        audit_path = os.path.join(orch2.paths["root_output_dir"], "audit_log.jsonl")
+        if os.path.exists(audit_path):
+            with open(audit_path) as f:
+                events = [json.loads(line) for line in f if line.strip()]
+            assert not any(e.get("event") == "pipeline.force_resume" for e in events), (
+                "pipeline.force_resume must not fire when topology has changed."
+            )
+
+    def test_resume_with_unknown_stage_name_fails(self, tmp_path, monkeypatch):
+        cfg = _three_stage_config(tmp_path)
+        _install_trainer_mocks(monkeypatch, [TrainResult(success=False, error="x")])
+        orch1 = PipelineOrchestrator(cfg, b"yaml")
+        orch1.run()
+        orch2 = PipelineOrchestrator(cfg, b"yaml")
+        code = orch2.run(resume_from="nonexistent")
+        assert code == EXIT_CONFIG_ERROR
+
+
+# ---------------------------------------------------------------------------
+# Orchestrator behaviour — audit + webhook (best-effort emission, no crash)
+# ---------------------------------------------------------------------------
+
+
+class TestAuditAndWebhook:
+    def test_audit_events_emitted_on_happy_path(self, tmp_path, monkeypatch):
+        cfg = _three_stage_config(tmp_path)
+        results = [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)]
+        _install_trainer_mocks(monkeypatch, results)
+        orch = PipelineOrchestrator(cfg, b"yaml")
+        orch.run()
+        # Audit log file lives under the root output dir.
+        audit_path = os.path.join(orch.paths["root_output_dir"], "audit_log.jsonl")
+        assert os.path.exists(audit_path)
+        with open(audit_path) as f:
+            events = [json.loads(line) for line in f if line.strip()]
+        event_names = [e["event"] for e in events]
+        assert "pipeline.started" in event_names
+        assert event_names.count("pipeline.stage_started") == 3
+        assert event_names.count("pipeline.stage_completed") == 3
+        assert "pipeline.completed" in event_names
+
+    def test_all_pipeline_events_share_same_top_level_run_id(self, tmp_path, monkeypatch):
+        """Phase 14 review final-round F-B-1 regression: every entry in
+        the pipeline-level audit log must carry the same top-level
+        ``run_id`` (the pipeline run id), so SIEM filters and Article 12
+        correlation work on a single field.  Pre-fix, ``_audit_event``
+        constructed a fresh ``AuditLogger`` per call → each entry got a
+        different auto-generated ``fg-``."""
+        cfg = _three_stage_config(tmp_path)
+        results = [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)]
+        _install_trainer_mocks(monkeypatch, results)
+        orch = PipelineOrchestrator(cfg, b"yaml")
+        orch.run()
+        audit_path = os.path.join(orch.paths["root_output_dir"], "audit_log.jsonl")
+        with open(audit_path) as f:
+            entries = [json.loads(line) for line in f if line.strip()]
+        # Every entry carries a top-level ``run_id`` field; collect the
+        # set across all pipeline.* events.
+        pipeline_entries = [e for e in entries if e.get("event", "").startswith("pipeline.")]
+        assert len(pipeline_entries) >= 5, "expected at least 5 pipeline.* events on a happy 3-stage run"
+        top_level_ids = {e["run_id"] for e in pipeline_entries}
+        assert len(top_level_ids) == 1, f"All pipeline.* events must share one top-level run_id, got {top_level_ids!r}"
+        # The pinned run_id must equal the pipeline_run_id field on the
+        # entries (which is the contract: pipeline.* events carry the
+        # pipeline run id in both surfaces).
+        (top_level_id,) = top_level_ids
+        pipeline_run_ids = {e["pipeline_run_id"] for e in pipeline_entries}
+        assert pipeline_run_ids == {top_level_id}
+
+    def test_webhook_failures_do_not_abort_pipeline(self, tmp_path, monkeypatch):
+        """A crashing webhook notifier must not derail the pipeline.
+
+        Mirrors the existing per-stage webhook discipline — best-effort
+        telemetry never blocks the operator's actual work."""
+        cfg = _three_stage_config(tmp_path)
+        _install_trainer_mocks(
+            monkeypatch, [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)]
+        )
+        with patch("forgelm.webhook.WebhookNotifier", side_effect=RuntimeError("notifier down")):
+            orch = PipelineOrchestrator(cfg, b"yaml")
+            code = orch.run()
+        assert code == EXIT_SUCCESS
+
+    def test_audit_events_share_one_pipeline_run_id(self, tmp_path, monkeypatch):
+        """Phase 14 review final-round F-B-1 regression: every entry in
+        ``audit_log.jsonl`` emitted by the orchestrator must carry the
+        **same** top-level ``run_id`` field (= the pipeline run id), so
+        SIEM dashboards can group all events for one pipeline run by
+        that single key.  Pre-fix, ``_audit_event`` constructed a fresh
+        ``AuditLogger`` per call and each got a different auto-
+        generated ``fg-`` run_id."""
+        cfg = _three_stage_config(tmp_path)
+        _install_trainer_mocks(monkeypatch, [TrainResult(success=True)] * 3)
+        orch = PipelineOrchestrator(cfg, b"yaml")
+        orch.run()
+        audit_path = os.path.join(orch.paths["root_output_dir"], "audit_log.jsonl")
+        with open(audit_path) as f:
+            events = [json.loads(line) for line in f if line.strip()]
+        # All pipeline.* events must have identical top-level run_id.
+        run_ids = {e.get("run_id") for e in events if e.get("event", "").startswith("pipeline.")}
+        assert len(run_ids) == 1, (
+            f"Pipeline events must share one top-level run_id; got {run_ids!r} across {len(events)} entries."
+        )
+        # And that single run_id must equal the pipeline run id surfaced
+        # in each event's ``pipeline_run_id`` field.
+        pipeline_run_ids = {e.get("pipeline_run_id") for e in events if e.get("event", "").startswith("pipeline.")}
+        assert pipeline_run_ids == run_ids, (
+            f"Top-level run_id {run_ids!r} must equal pipeline_run_id {pipeline_run_ids!r}."
+        )
+
+    def test_audit_event_failures_do_not_abort_pipeline(self, tmp_path, monkeypatch):
+        """Phase 14 review F-N-5 regression: ``_audit_event``'s
+        documented ``except Exception`` swallow path must be reached and
+        the pipeline must continue.  An audit-write failure (read-only
+        filesystem, disk full, malformed audit logger config) is
+        documented as best-effort — the orchestrator emits a WARNING and
+        carries on; the test patches ``AuditLogger.log_event`` to raise
+        and asserts the run completes with the expected exit code."""
+        cfg = _three_stage_config(tmp_path)
+        _install_trainer_mocks(monkeypatch, [TrainResult(success=True)] * 3)
+
+        from forgelm.compliance import AuditLogger as _RealAuditLogger
+
+        original_log_event = _RealAuditLogger.log_event
+
+        def _raising_log_event(self, event, **fields):
+            if event.startswith("pipeline."):
+                raise OSError("Disk full / read-only audit log")
+            return original_log_event(self, event, **fields)
+
+        monkeypatch.setattr(_RealAuditLogger, "log_event", _raising_log_event)
+        orch = PipelineOrchestrator(cfg, b"yaml")
+        code = orch.run()
+        assert code == EXIT_SUCCESS
+
+    def test_emit_summary_json_output_is_round_trippable(self, tmp_path, monkeypatch, capsys):
+        """Phase 14 review F-N-7 regression: the ``output_format='json'``
+        branch of ``_emit_summary`` is the dispatcher contract for
+        ``forgelm --config pipeline.yaml --output-format json`` — it
+        must print a single JSON object that round-trips through
+        ``json.loads`` and carries the per-stage payload reviewers
+        consume programmatically."""
+        cfg = _three_stage_config(tmp_path)
+        _install_trainer_mocks(monkeypatch, [TrainResult(success=True)] * 3)
+        orch = PipelineOrchestrator(cfg, b"yaml", output_format="json")
+        code = orch.run()
+        assert code == EXIT_SUCCESS
+        out = capsys.readouterr().out
+        # The JSON object is the only thing on stdout; logger goes to stderr.
+        payload = json.loads(out)
+        assert payload["final_status"] == "completed"
+        assert len(payload["stages"]) == 3
+        assert all(s["status"] == "completed" for s in payload["stages"])
diff --git a/tools/check_bilingual_parity.py b/tools/check_bilingual_parity.py
index 28743bad..15bf446e 100644
--- a/tools/check_bilingual_parity.py
+++ b/tools/check_bilingual_parity.py
@@ -74,6 +74,7 @@
     ("docs/guides/iso_soc2_deployer_guide.md", "docs/guides/iso_soc2_deployer_guide-tr.md"),
     ("docs/guides/library_api.md", "docs/guides/library_api-tr.md"),
     ("docs/guides/performance.md", "docs/guides/performance-tr.md"),
+    ("docs/guides/pipeline.md", "docs/guides/pipeline-tr.md"),
     ("docs/guides/quickstart.md", "docs/guides/quickstart-tr.md"),
     ("docs/guides/safety_compliance.md", "docs/guides/safety_compliance-tr.md"),
     ("docs/guides/troubleshooting.md", "docs/guides/troubleshooting-tr.md"),
@@ -111,6 +112,14 @@
     ("docs/reference/verify_annex_iv_subcommand.md", "docs/reference/verify_annex_iv_subcommand-tr.md"),
     ("docs/reference/verify_audit.md", "docs/reference/verify_audit-tr.md"),
     ("docs/reference/verify_gguf_subcommand.md", "docs/reference/verify_gguf_subcommand-tr.md"),
+    # docs/usermanuals/ — newly mirrored Phase 14 page.  The rest of
+    # docs/usermanuals/{en,tr}/ is structural-parity-validated by the
+    # ``_meta.yaml``-driven nav build in ``tools/build_usermanuals.py``
+    # (every EN page MUST have a TR counterpart at the same relative
+    # path); registering pipelines.md here adds the *heading-spine*
+    # parity check on top, matching the discipline applied to
+    # docs/guides/pipeline{,-tr}.md.
+    ("docs/usermanuals/en/training/pipelines.md", "docs/usermanuals/tr/training/pipelines.md"),
 )
 
 # Match a Markdown ATX heading prefix: 1-6 leading hashes followed by
diff --git a/tools/check_module_size.py b/tools/check_module_size.py
index 6b307aa7..2c28ff51 100644
--- a/tools/check_module_size.py
+++ b/tools/check_module_size.py
@@ -9,10 +9,33 @@
 (``module_name/`` directory with the same public import path).
 
 This guard catches **future drift** without forcing an immediate
-refactor of the seven modules that already sit over the ceiling at
-PR #29 HEAD.  Those seven are recorded in
-:data:`_GRANDFATHERED_OVER_CEILING`; their splits are tracked for
-the v0.6.x cycle (see ``docs/roadmap/risks-and-decisions.md``).
+refactor of the modules that already sit over the ceiling, recorded
+in :data:`_GRANDFATHERED_OVER_CEILING`.
+
+Grandfather policy
+~~~~~~~~~~~~~~~~~~
+
+The initial seven entries (Wave 2-9, PR #29 HEAD audit) are tracked
+for a v0.6.x sub-package split (see
+``docs/roadmap/risks-and-decisions.md``).
+
+New entries added after PR #29 are admitted **only** when the
+sub-package split is non-trivial enough to materially risk
+behavioural regression at the release-prep stage — i.e. the kind of
+split that needs its own PR / Wave with isolated tests rather than
+being bundled into a feature PR.  Every new entry MUST carry:
+
+* an inline comment naming the **phase** + **release cycle** in
+  which the split lands;
+* a follow-up tracking artefact (roadmap entry, issue, or
+  ``risks-and-decisions.md`` row) so the deferred work can't get
+  lost.
+
+The current addition beyond the original seven is
+``forgelm/cli/_pipeline.py`` (Phase 14, v0.7.0; split tracked for
+v0.7.x alongside the Phase 15 audit-package split pattern).  See
+that file's inline comment in :data:`_GRANDFATHERED_OVER_CEILING`
+below for the tracking pointer.
 
 LOC metric
 ----------
@@ -85,6 +108,15 @@
         "forgelm/config.py",
         "forgelm/cli/_parser.py",
         "forgelm/cli/subcommands/_doctor.py",
+        # Phase 14 (v0.7.0) — multi-stage pipeline orchestrator
+        # at ~1060 LOC: orchestrator state machine + manifest
+        # builder + audit/webhook hooks + 6 helper methods that the
+        # SonarCloud cognitive-complexity refactor cycle pulled out
+        # of the original ``run()`` body.  A sub-package split
+        # (``forgelm/cli/_pipeline/{__init__,_state,_events,
+        # _verify}.py``) is tracked for the v0.7.x cycle and will
+        # land alongside the Phase 15 audit-package split pattern.
+        "forgelm/cli/_pipeline.py",
     }
 )