diff --git a/CHANGELOG.md b/CHANGELOG.md index e63de8d3..7ee7578f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,133 @@ All notable changes to ForgeLM are documented here. _(v0.6.1 dev cycle — entries will land here as PRs merge.)_ +### Added (Phase 14 — Multi-Stage Pipeline Chains) + +- **`pipeline:` config block** at the root of `ForgeConfig` chains + one or more training stages (typically SFT → DPO → GRPO) into one + config-driven run. New Pydantic models `PipelineStage` and + `PipelineConfig` enforce stage-name uniqueness, `^[a-z0-9_]{1,32}$` + identifier shape, **at least 1 stage** (`min_length=1` — single-stage + ergonomics are still better served by omitting the `pipeline:` + block, but the schema accepts a 1-element pipeline so an operator + can iterate up to a multi-stage chain without re-shaping the YAML), + and explicit-`trainer_type` audit-clarity validation per stage. Section-wholesale inheritance: + `model` / `lora` / `training` / `data` / `evaluation` blocks + inherit from the root if omitted or fully replace it when set. + `distributed` / `webhook` / `compliance` / `risk_assessment` / + `monitoring` / `retention` / `synthetic` / `merge` / `auth` are + root-only and rejected per-stage with `EXIT_CONFIG_ERROR (1)`. +- **`forgelm/cli/_pipeline.py` orchestrator** drives the chain + end-to-end. Auto-chains each stage's `model.name_or_path` to the + previous stage's output, persists state atomically to + `/pipeline_state.json` after every transition, + and emits 7 new audit events: `pipeline.started`, + `pipeline.stage_started`, `pipeline.stage_completed`, + `pipeline.stage_gated` (when a stage exits + `EXIT_AWAITING_APPROVAL` — review-cycle F-N-1), + `pipeline.stage_reverted`, `pipeline.force_resume` (operator- + approved stale-config override — review-cycle F-B-2), and + `pipeline.completed`. Every entry's top-level `run_id` is pinned + to the pipeline run id (review-cycle final-round F-B-1). Existing + `training.*` per-stage events from `ForgeTrainer` are preserved — + pre-existing Slack / Teams dashboards filtering on `training.failure` + keep working unchanged. +- **CLI flags** `--stage ` (single-stage filter for audit / + re-run), `--resume-from ` (stage-boundary resume), + `--force-resume` (stale-config-hash override), and `--input-model + ` (auto-chain escape hatch, recorded with `input_source: + cli_override` in the audit log). +- **`--dry-run` multi-stage validation** — when the config carries a + `pipeline:` block, dry-run validates every stage's merged config + + the chain-integrity assertion (stage N's auto-chained input is + under stage N-1's `output_dir`). Errors are collected before + exiting (à la `pytest --collectonly`). +- **Pipeline Annex IV manifest** — `compliance/pipeline_manifest.json` + at `` indexes per-stage `training_manifest.json` + files into one verifiable chain artefact, carrying + `pipeline_run_id` + `pipeline_config_hash` (SHA-256 of the YAML + bytes) for reproducibility. `forgelm verify-annex-iv --pipeline + ` walks the index, checks chain integrity, and asserts + every referenced per-stage manifest exists on disk. +- **Webhook notifications** — `WebhookNotifier.notify_pipeline_started`, + `notify_pipeline_completed`, and `notify_pipeline_reverted` + mirror the orchestrator's audit events; gated by the existing + `notify_on_start` / `notify_on_success` / `notify_on_failure` + config flags so operators silencing per-stage events also silence + the pipeline-level pings. +- **Bilingual operator guide** — new `docs/guides/pipeline.md` + + `docs/guides/pipeline-tr.md` (canonical "first pipeline" walkthrough, + inheritance matrix, CLI semantics, Annex IV verifier flow, + Limitations section). `config_template.yaml` gains a commented-out + `pipeline:` example block. + +### Changed (Phase 14) + +- **Backward compatibility preserved.** A config file without a + `pipeline:` key reaches `forgelm/trainer.py` byte-identically to + v0.6.0; the orchestrator module is never imported. +- `compliance.py` exports four new symbols: `generate_pipeline_manifest` + (called by the orchestrator after every transition), + `export_pipeline_manifest` (atomic write to + `/compliance/pipeline_manifest.json`), + `verify_pipeline_manifest(manifest: dict) -> List[str]` (in-memory + structural + chain-integrity check, returns a list of human-readable + violations — empty when valid), and `verify_pipeline_manifest_at_path + (pipeline_dir: str) -> List[str]` (disk-backed wrapper used by the + `forgelm verify-annex-iv --pipeline ` CLI mode). The shared + validator lives in private helper `_verify_manifest_payload` so the + in-memory and disk-bound entry points cannot drift. + +### Fixed (Phase 14 review-response — PR #53) + +Addresses the consolidated reviewer-pass findings (3 blocking + +4 significant + Gemini's 3 inline comments) against the initial Phase +14 merge candidate: + +- **F-B-1** — `forgelm --config pipeline.yaml --dry-run` now reaches + the pipeline orchestrator's per-stage validator instead of the + legacy single-stage dry-run path; the `_dispatch.py` ordering was + inverted (no-train-mode ran before the pipeline branch). +- **F-B-2** — `--force-resume` now emits a `pipeline.force_resume` + audit event carrying `old_config_hash` + `new_config_hash` so + compliance reviewers can distinguish an operator-approved override + from a normal resume (was previously a WARNING log only — + invisible in the append-only JSONL stream). +- **F-B-3** — pipeline manifest verifier now compares every chain + stage against its **immediate** predecessor, not the most-recent + stage with an `output_model`. Pre-fix, a manifest where stage N-1 + failed without saving an `output_model` could appear to chain + stage N from stage N-2, masking the broken link. +- **F-F-1 / F-G-3 (Gemini)** — auto-chain existence guard now + checks the resolved model directory itself, not its parent — used + to accept `./stage1/` even when `./stage1/final_model` was + missing. +- **F-F-2** — `--input-model ""` (empty string) is normalised to + `None` at dispatch time so it no longer slips past the truthy + guard and silently overwrites the auto-chained model path with an + empty string. +- **F-F-3** — `_validate_resume_state` now returns its refusal + through the normal `run()` flow instead of `sys.exit`-ing mid-method, + so every refusal path emits the same audit-log finalisation and + the test surface uses one uniform `assert code == EXIT_CONFIG_ERROR` + shape. +- **F-S-2 / F-N-2** — verifier surfaces stages still in `running` + status on a finalised manifest as a `chain_integrity_violation` + candidate (a tell of a crashed orchestrator). Also `_load_state_file` + catches `KeyError` / `TypeError` / `ValueError` / `AttributeError` + around `_deserialise_state` so a tampered state file produces an + actionable config error rather than a Python traceback. +- **F-N-1** — gated stages now emit a dedicated + `pipeline.stage_gated` audit event (was `pipeline.stage_completed` + with a `gate_decision=approval_pending` sub-field). Lets + dashboard / SIEM filters distinguish the gate flow on the event + name alone. +- **F-G-1 (Gemini)** — pipeline dry-run now flags `training.output_dir` + collisions across stages (two stages writing to the same dir would + silently overwrite each other's checkpoints + per-stage Annex IV + manifests). Covers both the explicit-override and inherited-from- + root collision cases. + ### Security - **Webhook / judge / synthetic SSRF guard — DNS-rebinding TOCTOU diff --git a/config_template.yaml b/config_template.yaml index 6a7cfd83..fe35b495 100644 --- a/config_template.yaml +++ b/config_template.yaml @@ -223,3 +223,33 @@ data: # ephemeral_artefact_retention_days: 90 # compliance bundles + audit reports # raw_documents_retention_days: 90 # ingested raw docs (PDF/DOCX/EPUB/TXT/MD) # enforce: "log_only" # "log_only" | "warn_on_excess" | "block_on_excess" + +# Phase 14 — Multi-stage training pipeline chains (SFT → DPO → GRPO). +# Uncomment the ``pipeline:`` block below to run multiple training stages +# back-to-back; each stage's input model is auto-chained to the previous +# stage's output_dir/final_model. Stages inherit unset config blocks +# from the root (above). See docs/guides/pipeline.md. +# pipeline: +# output_dir: "./pipeline_run" # pipeline-level dir (state + manifest live here) +# stages: +# - name: sft_stage +# training: +# trainer_type: "sft" +# output_dir: "./pipeline_run/stage1_sft" +# num_train_epochs: 3 +# data: +# dataset_name_or_path: "./data/sft.jsonl" +# - name: dpo_stage +# training: +# trainer_type: "dpo" +# output_dir: "./pipeline_run/stage2_dpo" +# num_train_epochs: 1 +# dpo_beta: 0.1 +# data: +# dataset_name_or_path: "./data/preferences.jsonl" +# - name: grpo_stage +# training: +# trainer_type: "grpo" +# output_dir: "./pipeline_run/stage3_grpo" +# data: +# dataset_name_or_path: "./data/math_prompts.jsonl" diff --git a/docs/guides/pipeline-tr.md b/docs/guides/pipeline-tr.md new file mode 100644 index 00000000..f01c1b88 --- /dev/null +++ b/docs/guides/pipeline-tr.md @@ -0,0 +1,316 @@ +# Çok Aşamalı Eğitim Pipeline'ları + +Faz 14 — `v0.7.0` için hedefleniyor (şu an CHANGELOG'un `Unreleased` satırında). + +ForgeLM'in `pipeline:` config bloğu, 2 veya daha fazla eğitim aşamasını +(genellikle SFT → DPO → GRPO) tek bir config-tabanlı, dry-run-doğrulanabilir, +Annex IV-izlenebilir koşuya zincirler. Faz 14'ten önce aynı iş 3 ya da +daha fazla ayrı config dosyası, aşamalar arası elle `model.name_or_path` +düzenleme ve dışarıdan bir shell betiğiyle orkestrasyon gerektiriyordu. +Faz 14'ten sonra tek bir `forgelm --config pipeline.yaml` çağrısı tüm +zinciri uçtan uca yürütür. + +--- + +## When to use a pipeline (and when not to) + +`pipeline:` bloğunu **tüm** aşağıdaki koşullar geçerliyse kullanın: + +- Sıralı olarak 2 veya daha fazla eğitim aşaması koşturuyorsunuz + (örn. SFT sonrası DPO). +- Her aşamanın giriş modeli, önceki aşamanın çıkış modelidir. +- Tüm zinciri kapsayan tek bir Annex IV manifest istiyorsunuz. + +`pipeline:` bloğunu kullanmayın eğer: + +- Sadece tek bir eğitim paradigması koşturuyorsanız. Tek aşamalı + config'ler (v0.6.0 default'u) hâlâ kanonik kullanım örneğidir ve + Faz-14 öncesi davranışlarını byte-byte korur. +- Doğrusal olmayan aşama bağımlılıkları (DAG-şekilli pipeline'lar) + istiyorsanız. Faz 14 sadece sıralı pipeline'lar gönderir; şema + ilerideki bir DAG genişlemesi için yüzeyi koruyor. +- Paralel aşama yürütmesi (eşzamanlı bağımsız dallar) istiyorsanız. + DAG desteğiyle aynı zaman ufkunda — Wave 2 veya sonrası. + +--- + +## Anatomy of a pipeline config + +```yaml +# Root config — her aşamanın aşmadığı sürece miras aldığı varsayılanları +# sağlar. +model: + name_or_path: "meta-llama/Llama-3-8B" # Aşama 0'ın başlangıç modeli +lora: + r: 8 + alpha: 16 +training: + trainer_type: "sft" # Training bloğunu miras alan + # aşamalar bunu kullanır +data: + dataset_name_or_path: "./placeholder.jsonl" # Root'ta zorunlu; her + # aşama kendi data'sını verir + +# Pipeline-level blok — zinciri tanımlar. +pipeline: + output_dir: "./pipeline_run" # pipeline_state.json, + # compliance/pipeline_manifest.json + # ve pipeline-level audit_log.jsonl + # buraya yazılır + stages: + - name: sft_stage # ^[a-z0-9_]{1,32}$ + training: + trainer_type: "sft" # Her aşama için zorunlu + output_dir: "./pipeline_run/stage1_sft" + num_train_epochs: 3 + data: + dataset_name_or_path: "./data/sft.jsonl" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./pipeline_run/stage2_dpo" + num_train_epochs: 1 + dpo_beta: 0.1 + data: + dataset_name_or_path: "./data/preferences.jsonl" + + - name: grpo_stage + training: + trainer_type: "grpo" + output_dir: "./pipeline_run/stage3_grpo" + grpo_num_generations: 4 + data: + dataset_name_or_path: "./data/math_prompts.jsonl" +``` + +**Otomatik zincirleme:** her aşamanın `model.name_or_path` değeri, +önceki aşamanın `training.output_dir/final_model` yoluna otomatik +olarak ayarlanır — bu yolu elinizle yazmazsınız ve aşama N'in DPO +eğiticisini yanlışlıkla base modele (aşama N-1'in SFT çıktısı yerine) +yönlendiremezsiniz. + +--- + +## Inheritance matrix + +Section-wholesale override semantiği — bir aşama üst seviye blok (`model`, +`lora`, `training`, `data`, `evaluation`) tanımlarsa **tüm** blok root'unkini +değiştirir; aşama bloğu atlarsa root'un bloğu birebir miras alınır. Field +seviyesinde deep-merge yok: "miras almak istiyorsan bloğu yazma; override +etmek istiyorsan tam bloğu yaz." + +| Bölüm | Miras davranışı | Notlar | +|---|---|---| +| `model.name_or_path` | **Otomatik zincirlenir** (aşama 1'den itibaren root'u geçer) | Önceki aşamanın `training.output_dir/final_model` değerine ayarlanır. Aşama 0 root'un `model.name_or_path`'ini okur. Aşamada explicit `model:` bloğu otomatik zincirlemeyi o aşama için devre dışı bırakır (kaçış kapısı). | +| `model.*` (diğer alanlar) | `model:` bloğu override edilmediyse miras alınır | `backend`, `load_in_4bit`, `trust_remote_code`, `max_length`, `chat_template` root'u izler. | +| `lora` | `lora:` bloğu override edilmediyse miras alınır | **Kritik edge case:** aşama 2 DPO, aşama 1 SFT'den *farklı* bir `lora.r` ile gelirse aşama 2 **birleştirilmiş SFT modelinin üstüne taze bir LoRA**dır, SFT'nin LoRA'sının devamı değil. Operatör-görünür davranış. | +| `data` | `data:` bloğu override edilmediyse miras alınır (önerilmez) | Şema aşama bazında override'ı **zorunlu kılmıyor** — operatörler farklı bir trainer'a karşı aynı veri setini ablation için yeniden koşturabilir — ama production'da aşamalar arası aynı dataset'i kullanan pipeline'lar son derece nadirdir. Operator kılavuzu inheritance'ı bir kötü-koku olarak işaretler; gerçek pipeline'ların neredeyse tümünde SFT küratör SFT dataset'i kullanır, DPO preference-pairs dataset'i, GRPO matematik/ödül dataset'i. | +| `training` | `training:` bloğu override edilmediyse miras alınır | `trainer_type` her aşamada explicit set EDİLMEK ZORUNDADIR (audit-clarity doğrulayıcısı). Diğer alanlar blok override edildiğinde wholesale yer değiştirir. | +| `evaluation` | `evaluation:` bloğu override edilmediyse miras alınır | Aşama bazında gate'ler (loss eşikleri, auto_revert, safety, judge, human-approval) burada yaşar. | +| `distributed` | Sadece root — **aşamada reddedilir** | Distributed strateji koşu boyunca tutarlı olmak zorundadır. | +| `webhook` | Sadece root — **aşamada reddedilir** | Aşama bazında olaylar payload'da aşama adını taşır. | +| `compliance` | Sadece root — **aşamada reddedilir** | Provider/system/risk metadata pipeline seviyesindedir. | +| `risk_assessment`, `monitoring`, `retention`, `synthetic`, `merge`, `auth` | Sadece root — **aşamada reddedilir** | Pipeline seviyesi sorumlulukları. | + +Bir aşama root-only bölümlerden birini tanımlarsa, config yükleme +zamanında `EXIT_CONFIG_ERROR (1)` ile reddedilir ve hata mesajı +ihlal eden bölüm adını taşır. + +--- + +## CLI + +```bash +# Uçtan uca pipeline koşusu. +forgelm --config pipeline.yaml + +# Dry-run her aşamayı (Pydantic + chain integrity) hiçbir GPU +# tahsisi olmadan doğrular. `pytest --collectonly` gibi tüm +# hataları çıkmadan önce toplar. +forgelm --config pipeline.yaml --dry-run + +# Tek bir adlı aşamayı yalıtılmış olarak koştur (audit / yeniden +# koşturma senaryoları). İlk olmayan aşamalar önceki aşamanın +# disk üzerindeki çıktısını veya explicit --input-model override +# gerektirir. +forgelm --config pipeline.yaml --stage dpo_stage + +# Başarısız / kesintiye uğramış bir koşudan adlı aşamadan itibaren +# devam et. output_model yolu disk üzerinde var olan +# already-completed aşamalar atlanır (INFO log'lanır). +forgelm --config pipeline.yaml --resume-from dpo_stage + +# Tek aşama için otomatik zincirlenen input modeli override et +# (kaçış kapısı). Audit log `input_source: cli_override` kaydeder. +forgelm --config pipeline.yaml --stage dpo_stage --input-model ./other/checkpoint +``` + +### `--stage ` partial-run rules + +| Senaryo | Davranış | +|---|---| +| `--stage ` ve `` ilk aşama | Root `model.name_or_path` okunur; normal koşar. | +| `--stage ` ve önceki aşamanın `output_dir/final_model` diskte var | Önceki aşama yeni bitmiş gibi otomatik zincirlenir. | +| `--stage ` ve önceki aşamanın çıktısı eksik | `EXIT_CONFIG_ERROR (1)` ile hard-fail: `Stage requires output at ; pass --input-model to override or run the full pipeline first.` Root `model.name_or_path`'e sessiz geri dönüş yok. | +| `--stage --input-model ` | Operatör kaçış kapısı: otomatik zincirlemeyi atlar, ``'i kullanır. Audit log `input_source: cli_override` kaydeder. | +| `--stage ` ve `` config'te yok | Parse zamanında valid aşama adları listesiyle hard-fail. | + +### `--resume-from ` semantics + +- State dosyası: `/pipeline_state.json` (atomic-write). +- `status: completed` ve `output_model` yolu hâlâ diskte olan aşamalar + **atlanır** ve aşama bazında training manifest'leri korunur. Adlı + aşama ve sonrasındaki her aşama yeniden koşturulur. +- **Stale-state koruması:** disk üzerindeki state dosyasının + `pipeline_config_hash`'i mevcut YAML byte'larından farklıysa, resume + `EXIT_CONFIG_ERROR (1)` ile başarısız olur — bu "koşu sırasında + config'i düzenledim ve resume ettim" sessiz divergenc'yi engeller. + `--force-resume` ile override edilebilir (WARNING log'lanır, audit + event kaydedilir). +- Faz 14 sadece **aşama sınırlarında** resume eder. Aşama içi HF + `Trainer.train(resume_from_checkpoint=...)` entegrasyonu bir Faz 14.x + takipçisine ertelendi — Sınırlamalar bölümüne bakın. + +--- + +## Human-approval gate within a pipeline + +Bir aşama `evaluation.require_human_approval: true` taşıyorsa, ForgeLM'in +mevcut Faz 9 akışı aynen koşar: model `final_model.staging./`'ye +iner, orkestratör staging yolunu pipeline state'inde +`status: gated_pending_approval` ile yakalar ve koşu +`EXIT_AWAITING_APPROVAL (4)` ile çıkar. + +Sonraki aşamalar `pending` kalır (`skipped_due_to_prior_revert` değil) +böylece onaydan sonra resume onları alır: + +```bash +# Aşama 1 SFT onay bekliyor — pipeline 4 ile çıkar. +$ forgelm --config pipeline.yaml +# ... → exit 4 + +# Operatör staged modeli inceler, sonra onaylar. +$ forgelm approve --output-dir ./pipeline_run/stage1_sft +# ... → final_model/ promote edilir, exit 0 + +# Pipeline'ı DPO'dan resume et; SFT atlanır (status=completed on disk). +# DPO'nun input_model'i staging değil *promote edilen* final_model/ +# yolunu işaret eder. +$ forgelm --config pipeline.yaml --resume-from dpo_stage +# ... → zincirin geri kalanı başarılıysa exit 0 +``` + +--- + +## Audit events + +Orkestratör bu olayları pipeline-level `audit_log.jsonl`'a (pipeline.output_dir +altında) emit eder: + +- `pipeline.started` — run id, config hash, stage count, stage names +- `pipeline.stage_started` — stage name, index, input model, input source +- `pipeline.stage_completed` — stage name, gate decision (`passed` / `failed`), metrics summary +- `pipeline.stage_gated` — stage name, gate decision `approval_pending`, staging path (bir aşama `EXIT_AWAITING_APPROVAL` ile çıkış yaptığında emit edilir) +- `pipeline.stage_reverted` — stage name, auto-revert reason +- `pipeline.force_resume` — operatör onaylı stale-hash override, `old_config_hash` + `new_config_hash` taşır +- `pipeline.completed` — final status, stopped-at stage (varsa) + +Bu olaylar mevcut her aşamanın `ForgeTrainer`'inin emit etmeye devam +ettiği `training.*` olaylarının yanında yaşar; `training.failure` +üzerinde filtreleyen Slack / Teams dashboard'ları değişmeden çalışır. + +İlgili webhook metotları `notify_pipeline_started`, +`notify_pipeline_completed` ve `notify_pipeline_reverted` — +`forgelm/webhook.py`'a bakın. + +--- + +## Annex IV manifest + +Her aşama geçişi `/compliance/pipeline_manifest.json` +dosyasını (atomic-write) yeniden yazar. Pipeline manifest, aşama bazında +`training_manifest.json` dosyalarını doğrulanabilir bir zincire bağlayan +**indekstir**; aşama bazında manifest'ler tek aşamalı Annex IV şemasına +karşı bireysel olarak geçerli kalmaya devam eder. + +Tam bir koşuyu doğrulayıcı ile geçerleyin: + +```bash +forgelm verify-annex-iv --pipeline ./pipeline_run +``` + +Doğrulayıcı şu koşullar sağlandığında exit 0 döner: + +1. Her zorunlu üst seviye anahtar mevcut ve iyi biçimli. +2. Aşama indeksleri sırayla `0..N-1`. +3. Her chain aşamasının `input_model`'i önceki yürütülen aşamanın + `output_model`'iyle eşit (operatör `--input-model` override'ları + `input_source: cli_override` ile kaydedilir ve bu kontrolden + tasarımla muaftır — auditor'lar legitimate override'ı bozuk + manifest'ten ayırmak için audit log'a bakar). +4. Her aşama bazında `training_manifest` yolu gerçek bir dosyayı + işaret eder. +5. `stopped_at` (set ise) gerçek bir aşamayı adlandırır ve o aşamanın + status'u `failed` veya `gated_pending_approval`. + +Sıfır olmayan exit kodu ihlalleri keşfedildikleri sırayla listeler. + +--- + +## Limitations (Phase 14 Wave 1) + +- **Aşama içi checkpoint resume yok.** `--resume-from` sadece aşama + sınırlarında devam alır. Bir aşamanın `ForgeTrainer.train()` + yarıda crash ederse, resume o aşamayı epoch 0'dan yeniden koşturur. + Wave 2. +- **Sadece sıralı — DAG semantiği yok.** Aşamalar `pipeline.stages` + içinde görünme sıralarında yürütülür. Branch / fan-out / merge + ertelendi. +- **Paralel aşama yürütmesi yok.** İki aşama mantıken bağımsız olsa + bile orkestratör onları sıralı koşturur. +- **`pipeline.output_dir` üzerinde multi-process lock yok.** Aynı + `pipeline.output_dir`'a karşı eş zamanlı iki `forgelm --config + pipeline.yaml` çağrısı `pipeline_state.json` + manifest üzerinde + race oluşturur — eş zamanlı koşular için ayrı `pipeline.output_dir` + seçin. Atomic-write helper'ı writer başına benzersiz temp adı + kullanır, dolayısıyla race son-yazan-kazanır şeklinde yüzeyleşir + (FileNotFoundError traceback değil), ama kaybedenin per-stage + ilerlemesi kendi process'inde görünmez kalır. +- **Stage ve pipeline `output_dir` çakışması mümkün ama audit log'u + iç içe geçirir.** Bir aşamanın çözülen `training.output_dir`'ı + `pipeline.output_dir`'a eşit olduğunda, aşamanın `training.*` + olayları ile pipeline'ın `pipeline.*` olayları aynı + `audit_log.jsonl` dosyasına düşer (dizin başına bir dosya). + Hash-chain + satır-bazında flock dosyayı iç tutarlı tutar ve + `tests/fixtures/pipeline/inheritance_matrix.yaml` fixture'ı bu + layout'u kasten icra eder; ama dosya başına tek `event_kind` + bekleyen SIEM filtreleri ayrı dizinler kullanmalı. +- **`--force-resume` topology guard yalnızca pozisyonel kırılmaları + yakalar.** Stage ekleme, silme veya yeniden adlandırma + `--force-resume` altında bile reddedilir (state.stages[i] başka + bir aşamayı adresliyor olur). *Value-level* düzenlemeler — aynı + stage adı altında farklı `trainer_type`, `training.output_dir`, + hyperparameter — tasarımı gereği kabul edilir; `--force-resume` + operatörün ayrışmayı anladığını bildiren sinyaldir. +- **`forgelm wizard` entegrasyonu yok.** Tek aşamalı config'ler + wizard'la üretilebilir; pipeline'lar operatör-seviyesi ve manuel + YAML yüzeyini kullanır. Wizard'ın işi "çalışan bir tek aşama + config'i üret, büyüdüğünde elle pipeline'a düzenle" olarak kalır. +- **Notebook entegrasyonu yok.** `notebooks/` altındaki 11 demo + notebook bireysel eğitim paradigmalarını kapsar; uçtan uca pipeline + demosu her notebook'un setup boilerplate'ini üç kez tekrar ederdi. + `tests/fixtures/pipeline/` altındaki fixture suite, reviewer'lara + bir notebook'la aynı yüzeyi verir — üstüne golden manifest'le + byte-comparable olma avantajıyla. + +--- + +## Cross-references + +- Faz 14 tasarım dokümanı: [docs/roadmap/phase-14-pipeline-chains.md](../roadmap/phase-14-pipeline-chains.md) +- Roadmap girişi: [docs/roadmap-tr.md](../roadmap-tr.md) +- Annex IV doğrulayıcısı: `forgelm verify-annex-iv --pipeline ` (CLI help'e bakın) +- Audit log standardı: [docs/standards/logging-observability.md](../standards/logging-observability.md) +- Tek aşamalı eğitici kılavuzu (her şey bu yüzeyden miras alır): + [docs/guides/alignment-tr.md](alignment-tr.md) diff --git a/docs/guides/pipeline.md b/docs/guides/pipeline.md new file mode 100644 index 00000000..a1f94e7d --- /dev/null +++ b/docs/guides/pipeline.md @@ -0,0 +1,313 @@ +# Multi-Stage Training Pipelines + +Phase 14 — targeted for `v0.7.0` (currently on the `Unreleased` line of the CHANGELOG). + +ForgeLM's `pipeline:` config block chains 2 or more training stages +(typically SFT → DPO → GRPO) into one config-driven, dry-run-validatable, +Annex-IV-traceable run. Before Phase 14, the same workflow required +3+ separate config files, manual `model.name_or_path` editing between +stages, and an external shell script for orchestration. After Phase 14, +a single `forgelm --config pipeline.yaml` invocation runs the entire +chain end-to-end. + +--- + +## When to use a pipeline (and when not to) + +Use the `pipeline:` block when **all** of the following hold: + +- You are running 2 or more training stages in sequence (e.g. SFT + followed by DPO). +- Each stage's input model is the previous stage's output model. +- You want a single Annex IV manifest covering the entire chain. + +Do **not** use the `pipeline:` block when: + +- You only need to run a single training paradigm. Single-stage configs + (the v0.6.0 default) remain the canonical case and stay byte-identical + to their pre-Phase-14 behaviour. +- You need non-linear stage dependencies (DAG-shaped pipelines). + Phase 14 ships sequential pipelines only; the schema reserves the + surface for a future DAG extension. +- You want parallel stage execution (independent branches running + concurrently). Same horizon as the DAG support — Wave 2 or later. + +--- + +## Anatomy of a pipeline config + +```yaml +# Root config — provides defaults that every stage inherits from +# unless the stage overrides them. +model: + name_or_path: "meta-llama/Llama-3-8B" # Stage 0's starting model +lora: + r: 8 + alpha: 16 +training: + trainer_type: "sft" # Used by stages that + # inherit the training block +data: + dataset_name_or_path: "./placeholder.jsonl" # Required at root; each + # stage overrides per-stage + +# Pipeline-level block — declares the chain. +pipeline: + output_dir: "./pipeline_run" # Hosts pipeline_state.json, + # compliance/pipeline_manifest.json, + # and the pipeline-level + # audit_log.jsonl + stages: + - name: sft_stage # ^[a-z0-9_]{1,32}$ + training: + trainer_type: "sft" # Required per stage + output_dir: "./pipeline_run/stage1_sft" + num_train_epochs: 3 + data: + dataset_name_or_path: "./data/sft.jsonl" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./pipeline_run/stage2_dpo" + num_train_epochs: 1 + dpo_beta: 0.1 + data: + dataset_name_or_path: "./data/preferences.jsonl" + + - name: grpo_stage + training: + trainer_type: "grpo" + output_dir: "./pipeline_run/stage3_grpo" + grpo_num_generations: 4 + data: + dataset_name_or_path: "./data/math_prompts.jsonl" +``` + +**Auto-chain:** each stage's `model.name_or_path` is automatically set to +the previous stage's `training.output_dir/final_model` — you do not +write that path by hand, and you cannot accidentally point stage N's +DPO trainer at the base model instead of stage N-1's SFT output. + +--- + +## Inheritance matrix + +Section-wholesale override semantics — if a stage declares a top-level +block (`model`, `lora`, `training`, `data`, `evaluation`), the **entire** +block replaces the root's; if the stage omits the block, the root's +block is inherited verbatim. No field-level deep-merge: "if you want to +inherit, omit the block; if you want to override, supply the full block." + +| Section | Inheritance | Notes | +|---|---|---| +| `model.name_or_path` | **Auto-chained** (overrides root from stage 1 onward) | Set to previous stage's `training.output_dir/final_model`. Stage 0 reads root `model.name_or_path`. An explicit per-stage `model:` block disables auto-chaining for that stage (escape hatch). | +| `model.*` (other fields) | Inherited unless `model:` block overridden | `backend`, `load_in_4bit`, `trust_remote_code`, `max_length`, `chat_template` follow the root. | +| `lora` | Inherited unless `lora:` block overridden | **Critical edge case:** stage 2 DPO with a *different* `lora.r` than stage 1 SFT means stage 2 is **a fresh LoRA over the merged SFT model**, not a continuation of SFT's LoRA. Operator-visible behaviour. | +| `data` | Inherited unless `data:` block overridden (strongly discouraged) | The schema does **not** force a per-stage override — operators may rerun the same dataset against a different trainer for ablation studies — but pipelines that legitimately reuse the same dataset across stages are vanishingly rare in production. The operator guide flags inheritance as a smell; almost every real chain has SFT use a curated SFT dataset, DPO use a preference-pairs dataset, and GRPO use a math/reward dataset. | +| `training` | Inherited unless `training:` block overridden | `trainer_type` MUST be set explicitly per stage (audit-clarity validator). Other fields wholesale-replaced when the block is overridden. | +| `evaluation` | Inherited unless `evaluation:` block overridden | Per-stage gates (loss thresholds, auto_revert, safety, judge, human-approval) live here. | +| `distributed` | Root only — **per-stage rejected** | Distributed strategy must be consistent across the run. | +| `webhook` | Root only — **per-stage rejected** | Per-stage events carry the stage name in the payload. | +| `compliance` | Root only — **per-stage rejected** | Provider/system/risk metadata is pipeline-level. | +| `risk_assessment`, `monitoring`, `retention`, `synthetic`, `merge`, `auth` | Root only — **per-stage rejected** | Pipeline-level concerns. | + +A stage that declares any of the root-only sections is rejected at +config-load time with `EXIT_CONFIG_ERROR (1)` and the offending section +name in the message. + +--- + +## CLI + +```bash +# End-to-end pipeline run. +forgelm --config pipeline.yaml + +# Dry-run validates every stage (Pydantic + chain integrity) before any +# GPU is allocated. Collects all errors before exiting, like +# `pytest --collectonly`. +forgelm --config pipeline.yaml --dry-run + +# Run a single named stage in isolation (audit / re-run scenarios). +# Non-first stages require the previous stage's on-disk output, or +# an explicit --input-model override. +forgelm --config pipeline.yaml --stage dpo_stage + +# Resume after a failed / interrupted run from a named stage onward. +# Already-completed stages whose output paths exist on disk are +# skipped (logged at INFO). +forgelm --config pipeline.yaml --resume-from dpo_stage + +# Override the auto-chained input model for a single stage (escape hatch). +# The audit log records `input_source: cli_override`. +forgelm --config pipeline.yaml --stage dpo_stage --input-model ./other/checkpoint +``` + +### `--stage ` partial-run rules + +| Scenario | Behaviour | +|---|---| +| `--stage ` and `` is the first stage | Reads root `model.name_or_path`; runs normally. | +| `--stage ` and the previous stage's `output_dir/final_model` exists on disk | Auto-chains as if the previous stage had just finished. | +| `--stage ` and the previous stage's output is missing | Hard-fails with `EXIT_CONFIG_ERROR (1)`: `Stage requires output at ; pass --input-model to override or run the full pipeline first.` No silent fallback to root `model.name_or_path`. | +| `--stage --input-model ` | Operator escape hatch: skips the auto-chain, uses ``. Audit-log records `input_source: cli_override`. | +| `--stage ` where `` is not in the config | Hard-fails at parse time with the list of valid stage names. | + +### `--resume-from ` semantics + +- State file: `/pipeline_state.json` (atomic-write). +- Stages with `status: completed` whose `output_model` path still exists + on disk are **skipped** and their per-stage training manifests are + preserved. The named stage and every stage after it are re-run. +- **Stale-state guard:** if the on-disk state file's + `pipeline_config_hash` differs from the current parse of the YAML + bytes, resume fails with `EXIT_CONFIG_ERROR (1)` — preventing + "I resumed against a config that was edited mid-flight" silent + divergence. Override via `--force-resume` (logged at WARNING, + recorded in the audit event). +- Phase 14 resumes at **stage boundaries** only. Intra-stage HF + `Trainer.train(resume_from_checkpoint=...)` integration is deferred + to a Phase 14.x follow-up — see Limitations below. + +--- + +## Human-approval gate within a pipeline + +If a stage carries `evaluation.require_human_approval: true`, ForgeLM's +existing Phase 9 flow runs unchanged: the model lands in +`final_model.staging./`, the orchestrator captures the staging +path in the pipeline state with `status: gated_pending_approval`, and +the run exits with `EXIT_AWAITING_APPROVAL (4)`. + +Downstream stages stay `pending` (not `skipped_due_to_prior_revert`) +so a subsequent resume picks them up after the approval: + +```bash +# Stage 1 SFT gates pending approval — pipeline exits 4. +$ forgelm --config pipeline.yaml +# ... → exit 4 + +# Operator inspects the staged model, then approves. +$ forgelm approve --output-dir ./pipeline_run/stage1_sft +# ... → final_model/ is promoted, exit 0 + +# Resume the pipeline from DPO; SFT is skipped (status=completed +# on disk). DPO's input_model points at the *promoted* final_model/ +# path, not the staging path. +$ forgelm --config pipeline.yaml --resume-from dpo_stage +# ... → exit 0 when the rest of the chain succeeds +``` + +--- + +## Audit events + +The orchestrator emits these events into the pipeline-level +`audit_log.jsonl` (under `pipeline.output_dir`): + +- `pipeline.started` — run id, config hash, stage count, stage names +- `pipeline.stage_started` — stage name, index, input model, input source +- `pipeline.stage_completed` — stage name, gate decision (`passed` / `failed`), metrics summary +- `pipeline.stage_gated` — stage name, gate decision `approval_pending`, staging path (emitted when a stage exits `EXIT_AWAITING_APPROVAL`) +- `pipeline.stage_reverted` — stage name, auto-revert reason +- `pipeline.force_resume` — operator-approved stale-hash override with `old_config_hash` + `new_config_hash` +- `pipeline.completed` — final status, stopped-at stage (if any) + +These events live alongside the existing `training.*` events that each +stage's `ForgeTrainer` continues to emit; pre-existing Slack / Teams +dashboards filtering on `training.failure` keep working unchanged. + +The matching webhook methods are `notify_pipeline_started`, +`notify_pipeline_completed`, and `notify_pipeline_reverted` — see +`forgelm/webhook.py`. + +--- + +## Annex IV manifest + +Every stage transition rewrites `/compliance/ +pipeline_manifest.json` (atomic-write). The pipeline manifest is the +**index** that ties the per-stage `training_manifest.json` files into +one verifiable chain; per-stage manifests remain individually valid +against the existing single-stage Annex IV schema. + +Validate a complete run with the verifier: + +```bash +forgelm verify-annex-iv --pipeline ./pipeline_run +``` + +The verifier returns exit 0 when: + +1. Every required top-level key is present and well-formed. +2. Stage indices form `0..N-1` in order. +3. Every chain stage's `input_model` equals the previous executed + stage's `output_model` (operator `--input-model` overrides are + recorded with `input_source: cli_override` and exempted from this + check, by design — auditors cross-reference the audit log to + distinguish a legitimate override from a corrupt manifest). +4. Every per-stage `training_manifest` path points at a real file. +5. `stopped_at` (if set) names a real stage whose status is `failed` + or `gated_pending_approval`. + +A non-zero exit code lists the violations in the order they were +discovered. + +--- + +## Limitations (Phase 14 Wave 1) + +- **No intra-stage checkpoint resume.** `--resume-from` picks up at + stage boundaries only. If a stage's `ForgeTrainer.train()` crashes + half-way, the resume re-runs that stage from epoch 0. Wave 2. +- **Sequential only — no DAG semantics.** Stages execute in the order + they appear in `pipeline.stages`. Branches / fan-out / merge are + deferred. +- **No parallel stage execution.** Even when two stages are logically + independent, the orchestrator runs them sequentially. +- **No multi-process locking on `pipeline.output_dir`.** Running two + `forgelm --config pipeline.yaml` invocations against the same + `pipeline.output_dir` simultaneously will race on + `pipeline_state.json` + the manifest — pick distinct + `pipeline.output_dir` per concurrent run. The atomic-write helper + uses per-writer-unique temp filenames so the race surfaces as + last-writer-wins (not a `FileNotFoundError` traceback), but the + loser's per-stage progress is invisible to its own process. +- **Stage-vs-pipeline `output_dir` overlap is allowed but interleaves + the audit log.** When a stage's resolved `training.output_dir` + equals `pipeline.output_dir`, the stage's `training.*` events and + the pipeline's `pipeline.*` events land in the same + `audit_log.jsonl` (one file per directory). The hash-chain + per- + line flock keep the merged file internally consistent and the + `tests/fixtures/pipeline/inheritance_matrix.yaml` fixture + deliberately exercises this layout, but SIEM filters that pin a + single `event_kind` per file should pick distinct directories. +- **`--force-resume` topology guard catches positional breaks only.** + Adding, deleting, or renaming a stage between runs is refused even + under `--force-resume` (state.stages[i] would address a different + stage than pipeline.stages[i]). *Value-level* edits — different + `trainer_type`, `training.output_dir`, hyperparameters under + identical stage names — are accepted by design; `--force-resume` + is the operator's signal that they understand the divergence. +- **No `forgelm wizard` integration.** Single-stage configs are + wizard-buildable; pipelines are operator-grade and use the manual + YAML surface. The wizard's job stays "produce a working single-stage + config you can hand-edit into a pipeline if you outgrow it." +- **No notebook integration.** The 11 demo notebooks under + `notebooks/` cover individual training paradigms; an end-to-end + pipeline demo would duplicate every notebook's setup boilerplate + three times. The fixture suite under `tests/fixtures/pipeline/` + gives reviewers exactly the same surface as a notebook would, with + the advantage of being byte-comparable to a golden manifest. + +--- + +## Cross-references + +- Phase 14 design doc: [docs/roadmap/phase-14-pipeline-chains.md](../roadmap/phase-14-pipeline-chains.md) +- Roadmap entry: [docs/roadmap.md](../roadmap.md) +- Annex IV verifier: `forgelm verify-annex-iv --pipeline ` (see CLI help) +- Audit log standard: [docs/standards/logging-observability.md](../standards/logging-observability.md) +- Single-stage trainer guide (everything inherits from this surface): + [docs/guides/alignment.md](alignment.md) diff --git a/docs/reference/data_ingestion_architecture-tr.md b/docs/reference/data_ingestion_architecture-tr.md index 29d71f04..0664b612 100644 --- a/docs/reference/data_ingestion_architecture-tr.md +++ b/docs/reference/data_ingestion_architecture-tr.md @@ -209,7 +209,7 @@ graph TD PAGES[sayfa başına metin] --> SPLIT[Satırlara böl] SPLIT --> WINDOW[Sayfa başına üst-3 / alt-3 satırı incele
_PDF_EDGE_WINDOW = 3] WINDOW --> COUNT[Tüm sayfalarda satır
frekansını say] - COUNT --> CHECK{>= yüzde 70 sayfa?} + COUNT --> CHECK{">= yüzde 70 sayfa?"} CHECK -- evet --> POP[Eşleşen window satırlarını çıkar
window içinde herhangi bir offset] POP --> WINDOW CHECK -- hayır --> JOIN[Sayfaları çift yeni satırla birleştir] diff --git a/docs/reference/data_ingestion_architecture.md b/docs/reference/data_ingestion_architecture.md index fbc94c20..44dbdb6d 100644 --- a/docs/reference/data_ingestion_architecture.md +++ b/docs/reference/data_ingestion_architecture.md @@ -207,7 +207,7 @@ graph TD PAGES[per-page text] --> SPLIT[Split into lines] SPLIT --> WINDOW[Inspect top-3 / bottom-3 rows
per page _PDF_EDGE_WINDOW = 3] WINDOW --> COUNT[Count line frequency
across pages] - COUNT --> CHECK{>= 70 percent of pages?} + COUNT --> CHECK{">= 70 percent of pages?"} CHECK -- yes --> POP[Pop matching window lines
any offset within the window] POP --> WINDOW CHECK -- no --> JOIN[Re-join pages with double newline] diff --git a/docs/roadmap/completed-phases.md b/docs/roadmap/completed-phases.md index 4ea6faf5..68f45d2a 100644 --- a/docs/roadmap/completed-phases.md +++ b/docs/roadmap/completed-phases.md @@ -422,7 +422,7 @@ graph LR B -->|classify| C[Llama Guard] C --> D{safe / unsafe} D -->|count| E[unsafe_ratio] - E -->|compare| F{> threshold?} + E -->|compare| F{"> threshold?"} F -->|yes| G[❌ REVERT] F -->|no| H[✅ PASS] ``` diff --git a/docs/roadmap/phase-14-pipeline-chains.md b/docs/roadmap/phase-14-pipeline-chains.md index 014e1e84..501ff6cf 100644 --- a/docs/roadmap/phase-14-pipeline-chains.md +++ b/docs/roadmap/phase-14-pipeline-chains.md @@ -1,6 +1,6 @@ # Phase 14: Multi-Stage Training Pipeline Chains -> **Status:** Planning. Targeted for `v0.7.0` (originally `v0.6.0`; Phase 15 displaced it after the 2026-05-11 ingestion pilot — see [completed-phases.md#phase-15-ingestion-pipeline-reliability-v060](completed-phases.md#phase-15-ingestion-pipeline-reliability-v060)). No hard blockers; `v0.6.0` has shipped, Phase 14 can start. +> **Status:** Implementation complete on `development` (branch `feat/phase-14-pipeline-chains`). Awaiting `v0.7.0` PyPI tag for the final "Done" promotion to [completed-phases.md](completed-phases.md). Originally targeted for `v0.6.0`; Phase 15 displaced it after the 2026-05-11 ingestion pilot — see [completed-phases.md#phase-15-ingestion-pipeline-reliability-v060](completed-phases.md#phase-15-ingestion-pipeline-reliability-v060). > **Note:** This file details a single planned phase. See [../roadmap.md](../roadmap.md) for the cross-phase summary. @@ -62,7 +62,7 @@ | `model.name_or_path` | **Auto-chained** (overrides root after stage 0) | Set to previous stage's `training.output_dir/final_model`. Stage 0 still reads root `model.name_or_path`. An explicit per-stage `model.name_or_path` is allowed and disables auto-chaining for that stage (operator escape hatch). | | `model.*` (other fields) | Inherited unless `model:` block overridden | `backend`, `load_in_4bit`, `trust_remote_code`, `max_length`, `chat_template` follow the root unless the stage supplies a full `model:` block. | | `lora` | Inherited unless `lora:` block overridden | Critical edge case: stage 2 DPO with a *different* `lora.r` than stage 1 SFT means stage 2 is **a fresh LoRA over the merged SFT model**, not a continuation of SFT's LoRA. Documented in the operator guide. | - | `data` | **Must be overridden per stage** | Pipelines that don't change the dataset between stages are vanishingly rare; explicit per-stage data is a forcing function against accidental dataset reuse. | + | `data` | Inherited unless `data:` block overridden (strongly discouraged) | Pipelines that don't change the dataset between stages are vanishingly rare — almost every real chain has SFT use a curated SFT dataset, DPO use a preference pairs dataset, and GRPO use a math/reward dataset. The schema does not *force* a per-stage override (operators may legitimately rerun the same dataset against a different trainer for ablation studies), but the operator guide flags the inheritance case as a smell and the dry-run summary prints a warning when two consecutive stages share the same `dataset_name_or_path`. | | `training` | Inherited unless `training:` block overridden | `trainer_type` is required per stage; the validator rejects a stage with no `training.trainer_type` even if the root supplies one (each stage explicitly states its alignment paradigm for audit clarity). | | `evaluation` | Inherited unless `evaluation:` block overridden | Per-stage gates (Task 5) live here; a stage that wants different `auto_revert` / `safety` config supplies a full `evaluation:` block. | | `distributed` | Inherited (always); no per-stage override | Distributed strategy must be consistent across the pipeline run. Per-stage `distributed:` blocks rejected at load with a clear error. | @@ -181,8 +181,10 @@ **Audit-log events (extends the existing 5-event vocabulary):** - `pipeline.started` — run id, config hash, stage count, stage names - `pipeline.stage_started` — stage name, index, input model, input source (chain/cli_override/root) - - `pipeline.stage_completed` — stage name, metrics summary, gate decision + - `pipeline.stage_completed` — stage name, metrics summary, gate decision (`passed` / `failed`) + - `pipeline.stage_gated` — stage name, gate decision `approval_pending`, staging path (emitted instead of `stage_completed` when a stage exits `EXIT_AWAITING_APPROVAL`; lets dashboard / SIEM rules filter on the event name alone — Phase 14 review F-N-1) - `pipeline.stage_reverted` — stage name, auto-revert reason, halt-pipeline=true + - `pipeline.force_resume` — operator-approved stale-hash override; carries `old_config_hash` + `new_config_hash` so reviewers can correlate to the audit trail (Phase 14 review F-B-2) - `pipeline.completed` — final status, total duration, stage count These events live alongside (not replacing) the existing per-stage `training.*` events emitted by `ForgeTrainer`. Webhook notifier (`forgelm/webhook.py`) gains `notify_pipeline_started` / `notify_pipeline_completed` / `notify_pipeline_reverted` methods that wrap the existing 5-event vocabulary so Slack / Teams dashboards filtering on `event=training.failure` continue to work; pipeline-aware dashboards can additionally filter on the new `pipeline.*` events. @@ -246,4 +248,17 @@ Each fixture has a golden pipeline manifest committed alongside; regression suit - **Standards consulted:** [`docs/standards/error-handling.md`](../standards/error-handling.md) (exit code stability), [`docs/standards/architecture.md`](../standards/architecture.md) (layering — orchestrator above trainer), [`docs/standards/testing.md`](../standards/testing.md) (fixture matrix discipline), [`docs/standards/localization.md`](../standards/localization.md) (EN ↔ TR mirror). - **Pattern reference:** Phase 15's review-absorption discipline + fixture matrix is the working model — see [completed-phases.md#phase-15-ingestion-pipeline-reliability-v060](completed-phases.md#phase-15-ingestion-pipeline-reliability-v060). +### Release-time follow-up (deferred to the `v0.7.0` `cut-release` cycle) + +Phase 14 implementation ships on the `development` branch ahead of the +PyPI tag. The two operator-facing surfaces below are intentionally not +edited inside this PR — they are released-product surfaces and stay +v0.6.0-shaped until `cut-release` actually tags `v0.7.0`: + +- **`docs/usermanuals/_meta.yaml` + `docs/usermanuals/{en,tr,de,fr,es,zh}/training/pipeline.md`** — new training-section page mirroring `docs/guides/pipeline.md`. TOC entry needs 6 locale titles (`en`/`tr` mandatory; `de`/`fr`/`es`/`zh` fall back to EN per `docs/standards/localization.md`). +- **`site/features.html` + `site/index.html` + `site/js/translations.js`** — new "Multi-stage training pipelines" feature card and a hook into the existing `home.pipeline` (alignment-stack) section so the two distinct senses of the word "pipeline" don't collide visually. 6-language i18n keys (`features.pipeline.title`, `features.pipeline.body`, `home.pipeline.chain.cta`) gated at full parity by `tools/check_site_claims.py`. +- **`tools/update_site_version.py`** — Phase 14 ships under the `v0.7.0` literal bump, which the version guard rewrites across `site/*.html` and `site/js/translations.js` automatically. + +The `cut-release` skill ([.claude/skills/cut-release/SKILL.md](../../.claude/skills/cut-release/SKILL.md)) walks these in order at `v0.7.0` tag time. + --- diff --git a/docs/usermanuals/en/concepts/alignment-overview.md b/docs/usermanuals/en/concepts/alignment-overview.md index 844d8689..5827cd68 100644 --- a/docs/usermanuals/en/concepts/alignment-overview.md +++ b/docs/usermanuals/en/concepts/alignment-overview.md @@ -10,7 +10,7 @@ Modern LLM fine-tuning is no longer a single algorithm — it's a *stack* of pos ```mermaid flowchart TD Base[Base model
e.g. Qwen 2.5 / Llama 3] - SFT[SFT
format & content] + SFT["SFT
format & content"] DPO[DPO
preference alignment] SimPO[SimPO
preference, ref-free] KTO[KTO
binary feedback] diff --git a/docs/usermanuals/en/getting-started/introduction.md b/docs/usermanuals/en/getting-started/introduction.md index 5b4c1cc8..1cfed53e 100644 --- a/docs/usermanuals/en/getting-started/introduction.md +++ b/docs/usermanuals/en/getting-started/introduction.md @@ -32,7 +32,7 @@ flowchart LR F -->|regression| G[Auto-revert] G --> E F -->|pass| H[Annex IV
artifacts] - H --> I[Export &
deploy] + H --> I["Export &
deploy"] classDef start fill:#1c2030,stroke:#f97316,color:#e6e7ec classDef ok fill:#161a24,stroke:#22c55e,color:#e6e7ec classDef warn fill:#161a24,stroke:#eab308,color:#e6e7ec diff --git a/docs/usermanuals/en/training/lora.md b/docs/usermanuals/en/training/lora.md index 5221afb8..2d051c65 100644 --- a/docs/usermanuals/en/training/lora.md +++ b/docs/usermanuals/en/training/lora.md @@ -13,7 +13,7 @@ ForgeLM applies LoRA / QLoRA / DoRA to *every* trainer (SFT, DPO, SimPO, KTO, OR ```mermaid flowchart TD - Q1{Need to fit on a
consumer GPU
(<24 GB)?} + Q1{"Need to fit on a
consumer GPU
(<24 GB)?"} Q2{Want full-precision
weights kept frozen?} Q3{Need maximum
quality at LoRA cost?} Full[Full fine-tuning] diff --git a/docs/usermanuals/tr/concepts/alignment-overview.md b/docs/usermanuals/tr/concepts/alignment-overview.md index 3535f10a..7cdddbd7 100644 --- a/docs/usermanuals/tr/concepts/alignment-overview.md +++ b/docs/usermanuals/tr/concepts/alignment-overview.md @@ -10,7 +10,7 @@ Modern LLM fine-tuning'i artık tek bir algoritma değil — her biri farklı bi ```mermaid flowchart TD Base[Temel model
ör. Qwen 2.5 / Llama 3] - SFT[SFT
format & içerik] + SFT["SFT
format & içerik"] DPO[DPO
tercih hizalaması] SimPO[SimPO
tercih, ref-free] KTO[KTO
ikili geri bildirim] diff --git a/docs/usermanuals/tr/training/lora.md b/docs/usermanuals/tr/training/lora.md index 64d7d984..04516e3d 100644 --- a/docs/usermanuals/tr/training/lora.md +++ b/docs/usermanuals/tr/training/lora.md @@ -13,7 +13,7 @@ ForgeLM LoRA / QLoRA / DoRA'yı *her* trainer'a uygular (SFT, DPO, SimPO, KTO, O ```mermaid flowchart TD - Q1{Consumer GPU'da
(<24 GB) sığmalı mı?} + Q1{"Consumer GPU'da
(<24 GB) sığmalı mı?"} Q2{Full-precision
ağırlıklar dondurulsun mu?} Q3{LoRA maliyetinde
maksimum kalite mi?} Full[Full fine-tuning] diff --git a/forgelm/cli/_dispatch.py b/forgelm/cli/_dispatch.py index da2ecbec..d874cbaa 100644 --- a/forgelm/cli/_dispatch.py +++ b/forgelm/cli/_dispatch.py @@ -163,6 +163,97 @@ def main(): sys.exit(EXIT_TRAINING_ERROR) +def _dispatch_legacy_data_audit(args) -> None: + """Handle ``forgelm --data-audit PATH`` (deprecated flag). + + Pulled out of ``_main_inner`` for Sonar python:S3776 cognitive- + complexity hygiene. Emits the standard deprecation + ``DeprecationWarning`` + an audit-log breadcrumb (best-effort), then + delegates to the canonical ``_run_data_audit`` and ``sys.exit``s. + Calls back to ``_main_inner`` are impossible — this function always + terminates the process. + """ + json_output = args.output_format == "json" + log_level = "WARNING" if args.quiet else args.log_level + _setup_logging(log_level, json_format=json_output) + # Phase 13 (Faz 13): emit a structured Python ``DeprecationWarning`` + # so `pytest -W error::DeprecationWarning` and `python -Wd` tooling + # surface it, plus an append-only audit-log event so operators who + # only read the JSONL trail see the migration signal too. Cadence + # for the v0.7.0 removal follows the "Deprecation cadence" section + # in ``docs/standards/release.md`` (one-minor warning window + # minimum). + warnings.warn( + "`forgelm --data-audit PATH` is deprecated and will be removed " + "in v0.7.0. Use the `forgelm audit PATH` subcommand instead — " + "same behaviour, same output. " + "See docs/standards/release.md#deprecation-cadence for the removal timeline.", + DeprecationWarning, + stacklevel=2, + ) + # Audit-log event: append-only Article 12 record of the legacy + # invocation so compliance reviewers can prove the operator was + # warned. Best-effort — a read-only output dir must not abort the + # actual audit run. + legacy_target = args.output or "./audit" + try: + from ..compliance import AuditLogger + from ..config import ConfigError + + AuditLogger(legacy_target).log_event( + "cli.legacy_flag_invoked", + flag="--data-audit", + replacement="forgelm audit", + version="v0.7.0 removal", + ) + except (OSError, ConfigError) as audit_exc: + logger.warning( + "Failed to emit deprecation audit event to %s: %s", + legacy_target, + audit_exc, + ) + # Late import via the package facade so monkeypatched + # ``forgelm.cli._run_data_audit`` references resolve correctly. + from forgelm import cli as _cli_facade + + _cli_facade._run_data_audit( + args.data_audit, + args.output, + args.output_format, + ) + sys.exit(EXIT_SUCCESS) + + +def _dispatch_pipeline_mode(config, args) -> None: + """Handle a config that carries a ``pipeline:`` block. + + Pulled out of ``_main_inner`` for Sonar python:S3776 cognitive- + complexity hygiene. Re-reads the YAML *bytes* (not the parsed + semantic content — regulators audit the on-disk artefact) and + routes to :func:`forgelm.cli._pipeline.run_pipeline_from_args`. + Always terminates the process via ``sys.exit``. + + Layering rule (Phase 14 F-B-1): this branch MUST run before + ``_maybe_run_no_train_mode``. The single-stage no-train modes + (``--dry-run``, ``--fit-check``, ``--benchmark-only``, + ``--merge``, ``--generate-data``, ``--compliance-export``) + ``sys.exit`` internally; if they ran first on a pipeline config, + the orchestrator would never reach the multi-stage validation path + and the documented ``--dry-run`` per-stage error-collection + contract would be silently violated. + """ + from ._pipeline import run_pipeline_from_args + + try: + with open(args.config, "rb") as f: + pipeline_yaml_bytes = f.read() + except OSError as e: + logger.error("Failed to re-read pipeline YAML for hashing: %s", e) + sys.exit(EXIT_CONFIG_ERROR) + + sys.exit(run_pipeline_from_args(config, pipeline_yaml_bytes, args)) + + def _main_inner() -> None: """Entry-point body — extracted so ``main`` can wrap a single ``KeyboardInterrupt`` handler around every step from argparse to @@ -181,70 +272,9 @@ def _main_inner() -> None: # Run before the config-required check so operators can audit raw data # without writing a YAML. Phase 11.5 promoted the same code path to # `forgelm audit PATH` (a real subcommand); the legacy flag is preserved - # as an alias and slated for removal in v0.7.0 — Phase 13 wires up the - # deprecation signalling at this dispatch site. + # as an alias and slated for removal in v0.7.0. if getattr(args, "data_audit", None): - json_output = args.output_format == "json" - log_level = "WARNING" if args.quiet else args.log_level - _setup_logging(log_level, json_format=json_output) - # Phase 13 (Faz 13): emit a structured Python ``DeprecationWarning`` - # so `pytest -W error::DeprecationWarning` and `python -Wd` tooling - # surface it, plus an append-only audit-log event so operators who - # only read the JSONL trail see the migration signal too. Cadence - # for the v0.7.0 removal follows the "Deprecation cadence" section - # in ``docs/standards/release.md`` (one-minor warning window - # minimum). - warnings.warn( - "`forgelm --data-audit PATH` is deprecated and will be removed " - "in v0.7.0. Use the `forgelm audit PATH` subcommand instead — " - "same behaviour, same output. " - "See docs/standards/release.md#deprecation-cadence for the removal timeline.", - DeprecationWarning, - stacklevel=2, - ) - # Audit-log event: append-only Article 12 record of the legacy - # invocation so compliance reviewers can prove the operator was - # warned. The logger writes to ``/audit_log.jsonl`` next to - # the report this run is about to produce. Resolve the target the - # same way ``_run_data_audit`` does (default ``./audit``) so the - # event lands in the same directory as the report. - legacy_target = args.output or "./audit" - try: - from ..compliance import AuditLogger - from ..config import ConfigError - - AuditLogger(legacy_target).log_event( - "cli.legacy_flag_invoked", - flag="--data-audit", - replacement="forgelm audit", - version="v0.7.0 removal", - ) - except (OSError, ConfigError) as audit_exc: - # Non-fatal: the audit log is a best-effort telemetry record - # for the deprecation notice. The DeprecationWarning has - # already fired; the audit run itself must still proceed — - # losing the legacy-flag breadcrumb is preferable to aborting - # a working pipeline because the output dir is read-only. - # Logged at WARNING (not debug) per - # docs/standards/error-handling.md "Best-effort artefact - # carve-out" hygiene item 2: audit-trail visibility trumps - # log noise for low-frequency, structured events like a - # legacy-flag invocation against a read-only output dir. - logger.warning( - "Failed to emit deprecation audit event to %s: %s", - legacy_target, - audit_exc, - ) - # Late import via the package facade so monkeypatched - # ``forgelm.cli._run_data_audit`` references resolve correctly. - from forgelm import cli as _cli_facade - - _cli_facade._run_data_audit( - args.data_audit, - args.output, - args.output_format, - ) - sys.exit(EXIT_SUCCESS) + _dispatch_legacy_data_audit(args) _maybe_run_wizard(args) @@ -261,5 +291,10 @@ def _main_inner() -> None: config = _load_config_or_exit(args.config, json_output) _apply_offline_flag(config, args.offline) + + if config.pipeline is not None: + _dispatch_pipeline_mode(config, args) + + # Single-stage path — unchanged from v0.6.0. _maybe_run_no_train_mode(config, args) _run_training_pipeline(config, args, json_output) diff --git a/forgelm/cli/_parser.py b/forgelm/cli/_parser.py index 2e916d56..d0d485d5 100644 --- a/forgelm/cli/_parser.py +++ b/forgelm/cli/_parser.py @@ -754,7 +754,25 @@ def _add_verify_annex_iv_subcommand(subparsers) -> None: "Exits 0 on valid; 1 on missing field or hash mismatch." ), ) - p.add_argument("path", help="Path to the Annex IV JSON artifact (typically `compliance/annex_iv_.json`).") + p.add_argument( + "path", + help=( + "Path to the Annex IV JSON artifact (typically " + "``compliance/annex_iv_.json``). When ``--pipeline`` is set, " + "this is interpreted as a pipeline run directory containing " + "``compliance/pipeline_manifest.json`` instead." + ), + ) + p.add_argument( + "--pipeline", + action="store_true", + help=( + "Phase 14: interpret ``path`` as a pipeline run directory and " + "validate the chain-level ``pipeline_manifest.json`` (chain " + "integrity, stage-index ordering, ``stopped_at`` coherence, " + "per-stage training_manifest existence)." + ), + ) _add_common_subparser_flags(p, include_output_format=True) @@ -1230,6 +1248,54 @@ def parse_args(): parser.add_argument( "--merge", action="store_true", help="Run model merging from the merge section of your config. No training." ) + # Phase 14 — multi-stage pipeline chains. Active only when the + # ``pipeline:`` block is present in the config; documented at + # docs/guides/pipeline.md. + parser.add_argument( + "--stage", + type=str, + default=None, + metavar="NAME", + help=( + "Multi-stage pipelines only: run a single named stage in isolation " + "(audit / re-run scenarios). Non-first stages require the previous " + "stage's on-disk output, or an explicit --input-model override." + ), + ) + parser.add_argument( + "--resume-from", + type=str, + default=None, + metavar="NAME", + help=( + "Multi-stage pipelines only: resume an interrupted run from a named " + "stage onward. Already-completed stages whose output_model paths " + "exist on disk are skipped; refuses to run if the on-disk " + "pipeline_config_hash differs from the current YAML (unless " + "--force-resume is also set)." + ), + ) + parser.add_argument( + "--force-resume", + action="store_true", + help=( + "Multi-stage pipelines only: bypass the --resume-from stale-config " + "guard. The override is logged at WARNING and recorded in the " + "audit event." + ), + ) + parser.add_argument( + "--input-model", + type=str, + default=None, + metavar="PATH", + help=( + "Multi-stage pipelines only: when used with --stage, replaces the " + "auto-chained input model with this path. The audit-log entry " + "records ``input_source: cli_override`` so reviewers see the chain " + "was broken intentionally." + ), + ) parser.add_argument( "--output-format", type=str, diff --git a/forgelm/cli/_pipeline.py b/forgelm/cli/_pipeline.py new file mode 100644 index 00000000..5302a207 --- /dev/null +++ b/forgelm/cli/_pipeline.py @@ -0,0 +1,1347 @@ +"""Phase 14 — Multi-stage training pipeline orchestrator. + +Top-level entry point for ``forgelm --config pipeline.yaml`` runs. The +orchestrator iterates over the ordered :class:`PipelineConfig.stages`, +materialises a flat :class:`ForgeConfig` per stage via +:func:`forgelm.config.merge_pipeline_stage_config`, hands it to a fresh +:class:`forgelm.trainer.ForgeTrainer`, and threads the state through a +crash-safe JSON state file + an append-only audit-log event stream. + +Layering rule — see ``docs/roadmap/phase-14-pipeline-chains.md`` Task 2: +``forgelm/trainer.py`` is **not** aware of the pipeline layer. Each +stage is a single-stage run from the trainer's point of view; only this +module knows there is an outer loop. When a config carries no +``pipeline:`` block, this module is never imported. + +Key responsibilities: + +- **Auto-chain** each stage's ``model.name_or_path`` to the previous + stage's output path (or honour an explicit stage override / CLI + ``--input-model`` override). +- **Per-stage gates** — auto-revert and human-approval are composed + one stage at a time; a failed gate stops the chain and marks + downstream stages ``skipped_due_to_prior_revert``. +- **Crash-safe state** — every stage transition is atomically written to + ``/pipeline_state.json`` (tmp file + ``os.replace``). + ``--resume-from`` reads that file and skips ``completed`` stages whose + outputs still exist on disk. +- **Annex-IV manifest** — every transition also rewrites + ``/compliance/pipeline_manifest.json`` (delegated to + :mod:`forgelm.compliance`) so reviewers can correlate every chain + failure to a frozen artefact. +- **Audit + webhook** — emits ``pipeline.started`` / ``.stage_started`` + / ``.stage_completed`` / ``.stage_reverted`` / ``.completed`` audit + events alongside the existing per-stage ``training.*`` vocabulary; the + webhook notifier gains ``notify_pipeline_*`` methods that the orchestrator + calls at the same transitions. +- **Exit-code routing** — the pipeline aggregates per-stage outcomes + via ``worst_exit = max(worst_exit, code)``, which gives the + precedence order + ``EXIT_AWAITING_APPROVAL (4)`` > ``EXIT_EVAL_FAILURE (3)`` > + ``EXIT_TRAINING_ERROR (2)`` > ``EXIT_CONFIG_ERROR (1)`` > + ``EXIT_SUCCESS (0)``. In practice an approval gate causes an + immediate ``break`` (no later stages run) and any other failure + sets ``chain_broken = True`` (downstream stages skip), so the + precedence rule is reached only when a single stage produces a + single non-zero code. Numeric ordering of the exit-code constants + matches the documented severity ramp — operators reading the exit + code in a shell only see the "worst" outcome from a single stage + that ran. +""" + +from __future__ import annotations + +import hashlib +import json +import logging +import os +import secrets +from dataclasses import asdict, dataclass, field +from datetime import datetime, timezone +from typing import Any, Dict, List, Literal, Optional + +from ..config import ForgeConfig, PipelineConfig, PipelineStage, merge_pipeline_stage_config +from ._exit_codes import ( + EXIT_AWAITING_APPROVAL, + EXIT_CONFIG_ERROR, + EXIT_EVAL_FAILURE, + EXIT_SUCCESS, + EXIT_TRAINING_ERROR, +) + +logger = logging.getLogger("forgelm.pipeline") + + +# --------------------------------------------------------------------------- +# State enums + dataclasses +# --------------------------------------------------------------------------- + + +StageStatusLiteral = Literal[ + "pending", + "running", + "completed", + "failed", + "gated_pending_approval", + "skipped_due_to_prior_revert", + "skipped_by_filter", +] +"""Per-stage status values. See Phase 14 Task 3 for the meaning of each.""" + + +InputSourceLiteral = Literal["root", "chain", "stage_explicit", "cli_override"] +"""How the orchestrator resolved a stage's input model path. Recorded on +the per-stage state so reviewers can trace any chain interruption +(``cli_override`` is the operator escape hatch that intentionally breaks +the chain).""" + + +FinalStatusLiteral = Literal[ + "in_progress", + "completed", + "stopped_at_stage", + "gated_pending_approval", +] + + +@dataclass +class PipelineStageState: + """Per-stage state snapshot persisted to disk + included in the manifest.""" + + name: str + index: int + trainer_type: str + status: StageStatusLiteral = "pending" + input_model: Optional[str] = None + input_source: Optional[InputSourceLiteral] = None + output_model: Optional[str] = None + started_at: Optional[str] = None + finished_at: Optional[str] = None + duration_seconds: Optional[float] = None + training_manifest: Optional[str] = None + metrics: Dict[str, float] = field(default_factory=dict) + gate_decision: Optional[str] = None + auto_revert_triggered: bool = False + skipped_reason: Optional[str] = None + exit_code: Optional[int] = None + error: Optional[str] = None + + +@dataclass +class PipelineState: + """Top-level pipeline-run state snapshot. + + Round-tripped to ``pipeline_state.json`` after every stage transition. + Compatible with :mod:`json` (atomic-write via tmp + ``os.replace``); + no nested non-serialisable types. + """ + + pipeline_run_id: str + pipeline_config_hash: str + forgelm_version: str + started_at: str + finished_at: Optional[str] = None + final_status: FinalStatusLiteral = "in_progress" + stopped_at: Optional[str] = None + stages: List[PipelineStageState] = field(default_factory=list) + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _utc_iso_now() -> str: + """UTC ISO-8601 timestamp with seconds precision. + + Mirrors the format ``forgelm/compliance.py`` already uses for audit + log entries so timestamps line up across the two surfaces. + """ + return datetime.now(timezone.utc).isoformat(timespec="seconds") + + +def _generate_run_id() -> str: + """Return a pipeline run id of shape ``pl_YYYY-MM-DD_<6-hex>``. + + The date prefix lets operators eyeball runs by week without consulting + the manifest; the 6-hex suffix gives ~16 million combinations per day, + which is more than enough to disambiguate same-day re-runs even in + aggressive CI loops. + """ + today = datetime.now(timezone.utc).strftime("%Y-%m-%d") + return f"pl_{today}_{secrets.token_hex(3)}" + + +def _compute_pipeline_config_hash(pipeline_yaml_bytes: bytes) -> str: + """SHA-256 of the *raw bytes* of the pipeline YAML file. + + Hashing the bytes (not the parsed dict) is intentional: it locks the + on-disk artefact, which is what regulators audit. YAML + comments / key ordering / whitespace are all in scope; an operator + who edits the file between runs gets a different hash even if the + semantic content is unchanged, and the ``--resume-from`` stale-state + guard catches it. + """ + return "sha256:" + hashlib.sha256(pipeline_yaml_bytes).hexdigest() + + +def _pipeline_paths(root_cfg: ForgeConfig) -> Dict[str, str]: + """Resolve the canonical filesystem paths for the pipeline run. + + All pipeline-level artefacts (state file, manifest, audit log) live + under :attr:`PipelineConfig.output_dir` (defaults to + ``./pipeline_run``). Per-stage trainer artefacts continue to live + under each stage's own ``training.output_dir`` (as in single-stage + runs) — the pipeline output dir is the *index* directory, distinct + from the stage checkpoint directories so operators can ``rm -rf`` + individual stages without nuking the pipeline manifest. + """ + if root_cfg.pipeline is None: # defensive — orchestrator constructor catches this earlier + raise ValueError("Cannot derive pipeline paths from a config with no `pipeline` block.") + root_output_dir = root_cfg.pipeline.output_dir + return { + "root_output_dir": root_output_dir, + "state_file": os.path.join(root_output_dir, "pipeline_state.json"), + "manifest_dir": os.path.join(root_output_dir, "compliance"), + "manifest_file": os.path.join(root_output_dir, "compliance", "pipeline_manifest.json"), + } + + +def _atomic_write_json(path: str, payload: Dict[str, Any]) -> None: + """Write ``payload`` to ``path`` atomically. + + Pattern: serialise to a sibling temp file then ``os.replace`` it + onto the final name — guarantees a reader on any concurrent process + either sees the previous version or the new version, never a half- + written file. + + The temp filename is per-writer-unique (``...tmp``) + rather than a shared ``.tmp`` so two writers targeting the same + final path don't truncate each other's tmp mid-write — pre-fix, a + concurrent invocation produced ``FileNotFoundError`` from + ``os.replace`` (the loser's tmp was unlinked by the winner's + rename). Phase 14 doesn't *support* concurrent orchestrators on + the same ``pipeline.output_dir`` (see ``docs/guides/pipeline.md`` + "Limitations"), but the per-writer tmp is cheap defence so an + accidental double-invocation surfaces as last-writer-wins rather + than a Python traceback. The temp lives in the same directory as + ``path`` so ``os.replace`` stays a same-filesystem atomic rename. + """ + target_dir = os.path.dirname(path) or "." + os.makedirs(target_dir, exist_ok=True) + tmp = f"{path}.{os.getpid()}.{secrets.token_hex(4)}.tmp" + try: + with open(tmp, "w", encoding="utf-8") as f: + json.dump(payload, f, indent=2, sort_keys=False) + os.replace(tmp, path) + except Exception: + # If the write failed before the replace, clean up the orphan + # temp. After a successful replace there's nothing to remove + # (the tmp name no longer exists), so silently ignore that + # branch. Don't swallow the original exception either; we + # only swallow the cleanup-of-orphan-tmp failure. + try: + os.unlink(tmp) + except OSError: + pass + raise + + +def _serialise_state(state: PipelineState) -> Dict[str, Any]: + """Return a JSON-ready dict representation of a :class:`PipelineState`.""" + return asdict(state) + + +def _deserialise_state(payload: Dict[str, Any]) -> PipelineState: + """Round-trip :class:`PipelineState` from its on-disk JSON shape. + + Tolerates unknown keys (future-forward compat) and missing optionals + (legacy state files). Type-coerces ``stages[]`` back into + :class:`PipelineStageState` instances so the rest of the orchestrator + sees a strongly-typed object. + """ + stages_raw = payload.get("stages", []) + stages = [ + PipelineStageState(**{k: v for k, v in s.items() if k in PipelineStageState.__dataclass_fields__}) + for s in stages_raw + ] + return PipelineState( + pipeline_run_id=payload["pipeline_run_id"], + pipeline_config_hash=payload["pipeline_config_hash"], + forgelm_version=payload.get("forgelm_version", "unknown"), + started_at=payload["started_at"], + finished_at=payload.get("finished_at"), + final_status=payload.get("final_status", "in_progress"), + stopped_at=payload.get("stopped_at"), + stages=stages, + ) + + +# --------------------------------------------------------------------------- +# Orchestrator +# --------------------------------------------------------------------------- + + +class PipelineOrchestrator: + """Drives a multi-stage pipeline run end-to-end. + + Constructor takes the root config and the *bytes* of the pipeline YAML + (the bytes are hashed for the stale-resume guard; the parsed config + drives the actual stage iteration). + """ + + def __init__( + self, + root_cfg: ForgeConfig, + pipeline_yaml_bytes: bytes, + *, + output_format: str = "text", + ) -> None: + if root_cfg.pipeline is None: + # Defensive — callers should only reach this path when a + # ``pipeline:`` block is present. A None ``pipeline`` here + # signals a CLI dispatch bug; fail loud rather than running + # an empty pipeline. + raise ValueError("PipelineOrchestrator instantiated with a config that has no `pipeline` block.") + self.root_cfg: ForgeConfig = root_cfg + self.pipeline: PipelineConfig = root_cfg.pipeline + self.config_hash: str = _compute_pipeline_config_hash(pipeline_yaml_bytes) + self.output_format: str = output_format + self.paths: Dict[str, str] = _pipeline_paths(root_cfg) + # Lazy-initialised audit-log writer. Pinned to the pipeline run + # id on first event so every ``pipeline.*`` entry in the JSONL + # stream carries the same top-level ``run_id`` (Phase 14 review + # final-round F-B-1). Pre-fix, ``_audit_event`` constructed a + # fresh ``AuditLogger`` per call → every event got a different + # auto-generated ``fg-`` run_id, breaking SIEM-style + # grouping and the Article 12 record-keeping correlation + # contract. The pipeline_run_id is set in state.pipeline_run_id + # before any event fires, so the caller passes it through. + self._audit_logger: Optional[Any] = None + + # -- run-mode entry points ------------------------------------------------ + + def _prepare_resume_or_init_state( + self, + *, + resume_from: Optional[str], + force_resume: bool, + ) -> "tuple[Optional[PipelineState], Optional[int]]": + """Load + validate state for ``--resume-from`` or build a fresh state. + + Returns ``(state, None)`` on success; ``(None, EXIT_CONFIG_ERROR)`` + when the resume path is invalid (missing state file or stale-hash + refused without ``--force-resume``). Extracted from + :meth:`run` for Sonar python:S3776 cognitive-complexity hygiene. + """ + existing = self._load_state_file() + if resume_from is not None and existing is None: + logger.error( + "Cannot --resume-from %r: no pipeline_state.json found at %s. Run the pipeline once first.", + resume_from, + self.paths["state_file"], + ) + return None, EXIT_CONFIG_ERROR + if resume_from is not None: + refusal = self._validate_resume_state(existing, force=force_resume) + if refusal is not None: + return None, EXIT_CONFIG_ERROR + return existing, None + return self._init_state(), None + + def _resolve_resume_skiplist( + self, + *, + state: PipelineState, + resume_from: str, + ) -> "tuple[Optional[List[str]], Optional[int]]": + """Compute the list of already-completed stages to skip on resume. + + Returns ``(skiplist, None)`` on success or ``(None, + EXIT_CONFIG_ERROR)`` when ``resume_from`` names a stage that + doesn't exist. Stages before the resume index whose status is + ``completed`` and whose ``output_model`` directory exists on + disk are kept-as-is. Extracted from :meth:`run` for Sonar + python:S3776 cognitive-complexity hygiene. + """ + try: + resume_idx = next(i for i, s in enumerate(self.pipeline.stages) if s.name == resume_from) + except StopIteration: + logger.error( + "Cannot --resume-from %r: stage name not found in pipeline (valid names: %s).", + resume_from, + ", ".join(s.name for s in self.pipeline.stages), + ) + return None, EXIT_CONFIG_ERROR + stages_to_skip_completed: List[str] = [] + for prior_state in state.stages[:resume_idx]: + if ( + prior_state.status == "completed" + and prior_state.output_model + and os.path.isdir(prior_state.output_model) + ): + stages_to_skip_completed.append(prior_state.name) + logger.info( + "Resuming: stage %r already completed at %s; skipping.", + prior_state.name, + prior_state.output_model, + ) + return stages_to_skip_completed, None + + def _resolve_stage_filter_setup( + self, + *, + stage_filter: str, + input_model_override: Optional[str], + ) -> "tuple[Optional[str], Optional[int]]": + """Validate ``--stage `` + seed the auto-chain on a non-first stage. + + Returns ``(prev_output_for_filter, None)`` on success or + ``(None, EXIT_CONFIG_ERROR)`` when the stage name is unknown or + the previous stage's output directory is missing (and no + ``--input-model`` override was supplied to bypass the chain). + Extracted from :meth:`run` for Sonar python:S3776 cognitive- + complexity hygiene. + """ + if not any(s.name == stage_filter for s in self.pipeline.stages): + logger.error( + "Cannot --stage %r: stage name not found in pipeline (valid names: %s).", + stage_filter, + ", ".join(s.name for s in self.pipeline.stages), + ) + return None, EXIT_CONFIG_ERROR + + if input_model_override is not None: + return None, None + + filter_idx = next(i for i, s in enumerate(self.pipeline.stages) if s.name == stage_filter) + if filter_idx == 0: + return None, None + + prev_stage = self.pipeline.stages[filter_idx - 1] + prev_merged = merge_pipeline_stage_config(self.root_cfg, prev_stage, prev_output_model=None) + candidate = os.path.join(prev_merged.training.output_dir, "final_model") + if not os.path.isdir(candidate): + logger.error( + "Stage %r requires the previous stage's output at %s; " + "pass --input-model to override or run the full pipeline first.", + stage_filter, + candidate, + ) + return None, EXIT_CONFIG_ERROR + return candidate, None + + def _handle_stage_outcome( + self, + *, + stage: PipelineStage, + stage_state: PipelineStageState, + exit_code: int, + state: PipelineState, + worst_exit: int, + ) -> "tuple[int, Optional[str], bool, bool]": + """Persist + audit + webhook the outcome of a single executed stage. + + Returns ``(worst_exit, new_prev_output, chain_broken, should_break)``: + + - ``worst_exit`` — the updated highest-priority exit code so far. + - ``new_prev_output`` — the auto-chain seed for the next stage + (``stage_state.output_model`` on success, ``None`` otherwise). + - ``chain_broken`` — ``True`` for any non-success outcome; the + caller marks downstream stages skipped on it. + - ``should_break`` — ``True`` only for ``EXIT_AWAITING_APPROVAL``; + the caller exits the stage loop immediately so a subsequent + ``--resume-from`` picks up the remaining stages post-approval. + + Extracted from :meth:`run` for Sonar python:S3776 cognitive- + complexity hygiene. + """ + if exit_code == EXIT_AWAITING_APPROVAL: + # Human-approval gate fired. Pipeline exits; remaining + # stages stay ``pending`` so a subsequent ``--resume-from`` + # picks them up after operator action. + stage_state.status = "gated_pending_approval" + state.final_status = "gated_pending_approval" + state.stopped_at = stage.name + state.finished_at = _utc_iso_now() + new_worst = max(worst_exit, EXIT_AWAITING_APPROVAL) + self._save_state(state) + # Phase 14 review F-N-1: dedicated ``pipeline.stage_gated`` + # event (instead of ``pipeline.stage_completed`` with a + # ``gate_decision=approval_pending`` sub-field). Lets + # dashboard / SIEM rules filter on the event name alone. + self._audit_event( + "pipeline.stage_gated", + pipeline_run_id=state.pipeline_run_id, + stage_name=stage.name, + gate_decision="approval_pending", + staging_path=stage_state.output_model, + ) + return new_worst, None, True, True + + if exit_code == EXIT_SUCCESS: + stage_state.status = "completed" + stage_state.gate_decision = "passed" + self._save_state(state) + self._audit_event( + "pipeline.stage_completed", + pipeline_run_id=state.pipeline_run_id, + stage_name=stage.name, + gate_decision="passed", + metrics=stage_state.metrics, + ) + return worst_exit, stage_state.output_model, False, False + + # Anything else (EXIT_TRAINING_ERROR / EXIT_EVAL_FAILURE): + # chain stops; downstream stages will skip with + # ``skipped_due_to_prior_revert``. + stage_state.status = "failed" + stage_state.gate_decision = "failed" + state.stopped_at = stage.name + new_worst = max(worst_exit, exit_code) + self._save_state(state) + self._audit_event( + "pipeline.stage_reverted" if stage_state.auto_revert_triggered else "pipeline.stage_completed", + pipeline_run_id=state.pipeline_run_id, + stage_name=stage.name, + gate_decision="failed", + auto_revert_triggered=stage_state.auto_revert_triggered, + ) + self._notify_pipeline( + "reverted" if stage_state.auto_revert_triggered else "stage_failed", + state, + stage_state=stage_state, + ) + return new_worst, None, True, False + + def _execute_one_stage( + self, + *, + index: int, + stage: PipelineStage, + state: PipelineState, + prev_output: Optional[str], + stage_filter: Optional[str], + input_model_override: Optional[str], + ) -> int: + """Setup + invoke a single stage's trainer. Returns the per-stage exit code. + + Caller is responsible for translating the returned exit code + into the next-state transition via :meth:`_handle_stage_outcome`. + Extracted from :meth:`run` for Sonar python:S3776 hygiene. + """ + stage_state = state.stages[index] + stage_state.index = index + stage_state.name = stage.name + stage_state.trainer_type = ( + stage.training.trainer_type if stage.training else self.root_cfg.training.trainer_type + ) + stage_state.started_at = _utc_iso_now() + stage_state.status = "running" + self._save_state(state) + + # ``--input-model`` attaches to a single filtered stage only. + override_for_this_stage = ( + input_model_override if (stage_filter is not None and stage.name == stage_filter) else None + ) + + exit_code = self._run_single_stage( + stage=stage, + stage_state=stage_state, + prev_output=prev_output, + input_model_override=override_for_this_stage, + state=state, + ) + stage_state.exit_code = exit_code + stage_state.finished_at = _utc_iso_now() + try: + start_dt = datetime.fromisoformat(stage_state.started_at) + end_dt = datetime.fromisoformat(stage_state.finished_at) + stage_state.duration_seconds = (end_dt - start_dt).total_seconds() + except (TypeError, ValueError): + stage_state.duration_seconds = None + return exit_code + + def _finalise_pipeline(self, state: PipelineState, worst_exit: int) -> None: + """Mark the pipeline finished + emit the terminal events. + + Mirrors the run-loop tail. Extracted from :meth:`run` for + Sonar python:S3776 cognitive-complexity hygiene. + """ + if state.final_status == "in_progress": + state.final_status = "completed" if worst_exit == EXIT_SUCCESS else "stopped_at_stage" + state.finished_at = _utc_iso_now() + self._save_state(state) + + if state.final_status in ("completed", "stopped_at_stage"): + self._audit_event( + "pipeline.completed", + pipeline_run_id=state.pipeline_run_id, + final_status=state.final_status, + stopped_at=state.stopped_at, + ) + self._notify_pipeline("completed", state) + + def _stage_skip_reason( + self, + *, + stage: PipelineStage, + stage_state: PipelineStageState, + state: PipelineState, + stage_filter: Optional[str], + stages_to_skip_completed: List[str], + chain_broken: bool, + ) -> Optional[str]: + """Decide whether the stage should be skipped, and update state accordingly. + + Returns one of: + + - ``"filtered"`` — ``--stage X`` was set and this is not the + named stage. Status set to ``skipped_by_filter``. + - ``"already_completed"`` — ``--resume-from`` picked this stage + up as already-done; caller seeds ``prev_output`` from + ``stage_state.output_model``. + - ``"chain_broken"`` — a prior stage in the same run reverted + or failed; status set to ``skipped_due_to_prior_revert``. + - ``None`` — proceed to execute. + + Extracted from :meth:`run` for Sonar python:S3776 cognitive- + complexity hygiene. + """ + if stage_filter is not None and stage.name != stage_filter: + stage_state.status = "skipped_by_filter" + stage_state.skipped_reason = f"--stage {stage_filter!r} was specified; only that stage runs." + self._save_state(state) + return "filtered" + if stage.name in stages_to_skip_completed: + return "already_completed" + if chain_broken: + stage_state.status = "skipped_due_to_prior_revert" + stage_state.skipped_reason = ( + f"Stage {state.stopped_at!r} triggered auto_revert; downstream stages did not run." + ) + self._save_state(state) + return "chain_broken" + return None + + def _run_stage_loop( + self, + *, + state: PipelineState, + stage_filter: Optional[str], + stages_to_skip_completed: List[str], + input_model_override: Optional[str], + prev_output_for_filter: Optional[str], + ) -> int: + """Iterate over ``self.pipeline.stages`` and return the aggregated ``worst_exit``. + + Each iteration is one of three things: + + 1. A skipped stage (filter / already-done / chain broken) — see + :meth:`_stage_skip_reason` for the routing rules. + 2. An executed stage — :meth:`_execute_one_stage` runs the + trainer and :meth:`_handle_stage_outcome` records the + transition + audit events. + 3. A gated stage (``EXIT_AWAITING_APPROVAL``) — the outcome + handler sets ``should_break`` and the loop returns early so + downstream stages stay ``pending`` for a subsequent + ``--resume-from``. + + Extracted from :meth:`run` for Sonar python:S3776 cognitive- + complexity hygiene. + """ + worst_exit: int = EXIT_SUCCESS + prev_output: Optional[str] = prev_output_for_filter + chain_broken: bool = False + for i, stage in enumerate(self.pipeline.stages): + stage_state = state.stages[i] + skip_reason = self._stage_skip_reason( + stage=stage, + stage_state=stage_state, + state=state, + stage_filter=stage_filter, + stages_to_skip_completed=stages_to_skip_completed, + chain_broken=chain_broken, + ) + if skip_reason == "already_completed": + prev_output = stage_state.output_model + continue + if skip_reason is not None: + continue + + exit_code = self._execute_one_stage( + index=i, + stage=stage, + state=state, + prev_output=prev_output, + stage_filter=stage_filter, + input_model_override=input_model_override, + ) + worst_exit, new_prev_output, chain_broken_now, should_break = self._handle_stage_outcome( + stage=stage, + stage_state=stage_state, + exit_code=exit_code, + state=state, + worst_exit=worst_exit, + ) + prev_output = new_prev_output if new_prev_output is not None else prev_output + chain_broken = chain_broken or chain_broken_now + if should_break: + break + return worst_exit + + def run( + self, + *, + stage_filter: Optional[str] = None, + resume_from: Optional[str] = None, + force_resume: bool = False, + input_model_override: Optional[str] = None, + ) -> int: + """Execute the pipeline. Returns a process exit code. + + ``stage_filter`` and ``resume_from`` are mutually exclusive; the + caller (CLI parser) validates this before reaching here. + + Exit-code aggregation uses ``worst_exit = max(worst_exit, code)`` + across stages, giving the numeric precedence: + ``EXIT_AWAITING_APPROVAL (4)`` > ``EXIT_EVAL_FAILURE (3)`` > + ``EXIT_TRAINING_ERROR (2)`` > ``EXIT_CONFIG_ERROR (1)`` > + ``EXIT_SUCCESS (0)``. An approval gate triggers an immediate + ``break`` (later stages stay ``pending``) and any other non-zero + outcome sets ``chain_broken = True`` (later stages skip with + ``skipped_due_to_prior_revert``), so multi-stage compounding is + rare — in practice the return value reflects the single failing + stage's exit code. + """ + # 1. Load (or initialise) state. + state, state_err = self._prepare_resume_or_init_state(resume_from=resume_from, force_resume=force_resume) + if state is None: + return state_err # type: ignore[return-value] + + # 2. Determine which stages to execute. + stages_to_skip_completed: List[str] = [] + if resume_from is not None: + skiplist, skiplist_err = self._resolve_resume_skiplist(state=state, resume_from=resume_from) + if skiplist is None: + return skiplist_err # type: ignore[return-value] + stages_to_skip_completed = skiplist + + # 2a. ``--stage`` validation + auto-chain seeding. + prev_output_for_filter: Optional[str] = None + if stage_filter is not None: + seed, filter_err = self._resolve_stage_filter_setup( + stage_filter=stage_filter, input_model_override=input_model_override + ) + if filter_err is not None: + return filter_err + prev_output_for_filter = seed + # Phase 14 review final-round F-B-2: surface the disk-seed + # path on the *predecessor* stage's manifest entry as well. + # When ``--stage X`` re-runs only the named stage, its + # predecessor is recorded as ``skipped_by_filter`` with no + # ``output_model``. The filtered stage records + # ``input_source: "chain"`` (we auto-chained from the seed), + # so the strict chain verifier (F-B-3) compares stage X's + # ``input_model`` against ``stages[X-1].output_model`` and + # flags a ``chain_integrity_violation`` even though the + # chain is intact on disk. We seed the predecessor's + # ``output_model`` here so the manifest faithfully + # represents "stage X-1 produced this path (in a prior + # run); stage X chained from it now". + if seed is not None: + filter_idx = next(i for i, s in enumerate(self.pipeline.stages) if s.name == stage_filter) + if filter_idx > 0: + state.stages[filter_idx - 1].output_model = seed + + # 3. Emit pipeline.started + webhook (fresh runs only). + if resume_from is None: + self._audit_event( + "pipeline.started", + pipeline_run_id=state.pipeline_run_id, + config_hash=self.config_hash, + stage_count=len(self.pipeline.stages), + stage_names=[s.name for s in self.pipeline.stages], + ) + self._notify_pipeline("started", state) + + # 4. Iterate stages (delegated so this method stays at a + # bounded cognitive-complexity score). + worst_exit = self._run_stage_loop( + state=state, + stage_filter=stage_filter, + stages_to_skip_completed=stages_to_skip_completed, + input_model_override=input_model_override, + prev_output_for_filter=prev_output_for_filter, + ) + + # 5+6. Final state + manifest + summary. + self._finalise_pipeline(state, worst_exit) + self._emit_summary(state) + return worst_exit + + def dry_run(self) -> int: + """Validate every stage without allocating any GPU resources. + + Collects all per-stage Pydantic / merge errors before exiting, + mirroring ``pytest --collectonly``. Returns + :data:`EXIT_CONFIG_ERROR (1)` if any stage fails validation, + :data:`EXIT_SUCCESS (0)` otherwise. + + Additionally checks that no two stages resolve to the same + ``training.output_dir`` — colliding output dirs would cause one + stage's checkpoints and per-stage Annex-IV manifest to overwrite + another's, breaking the chain of custody silently (Phase 14 + review F-G-1). Because stage-level ``training`` blocks + *inherit* the root ``training`` block wholesale unless the + stage supplies a full override, two stages that both inherit + end up sharing the root's ``output_dir`` — that's the canonical + misconfiguration this guard catches. + """ + errors: List[str] = [] + prev_output: Optional[str] = None + seen_dirs: Dict[str, str] = {} # abs output_dir -> first stage that claimed it + for stage in self.pipeline.stages: + try: + merged = merge_pipeline_stage_config( + self.root_cfg, + stage, + prev_output_model=prev_output, + ) + except Exception as exc: # noqa: BLE001 — best-effort: collect every per-stage validation error for a single operator report instead of stopping at the first. + errors.append(f"Stage {stage.name!r}: merge failed: {exc}") + continue + + # Collision check (Phase 14 review F-G-1): every stage must + # write its checkpoints + per-stage manifest into a distinct + # directory. Comparing absolute paths so ``./out`` and + # ``out/`` collide as one. + abs_out = os.path.abspath(merged.training.output_dir) + if abs_out in seen_dirs: + errors.append( + f"Stage {stage.name!r}: training.output_dir collides with stage " + f"{seen_dirs[abs_out]!r} (both resolve to {abs_out!r}); per-stage " + "checkpoints and manifests would overwrite each other. Set a " + "unique 'training.output_dir' in each stage." + ) + else: + seen_dirs[abs_out] = stage.name + + # Project the output path the next stage would auto-chain to. + prev_output = os.path.join(merged.training.output_dir, "final_model") + + if errors: + logger.error("Pipeline dry-run found %d stage error(s):", len(errors)) + for e in errors: + logger.error(" %s", e) + return EXIT_CONFIG_ERROR + + logger.info( + "Pipeline dry-run OK: %d stage(s) validated; no GPU was allocated.", + len(self.pipeline.stages), + ) + return EXIT_SUCCESS + + # -- internals ------------------------------------------------------------ + + def _init_state(self) -> PipelineState: + """Build a fresh :class:`PipelineState` from the parsed pipeline.""" + from .._version import __version__ + + stages = [ + PipelineStageState( + name=s.name, + index=i, + trainer_type=(s.training.trainer_type if s.training else self.root_cfg.training.trainer_type), + ) + for i, s in enumerate(self.pipeline.stages) + ] + return PipelineState( + pipeline_run_id=_generate_run_id(), + pipeline_config_hash=self.config_hash, + forgelm_version=__version__, + started_at=_utc_iso_now(), + stages=stages, + ) + + def _validate_resume_state(self, existing: PipelineState, *, force: bool) -> Optional[str]: + """Stale-state guard for ``--resume-from``. + + If the on-disk state file's ``pipeline_config_hash`` differs from + the current parse of the pipeline YAML, the operator edited the + file between the failed run and the resume call — resuming + against a different config silently produces a chain whose + history doesn't match its current shape. We refuse unless + ``--force-resume`` is set (logged at WARNING **and** emitted as a + ``pipeline.force_resume`` audit event so reviewers see why the + divergence was accepted). + + Topology vs value-level edits (Phase 14 review final-round + clarification): + + - **Topology change** (adding, deleting, or renaming a stage — + the ordered ``[s.name for s in stages]`` list changes) is + refused **even under** ``--force-resume`` because + ``state.stages[i]`` is positionally addressed and a topology + shift would silently overwrite a different stage's history. + - **Value-level edits** (different ``trainer_type``, + ``training.output_dir``, hyperparameters under unchanged + stage names) trip the hash check but are *accepted* under + ``--force-resume``. This is the documented escape hatch: + the operator is signalling "I know the on-disk state and the + current config diverge; proceed and record the + ``pipeline.force_resume`` audit breadcrumb." Reviewers can + reconstruct the divergence from the event payload's + ``old_config_hash`` / ``new_config_hash`` fields. + + Return semantics: + + - ``None`` — the resume is OK to proceed (no hash divergence, or + divergence accepted via ``--force-resume``). + - error-string — the resume must abort; the caller maps this to + ``EXIT_CONFIG_ERROR``. Using a return value (rather than + ``sys.exit`` mid-method) keeps the orchestrator's exit policy + consistent with the rest of ``run()`` — every refusal flows + back through the same ``return`` branch, the same audit-log + finalisation runs, and the test surface uses a uniform + ``assert code == EXIT_CONFIG_ERROR`` shape. + """ + if existing.pipeline_config_hash == self.config_hash: + return None + if not force: + msg = ( + f"Refusing to --resume-from: pipeline_config_hash mismatch.\n" + f" on-disk: {existing.pipeline_config_hash}\n" + f" current: {self.config_hash}\n" + "Pass --force-resume to override (logged for audit)." + ) + logger.error(msg) + return msg + + # Topology guard (Phase 14 review-response): even under + # ``--force-resume`` we refuse if the stage *topology* has + # changed — count or ordered names — because the on-disk state + # file's ``stages[i]`` payload is positionally addressed. A + # stage inserted, deleted, or renamed between runs would make + # ``state.stages[i]`` describe a different stage than + # ``self.pipeline.stages[i]`` and the resume would silently + # corrupt the audit trail. + # + # *Value-level* edits — different ``trainer_type``, + # ``training.output_dir``, hyperparameters under identical + # stage names — trip the hash check but are *accepted by + # design* under ``--force-resume``. The flag is the operator's + # explicit signal that they understand the on-disk state and + # the current config diverge; the guard's job is only to + # protect the positional-addressing invariant. See + # ``docs/guides/pipeline.md`` "Limitations" for the documented + # contract. + on_disk_names = [s.name for s in existing.stages] + current_names = [s.name for s in self.pipeline.stages] + if on_disk_names != current_names: + msg = ( + f"Refusing to --resume-from even with --force-resume: pipeline " + f"stage topology has changed.\n" + f" on-disk stages: {on_disk_names!r}\n" + f" current stages: {current_names!r}\n" + "Stage names / count / order must match the on-disk state. " + "Start a fresh pipeline run instead." + ) + logger.error(msg) + return msg + + old_hash = existing.pipeline_config_hash + logger.warning( + "Resuming with --force-resume despite pipeline_config_hash mismatch: %s → %s.", + old_hash, + self.config_hash, + ) + # Append-only audit event so a reviewer reading the JSONL stream + # later can distinguish an operator-approved stale-config resume + # from a normal resume. Emitted BEFORE the stored hash is + # updated so the event payload still records both sides of the + # divergence. + self._audit_event( + "pipeline.force_resume", + pipeline_run_id=existing.pipeline_run_id, + old_config_hash=old_hash, + new_config_hash=self.config_hash, + ) + # Update the stored hash to match the now-trusted current config + # so subsequent transitions don't trip the guard again. + existing.pipeline_config_hash = self.config_hash + return None + + def _load_state_file(self) -> Optional[PipelineState]: + path = self.paths["state_file"] + if not os.path.isfile(path): + return None + try: + with open(path, "r", encoding="utf-8") as f: + payload = json.load(f) + except (OSError, json.JSONDecodeError) as e: + logger.warning("Failed to read pipeline_state.json at %s: %s. Treating as missing.", path, e) + return None + # Phase 14 review F-R1-S4: a tampered state file (wrong shape, + # missing required keys, ``stages`` not a list of dicts, etc.) + # could crash ``_deserialise_state`` with an opaque + # ``TypeError`` / ``KeyError`` traceback instead of producing + # an actionable config-error path. Catch the common shape + # failures here and treat the file as missing — the caller's + # ``--resume-from`` branch then surfaces ``EXIT_CONFIG_ERROR`` + # with a clear message. + try: + return _deserialise_state(payload) + except (KeyError, TypeError, ValueError, AttributeError) as e: + # AttributeError covers the case where ``stages`` items aren't + # dicts (e.g. an attacker wrote ``stages: [null, "foo"]``) — + # ``s.items()`` then raises AttributeError on the str / None. + logger.warning( + "pipeline_state.json at %s is structurally invalid (%s); treating as missing.", + path, + e, + ) + return None + + def _save_state(self, state: PipelineState) -> None: + """Persist state file + refresh pipeline manifest. + + Both writes are atomic. Manifest export is delegated to + :mod:`forgelm.compliance` so the schema lives in one place. + """ + _atomic_write_json(self.paths["state_file"], _serialise_state(state)) + # Lazy import — compliance has its own heavy deps for the + # Annex-IV verifier path; only needed when we actually export. + from ..compliance import generate_pipeline_manifest + + manifest = generate_pipeline_manifest(state=state, root_cfg=self.root_cfg) + _atomic_write_json(self.paths["manifest_file"], manifest) + + def _audit_event(self, event: str, *, pipeline_run_id: str, **fields: Any) -> None: + """Append-only audit log entry under the **pipeline** root output dir. + + Two separate audit-log streams exist on a multi-stage run: + + - **Pipeline-level** (this method): ``/ + audit_log.jsonl`` — carries ``pipeline.*`` lifecycle events + (started / stage_started / stage_completed / stage_gated / + stage_reverted / force_resume / completed). + - **Stage-level** (emitted by each ``ForgeTrainer``): + ``/audit_log.jsonl`` — carries the + existing ``training.*`` vocabulary unchanged from a single- + stage v0.6.0 run. + + Both files are append-only JSONL. Reviewers can correlate by + joining on ``pipeline_run_id`` (present in every pipeline event) + with the per-stage ``training_manifest.json`` ``training_run_id``. + See ``docs/guides/pipeline.md`` "Audit events" for the full + join key matrix. + + Phase 14 review final-round F-B-1: the ``AuditLogger`` instance + is cached on the orchestrator and pinned to the pipeline run + id, so every entry's top-level ``run_id`` field equals the + pipeline run id (instead of a fresh auto-generated + ``fg-`` per event, which broke SIEM-style grouping and + the Article 12 correlation contract). ``pipeline_run_id`` is + now a required keyword argument so callers cannot silently + regress this. + """ + try: + from ..compliance import AuditLogger + + if self._audit_logger is None: + self._audit_logger = AuditLogger(self.paths["root_output_dir"], run_id=pipeline_run_id) + self._audit_logger.log_event(event, pipeline_run_id=pipeline_run_id, **fields) + except Exception as exc: # noqa: BLE001 — audit emission is best-effort telemetry; a write failure must not abort the pipeline. + logger.warning("Failed to emit audit event %r: %s", event, exc) + + def _notify_pipeline( + self, + kind: Literal["started", "completed", "reverted", "stage_failed"], + state: PipelineState, + *, + stage_state: Optional[PipelineStageState] = None, + ) -> None: + """Webhook fan-out for pipeline lifecycle transitions. + + Notifier methods are optional (added in Phase 14, defended via + ``getattr`` so a fork that pinned an older webhook module + doesn't crash here). + """ + try: + from ..webhook import WebhookNotifier + + notifier = WebhookNotifier(self.root_cfg) + stage_count = len(self.pipeline.stages) + if kind == "started": + method = getattr(notifier, "notify_pipeline_started", None) + if method: + method(run_id=state.pipeline_run_id, stage_count=stage_count) + elif kind == "completed": + method = getattr(notifier, "notify_pipeline_completed", None) + if method: + method(run_id=state.pipeline_run_id, final_status=state.final_status, stopped_at=state.stopped_at) + elif kind == "reverted": + method = getattr(notifier, "notify_pipeline_reverted", None) + if method and stage_state is not None: + method( + run_id=state.pipeline_run_id, + stage_name=stage_state.name, + reason=stage_state.error or "auto_revert", + ) + # ``stage_failed`` is currently only an audit event — no + # dedicated webhook method. Existing per-stage + # ``training.failure`` from ForgeTrainer's notifier covers + # the operator surface; the pipeline-level event would be + # redundant noise on Slack. + except Exception as exc: # noqa: BLE001 — best-effort notification; webhook outage must not derail the pipeline. + logger.warning("Failed to emit pipeline webhook %r: %s", kind, exc) + + @staticmethod + def _resolve_input_source( + *, + input_model_override: Optional[str], + stage_model_set: bool, + prev_output: Optional[str], + ) -> InputSourceLiteral: + """Resolve a stage's ``input_source`` from the priority ladder. + + Priority order: CLI override > stage explicit > chain (prev + output) > root. See the ``InputSourceLiteral`` docstring for + what each label means. Extracted from ``_run_single_stage`` + for Sonar python:S3776 cognitive-complexity hygiene; pure + function so the caller stays linear. + """ + if input_model_override is not None: + return "cli_override" + if stage_model_set: + return "stage_explicit" + if prev_output is not None: + return "chain" + return "root" + + def _merge_and_validate_stage( + self, + *, + stage: PipelineStage, + stage_state: PipelineStageState, + prev_output: Optional[str], + input_model_override: Optional[str], + ) -> Optional[Any]: + """Merge the per-stage ForgeConfig + run the chain-presence guard. + + Returns the merged ``ForgeConfig`` on success, or ``None`` if a + config-error path already populated ``stage_state.error`` and + the caller should return ``EXIT_CONFIG_ERROR``. Extracted from + ``_run_single_stage`` for Sonar python:S3776 cognitive- + complexity hygiene. + """ + try: + stage_cfg = merge_pipeline_stage_config( + self.root_cfg, + stage, + prev_output_model=prev_output, + input_model_override=input_model_override, + ) + except Exception as exc: # noqa: BLE001 + stage_state.error = f"Config merge failed: {exc}" + logger.exception("Stage %r config merge failed.", stage.name) + return None + + stage_state.input_model = stage_cfg.model.name_or_path + + # Chain-presence guard: when auto-chained, the previous stage's + # output_dir/final_model leaf must exist. Phase 14 review F-S-1. + if stage_state.input_source == "chain" and not os.path.isdir(stage_cfg.model.name_or_path): + stage_state.error = ( + f"Stage {stage.name!r} requires the previous stage's output at " + f"{stage_cfg.model.name_or_path}; pass --input-model to " + f"override or run the full pipeline first." + ) + logger.error(stage_state.error) + return None + + return stage_cfg + + def _invoke_trainer( + self, + *, + stage_cfg: Any, + stage_name: str, + stage_state: PipelineStageState, + ) -> Optional[Any]: + """Run a single stage's :class:`ForgeTrainer` lifecycle. + + Returns the ``TrainResult`` on success, ``None`` on import / + runtime failure (caller maps to the right exit code via + ``stage_state.error``). Extracted from ``_run_single_stage`` + for Sonar python:S3776 cognitive-complexity hygiene. + """ + # Defer heavy imports so dry-run / --stage on a different stage + # doesn't load torch. + try: + from ..data import prepare_dataset + from ..model import get_model_and_tokenizer + from ..trainer import ForgeTrainer + from ..utils import setup_authentication + except ImportError as e: + stage_state.error = f"Missing training dependency: {e}" + logger.error(stage_state.error) + return None + + # Top-of-stage catch. ForgeTrainer.train() crosses every + # concern (HF load, dataset, TRL, safety, judge, audit, + # compliance, webhook); any uncaught exception must surface as + # a per-stage failure rather than a Python traceback that + # strands the pipeline state. + try: + if not stage_cfg.model.offline: + setup_authentication(stage_cfg.auth.hf_token if stage_cfg.auth else None) + model, tokenizer = get_model_and_tokenizer(stage_cfg) + dataset = prepare_dataset(stage_cfg, tokenizer) + trainer = ForgeTrainer(model=model, tokenizer=tokenizer, config=stage_cfg, dataset=dataset) + return trainer.train() + except Exception as exc: # noqa: BLE001 + stage_state.error = f"Trainer crashed: {exc}" + logger.exception("Stage %r trainer crashed.", stage_name) + return None + + def _run_single_stage( + self, + *, + stage: PipelineStage, + stage_state: PipelineStageState, + prev_output: Optional[str], + input_model_override: Optional[str], + state: PipelineState, + ) -> int: + """Materialise the per-stage ForgeConfig, run the trainer, record results. + + Returns the per-stage exit code; the caller decides what to do + with it (continue, halt, gate). + """ + stage_state.input_source = self._resolve_input_source( + input_model_override=input_model_override, + stage_model_set=stage.model is not None, + prev_output=prev_output, + ) + + stage_cfg = self._merge_and_validate_stage( + stage=stage, + stage_state=stage_state, + prev_output=prev_output, + input_model_override=input_model_override, + ) + if stage_cfg is None: + return EXIT_CONFIG_ERROR + + self._audit_event( + "pipeline.stage_started", + pipeline_run_id=state.pipeline_run_id, + stage_name=stage.name, + stage_index=stage_state.index, + input_model=stage_state.input_model, + input_source=stage_state.input_source, + ) + + result = self._invoke_trainer( + stage_cfg=stage_cfg, + stage_name=stage.name, + stage_state=stage_state, + ) + if result is None: + return EXIT_TRAINING_ERROR + + # Capture results onto the stage state. + stage_state.metrics = {k: float(v) for k, v in (result.metrics or {}).items() if isinstance(v, (int, float))} + stage_state.auto_revert_triggered = bool(result.reverted) + stage_state.training_manifest = os.path.join( + stage_cfg.training.output_dir, "compliance", "training_manifest.json" + ) + + # Human-approval gate produces a non-zero exit + a staging_path. + if result.staging_path: + stage_state.output_model = result.staging_path + return EXIT_AWAITING_APPROVAL + + # Auto-revert or gate failure: trainer returns success=False. + if not result.success: + stage_state.error = result.error or "Stage gate failed." + return EXIT_EVAL_FAILURE + + # Normal success path. + stage_state.output_model = result.final_model_path or os.path.join(stage_cfg.training.output_dir, "final_model") + return EXIT_SUCCESS + + # -- summary -------------------------------------------------------------- + + def _emit_summary(self, state: PipelineState) -> None: + """Operator-facing run summary (text or JSON envelope).""" + if self.output_format == "json": + print(json.dumps(_serialise_state(state), indent=2)) + return + + logger.info("Pipeline %s — final_status=%s", state.pipeline_run_id, state.final_status) + for s in state.stages: + logger.info( + " [%s] %s — trainer=%s, exit=%s, output=%s", + s.status, + s.name, + s.trainer_type, + s.exit_code if s.exit_code is not None else "—", + s.output_model or "—", + ) + + +# --------------------------------------------------------------------------- +# Entry point (CLI dispatcher delegates here) +# --------------------------------------------------------------------------- + + +def run_pipeline_from_args( + root_cfg: ForgeConfig, + pipeline_yaml_bytes: bytes, + args: Any, +) -> int: + """Dispatcher entry — orchestrates flag → orchestrator wiring. + + ``args`` is the parsed ``argparse.Namespace`` from the CLI; only the + pipeline-relevant attributes are read. Keeping this glue thin makes + the orchestrator itself trivial to unit-test against a fake args + namespace. + """ + stage_filter = getattr(args, "stage", None) + resume_from = getattr(args, "resume_from", None) + force_resume = bool(getattr(args, "force_resume", False)) + # Phase 14 review F-F-2: argparse cannot reject an empty string + # value for ``--input-model "" --stage X`` at parse time, and the + # downstream merge helper uses ``is not None`` rather than a truthy + # check, so the empty string would silently overwrite the auto- + # chained model path. Normalise to ``None`` here so the rest of the + # orchestrator sees "no override" rather than "override with nothing". + raw_input_model = getattr(args, "input_model", None) + input_model_override = raw_input_model if raw_input_model else None + output_format = getattr(args, "output_format", "text") + dry_run = bool(getattr(args, "dry_run", False)) + + if stage_filter and resume_from: + logger.error("`--stage` and `--resume-from` are mutually exclusive.") + return EXIT_CONFIG_ERROR + if input_model_override and not stage_filter: + logger.error("`--input-model` requires `--stage `.") + return EXIT_CONFIG_ERROR + + orchestrator = PipelineOrchestrator( + root_cfg=root_cfg, + pipeline_yaml_bytes=pipeline_yaml_bytes, + output_format=output_format, + ) + + if dry_run: + return orchestrator.dry_run() + return orchestrator.run( + stage_filter=stage_filter, + resume_from=resume_from, + force_resume=force_resume, + input_model_override=input_model_override, + ) diff --git a/forgelm/cli/subcommands/_verify_annex_iv.py b/forgelm/cli/subcommands/_verify_annex_iv.py index 4458beb8..79681496 100644 --- a/forgelm/cli/subcommands/_verify_annex_iv.py +++ b/forgelm/cli/subcommands/_verify_annex_iv.py @@ -185,71 +185,169 @@ def _compute_manifest_hash(artifact: Dict[str, Any]) -> str: return compute_annex_iv_manifest_hash(artifact) -def _run_verify_annex_iv_cmd(args, output_format: str) -> None: - """Top-level dispatcher for ``forgelm verify-annex-iv ``.""" - path = getattr(args, "path", None) - if not path: - _output_error_and_exit( - output_format, - "verify-annex-iv requires a path argument: `forgelm verify-annex-iv `.", - EXIT_CONFIG_ERROR, +def _run_pipeline_mode(path: str, output_format: str) -> None: + """Verify a pipeline run directory's manifest + chain integrity. + + Reads ``/compliance/pipeline_manifest.json``, runs the in- + memory verifier + per-stage training_manifest existence check, and + prints / exits. Extracted from :func:`_run_verify_annex_iv_cmd` + for Sonar python:S3776 cognitive-complexity hygiene. + + Exit-code mapping (Phase 14 review-response — aligns with the + single-artefact path's policy in + :func:`_verify_artefact_and_handle_io_errors`): + + - ``EXIT_SUCCESS (0)`` — empty violation list. + - ``EXIT_TRAINING_ERROR (2)`` — genuine runtime I/O failure on a + reachable path (mid-read OSError, locked manifest, …) or the + ``pipeline_manifest.json unreadable: …`` sentinel that + :func:`forgelm.compliance.verify_pipeline_manifest_at_path` + emits when the JSON file is reachable but unreadable. + - ``EXIT_CONFIG_ERROR (1)`` — everything else, including the + ``pipeline_manifest.json not found …`` sentinel and all + structural / chain-integrity violations. + """ + from forgelm.compliance import verify_pipeline_manifest_at_path + + # Defensive try/except: the verifier already maps OSError / + # JSONDecodeError to violation strings, but a future change there + # could let an exception bubble — fail loud rather than swallow. + try: + violations = verify_pipeline_manifest_at_path(path) + except OSError as exc: + msg = f"FAIL: pipeline manifest at {path} — runtime I/O error: {exc}" + if output_format == "json": + print( + json.dumps( + { + "success": False, + "mode": "pipeline", + "path": os.path.abspath(path), + "violations": [str(exc)], + }, + indent=2, + ) + ) + else: + print(msg) + sys.exit(EXIT_TRAINING_ERROR) + + # An "unreadable" sentinel from the verifier means the file is + # reachable but I/O / parse fails — runtime error, not config + # error. "not found" is operator-input → config error. + unreadable = any("unreadable" in v for v in violations) + + if output_format == "json": + print( + json.dumps( + { + "success": not violations, + "mode": "pipeline", + "path": os.path.abspath(path), + "violations": violations, + }, + indent=2, + ) ) - # Round 6 absorption: the prior `os.path.isfile()` pre-check - # collapsed two distinct conditions ("path doesn't exist" and - # "path exists but the user can't stat it") into a single - # `EXIT_CONFIG_ERROR`, breaking the documented exit-code - # contract for the permission-denied case. Move all path - # validation inside the try block and let Python's exception - # hierarchy disambiguate: FileNotFoundError + IsADirectoryError - # = caller-input error (exit 1); OSError (incl. PermissionError) - # = genuine I/O failure on a reachable path (exit 2). + elif not violations: + print(f"OK: pipeline manifest at {path}") + else: + print(f"FAIL: pipeline manifest at {path}") + for v in violations: + print(f" - {v}") + + if not violations: + sys.exit(EXIT_SUCCESS) + sys.exit(EXIT_TRAINING_ERROR if unreadable else EXIT_CONFIG_ERROR) + + +def _verify_artefact_and_handle_io_errors(path: str, output_format: str) -> "VerifyAnnexIVResult": + """Run :func:`verify_annex_iv_artifact` with the documented I/O + error-mapping policy. + + Extracted from :func:`_run_verify_annex_iv_cmd` for Sonar + python:S3776 cognitive-complexity hygiene. Each ``except`` branch + exits the process via ``_output_error_and_exit`` — the function + only returns on success. + + Exit-code mapping (per ``docs/reference/verify_annex_iv_subcommand.md``): + + - ``FileNotFoundError`` / ``IsADirectoryError`` / ``JSONDecodeError`` → + ``EXIT_CONFIG_ERROR (1)`` (operator-actionable input error). + - ``OSError`` (catch-all, must follow ``FileNotFoundError`` because + Python's OSError hierarchy makes it a subclass) → + ``EXIT_TRAINING_ERROR (2)`` (genuine runtime I/O failure). + """ try: - result = verify_annex_iv_artifact(path) + return verify_annex_iv_artifact(path) except (FileNotFoundError, IsADirectoryError) as exc: - # Caller-input error: the path is missing or refers to a - # directory. Per docs/standards/error-handling.md and the - # public exit-code contract in - # docs/reference/verify_annex_iv_subcommand.md, this is - # exit 1 (config / caller error). _output_error_and_exit( output_format, f"Annex IV artifact not found or not a regular file: {path!r} ({exc.__class__.__name__}).", EXIT_CONFIG_ERROR, ) except json.JSONDecodeError as exc: - # Validation failure: the artifact is reachable but not parseable - # as JSON. Operator-actionable input error per the public - # contract; routes to exit 1. _output_error_and_exit( output_format, f"Annex IV artifact at {path!r} is not valid JSON: {exc.msg} (line {exc.lineno}).", EXIT_CONFIG_ERROR, ) except OSError as exc: - # Genuine runtime I/O failure on a reachable path (permission - # denied, mid-read I/O error, locked file, etc.). This is the - # post-FileNotFoundError catch-all because Python's OSError - # hierarchy puts FileNotFoundError as a subclass — order - # matters. _output_error_and_exit( output_format, f"Could not read Annex IV artifact {path!r}: {exc}.", EXIT_TRAINING_ERROR, ) + +def _print_artefact_result(result: "VerifyAnnexIVResult", path: str, output_format: str) -> None: + """Render the per-artefact verify result to stdout (JSON or text).""" payload = result.to_dict() payload["path"] = os.path.abspath(path) if output_format == "json": print(json.dumps({"success": result.valid, **payload}, indent=2)) - else: - if result.valid: - print(f"OK: {path}") - print(f" {result.reason}") - else: - print(f"FAIL: {path}") - print(f" {result.reason}") - for missing in result.missing_fields: - print(f" - missing: {missing}") + return + if result.valid: + print(f"OK: {path}") + print(f" {result.reason}") + return + print(f"FAIL: {path}") + print(f" {result.reason}") + for missing in result.missing_fields: + print(f" - missing: {missing}") + + +def _run_verify_annex_iv_cmd(args, output_format: str) -> None: + """Top-level dispatcher for ``forgelm verify-annex-iv ``. + + Two modes: + + - **Single artefact** (default): ```` is an Annex IV JSON file + and the verifier checks field completeness + manifest hash. + - **Pipeline** (``--pipeline`` flag): ```` is a pipeline run + directory and the verifier reads + ``/compliance/pipeline_manifest.json`` and runs chain- + integrity + stage-index + ``stopped_at`` coherence + per-stage + training_manifest existence checks. Returns a list of violations + and exits 0 only when the list is empty. + """ + path = getattr(args, "path", None) + if not path: + _output_error_and_exit( + output_format, + "verify-annex-iv requires a path argument: `forgelm verify-annex-iv `.", + EXIT_CONFIG_ERROR, + ) + + # Phase 14: ``--pipeline`` mode validates the chain-level manifest. + if getattr(args, "pipeline", False): + _run_pipeline_mode(path, output_format) + + # Single-artefact path: I/O errors map to documented exit codes via + # the helper (which sys.exits on failure); on success we render the + # result and exit with the artefact's validity verdict. + result = _verify_artefact_and_handle_io_errors(path, output_format) + _print_artefact_result(result, path, output_format) sys.exit(EXIT_SUCCESS if result.valid else EXIT_CONFIG_ERROR) diff --git a/forgelm/compliance.py b/forgelm/compliance.py index 873fabe4..2ad47f10 100644 --- a/forgelm/compliance.py +++ b/forgelm/compliance.py @@ -1041,6 +1041,168 @@ def compute_annex_iv_manifest_hash(artifact: Dict[str, Any]) -> str: return _hashlib.sha256(canonical.encode("utf-8")).hexdigest() +# --------------------------------------------------------------------------- +# Phase 14 — Pipeline-level Annex IV manifest +# --------------------------------------------------------------------------- +# +# The canonical ``generate_pipeline_manifest`` + ``export_pipeline_manifest`` +# definitions live further down in this module, alongside the +# ``_provider_metadata_from_config`` helper and ``verify_pipeline_manifest`` +# disk-bound verifier. Only the structural / chain-integrity helper +# (``_verify_manifest_payload``) is declared here; the rest of the +# pipeline-manifest surface clusters near the verifier so reviewers can +# eyeball the schema and its validator together. + + +_PIPELINE_MANIFEST_REQUIRED_KEYS = ( + "forgelm_version", + "pipeline_run_id", + "pipeline_config_hash", + "started_at", + "final_status", + "stages", +) + + +def _check_chain_link(idx: int, stage: Dict[str, Any], stages: List[Dict[str, Any]]) -> Optional[str]: + """Single-stage chain-integrity check. + + Returns a violation string when stage *idx* (claiming + ``input_source == "chain"``) cannot be validated against its + immediate predecessor's ``output_model``; ``None`` when the link + is intact (or when the stage's ``input_source`` is not ``chain`` + in which case the caller should never have invoked this). + Extracted from :func:`_verify_manifest_payload` for Sonar + python:S3776 cognitive-complexity hygiene. + """ + if idx == 0: + return f"Stage {stage.get('name')!r}: input_source='chain' but no previous stage exists (stage 0 cannot chain)." + prev = stages[idx - 1] + prev_output_model = prev.get("output_model") + if not prev_output_model: + return ( + f"chain_integrity_violation at stage {stage.get('name')!r}: claims " + f"input_source='chain' but previous stage {prev.get('name')!r} has " + f"no output_model (status={prev.get('status')!r})." + ) + if stage.get("input_model") != prev_output_model: + return ( + f"chain_integrity_violation at stage {stage.get('name')!r}: input_model={stage.get('input_model')!r} " + f"≠ previous output_model={prev_output_model!r}" + ) + return None + + +def _check_status_consistency(manifest: Dict[str, Any], stages: List[Dict[str, Any]]) -> List[str]: + """Status-consistency checks for a finalised pipeline manifest. + + Covers (a) ``stopped_at`` referent existence + expected status and + (b) the no-``running``-stages-on-a-finalised-manifest rule (F-N-2). + Extracted from :func:`_verify_manifest_payload` for Sonar + python:S3776 cognitive-complexity hygiene. + """ + violations: List[str] = [] + stopped_at = manifest.get("stopped_at") + if stopped_at is not None: + matching = [s for s in stages if s.get("name") == stopped_at] + if not matching: + violations.append(f"stopped_at refers to unknown stage {stopped_at!r}") + elif matching[0].get("status") not in ("failed", "gated_pending_approval"): + violations.append( + f"stopped_at stage {stopped_at!r} has status {matching[0].get('status')!r} " + f"(expected `failed` or `gated_pending_approval`)" + ) + + # Running-stage consistency (N-2): a finalised manifest with a stage + # still in ``running`` indicates the orchestrator crashed mid-stage. + final_status = manifest.get("final_status") + if final_status and final_status != "in_progress": + running_stages = [s.get("name") for s in stages if s.get("status") == "running"] + if running_stages: + violations.append( + f"stage(s) {running_stages!r} still in `running` status on a " + f"finalised manifest (final_status={final_status!r}); the " + f"orchestrator likely crashed mid-stage without updating state." + ) + return violations + + +def _verify_manifest_payload(manifest: Dict[str, Any]) -> List[str]: + """Validate a pipeline manifest's structural + chain-integrity rules. + + Returns a list of human-readable violation strings (empty list ⇒ + manifest is valid). Used by ``forgelm verify-annex-iv --pipeline + `` to surface integrity issues to operators / regulators. + + Checks: + + 1. **Required top-level keys** are present and of the right shape. + 2. **Chain integrity** — for every stage N with + ``input_source == "chain"``, the *immediate* previous stage's + ``output_model`` must match its ``input_model``. If the previous + stage has no ``output_model`` (e.g. failed before saving) the + chain link is unreconstructible and the verifier flags it as a + ``chain_integrity_violation`` (Phase 14 review F-B-3 hardening: + pre-fix the verifier walked across stages that *had* an + output_model, silently accepting a manifest whose chain could + not actually be reconstructed). Stages with + ``input_source != "chain"`` (``root`` / ``stage_explicit`` / + ``cli_override``) intentionally break the chain — by design, + reviewers inspect the audit log to validate them. + 3. **Status consistency** — at most one ``stopped_at`` stage; if + set, that stage's status must be one of ``failed`` / + ``gated_pending_approval``. Additionally, a finalised manifest + (``final_status != "in_progress"``) must not carry any stage + still in ``running`` status — that indicates a process crash + mid-stage that the archive must surface (Phase 14 review F-N-2). + 4. **Index monotonicity** — stage indices form 0..N-1 in order. + """ + violations: List[str] = [] + + for key in _PIPELINE_MANIFEST_REQUIRED_KEYS: + if key not in manifest: + violations.append(f"missing required top-level key: {key!r}") + + stages = manifest.get("stages", []) + if not isinstance(stages, list): + violations.append("`stages` must be a list") + return violations + + # Phase 14 review-response: per-item type-guard. A tampered + # manifest (``stages: [null, "foo"]``) would otherwise raise + # ``AttributeError`` on ``s.get(...)`` partway through the + # verifier. Surface as a structured violation so the disk wrapper + # (and any future caller) gets a clean list back. We filter the + # malformed items out of the rest of the checks because the + # downstream helpers also expect dicts. + well_formed_stages: List[Dict[str, Any]] = [] + for idx, s in enumerate(stages): + if not isinstance(s, dict): + violations.append(f"stage at index {idx} is not an object (got {type(s).__name__})") + continue + well_formed_stages.append(s) + + # Index monotonicity (on the well-formed subset; mis-indexing of + # a malformed entry would be a noise violation on top of the + # already-reported type error). + for expected, s in enumerate(well_formed_stages): + if s.get("index") != expected: + violations.append(f"stage index out of order at position {expected}: got {s.get('index')!r}") + + # Chain integrity (one check per chain-stage; helper is pure). + for idx, s in enumerate(well_formed_stages): + if s.get("input_source") != "chain": + continue + link_violation = _check_chain_link(idx, s, well_formed_stages) + if link_violation is not None: + violations.append(link_violation) + + # Status consistency (stopped_at + running-on-finalised). + violations.extend(_check_status_consistency(manifest, well_formed_stages)) + + return violations + + # --------------------------------------------------------------------------- # Export: All Compliance Artifacts # --------------------------------------------------------------------------- @@ -1484,3 +1646,184 @@ def verify_audit_log( return manifest_failure return chain_result + + +# --------------------------------------------------------------------------- +# Phase 14 — Pipeline manifest (chain-level Annex IV artefact) +# --------------------------------------------------------------------------- +# +# The pipeline manifest is the *index* over a multi-stage training run. +# Per-stage ``training_manifest.json`` files (produced by +# :func:`generate_training_manifest` + :func:`export_compliance_artifacts`) +# remain individually valid against the existing single-stage Annex IV +# schema; the pipeline manifest ties them together at the chain level so +# auditors can verify both the per-stage evidence AND the chain +# integrity that connects the records. +# +# Lives in ``compliance.py`` (alongside the single-stage manifest) so +# Annex IV schema decisions live in one module. The orchestrator +# imports these functions from here and never touches the schema +# directly. + + +def _provider_metadata_from_config(root_cfg: Any) -> Dict[str, Any]: + """Extract the ``annex_iv`` and ``risk_assessment`` provider metadata + from a root :class:`ForgeConfig`, in the shape the pipeline manifest + embeds verbatim. + + Defensive — both blocks are optional; an absent block produces an + absent key in the returned dict rather than a half-populated record. + """ + payload: Dict[str, Any] = {} + comp_cfg = getattr(root_cfg, "compliance", None) + if comp_cfg: + payload["annex_iv"] = { + "provider_name": comp_cfg.provider_name, + "provider_contact": comp_cfg.provider_contact, + "system_name": comp_cfg.system_name, + "intended_purpose": comp_cfg.intended_purpose, + "known_limitations": comp_cfg.known_limitations, + "system_version": comp_cfg.system_version, + "risk_classification": comp_cfg.risk_classification, + } + risk_cfg = getattr(root_cfg, "risk_assessment", None) + if risk_cfg: + payload["risk_assessment"] = { + "intended_use": risk_cfg.intended_use, + "foreseeable_misuse": risk_cfg.foreseeable_misuse, + "risk_category": risk_cfg.risk_category, + "mitigation_measures": risk_cfg.mitigation_measures, + "vulnerable_groups_considered": risk_cfg.vulnerable_groups_considered, + } + return payload + + +def generate_pipeline_manifest(state: Any, root_cfg: Any) -> Dict[str, Any]: + """Build the chain-level Annex IV manifest. + + Accepts the orchestrator's :class:`PipelineState` (any object + exposing the same field names — duck-typed to avoid a circular + import from :mod:`forgelm.cli._pipeline`) and the root + :class:`ForgeConfig`. Returns a JSON-serialisable dict matching the + schema documented in + ``docs/roadmap/phase-14-pipeline-chains.md`` Task 3. + + The per-stage rows are taken from ``state.stages`` verbatim — the + on-disk state file and the manifest carry the same stage payload, so + a reviewer can correlate the two without translating. Provider / + risk metadata is copied from the root config (immutable across the + pipeline run by the per-stage inheritance matrix). + """ + stages_payload: List[Dict[str, Any]] = [] + for s in state.stages: + stages_payload.append( + { + "name": s.name, + "index": s.index, + "trainer_type": s.trainer_type, + "status": s.status, + "input_model": s.input_model, + "input_source": s.input_source, + "output_model": s.output_model, + "started_at": s.started_at, + "finished_at": s.finished_at, + "duration_seconds": s.duration_seconds, + "training_manifest": s.training_manifest, + "metrics": dict(s.metrics or {}), + "gate_decision": s.gate_decision, + "auto_revert_triggered": bool(s.auto_revert_triggered), + "skipped_reason": s.skipped_reason, + "exit_code": s.exit_code, + "error": s.error, + } + ) + + manifest: Dict[str, Any] = { + "forgelm_version": state.forgelm_version, + "generated_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "pipeline_run_id": state.pipeline_run_id, + "pipeline_config_hash": state.pipeline_config_hash, + "started_at": state.started_at, + "finished_at": state.finished_at, + "final_status": state.final_status, + "stopped_at": state.stopped_at, + "stages": stages_payload, + } + manifest.update(_provider_metadata_from_config(root_cfg)) + return manifest + + +def export_pipeline_manifest(manifest: Dict[str, Any], pipeline_output_dir: str) -> str: + """Write *manifest* to ``/compliance/pipeline_manifest.json``. + + Atomic write via tmp file + ``os.replace`` so a partial write on + SIGKILL leaves the previous valid manifest intact — the orchestrator + calls this on every stage transition; an interrupted transition must + not corrupt the artefact. + """ + target_dir = os.path.join(pipeline_output_dir, "compliance") + os.makedirs(target_dir, exist_ok=True) + target_path = os.path.join(target_dir, "pipeline_manifest.json") + tmp_path = target_path + ".tmp" + with open(tmp_path, "w", encoding="utf-8") as f: + json.dump(manifest, f, indent=2, default=str) + os.replace(tmp_path, target_path) + return target_path + + +def verify_pipeline_manifest(manifest: Dict[str, Any]) -> List[str]: + """Public Annex IV verifier for a pipeline manifest *dict*. + + Returns a list of human-readable violation strings; an empty list + means the manifest is structurally and chain-consistently valid. + See :func:`_verify_manifest_payload` for the complete check matrix. + + Library callers (test harnesses, internal integrity checks) consume + this directly; the CLI command ``forgelm verify-annex-iv --pipeline + `` reads the file via :func:`verify_pipeline_manifest_at_path`, + which adds the disk-bound checks (per-stage training_manifest + existence) on top. + """ + return _verify_manifest_payload(manifest) + + +def verify_pipeline_manifest_at_path(pipeline_dir: str) -> List[str]: + """Disk-backed wrapper around :func:`verify_pipeline_manifest`. + + Reads ``/compliance/pipeline_manifest.json``, runs the + in-memory verifier on the parsed payload, then layers on disk-only + checks (per-stage ``training_manifest`` file existence). Pre-flight + failures (missing manifest file, malformed JSON) surface as a single- + entry violation list so the CLI's exit-code mapping is uniform. + """ + manifest_path = os.path.join(pipeline_dir, "compliance", "pipeline_manifest.json") + if not os.path.isfile(manifest_path): + return [f"pipeline_manifest.json not found at {manifest_path}"] + try: + with open(manifest_path, "r", encoding="utf-8") as f: + manifest = json.load(f) + except (OSError, json.JSONDecodeError) as e: + return [f"pipeline_manifest.json unreadable: {e}"] + + violations: List[str] = list(_verify_manifest_payload(manifest)) + + # Disk-only check: each completed stage's training_manifest must + # exist. The in-memory verifier cannot see the filesystem. + # Phase 14 review-response: type-guard each stage entry so a + # tampered manifest (``stages: [null, "foo"]``) surfaces as a + # violation rather than raising ``AttributeError`` on ``s.get``. + raw_stages = manifest.get("stages") + if not isinstance(raw_stages, list): + # In-memory verifier already flagged this; nothing more to + # check on disk. + return violations + for idx, s in enumerate(raw_stages): + if not isinstance(s, dict): + violations.append(f"stage at index {idx} is not an object (got {type(s).__name__})") + continue + name = s.get("name", "") + per_stage_manifest = s.get("training_manifest") + if per_stage_manifest and s.get("status") == "completed" and not os.path.isfile(per_stage_manifest): + violations.append(f"Stage {name!r}: training_manifest at {per_stage_manifest!r} is missing.") + + return violations diff --git a/forgelm/config.py b/forgelm/config.py index 341afe4b..a62d2a04 100644 --- a/forgelm/config.py +++ b/forgelm/config.py @@ -847,6 +847,16 @@ class ForgeConfig(BaseModel): default=None, description="Phase 21 / GDPR Article 5(1)(e) storage limitation + Article 17 erasure horizons block.", ) + pipeline: Optional["PipelineConfig"] = Field( + default=None, + description=( + "Phase 14 — multi-stage training pipeline chain (e.g. SFT → DPO → " + "GRPO). When present, the orchestrator at " + "``forgelm/cli/_pipeline.py`` runs each stage sequentially with " + "section-wholesale inheritance from this root config. When absent, " + "the run path is byte-identical to v0.6.0." + ), + ) def _warn_general_consistency(self) -> None: """Emit warnings for the broad cross-field config inconsistencies.""" @@ -1191,6 +1201,265 @@ class ConfigError(Exception): pass +# --------------------------------------------------------------------------- +# Phase 14 — Multi-Stage Training Pipeline Chains +# --------------------------------------------------------------------------- +# +# A ``pipeline`` block at the root level chains 2+ training stages (e.g. +# SFT → DPO → GRPO) into one config-driven run. See +# ``docs/roadmap/phase-14-pipeline-chains.md`` for the full design spec +# (inheritance matrix, CLI semantics, audit-log events, Annex IV manifest). +# +# Layering rule: the pipeline orchestrator (``forgelm/cli/_pipeline.py``) +# produces one flat :class:`ForgeConfig` per stage via +# :func:`merge_pipeline_stage_config`, then hands that to a fresh +# :class:`ForgeTrainer`. ``forgelm/trainer.py`` is **not** aware of the +# pipeline layer — single-stage runs remain byte-identical to v0.6.0. + + +_STAGE_NAME_PATTERN = r"^[a-z0-9_]{1,32}$" +"""Regex enforcing audit-safe pipeline stage names. + +The name is used in CLI flags (``--stage ``, ``--resume-from ``), +audit-log fields, state-file keys, and webhook payloads — pinning it to a +narrow lowercase / digit / underscore alphabet removes every escaping +concern downstream. +""" + + +class PipelineStage(BaseModel): + """A single stage in a multi-stage training pipeline. + + Section-wholesale override semantics — if a block is present in the + stage YAML it **fully replaces** the root config's block for that + section; if the block is omitted, the stage inherits the root. No + field-level deep-merge: "if you want to inherit, omit the block; if + you want to override, supply the full block." + + Auto-chaining: when the stage does not supply its own ``model:`` block + the orchestrator sets ``model.name_or_path`` to the *previous* + stage's ``training.output_dir/final_model`` path. Stage 0 still reads + the root's ``model.name_or_path``. An explicit per-stage ``model:`` + block (with its required ``name_or_path``) disables auto-chaining for + that stage (operator escape hatch). + + The pipeline-level concerns (``distributed``, ``webhook``, + ``compliance``, ``risk_assessment``, ``monitoring``, ``retention``, + ``synthetic``, ``merge``, ``auth``) live at the root only and have no + per-stage override field — ``extra="forbid"`` causes Pydantic to + reject any attempt to declare them inside a stage, with the + section-name preserved in the error message. + """ + + model_config = ConfigDict(extra="forbid") + + name: str = Field( + ..., + pattern=_STAGE_NAME_PATTERN, + description=( + "Stage identifier. Must match ``[a-z0-9_]{1,32}`` so it can serve as " + "an identifier in CLI flags, audit-log fields, and state-file keys " + "without escaping. Must be unique within the pipeline." + ), + ) + + # Section-wholesale override slots. ``None`` (the default) means + # "inherit this section from the root config"; a populated value + # replaces the root section in full. + model: Optional[ModelConfig] = Field( + default=None, + description=( + "Per-stage model block override. When None, the stage inherits the " + "root ``model`` block AND auto-chains ``model.name_or_path`` to the " + "previous stage's output path." + ), + ) + lora: Optional[LoraConfigModel] = Field( + default=None, + description="Per-stage LoRA / PEFT block override. None → inherit from root.", + ) + training: Optional[TrainingConfig] = Field( + default=None, + description=( + "Per-stage training block override. None → inherit from root. " + "When present, ``trainer_type`` is required (Pydantic's existing " + "``TrainingConfig`` validation enforces this)." + ), + ) + data: Optional[DataConfig] = Field( + default=None, + description=( + "Per-stage data block override. None → inherit from root. Pipelines " + "rarely reuse the same dataset across stages; supplying an explicit " + "per-stage data block is the common case." + ), + ) + evaluation: Optional[EvaluationConfig] = Field( + default=None, + description=( + "Per-stage evaluation block override. None → inherit from root. " + "Per-stage gates (loss thresholds, auto_revert, safety, judge, human-" + "approval) live here; each stage may independently configure its gate." + ), + ) + + @model_validator(mode="after") + def _training_block_must_set_trainer_type_explicitly(self) -> "PipelineStage": + """When a stage supplies its own ``training:`` block, the YAML + MUST explicitly set ``trainer_type`` (each stage states its + alignment paradigm in the YAML — and therefore in the pipeline + manifest — for audit clarity). Without this guard, + ``TrainingConfig.trainer_type`` would silently default to + ``"sft"`` and a DPO stage's manifest could carry + ``trainer_type: sft`` if the operator forgot the field. + """ + if self.training is None: + return self + if "trainer_type" not in self.training.model_fields_set: + raise ValueError( + f"Pipeline stage {self.name!r}: when a 'training:' block is " + f"supplied per stage, 'trainer_type' must be set explicitly " + f"(each stage states its alignment paradigm for audit clarity)." + ) + return self + + +class PipelineConfig(BaseModel): + """Top-level pipeline block. + + Carries the ordered list of stages (≥ 1) that the orchestrator + executes sequentially. The orchestrator is responsible for chaining + each stage's input model to the previous stage's output and emitting + the ``pipeline.*`` audit events documented in the Phase 14 spec. + """ + + model_config = ConfigDict(extra="forbid") + + output_dir: str = Field( + default="./pipeline_run", + description=( + "Pipeline-level output directory. Hosts the pipeline-level audit " + "log, the pipeline state file (``pipeline_state.json``), and the " + "pipeline manifest (``compliance/pipeline_manifest.json``). " + "Distinct from each stage's ``training.output_dir`` — those keep " + "the per-stage checkpoints + per-stage training manifest." + ), + ) + + stages: List[PipelineStage] = Field( + ..., + min_length=1, + description=( + "Ordered list of training stages. Must contain at least one stage; " + "operators who want a single-stage run should omit the ``pipeline:`` " + "block entirely rather than declaring a 1-element pipeline." + ), + ) + + @field_validator("stages") + @classmethod + def _unique_stage_names(cls, v: List[PipelineStage]) -> List[PipelineStage]: + """Reject duplicate stage names early. + + Duplicate names would make ``--stage `` and + ``--resume-from `` ambiguous, and would corrupt the pipeline + state file (which keys per-stage status on the name). + """ + names = [s.name for s in v] + seen = set() + duplicates: List[str] = [] + for n in names: + if n in seen and n not in duplicates: + duplicates.append(n) + seen.add(n) + if duplicates: + raise ValueError( + "Duplicate pipeline stage name(s): " + + ", ".join(repr(d) for d in duplicates) + + ". Stage names must be unique within a pipeline." + ) + return v + + +def merge_pipeline_stage_config( + root_cfg: "ForgeConfig", + stage: PipelineStage, + *, + prev_output_model: Optional[str] = None, + input_model_override: Optional[str] = None, +) -> "ForgeConfig": + """Materialise a flat :class:`ForgeConfig` for a single pipeline stage. + + Implements the section-wholesale inheritance rule documented in + ``docs/roadmap/phase-14-pipeline-chains.md`` Task 2: + + - For each of ``model`` / ``lora`` / ``training`` / ``data`` / + ``evaluation``: if the stage sets the block, the stage's block + replaces the root's; otherwise the root's block is preserved. + - The pipeline-level sections (``distributed``, ``webhook``, + ``compliance``, ``risk_assessment``, ``monitoring``, ``retention``, + ``synthetic``, ``merge``, ``auth``) are kept from the root unchanged + — they cannot be overridden per stage. + + Auto-chain resolution (in priority order): + + 1. ``input_model_override`` (CLI ``--input-model`` flag) — operator + escape hatch. Set ``model.name_or_path`` to this value, regardless + of whether the stage declared its own ``model:`` block. The + caller is responsible for logging the override. + 2. The stage declared its own ``model:`` block — keep the stage's + ``name_or_path``. No auto-chain. + 3. ``prev_output_model`` is not None — auto-chain. Set + ``model.name_or_path`` to this value (typically + ``/final_model``). + 4. Stage 0 with no override — keep the root's ``model.name_or_path``. + + The returned :class:`ForgeConfig` carries no ``pipeline`` block (it + is stripped during the merge) so a downstream :class:`ForgeTrainer` + sees an ordinary single-stage config and behaves byte-identically to + a v0.6.0 single-stage run. + """ + # ``model_dump(exclude_none=True)`` so we don't materialise inherited + # ``Optional[T] = None`` fields the root never set — the + # ``extra="forbid"`` re-validation below would tolerate them, but + # round-tripping no-op None values inflates the per-stage manifest + # and confuses operators reading the merged config. + base = root_cfg.model_dump(exclude_none=True) + base.pop("pipeline", None) + + if stage.model is not None: + base["model"] = stage.model.model_dump(exclude_none=True) + if stage.lora is not None: + base["lora"] = stage.lora.model_dump(exclude_none=True) + if stage.training is not None: + base["training"] = stage.training.model_dump(exclude_none=True) + if stage.data is not None: + base["data"] = stage.data.model_dump(exclude_none=True) + if stage.evaluation is not None: + base["evaluation"] = stage.evaluation.model_dump(exclude_none=True) + + # Auto-chain resolution. See docstring above for priority order. + if input_model_override is not None: + base["model"]["name_or_path"] = input_model_override + elif stage.model is None and prev_output_model is not None: + # Stage inherited the model block AND a previous output exists — + # the canonical auto-chain case. + base["model"]["name_or_path"] = prev_output_model + # else: keep whatever is already in ``base["model"]["name_or_path"]`` + # (either the stage's explicit value or the root's). + + return ForgeConfig(**base) + + +# Resolve the ``pipeline: Optional["PipelineConfig"]`` forward reference in +# :class:`ForgeConfig` now that :class:`PipelineConfig` is in scope. +# Without this, the first ``ForgeConfig(**yaml_data)`` instantiation that +# carries a populated ``pipeline`` block fails at validation time with a +# ``PydanticUndefinedAnnotation`` because the forward reference was never +# late-bound. +ForgeConfig.model_rebuild() + + def load_config(config_path: str) -> ForgeConfig: """Loads and validates a YAML configuration file.""" if not os.path.exists(config_path): diff --git a/forgelm/webhook.py b/forgelm/webhook.py index cfe7c513..ded6f5db 100644 --- a/forgelm/webhook.py +++ b/forgelm/webhook.py @@ -1,7 +1,7 @@ import json import logging import os -from typing import Dict, Optional +from typing import Any, Dict, Optional from urllib.parse import urlparse import requests @@ -173,7 +173,20 @@ def _send( metrics: Optional[Dict[str, float]] = None, reason: Optional[str] = None, model_path: Optional[str] = None, + **extra: Any, ) -> None: + """Build + post the webhook payload. + + ``**extra`` carries event-specific fields the + ``notify_pipeline_*`` methods forward (stage_count, final_status, + stopped_at, stage_name, …) — Phase 14 review-response fix: pre- + fix the pipeline notifiers passed unknown kwargs to a fixed + signature, the resulting ``TypeError`` was swallowed by the + orchestrator's best-effort try/except, and pipeline webhooks + silently never fired. Extras are merged into the payload under + their original key names so existing Slack / Teams receivers + that pick fields by name keep working. + """ url = self._resolve_url() if not url: return @@ -188,7 +201,7 @@ def _send( # ``model_path`` is included only for ``approval.required`` events; # we add the key unconditionally (even as None) to keep the schema # stable so downstream consumers can rely on its presence. - payload = { + payload: Dict[str, Any] = { "event": event, "run_name": run_name, "status": status, @@ -198,6 +211,15 @@ def _send( # Slack-compatible formatting (receivers can ignore) "attachments": [{"title": title, "text": text, "color": color}], } + # Merge event-specific extras (pipeline.* events carry + # stage_count / final_status / stopped_at / stage_name). Drop + # any extra whose key collides with a base-payload field so the + # contract stays stable; we don't expect collisions in practice + # since the pipeline notifier names are disjoint, but the guard + # makes the merge order explicit. + for key, value in extra.items(): + if key not in payload: + payload[key] = value self._post_payload(url, payload, event) @@ -333,3 +355,105 @@ def notify_reverted(self, run_name: str, reason: str) -> None: color="#ff9900", reason=masked_reason, ) + + # ---------------------------------------------------------------------- + # Phase 14 — pipeline-level notifications + # ---------------------------------------------------------------------- + # + # The pipeline orchestrator drives multi-stage runs and emits its own + # ``pipeline.*`` events alongside (not replacing) the existing per-stage + # ``training.*`` events that each ``ForgeTrainer`` instance still fires. + # Pre-existing Slack / Teams dashboards filtering on ``training.failure`` + # therefore keep working unchanged; pipeline-aware dashboards can + # additionally subscribe to the new ``pipeline.*`` event vocabulary. + + def notify_pipeline_started(self, run_id: str, stage_count: int) -> None: + """Post a "pipeline started" notification. + + Fires once per pipeline run, before any stage executes. Operators + running long chains use this signal to confirm the orchestrator + accepted the config and started the first stage. + """ + if not (self.config and self.config.notify_on_start): + return + self._send( + event="pipeline.started", + run_name=run_id, + status="started", + title=f"Pipeline Started: {run_id}", + text=f"Multi-stage training pipeline began with {stage_count} stage(s).", + color="#0052cc", + stage_count=stage_count, + ) + + def notify_pipeline_completed( + self, + run_id: str, + final_status: str, + stopped_at: Optional[str], + ) -> None: + """Post a "pipeline completed" notification. + + Fires once per pipeline run, after the final stage transition + (success, failure, or revert). ``stopped_at`` names the failing + stage when the chain halted; ``None`` on full success. + + Piggy-backs on ``notify_on_success`` when the pipeline finished + cleanly and on ``notify_on_failure`` when it stopped early — + matches the existing single-stage notification policy so + operators don't see pipeline pings they explicitly silenced. + """ + succeeded = final_status == "completed" + if succeeded: + if not (self.config and self.config.notify_on_success): + return + color = "#36a64f" + title = f"Pipeline Succeeded: {run_id}" + text = "All stages completed successfully." + else: + if not (self.config and self.config.notify_on_failure): + return + color = "#cc0000" + title = f"Pipeline Stopped: {run_id}" + text = f"Pipeline halted at stage {stopped_at!r} with final_status={final_status!r}." + + self._send( + event="pipeline.completed", + run_name=run_id, + status=final_status, + title=title, + text=text, + color=color, + final_status=final_status, + stopped_at=stopped_at, + ) + + def notify_pipeline_reverted(self, run_id: str, stage_name: str, reason: str) -> None: + """Post a "pipeline stage auto-reverted" notification. + + Distinct from ``notify_pipeline_completed(final_status='stopped_at_stage')``: + this fires *at the moment* a stage auto-reverts, before downstream + stages are marked skipped. Operators monitoring a long chain see + the revert event in near-real-time rather than waiting for the + final summary at the end of the run. + + The reason is masked + truncated identically to + :meth:`notify_failure` so a leaked stack trace cannot smuggle + secrets via this path either. + """ + if not (self.config and self.config.notify_on_failure): + return + masked_reason = self._mask_and_truncate_reason(reason) + self._send( + event="pipeline.stage_reverted", + run_name=run_id, + status="reverted", + title=f"Pipeline Stage Reverted: {run_id}", + text=( + f"Stage {stage_name!r} triggered auto-revert; downstream stages " + f"will not run.\n\nReason: {masked_reason}" + ), + color="#ff9900", + stage_name=stage_name, + reason=masked_reason, + ) diff --git a/tests/fixtures/pipeline/auto_revert_at_stage_2.yaml b/tests/fixtures/pipeline/auto_revert_at_stage_2.yaml new file mode 100644 index 00000000..038c50df --- /dev/null +++ b/tests/fixtures/pipeline/auto_revert_at_stage_2.yaml @@ -0,0 +1,43 @@ +# Phase 14 fixture — stage 2 carries an `auto_revert` gate with a +# trivially-impossible `max_acceptable_loss`. In integration tests the +# synthetic eval is wired to exceed that threshold, the trainer +# auto-reverts stage 2's adapters, the orchestrator marks stage 3 +# `skipped_due_to_prior_revert`, and the run exits with +# EXIT_EVAL_FAILURE (3). + +model: + name_or_path: "org/base" +lora: {} +training: + trainer_type: "sft" + output_dir: "./out/revert" +data: + dataset_name_or_path: "org/sft" + +pipeline: + output_dir: "./out/revert" + stages: + - name: sft_stage + training: + trainer_type: "sft" + output_dir: "./out/revert/stage1" + data: + dataset_name_or_path: "org/sft_data" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./out/revert/stage2" + data: + dataset_name_or_path: "org/dpo_data" + evaluation: + auto_revert: true + max_acceptable_loss: 0.001 + baseline_loss: 1.0 + + - name: grpo_stage + training: + trainer_type: "grpo" + output_dir: "./out/revert/stage3" + data: + dataset_name_or_path: "org/math_data" diff --git a/tests/fixtures/pipeline/gated_pending_approval.yaml b/tests/fixtures/pipeline/gated_pending_approval.yaml new file mode 100644 index 00000000..eeec4a39 --- /dev/null +++ b/tests/fixtures/pipeline/gated_pending_approval.yaml @@ -0,0 +1,33 @@ +# Phase 14 fixture — stage 1 carries `require_human_approval: true`. +# Orchestrator must exit EXIT_AWAITING_APPROVAL (4); downstream stages +# must stay `pending` (not `skipped_due_to_prior_revert`) so a +# subsequent `forgelm approve ` + `--resume-from dpo_stage` +# picks the chain back up. + +model: + name_or_path: "org/base" +lora: {} +training: + trainer_type: "sft" + output_dir: "./out/gated" +data: + dataset_name_or_path: "org/sft" + +pipeline: + output_dir: "./out/gated" + stages: + - name: sft_stage + training: + trainer_type: "sft" + output_dir: "./out/gated/stage1" + data: + dataset_name_or_path: "org/sft_data" + evaluation: + require_human_approval: true + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./out/gated/stage2" + data: + dataset_name_or_path: "org/dpo_data" diff --git a/tests/fixtures/pipeline/inheritance_matrix.yaml b/tests/fixtures/pipeline/inheritance_matrix.yaml new file mode 100644 index 00000000..9ee3013b --- /dev/null +++ b/tests/fixtures/pipeline/inheritance_matrix.yaml @@ -0,0 +1,58 @@ +# Phase 14 fixture — exercises every row of the inheritance matrix +# documented in docs/roadmap/phase-14-pipeline-chains.md Task 2: +# auto-chain, section-wholesale override, root-only sections preserved. + +model: + name_or_path: "org/root-base" +lora: + r: 8 + alpha: 16 +training: + trainer_type: "sft" + output_dir: "./out/inheritance" + learning_rate: 2.0e-5 + num_train_epochs: 3 +data: + dataset_name_or_path: "org/root_data" +compliance: + provider_name: "Acme Inc" + provider_contact: "compliance@acme.test" + system_name: "Inheritance Demo" + intended_purpose: "Inheritance-matrix coverage fixture." +webhook: + url: "https://example.test/webhook" + +pipeline: + output_dir: "./out/inheritance" + stages: + # Stage 0 — inherits everything from root; no overrides; reads root model. + - name: stage_inherit_all + + # Stage 1 — overrides lora wholesale; training inherited; auto-chains + # input model to stage_inherit_all's final_model. + - name: stage_lora_override + lora: + r: 32 + alpha: 64 + training: + trainer_type: "sft" + + # Stage 2 — overrides training wholesale (must include trainer_type + # explicitly per Phase 14's audit-clarity validator); overrides data. + - name: stage_training_override + training: + trainer_type: "dpo" + output_dir: "./out/inheritance/stage2" + num_train_epochs: 1 + dpo_beta: 0.1 + data: + dataset_name_or_path: "org/dpo_data" + + # Stage 3 — explicit model block; disables auto-chain; subsequent + # stages still chain from this stage's output. + - name: stage_explicit_model + model: + name_or_path: "meta-llama/Llama-3-8B" + training: + trainer_type: "grpo" + output_dir: "./out/inheritance/stage3" diff --git a/tests/fixtures/pipeline/invalid_distributed_per_stage.yaml b/tests/fixtures/pipeline/invalid_distributed_per_stage.yaml new file mode 100644 index 00000000..14e0a420 --- /dev/null +++ b/tests/fixtures/pipeline/invalid_distributed_per_stage.yaml @@ -0,0 +1,31 @@ +# Phase 14 fixture — invalid per-stage `distributed:` block. +# Phase 14 reserves `distributed`, `webhook`, `compliance`, +# `risk_assessment`, `monitoring`, `retention`, `synthetic`, `merge`, +# and `auth` for the root level only — `PipelineStage`'s +# ``extra="forbid"`` rejects them with the offending section name in +# the Pydantic error. Config load on this fixture must raise +# `ConfigError` (mapped to EXIT_CONFIG_ERROR=1 at the CLI surface). + +model: + name_or_path: "org/base" +lora: {} +training: + trainer_type: "sft" + output_dir: "./out/bad" +data: + dataset_name_or_path: "org/sft" + +pipeline: + output_dir: "./out/bad" + stages: + - name: sft_stage + training: + trainer_type: "sft" + output_dir: "./out/bad/stage1" + data: + dataset_name_or_path: "org/sft_data" + # The next key is the violation — distributed config is + # pipeline-level only. Pydantic rejects with the key name. + distributed: + strategy: "deepspeed" + config_path: "./deepspeed_config.json" diff --git a/tests/fixtures/pipeline/minimal_3_stage.yaml b/tests/fixtures/pipeline/minimal_3_stage.yaml new file mode 100644 index 00000000..bd215afb --- /dev/null +++ b/tests/fixtures/pipeline/minimal_3_stage.yaml @@ -0,0 +1,42 @@ +# Phase 14 fixture — minimal 3-stage SFT → DPO → GRPO pipeline. +# Used as the happy-path smoke baseline for parser + orchestrator tests. + +model: + name_or_path: "org/base-model" +lora: + r: 8 + alpha: 16 +training: + trainer_type: "sft" + output_dir: "./out/pipeline_smoke" + num_train_epochs: 1 +data: + dataset_name_or_path: "org/sft_dataset" + +pipeline: + output_dir: "./out/pipeline_smoke" + stages: + - name: sft_stage + training: + trainer_type: "sft" + output_dir: "./out/pipeline_smoke/stage1_sft" + num_train_epochs: 3 + data: + dataset_name_or_path: "org/sft_dataset" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./out/pipeline_smoke/stage2_dpo" + num_train_epochs: 1 + dpo_beta: 0.1 + data: + dataset_name_or_path: "org/dpo_preferences" + + - name: grpo_stage + training: + trainer_type: "grpo" + output_dir: "./out/pipeline_smoke/stage3_grpo" + grpo_num_generations: 4 + data: + dataset_name_or_path: "org/math_prompts" diff --git a/tests/fixtures/pipeline/stale_state_resume_v1.yaml b/tests/fixtures/pipeline/stale_state_resume_v1.yaml new file mode 100644 index 00000000..fff21fc3 --- /dev/null +++ b/tests/fixtures/pipeline/stale_state_resume_v1.yaml @@ -0,0 +1,33 @@ +# Phase 14 fixture (v1) — used to seed a `pipeline_state.json` whose +# `pipeline_config_hash` was computed against this exact byte sequence. +# Paired with `stale_state_resume_v2.yaml`: re-running with v2 against +# the v1 state file must trigger the stale-config guard (config-hash +# mismatch) unless `--force-resume` is set. + +model: + name_or_path: "org/base" +lora: {} +training: + trainer_type: "sft" + output_dir: "./out/stale" + learning_rate: 2.0e-5 +data: + dataset_name_or_path: "org/sft" + +pipeline: + output_dir: "./out/stale" + stages: + - name: sft_stage + training: + trainer_type: "sft" + output_dir: "./out/stale/stage1" + learning_rate: 2.0e-5 + data: + dataset_name_or_path: "org/sft_data" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./out/stale/stage2" + data: + dataset_name_or_path: "org/dpo_data" diff --git a/tests/fixtures/pipeline/stale_state_resume_v2.yaml b/tests/fixtures/pipeline/stale_state_resume_v2.yaml new file mode 100644 index 00000000..1fdb2398 --- /dev/null +++ b/tests/fixtures/pipeline/stale_state_resume_v2.yaml @@ -0,0 +1,34 @@ +# Phase 14 fixture (v2) — paired with `stale_state_resume_v1.yaml` to +# demonstrate the stale-state guard. Differs from v1 in one numeric +# value (learning_rate); the bytes are different so the SHA-256 over +# the file content differs. Re-running with this file against a state +# file produced by v1 must surface the `pipeline_config_hash mismatch` +# refusal. + +model: + name_or_path: "org/base" +lora: {} +training: + trainer_type: "sft" + output_dir: "./out/stale" + learning_rate: 1.0e-4 # ← differs from v1 (2.0e-5) +data: + dataset_name_or_path: "org/sft" + +pipeline: + output_dir: "./out/stale" + stages: + - name: sft_stage + training: + trainer_type: "sft" + output_dir: "./out/stale/stage1" + learning_rate: 1.0e-4 # ← differs from v1 + data: + dataset_name_or_path: "org/sft_data" + + - name: dpo_stage + training: + trainer_type: "dpo" + output_dir: "./out/stale/stage2" + data: + dataset_name_or_path: "org/dpo_data" diff --git a/tests/test_pipeline_cli.py b/tests/test_pipeline_cli.py new file mode 100644 index 00000000..e23c712f --- /dev/null +++ b/tests/test_pipeline_cli.py @@ -0,0 +1,319 @@ +"""Phase 14 — CLI flag interaction + dispatcher tests. + +Covers :func:`forgelm.cli._pipeline.run_pipeline_from_args`, the argparse +glue layer between the top-level parser and +:class:`forgelm.cli._pipeline.PipelineOrchestrator`. Focus: flag +combinatorial rules + dispatch routing. The orchestrator's own state +machine has dedicated coverage in ``tests/test_pipeline_orchestrator.py``; +this file pins the *boundary* between argparse and the orchestrator. +""" + +from __future__ import annotations + +import argparse +import os +import types +from unittest.mock import MagicMock + +import pytest + +from forgelm.cli._exit_codes import EXIT_CONFIG_ERROR, EXIT_SUCCESS +from forgelm.cli._pipeline import run_pipeline_from_args +from forgelm.config import ForgeConfig +from forgelm.results import TrainResult + + +def _three_stage_cfg(tmp_path): + return ForgeConfig( + model={"name_or_path": "org/base"}, + lora={"r": 8}, + training={"trainer_type": "sft"}, + data={"dataset_name_or_path": "org/data"}, + pipeline={ + "output_dir": str(tmp_path / "pipeline_run"), + "stages": [ + { + "name": "sft_stage", + "training": {"trainer_type": "sft", "output_dir": str(tmp_path / "stage1")}, + "data": {"dataset_name_or_path": "org/sft"}, + }, + { + "name": "dpo_stage", + "training": {"trainer_type": "dpo", "output_dir": str(tmp_path / "stage2")}, + "data": {"dataset_name_or_path": "org/dpo"}, + }, + { + "name": "grpo_stage", + "training": {"trainer_type": "grpo", "output_dir": str(tmp_path / "stage3")}, + "data": {"dataset_name_or_path": "org/math"}, + }, + ], + }, + ) + + +def _ns(**overrides): + """Build an argparse.Namespace with the orchestrator-relevant fields.""" + defaults = { + "stage": None, + "resume_from": None, + "force_resume": False, + "input_model": None, + "output_format": "text", + "dry_run": False, + } + defaults.update(overrides) + return argparse.Namespace(**defaults) + + +def _install_fake_trainer(monkeypatch, results): + iterator = iter(results) + instantiated_configs: list = [] + + class _FakeForgeTrainer: + def __init__(self, *, model, tokenizer, config, dataset): + instantiated_configs.append(config) + os.makedirs(os.path.join(config.training.output_dir, "final_model"), exist_ok=True) + self.config = config + + def train(self, resume_from_checkpoint=None): + try: + return next(iterator) + except StopIteration: + return TrainResult(success=True) + + fake_trainer = types.ModuleType("forgelm.trainer") + fake_trainer.ForgeTrainer = _FakeForgeTrainer + monkeypatch.setitem(__import__("sys").modules, "forgelm.trainer", fake_trainer) + + fake_model = types.ModuleType("forgelm.model") + fake_model.get_model_and_tokenizer = lambda config: (MagicMock(), MagicMock()) + monkeypatch.setitem(__import__("sys").modules, "forgelm.model", fake_model) + + fake_data = types.ModuleType("forgelm.data") + fake_data.prepare_dataset = lambda config, tokenizer: {"train": [{"text": "x"}]} + monkeypatch.setitem(__import__("sys").modules, "forgelm.data", fake_data) + + fake_utils = types.ModuleType("forgelm.utils") + fake_utils.setup_authentication = lambda token: None + monkeypatch.setitem(__import__("sys").modules, "forgelm.utils", fake_utils) + + return instantiated_configs + + +class TestFlagInteractionGuards: + """Argparse alone cannot express "X and Y are mutually exclusive + *only when both are non-None*"; the dispatcher enforces it.""" + + def test_stage_and_resume_from_mutually_exclusive(self, tmp_path): + cfg = _three_stage_cfg(tmp_path) + args = _ns(stage="sft_stage", resume_from="dpo_stage") + code = run_pipeline_from_args(cfg, b"yaml", args) + assert code == EXIT_CONFIG_ERROR + + def test_input_model_requires_stage(self, tmp_path): + cfg = _three_stage_cfg(tmp_path) + args = _ns(input_model="some/path") + code = run_pipeline_from_args(cfg, b"yaml", args) + assert code == EXIT_CONFIG_ERROR + + def test_empty_input_model_normalised_to_none(self, tmp_path, monkeypatch): + """Phase 14 review F-F-2 regression: ``--input-model ""`` + (empty string) used to slip past the dispatcher's truthy + check and propagate through ``merge_pipeline_stage_config`` + (``is not None`` branch), silently overwriting the auto-chained + model path with the empty string. The orchestrator now + normalises a falsy ``--input-model`` value to ``None`` before + dispatch so it never reaches the merge helper. + """ + cfg = _three_stage_cfg(tmp_path) + # Pre-create stage 1's output so the chain check on stage 2 is + # not the failure point — the test asserts the empty override + # is normalised, not that the chain is broken. + (tmp_path / "stage1" / "final_model").mkdir(parents=True, exist_ok=True) + configs_seen = _install_fake_trainer(monkeypatch, [TrainResult(success=True)]) + args = _ns(stage="dpo_stage", input_model="") + code = run_pipeline_from_args(cfg, b"yaml", args) + assert code == EXIT_SUCCESS + # The stage's input_model must come from the auto-chain (stage + # 1's on-disk final_model), NOT from the empty string override. + assert configs_seen[0].model.name_or_path == str(tmp_path / "stage1" / "final_model") + + def test_force_resume_without_resume_from_is_noop(self, tmp_path, monkeypatch): + """``--force-resume`` without ``--resume-from`` is meaningless + but not an error — the orchestrator simply ignores it on a + fresh run. We pin this so a future overzealous validator + doesn't start rejecting harmless flag combinations.""" + cfg = _three_stage_cfg(tmp_path) + _install_fake_trainer( + monkeypatch, [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)] + ) + args = _ns(force_resume=True) + code = run_pipeline_from_args(cfg, b"yaml", args) + assert code == EXIT_SUCCESS + + +class TestDispatchToDryRun: + def test_dry_run_routes_to_orchestrator_dry_run(self, tmp_path, monkeypatch): + """``--dry-run`` must take the dry-run branch even when other + pipeline flags are set — keeps operators from accidentally + kicking off a real run when they intended to validate.""" + cfg = _three_stage_cfg(tmp_path) + _install_fake_trainer(monkeypatch, []) + args = _ns(dry_run=True) + code = run_pipeline_from_args(cfg, b"yaml", args) + assert code == EXIT_SUCCESS + # No state / manifest written by dry-run. + assert not os.path.exists(tmp_path / "pipeline_run" / "pipeline_state.json") + + +class TestDispatchToRun: + def test_full_run_routes_to_orchestrator_run(self, tmp_path, monkeypatch): + cfg = _three_stage_cfg(tmp_path) + _install_fake_trainer(monkeypatch, [TrainResult(success=True)] * 3) + args = _ns() + code = run_pipeline_from_args(cfg, b"yaml", args) + assert code == EXIT_SUCCESS + assert os.path.exists(tmp_path / "pipeline_run" / "pipeline_state.json") + + def test_stage_filter_propagates_to_orchestrator(self, tmp_path, monkeypatch): + cfg = _three_stage_cfg(tmp_path) + _install_fake_trainer(monkeypatch, [TrainResult(success=True)]) + args = _ns(stage="sft_stage") + code = run_pipeline_from_args(cfg, b"yaml", args) + assert code == EXIT_SUCCESS + + def test_unknown_stage_filter_rejected_via_orchestrator(self, tmp_path, monkeypatch): + cfg = _three_stage_cfg(tmp_path) + _install_fake_trainer(monkeypatch, []) + args = _ns(stage="ghost") + code = run_pipeline_from_args(cfg, b"yaml", args) + assert code == EXIT_CONFIG_ERROR + + +class TestTopLevelDispatchOrdering: + """Regression for the Phase 14 review F-B-1: the single-stage + ``--dry-run`` path inside ``_maybe_run_no_train_mode`` MUST NOT run + before the orchestrator dispatch when the YAML carries a + ``pipeline:`` block. Otherwise ``forgelm --config pipeline.yaml + --dry-run`` falls through to the legacy single-stage dry-run summary + and the documented per-stage chain-integrity validation never fires. + + Exercises the actual top-level ``main()`` (not just + ``run_pipeline_from_args``) by writing a YAML to disk and invoking + the CLI in-process via the installed entry point. + """ + + def test_pipeline_dry_run_routes_to_orchestrator_not_legacy(self, tmp_path, monkeypatch, caplog): + import logging + import sys as _sys + + import yaml + + yaml_path = tmp_path / "pipeline.yaml" + yaml_path.write_text( + yaml.safe_dump( + { + "model": {"name_or_path": "org/base"}, + "lora": {}, + "training": {"trainer_type": "sft"}, + "data": {"dataset_name_or_path": "org/data"}, + "pipeline": { + "output_dir": str(tmp_path / "pipeline_run"), + "stages": [ + { + "name": "sft_stage", + "training": {"trainer_type": "sft", "output_dir": str(tmp_path / "s1")}, + "data": {"dataset_name_or_path": "org/sft"}, + }, + { + "name": "dpo_stage", + "training": {"trainer_type": "dpo", "output_dir": str(tmp_path / "s2")}, + "data": {"dataset_name_or_path": "org/dpo"}, + }, + ], + }, + } + ) + ) + + from forgelm.cli._dispatch import main as cli_main + + monkeypatch.setattr(_sys, "argv", ["forgelm", "--config", str(yaml_path), "--dry-run"]) + with caplog.at_level(logging.INFO, logger="forgelm.pipeline"): + with pytest.raises(SystemExit) as exc_info: + cli_main() + # The orchestrator's dry-run exits 0 on a clean validation; the + # legacy single-stage dry-run would also exit 0 here, so the + # discriminator below is the orchestrator's INFO log line — + # ``Pipeline dry-run OK: stage(s) validated`` is emitted + # only by ``PipelineOrchestrator.dry_run``. The legacy + # single-stage dry-run path lives at ``forgelm.cli._dry_run`` and + # emits a different log surface. + assert exc_info.value.code == 0 + orchestrator_log_emitted = any("pipeline dry-run ok" in r.message.lower() for r in caplog.records) + assert orchestrator_log_emitted, ( + "Dispatcher routed --dry-run through the legacy single-stage path " + "instead of the pipeline orchestrator (Phase 14 F-B-1 regression). " + f"Captured records: {[r.message for r in caplog.records]!r}" + ) + + def test_pipeline_branch_runs_before_no_train_modes_on_pipeline_config(self, tmp_path, monkeypatch): + """Structural assertion: importing ``_dispatch`` and following + the code path on a ``config.pipeline is not None`` config must + invoke ``run_pipeline_from_args`` strictly before + ``_maybe_run_no_train_mode``. Sentinels track the call order.""" + import yaml + + from forgelm.cli import _dispatch + + yaml_path = tmp_path / "pipeline.yaml" + yaml_path.write_text( + yaml.safe_dump( + { + "model": {"name_or_path": "org/base"}, + "lora": {}, + "training": {"trainer_type": "sft"}, + "data": {"dataset_name_or_path": "org/data"}, + "pipeline": { + "output_dir": str(tmp_path / "pipeline_run"), + "stages": [ + { + "name": "sft_stage", + "training": {"trainer_type": "sft", "output_dir": str(tmp_path / "s1")}, + "data": {"dataset_name_or_path": "org/sft"}, + }, + ], + }, + } + ) + ) + + call_order: list[str] = [] + + def _fake_maybe_no_train(config, args): + call_order.append("no_train_mode") + + def _fake_run_pipeline(config, yaml_bytes, args): + call_order.append("pipeline_dispatch") + return 0 + + monkeypatch.setattr(_dispatch, "_maybe_run_no_train_mode", _fake_maybe_no_train) + import forgelm.cli._pipeline as _pipeline_mod + + monkeypatch.setattr(_pipeline_mod, "run_pipeline_from_args", _fake_run_pipeline) + # The dispatcher imports run_pipeline_from_args lazily inside + # main() — patch the function on the source module so the lazy + # import resolves to our fake. + import sys + + monkeypatch.setattr(sys, "argv", ["forgelm", "--config", str(yaml_path)]) + with pytest.raises(SystemExit): + _dispatch.main() + # Pipeline dispatch must fire; no_train_mode must NOT have run + # before it. + assert call_order == ["pipeline_dispatch"], ( + f"Expected pipeline_dispatch first (and only), got {call_order!r}. " + f"Phase 14 F-B-1 regression: legacy no-train path executed before " + f"the pipeline branch on a pipeline config." + ) diff --git a/tests/test_pipeline_compliance.py b/tests/test_pipeline_compliance.py new file mode 100644 index 00000000..00198ca9 --- /dev/null +++ b/tests/test_pipeline_compliance.py @@ -0,0 +1,592 @@ +"""Phase 14 — Pipeline manifest schema + chain-integrity verifier tests. + +Covers :func:`forgelm.compliance.generate_pipeline_manifest` and +:func:`forgelm.compliance.verify_pipeline_manifest`. The pipeline +manifest is the EU AI Act Annex IV chain-of-custody artefact that ties +the per-stage ``training_manifest.json`` files into one verifiable +provenance index. +""" + +from __future__ import annotations + +from forgelm.cli._pipeline import PipelineStageState, PipelineState +from forgelm.compliance import _verify_manifest_payload, generate_pipeline_manifest +from forgelm.config import ForgeConfig + + +def _root_with_compliance() -> ForgeConfig: + return ForgeConfig( + model={"name_or_path": "org/base"}, + lora={"r": 8}, + training={"trainer_type": "sft"}, + data={"dataset_name_or_path": "org/data"}, + compliance={ + "provider_name": "Acme Inc", + "provider_contact": "compliance@acme.test", + "system_name": "Acme Pipeline System", + "intended_purpose": "Customer-service assistant fine-tune", + "system_version": "v0.7.0", + }, + pipeline={ + "stages": [{"name": "sft_stage"}, {"name": "dpo_stage"}, {"name": "grpo_stage"}], + }, + ) + + +def _three_stage_state() -> PipelineState: + """Build a representative 3-stage state for happy-path schema tests.""" + s1 = PipelineStageState( + name="sft_stage", + index=0, + trainer_type="sft", + status="completed", + input_model="org/base", + input_source="root", + output_model="./out/stage1/final_model", + started_at="2026-06-15T12:00:00+00:00", + finished_at="2026-06-15T13:00:00+00:00", + duration_seconds=3600.0, + metrics={"eval_loss": 0.5}, + gate_decision="passed", + exit_code=0, + ) + s2 = PipelineStageState( + name="dpo_stage", + index=1, + trainer_type="dpo", + status="completed", + input_model="./out/stage1/final_model", + input_source="chain", + output_model="./out/stage2/final_model", + started_at="2026-06-15T13:00:00+00:00", + finished_at="2026-06-15T13:45:00+00:00", + duration_seconds=2700.0, + metrics={"eval_loss": 0.3}, + gate_decision="passed", + exit_code=0, + ) + s3 = PipelineStageState( + name="grpo_stage", + index=2, + trainer_type="grpo", + status="completed", + input_model="./out/stage2/final_model", + input_source="chain", + output_model="./out/stage3/final_model", + started_at="2026-06-15T13:45:00+00:00", + finished_at="2026-06-15T14:30:00+00:00", + duration_seconds=2700.0, + metrics={"eval_loss": 0.2}, + gate_decision="passed", + exit_code=0, + ) + return PipelineState( + pipeline_run_id="pl_2026-06-15_a1b2c3", + pipeline_config_hash="sha256:abc", + forgelm_version="0.7.0", + started_at="2026-06-15T12:00:00+00:00", + finished_at="2026-06-15T14:30:00+00:00", + final_status="completed", + stages=[s1, s2, s3], + ) + + +# --------------------------------------------------------------------------- +# generate_pipeline_manifest — schema coverage +# --------------------------------------------------------------------------- + + +class TestManifestSchema: + def test_required_top_level_keys_present(self): + manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance()) + required = { + "forgelm_version", + "generated_at", + "pipeline_run_id", + "pipeline_config_hash", + "started_at", + "finished_at", + "final_status", + "stages", + } + assert required.issubset(manifest.keys()) + + def test_annex_iv_block_propagated_from_root_compliance(self): + manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance()) + assert "annex_iv" in manifest + assert manifest["annex_iv"]["provider_name"] == "Acme Inc" + assert manifest["annex_iv"]["system_name"] == "Acme Pipeline System" + + def test_annex_iv_omitted_when_no_compliance_block(self): + root = ForgeConfig( + model={"name_or_path": "x"}, + lora={}, + training={"trainer_type": "sft"}, + data={"dataset_name_or_path": "y"}, + pipeline={"stages": [{"name": "s1"}]}, + ) + manifest = generate_pipeline_manifest(_three_stage_state(), root) + assert "annex_iv" not in manifest + + def test_stage_payload_carries_chain_fields(self): + manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance()) + s1, s2, s3 = manifest["stages"] + assert s1["index"] == 0 and s2["index"] == 1 and s3["index"] == 2 + assert s2["input_model"] == s1["output_model"] + assert s3["input_model"] == s2["output_model"] + assert all(s["gate_decision"] == "passed" for s in (s1, s2, s3)) + + def test_stage_metrics_are_a_plain_dict_in_payload(self): + """Manifest must be JSON-serialisable; metrics dict must round- + trip through ``json.dumps``.""" + import json as _json + + manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance()) + # No round-trip failure. + _json.dumps(manifest) + + +# --------------------------------------------------------------------------- +# verify_pipeline_manifest — chain integrity +# --------------------------------------------------------------------------- + + +class TestManifestVerification: + def test_clean_manifest_passes(self): + manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance()) + assert _verify_manifest_payload(manifest) == [] + + def test_missing_required_key_flagged(self): + manifest = generate_pipeline_manifest(_three_stage_state(), _root_with_compliance()) + manifest.pop("pipeline_run_id") + violations = _verify_manifest_payload(manifest) + assert any("pipeline_run_id" in v for v in violations) + + def test_chain_integrity_violation_flagged(self): + """Stage 2's input_model ≠ stage 1's output_model on a chain + stage must surface a ``chain_integrity_violation``.""" + state = _three_stage_state() + state.stages[1].input_model = "tampered/value" + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + assert any("chain_integrity_violation" in v for v in violations) + assert any("tampered/value" in v for v in violations) + + def test_cli_override_chain_break_still_flagged(self): + """Even when ``input_source: cli_override`` legitimately breaks + the chain, the verifier still surfaces it (with the same message) + so reviewers can correlate against the audit log to decide + legitimate vs. corrupt. + + Note: only *chain* stages contribute to the integrity check; an + explicit cli_override stage is recorded with ``input_source != + "chain"`` and therefore skipped by design. This test asserts + that contract.""" + state = _three_stage_state() + state.stages[1].input_model = "operator/manual" + state.stages[1].input_source = "cli_override" + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + # cli_override stages don't trip the chain check. + assert all("chain_integrity_violation" not in v for v in violations) + + def test_index_out_of_order_flagged(self): + state = _three_stage_state() + state.stages[0], state.stages[1] = state.stages[1], state.stages[0] + # Indices stay 0/1 but names are swapped — verifier checks index + # vs. positional order. + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + assert any("index out of order" in v for v in violations) + + def test_stopped_at_unknown_stage_flagged(self): + state = _three_stage_state() + state.stopped_at = "ghost_stage" + state.final_status = "stopped_at_stage" + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + assert any("unknown stage" in v and "ghost_stage" in v for v in violations) + + def test_stopped_at_completed_stage_flagged(self): + """If ``stopped_at`` points at a stage whose status is + ``completed`` rather than ``failed`` / ``gated_pending_approval``, + the manifest is internally inconsistent.""" + state = _three_stage_state() + state.stopped_at = "sft_stage" # which has status=completed + state.final_status = "stopped_at_stage" + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + assert any("expected `failed` or `gated_pending_approval`" in v for v in violations) + + +class TestVerifyOnAutoRevertScenario: + """End-to-end: build a state where stage 2 failed and stage 3 was + skipped; the resulting manifest should verify cleanly (it's a valid + record of a real failure, not an integrity violation).""" + + def test_auto_revert_manifest_verifies(self): + s1 = PipelineStageState( + name="sft_stage", + index=0, + trainer_type="sft", + status="completed", + input_model="org/base", + input_source="root", + output_model="./out/stage1/final_model", + exit_code=0, + ) + s2 = PipelineStageState( + name="dpo_stage", + index=1, + trainer_type="dpo", + status="failed", + input_model="./out/stage1/final_model", + input_source="chain", + output_model="./out/stage2/final_model", + auto_revert_triggered=True, + exit_code=3, + error="loss regression", + ) + s3 = PipelineStageState( + name="grpo_stage", + index=2, + trainer_type="grpo", + status="skipped_due_to_prior_revert", + skipped_reason="Stage 'dpo_stage' triggered auto_revert.", + ) + state = PipelineState( + pipeline_run_id="pl_x", + pipeline_config_hash="sha256:abc", + forgelm_version="0.7.0", + started_at="2026-06-15T12:00:00+00:00", + finished_at="2026-06-15T13:30:00+00:00", + final_status="stopped_at_stage", + stopped_at="dpo_stage", + stages=[s1, s2, s3], + ) + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + assert _verify_manifest_payload(manifest) == [] + + +class TestStrictChainIntegrity: + """Phase 14 review F-B-3 + F-N-3 regression: the verifier must + compare every chain stage against its **immediate** predecessor, + not the most-recent stage that happens to carry an + ``output_model``. Without this, a broken/missing prev output is + silently bridged. + """ + + def test_chain_stage_with_prev_missing_output_flagged(self): + """Stage 0 completed normally, stage 1 failed without saving an + output, stage 2 claims input_source='chain'. Pre-fix the + verifier compared stage 2 against stage 0's output, masking the + gap; the strict check now flags it.""" + s0 = PipelineStageState( + name="s0", + index=0, + trainer_type="sft", + status="completed", + input_source="root", + output_model="./out/s0/final_model", + ) + s1 = PipelineStageState( + name="s1", + index=1, + trainer_type="dpo", + status="failed", + input_model="./out/s0/final_model", + input_source="chain", + output_model=None, # crashed before save + ) + s2 = PipelineStageState( + name="s2", + index=2, + trainer_type="grpo", + status="completed", + input_model="./out/s0/final_model", # plausibly stale + input_source="chain", + output_model="./out/s2/final_model", + ) + state = PipelineState( + pipeline_run_id="pl_x", + pipeline_config_hash="sha256:abc", + forgelm_version="0.7.0", + started_at="2026-06-15T12:00:00+00:00", + final_status="stopped_at_stage", + stopped_at="s1", + stages=[s0, s1, s2], + ) + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + assert any("chain_integrity_violation" in v and "'s2'" in v for v in violations), ( + f"Expected stage 's2' to fail chain integrity due to gap; got: {violations!r}" + ) + + def test_chain_stage_at_index_zero_flagged(self): + """Stage 0 cannot have input_source='chain' (there is no + previous stage).""" + s0 = PipelineStageState( + name="s0", + index=0, + trainer_type="sft", + status="completed", + input_source="chain", # wrong — no prev exists + input_model="./somewhere", + output_model="./out/s0/final_model", + ) + state = PipelineState( + pipeline_run_id="pl_x", + pipeline_config_hash="sha256:abc", + forgelm_version="0.7.0", + started_at="2026-06-15T12:00:00+00:00", + final_status="completed", + stages=[s0], + ) + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + assert any("'s0'" in v and "stage 0 cannot chain" in v for v in violations) + + +class TestVerifierFlagsRunningOnFinalisedManifest: + """Phase 14 review F-N-2: a finalised manifest carrying a stage in + ``running`` status is a tell that the orchestrator crashed + mid-stage. The verifier surfaces it so an archival audit catches + the orphan.""" + + def test_running_stage_with_completed_final_status_flagged(self): + s0 = PipelineStageState( + name="s0", + index=0, + trainer_type="sft", + status="completed", + input_source="root", + output_model="./out/s0/final_model", + ) + s1 = PipelineStageState( + name="s1", + index=1, + trainer_type="dpo", + status="running", + input_source="chain", + input_model="./out/s0/final_model", + ) + state = PipelineState( + pipeline_run_id="pl_x", + pipeline_config_hash="sha256:abc", + forgelm_version="0.7.0", + started_at="2026-06-15T12:00:00+00:00", + final_status="completed", + stages=[s0, s1], + ) + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + assert any("running" in v and "'s1'" in v for v in violations) + + def test_running_stage_with_in_progress_final_status_is_ok(self): + """A live run is allowed to carry a ``running`` stage — the + verifier only flags ``running`` on a *finalised* manifest.""" + s0 = PipelineStageState( + name="s0", + index=0, + trainer_type="sft", + status="running", + input_source="root", + ) + state = PipelineState( + pipeline_run_id="pl_x", + pipeline_config_hash="sha256:abc", + forgelm_version="0.7.0", + started_at="2026-06-15T12:00:00+00:00", + final_status="in_progress", + stages=[s0], + ) + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + violations = _verify_manifest_payload(manifest) + assert all("running" not in v for v in violations) + + +class TestVerifyPipelineManifestAtPath: + """Phase 14 review F-N-6: cover the disk-backed wrapper that the + CLI ``forgelm verify-annex-iv --pipeline`` actually invokes.""" + + def test_missing_manifest_returns_single_violation(self, tmp_path): + from forgelm.compliance import verify_pipeline_manifest_at_path + + violations = verify_pipeline_manifest_at_path(str(tmp_path)) + assert len(violations) == 1 + assert "pipeline_manifest.json not found" in violations[0] + + def test_malformed_manifest_returns_single_violation(self, tmp_path): + from forgelm.compliance import verify_pipeline_manifest_at_path + + manifest_dir = tmp_path / "compliance" + manifest_dir.mkdir() + (manifest_dir / "pipeline_manifest.json").write_text("{not valid json") + violations = verify_pipeline_manifest_at_path(str(tmp_path)) + assert len(violations) == 1 + assert "unreadable" in violations[0] + + def test_disk_wrapper_type_guards_non_dict_stage_items(self, tmp_path): + """Phase 14 review-response regression: a tampered manifest where + ``stages`` contains non-dict items (``null`` / a string / etc.) + must surface as a violation, not crash with ``AttributeError`` + inside the disk-only loop's ``s.get(...)`` calls.""" + from forgelm.compliance import verify_pipeline_manifest_at_path + + manifest_dir = tmp_path / "compliance" + manifest_dir.mkdir() + # Build a manifest payload with two malformed stage entries. + bad_manifest = { + "forgelm_version": "0.7.0", + "pipeline_run_id": "pl_x", + "pipeline_config_hash": "sha256:abc", + "started_at": "2026-06-15T12:00:00+00:00", + "final_status": "in_progress", + "stages": [ + None, + "this-should-be-a-dict", + {"name": "ok", "index": 2, "trainer_type": "sft", "status": "pending"}, + ], + } + import json as _json + + (manifest_dir / "pipeline_manifest.json").write_text(_json.dumps(bad_manifest)) + violations = verify_pipeline_manifest_at_path(str(tmp_path)) + assert any("stage at index 0 is not an object" in v for v in violations) + assert any("stage at index 1 is not an object" in v for v in violations) + + def test_completed_stage_with_missing_training_manifest_flagged(self, tmp_path): + """The disk wrapper layers a per-stage training_manifest + existence check on top of the in-memory verifier — the + difference between the two surfaces. This test pins that the + wrapper actually reaches that branch.""" + from forgelm.compliance import verify_pipeline_manifest_at_path + + manifest_dir = tmp_path / "compliance" + manifest_dir.mkdir() + state = _three_stage_state() + # Wire training_manifest paths that don't exist on disk so the + # disk wrapper's existence check has something to fail against. + for s in state.stages: + s.training_manifest = str(tmp_path / s.name / "compliance" / "training_manifest.json") + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + import json as _json + + (manifest_dir / "pipeline_manifest.json").write_text(_json.dumps(manifest)) + violations = verify_pipeline_manifest_at_path(str(tmp_path)) + assert any("training_manifest" in v and "is missing" in v for v in violations) + + +class TestVerifyAnnexIvPipelineModeExitCodes: + """Phase 14 review-response regression: ``forgelm verify-annex-iv + --pipeline `` must map I/O failures to ``EXIT_TRAINING_ERROR`` + (2) and operator-input errors (``not found``, structural / chain- + integrity violations) to ``EXIT_CONFIG_ERROR`` (1). Mirrors the + single-artefact path's exit-code policy.""" + + def _run(self, tmp_path, args_overrides: dict) -> int: + """Invoke ``_run_pipeline_mode`` and capture the ``SystemExit`` code. + + Uses ``pytest.raises(SystemExit)`` rather than a bare + ``try / except SystemExit`` (Sonar python:S5754 / pylint + ``broad-except``). The CLI command always ``sys.exit``s, so we + expect ``SystemExit`` every invocation — a leak past the + context manager would be a real bug worth surfacing. + """ + import argparse as _argparse + + import pytest as _pytest + + from forgelm.cli.subcommands._verify_annex_iv import _run_verify_annex_iv_cmd + + args = _argparse.Namespace(path=str(tmp_path), pipeline=True, **args_overrides) + with _pytest.raises(SystemExit) as exc_info: + _run_verify_annex_iv_cmd(args, "text") + code = exc_info.value.code + return int(code) if code is not None else 0 + + def test_missing_manifest_exits_config_error(self, tmp_path, capsys): + """``not found`` is operator-actionable input → exit 1.""" + code = self._run(tmp_path, {}) + assert code == 1 # EXIT_CONFIG_ERROR + captured = capsys.readouterr().out + assert "FAIL: pipeline manifest" in captured + assert "not found" in captured + + def test_unreadable_manifest_exits_training_error(self, tmp_path, capsys): + """Reachable file that can't be parsed → exit 2 (runtime I/O).""" + manifest_dir = tmp_path / "compliance" + manifest_dir.mkdir() + (manifest_dir / "pipeline_manifest.json").write_text("{not valid json") + code = self._run(tmp_path, {}) + assert code == 2 # EXIT_TRAINING_ERROR + captured = capsys.readouterr().out + assert "unreadable" in captured + + def test_chain_integrity_violation_exits_config_error(self, tmp_path, capsys): + """Structural / chain violations on a readable manifest → + exit 1 (operator-fixable config error).""" + import json as _json + + manifest_dir = tmp_path / "compliance" + manifest_dir.mkdir() + bad_chain_manifest = { + "forgelm_version": "0.7.0", + "pipeline_run_id": "pl_x", + "pipeline_config_hash": "sha256:abc", + "started_at": "2026-06-15T12:00:00+00:00", + "final_status": "completed", + "stages": [ + { + "name": "s0", + "index": 0, + "trainer_type": "sft", + "status": "completed", + "input_source": "root", + "output_model": "./s0/out", + }, + { + "name": "s1", + "index": 1, + "trainer_type": "dpo", + "status": "completed", + "input_source": "chain", + "input_model": "tampered/different/path", # ≠ s0.output_model + "output_model": "./s1/out", + }, + ], + } + (manifest_dir / "pipeline_manifest.json").write_text(_json.dumps(bad_chain_manifest)) + code = self._run(tmp_path, {}) + assert code == 1 # EXIT_CONFIG_ERROR + captured = capsys.readouterr().out + assert "chain_integrity_violation" in captured + + +class TestVerifyOnPartialFilterRun: + """A ``--stage X`` run produces a manifest where other stages have + ``status: skipped_by_filter``; verifier should accept it.""" + + def test_partial_filter_manifest_verifies(self): + s1 = PipelineStageState(name="s1", index=0, trainer_type="sft", status="skipped_by_filter") + s2 = PipelineStageState( + name="s2", + index=1, + trainer_type="dpo", + status="completed", + input_model="./prev/output", + input_source="cli_override", + output_model="./out/stage2/final_model", + exit_code=0, + ) + s3 = PipelineStageState(name="s3", index=2, trainer_type="grpo", status="skipped_by_filter") + state = PipelineState( + pipeline_run_id="pl_x", + pipeline_config_hash="sha256:abc", + forgelm_version="0.7.0", + started_at="2026-06-15T12:00:00+00:00", + final_status="completed", + stages=[s1, s2, s3], + ) + manifest = generate_pipeline_manifest(state, _root_with_compliance()) + assert _verify_manifest_payload(manifest) == [] diff --git a/tests/test_pipeline_config.py b/tests/test_pipeline_config.py new file mode 100644 index 00000000..584e519f --- /dev/null +++ b/tests/test_pipeline_config.py @@ -0,0 +1,369 @@ +"""Phase 14 — Pydantic schema + inheritance-merge tests. + +Covers: + +- :class:`forgelm.config.PipelineStage` field validation (name pattern, + ``extra="forbid"`` rejecting pipeline-only sections). +- :class:`forgelm.config.PipelineConfig` (minimum-1-stage, unique-name + validator). +- :func:`forgelm.config.merge_pipeline_stage_config` — the section- + wholesale inheritance rule + auto-chain priority order documented in + ``docs/roadmap/phase-14-pipeline-chains.md`` Task 2. +- Backward compatibility: a config without a ``pipeline:`` block produces + ``config.pipeline is None``; an existing single-stage config is + byte-identical to v0.6.0 after the schema change. +""" + +from __future__ import annotations + +import pytest +from pydantic import ValidationError + +from forgelm.config import ( + ForgeConfig, + PipelineConfig, + PipelineStage, + merge_pipeline_stage_config, +) + + +def _root_cfg(**overrides): + """Build a minimal valid root ForgeConfig with sensible defaults.""" + base = { + "model": {"name_or_path": "org/base"}, + "lora": {"r": 8, "alpha": 16}, + "training": {"trainer_type": "sft", "num_train_epochs": 3, "learning_rate": 2e-5}, + "data": {"dataset_name_or_path": "org/sft_data"}, + } + base.update(overrides) + return ForgeConfig(**base) + + +# --------------------------------------------------------------------------- +# PipelineStage — name + per-section validation +# --------------------------------------------------------------------------- + + +class TestPipelineStageName: + @pytest.mark.parametrize( + "name", + ["sft_stage", "stage_1", "s", "a" * 32, "abc_def_123"], + ) + def test_valid_names_accepted(self, name): + stage = PipelineStage(name=name) + assert stage.name == name + + @pytest.mark.parametrize( + "name", + [ + "", # empty + "a" * 33, # > 32 chars + "Stage1", # uppercase + "stage-1", # hyphen + "stage 1", # space + "stage.1", # dot + "stage/1", # slash + "stage@1", # at-sign + "stage_!", # punctuation + ], + ) + def test_invalid_names_rejected(self, name): + with pytest.raises(ValidationError): + PipelineStage(name=name) + + def test_name_is_required(self): + with pytest.raises(ValidationError): + PipelineStage() + + +class TestPipelineStageExtraForbid: + """Pipeline-only sections (distributed / webhook / compliance / etc.) + must not appear inside a stage. ``extra="forbid"`` makes Pydantic + reject them with the offending field name in the error. This is the + primary defence against operators putting root-only config inside a + stage by mistake. + """ + + @pytest.mark.parametrize( + "forbidden_section", + [ + "distributed", + "webhook", + "compliance", + "risk_assessment", + "monitoring", + "retention", + "synthetic", + "merge", + "auth", + "pipeline", # no nested pipelines + ], + ) + def test_pipeline_only_section_rejected_in_stage(self, forbidden_section): + with pytest.raises(ValidationError) as exc_info: + PipelineStage(name="s1", **{forbidden_section: {}}) + # The forbidden section name appears in the error so the operator + # knows which key to remove. + assert forbidden_section in str(exc_info.value) + + +class TestPipelineStageOverrides: + """All allowed override slots accept their corresponding config block.""" + + def test_all_override_slots_default_to_none(self): + stage = PipelineStage(name="s1") + assert stage.model is None + assert stage.lora is None + assert stage.training is None + assert stage.data is None + assert stage.evaluation is None + + def test_model_block_override(self): + stage = PipelineStage(name="s1", model={"name_or_path": "org/other"}) + assert stage.model is not None + assert stage.model.name_or_path == "org/other" + + def test_lora_block_override(self): + stage = PipelineStage(name="s1", lora={"r": 32, "alpha": 64}) + assert stage.lora is not None + assert stage.lora.r == 32 + + def test_training_block_override_requires_trainer_type(self): + """``trainer_type`` is required by the existing ``TrainingConfig`` + schema; a stage's training block must therefore supply it. This + is the Phase 14 spec's "each stage explicitly states its + alignment paradigm" rule, enforced via Pydantic's existing + validator rather than a duplicate check.""" + with pytest.raises(ValidationError): + PipelineStage(name="s1", training={"num_train_epochs": 1}) + + def test_data_block_override(self): + stage = PipelineStage(name="s1", data={"dataset_name_or_path": "org/dpo_prefs"}) + assert stage.data is not None + assert stage.data.dataset_name_or_path == "org/dpo_prefs" + + def test_evaluation_block_override(self): + stage = PipelineStage( + name="s1", + evaluation={"auto_revert": True, "max_acceptable_loss": 2.0}, + ) + assert stage.evaluation is not None + assert stage.evaluation.auto_revert is True + + +# --------------------------------------------------------------------------- +# PipelineConfig — list validators +# --------------------------------------------------------------------------- + + +class TestPipelineConfig: + def test_minimum_one_stage_required(self): + with pytest.raises(ValidationError): + PipelineConfig(stages=[]) + + def test_single_stage_pipeline_accepted(self): + """A 1-stage pipeline is technically valid (the spec only forbids + empty pipelines). Whether operators *should* declare one is a + documentation matter, not a schema matter.""" + pl = PipelineConfig(stages=[PipelineStage(name="only")]) + assert len(pl.stages) == 1 + + def test_multi_stage_pipeline(self): + pl = PipelineConfig( + stages=[ + PipelineStage(name="sft_stage"), + PipelineStage(name="dpo_stage"), + PipelineStage(name="grpo_stage"), + ] + ) + assert [s.name for s in pl.stages] == ["sft_stage", "dpo_stage", "grpo_stage"] + + def test_duplicate_stage_names_rejected(self): + with pytest.raises(ValidationError) as exc_info: + PipelineConfig( + stages=[ + PipelineStage(name="dup"), + PipelineStage(name="other"), + PipelineStage(name="dup"), + ] + ) + assert "Duplicate" in str(exc_info.value) + assert "dup" in str(exc_info.value) + + def test_extra_keys_rejected(self): + """``extra="forbid"`` blocks typos like ``stagess`` from being + silently accepted as ignored fields.""" + with pytest.raises(ValidationError): + PipelineConfig(stages=[PipelineStage(name="s1")], stagess=[]) + + +# --------------------------------------------------------------------------- +# ForgeConfig.pipeline wiring + backward compatibility +# --------------------------------------------------------------------------- + + +class TestForgeConfigPipelineField: + def test_pipeline_defaults_to_none(self): + cfg = _root_cfg() + assert cfg.pipeline is None + + def test_pipeline_populated_from_yaml_dict(self): + cfg = _root_cfg(pipeline={"stages": [{"name": "s1"}, {"name": "s2"}]}) + assert cfg.pipeline is not None + assert len(cfg.pipeline.stages) == 2 + + def test_pipeline_section_round_trips_through_model_dump(self): + cfg = _root_cfg(pipeline={"stages": [{"name": "s1", "training": {"trainer_type": "dpo"}}]}) + dumped = cfg.model_dump(exclude_none=True) + assert "pipeline" in dumped + assert dumped["pipeline"]["stages"][0]["name"] == "s1" + + def test_single_stage_config_byte_identical_without_pipeline(self): + """A pre-Phase-14 single-stage config (no ``pipeline:`` key) + must produce a ``ForgeConfig`` indistinguishable from v0.6.0 + for the trainer's purposes — the ``pipeline`` field defaults to + None and is excluded from ``model_dump(exclude_none=True)``.""" + cfg = _root_cfg() + dumped = cfg.model_dump(exclude_none=True) + assert "pipeline" not in dumped + + +# --------------------------------------------------------------------------- +# merge_pipeline_stage_config — section-wholesale + auto-chain priority +# --------------------------------------------------------------------------- + + +class TestMergeSectionWholesale: + def test_stage_with_no_overrides_inherits_root_entirely(self): + root = _root_cfg() + stage = PipelineStage(name="s0") + merged = merge_pipeline_stage_config(root, stage, prev_output_model=None) + assert merged.model.name_or_path == root.model.name_or_path + assert merged.lora.r == root.lora.r + assert merged.training.trainer_type == root.training.trainer_type + assert merged.training.num_train_epochs == root.training.num_train_epochs + assert merged.data.dataset_name_or_path == root.data.dataset_name_or_path + + def test_stage_lora_block_wholesale_replaces_root(self): + root = _root_cfg() + stage = PipelineStage(name="s1", lora={"r": 64, "alpha": 128}) + merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model") + # The stage's lora block fully replaces — fields the stage didn't + # mention fall back to ``LoraConfigModel`` defaults, NOT to the + # root's ``lora`` block's values. + assert merged.lora.r == 64 + assert merged.lora.alpha == 128 + + def test_stage_training_block_wholesale_replaces_root(self): + root = _root_cfg() + stage = PipelineStage(name="s1", training={"trainer_type": "dpo", "num_train_epochs": 1}) + merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model") + assert merged.training.trainer_type == "dpo" + assert merged.training.num_train_epochs == 1 + + def test_stage_data_block_wholesale_replaces_root(self): + root = _root_cfg() + stage = PipelineStage(name="s1", data={"dataset_name_or_path": "org/dpo_prefs"}) + merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model") + assert merged.data.dataset_name_or_path == "org/dpo_prefs" + + def test_stage_evaluation_block_wholesale_replaces_root(self): + root = _root_cfg(evaluation={"auto_revert": False}) + stage = PipelineStage(name="s1", evaluation={"auto_revert": True, "max_acceptable_loss": 1.5}) + merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model") + assert merged.evaluation is not None + assert merged.evaluation.auto_revert is True + assert merged.evaluation.max_acceptable_loss == pytest.approx(1.5) + + def test_pipeline_section_stripped_from_merged_config(self): + """The orchestrator hands the merged ForgeConfig to a single-stage + ``ForgeTrainer`` that has no awareness of pipelines. The + ``pipeline`` block must be absent from the merged config so the + trainer's lifecycle is byte-identical to a v0.6.0 single-stage + run.""" + root = _root_cfg(pipeline={"stages": [{"name": "s0"}, {"name": "s1"}]}) + stage = PipelineStage(name="s0") + merged = merge_pipeline_stage_config(root, stage, prev_output_model=None) + assert merged.pipeline is None + + +class TestMergeAutoChainPriorityOrder: + """The auto-chain resolution rule has four priority levels. These + tests pin every level so a future refactor cannot silently change + the order — operators rely on the documented behaviour for + ``--input-model`` to actually override an in-config ``model:`` block. + """ + + def test_priority_1_input_model_override_wins_over_everything(self): + root = _root_cfg() + stage = PipelineStage(name="s1", model={"name_or_path": "stage/value"}) + merged = merge_pipeline_stage_config( + root, + stage, + prev_output_model="./prev/model", + input_model_override="cli/override", + ) + assert merged.model.name_or_path == "cli/override" + + def test_priority_2_explicit_stage_model_disables_auto_chain(self): + root = _root_cfg() + stage = PipelineStage(name="s1", model={"name_or_path": "stage/explicit"}) + merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model") + assert merged.model.name_or_path == "stage/explicit" + + def test_priority_3_auto_chain_to_prev_output_when_no_overrides(self): + root = _root_cfg() + stage = PipelineStage(name="s1") + merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/final") + assert merged.model.name_or_path == "./prev/final" + + def test_priority_4_stage_zero_inherits_root_model(self): + """Stage 0 of a pipeline (or any stage launched standalone with + ``--stage `` without a prior output) reads the root's + ``model.name_or_path`` unchanged.""" + root = _root_cfg() + stage = PipelineStage(name="s0") + merged = merge_pipeline_stage_config(root, stage, prev_output_model=None) + assert merged.model.name_or_path == root.model.name_or_path + + def test_input_model_override_beats_stage_zero_root_value(self): + """The CLI escape hatch (``--input-model``) must work even on the + first stage — operators using ``--stage + --input-model `` to re-run with a different base model.""" + root = _root_cfg() + stage = PipelineStage(name="s0") + merged = merge_pipeline_stage_config( + root, + stage, + prev_output_model=None, + input_model_override="cli/override", + ) + assert merged.model.name_or_path == "cli/override" + + +class TestMergePreservesRootOnlyBlocks: + """The pipeline-level config sections (distributed, webhook, + compliance, etc.) cannot be overridden per stage by design. After + merge, those blocks must come through from the root verbatim. + """ + + def test_root_webhook_survives_merge(self): + root = _root_cfg(webhook={"url": "https://example.com/hook"}) + stage = PipelineStage(name="s1", training={"trainer_type": "dpo"}) + merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model") + assert merged.webhook is not None + assert merged.webhook.url == "https://example.com/hook" + + def test_root_compliance_metadata_survives_merge(self): + root = _root_cfg( + compliance={ + "provider_name": "Acme Corp", + "provider_contact": "compliance@acme.test", + "system_name": "Pipeline Demo", + "intended_purpose": "Internal eval", + } + ) + stage = PipelineStage(name="s1", training={"trainer_type": "dpo"}) + merged = merge_pipeline_stage_config(root, stage, prev_output_model="./prev/model") + assert merged.compliance is not None + assert merged.compliance.provider_name == "Acme Corp" diff --git a/tests/test_pipeline_orchestrator.py b/tests/test_pipeline_orchestrator.py new file mode 100644 index 00000000..be1c3621 --- /dev/null +++ b/tests/test_pipeline_orchestrator.py @@ -0,0 +1,920 @@ +"""Phase 14 — Pipeline orchestrator state-machine tests. + +Covers the documented edge cases in ``docs/roadmap/phase-14-pipeline-chains.md`` +Tasks 4 + 5: + +- Happy-path 3-stage chain ⇒ all three trainers run, state file + + pipeline manifest written, exit 0. +- Auto-revert at stage 2 ⇒ stage 3 enters ``skipped_due_to_prior_revert``, + ``pipeline.stage_reverted`` audit event fires, exit 3. +- Human-approval gate at stage 1 ⇒ exit 4, downstream stages still + ``pending``, state preserved for ``--resume-from``. +- ``--resume-from `` skips already-completed stages whose output + directory still exists; stale-state guard rejects a config-hash + mismatch unless ``--force-resume`` is set. +- ``--stage `` partial-run filters; missing-prev-output produces + a clear config error. +- Dry-run validates every stage without touching a trainer. + +The :class:`forgelm.trainer.ForgeTrainer` is replaced via lazy-import +monkeypatch — the orchestrator only reaches the heavy modules through +``from ..trainer import ForgeTrainer`` inside ``_run_single_stage``, so +patching ``sys.modules`` and the dataset / model helpers is enough. +""" + +from __future__ import annotations + +import json +import os +import types +from unittest.mock import MagicMock, patch + +from forgelm.cli._exit_codes import ( + EXIT_AWAITING_APPROVAL, + EXIT_CONFIG_ERROR, + EXIT_EVAL_FAILURE, + EXIT_SUCCESS, +) +from forgelm.cli._pipeline import ( + PipelineOrchestrator, + PipelineStageState, + PipelineState, + _compute_pipeline_config_hash, + _deserialise_state, + _generate_run_id, + _serialise_state, +) +from forgelm.config import ForgeConfig +from forgelm.results import TrainResult + +# --------------------------------------------------------------------------- +# Shared fixtures +# --------------------------------------------------------------------------- + + +def _three_stage_config(tmp_path) -> ForgeConfig: + """Build a minimal valid 3-stage pipeline ForgeConfig.""" + return ForgeConfig( + model={"name_or_path": "org/base"}, + lora={"r": 8}, + training={"trainer_type": "sft", "output_dir": str(tmp_path)}, + data={"dataset_name_or_path": "org/sft"}, + pipeline={ + "output_dir": str(tmp_path / "pipeline_run"), + "stages": [ + { + "name": "sft_stage", + "training": {"trainer_type": "sft", "output_dir": str(tmp_path / "stage1")}, + "data": {"dataset_name_or_path": "org/sft_data"}, + }, + { + "name": "dpo_stage", + "training": {"trainer_type": "dpo", "output_dir": str(tmp_path / "stage2")}, + "data": {"dataset_name_or_path": "org/dpo_data"}, + }, + { + "name": "grpo_stage", + "training": {"trainer_type": "grpo", "output_dir": str(tmp_path / "stage3")}, + "data": {"dataset_name_or_path": "org/math_data"}, + }, + ], + }, + ) + + +def _install_trainer_mocks(monkeypatch, train_results): + """Inject mock trainer/model/data modules. + + ``train_results`` is a list of :class:`TrainResult` returned in order + by successive ``ForgeTrainer.train()`` calls. The mock also + materialises the per-stage output dir on disk so the chain-integrity + guard in the orchestrator does not trip when subsequent stages + auto-chain from a "fresh" directory. + """ + iterator = iter(train_results) + instantiated_configs = [] + + class _FakeForgeTrainer: + def __init__(self, *, model, tokenizer, config, dataset): + instantiated_configs.append(config) + # Materialise final_model dir on disk so the next stage's + # chain-integrity guard sees the path it auto-chains to. + final_dir = os.path.join(config.training.output_dir, "final_model") + os.makedirs(final_dir, exist_ok=True) + self.config = config + + def train(self, resume_from_checkpoint=None): + try: + return next(iterator) + except StopIteration: + return TrainResult(success=True) + + fake_trainer_mod = types.ModuleType("forgelm.trainer") + fake_trainer_mod.ForgeTrainer = _FakeForgeTrainer + monkeypatch.setitem(__import__("sys").modules, "forgelm.trainer", fake_trainer_mod) + + fake_model_mod = types.ModuleType("forgelm.model") + fake_model_mod.get_model_and_tokenizer = lambda config: (MagicMock(), MagicMock()) + monkeypatch.setitem(__import__("sys").modules, "forgelm.model", fake_model_mod) + + fake_data_mod = types.ModuleType("forgelm.data") + fake_data_mod.prepare_dataset = lambda config, tokenizer: {"train": [{"text": "x"}]} + monkeypatch.setitem(__import__("sys").modules, "forgelm.data", fake_data_mod) + + fake_utils_mod = types.ModuleType("forgelm.utils") + fake_utils_mod.setup_authentication = lambda token: None + monkeypatch.setitem(__import__("sys").modules, "forgelm.utils", fake_utils_mod) + + return instantiated_configs + + +# --------------------------------------------------------------------------- +# Pure helpers +# --------------------------------------------------------------------------- + + +class TestPipelineConfigHash: + def test_deterministic(self): + h1 = _compute_pipeline_config_hash(b"some yaml bytes") + h2 = _compute_pipeline_config_hash(b"some yaml bytes") + assert h1 == h2 + assert h1.startswith("sha256:") + + def test_different_inputs_different_hashes(self): + assert _compute_pipeline_config_hash(b"v1") != _compute_pipeline_config_hash(b"v2") + + def test_whitespace_sensitive(self): + """Hashing raw bytes — whitespace / key order / comments all matter. + + This is intentional: regulators audit the on-disk artefact, not + the parsed semantic content. An operator who edits the file + between runs gets a new hash even if the YAML is semantically + equivalent.""" + assert _compute_pipeline_config_hash(b"a: 1\nb: 2") != _compute_pipeline_config_hash(b"b: 2\na: 1") + + +class TestRunIdShape: + def test_run_id_format(self): + run_id = _generate_run_id() + # ``pl_YYYY-MM-DD_<6-hex>`` + assert run_id.startswith("pl_") + parts = run_id.split("_") + assert len(parts) == 3 + assert len(parts[1]) == 10 # YYYY-MM-DD + assert len(parts[2]) == 6 # 6-hex + + def test_run_id_unique(self): + ids = {_generate_run_id() for _ in range(50)} + assert len(ids) == 50, "run_id collisions in 50 samples — too narrow randomness?" + + +class TestStateRoundTrip: + def test_serialise_deserialise(self): + state = PipelineState( + pipeline_run_id="pl_test", + pipeline_config_hash="sha256:abc", + forgelm_version="0.7.0-dev", + started_at="2026-06-15T12:00:00+00:00", + stages=[ + PipelineStageState(name="s1", index=0, trainer_type="sft", status="completed"), + PipelineStageState(name="s2", index=1, trainer_type="dpo", status="pending"), + ], + ) + payload = _serialise_state(state) + round_tripped = _deserialise_state(payload) + assert round_tripped.pipeline_run_id == state.pipeline_run_id + assert len(round_tripped.stages) == 2 + assert round_tripped.stages[0].name == "s1" + assert round_tripped.stages[0].status == "completed" + + def test_deserialise_tolerates_unknown_keys(self): + """Future-forward compatibility: a newer manifest schema with + extra fields must not crash an older reader.""" + payload = { + "pipeline_run_id": "pl_test", + "pipeline_config_hash": "sha256:abc", + "forgelm_version": "0.7.0", + "started_at": "2026-06-15T12:00:00+00:00", + "final_status": "completed", + "stages": [ + {"name": "s1", "index": 0, "trainer_type": "sft", "status": "completed", "future_field": "ignore me"} + ], + "future_top_level": "also ignore", + } + state = _deserialise_state(payload) + assert state.stages[0].name == "s1" + + +# --------------------------------------------------------------------------- +# Atomic write helper +# --------------------------------------------------------------------------- + + +class TestAtomicWriteJson: + """Phase 14 review final-round F-S-1 regression: ``_atomic_write_json`` + must use a per-writer-unique temp filename so two writers targeting + the same final path don't truncate each other's tmp mid-write.""" + + def test_per_writer_temp_path_does_not_race_on_shared_tmp(self, tmp_path): + """Pre-fix, both writers wrote to ``.tmp`` — a 16-thread + stress probe produced ``FileNotFoundError`` from ``os.replace`` + (the loser's tmp was unlinked by the winner's rename). Post- + fix, each writer's tmp is unique so no traceback escapes; the + race surfaces as last-writer-wins on the final ``path``.""" + from concurrent.futures import ThreadPoolExecutor + + from forgelm.cli._pipeline import _atomic_write_json + + target = str(tmp_path / "shared.json") + errors: list[Exception] = [] + + def writer(i: int) -> None: + # OSError covers the FileNotFoundError race that the + # shared-tmp implementation produced; pre-fix this branch + # captured the escapes for the assertion below. ``Exception`` + # is wide-enough to surface any unexpected escape too without + # tripping the Sonar ``catch-BaseException`` rule. + try: + _atomic_write_json(target, {"writer": i, "value": "x" * 1024}) + except Exception as exc: # noqa: BLE001 — test probe: record any escape + errors.append(exc) + + with ThreadPoolExecutor(max_workers=8) as pool: + list(pool.map(writer, range(32))) + + # No writer's tmp-vs-replace race should have escaped. + assert errors == [], f"Concurrent writers produced {len(errors)} exception(s): {errors[:3]}" + # And the final path holds *some* valid JSON (last writer wins). + with open(target) as f: + payload = json.load(f) + assert isinstance(payload, dict) + assert "writer" in payload + + def test_orphan_tmp_cleaned_up_on_serialise_failure(self, tmp_path, monkeypatch): + """If the tmp write itself fails (disk full, permission, etc.), + the orphan tmp must not be left behind under the target dir. + Post-fix the helper's bare-except cleanup ensures we don't + accumulate ``*.tmp`` debris when an upstream caller passes a + non-JSON-serialisable payload.""" + from forgelm.cli._pipeline import _atomic_write_json + + target = str(tmp_path / "bad.json") + # ``json.dump`` cannot serialise a set; this triggers the + # try/except in the helper after the tmp file is created but + # before the replace. + try: + _atomic_write_json(target, {"bad": {1, 2, 3}}) + except TypeError: + pass + # No orphan files left in tmp_path. + orphans = [p for p in os.listdir(str(tmp_path)) if p.endswith(".tmp")] + assert orphans == [], f"Orphan tmp file(s) left behind: {orphans!r}" + + +# --------------------------------------------------------------------------- +# Orchestrator behaviour — dry-run +# --------------------------------------------------------------------------- + + +class TestDryRun: + def test_dry_run_validates_every_stage_without_running_trainer(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + configs_seen = _install_trainer_mocks(monkeypatch, []) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch.dry_run() + assert code == EXIT_SUCCESS + # Trainer must not have been instantiated. + assert configs_seen == [] + + def test_dry_run_does_not_write_state_or_manifest(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, []) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + orch.dry_run() + assert not os.path.exists(orch.paths["state_file"]) + assert not os.path.exists(orch.paths["manifest_file"]) + + def test_dry_run_flags_output_dir_collision(self, tmp_path, monkeypatch): + """Phase 14 review F-G-1 (Gemini): two stages resolving to the + same ``training.output_dir`` would silently overwrite each + other's checkpoints and per-stage Annex-IV manifests, breaking + the chain of custody. Dry-run must surface the collision as + EXIT_CONFIG_ERROR before any GPU work.""" + shared_dir = str(tmp_path / "shared_out") + cfg = ForgeConfig( + model={"name_or_path": "org/base"}, + lora={"r": 8}, + training={"trainer_type": "sft", "output_dir": str(tmp_path)}, + data={"dataset_name_or_path": "org/sft"}, + pipeline={ + "output_dir": str(tmp_path / "pipeline_run"), + "stages": [ + { + "name": "sft_stage", + "training": {"trainer_type": "sft", "output_dir": shared_dir}, + "data": {"dataset_name_or_path": "org/sft"}, + }, + { + "name": "dpo_stage", + "training": {"trainer_type": "dpo", "output_dir": shared_dir}, + "data": {"dataset_name_or_path": "org/dpo"}, + }, + ], + }, + ) + _install_trainer_mocks(monkeypatch, []) + orch = PipelineOrchestrator(cfg, b"yaml") + code = orch.dry_run() + assert code == EXIT_CONFIG_ERROR + + def test_dry_run_flags_inherited_output_dir_collision(self, tmp_path, monkeypatch): + """Two stages that both *inherit* the root ``training`` block + (no per-stage override) end up sharing the root's output_dir + by construction — the most common form of the F-G-1 footgun. + Verify the guard catches it even when the collision is + inherited rather than explicit.""" + cfg = ForgeConfig( + model={"name_or_path": "org/base"}, + lora={"r": 8}, + training={"trainer_type": "sft", "output_dir": str(tmp_path / "root_out")}, + data={"dataset_name_or_path": "org/sft"}, + pipeline={ + "output_dir": str(tmp_path / "pipeline_run"), + # Note: no per-stage training: block — both stages inherit + # the root training.output_dir. Setting data: per stage + # so the merge succeeds, isolating the collision. + "stages": [ + {"name": "s1", "data": {"dataset_name_or_path": "org/d1"}}, + {"name": "s2", "data": {"dataset_name_or_path": "org/d2"}}, + ], + }, + ) + _install_trainer_mocks(monkeypatch, []) + orch = PipelineOrchestrator(cfg, b"yaml") + code = orch.dry_run() + assert code == EXIT_CONFIG_ERROR + + +# --------------------------------------------------------------------------- +# Orchestrator behaviour — happy path +# --------------------------------------------------------------------------- + + +class TestHappyPath: + def test_three_stage_chain_runs_all_stages(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + results = [ + TrainResult( + success=True, metrics={"eval_loss": 0.5}, final_model_path=str(tmp_path / "stage1" / "final_model") + ), + TrainResult( + success=True, metrics={"eval_loss": 0.3}, final_model_path=str(tmp_path / "stage2" / "final_model") + ), + TrainResult( + success=True, metrics={"eval_loss": 0.2}, final_model_path=str(tmp_path / "stage3" / "final_model") + ), + ] + configs_seen = _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch.run() + assert code == EXIT_SUCCESS + assert len(configs_seen) == 3 + # Stage 2's input model must be stage 1's output (auto-chain). + assert configs_seen[1].model.name_or_path == str(tmp_path / "stage1" / "final_model") + assert configs_seen[2].model.name_or_path == str(tmp_path / "stage2" / "final_model") + + def test_state_file_written_with_final_status_completed(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + results = [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)] + _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + orch.run() + assert os.path.exists(orch.paths["state_file"]) + with open(orch.paths["state_file"]) as f: + payload = json.load(f) + assert payload["final_status"] == "completed" + assert payload["stopped_at"] is None + assert all(s["status"] == "completed" for s in payload["stages"]) + + def test_manifest_written_with_chain_intact(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + results = [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)] + _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + orch.run() + assert os.path.exists(orch.paths["manifest_file"]) + with open(orch.paths["manifest_file"]) as f: + manifest = json.load(f) + assert manifest["pipeline_run_id"].startswith("pl_") + assert manifest["final_status"] == "completed" + # Chain integrity in the manifest payload. + s1, s2, s3 = manifest["stages"] + assert s2["input_model"] == s1["output_model"] + assert s3["input_model"] == s2["output_model"] + + +# --------------------------------------------------------------------------- +# Orchestrator behaviour — auto-revert + gate failures +# --------------------------------------------------------------------------- + + +class TestAutoRevert: + def test_stage_2_auto_revert_stops_pipeline(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + results = [ + TrainResult(success=True), + TrainResult(success=False, reverted=True, error="loss regression"), + ] + configs_seen = _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch.run() + assert code == EXIT_EVAL_FAILURE + # Stage 3 must NOT have been instantiated. + assert len(configs_seen) == 2 + # State file records the stop-at + skipped status. + with open(orch.paths["state_file"]) as f: + payload = json.load(f) + assert payload["final_status"] == "stopped_at_stage" + assert payload["stopped_at"] == "dpo_stage" + statuses = [s["status"] for s in payload["stages"]] + assert statuses == ["completed", "failed", "skipped_due_to_prior_revert"] + + def test_gate_failure_without_revert_also_stops_pipeline(self, tmp_path, monkeypatch): + """A stage that returns ``success=False`` but ``reverted=False`` + (e.g., a benchmark min-score failure that didn't auto-revert) + still halts the chain — downstream stages would be operating + on an explicitly-failed checkpoint.""" + cfg = _three_stage_config(tmp_path) + results = [TrainResult(success=False, reverted=False, error="benchmark below min")] + _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch.run() + assert code == EXIT_EVAL_FAILURE + with open(orch.paths["state_file"]) as f: + payload = json.load(f) + assert payload["stages"][0]["status"] == "failed" + + +# --------------------------------------------------------------------------- +# Orchestrator behaviour — human-approval gate +# --------------------------------------------------------------------------- + + +class TestHumanApprovalGate: + def test_gated_stage_exits_with_awaiting_approval(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + staging = str(tmp_path / "stage1" / "final_model.staging") + os.makedirs(staging, exist_ok=True) + results = [TrainResult(success=True, staging_path=staging)] + _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch.run() + assert code == EXIT_AWAITING_APPROVAL + + def test_gated_pending_approval_preserves_downstream_pending(self, tmp_path, monkeypatch): + """After a gated stage, downstream stages must stay ``pending`` + (not ``skipped_due_to_prior_revert``) so a subsequent + ``--resume-from`` picks them up post-approval.""" + cfg = _three_stage_config(tmp_path) + staging = str(tmp_path / "stage1" / "final_model.staging") + os.makedirs(staging, exist_ok=True) + results = [TrainResult(success=True, staging_path=staging)] + _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + orch.run() + with open(orch.paths["state_file"]) as f: + payload = json.load(f) + assert payload["final_status"] == "gated_pending_approval" + assert payload["stages"][0]["status"] == "gated_pending_approval" + assert payload["stages"][1]["status"] == "pending" + assert payload["stages"][2]["status"] == "pending" + + def test_gated_stage_emits_dedicated_stage_gated_audit_event(self, tmp_path, monkeypatch): + """Phase 14 review F-N-1 regression: a stage exiting + ``EXIT_AWAITING_APPROVAL`` must emit a dedicated + ``pipeline.stage_gated`` audit event (not + ``pipeline.stage_completed`` with a sub-field). This lets + dashboard / SIEM filters distinguish the gate flow on the event + name alone.""" + cfg = _three_stage_config(tmp_path) + staging = str(tmp_path / "stage1" / "final_model.staging") + os.makedirs(staging, exist_ok=True) + _install_trainer_mocks(monkeypatch, [TrainResult(success=True, staging_path=staging)]) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + orch.run() + audit_path = os.path.join(orch.paths["root_output_dir"], "audit_log.jsonl") + with open(audit_path) as f: + events = [json.loads(line) for line in f if line.strip()] + gated_events = [e for e in events if e.get("event") == "pipeline.stage_gated"] + assert len(gated_events) == 1, ( + f"Expected exactly one pipeline.stage_gated event, got {[e.get('event') for e in events]!r}" + ) + evt = gated_events[0] + assert evt["stage_name"] == "sft_stage" + assert evt["gate_decision"] == "approval_pending" + assert evt["staging_path"] == staging + # Counter-assert: the legacy stage_completed event must NOT have + # fired for this stage (preventing a future regression where both + # events are emitted in parallel and the dashboard double-counts). + completed_for_sft = [ + e for e in events if e.get("event") == "pipeline.stage_completed" and e.get("stage_name") == "sft_stage" + ] + assert completed_for_sft == [] + + +# --------------------------------------------------------------------------- +# Orchestrator behaviour — --stage filter +# --------------------------------------------------------------------------- + + +class TestStageFilter: + def test_unknown_stage_name_rejected(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, []) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch.run(stage_filter="nonexistent_stage") + assert code == EXIT_CONFIG_ERROR + + def test_filter_to_first_stage_runs_only_it(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + results = [TrainResult(success=True)] + configs_seen = _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch.run(stage_filter="sft_stage") + assert code == EXIT_SUCCESS + assert len(configs_seen) == 1 + with open(orch.paths["state_file"]) as f: + payload = json.load(f) + statuses = [s["status"] for s in payload["stages"]] + assert statuses == ["completed", "skipped_by_filter", "skipped_by_filter"] + + def test_filter_to_middle_stage_without_prev_output_fails_cleanly(self, tmp_path, monkeypatch): + """``--stage dpo_stage`` without sft_stage's output on disk must + fail with a clear EXIT_CONFIG_ERROR — not silently fall back to + the root ``model.name_or_path``.""" + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=True)]) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + # Make sure the prev output dir does NOT exist. + assert not os.path.exists(tmp_path / "stage1" / "final_model") + code = orch.run(stage_filter="dpo_stage") + assert code == EXIT_CONFIG_ERROR + + def test_filter_with_input_model_override_succeeds(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + configs_seen = _install_trainer_mocks(monkeypatch, [TrainResult(success=True)]) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + override_path = str(tmp_path / "manual_input") + os.makedirs(override_path, exist_ok=True) + code = orch.run(stage_filter="dpo_stage", input_model_override=override_path) + assert code == EXIT_SUCCESS + assert len(configs_seen) == 1 + assert configs_seen[0].model.name_or_path == override_path + + def test_filter_middle_stage_with_disk_seed_passes_chain_verifier(self, tmp_path, monkeypatch): + """Phase 14 review final-round F-B-2 regression: ``--stage X`` + on a non-first stage whose predecessor's ``final_model`` exists + on disk auto-chains correctly, AND the resulting pipeline + manifest must NOT report a ``chain_integrity_violation``. + + Pre-fix, the ``skipped_by_filter`` predecessor had + ``output_model=null`` while the executed stage had + ``input_source=chain``, so the strict chain verifier (F-B-3) + emitted a ``chain_integrity_violation`` on every legitimate + ``--stage `` run. The fix seeds the predecessor's + ``output_model`` in the manifest from the resolved disk path. + """ + from forgelm.compliance import verify_pipeline_manifest_at_path + + cfg = _three_stage_config(tmp_path) + # Pre-create stage 1's on-disk output so the --stage filter can + # seed the chain. + prev_output_dir = tmp_path / "stage1" / "final_model" + prev_output_dir.mkdir(parents=True, exist_ok=True) + configs_seen = _install_trainer_mocks(monkeypatch, [TrainResult(success=True)]) + orch = PipelineOrchestrator(cfg, b"yaml") + code = orch.run(stage_filter="dpo_stage") + assert code == EXIT_SUCCESS + assert len(configs_seen) == 1 + # The DPO stage must have auto-chained from stage 1's on-disk output. + assert configs_seen[0].model.name_or_path == str(prev_output_dir) + + # Inspect the manifest directly to confirm chain integrity is + # intact. ``verify_pipeline_manifest_at_path`` additionally + # checks each completed stage's per-stage training_manifest + # exists; with a mocked trainer that file isn't actually + # written, so we don't assert ``violations == []`` here — we + # assert only the chain-integrity contract that's the subject + # of the F-B-2 regression. Per-stage training_manifest + # existence is covered by ``test_completed_stage_with_missing_ + # training_manifest_flagged`` in test_pipeline_compliance.py. + import json as _json + + with open(orch.paths["manifest_file"]) as f: + manifest = _json.load(f) + sft_stage_payload, dpo_stage_payload, _grpo = manifest["stages"] + assert sft_stage_payload["status"] == "skipped_by_filter" + # The fix: predecessor's output_model is surfaced from the resolved disk path. + assert sft_stage_payload["output_model"] == str(prev_output_dir) + assert dpo_stage_payload["input_source"] == "chain" + assert dpo_stage_payload["input_model"] == str(prev_output_dir) + + violations = verify_pipeline_manifest_at_path(str(orch.paths["root_output_dir"])) + assert not any("chain_integrity_violation" in v for v in violations), ( + f"Chain integrity must hold for --stage with disk seed; got: {violations!r}" + ) + + +# --------------------------------------------------------------------------- +# Orchestrator behaviour — --resume-from +# --------------------------------------------------------------------------- + + +class TestResumeFrom: + def test_resume_without_state_file_fails(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, []) + orch = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch.run(resume_from="dpo_stage") + assert code == EXIT_CONFIG_ERROR + + def test_resume_skips_completed_stages(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + # First run: stage 1 completes, stage 2 fails (training crash). + results_run1 = [ + TrainResult(success=True), + TrainResult(success=False, error="oom"), + ] + _install_trainer_mocks(monkeypatch, results_run1) + orch1 = PipelineOrchestrator(cfg, b"yaml bytes") + orch1.run() + # Second run: resume from dpo_stage. Stage 1 already completed + # AND its output_model dir exists (mock created it) — must skip. + results_run2 = [TrainResult(success=True), TrainResult(success=True)] + configs_seen2 = _install_trainer_mocks(monkeypatch, results_run2) + orch2 = PipelineOrchestrator(cfg, b"yaml bytes") + code = orch2.run(resume_from="dpo_stage") + assert code == EXIT_SUCCESS + # Only stage 2 + stage 3 ran — stage 1 was skipped. + assert len(configs_seen2) == 2 + + def test_stale_config_hash_rejects_resume(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=False, error="x")]) + orch1 = PipelineOrchestrator(cfg, b"original yaml") + orch1.run() + # Operator edited the YAML; resume against a different hash. + # Post-Phase-14-review: refusal flows back through ``run()`` as a + # plain return value (not ``sys.exit`` mid-method) so the audit- + # log + summary path runs uniformly on every refusal. + orch2 = PipelineOrchestrator(cfg, b"edited yaml") + code = orch2.run(resume_from="dpo_stage") + assert code == EXIT_CONFIG_ERROR + + def test_force_resume_accepts_stale_hash_with_warning(self, tmp_path, monkeypatch, caplog): + import logging + + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=False, error="x")]) + orch1 = PipelineOrchestrator(cfg, b"original yaml") + orch1.run() + _install_trainer_mocks(monkeypatch, [TrainResult(success=True), TrainResult(success=True)]) + orch2 = PipelineOrchestrator(cfg, b"edited yaml") + with caplog.at_level(logging.WARNING, logger="forgelm.pipeline"): + code = orch2.run(resume_from="dpo_stage", force_resume=True) + assert code == EXIT_SUCCESS + + def test_force_resume_emits_audit_event_with_both_hashes(self, tmp_path, monkeypatch): + """Phase 14 review F-B-2 regression: ``--force-resume`` must + emit a ``pipeline.force_resume`` audit event carrying both the + old and new ``pipeline_config_hash`` so a compliance reviewer + can distinguish an operator-approved override from a normal + resume. Pre-fix, only a WARNING log line was emitted — invisible + in the append-only audit-log JSONL stream.""" + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=False, error="x")]) + orch1 = PipelineOrchestrator(cfg, b"yaml v1") + orch1.run() + + _install_trainer_mocks(monkeypatch, [TrainResult(success=True), TrainResult(success=True)]) + orch2 = PipelineOrchestrator(cfg, b"yaml v2 (operator edited)") + orch2.run(resume_from="dpo_stage", force_resume=True) + + audit_path = os.path.join(orch2.paths["root_output_dir"], "audit_log.jsonl") + assert os.path.exists(audit_path) + with open(audit_path) as f: + events = [json.loads(line) for line in f if line.strip()] + force_resume_events = [e for e in events if e.get("event") == "pipeline.force_resume"] + assert len(force_resume_events) == 1, ( + f"Expected exactly one pipeline.force_resume audit event, got " + f"{len(force_resume_events)}: {[e.get('event') for e in events]!r}" + ) + evt = force_resume_events[0] + assert "old_config_hash" in evt + assert "new_config_hash" in evt + assert evt["old_config_hash"] != evt["new_config_hash"] + assert "yaml v1" not in str(evt) and "yaml v2" not in str(evt), ( + "Audit event must record hashes, not raw YAML bytes." + ) + + def test_force_resume_refuses_when_stage_topology_changed(self, tmp_path, monkeypatch): + """Phase 14 review-response regression: ``--force-resume`` must + still refuse when the on-disk state's stage list disagrees with + the current pipeline (count, names, or order). Without this + guard, a renamed/inserted/deleted stage between runs would let + the orchestrator address ``state.stages[i]`` from the old shape + while iterating ``self.pipeline.stages[i]`` from the new shape, + silently corrupting the audit trail.""" + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=False, error="x")]) + orch1 = PipelineOrchestrator(cfg, b"yaml v1") + orch1.run() + + # Build a *different* topology (rename middle stage) while keeping + # the YAML hash changed too, so force_resume is the only thing + # standing between us and a resume. + renamed_cfg = ForgeConfig( + model={"name_or_path": "org/base"}, + lora={"r": 8}, + training={"trainer_type": "sft", "output_dir": str(tmp_path)}, + data={"dataset_name_or_path": "org/sft"}, + pipeline={ + "output_dir": str(tmp_path / "pipeline_run"), + "stages": [ + { + "name": "sft_stage", + "training": {"trainer_type": "sft", "output_dir": str(tmp_path / "stage1")}, + "data": {"dataset_name_or_path": "org/sft_data"}, + }, + { + "name": "renamed_middle", # was 'dpo_stage' + "training": {"trainer_type": "dpo", "output_dir": str(tmp_path / "stage2")}, + "data": {"dataset_name_or_path": "org/dpo_data"}, + }, + { + "name": "grpo_stage", + "training": {"trainer_type": "grpo", "output_dir": str(tmp_path / "stage3")}, + "data": {"dataset_name_or_path": "org/math_data"}, + }, + ], + }, + ) + _install_trainer_mocks(monkeypatch, [TrainResult(success=True), TrainResult(success=True)]) + orch2 = PipelineOrchestrator(renamed_cfg, b"yaml v2 (renamed)") + code = orch2.run(resume_from="grpo_stage", force_resume=True) + assert code == EXIT_CONFIG_ERROR + # No pipeline.force_resume audit event should have been emitted — + # topology mismatch is refused *before* the audit hook. + audit_path = os.path.join(orch2.paths["root_output_dir"], "audit_log.jsonl") + if os.path.exists(audit_path): + with open(audit_path) as f: + events = [json.loads(line) for line in f if line.strip()] + assert not any(e.get("event") == "pipeline.force_resume" for e in events), ( + "pipeline.force_resume must not fire when topology has changed." + ) + + def test_resume_with_unknown_stage_name_fails(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=False, error="x")]) + orch1 = PipelineOrchestrator(cfg, b"yaml") + orch1.run() + orch2 = PipelineOrchestrator(cfg, b"yaml") + code = orch2.run(resume_from="nonexistent") + assert code == EXIT_CONFIG_ERROR + + +# --------------------------------------------------------------------------- +# Orchestrator behaviour — audit + webhook (best-effort emission, no crash) +# --------------------------------------------------------------------------- + + +class TestAuditAndWebhook: + def test_audit_events_emitted_on_happy_path(self, tmp_path, monkeypatch): + cfg = _three_stage_config(tmp_path) + results = [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)] + _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml") + orch.run() + # Audit log file lives under the root output dir. + audit_path = os.path.join(orch.paths["root_output_dir"], "audit_log.jsonl") + assert os.path.exists(audit_path) + with open(audit_path) as f: + events = [json.loads(line) for line in f if line.strip()] + event_names = [e["event"] for e in events] + assert "pipeline.started" in event_names + assert event_names.count("pipeline.stage_started") == 3 + assert event_names.count("pipeline.stage_completed") == 3 + assert "pipeline.completed" in event_names + + def test_all_pipeline_events_share_same_top_level_run_id(self, tmp_path, monkeypatch): + """Phase 14 review final-round F-B-1 regression: every entry in + the pipeline-level audit log must carry the same top-level + ``run_id`` (the pipeline run id), so SIEM filters and Article 12 + correlation work on a single field. Pre-fix, ``_audit_event`` + constructed a fresh ``AuditLogger`` per call → each entry got a + different auto-generated ``fg-``.""" + cfg = _three_stage_config(tmp_path) + results = [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)] + _install_trainer_mocks(monkeypatch, results) + orch = PipelineOrchestrator(cfg, b"yaml") + orch.run() + audit_path = os.path.join(orch.paths["root_output_dir"], "audit_log.jsonl") + with open(audit_path) as f: + entries = [json.loads(line) for line in f if line.strip()] + # Every entry carries a top-level ``run_id`` field; collect the + # set across all pipeline.* events. + pipeline_entries = [e for e in entries if e.get("event", "").startswith("pipeline.")] + assert len(pipeline_entries) >= 5, "expected at least 5 pipeline.* events on a happy 3-stage run" + top_level_ids = {e["run_id"] for e in pipeline_entries} + assert len(top_level_ids) == 1, f"All pipeline.* events must share one top-level run_id, got {top_level_ids!r}" + # The pinned run_id must equal the pipeline_run_id field on the + # entries (which is the contract: pipeline.* events carry the + # pipeline run id in both surfaces). + (top_level_id,) = top_level_ids + pipeline_run_ids = {e["pipeline_run_id"] for e in pipeline_entries} + assert pipeline_run_ids == {top_level_id} + + def test_webhook_failures_do_not_abort_pipeline(self, tmp_path, monkeypatch): + """A crashing webhook notifier must not derail the pipeline. + + Mirrors the existing per-stage webhook discipline — best-effort + telemetry never blocks the operator's actual work.""" + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks( + monkeypatch, [TrainResult(success=True), TrainResult(success=True), TrainResult(success=True)] + ) + with patch("forgelm.webhook.WebhookNotifier", side_effect=RuntimeError("notifier down")): + orch = PipelineOrchestrator(cfg, b"yaml") + code = orch.run() + assert code == EXIT_SUCCESS + + def test_audit_events_share_one_pipeline_run_id(self, tmp_path, monkeypatch): + """Phase 14 review final-round F-B-1 regression: every entry in + ``audit_log.jsonl`` emitted by the orchestrator must carry the + **same** top-level ``run_id`` field (= the pipeline run id), so + SIEM dashboards can group all events for one pipeline run by + that single key. Pre-fix, ``_audit_event`` constructed a fresh + ``AuditLogger`` per call and each got a different auto- + generated ``fg-`` run_id.""" + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=True)] * 3) + orch = PipelineOrchestrator(cfg, b"yaml") + orch.run() + audit_path = os.path.join(orch.paths["root_output_dir"], "audit_log.jsonl") + with open(audit_path) as f: + events = [json.loads(line) for line in f if line.strip()] + # All pipeline.* events must have identical top-level run_id. + run_ids = {e.get("run_id") for e in events if e.get("event", "").startswith("pipeline.")} + assert len(run_ids) == 1, ( + f"Pipeline events must share one top-level run_id; got {run_ids!r} across {len(events)} entries." + ) + # And that single run_id must equal the pipeline run id surfaced + # in each event's ``pipeline_run_id`` field. + pipeline_run_ids = {e.get("pipeline_run_id") for e in events if e.get("event", "").startswith("pipeline.")} + assert pipeline_run_ids == run_ids, ( + f"Top-level run_id {run_ids!r} must equal pipeline_run_id {pipeline_run_ids!r}." + ) + + def test_audit_event_failures_do_not_abort_pipeline(self, tmp_path, monkeypatch): + """Phase 14 review F-N-5 regression: ``_audit_event``'s + documented ``except Exception`` swallow path must be reached and + the pipeline must continue. An audit-write failure (read-only + filesystem, disk full, malformed audit logger config) is + documented as best-effort — the orchestrator emits a WARNING and + carries on; the test patches ``AuditLogger.log_event`` to raise + and asserts the run completes with the expected exit code.""" + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=True)] * 3) + + from forgelm.compliance import AuditLogger as _RealAuditLogger + + original_log_event = _RealAuditLogger.log_event + + def _raising_log_event(self, event, **fields): + if event.startswith("pipeline."): + raise OSError("Disk full / read-only audit log") + return original_log_event(self, event, **fields) + + monkeypatch.setattr(_RealAuditLogger, "log_event", _raising_log_event) + orch = PipelineOrchestrator(cfg, b"yaml") + code = orch.run() + assert code == EXIT_SUCCESS + + def test_emit_summary_json_output_is_round_trippable(self, tmp_path, monkeypatch, capsys): + """Phase 14 review F-N-7 regression: the ``output_format='json'`` + branch of ``_emit_summary`` is the dispatcher contract for + ``forgelm --config pipeline.yaml --output-format json`` — it + must print a single JSON object that round-trips through + ``json.loads`` and carries the per-stage payload reviewers + consume programmatically.""" + cfg = _three_stage_config(tmp_path) + _install_trainer_mocks(monkeypatch, [TrainResult(success=True)] * 3) + orch = PipelineOrchestrator(cfg, b"yaml", output_format="json") + code = orch.run() + assert code == EXIT_SUCCESS + out = capsys.readouterr().out + # The JSON object is the only thing on stdout; logger goes to stderr. + payload = json.loads(out) + assert payload["final_status"] == "completed" + assert len(payload["stages"]) == 3 + assert all(s["status"] == "completed" for s in payload["stages"]) diff --git a/tools/check_bilingual_parity.py b/tools/check_bilingual_parity.py index 28743bad..06553b9e 100644 --- a/tools/check_bilingual_parity.py +++ b/tools/check_bilingual_parity.py @@ -74,6 +74,7 @@ ("docs/guides/iso_soc2_deployer_guide.md", "docs/guides/iso_soc2_deployer_guide-tr.md"), ("docs/guides/library_api.md", "docs/guides/library_api-tr.md"), ("docs/guides/performance.md", "docs/guides/performance-tr.md"), + ("docs/guides/pipeline.md", "docs/guides/pipeline-tr.md"), ("docs/guides/quickstart.md", "docs/guides/quickstart-tr.md"), ("docs/guides/safety_compliance.md", "docs/guides/safety_compliance-tr.md"), ("docs/guides/troubleshooting.md", "docs/guides/troubleshooting-tr.md"),