From 885395da163d623f9b49ef4d1d391bc86896c091 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 30 Apr 2026 17:19:11 +0300 Subject: [PATCH 01/13] =?UTF-8?q?feat(closure):=20foundation=20bundle=20?= =?UTF-8?q?=E2=80=94=20Faz=201-8=20(8=20phase=20consolidation)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This single commit consolidates the first 8 phases of the v0.5.5 closure plan (docs/analysis/code_reviews/closure-plan-202604300906.md). All 8 phases were implemented in parallel git worktrees and merged onto a single branch for review. PHASE BREAKDOWN Faz 1 — Site & doc honesty + count drift sweep Critical closes: F-business-001 (site/compliance.html sahte artefakt isimlerini gerçek _export_compliance_artifacts çıktısıyla rewrite), F-business-002 (site/quickstart.html mock template names → 5 real template), F-business-003 (PyPI publish status sweep — v0.5.0 ✅). Major: F-business-004 (16 GPU vs claimed 18), F-business-007/M-DOC-010 (CONTRIBUTING project structure → architecture.md pointer), F-business-014 (domain-expert template README), M-DOC-011 (CLAUDE.md "26 → 47 test modules"), M-DOC-001 (QMS sop_data_management body rewrite — single v0.5.0 story, no v0.5.1+/v0.5.2 splits). Faz 2 — CI gates & standards drift cleanup Critical closes: F-test-001 (CI cov-fail-under enforcement via pyproject [tool.pytest.ini_options] addopts). Major: F-test-006 (fail-fast: false in matrix), F-test-010 (testing.md cov-fail-under=40 sync), F-test-017 (usermanuals-validate push trigger), F-business-011 (Deprecation cadence section in release.md), FORGELM_OPERATOR=ci-smoke env var added to ci/nightly/usermanuals workflows so distroless containers don't trip Faz 3's operator-raise. Faz 3 — Operator identity + audit forensic completeness Critical: F-compliance-102 (operator identity raises ConfigError instead of "unknown" fallback; getpass + socket.gethostname() chain; FORGELM_ALLOW_ANONYMOUS_OPERATOR=1 escape hatch). Major: F-compliance-114 (os.fsync after flush), F-compliance-117 (HF Hub revision pin in dataset fingerprint), F-compliance-120 (audit.classifier_load_failed event in safety.py), F-compliance-121 (cross-run continuity claim narrowed). New artifact: docs/reference/audit_event_catalog.md (+ -tr.md mirror) — 12 audit events documented with payload schemas + Article mapping. Faz 4 (v2) — Lazy torch + batching + paragraph chunker + lang_sample slice Major: F-performance-101 (lazy torch/transformers/trl in trainer.py + model.py — regression test in tests/test_lazy_imports.py via subprocess pattern), F-performance-102 (safety + judge batch_size=8 with token-pad-longest + per-batch OOM fallback to single-prompt), F-performance-103 (ingestion.py paragraph chunker single-encode + offset slicing via _count_section_tokens reuse), F-performance-106 (data_audit lang_sample[:512]), organisations → organizations (data_audit.py:104), Windows cp1252 portability (build_usermanuals.py line 404 ✓ → OK). Note: build_usermanuals.py incremental cache + ProcessPoolExecutor (F-performance-105) deferred to Faz 30 final-doc-sweep. Faz 5 — Notebook PyPI pin + nightly grep guard Major: F-business-005 / M-DOC-005 — 7 notebooks migrated from git+https://github.com/cemililik/ForgeLM.git → 'forgelm[qlora]==0.5.0' PyPI pin (with comment "Pinned to v0.5.0; bump on each release"). Plus nightly.yml grep step that fails CI if any git+https reappears. Note: Notebook pins held at ==0.5.0 (current PyPI). Faz 33 release PR bumps to ==0.5.5 once published. Faz 6 — forgelm verify-audit subcommand + library function Critical: F-compliance-103 — forgelm.compliance.verify_audit_log() library function (SHA-256 chain + optional HMAC + manifest sidecar) + forgelm verify-audit CLI subcommand (exit 0/1/2). Mirrors AuditLogger.log_event canonicalisation byte-for-byte. 8 integration tests cover valid/tampered/truncated/missing-manifest/HMAC valid /HMAC invalid/--require-hmac no-secret/empty-log paths. Note: docs/guides/safety_compliance-tr.md is feature-scoped excerpt; full structural mirror Faz 30. Faz 7 — safe_post HTTP discipline (Theme ε) New module: forgelm/_http.py (228 lines) — safe_post() with SSRF guard (RFC1918 + loopback + IMDS + multicast + link-local), redirect refusal, http:// refusal (allow_insecure_http opt-in), timeout floor (default 10s; min_timeout=1.0 for webhook backwards-compat), TLS verify=True default, secret-mask Authorization headers on errors. Major: M-201 — extracted from webhook._post_payload, plumbed through judge._call_api_judge and synthetic._call_api_teacher. Major: M-202 — trainer.py:644-646 GRPO classifier reward from_pretrained gains explicit trust_remote_code=False. Tests: 18 new (11 _http + 3 judge + 3 synthetic + 1 trainer). Faz 8 — Webhook lifecycle vocabulary Major: F-compliance-104 — WebhookNotifier.notify_reverted (event= training.reverted, color #ff9900) and notify_awaiting_approval (event=approval.required, status=awaiting_approval, color #ffcc00). Both delegate to _post_payload (so Faz 7's safe_post discipline applies). Wired in trainer.py: _revert_model now emits notify_reverted (replacing the old notify_failure call so the "trainer crashed" vs "trainer succeeded but gate rejected" cases have distinct webhook events); _handle_human_approval_gate emits notify_awaiting_approval. Schema change: _send signature gains model_path: Optional[str] = None (permanent key, populated only for approval.required event). Tests: 4 new (notify_reverted payload + masking + 2048-char truncation + status code != failure; notify_awaiting_approval payload schema + no-weights-leakage assertion). Audit event catalog (Faz 3 + Faz 8 union): 5 webhook events documented with payload schema, security guarantees (5 items), retention guidance. CROSS-CUT FIXES - audit_event_catalog.md + -tr.md: Faz 3 created audit-event vocabulary scaffold; Faz 8 added comprehensive webhook events section. Manually merged so the catalog is one coherent document, not two competing scaffolds. - Patch path migration: All 32 webhook tests now patch forgelm._http.requests.post (the new central HTTP boundary), not the legacy forgelm.webhook.requests.post path. Faz 7 introduced this; Faz 8 tests were aligned during merge so the lifecycle tests still mock correctly. - Slack token literal in test_webhook.py:342 broken into fragments per docs/standards/regex.md Rule 7 — github secret scanning rejected the joined literal even though it's a clearly-fake test fixture. ACCEPTANCE GATES - pytest tests/ (excluding pre-existing test_cost_estimation.py collection error, F-test-005 cluster Faz 12'de kapanacak): 881 passed, 13 skipped, 0 failed - ruff format --check forgelm/ tests/ tools/: 81 files clean - ruff check forgelm/ tests/ tools/: All checks passed - forgelm --config config_template.yaml --dry-run: valid - coverage gate enforced (pyproject addopts --cov-fail-under=40); full-suite total coverage well above gate. CRITICAL CLOSED (8 of master review's 8 critical): F-business-001 (site compliance.html artefacts) F-business-002 (site quickstart.html template names) F-business-003 (PyPI publish status drift) M-DOC-001 (QMS sop_data_management v0.5.x contradiction) F-test-001 (CI cov-fail-under enforcement) F-compliance-102 (operator identity) F-compliance-103 (verify-audit subcommand + library) F-compliance-104 / lifecycle (covered by Faz 8 + downstream notify_reverted/notify_awaiting_approval; F-compliance-101 Article 14 staging staged for Faz 9 — next sprint). DEFERRED (within plan, not silently dropped): - F-performance-105 build_usermanuals.py incremental cache → Faz 30 (final doc sweep already touches this file) - safety_compliance-tr.md full structural mirror → Faz 30 - docs/usermanuals/** generated drift (byod-domain-expert, math-reasoning, "18 GPU profiles", annex_iv.json) → Faz 30 rebuild - docs/reference/usage*.md "18 GPU" → Faz 30 - F-compliance-101 Article 14 path-b real staging directory → Faz 9 - post_training_workflow.ipynb 'forgelm>=0.5.2' pin → Faz 33 release - pre-existing tests/test_cost_estimation.py:7 collection error (F-code-015 / F-test-005 cluster) → Faz 12 fixture consolidation Tracking plan: docs/analysis/code_reviews/closure-plan-202604300906.md Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/ci.yml | 14 +- .github/workflows/nightly.yml | 17 + .github/workflows/usermanuals-validate.yml | 10 + .gitignore | 1 + CHANGELOG.md | 4 +- CLAUDE.md | 2 +- CONTRIBUTING.md | 28 +- README.md | 6 +- docs/guides/safety_compliance-tr.md | 65 ++++ docs/guides/safety_compliance.md | 19 ++ docs/qms/sop_data_management.md | 51 +-- docs/reference/audit_event_catalog-tr.md | 168 ++++++++++ docs/reference/audit_event_catalog.md | 165 ++++++++++ docs/roadmap-tr.md | 12 +- docs/roadmap.md | 12 +- docs/roadmap/releases.md | 4 +- docs/roadmap/risks-and-decisions.md | 2 + docs/standards/logging-observability.md | 38 +++ docs/standards/release.md | 30 ++ docs/standards/testing.md | 4 +- forgelm/_http.py | 228 +++++++++++++ forgelm/cli.py | 111 ++++++- forgelm/compliance.py | 306 +++++++++++++++++- forgelm/config.py | 2 + forgelm/data_audit.py | 7 +- forgelm/ingestion.py | 8 +- forgelm/judge.py | 102 +++++- forgelm/model.py | 28 +- forgelm/safety.py | 122 ++++++- forgelm/synthetic.py | 12 +- forgelm/templates/domain-expert/README.md | 2 +- forgelm/trainer.py | 37 ++- forgelm/webhook.py | 245 ++++++++------ notebooks/dpo_alignment.ipynb | 7 +- notebooks/galore_memory_optimization.ipynb | 8 +- notebooks/grpo_reasoning.ipynb | 7 +- notebooks/kto_binary_feedback.ipynb | 7 +- notebooks/multi_dataset.ipynb | 7 +- notebooks/quickstart_sft.ipynb | 2 +- notebooks/safety_evaluation.ipynb | 9 +- notebooks/synthetic_data_training.ipynb | 9 +- pyproject.toml | 1 + site/compliance.html | 22 +- site/features.html | 2 +- site/index.html | 6 +- site/js/translations.js | 12 +- site/quickstart.html | 6 +- tests/conftest.py | 21 ++ tests/test_compliance.py | 353 +++++++++++++++++++++ tests/test_grpo_reward.py | 35 ++ tests/test_judge_functions.py | 55 ++++ tests/test_lazy_imports.py | 42 +++ tests/test_synthetic.py | 70 ++++ tests/test_webhook.py | 272 +++++++++++++++- tools/build_usermanuals.py | 2 +- 55 files changed, 2521 insertions(+), 296 deletions(-) create mode 100644 docs/guides/safety_compliance-tr.md create mode 100644 docs/reference/audit_event_catalog-tr.md create mode 100644 docs/reference/audit_event_catalog.md create mode 100644 forgelm/_http.py create mode 100644 tests/test_lazy_imports.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7c3ee26d..ded461b9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -6,6 +6,14 @@ on: pull_request: branches: [main] +# Coverage gate (--cov-fail-under=40) is enforced via pyproject.toml's +# [tool.pytest.ini_options].addopts; the test job intentionally does not +# duplicate the flag here. FORGELM_OPERATOR is set so future operator-identity +# checks (Phase 3 hardening) do not raise on distroless runners where USER is +# unset. +env: + FORGELM_OPERATOR: ci-smoke + jobs: # --- Job 1: Lint (fast, no heavy deps) --- lint: @@ -30,6 +38,7 @@ jobs: test: runs-on: ubuntu-latest strategy: + fail-fast: false matrix: python-version: ["3.10", "3.11", "3.12", "3.13"] steps: @@ -46,7 +55,10 @@ jobs: python -m pip install -e ".[dev]" - name: Run tests with coverage - run: pytest -q --tb=short --cov=forgelm --cov-report=term-missing --cov-report=xml:coverage.xml + # --cov, --cov-report=term-missing, and --cov-fail-under=40 come from + # pyproject.toml addopts (single source of truth). Only the XML report + # is added here because it is CI-specific (artifact upload). + run: pytest -q --tb=short --cov-report=xml:coverage.xml - name: Upload coverage (3.11 only) if: matrix.python-version == '3.11' && always() diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 385949d5..ab9d1195 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -7,6 +7,11 @@ on: # No workflow-level permissions — each job declares the minimal set it needs. +# Set FORGELM_OPERATOR so future operator-identity checks (Phase 3 hardening) +# do not raise on distroless runners where USER is unset. +env: + FORGELM_OPERATOR: ci-smoke + jobs: test-latest-deps: name: Test with latest dependencies @@ -71,6 +76,18 @@ jobs: forgelm --config "/tmp/qs-$tpl.yaml" --dry-run done + - name: Forbid git+https in notebooks + run: | + # Notebooks must install ForgeLM from PyPI with a pinned version, + # never from a git+https URL. Closure plan Faz 5 — git+https makes + # Colab installs depend on a specific branch HEAD, breaking + # reproducibility and confusing users when main moves ahead of + # the latest released wheel. + if grep -rn 'git+https://github.com/cemililik/ForgeLM' notebooks/; then + echo "::error::Notebooks must install from PyPI, not git+https" + exit 1 + fi + - name: Ingestion + audit smoke test (Phase 11) run: | # Minimal end-to-end: TXT in → JSONL out → audit report out. diff --git a/.github/workflows/usermanuals-validate.yml b/.github/workflows/usermanuals-validate.yml index 985beb0a..5b89e29f 100644 --- a/.github/workflows/usermanuals-validate.yml +++ b/.github/workflows/usermanuals-validate.yml @@ -6,12 +6,22 @@ name: Validate user-manuals # source. Catches structural problems before they hit main. on: + push: + branches: [main, development] + paths: + - "docs/usermanuals/**" + - "tools/build_usermanuals.py" pull_request: branches: [main] paths: - "docs/usermanuals/**" - "tools/build_usermanuals.py" +# Mirror CI's operator identity so any future pytest invocation does not +# raise on distroless runners where USER is unset. +env: + FORGELM_OPERATOR: ci-smoke + jobs: validate: runs-on: ubuntu-latest diff --git a/.gitignore b/.gitignore index 4cdef8af..590a24d9 100644 --- a/.gitignore +++ b/.gitignore @@ -70,3 +70,4 @@ logs/ # Claude Code runtime artifacts .claude/scheduled_tasks.lock .claude/state/ +.claude/worktrees/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 0107b0ff..374c4997 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -153,9 +153,7 @@ Touch points (so the next reviewer can audit blast radius quickly): Inline-comment batches landing on top of PR #13 (now merged to `main`). Same review surface as rounds 4-7; further hardening on top of the -`v0.5.2` content. The `v0.5.2` git tag + PyPI publish are the -remaining release-engineering steps before this section can be -frozen into a release. +`v0.5.2` content. - **Audit log hardening** (`forgelm/compliance.py`) — HMAC `_hmac` field is now emitted only when `FORGELM_AUDIT_SECRET` is set; without a secret, a key diff --git a/CLAUDE.md b/CLAUDE.md index d9c8d64f..ecea7c0f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -57,7 +57,7 @@ ForgeLM/ │ ├── webhook.py # Slack/Teams notifications │ ├── grpo_rewards.py # Built-in GRPO format/length shaping reward fallback │ └── ... # benchmark, judge, merging, synthetic, wizard, ... -├── tests/ # pytest, 26 test modules +├── tests/ # pytest, 47 test modules ├── docs/ │ ├── roadmap.md # Public roadmap (short index) │ ├── roadmap/ # Detailed phase files + archive diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a0cad91d..ca99966a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -59,30 +59,10 @@ Push your branch and open a Pull Request against `main`. ### Project Structure -``` -forgelm/ -├── cli.py # CLI entry point -├── config.py # Pydantic config models -├── data.py # Dataset loading -├── model.py # Model + LoRA setup -├── trainer.py # Training orchestration -├── results.py # TrainResult dataclass -├── benchmark.py # lm-eval-harness -├── safety.py # Safety evaluation -├── judge.py # LLM-as-Judge -├── compliance.py # EU AI Act artifacts -├── model_card.py # Model card generation -├── merging.py # Model merging (TIES/DARE/SLERP) -├── wizard.py # Interactive config wizard -├── synthetic.py # Synthetic data pipeline -├── webhook.py # Notifications -└── utils.py # Auth & checkpoints - -tests/ # 47 test files -notebooks/ # 10 Colab notebooks -configs/deepspeed/ # ZeRO presets -docs/guides/ # user guides -``` +ForgeLM is a single-package layout: ~26 single-file modules under `forgelm/`, +~47 test files under `tests/`, plus `configs/`, `docs/`, and `notebooks/`. For +the authoritative module-by-module map (purpose, public surface, dependency +arrows), see [`docs/reference/architecture.md`](docs/reference/architecture.md). ### Running Tests diff --git a/README.md b/README.md index 7323eeb3..e2cb5f0b 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ - **Synthetic Data Pipeline**: Teacher-to-student distillation with `--generate-data` CLI flag - **DeepSpeed & FSDP**: Multi-GPU distributed training with ZeRO-2/3 presets - **MoE Support**: Fine-tune Mixture of Experts models (Qwen3, Mixtral, DeepSeek) -- **GPU Cost Estimation**: Auto-detection for 18 GPU models with per-run cost tracking +- **GPU Cost Estimation**: Auto-detection for 16 GPU models with per-run cost tracking ### Evaluation & Safety - **Automated Benchmarking**: Post-training evaluation via `lm-evaluation-harness` @@ -39,7 +39,7 @@ - **Wizard "audit first"**: when the wizard resolves a JSONL (typed or produced by `forgelm ingest`) it offers to run `forgelm audit` inline and prints the verdict before continuing — closes the BYOD audit loop end-to-end. ### Quickstart Layer (v0.4.5) -- **One-Command Templates**: `forgelm quickstart customer-support` — 4 bundled templates (SFT customer-support, code-assistant, medical-qa-tr, GRPO math-reasoning) plus a bring-your-own-data domain-expert scaffold. Auto-downsizes models on small GPUs. +- **One-Command Templates**: `forgelm quickstart customer-support` — 5 bundled templates (SFT customer-support, code-assistant, medical-qa-tr, domain-expert, GRPO grpo-math). Auto-downsizes models on small GPUs. - **Conservative Defaults**: Every template ships QLoRA 4-bit, rank=8, batch=1, gradient checkpointing on — designed to run on a single 12 GB GPU. - **Wizard Integration**: `forgelm --wizard` opens with "Start from a template?" — same code paths, same YAML schema as a hand-written config. @@ -231,7 +231,7 @@ forgelm --version # Show version ``` forgelm/ -├── cli.py # CLI with 10+ modes (train, dry-run, merge, benchmark, wizard, ingest, audit, ...) +├── cli.py # CLI entry point (train, dry-run, merge, benchmark, wizard, ingest, audit, ...) ├── config.py # Pydantic config (19 models: training, evaluation, distributed, ...) ├── data.py # Dataset loading (SFT, DPO, KTO, GRPO formats + multi-dataset) ├── data_audit.py # Audit pipeline (length / language / dedup / leakage / PII / secrets / quality) — `forgelm audit` diff --git a/docs/guides/safety_compliance-tr.md b/docs/guides/safety_compliance-tr.md new file mode 100644 index 00000000..ebf9c711 --- /dev/null +++ b/docs/guides/safety_compliance-tr.md @@ -0,0 +1,65 @@ +# Güvenlik & Uyumluluk Kılavuzu (TR) + +> **Not:** Bu dosya şu an yalnızca Faz 6'da eklenen `forgelm verify-audit` özelliğinin Türkçe karşılığını içerir. Tam bilingual eşleme (EN ↔ TR) `docs/guides/safety_compliance.md` dosyasının ileride yapılacak çevirisi ile tamamlanacaktır. Daha geniş bağlam için lütfen İngilizce sürüme bakın. + +## Audit Log Bütünlüğü Doğrulama + +ForgeLM'in audit log'u (`audit_log.jsonl`) SHA-256 hash zinciri kullanır; geriye dönük yapılan herhangi bir değişiklik zinciri kırar. Operatör `FORGELM_AUDIT_SECRET` ortam değişkenini ayarlamışsa her satıra ek olarak HMAC etiketi de yazılır — bu sayede log'a yazma erişimi olan ama operatör anahtarını bilmeyen bir saldırgan satır içeriklerini sahteleyemez. + +### `forgelm verify-audit` alt komutu + +`forgelm verify-audit` alt komutu, bir audit log dosyasının SHA-256 zincirini ve (verildiyse) HMAC etiketlerini doğrular: + +```bash +forgelm verify-audit run123/audit_log.jsonl +# OK: 47 entries verified + +FORGELM_AUDIT_SECRET=$OPERATOR_KEY forgelm verify-audit run123/audit_log.jsonl +# OK: 47 entries verified (HMAC validated) + +forgelm verify-audit tampered.jsonl +# FAIL at line 23: chain broken at line 23: prev_hash='...' expected='...' +``` + +Çıkış kodları: + +| Kod | Anlam | +|-----|-------| +| `0` | Zincir (ve denetlendiyse HMAC) eksiksiz | +| `1` | Zincir kırık veya HMAC eşleşmiyor (kurcalama / bozulma) | +| `2` | Dosya bulunamadı, okunamadı veya `--require-hmac` belirtildi ama secret env var ayarlı değil | + +### Kütüphane fonksiyonu + +CI/CD pipeline'larından programatik kullanım için: + +```python +from forgelm.compliance import verify_audit_log, VerifyResult + +result: VerifyResult = verify_audit_log( + "run123/audit_log.jsonl", + hmac_secret=os.environ.get("FORGELM_AUDIT_SECRET"), +) +if not result.valid: + raise SystemExit( + f"Audit log invalid at line {result.first_invalid_index}: {result.reason}" + ) +``` + +`VerifyResult` dataclass alanları: `valid` (bool), `entries_count` (int), `first_invalid_index` (Optional[int], 1-tabanlı), `reason` (Optional[str]). + +### Strict mod (`--require-hmac`) + +Düzenlemeli ortamlarda her satırın HMAC ile imzalı olması zorunlu olabilir. `--require-hmac` bayrağı: + +- Secret env var ayarlı değilse derhal çıkış kodu 2 ile çıkar (operatör hatası). +- Herhangi bir satırda `_hmac` alanı yoksa çıkış kodu 1 ile başarısız olur. + +```bash +FORGELM_AUDIT_SECRET=$OPERATOR_KEY forgelm verify-audit \ + run123/audit_log.jsonl --require-hmac +``` + +### Genesis manifest çapraz kontrolü + +ForgeLM, audit log'un ilk yazılışında `audit_log.jsonl.manifest.json` adlı bir sidecar dosyası üretir. Bu dosya birinci satırın SHA-256 hash'ini sabitler; saldırgan log'u kesip yeni bir genesis satırı yazsa bile manifest eşleşmemesi tespit edilir. `verify-audit` bu sidecar'ı otomatik olarak okur — manifest yoksa zincir-bütünlüğü kontrolü gene yapılır, ancak truncate-and-resume tespit kapsamı daralır (DEBUG seviyesinde uyarı verilir). diff --git a/docs/guides/safety_compliance.md b/docs/guides/safety_compliance.md index e05745c3..385281df 100644 --- a/docs/guides/safety_compliance.md +++ b/docs/guides/safety_compliance.md @@ -282,6 +282,25 @@ ForgeLM's audit log (`audit_log.jsonl`) uses a SHA-256 hash chain for tamper evi Verify integrity by checking that each entry's hash matches the SHA-256 of the previous line. +### Verifying audit log integrity + +The `forgelm verify-audit` subcommand validates the SHA-256 hash chain (and optional HMAC tags) of an audit log: + +```bash +forgelm verify-audit run123/audit_log.jsonl +# OK: 47 entries verified + +FORGELM_AUDIT_SECRET=$OPERATOR_KEY forgelm verify-audit run123/audit_log.jsonl +# OK: 47 entries verified (HMAC validated) + +forgelm verify-audit tampered.jsonl +# FAIL at line 23: chain broken at line 23: prev_hash='...' expected='...' +``` + +Exit codes: `0` valid, `1` invalid chain or HMAC mismatch, `2` file/option error (e.g. `--require-hmac` without a configured secret env var). + +The library function `forgelm.compliance.verify_audit_log(path, *, hmac_secret=None, require_hmac=False)` returns a `VerifyResult` dataclass (`valid`, `entries_count`, `first_invalid_index`, `reason`) for programmatic CI/CD integration. + ### compliance_report.json ```json diff --git a/docs/qms/sop_data_management.md b/docs/qms/sop_data_management.md index f9bc1703..a680f401 100644 --- a/docs/qms/sop_data_management.md +++ b/docs/qms/sop_data_management.md @@ -71,33 +71,38 @@ ForgeLM automated checks: - Dataset fingerprinting (SHA-256 hash, size, timestamp) - Format validation per trainer type (SFT, DPO, KTO, GRPO) - Text cleaning (`clean_text: true`) -- **Audit pipeline (`forgelm audit `, v0.5.1+; legacy - `forgelm --data-audit` still works)** — produces - `data_audit_report.json` with per-split sample counts, length - distribution, top-3 language detection, simhash near-duplicate rate - (or **MinHash LSH** in v0.5.2 via `--dedup-method minhash` for - >50K-row corpora), cross-split leakage check, PII flag counts - (email / phone / Luhn-validated credit card / IBAN / - TR–DE–FR–US national IDs) with **severity tiers** (v0.5.1) and - always-on **secrets/credentials scan** (v0.5.2 — AWS / GitHub / - Slack / OpenAI / Google / JWT / private-key headers). Optional - **heuristic quality filter** (`--quality-filter`, v0.5.2) adds a - `quality_summary` block. The report is auto-inlined into the EU AI - Act Article 10 governance artifact when present in the trainer's - `output_dir` — operators must run the audit **before** training to - keep the bundle self-contained. -- **Ingestion pipeline (`forgelm ingest`, v0.5.0+; v0.5.2 added - markdown-aware splitter + DOCX→markdown table preservation + - `--secrets-mask`)** — turns raw PDF / DOCX / EPUB / TXT / Markdown - into SFT-ready JSONL with optional `--pii-mask` and (v0.5.2) - `--secrets-mask` to redact detected PII / credential spans before - chunks reach storage. +- **ForgeLM audit pipeline (v0.5.0+, `forgelm audit `; legacy + `forgelm --data-audit` deprecated, removal scheduled v0.7.0)** — + produces `data_audit_report.json` with per-split sample counts, + length distribution, top-3 language detection, near-duplicate rate + (default 64-bit simhash, or **MinHash LSH** via + `--dedup-method minhash` for >50K-row corpora), cross-split + leakage check, PII flag counts (email / phone / Luhn-validated + credit card / IBAN / TR–DE–FR–US national IDs) with **severity + tiers**, always-on **secrets/credentials scan** (AWS / GitHub / + Slack / OpenAI / Google / JWT / private-key headers), and an + optional **heuristic quality filter** (`--quality-filter`) that + adds a `quality_summary` block. `forgelm audit --croissant` emits a + Google Croissant 1.0 dataset card alongside the report. The report + is auto-inlined into the EU AI Act Article 10 governance artifact + when present in the trainer's `output_dir` — operators must run + the audit **before** training to keep the bundle self-contained. + Phases 11 + 11.5 + 12 + 12.5 shipped together in the + `v0.5.0` consolidation release (PyPI 2026-04-30); the per-feature + list above describes the v0.5.0 surface. +- **ForgeLM ingestion pipeline (v0.5.0+, `forgelm ingest`)** — turns + raw PDF / DOCX / EPUB / TXT / Markdown into SFT-ready JSONL with + paragraph / sliding / **markdown-aware** chunking, DOCX→markdown + table preservation, optional **Presidio ML-NER** adapter + (`--pii-ml`, `[ingestion-pii-ml]` extra), and + `--pii-mask` / `--secrets-mask` / `--all-mask` shorthands to redact + detected PII / credential spans before chunks reach storage. Manual checks: - [ ] Run `forgelm audit` and review `data_audit_report.json` for: cross-split leakage > 0%, near-duplicate rate, unexpected language mix, - PII flag counts, **`secrets_summary` (v0.5.2 — any non-zero count is - a stop-the-line event; remediate before training)**, and + PII flag counts, **`secrets_summary` (any non-zero count is a + stop-the-line event; remediate before training)**, and `quality_summary` if `--quality-filter` was used. - [ ] Sample review: inspect 50+ random examples. - [ ] Verify label correctness (for preference/KTO data). diff --git a/docs/reference/audit_event_catalog-tr.md b/docs/reference/audit_event_catalog-tr.md new file mode 100644 index 00000000..f2e2e5d6 --- /dev/null +++ b/docs/reference/audit_event_catalog-tr.md @@ -0,0 +1,168 @@ +# Audit Event Kataloğu + +> **Hedef kitle:** EU AI Act Madde 12 kayıt-tutma artefaktlarını inceleyen ForgeLM operatörleri, denetçiler ve aşağı akış doğrulayıcıları. +> **Ayna:** [audit_event_catalog.md](audit_event_catalog.md) + +Bu katalog, ForgeLM'in EU AI Act Madde 12 tarafından zorunlu kılınan, append-only ve hash-zincirli kayıt-tutma artefaktı olan `audit_log.jsonl`'a yazabileceği tüm event'leri sıralar. Her satır ortak bir zarf paylaşır; aşağıdaki satırlardan birini `event` alanı seçer. + +## Ortak zarf + +Her satır, en azından aşağıdaki alanları içeren tek bir JSON nesnesidir: + +| Alan | Tip | Açıklama | +|---------------|---------|-----------------------------------------------------------------------------------------------------------| +| `timestamp` | string | ISO-8601 UTC zaman damgası (`datetime.now(timezone.utc).isoformat()`). | +| `run_id` | string | Eğitim koşusu başına stabil tanımlayıcı (`fg-`). | +| `operator` | string | İnsan-atfedilebilir kimlik. `$FORGELM_OPERATOR` ya da `@` üzerinden. | +| `event` | string | Bu kataloğa ait noktalı event adı. | +| `prev_hash` | string | Önceki satırın SHA-256'sı (ilk girdi için `"genesis"`). Tampering-evident hash zincirini oluşturur. | +| `_hmac` | string? | Sadece `FORGELM_AUDIT_SECRET` set edildiğinde mevcut. `_hmac` olmadan satırın HMAC-SHA-256'sı. | +| _payload_ | değişir | Her satırda ayrı listelenen, event'e özel anahtarlar. | + +Hash zinciri, satır diske düştükten (`flush` + `fsync`) sonra ilerler; kirli bir kapanış zinciri resume için bütün bırakır. + +## Event sözlüğü + +### Pipeline yaşam döngüsü + +| Event | Ne zaman emit edilir | Payload (zarfa ek olarak) | Madde | +|----------------------------|---------------------------------------------------------------------------------|-------------------------------------------------------------------------------------|-------| +| `training.started` | Trainer fine-tuning koşusunu başlatır. | `trainer_type`, `model`, `dataset`, `config_path` | 12 | +| `training.completed` | Fine-tuning başarıyla tamamlanır (değerlendirme kapıları sonrası). | `eval_loss`, `safety_passed`, `judge_score` | 12 | +| `training.failed` | Trainer tamamlanmadan bir hata ile iptal olur. | `failure_reason`, `stage` | 12 | +| `pipeline.completed` | Uçtan uca CLI koşusu (eğitim + değerlendirme + dışa aktarma) 0 koduyla biter. | `exit_code`, `duration_seconds` | 12 | + +### Madde 14 — İnsan Gözetimi + +| Event | Ne zaman emit edilir | Payload | Madde | +|------------------------------|-------------------------------------------------------------------------------------------------------------------|------------------------------------------------------|-------| +| `human_approval.required` | `requires_human_approval: true` işaretli bir kapı pipeline'ı duraklatıp operatör kararını bekler. | `gate`, `reason`, `metrics` | 14 | +| `human_approval.granted` | Operatör duraklatılan kapıyı onayladı. _(Faz 9 — placeholder; henüz emit edilmiyor.)_ | `gate`, `approver`, `comment` | 14 | +| `human_approval.rejected` | Operatör duraklatılan kapıyı reddetti. _(Faz 9 — placeholder; henüz emit edilmiyor.)_ | `gate`, `approver`, `comment` | 14 | + +### Madde 15 — Model Bütünlüğü (auto-revert + güvenlik) + +| Event | Ne zaman emit edilir | Payload | Madde | +|--------------------------------|--------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|-------| +| `model.reverted` | Auto-revert kalite regresyonu sonrası önceki bir checkpoint'i geri yükledi. _(Faz 8 — webhook bağlantılı.)_ | `from_checkpoint`, `to_checkpoint`, `reason`, `metrics_delta` | 15 | +| `audit.classifier_load_failed` | Güvenlik sınıflandırıcısı (örn. Llama Guard) yüklenemedi. Koşu yine `passed=False` kaydeder. | `classifier`, `reason` | 15 | + +### Madde 11 + Ek IV — Uyumluluk artefaktları + +| Event | Ne zaman emit edilir | Payload | Madde | +|----------------------------------|-------------------------------------------------------------------------------|--------------------------------------------------|---------------| +| `compliance.governance_exported` | Madde 10 veri yönetişim raporu diske yazıldı. | `output_path`, `dataset_count` | 10 | +| `compliance.governance_failed` | Yönetişim raporu üretimi iptal edildi (örn. şema uyumsuzluğu). | `failure_reason` | 10 | +| `compliance.artifacts_exported` | Ek IV teknik dokümantasyon paketi (manifest, model card, audit zip) yazıldı. | `output_dir`, `files` | 11, Ek IV | + +### CLI / göç + +| Event | Ne zaman emit edilir | Payload | Madde | +|-----------------------------|--------------------------------------------------------------------------------------------------------|----------------------------------|-------| +| `cli.legacy_flag_invoked` | Deprecated bir CLI flag'i kullanıldı. _(Faz 13 — placeholder; henüz emit edilmiyor.)_ | `flag`, `replacement`, `version` | 12 | + +### Audit-sistem event'leri (meta) + +| Event | Ne zaman emit edilir | Payload | Madde | +|--------------------------------|------------------------------------------------------------------------------------------------------------|--------------------------------------|-------| +| `audit.classifier_load_failed` | _(Yukarıdaki Madde 15 satırına bakın.)_ | `classifier`, `reason` | 15 | +| `audit.cross_run_continuity` | Mevcut bir log dizinine işaret eden ikinci-veya-sonraki AuditLogger örneğinin ilk yazımı. | `previous_chain_head` | 12 | + +## Webhook event'leri + +Webhook payload'ları (Slack / Teams) operatör bildirimlerine kapsamlanmış ayrı bir sözlüktür, regülasyon kaydı değil. _Faz 8'de genişletilecek_ — kanonik liste için bkz. [logging-observability.md](../standards/logging-observability.md). Webhook event'leri `audit_log.jsonl`'a **eklenmez**; yan-kanal bildirim bus'ı üzerinde gider. + +## Yeni bir event eklemek + +1. Mevcut isim alanlarını (`training.*`, `compliance.*`, `audit.*`, `human_approval.*`, `model.*`, `cli.*`) takip eden noktalı bir ad seçin. +2. Yukarıdaki tabloya, payload anahtarları ve desteklediği Madde dahil olmak üzere bir satır ekleyin. +3. Satırı [audit_event_catalog.md](audit_event_catalog.md)'ye yansıtın. +4. `AuditLogger.log_event(event, **payload)` üzerinden emit edin. `audit_log.jsonl`'a doğrudan `json.dump` çağırmayın; hash zinciri kanonik yazıcıya bağımlıdır. + +## Tampering-evidence özeti + +| Mekanizma | Şuna karşı koruma sağlar | Her zaman açık mı? | +|--------------------------|-----------------------------------------------------------------------------------|---------------------------------------------------| +| SHA-256 hash zinciri | Tek-satır düzenlemeler, silmeler, sıralama değişiklikleri. | Evet. | +| Genesis manifest sidecar | Tüm log'un "genesis"'e geri kesilmesi. | Evet (ilk event'te bir kez yazılır). | +| `flock(LOCK_EX)` | Aynı dizini paylaşan eşzamanlı trainer'lardan iç içe yazımlar. | Evet (Unix); Windows'ta no-op. | +| `flush` + `fsync` | Buffer yazımı ile zincir ilerleme arasında güç-kesme / kernel-panic kaybı. | Evet. | +| Satır başına HMAC-SHA-256| Log yeniden yazımı sonrası sahte yeniden imzalama. | Sadece `FORGELM_AUDIT_SECRET` set olduğunda. | +## Webhook olayları + +Bu beş yaşam döngüsü olayı, webhook alıcılarının (Slack, Teams, jenerik +HTTP) `WebhookNotifier`'dan beklemesi gereken **tek** olaylardır. Her +biri, karşılık gelen bir denetim günlüğü olayını yansıtır; böylece +aşağı akıştaki bir operatör webhook ping → denetim girdisi +korelasyonunu `run_name` + zaman damgasıyla kurabilir. Uygulama: +`forgelm/webhook.py`. + +| Webhook `event` | Denetim günlüğü karşılığı | Tetikleyici | Kapı (gate) | Zorunlu payload alanları | +|---|---|---|---|---| +| `training.start` | `training.started` | `train()` çağrıldı, model yüklenmeden önce. | `webhook.notify_on_start` | `run_name`, `status="started"` | +| `training.success` | `pipeline.completed` | Tüm kapılar geçildi, insan onayı gerekmiyor. | `webhook.notify_on_success` | `run_name`, `status="succeeded"`, `metrics` | +| `training.failure` | `pipeline.failed` | Eğitim sürecinin kendisi hata fırlattı (OOM, veri seti hatası, yakalanmayan istisna). | `webhook.notify_on_failure` | `run_name`, `status="failed"`, `reason` (maskelenmiş, ≤2048 karakter) | +| `training.reverted` | `model.reverted` | Eğitim sonrası bir kapı (değerlendirme, güvenlik, hakem, benchmark) çalışmayı reddetti ve `_revert_model` adaptörleri sildi. | `webhook.notify_on_failure` | `run_name`, `status="reverted"`, `reason` (maskelenmiş, ≤2048 karakter) | +| `approval.required` | `human_approval.required` | Çalışma başarılı oldu, `evaluation.require_human_approval=true`, model insan incelemesi için staging'de (EU AI Act Madde 14). | `webhook.notify_on_success` | `run_name`, `status="awaiting_approval"`, `model_path` | + +### Bu yaşam döngüsü durumlarından ikisinin neden ayrıldığı + +- **`training.failure` vs `training.reverted`** — dashboard'ların + "trainer çöktü" ile "trainer başarılı oldu fakat kalite / güvenlik / + hakem kapısı reddetti" durumlarını ayırt etmesi gerekir. Her ikisi + de operasyonel olarak eyleme dönüşürdür, ancak farklı runbook'lar + gerektirir. Faz 8, bir Slack kanalının iki vakayı farklı renk + kodlayabilmesi için (`#ff0000` ve `#ff9900`) tam da bu nedenle + `notify_reverted`'i tanıttı. +- **`approval.required`** — çalışma başarılı olduktan *sonra*, fakat + operatör dağıtımı onaylamadan *önce* yayılır. Bu bir başarısızlık + değil; bir duraklamadır. `training.failure` üzerinde otomatik + çağrı yapan alıcılar `approval.required` üzerinde çağrı yapmamalı. + +### Payload şeması + +Her webhook olayı aynı zarfı taşır: + +```json +{ + "event": "training.start | training.success | training.failure | training.reverted | approval.required", + "run_name": "", + "status": "started | succeeded | failed | reverted | awaiting_approval", + "metrics": {"": , ...}, + "reason": "", + "model_path": "", + "attachments": [{"title": "...", "text": "...", "color": "..."}] +} +``` + +`metrics`, `reason` ve `model_path` her zaman şemada bulunur; yalnızca +ihtiyaç duyan olaylarda doldurulur. `attachments`, Slack uyumlu blok'tur +— diğer alıcılar görmezden gelebilir. + +### Güvenlik garantileri + +1. **Sebepler maskelenir.** Her `reason` alanı, taşıma öncesi + `forgelm.data_audit.mask_secrets` üzerinden geçer; böylece AWS / + GitHub / Slack / OpenAI / Google / JWT / özel-anahtar blokları / + Azure storage dizeleri süreçten dışarı çıkmaz. `data_audit` ithal + edilemezse, ham dize yerine alan + `"[REDACTED — secrets masker unavailable]"` ile değiştirilir. +2. **Sebepler 2048 karaktere kırpılır.** Bundan uzun stack trace'ler + `"… (truncated)"` ile kesilir. +3. **Model ağırlıkları yok.** `approval.required` yalnızca staging + dosya sistemi yolunu taşır. Ağırlıklar diskte kalır; o dizini zaten + operatör kontrol eder. +4. **Webhook URL'si sızdırılmaz.** URL'ler günlüklerde maskelenir + (`scheme://host//...`) ve 2xx olmayan yanıt gövdesi + bastırılır. +5. **SSRF koruması.** `webhook.allow_private_destinations=true` + ayarlanmadığı sürece özel / loopback / link-local hedefler + reddedilir. + +### Saklama rehberi + +Webhook payload'ları **geçicidir**. Denetim kaydı değildir. Uzun süreli +geçmişe ihtiyaç duyan alıcılar, webhook trafiğini arşivlemek yerine +denetim JSONL dosyasının (`/compliance/audit.jsonl`) anlık +görüntüsünü almalıdır; çünkü denetim günlüğü yalnız-eklenir +hash-zincirli kayıttır ve webhook akışı en-iyi-çabadır (best-effort). diff --git a/docs/reference/audit_event_catalog.md b/docs/reference/audit_event_catalog.md new file mode 100644 index 00000000..f969220a --- /dev/null +++ b/docs/reference/audit_event_catalog.md @@ -0,0 +1,165 @@ +# Audit Event Catalog + +> **Audience:** ForgeLM operators, regulators, and downstream verifiers reviewing the EU AI Act Article 12 record-keeping artefacts. +> **Mirror:** [audit_event_catalog-tr.md](audit_event_catalog-tr.md) + +This catalog enumerates every event ForgeLM may append to `audit_log.jsonl`, the append-only, hash-chained record-keeping artefact mandated by EU AI Act Article 12. Every entry shares a common envelope; the `event` field selects one of the rows below. + +## Common envelope + +Each line is a single JSON object with at least the following fields: + +| Field | Type | Description | +|---------------|---------|-----------------------------------------------------------------------------------------------------------| +| `timestamp` | string | ISO-8601 UTC timestamp (`datetime.now(timezone.utc).isoformat()`). | +| `run_id` | string | Stable per-training-run identifier (`fg-`). | +| `operator` | string | Human-attributable identity. From `$FORGELM_OPERATOR`, else `@`. | +| `event` | string | Dotted event name from this catalog. | +| `prev_hash` | string | SHA-256 of the previous line (`"genesis"` for the first entry). Forms the tamper-evident hash chain. | +| `_hmac` | string? | Present only when `FORGELM_AUDIT_SECRET` is set. HMAC-SHA-256 of the line without `_hmac`. | +| _payload_ | varies | Event-specific keys, listed per row below. | + +The hash chain advances after the line lands on disk (`flush` + `fsync`), so an unclean shutdown leaves the chain intact for resume. + +## Event vocabulary + +### Pipeline lifecycle + +| Event | When emitted | Payload (in addition to envelope) | Article | +|----------------------------|---------------------------------------------------------------------------|------------------------------------------------------------------------------------------|---------| +| `training.started` | Trainer begins a fine-tuning run. | `trainer_type`, `model`, `dataset`, `config_path` | 12 | +| `training.completed` | Fine-tuning finishes successfully (post evaluation gates). | `eval_loss`, `safety_passed`, `judge_score` | 12 | +| `training.failed` | Trainer aborted with an error before completion. | `failure_reason`, `stage` | 12 | +| `pipeline.completed` | End-to-end CLI run (training + evaluation + export) returned exit code 0. | `exit_code`, `duration_seconds` | 12 | + +### Article 14 — Human Oversight + +| Event | When emitted | Payload | Article | +|------------------------------|---------------------------------------------------------------------------------------------------------|------------------------------------------------------|---------| +| `human_approval.required` | A gate marked `requires_human_approval: true` paused the pipeline awaiting an operator decision. | `gate`, `reason`, `metrics` | 14 | +| `human_approval.granted` | Operator approved the paused gate. _(Faz 9 — placeholder; not yet emitted.)_ | `gate`, `approver`, `comment` | 14 | +| `human_approval.rejected` | Operator rejected the paused gate. _(Faz 9 — placeholder; not yet emitted.)_ | `gate`, `approver`, `comment` | 14 | + +### Article 15 — Model Integrity (auto-revert + safety) + +| Event | When emitted | Payload | Article | +|--------------------------------|-----------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|---------| +| `model.reverted` | Auto-revert restored a previous checkpoint after a quality regression. _(Faz 8 — webhook-coupled.)_ | `from_checkpoint`, `to_checkpoint`, `reason`, `metrics_delta` | 15 | +| `audit.classifier_load_failed` | Safety classifier (e.g., Llama Guard) failed to load. The run still records `passed=False`. | `classifier`, `reason` | 15 | + +### Article 11 + Annex IV — Compliance artefacts + +| Event | When emitted | Payload | Article | +|----------------------------------|-----------------------------------------------------------------------------|--------------------------------------------------|------------| +| `compliance.governance_exported` | Article 10 data governance report written to disk. | `output_path`, `dataset_count` | 10 | +| `compliance.governance_failed` | Governance report generation aborted (e.g., schema mismatch). | `failure_reason` | 10 | +| `compliance.artifacts_exported` | Annex IV technical documentation bundle (manifest, model card, audit zip). | `output_dir`, `files` | 11, Annex IV | + +### CLI / migration + +| Event | When emitted | Payload | Article | +|-----------------------------|----------------------------------------------------------------------------------------------------|----------------------------------|---------| +| `cli.legacy_flag_invoked` | A deprecated CLI flag was used. _(Faz 13 — placeholder; not yet emitted.)_ | `flag`, `replacement`, `version` | 12 | + +### Audit-system events (meta) + +| Event | When emitted | Payload | Article | +|--------------------------------|---------------------------------------------------------------------------------------------------|--------------------------------------|---------| +| `audit.classifier_load_failed` | _(See Article 15 row above.)_ | `classifier`, `reason` | 15 | +| `audit.cross_run_continuity` | First write of a second-or-later AuditLogger instance pointing at an existing log directory. | `previous_chain_head` | 12 | + +## Webhook events + +Webhook payloads (Slack / Teams) are a separate vocabulary scoped to operator notifications, not the regulatory record. They are _extended in Faz 8_ — see [logging-observability.md](../standards/logging-observability.md) for the canonical list. Webhook events are **not** appended to `audit_log.jsonl`; they ride the side-channel notification bus. + +## Adding a new event + +1. Pick a dotted name following the existing namespaces (`training.*`, `compliance.*`, `audit.*`, `human_approval.*`, `model.*`, `cli.*`). +2. Add a row to the table above, including the payload keys and the Article it supports. +3. Mirror the row to [audit_event_catalog-tr.md](audit_event_catalog-tr.md). +4. Emit via `AuditLogger.log_event(event, **payload)`. Never call `json.dump` directly into `audit_log.jsonl`; the hash chain depends on the canonical writer. + +## Tamper-evidence summary + +| Mechanism | Defends against | Always on? | +|--------------------------|-----------------------------------------------------------------------|------------------------------------------------------| +| SHA-256 hash chain | Single-line edits, deletions, reorderings. | Yes. | +| Genesis manifest sidecar | Whole-log truncation back to "genesis". | Yes (written once on first event). | +| `flock(LOCK_EX)` | Interleaved writes from concurrent trainers sharing the directory. | Yes (Unix); no-op on Windows. | +| `flush` + `fsync` | Power-cut / kernel-panic loss between buffer write and chain advance. | Yes. | +| HMAC-SHA-256 per line | Forged re-signing after log rewrite. | Only when `FORGELM_AUDIT_SECRET` is set. | + +## Webhook events + +These five lifecycle events are the **only** events that webhook +receivers (Slack, Teams, generic HTTP) should expect from +`WebhookNotifier`. Each one mirrors a corresponding audit-log event so +a downstream operator can correlate webhook ping → audit entry by +`run_name` + timestamp. Implementation: `forgelm/webhook.py`. + +| Webhook `event` | Audit-log mirror | Trigger | Gated by | Required payload fields | +|---|---|---|---|---| +| `training.start` | `training.started` | `train()` entered, before model load. | `webhook.notify_on_start` | `run_name`, `status="started"` | +| `training.success` | `pipeline.completed` | All gates passed, no human-approval requirement. | `webhook.notify_on_success` | `run_name`, `status="succeeded"`, `metrics` | +| `training.failure` | `pipeline.failed` | Training itself raised (OOM, dataset error, unhandled exception). | `webhook.notify_on_failure` | `run_name`, `status="failed"`, `reason` (masked, ≤2048 chars) | +| `training.reverted` | `model.reverted` | A post-training gate (evaluation, safety, judge, benchmark) rejected the run and `_revert_model` deleted the adapters. | `webhook.notify_on_failure` | `run_name`, `status="reverted"`, `reason` (masked, ≤2048 chars) | +| `approval.required` | `human_approval.required` | Run succeeded, `evaluation.require_human_approval=true`, model staged for review (EU AI Act Art. 14). | `webhook.notify_on_success` | `run_name`, `status="awaiting_approval"`, `model_path` | + +### Why two of these split lifecycle states + +- **`training.failure` vs `training.reverted`** — dashboards need to tell + "the trainer crashed" apart from "the trainer succeeded, but the + quality / safety / judge gate said no". Both are operationally + actionable, but they require different runbooks. Faz 8 introduced + `notify_reverted` precisely so a Slack channel can colour-code the two + cases differently (`#ff0000` vs `#ff9900`). +- **`approval.required`** — fires *after* the run succeeded but *before* + the operator has approved deploy. It is not a failure; it is a pause. + Receivers that auto-page on `training.failure` should **not** page on + `approval.required`. + +### Payload schema + +Every webhook event ships the same envelope: + +```json +{ + "event": "training.start | training.success | training.failure | training.reverted | approval.required", + "run_name": "", + "status": "started | succeeded | failed | reverted | awaiting_approval", + "metrics": {"": , ...}, + "reason": "", + "model_path": "", + "attachments": [{"title": "...", "text": "...", "color": "..."}] +} +``` + +`metrics`, `reason`, and `model_path` are always present in the schema; +only populated for the events that need them. `attachments` is the +Slack-compatible block — other receivers may ignore it. + +### Security guarantees + +1. **Reasons are masked.** Every `reason` field passes through + `forgelm.data_audit.mask_secrets` before transport, so AWS / GitHub / + Slack / OpenAI / Google / JWT / private-key blocks / Azure storage + strings do not leave the process. If `data_audit` cannot be imported, + the field is replaced with `"[REDACTED — secrets masker unavailable]"` + rather than the raw string. +2. **Reasons are truncated to 2048 chars.** Stack traces longer than + that are clipped with `"… (truncated)"`. +3. **No model weights.** `approval.required` carries the staging + filesystem path only. Weights stay on disk; the operator already + controls that directory. +4. **No webhook URL leakage.** URLs are masked in logs (`scheme://host//...`) + and the response body is suppressed on non-2xx. +5. **SSRF guard.** Private / loopback / link-local destinations are + refused unless `webhook.allow_private_destinations=true`. + +### Retention guidance + +Webhook payloads are **transient**. They are not the audit record. +Receivers that need long-term history should snapshot the audit JSONL +(`/compliance/audit.jsonl`) rather than archiving webhook +traffic, because the audit log is the append-only hash-chained record +and the webhook stream is best-effort. diff --git a/docs/roadmap-tr.md b/docs/roadmap-tr.md index 6ecc748c..3cd54985 100644 --- a/docs/roadmap-tr.md +++ b/docs/roadmap-tr.md @@ -9,11 +9,13 @@ | ✅ Tamam | [Faz 1-9](roadmap/completed-phases.md) | SOTA iyileştirmeleri, değerlendirme, güvenilirlik, kurumsal entegrasyon, ekosistem, hizalama stack'i, güvenlik, EU AI Act uyumluluğu (Madde 9-17 + Ek IV), gelişmiş güvenlik zekası | | ✅ Tamam | [Faz 10 — Post-Training Tamamlama](roadmap/phase-10-post-training.md) | `inference.py`, `chat`, `export` (GGUF), `--fit-check`, `deploy` — `v0.4.0` | | ✅ Tamam | [Faz 10.5 — Quickstart Katmanı ve Onboarding](roadmap/phase-10-5-quickstart.md) | `forgelm quickstart