diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9bface50..3e547232 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -161,6 +161,11 @@ jobs: steps: - uses: actions/checkout@v5 + - uses: actions/setup-node@v7 + with: + node-version: "24" + package-manager-cache: false + - uses: actions/setup-python@v6 with: python-version: "3.11" @@ -181,6 +186,9 @@ jobs: - name: Import-origin guard run: python3 tools/check_import_origin.py --strict + - name: Browser wizard regression tests + run: node --test tests/site/*.test.cjs + - name: CLI smoke checks run: | forgelm --version diff --git a/AGENTS.md b/AGENTS.md index 629b12a7..d3b0470b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -166,6 +166,7 @@ Default workflow for a non-trivial change: ```bash python3 tools/check_import_origin.py --strict && \ ruff format . && ruff check . && pytest tests/ && \ + node --test tests/site/*.test.cjs && \ python3 -m forgelm --config config_template.yaml --dry-run && \ python3 -m mypy --strict --follow-imports=silent forgelm/__init__.py forgelm/_version.py tests/typing/public_surface_probe.py && \ python3 tools/check_field_descriptions.py --strict forgelm/config.py && \ diff --git a/CHANGELOG.md b/CHANGELOG.md index febf460c..818f6667 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,8 +4,36 @@ All notable changes to ForgeLM are documented here. ## [Unreleased] +### Breaking + +- Tighten `monitoring.endpoint_env` to trimmed ASCII identifiers; URLs, `$`/`env:` prefixes, + internal whitespace, and non-ASCII names now fail validation. Blank names remain absent. + This changes acceptance of previously valid YAML; use a variable name, such as `MONITORING_PLAN_URL`. +- Require non-blank `monitoring.endpoint` literals to be absolute HTTP(S) URLs with a host + and valid port; schemeless addresses are rejected instead of silently becoming `` + in evidence. Use `https://monitor.example.com:8443/plan` or an `endpoint_env` name (#93). + ### Changed +- Record monitoring default provenance in `defaulted_fields` and mark literal URL projection with + `endpoint_redacted` (also true when the URL already contains only scheme/host/port). Literal + endpoints retain scheme/host/port in JSON evidence, including IPv6 brackets; userinfo, path, + query and fragment are removed. Env references remain unresolved. Absent endpoints record `null` + instead of the legacy `"$None"`, and non-empty `endpoint_env` takes precedence. + Monitoring metadata is exported to `compliance_report.json` and, when emitted, + `annex_iv_metadata.json`; the flattened `training_manifest.yaml` summary does not contain it. + Monitoring declarations remain optional in ForgeLM at every risk tier, including strict tiers; + providers’ Article 72 obligations apply independently. Treat legacy monitoring blocks without + `declarative_only` as declarations; their default provenance was not recorded (#93). +- Default CLI/browser wizard drift-alert answers to No / unchecked; only declare alerts when + the operator confirms them. Preserve the schema's `true` default with `defaulted_fields` + provenance. Both wizards explicitly record the selected `none` exporter (#93). +- Accept case-insensitive `env:` references with surrounding whitespace, lowercase letters and + initial underscores in both CLI wizard monitoring and webhook inputs; reject malformed names. +- Omit Pydantic `input_value` / `input_type` excerpts from all rendered `ForgeConfig` validation + errors, preserving field paths and reasons. `MonitoringConfig` also applies this protection + standalone; other standalone submodels retain their existing diagnostics (#93). + - Redesign the root README around a GPU-free first run, a concise capability overview, optional evidence controls, and clear contribution and documentation paths. Reuse the site's ForgeLM mark and distinguish the Apache-licensed code from CC-BY-SA seed datasets. @@ -15,6 +43,15 @@ All notable changes to ForgeLM are documented here. ### Fixed +- Restore the monitoring-plan evidence contract: `monitoring.*` declares an external plan, + labelled `declarative_only: true`; ForgeLM runs no exporter, drift detector, alert, or scheduler. + Warn once per config-file load for an enabled exporter declaration, without repeating during + pipeline stage validation. Align CLI/web wizards and EN/TR guidance, correct Annex IV monitoring + point 9 / risk-management point 5 labels, and verify exported declarations end to end (#93). +- Correct webhook troubleshooting in the site and EN/TR manuals: delivery failures appear as + `forgelm.webhook` warnings, never `webhook_failed` audit events; use the actual + `allow_private_destinations` setting (#231). Clarify that `--offline` sets HF offline flags; + reporting, API judges, and webhooks require separate disconnected-run configuration (refs #100). - Rework the homepage terminal into a first-run guide with separately copyable installation and config-generation commands, an optional training step, and readable code in both themes and on narrow screens, with interface copy in all six site languages. Keep the dated CPU recording and diff --git a/CLAUDE.md b/CLAUDE.md index e293ab9c..67197574 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -166,6 +166,7 @@ Default workflow for a non-trivial change: ```bash python3 tools/check_import_origin.py --strict && \ ruff format . && ruff check . && pytest tests/ && \ + node --test tests/site/*.test.cjs && \ python3 -m forgelm --config config_template.yaml --dry-run && \ python3 -m mypy --strict --follow-imports=silent forgelm/__init__.py forgelm/_version.py tests/typing/public_surface_probe.py && \ python3 tools/check_field_descriptions.py --strict forgelm/config.py && \ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 900a46a0..dbf0da6a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -165,6 +165,7 @@ supported Python matrix: ```bash python3 tools/check_import_origin.py --strict && \ ruff format . && ruff check . && pytest tests/ && \ + node --test tests/site/*.test.cjs && \ python3 -m forgelm --config config_template.yaml --dry-run && \ python3 -m mypy --strict --follow-imports=silent forgelm/__init__.py forgelm/_version.py tests/typing/public_surface_probe.py && \ python3 tools/check_field_descriptions.py --strict forgelm/config.py && \ diff --git a/config_template.yaml b/config_template.yaml index ce08c2d6..8aec1398 100644 --- a/config_template.yaml +++ b/config_template.yaml @@ -256,11 +256,15 @@ data: # - "Human review required for policy-related responses" # vulnerable_groups_considered: true -# Post-Market Monitoring (Art. 12+17) +# External post-market monitoring plan (Art. 72 / 17(1)(h)) — declarative metadata only. +# Recorded in the compliance manifest / Annex IV; ForgeLM does not contact the endpoint, +# export metrics, detect drift, send drift alerts, or schedule checks. Operate your own stack. +# Uncommented values are explicit declarations about your external stack; review them before use. # monitoring: # enabled: true -# endpoint_env: "MONITORING_WEBHOOK_URL" -# metrics_export: "prometheus" # "none", "prometheus", "datadog", "custom_webhook" +# endpoint: "" # optional literal URL; evidence keeps scheme/host/port only; must be http(s)://host +# endpoint_env: "MONITORING_PLAN_URL" # recorded as "$NAME", never read; overrides endpoint +# metrics_export: "none" # other exporters emit a metadata-only warning at config load # alert_on_drift: true # check_interval_hours: 24 diff --git a/docs/design/iso27001_soc2_alignment.md b/docs/design/iso27001_soc2_alignment.md index 49483942..19003e2f 100644 --- a/docs/design/iso27001_soc2_alignment.md +++ b/docs/design/iso27001_soc2_alignment.md @@ -157,7 +157,7 @@ Each row classifies coverage as one of: | A.5.5 Contact with authorities | OOS | — | Establish regulator contact for serious-incident reporting (EU AI Act Art. 73) | ForgeLM cannot interface with regulators | | A.5.6 Contact with special interest groups | OOS | — | ML safety / red-team community engagement | Out of scope | | A.5.7 Threat intelligence | OOS | — | Subscribe to ML supply-chain CVE feeds, model-poisoning advisories | Out of scope | -| A.5.8 Information security in project management | FL-helps | `risk_assessment` config block; F-compliance-110 strict gate; Annex IV §9 metadata | Embed ForgeLM dry-run + safety-eval gates in project sign-off checklist | Project methodology deployer-side | +| A.5.8 Information security in project management | FL-helps | `risk_assessment` config block; F-compliance-110 strict gate; Annex IV point 5 metadata | Embed ForgeLM dry-run + safety-eval gates in project sign-off checklist | Project methodology deployer-side | | A.5.9 Inventory of information & associated assets | FL-helps | `data_provenance.json`, `model_integrity.json`, SBOM | Maintain corporate asset register linking model SHA + data fingerprint | Org asset registry deployer-side | | A.5.10 Acceptable use of information | OOS | — | Acceptable-use policy for trained-model outputs | Out of scope | | A.5.11 Return of assets | OOS | — | Decommissioning of deployed models | Out of scope | @@ -244,7 +244,7 @@ ForgeLM contributes nothing here. | A.8.13 Information backup | FL-helps | `audit_log.jsonl` + `.manifest.json` form a backupable unit; Annex IV bundle (`forgelm --compliance-export`) | Off-site backup with encryption | | A.8.14 Redundancy of information processing facilities | OOS | — | Multi-AZ infra | | A.8.15 Logging | FL | `audit_log.jsonl` JSON Lines + HMAC chain + genesis manifest; `forgelm verify-audit` validates | Ship to SIEM | -| A.8.16 Monitoring activities | FL-helps | Webhook lifecycle events; `safety_trend.jsonl` cross-run trend tracking | Define alert thresholds | +| A.8.16 Monitoring activities | FL-helps | Webhook lifecycle notifications and `safety_trend.jsonl` evaluation trends; external monitoring-plan metadata only | Operate monitoring and define alert thresholds | | A.8.17 Clock synchronisation | FL-helps | All audit entries ISO-8601 UTC | NTP on training hosts | | A.8.18 Use of privileged utility programs | OOS | — | OS-level privilege management | | A.8.19 Installation of software on operational systems | FL-helps | `forgelm doctor` reports installed packages + versions; `pyproject.toml` pins; `pip install forgelm==X.Y.Z` | Package allowlist | diff --git a/docs/guides/quickstart-tr.md b/docs/guides/quickstart-tr.md index ed185a57..43b6ab2a 100644 --- a/docs/guides/quickstart-tr.md +++ b/docs/guides/quickstart-tr.md @@ -72,7 +72,7 @@ Bundled seed dataset'lerin lisansları için bkz. [LICENSES.md](https://github.c forgelm --wizard ``` -Sihirbaz önce curated quickstart-template kısayolu önerir; reddedilirse 9 adımlı etkileşimli akış açılır (welcome / use-case / model / strategy / trainer / dataset / training-params / compliance / operations) ve her `ForgeConfig` bloğunu kapsar — model, LoRA / DoRA / PiSSA / rsLoRA / GaLore stratejisi, trainer-spesifik hyperparam'lar (`dpo_beta` / `simpo_*` / `kto_beta` / `orpo_beta` / `grpo_*`), EU AI Act Madde 9 / 10 / 11 / 12+17 uyumluluk metadata, retention, monitoring, evaluation kapıları, webhook, sentetik veri — ve kullanıma-hazır bir YAML yazar. Geri dönmek için `back` / `b`, sıfırlamak için `reset` / `r`; state `~/.cache/forgelm/wizard_state.yaml`'da persistent, Ctrl-C / yeni oturum kaldığı yerden devam edebilir. +Sihirbaz önce curated quickstart-template kısayolu önerir; reddedilirse 9 adımlı etkileşimli akış açılır (welcome / use-case / model / strategy / trainer / dataset / training-params / compliance / operations) ve her `ForgeConfig` bloğunu kapsar — model, LoRA / DoRA / PiSSA / rsLoRA / GaLore stratejisi, trainer-spesifik hyperparam'lar (`dpo_beta` / `simpo_*` / `kto_beta` / `orpo_beta` / `grpo_*`), EU AI Act Madde 9 / 10 / 11 / 12 / 17(1)(h) / 72 uyumluluk metadata, retention, monitoring, evaluation kapıları, webhook, sentetik veri — ve kullanıma-hazır bir YAML yazar. Geri dönmek için `back` / `b`, sıfırlamak için `reset` / `r`; state `~/.cache/forgelm/wizard_state.yaml`'da persistent, Ctrl-C / yeni oturum kaldığı yerden devam edebilir. Review-cycle 2 (2026-05-09) ile eklenen operatör guardrail'ları: sihirbaz çıkmadan önce kayıtlı YAML'ı `ForgeConfig.model_validate` ile doğrular (schema reddetmeleri 30 dakika sonra eğitim sırasında değil, anında yüzeylenir), mevcut config dosyasını üzerine yazmadan önce sorar (reddederseniz `_2.yaml` / `_3.yaml` ile auto-suffix'ler), non-tty stdin altında çalışmayı reddeder (scriptli koşumlar için `forgelm quickstart